ONB Web Archive Fits Characterisation using ToMaR

Created: 2013-12-09 15:58:54 Last updated: 2013-12-10 17:06:09

Download Workflow

Workflow for Web Archive Content Characterisation using Fits.

The workflow uses Spacip (https://github.com/shsdev/spacip) to prepare ARC web is a tool to prepare ARC web archive container files by unpacking the compressed files in HDFS and creates input files which can be used by Tomar (https://github.com/openplanets/tomar). After merging the mapper output files from Spacip (MergeTomarInput) into one single file, it can be used by Tomar as input. The tool invokation depends on a tool specification file which must be available in HDFS, this is explained in the Tomar documentation.

Preview

Download as scalable diagram (SVG)

Run

Run this Workflow in the Taverna Workbench...

Option 1:

Copy and paste this link into File > 'Open workflow location...'
http://www.myexperiment.org/workflows/3933/download?version=1
[ More Info Expand ]

Workflow Components

Authors (1)

Titles (1)

Descriptions (1)

Dependencies (0)

Inputs (2)

Name	Description
hdfs_input_path	HDFS input path
num_files_per_invokation	Number of files per invokation

Processors (4)

Name	Type	Description
Spacip	externaltool
MergeTomarInput	externaltool
Tomar	externaltool
toolspecs_hdfs_dir_value	stringconstant	Value /user/scape/scape-toolspecs

Beanshells (0)

Outputs (1)

Name	Description
Tomar_STDOUT

Datalinks (6)

Source	Sink
hdfs_input_path	Spacip:hdfs_input_path
num_files_per_invokation	Spacip:num_files_per_invokation
Spacip:STDOUT	MergeTomarInput:spacip_joboutput_hdfs_dir
toolspecs_hdfs_dir_value:value	Tomar:toolspecs_hdfs_dir
MergeTomarInput:STDOUT	Tomar:merged_tomar_input
Tomar:STDOUT	Tomar_STDOUT