ToMaR HDFS Input Directory Processing

This workflow allows processing an HDFS input directory using ToMaR.

The "hdfs_working_dir" input port is the HDFS input directory which containes the data to be processed by ToMaR.

The "toolspec" input port contains the toolspec XML describing operations that can be used (see "operation" input port).

The "operation" input port defines the operation to be used in the current ToMaR job execution (see "toolspec" input port, an operation port used here must be defined in the tool specification).

The "hdfs_working_dir" input port defines the directory where the outputs will be stored in a date/time-subdirectory.

For example:

tomarworkingdir/20140304130007/dataout tomarworkingdir/20140304130007/joboutput tomarworkingdir/20140304130007/tomar-controlfile.txt tomarworkingdir/20140304130007/toolspec

The "dataout" directory contains the output data of the ToMaR process. Depending on the operation used, this can be the result of a file format identification or a data migration process. The "joboutput" directory contains the Hadoop job output of the ToMaR Hadoop job. The "tomar-controlfile.txt" file is the input file for the ToMaR Hadoop job execution. The "toolspec" directory contains the tool specification file given by the "toolspec" input port.