ONB Web Archive Fits Characterisation using ToMaR

Hadoop based workflow for applying FITS on the files contained in ARC web archive container files and ingest the FITS output in a MongoDB using C3PO.

Dependencies:

- Spacip (https://github.com/shsdev/spacip) - Tomar (https://github.com/openplanets/tomar) - C3PO (https://github.com/peshkira/c3po)

Parameters:

- hdfs_input_path: Path to a directory which contains textfile(s) with absolute HDFS paths to ARC files - num_files_per_invokation: Number of items to be processed per invokation - fits_local_tmp_dir: Local directory where the FITS output XML files will be stored - c3po_collection_name: Name of the C3P0 collection

The workflow uses Spacip to unpackage the ARC container files into HDFS and creating input files which can be used by ToMaR. After merging the mapper output files from Spacip (MergeTomarInput) into one single file, the FITS characterisation process is invoked by ToMaR as a MapReduce job. The tool invokation depends on a tool specification file which must be available in HDFS, this is explained in the Tomar documentation.