Concept Profile Generation Pipeline

Created: 2014-04-30 19:14:51 Last updated: 2014-06-15 17:55:11

Download Workflow

Requirements:

Have MySQL installed with a database called mydb or please create one. Have a mysql connector file. This can be downloaded from mysql.com. Rename the downloaded file to mysql-connector-java.jar and place it in the the local lib file of Taverna as written on the dependencies tab. Choose this dependency by rightclicking a beanshell script and selecting it for the whole workflow as system classloader.

Download the Peregrine SKOS CLI from: https://trac.nbic.nl/biosemantics/downloads Install LVG2013Lite. See: https://trac.nbic.nl/biosemantics/wiki/Peregrine%20SKOS%20CLI

Change lvg.properties LVG_DIR=/home/path/to/lvg2013lite/ This file can be found in:

/home/path/to/lvg2013lite/data/config/lvg.properties

Copy properties file from production.properties which can be found on: https://trac.nbic.nl/biosemantics/wiki/Peregrine%20SKOS%20CLI

Change the normalizer.lvg.properties and normalizer.lvg.binaryCache properties to point to the LVG installation path. This file can be obtained from https://trac.nbic.nl/biosemantics/wiki/Peregrine%20SKOS%20CLI

Have Peregrine indexer installed locally and give it the correct path in the component named "Indexer_tool". Which can be found in the nested workflow named Peregrine indexer.

If using a Windows machine please install cygwin. Set cygwin in the $PATH of the environment variabale. Go to taverna File --> preferences --> Tool invocation --> set modify on explicit locations --> edit a default local to Shell: C:\Windows\system32\cmd.exe /c and save that. -->Set Modify on: symbolic locations --> edit --> choose default local and save.

Have JAVA_HOME C:\Program Files\jdk 1.7.0 in environment variabale when using Windows.

The nested workflow Get_abstract uses a sql command to retrieve abstracts which presumes that it uses the mydb database. Please change this when using a other sql database.

Preview

Download as scalable diagram (SVG)

Run

Run this Workflow in the Taverna Workbench...

Option 1:

Copy and paste this link into File > 'Open workflow location...'
http://myexperiment.org/workflows/4272/download?version=5
[ More Info Expand ]

Workflow Components

Authors (1)

Titles (1)

Descriptions (1)

Dependencies (1)

Inputs (2)

Name	Description
Ontology_input	A Skos formated dictionary/thesaurus of predefined concepts.
db_properties	Please load a file in as seen in the example where the user, password, url (which is the database it should use), and driver is stated.

Processors (7)

Name	Type	Description
Peregrine indexer	workflow	The nested workflow Peregrine Indexer uses a java peregrine skos cli.jar file to index the occurence of concepts in documents. The Peregrine indexer program can be downloaded from trac.nbic.nl/biosemantics/downloads. Install LVG2013Lite. See: https://trac.nbic.nl/biosemantics/wiki/Peregrine%20SKOS%20CLI Copy properties file from production.properties which can be downloaded from: https://trac.nbic.nl/biosemantics/wiki/Peregrine%20SKOS%20CLI Change lvg.properties LVG_DIR=/home/path/to/lvg2013lite/ This file can be found in: /home/path/to/lvg2013lite/data/config/lvg.properties Change the normalizer.lvg.properties and normalizer.lvg.binaryCache properties to point to the LVG installation path. normalizer.lvg.properties =/home/path/to/lvg2013lite/data/config/config/lvg.properties normalizer.lvg.binaryCahce=/home/path/to/lvg2013lite/standartNormCache2013.bin Have MySQL installed with a database called mydb or please create one. Have mysql-connector-java-5.1.28-bin.jar or better placed in the local JAR files.
Calculate Uncertainty Coefficient	workflow	A nested workflow to calculate the Uncertainty coefficient with the contingency values of the concept pairs. The calculated values are afterwards inserted into a SQL table.
Calculate Contincengy table	workflow	A nested workflow that retrieves the values of a 2x2 contingency table and puts the co-occurence of made concept pairs into the SQL table TableName_co_occurence. This is done by creating concept pairs which is retrieved from the Sql table TableName_occurence. This contincengy table consits out of a matrix where two concepts co-occur, concept A only occurs and not concept B, concept B occurs and not concept A and neither concepts occur in literature. B Not B A AB A not A B not A not B This is calculated for every possible concept pair.
Calculate Inner Product	workflow	A nested workflow to calculate the inner product of a concept profile.
Create Tables	workflow	This nested workflows create the necessary tables for inserting the generated values. The following tables are created: - Co-occurence table This table will consist out of - Occurence table - Uncertainty coefficient table - Inner product table - Big table
Summarize Big Table	workflow	A nested workflow to retrieve the calculated co-occurence, occurence, uncertainty coefficient and inner product values. These values are inserted into one table to summarize and quick access.
Get_abstracts	workflow	A nested workflow to retrieve abstracts from a sql table.

Beanshells (20)

Name	Description	Inputs	Outputs
Beanshell_SQL_INSERT_UC_values_INTO_table	A Beanshell script that generates a SQL statement to insert the calculated values of uncertainty coefficient per concept pair into the table.	timestamp concept_pairs UC_value	output_SQL
Beanshell_calculate_Uncertainty_Coefficient	A Beanshell script that calculates the Uncertainty coefficient based on the script of Herman van Haagen.	corrected_contingency_values	UC_value
Beanshell_generate_SQL_occurence_concept_doc_table	A Beanshell script which generates a sql statement to CREATE a TABLE. TABLE name is the input of the timestamp. The table consists out of 4 columns: concept_occ_doc_id, concept_uri, occurence_uri and doc_uri. Where as concept_occ_doc_id is the PRIMARY KEY and AUTOINCREMENTED. Every other column has a VARCHAR(200)	timeStamp	Table_sql
Beanshell_SQL_Generate_INSERT_INTO_big_table	A Beanshell script that generates SQL statements to insert two concepts and their given co-occurence, uncertainty coefficientcy and inner product values into the table.	timestamp in1	out1
Beanshell_SQL_statement_get_last_table_data_input	A Beanshell that generates SQL statements to retrieve the inner product of two concept profiles, the co-occurence of the concepts and the uncertainty coefficient.	timestamp	out1
Beanshell_SQL_generate_contingency_table	A Beanshell script that generates a SQL statement to retrieve the amount two concepts co-occur, concept A only occurs and not concept B, concept B occurs and not concept A and neither concepts occur in literature. The output of only concept A and only concept B needs to be corrected. This is done in the Beanshell script called: "Beanshell_correct_contincengy_table". Because the output of those two values were with the co-occurence. This needed to be subtracted.	concept_pairs table_name	out1
Beanshell_SQL_insert_into_co_occurence_table	A Beanshell script that generates SQL statement to insert into the co-occurence table the amount two given concepts co-occur together in literature.	contingency_values timestamp concept_pairs	out1
Beanshell_correct_contincengy_table	A Beanshell script that corrects the output SQL retrieved contincengy table by subtracting the co-occurence values of the concept only a and of concept only b value.	contingency_values	corrected_contingency_values
Beanshell_generate_SQL_statement_Get_destinct_concept	A Beanshell script that retrieves every concept from the Sql table TableName_occurence.	table_name	out1
Beanshell_make_concept_pairs	A Beanshell script that gets as input a list of concepts to create concept pairs.	concept_B concept_A	concept_pair
Beanshell_SQL_CREATE_last_big_table		timestamp	out1
Beanshell_SQL_generate_table_UC	A Beanshell script that generates a table for the uncertainty coefficient values.	time_stamp	output
Beanshell_SQL_create_table_co_occurence		timestamp	out1
Beanshell_generate_SQL_table_inner_product		timestamp	out1
Beanshell_get_timestamp	A Beanshell script which retreives the time the program started. As in year, month, day, hour, minutes and seconds in military time.		timeStamp
Beanshell_SQL_query_generator_Concept_Doc_occurence	A Beanshell script that generates a Sql statement to INSERT the concept URI and document URI in the table named after the timestamp/tablename the workflow started. It uses a database called mydb. If there is not a database/schema named mydb, please create one.	table_name concept_occurence_document	out1
Beanshell_extract_uri_doc_and_abstract	A Beanshell script which loops through the input to seperate the Document URIs and Abstract URIs.	Input	doc_uri abstracts
Beanshell_remove_dot_and_put_correct_doc_uri_in_con_occ_doc	A Beanshell script that replaces the temp created Taverna file with the correct Document URI.	concept_occurence_document doc_uri	out1
Beanshell_SQL_generate_Get_Inner_Product	A Beanshell script that generates a SQL statement to calculate the inner product between two concept profiles. These concept profiles only have concepts that both have in common. The concepts that are in common in the concept profiles are multiplied with each others UC value and added. This summed up value of two concept profiles is the inner product. For example: Concept profile A, has the concept A, B, C and D. Concept profile B has the concepts B, C, E. This means the common concepts for both concept profiles are B, C. Deze waardes worden gemultipliceerd en opgeteld.	concept_pairs timestamp	out1
Beanshell_SQL_generate_INSERT_INTO_Inner_Product_table	A Beanshell script that generates a SQL statement to insert the inner product values calculated of two concept profiles into the inner product table.	concept_pairs timestamp inner_product_value	out1

Outputs (1)

Name	Description
end

Datalinks (18)

Source	Sink
Create Tables:timeStamp	Peregrine indexer:table_name
Ontology_input	Peregrine indexer:ontology
Get_abstracts:db_results	Peregrine indexer:Input
db_properties	Peregrine indexer:propertyString
Calculate Contincengy table:outputlist	Calculate Uncertainty Coefficient:concept_pairs
Calculate Contincengy table:corrected_contingency_values	Calculate Uncertainty Coefficient:corrected_contingency_values
Create Tables:timeStamp	Calculate Uncertainty Coefficient:timestamp
db_properties	Calculate Uncertainty Coefficient:propertyString
Create Tables:timeStamp	Calculate Contincengy table:table_name
db_properties	Calculate Contincengy table:propertyString
Calculate Contincengy table:outputlist	Calculate Inner Product:concept_pairs
Create Tables:timeStamp	Calculate Inner Product:timestamp
db_properties	Calculate Inner Product:propertyString
db_properties	Create Tables:db_properties
Create Tables:timeStamp	Summarize Big Table:timestamp
db_properties	Summarize Big Table:propertyString
db_properties	Get_abstracts:propertyString
Summarize Big Table:resultList	end

Coordinations (3)

Controller	Target
Peregrine indexer	Calculate Contincengy table
Calculate Inner Product	Summarize Big Table
Calculate Uncertainty Coefficient	Calculate Inner Product

Information Workflow Type

Taverna 2

Information Uploader

Amrish Mahes

Information License

All versions of this Workflow are licensed under:

Information Version 5 (of 9)

Information Credits (1)

(People/Groups)

Amrish Mahes

Information Attributions (0)

(Workflows/Files)

None

Information Tags (0)

None

Log in to add Tags

Information Shared with Groups (0)

None

Information Featured In Packs (1)

BioSemantics Concept Profile Generation Workflows

Log in to add to one of your Packs

Information Attributed By (0)

(Workflows/Files)

None

Information Favourited By (0)

No one

Information Statistics

2549 viewings

876 downloads

[ see breakdown ]

Citations (0)

None

Version History

In chronological order:

Full_CPGP

Created by Amrish Mahes on Wednesday 30 April 2014 19:14:47 (UTC)
Full_CPGP

Created by Amrish Mahes on Wednesday 30 April 2014 19:18:01 (UTC)
Full_CPGP

Created by Amrish Mahes on Wednesday 30 April 2014 19:18:42 (UTC)
Concept Profile Generation Pipeline

Created by Amrish Mahes on Wednesday 07 May 2014 15:33:38 (UTC)

Revision comment:

New version with comment and the right websites to download dependencies
Concept Profile Generation Pipeline

Created by Amrish Mahes on Monday 12 May 2014 15:34:00 (UTC)
Concept Profile Generation Pipeline

Created by Amrish Mahes on Wednesday 11 June 2014 13:23:36 (UTC)
Concept Profile Generation Pipeline

Created by Amrish Mahes on Sunday 15 June 2014 17:45:34 (UTC)

Revision comment:

Changed Table construction workflow
For peregrine tool invocation it is now mandatory to give path to the prodcution.properties and peregrine-skos-cli.jar
Concept Profile Generation Pipeline

Created by Amrish Mahes on Sunday 15 June 2014 17:50:32 (UTC)
Concept Profile Generation Pipeline

Created by Amrish Mahes on Sunday 15 June 2014 17:55:11 (UTC)

Revision comment:

Removed the LIMIT 50 in the sql_value file

Reviews (0)

No reviews yet

Be the first to review!

Comments (0)

View Timeline

No comments yet

Log in to make a comment

Other workflows that use similar services (0)

There are no workflows in myExperiment that use similar services to this Workflow.