Concept Profile Generation Pipeline

Created: 2014-04-30 19:14:51 Last updated: 2014-06-15 17:55:11

Download Workflow

Requirements:

Have MySQL installed with a database called mydb or please create one. Have a mysql connector file. This can be downloaded from mysql.com. Rename the downloaded file to mysql-connector-java.jar and place it in the the local lib file of Taverna as written on the dependencies tab. Choose this dependency by rightclicking a beanshell script and selecting it for the whole workflow as system classloader.

Download the Peregrine SKOS CLI from: https://trac.nbic.nl/biosemantics/downloads Install LVG2013Lite. See: https://trac.nbic.nl/biosemantics/wiki/Peregrine%20SKOS%20CLI

Change lvg.properties LVG_DIR=/home/path/to/lvg2013lite/ This file can be found in:

/home/path/to/lvg2013lite/data/config/lvg.properties

Copy properties file from production.properties which can be found on: https://trac.nbic.nl/biosemantics/wiki/Peregrine%20SKOS%20CLI

Change the normalizer.lvg.properties and normalizer.lvg.binaryCache properties to point to the LVG installation path. This file can be obtained from https://trac.nbic.nl/biosemantics/wiki/Peregrine%20SKOS%20CLI

Have Peregrine indexer installed locally and give it the correct path in the component named "Indexer_tool". Which can be found in the nested workflow named Peregrine indexer.

If using a Windows machine please install cygwin. Set cygwin in the $PATH of the environment variabale. Go to taverna File --> preferences --> Tool invocation --> set modify on explicit locations --> edit a default local to Shell: C:\Windows\system32\cmd.exe /c and save that. -->Set Modify on: symbolic locations --> edit --> choose default local and save. When using a unix environment make sure the default local is set Shell into /bin/sh -c. Please make sure this is also set into the indexer tool part.

Have JAVA_HOME C:\Program Files\jdk 1.7.0 in environment variabale when using Windows.

The nested workflow Get_abstract uses a sql command to retrieve abstracts which assumes are located in the database mydb. If this is not the case, please change the database in the SQL value.

Preview

Download as scalable diagram (SVG)

Run

Run this Workflow in the Taverna Workbench...

Option 1:

Copy and paste this link into File > 'Open workflow location...'
http://www.myexperiment.org/workflows/4272/download?version=6
[ More Info Expand ]

Workflow Components

Authors (1)

Titles (1)

Descriptions (1)

Requirements: Have MySQL installed with a database called mydb or please create one. Have a mysql connector file. This can be downloaded from mysql.com. Rename the downloaded file to mysql-connector-java.jar and place it in the the local lib file of Taverna as written on the dependencies tab. Choose this dependency by rightclicking a beanshell script and selecting it for the whole workflow as system classloader. Download the Peregrine SKOS CLI from: https://trac.nbic.nl/biosemantics/downloads Install LVG2013Lite. See: https://trac.nbic.nl/biosemantics/wiki/Peregrine%20SKOS%20CLI Change lvg.properties LVG_DIR=/home/path/to/lvg2013lite/ This file can be found in: /home/path/to/lvg2013lite/data/config/lvg.properties Copy properties file from production.properties which can be found on: https://trac.nbic.nl/biosemantics/wiki/Peregrine%20SKOS%20CLI Change the normalizer.lvg.properties and normalizer.lvg.binaryCache properties to point to the LVG installation path. This file can be obtained from https://trac.nbic.nl/biosemantics/wiki/Peregrine%20SKOS%20CLI Have Peregrine indexer installed locally and give it the correct path in the component named "Indexer_tool". Which can be found in the nested workflow named Peregrine indexer. If using a Windows machine please install cygwin. Set cygwin in the $PATH of the environment variabale. Go to taverna File --> preferences --> Tool invocation --> set modify on explicit locations --> edit a default local to Shell: C:\Windows\system32\cmd.exe /c and save that. -->Set Modify on: symbolic locations --> edit --> choose default local and save. When using a unix environment make sure the default local is set Shell into /bin/sh -c. Please make sure this is also set into the indexer tool part. Have JAVA_HOME C:\Program Files\jdk 1.7.0 in environment variabale when using Windows. The nested workflow Get_abstract uses a sql command to retrieve abstracts which assumes are located in the database mydb. If this is not the case, please change the database in the SQL value.

Dependencies (1)

Inputs (2)

Name	Description
Ontology_input	A Skos formated dictionary/thesaurus of predefined concepts.
db_properties	Please load a file in as seen in the example where the user, password, url (which is the database it should use), and driver is stated.

Processors (7)

Name	Type	Description
Peregrine_Indexer	workflow	The nested workflow Peregrine Indexer uses a java peregrine skos cli.jar file to index the occurrence of concepts in documents. The Peregrine indexer program can be downloaded from trac.nbic.nl/biosemantics/downloads. Install LVG2013Lite. See: https://trac.nbic.nl/biosemantics/wiki/Peregrine%20SKOS%20CLI Copy properties file from production.properties which can be downloaded from: https://trac.nbic.nl/biosemantics/wiki/Peregrine%20SKOS%20CLI Change lvg.properties LVG_DIR=/home/path/to/lvg2013lite/ This file can be found in: /home/path/to/lvg2013lite/data/config/lvg.properties Change the normalizer.lvg.properties and normalizer.lvg.binaryCache properties to point to the LVG installation path. normalizer.lvg.properties =/home/path/to/lvg2013lite/data/config/config/lvg.properties normalizer.lvg.binaryCahce=/home/path/to/lvg2013lite/standartNormCache2013.bin Have MySQL installed with a database called mydb or please create one. Have mysql-connector-java-5.1.28-bin.jar or better placed in the local JAR files.
Calculate_Uncertainty_Coefficient	workflow	A nested workflow to calculate the Uncertainty coefficient with the contingency table values of the concept pairs. The calculated values are afterwards inserted into the uncertainty coefficient SQL table. The row with the same concepts in column name concept_uri_1 is the concept profile. The values in the columns name concept_uri_2 are the concepts that are used for calculating the uncertainty coefficient (similarity measure). Example of the constructed SQL table: 1 A A UC value of concept pair AA 2 A B UC value of concept pair AB 3 A C UC value of concept pair AC Example of a concept profile made of concept A: A \| \|__UC value__A \|__UC value__B \|__UC value__C
Calculate_Contingency_Table	workflow	A nested workflow that retrieves the values of a 2x2 contingency table and puts the co-occurrence of made concept pairs into the SQL table TableName_co_occurrence. This is done by creating concept pairs which is retrieved from the Sql table TableName_occurrence. This contingency table consits out of a matrix where two concepts co-occur, concept A only occurs and not concept B, concept B occurs and not concept A and neither concepts occur in literature. Example of a 2x2 contingency table of concept pair A B: B Not B A AB A not A B not A not B This is calculated for every possible concept pair.
Calculate_Inner_Product	workflow	A nested workflow to calculate the inner product between concept profile pairs for concept profile matching. The calculated values are afterwards inserted into the inner product SQL table. Example of calculating the inner product between two concept profiles: A B \| \| \|__UC value__A x A__UC value__\| \| + \| \|__UC value__B x B__UC value__\| \| \| \|__UC value__C D__UC value__\| The inner product of the two concept profile A and B is the common concept multplied between each other and added: (UC value AA x UC value BA) + (UC value AB x UC value BB) = inner product
Create_Tables	workflow	This nested workflows creates the necessary tables for inserting the generated values. The following tables are created: - Co-occurrence table - Occurrence table - Uncertainty coefficient table - Inner product table - Summary table
Get_Summary_Table	workflow	A nested workflow to retrieve the calculated co-occurrence, occurrence, uncertainty coefficient and inner product values. These values are inserted into one table to summarize and quick access.
Get_abstracts	workflow	A nested workflow to retrieve abstracts from a sql table. This table is assumed to have atleast two columns consisting out of pmid and abstract text.

Beanshells (21)

Name	Description	Inputs	Outputs
Beanshell_SQL_Generate_INSERT_INTO_summary_table	A Beanshell script that generates SQL statements to insert two concepts and their given co-occurence, uncertainty coefficientcy and inner product values into the table.	timestamp in1	out1
Beanshell_SQL_statement_get_last_table_data_input	A Beanshell that generates SQL statements to retrieve the inner product of two concept profiles, the co-occurence of the concepts and the uncertainty coefficient.	timestamp	out1
Beanshell_generate_SQL_table_inner_product	A Beanshell script which generates a sql statement to CREATE a TABLE for the inner product values. TABLE name is a combination of the input timestamp and _inner_product. The table consists out of 4 columns: concept_record_id, concept_uri_1, concept_uri_2 and Inner_product. Where as concept_record_id is the PRIMARY KEY and AUTOINCREMENTED.	timestamp	out1
Beanshell_SQL_create_table_co_occurrence	A Beanshell script which generates a sql statement to CREATE a TABLE for the co-occurrences. TABLE name is a combination of the input timestamp and _co_occurrence. The table consists out of 4 columns: concept_record_id, concept_uri_1, concept_uri_2 and Co-occurrence. Where as concept_record_id is the PRIMARY KEY and AUTOINCREMENTED.	timestamp	out1
Beanshell_SQL_INSERT_UC_values_INTO_table	A Beanshell script that generates a SQL statement to insert the calculated values of uncertainty coefficient per concept pair into the table.	timestamp concept_pairs UC_value	output_SQL
Beanshell_calculate_Uncertainty_Coefficient	A Beanshell script that calculates the Uncertainty coefficient based on a script of Herman van Haagen.	corrected_contingency_values	UC_value
Beanshell_get_timestamp	A Beanshell script which retreives the time the program started. As in year, month, day, hour, minutes and seconds in military time.		timeStamp
Beanshell_SQL_generate_Get_Inner_Product	A Beanshell script that generates a SQL statement to calculate the inner product between two concept profiles. These concept profiles only have concepts that both have in common. The concepts that are in common in the concept profiles are multiplied with each others UC value and added. This summed up value of two concept profiles is the inner product. For example: Concept profile A, has the concept A, B, C and D. Concept profile B has the concepts B, C, E. This means the common concepts for both concept profiles are B, C. These values are multiplied and added. This final value is the inner product between two concept profiles.	concept_pairs timestamp	out1
Beanshell_SQL_generate_INSERT_INTO_Inner_Product_table	A Beanshell script that generates a SQL statement to insert the inner product values calculated of two concept profiles into the inner product table.	concept_pairs timestamp inner_product_value	out1
Beanshell_SQL_generate_contingency_table	A Beanshell script that generates a SQL statement to retrieve the amount two concepts co-occur, concept A only occurs and not concept B, concept B occurs and not concept A. When a concept pair is the same. As in concept pair A A or B B etc. It will use the first SQL statement. Else it will use the second SQL statement. The output of only concept A and only concept B needs to be corrected and the value to get when neither concepts occur in literature is done in the Beanshell script called: "Beanshell_correct_contincengy_table".	concept_pairs table_name	out1
Beanshell_SQL_insert_into_co_occurrence_table	A Beanshell script that generates SQL statement to insert into the co-occurrence table the amount two given concepts co-occur together in literature.	contingency_values timestamp concept_pairs	out1
Beanshell_correct_contingency_table	A Beanshell script that corrects the output SQL retrieved contingency table by subtracting the co-occurence values of the concept only A and of concept only B value. It also uses the value of the total amount of abstracts there are in the dataset for calculating the value of when neither concepts occure.	contingency_values total_nr_abtracts	corrected_contingency_values
Beanshell_generate_SQL_statement_Get_distinct_concept	A Beanshell script that retrieves every concept from the Sql table TableName_occurrence.	table_name	out1
Beanshell_make_concept_pairs	A Beanshell script that gets as input a list of concepts to create concept pairs.	concept_B concept_A	concept_pair
Beanshell_SQL_count_abstracts			out1
Beanshell_generate_SQL_occurrence_concept_doc_table	A Beanshell script which generates a sql statement to CREATE a TABLE for the occurrences. TABLE name is a combination of the input timestamp and _occurrence. The table consists out of 4 columns: concept_record_id, concept_uri and document_uri. Where as concept_record_id is the PRIMARY KEY and AUTOINCREMENTED.	timeStamp	Table_sql
Beanshell_SQL_generate_table_UC	A Beanshell script which generates a sql statement to CREATE a TABLE for the uncertainty coefficient values. TABLE name is a combination of the input timestamp and _uncertainty_coefficient. The table consists out of 4 columns: concept_record_id, concept_uri_1, concept_uri_2 and Uncertainty_coefficient. Where as concept_record_id is the PRIMARY KEY and AUTOINCREMENTED.	time_stamp	output
Beanshell_SQL_CREATE_Summary_table	A Beanshell script which generates a sql statement to CREATE a TABLE for the summary table. TABLE name is a combination of the input timestamp and _summary. The table consists out of 4 columns: concept_record_id, concept_uri_1, concept_uri_2, Inner_product, Uncertainty_coefficient, Co-occurrence. Where as concept_record_id is the PRIMARY KEY and AUTOINCREMENTED.	timestamp	out1
Beanshell_SQL_query_generator_Concept_Doc_occurrence	A Beanshell script that generates a Sql statement to INSERT the concept URI and document URI in the table named after the timestamp/tablename the workflow started. It uses a database called mydb. If there is not a database/schema named mydb, please create one.	table_name concept_occurence_document	out1
Beanshell_extract_uri_doc_and_abstract	A Beanshell script which loops through the input to seperate the Document URIs and Abstract URIs.	Input	doc_uri abstracts
Beanshell_remove_dot_and_correct_doc_uri_in_con_doc	A Beanshell script that replaces the temp created Taverna file with the correct Document URI.	concept_occurence_document doc_uri	out1

Outputs (1)

Name	Description
end

Datalinks (18)

Source	Sink
Create_Tables:timeStamp	Peregrine_Indexer:table_name
Ontology_input	Peregrine_Indexer:ontology
Get_abstracts:db_results	Peregrine_Indexer:Input
db_properties	Peregrine_Indexer:propertyString
Calculate_Contingency_Table:outputlist	Calculate_Uncertainty_Coefficient:concept_pairs
Calculate_Contingency_Table:corrected_contingency_values	Calculate_Uncertainty_Coefficient:corrected_contingency_values
Create_Tables:timeStamp	Calculate_Uncertainty_Coefficient:timestamp
db_properties	Calculate_Uncertainty_Coefficient:propertyString
Create_Tables:timeStamp	Calculate_Contingency_Table:table_name
db_properties	Calculate_Contingency_Table:propertyString
Calculate_Contingency_Table:outputlist	Calculate_Inner_Product:concept_pairs
Create_Tables:timeStamp	Calculate_Inner_Product:timestamp
db_properties	Calculate_Inner_Product:propertyString
db_properties	Create_Tables:db_properties
Create_Tables:timeStamp	Get_Summary_Table:timestamp
db_properties	Get_Summary_Table:propertyString
db_properties	Get_abstracts:propertyString
Get_Summary_Table:resultList	end

Coordinations (3)

Controller	Target
Peregrine_Indexer	Calculate_Contingency_Table
Calculate_Uncertainty_Coefficient	Calculate_Inner_Product
Calculate_Inner_Product	Get_Summary_Table

Information Workflow Type

Taverna 2

Information Uploader

Amrish Mahes

Information License

All versions of this Workflow are licensed under:

Information Version 6 (of 9)

Information Credits (1)

(People/Groups)

Amrish Mahes

Information Attributions (0)

(Workflows/Files)

None

Information Tags (0)

None

Log in to add Tags

Information Shared with Groups (0)

None

Information Featured In Packs (1)

BioSemantics Concept Profile Generation Workflows

Log in to add to one of your Packs

Information Attributed By (0)

(Workflows/Files)

None

Information Favourited By (0)

No one

Information Statistics

2545 viewings

876 downloads

[ see breakdown ]

Citations (0)

None

Version History

In chronological order:

Full_CPGP

Created by Amrish Mahes on Wednesday 30 April 2014 19:14:47 (UTC)
Full_CPGP

Created by Amrish Mahes on Wednesday 30 April 2014 19:18:01 (UTC)
Full_CPGP

Created by Amrish Mahes on Wednesday 30 April 2014 19:18:42 (UTC)
Concept Profile Generation Pipeline

Created by Amrish Mahes on Wednesday 07 May 2014 15:33:38 (UTC)

Revision comment:

New version with comment and the right websites to download dependencies
Concept Profile Generation Pipeline

Created by Amrish Mahes on Monday 12 May 2014 15:34:00 (UTC)
Concept Profile Generation Pipeline

Created by Amrish Mahes on Wednesday 11 June 2014 13:23:36 (UTC)
Concept Profile Generation Pipeline

Created by Amrish Mahes on Sunday 15 June 2014 17:45:34 (UTC)

Revision comment:

Changed Table construction workflow
For peregrine tool invocation it is now mandatory to give path to the prodcution.properties and peregrine-skos-cli.jar
Concept Profile Generation Pipeline

Created by Amrish Mahes on Sunday 15 June 2014 17:50:32 (UTC)
Concept Profile Generation Pipeline

Created by Amrish Mahes on Sunday 15 June 2014 17:55:11 (UTC)

Revision comment:

Removed the LIMIT 50 in the sql_value file

Reviews (0)

No reviews yet

Be the first to review!

Comments (0)

View Timeline

No comments yet

Log in to make a comment

Other workflows that use similar services (0)

There are no workflows in myExperiment that use similar services to this Workflow.