06 April 2012

Indexing the content of Gene Ontology with apache SOLR

Via Wikipedia:"Solr (http://lucene.apache.org/solr/) is an open source enterprise search platform from the Apache Lucene project. Its major features include powerful full-text search, hit highlighting, faceted search, dynamic clustering, database integration, and rich document (e.g., Word, PDF) handling. Providing distributed search and index replication, Solr is highly scalable." In the this post, I'll show how I've used SOLR to index the content of GeneOntology.

Download and install SOLR

Download from http://mirrors.ircam.fr/pub/apache/lucene/solr/3.5.0/apache-solr-3.5.0.tgz.
tar xvfz apache-solr-3.5.0.tgz
rm apache-solr-3.5.0.tgz

Configure schema.xml

We need to tell SOLR about the which fields of GO will be indexed, what are their type, how they should be tokenized and parsed. This information is defined in the schema.xml. The following components will be indexed: accession, name, synonym and definition. Edit apache-solr-3.5.0/example/solr/conf/schema.xml and add the following <fields>:

<field name="go_name" type="text_general" indexed="true" stored="true" multiValued="false"/>
<field name="go_synonym" type="text_general" indexed="true" stored="true" multiValued="true"/>
<field name="go_definition" type="text_general" indexed="true" stored="true" multiValued="false"/>

Start the SOLR server

In this example, the SOLR server is started using the simple Jetty server provided in the distribution:

$ cd apache-solr-3.5.0/example/example
$ java -jar start.jar

(...)

Indexing Gene Ontology

Go is downloaded as RDF/XML from http://archive.geneontology.org/latest-termdb/go_daily-termdb.rdf-xml.gz
 
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE go:go PUBLIC "-//Gene Ontology//Custom XML/RDF Version 2.0//EN" "http://www.geneontology.org/dtd/go.dtd">

<go:go xmlns:go="http://www.geneontology.org/dtds/go.dtd#" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
    <rdf:RDF>
        <go:term rdf:about="http://www.geneontology.org/go#GO:0000001">
            <go:accession>GO:0000001</go:accession>
            <go:name>mitochondrion inheritance</go:name>
            <go:synonym>mitochondrial inheritance</go:synonym>
            <go:definition>The distribution of mitochondria, including the mitochondrial genome, into daughter cells after mitosis or meiosis, mediated by interactions between mitochondria and the cytoskeleton.</go:definition>
            <go:is_a rdf:resource="http://www.geneontology.org/go#GO:0048308" />
            <go:is_a rdf:resource="http://www.geneontology.org/go#GO:0048311" />
        </go:term>
        <go:term rdf:about="http://www.geneontology.org/go#GO:0000002">
            <go:accession>GO:0000002</go:accession>
            <go:name>mitochondrial genome maintenance</go:name>
            <go:definition>The maintenance of the structure and integrity of the mitochondrial genome; includes replication and segregation of the mitochondrial chromosome.</go:definition>
            <go:is_a rdf:resource="http://www.geneontology.org/go#GO:0007005" />
            <go:dbxref rdf:parseType="Resource">
                <go:database_symbol>InterPro</go:database_symbol>
(...)
 
We now need to transform this XML file to another XML file that can be indexed by the SOLR server.  

"You can modify a Solr index by POSTing XML Documents containing instructions to add (or update) documents, delete documents, commit pending adds and deletes, and optimize your index."

The following XSLT stylesheet is used to transform the RDF/XML for GO:


$ xsltproc --novalid go2solr.xsl go_daily-termdb.rdf-xml.gz > add.xml
$ cat add.xml

Before indexing the current disk usage under apache-solr-3.5.0 is 136Mo. We can now use the java utiliy post.jar to index GeneOntology.

 $ cd  ~/package/apache-solr-3.5.0/example/exampledocs
 $ java -jar post.jar  add.xml

SimplePostTool: version 1.4
SimplePostTool: POSTing files to http://localhost:8983/solr/update..
SimplePostTool: POSTing file jeter.xml
SimplePostTool: COMMITting Solr index changes..

After indexing, the disk usage under apache-solr-3.5.0 is 153Mo.

Querying

Search for the GO terms having go:definition containing "cancer" a go:name containing "genome" but discard those having go:definition containing "metabolism".
 curl "http://localhost:8983/solr/select/?q=go_definition%3Acancer+go_name%3Agenome+-go_definition%3Ametabolism&version=2.2&start=0&rows=10&indent=on"
Same query, but return the result as a JSON structure:
 curl "http://localhost:8983/solr/select/?q=go_definition%3Acancer+go_name%3Agenome+-go_definition%3Ametabolism&version=2.2&start=0&rows=10&indent=on&wt=json"
That's it, Pierre

4 comments:

Chris Mungall said...

Hi Pierre

As it happens we're investigating Solr as a backend for both QuickGO and AmiGO. We're looking at storing not only the ontology terms, but also all genes together with the pre-computed inferences over the whole ontology. We should have a demo server available soon.

harsha said...

hi
pierre, I have to use a custom genome (silkworm) and go annotation to identify binding sites and annotate peaks of chipseq data.

I have go and interpro annotations.I was browsing to manually do the annotation, a program called ceas needs sqlite3 format. could you please suggest how to convert a go annotation to a sqlite3 file.
or an alternative where i can annotate peaks (bed) with a go or interpro file.
thanks
harsha
pune, india

Pierre Lindenbaum said...

harsha, ask biostars.org please

Algy said...

Just a quick thank you / merci beaucoup - almost 4 years from you posting it, this has been particularly helpful ... can look forward to a bright ontology-searching future ;)