elastic search in genome apollo

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

elastic search in genome apollo

Michael Poeschla
Hi,

has anyone implemented the jbrowse_elasticsearch plugin in Apollo? I'm running from the docker container with an nginx reverse proxy and having trouble adding this plugin. How would I add this to the docker image, with the middleware as well?

The reason we want to do this is the predicted gene model names we have in our tracks are not always intuitive so it's hard to find them by searching in Apollo (ie mTOR is ID'd as "serine/threonine-protein kinase mTOR", so if you just start typing mTOR you get no results). Wondering also if anyone else has encountered this problem and found good solutions, besides manipulating all of our tracks.

Thanks so much,
Michael




This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

Reply | Threaded
Open this post in threaded view
|

Re: elastic search in genome apollo

Colin
Aside from changing your gene identifiers, I think the full text search from elasticsearch could help. I forwarded some comments here about that https://github.com/elsiklab/jbrowse_elasticsearch/issues/7

Potentially data that is loaded in Apollo backend could use text similarity %like% or full text sql queries but for normal jbrowse tracks that are not loaded in the sql database, the jbrowse_elasticsearch plugin seems reasonable.

-Colin

On Tue, Nov 13, 2018 at 7:35 AM Michael Poeschla <[hidden email]> wrote:
Hi,

has anyone implemented the jbrowse_elasticsearch plugin in Apollo? I'm running from the docker container with an nginx reverse proxy and having trouble adding this plugin. How would I add this to the docker image, with the middleware as well?

The reason we want to do this is the predicted gene model names we have in our tracks are not always intuitive so it's hard to find them by searching in Apollo (ie mTOR is ID'd as "serine/threonine-protein kinase mTOR", so if you just start typing mTOR you get no results). Wondering also if anyone else has encountered this problem and found good solutions, besides manipulating all of our tracks.

Thanks so much,
Michael



This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

Reply | Threaded
Open this post in threaded view
|

Re: elastic search in genome apollo

nathandunn

Michael,

What problems are you running it when adding the plugins? 

Nathan


On Nov 13, 2018, at 4:55 PM, Colin <[hidden email]> wrote:

Aside from changing your gene identifiers, I think the full text search from elasticsearch could help. I forwarded some comments here about that https://github.com/elsiklab/jbrowse_elasticsearch/issues/7

Potentially data that is loaded in Apollo backend could use text similarity %like% or full text sql queries but for normal jbrowse tracks that are not loaded in the sql database, the jbrowse_elasticsearch plugin seems reasonable.

-Colin

On Tue, Nov 13, 2018 at 7:35 AM Michael Poeschla <[hidden email]> wrote:
Hi,

has anyone implemented the jbrowse_elasticsearch plugin in Apollo? I'm running from the docker container with an nginx reverse proxy and having trouble adding this plugin. How would I add this to the docker image, with the middleware as well?

The reason we want to do this is the predicted gene model names we have in our tracks are not always intuitive so it's hard to find them by searching in Apollo (ie mTOR is ID'd as "serine/threonine-protein kinase mTOR", so if you just start typing mTOR you get no results). Wondering also if anyone else has encountered this problem and found good solutions, besides manipulating all of our tracks.

Thanks so much,
Michael



This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.




This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.






This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

Reply | Threaded
Open this post in threaded view
|

Re: elastic search in genome apollo

Michael Poeschla
Hi, 
I'm following the instructions from https://github.com/elsiklab/jbrowse_elasticsearch

we have elasticsearch and the middleware app.js running. I was able to execute generate-names.pl and this seemed to work correctly from the terminal output. The changes also got picked up in the tracklist.json for the genome. But there is no sign of the elasticsearch plugin in web apollo...added the plugin to the config.groovy, and we have another jbrowse plugin (GC content) working fine so not sure what's the problem here. 


Thanks for all of your help! :)
Michael 

From: "Nathan Dunn" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Tuesday, November 13, 2018 6:33:05 PM
Subject: Re: [apollo] elastic search in genome apollo


Michael,

What problems are you running it when adding the plugins? 

Nathan


On Nov 13, 2018, at 4:55 PM, Colin <[hidden email]> wrote:

Aside from changing your gene identifiers, I think the full text search from elasticsearch could help. I forwarded some comments here about that https://github.com/elsiklab/jbrowse_elasticsearch/issues/7

Potentially data that is loaded in Apollo backend could use text similarity %like% or full text sql queries but for normal jbrowse tracks that are not loaded in the sql database, the jbrowse_elasticsearch plugin seems reasonable.

-Colin

On Tue, Nov 13, 2018 at 7:35 AM Michael Poeschla <[hidden email]> wrote:
Hi,

has anyone implemented the jbrowse_elasticsearch plugin in Apollo? I'm running from the docker container with an nginx reverse proxy and having trouble adding this plugin. How would I add this to the docker image, with the middleware as well?

The reason we want to do this is the predicted gene model names we have in our tracks are not always intuitive so it's hard to find them by searching in Apollo (ie mTOR is ID'd as "serine/threonine-protein kinase mTOR", so if you just start typing mTOR you get no results). Wondering also if anyone else has encountered this problem and found good solutions, besides manipulating all of our tracks.

Thanks so much,
Michael



This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.




This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.






This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

Reply | Threaded
Open this post in threaded view
|

Re: elastic search in genome apollo

nathandunn
Can you post your apollo config file? 

Nathan

On Nov 14, 2018, at 6:45 PM, Michael Poeschla <[hidden email]> wrote:

Hi, 
I'm following the instructions from https://github.com/elsiklab/jbrowse_elasticsearch

we have elasticsearch and the middleware app.js running. I was able to execute generate-names.pl and this seemed to work correctly from the terminal output. The changes also got picked up in the tracklist.json for the genome. But there is no sign of the elasticsearch plugin in web apollo...added the plugin to the config.groovy, and we have another jbrowse plugin (GC content) working fine so not sure what's the problem here. 


Thanks for all of your help! :)
Michael 

From: "Nathan Dunn" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Tuesday, November 13, 2018 6:33:05 PM
Subject: Re: [apollo] elastic search in genome apollo


Michael,

What problems are you running it when adding the plugins? 

Nathan


On Nov 13, 2018, at 4:55 PM, Colin <[hidden email]> wrote:

Aside from changing your gene identifiers, I think the full text search from elasticsearch could help. I forwarded some comments here about that https://github.com/elsiklab/jbrowse_elasticsearch/issues/7

Potentially data that is loaded in Apollo backend could use text similarity %like% or full text sql queries but for normal jbrowse tracks that are not loaded in the sql database, the jbrowse_elasticsearch plugin seems reasonable.

-Colin

On Tue, Nov 13, 2018 at 7:35 AM Michael Poeschla <[hidden email]> wrote:
Hi,

has anyone implemented the jbrowse_elasticsearch plugin in Apollo? I'm running from the docker container with an nginx reverse proxy and having trouble adding this plugin. How would I add this to the docker image, with the middleware as well?

The reason we want to do this is the predicted gene model names we have in our tracks are not always intuitive so it's hard to find them by searching in Apollo (ie mTOR is ID'd as "serine/threonine-protein kinase mTOR", so if you just start typing mTOR you get no results). Wondering also if anyone else has encountered this problem and found good solutions, besides manipulating all of our tracks.

Thanks so much,
Michael



This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.




This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.






This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.




This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

Reply | Threaded
Open this post in threaded view
|

Re: elastic search in genome apollo

Michael Poeschla
Here it is - sorry for the late replies, I'm on europe time.

environments {
    development {
    }
    test {
    }
    production {
        dataSource {
            dbCreate = "update"
            username = System.getenv("WEBAPOLLO_DB_USERNAME") ?: *
            password = System.getenv("WEBAPOLLO_DB_PASSWORD") ?: *

            driverClassName = "org.postgresql.Driver"
            dialect = "org.hibernate.dialect.PostgresPlusDialect"
            url = "jdbc:postgresql://${System.getenv("WEBAPOLLO_DB_HOST")?:"127.0.0.1"}/${System.getenv("WEBAPOLLO_DB_NAME")?:"apollo"}"

            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = false
                initialSize                   = 5
                maxActive                     = 50
                minIdle                       = 5
                maxIdle                       = 25
                maxWait                       = 10000
                maxAge                        = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis    = 60000
                validationQuery               = "SELECT 1"
                validationQueryTimeout        = 3
                validationInterval            = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
        dataSource_chado {
            dbCreate = "update"
            username = System.getenv("CHADO_DB_USERNAME") ?: *
            password = System.getenv("CHADO_DB_PASSWORD") ?: *

            driverClassName = "org.postgresql.Driver"
            dialect = "org.hibernate.dialect.PostgresPlusDialect"

            url = "jdbc:postgresql://${System.getenv("CHADO_DB_HOST")?:"127.0.0.1"}/${System.getenv("CHADO_DB_NAME")?:"chado"}"

            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = false
                initialSize                   = 5
                maxActive                     = 50
                minIdle                       = 5
                maxIdle                       = 25
                maxWait                       = 10000
                maxAge                        = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis    = 60000
                validationQuery               = "SELECT 1"
                validationQueryTimeout        = 3
                validationInterval            = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
    }
}

apollo {
    default_minimum_intron_size = System.getenv("WEBAPOLLO_MINIMUM_INTRON_SIZE") ? System.getenv("WEBAPOLLO_MINIMUM_INTRON_SIZE").toInteger() : 1
    history_size = System.getenv("WEBAPOLLO_HISTORY_SIZE") ? System.getenv("WEBAPOLLO_HISTORY_SIZE").toInteger() : 0
    overlapper_class = System.getenv("WEBAPOLLO_OVERLAPPER_CLASS") ?: "org.bbop.apollo.sequence.OrfOverlapper"
    use_cds_for_new_transcripts = System.getenv("WEBAPOLLO_CDS_FOR_NEW_TRANSCRIPTS").equals("true") // will default to false
    feature_has_dbxrefs = System.getenv("WEBAPOLLO_FEATURE_HAS_DBXREFS") ?: true
    feature_has_attributes = System.getenv("WEBAPOLLO_FEATURE_HAS_ATTRS") ?: true
    feature_has_pubmed_ids = System.getenv("WEBAPOLLO_FEATURE_HAS_PUBMED") ?: true
    feature_has_go_ids = System.getenv("WEBAPOLLO_FEATURE_HAS_GO") ?: true
    feature_has_comments = System.getenv("WEBAPOLLO_FEATURE_HAS_COMMENTS") ?: true
    feature_has_status = System.getenv("WEBAPOLLO_FEATURE_HAS_STATUS") ?: true
    translation_table = "/config/translation_tables/ncbi_" + (System.getenv("WEBAPOLLO_TRANSLATION_TABLE") ?: "1") + "_translation_table.txt"
    get_translation_code = System.getenv("WEBAPOLLO_TRANSLATION_TABLE") ? System.getenv("WEBAPOLLO_TRANSLATION_TABLE").toInteger() : 1

    // ADDED BLAT AND SET PATH TO BIN
    sequence_search_tools = [
        blat_nuc: [
          search_exe: "/usr/local/bin/blat",
          search_class: "org.bbop.apollo.sequence.search.blat.BlatCommandLineNucleotideToNucleotide",
          name: "Blat nucleotide",
          params: ""
        ],
        blat_prot: [
          search_exe: "/usr/local/bin/blat",
          search_class: "org.bbop.apollo.sequence.search.blat.BlatCommandLineProteinToNucleotide",
          name: "Blat protein",
          params: ""
        ]
      ]
    // TODO: should come from config or via preferences database
    splice_donor_sites = System.getenv("WEBAPOLLO_SPLICE_DONOR_SITES") ? System.getenv("WEBAPOLLO_SPLICE_DONOR_SITES").split(",") : ["GT"]
    splice_acceptor_sites = System.getenv("WEBAPOLLO_SPLICE_ACCEPTOR_SITES") ? System.getenv("WEBAPOLLO_SPLICE_ACCEPTOR_SITES").split(",") : ["AG"]
    gff3.source = System.getenv("WEBAPOLLO_GFF3_SOURCE") ?: "."

    google_analytics = System.getenv("WEBAPOLLO_GOOGLE_ANALYTICS_ID") ?: ["UA-62921593-1"]

    admin{
        username = System.getenv("APOLLO_ADMIN_EMAIL") ?: *
        password = System.getenv("APOLLO_ADMIN_PASSWORD") ?: *
        firstName = System.getenv("APOLLO_ADMIN_FIRST_NAME") ?: *
        lastName = System.getenv("APOLLO_ADMIN_LAST_NAME") ?: *
    }
}

jbrowse {
    git {
        url = "https://github.com/GMOD/jbrowse"
        tag = "maint/1.12.5-apollo"
    }
    plugins {
        WebApollo{
            included = true
        }
        NeatHTMLFeatures{
            included = System.getenv("WEBAPOLLO_JBROWSE_PLUGIN_NEATHTML") ?: true
        }
        NeatCanvasFeatures{
            included = System.getenv("WEBAPOLLO_JBROWSE_PLUGIN_NEATCANVAS") ?: true
        }
        RegexSequenceSearch{
            included = true
        }
        HideTrackLabels{
            included = true
        }
        //EDIT: ADDED GC CONTENT
        GCContent{
            git = 'https://github.com/cmdcolin/GCContent'
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
        //EDIT: ADDS ELASTIC SEARCH PLUGIN
        ElasticSearch{
            git = 'https://github.com/elsiklab/jbrowse_elasticsearch'
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
    }
}

From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 12:30:50 AM
Subject: Re: [apollo] elastic search in genome apollo

Can you post your apollo config file? 

Nathan

On Nov 14, 2018, at 6:45 PM, Michael Poeschla <[hidden email]> wrote:

Hi, 
I'm following the instructions from https://github.com/elsiklab/jbrowse_elasticsearch

we have elasticsearch and the middleware app.js running. I was able to execute generate-names.pl and this seemed to work correctly from the terminal output. The changes also got picked up in the tracklist.json for the genome. But there is no sign of the elasticsearch plugin in web apollo...added the plugin to the config.groovy, and we have another jbrowse plugin (GC content) working fine so not sure what's the problem here. 


Thanks for all of your help! :)
Michael 

From: "Nathan Dunn" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Tuesday, November 13, 2018 6:33:05 PM
Subject: Re: [apollo] elastic search in genome apollo


Michael,

What problems are you running it when adding the plugins? 

Nathan


On Nov 13, 2018, at 4:55 PM, Colin <[hidden email]> wrote:

Aside from changing your gene identifiers, I think the full text search from elasticsearch could help. I forwarded some comments here about that https://github.com/elsiklab/jbrowse_elasticsearch/issues/7

Potentially data that is loaded in Apollo backend could use text similarity %like% or full text sql queries but for normal jbrowse tracks that are not loaded in the sql database, the jbrowse_elasticsearch plugin seems reasonable.

-Colin

On Tue, Nov 13, 2018 at 7:35 AM Michael Poeschla <[hidden email]> wrote:
Hi,

has anyone implemented the jbrowse_elasticsearch plugin in Apollo? I'm running from the docker container with an nginx reverse proxy and having trouble adding this plugin. How would I add this to the docker image, with the middleware as well?

The reason we want to do this is the predicted gene model names we have in our tracks are not always intuitive so it's hard to find them by searching in Apollo (ie mTOR is ID'd as "serine/threonine-protein kinase mTOR", so if you just start typing mTOR you get no results). Wondering also if anyone else has encountered this problem and found good solutions, besides manipulating all of our tracks.

Thanks so much,
Michael



This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.




This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.






This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.




This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.




This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

Reply | Threaded
Open this post in threaded view
|

Re: elastic search in genome apollo

nathandunn
Thanks.  Also on Europe time for another 20 minutes.

Can you also post any other specifics of failure?

Have you installed elasticsearch via docker or manually ?

Nathan

On Nov 15, 2018, at 11:10 AM, Michael Poeschla <[hidden email]> wrote:

Here it is - sorry for the late replies, I'm on europe time.

environments {
    development {
    }
    test {
    }
    production {
        dataSource {
            dbCreate = "update"
            username = System.getenv("WEBAPOLLO_DB_USERNAME") ?: *
            password = System.getenv("WEBAPOLLO_DB_PASSWORD") ?: *

            driverClassName = "org.postgresql.Driver"
            dialect = "org.hibernate.dialect.PostgresPlusDialect"
            url = "jdbc:postgresql://${System.getenv("WEBAPOLLO_DB_HOST")?:"127.0.0.1"}/${System.getenv("WEBAPOLLO_DB_NAME")?:"apollo"}"

            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = false
                initialSize                   = 5
                maxActive                     = 50
                minIdle                       = 5
                maxIdle                       = 25
                maxWait                       = 10000
                maxAge                        = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis    = 60000
                validationQuery               = "SELECT 1"
                validationQueryTimeout        = 3
                validationInterval            = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
        dataSource_chado {
            dbCreate = "update"
            username = System.getenv("CHADO_DB_USERNAME") ?: *
            password = System.getenv("CHADO_DB_PASSWORD") ?: *

            driverClassName = "org.postgresql.Driver"
            dialect = "org.hibernate.dialect.PostgresPlusDialect"

            url = "jdbc:postgresql://${System.getenv("CHADO_DB_HOST")?:"127.0.0.1"}/${System.getenv("CHADO_DB_NAME")?:"chado"}"

            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = false
                initialSize                   = 5
                maxActive                     = 50
                minIdle                       = 5
                maxIdle                       = 25
                maxWait                       = 10000
                maxAge                        = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis    = 60000
                validationQuery               = "SELECT 1"
                validationQueryTimeout        = 3
                validationInterval            = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
    }
}

apollo {
    default_minimum_intron_size = System.getenv("WEBAPOLLO_MINIMUM_INTRON_SIZE") ? System.getenv("WEBAPOLLO_MINIMUM_INTRON_SIZE").toInteger() : 1
    history_size = System.getenv("WEBAPOLLO_HISTORY_SIZE") ? System.getenv("WEBAPOLLO_HISTORY_SIZE").toInteger() : 0
    overlapper_class = System.getenv("WEBAPOLLO_OVERLAPPER_CLASS") ?: "org.bbop.apollo.sequence.OrfOverlapper"
    use_cds_for_new_transcripts = System.getenv("WEBAPOLLO_CDS_FOR_NEW_TRANSCRIPTS").equals("true") // will default to false
    feature_has_dbxrefs = System.getenv("WEBAPOLLO_FEATURE_HAS_DBXREFS") ?: true
    feature_has_attributes = System.getenv("WEBAPOLLO_FEATURE_HAS_ATTRS") ?: true
    feature_has_pubmed_ids = System.getenv("WEBAPOLLO_FEATURE_HAS_PUBMED") ?: true
    feature_has_go_ids = System.getenv("WEBAPOLLO_FEATURE_HAS_GO") ?: true
    feature_has_comments = System.getenv("WEBAPOLLO_FEATURE_HAS_COMMENTS") ?: true
    feature_has_status = System.getenv("WEBAPOLLO_FEATURE_HAS_STATUS") ?: true
    translation_table = "/config/translation_tables/ncbi_" + (System.getenv("WEBAPOLLO_TRANSLATION_TABLE") ?: "1") + "_translation_table.txt"
    get_translation_code = System.getenv("WEBAPOLLO_TRANSLATION_TABLE") ? System.getenv("WEBAPOLLO_TRANSLATION_TABLE").toInteger() : 1

    // ADDED BLAT AND SET PATH TO BIN
    sequence_search_tools = [
        blat_nuc: [
          search_exe: "/usr/local/bin/blat",
          search_class: "org.bbop.apollo.sequence.search.blat.BlatCommandLineNucleotideToNucleotide",
          name: "Blat nucleotide",
          params: ""
        ],
        blat_prot: [
          search_exe: "/usr/local/bin/blat",
          search_class: "org.bbop.apollo.sequence.search.blat.BlatCommandLineProteinToNucleotide",
          name: "Blat protein",
          params: ""
        ]
      ]
    // TODO: should come from config or via preferences database
    splice_donor_sites = System.getenv("WEBAPOLLO_SPLICE_DONOR_SITES") ? System.getenv("WEBAPOLLO_SPLICE_DONOR_SITES").split(",") : ["GT"]
    splice_acceptor_sites = System.getenv("WEBAPOLLO_SPLICE_ACCEPTOR_SITES") ? System.getenv("WEBAPOLLO_SPLICE_ACCEPTOR_SITES").split(",") : ["AG"]
    gff3.source = System.getenv("WEBAPOLLO_GFF3_SOURCE") ?: "."

    google_analytics = System.getenv("WEBAPOLLO_GOOGLE_ANALYTICS_ID") ?: ["UA-62921593-1"]

    admin{
        username = System.getenv("APOLLO_ADMIN_EMAIL") ?: *
        password = System.getenv("APOLLO_ADMIN_PASSWORD") ?: *
        firstName = System.getenv("APOLLO_ADMIN_FIRST_NAME") ?: *
        lastName = System.getenv("APOLLO_ADMIN_LAST_NAME") ?: *
    }
}

jbrowse {
    git {
        url = "https://github.com/GMOD/jbrowse"
        tag = "maint/1.12.5-apollo"
    }
    plugins {
        WebApollo{
            included = true
        }
        NeatHTMLFeatures{
            included = System.getenv("WEBAPOLLO_JBROWSE_PLUGIN_NEATHTML") ?: true
        }
        NeatCanvasFeatures{
            included = System.getenv("WEBAPOLLO_JBROWSE_PLUGIN_NEATCANVAS") ?: true
        }
        RegexSequenceSearch{
            included = true
        }
        HideTrackLabels{
            included = true
        }
        //EDIT: ADDED GC CONTENT
        GCContent{
            git = 'https://github.com/cmdcolin/GCContent'
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
        //EDIT: ADDS ELASTIC SEARCH PLUGIN
        ElasticSearch{
            git = 'https://github.com/elsiklab/jbrowse_elasticsearch'
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
    }
}

From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 12:30:50 AM
Subject: Re: [apollo] elastic search in genome apollo

Can you post your apollo config file? 

Nathan

On Nov 14, 2018, at 6:45 PM, Michael Poeschla <[hidden email]> wrote:

Hi, 
I'm following the instructions from https://github.com/elsiklab/jbrowse_elasticsearch

we have elasticsearch and the middleware app.js running. I was able to execute generate-names.pl and this seemed to work correctly from the terminal output. The changes also got picked up in the tracklist.json for the genome. But there is no sign of the elasticsearch plugin in web apollo...added the plugin to the config.groovy, and we have another jbrowse plugin (GC content) working fine so not sure what's the problem here. 


Thanks for all of your help! :)
Michael 

From: "Nathan Dunn" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Tuesday, November 13, 2018 6:33:05 PM
Subject: Re: [apollo] elastic search in genome apollo


Michael,

What problems are you running it when adding the plugins? 

Nathan


On Nov 13, 2018, at 4:55 PM, Colin <[hidden email]> wrote:

Aside from changing your gene identifiers, I think the full text search from elasticsearch could help. I forwarded some comments here about that https://github.com/elsiklab/jbrowse_elasticsearch/issues/7

Potentially data that is loaded in Apollo backend could use text similarity %like% or full text sql queries but for normal jbrowse tracks that are not loaded in the sql database, the jbrowse_elasticsearch plugin seems reasonable.

-Colin

On Tue, Nov 13, 2018 at 7:35 AM Michael Poeschla <[hidden email]> wrote:
Hi,

has anyone implemented the jbrowse_elasticsearch plugin in Apollo? I'm running from the docker container with an nginx reverse proxy and having trouble adding this plugin. How would I add this to the docker image, with the middleware as well?

The reason we want to do this is the predicted gene model names we have in our tracks are not always intuitive so it's hard to find them by searching in Apollo (ie mTOR is ID'd as "serine/threonine-protein kinase mTOR", so if you just start typing mTOR you get no results). Wondering also if anyone else has encountered this problem and found good solutions, besides manipulating all of our tracks.

Thanks so much,
Michael



This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.




This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.






This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.




This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.



This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

Reply | Threaded
Open this post in threaded view
|

Re: elastic search in genome apollo

Michael Poeschla
hi,
elasticsearch was installed with docker. Here's the tracklist.json for the organism that we generated the index for. Is there any other info I could provide that would help?

{
   "elasticIndexName" : "nfz",
   "elasticSearchUrl" : "localhost:3000",
   "formatVersion" : 1,
   "names" : {
      "type" : "Hash",
      "url" : "names/"
   },
   "plugins" : [
      "ElasticSearch"
   ],
   "tracks" : [
      {
         "category" : "Reference sequence",
         "chunkSize" : 20000,
         "key" : "Reference sequence",
         "label" : "DNA",
         "seqType" : "dna",
         "storeClass" : "JBrowse/Store/Sequence/StaticChunked",
         "type" : "SequenceTrack",
         "urlTemplate" : "seq/{refseq_dirpath}/{refseq}-"
      },
      {
         "bicolor_pivot" : 0.5,
         "label" : "GC Content",
         "storeClass" : "JBrowse/Store/SeqFeature/SequenceChunks",
         "type" : "GCContent/View/Track/GCContentXY",
         "urlTemplate" : "seq/{refseq_dirpath}/{refseq}-"
      },
      {
         "compress" : 0,
         "key" : "Repeats",
         "label" : "Repeats",
         "storeClass" : "JBrowse/Store/SeqFeature/NCList",
         "style" : {
            "className" : "feature"
         },
         "trackType" : "JBrowse/View/Track/CanvasFeatures",
         "type" : "JBrowse/View/Track/CanvasFeatures",
         "urlTemplate" : "tracks/Repeats/{refseq}/trackData.json"
      },
      {
         "compress" : 0,
         "key" : "Annotation",
         "label" : "Annotationfromsumgff",
         "storeClass" : "JBrowse/Store/SeqFeature/NCList",
         "style" : {
            "className" : "feature"
         },
         "trackType" : "CanvasFeatures",
         "type" : "CanvasFeatures",
         "urlTemplate" : "tracks/Annotationfromsumgff/{refseq}/trackData.json"
      }
   ]
}


From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 11:33:03 AM
Subject: Re: [apollo] elastic search in genome apollo

Thanks.  Also on Europe time for another 20 minutes.
Can you also post any other specifics of failure?

Have you installed elasticsearch via docker or manually ?

Nathan

On Nov 15, 2018, at 11:10 AM, Michael Poeschla <[hidden email]> wrote:

Here it is - sorry for the late replies, I'm on europe time.

environments {
    development {
    }
    test {
    }
    production {
        dataSource {
            dbCreate = "update"
            username = System.getenv("WEBAPOLLO_DB_USERNAME") ?: *
            password = System.getenv("WEBAPOLLO_DB_PASSWORD") ?: *

            driverClassName = "org.postgresql.Driver"
            dialect = "org.hibernate.dialect.PostgresPlusDialect"
            url = "jdbc:postgresql://${System.getenv("WEBAPOLLO_DB_HOST")?:"127.0.0.1"}/${System.getenv("WEBAPOLLO_DB_NAME")?:"apollo"}"

            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = false
                initialSize                   = 5
                maxActive                     = 50
                minIdle                       = 5
                maxIdle                       = 25
                maxWait                       = 10000
                maxAge                        = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis    = 60000
                validationQuery               = "SELECT 1"
                validationQueryTimeout        = 3
                validationInterval            = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
        dataSource_chado {
            dbCreate = "update"
            username = System.getenv("CHADO_DB_USERNAME") ?: *
            password = System.getenv("CHADO_DB_PASSWORD") ?: *

            driverClassName = "org.postgresql.Driver"
            dialect = "org.hibernate.dialect.PostgresPlusDialect"

            url = "jdbc:postgresql://${System.getenv("CHADO_DB_HOST")?:"127.0.0.1"}/${System.getenv("CHADO_DB_NAME")?:"chado"}"

            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = false
                initialSize                   = 5
                maxActive                     = 50
                minIdle                       = 5
                maxIdle                       = 25
                maxWait                       = 10000
                maxAge                        = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis    = 60000
                validationQuery               = "SELECT 1"
                validationQueryTimeout        = 3
                validationInterval            = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
    }
}

apollo {
    default_minimum_intron_size = System.getenv("WEBAPOLLO_MINIMUM_INTRON_SIZE") ? System.getenv("WEBAPOLLO_MINIMUM_INTRON_SIZE").toInteger() : 1
    history_size = System.getenv("WEBAPOLLO_HISTORY_SIZE") ? System.getenv("WEBAPOLLO_HISTORY_SIZE").toInteger() : 0
    overlapper_class = System.getenv("WEBAPOLLO_OVERLAPPER_CLASS") ?: "org.bbop.apollo.sequence.OrfOverlapper"
    use_cds_for_new_transcripts = System.getenv("WEBAPOLLO_CDS_FOR_NEW_TRANSCRIPTS").equals("true") // will default to false
    feature_has_dbxrefs = System.getenv("WEBAPOLLO_FEATURE_HAS_DBXREFS") ?: true
    feature_has_attributes = System.getenv("WEBAPOLLO_FEATURE_HAS_ATTRS") ?: true
    feature_has_pubmed_ids = System.getenv("WEBAPOLLO_FEATURE_HAS_PUBMED") ?: true
    feature_has_go_ids = System.getenv("WEBAPOLLO_FEATURE_HAS_GO") ?: true
    feature_has_comments = System.getenv("WEBAPOLLO_FEATURE_HAS_COMMENTS") ?: true
    feature_has_status = System.getenv("WEBAPOLLO_FEATURE_HAS_STATUS") ?: true
    translation_table = "/config/translation_tables/ncbi_" + (System.getenv("WEBAPOLLO_TRANSLATION_TABLE") ?: "1") + "_translation_table.txt"
    get_translation_code = System.getenv("WEBAPOLLO_TRANSLATION_TABLE") ? System.getenv("WEBAPOLLO_TRANSLATION_TABLE").toInteger() : 1

    // ADDED BLAT AND SET PATH TO BIN
    sequence_search_tools = [
        blat_nuc: [
          search_exe: "/usr/local/bin/blat",
          search_class: "org.bbop.apollo.sequence.search.blat.BlatCommandLineNucleotideToNucleotide",
          name: "Blat nucleotide",
          params: ""
        ],
        blat_prot: [
          search_exe: "/usr/local/bin/blat",
          search_class: "org.bbop.apollo.sequence.search.blat.BlatCommandLineProteinToNucleotide",
          name: "Blat protein",
          params: ""
        ]
      ]
    // TODO: should come from config or via preferences database
    splice_donor_sites = System.getenv("WEBAPOLLO_SPLICE_DONOR_SITES") ? System.getenv("WEBAPOLLO_SPLICE_DONOR_SITES").split(",") : ["GT"]
    splice_acceptor_sites = System.getenv("WEBAPOLLO_SPLICE_ACCEPTOR_SITES") ? System.getenv("WEBAPOLLO_SPLICE_ACCEPTOR_SITES").split(",") : ["AG"]
    gff3.source = System.getenv("WEBAPOLLO_GFF3_SOURCE") ?: "."

    google_analytics = System.getenv("WEBAPOLLO_GOOGLE_ANALYTICS_ID") ?: ["UA-62921593-1"]

    admin{
        username = System.getenv("APOLLO_ADMIN_EMAIL") ?: *
        password = System.getenv("APOLLO_ADMIN_PASSWORD") ?: *
        firstName = System.getenv("APOLLO_ADMIN_FIRST_NAME") ?: *
        lastName = System.getenv("APOLLO_ADMIN_LAST_NAME") ?: *
    }
}

jbrowse {
    git {
        url = "https://github.com/GMOD/jbrowse"
        tag = "maint/1.12.5-apollo"
    }
    plugins {
        WebApollo{
            included = true
        }
        NeatHTMLFeatures{
            included = System.getenv("WEBAPOLLO_JBROWSE_PLUGIN_NEATHTML") ?: true
        }
        NeatCanvasFeatures{
            included = System.getenv("WEBAPOLLO_JBROWSE_PLUGIN_NEATCANVAS") ?: true
        }
        RegexSequenceSearch{
            included = true
        }
        HideTrackLabels{
            included = true
        }
        //EDIT: ADDED GC CONTENT
        GCContent{
            git = 'https://github.com/cmdcolin/GCContent'
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
        //EDIT: ADDS ELASTIC SEARCH PLUGIN
        ElasticSearch{
            git = 'https://github.com/elsiklab/jbrowse_elasticsearch'
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
    }
}

From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 12:30:50 AM
Subject: Re: [apollo] elastic search in genome apollo

Can you post your apollo config file? 

Nathan

On Nov 14, 2018, at 6:45 PM, Michael Poeschla <[hidden email]> wrote:

Hi, 
I'm following the instructions from https://github.com/elsiklab/jbrowse_elasticsearch

we have elasticsearch and the middleware app.js running. I was able to execute generate-names.pl and this seemed to work correctly from the terminal output. The changes also got picked up in the tracklist.json for the genome. But there is no sign of the elasticsearch plugin in web apollo...added the plugin to the config.groovy, and we have another jbrowse plugin (GC content) working fine so not sure what's the problem here. 


Thanks for all of your help! :)
Michael 

From: "Nathan Dunn" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Tuesday, November 13, 2018 6:33:05 PM
Subject: Re: [apollo] elastic search in genome apollo


Michael,

What problems are you running it when adding the plugins? 

Nathan


On Nov 13, 2018, at 4:55 PM, Colin <[hidden email]> wrote:

Aside from changing your gene identifiers, I think the full text search from elasticsearch could help. I forwarded some comments here about that https://github.com/elsiklab/jbrowse_elasticsearch/issues/7

Potentially data that is loaded in Apollo backend could use text similarity %like% or full text sql queries but for normal jbrowse tracks that are not loaded in the sql database, the jbrowse_elasticsearch plugin seems reasonable.

-Colin

On Tue, Nov 13, 2018 at 7:35 AM Michael Poeschla <[hidden email]> wrote:
Hi,

has anyone implemented the jbrowse_elasticsearch plugin in Apollo? I'm running from the docker container with an nginx reverse proxy and having trouble adding this plugin. How would I add this to the docker image, with the middleware as well?

The reason we want to do this is the predicted gene model names we have in our tracks are not always intuitive so it's hard to find them by searching in Apollo (ie mTOR is ID'd as "serine/threonine-protein kinase mTOR", so if you just start typing mTOR you get no results). Wondering also if anyone else has encountered this problem and found good solutions, besides manipulating all of our tracks.

Thanks so much,
Michael



This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.




This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.






This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.




This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.



This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.




This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

Reply | Threaded
Open this post in threaded view
|

Re: elastic search in genome apollo

Colin
So you are saying that the plugin is not being added e.g. you don't have a menu item for the ElasticSearch dialog in jbrowse?

-Colin

On Thu, Nov 15, 2018 at 9:07 AM Michael Poeschla <[hidden email]> wrote:
hi,
elasticsearch was installed with docker. Here's the tracklist.json for the organism that we generated the index for. Is there any other info I could provide that would help?

{
   "elasticIndexName" : "nfz",
   "elasticSearchUrl" : "localhost:3000",
   "formatVersion" : 1,
   "names" : {
      "type" : "Hash",
      "url" : "names/"
   },
   "plugins" : [
      "ElasticSearch"
   ],
   "tracks" : [
      {
         "category" : "Reference sequence",
         "chunkSize" : 20000,
         "key" : "Reference sequence",
         "label" : "DNA",
         "seqType" : "dna",
         "storeClass" : "JBrowse/Store/Sequence/StaticChunked",
         "type" : "SequenceTrack",
         "urlTemplate" : "seq/{refseq_dirpath}/{refseq}-"
      },
      {
         "bicolor_pivot" : 0.5,
         "label" : "GC Content",
         "storeClass" : "JBrowse/Store/SeqFeature/SequenceChunks",
         "type" : "GCContent/View/Track/GCContentXY",
         "urlTemplate" : "seq/{refseq_dirpath}/{refseq}-"
      },
      {
         "compress" : 0,
         "key" : "Repeats",
         "label" : "Repeats",
         "storeClass" : "JBrowse/Store/SeqFeature/NCList",
         "style" : {
            "className" : "feature"
         },
         "trackType" : "JBrowse/View/Track/CanvasFeatures",
         "type" : "JBrowse/View/Track/CanvasFeatures",
         "urlTemplate" : "tracks/Repeats/{refseq}/trackData.json"
      },
      {
         "compress" : 0,
         "key" : "Annotation",
         "label" : "Annotationfromsumgff",
         "storeClass" : "JBrowse/Store/SeqFeature/NCList",
         "style" : {
            "className" : "feature"
         },
         "trackType" : "CanvasFeatures",
         "type" : "CanvasFeatures",
         "urlTemplate" : "tracks/Annotationfromsumgff/{refseq}/trackData.json"
      }
   ]
}


From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 11:33:03 AM
Subject: Re: [apollo] elastic search in genome apollo

Thanks.  Also on Europe time for another 20 minutes.
Can you also post any other specifics of failure?

Have you installed elasticsearch via docker or manually ?

Nathan

On Nov 15, 2018, at 11:10 AM, Michael Poeschla <[hidden email]> wrote:

Here it is - sorry for the late replies, I'm on europe time.

environments {
    development {
    }
    test {
    }
    production {
        dataSource {
            dbCreate = "update"
            username = System.getenv("WEBAPOLLO_DB_USERNAME") ?: *
            password = System.getenv("WEBAPOLLO_DB_PASSWORD") ?: *

            driverClassName = "org.postgresql.Driver"
            dialect = "org.hibernate.dialect.PostgresPlusDialect"
            url = "jdbc:postgresql://${System.getenv("WEBAPOLLO_DB_HOST")?:"127.0.0.1"}/${System.getenv("WEBAPOLLO_DB_NAME")?:"apollo"}"

            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = false
                initialSize                   = 5
                maxActive                     = 50
                minIdle                       = 5
                maxIdle                       = 25
                maxWait                       = 10000
                maxAge                        = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis    = 60000
                validationQuery               = "SELECT 1"
                validationQueryTimeout        = 3
                validationInterval            = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
        dataSource_chado {
            dbCreate = "update"
            username = System.getenv("CHADO_DB_USERNAME") ?: *
            password = System.getenv("CHADO_DB_PASSWORD") ?: *

            driverClassName = "org.postgresql.Driver"
            dialect = "org.hibernate.dialect.PostgresPlusDialect"

            url = "jdbc:postgresql://${System.getenv("CHADO_DB_HOST")?:"127.0.0.1"}/${System.getenv("CHADO_DB_NAME")?:"chado"}"

            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = false
                initialSize                   = 5
                maxActive                     = 50
                minIdle                       = 5
                maxIdle                       = 25
                maxWait                       = 10000
                maxAge                        = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis    = 60000
                validationQuery               = "SELECT 1"
                validationQueryTimeout        = 3
                validationInterval            = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
    }
}

apollo {
    default_minimum_intron_size = System.getenv("WEBAPOLLO_MINIMUM_INTRON_SIZE") ? System.getenv("WEBAPOLLO_MINIMUM_INTRON_SIZE").toInteger() : 1
    history_size = System.getenv("WEBAPOLLO_HISTORY_SIZE") ? System.getenv("WEBAPOLLO_HISTORY_SIZE").toInteger() : 0
    overlapper_class = System.getenv("WEBAPOLLO_OVERLAPPER_CLASS") ?: "org.bbop.apollo.sequence.OrfOverlapper"
    use_cds_for_new_transcripts = System.getenv("WEBAPOLLO_CDS_FOR_NEW_TRANSCRIPTS").equals("true") // will default to false
    feature_has_dbxrefs = System.getenv("WEBAPOLLO_FEATURE_HAS_DBXREFS") ?: true
    feature_has_attributes = System.getenv("WEBAPOLLO_FEATURE_HAS_ATTRS") ?: true
    feature_has_pubmed_ids = System.getenv("WEBAPOLLO_FEATURE_HAS_PUBMED") ?: true
    feature_has_go_ids = System.getenv("WEBAPOLLO_FEATURE_HAS_GO") ?: true
    feature_has_comments = System.getenv("WEBAPOLLO_FEATURE_HAS_COMMENTS") ?: true
    feature_has_status = System.getenv("WEBAPOLLO_FEATURE_HAS_STATUS") ?: true
    translation_table = "/config/translation_tables/ncbi_" + (System.getenv("WEBAPOLLO_TRANSLATION_TABLE") ?: "1") + "_translation_table.txt"
    get_translation_code = System.getenv("WEBAPOLLO_TRANSLATION_TABLE") ? System.getenv("WEBAPOLLO_TRANSLATION_TABLE").toInteger() : 1

    // ADDED BLAT AND SET PATH TO BIN
    sequence_search_tools = [
        blat_nuc: [
          search_exe: "/usr/local/bin/blat",
          search_class: "org.bbop.apollo.sequence.search.blat.BlatCommandLineNucleotideToNucleotide",
          name: "Blat nucleotide",
          params: ""
        ],
        blat_prot: [
          search_exe: "/usr/local/bin/blat",
          search_class: "org.bbop.apollo.sequence.search.blat.BlatCommandLineProteinToNucleotide",
          name: "Blat protein",
          params: ""
        ]
      ]
    // TODO: should come from config or via preferences database
    splice_donor_sites = System.getenv("WEBAPOLLO_SPLICE_DONOR_SITES") ? System.getenv("WEBAPOLLO_SPLICE_DONOR_SITES").split(",") : ["GT"]
    splice_acceptor_sites = System.getenv("WEBAPOLLO_SPLICE_ACCEPTOR_SITES") ? System.getenv("WEBAPOLLO_SPLICE_ACCEPTOR_SITES").split(",") : ["AG"]
    gff3.source = System.getenv("WEBAPOLLO_GFF3_SOURCE") ?: "."

    google_analytics = System.getenv("WEBAPOLLO_GOOGLE_ANALYTICS_ID") ?: ["UA-62921593-1"]

    admin{
        username = System.getenv("APOLLO_ADMIN_EMAIL") ?: *
        password = System.getenv("APOLLO_ADMIN_PASSWORD") ?: *
        firstName = System.getenv("APOLLO_ADMIN_FIRST_NAME") ?: *
        lastName = System.getenv("APOLLO_ADMIN_LAST_NAME") ?: *
    }
}

jbrowse {
    git {
        url = "https://github.com/GMOD/jbrowse"
        tag = "maint/1.12.5-apollo"
    }
    plugins {
        WebApollo{
            included = true
        }
        NeatHTMLFeatures{
            included = System.getenv("WEBAPOLLO_JBROWSE_PLUGIN_NEATHTML") ?: true
        }
        NeatCanvasFeatures{
            included = System.getenv("WEBAPOLLO_JBROWSE_PLUGIN_NEATCANVAS") ?: true
        }
        RegexSequenceSearch{
            included = true
        }
        HideTrackLabels{
            included = true
        }
        //EDIT: ADDED GC CONTENT
        GCContent{
            git = 'https://github.com/cmdcolin/GCContent'
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
        //EDIT: ADDS ELASTIC SEARCH PLUGIN
        ElasticSearch{
            git = 'https://github.com/elsiklab/jbrowse_elasticsearch'
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
    }
}

From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 12:30:50 AM
Subject: Re: [apollo] elastic search in genome apollo

Can you post your apollo config file? 

Nathan

On Nov 14, 2018, at 6:45 PM, Michael Poeschla <[hidden email]> wrote:

Hi, 
I'm following the instructions from https://github.com/elsiklab/jbrowse_elasticsearch

we have elasticsearch and the middleware app.js running. I was able to execute generate-names.pl and this seemed to work correctly from the terminal output. The changes also got picked up in the tracklist.json for the genome. But there is no sign of the elasticsearch plugin in web apollo...added the plugin to the config.groovy, and we have another jbrowse plugin (GC content) working fine so not sure what's the problem here. 


Thanks for all of your help! :)
Michael 

From: "Nathan Dunn" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Tuesday, November 13, 2018 6:33:05 PM
Subject: Re: [apollo] elastic search in genome apollo


Michael,

What problems are you running it when adding the plugins? 

Nathan


On Nov 13, 2018, at 4:55 PM, Colin <[hidden email]> wrote:

Aside from changing your gene identifiers, I think the full text search from elasticsearch could help. I forwarded some comments here about that https://github.com/elsiklab/jbrowse_elasticsearch/issues/7

Potentially data that is loaded in Apollo backend could use text similarity %like% or full text sql queries but for normal jbrowse tracks that are not loaded in the sql database, the jbrowse_elasticsearch plugin seems reasonable.

-Colin

On Tue, Nov 13, 2018 at 7:35 AM Michael Poeschla <[hidden email]> wrote:
Hi,

has anyone implemented the jbrowse_elasticsearch plugin in Apollo? I'm running from the docker container with an nginx reverse proxy and having trouble adding this plugin. How would I add this to the docker image, with the middleware as well?

The reason we want to do this is the predicted gene model names we have in our tracks are not always intuitive so it's hard to find them by searching in Apollo (ie mTOR is ID'd as "serine/threonine-protein kinase mTOR", so if you just start typing mTOR you get no results). Wondering also if anyone else has encountered this problem and found good solutions, besides manipulating all of our tracks.

Thanks so much,
Michael



This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.




This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.






This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.




This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.



This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.



This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

Reply | Threaded
Open this post in threaded view
|

Re: elastic search in genome apollo

Michael Poeschla
yes - we don't see anything in the browser 
Thanks,
Michael

From: "Colin" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 7:09:45 PM
Subject: Re: [apollo] elastic search in genome apollo

So you are saying that the plugin is not being added e.g. you don't have a menu item for the ElasticSearch dialog in jbrowse?
-Colin

On Thu, Nov 15, 2018 at 9:07 AM Michael Poeschla <[hidden email]> wrote:
hi,
elasticsearch was installed with docker. Here's the tracklist.json for the organism that we generated the index for. Is there any other info I could provide that would help?

{
   "elasticIndexName" : "nfz",
   "elasticSearchUrl" : "localhost:3000",
   "formatVersion" : 1,
   "names" : {
      "type" : "Hash",
      "url" : "names/"
   },
   "plugins" : [
      "ElasticSearch"
   ],
   "tracks" : [
      {
         "category" : "Reference sequence",
         "chunkSize" : 20000,
         "key" : "Reference sequence",
         "label" : "DNA",
         "seqType" : "dna",
         "storeClass" : "JBrowse/Store/Sequence/StaticChunked",
         "type" : "SequenceTrack",
         "urlTemplate" : "seq/{refseq_dirpath}/{refseq}-"
      },
      {
         "bicolor_pivot" : 0.5,
         "label" : "GC Content",
         "storeClass" : "JBrowse/Store/SeqFeature/SequenceChunks",
         "type" : "GCContent/View/Track/GCContentXY",
         "urlTemplate" : "seq/{refseq_dirpath}/{refseq}-"
      },
      {
         "compress" : 0,
         "key" : "Repeats",
         "label" : "Repeats",
         "storeClass" : "JBrowse/Store/SeqFeature/NCList",
         "style" : {
            "className" : "feature"
         },
         "trackType" : "JBrowse/View/Track/CanvasFeatures",
         "type" : "JBrowse/View/Track/CanvasFeatures",
         "urlTemplate" : "tracks/Repeats/{refseq}/trackData.json"
      },
      {
         "compress" : 0,
         "key" : "Annotation",
         "label" : "Annotationfromsumgff",
         "storeClass" : "JBrowse/Store/SeqFeature/NCList",
         "style" : {
            "className" : "feature"
         },
         "trackType" : "CanvasFeatures",
         "type" : "CanvasFeatures",
         "urlTemplate" : "tracks/Annotationfromsumgff/{refseq}/trackData.json"
      }
   ]
}


From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 11:33:03 AM
Subject: Re: [apollo] elastic search in genome apollo

Thanks.  Also on Europe time for another 20 minutes.
Can you also post any other specifics of failure?

Have you installed elasticsearch via docker or manually ?

Nathan

On Nov 15, 2018, at 11:10 AM, Michael Poeschla <[hidden email]> wrote:

Here it is - sorry for the late replies, I'm on europe time.

environments {
    development {
    }
    test {
    }
    production {
        dataSource {
            dbCreate = "update"
            username = System.getenv("WEBAPOLLO_DB_USERNAME") ?: *
            password = System.getenv("WEBAPOLLO_DB_PASSWORD") ?: *

            driverClassName = "org.postgresql.Driver"
            dialect = "org.hibernate.dialect.PostgresPlusDialect"
            url = "jdbc:postgresql://${System.getenv("WEBAPOLLO_DB_HOST")?:"127.0.0.1"}/${System.getenv("WEBAPOLLO_DB_NAME")?:"apollo"}"

            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = false
                initialSize                   = 5
                maxActive                     = 50
                minIdle                       = 5
                maxIdle                       = 25
                maxWait                       = 10000
                maxAge                        = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis    = 60000
                validationQuery               = "SELECT 1"
                validationQueryTimeout        = 3
                validationInterval            = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
        dataSource_chado {
            dbCreate = "update"
            username = System.getenv("CHADO_DB_USERNAME") ?: *
            password = System.getenv("CHADO_DB_PASSWORD") ?: *

            driverClassName = "org.postgresql.Driver"
            dialect = "org.hibernate.dialect.PostgresPlusDialect"

            url = "jdbc:postgresql://${System.getenv("CHADO_DB_HOST")?:"127.0.0.1"}/${System.getenv("CHADO_DB_NAME")?:"chado"}"

            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = false
                initialSize                   = 5
                maxActive                     = 50
                minIdle                       = 5
                maxIdle                       = 25
                maxWait                       = 10000
                maxAge                        = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis    = 60000
                validationQuery               = "SELECT 1"
                validationQueryTimeout        = 3
                validationInterval            = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
    }
}

apollo {
    default_minimum_intron_size = System.getenv("WEBAPOLLO_MINIMUM_INTRON_SIZE") ? System.getenv("WEBAPOLLO_MINIMUM_INTRON_SIZE").toInteger() : 1
    history_size = System.getenv("WEBAPOLLO_HISTORY_SIZE") ? System.getenv("WEBAPOLLO_HISTORY_SIZE").toInteger() : 0
    overlapper_class = System.getenv("WEBAPOLLO_OVERLAPPER_CLASS") ?: "org.bbop.apollo.sequence.OrfOverlapper"
    use_cds_for_new_transcripts = System.getenv("WEBAPOLLO_CDS_FOR_NEW_TRANSCRIPTS").equals("true") // will default to false
    feature_has_dbxrefs = System.getenv("WEBAPOLLO_FEATURE_HAS_DBXREFS") ?: true
    feature_has_attributes = System.getenv("WEBAPOLLO_FEATURE_HAS_ATTRS") ?: true
    feature_has_pubmed_ids = System.getenv("WEBAPOLLO_FEATURE_HAS_PUBMED") ?: true
    feature_has_go_ids = System.getenv("WEBAPOLLO_FEATURE_HAS_GO") ?: true
    feature_has_comments = System.getenv("WEBAPOLLO_FEATURE_HAS_COMMENTS") ?: true
    feature_has_status = System.getenv("WEBAPOLLO_FEATURE_HAS_STATUS") ?: true
    translation_table = "/config/translation_tables/ncbi_" + (System.getenv("WEBAPOLLO_TRANSLATION_TABLE") ?: "1") + "_translation_table.txt"
    get_translation_code = System.getenv("WEBAPOLLO_TRANSLATION_TABLE") ? System.getenv("WEBAPOLLO_TRANSLATION_TABLE").toInteger() : 1

    // ADDED BLAT AND SET PATH TO BIN
    sequence_search_tools = [
        blat_nuc: [
          search_exe: "/usr/local/bin/blat",
          search_class: "org.bbop.apollo.sequence.search.blat.BlatCommandLineNucleotideToNucleotide",
          name: "Blat nucleotide",
          params: ""
        ],
        blat_prot: [
          search_exe: "/usr/local/bin/blat",
          search_class: "org.bbop.apollo.sequence.search.blat.BlatCommandLineProteinToNucleotide",
          name: "Blat protein",
          params: ""
        ]
      ]
    // TODO: should come from config or via preferences database
    splice_donor_sites = System.getenv("WEBAPOLLO_SPLICE_DONOR_SITES") ? System.getenv("WEBAPOLLO_SPLICE_DONOR_SITES").split(",") : ["GT"]
    splice_acceptor_sites = System.getenv("WEBAPOLLO_SPLICE_ACCEPTOR_SITES") ? System.getenv("WEBAPOLLO_SPLICE_ACCEPTOR_SITES").split(",") : ["AG"]
    gff3.source = System.getenv("WEBAPOLLO_GFF3_SOURCE") ?: "."

    google_analytics = System.getenv("WEBAPOLLO_GOOGLE_ANALYTICS_ID") ?: ["UA-62921593-1"]

    admin{
        username = System.getenv("APOLLO_ADMIN_EMAIL") ?: *
        password = System.getenv("APOLLO_ADMIN_PASSWORD") ?: *
        firstName = System.getenv("APOLLO_ADMIN_FIRST_NAME") ?: *
        lastName = System.getenv("APOLLO_ADMIN_LAST_NAME") ?: *
    }
}

jbrowse {
    git {
        url = "https://github.com/GMOD/jbrowse"
        tag = "maint/1.12.5-apollo"
    }
    plugins {
        WebApollo{
            included = true
        }
        NeatHTMLFeatures{
            included = System.getenv("WEBAPOLLO_JBROWSE_PLUGIN_NEATHTML") ?: true
        }
        NeatCanvasFeatures{
            included = System.getenv("WEBAPOLLO_JBROWSE_PLUGIN_NEATCANVAS") ?: true
        }
        RegexSequenceSearch{
            included = true
        }
        HideTrackLabels{
            included = true
        }
        //EDIT: ADDED GC CONTENT
        GCContent{
            git = 'https://github.com/cmdcolin/GCContent'
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
        //EDIT: ADDS ELASTIC SEARCH PLUGIN
        ElasticSearch{
            git = 'https://github.com/elsiklab/jbrowse_elasticsearch'
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
    }
}

From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 12:30:50 AM
Subject: Re: [apollo] elastic search in genome apollo

Can you post your apollo config file? 

Nathan

On Nov 14, 2018, at 6:45 PM, Michael Poeschla <[hidden email]> wrote:

Hi, 
I'm following the instructions from https://github.com/elsiklab/jbrowse_elasticsearch

we have elasticsearch and the middleware app.js running. I was able to execute generate-names.pl and this seemed to work correctly from the terminal output. The changes also got picked up in the tracklist.json for the genome. But there is no sign of the elasticsearch plugin in web apollo...added the plugin to the config.groovy, and we have another jbrowse plugin (GC content) working fine so not sure what's the problem here. 


Thanks for all of your help! :)
Michael 

From: "Nathan Dunn" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Tuesday, November 13, 2018 6:33:05 PM
Subject: Re: [apollo] elastic search in genome apollo


Michael,

What problems are you running it when adding the plugins? 

Nathan


On Nov 13, 2018, at 4:55 PM, Colin <[hidden email]> wrote:

Aside from changing your gene identifiers, I think the full text search from elasticsearch could help. I forwarded some comments here about that https://github.com/elsiklab/jbrowse_elasticsearch/issues/7

Potentially data that is loaded in Apollo backend could use text similarity %like% or full text sql queries but for normal jbrowse tracks that are not loaded in the sql database, the jbrowse_elasticsearch plugin seems reasonable.

-Colin

On Tue, Nov 13, 2018 at 7:35 AM Michael Poeschla <[hidden email]> wrote:
Hi,

has anyone implemented the jbrowse_elasticsearch plugin in Apollo? I'm running from the docker container with an nginx reverse proxy and having trouble adding this plugin. How would I add this to the docker image, with the middleware as well?

The reason we want to do this is the predicted gene model names we have in our tracks are not always intuitive so it's hard to find them by searching in Apollo (ie mTOR is ID'd as "serine/threonine-protein kinase mTOR", so if you just start typing mTOR you get no results). Wondering also if anyone else has encountered this problem and found good solutions, besides manipulating all of our tracks.

Thanks so much,
Michael



This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.




This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.






This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.




This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.



This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.



This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.




This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

Reply | Threaded
Open this post in threaded view
|

Re: elastic search in genome apollo

nathandunn
Sorry for the late response. 

Can you provide the browser console logs and any tomcat logs (catalina.out)? 

Your modified docket file may be helpful as well.

Nathan

On Nov 15, 2018, at 10:26 AM, Michael Poeschla <[hidden email]> wrote:

yes - we don't see anything in the browser 
Thanks,
Michael

From: "Colin" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 7:09:45 PM
Subject: Re: [apollo] elastic search in genome apollo

So you are saying that the plugin is not being added e.g. you don't have a menu item for the ElasticSearch dialog in jbrowse?
-Colin

On Thu, Nov 15, 2018 at 9:07 AM Michael Poeschla <[hidden email]> wrote:
hi,
elasticsearch was installed with docker. Here's the tracklist.json for the organism that we generated the index for. Is there any other info I could provide that would help?

{
   "elasticIndexName" : "nfz",
   "elasticSearchUrl" : "localhost:3000",
   "formatVersion" : 1,
   "names" : {
      "type" : "Hash",
      "url" : "names/"
   },
   "plugins" : [
      "ElasticSearch"
   ],
   "tracks" : [
      {
         "category" : "Reference sequence",
         "chunkSize" : 20000,
         "key" : "Reference sequence",
         "label" : "DNA",
         "seqType" : "dna",
         "storeClass" : "JBrowse/Store/Sequence/StaticChunked",
         "type" : "SequenceTrack",
         "urlTemplate" : "seq/{refseq_dirpath}/{refseq}-"
      },
      {
         "bicolor_pivot" : 0.5,
         "label" : "GC Content",
         "storeClass" : "JBrowse/Store/SeqFeature/SequenceChunks",
         "type" : "GCContent/View/Track/GCContentXY",
         "urlTemplate" : "seq/{refseq_dirpath}/{refseq}-"
      },
      {
         "compress" : 0,
         "key" : "Repeats",
         "label" : "Repeats",
         "storeClass" : "JBrowse/Store/SeqFeature/NCList",
         "style" : {
            "className" : "feature"
         },
         "trackType" : "JBrowse/View/Track/CanvasFeatures",
         "type" : "JBrowse/View/Track/CanvasFeatures",
         "urlTemplate" : "tracks/Repeats/{refseq}/trackData.json"
      },
      {
         "compress" : 0,
         "key" : "Annotation",
         "label" : "Annotationfromsumgff",
         "storeClass" : "JBrowse/Store/SeqFeature/NCList",
         "style" : {
            "className" : "feature"
         },
         "trackType" : "CanvasFeatures",
         "type" : "CanvasFeatures",
         "urlTemplate" : "tracks/Annotationfromsumgff/{refseq}/trackData.json"
      }
   ]
}


From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 11:33:03 AM
Subject: Re: [apollo] elastic search in genome apollo

Thanks.  Also on Europe time for another 20 minutes.
Can you also post any other specifics of failure?

Have you installed elasticsearch via docker or manually ?

Nathan

On Nov 15, 2018, at 11:10 AM, Michael Poeschla <[hidden email]> wrote:

Here it is - sorry for the late replies, I'm on europe time.

environments {
    development {
    }
    test {
    }
    production {
        dataSource {
            dbCreate = "update"
            username = System.getenv("WEBAPOLLO_DB_USERNAME") ?: *
            password = System.getenv("WEBAPOLLO_DB_PASSWORD") ?: *

            driverClassName = "org.postgresql.Driver"
            dialect = "org.hibernate.dialect.PostgresPlusDialect"
            url = "jdbc:postgresql://${System.getenv("WEBAPOLLO_DB_HOST")?:"127.0.0.1"}/${System.getenv("WEBAPOLLO_DB_NAME")?:"apollo"}"

            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = false
                initialSize                   = 5
                maxActive                     = 50
                minIdle                       = 5
                maxIdle                       = 25
                maxWait                       = 10000
                maxAge                        = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis    = 60000
                validationQuery               = "SELECT 1"
                validationQueryTimeout        = 3
                validationInterval            = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
        dataSource_chado {
            dbCreate = "update"
            username = System.getenv("CHADO_DB_USERNAME") ?: *
            password = System.getenv("CHADO_DB_PASSWORD") ?: *

            driverClassName = "org.postgresql.Driver"
            dialect = "org.hibernate.dialect.PostgresPlusDialect"

            url = "jdbc:postgresql://${System.getenv("CHADO_DB_HOST")?:"127.0.0.1"}/${System.getenv("CHADO_DB_NAME")?:"chado"}"

            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = false
                initialSize                   = 5
                maxActive                     = 50
                minIdle                       = 5
                maxIdle                       = 25
                maxWait                       = 10000
                maxAge                        = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis    = 60000
                validationQuery               = "SELECT 1"
                validationQueryTimeout        = 3
                validationInterval            = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
    }
}

apollo {
    default_minimum_intron_size = System.getenv("WEBAPOLLO_MINIMUM_INTRON_SIZE") ? System.getenv("WEBAPOLLO_MINIMUM_INTRON_SIZE").toInteger() : 1
    history_size = System.getenv("WEBAPOLLO_HISTORY_SIZE") ? System.getenv("WEBAPOLLO_HISTORY_SIZE").toInteger() : 0
    overlapper_class = System.getenv("WEBAPOLLO_OVERLAPPER_CLASS") ?: "org.bbop.apollo.sequence.OrfOverlapper"
    use_cds_for_new_transcripts = System.getenv("WEBAPOLLO_CDS_FOR_NEW_TRANSCRIPTS").equals("true") // will default to false
    feature_has_dbxrefs = System.getenv("WEBAPOLLO_FEATURE_HAS_DBXREFS") ?: true
    feature_has_attributes = System.getenv("WEBAPOLLO_FEATURE_HAS_ATTRS") ?: true
    feature_has_pubmed_ids = System.getenv("WEBAPOLLO_FEATURE_HAS_PUBMED") ?: true
    feature_has_go_ids = System.getenv("WEBAPOLLO_FEATURE_HAS_GO") ?: true
    feature_has_comments = System.getenv("WEBAPOLLO_FEATURE_HAS_COMMENTS") ?: true
    feature_has_status = System.getenv("WEBAPOLLO_FEATURE_HAS_STATUS") ?: true
    translation_table = "/config/translation_tables/ncbi_" + (System.getenv("WEBAPOLLO_TRANSLATION_TABLE") ?: "1") + "_translation_table.txt"
    get_translation_code = System.getenv("WEBAPOLLO_TRANSLATION_TABLE") ? System.getenv("WEBAPOLLO_TRANSLATION_TABLE").toInteger() : 1

    // ADDED BLAT AND SET PATH TO BIN
    sequence_search_tools = [
        blat_nuc: [
          search_exe: "/usr/local/bin/blat",
          search_class: "org.bbop.apollo.sequence.search.blat.BlatCommandLineNucleotideToNucleotide",
          name: "Blat nucleotide",
          params: ""
        ],
        blat_prot: [
          search_exe: "/usr/local/bin/blat",
          search_class: "org.bbop.apollo.sequence.search.blat.BlatCommandLineProteinToNucleotide",
          name: "Blat protein",
          params: ""
        ]
      ]
    // TODO: should come from config or via preferences database
    splice_donor_sites = System.getenv("WEBAPOLLO_SPLICE_DONOR_SITES") ? System.getenv("WEBAPOLLO_SPLICE_DONOR_SITES").split(",") : ["GT"]
    splice_acceptor_sites = System.getenv("WEBAPOLLO_SPLICE_ACCEPTOR_SITES") ? System.getenv("WEBAPOLLO_SPLICE_ACCEPTOR_SITES").split(",") : ["AG"]
    gff3.source = System.getenv("WEBAPOLLO_GFF3_SOURCE") ?: "."

    google_analytics = System.getenv("WEBAPOLLO_GOOGLE_ANALYTICS_ID") ?: ["UA-62921593-1"]

    admin{
        username = System.getenv("APOLLO_ADMIN_EMAIL") ?: *
        password = System.getenv("APOLLO_ADMIN_PASSWORD") ?: *
        firstName = System.getenv("APOLLO_ADMIN_FIRST_NAME") ?: *
        lastName = System.getenv("APOLLO_ADMIN_LAST_NAME") ?: *
    }
}

jbrowse {
    git {
        url = "https://github.com/GMOD/jbrowse"
        tag = "maint/1.12.5-apollo"
    }
    plugins {
        WebApollo{
            included = true
        }
        NeatHTMLFeatures{
            included = System.getenv("WEBAPOLLO_JBROWSE_PLUGIN_NEATHTML") ?: true
        }
        NeatCanvasFeatures{
            included = System.getenv("WEBAPOLLO_JBROWSE_PLUGIN_NEATCANVAS") ?: true
        }
        RegexSequenceSearch{
            included = true
        }
        HideTrackLabels{
            included = true
        }
        //EDIT: ADDED GC CONTENT
        GCContent{
            git = 'https://github.com/cmdcolin/GCContent'
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
        //EDIT: ADDS ELASTIC SEARCH PLUGIN
        ElasticSearch{
            git = 'https://github.com/elsiklab/jbrowse_elasticsearch'
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
    }
}

From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 12:30:50 AM
Subject: Re: [apollo] elastic search in genome apollo

Can you post your apollo config file? 

Nathan

On Nov 14, 2018, at 6:45 PM, Michael Poeschla <[hidden email]> wrote:

Hi, 
I'm following the instructions from https://github.com/elsiklab/jbrowse_elasticsearch

we have elasticsearch and the middleware app.js running. I was able to execute generate-names.pl and this seemed to work correctly from the terminal output. The changes also got picked up in the tracklist.json for the genome. But there is no sign of the elasticsearch plugin in web apollo...added the plugin to the config.groovy, and we have another jbrowse plugin (GC content) working fine so not sure what's the problem here. 


Thanks for all of your help! :)
Michael 

From: "Nathan Dunn" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Tuesday, November 13, 2018 6:33:05 PM
Subject: Re: [apollo] elastic search in genome apollo


Michael,

What problems are you running it when adding the plugins? 

Nathan


On Nov 13, 2018, at 4:55 PM, Colin <[hidden email]> wrote:

Aside from changing your gene identifiers, I think the full text search from elasticsearch could help. I forwarded some comments here about that https://github.com/elsiklab/jbrowse_elasticsearch/issues/7

Potentially data that is loaded in Apollo backend could use text similarity %like% or full text sql queries but for normal jbrowse tracks that are not loaded in the sql database, the jbrowse_elasticsearch plugin seems reasonable.

-Colin

On Tue, Nov 13, 2018 at 7:35 AM Michael Poeschla <[hidden email]> wrote:
Hi,

has anyone implemented the jbrowse_elasticsearch plugin in Apollo? I'm running from the docker container with an nginx reverse proxy and having trouble adding this plugin. How would I add this to the docker image, with the middleware as well?

The reason we want to do this is the predicted gene model names we have in our tracks are not always intuitive so it's hard to find them by searching in Apollo (ie mTOR is ID'd as "serine/threonine-protein kinase mTOR", so if you just start typing mTOR you get no results). Wondering also if anyone else has encountered this problem and found good solutions, besides manipulating all of our tracks.

Thanks so much,
Michael



This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.




This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.






This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.




This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.



This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.



This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.



This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

Reply | Threaded
Open this post in threaded view
|

Re: elastic search in genome apollo

Michael Poeschla
Hi,

here are the logs, docker-compose, actual dockerfile and start and testing of npm (below). I've also added the tomcat logs as an attachment.

docker-compose:
--
version: "3.2"
services:

  apollo:
    volumes:
      - ./data/extdata/:/data
      - postgres-data:/var/lib/postgresql
      - ./config/tomcat/server.xml:/var/lib/tomcat8/conf/server.xml:ro
      - ./config/tomcat/setenv.sh:/usr/share/tomcat8/bin/setenv.sh
    environment:
      APOLLO_ADMIN_[hidden email]
    restart: always
    links:
      - elasticsearch:elasticsearch

  nginx:
    image: library/nginx:alpine
    restart: always
    volumes:
      - ./config/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
      - ./config/secrets:/secrets:ro
    ports:
      - 80:80
      - 443:443
    links:
      - apollo:apollo

  elasticsearch:
    image: elasticsearch:6.4.2
    container_name: elasticsearch
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata1:/usr/share/elasticsearch/data

volumes:
  postgres-data:
  esdata1:
    driver: local
--
The actual Dockerfile:
--
# WebApollo
# VERSION 2.1.X
FROM tomcat:8-jre8
MAINTAINER Eric Rasche <[hidden email]>, Anthony Bretaudeau <[hidden email]>, Nathan Dunn <[hidden email]>
ENV DEBIAN_FRONTEND noninteractive 

RUN apt-get -qq update --fix-missing && \
apt-get --no-install-recommends -y install \
git build-essential maven tomcat8 libpq-dev postgresql-common openjdk-8-jdk wget \
postgresql postgresql-client xmlstarlet netcat libpng-dev \
zlib1g-dev libexpat1-dev ant curl ssl-cert zip unzip

RUN apt-get -qq update --fix-missing && \
apt-get --no-install-recommends -y install nodejs && \
apt-get autoremove -y && apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

RUN cp /usr/lib/jvm/java-8-openjdk-amd64/lib/tools.jar /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext/tools.jar && \
useradd -ms /bin/bash -d /apollo apollo

ENV WEBAPOLLO_VERSION 99c7e54c6e74fbc6705d57de216c9b14b2bfb03b
RUN curl -L <a href="https://github.com/GMOD/Apollo/archive/${WEBAPOLLO_VERSION}.tar.gz" class="" target="_blank" data-mce-href="https://github.com/GMOD/Apollo/archive/${WEBAPOLLO_VERSION}.tar.gz">https://github.com/GMOD/Apollo/archive/${WEBAPOLLO_VERSION}.tar.gz | tar xzf - --strip-components=1 -C /apollo

# install grails
COPY build.sh /bin/build.sh
ADD apollo-config.groovy /apollo/apollo-config.groovy

RUN chown -R apollo:apollo /apollo
RUN chmod +x /usr/local/bin/blat


USER apollo
RUN curl -s get.sdkman.io | bash
RUN /bin/bash -c "source $HOME/.sdkman/bin/sdkman-init.sh && yes | sdk install grails 2.5.5"
RUN /bin/bash -c "source $HOME/.sdkman/bin/sdkman-init.sh && yes | sdk install gradle 3.2.1"


RUN /bin/bash -c "source $HOME/.profile && source $HOME/.sdkman/bin/sdkman-init.sh && /bin/bash /bin/build.sh"

USER root
ENV CATALINA_HOME=/var/lib/tomcat8
RUN rm -rf ${CATALINA_HOME}/webapps/* && \
cp /apollo/apollo*.war ${CATALINA_HOME}/apollo.war

ENV CONTEXT_PATH ROOT

# Download chado schema
gunzip /chado.sql.gz


ADD launch.sh /launch.sh
CMD "/launch.sh"
--
Starting npm inside the container:
root@995955a711e1:/var/lib/tomcat8/webapps/ROOT/jbrowse/plugins/ElasticSearch# npm start

> jbrowse_elasticsearch@1.0.4 start /var/lib/tomcat8/webapps/ROOT/jbrowse/plugins/ElasticSearch
> node ./bin/www

testing access with curl:
root@995955a711e1:/usr/local/tomcat# curl http://localhost:3000
{"hits":[],"total":0}root@995955a711e1:/usr/local/tomcat#

content of ElasticSearchPlugin:
 ^Croot@995955a711e1:/var/lib/tomcat8/webapps/ROOT/jbrowse/plugins/ElasticSearch# ls -la
total 68
drwxr-x--- 10 tomcat8 tomcat8   271 Nov 14 17:10 .
drwxr-x--- 12 tomcat8 tomcat8   223 Nov 14 16:34 ..
-rw-r-----  1 tomcat8 tomcat8   594 Nov 14 16:16 .eslintrc
-rw-r-----  1 tomcat8 tomcat8   290 Nov 14 16:16 .travis.yml
-rw-r-----  1 tomcat8 tomcat8   686 Nov 14 16:16 ChangeLog.md
-rw-r-----  1 tomcat8 tomcat8   810 Nov 14 16:16 Makefile.PL
-rw-r-----  1 tomcat8 tomcat8  5157 Nov 14 16:16 README.md
-rw-r-----  1 tomcat8 tomcat8  1888 Nov 14 16:44 app.js
drwxr-x---  2 tomcat8 tomcat8    66 Nov 14 16:34 bin
drwxr-x---  2 tomcat8 tomcat8    22 Nov 14 16:34 css
drwxr-x---  2 tomcat8 tomcat8    22 Nov 14 16:34 docs
drwxr-x---  2 tomcat8 tomcat8    45 Nov 14 16:34 img
drwxr-x---  3 tomcat8 tomcat8    65 Nov 14 16:34 js
drwxr-xr-x 24 root    root     4096 Nov 14 17:10 node_modules
-rw-r-----  1 tomcat8 tomcat8 25820 Nov 14 17:10 package-lock.json
-rw-r-----  1 tomcat8 tomcat8   725 Nov 14 16:16 package.json
-rw-r-----  1 tomcat8 tomcat8   727 Nov 14 16:16 setup.sh
drwxr-x---  3 tomcat8 tomcat8    19 Nov 14 16:34 src
drwxr-x---  3 tomcat8 tomcat8    33 Nov 14 16:34 test

..And the logs are attached.
Thanks!
Michael



From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Saturday, November 17, 2018 4:06:58 AM
Subject: Re: [apollo] elastic search in genome apollo

Sorry for the late response. 
Can you provide the browser console logs and any tomcat logs (catalina.out)? 
Your modified docket file may be helpful as well.

Nathan

On Nov 15, 2018, at 10:26 AM, Michael Poeschla <[hidden email]> wrote:

yes - we don't see anything in the browser 
Thanks,
Michael

From: "Colin" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 7:09:45 PM
Subject: Re: [apollo] elastic search in genome apollo

So you are saying that the plugin is not being added e.g. you don't have a menu item for the ElasticSearch dialog in jbrowse?
-Colin

On Thu, Nov 15, 2018 at 9:07 AM Michael Poeschla <[hidden email]> wrote:
hi,
elasticsearch was installed with docker. Here's the tracklist.json for the organism that we generated the index for. Is there any other info I could provide that would help?

{
   "elasticIndexName" : "nfz",
   "elasticSearchUrl" : "localhost:3000",
   "formatVersion" : 1,
   "names" : {
      "type" : "Hash",
      "url" : "names/"
   },
   "plugins" : [
      "ElasticSearch"
   ],
   "tracks" : [
      {
         "category" : "Reference sequence",
         "chunkSize" : 20000,
         "key" : "Reference sequence",
         "label" : "DNA",
         "seqType" : "dna",
         "storeClass" : "JBrowse/Store/Sequence/StaticChunked",
         "type" : "SequenceTrack",
         "urlTemplate" : "seq/{refseq_dirpath}/{refseq}-"
      },
      {
         "bicolor_pivot" : 0.5,
         "label" : "GC Content",
         "storeClass" : "JBrowse/Store/SeqFeature/SequenceChunks",
         "type" : "GCContent/View/Track/GCContentXY",
         "urlTemplate" : "seq/{refseq_dirpath}/{refseq}-"
      },
      {
         "compress" : 0,
         "key" : "Repeats",
         "label" : "Repeats",
         "storeClass" : "JBrowse/Store/SeqFeature/NCList",
         "style" : {
            "className" : "feature"
         },
         "trackType" : "JBrowse/View/Track/CanvasFeatures",
         "type" : "JBrowse/View/Track/CanvasFeatures",
         "urlTemplate" : "tracks/Repeats/{refseq}/trackData.json"
      },
      {
         "compress" : 0,
         "key" : "Annotation",
         "label" : "Annotationfromsumgff",
         "storeClass" : "JBrowse/Store/SeqFeature/NCList",
         "style" : {
            "className" : "feature"
         },
         "trackType" : "CanvasFeatures",
         "type" : "CanvasFeatures",
         "urlTemplate" : "tracks/Annotationfromsumgff/{refseq}/trackData.json"
      }
   ]
}


From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 11:33:03 AM
Subject: Re: [apollo] elastic search in genome apollo

Thanks.  Also on Europe time for another 20 minutes.
Can you also post any other specifics of failure?

Have you installed elasticsearch via docker or manually ?

Nathan

On Nov 15, 2018, at 11:10 AM, Michael Poeschla <[hidden email]> wrote:

Here it is - sorry for the late replies, I'm on europe time.

environments {
    development {
    }
    test {
    }
    production {
        dataSource {
            dbCreate = "update"
            username = System.getenv("WEBAPOLLO_DB_USERNAME") ?: *
            password = System.getenv("WEBAPOLLO_DB_PASSWORD") ?: *

            driverClassName = "org.postgresql.Driver"
            dialect = "org.hibernate.dialect.PostgresPlusDialect"
            url = "jdbc:postgresql://${System.getenv("WEBAPOLLO_DB_HOST")?:"127.0.0.1"}/${System.getenv("WEBAPOLLO_DB_NAME")?:"apollo"}"

            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = false
                initialSize                   = 5
                maxActive                     = 50
                minIdle                       = 5
                maxIdle                       = 25
                maxWait                       = 10000
                maxAge                        = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis    = 60000
                validationQuery               = "SELECT 1"
                validationQueryTimeout        = 3
                validationInterval            = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
        dataSource_chado {
            dbCreate = "update"
            username = System.getenv("CHADO_DB_USERNAME") ?: *
            password = System.getenv("CHADO_DB_PASSWORD") ?: *

            driverClassName = "org.postgresql.Driver"
            dialect = "org.hibernate.dialect.PostgresPlusDialect"

            url = "jdbc:postgresql://${System.getenv("CHADO_DB_HOST")?:"127.0.0.1"}/${System.getenv("CHADO_DB_NAME")?:"chado"}"

            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = false
                initialSize                   = 5
                maxActive                     = 50
                minIdle                       = 5
                maxIdle                       = 25
                maxWait                       = 10000
                maxAge                        = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis    = 60000
                validationQuery               = "SELECT 1"
                validationQueryTimeout        = 3
                validationInterval            = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
    }
}

apollo {
    default_minimum_intron_size = System.getenv("WEBAPOLLO_MINIMUM_INTRON_SIZE") ? System.getenv("WEBAPOLLO_MINIMUM_INTRON_SIZE").toInteger() : 1
    history_size = System.getenv("WEBAPOLLO_HISTORY_SIZE") ? System.getenv("WEBAPOLLO_HISTORY_SIZE").toInteger() : 0
    overlapper_class = System.getenv("WEBAPOLLO_OVERLAPPER_CLASS") ?: "org.bbop.apollo.sequence.OrfOverlapper"
    use_cds_for_new_transcripts = System.getenv("WEBAPOLLO_CDS_FOR_NEW_TRANSCRIPTS").equals("true") // will default to false
    feature_has_dbxrefs = System.getenv("WEBAPOLLO_FEATURE_HAS_DBXREFS") ?: true
    feature_has_attributes = System.getenv("WEBAPOLLO_FEATURE_HAS_ATTRS") ?: true
    feature_has_pubmed_ids = System.getenv("WEBAPOLLO_FEATURE_HAS_PUBMED") ?: true
    feature_has_go_ids = System.getenv("WEBAPOLLO_FEATURE_HAS_GO") ?: true
    feature_has_comments = System.getenv("WEBAPOLLO_FEATURE_HAS_COMMENTS") ?: true
    feature_has_status = System.getenv("WEBAPOLLO_FEATURE_HAS_STATUS") ?: true
    translation_table = "/config/translation_tables/ncbi_" + (System.getenv("WEBAPOLLO_TRANSLATION_TABLE") ?: "1") + "_translation_table.txt"
    get_translation_code = System.getenv("WEBAPOLLO_TRANSLATION_TABLE") ? System.getenv("WEBAPOLLO_TRANSLATION_TABLE").toInteger() : 1

    // ADDED BLAT AND SET PATH TO BIN
    sequence_search_tools = [
        blat_nuc: [
          search_exe: "/usr/local/bin/blat",
          search_class: "org.bbop.apollo.sequence.search.blat.BlatCommandLineNucleotideToNucleotide",
          name: "Blat nucleotide",
          params: ""
        ],
        blat_prot: [
          search_exe: "/usr/local/bin/blat",
          search_class: "org.bbop.apollo.sequence.search.blat.BlatCommandLineProteinToNucleotide",
          name: "Blat protein",
          params: ""
        ]
      ]
    // TODO: should come from config or via preferences database
    splice_donor_sites = System.getenv("WEBAPOLLO_SPLICE_DONOR_SITES") ? System.getenv("WEBAPOLLO_SPLICE_DONOR_SITES").split(",") : ["GT"]
    splice_acceptor_sites = System.getenv("WEBAPOLLO_SPLICE_ACCEPTOR_SITES") ? System.getenv("WEBAPOLLO_SPLICE_ACCEPTOR_SITES").split(",") : ["AG"]
    gff3.source = System.getenv("WEBAPOLLO_GFF3_SOURCE") ?: "."

    google_analytics = System.getenv("WEBAPOLLO_GOOGLE_ANALYTICS_ID") ?: ["UA-62921593-1"]

    admin{
        username = System.getenv("APOLLO_ADMIN_EMAIL") ?: *
        password = System.getenv("APOLLO_ADMIN_PASSWORD") ?: *
        firstName = System.getenv("APOLLO_ADMIN_FIRST_NAME") ?: *
        lastName = System.getenv("APOLLO_ADMIN_LAST_NAME") ?: *
    }
}

jbrowse {
    git {
        url = "https://github.com/GMOD/jbrowse"
        tag = "maint/1.12.5-apollo"
    }
    plugins {
        WebApollo{
            included = true
        }
        NeatHTMLFeatures{
            included = System.getenv("WEBAPOLLO_JBROWSE_PLUGIN_NEATHTML") ?: true
        }
        NeatCanvasFeatures{
            included = System.getenv("WEBAPOLLO_JBROWSE_PLUGIN_NEATCANVAS") ?: true
        }
        RegexSequenceSearch{
            included = true
        }
        HideTrackLabels{
            included = true
        }
        //EDIT: ADDED GC CONTENT
        GCContent{
            git = 'https://github.com/cmdcolin/GCContent'
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
        //EDIT: ADDS ELASTIC SEARCH PLUGIN
        ElasticSearch{
            git = 'https://github.com/elsiklab/jbrowse_elasticsearch'
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
    }
}

From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 12:30:50 AM
Subject: Re: [apollo] elastic search in genome apollo

Can you post your apollo config file? 

Nathan

On Nov 14, 2018, at 6:45 PM, Michael Poeschla <[hidden email]> wrote:

Hi, 
I'm following the instructions from https://github.com/elsiklab/jbrowse_elasticsearch

we have elasticsearch and the middleware app.js running. I was able to execute generate-names.pl and this seemed to work correctly from the terminal output. The changes also got picked up in the tracklist.json for the genome. But there is no sign of the elasticsearch plugin in web apollo...added the plugin to the config.groovy, and we have another jbrowse plugin (GC content) working fine so not sure what's the problem here. 


Thanks for all of your help! :)
Michael 

From: "Nathan Dunn" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Tuesday, November 13, 2018 6:33:05 PM
Subject: Re: [apollo] elastic search in genome apollo


Michael,

What problems are you running it when adding the plugins? 

Nathan


On Nov 13, 2018, at 4:55 PM, Colin <[hidden email]> wrote:

Aside from changing your gene identifiers, I think the full text search from elasticsearch could help. I forwarded some comments here about that https://github.com/elsiklab/jbrowse_elasticsearch/issues/7

Potentially data that is loaded in Apollo backend could use text similarity %like% or full text sql queries but for normal jbrowse tracks that are not loaded in the sql database, the jbrowse_elasticsearch plugin seems reasonable.

-Colin

On Tue, Nov 13, 2018 at 7:35 AM Michael Poeschla <[hidden email]> wrote:
Hi,

has anyone implemented the jbrowse_elasticsearch plugin in Apollo? I'm running from the docker container with an nginx reverse proxy and having trouble adding this plugin. How would I add this to the docker image, with the middleware as well?

The reason we want to do this is the predicted gene model names we have in our tracks are not always intuitive so it's hard to find them by searching in Apollo (ie mTOR is ID'd as "serine/threonine-protein kinase mTOR", so if you just start typing mTOR you get no results). Wondering also if anyone else has encountered this problem and found good solutions, besides manipulating all of our tracks.

Thanks so much,
Michael



This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.




This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.






This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.




This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.



This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.



This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.



This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.




This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.


tomcat.tar.gz (396K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: elastic search in genome apollo

nathandunn

Thanks.  Will have a look.


On Nov 20, 2018, at 5:03 AM, Michael Poeschla <[hidden email]> wrote:

Hi,

here are the logs, docker-compose, actual dockerfile and start and testing of npm (below). I've also added the tomcat logs as an attachment.

docker-compose:
--
version: "3.2"
services:

  apollo:
    volumes:
      - ./data/extdata/:/data
      - postgres-data:/var/lib/postgresql
      - ./config/tomcat/server.xml:/var/lib/tomcat8/conf/server.xml:ro
      - ./config/tomcat/setenv.sh:/usr/share/tomcat8/bin/setenv.sh
    environment:
      APOLLO_ADMIN_[hidden email]
    restart: always
    links:
      - elasticsearch:elasticsearch

  nginx:
    image: library/nginx:alpine
    restart: always
    volumes:
      - ./config/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
      - ./config/secrets:/secrets:ro
    ports:
      - 80:80
      - 443:443
    links:
      - apollo:apollo

  elasticsearch:
    image: elasticsearch:6.4.2
    container_name: elasticsearch
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata1:/usr/share/elasticsearch/data

volumes:
  postgres-data:
  esdata1:
    driver: local
--
The actual Dockerfile:
--
# WebApollo
# VERSION 2.1.X
FROM tomcat:8-jre8
MAINTAINER Eric Rasche <[hidden email]>, Anthony Bretaudeau <[hidden email]>, Nathan Dunn <[hidden email]>
ENV DEBIAN_FRONTEND noninteractive 

RUN apt-get -qq update --fix-missing && \
apt-get --no-install-recommends -y install \
git build-essential maven tomcat8 libpq-dev postgresql-common openjdk-8-jdk wget \
postgresql postgresql-client xmlstarlet netcat libpng-dev \
zlib1g-dev libexpat1-dev ant curl ssl-cert zip unzip

RUN apt-get -qq update --fix-missing && \
apt-get --no-install-recommends -y install nodejs && \
apt-get autoremove -y && apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

RUN cp /usr/lib/jvm/java-8-openjdk-amd64/lib/tools.jar /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext/tools.jar && \
useradd -ms /bin/bash -d /apollo apollo

ENV WEBAPOLLO_VERSION 99c7e54c6e74fbc6705d57de216c9b14b2bfb03b
RUN curl -L https://github.com/GMOD/Apollo/archive/${WEBAPOLLO_VERSION}.tar.gz | tar xzf - --strip-components=1 -C /apollo

# install grails
COPY build.sh /bin/build.sh
ADD apollo-config.groovy /apollo/apollo-config.groovy

RUN chown -R apollo:apollo /apollo
RUN chmod +x /usr/local/bin/blat


USER apollo
RUN curl -s get.sdkman.io | bash
RUN /bin/bash -c "source $HOME/.sdkman/bin/sdkman-init.sh && yes | sdk install grails 2.5.5"
RUN /bin/bash -c "source $HOME/.sdkman/bin/sdkman-init.sh && yes | sdk install gradle 3.2.1"


RUN /bin/bash -c "source $HOME/.profile && source $HOME/.sdkman/bin/sdkman-init.sh && /bin/bash /bin/build.sh"

USER root
ENV CATALINA_HOME=/var/lib/tomcat8
RUN rm -rf ${CATALINA_HOME}/webapps/* && \
cp /apollo/apollo*.war ${CATALINA_HOME}/apollo.war

ENV CONTEXT_PATH ROOT

# Download chado schema
gunzip /chado.sql.gz


ADD launch.sh /launch.sh
CMD "/launch.sh"
--
Starting npm inside the container:
root@995955a711e1:/var/lib/tomcat8/webapps/ROOT/jbrowse/plugins/ElasticSearch# npm start

> jbrowse_elasticsearch@1.0.4 start /var/lib/tomcat8/webapps/ROOT/jbrowse/plugins/ElasticSearch
> node ./bin/www

testing access with curl:
root@995955a711e1:/usr/local/tomcat# curl http://localhost:3000
{"hits":[],"total":0}root@995955a711e1:/usr/local/tomcat#

content of ElasticSearchPlugin:
 ^Croot@995955a711e1:/var/lib/tomcat8/webapps/ROOT/jbrowse/plugins/ElasticSearch# ls -la
total 68
drwxr-x--- 10 tomcat8 tomcat8   271 Nov 14 17:10 .
drwxr-x--- 12 tomcat8 tomcat8   223 Nov 14 16:34 ..
-rw-r-----  1 tomcat8 tomcat8   594 Nov 14 16:16 .eslintrc
-rw-r-----  1 tomcat8 tomcat8   290 Nov 14 16:16 .travis.yml
-rw-r-----  1 tomcat8 tomcat8   686 Nov 14 16:16 ChangeLog.md
-rw-r-----  1 tomcat8 tomcat8   810 Nov 14 16:16 Makefile.PL
-rw-r-----  1 tomcat8 tomcat8  5157 Nov 14 16:16 README.md
-rw-r-----  1 tomcat8 tomcat8  1888 Nov 14 16:44 app.js
drwxr-x---  2 tomcat8 tomcat8    66 Nov 14 16:34 bin
drwxr-x---  2 tomcat8 tomcat8    22 Nov 14 16:34 css
drwxr-x---  2 tomcat8 tomcat8    22 Nov 14 16:34 docs
drwxr-x---  2 tomcat8 tomcat8    45 Nov 14 16:34 img
drwxr-x---  3 tomcat8 tomcat8    65 Nov 14 16:34 js
drwxr-xr-x 24 root    root     4096 Nov 14 17:10 node_modules
-rw-r-----  1 tomcat8 tomcat8 25820 Nov 14 17:10 package-lock.json
-rw-r-----  1 tomcat8 tomcat8   725 Nov 14 16:16 package.json
-rw-r-----  1 tomcat8 tomcat8   727 Nov 14 16:16 setup.sh
drwxr-x---  3 tomcat8 tomcat8    19 Nov 14 16:34 src
drwxr-x---  3 tomcat8 tomcat8    33 Nov 14 16:34 test

..And the logs are attached.
Thanks!
Michael



From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Saturday, November 17, 2018 4:06:58 AM
Subject: Re: [apollo] elastic search in genome apollo

Sorry for the late response. 
Can you provide the browser console logs and any tomcat logs (catalina.out)? 
Your modified docket file may be helpful as well.

Nathan

On Nov 15, 2018, at 10:26 AM, Michael Poeschla <[hidden email]> wrote:

yes - we don't see anything in the browser 
Thanks,
Michael

From: "Colin" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 7:09:45 PM
Subject: Re: [apollo] elastic search in genome apollo

So you are saying that the plugin is not being added e.g. you don't have a menu item for the ElasticSearch dialog in jbrowse?
-Colin

On Thu, Nov 15, 2018 at 9:07 AM Michael Poeschla <[hidden email]> wrote:
hi,
elasticsearch was installed with docker. Here's the tracklist.json for the organism that we generated the index for. Is there any other info I could provide that would help?

{
   "elasticIndexName" : "nfz",
   "elasticSearchUrl" : "localhost:3000",
   "formatVersion" : 1,
   "names" : {
      "type" : "Hash",
      "url" : "names/"
   },
   "plugins" : [
      "ElasticSearch"
   ],
   "tracks" : [
      {
         "category" : "Reference sequence",
         "chunkSize" : 20000,
         "key" : "Reference sequence",
         "label" : "DNA",
         "seqType" : "dna",
         "storeClass" : "JBrowse/Store/Sequence/StaticChunked",
         "type" : "SequenceTrack",
         "urlTemplate" : "seq/{refseq_dirpath}/{refseq}-"
      },
      {
         "bicolor_pivot" : 0.5,
         "label" : "GC Content",
         "storeClass" : "JBrowse/Store/SeqFeature/SequenceChunks",
         "type" : "GCContent/View/Track/GCContentXY",
         "urlTemplate" : "seq/{refseq_dirpath}/{refseq}-"
      },
      {
         "compress" : 0,
         "key" : "Repeats",
         "label" : "Repeats",
         "storeClass" : "JBrowse/Store/SeqFeature/NCList",
         "style" : {
            "className" : "feature"
         },
         "trackType" : "JBrowse/View/Track/CanvasFeatures",
         "type" : "JBrowse/View/Track/CanvasFeatures",
         "urlTemplate" : "tracks/Repeats/{refseq}/trackData.json"
      },
      {
         "compress" : 0,
         "key" : "Annotation",
         "label" : "Annotationfromsumgff",
         "storeClass" : "JBrowse/Store/SeqFeature/NCList",
         "style" : {
            "className" : "feature"
         },
         "trackType" : "CanvasFeatures",
         "type" : "CanvasFeatures",
         "urlTemplate" : "tracks/Annotationfromsumgff/{refseq}/trackData.json"
      }
   ]
}


From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 11:33:03 AM
Subject: Re: [apollo] elastic search in genome apollo

Thanks.  Also on Europe time for another 20 minutes.
Can you also post any other specifics of failure?

Have you installed elasticsearch via docker or manually ?

Nathan

On Nov 15, 2018, at 11:10 AM, Michael Poeschla <[hidden email]> wrote:

Here it is - sorry for the late replies, I'm on europe time.

environments {
    development {
    }
    test {
    }
    production {
        dataSource {
            dbCreate = "update"
            username = System.getenv("WEBAPOLLO_DB_USERNAME") ?: *
            password = System.getenv("WEBAPOLLO_DB_PASSWORD") ?: *

            driverClassName = "org.postgresql.Driver"
            dialect = "org.hibernate.dialect.PostgresPlusDialect"
            url = "jdbc:<a href="postgresql://$" class="">postgresql://${System.getenv("WEBAPOLLO_DB_HOST")?:"127.0.0.1"}/${System.getenv("WEBAPOLLO_DB_NAME")?:"apollo"}"

            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = false
                initialSize                   = 5
                maxActive                     = 50
                minIdle                       = 5
                maxIdle                       = 25
                maxWait                       = 10000
                maxAge                        = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis    = 60000
                validationQuery               = "SELECT 1"
                validationQueryTimeout        = 3
                validationInterval            = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
        dataSource_chado {
            dbCreate = "update"
            username = System.getenv("CHADO_DB_USERNAME") ?: *
            password = System.getenv("CHADO_DB_PASSWORD") ?: *

            driverClassName = "org.postgresql.Driver"
            dialect = "org.hibernate.dialect.PostgresPlusDialect"

            url = "jdbc:<a href="postgresql://$" class="">postgresql://${System.getenv("CHADO_DB_HOST")?:"127.0.0.1"}/${System.getenv("CHADO_DB_NAME")?:"chado"}"

            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = false
                initialSize                   = 5
                maxActive                     = 50
                minIdle                       = 5
                maxIdle                       = 25
                maxWait                       = 10000
                maxAge                        = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis    = 60000
                validationQuery               = "SELECT 1"
                validationQueryTimeout        = 3
                validationInterval            = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
    }
}

apollo {
    default_minimum_intron_size = System.getenv("WEBAPOLLO_MINIMUM_INTRON_SIZE") ? System.getenv("WEBAPOLLO_MINIMUM_INTRON_SIZE").toInteger() : 1
    history_size = System.getenv("WEBAPOLLO_HISTORY_SIZE") ? System.getenv("WEBAPOLLO_HISTORY_SIZE").toInteger() : 0
    overlapper_class = System.getenv("WEBAPOLLO_OVERLAPPER_CLASS") ?: "org.bbop.apollo.sequence.OrfOverlapper"
    use_cds_for_new_transcripts = System.getenv("WEBAPOLLO_CDS_FOR_NEW_TRANSCRIPTS").equals("true") // will default to false
    feature_has_dbxrefs = System.getenv("WEBAPOLLO_FEATURE_HAS_DBXREFS") ?: true
    feature_has_attributes = System.getenv("WEBAPOLLO_FEATURE_HAS_ATTRS") ?: true
    feature_has_pubmed_ids = System.getenv("WEBAPOLLO_FEATURE_HAS_PUBMED") ?: true
    feature_has_go_ids = System.getenv("WEBAPOLLO_FEATURE_HAS_GO") ?: true
    feature_has_comments = System.getenv("WEBAPOLLO_FEATURE_HAS_COMMENTS") ?: true
    feature_has_status = System.getenv("WEBAPOLLO_FEATURE_HAS_STATUS") ?: true
    translation_table = "/config/translation_tables/ncbi_" + (System.getenv("WEBAPOLLO_TRANSLATION_TABLE") ?: "1") + "_translation_table.txt"
    get_translation_code = System.getenv("WEBAPOLLO_TRANSLATION_TABLE") ? System.getenv("WEBAPOLLO_TRANSLATION_TABLE").toInteger() : 1

    // ADDED BLAT AND SET PATH TO BIN
    sequence_search_tools = [
        blat_nuc: [
          search_exe: "/usr/local/bin/blat",
          search_class: "org.bbop.apollo.sequence.search.blat.BlatCommandLineNucleotideToNucleotide",
          name: "Blat nucleotide",
          params: ""
        ],
        blat_prot: [
          search_exe: "/usr/local/bin/blat",
          search_class: "org.bbop.apollo.sequence.search.blat.BlatCommandLineProteinToNucleotide",
          name: "Blat protein",
          params: ""
        ]
      ]
    // TODO: should come from config or via preferences database
    splice_donor_sites = System.getenv("WEBAPOLLO_SPLICE_DONOR_SITES") ? System.getenv("WEBAPOLLO_SPLICE_DONOR_SITES").split(",") : ["GT"]
    splice_acceptor_sites = System.getenv("WEBAPOLLO_SPLICE_ACCEPTOR_SITES") ? System.getenv("WEBAPOLLO_SPLICE_ACCEPTOR_SITES").split(",") : ["AG"]
    gff3.source = System.getenv("WEBAPOLLO_GFF3_SOURCE") ?: "."

    google_analytics = System.getenv("WEBAPOLLO_GOOGLE_ANALYTICS_ID") ?: ["UA-62921593-1"]

    admin{
        username = System.getenv("APOLLO_ADMIN_EMAIL") ?: *
        password = System.getenv("APOLLO_ADMIN_PASSWORD") ?: *
        firstName = System.getenv("APOLLO_ADMIN_FIRST_NAME") ?: *
        lastName = System.getenv("APOLLO_ADMIN_LAST_NAME") ?: *
    }
}

jbrowse {
    git {
        url = "https://github.com/GMOD/jbrowse"
        tag = "maint/1.12.5-apollo"
    }
    plugins {
        WebApollo{
            included = true
        }
        NeatHTMLFeatures{
            included = System.getenv("WEBAPOLLO_JBROWSE_PLUGIN_NEATHTML") ?: true
        }
        NeatCanvasFeatures{
            included = System.getenv("WEBAPOLLO_JBROWSE_PLUGIN_NEATCANVAS") ?: true
        }
        RegexSequenceSearch{
            included = true
        }
        HideTrackLabels{
            included = true
        }
        //EDIT: ADDED GC CONTENT
        GCContent{
            git = 'https://github.com/cmdcolin/GCContent'
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
        //EDIT: ADDS ELASTIC SEARCH PLUGIN
        ElasticSearch{
            git = 'https://github.com/elsiklab/jbrowse_elasticsearch'
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
    }
}

From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 12:30:50 AM
Subject: Re: [apollo] elastic search in genome apollo

Can you post your apollo config file? 

Nathan

On Nov 14, 2018, at 6:45 PM, Michael Poeschla <[hidden email]> wrote:

Hi, 
I'm following the instructions from https://github.com/elsiklab/jbrowse_elasticsearch

we have elasticsearch and the middleware app.js running. I was able to execute generate-names.pl and this seemed to work correctly from the terminal output. The changes also got picked up in the tracklist.json for the genome. But there is no sign of the elasticsearch plugin in web apollo...added the plugin to the config.groovy, and we have another jbrowse plugin (GC content) working fine so not sure what's the problem here. 


Thanks for all of your help! :)
Michael 

From: "Nathan Dunn" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Tuesday, November 13, 2018 6:33:05 PM
Subject: Re: [apollo] elastic search in genome apollo


Michael,

What problems are you running it when adding the plugins? 

Nathan


On Nov 13, 2018, at 4:55 PM, Colin <[hidden email]> wrote:

Aside from changing your gene identifiers, I think the full text search from elasticsearch could help. I forwarded some comments here about that https://github.com/elsiklab/jbrowse_elasticsearch/issues/7

Potentially data that is loaded in Apollo backend could use text similarity %like% or full text sql queries but for normal jbrowse tracks that are not loaded in the sql database, the jbrowse_elasticsearch plugin seems reasonable.

-Colin

On Tue, Nov 13, 2018 at 7:35 AM Michael Poeschla <[hidden email]> wrote:
Hi,

has anyone implemented the jbrowse_elasticsearch plugin in Apollo? I'm running from the docker container with an nginx reverse proxy and having trouble adding this plugin. How would I add this to the docker image, with the middleware as well?

The reason we want to do this is the predicted gene model names we have in our tracks are not always intuitive so it's hard to find them by searching in Apollo (ie mTOR is ID'd as "serine/threonine-protein kinase mTOR", so if you just start typing mTOR you get no results). Wondering also if anyone else has encountered this problem and found good solutions, besides manipulating all of our tracks.

Thanks so much,
Michael



This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.




This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.






This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.




This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.



This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.



This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.



This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.
<tomcat.tar.gz>


This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.






This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

Reply | Threaded
Open this post in threaded view
|

Re: elastic search in genome apollo

Colin
I went and tested this and I tried testing adding the plugin to apollo-config.groovy and was getting an error about plugins/ElasticSearch/js/main.js not being loaded apollo run-local. I can see that it gets downloaded to jbrowse-download/plugins/ElasticSearch but it was giving me that error there still. I also ran apollo deploy and then unzipped and the jbrowse plugins folder did not contain ElasticSearch plugin

Screenshot attached and apollo-config.groovy snippet here

jbrowse {
    plugins {
        ElasticSearch {
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
    }
}

-Colin

On Tue, Nov 20, 2018 at 6:57 PM Nathan Dunn <[hidden email]> wrote:

Thanks.  Will have a look.


On Nov 20, 2018, at 5:03 AM, Michael Poeschla <[hidden email]> wrote:

Hi,

here are the logs, docker-compose, actual dockerfile and start and testing of npm (below). I've also added the tomcat logs as an attachment.

docker-compose:
--
version: "3.2"
services:

  apollo:
    volumes:
      - ./data/extdata/:/data
      - postgres-data:/var/lib/postgresql
      - ./config/tomcat/server.xml:/var/lib/tomcat8/conf/server.xml:ro
      - ./config/tomcat/setenv.sh:/usr/share/tomcat8/bin/setenv.sh
    environment:
      APOLLO_ADMIN_[hidden email]
    restart: always
    links:
      - elasticsearch:elasticsearch

  nginx:
    image: library/nginx:alpine
    restart: always
    volumes:
      - ./config/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
      - ./config/secrets:/secrets:ro
    ports:
      - 80:80
      - 443:443
    links:
      - apollo:apollo

  elasticsearch:
    image: elasticsearch:6.4.2
    container_name: elasticsearch
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata1:/usr/share/elasticsearch/data

volumes:
  postgres-data:
  esdata1:
    driver: local
--
The actual Dockerfile:
--
# WebApollo
# VERSION 2.1.X
FROM tomcat:8-jre8
MAINTAINER Eric Rasche <[hidden email]>, Anthony Bretaudeau <[hidden email]>, Nathan Dunn <[hidden email]>
ENV DEBIAN_FRONTEND noninteractive 

RUN apt-get -qq update --fix-missing && \
apt-get --no-install-recommends -y install \
git build-essential maven tomcat8 libpq-dev postgresql-common openjdk-8-jdk wget \
postgresql postgresql-client xmlstarlet netcat libpng-dev \
zlib1g-dev libexpat1-dev ant curl ssl-cert zip unzip

RUN apt-get -qq update --fix-missing && \
apt-get --no-install-recommends -y install nodejs && \
apt-get autoremove -y && apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

RUN cp /usr/lib/jvm/java-8-openjdk-amd64/lib/tools.jar /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext/tools.jar && \
useradd -ms /bin/bash -d /apollo apollo

ENV WEBAPOLLO_VERSION 99c7e54c6e74fbc6705d57de216c9b14b2bfb03b
RUN curl -L https://github.com/GMOD/Apollo/archive/${WEBAPOLLO_VERSION}.tar.gz | tar xzf - --strip-components=1 -C /apollo

# install grails
COPY build.sh /bin/build.sh
ADD apollo-config.groovy /apollo/apollo-config.groovy

RUN chown -R apollo:apollo /apollo
RUN chmod +x /usr/local/bin/blat


USER apollo
RUN curl -s get.sdkman.io | bash
RUN /bin/bash -c "source $HOME/.sdkman/bin/sdkman-init.sh && yes | sdk install grails 2.5.5"
RUN /bin/bash -c "source $HOME/.sdkman/bin/sdkman-init.sh && yes | sdk install gradle 3.2.1"


RUN /bin/bash -c "source $HOME/.profile && source $HOME/.sdkman/bin/sdkman-init.sh && /bin/bash /bin/build.sh"

USER root
ENV CATALINA_HOME=/var/lib/tomcat8
RUN rm -rf ${CATALINA_HOME}/webapps/* && \
cp /apollo/apollo*.war ${CATALINA_HOME}/apollo.war

ENV CONTEXT_PATH ROOT

# Download chado schema
gunzip /chado.sql.gz


ADD launch.sh /launch.sh
CMD "/launch.sh"
--
Starting npm inside the container:
root@995955a711e1:/var/lib/tomcat8/webapps/ROOT/jbrowse/plugins/ElasticSearch# npm start

> jbrowse_elasticsearch@1.0.4 start /var/lib/tomcat8/webapps/ROOT/jbrowse/plugins/ElasticSearch
> node ./bin/www

testing access with curl:
root@995955a711e1:/usr/local/tomcat# curl http://localhost:3000
{"hits":[],"total":0}root@995955a711e1:/usr/local/tomcat#

content of ElasticSearchPlugin:
 ^Croot@995955a711e1:/var/lib/tomcat8/webapps/ROOT/jbrowse/plugins/ElasticSearch# ls -la
total 68
drwxr-x--- 10 tomcat8 tomcat8   271 Nov 14 17:10 .
drwxr-x--- 12 tomcat8 tomcat8   223 Nov 14 16:34 ..
-rw-r-----  1 tomcat8 tomcat8   594 Nov 14 16:16 .eslintrc
-rw-r-----  1 tomcat8 tomcat8   290 Nov 14 16:16 .travis.yml
-rw-r-----  1 tomcat8 tomcat8   686 Nov 14 16:16 ChangeLog.md
-rw-r-----  1 tomcat8 tomcat8   810 Nov 14 16:16 Makefile.PL
-rw-r-----  1 tomcat8 tomcat8  5157 Nov 14 16:16 README.md
-rw-r-----  1 tomcat8 tomcat8  1888 Nov 14 16:44 app.js
drwxr-x---  2 tomcat8 tomcat8    66 Nov 14 16:34 bin
drwxr-x---  2 tomcat8 tomcat8    22 Nov 14 16:34 css
drwxr-x---  2 tomcat8 tomcat8    22 Nov 14 16:34 docs
drwxr-x---  2 tomcat8 tomcat8    45 Nov 14 16:34 img
drwxr-x---  3 tomcat8 tomcat8    65 Nov 14 16:34 js
drwxr-xr-x 24 root    root     4096 Nov 14 17:10 node_modules
-rw-r-----  1 tomcat8 tomcat8 25820 Nov 14 17:10 package-lock.json
-rw-r-----  1 tomcat8 tomcat8   725 Nov 14 16:16 package.json
-rw-r-----  1 tomcat8 tomcat8   727 Nov 14 16:16 setup.sh
drwxr-x---  3 tomcat8 tomcat8    19 Nov 14 16:34 src
drwxr-x---  3 tomcat8 tomcat8    33 Nov 14 16:34 test

..And the logs are attached.
Thanks!
Michael



From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Saturday, November 17, 2018 4:06:58 AM
Subject: Re: [apollo] elastic search in genome apollo

Sorry for the late response. 
Can you provide the browser console logs and any tomcat logs (catalina.out)? 
Your modified docket file may be helpful as well.

Nathan

On Nov 15, 2018, at 10:26 AM, Michael Poeschla <[hidden email]> wrote:

yes - we don't see anything in the browser 
Thanks,
Michael

From: "Colin" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 7:09:45 PM
Subject: Re: [apollo] elastic search in genome apollo

So you are saying that the plugin is not being added e.g. you don't have a menu item for the ElasticSearch dialog in jbrowse?
-Colin

On Thu, Nov 15, 2018 at 9:07 AM Michael Poeschla <[hidden email]> wrote:
hi,
elasticsearch was installed with docker. Here's the tracklist.json for the organism that we generated the index for. Is there any other info I could provide that would help?

{
   "elasticIndexName" : "nfz",
   "elasticSearchUrl" : "localhost:3000",
   "formatVersion" : 1,
   "names" : {
      "type" : "Hash",
      "url" : "names/"
   },
   "plugins" : [
      "ElasticSearch"
   ],
   "tracks" : [
      {
         "category" : "Reference sequence",
         "chunkSize" : 20000,
         "key" : "Reference sequence",
         "label" : "DNA",
         "seqType" : "dna",
         "storeClass" : "JBrowse/Store/Sequence/StaticChunked",
         "type" : "SequenceTrack",
         "urlTemplate" : "seq/{refseq_dirpath}/{refseq}-"
      },
      {
         "bicolor_pivot" : 0.5,
         "label" : "GC Content",
         "storeClass" : "JBrowse/Store/SeqFeature/SequenceChunks",
         "type" : "GCContent/View/Track/GCContentXY",
         "urlTemplate" : "seq/{refseq_dirpath}/{refseq}-"
      },
      {
         "compress" : 0,
         "key" : "Repeats",
         "label" : "Repeats",
         "storeClass" : "JBrowse/Store/SeqFeature/NCList",
         "style" : {
            "className" : "feature"
         },
         "trackType" : "JBrowse/View/Track/CanvasFeatures",
         "type" : "JBrowse/View/Track/CanvasFeatures",
         "urlTemplate" : "tracks/Repeats/{refseq}/trackData.json"
      },
      {
         "compress" : 0,
         "key" : "Annotation",
         "label" : "Annotationfromsumgff",
         "storeClass" : "JBrowse/Store/SeqFeature/NCList",
         "style" : {
            "className" : "feature"
         },
         "trackType" : "CanvasFeatures",
         "type" : "CanvasFeatures",
         "urlTemplate" : "tracks/Annotationfromsumgff/{refseq}/trackData.json"
      }
   ]
}


From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 11:33:03 AM
Subject: Re: [apollo] elastic search in genome apollo

Thanks.  Also on Europe time for another 20 minutes.
Can you also post any other specifics of failure?

Have you installed elasticsearch via docker or manually ?

Nathan

On Nov 15, 2018, at 11:10 AM, Michael Poeschla <[hidden email]> wrote:

Here it is - sorry for the late replies, I'm on europe time.

environments {
    development {
    }
    test {
    }
    production {
        dataSource {
            dbCreate = "update"
            username = System.getenv("WEBAPOLLO_DB_USERNAME") ?: *
            password = System.getenv("WEBAPOLLO_DB_PASSWORD") ?: *

            driverClassName = "org.postgresql.Driver"
            dialect = "org.hibernate.dialect.PostgresPlusDialect"
            url = "jdbc:postgresql://${System.getenv("WEBAPOLLO_DB_HOST")?:"127.0.0.1"}/${System.getenv("WEBAPOLLO_DB_NAME")?:"apollo"}"

            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = false
                initialSize                   = 5
                maxActive                     = 50
                minIdle                       = 5
                maxIdle                       = 25
                maxWait                       = 10000
                maxAge                        = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis    = 60000
                validationQuery               = "SELECT 1"
                validationQueryTimeout        = 3
                validationInterval            = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
        dataSource_chado {
            dbCreate = "update"
            username = System.getenv("CHADO_DB_USERNAME") ?: *
            password = System.getenv("CHADO_DB_PASSWORD") ?: *

            driverClassName = "org.postgresql.Driver"
            dialect = "org.hibernate.dialect.PostgresPlusDialect"

            url = "jdbc:postgresql://${System.getenv("CHADO_DB_HOST")?:"127.0.0.1"}/${System.getenv("CHADO_DB_NAME")?:"chado"}"

            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = false
                initialSize                   = 5
                maxActive                     = 50
                minIdle                       = 5
                maxIdle                       = 25
                maxWait                       = 10000
                maxAge                        = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis    = 60000
                validationQuery               = "SELECT 1"
                validationQueryTimeout        = 3
                validationInterval            = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
    }
}

apollo {
    default_minimum_intron_size = System.getenv("WEBAPOLLO_MINIMUM_INTRON_SIZE") ? System.getenv("WEBAPOLLO_MINIMUM_INTRON_SIZE").toInteger() : 1
    history_size = System.getenv("WEBAPOLLO_HISTORY_SIZE") ? System.getenv("WEBAPOLLO_HISTORY_SIZE").toInteger() : 0
    overlapper_class = System.getenv("WEBAPOLLO_OVERLAPPER_CLASS") ?: "org.bbop.apollo.sequence.OrfOverlapper"
    use_cds_for_new_transcripts = System.getenv("WEBAPOLLO_CDS_FOR_NEW_TRANSCRIPTS").equals("true") // will default to false
    feature_has_dbxrefs = System.getenv("WEBAPOLLO_FEATURE_HAS_DBXREFS") ?: true
    feature_has_attributes = System.getenv("WEBAPOLLO_FEATURE_HAS_ATTRS") ?: true
    feature_has_pubmed_ids = System.getenv("WEBAPOLLO_FEATURE_HAS_PUBMED") ?: true
    feature_has_go_ids = System.getenv("WEBAPOLLO_FEATURE_HAS_GO") ?: true
    feature_has_comments = System.getenv("WEBAPOLLO_FEATURE_HAS_COMMENTS") ?: true
    feature_has_status = System.getenv("WEBAPOLLO_FEATURE_HAS_STATUS") ?: true
    translation_table = "/config/translation_tables/ncbi_" + (System.getenv("WEBAPOLLO_TRANSLATION_TABLE") ?: "1") + "_translation_table.txt"
    get_translation_code = System.getenv("WEBAPOLLO_TRANSLATION_TABLE") ? System.getenv("WEBAPOLLO_TRANSLATION_TABLE").toInteger() : 1

    // ADDED BLAT AND SET PATH TO BIN
    sequence_search_tools = [
        blat_nuc: [
          search_exe: "/usr/local/bin/blat",
          search_class: "org.bbop.apollo.sequence.search.blat.BlatCommandLineNucleotideToNucleotide",
          name: "Blat nucleotide",
          params: ""
        ],
        blat_prot: [
          search_exe: "/usr/local/bin/blat",
          search_class: "org.bbop.apollo.sequence.search.blat.BlatCommandLineProteinToNucleotide",
          name: "Blat protein",
          params: ""
        ]
      ]
    // TODO: should come from config or via preferences database
    splice_donor_sites = System.getenv("WEBAPOLLO_SPLICE_DONOR_SITES") ? System.getenv("WEBAPOLLO_SPLICE_DONOR_SITES").split(",") : ["GT"]
    splice_acceptor_sites = System.getenv("WEBAPOLLO_SPLICE_ACCEPTOR_SITES") ? System.getenv("WEBAPOLLO_SPLICE_ACCEPTOR_SITES").split(",") : ["AG"]
    gff3.source = System.getenv("WEBAPOLLO_GFF3_SOURCE") ?: "."

    google_analytics = System.getenv("WEBAPOLLO_GOOGLE_ANALYTICS_ID") ?: ["UA-62921593-1"]

    admin{
        username = System.getenv("APOLLO_ADMIN_EMAIL") ?: *
        password = System.getenv("APOLLO_ADMIN_PASSWORD") ?: *
        firstName = System.getenv("APOLLO_ADMIN_FIRST_NAME") ?: *
        lastName = System.getenv("APOLLO_ADMIN_LAST_NAME") ?: *
    }
}

jbrowse {
    git {
        url = "https://github.com/GMOD/jbrowse"
        tag = "maint/1.12.5-apollo"
    }
    plugins {
        WebApollo{
            included = true
        }
        NeatHTMLFeatures{
            included = System.getenv("WEBAPOLLO_JBROWSE_PLUGIN_NEATHTML") ?: true
        }
        NeatCanvasFeatures{
            included = System.getenv("WEBAPOLLO_JBROWSE_PLUGIN_NEATCANVAS") ?: true
        }
        RegexSequenceSearch{
            included = true
        }
        HideTrackLabels{
            included = true
        }
        //EDIT: ADDED GC CONTENT
        GCContent{
            git = 'https://github.com/cmdcolin/GCContent'
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
        //EDIT: ADDS ELASTIC SEARCH PLUGIN
        ElasticSearch{
            git = 'https://github.com/elsiklab/jbrowse_elasticsearch'
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
    }
}

From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 12:30:50 AM
Subject: Re: [apollo] elastic search in genome apollo

Can you post your apollo config file? 

Nathan

On Nov 14, 2018, at 6:45 PM, Michael Poeschla <[hidden email]> wrote:

Hi, 
I'm following the instructions from https://github.com/elsiklab/jbrowse_elasticsearch

we have elasticsearch and the middleware app.js running. I was able to execute generate-names.pl and this seemed to work correctly from the terminal output. The changes also got picked up in the tracklist.json for the genome. But there is no sign of the elasticsearch plugin in web apollo...added the plugin to the config.groovy, and we have another jbrowse plugin (GC content) working fine so not sure what's the problem here. 


Thanks for all of your help! :)
Michael 

From: "Nathan Dunn" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Tuesday, November 13, 2018 6:33:05 PM
Subject: Re: [apollo] elastic search in genome apollo


Michael,

What problems are you running it when adding the plugins? 

Nathan


On Nov 13, 2018, at 4:55 PM, Colin <[hidden email]> wrote:

Aside from changing your gene identifiers, I think the full text search from elasticsearch could help. I forwarded some comments here about that https://github.com/elsiklab/jbrowse_elasticsearch/issues/7

Potentially data that is loaded in Apollo backend could use text similarity %like% or full text sql queries but for normal jbrowse tracks that are not loaded in the sql database, the jbrowse_elasticsearch plugin seems reasonable.

-Colin

On Tue, Nov 13, 2018 at 7:35 AM Michael Poeschla <[hidden email]> wrote:
Hi,

has anyone implemented the jbrowse_elasticsearch plugin in Apollo? I'm running from the docker container with an nginx reverse proxy and having trouble adding this plugin. How would I add this to the docker image, with the middleware as well?

The reason we want to do this is the predicted gene model names we have in our tracks are not always intuitive so it's hard to find them by searching in Apollo (ie mTOR is ID'd as "serine/threonine-protein kinase mTOR", so if you just start typing mTOR you get no results). Wondering also if anyone else has encountered this problem and found good solutions, besides manipulating all of our tracks.

Thanks so much,
Michael



This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.




This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.






This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.




This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.



This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.



This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.



This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.
<tomcat.tar.gz>


This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.


VirtualBox_Ubuntu5_21_11_2018_13_06_07.png (293K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: elastic search in genome apollo

nathandunn

Thanks for testing Colin.  I wasn’t sure the exact plugin to test with. 

I did get it to deploy off of master.  I did an ./apollo clean-all 

I did an ./apollo deploy first  (not sure if that made any difference, but it should not have). 

My config is the sample-h2-config.groovy file with the snippet that Colin provided below.

That being said, I didn’t do a full test with the elastic search plugin, but this seems to install it correctly and I get an organism window, etc. when I load the volvox data. 



Nathan


// default username/password on h2 are given here. see docs for adjusting
dataSource {
    pooled = true
    jmxExport = true
    driverClassName = "org.h2.Driver"
    username = "sa"
    password = ""
}
// environment specific settings
environments {
    development {
        // sample config to turn on debug logging in development e.g. for apollo run-local
        log4j.main = {
            debug "grails.app"
        }
        // sample config to edit apollo specific configs in development mode
        apollo {
            gff3.source = "testing"
        }
        dataSource {
            // NOTE: this is in memory, so it will be deleted.
            dbCreate = "update" // one of 'create', 'create-drop', 'update', 'validate', ''
            url = "jdbc:h2:AnnotationDatabase:devDb;MVCC=TRUE;LOCK_TIMEOUT=10000;DB_CLOSE_ON_EXIT=FALSE"
        }
    }
    test {
        dataSource {
            dbCreate = "create-drop" // one of 'create', 'create-drop', 'update', 'validate', ''
            url = "jdbc:h2:mem:testDb;MVCC=TRUE;LOCK_TIMEOUT=10000;DB_CLOSE_ON_EXIT=FALSE"
        }
    }

    //note: not necessarily recommended to use h2 in production mode. see precautions
    production {
        dataSource {
            dbCreate = "update"
            //NOTE: production mode uses file instead of mem database
            //NOTE: Please specify the appropriate file path, otherwise /tmp/prodDb will be used.
            url = "jdbc:h2:/opt/apollo/h2/prodDb;MVCC=TRUE;LOCK_TIMEOUT=10000;DB_CLOSE_ON_EXIT=FALSE"
            properties {
               // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
               jmxEnabled = true
               initialSize = 5
               maxActive = 50
               minIdle = 5
               maxIdle = 25
               maxWait = 10000
               maxAge = 10 * 60000
               timeBetweenEvictionRunsMillis = 5000
               minEvictableIdleTimeMillis = 60000
               validationQuery = "SELECT 1"
               validationQueryTimeout = 3
               validationInterval = 15000
               testOnBorrow = true
               testWhileIdle = true
               testOnReturn = false
               jdbcInterceptors = "ConnectionState"
               defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
    }
}

// Uncomment to change the default memory configurations
//grails.project.fork = [
//        test   : false,
//        // configure settings for the run-app JVM
//        run    : [maxMemory: 2048, minMemory: 64, debug: false, maxPerm: 1024, forkReserve: false],
//        // configure settings for the run-war JVM
//        war    : [maxMemory: 2048, minMemory: 64, debug: false, maxPerm: 1024, forkReserve: false],
//        // configure settings for the Console UI JVM
//        console: [maxMemory: 2048, minMemory: 64, debug: false, maxPerm: 1024]
//]

// Uncomment to make changes
//
jbrowse {
git {
tag = "maint/1.12.5-apollo"
alwaysPull = true
alwaysRecheck = true
}
plugins {
NeatHTMLFeatures{
included = true
}
NeatCanvasFeatures{
included = true
}
RegexSequenceSearch{
included = true
}
HideTrackLabels{
included = true
}
        ElasticSearch {
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
//        MyVariantInfo {
//            git = 'https://github.com/GMOD/myvariantviewer'
//            branch = 'master'
//            alwaysRecheck = "true"
//            alwaysPull = "true"
//        }
//        SashimiPlot {
//            git = 'https://github.com/cmdcolin/sashimiplot'
//            branch = 'master'
//            alwaysPull = "true"
//        }
}
}



On Nov 21, 2018, at 10:09 AM, Colin <[hidden email]> wrote:

I went and tested this and I tried testing adding the plugin to apollo-config.groovy and was getting an error about plugins/ElasticSearch/js/main.js not being loaded apollo run-local. I can see that it gets downloaded to jbrowse-download/plugins/ElasticSearch but it was giving me that error there still. I also ran apollo deploy and then unzipped and the jbrowse plugins folder did not contain ElasticSearch plugin

Screenshot attached and apollo-config.groovy snippet here

jbrowse {
    plugins {
        ElasticSearch {
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
    }
}

-Colin

On Tue, Nov 20, 2018 at 6:57 PM Nathan Dunn <[hidden email]> wrote:

Thanks.  Will have a look.


On Nov 20, 2018, at 5:03 AM, Michael Poeschla <[hidden email]> wrote:

Hi,

here are the logs, docker-compose, actual dockerfile and start and testing of npm (below). I've also added the tomcat logs as an attachment.

docker-compose:
--
version: "3.2"
services:

  apollo:
    volumes:
      - ./data/extdata/:/data
      - postgres-data:/var/lib/postgresql
      - ./config/tomcat/server.xml:/var/lib/tomcat8/conf/server.xml:ro
      - ./config/tomcat/setenv.sh:/usr/share/tomcat8/bin/setenv.sh
    environment:
      APOLLO_ADMIN_[hidden email]
    restart: always
    links:
      - elasticsearch:elasticsearch

  nginx:
    image: library/nginx:alpine
    restart: always
    volumes:
      - ./config/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
      - ./config/secrets:/secrets:ro
    ports:
      - 80:80
      - 443:443
    links:
      - apollo:apollo

  elasticsearch:
    image: elasticsearch:6.4.2
    container_name: elasticsearch
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata1:/usr/share/elasticsearch/data

volumes:
  postgres-data:
  esdata1:
    driver: local
--
The actual Dockerfile:
--
# WebApollo
# VERSION 2.1.X
FROM tomcat:8-jre8
MAINTAINER Eric Rasche <[hidden email]>, Anthony Bretaudeau <[hidden email]>, Nathan Dunn <[hidden email]>
ENV DEBIAN_FRONTEND noninteractive 

RUN apt-get -qq update --fix-missing && \
apt-get --no-install-recommends -y install \
git build-essential maven tomcat8 libpq-dev postgresql-common openjdk-8-jdk wget \
postgresql postgresql-client xmlstarlet netcat libpng-dev \
zlib1g-dev libexpat1-dev ant curl ssl-cert zip unzip

RUN apt-get -qq update --fix-missing && \
apt-get --no-install-recommends -y install nodejs && \
apt-get autoremove -y && apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

RUN cp /usr/lib/jvm/java-8-openjdk-amd64/lib/tools.jar /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext/tools.jar && \
useradd -ms /bin/bash -d /apollo apollo

ENV WEBAPOLLO_VERSION 99c7e54c6e74fbc6705d57de216c9b14b2bfb03b
RUN curl -L https://github.com/GMOD/Apollo/archive/${WEBAPOLLO_VERSION}.tar.gz | tar xzf - --strip-components=1 -C /apollo

# install grails
COPY build.sh /bin/build.sh
ADD apollo-config.groovy /apollo/apollo-config.groovy

RUN chown -R apollo:apollo /apollo
RUN chmod +x /usr/local/bin/blat


USER apollo
RUN curl -s get.sdkman.io | bash
RUN /bin/bash -c "source $HOME/.sdkman/bin/sdkman-init.sh && yes | sdk install grails 2.5.5"
RUN /bin/bash -c "source $HOME/.sdkman/bin/sdkman-init.sh && yes | sdk install gradle 3.2.1"


RUN /bin/bash -c "source $HOME/.profile && source $HOME/.sdkman/bin/sdkman-init.sh && /bin/bash /bin/build.sh"

USER root
ENV CATALINA_HOME=/var/lib/tomcat8
RUN rm -rf ${CATALINA_HOME}/webapps/* && \
cp /apollo/apollo*.war ${CATALINA_HOME}/apollo.war

ENV CONTEXT_PATH ROOT

# Download chado schema
gunzip /chado.sql.gz


ADD launch.sh /launch.sh
CMD "/launch.sh"
--
Starting npm inside the container:
root@995955a711e1:/var/lib/tomcat8/webapps/ROOT/jbrowse/plugins/ElasticSearch# npm start

> jbrowse_elasticsearch@1.0.4 start /var/lib/tomcat8/webapps/ROOT/jbrowse/plugins/ElasticSearch
> node ./bin/www

testing access with curl:
root@995955a711e1:/usr/local/tomcat# curl http://localhost:3000
{"hits":[],"total":0}root@995955a711e1:/usr/local/tomcat#

content of ElasticSearchPlugin:
 ^Croot@995955a711e1:/var/lib/tomcat8/webapps/ROOT/jbrowse/plugins/ElasticSearch# ls -la
total 68
drwxr-x--- 10 tomcat8 tomcat8   271 Nov 14 17:10 .
drwxr-x--- 12 tomcat8 tomcat8   223 Nov 14 16:34 ..
-rw-r-----  1 tomcat8 tomcat8   594 Nov 14 16:16 .eslintrc
-rw-r-----  1 tomcat8 tomcat8   290 Nov 14 16:16 .travis.yml
-rw-r-----  1 tomcat8 tomcat8   686 Nov 14 16:16 ChangeLog.md
-rw-r-----  1 tomcat8 tomcat8   810 Nov 14 16:16 Makefile.PL
-rw-r-----  1 tomcat8 tomcat8  5157 Nov 14 16:16 README.md
-rw-r-----  1 tomcat8 tomcat8  1888 Nov 14 16:44 app.js
drwxr-x---  2 tomcat8 tomcat8    66 Nov 14 16:34 bin
drwxr-x---  2 tomcat8 tomcat8    22 Nov 14 16:34 css
drwxr-x---  2 tomcat8 tomcat8    22 Nov 14 16:34 docs
drwxr-x---  2 tomcat8 tomcat8    45 Nov 14 16:34 img
drwxr-x---  3 tomcat8 tomcat8    65 Nov 14 16:34 js
drwxr-xr-x 24 root    root     4096 Nov 14 17:10 node_modules
-rw-r-----  1 tomcat8 tomcat8 25820 Nov 14 17:10 package-lock.json
-rw-r-----  1 tomcat8 tomcat8   725 Nov 14 16:16 package.json
-rw-r-----  1 tomcat8 tomcat8   727 Nov 14 16:16 setup.sh
drwxr-x---  3 tomcat8 tomcat8    19 Nov 14 16:34 src
drwxr-x---  3 tomcat8 tomcat8    33 Nov 14 16:34 test

..And the logs are attached.
Thanks!
Michael



From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Saturday, November 17, 2018 4:06:58 AM
Subject: Re: [apollo] elastic search in genome apollo

Sorry for the late response. 
Can you provide the browser console logs and any tomcat logs (catalina.out)? 
Your modified docket file may be helpful as well.

Nathan

On Nov 15, 2018, at 10:26 AM, Michael Poeschla <[hidden email]> wrote:

yes - we don't see anything in the browser 
Thanks,
Michael

From: "Colin" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 7:09:45 PM
Subject: Re: [apollo] elastic search in genome apollo

So you are saying that the plugin is not being added e.g. you don't have a menu item for the ElasticSearch dialog in jbrowse?
-Colin

On Thu, Nov 15, 2018 at 9:07 AM Michael Poeschla <[hidden email]> wrote:
hi,
elasticsearch was installed with docker. Here's the tracklist.json for the organism that we generated the index for. Is there any other info I could provide that would help?

{
   "elasticIndexName" : "nfz",
   "elasticSearchUrl" : "localhost:3000",
   "formatVersion" : 1,
   "names" : {
      "type" : "Hash",
      "url" : "names/"
   },
   "plugins" : [
      "ElasticSearch"
   ],
   "tracks" : [
      {
         "category" : "Reference sequence",
         "chunkSize" : 20000,
         "key" : "Reference sequence",
         "label" : "DNA",
         "seqType" : "dna",
         "storeClass" : "JBrowse/Store/Sequence/StaticChunked",
         "type" : "SequenceTrack",
         "urlTemplate" : "seq/{refseq_dirpath}/{refseq}-"
      },
      {
         "bicolor_pivot" : 0.5,
         "label" : "GC Content",
         "storeClass" : "JBrowse/Store/SeqFeature/SequenceChunks",
         "type" : "GCContent/View/Track/GCContentXY",
         "urlTemplate" : "seq/{refseq_dirpath}/{refseq}-"
      },
      {
         "compress" : 0,
         "key" : "Repeats",
         "label" : "Repeats",
         "storeClass" : "JBrowse/Store/SeqFeature/NCList",
         "style" : {
            "className" : "feature"
         },
         "trackType" : "JBrowse/View/Track/CanvasFeatures",
         "type" : "JBrowse/View/Track/CanvasFeatures",
         "urlTemplate" : "tracks/Repeats/{refseq}/trackData.json"
      },
      {
         "compress" : 0,
         "key" : "Annotation",
         "label" : "Annotationfromsumgff",
         "storeClass" : "JBrowse/Store/SeqFeature/NCList",
         "style" : {
            "className" : "feature"
         },
         "trackType" : "CanvasFeatures",
         "type" : "CanvasFeatures",
         "urlTemplate" : "tracks/Annotationfromsumgff/{refseq}/trackData.json"
      }
   ]
}


From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 11:33:03 AM
Subject: Re: [apollo] elastic search in genome apollo

Thanks.  Also on Europe time for another 20 minutes.
Can you also post any other specifics of failure?

Have you installed elasticsearch via docker or manually ?

Nathan

On Nov 15, 2018, at 11:10 AM, Michael Poeschla <[hidden email]> wrote:

Here it is - sorry for the late replies, I'm on europe time.

environments {
    development {
    }
    test {
    }
    production {
        dataSource {
            dbCreate = "update"
            username = System.getenv("WEBAPOLLO_DB_USERNAME") ?: *
            password = System.getenv("WEBAPOLLO_DB_PASSWORD") ?: *

            driverClassName = "org.postgresql.Driver"
            dialect = "org.hibernate.dialect.PostgresPlusDialect"
            url = "jdbc:postgresql://${System.getenv("WEBAPOLLO_DB_HOST")?:"127.0.0.1"}/${System.getenv("WEBAPOLLO_DB_NAME")?:"apollo"}"

            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = false
                initialSize                   = 5
                maxActive                     = 50
                minIdle                       = 5
                maxIdle                       = 25
                maxWait                       = 10000
                maxAge                        = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis    = 60000
                validationQuery               = "SELECT 1"
                validationQueryTimeout        = 3
                validationInterval            = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
        dataSource_chado {
            dbCreate = "update"
            username = System.getenv("CHADO_DB_USERNAME") ?: *
            password = System.getenv("CHADO_DB_PASSWORD") ?: *

            driverClassName = "org.postgresql.Driver"
            dialect = "org.hibernate.dialect.PostgresPlusDialect"

            url = "jdbc:postgresql://${System.getenv("CHADO_DB_HOST")?:"127.0.0.1"}/${System.getenv("CHADO_DB_NAME")?:"chado"}"

            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = false
                initialSize                   = 5
                maxActive                     = 50
                minIdle                       = 5
                maxIdle                       = 25
                maxWait                       = 10000
                maxAge                        = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis    = 60000
                validationQuery               = "SELECT 1"
                validationQueryTimeout        = 3
                validationInterval            = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
    }
}

apollo {
    default_minimum_intron_size = System.getenv("WEBAPOLLO_MINIMUM_INTRON_SIZE") ? System.getenv("WEBAPOLLO_MINIMUM_INTRON_SIZE").toInteger() : 1
    history_size = System.getenv("WEBAPOLLO_HISTORY_SIZE") ? System.getenv("WEBAPOLLO_HISTORY_SIZE").toInteger() : 0
    overlapper_class = System.getenv("WEBAPOLLO_OVERLAPPER_CLASS") ?: "org.bbop.apollo.sequence.OrfOverlapper"
    use_cds_for_new_transcripts = System.getenv("WEBAPOLLO_CDS_FOR_NEW_TRANSCRIPTS").equals("true") // will default to false
    feature_has_dbxrefs = System.getenv("WEBAPOLLO_FEATURE_HAS_DBXREFS") ?: true
    feature_has_attributes = System.getenv("WEBAPOLLO_FEATURE_HAS_ATTRS") ?: true
    feature_has_pubmed_ids = System.getenv("WEBAPOLLO_FEATURE_HAS_PUBMED") ?: true
    feature_has_go_ids = System.getenv("WEBAPOLLO_FEATURE_HAS_GO") ?: true
    feature_has_comments = System.getenv("WEBAPOLLO_FEATURE_HAS_COMMENTS") ?: true
    feature_has_status = System.getenv("WEBAPOLLO_FEATURE_HAS_STATUS") ?: true
    translation_table = "/config/translation_tables/ncbi_" + (System.getenv("WEBAPOLLO_TRANSLATION_TABLE") ?: "1") + "_translation_table.txt"
    get_translation_code = System.getenv("WEBAPOLLO_TRANSLATION_TABLE") ? System.getenv("WEBAPOLLO_TRANSLATION_TABLE").toInteger() : 1

    // ADDED BLAT AND SET PATH TO BIN
    sequence_search_tools = [
        blat_nuc: [
          search_exe: "/usr/local/bin/blat",
          search_class: "org.bbop.apollo.sequence.search.blat.BlatCommandLineNucleotideToNucleotide",
          name: "Blat nucleotide",
          params: ""
        ],
        blat_prot: [
          search_exe: "/usr/local/bin/blat",
          search_class: "org.bbop.apollo.sequence.search.blat.BlatCommandLineProteinToNucleotide",
          name: "Blat protein",
          params: ""
        ]
      ]
    // TODO: should come from config or via preferences database
    splice_donor_sites = System.getenv("WEBAPOLLO_SPLICE_DONOR_SITES") ? System.getenv("WEBAPOLLO_SPLICE_DONOR_SITES").split(",") : ["GT"]
    splice_acceptor_sites = System.getenv("WEBAPOLLO_SPLICE_ACCEPTOR_SITES") ? System.getenv("WEBAPOLLO_SPLICE_ACCEPTOR_SITES").split(",") : ["AG"]
    gff3.source = System.getenv("WEBAPOLLO_GFF3_SOURCE") ?: "."

    google_analytics = System.getenv("WEBAPOLLO_GOOGLE_ANALYTICS_ID") ?: ["UA-62921593-1"]

    admin{
        username = System.getenv("APOLLO_ADMIN_EMAIL") ?: *
        password = System.getenv("APOLLO_ADMIN_PASSWORD") ?: *
        firstName = System.getenv("APOLLO_ADMIN_FIRST_NAME") ?: *
        lastName = System.getenv("APOLLO_ADMIN_LAST_NAME") ?: *
    }
}

jbrowse {
    git {
        url = "https://github.com/GMOD/jbrowse"
        tag = "maint/1.12.5-apollo"
    }
    plugins {
        WebApollo{
            included = true
        }
        NeatHTMLFeatures{
            included = System.getenv("WEBAPOLLO_JBROWSE_PLUGIN_NEATHTML") ?: true
        }
        NeatCanvasFeatures{
            included = System.getenv("WEBAPOLLO_JBROWSE_PLUGIN_NEATCANVAS") ?: true
        }
        RegexSequenceSearch{
            included = true
        }
        HideTrackLabels{
            included = true
        }
        //EDIT: ADDED GC CONTENT
        GCContent{
            git = 'https://github.com/cmdcolin/GCContent'
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
        //EDIT: ADDS ELASTIC SEARCH PLUGIN
        ElasticSearch{
            git = 'https://github.com/elsiklab/jbrowse_elasticsearch'
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
    }
}

From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 12:30:50 AM
Subject: Re: [apollo] elastic search in genome apollo

Can you post your apollo config file? 

Nathan

On Nov 14, 2018, at 6:45 PM, Michael Poeschla <[hidden email]> wrote:

Hi, 
I'm following the instructions from https://github.com/elsiklab/jbrowse_elasticsearch

we have elasticsearch and the middleware app.js running. I was able to execute generate-names.pl and this seemed to work correctly from the terminal output. The changes also got picked up in the tracklist.json for the genome. But there is no sign of the elasticsearch plugin in web apollo...added the plugin to the config.groovy, and we have another jbrowse plugin (GC content) working fine so not sure what's the problem here. 


Thanks for all of your help! :)
Michael 

From: "Nathan Dunn" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Tuesday, November 13, 2018 6:33:05 PM
Subject: Re: [apollo] elastic search in genome apollo


Michael,

What problems are you running it when adding the plugins? 

Nathan


On Nov 13, 2018, at 4:55 PM, Colin <[hidden email]> wrote:

Aside from changing your gene identifiers, I think the full text search from elasticsearch could help. I forwarded some comments here about that https://github.com/elsiklab/jbrowse_elasticsearch/issues/7

Potentially data that is loaded in Apollo backend could use text similarity %like% or full text sql queries but for normal jbrowse tracks that are not loaded in the sql database, the jbrowse_elasticsearch plugin seems reasonable.

-Colin

On Tue, Nov 13, 2018 at 7:35 AM Michael Poeschla <[hidden email]> wrote:
Hi,

has anyone implemented the jbrowse_elasticsearch plugin in Apollo? I'm running from the docker container with an nginx reverse proxy and having trouble adding this plugin. How would I add this to the docker image, with the middleware as well?

The reason we want to do this is the predicted gene model names we have in our tracks are not always intuitive so it's hard to find them by searching in Apollo (ie mTOR is ID'd as "serine/threonine-protein kinase mTOR", so if you just start typing mTOR you get no results). Wondering also if anyone else has encountered this problem and found good solutions, besides manipulating all of our tracks.

Thanks so much,
Michael



This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.




This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.






This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.




This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.



This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.



This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.



This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.
<tomcat.tar.gz>


This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

<VirtualBox_Ubuntu5_21_11_2018_13_06_07.png>


This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.






This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.