GMOD › Apollo

elastic search in genome apollo

Classic

List

Threaded

17 messages Options

Michael Poeschla

elastic search in genome apollo

Hi,

has anyone implemented the jbrowse_elasticsearch plugin in Apollo? I'm running from the docker container with an nginx reverse proxy and having trouble adding this plugin. How would I add this to the docker image, with the middleware as well?

The reason we want to do this is the predicted gene model names we have in our tracks are not always intuitive so it's hard to find them by searching in Apollo (ie mTOR is ID'd as "serine/threonine-protein kinase mTOR", so if you just start typing mTOR you get no results). Wondering also if anyone else has encountered this problem and found good solutions, besides manipulating all of our tracks.

Thanks so much,
Michael

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

Colin

Re: elastic search in genome apollo

Aside from changing your gene identifiers, I think the full text search from elasticsearch could help. I forwarded some comments here about that https://github.com/elsiklab/jbrowse_elasticsearch/issues/7

Potentially data that is loaded in Apollo backend could use text similarity %like% or full text sql queries but for normal jbrowse tracks that are not loaded in the sql database, the jbrowse_elasticsearch plugin seems reasonable.

-Colin

On Tue, Nov 13, 2018 at 7:35 AM Michael Poeschla <[hidden email]> wrote:

Hi,

has anyone implemented the jbrowse_elasticsearch plugin in Apollo? I'm running from the docker container with an nginx reverse proxy and having trouble adding this plugin. How would I add this to the docker image, with the middleware as well?

The reason we want to do this is the predicted gene model names we have in our tracks are not always intuitive so it's hard to find them by searching in Apollo (ie mTOR is ID'd as "serine/threonine-protein kinase mTOR", so if you just start typing mTOR you get no results). Wondering also if anyone else has encountered this problem and found good solutions, besides manipulating all of our tracks.

Thanks so much,
Michael

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

nathandunn

Re: elastic search in genome apollo

Michael,

What problems are you running it when adding the plugins?

Nathan

On Nov 13, 2018, at 4:55 PM, Colin <[hidden email]> wrote:

Aside from changing your gene identifiers, I think the full text search from elasticsearch could help. I forwarded some comments here about that https://github.com/elsiklab/jbrowse_elasticsearch/issues/7

Potentially data that is loaded in Apollo backend could use text similarity %like% or full text sql queries but for normal jbrowse tracks that are not loaded in the sql database, the jbrowse_elasticsearch plugin seems reasonable.

-Colin

On Tue, Nov 13, 2018 at 7:35 AM Michael Poeschla <[hidden email]> wrote:
Hi,

has anyone implemented the jbrowse_elasticsearch plugin in Apollo? I'm running from the docker container with an nginx reverse proxy and having trouble adding this plugin. How would I add this to the docker image, with the middleware as well?

The reason we want to do this is the predicted gene model names we have in our tracks are not always intuitive so it's hard to find them by searching in Apollo (ie mTOR is ID'd as "serine/threonine-protein kinase mTOR", so if you just start typing mTOR you get no results). Wondering also if anyone else has encountered this problem and found good solutions, besides manipulating all of our tracks.

Thanks so much,
Michael

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

Michael Poeschla

Re: elastic search in genome apollo

Hi,

I'm following the instructions from https://github.com/elsiklab/jbrowse_elasticsearch

we have elasticsearch and the middleware app.js running. I was able to execute generate-names.pl and this seemed to work correctly from the terminal output. The changes also got picked up in the tracklist.json for the genome. But there is no sign of the elasticsearch plugin in web apollo...added the plugin to the config.groovy, and we have another jbrowse plugin (GC content) working fine so not sure what's the problem here.

Thanks for all of your help! :)

Michael

From: "Nathan Dunn" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Tuesday, November 13, 2018 6:33:05 PM
Subject: Re: [apollo] elastic search in genome apollo

Michael,

What problems are you running it when adding the plugins?

Nathan

On Nov 13, 2018, at 4:55 PM, Colin <[hidden email]> wrote:

Aside from changing your gene identifiers, I think the full text search from elasticsearch could help. I forwarded some comments here about that https://github.com/elsiklab/jbrowse_elasticsearch/issues/7

Potentially data that is loaded in Apollo backend could use text similarity %like% or full text sql queries but for normal jbrowse tracks that are not loaded in the sql database, the jbrowse_elasticsearch plugin seems reasonable.

-Colin

On Tue, Nov 13, 2018 at 7:35 AM Michael Poeschla <[hidden email]> wrote:
Hi,

has anyone implemented the jbrowse_elasticsearch plugin in Apollo? I'm running from the docker container with an nginx reverse proxy and having trouble adding this plugin. How would I add this to the docker image, with the middleware as well?

The reason we want to do this is the predicted gene model names we have in our tracks are not always intuitive so it's hard to find them by searching in Apollo (ie mTOR is ID'd as "serine/threonine-protein kinase mTOR", so if you just start typing mTOR you get no results). Wondering also if anyone else has encountered this problem and found good solutions, besides manipulating all of our tracks.

Thanks so much,
Michael

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

nathandunn

Re: elastic search in genome apollo

Can you post your apollo config file?

Nathan

On Nov 14, 2018, at 6:45 PM, Michael Poeschla <[hidden email]> wrote:

Hi,
I'm following the instructions from https://github.com/elsiklab/jbrowse_elasticsearch

we have elasticsearch and the middleware app.js running. I was able to execute generate-names.pl and this seemed to work correctly from the terminal output. The changes also got picked up in the tracklist.json for the genome. But there is no sign of the elasticsearch plugin in web apollo...added the plugin to the config.groovy, and we have another jbrowse plugin (GC content) working fine so not sure what's the problem here.

Thanks for all of your help! :)
Michael
From: "Nathan Dunn" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Tuesday, November 13, 2018 6:33:05 PM
Subject: Re: [apollo] elastic search in genome apollo

Michael,

What problems are you running it when adding the plugins?

Nathan

On Nov 13, 2018, at 4:55 PM, Colin <[hidden email]> wrote:

Aside from changing your gene identifiers, I think the full text search from elasticsearch could help. I forwarded some comments here about that https://github.com/elsiklab/jbrowse_elasticsearch/issues/7

Potentially data that is loaded in Apollo backend could use text similarity %like% or full text sql queries but for normal jbrowse tracks that are not loaded in the sql database, the jbrowse_elasticsearch plugin seems reasonable.

-Colin

On Tue, Nov 13, 2018 at 7:35 AM Michael Poeschla <[hidden email]> wrote:
Hi,

has anyone implemented the jbrowse_elasticsearch plugin in Apollo? I'm running from the docker container with an nginx reverse proxy and having trouble adding this plugin. How would I add this to the docker image, with the middleware as well?

The reason we want to do this is the predicted gene model names we have in our tracks are not always intuitive so it's hard to find them by searching in Apollo (ie mTOR is ID'd as "serine/threonine-protein kinase mTOR", so if you just start typing mTOR you get no results). Wondering also if anyone else has encountered this problem and found good solutions, besides manipulating all of our tracks.

Thanks so much,
Michael

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

Michael Poeschla

Re: elastic search in genome apollo

Here it is - sorry for the late replies, I'm on europe time.

environments {
    development {
    }
    test {
    }
    production {
        dataSource {
            dbCreate = "update"
            username = System.getenv("WEBAPOLLO_DB_USERNAME") ?: *
            password = System.getenv("WEBAPOLLO_DB_PASSWORD") ?: *

            driverClassName = "org.postgresql.Driver"
            dialect = "org.hibernate.dialect.PostgresPlusDialect"
            url = "jdbc:postgresql://${System.getenv("WEBAPOLLO_DB_HOST")?:"127.0.0.1"}/${System.getenv("WEBAPOLLO_DB_NAME")?:"apollo"}"

            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = false
                initialSize                   = 5
                maxActive                     = 50
                minIdle                       = 5
                maxIdle                       = 25
                maxWait                       = 10000
                maxAge                        = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis    = 60000
                validationQuery               = "SELECT 1"
                validationQueryTimeout        = 3
                validationInterval            = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
        dataSource_chado {
            dbCreate = "update"
            username = System.getenv("CHADO_DB_USERNAME") ?: *
            password = System.getenv("CHADO_DB_PASSWORD") ?: *

            driverClassName = "org.postgresql.Driver"
            dialect = "org.hibernate.dialect.PostgresPlusDialect"

            url = "jdbc:postgresql://${System.getenv("CHADO_DB_HOST")?:"127.0.0.1"}/${System.getenv("CHADO_DB_NAME")?:"chado"}"

            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = false
                initialSize                   = 5
                maxActive                     = 50
                minIdle                       = 5
                maxIdle                       = 25
                maxWait                       = 10000
                maxAge                        = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis    = 60000
                validationQuery               = "SELECT 1"
                validationQueryTimeout        = 3
                validationInterval            = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
    }
}

apollo {
    default_minimum_intron_size = System.getenv("WEBAPOLLO_MINIMUM_INTRON_SIZE") ? System.getenv("WEBAPOLLO_MINIMUM_INTRON_SIZE").toInteger() : 1
    history_size = System.getenv("WEBAPOLLO_HISTORY_SIZE") ? System.getenv("WEBAPOLLO_HISTORY_SIZE").toInteger() : 0
    overlapper_class = System.getenv("WEBAPOLLO_OVERLAPPER_CLASS") ?: "org.bbop.apollo.sequence.OrfOverlapper"
    use_cds_for_new_transcripts = System.getenv("WEBAPOLLO_CDS_FOR_NEW_TRANSCRIPTS").equals("true") // will default to false
    feature_has_dbxrefs = System.getenv("WEBAPOLLO_FEATURE_HAS_DBXREFS") ?: true
    feature_has_attributes = System.getenv("WEBAPOLLO_FEATURE_HAS_ATTRS") ?: true
    feature_has_pubmed_ids = System.getenv("WEBAPOLLO_FEATURE_HAS_PUBMED") ?: true
    feature_has_go_ids = System.getenv("WEBAPOLLO_FEATURE_HAS_GO") ?: true
    feature_has_comments = System.getenv("WEBAPOLLO_FEATURE_HAS_COMMENTS") ?: true
    feature_has_status = System.getenv("WEBAPOLLO_FEATURE_HAS_STATUS") ?: true
    translation_table = "/config/translation_tables/ncbi_" + (System.getenv("WEBAPOLLO_TRANSLATION_TABLE") ?: "1") + "_translation_table.txt"
    get_translation_code = System.getenv("WEBAPOLLO_TRANSLATION_TABLE") ? System.getenv("WEBAPOLLO_TRANSLATION_TABLE").toInteger() : 1

    // ADDED BLAT AND SET PATH TO BIN
    sequence_search_tools = [
        blat_nuc: [
          search_exe: "/usr/local/bin/blat",
          search_class: "org.bbop.apollo.sequence.search.blat.BlatCommandLineNucleotideToNucleotide",
          name: "Blat nucleotide",
          params: ""
        ],
        blat_prot: [
          search_exe: "/usr/local/bin/blat",
          search_class: "org.bbop.apollo.sequence.search.blat.BlatCommandLineProteinToNucleotide",
          name: "Blat protein",
          params: ""
        ]
      ]
    // TODO: should come from config or via preferences database
    splice_donor_sites = System.getenv("WEBAPOLLO_SPLICE_DONOR_SITES") ? System.getenv("WEBAPOLLO_SPLICE_DONOR_SITES").split(",") : ["GT"]
    splice_acceptor_sites = System.getenv("WEBAPOLLO_SPLICE_ACCEPTOR_SITES") ? System.getenv("WEBAPOLLO_SPLICE_ACCEPTOR_SITES").split(",") : ["AG"]
    gff3.source = System.getenv("WEBAPOLLO_GFF3_SOURCE") ?: "."

    google_analytics = System.getenv("WEBAPOLLO_GOOGLE_ANALYTICS_ID") ?: ["UA-62921593-1"]

    admin{
        username = System.getenv("APOLLO_ADMIN_EMAIL") ?: *
        password = System.getenv("APOLLO_ADMIN_PASSWORD") ?: *
        firstName = System.getenv("APOLLO_ADMIN_FIRST_NAME") ?: *
        lastName = System.getenv("APOLLO_ADMIN_LAST_NAME") ?: *
    }
}

jbrowse {
    git {
        url = "https://github.com/GMOD/jbrowse"
        tag = "maint/1.12.5-apollo"
    }
    plugins {
        WebApollo{
            included = true
        }
        NeatHTMLFeatures{
            included = System.getenv("WEBAPOLLO_JBROWSE_PLUGIN_NEATHTML") ?: true
        }
        NeatCanvasFeatures{
            included = System.getenv("WEBAPOLLO_JBROWSE_PLUGIN_NEATCANVAS") ?: true
        }
        RegexSequenceSearch{
            included = true
        }
        HideTrackLabels{
            included = true
        }
        //EDIT: ADDED GC CONTENT
        GCContent{
            git = 'https://github.com/cmdcolin/GCContent'
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
        //EDIT: ADDS ELASTIC SEARCH PLUGIN
        ElasticSearch{
            git = 'https://github.com/elsiklab/jbrowse_elasticsearch'
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
    }

}

From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 12:30:50 AM
Subject: Re: [apollo] elastic search in genome apollo

Can you post your apollo config file?

Nathan

On Nov 14, 2018, at 6:45 PM, Michael Poeschla <[hidden email]> wrote:

Hi,
I'm following the instructions from https://github.com/elsiklab/jbrowse_elasticsearch

we have elasticsearch and the middleware app.js running. I was able to execute generate-names.pl and this seemed to work correctly from the terminal output. The changes also got picked up in the tracklist.json for the genome. But there is no sign of the elasticsearch plugin in web apollo...added the plugin to the config.groovy, and we have another jbrowse plugin (GC content) working fine so not sure what's the problem here.

Thanks for all of your help! :)
Michael
From: "Nathan Dunn" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Tuesday, November 13, 2018 6:33:05 PM
Subject: Re: [apollo] elastic search in genome apollo

Michael,

What problems are you running it when adding the plugins?

Nathan

On Nov 13, 2018, at 4:55 PM, Colin <[hidden email]> wrote:

Aside from changing your gene identifiers, I think the full text search from elasticsearch could help. I forwarded some comments here about that https://github.com/elsiklab/jbrowse_elasticsearch/issues/7

Potentially data that is loaded in Apollo backend could use text similarity %like% or full text sql queries but for normal jbrowse tracks that are not loaded in the sql database, the jbrowse_elasticsearch plugin seems reasonable.

-Colin

On Tue, Nov 13, 2018 at 7:35 AM Michael Poeschla <[hidden email]> wrote:
Hi,

has anyone implemented the jbrowse_elasticsearch plugin in Apollo? I'm running from the docker container with an nginx reverse proxy and having trouble adding this plugin. How would I add this to the docker image, with the middleware as well?

The reason we want to do this is the predicted gene model names we have in our tracks are not always intuitive so it's hard to find them by searching in Apollo (ie mTOR is ID'd as "serine/threonine-protein kinase mTOR", so if you just start typing mTOR you get no results). Wondering also if anyone else has encountered this problem and found good solutions, besides manipulating all of our tracks.

Thanks so much,
Michael

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

nathandunn

Re: elastic search in genome apollo

Thanks. Also on Europe time for another 20 minutes.

Can you also post any other specifics of failure?

Have you installed elasticsearch via docker or manually ?

Nathan

On Nov 15, 2018, at 11:10 AM, Michael Poeschla <[hidden email]> wrote:

Here it is - sorry for the late replies, I'm on europe time.

environments {
    development {
    }
    test {
    }
    production {
        dataSource {
            dbCreate = "update"
            username = System.getenv("WEBAPOLLO_DB_USERNAME") ?: *
            password = System.getenv("WEBAPOLLO_DB_PASSWORD") ?: *

            driverClassName = "org.postgresql.Driver"
            dialect = "org.hibernate.dialect.PostgresPlusDialect"
            url = "jdbc:postgresql://${System.getenv("WEBAPOLLO_DB_HOST")?:"127.0.0.1"}/${System.getenv("WEBAPOLLO_DB_NAME")?:"apollo"}"

            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = false
                initialSize                   = 5
                maxActive                     = 50
                minIdle                       = 5
                maxIdle                       = 25
                maxWait                       = 10000
                maxAge                        = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis    = 60000
                validationQuery               = "SELECT 1"
                validationQueryTimeout        = 3
                validationInterval            = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
        dataSource_chado {
            dbCreate = "update"
            username = System.getenv("CHADO_DB_USERNAME") ?: *
            password = System.getenv("CHADO_DB_PASSWORD") ?: *

            driverClassName = "org.postgresql.Driver"
            dialect = "org.hibernate.dialect.PostgresPlusDialect"

            url = "jdbc:postgresql://${System.getenv("CHADO_DB_HOST")?:"127.0.0.1"}/${System.getenv("CHADO_DB_NAME")?:"chado"}"

            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = false
                initialSize                   = 5
                maxActive                     = 50
                minIdle                       = 5
                maxIdle                       = 25
                maxWait                       = 10000
                maxAge                        = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis    = 60000
                validationQuery               = "SELECT 1"
                validationQueryTimeout        = 3
                validationInterval            = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
    }
}

apollo {
    default_minimum_intron_size = System.getenv("WEBAPOLLO_MINIMUM_INTRON_SIZE") ? System.getenv("WEBAPOLLO_MINIMUM_INTRON_SIZE").toInteger() : 1
    history_size = System.getenv("WEBAPOLLO_HISTORY_SIZE") ? System.getenv("WEBAPOLLO_HISTORY_SIZE").toInteger() : 0
    overlapper_class = System.getenv("WEBAPOLLO_OVERLAPPER_CLASS") ?: "org.bbop.apollo.sequence.OrfOverlapper"
    use_cds_for_new_transcripts = System.getenv("WEBAPOLLO_CDS_FOR_NEW_TRANSCRIPTS").equals("true") // will default to false
    feature_has_dbxrefs = System.getenv("WEBAPOLLO_FEATURE_HAS_DBXREFS") ?: true
    feature_has_attributes = System.getenv("WEBAPOLLO_FEATURE_HAS_ATTRS") ?: true
    feature_has_pubmed_ids = System.getenv("WEBAPOLLO_FEATURE_HAS_PUBMED") ?: true
    feature_has_go_ids = System.getenv("WEBAPOLLO_FEATURE_HAS_GO") ?: true
    feature_has_comments = System.getenv("WEBAPOLLO_FEATURE_HAS_COMMENTS") ?: true
    feature_has_status = System.getenv("WEBAPOLLO_FEATURE_HAS_STATUS") ?: true
    translation_table = "/config/translation_tables/ncbi_" + (System.getenv("WEBAPOLLO_TRANSLATION_TABLE") ?: "1") + "_translation_table.txt"
    get_translation_code = System.getenv("WEBAPOLLO_TRANSLATION_TABLE") ? System.getenv("WEBAPOLLO_TRANSLATION_TABLE").toInteger() : 1

    // ADDED BLAT AND SET PATH TO BIN
    sequence_search_tools = [
        blat_nuc: [
          search_exe: "/usr/local/bin/blat",
          search_class: "org.bbop.apollo.sequence.search.blat.BlatCommandLineNucleotideToNucleotide",
          name: "Blat nucleotide",
          params: ""
        ],
        blat_prot: [
          search_exe: "/usr/local/bin/blat",
          search_class: "org.bbop.apollo.sequence.search.blat.BlatCommandLineProteinToNucleotide",
          name: "Blat protein",
          params: ""
        ]
      ]
    // TODO: should come from config or via preferences database
    splice_donor_sites = System.getenv("WEBAPOLLO_SPLICE_DONOR_SITES") ? System.getenv("WEBAPOLLO_SPLICE_DONOR_SITES").split(",") : ["GT"]
    splice_acceptor_sites = System.getenv("WEBAPOLLO_SPLICE_ACCEPTOR_SITES") ? System.getenv("WEBAPOLLO_SPLICE_ACCEPTOR_SITES").split(",") : ["AG"]
    gff3.source = System.getenv("WEBAPOLLO_GFF3_SOURCE") ?: "."

    google_analytics = System.getenv("WEBAPOLLO_GOOGLE_ANALYTICS_ID") ?: ["UA-62921593-1"]

    admin{
        username = System.getenv("APOLLO_ADMIN_EMAIL") ?: *
        password = System.getenv("APOLLO_ADMIN_PASSWORD") ?: *
        firstName = System.getenv("APOLLO_ADMIN_FIRST_NAME") ?: *
        lastName = System.getenv("APOLLO_ADMIN_LAST_NAME") ?: *
    }
}

jbrowse {
    git {
        url = "https://github.com/GMOD/jbrowse"
        tag = "maint/1.12.5-apollo"
    }
    plugins {
        WebApollo{
            included = true
        }
        NeatHTMLFeatures{
            included = System.getenv("WEBAPOLLO_JBROWSE_PLUGIN_NEATHTML") ?: true
        }
        NeatCanvasFeatures{
            included = System.getenv("WEBAPOLLO_JBROWSE_PLUGIN_NEATCANVAS") ?: true
        }
        RegexSequenceSearch{
            included = true
        }
        HideTrackLabels{
            included = true
        }
        //EDIT: ADDED GC CONTENT
        GCContent{
            git = 'https://github.com/cmdcolin/GCContent'
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
        //EDIT: ADDS ELASTIC SEARCH PLUGIN
        ElasticSearch{
            git = 'https://github.com/elsiklab/jbrowse_elasticsearch'
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
    }
}
From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 12:30:50 AM
Subject: Re: [apollo] elastic search in genome apollo

Can you post your apollo config file?

Nathan

On Nov 14, 2018, at 6:45 PM, Michael Poeschla <[hidden email]> wrote:

Hi,
I'm following the instructions from https://github.com/elsiklab/jbrowse_elasticsearch

we have elasticsearch and the middleware app.js running. I was able to execute generate-names.pl and this seemed to work correctly from the terminal output. The changes also got picked up in the tracklist.json for the genome. But there is no sign of the elasticsearch plugin in web apollo...added the plugin to the config.groovy, and we have another jbrowse plugin (GC content) working fine so not sure what's the problem here.

Thanks for all of your help! :)
Michael
From: "Nathan Dunn" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Tuesday, November 13, 2018 6:33:05 PM
Subject: Re: [apollo] elastic search in genome apollo

Michael,

What problems are you running it when adding the plugins?

Nathan

On Nov 13, 2018, at 4:55 PM, Colin <[hidden email]> wrote:

Aside from changing your gene identifiers, I think the full text search from elasticsearch could help. I forwarded some comments here about that https://github.com/elsiklab/jbrowse_elasticsearch/issues/7

Potentially data that is loaded in Apollo backend could use text similarity %like% or full text sql queries but for normal jbrowse tracks that are not loaded in the sql database, the jbrowse_elasticsearch plugin seems reasonable.

-Colin

On Tue, Nov 13, 2018 at 7:35 AM Michael Poeschla <[hidden email]> wrote:
Hi,

has anyone implemented the jbrowse_elasticsearch plugin in Apollo? I'm running from the docker container with an nginx reverse proxy and having trouble adding this plugin. How would I add this to the docker image, with the middleware as well?

The reason we want to do this is the predicted gene model names we have in our tracks are not always intuitive so it's hard to find them by searching in Apollo (ie mTOR is ID'd as "serine/threonine-protein kinase mTOR", so if you just start typing mTOR you get no results). Wondering also if anyone else has encountered this problem and found good solutions, besides manipulating all of our tracks.

Thanks so much,
Michael

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

Michael Poeschla

Re: elastic search in genome apollo

hi,

elasticsearch was installed with docker. Here's the tracklist.json for the organism that we generated the index for. Is there any other info I could provide that would help?

{
   "elasticIndexName" : "nfz",
   "elasticSearchUrl" : "localhost:3000",
   "formatVersion" : 1,
   "names" : {
      "type" : "Hash",
      "url" : "names/"
   },
   "plugins" : [
      "ElasticSearch"
   ],
   "tracks" : [
      {
         "category" : "Reference sequence",
         "chunkSize" : 20000,
         "key" : "Reference sequence",
         "label" : "DNA",
         "seqType" : "dna",
         "storeClass" : "JBrowse/Store/Sequence/StaticChunked",
         "type" : "SequenceTrack",
         "urlTemplate" : "seq/{refseq_dirpath}/{refseq}-"
      },
      {
         "bicolor_pivot" : 0.5,
         "label" : "GC Content",
         "storeClass" : "JBrowse/Store/SeqFeature/SequenceChunks",
         "type" : "GCContent/View/Track/GCContentXY",
         "urlTemplate" : "seq/{refseq_dirpath}/{refseq}-"
      },
      {
         "compress" : 0,
         "key" : "Repeats",
         "label" : "Repeats",
         "storeClass" : "JBrowse/Store/SeqFeature/NCList",
         "style" : {
            "className" : "feature"
         },
         "trackType" : "JBrowse/View/Track/CanvasFeatures",
         "type" : "JBrowse/View/Track/CanvasFeatures",
         "urlTemplate" : "tracks/Repeats/{refseq}/trackData.json"
      },
      {
         "compress" : 0,
         "key" : "Annotation",
         "label" : "Annotationfromsumgff",
         "storeClass" : "JBrowse/Store/SeqFeature/NCList",
         "style" : {
            "className" : "feature"
         },
         "trackType" : "CanvasFeatures",
         "type" : "CanvasFeatures",
         "urlTemplate" : "tracks/Annotationfromsumgff/{refseq}/trackData.json"
      }
   ]
}

From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 11:33:03 AM
Subject: Re: [apollo] elastic search in genome apollo

Thanks. Also on Europe time for another 20 minutes.

Can you also post any other specifics of failure?

Have you installed elasticsearch via docker or manually ?

Nathan

On Nov 15, 2018, at 11:10 AM, Michael Poeschla <[hidden email]> wrote:

Here it is - sorry for the late replies, I'm on europe time.

environments {
    development {
    }
    test {
    }
    production {
        dataSource {
            dbCreate = "update"
            username = System.getenv("WEBAPOLLO_DB_USERNAME") ?: *
            password = System.getenv("WEBAPOLLO_DB_PASSWORD") ?: *

            driverClassName = "org.postgresql.Driver"
            dialect = "org.hibernate.dialect.PostgresPlusDialect"
            url = "jdbc:postgresql://${System.getenv("WEBAPOLLO_DB_HOST")?:"127.0.0.1"}/${System.getenv("WEBAPOLLO_DB_NAME")?:"apollo"}"

            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = false
                initialSize                   = 5
                maxActive                     = 50
                minIdle                       = 5
                maxIdle                       = 25
                maxWait                       = 10000
                maxAge                        = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis    = 60000
                validationQuery               = "SELECT 1"
                validationQueryTimeout        = 3
                validationInterval            = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
        dataSource_chado {
            dbCreate = "update"
            username = System.getenv("CHADO_DB_USERNAME") ?: *
            password = System.getenv("CHADO_DB_PASSWORD") ?: *

            driverClassName = "org.postgresql.Driver"
            dialect = "org.hibernate.dialect.PostgresPlusDialect"

            url = "jdbc:postgresql://${System.getenv("CHADO_DB_HOST")?:"127.0.0.1"}/${System.getenv("CHADO_DB_NAME")?:"chado"}"

            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = false
                initialSize                   = 5
                maxActive                     = 50
                minIdle                       = 5
                maxIdle                       = 25
                maxWait                       = 10000
                maxAge                        = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis    = 60000
                validationQuery               = "SELECT 1"
                validationQueryTimeout        = 3
                validationInterval            = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
    }
}

apollo {
    default_minimum_intron_size = System.getenv("WEBAPOLLO_MINIMUM_INTRON_SIZE") ? System.getenv("WEBAPOLLO_MINIMUM_INTRON_SIZE").toInteger() : 1
    history_size = System.getenv("WEBAPOLLO_HISTORY_SIZE") ? System.getenv("WEBAPOLLO_HISTORY_SIZE").toInteger() : 0
    overlapper_class = System.getenv("WEBAPOLLO_OVERLAPPER_CLASS") ?: "org.bbop.apollo.sequence.OrfOverlapper"
    use_cds_for_new_transcripts = System.getenv("WEBAPOLLO_CDS_FOR_NEW_TRANSCRIPTS").equals("true") // will default to false
    feature_has_dbxrefs = System.getenv("WEBAPOLLO_FEATURE_HAS_DBXREFS") ?: true
    feature_has_attributes = System.getenv("WEBAPOLLO_FEATURE_HAS_ATTRS") ?: true
    feature_has_pubmed_ids = System.getenv("WEBAPOLLO_FEATURE_HAS_PUBMED") ?: true
    feature_has_go_ids = System.getenv("WEBAPOLLO_FEATURE_HAS_GO") ?: true
    feature_has_comments = System.getenv("WEBAPOLLO_FEATURE_HAS_COMMENTS") ?: true
    feature_has_status = System.getenv("WEBAPOLLO_FEATURE_HAS_STATUS") ?: true
    translation_table = "/config/translation_tables/ncbi_" + (System.getenv("WEBAPOLLO_TRANSLATION_TABLE") ?: "1") + "_translation_table.txt"
    get_translation_code = System.getenv("WEBAPOLLO_TRANSLATION_TABLE") ? System.getenv("WEBAPOLLO_TRANSLATION_TABLE").toInteger() : 1

    // ADDED BLAT AND SET PATH TO BIN
    sequence_search_tools = [
        blat_nuc: [
          search_exe: "/usr/local/bin/blat",
          search_class: "org.bbop.apollo.sequence.search.blat.BlatCommandLineNucleotideToNucleotide",
          name: "Blat nucleotide",
          params: ""
        ],
        blat_prot: [
          search_exe: "/usr/local/bin/blat",
          search_class: "org.bbop.apollo.sequence.search.blat.BlatCommandLineProteinToNucleotide",
          name: "Blat protein",
          params: ""
        ]
      ]
    // TODO: should come from config or via preferences database
    splice_donor_sites = System.getenv("WEBAPOLLO_SPLICE_DONOR_SITES") ? System.getenv("WEBAPOLLO_SPLICE_DONOR_SITES").split(",") : ["GT"]
    splice_acceptor_sites = System.getenv("WEBAPOLLO_SPLICE_ACCEPTOR_SITES") ? System.getenv("WEBAPOLLO_SPLICE_ACCEPTOR_SITES").split(",") : ["AG"]
    gff3.source = System.getenv("WEBAPOLLO_GFF3_SOURCE") ?: "."

    google_analytics = System.getenv("WEBAPOLLO_GOOGLE_ANALYTICS_ID") ?: ["UA-62921593-1"]

    admin{
        username = System.getenv("APOLLO_ADMIN_EMAIL") ?: *
        password = System.getenv("APOLLO_ADMIN_PASSWORD") ?: *
        firstName = System.getenv("APOLLO_ADMIN_FIRST_NAME") ?: *
        lastName = System.getenv("APOLLO_ADMIN_LAST_NAME") ?: *
    }
}

jbrowse {
    git {
        url = "https://github.com/GMOD/jbrowse"
        tag = "maint/1.12.5-apollo"
    }
    plugins {
        WebApollo{
            included = true
        }
        NeatHTMLFeatures{
            included = System.getenv("WEBAPOLLO_JBROWSE_PLUGIN_NEATHTML") ?: true
        }
        NeatCanvasFeatures{
            included = System.getenv("WEBAPOLLO_JBROWSE_PLUGIN_NEATCANVAS") ?: true
        }
        RegexSequenceSearch{
            included = true
        }
        HideTrackLabels{
            included = true
        }
        //EDIT: ADDED GC CONTENT
        GCContent{
            git = 'https://github.com/cmdcolin/GCContent'
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
        //EDIT: ADDS ELASTIC SEARCH PLUGIN
        ElasticSearch{
            git = 'https://github.com/elsiklab/jbrowse_elasticsearch'
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
    }
}
From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 12:30:50 AM
Subject: Re: [apollo] elastic search in genome apollo

Can you post your apollo config file?

Nathan

On Nov 14, 2018, at 6:45 PM, Michael Poeschla <[hidden email]> wrote:

Hi,
I'm following the instructions from https://github.com/elsiklab/jbrowse_elasticsearch

we have elasticsearch and the middleware app.js running. I was able to execute generate-names.pl and this seemed to work correctly from the terminal output. The changes also got picked up in the tracklist.json for the genome. But there is no sign of the elasticsearch plugin in web apollo...added the plugin to the config.groovy, and we have another jbrowse plugin (GC content) working fine so not sure what's the problem here.

Thanks for all of your help! :)
Michael
From: "Nathan Dunn" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Tuesday, November 13, 2018 6:33:05 PM
Subject: Re: [apollo] elastic search in genome apollo

Michael,

What problems are you running it when adding the plugins?

Nathan

On Nov 13, 2018, at 4:55 PM, Colin <[hidden email]> wrote:

Aside from changing your gene identifiers, I think the full text search from elasticsearch could help. I forwarded some comments here about that https://github.com/elsiklab/jbrowse_elasticsearch/issues/7

Potentially data that is loaded in Apollo backend could use text similarity %like% or full text sql queries but for normal jbrowse tracks that are not loaded in the sql database, the jbrowse_elasticsearch plugin seems reasonable.

-Colin

On Tue, Nov 13, 2018 at 7:35 AM Michael Poeschla <[hidden email]> wrote:
Hi,

has anyone implemented the jbrowse_elasticsearch plugin in Apollo? I'm running from the docker container with an nginx reverse proxy and having trouble adding this plugin. How would I add this to the docker image, with the middleware as well?

The reason we want to do this is the predicted gene model names we have in our tracks are not always intuitive so it's hard to find them by searching in Apollo (ie mTOR is ID'd as "serine/threonine-protein kinase mTOR", so if you just start typing mTOR you get no results). Wondering also if anyone else has encountered this problem and found good solutions, besides manipulating all of our tracks.

Thanks so much,
Michael

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

Colin

Re: elastic search in genome apollo

So you are saying that the plugin is not being added e.g. you don't have a menu item for the ElasticSearch dialog in jbrowse?

-Colin

On Thu, Nov 15, 2018 at 9:07 AM Michael Poeschla <[hidden email]> wrote:

hi,
elasticsearch was installed with docker. Here's the tracklist.json for the organism that we generated the index for. Is there any other info I could provide that would help?

{
   "elasticIndexName" : "nfz",
   "elasticSearchUrl" : "localhost:3000",
   "formatVersion" : 1,
   "names" : {
      "type" : "Hash",
      "url" : "names/"
   },
   "plugins" : [
      "ElasticSearch"
   ],
   "tracks" : [
      {
         "category" : "Reference sequence",
         "chunkSize" : 20000,
         "key" : "Reference sequence",
         "label" : "DNA",
         "seqType" : "dna",
         "storeClass" : "JBrowse/Store/Sequence/StaticChunked",
         "type" : "SequenceTrack",
         "urlTemplate" : "seq/{refseq_dirpath}/{refseq}-"
      },
      {
         "bicolor_pivot" : 0.5,
         "label" : "GC Content",
         "storeClass" : "JBrowse/Store/SeqFeature/SequenceChunks",
         "type" : "GCContent/View/Track/GCContentXY",
         "urlTemplate" : "seq/{refseq_dirpath}/{refseq}-"
      },
      {
         "compress" : 0,
         "key" : "Repeats",
         "label" : "Repeats",
         "storeClass" : "JBrowse/Store/SeqFeature/NCList",
         "style" : {
            "className" : "feature"
         },
         "trackType" : "JBrowse/View/Track/CanvasFeatures",
         "type" : "JBrowse/View/Track/CanvasFeatures",
         "urlTemplate" : "tracks/Repeats/{refseq}/trackData.json"
      },
      {
         "compress" : 0,
         "key" : "Annotation",
         "label" : "Annotationfromsumgff",
         "storeClass" : "JBrowse/Store/SeqFeature/NCList",
         "style" : {
            "className" : "feature"
         },
         "trackType" : "CanvasFeatures",
         "type" : "CanvasFeatures",
         "urlTemplate" : "tracks/Annotationfromsumgff/{refseq}/trackData.json"
      }
   ]
}

From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 11:33:03 AM
Subject: Re: [apollo] elastic search in genome apollo

Thanks. Also on Europe time for another 20 minutes.
Can you also post any other specifics of failure?

Have you installed elasticsearch via docker or manually ?

Nathan

On Nov 15, 2018, at 11:10 AM, Michael Poeschla <[hidden email]> wrote:

Here it is - sorry for the late replies, I'm on europe time.

environments {
    development {
    }
    test {
    }
    production {
        dataSource {
            dbCreate = "update"
            username = System.getenv("WEBAPOLLO_DB_USERNAME") ?: *
            password = System.getenv("WEBAPOLLO_DB_PASSWORD") ?: *

            driverClassName = "org.postgresql.Driver"
            dialect = "org.hibernate.dialect.PostgresPlusDialect"
            url = "jdbc:postgresql://${System.getenv("WEBAPOLLO_DB_HOST")?:"127.0.0.1"}/${System.getenv("WEBAPOLLO_DB_NAME")?:"apollo"}"

            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = false
                initialSize                   = 5
                maxActive                     = 50
                minIdle                       = 5
                maxIdle                       = 25
                maxWait                       = 10000
                maxAge                        = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis    = 60000
                validationQuery               = "SELECT 1"
                validationQueryTimeout        = 3
                validationInterval            = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
        dataSource_chado {
            dbCreate = "update"
            username = System.getenv("CHADO_DB_USERNAME") ?: *
            password = System.getenv("CHADO_DB_PASSWORD") ?: *

            driverClassName = "org.postgresql.Driver"
            dialect = "org.hibernate.dialect.PostgresPlusDialect"

            url = "jdbc:postgresql://${System.getenv("CHADO_DB_HOST")?:"127.0.0.1"}/${System.getenv("CHADO_DB_NAME")?:"chado"}"

            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = false
                initialSize                   = 5
                maxActive                     = 50
                minIdle                       = 5
                maxIdle                       = 25
                maxWait                       = 10000
                maxAge                        = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis    = 60000
                validationQuery               = "SELECT 1"
                validationQueryTimeout        = 3
                validationInterval            = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
    }
}

apollo {
    default_minimum_intron_size = System.getenv("WEBAPOLLO_MINIMUM_INTRON_SIZE") ? System.getenv("WEBAPOLLO_MINIMUM_INTRON_SIZE").toInteger() : 1
    history_size = System.getenv("WEBAPOLLO_HISTORY_SIZE") ? System.getenv("WEBAPOLLO_HISTORY_SIZE").toInteger() : 0
    overlapper_class = System.getenv("WEBAPOLLO_OVERLAPPER_CLASS") ?: "org.bbop.apollo.sequence.OrfOverlapper"
    use_cds_for_new_transcripts = System.getenv("WEBAPOLLO_CDS_FOR_NEW_TRANSCRIPTS").equals("true") // will default to false
    feature_has_dbxrefs = System.getenv("WEBAPOLLO_FEATURE_HAS_DBXREFS") ?: true
    feature_has_attributes = System.getenv("WEBAPOLLO_FEATURE_HAS_ATTRS") ?: true
    feature_has_pubmed_ids = System.getenv("WEBAPOLLO_FEATURE_HAS_PUBMED") ?: true
    feature_has_go_ids = System.getenv("WEBAPOLLO_FEATURE_HAS_GO") ?: true
    feature_has_comments = System.getenv("WEBAPOLLO_FEATURE_HAS_COMMENTS") ?: true
    feature_has_status = System.getenv("WEBAPOLLO_FEATURE_HAS_STATUS") ?: true
    translation_table = "/config/translation_tables/ncbi_" + (System.getenv("WEBAPOLLO_TRANSLATION_TABLE") ?: "1") + "_translation_table.txt"
    get_translation_code = System.getenv("WEBAPOLLO_TRANSLATION_TABLE") ? System.getenv("WEBAPOLLO_TRANSLATION_TABLE").toInteger() : 1

    // ADDED BLAT AND SET PATH TO BIN
    sequence_search_tools = [
        blat_nuc: [
          search_exe: "/usr/local/bin/blat",
          search_class: "org.bbop.apollo.sequence.search.blat.BlatCommandLineNucleotideToNucleotide",
          name: "Blat nucleotide",
          params: ""
        ],
        blat_prot: [
          search_exe: "/usr/local/bin/blat",
          search_class: "org.bbop.apollo.sequence.search.blat.BlatCommandLineProteinToNucleotide",
          name: "Blat protein",
          params: ""
        ]
      ]
    // TODO: should come from config or via preferences database
    splice_donor_sites = System.getenv("WEBAPOLLO_SPLICE_DONOR_SITES") ? System.getenv("WEBAPOLLO_SPLICE_DONOR_SITES").split(",") : ["GT"]
    splice_acceptor_sites = System.getenv("WEBAPOLLO_SPLICE_ACCEPTOR_SITES") ? System.getenv("WEBAPOLLO_SPLICE_ACCEPTOR_SITES").split(",") : ["AG"]
    gff3.source = System.getenv("WEBAPOLLO_GFF3_SOURCE") ?: "."

    google_analytics = System.getenv("WEBAPOLLO_GOOGLE_ANALYTICS_ID") ?: ["UA-62921593-1"]

    admin{
        username = System.getenv("APOLLO_ADMIN_EMAIL") ?: *
        password = System.getenv("APOLLO_ADMIN_PASSWORD") ?: *
        firstName = System.getenv("APOLLO_ADMIN_FIRST_NAME") ?: *
        lastName = System.getenv("APOLLO_ADMIN_LAST_NAME") ?: *
    }
}

jbrowse {
    git {
        url = "https://github.com/GMOD/jbrowse"
        tag = "maint/1.12.5-apollo"
    }
    plugins {
        WebApollo{
            included = true
        }
        NeatHTMLFeatures{
            included = System.getenv("WEBAPOLLO_JBROWSE_PLUGIN_NEATHTML") ?: true
        }
        NeatCanvasFeatures{
            included = System.getenv("WEBAPOLLO_JBROWSE_PLUGIN_NEATCANVAS") ?: true
        }
        RegexSequenceSearch{
            included = true
        }
        HideTrackLabels{
            included = true
        }
        //EDIT: ADDED GC CONTENT
        GCContent{
            git = 'https://github.com/cmdcolin/GCContent'
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
        //EDIT: ADDS ELASTIC SEARCH PLUGIN
        ElasticSearch{
            git = 'https://github.com/elsiklab/jbrowse_elasticsearch'
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
    }
}
From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 12:30:50 AM
Subject: Re: [apollo] elastic search in genome apollo

Can you post your apollo config file?

Nathan

On Nov 14, 2018, at 6:45 PM, Michael Poeschla <[hidden email]> wrote:

Hi,
I'm following the instructions from https://github.com/elsiklab/jbrowse_elasticsearch

we have elasticsearch and the middleware app.js running. I was able to execute generate-names.pl and this seemed to work correctly from the terminal output. The changes also got picked up in the tracklist.json for the genome. But there is no sign of the elasticsearch plugin in web apollo...added the plugin to the config.groovy, and we have another jbrowse plugin (GC content) working fine so not sure what's the problem here.

Thanks for all of your help! :)
Michael
From: "Nathan Dunn" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Tuesday, November 13, 2018 6:33:05 PM
Subject: Re: [apollo] elastic search in genome apollo

Michael,

What problems are you running it when adding the plugins?

Nathan

On Nov 13, 2018, at 4:55 PM, Colin <[hidden email]> wrote:

Aside from changing your gene identifiers, I think the full text search from elasticsearch could help. I forwarded some comments here about that https://github.com/elsiklab/jbrowse_elasticsearch/issues/7

Potentially data that is loaded in Apollo backend could use text similarity %like% or full text sql queries but for normal jbrowse tracks that are not loaded in the sql database, the jbrowse_elasticsearch plugin seems reasonable.

-Colin

On Tue, Nov 13, 2018 at 7:35 AM Michael Poeschla <[hidden email]> wrote:
Hi,

has anyone implemented the jbrowse_elasticsearch plugin in Apollo? I'm running from the docker container with an nginx reverse proxy and having trouble adding this plugin. How would I add this to the docker image, with the middleware as well?

The reason we want to do this is the predicted gene model names we have in our tracks are not always intuitive so it's hard to find them by searching in Apollo (ie mTOR is ID'd as "serine/threonine-protein kinase mTOR", so if you just start typing mTOR you get no results). Wondering also if anyone else has encountered this problem and found good solutions, besides manipulating all of our tracks.

Thanks so much,
Michael

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

Michael Poeschla

Re: elastic search in genome apollo

yes - we don't see anything in the browser

Thanks,

Michael

From: "Colin" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 7:09:45 PM
Subject: Re: [apollo] elastic search in genome apollo

So you are saying that the plugin is not being added e.g. you don't have a menu item for the ElasticSearch dialog in jbrowse?

-Colin

On Thu, Nov 15, 2018 at 9:07 AM Michael Poeschla <[hidden email]> wrote:

hi,
elasticsearch was installed with docker. Here's the tracklist.json for the organism that we generated the index for. Is there any other info I could provide that would help?

{
   "elasticIndexName" : "nfz",
   "elasticSearchUrl" : "localhost:3000",
   "formatVersion" : 1,
   "names" : {
      "type" : "Hash",
      "url" : "names/"
   },
   "plugins" : [
      "ElasticSearch"
   ],
   "tracks" : [
      {
         "category" : "Reference sequence",
         "chunkSize" : 20000,
         "key" : "Reference sequence",
         "label" : "DNA",
         "seqType" : "dna",
         "storeClass" : "JBrowse/Store/Sequence/StaticChunked",
         "type" : "SequenceTrack",
         "urlTemplate" : "seq/{refseq_dirpath}/{refseq}-"
      },
      {
         "bicolor_pivot" : 0.5,
         "label" : "GC Content",
         "storeClass" : "JBrowse/Store/SeqFeature/SequenceChunks",
         "type" : "GCContent/View/Track/GCContentXY",
         "urlTemplate" : "seq/{refseq_dirpath}/{refseq}-"
      },
      {
         "compress" : 0,
         "key" : "Repeats",
         "label" : "Repeats",
         "storeClass" : "JBrowse/Store/SeqFeature/NCList",
         "style" : {
            "className" : "feature"
         },
         "trackType" : "JBrowse/View/Track/CanvasFeatures",
         "type" : "JBrowse/View/Track/CanvasFeatures",
         "urlTemplate" : "tracks/Repeats/{refseq}/trackData.json"
      },
      {
         "compress" : 0,
         "key" : "Annotation",
         "label" : "Annotationfromsumgff",
         "storeClass" : "JBrowse/Store/SeqFeature/NCList",
         "style" : {
            "className" : "feature"
         },
         "trackType" : "CanvasFeatures",
         "type" : "CanvasFeatures",
         "urlTemplate" : "tracks/Annotationfromsumgff/{refseq}/trackData.json"
      }
   ]
}

From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 11:33:03 AM
Subject: Re: [apollo] elastic search in genome apollo

Thanks. Also on Europe time for another 20 minutes.
Can you also post any other specifics of failure?

Have you installed elasticsearch via docker or manually ?

Nathan

On Nov 15, 2018, at 11:10 AM, Michael Poeschla <[hidden email]> wrote:

Here it is - sorry for the late replies, I'm on europe time.

environments {
    development {
    }
    test {
    }
    production {
        dataSource {
            dbCreate = "update"
            username = System.getenv("WEBAPOLLO_DB_USERNAME") ?: *
            password = System.getenv("WEBAPOLLO_DB_PASSWORD") ?: *

            driverClassName = "org.postgresql.Driver"
            dialect = "org.hibernate.dialect.PostgresPlusDialect"
            url = "jdbc:postgresql://${System.getenv("WEBAPOLLO_DB_HOST")?:"127.0.0.1"}/${System.getenv("WEBAPOLLO_DB_NAME")?:"apollo"}"

            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = false
                initialSize                   = 5
                maxActive                     = 50
                minIdle                       = 5
                maxIdle                       = 25
                maxWait                       = 10000
                maxAge                        = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis    = 60000
                validationQuery               = "SELECT 1"
                validationQueryTimeout        = 3
                validationInterval            = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
        dataSource_chado {
            dbCreate = "update"
            username = System.getenv("CHADO_DB_USERNAME") ?: *
            password = System.getenv("CHADO_DB_PASSWORD") ?: *

            driverClassName = "org.postgresql.Driver"
            dialect = "org.hibernate.dialect.PostgresPlusDialect"

            url = "jdbc:postgresql://${System.getenv("CHADO_DB_HOST")?:"127.0.0.1"}/${System.getenv("CHADO_DB_NAME")?:"chado"}"

            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = false
                initialSize                   = 5
                maxActive                     = 50
                minIdle                       = 5
                maxIdle                       = 25
                maxWait                       = 10000
                maxAge                        = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis    = 60000
                validationQuery               = "SELECT 1"
                validationQueryTimeout        = 3
                validationInterval            = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
    }
}

apollo {
    default_minimum_intron_size = System.getenv("WEBAPOLLO_MINIMUM_INTRON_SIZE") ? System.getenv("WEBAPOLLO_MINIMUM_INTRON_SIZE").toInteger() : 1
    history_size = System.getenv("WEBAPOLLO_HISTORY_SIZE") ? System.getenv("WEBAPOLLO_HISTORY_SIZE").toInteger() : 0
    overlapper_class = System.getenv("WEBAPOLLO_OVERLAPPER_CLASS") ?: "org.bbop.apollo.sequence.OrfOverlapper"
    use_cds_for_new_transcripts = System.getenv("WEBAPOLLO_CDS_FOR_NEW_TRANSCRIPTS").equals("true") // will default to false
    feature_has_dbxrefs = System.getenv("WEBAPOLLO_FEATURE_HAS_DBXREFS") ?: true
    feature_has_attributes = System.getenv("WEBAPOLLO_FEATURE_HAS_ATTRS") ?: true
    feature_has_pubmed_ids = System.getenv("WEBAPOLLO_FEATURE_HAS_PUBMED") ?: true
    feature_has_go_ids = System.getenv("WEBAPOLLO_FEATURE_HAS_GO") ?: true
    feature_has_comments = System.getenv("WEBAPOLLO_FEATURE_HAS_COMMENTS") ?: true
    feature_has_status = System.getenv("WEBAPOLLO_FEATURE_HAS_STATUS") ?: true
    translation_table = "/config/translation_tables/ncbi_" + (System.getenv("WEBAPOLLO_TRANSLATION_TABLE") ?: "1") + "_translation_table.txt"
    get_translation_code = System.getenv("WEBAPOLLO_TRANSLATION_TABLE") ? System.getenv("WEBAPOLLO_TRANSLATION_TABLE").toInteger() : 1

    // ADDED BLAT AND SET PATH TO BIN
    sequence_search_tools = [
        blat_nuc: [
          search_exe: "/usr/local/bin/blat",
          search_class: "org.bbop.apollo.sequence.search.blat.BlatCommandLineNucleotideToNucleotide",
          name: "Blat nucleotide",
          params: ""
        ],
        blat_prot: [
          search_exe: "/usr/local/bin/blat",
          search_class: "org.bbop.apollo.sequence.search.blat.BlatCommandLineProteinToNucleotide",
          name: "Blat protein",
          params: ""
        ]
      ]
    // TODO: should come from config or via preferences database
    splice_donor_sites = System.getenv("WEBAPOLLO_SPLICE_DONOR_SITES") ? System.getenv("WEBAPOLLO_SPLICE_DONOR_SITES").split(",") : ["GT"]
    splice_acceptor_sites = System.getenv("WEBAPOLLO_SPLICE_ACCEPTOR_SITES") ? System.getenv("WEBAPOLLO_SPLICE_ACCEPTOR_SITES").split(",") : ["AG"]
    gff3.source = System.getenv("WEBAPOLLO_GFF3_SOURCE") ?: "."

    google_analytics = System.getenv("WEBAPOLLO_GOOGLE_ANALYTICS_ID") ?: ["UA-62921593-1"]

    admin{
        username = System.getenv("APOLLO_ADMIN_EMAIL") ?: *
        password = System.getenv("APOLLO_ADMIN_PASSWORD") ?: *
        firstName = System.getenv("APOLLO_ADMIN_FIRST_NAME") ?: *
        lastName = System.getenv("APOLLO_ADMIN_LAST_NAME") ?: *
    }
}

jbrowse {
    git {
        url = "https://github.com/GMOD/jbrowse"
        tag = "maint/1.12.5-apollo"
    }
    plugins {
        WebApollo{
            included = true
        }
        NeatHTMLFeatures{
            included = System.getenv("WEBAPOLLO_JBROWSE_PLUGIN_NEATHTML") ?: true
        }
        NeatCanvasFeatures{
            included = System.getenv("WEBAPOLLO_JBROWSE_PLUGIN_NEATCANVAS") ?: true
        }
        RegexSequenceSearch{
            included = true
        }
        HideTrackLabels{
            included = true
        }
        //EDIT: ADDED GC CONTENT
        GCContent{
            git = 'https://github.com/cmdcolin/GCContent'
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
        //EDIT: ADDS ELASTIC SEARCH PLUGIN
        ElasticSearch{
            git = 'https://github.com/elsiklab/jbrowse_elasticsearch'
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
    }
}
From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 12:30:50 AM
Subject: Re: [apollo] elastic search in genome apollo

Can you post your apollo config file?

Nathan

On Nov 14, 2018, at 6:45 PM, Michael Poeschla <[hidden email]> wrote:

Hi,
I'm following the instructions from https://github.com/elsiklab/jbrowse_elasticsearch

we have elasticsearch and the middleware app.js running. I was able to execute generate-names.pl and this seemed to work correctly from the terminal output. The changes also got picked up in the tracklist.json for the genome. But there is no sign of the elasticsearch plugin in web apollo...added the plugin to the config.groovy, and we have another jbrowse plugin (GC content) working fine so not sure what's the problem here.

Thanks for all of your help! :)
Michael
From: "Nathan Dunn" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Tuesday, November 13, 2018 6:33:05 PM
Subject: Re: [apollo] elastic search in genome apollo

Michael,

What problems are you running it when adding the plugins?

Nathan

On Nov 13, 2018, at 4:55 PM, Colin <[hidden email]> wrote:

Aside from changing your gene identifiers, I think the full text search from elasticsearch could help. I forwarded some comments here about that https://github.com/elsiklab/jbrowse_elasticsearch/issues/7

Potentially data that is loaded in Apollo backend could use text similarity %like% or full text sql queries but for normal jbrowse tracks that are not loaded in the sql database, the jbrowse_elasticsearch plugin seems reasonable.

-Colin

On Tue, Nov 13, 2018 at 7:35 AM Michael Poeschla <[hidden email]> wrote:
Hi,

has anyone implemented the jbrowse_elasticsearch plugin in Apollo? I'm running from the docker container with an nginx reverse proxy and having trouble adding this plugin. How would I add this to the docker image, with the middleware as well?

The reason we want to do this is the predicted gene model names we have in our tracks are not always intuitive so it's hard to find them by searching in Apollo (ie mTOR is ID'd as "serine/threonine-protein kinase mTOR", so if you just start typing mTOR you get no results). Wondering also if anyone else has encountered this problem and found good solutions, besides manipulating all of our tracks.

Thanks so much,
Michael

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

nathandunn

Re: elastic search in genome apollo

Sorry for the late response.

Can you provide the browser console logs and any tomcat logs (catalina.out)?

Your modified docket file may be helpful as well.

Nathan

On Nov 15, 2018, at 10:26 AM, Michael Poeschla <[hidden email]> wrote:

yes - we don't see anything in the browser
Thanks,
Michael
From: "Colin" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 7:09:45 PM
Subject: Re: [apollo] elastic search in genome apollo

So you are saying that the plugin is not being added e.g. you don't have a menu item for the ElasticSearch dialog in jbrowse?
-Colin

On Thu, Nov 15, 2018 at 9:07 AM Michael Poeschla <[hidden email]> wrote:
hi,
elasticsearch was installed with docker. Here's the tracklist.json for the organism that we generated the index for. Is there any other info I could provide that would help?

{
   "elasticIndexName" : "nfz",
   "elasticSearchUrl" : "localhost:3000",
   "formatVersion" : 1,
   "names" : {
      "type" : "Hash",
      "url" : "names/"
   },
   "plugins" : [
      "ElasticSearch"
   ],
   "tracks" : [
      {
         "category" : "Reference sequence",
         "chunkSize" : 20000,
         "key" : "Reference sequence",
         "label" : "DNA",
         "seqType" : "dna",
         "storeClass" : "JBrowse/Store/Sequence/StaticChunked",
         "type" : "SequenceTrack",
         "urlTemplate" : "seq/{refseq_dirpath}/{refseq}-"
      },
      {
         "bicolor_pivot" : 0.5,
         "label" : "GC Content",
         "storeClass" : "JBrowse/Store/SeqFeature/SequenceChunks",
         "type" : "GCContent/View/Track/GCContentXY",
         "urlTemplate" : "seq/{refseq_dirpath}/{refseq}-"
      },
      {
         "compress" : 0,
         "key" : "Repeats",
         "label" : "Repeats",
         "storeClass" : "JBrowse/Store/SeqFeature/NCList",
         "style" : {
            "className" : "feature"
         },
         "trackType" : "JBrowse/View/Track/CanvasFeatures",
         "type" : "JBrowse/View/Track/CanvasFeatures",
         "urlTemplate" : "tracks/Repeats/{refseq}/trackData.json"
      },
      {
         "compress" : 0,
         "key" : "Annotation",
         "label" : "Annotationfromsumgff",
         "storeClass" : "JBrowse/Store/SeqFeature/NCList",
         "style" : {
            "className" : "feature"
         },
         "trackType" : "CanvasFeatures",
         "type" : "CanvasFeatures",
         "urlTemplate" : "tracks/Annotationfromsumgff/{refseq}/trackData.json"
      }
   ]
}

From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 11:33:03 AM
Subject: Re: [apollo] elastic search in genome apollo

Thanks. Also on Europe time for another 20 minutes.
Can you also post any other specifics of failure?

Have you installed elasticsearch via docker or manually ?

Nathan

On Nov 15, 2018, at 11:10 AM, Michael Poeschla <[hidden email]> wrote:

Here it is - sorry for the late replies, I'm on europe time.

environments {
    development {
    }
    test {
    }
    production {
        dataSource {
            dbCreate = "update"
            username = System.getenv("WEBAPOLLO_DB_USERNAME") ?: *
            password = System.getenv("WEBAPOLLO_DB_PASSWORD") ?: *

            driverClassName = "org.postgresql.Driver"
            dialect = "org.hibernate.dialect.PostgresPlusDialect"
            url = "jdbc:postgresql://${System.getenv("WEBAPOLLO_DB_HOST")?:"127.0.0.1"}/${System.getenv("WEBAPOLLO_DB_NAME")?:"apollo"}"

            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = false
                initialSize                   = 5
                maxActive                     = 50
                minIdle                       = 5
                maxIdle                       = 25
                maxWait                       = 10000
                maxAge                        = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis    = 60000
                validationQuery               = "SELECT 1"
                validationQueryTimeout        = 3
                validationInterval            = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
        dataSource_chado {
            dbCreate = "update"
            username = System.getenv("CHADO_DB_USERNAME") ?: *
            password = System.getenv("CHADO_DB_PASSWORD") ?: *

            driverClassName = "org.postgresql.Driver"
            dialect = "org.hibernate.dialect.PostgresPlusDialect"

            url = "jdbc:postgresql://${System.getenv("CHADO_DB_HOST")?:"127.0.0.1"}/${System.getenv("CHADO_DB_NAME")?:"chado"}"

            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = false
                initialSize                   = 5
                maxActive                     = 50
                minIdle                       = 5
                maxIdle                       = 25
                maxWait                       = 10000
                maxAge                        = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis    = 60000
                validationQuery               = "SELECT 1"
                validationQueryTimeout        = 3
                validationInterval            = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
    }
}

apollo {
    default_minimum_intron_size = System.getenv("WEBAPOLLO_MINIMUM_INTRON_SIZE") ? System.getenv("WEBAPOLLO_MINIMUM_INTRON_SIZE").toInteger() : 1
    history_size = System.getenv("WEBAPOLLO_HISTORY_SIZE") ? System.getenv("WEBAPOLLO_HISTORY_SIZE").toInteger() : 0
    overlapper_class = System.getenv("WEBAPOLLO_OVERLAPPER_CLASS") ?: "org.bbop.apollo.sequence.OrfOverlapper"
    use_cds_for_new_transcripts = System.getenv("WEBAPOLLO_CDS_FOR_NEW_TRANSCRIPTS").equals("true") // will default to false
    feature_has_dbxrefs = System.getenv("WEBAPOLLO_FEATURE_HAS_DBXREFS") ?: true
    feature_has_attributes = System.getenv("WEBAPOLLO_FEATURE_HAS_ATTRS") ?: true
    feature_has_pubmed_ids = System.getenv("WEBAPOLLO_FEATURE_HAS_PUBMED") ?: true
    feature_has_go_ids = System.getenv("WEBAPOLLO_FEATURE_HAS_GO") ?: true
    feature_has_comments = System.getenv("WEBAPOLLO_FEATURE_HAS_COMMENTS") ?: true
    feature_has_status = System.getenv("WEBAPOLLO_FEATURE_HAS_STATUS") ?: true
    translation_table = "/config/translation_tables/ncbi_" + (System.getenv("WEBAPOLLO_TRANSLATION_TABLE") ?: "1") + "_translation_table.txt"
    get_translation_code = System.getenv("WEBAPOLLO_TRANSLATION_TABLE") ? System.getenv("WEBAPOLLO_TRANSLATION_TABLE").toInteger() : 1

    // ADDED BLAT AND SET PATH TO BIN
    sequence_search_tools = [
        blat_nuc: [
          search_exe: "/usr/local/bin/blat",
          search_class: "org.bbop.apollo.sequence.search.blat.BlatCommandLineNucleotideToNucleotide",
          name: "Blat nucleotide",
          params: ""
        ],
        blat_prot: [
          search_exe: "/usr/local/bin/blat",
          search_class: "org.bbop.apollo.sequence.search.blat.BlatCommandLineProteinToNucleotide",
          name: "Blat protein",
          params: ""
        ]
      ]
    // TODO: should come from config or via preferences database
    splice_donor_sites = System.getenv("WEBAPOLLO_SPLICE_DONOR_SITES") ? System.getenv("WEBAPOLLO_SPLICE_DONOR_SITES").split(",") : ["GT"]
    splice_acceptor_sites = System.getenv("WEBAPOLLO_SPLICE_ACCEPTOR_SITES") ? System.getenv("WEBAPOLLO_SPLICE_ACCEPTOR_SITES").split(",") : ["AG"]
    gff3.source = System.getenv("WEBAPOLLO_GFF3_SOURCE") ?: "."

    google_analytics = System.getenv("WEBAPOLLO_GOOGLE_ANALYTICS_ID") ?: ["UA-62921593-1"]

    admin{
        username = System.getenv("APOLLO_ADMIN_EMAIL") ?: *
        password = System.getenv("APOLLO_ADMIN_PASSWORD") ?: *
        firstName = System.getenv("APOLLO_ADMIN_FIRST_NAME") ?: *
        lastName = System.getenv("APOLLO_ADMIN_LAST_NAME") ?: *
    }
}

jbrowse {
    git {
        url = "https://github.com/GMOD/jbrowse"
        tag = "maint/1.12.5-apollo"
    }
    plugins {
        WebApollo{
            included = true
        }
        NeatHTMLFeatures{
            included = System.getenv("WEBAPOLLO_JBROWSE_PLUGIN_NEATHTML") ?: true
        }
        NeatCanvasFeatures{
            included = System.getenv("WEBAPOLLO_JBROWSE_PLUGIN_NEATCANVAS") ?: true
        }
        RegexSequenceSearch{
            included = true
        }
        HideTrackLabels{
            included = true
        }
        //EDIT: ADDED GC CONTENT
        GCContent{
            git = 'https://github.com/cmdcolin/GCContent'
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
        //EDIT: ADDS ELASTIC SEARCH PLUGIN
        ElasticSearch{
            git = 'https://github.com/elsiklab/jbrowse_elasticsearch'
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
    }
}
From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 12:30:50 AM
Subject: Re: [apollo] elastic search in genome apollo

Can you post your apollo config file?

Nathan

On Nov 14, 2018, at 6:45 PM, Michael Poeschla <[hidden email]> wrote:

Hi,
I'm following the instructions from https://github.com/elsiklab/jbrowse_elasticsearch

we have elasticsearch and the middleware app.js running. I was able to execute generate-names.pl and this seemed to work correctly from the terminal output. The changes also got picked up in the tracklist.json for the genome. But there is no sign of the elasticsearch plugin in web apollo...added the plugin to the config.groovy, and we have another jbrowse plugin (GC content) working fine so not sure what's the problem here.

Thanks for all of your help! :)
Michael
From: "Nathan Dunn" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Tuesday, November 13, 2018 6:33:05 PM
Subject: Re: [apollo] elastic search in genome apollo

Michael,

What problems are you running it when adding the plugins?

Nathan

On Nov 13, 2018, at 4:55 PM, Colin <[hidden email]> wrote:

Aside from changing your gene identifiers, I think the full text search from elasticsearch could help. I forwarded some comments here about that https://github.com/elsiklab/jbrowse_elasticsearch/issues/7

Potentially data that is loaded in Apollo backend could use text similarity %like% or full text sql queries but for normal jbrowse tracks that are not loaded in the sql database, the jbrowse_elasticsearch plugin seems reasonable.

-Colin

On Tue, Nov 13, 2018 at 7:35 AM Michael Poeschla <[hidden email]> wrote:
Hi,

has anyone implemented the jbrowse_elasticsearch plugin in Apollo? I'm running from the docker container with an nginx reverse proxy and having trouble adding this plugin. How would I add this to the docker image, with the middleware as well?

The reason we want to do this is the predicted gene model names we have in our tracks are not always intuitive so it's hard to find them by searching in Apollo (ie mTOR is ID'd as "serine/threonine-protein kinase mTOR", so if you just start typing mTOR you get no results). Wondering also if anyone else has encountered this problem and found good solutions, besides manipulating all of our tracks.

Thanks so much,
Michael

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

Michael Poeschla

Re: elastic search in genome apollo

Hi,

here are the logs, docker-compose, actual dockerfile and start and testing of npm (below). I've also added the tomcat logs as an attachment.

docker-compose:

version: "3.2"

services:

apollo:

image: hub.age.mpg.de/mpoeschla/dv-kb:0.4

volumes:

- ./data/extdata/:/data

- postgres-data:/var/lib/postgresql

- ./config/tomcat/server.xml:/var/lib/tomcat8/conf/server.xml:ro

- ./config/tomcat/setenv.sh:/usr/share/tomcat8/bin/setenv.sh

environment:

APOLLO_ADMIN_[hidden email]

restart: always

links:

- elasticsearch:elasticsearch

nginx:

image: library/nginx:alpine

restart: always

volumes:

- ./config/nginx/nginx.conf:/etc/nginx/nginx.conf:ro

- ./config/secrets:/secrets:ro

ports:

- 80:80

- 443:443

links:

- apollo:apollo

elasticsearch:

image: elasticsearch:6.4.2

container_name: elasticsearch

environment:

- cluster.name=docker-cluster

- bootstrap.memory_lock=true

- "ES_JAVA_OPTS=-Xms512m -Xmx512m"

ulimits:

memlock:

soft: -1

hard: -1

volumes:

- esdata1:/usr/share/elasticsearch/data

volumes:

postgres-data:

esdata1:

driver: local

The actual Dockerfile:

# WebApollo

# VERSION 2.1.X

FROM tomcat:8-jre8

MAINTAINER Eric Rasche <[hidden email]>, Anthony Bretaudeau <[hidden email]>, Nathan Dunn <[hidden email]>

ENV DEBIAN_FRONTEND noninteractive

RUN apt-get -qq update --fix-missing && \

apt-get --no-install-recommends -y install \

git build-essential maven tomcat8 libpq-dev postgresql-common openjdk-8-jdk wget \

postgresql postgresql-client xmlstarlet netcat libpng-dev \

zlib1g-dev libexpat1-dev ant curl ssl-cert zip unzip

RUN curl -sL https://deb.nodesource.com/setup_8.x | bash -

RUN apt-get -qq update --fix-missing && \

apt-get --no-install-recommends -y install nodejs && \

apt-get autoremove -y && apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

RUN cp /usr/lib/jvm/java-8-openjdk-amd64/lib/tools.jar /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext/tools.jar && \

useradd -ms /bin/bash -d /apollo apollo

ENV WEBAPOLLO_VERSION 99c7e54c6e74fbc6705d57de216c9b14b2bfb03b

RUN curl -L <a href="https://github.com/GMOD/Apollo/archive/${WEBAPOLLO_VERSION}.tar.gz" class="" target="_blank" data-mce-href="https://github.com/GMOD/Apollo/archive/${WEBAPOLLO_VERSION}.tar.gz">https://github.com/GMOD/Apollo/archive/${WEBAPOLLO_VERSION}.tar.gz | tar xzf - --strip-components=1 -C /apollo

# install grails

COPY build.sh /bin/build.sh

ADD apollo-config.groovy /apollo/apollo-config.groovy

RUN chown -R apollo:apollo /apollo

RUN curl -s "http://hgdownload.soe.ucsc.edu/admin/exe/linux.x86_64/blat/blat" -o /usr/local/bin/blat

RUN chmod +x /usr/local/bin/blat

USER apollo

RUN curl -s get.sdkman.io | bash

RUN /bin/bash -c "source $HOME/.sdkman/bin/sdkman-init.sh && yes | sdk install grails 2.5.5"

RUN /bin/bash -c "source $HOME/.sdkman/bin/sdkman-init.sh && yes | sdk install gradle 3.2.1"

RUN /bin/bash -c "source $HOME/.profile && source $HOME/.sdkman/bin/sdkman-init.sh && /bin/bash /bin/build.sh"

USER root

ENV CATALINA_HOME=/var/lib/tomcat8

RUN rm -rf ${CATALINA_HOME}/webapps/* && \

cp /apollo/apollo*.war ${CATALINA_HOME}/apollo.war

ENV CONTEXT_PATH ROOT

# Download chado schema

RUN wget --quiet https://github.com/erasche/chado-schema-builder/releases/download/1.31-jenkins97/chado-1.31.sql.gz -O /chado.sql.gz && \

gunzip /chado.sql.gz

RUN wget https://optinsoft.net/pic/ipgen2.png -O test.png

ADD launch.sh /launch.sh

CMD "/launch.sh"

Starting npm inside the container:

root@995955a711e1:/var/lib/tomcat8/webapps/ROOT/jbrowse/plugins/ElasticSearch# npm start

> jbrowse_elasticsearch@1.0.4 start /var/lib/tomcat8/webapps/ROOT/jbrowse/plugins/ElasticSearch

> node ./bin/www

testing access with curl:

root@995955a711e1:/usr/local/tomcat# curl http://localhost:3000

{"hits":[],"total":0}root@995955a711e1:/usr/local/tomcat#

content of ElasticSearchPlugin:

^Croot@995955a711e1:/var/lib/tomcat8/webapps/ROOT/jbrowse/plugins/ElasticSearch# ls -la

total 68

drwxr-x--- 10 tomcat8 tomcat8 271 Nov 14 17:10 .

drwxr-x--- 12 tomcat8 tomcat8 223 Nov 14 16:34 ..

-rw-r----- 1 tomcat8 tomcat8 594 Nov 14 16:16 .eslintrc

-rw-r----- 1 tomcat8 tomcat8 290 Nov 14 16:16 .travis.yml

-rw-r----- 1 tomcat8 tomcat8 686 Nov 14 16:16 ChangeLog.md

-rw-r----- 1 tomcat8 tomcat8 810 Nov 14 16:16 Makefile.PL

-rw-r----- 1 tomcat8 tomcat8 5157 Nov 14 16:16 README.md

-rw-r----- 1 tomcat8 tomcat8 1888 Nov 14 16:44 app.js

drwxr-x--- 2 tomcat8 tomcat8 66 Nov 14 16:34 bin

drwxr-x--- 2 tomcat8 tomcat8 22 Nov 14 16:34 css

drwxr-x--- 2 tomcat8 tomcat8 22 Nov 14 16:34 docs

drwxr-x--- 2 tomcat8 tomcat8 45 Nov 14 16:34 img

drwxr-x--- 3 tomcat8 tomcat8 65 Nov 14 16:34 js

drwxr-xr-x 24 root root 4096 Nov 14 17:10 node_modules

-rw-r----- 1 tomcat8 tomcat8 25820 Nov 14 17:10 package-lock.json

-rw-r----- 1 tomcat8 tomcat8 725 Nov 14 16:16 package.json

-rw-r----- 1 tomcat8 tomcat8 727 Nov 14 16:16 setup.sh

drwxr-x--- 3 tomcat8 tomcat8 19 Nov 14 16:34 src

drwxr-x--- 3 tomcat8 tomcat8 33 Nov 14 16:34 test

..And the logs are attached.

Thanks!

Michael

From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Saturday, November 17, 2018 4:06:58 AM
Subject: Re: [apollo] elastic search in genome apollo

Sorry for the late response.

Can you provide the browser console logs and any tomcat logs (catalina.out)?

Your modified docket file may be helpful as well.

Nathan

On Nov 15, 2018, at 10:26 AM, Michael Poeschla <[hidden email]> wrote:

yes - we don't see anything in the browser
Thanks,
Michael
From: "Colin" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 7:09:45 PM
Subject: Re: [apollo] elastic search in genome apollo

So you are saying that the plugin is not being added e.g. you don't have a menu item for the ElasticSearch dialog in jbrowse?
-Colin

On Thu, Nov 15, 2018 at 9:07 AM Michael Poeschla <[hidden email]> wrote:
hi,
elasticsearch was installed with docker. Here's the tracklist.json for the organism that we generated the index for. Is there any other info I could provide that would help?

{
   "elasticIndexName" : "nfz",
   "elasticSearchUrl" : "localhost:3000",
   "formatVersion" : 1,
   "names" : {
      "type" : "Hash",
      "url" : "names/"
   },
   "plugins" : [
      "ElasticSearch"
   ],
   "tracks" : [
      {
         "category" : "Reference sequence",
         "chunkSize" : 20000,
         "key" : "Reference sequence",
         "label" : "DNA",
         "seqType" : "dna",
         "storeClass" : "JBrowse/Store/Sequence/StaticChunked",
         "type" : "SequenceTrack",
         "urlTemplate" : "seq/{refseq_dirpath}/{refseq}-"
      },
      {
         "bicolor_pivot" : 0.5,
         "label" : "GC Content",
         "storeClass" : "JBrowse/Store/SeqFeature/SequenceChunks",
         "type" : "GCContent/View/Track/GCContentXY",
         "urlTemplate" : "seq/{refseq_dirpath}/{refseq}-"
      },
      {
         "compress" : 0,
         "key" : "Repeats",
         "label" : "Repeats",
         "storeClass" : "JBrowse/Store/SeqFeature/NCList",
         "style" : {
            "className" : "feature"
         },
         "trackType" : "JBrowse/View/Track/CanvasFeatures",
         "type" : "JBrowse/View/Track/CanvasFeatures",
         "urlTemplate" : "tracks/Repeats/{refseq}/trackData.json"
      },
      {
         "compress" : 0,
         "key" : "Annotation",
         "label" : "Annotationfromsumgff",
         "storeClass" : "JBrowse/Store/SeqFeature/NCList",
         "style" : {
            "className" : "feature"
         },
         "trackType" : "CanvasFeatures",
         "type" : "CanvasFeatures",
         "urlTemplate" : "tracks/Annotationfromsumgff/{refseq}/trackData.json"
      }
   ]
}

From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 11:33:03 AM
Subject: Re: [apollo] elastic search in genome apollo

Thanks. Also on Europe time for another 20 minutes.
Can you also post any other specifics of failure?

Have you installed elasticsearch via docker or manually ?

Nathan

On Nov 15, 2018, at 11:10 AM, Michael Poeschla <[hidden email]> wrote:

Here it is - sorry for the late replies, I'm on europe time.

environments {
    development {
    }
    test {
    }
    production {
        dataSource {
            dbCreate = "update"
            username = System.getenv("WEBAPOLLO_DB_USERNAME") ?: *
            password = System.getenv("WEBAPOLLO_DB_PASSWORD") ?: *

            driverClassName = "org.postgresql.Driver"
            dialect = "org.hibernate.dialect.PostgresPlusDialect"
            url = "jdbc:postgresql://${System.getenv("WEBAPOLLO_DB_HOST")?:"127.0.0.1"}/${System.getenv("WEBAPOLLO_DB_NAME")?:"apollo"}"

            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = false
                initialSize                   = 5
                maxActive                     = 50
                minIdle                       = 5
                maxIdle                       = 25
                maxWait                       = 10000
                maxAge                        = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis    = 60000
                validationQuery               = "SELECT 1"
                validationQueryTimeout        = 3
                validationInterval            = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
        dataSource_chado {
            dbCreate = "update"
            username = System.getenv("CHADO_DB_USERNAME") ?: *
            password = System.getenv("CHADO_DB_PASSWORD") ?: *

            driverClassName = "org.postgresql.Driver"
            dialect = "org.hibernate.dialect.PostgresPlusDialect"

            url = "jdbc:postgresql://${System.getenv("CHADO_DB_HOST")?:"127.0.0.1"}/${System.getenv("CHADO_DB_NAME")?:"chado"}"

            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = false
                initialSize                   = 5
                maxActive                     = 50
                minIdle                       = 5
                maxIdle                       = 25
                maxWait                       = 10000
                maxAge                        = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis    = 60000
                validationQuery               = "SELECT 1"
                validationQueryTimeout        = 3
                validationInterval            = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
    }
}

apollo {
    default_minimum_intron_size = System.getenv("WEBAPOLLO_MINIMUM_INTRON_SIZE") ? System.getenv("WEBAPOLLO_MINIMUM_INTRON_SIZE").toInteger() : 1
    history_size = System.getenv("WEBAPOLLO_HISTORY_SIZE") ? System.getenv("WEBAPOLLO_HISTORY_SIZE").toInteger() : 0
    overlapper_class = System.getenv("WEBAPOLLO_OVERLAPPER_CLASS") ?: "org.bbop.apollo.sequence.OrfOverlapper"
    use_cds_for_new_transcripts = System.getenv("WEBAPOLLO_CDS_FOR_NEW_TRANSCRIPTS").equals("true") // will default to false
    feature_has_dbxrefs = System.getenv("WEBAPOLLO_FEATURE_HAS_DBXREFS") ?: true
    feature_has_attributes = System.getenv("WEBAPOLLO_FEATURE_HAS_ATTRS") ?: true
    feature_has_pubmed_ids = System.getenv("WEBAPOLLO_FEATURE_HAS_PUBMED") ?: true
    feature_has_go_ids = System.getenv("WEBAPOLLO_FEATURE_HAS_GO") ?: true
    feature_has_comments = System.getenv("WEBAPOLLO_FEATURE_HAS_COMMENTS") ?: true
    feature_has_status = System.getenv("WEBAPOLLO_FEATURE_HAS_STATUS") ?: true
    translation_table = "/config/translation_tables/ncbi_" + (System.getenv("WEBAPOLLO_TRANSLATION_TABLE") ?: "1") + "_translation_table.txt"
    get_translation_code = System.getenv("WEBAPOLLO_TRANSLATION_TABLE") ? System.getenv("WEBAPOLLO_TRANSLATION_TABLE").toInteger() : 1

    // ADDED BLAT AND SET PATH TO BIN
    sequence_search_tools = [
        blat_nuc: [
          search_exe: "/usr/local/bin/blat",
          search_class: "org.bbop.apollo.sequence.search.blat.BlatCommandLineNucleotideToNucleotide",
          name: "Blat nucleotide",
          params: ""
        ],
        blat_prot: [
          search_exe: "/usr/local/bin/blat",
          search_class: "org.bbop.apollo.sequence.search.blat.BlatCommandLineProteinToNucleotide",
          name: "Blat protein",
          params: ""
        ]
      ]
    // TODO: should come from config or via preferences database
    splice_donor_sites = System.getenv("WEBAPOLLO_SPLICE_DONOR_SITES") ? System.getenv("WEBAPOLLO_SPLICE_DONOR_SITES").split(",") : ["GT"]
    splice_acceptor_sites = System.getenv("WEBAPOLLO_SPLICE_ACCEPTOR_SITES") ? System.getenv("WEBAPOLLO_SPLICE_ACCEPTOR_SITES").split(",") : ["AG"]
    gff3.source = System.getenv("WEBAPOLLO_GFF3_SOURCE") ?: "."

    google_analytics = System.getenv("WEBAPOLLO_GOOGLE_ANALYTICS_ID") ?: ["UA-62921593-1"]

    admin{
        username = System.getenv("APOLLO_ADMIN_EMAIL") ?: *
        password = System.getenv("APOLLO_ADMIN_PASSWORD") ?: *
        firstName = System.getenv("APOLLO_ADMIN_FIRST_NAME") ?: *
        lastName = System.getenv("APOLLO_ADMIN_LAST_NAME") ?: *
    }
}

jbrowse {
    git {
        url = "https://github.com/GMOD/jbrowse"
        tag = "maint/1.12.5-apollo"
    }
    plugins {
        WebApollo{
            included = true
        }
        NeatHTMLFeatures{
            included = System.getenv("WEBAPOLLO_JBROWSE_PLUGIN_NEATHTML") ?: true
        }
        NeatCanvasFeatures{
            included = System.getenv("WEBAPOLLO_JBROWSE_PLUGIN_NEATCANVAS") ?: true
        }
        RegexSequenceSearch{
            included = true
        }
        HideTrackLabels{
            included = true
        }
        //EDIT: ADDED GC CONTENT
        GCContent{
            git = 'https://github.com/cmdcolin/GCContent'
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
        //EDIT: ADDS ELASTIC SEARCH PLUGIN
        ElasticSearch{
            git = 'https://github.com/elsiklab/jbrowse_elasticsearch'
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
    }
}
From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 12:30:50 AM
Subject: Re: [apollo] elastic search in genome apollo

Can you post your apollo config file?

Nathan

On Nov 14, 2018, at 6:45 PM, Michael Poeschla <[hidden email]> wrote:

Hi,
I'm following the instructions from https://github.com/elsiklab/jbrowse_elasticsearch

we have elasticsearch and the middleware app.js running. I was able to execute generate-names.pl and this seemed to work correctly from the terminal output. The changes also got picked up in the tracklist.json for the genome. But there is no sign of the elasticsearch plugin in web apollo...added the plugin to the config.groovy, and we have another jbrowse plugin (GC content) working fine so not sure what's the problem here.

Thanks for all of your help! :)
Michael
From: "Nathan Dunn" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Tuesday, November 13, 2018 6:33:05 PM
Subject: Re: [apollo] elastic search in genome apollo

Michael,

What problems are you running it when adding the plugins?

Nathan

On Nov 13, 2018, at 4:55 PM, Colin <[hidden email]> wrote:

Aside from changing your gene identifiers, I think the full text search from elasticsearch could help. I forwarded some comments here about that https://github.com/elsiklab/jbrowse_elasticsearch/issues/7

Potentially data that is loaded in Apollo backend could use text similarity %like% or full text sql queries but for normal jbrowse tracks that are not loaded in the sql database, the jbrowse_elasticsearch plugin seems reasonable.

-Colin

On Tue, Nov 13, 2018 at 7:35 AM Michael Poeschla <[hidden email]> wrote:
Hi,

has anyone implemented the jbrowse_elasticsearch plugin in Apollo? I'm running from the docker container with an nginx reverse proxy and having trouble adding this plugin. How would I add this to the docker image, with the middleware as well?

The reason we want to do this is the predicted gene model names we have in our tracks are not always intuitive so it's hard to find them by searching in Apollo (ie mTOR is ID'd as "serine/threonine-protein kinase mTOR", so if you just start typing mTOR you get no results). Wondering also if anyone else has encountered this problem and found good solutions, besides manipulating all of our tracks.

Thanks so much,
Michael

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

tomcat.tar.gz (396K) Download Attachment

nathandunn

Re: elastic search in genome apollo

Thanks. Will have a look.

On Nov 20, 2018, at 5:03 AM, Michael Poeschla <[hidden email]> wrote:

Hi,

here are the logs, docker-compose, actual dockerfile and start and testing of npm (below). I've also added the tomcat logs as an attachment.

docker-compose:
--
version: "3.2"
services:

apollo:
image: hub.age.mpg.de/mpoeschla/dv-kb:0.4
volumes:
- ./data/extdata/:/data
- postgres-data:/var/lib/postgresql
- ./config/tomcat/server.xml:/var/lib/tomcat8/conf/server.xml:ro
- ./config/tomcat/setenv.sh:/usr/share/tomcat8/bin/setenv.sh
environment:
APOLLO_ADMIN_[hidden email]
restart: always
links:
- elasticsearch:elasticsearch

nginx:
image: library/nginx:alpine
restart: always
volumes:
- ./config/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
- ./config/secrets:/secrets:ro
ports:
- 80:80
- 443:443
links:
- apollo:apollo

elasticsearch:
image: elasticsearch:6.4.2
container_name: elasticsearch
environment:
- cluster.name=docker-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- esdata1:/usr/share/elasticsearch/data

volumes:
postgres-data:
esdata1:
driver: local
--
The actual Dockerfile:
--
# WebApollo
# VERSION 2.1.X
FROM tomcat:8-jre8
MAINTAINER Eric Rasche <[hidden email]>, Anthony Bretaudeau <[hidden email]>, Nathan Dunn <[hidden email]>
ENV DEBIAN_FRONTEND noninteractive

RUN apt-get -qq update --fix-missing && \
apt-get --no-install-recommends -y install \
git build-essential maven tomcat8 libpq-dev postgresql-common openjdk-8-jdk wget \
postgresql postgresql-client xmlstarlet netcat libpng-dev \
zlib1g-dev libexpat1-dev ant curl ssl-cert zip unzip

RUN curl -sL https://deb.nodesource.com/setup_8.x | bash -
RUN apt-get -qq update --fix-missing && \
apt-get --no-install-recommends -y install nodejs && \
apt-get autoremove -y && apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

RUN cp /usr/lib/jvm/java-8-openjdk-amd64/lib/tools.jar /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext/tools.jar && \
useradd -ms /bin/bash -d /apollo apollo

ENV WEBAPOLLO_VERSION 99c7e54c6e74fbc6705d57de216c9b14b2bfb03b
RUN curl -L https://github.com/GMOD/Apollo/archive/${WEBAPOLLO_VERSION}.tar.gz | tar xzf - --strip-components=1 -C /apollo

# install grails
COPY build.sh /bin/build.sh
ADD apollo-config.groovy /apollo/apollo-config.groovy

RUN chown -R apollo:apollo /apollo
RUN curl -s "http://hgdownload.soe.ucsc.edu/admin/exe/linux.x86_64/blat/blat" -o /usr/local/bin/blat
RUN chmod +x /usr/local/bin/blat

USER apollo
RUN curl -s get.sdkman.io | bash
RUN /bin/bash -c "source $HOME/.sdkman/bin/sdkman-init.sh && yes | sdk install grails 2.5.5"
RUN /bin/bash -c "source $HOME/.sdkman/bin/sdkman-init.sh && yes | sdk install gradle 3.2.1"

RUN /bin/bash -c "source $HOME/.profile && source $HOME/.sdkman/bin/sdkman-init.sh && /bin/bash /bin/build.sh"

USER root
ENV CATALINA_HOME=/var/lib/tomcat8
RUN rm -rf ${CATALINA_HOME}/webapps/* && \
cp /apollo/apollo*.war ${CATALINA_HOME}/apollo.war

ENV CONTEXT_PATH ROOT

# Download chado schema
RUN wget --quiet https://github.com/erasche/chado-schema-builder/releases/download/1.31-jenkins97/chado-1.31.sql.gz -O /chado.sql.gz && \
gunzip /chado.sql.gz

RUN wget https://optinsoft.net/pic/ipgen2.png -O test.png

ADD launch.sh /launch.sh
CMD "/launch.sh"
--
Starting npm inside the container:
root@995955a711e1:/var/lib/tomcat8/webapps/ROOT/jbrowse/plugins/ElasticSearch# npm start

> jbrowse_elasticsearch@1.0.4 start /var/lib/tomcat8/webapps/ROOT/jbrowse/plugins/ElasticSearch
> node ./bin/www

testing access with curl:
root@995955a711e1:/usr/local/tomcat# curl http://localhost:3000
{"hits":[],"total":0}root@995955a711e1:/usr/local/tomcat#

content of ElasticSearchPlugin:
^Croot@995955a711e1:/var/lib/tomcat8/webapps/ROOT/jbrowse/plugins/ElasticSearch# ls -la
total 68
drwxr-x--- 10 tomcat8 tomcat8 271 Nov 14 17:10 .
drwxr-x--- 12 tomcat8 tomcat8 223 Nov 14 16:34 ..
-rw-r----- 1 tomcat8 tomcat8 594 Nov 14 16:16 .eslintrc
-rw-r----- 1 tomcat8 tomcat8 290 Nov 14 16:16 .travis.yml
-rw-r----- 1 tomcat8 tomcat8 686 Nov 14 16:16 ChangeLog.md
-rw-r----- 1 tomcat8 tomcat8 810 Nov 14 16:16 Makefile.PL
-rw-r----- 1 tomcat8 tomcat8 5157 Nov 14 16:16 README.md
-rw-r----- 1 tomcat8 tomcat8 1888 Nov 14 16:44 app.js
drwxr-x--- 2 tomcat8 tomcat8 66 Nov 14 16:34 bin
drwxr-x--- 2 tomcat8 tomcat8 22 Nov 14 16:34 css
drwxr-x--- 2 tomcat8 tomcat8 22 Nov 14 16:34 docs
drwxr-x--- 2 tomcat8 tomcat8 45 Nov 14 16:34 img
drwxr-x--- 3 tomcat8 tomcat8 65 Nov 14 16:34 js
drwxr-xr-x 24 root root 4096 Nov 14 17:10 node_modules
-rw-r----- 1 tomcat8 tomcat8 25820 Nov 14 17:10 package-lock.json
-rw-r----- 1 tomcat8 tomcat8 725 Nov 14 16:16 package.json
-rw-r----- 1 tomcat8 tomcat8 727 Nov 14 16:16 setup.sh
drwxr-x--- 3 tomcat8 tomcat8 19 Nov 14 16:34 src
drwxr-x--- 3 tomcat8 tomcat8 33 Nov 14 16:34 test

..And the logs are attached.
Thanks!
Michael

From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Saturday, November 17, 2018 4:06:58 AM
Subject: Re: [apollo] elastic search in genome apollo

Sorry for the late response.
Can you provide the browser console logs and any tomcat logs (catalina.out)?
Your modified docket file may be helpful as well.

Nathan

On Nov 15, 2018, at 10:26 AM, Michael Poeschla <[hidden email]> wrote:

yes - we don't see anything in the browser
Thanks,
Michael
From: "Colin" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 7:09:45 PM
Subject: Re: [apollo] elastic search in genome apollo

So you are saying that the plugin is not being added e.g. you don't have a menu item for the ElasticSearch dialog in jbrowse?
-Colin

On Thu, Nov 15, 2018 at 9:07 AM Michael Poeschla <[hidden email]> wrote:
hi,
elasticsearch was installed with docker. Here's the tracklist.json for the organism that we generated the index for. Is there any other info I could provide that would help?

{
   "elasticIndexName" : "nfz",
   "elasticSearchUrl" : "localhost:3000",
   "formatVersion" : 1,
   "names" : {
      "type" : "Hash",
      "url" : "names/"
   },
   "plugins" : [
      "ElasticSearch"
   ],
   "tracks" : [
      {
         "category" : "Reference sequence",
         "chunkSize" : 20000,
         "key" : "Reference sequence",
         "label" : "DNA",
         "seqType" : "dna",
         "storeClass" : "JBrowse/Store/Sequence/StaticChunked",
         "type" : "SequenceTrack",
         "urlTemplate" : "seq/{refseq_dirpath}/{refseq}-"
      },
      {
         "bicolor_pivot" : 0.5,
         "label" : "GC Content",
         "storeClass" : "JBrowse/Store/SeqFeature/SequenceChunks",
         "type" : "GCContent/View/Track/GCContentXY",
         "urlTemplate" : "seq/{refseq_dirpath}/{refseq}-"
      },
      {
         "compress" : 0,
         "key" : "Repeats",
         "label" : "Repeats",
         "storeClass" : "JBrowse/Store/SeqFeature/NCList",
         "style" : {
            "className" : "feature"
         },
         "trackType" : "JBrowse/View/Track/CanvasFeatures",
         "type" : "JBrowse/View/Track/CanvasFeatures",
         "urlTemplate" : "tracks/Repeats/{refseq}/trackData.json"
      },
      {
         "compress" : 0,
         "key" : "Annotation",
         "label" : "Annotationfromsumgff",
         "storeClass" : "JBrowse/Store/SeqFeature/NCList",
         "style" : {
            "className" : "feature"
         },
         "trackType" : "CanvasFeatures",
         "type" : "CanvasFeatures",
         "urlTemplate" : "tracks/Annotationfromsumgff/{refseq}/trackData.json"
      }
   ]
}

From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 11:33:03 AM
Subject: Re: [apollo] elastic search in genome apollo

Thanks. Also on Europe time for another 20 minutes.
Can you also post any other specifics of failure?

Have you installed elasticsearch via docker or manually ?

Nathan

On Nov 15, 2018, at 11:10 AM, Michael Poeschla <[hidden email]> wrote:

Here it is - sorry for the late replies, I'm on europe time.

environments {
    development {
    }
    test {
    }
    production {
        dataSource {
            dbCreate = "update"
            username = System.getenv("WEBAPOLLO_DB_USERNAME") ?: *
            password = System.getenv("WEBAPOLLO_DB_PASSWORD") ?: *

            driverClassName = "org.postgresql.Driver"
            dialect = "org.hibernate.dialect.PostgresPlusDialect"
            url = "jdbc:<a href="postgresql://$" class="">postgresql://${System.getenv("WEBAPOLLO_DB_HOST")?:"127.0.0.1"}/${System.getenv("WEBAPOLLO_DB_NAME")?:"apollo"}"

            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = false
                initialSize                   = 5
                maxActive                     = 50
                minIdle                       = 5
                maxIdle                       = 25
                maxWait                       = 10000
                maxAge                        = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis    = 60000
                validationQuery               = "SELECT 1"
                validationQueryTimeout        = 3
                validationInterval            = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
        dataSource_chado {
            dbCreate = "update"
            username = System.getenv("CHADO_DB_USERNAME") ?: *
            password = System.getenv("CHADO_DB_PASSWORD") ?: *

            driverClassName = "org.postgresql.Driver"
            dialect = "org.hibernate.dialect.PostgresPlusDialect"

            url = "jdbc:<a href="postgresql://$" class="">postgresql://${System.getenv("CHADO_DB_HOST")?:"127.0.0.1"}/${System.getenv("CHADO_DB_NAME")?:"chado"}"

            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = false
                initialSize                   = 5
                maxActive                     = 50
                minIdle                       = 5
                maxIdle                       = 25
                maxWait                       = 10000
                maxAge                        = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis    = 60000
                validationQuery               = "SELECT 1"
                validationQueryTimeout        = 3
                validationInterval            = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
    }
}

apollo {
    default_minimum_intron_size = System.getenv("WEBAPOLLO_MINIMUM_INTRON_SIZE") ? System.getenv("WEBAPOLLO_MINIMUM_INTRON_SIZE").toInteger() : 1
    history_size = System.getenv("WEBAPOLLO_HISTORY_SIZE") ? System.getenv("WEBAPOLLO_HISTORY_SIZE").toInteger() : 0
    overlapper_class = System.getenv("WEBAPOLLO_OVERLAPPER_CLASS") ?: "org.bbop.apollo.sequence.OrfOverlapper"
    use_cds_for_new_transcripts = System.getenv("WEBAPOLLO_CDS_FOR_NEW_TRANSCRIPTS").equals("true") // will default to false
    feature_has_dbxrefs = System.getenv("WEBAPOLLO_FEATURE_HAS_DBXREFS") ?: true
    feature_has_attributes = System.getenv("WEBAPOLLO_FEATURE_HAS_ATTRS") ?: true
    feature_has_pubmed_ids = System.getenv("WEBAPOLLO_FEATURE_HAS_PUBMED") ?: true
    feature_has_go_ids = System.getenv("WEBAPOLLO_FEATURE_HAS_GO") ?: true
    feature_has_comments = System.getenv("WEBAPOLLO_FEATURE_HAS_COMMENTS") ?: true
    feature_has_status = System.getenv("WEBAPOLLO_FEATURE_HAS_STATUS") ?: true
    translation_table = "/config/translation_tables/ncbi_" + (System.getenv("WEBAPOLLO_TRANSLATION_TABLE") ?: "1") + "_translation_table.txt"
    get_translation_code = System.getenv("WEBAPOLLO_TRANSLATION_TABLE") ? System.getenv("WEBAPOLLO_TRANSLATION_TABLE").toInteger() : 1

    // ADDED BLAT AND SET PATH TO BIN
    sequence_search_tools = [
        blat_nuc: [
          search_exe: "/usr/local/bin/blat",
          search_class: "org.bbop.apollo.sequence.search.blat.BlatCommandLineNucleotideToNucleotide",
          name: "Blat nucleotide",
          params: ""
        ],
        blat_prot: [
          search_exe: "/usr/local/bin/blat",
          search_class: "org.bbop.apollo.sequence.search.blat.BlatCommandLineProteinToNucleotide",
          name: "Blat protein",
          params: ""
        ]
      ]
    // TODO: should come from config or via preferences database
    splice_donor_sites = System.getenv("WEBAPOLLO_SPLICE_DONOR_SITES") ? System.getenv("WEBAPOLLO_SPLICE_DONOR_SITES").split(",") : ["GT"]
    splice_acceptor_sites = System.getenv("WEBAPOLLO_SPLICE_ACCEPTOR_SITES") ? System.getenv("WEBAPOLLO_SPLICE_ACCEPTOR_SITES").split(",") : ["AG"]
    gff3.source = System.getenv("WEBAPOLLO_GFF3_SOURCE") ?: "."

    google_analytics = System.getenv("WEBAPOLLO_GOOGLE_ANALYTICS_ID") ?: ["UA-62921593-1"]

    admin{
        username = System.getenv("APOLLO_ADMIN_EMAIL") ?: *
        password = System.getenv("APOLLO_ADMIN_PASSWORD") ?: *
        firstName = System.getenv("APOLLO_ADMIN_FIRST_NAME") ?: *
        lastName = System.getenv("APOLLO_ADMIN_LAST_NAME") ?: *
    }
}

jbrowse {
    git {
        url = "https://github.com/GMOD/jbrowse"
        tag = "maint/1.12.5-apollo"
    }
    plugins {
        WebApollo{
            included = true
        }
        NeatHTMLFeatures{
            included = System.getenv("WEBAPOLLO_JBROWSE_PLUGIN_NEATHTML") ?: true
        }
        NeatCanvasFeatures{
            included = System.getenv("WEBAPOLLO_JBROWSE_PLUGIN_NEATCANVAS") ?: true
        }
        RegexSequenceSearch{
            included = true
        }
        HideTrackLabels{
            included = true
        }
        //EDIT: ADDED GC CONTENT
        GCContent{
            git = 'https://github.com/cmdcolin/GCContent'
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
        //EDIT: ADDS ELASTIC SEARCH PLUGIN
        ElasticSearch{
            git = 'https://github.com/elsiklab/jbrowse_elasticsearch'
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
    }
}
From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 12:30:50 AM
Subject: Re: [apollo] elastic search in genome apollo

Can you post your apollo config file?

Nathan

On Nov 14, 2018, at 6:45 PM, Michael Poeschla <[hidden email]> wrote:

Hi,
I'm following the instructions from https://github.com/elsiklab/jbrowse_elasticsearch

we have elasticsearch and the middleware app.js running. I was able to execute generate-names.pl and this seemed to work correctly from the terminal output. The changes also got picked up in the tracklist.json for the genome. But there is no sign of the elasticsearch plugin in web apollo...added the plugin to the config.groovy, and we have another jbrowse plugin (GC content) working fine so not sure what's the problem here.

Thanks for all of your help! :)
Michael
From: "Nathan Dunn" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Tuesday, November 13, 2018 6:33:05 PM
Subject: Re: [apollo] elastic search in genome apollo

Michael,

What problems are you running it when adding the plugins?

Nathan

On Nov 13, 2018, at 4:55 PM, Colin <[hidden email]> wrote:

Aside from changing your gene identifiers, I think the full text search from elasticsearch could help. I forwarded some comments here about that https://github.com/elsiklab/jbrowse_elasticsearch/issues/7

Potentially data that is loaded in Apollo backend could use text similarity %like% or full text sql queries but for normal jbrowse tracks that are not loaded in the sql database, the jbrowse_elasticsearch plugin seems reasonable.

-Colin

On Tue, Nov 13, 2018 at 7:35 AM Michael Poeschla <[hidden email]> wrote:
Hi,

has anyone implemented the jbrowse_elasticsearch plugin in Apollo? I'm running from the docker container with an nginx reverse proxy and having trouble adding this plugin. How would I add this to the docker image, with the middleware as well?

The reason we want to do this is the predicted gene model names we have in our tracks are not always intuitive so it's hard to find them by searching in Apollo (ie mTOR is ID'd as "serine/threonine-protein kinase mTOR", so if you just start typing mTOR you get no results). Wondering also if anyone else has encountered this problem and found good solutions, besides manipulating all of our tracks.

Thanks so much,
Michael

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.
<tomcat.tar.gz>

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

Colin

Re: elastic search in genome apollo

I went and tested this and I tried testing adding the plugin to apollo-config.groovy and was getting an error about plugins/ElasticSearch/js/main.js not being loaded apollo run-local. I can see that it gets downloaded to jbrowse-download/plugins/ElasticSearch but it was giving me that error there still. I also ran apollo deploy and then unzipped and the jbrowse plugins folder did not contain ElasticSearch plugin

Screenshot attached and apollo-config.groovy snippet here

jbrowse {

plugins {

ElasticSearch {

git = 'https://github.com/elsiklab/jbrowse_elasticsearch'

branch = 'master'

alwaysRecheck = "true"

alwaysPull = "true"

}

-Colin

On Tue, Nov 20, 2018 at 6:57 PM Nathan Dunn <[hidden email]> wrote:

Thanks. Will have a look.

On Nov 20, 2018, at 5:03 AM, Michael Poeschla <[hidden email]> wrote:

Hi,

here are the logs, docker-compose, actual dockerfile and start and testing of npm (below). I've also added the tomcat logs as an attachment.

docker-compose:
--
version: "3.2"
services:

apollo:
image: hub.age.mpg.de/mpoeschla/dv-kb:0.4
volumes:
- ./data/extdata/:/data
- postgres-data:/var/lib/postgresql
- ./config/tomcat/server.xml:/var/lib/tomcat8/conf/server.xml:ro
- ./config/tomcat/setenv.sh:/usr/share/tomcat8/bin/setenv.sh
environment:
APOLLO_ADMIN_[hidden email]
restart: always
links:
- elasticsearch:elasticsearch

nginx:
image: library/nginx:alpine
restart: always
volumes:
- ./config/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
- ./config/secrets:/secrets:ro
ports:
- 80:80
- 443:443
links:
- apollo:apollo

elasticsearch:
image: elasticsearch:6.4.2
container_name: elasticsearch
environment:
- cluster.name=docker-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- esdata1:/usr/share/elasticsearch/data

volumes:
postgres-data:
esdata1:
driver: local
--
The actual Dockerfile:
--
# WebApollo
# VERSION 2.1.X
FROM tomcat:8-jre8
MAINTAINER Eric Rasche <[hidden email]>, Anthony Bretaudeau <[hidden email]>, Nathan Dunn <[hidden email]>
ENV DEBIAN_FRONTEND noninteractive

RUN apt-get -qq update --fix-missing && \
apt-get --no-install-recommends -y install \
git build-essential maven tomcat8 libpq-dev postgresql-common openjdk-8-jdk wget \
postgresql postgresql-client xmlstarlet netcat libpng-dev \
zlib1g-dev libexpat1-dev ant curl ssl-cert zip unzip

RUN curl -sL https://deb.nodesource.com/setup_8.x | bash -
RUN apt-get -qq update --fix-missing && \
apt-get --no-install-recommends -y install nodejs && \
apt-get autoremove -y && apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

RUN cp /usr/lib/jvm/java-8-openjdk-amd64/lib/tools.jar /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext/tools.jar && \
useradd -ms /bin/bash -d /apollo apollo

ENV WEBAPOLLO_VERSION 99c7e54c6e74fbc6705d57de216c9b14b2bfb03b
RUN curl -L https://github.com/GMOD/Apollo/archive/${WEBAPOLLO_VERSION}.tar.gz | tar xzf - --strip-components=1 -C /apollo

# install grails
COPY build.sh /bin/build.sh
ADD apollo-config.groovy /apollo/apollo-config.groovy

RUN chown -R apollo:apollo /apollo
RUN curl -s "http://hgdownload.soe.ucsc.edu/admin/exe/linux.x86_64/blat/blat" -o /usr/local/bin/blat
RUN chmod +x /usr/local/bin/blat

USER apollo
RUN curl -s get.sdkman.io | bash
RUN /bin/bash -c "source $HOME/.sdkman/bin/sdkman-init.sh && yes | sdk install grails 2.5.5"
RUN /bin/bash -c "source $HOME/.sdkman/bin/sdkman-init.sh && yes | sdk install gradle 3.2.1"

RUN /bin/bash -c "source $HOME/.profile && source $HOME/.sdkman/bin/sdkman-init.sh && /bin/bash /bin/build.sh"

USER root
ENV CATALINA_HOME=/var/lib/tomcat8
RUN rm -rf ${CATALINA_HOME}/webapps/* && \
cp /apollo/apollo*.war ${CATALINA_HOME}/apollo.war

ENV CONTEXT_PATH ROOT

# Download chado schema
RUN wget --quiet https://github.com/erasche/chado-schema-builder/releases/download/1.31-jenkins97/chado-1.31.sql.gz -O /chado.sql.gz && \
gunzip /chado.sql.gz

RUN wget https://optinsoft.net/pic/ipgen2.png -O test.png

ADD launch.sh /launch.sh
CMD "/launch.sh"
--
Starting npm inside the container:
root@995955a711e1:/var/lib/tomcat8/webapps/ROOT/jbrowse/plugins/ElasticSearch# npm start

> jbrowse_elasticsearch@1.0.4 start /var/lib/tomcat8/webapps/ROOT/jbrowse/plugins/ElasticSearch
> node ./bin/www

testing access with curl:
root@995955a711e1:/usr/local/tomcat# curl http://localhost:3000
{"hits":[],"total":0}root@995955a711e1:/usr/local/tomcat#

content of ElasticSearchPlugin:
^Croot@995955a711e1:/var/lib/tomcat8/webapps/ROOT/jbrowse/plugins/ElasticSearch# ls -la
total 68
drwxr-x--- 10 tomcat8 tomcat8 271 Nov 14 17:10 .
drwxr-x--- 12 tomcat8 tomcat8 223 Nov 14 16:34 ..
-rw-r----- 1 tomcat8 tomcat8 594 Nov 14 16:16 .eslintrc
-rw-r----- 1 tomcat8 tomcat8 290 Nov 14 16:16 .travis.yml
-rw-r----- 1 tomcat8 tomcat8 686 Nov 14 16:16 ChangeLog.md
-rw-r----- 1 tomcat8 tomcat8 810 Nov 14 16:16 Makefile.PL
-rw-r----- 1 tomcat8 tomcat8 5157 Nov 14 16:16 README.md
-rw-r----- 1 tomcat8 tomcat8 1888 Nov 14 16:44 app.js
drwxr-x--- 2 tomcat8 tomcat8 66 Nov 14 16:34 bin
drwxr-x--- 2 tomcat8 tomcat8 22 Nov 14 16:34 css
drwxr-x--- 2 tomcat8 tomcat8 22 Nov 14 16:34 docs
drwxr-x--- 2 tomcat8 tomcat8 45 Nov 14 16:34 img
drwxr-x--- 3 tomcat8 tomcat8 65 Nov 14 16:34 js
drwxr-xr-x 24 root root 4096 Nov 14 17:10 node_modules
-rw-r----- 1 tomcat8 tomcat8 25820 Nov 14 17:10 package-lock.json
-rw-r----- 1 tomcat8 tomcat8 725 Nov 14 16:16 package.json
-rw-r----- 1 tomcat8 tomcat8 727 Nov 14 16:16 setup.sh
drwxr-x--- 3 tomcat8 tomcat8 19 Nov 14 16:34 src
drwxr-x--- 3 tomcat8 tomcat8 33 Nov 14 16:34 test

..And the logs are attached.
Thanks!
Michael

From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Saturday, November 17, 2018 4:06:58 AM
Subject: Re: [apollo] elastic search in genome apollo

Sorry for the late response.
Can you provide the browser console logs and any tomcat logs (catalina.out)?
Your modified docket file may be helpful as well.

Nathan

On Nov 15, 2018, at 10:26 AM, Michael Poeschla <[hidden email]> wrote:

yes - we don't see anything in the browser
Thanks,
Michael
From: "Colin" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 7:09:45 PM
Subject: Re: [apollo] elastic search in genome apollo

So you are saying that the plugin is not being added e.g. you don't have a menu item for the ElasticSearch dialog in jbrowse?
-Colin

On Thu, Nov 15, 2018 at 9:07 AM Michael Poeschla <[hidden email]> wrote:
hi,
elasticsearch was installed with docker. Here's the tracklist.json for the organism that we generated the index for. Is there any other info I could provide that would help?

{
   "elasticIndexName" : "nfz",
   "elasticSearchUrl" : "localhost:3000",
   "formatVersion" : 1,
   "names" : {
      "type" : "Hash",
      "url" : "names/"
   },
   "plugins" : [
      "ElasticSearch"
   ],
   "tracks" : [
      {
         "category" : "Reference sequence",
         "chunkSize" : 20000,
         "key" : "Reference sequence",
         "label" : "DNA",
         "seqType" : "dna",
         "storeClass" : "JBrowse/Store/Sequence/StaticChunked",
         "type" : "SequenceTrack",
         "urlTemplate" : "seq/{refseq_dirpath}/{refseq}-"
      },
      {
         "bicolor_pivot" : 0.5,
         "label" : "GC Content",
         "storeClass" : "JBrowse/Store/SeqFeature/SequenceChunks",
         "type" : "GCContent/View/Track/GCContentXY",
         "urlTemplate" : "seq/{refseq_dirpath}/{refseq}-"
      },
      {
         "compress" : 0,
         "key" : "Repeats",
         "label" : "Repeats",
         "storeClass" : "JBrowse/Store/SeqFeature/NCList",
         "style" : {
            "className" : "feature"
         },
         "trackType" : "JBrowse/View/Track/CanvasFeatures",
         "type" : "JBrowse/View/Track/CanvasFeatures",
         "urlTemplate" : "tracks/Repeats/{refseq}/trackData.json"
      },
      {
         "compress" : 0,
         "key" : "Annotation",
         "label" : "Annotationfromsumgff",
         "storeClass" : "JBrowse/Store/SeqFeature/NCList",
         "style" : {
            "className" : "feature"
         },
         "trackType" : "CanvasFeatures",
         "type" : "CanvasFeatures",
         "urlTemplate" : "tracks/Annotationfromsumgff/{refseq}/trackData.json"
      }
   ]
}

From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 11:33:03 AM
Subject: Re: [apollo] elastic search in genome apollo

Thanks. Also on Europe time for another 20 minutes.
Can you also post any other specifics of failure?

Have you installed elasticsearch via docker or manually ?

Nathan

On Nov 15, 2018, at 11:10 AM, Michael Poeschla <[hidden email]> wrote:

Here it is - sorry for the late replies, I'm on europe time.

environments {
    development {
    }
    test {
    }
    production {
        dataSource {
            dbCreate = "update"
            username = System.getenv("WEBAPOLLO_DB_USERNAME") ?: *
            password = System.getenv("WEBAPOLLO_DB_PASSWORD") ?: *

            driverClassName = "org.postgresql.Driver"
            dialect = "org.hibernate.dialect.PostgresPlusDialect"
            url = "jdbc:postgresql://${System.getenv("WEBAPOLLO_DB_HOST")?:"127.0.0.1"}/${System.getenv("WEBAPOLLO_DB_NAME")?:"apollo"}"

            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = false
                initialSize                   = 5
                maxActive                     = 50
                minIdle                       = 5
                maxIdle                       = 25
                maxWait                       = 10000
                maxAge                        = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis    = 60000
                validationQuery               = "SELECT 1"
                validationQueryTimeout        = 3
                validationInterval            = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
        dataSource_chado {
            dbCreate = "update"
            username = System.getenv("CHADO_DB_USERNAME") ?: *
            password = System.getenv("CHADO_DB_PASSWORD") ?: *

            driverClassName = "org.postgresql.Driver"
            dialect = "org.hibernate.dialect.PostgresPlusDialect"

            url = "jdbc:postgresql://${System.getenv("CHADO_DB_HOST")?:"127.0.0.1"}/${System.getenv("CHADO_DB_NAME")?:"chado"}"

            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = false
                initialSize                   = 5
                maxActive                     = 50
                minIdle                       = 5
                maxIdle                       = 25
                maxWait                       = 10000
                maxAge                        = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis    = 60000
                validationQuery               = "SELECT 1"
                validationQueryTimeout        = 3
                validationInterval            = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
    }
}

apollo {
    default_minimum_intron_size = System.getenv("WEBAPOLLO_MINIMUM_INTRON_SIZE") ? System.getenv("WEBAPOLLO_MINIMUM_INTRON_SIZE").toInteger() : 1
    history_size = System.getenv("WEBAPOLLO_HISTORY_SIZE") ? System.getenv("WEBAPOLLO_HISTORY_SIZE").toInteger() : 0
    overlapper_class = System.getenv("WEBAPOLLO_OVERLAPPER_CLASS") ?: "org.bbop.apollo.sequence.OrfOverlapper"
    use_cds_for_new_transcripts = System.getenv("WEBAPOLLO_CDS_FOR_NEW_TRANSCRIPTS").equals("true") // will default to false
    feature_has_dbxrefs = System.getenv("WEBAPOLLO_FEATURE_HAS_DBXREFS") ?: true
    feature_has_attributes = System.getenv("WEBAPOLLO_FEATURE_HAS_ATTRS") ?: true
    feature_has_pubmed_ids = System.getenv("WEBAPOLLO_FEATURE_HAS_PUBMED") ?: true
    feature_has_go_ids = System.getenv("WEBAPOLLO_FEATURE_HAS_GO") ?: true
    feature_has_comments = System.getenv("WEBAPOLLO_FEATURE_HAS_COMMENTS") ?: true
    feature_has_status = System.getenv("WEBAPOLLO_FEATURE_HAS_STATUS") ?: true
    translation_table = "/config/translation_tables/ncbi_" + (System.getenv("WEBAPOLLO_TRANSLATION_TABLE") ?: "1") + "_translation_table.txt"
    get_translation_code = System.getenv("WEBAPOLLO_TRANSLATION_TABLE") ? System.getenv("WEBAPOLLO_TRANSLATION_TABLE").toInteger() : 1

    // ADDED BLAT AND SET PATH TO BIN
    sequence_search_tools = [
        blat_nuc: [
          search_exe: "/usr/local/bin/blat",
          search_class: "org.bbop.apollo.sequence.search.blat.BlatCommandLineNucleotideToNucleotide",
          name: "Blat nucleotide",
          params: ""
        ],
        blat_prot: [
          search_exe: "/usr/local/bin/blat",
          search_class: "org.bbop.apollo.sequence.search.blat.BlatCommandLineProteinToNucleotide",
          name: "Blat protein",
          params: ""
        ]
      ]
    // TODO: should come from config or via preferences database
    splice_donor_sites = System.getenv("WEBAPOLLO_SPLICE_DONOR_SITES") ? System.getenv("WEBAPOLLO_SPLICE_DONOR_SITES").split(",") : ["GT"]
    splice_acceptor_sites = System.getenv("WEBAPOLLO_SPLICE_ACCEPTOR_SITES") ? System.getenv("WEBAPOLLO_SPLICE_ACCEPTOR_SITES").split(",") : ["AG"]
    gff3.source = System.getenv("WEBAPOLLO_GFF3_SOURCE") ?: "."

    google_analytics = System.getenv("WEBAPOLLO_GOOGLE_ANALYTICS_ID") ?: ["UA-62921593-1"]

    admin{
        username = System.getenv("APOLLO_ADMIN_EMAIL") ?: *
        password = System.getenv("APOLLO_ADMIN_PASSWORD") ?: *
        firstName = System.getenv("APOLLO_ADMIN_FIRST_NAME") ?: *
        lastName = System.getenv("APOLLO_ADMIN_LAST_NAME") ?: *
    }
}

jbrowse {
    git {
        url = "https://github.com/GMOD/jbrowse"
        tag = "maint/1.12.5-apollo"
    }
    plugins {
        WebApollo{
            included = true
        }
        NeatHTMLFeatures{
            included = System.getenv("WEBAPOLLO_JBROWSE_PLUGIN_NEATHTML") ?: true
        }
        NeatCanvasFeatures{
            included = System.getenv("WEBAPOLLO_JBROWSE_PLUGIN_NEATCANVAS") ?: true
        }
        RegexSequenceSearch{
            included = true
        }
        HideTrackLabels{
            included = true
        }
        //EDIT: ADDED GC CONTENT
        GCContent{
            git = 'https://github.com/cmdcolin/GCContent'
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
        //EDIT: ADDS ELASTIC SEARCH PLUGIN
        ElasticSearch{
            git = 'https://github.com/elsiklab/jbrowse_elasticsearch'
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
    }
}
From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 12:30:50 AM
Subject: Re: [apollo] elastic search in genome apollo

Can you post your apollo config file?

Nathan

On Nov 14, 2018, at 6:45 PM, Michael Poeschla <[hidden email]> wrote:

Hi,
I'm following the instructions from https://github.com/elsiklab/jbrowse_elasticsearch

we have elasticsearch and the middleware app.js running. I was able to execute generate-names.pl and this seemed to work correctly from the terminal output. The changes also got picked up in the tracklist.json for the genome. But there is no sign of the elasticsearch plugin in web apollo...added the plugin to the config.groovy, and we have another jbrowse plugin (GC content) working fine so not sure what's the problem here.

Thanks for all of your help! :)
Michael
From: "Nathan Dunn" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Tuesday, November 13, 2018 6:33:05 PM
Subject: Re: [apollo] elastic search in genome apollo

Michael,

What problems are you running it when adding the plugins?

Nathan

On Nov 13, 2018, at 4:55 PM, Colin <[hidden email]> wrote:

Aside from changing your gene identifiers, I think the full text search from elasticsearch could help. I forwarded some comments here about that https://github.com/elsiklab/jbrowse_elasticsearch/issues/7

Potentially data that is loaded in Apollo backend could use text similarity %like% or full text sql queries but for normal jbrowse tracks that are not loaded in the sql database, the jbrowse_elasticsearch plugin seems reasonable.

-Colin

On Tue, Nov 13, 2018 at 7:35 AM Michael Poeschla <[hidden email]> wrote:
Hi,

has anyone implemented the jbrowse_elasticsearch plugin in Apollo? I'm running from the docker container with an nginx reverse proxy and having trouble adding this plugin. How would I add this to the docker image, with the middleware as well?

The reason we want to do this is the predicted gene model names we have in our tracks are not always intuitive so it's hard to find them by searching in Apollo (ie mTOR is ID'd as "serine/threonine-protein kinase mTOR", so if you just start typing mTOR you get no results). Wondering also if anyone else has encountered this problem and found good solutions, besides manipulating all of our tracks.

Thanks so much,
Michael

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.
<tomcat.tar.gz>

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

VirtualBox_Ubuntu5_21_11_2018_13_06_07.png (293K) Download Attachment

nathandunn

Re: elastic search in genome apollo

Thanks for testing Colin. I wasn’t sure the exact plugin to test with.

I did get it to deploy off of master. I did an ./apollo clean-all

I did an ./apollo deploy first (not sure if that made any difference, but it should not have).

My config is the sample-h2-config.groovy file with the snippet that Colin provided below.

That being said, I didn’t do a full test with the elastic search plugin, but this seems to install it correctly and I get an organism window, etc. when I load the volvox data.

Nathan

// default username/password on h2 are given here. see docs for adjusting

dataSource {

pooled = true

jmxExport = true

driverClassName = "org.h2.Driver"

username = "sa"

password = ""

}

// environment specific settings

environments {

development {

// sample config to turn on debug logging in development e.g. for apollo run-local

log4j.main = {

debug "grails.app"

}

// sample config to edit apollo specific configs in development mode

apollo {

gff3.source = "testing"

}

dataSource {

// NOTE: this is in memory, so it will be deleted.

dbCreate = "update" // one of 'create', 'create-drop', 'update', 'validate', ''

url = "jdbc:h2:AnnotationDatabase:devDb;MVCC=TRUE;LOCK_TIMEOUT=10000;DB_CLOSE_ON_EXIT=FALSE"

}

test {

dataSource {

dbCreate = "create-drop" // one of 'create', 'create-drop', 'update', 'validate', ''

url = "jdbc:h2:mem:testDb;MVCC=TRUE;LOCK_TIMEOUT=10000;DB_CLOSE_ON_EXIT=FALSE"

}

//note: not necessarily recommended to use h2 in production mode. see precautions

production {

dataSource {

dbCreate = "update"

//NOTE: production mode uses file instead of mem database

//NOTE: Please specify the appropriate file path, otherwise /tmp/prodDb will be used.

url = "jdbc:h2:/opt/apollo/h2/prodDb;MVCC=TRUE;LOCK_TIMEOUT=10000;DB_CLOSE_ON_EXIT=FALSE"

properties {

// See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation

jmxEnabled = true

initialSize = 5

maxActive = 50

minIdle = 5

maxIdle = 25

maxWait = 10000

maxAge = 10 * 60000

timeBetweenEvictionRunsMillis = 5000

minEvictableIdleTimeMillis = 60000

validationQuery = "SELECT 1"

validationQueryTimeout = 3

validationInterval = 15000

testOnBorrow = true

testWhileIdle = true

testOnReturn = false

jdbcInterceptors = "ConnectionState"

defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED

}

// Uncomment to change the default memory configurations

//grails.project.fork = [

// test : false,

// // configure settings for the run-app JVM

// run : [maxMemory: 2048, minMemory: 64, debug: false, maxPerm: 1024, forkReserve: false],

// // configure settings for the run-war JVM

// war : [maxMemory: 2048, minMemory: 64, debug: false, maxPerm: 1024, forkReserve: false],

// // configure settings for the Console UI JVM

// console: [maxMemory: 2048, minMemory: 64, debug: false, maxPerm: 1024]

//]

// Uncomment to make changes

jbrowse {

git {

url= "https://github.com/GMOD/jbrowse"

tag = "maint/1.12.5-apollo"

alwaysPull = true

alwaysRecheck = true

}

plugins {

NeatHTMLFeatures{

included = true

}

NeatCanvasFeatures{

included = true

}

RegexSequenceSearch{

included = true

}

HideTrackLabels{

included = true

}

ElasticSearch {

git = 'https://github.com/elsiklab/jbrowse_elasticsearch'

branch = 'master'

alwaysRecheck = "true"

alwaysPull = "true"

}

// MyVariantInfo {

// git = 'https://github.com/GMOD/myvariantviewer'

// branch = 'master'

// alwaysRecheck = "true"

// alwaysPull = "true"

// }

// SashimiPlot {

// git = 'https://github.com/cmdcolin/sashimiplot'

// branch = 'master'

// alwaysPull = "true"

// }

}

On Nov 21, 2018, at 10:09 AM, Colin <[hidden email]> wrote:

I went and tested this and I tried testing adding the plugin to apollo-config.groovy and was getting an error about plugins/ElasticSearch/js/main.js not being loaded apollo run-local. I can see that it gets downloaded to jbrowse-download/plugins/ElasticSearch but it was giving me that error there still. I also ran apollo deploy and then unzipped and the jbrowse plugins folder did not contain ElasticSearch plugin

Screenshot attached and apollo-config.groovy snippet here

jbrowse {
plugins {
ElasticSearch {
git = 'https://github.com/elsiklab/jbrowse_elasticsearch'
branch = 'master'
alwaysRecheck = "true"
alwaysPull = "true"
}
}
}

-Colin

On Tue, Nov 20, 2018 at 6:57 PM Nathan Dunn <[hidden email]> wrote:

Thanks. Will have a look.

On Nov 20, 2018, at 5:03 AM, Michael Poeschla <[hidden email]> wrote:

Hi,

here are the logs, docker-compose, actual dockerfile and start and testing of npm (below). I've also added the tomcat logs as an attachment.

docker-compose:
--
version: "3.2"
services:

apollo:
image: hub.age.mpg.de/mpoeschla/dv-kb:0.4
volumes:
- ./data/extdata/:/data
- postgres-data:/var/lib/postgresql
- ./config/tomcat/server.xml:/var/lib/tomcat8/conf/server.xml:ro
- ./config/tomcat/setenv.sh:/usr/share/tomcat8/bin/setenv.sh
environment:
APOLLO_ADMIN_[hidden email]
restart: always
links:
- elasticsearch:elasticsearch

nginx:
image: library/nginx:alpine
restart: always
volumes:
- ./config/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
- ./config/secrets:/secrets:ro
ports:
- 80:80
- 443:443
links:
- apollo:apollo

elasticsearch:
image: elasticsearch:6.4.2
container_name: elasticsearch
environment:
- cluster.name=docker-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- esdata1:/usr/share/elasticsearch/data

volumes:
postgres-data:
esdata1:
driver: local
--
The actual Dockerfile:
--
# WebApollo
# VERSION 2.1.X
FROM tomcat:8-jre8
MAINTAINER Eric Rasche <[hidden email]>, Anthony Bretaudeau <[hidden email]>, Nathan Dunn <[hidden email]>
ENV DEBIAN_FRONTEND noninteractive

RUN apt-get -qq update --fix-missing && \
apt-get --no-install-recommends -y install \
git build-essential maven tomcat8 libpq-dev postgresql-common openjdk-8-jdk wget \
postgresql postgresql-client xmlstarlet netcat libpng-dev \
zlib1g-dev libexpat1-dev ant curl ssl-cert zip unzip

RUN curl -sL https://deb.nodesource.com/setup_8.x | bash -
RUN apt-get -qq update --fix-missing && \
apt-get --no-install-recommends -y install nodejs && \
apt-get autoremove -y && apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

RUN cp /usr/lib/jvm/java-8-openjdk-amd64/lib/tools.jar /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext/tools.jar && \
useradd -ms /bin/bash -d /apollo apollo

ENV WEBAPOLLO_VERSION 99c7e54c6e74fbc6705d57de216c9b14b2bfb03b
RUN curl -L https://github.com/GMOD/Apollo/archive/${WEBAPOLLO_VERSION}.tar.gz | tar xzf - --strip-components=1 -C /apollo

# install grails
COPY build.sh /bin/build.sh
ADD apollo-config.groovy /apollo/apollo-config.groovy

RUN chown -R apollo:apollo /apollo
RUN curl -s "http://hgdownload.soe.ucsc.edu/admin/exe/linux.x86_64/blat/blat" -o /usr/local/bin/blat
RUN chmod +x /usr/local/bin/blat

USER apollo
RUN curl -s get.sdkman.io | bash
RUN /bin/bash -c "source $HOME/.sdkman/bin/sdkman-init.sh && yes | sdk install grails 2.5.5"
RUN /bin/bash -c "source $HOME/.sdkman/bin/sdkman-init.sh && yes | sdk install gradle 3.2.1"

RUN /bin/bash -c "source $HOME/.profile && source $HOME/.sdkman/bin/sdkman-init.sh && /bin/bash /bin/build.sh"

USER root
ENV CATALINA_HOME=/var/lib/tomcat8
RUN rm -rf ${CATALINA_HOME}/webapps/* && \
cp /apollo/apollo*.war ${CATALINA_HOME}/apollo.war

ENV CONTEXT_PATH ROOT

# Download chado schema
RUN wget --quiet https://github.com/erasche/chado-schema-builder/releases/download/1.31-jenkins97/chado-1.31.sql.gz -O /chado.sql.gz && \
gunzip /chado.sql.gz

RUN wget https://optinsoft.net/pic/ipgen2.png -O test.png

ADD launch.sh /launch.sh
CMD "/launch.sh"
--
Starting npm inside the container:
root@995955a711e1:/var/lib/tomcat8/webapps/ROOT/jbrowse/plugins/ElasticSearch# npm start

> jbrowse_elasticsearch@1.0.4 start /var/lib/tomcat8/webapps/ROOT/jbrowse/plugins/ElasticSearch
> node ./bin/www

testing access with curl:
root@995955a711e1:/usr/local/tomcat# curl http://localhost:3000
{"hits":[],"total":0}root@995955a711e1:/usr/local/tomcat#

content of ElasticSearchPlugin:
^Croot@995955a711e1:/var/lib/tomcat8/webapps/ROOT/jbrowse/plugins/ElasticSearch# ls -la
total 68
drwxr-x--- 10 tomcat8 tomcat8 271 Nov 14 17:10 .
drwxr-x--- 12 tomcat8 tomcat8 223 Nov 14 16:34 ..
-rw-r----- 1 tomcat8 tomcat8 594 Nov 14 16:16 .eslintrc
-rw-r----- 1 tomcat8 tomcat8 290 Nov 14 16:16 .travis.yml
-rw-r----- 1 tomcat8 tomcat8 686 Nov 14 16:16 ChangeLog.md
-rw-r----- 1 tomcat8 tomcat8 810 Nov 14 16:16 Makefile.PL
-rw-r----- 1 tomcat8 tomcat8 5157 Nov 14 16:16 README.md
-rw-r----- 1 tomcat8 tomcat8 1888 Nov 14 16:44 app.js
drwxr-x--- 2 tomcat8 tomcat8 66 Nov 14 16:34 bin
drwxr-x--- 2 tomcat8 tomcat8 22 Nov 14 16:34 css
drwxr-x--- 2 tomcat8 tomcat8 22 Nov 14 16:34 docs
drwxr-x--- 2 tomcat8 tomcat8 45 Nov 14 16:34 img
drwxr-x--- 3 tomcat8 tomcat8 65 Nov 14 16:34 js
drwxr-xr-x 24 root root 4096 Nov 14 17:10 node_modules
-rw-r----- 1 tomcat8 tomcat8 25820 Nov 14 17:10 package-lock.json
-rw-r----- 1 tomcat8 tomcat8 725 Nov 14 16:16 package.json
-rw-r----- 1 tomcat8 tomcat8 727 Nov 14 16:16 setup.sh
drwxr-x--- 3 tomcat8 tomcat8 19 Nov 14 16:34 src
drwxr-x--- 3 tomcat8 tomcat8 33 Nov 14 16:34 test

..And the logs are attached.
Thanks!
Michael

From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Saturday, November 17, 2018 4:06:58 AM
Subject: Re: [apollo] elastic search in genome apollo

Sorry for the late response.
Can you provide the browser console logs and any tomcat logs (catalina.out)?
Your modified docket file may be helpful as well.

Nathan

On Nov 15, 2018, at 10:26 AM, Michael Poeschla <[hidden email]> wrote:

yes - we don't see anything in the browser
Thanks,
Michael
From: "Colin" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 7:09:45 PM
Subject: Re: [apollo] elastic search in genome apollo

So you are saying that the plugin is not being added e.g. you don't have a menu item for the ElasticSearch dialog in jbrowse?
-Colin

On Thu, Nov 15, 2018 at 9:07 AM Michael Poeschla <[hidden email]> wrote:
hi,
elasticsearch was installed with docker. Here's the tracklist.json for the organism that we generated the index for. Is there any other info I could provide that would help?

{
   "elasticIndexName" : "nfz",
   "elasticSearchUrl" : "localhost:3000",
   "formatVersion" : 1,
   "names" : {
      "type" : "Hash",
      "url" : "names/"
   },
   "plugins" : [
      "ElasticSearch"
   ],
   "tracks" : [
      {
         "category" : "Reference sequence",
         "chunkSize" : 20000,
         "key" : "Reference sequence",
         "label" : "DNA",
         "seqType" : "dna",
         "storeClass" : "JBrowse/Store/Sequence/StaticChunked",
         "type" : "SequenceTrack",
         "urlTemplate" : "seq/{refseq_dirpath}/{refseq}-"
      },
      {
         "bicolor_pivot" : 0.5,
         "label" : "GC Content",
         "storeClass" : "JBrowse/Store/SeqFeature/SequenceChunks",
         "type" : "GCContent/View/Track/GCContentXY",
         "urlTemplate" : "seq/{refseq_dirpath}/{refseq}-"
      },
      {
         "compress" : 0,
         "key" : "Repeats",
         "label" : "Repeats",
         "storeClass" : "JBrowse/Store/SeqFeature/NCList",
         "style" : {
            "className" : "feature"
         },
         "trackType" : "JBrowse/View/Track/CanvasFeatures",
         "type" : "JBrowse/View/Track/CanvasFeatures",
         "urlTemplate" : "tracks/Repeats/{refseq}/trackData.json"
      },
      {
         "compress" : 0,
         "key" : "Annotation",
         "label" : "Annotationfromsumgff",
         "storeClass" : "JBrowse/Store/SeqFeature/NCList",
         "style" : {
            "className" : "feature"
         },
         "trackType" : "CanvasFeatures",
         "type" : "CanvasFeatures",
         "urlTemplate" : "tracks/Annotationfromsumgff/{refseq}/trackData.json"
      }
   ]
}

From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 11:33:03 AM
Subject: Re: [apollo] elastic search in genome apollo

Thanks. Also on Europe time for another 20 minutes.
Can you also post any other specifics of failure?

Have you installed elasticsearch via docker or manually ?

Nathan

On Nov 15, 2018, at 11:10 AM, Michael Poeschla <[hidden email]> wrote:

Here it is - sorry for the late replies, I'm on europe time.

environments {
    development {
    }
    test {
    }
    production {
        dataSource {
            dbCreate = "update"
            username = System.getenv("WEBAPOLLO_DB_USERNAME") ?: *
            password = System.getenv("WEBAPOLLO_DB_PASSWORD") ?: *

            driverClassName = "org.postgresql.Driver"
            dialect = "org.hibernate.dialect.PostgresPlusDialect"
            url = "jdbc:postgresql://${System.getenv("WEBAPOLLO_DB_HOST")?:"127.0.0.1"}/${System.getenv("WEBAPOLLO_DB_NAME")?:"apollo"}"

            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = false
                initialSize                   = 5
                maxActive                     = 50
                minIdle                       = 5
                maxIdle                       = 25
                maxWait                       = 10000
                maxAge                        = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis    = 60000
                validationQuery               = "SELECT 1"
                validationQueryTimeout        = 3
                validationInterval            = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
        dataSource_chado {
            dbCreate = "update"
            username = System.getenv("CHADO_DB_USERNAME") ?: *
            password = System.getenv("CHADO_DB_PASSWORD") ?: *

            driverClassName = "org.postgresql.Driver"
            dialect = "org.hibernate.dialect.PostgresPlusDialect"

            url = "jdbc:postgresql://${System.getenv("CHADO_DB_HOST")?:"127.0.0.1"}/${System.getenv("CHADO_DB_NAME")?:"chado"}"

            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = false
                initialSize                   = 5
                maxActive                     = 50
                minIdle                       = 5
                maxIdle                       = 25
                maxWait                       = 10000
                maxAge                        = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis    = 60000
                validationQuery               = "SELECT 1"
                validationQueryTimeout        = 3
                validationInterval            = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
    }
}

apollo {
    default_minimum_intron_size = System.getenv("WEBAPOLLO_MINIMUM_INTRON_SIZE") ? System.getenv("WEBAPOLLO_MINIMUM_INTRON_SIZE").toInteger() : 1
    history_size = System.getenv("WEBAPOLLO_HISTORY_SIZE") ? System.getenv("WEBAPOLLO_HISTORY_SIZE").toInteger() : 0
    overlapper_class = System.getenv("WEBAPOLLO_OVERLAPPER_CLASS") ?: "org.bbop.apollo.sequence.OrfOverlapper"
    use_cds_for_new_transcripts = System.getenv("WEBAPOLLO_CDS_FOR_NEW_TRANSCRIPTS").equals("true") // will default to false
    feature_has_dbxrefs = System.getenv("WEBAPOLLO_FEATURE_HAS_DBXREFS") ?: true
    feature_has_attributes = System.getenv("WEBAPOLLO_FEATURE_HAS_ATTRS") ?: true
    feature_has_pubmed_ids = System.getenv("WEBAPOLLO_FEATURE_HAS_PUBMED") ?: true
    feature_has_go_ids = System.getenv("WEBAPOLLO_FEATURE_HAS_GO") ?: true
    feature_has_comments = System.getenv("WEBAPOLLO_FEATURE_HAS_COMMENTS") ?: true
    feature_has_status = System.getenv("WEBAPOLLO_FEATURE_HAS_STATUS") ?: true
    translation_table = "/config/translation_tables/ncbi_" + (System.getenv("WEBAPOLLO_TRANSLATION_TABLE") ?: "1") + "_translation_table.txt"
    get_translation_code = System.getenv("WEBAPOLLO_TRANSLATION_TABLE") ? System.getenv("WEBAPOLLO_TRANSLATION_TABLE").toInteger() : 1

    // ADDED BLAT AND SET PATH TO BIN
    sequence_search_tools = [
        blat_nuc: [
          search_exe: "/usr/local/bin/blat",
          search_class: "org.bbop.apollo.sequence.search.blat.BlatCommandLineNucleotideToNucleotide",
          name: "Blat nucleotide",
          params: ""
        ],
        blat_prot: [
          search_exe: "/usr/local/bin/blat",
          search_class: "org.bbop.apollo.sequence.search.blat.BlatCommandLineProteinToNucleotide",
          name: "Blat protein",
          params: ""
        ]
      ]
    // TODO: should come from config or via preferences database
    splice_donor_sites = System.getenv("WEBAPOLLO_SPLICE_DONOR_SITES") ? System.getenv("WEBAPOLLO_SPLICE_DONOR_SITES").split(",") : ["GT"]
    splice_acceptor_sites = System.getenv("WEBAPOLLO_SPLICE_ACCEPTOR_SITES") ? System.getenv("WEBAPOLLO_SPLICE_ACCEPTOR_SITES").split(",") : ["AG"]
    gff3.source = System.getenv("WEBAPOLLO_GFF3_SOURCE") ?: "."

    google_analytics = System.getenv("WEBAPOLLO_GOOGLE_ANALYTICS_ID") ?: ["UA-62921593-1"]

    admin{
        username = System.getenv("APOLLO_ADMIN_EMAIL") ?: *
        password = System.getenv("APOLLO_ADMIN_PASSWORD") ?: *
        firstName = System.getenv("APOLLO_ADMIN_FIRST_NAME") ?: *
        lastName = System.getenv("APOLLO_ADMIN_LAST_NAME") ?: *
    }
}

jbrowse {
    git {
        url = "https://github.com/GMOD/jbrowse"
        tag = "maint/1.12.5-apollo"
    }
    plugins {
        WebApollo{
            included = true
        }
        NeatHTMLFeatures{
            included = System.getenv("WEBAPOLLO_JBROWSE_PLUGIN_NEATHTML") ?: true
        }
        NeatCanvasFeatures{
            included = System.getenv("WEBAPOLLO_JBROWSE_PLUGIN_NEATCANVAS") ?: true
        }
        RegexSequenceSearch{
            included = true
        }
        HideTrackLabels{
            included = true
        }
        //EDIT: ADDED GC CONTENT
        GCContent{
            git = 'https://github.com/cmdcolin/GCContent'
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
        //EDIT: ADDS ELASTIC SEARCH PLUGIN
        ElasticSearch{
            git = 'https://github.com/elsiklab/jbrowse_elasticsearch'
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
    }
}
From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 12:30:50 AM
Subject: Re: [apollo] elastic search in genome apollo

Can you post your apollo config file?

Nathan

On Nov 14, 2018, at 6:45 PM, Michael Poeschla <[hidden email]> wrote:

Hi,
I'm following the instructions from https://github.com/elsiklab/jbrowse_elasticsearch

we have elasticsearch and the middleware app.js running. I was able to execute generate-names.pl and this seemed to work correctly from the terminal output. The changes also got picked up in the tracklist.json for the genome. But there is no sign of the elasticsearch plugin in web apollo...added the plugin to the config.groovy, and we have another jbrowse plugin (GC content) working fine so not sure what's the problem here.

Thanks for all of your help! :)
Michael
From: "Nathan Dunn" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Tuesday, November 13, 2018 6:33:05 PM
Subject: Re: [apollo] elastic search in genome apollo

Michael,

What problems are you running it when adding the plugins?

Nathan

On Nov 13, 2018, at 4:55 PM, Colin <[hidden email]> wrote:

Aside from changing your gene identifiers, I think the full text search from elasticsearch could help. I forwarded some comments here about that https://github.com/elsiklab/jbrowse_elasticsearch/issues/7

Potentially data that is loaded in Apollo backend could use text similarity %like% or full text sql queries but for normal jbrowse tracks that are not loaded in the sql database, the jbrowse_elasticsearch plugin seems reasonable.

-Colin

On Tue, Nov 13, 2018 at 7:35 AM Michael Poeschla <[hidden email]> wrote:
Hi,

has anyone implemented the jbrowse_elasticsearch plugin in Apollo? I'm running from the docker container with an nginx reverse proxy and having trouble adding this plugin. How would I add this to the docker image, with the middleware as well?

The reason we want to do this is the predicted gene model names we have in our tracks are not always intuitive so it's hard to find them by searching in Apollo (ie mTOR is ID'd as "serine/threonine-protein kinase mTOR", so if you just start typing mTOR you get no results). Wondering also if anyone else has encountered this problem and found good solutions, besides manipulating all of our tracks.

Thanks so much,
Michael

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.
<tomcat.tar.gz>

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

<VirtualBox_Ubuntu5_21_11_2018_13_06_07.png>

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

Michael Poeschla

Re: elastic search in genome apollo

the reason the plugin didn't show up turned out to be because i was editing both the apollo-config.groovy and the tracklist.json as you would for jbrowse. When I removed that line from the tracklist.json, the elasticsearch popped up immediately. This was a while back, just remembered to add it here in case anyone else runs into this kind of issue.

Best,
Michael

From: "Nathan Dunn" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Wednesday, November 21, 2018 9:05:13 PM
Subject: Re: [apollo] elastic search in genome apollo

Thanks for testing Colin. I wasn’t sure the exact plugin to test with.

I did get it to deploy off of master. I did an ./apollo clean-all

I did an ./apollo deploy first (not sure if that made any difference, but it should not have).

My config is the sample-h2-config.groovy file with the snippet that Colin provided below.

That being said, I didn’t do a full test with the elastic search plugin, but this seems to install it correctly and I get an organism window, etc. when I load the volvox data.

Nathan

// default username/password on h2 are given here. see docs for adjusting

dataSource {

pooled = true

jmxExport = true

driverClassName = "org.h2.Driver"

username = "sa"

password = ""

}

// environment specific settings

environments {

development {

// sample config to turn on debug logging in development e.g. for apollo run-local

log4j.main = {

debug "grails.app"

}

// sample config to edit apollo specific configs in development mode

apollo {

gff3.source = "testing"

}

dataSource {

// NOTE: this is in memory, so it will be deleted.

dbCreate = "update" // one of 'create', 'create-drop', 'update', 'validate', ''

url = "jdbc:h2:AnnotationDatabase:devDb;MVCC=TRUE;LOCK_TIMEOUT=10000;DB_CLOSE_ON_EXIT=FALSE"

}

test {

dataSource {

dbCreate = "create-drop" // one of 'create', 'create-drop', 'update', 'validate', ''

url = "jdbc:h2:mem:testDb;MVCC=TRUE;LOCK_TIMEOUT=10000;DB_CLOSE_ON_EXIT=FALSE"

}

//note: not necessarily recommended to use h2 in production mode. see precautions

production {

dataSource {

dbCreate = "update"

//NOTE: production mode uses file instead of mem database

//NOTE: Please specify the appropriate file path, otherwise /tmp/prodDb will be used.

url = "jdbc:h2:/opt/apollo/h2/prodDb;MVCC=TRUE;LOCK_TIMEOUT=10000;DB_CLOSE_ON_EXIT=FALSE"

properties {

// See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation

jmxEnabled = true

initialSize = 5

maxActive = 50

minIdle = 5

maxIdle = 25

maxWait = 10000

maxAge = 10 * 60000

timeBetweenEvictionRunsMillis = 5000

minEvictableIdleTimeMillis = 60000

validationQuery = "SELECT 1"

validationQueryTimeout = 3

validationInterval = 15000

testOnBorrow = true

testWhileIdle = true

testOnReturn = false

jdbcInterceptors = "ConnectionState"

defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED

}

// Uncomment to change the default memory configurations

//grails.project.fork = [

// test : false,

// // configure settings for the run-app JVM

// run : [maxMemory: 2048, minMemory: 64, debug: false, maxPerm: 1024, forkReserve: false],

// // configure settings for the run-war JVM

// war : [maxMemory: 2048, minMemory: 64, debug: false, maxPerm: 1024, forkReserve: false],

// // configure settings for the Console UI JVM

// console: [maxMemory: 2048, minMemory: 64, debug: false, maxPerm: 1024]

//]

// Uncomment to make changes

jbrowse {

git {

url= "https://github.com/GMOD/jbrowse"

tag = "maint/1.12.5-apollo"

alwaysPull = true

alwaysRecheck = true

}

plugins {

NeatHTMLFeatures{

included = true

}

NeatCanvasFeatures{

included = true

}

RegexSequenceSearch{

included = true

}

HideTrackLabels{

included = true

}

ElasticSearch {

git = 'https://github.com/elsiklab/jbrowse_elasticsearch'

branch = 'master'

alwaysRecheck = "true"

alwaysPull = "true"

}

// MyVariantInfo {

// git = 'https://github.com/GMOD/myvariantviewer'

// branch = 'master'

// alwaysRecheck = "true"

// alwaysPull = "true"

// }

// SashimiPlot {

// git = 'https://github.com/cmdcolin/sashimiplot'

// branch = 'master'

// alwaysPull = "true"

// }

}

On Nov 21, 2018, at 10:09 AM, Colin <[hidden email]> wrote:

I went and tested this and I tried testing adding the plugin to apollo-config.groovy and was getting an error about plugins/ElasticSearch/js/main.js not being loaded apollo run-local. I can see that it gets downloaded to jbrowse-download/plugins/ElasticSearch but it was giving me that error there still. I also ran apollo deploy and then unzipped and the jbrowse plugins folder did not contain ElasticSearch plugin

Screenshot attached and apollo-config.groovy snippet here

jbrowse {
plugins {
ElasticSearch {
git = 'https://github.com/elsiklab/jbrowse_elasticsearch'
branch = 'master'
alwaysRecheck = "true"
alwaysPull = "true"
}
}
}

-Colin

On Tue, Nov 20, 2018 at 6:57 PM Nathan Dunn <[hidden email]> wrote:

Thanks. Will have a look.

On Nov 20, 2018, at 5:03 AM, Michael Poeschla <[hidden email]> wrote:

Hi,

here are the logs, docker-compose, actual dockerfile and start and testing of npm (below). I've also added the tomcat logs as an attachment.

docker-compose:
--
version: "3.2"
services:

apollo:
image: hub.age.mpg.de/mpoeschla/dv-kb:0.4
volumes:
- ./data/extdata/:/data
- postgres-data:/var/lib/postgresql
- ./config/tomcat/server.xml:/var/lib/tomcat8/conf/server.xml:ro
- ./config/tomcat/setenv.sh:/usr/share/tomcat8/bin/setenv.sh
environment:
APOLLO_ADMIN_[hidden email]
restart: always
links:
- elasticsearch:elasticsearch

nginx:
image: library/nginx:alpine
restart: always
volumes:
- ./config/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
- ./config/secrets:/secrets:ro
ports:
- 80:80
- 443:443
links:
- apollo:apollo

elasticsearch:
image: elasticsearch:6.4.2
container_name: elasticsearch
environment:
- cluster.name=docker-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- esdata1:/usr/share/elasticsearch/data

volumes:
postgres-data:
esdata1:
driver: local
--
The actual Dockerfile:
--
# WebApollo
# VERSION 2.1.X
FROM tomcat:8-jre8
MAINTAINER Eric Rasche <[hidden email]>, Anthony Bretaudeau <[hidden email]>, Nathan Dunn <[hidden email]>
ENV DEBIAN_FRONTEND noninteractive

RUN apt-get -qq update --fix-missing && \
apt-get --no-install-recommends -y install \
git build-essential maven tomcat8 libpq-dev postgresql-common openjdk-8-jdk wget \
postgresql postgresql-client xmlstarlet netcat libpng-dev \
zlib1g-dev libexpat1-dev ant curl ssl-cert zip unzip

RUN curl -sL https://deb.nodesource.com/setup_8.x | bash -
RUN apt-get -qq update --fix-missing && \
apt-get --no-install-recommends -y install nodejs && \
apt-get autoremove -y && apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

RUN cp /usr/lib/jvm/java-8-openjdk-amd64/lib/tools.jar /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext/tools.jar && \
useradd -ms /bin/bash -d /apollo apollo

ENV WEBAPOLLO_VERSION 99c7e54c6e74fbc6705d57de216c9b14b2bfb03b
RUN curl -L https://github.com/GMOD/Apollo/archive/${WEBAPOLLO_VERSION}.tar.gz | tar xzf - --strip-components=1 -C /apollo

# install grails
COPY build.sh /bin/build.sh
ADD apollo-config.groovy /apollo/apollo-config.groovy

RUN chown -R apollo:apollo /apollo
RUN curl -s "http://hgdownload.soe.ucsc.edu/admin/exe/linux.x86_64/blat/blat" -o /usr/local/bin/blat
RUN chmod +x /usr/local/bin/blat

USER apollo
RUN curl -s get.sdkman.io | bash
RUN /bin/bash -c "source $HOME/.sdkman/bin/sdkman-init.sh && yes | sdk install grails 2.5.5"
RUN /bin/bash -c "source $HOME/.sdkman/bin/sdkman-init.sh && yes | sdk install gradle 3.2.1"

RUN /bin/bash -c "source $HOME/.profile && source $HOME/.sdkman/bin/sdkman-init.sh && /bin/bash /bin/build.sh"

USER root
ENV CATALINA_HOME=/var/lib/tomcat8
RUN rm -rf ${CATALINA_HOME}/webapps/* && \
cp /apollo/apollo*.war ${CATALINA_HOME}/apollo.war

ENV CONTEXT_PATH ROOT

# Download chado schema
RUN wget --quiet https://github.com/erasche/chado-schema-builder/releases/download/1.31-jenkins97/chado-1.31.sql.gz -O /chado.sql.gz && \
gunzip /chado.sql.gz

RUN wget https://optinsoft.net/pic/ipgen2.png -O test.png

ADD launch.sh /launch.sh
CMD "/launch.sh"
--
Starting npm inside the container:
root@995955a711e1:/var/lib/tomcat8/webapps/ROOT/jbrowse/plugins/ElasticSearch# npm start

> jbrowse_elasticsearch@1.0.4 start /var/lib/tomcat8/webapps/ROOT/jbrowse/plugins/ElasticSearch
> node ./bin/www

testing access with curl:
root@995955a711e1:/usr/local/tomcat# curl http://localhost:3000
{"hits":[],"total":0}root@995955a711e1:/usr/local/tomcat#

content of ElasticSearchPlugin:
^Croot@995955a711e1:/var/lib/tomcat8/webapps/ROOT/jbrowse/plugins/ElasticSearch# ls -la
total 68
drwxr-x--- 10 tomcat8 tomcat8 271 Nov 14 17:10 .
drwxr-x--- 12 tomcat8 tomcat8 223 Nov 14 16:34 ..
-rw-r----- 1 tomcat8 tomcat8 594 Nov 14 16:16 .eslintrc
-rw-r----- 1 tomcat8 tomcat8 290 Nov 14 16:16 .travis.yml
-rw-r----- 1 tomcat8 tomcat8 686 Nov 14 16:16 ChangeLog.md
-rw-r----- 1 tomcat8 tomcat8 810 Nov 14 16:16 Makefile.PL
-rw-r----- 1 tomcat8 tomcat8 5157 Nov 14 16:16 README.md
-rw-r----- 1 tomcat8 tomcat8 1888 Nov 14 16:44 app.js
drwxr-x--- 2 tomcat8 tomcat8 66 Nov 14 16:34 bin
drwxr-x--- 2 tomcat8 tomcat8 22 Nov 14 16:34 css
drwxr-x--- 2 tomcat8 tomcat8 22 Nov 14 16:34 docs
drwxr-x--- 2 tomcat8 tomcat8 45 Nov 14 16:34 img
drwxr-x--- 3 tomcat8 tomcat8 65 Nov 14 16:34 js
drwxr-xr-x 24 root root 4096 Nov 14 17:10 node_modules
-rw-r----- 1 tomcat8 tomcat8 25820 Nov 14 17:10 package-lock.json
-rw-r----- 1 tomcat8 tomcat8 725 Nov 14 16:16 package.json
-rw-r----- 1 tomcat8 tomcat8 727 Nov 14 16:16 setup.sh
drwxr-x--- 3 tomcat8 tomcat8 19 Nov 14 16:34 src
drwxr-x--- 3 tomcat8 tomcat8 33 Nov 14 16:34 test

..And the logs are attached.
Thanks!
Michael

From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Saturday, November 17, 2018 4:06:58 AM
Subject: Re: [apollo] elastic search in genome apollo

Sorry for the late response.
Can you provide the browser console logs and any tomcat logs (catalina.out)?
Your modified docket file may be helpful as well.

Nathan

On Nov 15, 2018, at 10:26 AM, Michael Poeschla <[hidden email]> wrote:

yes - we don't see anything in the browser
Thanks,
Michael
From: "Colin" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 7:09:45 PM
Subject: Re: [apollo] elastic search in genome apollo

So you are saying that the plugin is not being added e.g. you don't have a menu item for the ElasticSearch dialog in jbrowse?
-Colin

On Thu, Nov 15, 2018 at 9:07 AM Michael Poeschla <[hidden email]> wrote:
hi,
elasticsearch was installed with docker. Here's the tracklist.json for the organism that we generated the index for. Is there any other info I could provide that would help?

{
   "elasticIndexName" : "nfz",
   "elasticSearchUrl" : "localhost:3000",
   "formatVersion" : 1,
   "names" : {
      "type" : "Hash",
      "url" : "names/"
   },
   "plugins" : [
      "ElasticSearch"
   ],
   "tracks" : [
      {
         "category" : "Reference sequence",
         "chunkSize" : 20000,
         "key" : "Reference sequence",
         "label" : "DNA",
         "seqType" : "dna",
         "storeClass" : "JBrowse/Store/Sequence/StaticChunked",
         "type" : "SequenceTrack",
         "urlTemplate" : "seq/{refseq_dirpath}/{refseq}-"
      },
      {
         "bicolor_pivot" : 0.5,
         "label" : "GC Content",
         "storeClass" : "JBrowse/Store/SeqFeature/SequenceChunks",
         "type" : "GCContent/View/Track/GCContentXY",
         "urlTemplate" : "seq/{refseq_dirpath}/{refseq}-"
      },
      {
         "compress" : 0,
         "key" : "Repeats",
         "label" : "Repeats",
         "storeClass" : "JBrowse/Store/SeqFeature/NCList",
         "style" : {
            "className" : "feature"
         },
         "trackType" : "JBrowse/View/Track/CanvasFeatures",
         "type" : "JBrowse/View/Track/CanvasFeatures",
         "urlTemplate" : "tracks/Repeats/{refseq}/trackData.json"
      },
      {
         "compress" : 0,
         "key" : "Annotation",
         "label" : "Annotationfromsumgff",
         "storeClass" : "JBrowse/Store/SeqFeature/NCList",
         "style" : {
            "className" : "feature"
         },
         "trackType" : "CanvasFeatures",
         "type" : "CanvasFeatures",
         "urlTemplate" : "tracks/Annotationfromsumgff/{refseq}/trackData.json"
      }
   ]
}

From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 11:33:03 AM
Subject: Re: [apollo] elastic search in genome apollo

Thanks. Also on Europe time for another 20 minutes.
Can you also post any other specifics of failure?

Have you installed elasticsearch via docker or manually ?

Nathan

On Nov 15, 2018, at 11:10 AM, Michael Poeschla <[hidden email]> wrote:

Here it is - sorry for the late replies, I'm on europe time.

environments {
    development {
    }
    test {
    }
    production {
        dataSource {
            dbCreate = "update"
            username = System.getenv("WEBAPOLLO_DB_USERNAME") ?: *
            password = System.getenv("WEBAPOLLO_DB_PASSWORD") ?: *

            driverClassName = "org.postgresql.Driver"
            dialect = "org.hibernate.dialect.PostgresPlusDialect"
            url = "jdbc:postgresql://${System.getenv("WEBAPOLLO_DB_HOST")?:"127.0.0.1"}/${System.getenv("WEBAPOLLO_DB_NAME")?:"apollo"}"

            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = false
                initialSize                   = 5
                maxActive                     = 50
                minIdle                       = 5
                maxIdle                       = 25
                maxWait                       = 10000
                maxAge                        = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis    = 60000
                validationQuery               = "SELECT 1"
                validationQueryTimeout        = 3
                validationInterval            = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
        dataSource_chado {
            dbCreate = "update"
            username = System.getenv("CHADO_DB_USERNAME") ?: *
            password = System.getenv("CHADO_DB_PASSWORD") ?: *

            driverClassName = "org.postgresql.Driver"
            dialect = "org.hibernate.dialect.PostgresPlusDialect"

            url = "jdbc:postgresql://${System.getenv("CHADO_DB_HOST")?:"127.0.0.1"}/${System.getenv("CHADO_DB_NAME")?:"chado"}"

            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = false
                initialSize                   = 5
                maxActive                     = 50
                minIdle                       = 5
                maxIdle                       = 25
                maxWait                       = 10000
                maxAge                        = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis    = 60000
                validationQuery               = "SELECT 1"
                validationQueryTimeout        = 3
                validationInterval            = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
    }
}

apollo {
    default_minimum_intron_size = System.getenv("WEBAPOLLO_MINIMUM_INTRON_SIZE") ? System.getenv("WEBAPOLLO_MINIMUM_INTRON_SIZE").toInteger() : 1
    history_size = System.getenv("WEBAPOLLO_HISTORY_SIZE") ? System.getenv("WEBAPOLLO_HISTORY_SIZE").toInteger() : 0
    overlapper_class = System.getenv("WEBAPOLLO_OVERLAPPER_CLASS") ?: "org.bbop.apollo.sequence.OrfOverlapper"
    use_cds_for_new_transcripts = System.getenv("WEBAPOLLO_CDS_FOR_NEW_TRANSCRIPTS").equals("true") // will default to false
    feature_has_dbxrefs = System.getenv("WEBAPOLLO_FEATURE_HAS_DBXREFS") ?: true
    feature_has_attributes = System.getenv("WEBAPOLLO_FEATURE_HAS_ATTRS") ?: true
    feature_has_pubmed_ids = System.getenv("WEBAPOLLO_FEATURE_HAS_PUBMED") ?: true
    feature_has_go_ids = System.getenv("WEBAPOLLO_FEATURE_HAS_GO") ?: true
    feature_has_comments = System.getenv("WEBAPOLLO_FEATURE_HAS_COMMENTS") ?: true
    feature_has_status = System.getenv("WEBAPOLLO_FEATURE_HAS_STATUS") ?: true
    translation_table = "/config/translation_tables/ncbi_" + (System.getenv("WEBAPOLLO_TRANSLATION_TABLE") ?: "1") + "_translation_table.txt"
    get_translation_code = System.getenv("WEBAPOLLO_TRANSLATION_TABLE") ? System.getenv("WEBAPOLLO_TRANSLATION_TABLE").toInteger() : 1

    // ADDED BLAT AND SET PATH TO BIN
    sequence_search_tools = [
        blat_nuc: [
          search_exe: "/usr/local/bin/blat",
          search_class: "org.bbop.apollo.sequence.search.blat.BlatCommandLineNucleotideToNucleotide",
          name: "Blat nucleotide",
          params: ""
        ],
        blat_prot: [
          search_exe: "/usr/local/bin/blat",
          search_class: "org.bbop.apollo.sequence.search.blat.BlatCommandLineProteinToNucleotide",
          name: "Blat protein",
          params: ""
        ]
      ]
    // TODO: should come from config or via preferences database
    splice_donor_sites = System.getenv("WEBAPOLLO_SPLICE_DONOR_SITES") ? System.getenv("WEBAPOLLO_SPLICE_DONOR_SITES").split(",") : ["GT"]
    splice_acceptor_sites = System.getenv("WEBAPOLLO_SPLICE_ACCEPTOR_SITES") ? System.getenv("WEBAPOLLO_SPLICE_ACCEPTOR_SITES").split(",") : ["AG"]
    gff3.source = System.getenv("WEBAPOLLO_GFF3_SOURCE") ?: "."

    google_analytics = System.getenv("WEBAPOLLO_GOOGLE_ANALYTICS_ID") ?: ["UA-62921593-1"]

    admin{
        username = System.getenv("APOLLO_ADMIN_EMAIL") ?: *
        password = System.getenv("APOLLO_ADMIN_PASSWORD") ?: *
        firstName = System.getenv("APOLLO_ADMIN_FIRST_NAME") ?: *
        lastName = System.getenv("APOLLO_ADMIN_LAST_NAME") ?: *
    }
}

jbrowse {
    git {
        url = "https://github.com/GMOD/jbrowse"
        tag = "maint/1.12.5-apollo"
    }
    plugins {
        WebApollo{
            included = true
        }
        NeatHTMLFeatures{
            included = System.getenv("WEBAPOLLO_JBROWSE_PLUGIN_NEATHTML") ?: true
        }
        NeatCanvasFeatures{
            included = System.getenv("WEBAPOLLO_JBROWSE_PLUGIN_NEATCANVAS") ?: true
        }
        RegexSequenceSearch{
            included = true
        }
        HideTrackLabels{
            included = true
        }
        //EDIT: ADDED GC CONTENT
        GCContent{
            git = 'https://github.com/cmdcolin/GCContent'
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
        //EDIT: ADDS ELASTIC SEARCH PLUGIN
        ElasticSearch{
            git = 'https://github.com/elsiklab/jbrowse_elasticsearch'
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
    }
}
From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 12:30:50 AM
Subject: Re: [apollo] elastic search in genome apollo

Can you post your apollo config file?

Nathan

On Nov 14, 2018, at 6:45 PM, Michael Poeschla <[hidden email]> wrote:

Hi,
I'm following the instructions from https://github.com/elsiklab/jbrowse_elasticsearch

we have elasticsearch and the middleware app.js running. I was able to execute generate-names.pl and this seemed to work correctly from the terminal output. The changes also got picked up in the tracklist.json for the genome. But there is no sign of the elasticsearch plugin in web apollo...added the plugin to the config.groovy, and we have another jbrowse plugin (GC content) working fine so not sure what's the problem here.

Thanks for all of your help! :)
Michael
From: "Nathan Dunn" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Tuesday, November 13, 2018 6:33:05 PM
Subject: Re: [apollo] elastic search in genome apollo

Michael,

What problems are you running it when adding the plugins?

Nathan

On Nov 13, 2018, at 4:55 PM, Colin <[hidden email]> wrote:

Aside from changing your gene identifiers, I think the full text search from elasticsearch could help. I forwarded some comments here about that https://github.com/elsiklab/jbrowse_elasticsearch/issues/7

Potentially data that is loaded in Apollo backend could use text similarity %like% or full text sql queries but for normal jbrowse tracks that are not loaded in the sql database, the jbrowse_elasticsearch plugin seems reasonable.

-Colin

On Tue, Nov 13, 2018 at 7:35 AM Michael Poeschla <[hidden email]> wrote:
Hi,

has anyone implemented the jbrowse_elasticsearch plugin in Apollo? I'm running from the docker container with an nginx reverse proxy and having trouble adding this plugin. How would I add this to the docker image, with the middleware as well?

The reason we want to do this is the predicted gene model names we have in our tracks are not always intuitive so it's hard to find them by searching in Apollo (ie mTOR is ID'd as "serine/threonine-protein kinase mTOR", so if you just start typing mTOR you get no results). Wondering also if anyone else has encountered this problem and found good solutions, besides manipulating all of our tracks.

Thanks so much,
Michael

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.
<tomcat.tar.gz>

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

<VirtualBox_Ubuntu5_21_11_2018_13_06_07.png>

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

nathandunn

Re: elastic search in genome apollo

Thanks for reporting the workaround for this.

In the future I’ll try to make sure that this doesn’t kill it so you can hopefully re-use that directory as a Browse directory.

https://github.com/GMOD/Apollo/issues/2014

Cheers,

Nathan

On Jan 17, 2019, at 10:19 AM, Michael Poeschla <[hidden email]> wrote:

the reason the plugin didn't show up turned out to be because i was editing both the apollo-config.groovy and the tracklist.json as you would for jbrowse. When I removed that line from the tracklist.json, the elasticsearch popped up immediately. This was a while back, just remembered to add it here in case anyone else runs into this kind of issue.
Best,
Michael
From: "Nathan Dunn" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Wednesday, November 21, 2018 9:05:13 PM
Subject: Re: [apollo] elastic search in genome apollo

Thanks for testing Colin. I wasn’t sure the exact plugin to test with.

I did get it to deploy off of master. I did an ./apollo clean-all

I did an ./apollo deploy first (not sure if that made any difference, but it should not have).

My config is the sample-h2-config.groovy file with the snippet that Colin provided below.

That being said, I didn’t do a full test with the elastic search plugin, but this seems to install it correctly and I get an organism window, etc. when I load the volvox data.

Nathan

// default username/password on h2 are given here. see docs for adjusting
dataSource {
pooled = true
jmxExport = true
driverClassName = "org.h2.Driver"
username = "sa"
password = ""
}
// environment specific settings
environments {
development {
// sample config to turn on debug logging in development e.g. for apollo run-local
log4j.main = {
debug "grails.app"
}
// sample config to edit apollo specific configs in development mode
apollo {
gff3.source = "testing"
}
dataSource {
// NOTE: this is in memory, so it will be deleted.
dbCreate = "update" // one of 'create', 'create-drop', 'update', 'validate', ''
url = "jdbc:h2:AnnotationDatabase:devDb;MVCC=TRUE;LOCK_TIMEOUT=10000;DB_CLOSE_ON_EXIT=FALSE"
}
}
test {
dataSource {
dbCreate = "create-drop" // one of 'create', 'create-drop', 'update', 'validate', ''
url = "jdbc:h2:mem:testDb;MVCC=TRUE;LOCK_TIMEOUT=10000;DB_CLOSE_ON_EXIT=FALSE"
}
}

//note: not necessarily recommended to use h2 in production mode. see precautions
production {
dataSource {
dbCreate = "update"
//NOTE: production mode uses file instead of mem database
//NOTE: Please specify the appropriate file path, otherwise /tmp/prodDb will be used.
url = "jdbc:h2:/opt/apollo/h2/prodDb;MVCC=TRUE;LOCK_TIMEOUT=10000;DB_CLOSE_ON_EXIT=FALSE"
properties {
// See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
jmxEnabled = true
initialSize = 5
maxActive = 50
minIdle = 5
maxIdle = 25
maxWait = 10000
maxAge = 10 * 60000
timeBetweenEvictionRunsMillis = 5000
minEvictableIdleTimeMillis = 60000
validationQuery = "SELECT 1"
validationQueryTimeout = 3
validationInterval = 15000
testOnBorrow = true
testWhileIdle = true
testOnReturn = false
jdbcInterceptors = "ConnectionState"
defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
}
}
}
}

// Uncomment to change the default memory configurations
//grails.project.fork = [
// test : false,
// // configure settings for the run-app JVM
// run : [maxMemory: 2048, minMemory: 64, debug: false, maxPerm: 1024, forkReserve: false],
// // configure settings for the run-war JVM
// war : [maxMemory: 2048, minMemory: 64, debug: false, maxPerm: 1024, forkReserve: false],
// // configure settings for the Console UI JVM
// console: [maxMemory: 2048, minMemory: 64, debug: false, maxPerm: 1024]
//]

// Uncomment to make changes
//
jbrowse {
git {
url= "https://github.com/GMOD/jbrowse"
tag = "maint/1.12.5-apollo"
alwaysPull = true
alwaysRecheck = true
}
plugins {
NeatHTMLFeatures{
included = true
}
NeatCanvasFeatures{
included = true
}
RegexSequenceSearch{
included = true
}
HideTrackLabels{
included = true
}
ElasticSearch {
git = 'https://github.com/elsiklab/jbrowse_elasticsearch'
branch = 'master'
alwaysRecheck = "true"
alwaysPull = "true"
}
// MyVariantInfo {
// git = 'https://github.com/GMOD/myvariantviewer'
// branch = 'master'
// alwaysRecheck = "true"
// alwaysPull = "true"
// }
// SashimiPlot {
// git = 'https://github.com/cmdcolin/sashimiplot'
// branch = 'master'
// alwaysPull = "true"
// }
}
}

On Nov 21, 2018, at 10:09 AM, Colin <[hidden email]> wrote:

I went and tested this and I tried testing adding the plugin to apollo-config.groovy and was getting an error about plugins/ElasticSearch/js/main.js not being loaded apollo run-local. I can see that it gets downloaded to jbrowse-download/plugins/ElasticSearch but it was giving me that error there still. I also ran apollo deploy and then unzipped and the jbrowse plugins folder did not contain ElasticSearch plugin

Screenshot attached and apollo-config.groovy snippet here

jbrowse {
plugins {
ElasticSearch {
git = 'https://github.com/elsiklab/jbrowse_elasticsearch'
branch = 'master'
alwaysRecheck = "true"
alwaysPull = "true"
}
}
}

-Colin

On Tue, Nov 20, 2018 at 6:57 PM Nathan Dunn <[hidden email]> wrote:

Thanks. Will have a look.

On Nov 20, 2018, at 5:03 AM, Michael Poeschla <[hidden email]> wrote:

Hi,

here are the logs, docker-compose, actual dockerfile and start and testing of npm (below). I've also added the tomcat logs as an attachment.

docker-compose:
--
version: "3.2"
services:

apollo:
image: hub.age.mpg.de/mpoeschla/dv-kb:0.4
volumes:
- ./data/extdata/:/data
- postgres-data:/var/lib/postgresql
- ./config/tomcat/server.xml:/var/lib/tomcat8/conf/server.xml:ro
- ./config/tomcat/setenv.sh:/usr/share/tomcat8/bin/setenv.sh
environment:
APOLLO_ADMIN_[hidden email]
restart: always
links:
- elasticsearch:elasticsearch

nginx:
image: library/nginx:alpine
restart: always
volumes:
- ./config/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
- ./config/secrets:/secrets:ro
ports:
- 80:80
- 443:443
links:
- apollo:apollo

elasticsearch:
image: elasticsearch:6.4.2
container_name: elasticsearch
environment:
- cluster.name=docker-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- esdata1:/usr/share/elasticsearch/data

volumes:
postgres-data:
esdata1:
driver: local
--
The actual Dockerfile:
--
# WebApollo
# VERSION 2.1.X
FROM tomcat:8-jre8
MAINTAINER Eric Rasche <[hidden email]>, Anthony Bretaudeau <[hidden email]>, Nathan Dunn <[hidden email]>
ENV DEBIAN_FRONTEND noninteractive

RUN apt-get -qq update --fix-missing && \
apt-get --no-install-recommends -y install \
git build-essential maven tomcat8 libpq-dev postgresql-common openjdk-8-jdk wget \
postgresql postgresql-client xmlstarlet netcat libpng-dev \
zlib1g-dev libexpat1-dev ant curl ssl-cert zip unzip

RUN curl -sL https://deb.nodesource.com/setup_8.x | bash -
RUN apt-get -qq update --fix-missing && \
apt-get --no-install-recommends -y install nodejs && \
apt-get autoremove -y && apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

RUN cp /usr/lib/jvm/java-8-openjdk-amd64/lib/tools.jar /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext/tools.jar && \
useradd -ms /bin/bash -d /apollo apollo

ENV WEBAPOLLO_VERSION 99c7e54c6e74fbc6705d57de216c9b14b2bfb03b
RUN curl -L https://github.com/GMOD/Apollo/archive/${WEBAPOLLO_VERSION}.tar.gz | tar xzf - --strip-components=1 -C /apollo

# install grails
COPY build.sh /bin/build.sh
ADD apollo-config.groovy /apollo/apollo-config.groovy

RUN chown -R apollo:apollo /apollo
RUN curl -s "http://hgdownload.soe.ucsc.edu/admin/exe/linux.x86_64/blat/blat" -o /usr/local/bin/blat
RUN chmod +x /usr/local/bin/blat

USER apollo
RUN curl -s get.sdkman.io | bash
RUN /bin/bash -c "source $HOME/.sdkman/bin/sdkman-init.sh && yes | sdk install grails 2.5.5"
RUN /bin/bash -c "source $HOME/.sdkman/bin/sdkman-init.sh && yes | sdk install gradle 3.2.1"

RUN /bin/bash -c "source $HOME/.profile && source $HOME/.sdkman/bin/sdkman-init.sh && /bin/bash /bin/build.sh"

USER root
ENV CATALINA_HOME=/var/lib/tomcat8
RUN rm -rf ${CATALINA_HOME}/webapps/* && \
cp /apollo/apollo*.war ${CATALINA_HOME}/apollo.war

ENV CONTEXT_PATH ROOT

# Download chado schema
RUN wget --quiet https://github.com/erasche/chado-schema-builder/releases/download/1.31-jenkins97/chado-1.31.sql.gz -O /chado.sql.gz && \
gunzip /chado.sql.gz

RUN wget https://optinsoft.net/pic/ipgen2.png -O test.png

ADD launch.sh /launch.sh
CMD "/launch.sh"
--
Starting npm inside the container:
root@995955a711e1:/var/lib/tomcat8/webapps/ROOT/jbrowse/plugins/ElasticSearch# npm start

> jbrowse_elasticsearch@1.0.4 start /var/lib/tomcat8/webapps/ROOT/jbrowse/plugins/ElasticSearch
> node ./bin/www

testing access with curl:
root@995955a711e1:/usr/local/tomcat# curl http://localhost:3000
{"hits":[],"total":0}root@995955a711e1:/usr/local/tomcat#

content of ElasticSearchPlugin:
^Croot@995955a711e1:/var/lib/tomcat8/webapps/ROOT/jbrowse/plugins/ElasticSearch# ls -la
total 68
drwxr-x--- 10 tomcat8 tomcat8 271 Nov 14 17:10 .
drwxr-x--- 12 tomcat8 tomcat8 223 Nov 14 16:34 ..
-rw-r----- 1 tomcat8 tomcat8 594 Nov 14 16:16 .eslintrc
-rw-r----- 1 tomcat8 tomcat8 290 Nov 14 16:16 .travis.yml
-rw-r----- 1 tomcat8 tomcat8 686 Nov 14 16:16 ChangeLog.md
-rw-r----- 1 tomcat8 tomcat8 810 Nov 14 16:16 Makefile.PL
-rw-r----- 1 tomcat8 tomcat8 5157 Nov 14 16:16 README.md
-rw-r----- 1 tomcat8 tomcat8 1888 Nov 14 16:44 app.js
drwxr-x--- 2 tomcat8 tomcat8 66 Nov 14 16:34 bin
drwxr-x--- 2 tomcat8 tomcat8 22 Nov 14 16:34 css
drwxr-x--- 2 tomcat8 tomcat8 22 Nov 14 16:34 docs
drwxr-x--- 2 tomcat8 tomcat8 45 Nov 14 16:34 img
drwxr-x--- 3 tomcat8 tomcat8 65 Nov 14 16:34 js
drwxr-xr-x 24 root root 4096 Nov 14 17:10 node_modules
-rw-r----- 1 tomcat8 tomcat8 25820 Nov 14 17:10 package-lock.json
-rw-r----- 1 tomcat8 tomcat8 725 Nov 14 16:16 package.json
-rw-r----- 1 tomcat8 tomcat8 727 Nov 14 16:16 setup.sh
drwxr-x--- 3 tomcat8 tomcat8 19 Nov 14 16:34 src
drwxr-x--- 3 tomcat8 tomcat8 33 Nov 14 16:34 test

..And the logs are attached.
Thanks!
Michael

From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Saturday, November 17, 2018 4:06:58 AM
Subject: Re: [apollo] elastic search in genome apollo

Sorry for the late response.
Can you provide the browser console logs and any tomcat logs (catalina.out)?
Your modified docket file may be helpful as well.

Nathan

On Nov 15, 2018, at 10:26 AM, Michael Poeschla <[hidden email]> wrote:

yes - we don't see anything in the browser
Thanks,
Michael
From: "Colin" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 7:09:45 PM
Subject: Re: [apollo] elastic search in genome apollo

So you are saying that the plugin is not being added e.g. you don't have a menu item for the ElasticSearch dialog in jbrowse?
-Colin

On Thu, Nov 15, 2018 at 9:07 AM Michael Poeschla <[hidden email]> wrote:
hi,
elasticsearch was installed with docker. Here's the tracklist.json for the organism that we generated the index for. Is there any other info I could provide that would help?

{
   "elasticIndexName" : "nfz",
   "elasticSearchUrl" : "localhost:3000",
   "formatVersion" : 1,
   "names" : {
      "type" : "Hash",
      "url" : "names/"
   },
   "plugins" : [
      "ElasticSearch"
   ],
   "tracks" : [
      {
         "category" : "Reference sequence",
         "chunkSize" : 20000,
         "key" : "Reference sequence",
         "label" : "DNA",
         "seqType" : "dna",
         "storeClass" : "JBrowse/Store/Sequence/StaticChunked",
         "type" : "SequenceTrack",
         "urlTemplate" : "seq/{refseq_dirpath}/{refseq}-"
      },
      {
         "bicolor_pivot" : 0.5,
         "label" : "GC Content",
         "storeClass" : "JBrowse/Store/SeqFeature/SequenceChunks",
         "type" : "GCContent/View/Track/GCContentXY",
         "urlTemplate" : "seq/{refseq_dirpath}/{refseq}-"
      },
      {
         "compress" : 0,
         "key" : "Repeats",
         "label" : "Repeats",
         "storeClass" : "JBrowse/Store/SeqFeature/NCList",
         "style" : {
            "className" : "feature"
         },
         "trackType" : "JBrowse/View/Track/CanvasFeatures",
         "type" : "JBrowse/View/Track/CanvasFeatures",
         "urlTemplate" : "tracks/Repeats/{refseq}/trackData.json"
      },
      {
         "compress" : 0,
         "key" : "Annotation",
         "label" : "Annotationfromsumgff",
         "storeClass" : "JBrowse/Store/SeqFeature/NCList",
         "style" : {
            "className" : "feature"
         },
         "trackType" : "CanvasFeatures",
         "type" : "CanvasFeatures",
         "urlTemplate" : "tracks/Annotationfromsumgff/{refseq}/trackData.json"
      }
   ]
}

From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 11:33:03 AM
Subject: Re: [apollo] elastic search in genome apollo

Thanks. Also on Europe time for another 20 minutes.
Can you also post any other specifics of failure?

Have you installed elasticsearch via docker or manually ?

Nathan

On Nov 15, 2018, at 11:10 AM, Michael Poeschla <[hidden email]> wrote:

Here it is - sorry for the late replies, I'm on europe time.

environments {
    development {
    }
    test {
    }
    production {
        dataSource {
            dbCreate = "update"
            username = System.getenv("WEBAPOLLO_DB_USERNAME") ?: *
            password = System.getenv("WEBAPOLLO_DB_PASSWORD") ?: *

            driverClassName = "org.postgresql.Driver"
            dialect = "org.hibernate.dialect.PostgresPlusDialect"
            url = "jdbc:postgresql://${System.getenv("WEBAPOLLO_DB_HOST")?:"127.0.0.1"}/${System.getenv("WEBAPOLLO_DB_NAME")?:"apollo"}"

            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = false
                initialSize                   = 5
                maxActive                     = 50
                minIdle                       = 5
                maxIdle                       = 25
                maxWait                       = 10000
                maxAge                        = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis    = 60000
                validationQuery               = "SELECT 1"
                validationQueryTimeout        = 3
                validationInterval            = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
        dataSource_chado {
            dbCreate = "update"
            username = System.getenv("CHADO_DB_USERNAME") ?: *
            password = System.getenv("CHADO_DB_PASSWORD") ?: *

            driverClassName = "org.postgresql.Driver"
            dialect = "org.hibernate.dialect.PostgresPlusDialect"

            url = "jdbc:postgresql://${System.getenv("CHADO_DB_HOST")?:"127.0.0.1"}/${System.getenv("CHADO_DB_NAME")?:"chado"}"

            properties {
                // See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
                jmxEnabled = false
                initialSize                   = 5
                maxActive                     = 50
                minIdle                       = 5
                maxIdle                       = 25
                maxWait                       = 10000
                maxAge                        = 10 * 60000
                timeBetweenEvictionRunsMillis = 5000
                minEvictableIdleTimeMillis    = 60000
                validationQuery               = "SELECT 1"
                validationQueryTimeout        = 3
                validationInterval            = 15000
                testOnBorrow = true
                testWhileIdle = true
                testOnReturn = false
                jdbcInterceptors = "ConnectionState"
                defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
            }
        }
    }
}

apollo {
    default_minimum_intron_size = System.getenv("WEBAPOLLO_MINIMUM_INTRON_SIZE") ? System.getenv("WEBAPOLLO_MINIMUM_INTRON_SIZE").toInteger() : 1
    history_size = System.getenv("WEBAPOLLO_HISTORY_SIZE") ? System.getenv("WEBAPOLLO_HISTORY_SIZE").toInteger() : 0
    overlapper_class = System.getenv("WEBAPOLLO_OVERLAPPER_CLASS") ?: "org.bbop.apollo.sequence.OrfOverlapper"
    use_cds_for_new_transcripts = System.getenv("WEBAPOLLO_CDS_FOR_NEW_TRANSCRIPTS").equals("true") // will default to false
    feature_has_dbxrefs = System.getenv("WEBAPOLLO_FEATURE_HAS_DBXREFS") ?: true
    feature_has_attributes = System.getenv("WEBAPOLLO_FEATURE_HAS_ATTRS") ?: true
    feature_has_pubmed_ids = System.getenv("WEBAPOLLO_FEATURE_HAS_PUBMED") ?: true
    feature_has_go_ids = System.getenv("WEBAPOLLO_FEATURE_HAS_GO") ?: true
    feature_has_comments = System.getenv("WEBAPOLLO_FEATURE_HAS_COMMENTS") ?: true
    feature_has_status = System.getenv("WEBAPOLLO_FEATURE_HAS_STATUS") ?: true
    translation_table = "/config/translation_tables/ncbi_" + (System.getenv("WEBAPOLLO_TRANSLATION_TABLE") ?: "1") + "_translation_table.txt"
    get_translation_code = System.getenv("WEBAPOLLO_TRANSLATION_TABLE") ? System.getenv("WEBAPOLLO_TRANSLATION_TABLE").toInteger() : 1

    // ADDED BLAT AND SET PATH TO BIN
    sequence_search_tools = [
        blat_nuc: [
          search_exe: "/usr/local/bin/blat",
          search_class: "org.bbop.apollo.sequence.search.blat.BlatCommandLineNucleotideToNucleotide",
          name: "Blat nucleotide",
          params: ""
        ],
        blat_prot: [
          search_exe: "/usr/local/bin/blat",
          search_class: "org.bbop.apollo.sequence.search.blat.BlatCommandLineProteinToNucleotide",
          name: "Blat protein",
          params: ""
        ]
      ]
    // TODO: should come from config or via preferences database
    splice_donor_sites = System.getenv("WEBAPOLLO_SPLICE_DONOR_SITES") ? System.getenv("WEBAPOLLO_SPLICE_DONOR_SITES").split(",") : ["GT"]
    splice_acceptor_sites = System.getenv("WEBAPOLLO_SPLICE_ACCEPTOR_SITES") ? System.getenv("WEBAPOLLO_SPLICE_ACCEPTOR_SITES").split(",") : ["AG"]
    gff3.source = System.getenv("WEBAPOLLO_GFF3_SOURCE") ?: "."

    google_analytics = System.getenv("WEBAPOLLO_GOOGLE_ANALYTICS_ID") ?: ["UA-62921593-1"]

    admin{
        username = System.getenv("APOLLO_ADMIN_EMAIL") ?: *
        password = System.getenv("APOLLO_ADMIN_PASSWORD") ?: *
        firstName = System.getenv("APOLLO_ADMIN_FIRST_NAME") ?: *
        lastName = System.getenv("APOLLO_ADMIN_LAST_NAME") ?: *
    }
}

jbrowse {
    git {
        url = "https://github.com/GMOD/jbrowse"
        tag = "maint/1.12.5-apollo"
    }
    plugins {
        WebApollo{
            included = true
        }
        NeatHTMLFeatures{
            included = System.getenv("WEBAPOLLO_JBROWSE_PLUGIN_NEATHTML") ?: true
        }
        NeatCanvasFeatures{
            included = System.getenv("WEBAPOLLO_JBROWSE_PLUGIN_NEATCANVAS") ?: true
        }
        RegexSequenceSearch{
            included = true
        }
        HideTrackLabels{
            included = true
        }
        //EDIT: ADDED GC CONTENT
        GCContent{
            git = 'https://github.com/cmdcolin/GCContent'
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
        //EDIT: ADDS ELASTIC SEARCH PLUGIN
        ElasticSearch{
            git = 'https://github.com/elsiklab/jbrowse_elasticsearch'
            branch = 'master'
            alwaysRecheck = "true"
            alwaysPull = "true"
        }
    }
}
From: "Lbl" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Thursday, November 15, 2018 12:30:50 AM
Subject: Re: [apollo] elastic search in genome apollo

Can you post your apollo config file?

Nathan

On Nov 14, 2018, at 6:45 PM, Michael Poeschla <[hidden email]> wrote:

Hi,
I'm following the instructions from https://github.com/elsiklab/jbrowse_elasticsearch

we have elasticsearch and the middleware app.js running. I was able to execute generate-names.pl and this seemed to work correctly from the terminal output. The changes also got picked up in the tracklist.json for the genome. But there is no sign of the elasticsearch plugin in web apollo...added the plugin to the config.groovy, and we have another jbrowse plugin (GC content) working fine so not sure what's the problem here.

Thanks for all of your help! :)
Michael
From: "Nathan Dunn" <[hidden email]>
To: "apollo" <[hidden email]>
Sent: Tuesday, November 13, 2018 6:33:05 PM
Subject: Re: [apollo] elastic search in genome apollo

Michael,

What problems are you running it when adding the plugins?

Nathan

On Nov 13, 2018, at 4:55 PM, Colin <[hidden email]> wrote:

Aside from changing your gene identifiers, I think the full text search from elasticsearch could help. I forwarded some comments here about that https://github.com/elsiklab/jbrowse_elasticsearch/issues/7

Potentially data that is loaded in Apollo backend could use text similarity %like% or full text sql queries but for normal jbrowse tracks that are not loaded in the sql database, the jbrowse_elasticsearch plugin seems reasonable.

-Colin

On Tue, Nov 13, 2018 at 7:35 AM Michael Poeschla <[hidden email]> wrote:
Hi,

has anyone implemented the jbrowse_elasticsearch plugin in Apollo? I'm running from the docker container with an nginx reverse proxy and having trouble adding this plugin. How would I add this to the docker image, with the middleware as well?

The reason we want to do this is the predicted gene model names we have in our tracks are not always intuitive so it's hard to find them by searching in Apollo (ie mTOR is ID'd as "serine/threonine-protein kinase mTOR", so if you just start typing mTOR you get no results). Wondering also if anyone else has encountered this problem and found good solutions, besides manipulating all of our tracks.

Thanks so much,
Michael

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.
<tomcat.tar.gz>

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

<VirtualBox_Ubuntu5_21_11_2018_13_06_07.png>

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.