Webapollo user annotation export problems with GMOD in the cloud

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Webapollo user annotation export problems with GMOD in the cloud

Bérénice Benayoun
Hi all,

I have set up a GMOD in the Cloud instance to perform some manual curation of genes in a genome project I am involved in. 

It took a while, but things are mostly working now (I can create new annotations, etc.), even if they are crazy slow (even with the machine having 30Gb of RAM).

However, it is now impossible to download any of the user created annotation - every time I attempt the gff3 export, I get an "Unable to export data" message, and the fixes for that proposed in the forums (manually creating a gff3 folder with tomcat7 as owner, etc) have not helped my case.


Any help on the matter would be greatly appreciated!

Thank you,

Berenice.


--
Bérénice A. BENAYOUN, Ph.D.
Stanford University/Genetics Department
BRUNET Laboratory, 'Molecular Basis of Longevity and Age Related Diseases'
M312 Alway Building
300, Pasteur Drive
MC 5120
Stanford, CA 94305-5120

USA
Email: [hidden email]
Web: www.stanford.edu/group/brunet/




This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

Reply | Threaded
Open this post in threaded view
|

Re: Webapollo user annotation export problems with GMOD in the cloud

nathandunn

Berenice, 

You have to setup tomcat with the appropriate amount of memory.  

If I go here:  


you’ll see that you get an out of memory error.

If you look here, you have to set the memory explicitly:


1 GB should be plenty, but you have a lot to play with. 

To confirm you have the appropriate memory when you restart tomcat do:

ps -ef | grep java 

You should see your memory options in that line. 

Let us know if you have further issues.

Nathan Dunn, PhD
Berkeley Bioinformatics Open-source Projects (BBOP)
Genomics Division, Lawrence Berkeley National Laboratory
[hidden email]


On Aug 24, 2015, at 9:50 AM, Bérénice Benayoun <[hidden email]> wrote:

Hi all,

I have set up a GMOD in the Cloud instance to perform some manual curation of genes in a genome project I am involved in. 

It took a while, but things are mostly working now (I can create new annotations, etc.), even if they are crazy slow (even with the machine having 30Gb of RAM).

However, it is now impossible to download any of the user created annotation - every time I attempt the gff3 export, I get an "Unable to export data" message, and the fixes for that proposed in the forums (manually creating a gff3 folder with tomcat7 as owner, etc) have not helped my case.


Any help on the matter would be greatly appreciated!

Thank you,

Berenice.


--
Bérénice A. BENAYOUN, Ph.D.
Stanford University/Genetics Department
BRUNET Laboratory, 'Molecular Basis of Longevity and Age Related Diseases'
M312 Alway Building
300, Pasteur Drive
MC 5120
Stanford, CA 94305-5120

USA
Email: [hidden email]
Web: www.stanford.edu/group/brunet/



This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.






This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

Reply | Threaded
Open this post in threaded view
|

Re: Webapollo user annotation export problems with GMOD in the cloud

Bérénice Benayoun
Hi Nathan,

thanks for the quick answer!

When I run that comamnd,  I get this unhelpful message:
ubuntu@ip-10-191-141-46:~$ ps -ef | grep java 
tomcat7  19284     1 23 16:51 ?        00:05:20 /usr/lib/jvm/default-java/bin/java -Djava.util.logging.config.file=/var/lib/tomcat7/conf/logging.properties -Djava.awt.headless=true -Xmx128m -XX:+UseConcMarkSweepGC -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/usr/share/tomcat7/endorsed -classpath /usr/share/tomcat7/bin/bootstrap.jar:/usr/share/tomcat7/bin/tomcat-juli.jar -Dcatalina.base=/var/lib/tomcat7 -Dcatalina.home=/usr/share/tomcat7 -Djava.io.tmpdir=/tmp/tomcat7-tomcat7-tmp org.apache.catalina.startup.Bootstrap start
ubuntu   19441 18542  0 17:14 pts/0    00:00:00 grep --color=auto java

I had suspected the memory, so I actually modified the Catalina memory in /usr/share/tomcat7/bin/setenv.sh.

This is my current setting:
CATALINA_OPTS=-Xms8g -Xmx25g -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -XX:+UseConcMarkSweepGC -XX:MaxPermSize=5g -XX:ReservedCodeCacheSize=64m

Is 25g not enough memory to get gff3 track export?

Thank you !!!

Berenice

2015-08-24 10:03 GMT-07:00 Nathan Dunn <[hidden email]>:

Berenice, 

You have to setup tomcat with the appropriate amount of memory.  

If I go here:  


you’ll see that you get an out of memory error.

If you look here, you have to set the memory explicitly:


1 GB should be plenty, but you have a lot to play with. 

To confirm you have the appropriate memory when you restart tomcat do:

ps -ef | grep java 

You should see your memory options in that line. 

Let us know if you have further issues.

Nathan Dunn, PhD
Berkeley Bioinformatics Open-source Projects (BBOP)
Genomics Division, Lawrence Berkeley National Laboratory
[hidden email]


On Aug 24, 2015, at 9:50 AM, Bérénice Benayoun <[hidden email]> wrote:

Hi all,

I have set up a GMOD in the Cloud instance to perform some manual curation of genes in a genome project I am involved in. 

It took a while, but things are mostly working now (I can create new annotations, etc.), even if they are crazy slow (even with the machine having 30Gb of RAM).

However, it is now impossible to download any of the user created annotation - every time I attempt the gff3 export, I get an "Unable to export data" message, and the fixes for that proposed in the forums (manually creating a gff3 folder with tomcat7 as owner, etc) have not helped my case.


Any help on the matter would be greatly appreciated!

Thank you,

Berenice.


--
Bérénice A. BENAYOUN, Ph.D.
Stanford University/Genetics Department
BRUNET Laboratory, 'Molecular Basis of Longevity and Age Related Diseases'
M312 Alway Building
300, Pasteur Drive
MC 5120
Stanford, CA 94305-5120

USA
Email: [hidden email]
Web: www.stanford.edu/group/brunet/



This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.






This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





--
Bérénice A. BENAYOUN, Ph.D.
Stanford University/Genetics Department
BRUNET Laboratory, 'Molecular Basis of Longevity and Age Related Diseases'
M312 Alway Building
300, Pasteur Drive
MC 5120
Stanford, CA 94305-5120

USA
Email: [hidden email]
Web: www.stanford.edu/group/brunet/




This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

Reply | Threaded
Open this post in threaded view
|

Re: Webapollo user annotation export problems with GMOD in the cloud

nathandunn

So, 1gb (or even 2 gb) should be sufficient.  I would not recommend more than 2 GB max unless you have a lot of instances or an especially fragmented scaffold. 

The problem you are having is that tomcat is not picking it up properly after you restart.  

So, a couple of things:

- if you stop tomcat and can confirm (using ps -ef | grep java) that it is not up and you restart and it is not getting the proper options then you need to hard-code it somewhere. 

The advice for tomcat 7 and 8 is the same as for tomcat 6.  I tend to edit the catalina.sh file that is responsible for starting and stopping tomcat.  


In my catalina.sh I have this at around line 275) . . . only the CATALINA_OPTS line is new  :

  shift
fi

CATALINA_OPTS="$CATALINA_OPTS -Xms512m -Xmx1g -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -XX:+UseConcMarkSweepGC "

if [ "$1" = "debug" ] ; then


In an ubuntu setup (probably the more likely scenario) I edited /etc/init.d/tomcat file, adding this at the top:

export CATALINA_OPTS="-Xms1g -Xmx4g -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -XX:+UseConcMarkSweepGC "

However for this, I am using this command to stop / start: “sudo service tomcat7 <command>”

Hope this helps,

Nathan

On Aug 24, 2015, at 10:16 AM, Bérénice Benayoun <[hidden email]> wrote:

Hi Nathan,

thanks for the quick answer!

When I run that comamnd,  I get this unhelpful message:
ubuntu@ip-10-191-141-46:~$ ps -ef | grep java 
tomcat7  19284     1 23 16:51 ?        00:05:20 /usr/lib/jvm/default-java/bin/java -Djava.util.logging.config.file=/var/lib/tomcat7/conf/logging.properties -Djava.awt.headless=true -Xmx128m -XX:+UseConcMarkSweepGC -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/usr/share/tomcat7/endorsed -classpath /usr/share/tomcat7/bin/bootstrap.jar:/usr/share/tomcat7/bin/tomcat-juli.jar -Dcatalina.base=/var/lib/tomcat7 -Dcatalina.home=/usr/share/tomcat7 -Djava.io.tmpdir=/tmp/tomcat7-tomcat7-tmp org.apache.catalina.startup.Bootstrap start
ubuntu   19441 18542  0 17:14 pts/0    00:00:00 grep --color=auto java

I had suspected the memory, so I actually modified the Catalina memory in /usr/share/tomcat7/bin/setenv.sh.

This is my current setting:
CATALINA_OPTS=-Xms8g -Xmx25g -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -XX:+UseConcMarkSweepGC -XX:MaxPermSize=5g -XX:ReservedCodeCacheSize=64m

Is 25g not enough memory to get gff3 track export?

Thank you !!!

Berenice

2015-08-24 10:03 GMT-07:00 Nathan Dunn <[hidden email]>:

Berenice, 

You have to setup tomcat with the appropriate amount of memory.  

If I go here:  


you’ll see that you get an out of memory error.

If you look here, you have to set the memory explicitly:


1 GB should be plenty, but you have a lot to play with. 

To confirm you have the appropriate memory when you restart tomcat do:

ps -ef | grep java 

You should see your memory options in that line. 

Let us know if you have further issues.

Nathan Dunn, PhD
Berkeley Bioinformatics Open-source Projects (BBOP)
Genomics Division, Lawrence Berkeley National Laboratory
[hidden email]


On Aug 24, 2015, at 9:50 AM, Bérénice Benayoun <[hidden email]> wrote:

Hi all,

I have set up a GMOD in the Cloud instance to perform some manual curation of genes in a genome project I am involved in. 

It took a while, but things are mostly working now (I can create new annotations, etc.), even if they are crazy slow (even with the machine having 30Gb of RAM).

However, it is now impossible to download any of the user created annotation - every time I attempt the gff3 export, I get an "Unable to export data" message, and the fixes for that proposed in the forums (manually creating a gff3 folder with tomcat7 as owner, etc) have not helped my case.


Any help on the matter would be greatly appreciated!

Thank you,

Berenice.


--
Bérénice A. BENAYOUN, Ph.D.
Stanford University/Genetics Department
BRUNET Laboratory, 'Molecular Basis of Longevity and Age Related Diseases'
M312 Alway Building
300, Pasteur Drive
MC 5120
Stanford, CA 94305-5120

USA
Email: [hidden email]
Web: www.stanford.edu/group/brunet/



This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.






This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





--
Bérénice A. BENAYOUN, Ph.D.
Stanford University/Genetics Department
BRUNET Laboratory, 'Molecular Basis of Longevity and Age Related Diseases'
M312 Alway Building
300, Pasteur Drive
MC 5120
Stanford, CA 94305-5120

USA
Email: [hidden email]
Web: www.stanford.edu/group/brunet/



This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.






This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

Reply | Threaded
Open this post in threaded view
|

Re: Webapollo user annotation export problems with GMOD in the cloud

Bérénice Benayoun
Hi Nathan,

indeed somehow tomcat was not picking up my modifications in setenv.sh and implementing them where you indicated solved the speed and ability to export gff3!

Thank you so much!

B

2015-08-24 10:49 GMT-07:00 Nathan Dunn <[hidden email]>:

So, 1gb (or even 2 gb) should be sufficient.  I would not recommend more than 2 GB max unless you have a lot of instances or an especially fragmented scaffold. 

The problem you are having is that tomcat is not picking it up properly after you restart.  

So, a couple of things:

- if you stop tomcat and can confirm (using ps -ef | grep java) that it is not up and you restart and it is not getting the proper options then you need to hard-code it somewhere. 

The advice for tomcat 7 and 8 is the same as for tomcat 6.  I tend to edit the catalina.sh file that is responsible for starting and stopping tomcat.  


In my catalina.sh I have this at around line 275) . . . only the CATALINA_OPTS line is new  :

  shift
fi

CATALINA_OPTS="$CATALINA_OPTS -Xms512m -Xmx1g -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -XX:+UseConcMarkSweepGC "

if [ "$1" = "debug" ] ; then


In an ubuntu setup (probably the more likely scenario) I edited /etc/init.d/tomcat file, adding this at the top:

export CATALINA_OPTS="-Xms1g -Xmx4g -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -XX:+UseConcMarkSweepGC "

However for this, I am using this command to stop / start: “sudo service tomcat7 <command>”

Hope this helps,

Nathan

On Aug 24, 2015, at 10:16 AM, Bérénice Benayoun <[hidden email]> wrote:

Hi Nathan,

thanks for the quick answer!

When I run that comamnd,  I get this unhelpful message:
ubuntu@ip-10-191-141-46:~$ ps -ef | grep java 
tomcat7  19284     1 23 16:51 ?        00:05:20 /usr/lib/jvm/default-java/bin/java -Djava.util.logging.config.file=/var/lib/tomcat7/conf/logging.properties -Djava.awt.headless=true -Xmx128m -XX:+UseConcMarkSweepGC -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/usr/share/tomcat7/endorsed -classpath /usr/share/tomcat7/bin/bootstrap.jar:/usr/share/tomcat7/bin/tomcat-juli.jar -Dcatalina.base=/var/lib/tomcat7 -Dcatalina.home=/usr/share/tomcat7 -Djava.io.tmpdir=/tmp/tomcat7-tomcat7-tmp org.apache.catalina.startup.Bootstrap start
ubuntu   19441 18542  0 17:14 pts/0    00:00:00 grep --color=auto java

I had suspected the memory, so I actually modified the Catalina memory in /usr/share/tomcat7/bin/setenv.sh.

This is my current setting:
CATALINA_OPTS=-Xms8g -Xmx25g -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -XX:+UseConcMarkSweepGC -XX:MaxPermSize=5g -XX:ReservedCodeCacheSize=64m

Is 25g not enough memory to get gff3 track export?

Thank you !!!

Berenice

2015-08-24 10:03 GMT-07:00 Nathan Dunn <[hidden email]>:

Berenice, 

You have to setup tomcat with the appropriate amount of memory.  

If I go here:  


you’ll see that you get an out of memory error.

If you look here, you have to set the memory explicitly:


1 GB should be plenty, but you have a lot to play with. 

To confirm you have the appropriate memory when you restart tomcat do:

ps -ef | grep java 

You should see your memory options in that line. 

Let us know if you have further issues.

Nathan Dunn, PhD
Berkeley Bioinformatics Open-source Projects (BBOP)
Genomics Division, Lawrence Berkeley National Laboratory
[hidden email]


On Aug 24, 2015, at 9:50 AM, Bérénice Benayoun <[hidden email]> wrote:

Hi all,

I have set up a GMOD in the Cloud instance to perform some manual curation of genes in a genome project I am involved in. 

It took a while, but things are mostly working now (I can create new annotations, etc.), even if they are crazy slow (even with the machine having 30Gb of RAM).

However, it is now impossible to download any of the user created annotation - every time I attempt the gff3 export, I get an "Unable to export data" message, and the fixes for that proposed in the forums (manually creating a gff3 folder with tomcat7 as owner, etc) have not helped my case.


Any help on the matter would be greatly appreciated!

Thank you,

Berenice.


--
Bérénice A. BENAYOUN, Ph.D.
Stanford University/Genetics Department
BRUNET Laboratory, 'Molecular Basis of Longevity and Age Related Diseases'
M312 Alway Building
300, Pasteur Drive
MC 5120
Stanford, CA 94305-5120

USA
Email: [hidden email]
Web: www.stanford.edu/group/brunet/



This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.






This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





--
Bérénice A. BENAYOUN, Ph.D.
Stanford University/Genetics Department
BRUNET Laboratory, 'Molecular Basis of Longevity and Age Related Diseases'
M312 Alway Building
300, Pasteur Drive
MC 5120
Stanford, CA 94305-5120

USA
Email: [hidden email]
Web: www.stanford.edu/group/brunet/



This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.






This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





--
Bérénice A. BENAYOUN, Ph.D.
Stanford University/Genetics Department
BRUNET Laboratory, 'Molecular Basis of Longevity and Age Related Diseases'
M312 Alway Building
300, Pasteur Drive
MC 5120
Stanford, CA 94305-5120

USA
Email: [hidden email]
Web: www.stanford.edu/group/brunet/




This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.