Quantcast

Webapollo 2 slow

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Webapollo 2 slow

Jacques Dainat-3
Hi all,

I’m using webapollo 2.0.4, and I find that modifying gene models is taking forever compare to older version (as 1.0.5). Like to stretch an exon I have to wait between 20 seconds and 1min that the modification is taken into account and displayed. I have 12 genomes, and I set up tomcat as advised within the documentation:
-Xmx12288m -Xms8192m -XX:PermSize=256m -XX:MaxPermSize=1024m -XX:ReservedCodeCacheSize=64m -XX:+UseG1GC -XX:+CMSClassUnloadingEnabled -Xloggc:/usr/share/tomcat/logs/gc.log -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps

Did I miss something ? Do you have some tip to optimise the speed ? Could it be the database that has to be optimised ?
Could a Javascript minimization fix it ?


Webapollo config:
  • Version: 2.0.4-SNAPSHOT
  • Grails version: 2.5.5
  • Groovy version: 2.4.4
  • JVM version: 1.7.0_111

Best regards,

Jacques Dainat, PhD
NBIS (National Bioinformatics Infrastructure Sweden)
Genome Annotation Service





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Webapollo 2 slow

nathandunn

Jacques,

Sorry that things are taking too long.  Thanks for posting the details.

A few things:

1 - it looks like you are using a snapshot instead of the official release version (https://github.com/GMOD/Apollo/releases/tag/2.0.4).  This is unlikely to be the issue, but its hard to say.  
2 - Can you confirm that the memory settings you setup for tomcat are being picked up in memory?  (ps -ef | grep java  . . . as well as top) 
3 - The min-memory is pretty large.  I would probably use 1GB (Xms1024), but I don’t think that would be a problem.
4 - What version of Java are you using? 
5 - Are you seeing spikes in database (again using top) during this?
6 - What is your user load (i.e., is it like this when no one is on the system, or just sometimes when everyone is)?

Nathan Dunn, PhD
Berkeley Bioinformatics Open-source Projects (BBOP)
Genomics Division, Lawrence Berkeley National Laboratory
[hidden email]


On Sep 21, 2016, at 7:46 AM, Jacques Dainat <[hidden email]> wrote:

Hi all,

I’m using webapollo 2.0.4, and I find that modifying gene models is taking forever compare to older version (as 1.0.5). Like to stretch an exon I have to wait between 20 seconds and 1min that the modification is taken into account and displayed. I have 12 genomes, and I set up tomcat as advised within the documentation:
-Xmx12288m -Xms8192m -XX:PermSize=256m -XX:MaxPermSize=1024m -XX:ReservedCodeCacheSize=64m -XX:+UseG1GC -XX:+CMSClassUnloadingEnabled -Xloggc:/usr/share/tomcat/logs/gc.log -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps

Did I miss something ? Do you have some tip to optimise the speed ? Could it be the database that has to be optimised ?
Could a Javascript minimization fix it ?


Webapollo config:
  • Version: 2.0.4-SNAPSHOT
  • Grails version: 2.5.5
  • Groovy version: 2.4.4
  • JVM version: 1.7.0_111

Best regards,

Jacques Dainat, PhD
NBIS (National Bioinformatics Infrastructure Sweden)
Genome Annotation Service




This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.






This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Webapollo 2 slow

Jacques Dainat-3
Hello,

Here the answers:

1 - it looks like you are using a snapshot instead of the official release version (https://github.com/GMOD/Apollo/releases/tag/2.0.4).  This is unlikely to be the issue, but its hard to say. 

I will change it soon.

2 - Can you confirm that the memory settings you setup for tomcat are being picked up in memory?  (ps -ef | grep java  . . . as well as top) 

the ps command gives the result I pasted in the previous email.
the top show:  2018 tomcat    20   0 15.9g 9.0g 7988 S  0.3 28.6  64:36.86 java

3 - The min-memory is pretty large.  I would probably use 1GB (Xms1024), but I don’t think that would be a problem.

Ok, I will give a try.

4 - What version of Java are you using?

java version "1.7.0_111"
OpenJDK Runtime Environment (rhel-2.6.7.2.el6_8-x86_64 u111-b01)
OpenJDK 64-Bit Server VM (build 24.111-b01, mixed mode) 

5 - Are you seeing spikes in database (again using top) during this?

Yes postgres is going up to 100% CPU.

6 - What is your user load (i.e., is it like this when no one is on the system, or just sometimes when everyone is)?

The user load is quite low. But no matter how many people are using it. It is slow even if I’m alone to use it.


Best regards,

Jacques




On Sep 21, 2016, at 7:46 AM, Jacques Dainat <[hidden email]> wrote:

Hi all,

I’m using webapollo 2.0.4, and I find that modifying gene models is taking forever compare to older version (as 1.0.5). Like to stretch an exon I have to wait between 20 seconds and 1min that the modification is taken into account and displayed. I have 12 genomes, and I set up tomcat as advised within the documentation:
-Xmx12288m -Xms8192m -XX:PermSize=256m -XX:MaxPermSize=1024m -XX:ReservedCodeCacheSize=64m -XX:+UseG1GC -XX:+CMSClassUnloadingEnabled -Xloggc:/usr/share/tomcat/logs/gc.log -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps

Did I miss something ? Do you have some tip to optimise the speed ? Could it be the database that has to be optimised ?
Could a Javascript minimization fix it ?


Webapollo config:
  • Version: 2.0.4-SNAPSHOT
  • Grails version: 2.5.5
  • Groovy version: 2.4.4
  • JVM version: 1.7.0_111

Best regards,

Jacques Dainat, PhD
NBIS (National Bioinformatics Infrastructure Sweden)
Genome Annotation Service




This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.






This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Webapollo 2 slow

nathandunn

Jacques,

It is probably a bug in one of the queries (PostgreSQL being pegged), though you should update to the current release (2.0.4) in case we have already fixed it.  

Just so I can reproduce it (it it is unfixed with the release), I had a few questions:

1 - do you see the same effect for ALL organisms? 
2 - roughly how many annotated features (genes, etc.) do you have in the organism in question? 
3 - if you create a new test organism with no annotated features, do you see the same effect? 
4 - how many organisms do you have? 
5 - how many users do you have registered on the system? 
6 - is it any worse / better in regions with lots of isoforms (screenshot would be helpful if possible) and if so, how many isoforms are we looking at? 

Thanks,

Nathan Dunn, PhD
Berkeley Bioinformatics Open-source Projects (BBOP)
Genomics Division, Lawrence Berkeley National Laboratory
[hidden email]


On Sep 22, 2016, at 12:54 AM, Jacques Dainat <[hidden email]> wrote:

Hello,

Here the answers:

1 - it looks like you are using a snapshot instead of the official release version (https://github.com/GMOD/Apollo/releases/tag/2.0.4).  This is unlikely to be the issue, but its hard to say. 

I will change it soon.

2 - Can you confirm that the memory settings you setup for tomcat are being picked up in memory?  (ps -ef | grep java  . . . as well as top) 

the ps command gives the result I pasted in the previous email.
the top show:  2018 tomcat    20   0 15.9g 9.0g 7988 S  0.3 28.6  64:36.86 java

3 - The min-memory is pretty large.  I would probably use 1GB (Xms1024), but I don’t think that would be a problem.

Ok, I will give a try.

4 - What version of Java are you using?

java version "1.7.0_111"
OpenJDK Runtime Environment (rhel-2.6.7.2.el6_8-x86_64 u111-b01)
OpenJDK 64-Bit Server VM (build 24.111-b01, mixed mode) 

5 - Are you seeing spikes in database (again using top) during this?

Yes postgres is going up to 100% CPU.

6 - What is your user load (i.e., is it like this when no one is on the system, or just sometimes when everyone is)?

The user load is quite low. But no matter how many people are using it. It is slow even if I’m alone to use it.


Best regards,

Jacques




On Sep 21, 2016, at 7:46 AM, Jacques Dainat <[hidden email]> wrote:

Hi all,

I’m using webapollo 2.0.4, and I find that modifying gene models is taking forever compare to older version (as 1.0.5). Like to stretch an exon I have to wait between 20 seconds and 1min that the modification is taken into account and displayed. I have 12 genomes, and I set up tomcat as advised within the documentation:
-Xmx12288m -Xms8192m -XX:PermSize=256m -XX:MaxPermSize=1024m -XX:ReservedCodeCacheSize=64m -XX:+UseG1GC -XX:+CMSClassUnloadingEnabled -Xloggc:/usr/share/tomcat/logs/gc.log -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps

Did I miss something ? Do you have some tip to optimise the speed ? Could it be the database that has to be optimised ?
Could a Javascript minimization fix it ?


Webapollo config:
  • Version: 2.0.4-SNAPSHOT
  • Grails version: 2.5.5
  • Groovy version: 2.4.4
  • JVM version: 1.7.0_111

Best regards,

Jacques Dainat, PhD
NBIS (National Bioinformatics Infrastructure Sweden)
Genome Annotation Service




This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.






This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Webapollo 2 slow

nathandunn

Also forgot to ask, how many scaffolds / sequences per organism are we looking at and what is the approximate size of these genomes? 

If any of this is non-public data, please feel free to reply to me directly. 

Nathan Dunn, PhD
Berkeley Bioinformatics Open-source Projects (BBOP)
Genomics Division, Lawrence Berkeley National Laboratory
[hidden email]


On Sep 22, 2016, at 8:45 AM, Nathan Dunn <[hidden email]> wrote:


Jacques,

It is probably a bug in one of the queries (PostgreSQL being pegged), though you should update to the current release (2.0.4) in case we have already fixed it.  

Just so I can reproduce it (it it is unfixed with the release), I had a few questions:

1 - do you see the same effect for ALL organisms? 
2 - roughly how many annotated features (genes, etc.) do you have in the organism in question? 
3 - if you create a new test organism with no annotated features, do you see the same effect? 
4 - how many organisms do you have? 
5 - how many users do you have registered on the system? 
6 - is it any worse / better in regions with lots of isoforms (screenshot would be helpful if possible) and if so, how many isoforms are we looking at? 

Thanks,

Nathan Dunn, PhD
Berkeley Bioinformatics Open-source Projects (BBOP)
Genomics Division, Lawrence Berkeley National Laboratory
[hidden email]


On Sep 22, 2016, at 12:54 AM, Jacques Dainat <[hidden email]> wrote:

Hello,

Here the answers:

1 - it looks like you are using a snapshot instead of the official release version (https://github.com/GMOD/Apollo/releases/tag/2.0.4).  This is unlikely to be the issue, but its hard to say. 

I will change it soon.

2 - Can you confirm that the memory settings you setup for tomcat are being picked up in memory?  (ps -ef | grep java  . . . as well as top) 

the ps command gives the result I pasted in the previous email.
the top show:  2018 tomcat    20   0 15.9g 9.0g 7988 S  0.3 28.6  64:36.86 java

3 - The min-memory is pretty large.  I would probably use 1GB (Xms1024), but I don’t think that would be a problem.

Ok, I will give a try.

4 - What version of Java are you using?

java version "1.7.0_111"
OpenJDK Runtime Environment (rhel-2.6.7.2.el6_8-x86_64 u111-b01)
OpenJDK 64-Bit Server VM (build 24.111-b01, mixed mode) 

5 - Are you seeing spikes in database (again using top) during this?

Yes postgres is going up to 100% CPU.

6 - What is your user load (i.e., is it like this when no one is on the system, or just sometimes when everyone is)?

The user load is quite low. But no matter how many people are using it. It is slow even if I’m alone to use it.


Best regards,

Jacques




On Sep 21, 2016, at 7:46 AM, Jacques Dainat <[hidden email]> wrote:

Hi all,

I’m using webapollo 2.0.4, and I find that modifying gene models is taking forever compare to older version (as 1.0.5). Like to stretch an exon I have to wait between 20 seconds and 1min that the modification is taken into account and displayed. I have 12 genomes, and I set up tomcat as advised within the documentation:
-Xmx12288m -Xms8192m -XX:PermSize=256m -XX:MaxPermSize=1024m -XX:ReservedCodeCacheSize=64m -XX:+UseG1GC -XX:+CMSClassUnloadingEnabled -Xloggc:/usr/share/tomcat/logs/gc.log -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps

Did I miss something ? Do you have some tip to optimise the speed ? Could it be the database that has to be optimised ?
Could a Javascript minimization fix it ?


Webapollo config:
  • Version: 2.0.4-SNAPSHOT
  • Grails version: 2.5.5
  • Groovy version: 2.4.4
  • JVM version: 1.7.0_111

Best regards,

Jacques Dainat, PhD
NBIS (National Bioinformatics Infrastructure Sweden)
Genome Annotation Service




This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.







This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Webapollo 2 slow

nathandunn
In reply to this post by Jacques Dainat-3

Just a quick update on this if you are experiencing a slowdown (operations that take >3 seconds to complete):

- I’ve tested this with mysql (but it should be the same) and 35K annotated genes and I have good performance (0.3-2 seconds) for all operations I’ve tried (setting exon boundaries, adding transcripts, merging, etc.).   I’ll test with postgresql and see if I notice anything different.

- I’ve moved most of the conversation to https://github.com/GMOD/Apollo/issues/1276 in case there is a coding fix

- Loading features using the add-transcript-from-gff3.pl may clog the preferences table: https://github.com/GMOD/Apollo/issues/1277  . . I can’t see why that would be a problem, but feel free to delete the contents of that table to see if it helps (and let me know if it does).

- You should have an index on unique_name on the feature_event table.  This only seems to affect mysql (postgresql does magic), so make sure that is there.   By default this is added, so you should not need to do anything. 

- I’ll let you know if there are any system-wide recommendations.  


Nathan


On Sep 22, 2016, at 12:54 AM, Jacques Dainat <[hidden email]> wrote:

Hello,

Here the answers:

1 - it looks like you are using a snapshot instead of the official release version (https://github.com/GMOD/Apollo/releases/tag/2.0.4).  This is unlikely to be the issue, but its hard to say. 

I will change it soon.

2 - Can you confirm that the memory settings you setup for tomcat are being picked up in memory?  (ps -ef | grep java  . . . as well as top) 

the ps command gives the result I pasted in the previous email.
the top show:  2018 tomcat    20   0 15.9g 9.0g 7988 S  0.3 28.6  64:36.86 java

3 - The min-memory is pretty large.  I would probably use 1GB (Xms1024), but I don’t think that would be a problem.

Ok, I will give a try.

4 - What version of Java are you using?

java version "1.7.0_111"
OpenJDK Runtime Environment (rhel-2.6.7.2.el6_8-x86_64 u111-b01)
OpenJDK 64-Bit Server VM (build 24.111-b01, mixed mode) 

5 - Are you seeing spikes in database (again using top) during this?

Yes postgres is going up to 100% CPU.

6 - What is your user load (i.e., is it like this when no one is on the system, or just sometimes when everyone is)?

The user load is quite low. But no matter how many people are using it. It is slow even if I’m alone to use it.


Best regards,

Jacques




On Sep 21, 2016, at 7:46 AM, Jacques Dainat <[hidden email]> wrote:

Hi all,

I’m using webapollo 2.0.4, and I find that modifying gene models is taking forever compare to older version (as 1.0.5). Like to stretch an exon I have to wait between 20 seconds and 1min that the modification is taken into account and displayed. I have 12 genomes, and I set up tomcat as advised within the documentation:
-Xmx12288m -Xms8192m -XX:PermSize=256m -XX:MaxPermSize=1024m -XX:ReservedCodeCacheSize=64m -XX:+UseG1GC -XX:+CMSClassUnloadingEnabled -Xloggc:/usr/share/tomcat/logs/gc.log -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps

Did I miss something ? Do you have some tip to optimise the speed ? Could it be the database that has to be optimised ?
Could a Javascript minimization fix it ?


Webapollo config:
  • Version: 2.0.4-SNAPSHOT
  • Grails version: 2.5.5
  • Groovy version: 2.4.4
  • JVM version: 1.7.0_111

Best regards,

Jacques Dainat, PhD
NBIS (National Bioinformatics Infrastructure Sweden)
Genome Annotation Service




This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.






This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

Loading...