Fwd: delete_annotations_from_organism.groovy crash

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Fwd: delete_annotations_from_organism.groovy crash

Jacques Dainat-3
Dear all,

We tried (many times) to remove annotations using the `delete_annotations_from_organism.groovy ` script but it make tomcat crashing: Caused by: java.lang.OutOfMemoryError: Java heap space
The heap space is already fairly high: -Xmx12288m
The database contains currenlty 1774385 features. It sounds a lot for only 4 species… 
Anyway, I was wondering if it was possible to avoid the use of the `delete_annotations_from_organism.groovy` and use instead a proper postgresql command to clean safely the db according to the species ID. 

Best regards,

Jacques
-------------------------------------------------
Jacques Dainat, Ph.D.
NBIS (National Bioinformatics Infrastructure Sweden)
Genome Annotation Service
http://nbis.se/about/staff/jacques-dainat/
http://nbis.se

Address:
Uppsala University, Biomedicinska Centrum
Department of Medical Biochemistry Microbiology, Genomics
Husargatan 3, box 582
S-75123 Uppsala Sweden
Phone: +46 18 471 46 25






This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

Reply | Threaded
Open this post in threaded view
|

Re: delete_annotations_from_organism.groovy crash

nathandunn

However, every Exon and CDS is a feature, so 44K annotated genes could generate this (which is also a lot).  

How many annotations is it counting in your annotations tab per species? 



If you change the logging to default debug, what do you see in your logging output when executing the delete? 


I created a PR that *should* fix this:


I haven’t loaded the full set of data in there, but it worked fine for smaller sets.   If you want to change the batch size to pull, just adjust this:



Nathan



On Apr 11, 2018, at 6:12 AM, Jacques Dainat <[hidden email]> wrote:

Dear all,

We tried (many times) to remove annotations using the `delete_annotations_from_organism.groovy ` script but it make tomcat crashing: Caused by: java.lang.OutOfMemoryError: Java heap space
The heap space is already fairly high: -Xmx12288m
The database contains currenlty 1774385 features. It sounds a lot for only 4 species… 
Anyway, I was wondering if it was possible to avoid the use of the `delete_annotations_from_organism.groovy` and use instead a proper postgresql command to clean safely the db according to the species ID. 

Best regards,

Jacques
-------------------------------------------------
Jacques Dainat, Ph.D.
NBIS (National Bioinformatics Infrastructure Sweden)
Genome Annotation Service
http://nbis.se/about/staff/jacques-dainat/
http://nbis.se

Address:
Uppsala University, Biomedicinska Centrum
Department of Medical Biochemistry Microbiology, Genomics
Husargatan 3, box 582
S-75123 Uppsala Sweden
Phone: +46 18 471 46 25





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.






This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

Reply | Threaded
Open this post in threaded view
|

Re: delete_annotations_from_organism.groovy crash

nathandunn

Are you trying to delete all of the feature annotations, or just for the single organism?

If all of them, I just use this script:


Its a bit of a blunt hammer, though and will remove ALL annotations from EVERY organism for the specified database. 

Nathan


On Apr 11, 2018, at 4:12 PM, Nathan Dunn <[hidden email]> wrote:


However, every Exon and CDS is a feature, so 44K annotated genes could generate this (which is also a lot).  

How many annotations is it counting in your annotations tab per species? 

<Screen Shot 2018-04-11 at 2.38.55 PM.png>


If you change the logging to default debug, what do you see in your logging output when executing the delete? 


I created a PR that *should* fix this:


I haven’t loaded the full set of data in there, but it worked fine for smaller sets.   If you want to change the batch size to pull, just adjust this:



Nathan



On Apr 11, 2018, at 6:12 AM, Jacques Dainat <[hidden email]> wrote:

Dear all,

We tried (many times) to remove annotations using the `delete_annotations_from_organism.groovy ` script but it make tomcat crashing: Caused by: java.lang.OutOfMemoryError: Java heap space
The heap space is already fairly high: -Xmx12288m
The database contains currenlty 1774385 features. It sounds a lot for only 4 species… 
Anyway, I was wondering if it was possible to avoid the use of the `delete_annotations_from_organism.groovy` and use instead a proper postgresql command to clean safely the db according to the species ID. 

Best regards,

Jacques
-------------------------------------------------
Jacques Dainat, Ph.D.
NBIS (National Bioinformatics Infrastructure Sweden)
Genome Annotation Service
http://nbis.se/about/staff/jacques-dainat/
http://nbis.se

Address:
Uppsala University, Biomedicinska Centrum
Department of Medical Biochemistry Microbiology, Genomics
Husargatan 3, box 582
S-75123 Uppsala Sweden
Phone: +46 18 471 46 25





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.







This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

Reply | Threaded
Open this post in threaded view
|

Re: delete_annotations_from_organism.groovy crash

Jacques Dainat
Hello,

Thank you for the tip, but no I was trying to remove only annotation for a single species
Nevertheless it’s good to know the existence of this script.

Jacques

On 12 Apr 2018, at 20:26, Nathan Dunn <[hidden email]> wrote:


Are you trying to delete all of the feature annotations, or just for the single organism?

If all of them, I just use this script:


Its a bit of a blunt hammer, though and will remove ALL annotations from EVERY organism for the specified database. 

Nathan


On Apr 11, 2018, at 4:12 PM, Nathan Dunn <[hidden email]> wrote:


However, every Exon and CDS is a feature, so 44K annotated genes could generate this (which is also a lot).  

How many annotations is it counting in your annotations tab per species? 

<Screen Shot 2018-04-11 at 2.38.55 PM.png>


If you change the logging to default debug, what do you see in your logging output when executing the delete? 


I created a PR that *should* fix this:


I haven’t loaded the full set of data in there, but it worked fine for smaller sets.   If you want to change the batch size to pull, just adjust this:



Nathan



On Apr 11, 2018, at 6:12 AM, Jacques Dainat <[hidden email]> wrote:

Dear all,

We tried (many times) to remove annotations using the `delete_annotations_from_organism.groovy ` script but it make tomcat crashing: Caused by: java.lang.OutOfMemoryError: Java heap space
The heap space is already fairly high: -Xmx12288m
The database contains currenlty 1774385 features. It sounds a lot for only 4 species… 
Anyway, I was wondering if it was possible to avoid the use of the `delete_annotations_from_organism.groovy` and use instead a proper postgresql command to clean safely the db according to the species ID. 

Best regards,

Jacques
-------------------------------------------------
Jacques Dainat, Ph.D.
NBIS (National Bioinformatics Infrastructure Sweden)
Genome Annotation Service
http://nbis.se/about/staff/jacques-dainat/
http://nbis.se

Address:
Uppsala University, Biomedicinska Centrum
Department of Medical Biochemistry Microbiology, Genomics
Husargatan 3, box 582
S-75123 Uppsala Sweden
Phone: +46 18 471 46 25





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.






This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.






This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

Reply | Threaded
Open this post in threaded view
|

Re: Fwd: delete_annotations_from_organism.groovy crash

nathandunn
In reply to this post by Jacques Dainat-3
Ah okay.  

Were you able to rest the pull request? 

Nathan

On Apr 11, 2018, at 6:12 AM, Jacques Dainat <[hidden email]> wrote:

Dear all,

We tried (many times) to remove annotations using the `delete_annotations_from_organism.groovy ` script but it make tomcat crashing: Caused by: java.lang.OutOfMemoryError: Java heap space
The heap space is already fairly high: -Xmx12288m
The database contains currenlty 1774385 features. It sounds a lot for only 4 species… 
Anyway, I was wondering if it was possible to avoid the use of the `delete_annotations_from_organism.groovy` and use instead a proper postgresql command to clean safely the db according to the species ID. 

Best regards,

Jacques
-------------------------------------------------
Jacques Dainat, Ph.D.
NBIS (National Bioinformatics Infrastructure Sweden)
Genome Annotation Service
http://nbis.se/about/staff/jacques-dainat/
http://nbis.se

Address:
Uppsala University, Biomedicinska Centrum
Department of Medical Biochemistry Microbiology, Genomics
Husargatan 3, box 582
S-75123 Uppsala Sweden
Phone: +46 18 471 46 25





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

Reply | Threaded
Open this post in threaded view
|

Re: delete_annotations_from_organism.groovy crash

Jacques Dainat
In reply to this post by nathandunn
Hi,

Activating the logging to default debug, I have seen then this message in the log:

INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path

So, I have installed the librairy and relaunched the delete_annotations_from_organism.groovy script to see if it changes something. Usually the Java heap space problem was occurring after ~1h, after the library installation it was running for 3 days. But I have killed the command by mistake before it has finished. So, I will now try your modification.

Best,

/Jacques

P.S: I have ~40K + 50k + 10K + 0K gene annotated for my 4 species. I have checked the number of exon and CDS for those annotations, indeed having 1774385 features in the database seems perfectly normal.

On 12 Apr 2018, at 01:12, Nathan Dunn <[hidden email]> wrote:


However, every Exon and CDS is a feature, so 44K annotated genes could generate this (which is also a lot).  

How many annotations is it counting in your annotations tab per species? 

<Screen Shot 2018-04-11 at 2.38.55 PM.png>


If you change the logging to default debug, what do you see in your logging output when executing the delete? 


I created a PR that *should* fix this:


I haven’t loaded the full set of data in there, but it worked fine for smaller sets.   If you want to change the batch size to pull, just adjust this:



Nathan



On Apr 11, 2018, at 6:12 AM, Jacques Dainat <[hidden email]> wrote:

Dear all,

We tried (many times) to remove annotations using the `delete_annotations_from_organism.groovy ` script but it make tomcat crashing: Caused by: java.lang.OutOfMemoryError: Java heap space
The heap space is already fairly high: -Xmx12288m
The database contains currenlty 1774385 features. It sounds a lot for only 4 species… 
Anyway, I was wondering if it was possible to avoid the use of the `delete_annotations_from_organism.groovy` and use instead a proper postgresql command to clean safely the db according to the species ID. 

Best regards,

Jacques
-------------------------------------------------
Jacques Dainat, Ph.D.
NBIS (National Bioinformatics Infrastructure Sweden)
Genome Annotation Service
http://nbis.se/about/staff/jacques-dainat/
http://nbis.se

Address:
Uppsala University, Biomedicinska Centrum
Department of Medical Biochemistry Microbiology, Genomics
Husargatan 3, box 582
S-75123 Uppsala Sweden
Phone: +46 18 471 46 25





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.






This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

Reply | Threaded
Open this post in threaded view
|

Re: delete_annotations_from_organism.groovy crash

nathandunn
What version of Tomcat are you running?

Also, if you type:

ps -ef | grep java 

What do you see for tomcat when running?

Nathan

On Apr 17, 2018, at 12:33 AM, Jacques Dainat <[hidden email]> wrote:

Hi,

Activating the logging to default debug, I have seen then this message in the log:

INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path

So, I have installed the librairy and relaunched the delete_annotations_from_organism.groovy script to see if it changes something. Usually the Java heap space problem was occurring after ~1h, after the library installation it was running for 3 days. But I have killed the command by mistake before it has finished. So, I will now try your modification.

Best,

/Jacques

P.S: I have ~40K + 50k + 10K + 0K gene annotated for my 4 species. I have checked the number of exon and CDS for those annotations, indeed having 1774385 features in the database seems perfectly normal.

On 12 Apr 2018, at 01:12, Nathan Dunn <[hidden email]> wrote:


However, every Exon and CDS is a feature, so 44K annotated genes could generate this (which is also a lot).  

How many annotations is it counting in your annotations tab per species? 

<Screen Shot 2018-04-11 at 2.38.55 PM.png>


If you change the logging to default debug, what do you see in your logging output when executing the delete? 


I created a PR that *should* fix this:


I haven’t loaded the full set of data in there, but it worked fine for smaller sets.   If you want to change the batch size to pull, just adjust this:



Nathan



On Apr 11, 2018, at 6:12 AM, Jacques Dainat <[hidden email]> wrote:

Dear all,

We tried (many times) to remove annotations using the `delete_annotations_from_organism.groovy ` script but it make tomcat crashing: Caused by: java.lang.OutOfMemoryError: Java heap space
The heap space is already fairly high: -Xmx12288m
The database contains currenlty 1774385 features. It sounds a lot for only 4 species… 
Anyway, I was wondering if it was possible to avoid the use of the `delete_annotations_from_organism.groovy` and use instead a proper postgresql command to clean safely the db according to the species ID. 

Best regards,

Jacques
-------------------------------------------------
Jacques Dainat, Ph.D.
NBIS (National Bioinformatics Infrastructure Sweden)
Genome Annotation Service
http://nbis.se/about/staff/jacques-dainat/
http://nbis.se

Address:
Uppsala University, Biomedicinska Centrum
Department of Medical Biochemistry Microbiology, Genomics
Husargatan 3, box 582
S-75123 Uppsala Sweden
Phone: +46 18 471 46 25





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

Reply | Threaded
Open this post in threaded view
|

Re: delete_annotations_from_organism.groovy crash

Jacques Dainat
Hi,

here the information you have asked for.

On 17 Apr 2018, at 15:48, Lbl <[hidden email]> wrote:

What version of Tomcat are you running?


Tomcat 7.0.85

Also, if you type:

ps -ef | grep java 

What do you see for tomcat when running?


tomcat    6586     1 12 Apr17 ?        02:55:12 /usr/lib/jvm/jre/bin/java -Xmx12288m -Xms1024m -XX:PermSize=256m -XX:MaxPermSize=1024m -XX:ReservedCodeCacheSize=64m -XX:+UseG1GC -XX:+CMSClassUnloadingEnabled -Xloggc:/usr/share/tomcat/logs/gc.log -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -classpath :/usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/usr/share/tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/usr/share/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start
jacques  17726 18069  0 Apr17 pts/16   00:01:48 /usr/lib/jvm/java-1.8.0-openjdk.x86_64/bin/java -classpath /home/jacques/.sdkman/candidates/groovy/current/lib/groovy-2.4.7.jar -Dscript.name=/home/jacques/.sdkman/candidates/groovy/current/bin/groovy -Dprogram.name=groovy -Dgroovy.starter.conf=/home/jacques/.sdkman/candidates/groovy/current/conf/groovy-starter.conf -Dgroovy.home=/home/jacques/.sdkman/candidates/groovy/current -Dtools.jar=/usr/lib/jvm/java-1.8.0-openjdk.x86_64/lib/tools.jar org.codehaus.groovy.tools.GroovyStarter --main groovy.ui.GroovyMain --conf /home/jacques/.sdkman/candidates/groovy/current/conf/groovy-starter.conf --classpath . /big/webapollo/build/NBIS/docs/web_services/examples/groovy/delete_annotations_from_organism.groovy -destinationurl http://localhost:8080/NBIS --organismname XXX -adminpassword XXX -adminusername XXX
Nathan


Btw, I have kept the version 2.0.8 of webapollo and just copy past the modification you have done on the OrganismService.groovy script and deployed this patched version. I have then relaunched the command delete_annotations_from_organism.groovy again, but it sounds that the result is the same… It’s running for 24 hours and when I check the number of feature in the DB it is still the same. Then trying to connect several times on Webapollo I have made tomcat crashing again. It doesn't answer anymore… I’m wondering if the problem doesn’t come instead from the authentifiaction, or a concomitance between authentifiaction and deleting in the same time… 
I’m a bit lost. 

2018-04-18 09:59:32,872 [http-apr-8080-exec-14] ERROR errors.GrailsExceptionResolver  - IllegalStateException occurred when processing request: [GET] /NBIS/1465026667935935802497283825/jbrowse/data/tracks/rc5_over50/NODE_1_length_534763_cov_98.635_ID_111564/lf-5.json
getOutputStream() has already been called for this response. Stacktrace follows:
org.codehaus.groovy.grails.web.pages.exceptions.GroovyPagesException: Error processing GroovyPageView: getOutputStream() has already been called for this response
        at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:198)
        at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
        at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
        at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
        at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
        at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
        at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
        at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
        at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: getOutputStream() has already been called for this response
        at java.io.FilterWriter.flush(FilterWriter.java:100)
        ... 12 more
2018-04-18 10:00:19,312 [http-apr-8080-exec-1] WARN  apollo.PreferenceService  - No user present, so using the client token
2018-04-18 10:00:19,369 [http-apr-8080-exec-10] ERROR apollo.PermissionService  - Username not supplied so can not authenticate.
2018-04-18 10:00:19,393 [http-apr-8080-exec-10] ERROR apollo.PermissionService  - User null does not exist in the database.
2018-04-18 10:00:19,511 [http-apr-8080-exec-3] ERROR authenticator.UsernamePasswordAuthenticatorService  - Problem authenticating: java.lang.IllegalArgumentException: Method argumet (authentication token) cannot be null.
2018-04-18 10:00:19,511 [http-apr-8080-exec-3] WARN  apollo.PermissionService  - Failed to authenticate user
2018-04-18 10:00:19,512 [http-apr-8080-exec-3] ERROR apollo.UserController  - Failed to authenticate
/opt/apollo/20-thielavia_terrestris/extendedTrackList.json
2018-04-18 10:01:52,781 [http-apr-8080-exec-1] ERROR apollo.JbrowseController  - File not found: /big/data/thielavia_terrestris/names/root.json
2018-04-18 10:04:13,777 [http-apr-8080-exec-21] ERROR apollo.JbrowseController  - File not found: /big/data/thielavia_terrestris/names/root.json
/opt/apollo/20-thielavia_terrestris/extendedTrackList.json
2018-04-18 10:04:56,641 [http-apr-8080-exec-19] WARN  apollo.PreferenceService  - No user present, so using the client token
2018-04-18 10:04:56,729 [http-apr-8080-exec-12] ERROR apollo.PermissionService  - Username not supplied so can not authenticate.
2018-04-18 10:04:56,870 [http-apr-8080-exec-12] ERROR apollo.PermissionService  - User null does not exist in the database.
2018-04-18 10:04:56,914 [http-apr-8080-exec-14] ERROR authenticator.UsernamePasswordAuthenticatorService  - Problem authenticating: java.lang.IllegalArgumentException: Method argumet (authentication token) cannot be null.
2018-04-18 10:04:56,915 [http-apr-8080-exec-14] WARN  apollo.PermissionService  - Failed to authenticate user
2018-04-18 10:04:56,915 [http-apr-8080-exec-14] ERROR apollo.UserController  - Failed to authenticate

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "http-apr-8080-exec-30"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DefaultPluginScheduler_QuartzSchedulerThread"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DefaultPluginScheduler_QuartzSchedulerThread"

On Apr 17, 2018, at 12:33 AM, Jacques Dainat <[hidden email]> wrote:

Hi,

Activating the logging to default debug, I have seen then this message in the log:

INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path

So, I have installed the librairy and relaunched the delete_annotations_from_organism.groovy script to see if it changes something. Usually the Java heap space problem was occurring after ~1h, after the library installation it was running for 3 days. But I have killed the command by mistake before it has finished. So, I will now try your modification.

Best,

/Jacques

P.S: I have ~40K + 50k + 10K + 0K gene annotated for my 4 species. I have checked the number of exon and CDS for those annotations, indeed having 1774385 features in the database seems perfectly normal.

On 12 Apr 2018, at 01:12, Nathan Dunn <[hidden email]> wrote:


However, every Exon and CDS is a feature, so 44K annotated genes could generate this (which is also a lot).  

How many annotations is it counting in your annotations tab per species? 

<Screen Shot 2018-04-11 at 2.38.55 PM.png>


If you change the logging to default debug, what do you see in your logging output when executing the delete? 


I created a PR that *should* fix this:


I haven’t loaded the full set of data in there, but it worked fine for smaller sets.   If you want to change the batch size to pull, just adjust this:



Nathan



On Apr 11, 2018, at 6:12 AM, Jacques Dainat <[hidden email]> wrote:

Dear all,

We tried (many times) to remove annotations using the `delete_annotations_from_organism.groovy ` script but it make tomcat crashing: Caused by: java.lang.OutOfMemoryError: Java heap space
The heap space is already fairly high: -Xmx12288m
The database contains currenlty 1774385 features. It sounds a lot for only 4 species… 
Anyway, I was wondering if it was possible to avoid the use of the `delete_annotations_from_organism.groovy` and use instead a proper postgresql command to clean safely the db according to the species ID. 

Best regards,

Jacques
-------------------------------------------------
Jacques Dainat, Ph.D.
NBIS (National Bioinformatics Infrastructure Sweden)
Genome Annotation Service
http://nbis.se/about/staff/jacques-dainat/
http://nbis.se

Address:
Uppsala University, Biomedicinska Centrum
Department of Medical Biochemistry Microbiology, Genomics
Husargatan 3, box 582
S-75123 Uppsala Sweden
Phone: +46 18 471 46 25





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.




This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.






This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.