heap space error

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

heap space error

Parul Kudtarkar
Dear Apollo community,

I have set CATALINA_OPTS  as recommend by webapollo, even though webapollo
is running, it's pretty slow and I get error messages logged into
catalina.out

 ps -ef | grep java
root     13952     1 99 12:11 pts/1    00:25:47 /usr/bin/java
-Djava.util.logging.config.file=/usr/share/apache-tomcat-7.0.59/conf/logging.properties
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms512m
-Xmx1g -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled
-XX:+UseConcMarkSweepGC -XX:MaxPermSize=256m
-Djava.endorsed.dirs=/usr/share/apache-tomcat-7.0.59/endorsed -classpath
/usr/share/apache-tomcat-7.0.59/bin/bootstrap.jar:/usr/share/apache-tomcat-7.0.59/bin/tomcat-juli.jar
-Dcatalina.base=/usr/share/apache-tomcat-7.0.59
-Dcatalina.home=/usr/share/apache-tomcat-7.0.59
-Djava.io.tmpdir=/usr/share/apache-tomcat-7.0.59/temp
org.apache.catalina.startup.Bootstrap start
----

Mar 24, 2015 11:18:37 AM
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor run
SEVERE: Unexpected death of background thread
ContainerBackgroundProcessor[StandardEngine[Catalina]]
java.lang.OutOfMemoryError: Java heap space

        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
Exception in thread
"ContainerBackgroundProcessor[StandardEngine[Catalina]]"
java.lang.OutOfMemoryError: Java heap space
Exception in thread "http-bio-8080-exec-5" Exception in thread
"http-bio-8080-exec-10" java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread "http-bio-8080-exec-4" java.lang.OutOfMemoryError:
Java heap space
----

--
Scientific Programmer
Center for Computational Regulatory Genomics
Beckman Institute,
Biology and Biological Engineering
California Institute of Technology
http://www.echinobase.org/

Reply | Threaded
Open this post in threaded view
|

Re: heap space error

nathandunn

Thanks for the report.  This is a known issue that will be completely address in 2.0, but we have to work around it in the interim.  

I think that the number one cause of this will likely be doing a search on the recent changes or sequence search if you have many sequences or changes or are hosting multiple organisms on the same tomcat.

You can also try using the old interface (in the upper-right-hand side) of either page and see if that is any better.  

Does this seem consistent with what you are seeing or are you seeing the memory creep up in other places?    

If you know the number of sequences you have, we might be able to get a better memory setting to tune.  Typically with around 5K sequences we have no problems (other than the above mentioned).  

Nathan

> On Mar 24, 2015, at 12:28 PM, Parul Kudtarkar <[hidden email]> wrote:
>
> Dear Apollo community,
>
> I have set CATALINA_OPTS  as recommend by webapollo, even though webapollo
> is running, it's pretty slow and I get error messages logged into
> catalina.out
>
> ps -ef | grep java
> root     13952     1 99 12:11 pts/1    00:25:47 /usr/bin/java
> -Djava.util.logging.config.file=/usr/share/apache-tomcat-7.0.59/conf/logging.properties
> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms512m
> -Xmx1g -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled
> -XX:+UseConcMarkSweepGC -XX:MaxPermSize=256m
> -Djava.endorsed.dirs=/usr/share/apache-tomcat-7.0.59/endorsed -classpath
> /usr/share/apache-tomcat-7.0.59/bin/bootstrap.jar:/usr/share/apache-tomcat-7.0.59/bin/tomcat-juli.jar
> -Dcatalina.base=/usr/share/apache-tomcat-7.0.59
> -Dcatalina.home=/usr/share/apache-tomcat-7.0.59
> -Djava.io.tmpdir=/usr/share/apache-tomcat-7.0.59/temp
> org.apache.catalina.startup.Bootstrap start
> ----
>
> Mar 24, 2015 11:18:37 AM
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor run
> SEVERE: Unexpected death of background thread
> ContainerBackgroundProcessor[StandardEngine[Catalina]]
> java.lang.OutOfMemoryError: Java heap space
>
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> Exception in thread
> "ContainerBackgroundProcessor[StandardEngine[Catalina]]"
> java.lang.OutOfMemoryError: Java heap space
> Exception in thread "http-bio-8080-exec-5" Exception in thread
> "http-bio-8080-exec-10" java.lang.OutOfMemoryError: Java heap space
> java.lang.OutOfMemoryError: Java heap space
> Exception in thread "http-bio-8080-exec-4" java.lang.OutOfMemoryError:
> Java heap space
> ----
>
> --
> Scientific Programmer
> Center for Computational Regulatory Genomics
> Beckman Institute,
> Biology and Biological Engineering
> California Institute of Technology
> http://www.echinobase.org/
>


Reply | Threaded
Open this post in threaded view
|

Re: heap space error

Parul Kudtarkar
Dear Nathan,

Thanks for a quick response. You are correct, using the new interface and
either doing sequence search or doing search on recent changes gives heap
search error.
At the moment we are hosting single organism and the number of sequences
are 336K, far beyond what welapollo can host. Could you suggest
appropriate memory setting?

Thanks and regards,
Parul Kudtarkar

>
> Thanks for the report.  This is a known issue that will be completely
> address in 2.0, but we have to work around it in the interim.
>
> I think that the number one cause of this will likely be doing a search on
> the recent changes or sequence search if you have many sequences or
> changes or are hosting multiple organisms on the same tomcat.
>
> You can also try using the old interface (in the upper-right-hand side) of
> either page and see if that is any better.
>
> Does this seem consistent with what you are seeing or are you seeing the
> memory creep up in other places?
>
> If you know the number of sequences you have, we might be able to get a
> better memory setting to tune.  Typically with around 5K sequences we have
> no problems (other than the above mentioned).
>
> Nathan
>
>> On Mar 24, 2015, at 12:28 PM, Parul Kudtarkar <[hidden email]>
>> wrote:
>>
>> Dear Apollo community,
>>
>> I have set CATALINA_OPTS  as recommend by webapollo, even though
>> webapollo
>> is running, it's pretty slow and I get error messages logged into
>> catalina.out
>>
>> ps -ef | grep java
>> root     13952     1 99 12:11 pts/1    00:25:47 /usr/bin/java
>> -Djava.util.logging.config.file=/usr/share/apache-tomcat-7.0.59/conf/logging.properties
>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>> -Xms512m
>> -Xmx1g -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled
>> -XX:+UseConcMarkSweepGC -XX:MaxPermSize=256m
>> -Djava.endorsed.dirs=/usr/share/apache-tomcat-7.0.59/endorsed -classpath
>> /usr/share/apache-tomcat-7.0.59/bin/bootstrap.jar:/usr/share/apache-tomcat-7.0.59/bin/tomcat-juli.jar
>> -Dcatalina.base=/usr/share/apache-tomcat-7.0.59
>> -Dcatalina.home=/usr/share/apache-tomcat-7.0.59
>> -Djava.io.tmpdir=/usr/share/apache-tomcat-7.0.59/temp
>> org.apache.catalina.startup.Bootstrap start
>> ----
>>
>> Mar 24, 2015 11:18:37 AM
>> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor run
>> SEVERE: Unexpected death of background thread
>> ContainerBackgroundProcessor[StandardEngine[Catalina]]
>> java.lang.OutOfMemoryError: Java heap space
>>
>>        at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> Exception in thread
>> "ContainerBackgroundProcessor[StandardEngine[Catalina]]"
>> java.lang.OutOfMemoryError: Java heap space
>> Exception in thread "http-bio-8080-exec-5" Exception in thread
>> "http-bio-8080-exec-10" java.lang.OutOfMemoryError: Java heap space
>> java.lang.OutOfMemoryError: Java heap space
>> Exception in thread "http-bio-8080-exec-4" java.lang.OutOfMemoryError:
>> Java heap space
>> ----
>>
>> --
>> Scientific Programmer
>> Center for Computational Regulatory Genomics
>> Beckman Institute,
>> Biology and Biological Engineering
>> California Institute of Technology
>> http://www.echinobase.org/
>>
>
>
>


--
Scientific Programmer
Center for Computational Regulatory Genomics
Beckman Institute,
Biology and Biological Engineering
California Institute of Technology
http://www.echinobase.org/

Reply | Threaded
Open this post in threaded view
|

Re: heap space error

Monica Munoz-Torres
Dear Parul, 

By 5,000 sequences we mean to say that the current Apollo architecture allows for mostly uneventful searchers through ~5,000 "reference sequences" or fragments / scaffolds / groups. 

Do you mean to say that your assembly has 336K fragments?

cheers, 
~moni. 





-- 
Mentorship Matters!
--
Monica Munoz-Torres, PhD.
Berkeley Bioinformatics Open-source Projects (BBOP)
Genomics Division, Lawrence Berkeley National Laboratory


Mailing Address:
Joint Genome Institute (400-0415)
for the US Department of Energy
2800 Mitchell Dr. Ste 100
Walnut Creek, CA 94598-1631




On Tue, Mar 24, 2015 at 2:39 PM, Parul Kudtarkar <[hidden email]> wrote:
Dear Nathan,

Thanks for a quick response. You are correct, using the new interface and
either doing sequence search or doing search on recent changes gives heap
search error.
At the moment we are hosting single organism and the number of sequences
are 336K, far beyond what welapollo can host. Could you suggest
appropriate memory setting?

Thanks and regards,
Parul Kudtarkar

>
> Thanks for the report.  This is a known issue that will be completely
> address in 2.0, but we have to work around it in the interim.
>
> I think that the number one cause of this will likely be doing a search on
> the recent changes or sequence search if you have many sequences or
> changes or are hosting multiple organisms on the same tomcat.
>
> You can also try using the old interface (in the upper-right-hand side) of
> either page and see if that is any better.
>
> Does this seem consistent with what you are seeing or are you seeing the
> memory creep up in other places?
>
> If you know the number of sequences you have, we might be able to get a
> better memory setting to tune.  Typically with around 5K sequences we have
> no problems (other than the above mentioned).
>
> Nathan
>
>> On Mar 24, 2015, at 12:28 PM, Parul Kudtarkar <[hidden email]>
>> wrote:
>>
>> Dear Apollo community,
>>
>> I have set CATALINA_OPTS  as recommend by webapollo, even though
>> webapollo
>> is running, it's pretty slow and I get error messages logged into
>> catalina.out
>>
>> ps -ef | grep java
>> root     13952     1 99 12:11 pts/1    00:25:47 /usr/bin/java
>> -Djava.util.logging.config.file=/usr/share/apache-tomcat-7.0.59/conf/logging.properties
>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>> -Xms512m
>> -Xmx1g -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled
>> -XX:+UseConcMarkSweepGC -XX:MaxPermSize=256m
>> -Djava.endorsed.dirs=/usr/share/apache-tomcat-7.0.59/endorsed -classpath
>> /usr/share/apache-tomcat-7.0.59/bin/bootstrap.jar:/usr/share/apache-tomcat-7.0.59/bin/tomcat-juli.jar
>> -Dcatalina.base=/usr/share/apache-tomcat-7.0.59
>> -Dcatalina.home=/usr/share/apache-tomcat-7.0.59
>> -Djava.io.tmpdir=/usr/share/apache-tomcat-7.0.59/temp
>> org.apache.catalina.startup.Bootstrap start
>> ----
>>
>> Mar 24, 2015 11:18:37 AM
>> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor run
>> SEVERE: Unexpected death of background thread
>> ContainerBackgroundProcessor[StandardEngine[Catalina]]
>> java.lang.OutOfMemoryError: Java heap space
>>
>>        at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> Exception in thread
>> "ContainerBackgroundProcessor[StandardEngine[Catalina]]"
>> java.lang.OutOfMemoryError: Java heap space
>> Exception in thread "http-bio-8080-exec-5" Exception in thread
>> "http-bio-8080-exec-10" java.lang.OutOfMemoryError: Java heap space
>> java.lang.OutOfMemoryError: Java heap space
>> Exception in thread "http-bio-8080-exec-4" java.lang.OutOfMemoryError:
>> Java heap space
>> ----
>>
>> --
>> Scientific Programmer
>> Center for Computational Regulatory Genomics
>> Beckman Institute,
>> Biology and Biological Engineering
>> California Institute of Technology
>> http://www.echinobase.org/
>>
>
>
>


--
Scientific Programmer
Center for Computational Regulatory Genomics
Beckman Institute,
Biology and Biological Engineering
California Institute of Technology
http://www.echinobase.org/



Reply | Threaded
Open this post in threaded view
|

Re: heap space error

Parul Kudtarkar
Dear Monica,

This is a poorly assembled genome(Lytechinus variegatus) with 336K
Scaffolds aka reference sequences. In past though webapollo has
successfully handled our other genome(Strongylocentrotus purpuratus) with
30K Scaffolds.

Thanks and regards,
Parul Kudtarkar

> Dear Parul,
>
> By 5,000 sequences we mean to say that the current Apollo architecture
> allows for *mostly uneventful* searchers through ~5,000 "reference
> sequences" or fragments / scaffolds / groups.
>
> Do you mean to say that your assembly has 336K fragments?
>
> cheers,
> ~moni.
>
>
>
>
>
> --
> Mentorship Matters!
> --
> Monica Munoz-Torres, PhD.
> Berkeley Bioinformatics Open-source Projects (BBOP)
> Genomics Division, Lawrence Berkeley National Laboratory
>
> Mailing Address:
> Joint Genome Institute (400-0415)
> for the US Department of Energy
> 2800 Mitchell Dr. Ste 100
> Walnut Creek, CA 94598-1631
>
>
>
>
> On Tue, Mar 24, 2015 at 2:39 PM, Parul Kudtarkar <[hidden email]>
> wrote:
>
>> Dear Nathan,
>>
>> Thanks for a quick response. You are correct, using the new interface
>> and
>> either doing sequence search or doing search on recent changes gives
>> heap
>> search error.
>> At the moment we are hosting single organism and the number of sequences
>> are 336K, far beyond what welapollo can host. Could you suggest
>> appropriate memory setting?
>>
>> Thanks and regards,
>> Parul Kudtarkar
>>
>> >
>> > Thanks for the report.  This is a known issue that will be completely
>> > address in 2.0, but we have to work around it in the interim.
>> >
>> > I think that the number one cause of this will likely be doing a
>> search
>> on
>> > the recent changes or sequence search if you have many sequences or
>> > changes or are hosting multiple organisms on the same tomcat.
>> >
>> > You can also try using the old interface (in the upper-right-hand
>> side)
>> of
>> > either page and see if that is any better.
>> >
>> > Does this seem consistent with what you are seeing or are you seeing
>> the
>> > memory creep up in other places?
>> >
>> > If you know the number of sequences you have, we might be able to get
>> a
>> > better memory setting to tune.  Typically with around 5K sequences we
>> have
>> > no problems (other than the above mentioned).
>> >
>> > Nathan
>> >
>> >> On Mar 24, 2015, at 12:28 PM, Parul Kudtarkar <[hidden email]>
>> >> wrote:
>> >>
>> >> Dear Apollo community,
>> >>
>> >> I have set CATALINA_OPTS  as recommend by webapollo, even though
>> >> webapollo
>> >> is running, it's pretty slow and I get error messages logged into
>> >> catalina.out
>> >>
>> >> ps -ef | grep java
>> >> root     13952     1 99 12:11 pts/1    00:25:47 /usr/bin/java
>> >>
>> -Djava.util.logging.config.file=/usr/share/apache-tomcat-7.0.59/conf/logging.properties
>> >> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>> >> -Xms512m
>> >> -Xmx1g -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled
>> >> -XX:+UseConcMarkSweepGC -XX:MaxPermSize=256m
>> >> -Djava.endorsed.dirs=/usr/share/apache-tomcat-7.0.59/endorsed
>> -classpath
>> >>
>> /usr/share/apache-tomcat-7.0.59/bin/bootstrap.jar:/usr/share/apache-tomcat-7.0.59/bin/tomcat-juli.jar
>> >> -Dcatalina.base=/usr/share/apache-tomcat-7.0.59
>> >> -Dcatalina.home=/usr/share/apache-tomcat-7.0.59
>> >> -Djava.io.tmpdir=/usr/share/apache-tomcat-7.0.59/temp
>> >> org.apache.catalina.startup.Bootstrap start
>> >> ----
>> >>
>> >> Mar 24, 2015 11:18:37 AM
>> >> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor
>> run
>> >> SEVERE: Unexpected death of background thread
>> >> ContainerBackgroundProcessor[StandardEngine[Catalina]]
>> >> java.lang.OutOfMemoryError: Java heap space
>> >>
>> >>        at
>> >>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> >> Exception in thread
>> >> "ContainerBackgroundProcessor[StandardEngine[Catalina]]"
>> >> java.lang.OutOfMemoryError: Java heap space
>> >> Exception in thread "http-bio-8080-exec-5" Exception in thread
>> >> "http-bio-8080-exec-10" java.lang.OutOfMemoryError: Java heap space
>> >> java.lang.OutOfMemoryError: Java heap space
>> >> Exception in thread "http-bio-8080-exec-4"
>> java.lang.OutOfMemoryError:
>> >> Java heap space
>> >> ----
>> >>
>> >> --
>> >> Scientific Programmer
>> >> Center for Computational Regulatory Genomics
>> >> Beckman Institute,
>> >> Biology and Biological Engineering
>> >> California Institute of Technology
>> >> http://www.echinobase.org/
>> >>
>> >
>> >
>> >
>>
>>
>> --
>> Scientific Programmer
>> Center for Computational Regulatory Genomics
>> Beckman Institute,
>> Biology and Biological Engineering
>> California Institute of Technology
>> http://www.echinobase.org/
>>
>>
>


--
Scientific Programmer
Center for Computational Regulatory Genomics
Beckman Institute,
Biology and Biological Engineering
California Institute of Technology
http://www.echinobase.org/

Reply | Threaded
Open this post in threaded view
|

Re: heap space error

nathandunn

When you have annotations over that many scaffolds you’ll have to increase your memory or split them up across separate genomes (at least for doing the searches).

With 2.0 we’ve fixed this for any size, but the alpha won’t be out for a couple of months.

Nathan


> On Mar 24, 2015, at 4:40 PM, Parul Kudtarkar <[hidden email]> wrote:
>
> Dear Monica,
>
> This is a poorly assembled genome(Lytechinus variegatus) with 336K
> Scaffolds aka reference sequences. In past though webapollo has
> successfully handled our other genome(Strongylocentrotus purpuratus) with
> 30K Scaffolds.
>
> Thanks and regards,
> Parul Kudtarkar
>
>> Dear Parul,
>>
>> By 5,000 sequences we mean to say that the current Apollo architecture
>> allows for *mostly uneventful* searchers through ~5,000 "reference
>> sequences" or fragments / scaffolds / groups.
>>
>> Do you mean to say that your assembly has 336K fragments?
>>
>> cheers,
>> ~moni.
>>
>>
>>
>>
>>
>> --
>> Mentorship Matters!
>> --
>> Monica Munoz-Torres, PhD.
>> Berkeley Bioinformatics Open-source Projects (BBOP)
>> Genomics Division, Lawrence Berkeley National Laboratory
>>
>> Mailing Address:
>> Joint Genome Institute (400-0415)
>> for the US Department of Energy
>> 2800 Mitchell Dr. Ste 100
>> Walnut Creek, CA 94598-1631
>>
>>
>>
>>
>> On Tue, Mar 24, 2015 at 2:39 PM, Parul Kudtarkar <[hidden email]>
>> wrote:
>>
>>> Dear Nathan,
>>>
>>> Thanks for a quick response. You are correct, using the new interface
>>> and
>>> either doing sequence search or doing search on recent changes gives
>>> heap
>>> search error.
>>> At the moment we are hosting single organism and the number of sequences
>>> are 336K, far beyond what welapollo can host. Could you suggest
>>> appropriate memory setting?
>>>
>>> Thanks and regards,
>>> Parul Kudtarkar
>>>
>>>>
>>>> Thanks for the report.  This is a known issue that will be completely
>>>> address in 2.0, but we have to work around it in the interim.
>>>>
>>>> I think that the number one cause of this will likely be doing a
>>> search
>>> on
>>>> the recent changes or sequence search if you have many sequences or
>>>> changes or are hosting multiple organisms on the same tomcat.
>>>>
>>>> You can also try using the old interface (in the upper-right-hand
>>> side)
>>> of
>>>> either page and see if that is any better.
>>>>
>>>> Does this seem consistent with what you are seeing or are you seeing
>>> the
>>>> memory creep up in other places?
>>>>
>>>> If you know the number of sequences you have, we might be able to get
>>> a
>>>> better memory setting to tune.  Typically with around 5K sequences we
>>> have
>>>> no problems (other than the above mentioned).
>>>>
>>>> Nathan
>>>>
>>>>> On Mar 24, 2015, at 12:28 PM, Parul Kudtarkar <[hidden email]>
>>>>> wrote:
>>>>>
>>>>> Dear Apollo community,
>>>>>
>>>>> I have set CATALINA_OPTS  as recommend by webapollo, even though
>>>>> webapollo
>>>>> is running, it's pretty slow and I get error messages logged into
>>>>> catalina.out
>>>>>
>>>>> ps -ef | grep java
>>>>> root     13952     1 99 12:11 pts/1    00:25:47 /usr/bin/java
>>>>>
>>> -Djava.util.logging.config.file=/usr/share/apache-tomcat-7.0.59/conf/logging.properties
>>>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>>>>> -Xms512m
>>>>> -Xmx1g -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled
>>>>> -XX:+UseConcMarkSweepGC -XX:MaxPermSize=256m
>>>>> -Djava.endorsed.dirs=/usr/share/apache-tomcat-7.0.59/endorsed
>>> -classpath
>>>>>
>>> /usr/share/apache-tomcat-7.0.59/bin/bootstrap.jar:/usr/share/apache-tomcat-7.0.59/bin/tomcat-juli.jar
>>>>> -Dcatalina.base=/usr/share/apache-tomcat-7.0.59
>>>>> -Dcatalina.home=/usr/share/apache-tomcat-7.0.59
>>>>> -Djava.io.tmpdir=/usr/share/apache-tomcat-7.0.59/temp
>>>>> org.apache.catalina.startup.Bootstrap start
>>>>> ----
>>>>>
>>>>> Mar 24, 2015 11:18:37 AM
>>>>> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor
>>> run
>>>>> SEVERE: Unexpected death of background thread
>>>>> ContainerBackgroundProcessor[StandardEngine[Catalina]]
>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>
>>>>>       at
>>>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>> Exception in thread
>>>>> "ContainerBackgroundProcessor[StandardEngine[Catalina]]"
>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>> Exception in thread "http-bio-8080-exec-5" Exception in thread
>>>>> "http-bio-8080-exec-10" java.lang.OutOfMemoryError: Java heap space
>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>> Exception in thread "http-bio-8080-exec-4"
>>> java.lang.OutOfMemoryError:
>>>>> Java heap space
>>>>> ----
>>>>>
>>>>> --
>>>>> Scientific Programmer
>>>>> Center for Computational Regulatory Genomics
>>>>> Beckman Institute,
>>>>> Biology and Biological Engineering
>>>>> California Institute of Technology
>>>>> http://www.echinobase.org/
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Scientific Programmer
>>> Center for Computational Regulatory Genomics
>>> Beckman Institute,
>>> Biology and Biological Engineering
>>> California Institute of Technology
>>> http://www.echinobase.org/
>>>
>>>
>>
>
>
> --
> Scientific Programmer
> Center for Computational Regulatory Genomics
> Beckman Institute,
> Biology and Biological Engineering
> California Institute of Technology
> http://www.echinobase.org/
>


Reply | Threaded
Open this post in threaded view
|

Re: heap space error

Parul Kudtarkar
Dear Nathan,

Thanks, what is the recommended increase in memory size? Could you give
exact parameters for CATALINA_OPTS or do we keep tuning till we get it
correct.
Looking forward to 2.0 WebApollo.

Best,
Parul Kudtarkar

>
> When you have annotations over that many scaffolds you’ll have to increase
> your memory or split them up across separate genomes (at least for doing
> the searches).
>
> With 2.0 we’ve fixed this for any size, but the alpha won’t be out for a
> couple of months.
>
> Nathan
>
>
>> On Mar 24, 2015, at 4:40 PM, Parul Kudtarkar <[hidden email]> wrote:
>>
>> Dear Monica,
>>
>> This is a poorly assembled genome(Lytechinus variegatus) with 336K
>> Scaffolds aka reference sequences. In past though webapollo has
>> successfully handled our other genome(Strongylocentrotus purpuratus)
>> with
>> 30K Scaffolds.
>>
>> Thanks and regards,
>> Parul Kudtarkar
>>
>>> Dear Parul,
>>>
>>> By 5,000 sequences we mean to say that the current Apollo architecture
>>> allows for *mostly uneventful* searchers through ~5,000 "reference
>>> sequences" or fragments / scaffolds / groups.
>>>
>>> Do you mean to say that your assembly has 336K fragments?
>>>
>>> cheers,
>>> ~moni.
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Mentorship Matters!
>>> --
>>> Monica Munoz-Torres, PhD.
>>> Berkeley Bioinformatics Open-source Projects (BBOP)
>>> Genomics Division, Lawrence Berkeley National Laboratory
>>>
>>> Mailing Address:
>>> Joint Genome Institute (400-0415)
>>> for the US Department of Energy
>>> 2800 Mitchell Dr. Ste 100
>>> Walnut Creek, CA 94598-1631
>>>
>>>
>>>
>>>
>>> On Tue, Mar 24, 2015 at 2:39 PM, Parul Kudtarkar <[hidden email]>
>>> wrote:
>>>
>>>> Dear Nathan,
>>>>
>>>> Thanks for a quick response. You are correct, using the new interface
>>>> and
>>>> either doing sequence search or doing search on recent changes gives
>>>> heap
>>>> search error.
>>>> At the moment we are hosting single organism and the number of
>>>> sequences
>>>> are 336K, far beyond what welapollo can host. Could you suggest
>>>> appropriate memory setting?
>>>>
>>>> Thanks and regards,
>>>> Parul Kudtarkar
>>>>
>>>>>
>>>>> Thanks for the report.  This is a known issue that will be completely
>>>>> address in 2.0, but we have to work around it in the interim.
>>>>>
>>>>> I think that the number one cause of this will likely be doing a
>>>> search
>>>> on
>>>>> the recent changes or sequence search if you have many sequences or
>>>>> changes or are hosting multiple organisms on the same tomcat.
>>>>>
>>>>> You can also try using the old interface (in the upper-right-hand
>>>> side)
>>>> of
>>>>> either page and see if that is any better.
>>>>>
>>>>> Does this seem consistent with what you are seeing or are you seeing
>>>> the
>>>>> memory creep up in other places?
>>>>>
>>>>> If you know the number of sequences you have, we might be able to get
>>>> a
>>>>> better memory setting to tune.  Typically with around 5K sequences we
>>>> have
>>>>> no problems (other than the above mentioned).
>>>>>
>>>>> Nathan
>>>>>
>>>>>> On Mar 24, 2015, at 12:28 PM, Parul Kudtarkar <[hidden email]>
>>>>>> wrote:
>>>>>>
>>>>>> Dear Apollo community,
>>>>>>
>>>>>> I have set CATALINA_OPTS  as recommend by webapollo, even though
>>>>>> webapollo
>>>>>> is running, it's pretty slow and I get error messages logged into
>>>>>> catalina.out
>>>>>>
>>>>>> ps -ef | grep java
>>>>>> root     13952     1 99 12:11 pts/1    00:25:47 /usr/bin/java
>>>>>>
>>>> -Djava.util.logging.config.file=/usr/share/apache-tomcat-7.0.59/conf/logging.properties
>>>>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>>>>>> -Xms512m
>>>>>> -Xmx1g -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled
>>>>>> -XX:+UseConcMarkSweepGC -XX:MaxPermSize=256m
>>>>>> -Djava.endorsed.dirs=/usr/share/apache-tomcat-7.0.59/endorsed
>>>> -classpath
>>>>>>
>>>> /usr/share/apache-tomcat-7.0.59/bin/bootstrap.jar:/usr/share/apache-tomcat-7.0.59/bin/tomcat-juli.jar
>>>>>> -Dcatalina.base=/usr/share/apache-tomcat-7.0.59
>>>>>> -Dcatalina.home=/usr/share/apache-tomcat-7.0.59
>>>>>> -Djava.io.tmpdir=/usr/share/apache-tomcat-7.0.59/temp
>>>>>> org.apache.catalina.startup.Bootstrap start
>>>>>> ----
>>>>>>
>>>>>> Mar 24, 2015 11:18:37 AM
>>>>>> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor
>>>> run
>>>>>> SEVERE: Unexpected death of background thread
>>>>>> ContainerBackgroundProcessor[StandardEngine[Catalina]]
>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>>
>>>>>>       at
>>>>>>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>> Exception in thread
>>>>>> "ContainerBackgroundProcessor[StandardEngine[Catalina]]"
>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>> Exception in thread "http-bio-8080-exec-5" Exception in thread
>>>>>> "http-bio-8080-exec-10" java.lang.OutOfMemoryError: Java heap space
>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>> Exception in thread "http-bio-8080-exec-4"
>>>> java.lang.OutOfMemoryError:
>>>>>> Java heap space
>>>>>> ----
>>>>>>
>>>>>> --
>>>>>> Scientific Programmer
>>>>>> Center for Computational Regulatory Genomics
>>>>>> Beckman Institute,
>>>>>> Biology and Biological Engineering
>>>>>> California Institute of Technology
>>>>>> http://www.echinobase.org/
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Scientific Programmer
>>>> Center for Computational Regulatory Genomics
>>>> Beckman Institute,
>>>> Biology and Biological Engineering
>>>> California Institute of Technology
>>>> http://www.echinobase.org/
>>>>
>>>>
>>>
>>
>>
>> --
>> Scientific Programmer
>> Center for Computational Regulatory Genomics
>> Beckman Institute,
>> Biology and Biological Engineering
>> California Institute of Technology
>> http://www.echinobase.org/
>>
>
>
>


--
Scientific Programmer
Center for Computational Regulatory Genomics
Beckman Institute,
Biology and Biological Engineering
California Institute of Technology
http://www.echinobase.org/

Reply | Threaded
Open this post in threaded view
|

Re: heap space error

nathandunn

I think you have to just tune it.   Mostly I would increase the max heap (Xmx).   Other folks in the community have more insight into working with fractured sequences.  

Nathan

> On Mar 24, 2015, at 5:20 PM, Parul Kudtarkar <[hidden email]> wrote:
>
> Dear Nathan,
>
> Thanks, what is the recommended increase in memory size? Could you give
> exact parameters for CATALINA_OPTS or do we keep tuning till we get it
> correct.
> Looking forward to 2.0 WebApollo.
>
> Best,
> Parul Kudtarkar
>
>>
>> When you have annotations over that many scaffolds you’ll have to increase
>> your memory or split them up across separate genomes (at least for doing
>> the searches).
>>
>> With 2.0 we’ve fixed this for any size, but the alpha won’t be out for a
>> couple of months.
>>
>> Nathan
>>
>>
>>> On Mar 24, 2015, at 4:40 PM, Parul Kudtarkar <[hidden email]> wrote:
>>>
>>> Dear Monica,
>>>
>>> This is a poorly assembled genome(Lytechinus variegatus) with 336K
>>> Scaffolds aka reference sequences. In past though webapollo has
>>> successfully handled our other genome(Strongylocentrotus purpuratus)
>>> with
>>> 30K Scaffolds.
>>>
>>> Thanks and regards,
>>> Parul Kudtarkar
>>>
>>>> Dear Parul,
>>>>
>>>> By 5,000 sequences we mean to say that the current Apollo architecture
>>>> allows for *mostly uneventful* searchers through ~5,000 "reference
>>>> sequences" or fragments / scaffolds / groups.
>>>>
>>>> Do you mean to say that your assembly has 336K fragments?
>>>>
>>>> cheers,
>>>> ~moni.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Mentorship Matters!
>>>> --
>>>> Monica Munoz-Torres, PhD.
>>>> Berkeley Bioinformatics Open-source Projects (BBOP)
>>>> Genomics Division, Lawrence Berkeley National Laboratory
>>>>
>>>> Mailing Address:
>>>> Joint Genome Institute (400-0415)
>>>> for the US Department of Energy
>>>> 2800 Mitchell Dr. Ste 100
>>>> Walnut Creek, CA 94598-1631
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, Mar 24, 2015 at 2:39 PM, Parul Kudtarkar <[hidden email]>
>>>> wrote:
>>>>
>>>>> Dear Nathan,
>>>>>
>>>>> Thanks for a quick response. You are correct, using the new interface
>>>>> and
>>>>> either doing sequence search or doing search on recent changes gives
>>>>> heap
>>>>> search error.
>>>>> At the moment we are hosting single organism and the number of
>>>>> sequences
>>>>> are 336K, far beyond what welapollo can host. Could you suggest
>>>>> appropriate memory setting?
>>>>>
>>>>> Thanks and regards,
>>>>> Parul Kudtarkar
>>>>>
>>>>>>
>>>>>> Thanks for the report.  This is a known issue that will be completely
>>>>>> address in 2.0, but we have to work around it in the interim.
>>>>>>
>>>>>> I think that the number one cause of this will likely be doing a
>>>>> search
>>>>> on
>>>>>> the recent changes or sequence search if you have many sequences or
>>>>>> changes or are hosting multiple organisms on the same tomcat.
>>>>>>
>>>>>> You can also try using the old interface (in the upper-right-hand
>>>>> side)
>>>>> of
>>>>>> either page and see if that is any better.
>>>>>>
>>>>>> Does this seem consistent with what you are seeing or are you seeing
>>>>> the
>>>>>> memory creep up in other places?
>>>>>>
>>>>>> If you know the number of sequences you have, we might be able to get
>>>>> a
>>>>>> better memory setting to tune.  Typically with around 5K sequences we
>>>>> have
>>>>>> no problems (other than the above mentioned).
>>>>>>
>>>>>> Nathan
>>>>>>
>>>>>>> On Mar 24, 2015, at 12:28 PM, Parul Kudtarkar <[hidden email]>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Dear Apollo community,
>>>>>>>
>>>>>>> I have set CATALINA_OPTS  as recommend by webapollo, even though
>>>>>>> webapollo
>>>>>>> is running, it's pretty slow and I get error messages logged into
>>>>>>> catalina.out
>>>>>>>
>>>>>>> ps -ef | grep java
>>>>>>> root     13952     1 99 12:11 pts/1    00:25:47 /usr/bin/java
>>>>>>>
>>>>> -Djava.util.logging.config.file=/usr/share/apache-tomcat-7.0.59/conf/logging.properties
>>>>>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>>>>>>> -Xms512m
>>>>>>> -Xmx1g -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled
>>>>>>> -XX:+UseConcMarkSweepGC -XX:MaxPermSize=256m
>>>>>>> -Djava.endorsed.dirs=/usr/share/apache-tomcat-7.0.59/endorsed
>>>>> -classpath
>>>>>>>
>>>>> /usr/share/apache-tomcat-7.0.59/bin/bootstrap.jar:/usr/share/apache-tomcat-7.0.59/bin/tomcat-juli.jar
>>>>>>> -Dcatalina.base=/usr/share/apache-tomcat-7.0.59
>>>>>>> -Dcatalina.home=/usr/share/apache-tomcat-7.0.59
>>>>>>> -Djava.io.tmpdir=/usr/share/apache-tomcat-7.0.59/temp
>>>>>>> org.apache.catalina.startup.Bootstrap start
>>>>>>> ----
>>>>>>>
>>>>>>> Mar 24, 2015 11:18:37 AM
>>>>>>> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor
>>>>> run
>>>>>>> SEVERE: Unexpected death of background thread
>>>>>>> ContainerBackgroundProcessor[StandardEngine[Catalina]]
>>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>>>
>>>>>>>      at
>>>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>> Exception in thread
>>>>>>> "ContainerBackgroundProcessor[StandardEngine[Catalina]]"
>>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>>> Exception in thread "http-bio-8080-exec-5" Exception in thread
>>>>>>> "http-bio-8080-exec-10" java.lang.OutOfMemoryError: Java heap space
>>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>>> Exception in thread "http-bio-8080-exec-4"
>>>>> java.lang.OutOfMemoryError:
>>>>>>> Java heap space
>>>>>>> ----
>>>>>>>
>>>>>>> --
>>>>>>> Scientific Programmer
>>>>>>> Center for Computational Regulatory Genomics
>>>>>>> Beckman Institute,
>>>>>>> Biology and Biological Engineering
>>>>>>> California Institute of Technology
>>>>>>> http://www.echinobase.org/
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Scientific Programmer
>>>>> Center for Computational Regulatory Genomics
>>>>> Beckman Institute,
>>>>> Biology and Biological Engineering
>>>>> California Institute of Technology
>>>>> http://www.echinobase.org/
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Scientific Programmer
>>> Center for Computational Regulatory Genomics
>>> Beckman Institute,
>>> Biology and Biological Engineering
>>> California Institute of Technology
>>> http://www.echinobase.org/
>>>
>>
>>
>>
>
>
> --
> Scientific Programmer
> Center for Computational Regulatory Genomics
> Beckman Institute,
> Biology and Biological Engineering
> California Institute of Technology
> http://www.echinobase.org/
>


Reply | Threaded
Open this post in threaded view
|

Re: heap space error

Parul Kudtarkar
Thanks once again, looking forward to more insights from the community.

>
> I think you have to just tune it.   Mostly I would increase the max heap
> (Xmx).   Other folks in the community have more insight into working with
> fractured sequences.
>
> Nathan
>
>> On Mar 24, 2015, at 5:20 PM, Parul Kudtarkar <[hidden email]> wrote:
>>
>> Dear Nathan,
>>
>> Thanks, what is the recommended increase in memory size? Could you give
>> exact parameters for CATALINA_OPTS or do we keep tuning till we get it
>> correct.
>> Looking forward to 2.0 WebApollo.
>>
>> Best,
>> Parul Kudtarkar
>>
>>>
>>> When you have annotations over that many scaffolds you’ll have to
>>> increase
>>> your memory or split them up across separate genomes (at least for
>>> doing
>>> the searches).
>>>
>>> With 2.0 we’ve fixed this for any size, but the alpha won’t be out for
>>> a
>>> couple of months.
>>>
>>> Nathan
>>>
>>>
>>>> On Mar 24, 2015, at 4:40 PM, Parul Kudtarkar <[hidden email]>
>>>> wrote:
>>>>
>>>> Dear Monica,
>>>>
>>>> This is a poorly assembled genome(Lytechinus variegatus) with 336K
>>>> Scaffolds aka reference sequences. In past though webapollo has
>>>> successfully handled our other genome(Strongylocentrotus purpuratus)
>>>> with
>>>> 30K Scaffolds.
>>>>
>>>> Thanks and regards,
>>>> Parul Kudtarkar
>>>>
>>>>> Dear Parul,
>>>>>
>>>>> By 5,000 sequences we mean to say that the current Apollo
>>>>> architecture
>>>>> allows for *mostly uneventful* searchers through ~5,000 "reference
>>>>> sequences" or fragments / scaffolds / groups.
>>>>>
>>>>> Do you mean to say that your assembly has 336K fragments?
>>>>>
>>>>> cheers,
>>>>> ~moni.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Mentorship Matters!
>>>>> --
>>>>> Monica Munoz-Torres, PhD.
>>>>> Berkeley Bioinformatics Open-source Projects (BBOP)
>>>>> Genomics Division, Lawrence Berkeley National Laboratory
>>>>>
>>>>> Mailing Address:
>>>>> Joint Genome Institute (400-0415)
>>>>> for the US Department of Energy
>>>>> 2800 Mitchell Dr. Ste 100
>>>>> Walnut Creek, CA 94598-1631
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Mar 24, 2015 at 2:39 PM, Parul Kudtarkar <[hidden email]>
>>>>> wrote:
>>>>>
>>>>>> Dear Nathan,
>>>>>>
>>>>>> Thanks for a quick response. You are correct, using the new
>>>>>> interface
>>>>>> and
>>>>>> either doing sequence search or doing search on recent changes gives
>>>>>> heap
>>>>>> search error.
>>>>>> At the moment we are hosting single organism and the number of
>>>>>> sequences
>>>>>> are 336K, far beyond what welapollo can host. Could you suggest
>>>>>> appropriate memory setting?
>>>>>>
>>>>>> Thanks and regards,
>>>>>> Parul Kudtarkar
>>>>>>
>>>>>>>
>>>>>>> Thanks for the report.  This is a known issue that will be
>>>>>>> completely
>>>>>>> address in 2.0, but we have to work around it in the interim.
>>>>>>>
>>>>>>> I think that the number one cause of this will likely be doing a
>>>>>> search
>>>>>> on
>>>>>>> the recent changes or sequence search if you have many sequences or
>>>>>>> changes or are hosting multiple organisms on the same tomcat.
>>>>>>>
>>>>>>> You can also try using the old interface (in the upper-right-hand
>>>>>> side)
>>>>>> of
>>>>>>> either page and see if that is any better.
>>>>>>>
>>>>>>> Does this seem consistent with what you are seeing or are you
>>>>>>> seeing
>>>>>> the
>>>>>>> memory creep up in other places?
>>>>>>>
>>>>>>> If you know the number of sequences you have, we might be able to
>>>>>>> get
>>>>>> a
>>>>>>> better memory setting to tune.  Typically with around 5K sequences
>>>>>>> we
>>>>>> have
>>>>>>> no problems (other than the above mentioned).
>>>>>>>
>>>>>>> Nathan
>>>>>>>
>>>>>>>> On Mar 24, 2015, at 12:28 PM, Parul Kudtarkar <[hidden email]>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Dear Apollo community,
>>>>>>>>
>>>>>>>> I have set CATALINA_OPTS  as recommend by webapollo, even though
>>>>>>>> webapollo
>>>>>>>> is running, it's pretty slow and I get error messages logged into
>>>>>>>> catalina.out
>>>>>>>>
>>>>>>>> ps -ef | grep java
>>>>>>>> root     13952     1 99 12:11 pts/1    00:25:47 /usr/bin/java
>>>>>>>>
>>>>>> -Djava.util.logging.config.file=/usr/share/apache-tomcat-7.0.59/conf/logging.properties
>>>>>>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>>>>>>>> -Xms512m
>>>>>>>> -Xmx1g -XX:+CMSClassUnloadingEnabled
>>>>>>>> -XX:+CMSPermGenSweepingEnabled
>>>>>>>> -XX:+UseConcMarkSweepGC -XX:MaxPermSize=256m
>>>>>>>> -Djava.endorsed.dirs=/usr/share/apache-tomcat-7.0.59/endorsed
>>>>>> -classpath
>>>>>>>>
>>>>>> /usr/share/apache-tomcat-7.0.59/bin/bootstrap.jar:/usr/share/apache-tomcat-7.0.59/bin/tomcat-juli.jar
>>>>>>>> -Dcatalina.base=/usr/share/apache-tomcat-7.0.59
>>>>>>>> -Dcatalina.home=/usr/share/apache-tomcat-7.0.59
>>>>>>>> -Djava.io.tmpdir=/usr/share/apache-tomcat-7.0.59/temp
>>>>>>>> org.apache.catalina.startup.Bootstrap start
>>>>>>>> ----
>>>>>>>>
>>>>>>>> Mar 24, 2015 11:18:37 AM
>>>>>>>> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor
>>>>>> run
>>>>>>>> SEVERE: Unexpected death of background thread
>>>>>>>> ContainerBackgroundProcessor[StandardEngine[Catalina]]
>>>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>>>>
>>>>>>>>      at
>>>>>>>>
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>>> Exception in thread
>>>>>>>> "ContainerBackgroundProcessor[StandardEngine[Catalina]]"
>>>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>>>> Exception in thread "http-bio-8080-exec-5" Exception in thread
>>>>>>>> "http-bio-8080-exec-10" java.lang.OutOfMemoryError: Java heap
>>>>>>>> space
>>>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>>>> Exception in thread "http-bio-8080-exec-4"
>>>>>> java.lang.OutOfMemoryError:
>>>>>>>> Java heap space
>>>>>>>> ----
>>>>>>>>
>>>>>>>> --
>>>>>>>> Scientific Programmer
>>>>>>>> Center for Computational Regulatory Genomics
>>>>>>>> Beckman Institute,
>>>>>>>> Biology and Biological Engineering
>>>>>>>> California Institute of Technology
>>>>>>>> http://www.echinobase.org/
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Scientific Programmer
>>>>>> Center for Computational Regulatory Genomics
>>>>>> Beckman Institute,
>>>>>> Biology and Biological Engineering
>>>>>> California Institute of Technology
>>>>>> http://www.echinobase.org/
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Scientific Programmer
>>>> Center for Computational Regulatory Genomics
>>>> Beckman Institute,
>>>> Biology and Biological Engineering
>>>> California Institute of Technology
>>>> http://www.echinobase.org/
>>>>
>>>
>>>
>>>
>>
>>
>> --
>> Scientific Programmer
>> Center for Computational Regulatory Genomics
>> Beckman Institute,
>> Biology and Biological Engineering
>> California Institute of Technology
>> http://www.echinobase.org/
>>
>
>
>


--
Scientific Programmer
Center for Computational Regulatory Genomics
Beckman Institute,
Biology and Biological Engineering
California Institute of Technology
http://www.echinobase.org/

Reply | Threaded
Open this post in threaded view
|

Re: heap space error

Chris Childers
Hi Parul,

Here at the i5k Workspace@NAL, we have 45 Web Apollo instances running, and we learned we needed to tune the memory and garbage collection to help keep the system stable.  I'm not sure we will ever stop tuning the system, but these settings seem to work pretty well for us:

 -Xmx12288m
-Xms8192m
-XX:PermSize=256m
-XX:MaxPermSize=1024m
-XX:ReservedCodeCacheSize=64m
-XX:+UseG1GC
-XX:+CMSClassUnloadingEnabled
-Xloggc:$CATALINA_HOME/logs/gc.log
-XX:+PrintHeapAtGC
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps

Hope this helps,
Chris


On Tue, Mar 24, 2015 at 8:24 PM, Parul Kudtarkar <[hidden email]> wrote:
Thanks once again, looking forward to more insights from the community.

>
> I think you have to just tune it.   Mostly I would increase the max heap
> (Xmx).   Other folks in the community have more insight into working with
> fractured sequences.
>
> Nathan
>
>> On Mar 24, 2015, at 5:20 PM, Parul Kudtarkar <[hidden email]> wrote:
>>
>> Dear Nathan,
>>
>> Thanks, what is the recommended increase in memory size? Could you give
>> exact parameters for CATALINA_OPTS or do we keep tuning till we get it
>> correct.
>> Looking forward to 2.0 WebApollo.
>>
>> Best,
>> Parul Kudtarkar
>>
>>>
>>> When you have annotations over that many scaffolds you’ll have to
>>> increase
>>> your memory or split them up across separate genomes (at least for
>>> doing
>>> the searches).
>>>
>>> With 2.0 we’ve fixed this for any size, but the alpha won’t be out for
>>> a
>>> couple of months.
>>>
>>> Nathan
>>>
>>>
>>>> On Mar 24, 2015, at 4:40 PM, Parul Kudtarkar <[hidden email]>
>>>> wrote:
>>>>
>>>> Dear Monica,
>>>>
>>>> This is a poorly assembled genome(Lytechinus variegatus) with 336K
>>>> Scaffolds aka reference sequences. In past though webapollo has
>>>> successfully handled our other genome(Strongylocentrotus purpuratus)
>>>> with
>>>> 30K Scaffolds.
>>>>
>>>> Thanks and regards,
>>>> Parul Kudtarkar
>>>>
>>>>> Dear Parul,
>>>>>
>>>>> By 5,000 sequences we mean to say that the current Apollo
>>>>> architecture
>>>>> allows for *mostly uneventful* searchers through ~5,000 "reference
>>>>> sequences" or fragments / scaffolds / groups.
>>>>>
>>>>> Do you mean to say that your assembly has 336K fragments?
>>>>>
>>>>> cheers,
>>>>> ~moni.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Mentorship Matters!
>>>>> --
>>>>> Monica Munoz-Torres, PhD.
>>>>> Berkeley Bioinformatics Open-source Projects (BBOP)
>>>>> Genomics Division, Lawrence Berkeley National Laboratory
>>>>>
>>>>> Mailing Address:
>>>>> Joint Genome Institute (400-0415)
>>>>> for the US Department of Energy
>>>>> 2800 Mitchell Dr. Ste 100
>>>>> Walnut Creek, CA 94598-1631
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Mar 24, 2015 at 2:39 PM, Parul Kudtarkar <[hidden email]>
>>>>> wrote:
>>>>>
>>>>>> Dear Nathan,
>>>>>>
>>>>>> Thanks for a quick response. You are correct, using the new
>>>>>> interface
>>>>>> and
>>>>>> either doing sequence search or doing search on recent changes gives
>>>>>> heap
>>>>>> search error.
>>>>>> At the moment we are hosting single organism and the number of
>>>>>> sequences
>>>>>> are 336K, far beyond what welapollo can host. Could you suggest
>>>>>> appropriate memory setting?
>>>>>>
>>>>>> Thanks and regards,
>>>>>> Parul Kudtarkar
>>>>>>
>>>>>>>
>>>>>>> Thanks for the report.  This is a known issue that will be
>>>>>>> completely
>>>>>>> address in 2.0, but we have to work around it in the interim.
>>>>>>>
>>>>>>> I think that the number one cause of this will likely be doing a
>>>>>> search
>>>>>> on
>>>>>>> the recent changes or sequence search if you have many sequences or
>>>>>>> changes or are hosting multiple organisms on the same tomcat.
>>>>>>>
>>>>>>> You can also try using the old interface (in the upper-right-hand
>>>>>> side)
>>>>>> of
>>>>>>> either page and see if that is any better.
>>>>>>>
>>>>>>> Does this seem consistent with what you are seeing or are you
>>>>>>> seeing
>>>>>> the
>>>>>>> memory creep up in other places?
>>>>>>>
>>>>>>> If you know the number of sequences you have, we might be able to
>>>>>>> get
>>>>>> a
>>>>>>> better memory setting to tune.  Typically with around 5K sequences
>>>>>>> we
>>>>>> have
>>>>>>> no problems (other than the above mentioned).
>>>>>>>
>>>>>>> Nathan
>>>>>>>
>>>>>>>> On Mar 24, 2015, at 12:28 PM, Parul Kudtarkar <[hidden email]>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Dear Apollo community,
>>>>>>>>
>>>>>>>> I have set CATALINA_OPTS  as recommend by webapollo, even though
>>>>>>>> webapollo
>>>>>>>> is running, it's pretty slow and I get error messages logged into
>>>>>>>> catalina.out
>>>>>>>>
>>>>>>>> ps -ef | grep java
>>>>>>>> root     13952     1 99 12:11 pts/1    00:25:47 /usr/bin/java
>>>>>>>>
>>>>>> -Djava.util.logging.config.file=/usr/share/apache-tomcat-7.0.59/conf/logging.properties
>>>>>>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>>>>>>>> -Xms512m
>>>>>>>> -Xmx1g -XX:+CMSClassUnloadingEnabled
>>>>>>>> -XX:+CMSPermGenSweepingEnabled
>>>>>>>> -XX:+UseConcMarkSweepGC -XX:MaxPermSize=256m
>>>>>>>> -Djava.endorsed.dirs=/usr/share/apache-tomcat-7.0.59/endorsed
>>>>>> -classpath
>>>>>>>>
>>>>>> /usr/share/apache-tomcat-7.0.59/bin/bootstrap.jar:/usr/share/apache-tomcat-7.0.59/bin/tomcat-juli.jar
>>>>>>>> -Dcatalina.base=/usr/share/apache-tomcat-7.0.59
>>>>>>>> -Dcatalina.home=/usr/share/apache-tomcat-7.0.59
>>>>>>>> -Djava.io.tmpdir=/usr/share/apache-tomcat-7.0.59/temp
>>>>>>>> org.apache.catalina.startup.Bootstrap start
>>>>>>>> ----
>>>>>>>>
>>>>>>>> Mar 24, 2015 11:18:37 AM
>>>>>>>> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor
>>>>>> run
>>>>>>>> SEVERE: Unexpected death of background thread
>>>>>>>> ContainerBackgroundProcessor[StandardEngine[Catalina]]
>>>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>>>>
>>>>>>>>      at
>>>>>>>>
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>>> Exception in thread
>>>>>>>> "ContainerBackgroundProcessor[StandardEngine[Catalina]]"
>>>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>>>> Exception in thread "http-bio-8080-exec-5" Exception in thread
>>>>>>>> "http-bio-8080-exec-10" java.lang.OutOfMemoryError: Java heap
>>>>>>>> space
>>>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>>>> Exception in thread "http-bio-8080-exec-4"
>>>>>> java.lang.OutOfMemoryError:
>>>>>>>> Java heap space
>>>>>>>> ----
>>>>>>>>
>>>>>>>> --
>>>>>>>> Scientific Programmer
>>>>>>>> Center for Computational Regulatory Genomics
>>>>>>>> Beckman Institute,
>>>>>>>> Biology and Biological Engineering
>>>>>>>> California Institute of Technology
>>>>>>>> http://www.echinobase.org/
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Scientific Programmer
>>>>>> Center for Computational Regulatory Genomics
>>>>>> Beckman Institute,
>>>>>> Biology and Biological Engineering
>>>>>> California Institute of Technology
>>>>>> http://www.echinobase.org/
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Scientific Programmer
>>>> Center for Computational Regulatory Genomics
>>>> Beckman Institute,
>>>> Biology and Biological Engineering
>>>> California Institute of Technology
>>>> http://www.echinobase.org/
>>>>
>>>
>>>
>>>
>>
>>
>> --
>> Scientific Programmer
>> Center for Computational Regulatory Genomics
>> Beckman Institute,
>> Biology and Biological Engineering
>> California Institute of Technology
>> http://www.echinobase.org/
>>
>
>
>


--
Scientific Programmer
Center for Computational Regulatory Genomics
Beckman Institute,
Biology and Biological Engineering
California Institute of Technology
http://www.echinobase.org/


Reply | Threaded
Open this post in threaded view
|

Re: heap space error

Parul Kudtarkar
Dear Chris,

Thanks for sharing memory setting, these work great for our other genome
which is 60K in size(reference sequences/Scaffold count)

> Hi Parul,
>
> Here at the i5k Workspace@NAL, we have 45 Web Apollo instances running,
> and
> we learned we needed to tune the memory and garbage collection to help
> keep
> the system stable.  I'm not sure we will ever stop tuning the system, but
> these settings seem to work pretty well for us:
>
>  -Xmx12288m
> -Xms8192m
> -XX:PermSize=256m
> -XX:MaxPermSize=1024m
> -XX:ReservedCodeCacheSize=64m
> -XX:+UseG1GC
> -XX:+CMSClassUnloadingEnabled
> -Xloggc:$CATALINA_HOME/logs/gc.log
> -XX:+PrintHeapAtGC
> -XX:+PrintGCDetails
> -XX:+PrintGCTimeStamps
>
> Hope this helps,
> Chris
>
>
> On Tue, Mar 24, 2015 at 8:24 PM, Parul Kudtarkar <[hidden email]>
> wrote:
>
>> Thanks once again, looking forward to more insights from the community.
>>
>> >
>> > I think you have to just tune it.   Mostly I would increase the max
>> heap
>> > (Xmx).   Other folks in the community have more insight into working
>> with
>> > fractured sequences.
>> >
>> > Nathan
>> >
>> >> On Mar 24, 2015, at 5:20 PM, Parul Kudtarkar <[hidden email]>
>> wrote:
>> >>
>> >> Dear Nathan,
>> >>
>> >> Thanks, what is the recommended increase in memory size? Could you
>> give
>> >> exact parameters for CATALINA_OPTS or do we keep tuning till we get
>> it
>> >> correct.
>> >> Looking forward to 2.0 WebApollo.
>> >>
>> >> Best,
>> >> Parul Kudtarkar
>> >>
>> >>>
>> >>> When you have annotations over that many scaffolds you’ll have to
>> >>> increase
>> >>> your memory or split them up across separate genomes (at least for
>> >>> doing
>> >>> the searches).
>> >>>
>> >>> With 2.0 we’ve fixed this for any size, but the alpha won’t be out
>> for
>> >>> a
>> >>> couple of months.
>> >>>
>> >>> Nathan
>> >>>
>> >>>
>> >>>> On Mar 24, 2015, at 4:40 PM, Parul Kudtarkar <[hidden email]>
>> >>>> wrote:
>> >>>>
>> >>>> Dear Monica,
>> >>>>
>> >>>> This is a poorly assembled genome(Lytechinus variegatus) with 336K
>> >>>> Scaffolds aka reference sequences. In past though webapollo has
>> >>>> successfully handled our other genome(Strongylocentrotus
>> purpuratus)
>> >>>> with
>> >>>> 30K Scaffolds.
>> >>>>
>> >>>> Thanks and regards,
>> >>>> Parul Kudtarkar
>> >>>>
>> >>>>> Dear Parul,
>> >>>>>
>> >>>>> By 5,000 sequences we mean to say that the current Apollo
>> >>>>> architecture
>> >>>>> allows for *mostly uneventful* searchers through ~5,000 "reference
>> >>>>> sequences" or fragments / scaffolds / groups.
>> >>>>>
>> >>>>> Do you mean to say that your assembly has 336K fragments?
>> >>>>>
>> >>>>> cheers,
>> >>>>> ~moni.
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> --
>> >>>>> Mentorship Matters!
>> >>>>> --
>> >>>>> Monica Munoz-Torres, PhD.
>> >>>>> Berkeley Bioinformatics Open-source Projects (BBOP)
>> >>>>> Genomics Division, Lawrence Berkeley National Laboratory
>> >>>>>
>> >>>>> Mailing Address:
>> >>>>> Joint Genome Institute (400-0415)
>> >>>>> for the US Department of Energy
>> >>>>> 2800 Mitchell Dr. Ste 100
>> >>>>> Walnut Creek, CA 94598-1631
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> On Tue, Mar 24, 2015 at 2:39 PM, Parul Kudtarkar
>> <[hidden email]
>> >
>> >>>>> wrote:
>> >>>>>
>> >>>>>> Dear Nathan,
>> >>>>>>
>> >>>>>> Thanks for a quick response. You are correct, using the new
>> >>>>>> interface
>> >>>>>> and
>> >>>>>> either doing sequence search or doing search on recent changes
>> gives
>> >>>>>> heap
>> >>>>>> search error.
>> >>>>>> At the moment we are hosting single organism and the number of
>> >>>>>> sequences
>> >>>>>> are 336K, far beyond what welapollo can host. Could you suggest
>> >>>>>> appropriate memory setting?
>> >>>>>>
>> >>>>>> Thanks and regards,
>> >>>>>> Parul Kudtarkar
>> >>>>>>
>> >>>>>>>
>> >>>>>>> Thanks for the report.  This is a known issue that will be
>> >>>>>>> completely
>> >>>>>>> address in 2.0, but we have to work around it in the interim.
>> >>>>>>>
>> >>>>>>> I think that the number one cause of this will likely be doing a
>> >>>>>> search
>> >>>>>> on
>> >>>>>>> the recent changes or sequence search if you have many sequences
>> or
>> >>>>>>> changes or are hosting multiple organisms on the same tomcat.
>> >>>>>>>
>> >>>>>>> You can also try using the old interface (in the
>> upper-right-hand
>> >>>>>> side)
>> >>>>>> of
>> >>>>>>> either page and see if that is any better.
>> >>>>>>>
>> >>>>>>> Does this seem consistent with what you are seeing or are you
>> >>>>>>> seeing
>> >>>>>> the
>> >>>>>>> memory creep up in other places?
>> >>>>>>>
>> >>>>>>> If you know the number of sequences you have, we might be able
>> to
>> >>>>>>> get
>> >>>>>> a
>> >>>>>>> better memory setting to tune.  Typically with around 5K
>> sequences
>> >>>>>>> we
>> >>>>>> have
>> >>>>>>> no problems (other than the above mentioned).
>> >>>>>>>
>> >>>>>>> Nathan
>> >>>>>>>
>> >>>>>>>> On Mar 24, 2015, at 12:28 PM, Parul Kudtarkar
>> <[hidden email]
>> >
>> >>>>>>>> wrote:
>> >>>>>>>>
>> >>>>>>>> Dear Apollo community,
>> >>>>>>>>
>> >>>>>>>> I have set CATALINA_OPTS  as recommend by webapollo, even
>> though
>> >>>>>>>> webapollo
>> >>>>>>>> is running, it's pretty slow and I get error messages logged
>> into
>> >>>>>>>> catalina.out
>> >>>>>>>>
>> >>>>>>>> ps -ef | grep java
>> >>>>>>>> root     13952     1 99 12:11 pts/1    00:25:47 /usr/bin/java
>> >>>>>>>>
>> >>>>>>
>> -Djava.util.logging.config.file=/usr/share/apache-tomcat-7.0.59/conf/logging.properties
>> >>>>>>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>> >>>>>>>> -Xms512m
>> >>>>>>>> -Xmx1g -XX:+CMSClassUnloadingEnabled
>> >>>>>>>> -XX:+CMSPermGenSweepingEnabled
>> >>>>>>>> -XX:+UseConcMarkSweepGC -XX:MaxPermSize=256m
>> >>>>>>>> -Djava.endorsed.dirs=/usr/share/apache-tomcat-7.0.59/endorsed
>> >>>>>> -classpath
>> >>>>>>>>
>> >>>>>>
>> /usr/share/apache-tomcat-7.0.59/bin/bootstrap.jar:/usr/share/apache-tomcat-7.0.59/bin/tomcat-juli.jar
>> >>>>>>>> -Dcatalina.base=/usr/share/apache-tomcat-7.0.59
>> >>>>>>>> -Dcatalina.home=/usr/share/apache-tomcat-7.0.59
>> >>>>>>>> -Djava.io.tmpdir=/usr/share/apache-tomcat-7.0.59/temp
>> >>>>>>>> org.apache.catalina.startup.Bootstrap start
>> >>>>>>>> ----
>> >>>>>>>>
>> >>>>>>>> Mar 24, 2015 11:18:37 AM
>> >>>>>>>>
>> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor
>> >>>>>> run
>> >>>>>>>> SEVERE: Unexpected death of background thread
>> >>>>>>>> ContainerBackgroundProcessor[StandardEngine[Catalina]]
>> >>>>>>>> java.lang.OutOfMemoryError: Java heap space
>> >>>>>>>>
>> >>>>>>>>      at
>> >>>>>>>>
>> >>>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> >>>>>>>> Exception in thread
>> >>>>>>>> "ContainerBackgroundProcessor[StandardEngine[Catalina]]"
>> >>>>>>>> java.lang.OutOfMemoryError: Java heap space
>> >>>>>>>> Exception in thread "http-bio-8080-exec-5" Exception in thread
>> >>>>>>>> "http-bio-8080-exec-10" java.lang.OutOfMemoryError: Java heap
>> >>>>>>>> space
>> >>>>>>>> java.lang.OutOfMemoryError: Java heap space
>> >>>>>>>> Exception in thread "http-bio-8080-exec-4"
>> >>>>>> java.lang.OutOfMemoryError:
>> >>>>>>>> Java heap space
>> >>>>>>>> ----
>> >>>>>>>>
>> >>>>>>>> --
>> >>>>>>>> Scientific Programmer
>> >>>>>>>> Center for Computational Regulatory Genomics
>> >>>>>>>> Beckman Institute,
>> >>>>>>>> Biology and Biological Engineering
>> >>>>>>>> California Institute of Technology
>> >>>>>>>> http://www.echinobase.org/
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> --
>> >>>>>> Scientific Programmer
>> >>>>>> Center for Computational Regulatory Genomics
>> >>>>>> Beckman Institute,
>> >>>>>> Biology and Biological Engineering
>> >>>>>> California Institute of Technology
>> >>>>>> http://www.echinobase.org/
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> Scientific Programmer
>> >>>> Center for Computational Regulatory Genomics
>> >>>> Beckman Institute,
>> >>>> Biology and Biological Engineering
>> >>>> California Institute of Technology
>> >>>> http://www.echinobase.org/
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>
>> >>
>> >> --
>> >> Scientific Programmer
>> >> Center for Computational Regulatory Genomics
>> >> Beckman Institute,
>> >> Biology and Biological Engineering
>> >> California Institute of Technology
>> >> http://www.echinobase.org/
>> >>
>> >
>> >
>> >
>>
>>
>> --
>> Scientific Programmer
>> Center for Computational Regulatory Genomics
>> Beckman Institute,
>> Biology and Biological Engineering
>> California Institute of Technology
>> http://www.echinobase.org/
>>
>>
>


--
Scientific Programmer
Center for Computational Regulatory Genomics
Beckman Institute,
Biology and Biological Engineering
California Institute of Technology
http://www.echinobase.org/

Reply | Threaded
Open this post in threaded view
|

Re: heap space error

Chris Childers
Hi Parul,

I'm glad that it was helpful!

Cheers,
Chris
https://i5k.nal.usda.gov


On Thu, Mar 26, 2015 at 4:31 PM, Parul Kudtarkar <[hidden email]> wrote:
Dear Chris,

Thanks for sharing memory setting, these work great for our other genome
which is 60K in size(reference sequences/Scaffold count)

> Hi Parul,
>
> Here at the i5k Workspace@NAL, we have 45 Web Apollo instances running,
> and
> we learned we needed to tune the memory and garbage collection to help
> keep
> the system stable.  I'm not sure we will ever stop tuning the system, but
> these settings seem to work pretty well for us:
>
>  -Xmx12288m
> -Xms8192m
> -XX:PermSize=256m
> -XX:MaxPermSize=1024m
> -XX:ReservedCodeCacheSize=64m
> -XX:+UseG1GC
> -XX:+CMSClassUnloadingEnabled
> -Xloggc:$CATALINA_HOME/logs/gc.log
> -XX:+PrintHeapAtGC
> -XX:+PrintGCDetails
> -XX:+PrintGCTimeStamps
>
> Hope this helps,
> Chris
>
>
> On Tue, Mar 24, 2015 at 8:24 PM, Parul Kudtarkar <[hidden email]>
> wrote:
>
>> Thanks once again, looking forward to more insights from the community.
>>
>> >
>> > I think you have to just tune it.   Mostly I would increase the max
>> heap
>> > (Xmx).   Other folks in the community have more insight into working
>> with
>> > fractured sequences.
>> >
>> > Nathan
>> >
>> >> On Mar 24, 2015, at 5:20 PM, Parul Kudtarkar <[hidden email]>
>> wrote:
>> >>
>> >> Dear Nathan,
>> >>
>> >> Thanks, what is the recommended increase in memory size? Could you
>> give
>> >> exact parameters for CATALINA_OPTS or do we keep tuning till we get
>> it
>> >> correct.
>> >> Looking forward to 2.0 WebApollo.
>> >>
>> >> Best,
>> >> Parul Kudtarkar
>> >>
>> >>>
>> >>> When you have annotations over that many scaffolds you’ll have to
>> >>> increase
>> >>> your memory or split them up across separate genomes (at least for
>> >>> doing
>> >>> the searches).
>> >>>
>> >>> With 2.0 we’ve fixed this for any size, but the alpha won’t be out
>> for
>> >>> a
>> >>> couple of months.
>> >>>
>> >>> Nathan
>> >>>
>> >>>
>> >>>> On Mar 24, 2015, at 4:40 PM, Parul Kudtarkar <[hidden email]>
>> >>>> wrote:
>> >>>>
>> >>>> Dear Monica,
>> >>>>
>> >>>> This is a poorly assembled genome(Lytechinus variegatus) with 336K
>> >>>> Scaffolds aka reference sequences. In past though webapollo has
>> >>>> successfully handled our other genome(Strongylocentrotus
>> purpuratus)
>> >>>> with
>> >>>> 30K Scaffolds.
>> >>>>
>> >>>> Thanks and regards,
>> >>>> Parul Kudtarkar
>> >>>>
>> >>>>> Dear Parul,
>> >>>>>
>> >>>>> By 5,000 sequences we mean to say that the current Apollo
>> >>>>> architecture
>> >>>>> allows for *mostly uneventful* searchers through ~5,000 "reference
>> >>>>> sequences" or fragments / scaffolds / groups.
>> >>>>>
>> >>>>> Do you mean to say that your assembly has 336K fragments?
>> >>>>>
>> >>>>> cheers,
>> >>>>> ~moni.
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> --
>> >>>>> Mentorship Matters!
>> >>>>> --
>> >>>>> Monica Munoz-Torres, PhD.
>> >>>>> Berkeley Bioinformatics Open-source Projects (BBOP)
>> >>>>> Genomics Division, Lawrence Berkeley National Laboratory
>> >>>>>
>> >>>>> Mailing Address:
>> >>>>> Joint Genome Institute (400-0415)
>> >>>>> for the US Department of Energy
>> >>>>> 2800 Mitchell Dr. Ste 100
>> >>>>> Walnut Creek, CA 94598-1631
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> On Tue, Mar 24, 2015 at 2:39 PM, Parul Kudtarkar
>> <[hidden email]
>> >
>> >>>>> wrote:
>> >>>>>
>> >>>>>> Dear Nathan,
>> >>>>>>
>> >>>>>> Thanks for a quick response. You are correct, using the new
>> >>>>>> interface
>> >>>>>> and
>> >>>>>> either doing sequence search or doing search on recent changes
>> gives
>> >>>>>> heap
>> >>>>>> search error.
>> >>>>>> At the moment we are hosting single organism and the number of
>> >>>>>> sequences
>> >>>>>> are 336K, far beyond what welapollo can host. Could you suggest
>> >>>>>> appropriate memory setting?
>> >>>>>>
>> >>>>>> Thanks and regards,
>> >>>>>> Parul Kudtarkar
>> >>>>>>
>> >>>>>>>
>> >>>>>>> Thanks for the report.  This is a known issue that will be
>> >>>>>>> completely
>> >>>>>>> address in 2.0, but we have to work around it in the interim.
>> >>>>>>>
>> >>>>>>> I think that the number one cause of this will likely be doing a
>> >>>>>> search
>> >>>>>> on
>> >>>>>>> the recent changes or sequence search if you have many sequences
>> or
>> >>>>>>> changes or are hosting multiple organisms on the same tomcat.
>> >>>>>>>
>> >>>>>>> You can also try using the old interface (in the
>> upper-right-hand
>> >>>>>> side)
>> >>>>>> of
>> >>>>>>> either page and see if that is any better.
>> >>>>>>>
>> >>>>>>> Does this seem consistent with what you are seeing or are you
>> >>>>>>> seeing
>> >>>>>> the
>> >>>>>>> memory creep up in other places?
>> >>>>>>>
>> >>>>>>> If you know the number of sequences you have, we might be able
>> to
>> >>>>>>> get
>> >>>>>> a
>> >>>>>>> better memory setting to tune.  Typically with around 5K
>> sequences
>> >>>>>>> we
>> >>>>>> have
>> >>>>>>> no problems (other than the above mentioned).
>> >>>>>>>
>> >>>>>>> Nathan
>> >>>>>>>
>> >>>>>>>> On Mar 24, 2015, at 12:28 PM, Parul Kudtarkar
>> <[hidden email]
>> >
>> >>>>>>>> wrote:
>> >>>>>>>>
>> >>>>>>>> Dear Apollo community,
>> >>>>>>>>
>> >>>>>>>> I have set CATALINA_OPTS  as recommend by webapollo, even
>> though
>> >>>>>>>> webapollo
>> >>>>>>>> is running, it's pretty slow and I get error messages logged
>> into
>> >>>>>>>> catalina.out
>> >>>>>>>>
>> >>>>>>>> ps -ef | grep java
>> >>>>>>>> root     13952     1 99 12:11 pts/1    00:25:47 /usr/bin/java
>> >>>>>>>>
>> >>>>>>
>> -Djava.util.logging.config.file=/usr/share/apache-tomcat-7.0.59/conf/logging.properties
>> >>>>>>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>> >>>>>>>> -Xms512m
>> >>>>>>>> -Xmx1g -XX:+CMSClassUnloadingEnabled
>> >>>>>>>> -XX:+CMSPermGenSweepingEnabled
>> >>>>>>>> -XX:+UseConcMarkSweepGC -XX:MaxPermSize=256m
>> >>>>>>>> -Djava.endorsed.dirs=/usr/share/apache-tomcat-7.0.59/endorsed
>> >>>>>> -classpath
>> >>>>>>>>
>> >>>>>>
>> /usr/share/apache-tomcat-7.0.59/bin/bootstrap.jar:/usr/share/apache-tomcat-7.0.59/bin/tomcat-juli.jar
>> >>>>>>>> -Dcatalina.base=/usr/share/apache-tomcat-7.0.59
>> >>>>>>>> -Dcatalina.home=/usr/share/apache-tomcat-7.0.59
>> >>>>>>>> -Djava.io.tmpdir=/usr/share/apache-tomcat-7.0.59/temp
>> >>>>>>>> org.apache.catalina.startup.Bootstrap start
>> >>>>>>>> ----
>> >>>>>>>>
>> >>>>>>>> Mar 24, 2015 11:18:37 AM
>> >>>>>>>>
>> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor
>> >>>>>> run
>> >>>>>>>> SEVERE: Unexpected death of background thread
>> >>>>>>>> ContainerBackgroundProcessor[StandardEngine[Catalina]]
>> >>>>>>>> java.lang.OutOfMemoryError: Java heap space
>> >>>>>>>>
>> >>>>>>>>      at
>> >>>>>>>>
>> >>>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> >>>>>>>> Exception in thread
>> >>>>>>>> "ContainerBackgroundProcessor[StandardEngine[Catalina]]"
>> >>>>>>>> java.lang.OutOfMemoryError: Java heap space
>> >>>>>>>> Exception in thread "http-bio-8080-exec-5" Exception in thread
>> >>>>>>>> "http-bio-8080-exec-10" java.lang.OutOfMemoryError: Java heap
>> >>>>>>>> space
>> >>>>>>>> java.lang.OutOfMemoryError: Java heap space
>> >>>>>>>> Exception in thread "http-bio-8080-exec-4"
>> >>>>>> java.lang.OutOfMemoryError:
>> >>>>>>>> Java heap space
>> >>>>>>>> ----
>> >>>>>>>>
>> >>>>>>>> --
>> >>>>>>>> Scientific Programmer
>> >>>>>>>> Center for Computational Regulatory Genomics
>> >>>>>>>> Beckman Institute,
>> >>>>>>>> Biology and Biological Engineering
>> >>>>>>>> California Institute of Technology
>> >>>>>>>> http://www.echinobase.org/
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> --
>> >>>>>> Scientific Programmer
>> >>>>>> Center for Computational Regulatory Genomics
>> >>>>>> Beckman Institute,
>> >>>>>> Biology and Biological Engineering
>> >>>>>> California Institute of Technology
>> >>>>>> http://www.echinobase.org/
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> Scientific Programmer
>> >>>> Center for Computational Regulatory Genomics
>> >>>> Beckman Institute,
>> >>>> Biology and Biological Engineering
>> >>>> California Institute of Technology
>> >>>> http://www.echinobase.org/
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>
>> >>
>> >> --
>> >> Scientific Programmer
>> >> Center for Computational Regulatory Genomics
>> >> Beckman Institute,
>> >> Biology and Biological Engineering
>> >> California Institute of Technology
>> >> http://www.echinobase.org/
>> >>
>> >
>> >
>> >
>>
>>
>> --
>> Scientific Programmer
>> Center for Computational Regulatory Genomics
>> Beckman Institute,
>> Biology and Biological Engineering
>> California Institute of Technology
>> http://www.echinobase.org/
>>
>>
>


--
Scientific Programmer
Center for Computational Regulatory Genomics
Beckman Institute,
Biology and Biological Engineering
California Institute of Technology
http://www.echinobase.org/