Requested array size exceeds VM limit

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Requested array size exceeds VM limit

Thomas LETELLIER
Hello,

I'm currently working on a big InterMine instance (with huge data files) and I've some troubles when loading fasta files:

This is the error I get:

BUILD FAILED
/home/bin/intermine/imbuild/integrate.xml:54: The following error occurred while executing this line:
/home/bin/intermine/bio/sources/fasta/build.xml:44: java.lang.OutOfMemoryError: Requested array size exceeds VM limit

The fasta file is about ~760Mo and contains only one sequence (the chromosome). For files between 450 and 650Mo the data loading is fine.

Here is my computer setting:
- 32Go of RAM
- 4 CPU

Here are some variables from my bashrc file:
export JAVA_OPTS="-d64 -Xms16G -Xmx24G -XX:MaxPermSize=8G -XX:+UseCompressedOops"
export ANT_OPTS=$JAVA_OPTS


I checked the RAM and CPU consumption using htop and it seem's that the RAM is not the limiting factor. However I saw that only one CPU was used during the data integration (used at 100%) and I suppose that this is why the loading failed.


Do somebody already got this error ? Is there a way to improve the CPU utilization during the data loading ?

Best regards,

Thomas

_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Requested array size exceeds VM limit

Julie Sullivan-2
Hi Thomas

Which version of InterMine are you on?

We use the wonderful BioJava library to parse FASTA files and this was a
bug in that library, fixed by upgrading.

        https://github.com/biojava/biojava/issues/338

If you don't want to update your whole mine, you can try just updating
the biojava.jar.

Try that? Let me know if you have issues.

Julie



On 07/19/2017 01:44 PM, Thomas LETELLIER wrote:

> Hello,
>
> I'm currently working on a big InterMine instance (with huge data files)
> and I've some troubles when loading fasta files:
>
> This is the error I get:
>
>     BUILD FAILED
>     /home/bin/intermine/imbuild/integrate.xml:54: The following error
>     occurred while executing this line:
>     /home/bin/intermine/bio/sources/fasta/build.xml:44:
>     java.lang.OutOfMemoryError: *Requested array size exceeds VM limit*
>
>
> The fasta file is about ~760Mo and contains only one sequence (the
> chromosome). For files between 450 and 650Mo the data loading is fine.
>
> Here is my computer setting:
> - 32Go of RAM
> - 4 CPU
>
> Here are some variables from my bashrc file:
>
>     export JAVA_OPTS="-d64 -Xms16G -Xmx24G -XX:MaxPermSize=8G
>     -XX:+UseCompressedOops"
>     export ANT_OPTS=$JAVA_OPTS
>
>
>
> I checked the RAM and CPU consumption using htop and it seem's that the
> RAM is not the limiting factor. However I saw that only one CPU was used
> during the data integration (used at 100%) and I suppose that this is
> why the loading failed.
>
>
> Do somebody already got this error ? Is there a way to improve the CPU
> utilization during the data loading ?
>
> Best regards,
>
> Thomas
>
>
> _______________________________________________
> dev mailing list
> [hidden email]
> https://lists.intermine.org/mailman/listinfo/dev
>
_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Requested array size exceeds VM limit

Thomas LETELLIER
Hi,

I tried to update the biojava with the last version from the official
repository (4.2.0) but I always get the same error.

Regards,

Thomas

On 07/19/2017 03:37 PM, Thomas Letellier wrote:

> Hello Julie,
>
> I'm using the 1.8.3 version from 2017 july 12th which uses the 1-7-1
> biojava version.
> I will try to update manually the biojava !
>
> Thank you,
>
> Regards,
>
> Thomas
>
>
>
>
> On 07/19/2017 03:10 PM, Julie Sullivan wrote:
>> Hi Thomas
>>
>> Which version of InterMine are you on?
>>
>> We use the wonderful BioJava library to parse FASTA files and this
>> was a bug in that library, fixed by upgrading.
>>
>>     https://github.com/biojava/biojava/issues/338
>>
>> If you don't want to update your whole mine, you can try just
>> updating the biojava.jar.
>>
>> Try that? Let me know if you have issues.
>>
>> Julie
>>
>>
>>
>> On 07/19/2017 01:44 PM, Thomas LETELLIER wrote:
>>> Hello,
>>>
>>> I'm currently working on a big InterMine instance (with huge data
>>> files)
>>> and I've some troubles when loading fasta files:
>>>
>>> This is the error I get:
>>>
>>>     BUILD FAILED
>>>     /home/bin/intermine/imbuild/integrate.xml:54: The following error
>>>     occurred while executing this line:
>>>     /home/bin/intermine/bio/sources/fasta/build.xml:44:
>>>     java.lang.OutOfMemoryError: *Requested array size exceeds VM limit*
>>>
>>>
>>> The fasta file is about ~760Mo and contains only one sequence (the
>>> chromosome). For files between 450 and 650Mo the data loading is fine.
>>>
>>> Here is my computer setting:
>>> - 32Go of RAM
>>> - 4 CPU
>>>
>>> Here are some variables from my bashrc file:
>>>
>>>     export JAVA_OPTS="-d64 -Xms16G -Xmx24G -XX:MaxPermSize=8G
>>>     -XX:+UseCompressedOops"
>>>     export ANT_OPTS=$JAVA_OPTS
>>>
>>>
>>>
>>> I checked the RAM and CPU consumption using htop and it seem's that the
>>> RAM is not the limiting factor. However I saw that only one CPU was
>>> used
>>> during the data integration (used at 100%) and I suppose that this is
>>> why the loading failed.
>>>
>>>
>>> Do somebody already got this error ? Is there a way to improve the CPU
>>> utilization during the data loading ?
>>>
>>> Best regards,
>>>
>>> Thomas
>>>
>>>
>>> _______________________________________________
>>> dev mailing list
>>> [hidden email]
>>> https://lists.intermine.org/mailman/listinfo/dev
>>>
>

_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Loading...