Really slow build process

classic Classic list List threaded Threaded
23 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Really slow build process

Paulo Nuin
Hi everyone

In my adventure of trying to build the DB with 2.x I started trying our large data sources (XML files from AceDB dumps) and I am noticing that the process is extremely slow, something like 400% or more. I started with our gene source, which is about 12Gb and usually takes a couple of hours or a little more to load. We are not into more than 10 hours on the build

<==========---> 80% EXECUTING [10h 31m 39s]

and I only see

idle in trans   SELECT nextval('serial’)

on my Postgres. Is that normal? How can I speed up the process, if there’s a way to speed it up. If the trend continues we will jump from a couple of days of build time, to more than one week.

Thanks

Paulo
_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Really slow build process

sergio contrino-2
hi paulo,
it does not look ok and i would consider stopping it if still going on.
i would check your memory settings and availability in your machine (i
suppose you are running postgres there as well, possibly anything else?).
if all seems fine, i could run a test here if you share your data file.
but it is our last work day here until 2nd jan, so it could take us a
while to assess this issue.
thanks (and merry christmas!)
sergio


On 21/12/2018 05:06, Paulo Nuin wrote:

> Hi everyone
>
> In my adventure of trying to build the DB with 2.x I started trying our large data sources (XML files from AceDB dumps) and I am noticing that the process is extremely slow, something like 400% or more. I started with our gene source, which is about 12Gb and usually takes a couple of hours or a little more to load. We are not into more than 10 hours on the build
>
> <==========---> 80% EXECUTING [10h 31m 39s]
>
> and I only see
>
> idle in trans   SELECT nextval('serial’)
>
> on my Postgres. Is that normal? How can I speed up the process, if there’s a way to speed it up. If the trend continues we will jump from a couple of days of build time, to more than one week.
>
> Thanks
>
> Paulo
> _______________________________________________
> dev mailing list
> [hidden email]
> https://lists.intermine.org/mailman/listinfo/dev
>

--
sergio contrino                  InterMine, University of Cambridge
https://sergiocontrino.github.io           http://www.intermine.org
_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Really slow build process

Paulo Nuin
Hi Sergio

It is still going on, and the only major things running on the machine are Java and Postgres, This is how it looks like

top - 13:01:13 up 187 days, 12:37,  2 users,  load average: 7.36, 7.44, 7.53
Tasks: 240 total,   1 running, 184 sleeping,   0 stopped,   0 zombie
Cpu(s): 94.2%us,  0.2%sy,  0.0%ni,  5.6%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  32943180k total, 32615884k used,   327296k free,   709564k buffers
Swap:        0k total,        0k used,        0k free, 24612256k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
29298 nuin      20   0 6768m 1.5g  28m S 752.3  4.7   8267:28 java


Cheers
Paulo


> On Dec 21, 2018, at 4:51 AM, sergio contrino <[hidden email]> wrote:
>
> hi paulo,
> it does not look ok and i would consider stopping it if still going on. i would check your memory settings and availability in your machine (i suppose you are running postgres there as well, possibly anything else?).
> if all seems fine, i could run a test here if you share your data file.
> but it is our last work day here until 2nd jan, so it could take us a while to assess this issue.
> thanks (and merry christmas!)
> sergio
>
>
> On 21/12/2018 05:06, Paulo Nuin wrote:
>> Hi everyone
>> In my adventure of trying to build the DB with 2.x I started trying our large data sources (XML files from AceDB dumps) and I am noticing that the process is extremely slow, something like 400% or more. I started with our gene source, which is about 12Gb and usually takes a couple of hours or a little more to load. We are not into more than 10 hours on the build
>> <==========---> 80% EXECUTING [10h 31m 39s]
>> and I only see
>> idle in trans   SELECT nextval('serial’)
>> on my Postgres. Is that normal? How can I speed up the process, if there’s a way to speed it up. If the trend continues we will jump from a couple of days of build time, to more than one week.
>> Thanks
>> Paulo
>> _______________________________________________
>> dev mailing list
>> [hidden email]
>> https://lists.intermine.org/mailman/listinfo/dev
>
> --
> sergio contrino                  InterMine, University of Cambridge
> https://sergiocontrino.github.io           http://www.intermine.org
> _______________________________________________
> dev mailing list
> [hidden email]
> https://lists.intermine.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Really slow build process

Paulo Nuin
Hi Sergio

And after 21 hours:

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':dbmodel:retrieveSingleSource'.
> java.lang.OutOfMemoryError: GC overhead limit exceeded

Cheers
Paulo




> On Dec 21, 2018, at 6:02 AM, Paulo Nuin <[hidden email]> wrote:
>
> Hi Sergio
>
> It is still going on, and the only major things running on the machine are Java and Postgres, This is how it looks like
>
> top - 13:01:13 up 187 days, 12:37,  2 users,  load average: 7.36, 7.44, 7.53
> Tasks: 240 total,   1 running, 184 sleeping,   0 stopped,   0 zombie
> Cpu(s): 94.2%us,  0.2%sy,  0.0%ni,  5.6%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
> Mem:  32943180k total, 32615884k used,   327296k free,   709564k buffers
> Swap:        0k total,        0k used,        0k free, 24612256k cached
>
>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 29298 nuin      20   0 6768m 1.5g  28m S 752.3  4.7   8267:28 java
>
>
> Cheers
> Paulo
>
>
>> On Dec 21, 2018, at 4:51 AM, sergio contrino <[hidden email]> wrote:
>>
>> hi paulo,
>> it does not look ok and i would consider stopping it if still going on. i would check your memory settings and availability in your machine (i suppose you are running postgres there as well, possibly anything else?).
>> if all seems fine, i could run a test here if you share your data file.
>> but it is our last work day here until 2nd jan, so it could take us a while to assess this issue.
>> thanks (and merry christmas!)
>> sergio
>>
>>
>> On 21/12/2018 05:06, Paulo Nuin wrote:
>>> Hi everyone
>>> In my adventure of trying to build the DB with 2.x I started trying our large data sources (XML files from AceDB dumps) and I am noticing that the process is extremely slow, something like 400% or more. I started with our gene source, which is about 12Gb and usually takes a couple of hours or a little more to load. We are not into more than 10 hours on the build
>>> <==========---> 80% EXECUTING [10h 31m 39s]
>>> and I only see
>>> idle in trans   SELECT nextval('serial’)
>>> on my Postgres. Is that normal? How can I speed up the process, if there’s a way to speed it up. If the trend continues we will jump from a couple of days of build time, to more than one week.
>>> Thanks
>>> Paulo
>>> _______________________________________________
>>> dev mailing list
>>> [hidden email]
>>> https://lists.intermine.org/mailman/listinfo/dev
>>
>> --
>> sergio contrino                  InterMine, University of Cambridge
>> https://sergiocontrino.github.io           http://www.intermine.org
>> _______________________________________________
>> dev mailing list
>> [hidden email]
>> https://lists.intermine.org/mailman/listinfo/dev
>

_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Really slow build process

Julie Sullivan-2
Hi Paulo,

For the build that failed, can you run two commands for me? In the same
console.

 > echo $GRADLE_OPTS

and

 > ./gradlew --stop

Can you tell me what those two commands produce?

(be sure to use the same console, I think we had issues with AWS and ENV
vars before? maybe?)

Thanks!
Julie

PS. Happy New Year! :)

On 21/12/2018 15:51, Paulo Nuin wrote:

> Hi Sergio
>
> And after 21 hours:
>
> FAILURE: Build failed with an exception.
>
> * What went wrong:
> Execution failed for task ':dbmodel:retrieveSingleSource'.
>> java.lang.OutOfMemoryError: GC overhead limit exceeded
>
> Cheers
> Paulo
>
>
>
>
>> On Dec 21, 2018, at 6:02 AM, Paulo Nuin <[hidden email]> wrote:
>>
>> Hi Sergio
>>
>> It is still going on, and the only major things running on the machine are Java and Postgres, This is how it looks like
>>
>> top - 13:01:13 up 187 days, 12:37,  2 users,  load average: 7.36, 7.44, 7.53
>> Tasks: 240 total,   1 running, 184 sleeping,   0 stopped,   0 zombie
>> Cpu(s): 94.2%us,  0.2%sy,  0.0%ni,  5.6%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
>> Mem:  32943180k total, 32615884k used,   327296k free,   709564k buffers
>> Swap:        0k total,        0k used,        0k free, 24612256k cached
>>
>>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>> 29298 nuin      20   0 6768m 1.5g  28m S 752.3  4.7   8267:28 java
>>
>>
>> Cheers
>> Paulo
>>
>>
>>> On Dec 21, 2018, at 4:51 AM, sergio contrino <[hidden email]> wrote:
>>>
>>> hi paulo,
>>> it does not look ok and i would consider stopping it if still going on. i would check your memory settings and availability in your machine (i suppose you are running postgres there as well, possibly anything else?).
>>> if all seems fine, i could run a test here if you share your data file.
>>> but it is our last work day here until 2nd jan, so it could take us a while to assess this issue.
>>> thanks (and merry christmas!)
>>> sergio
>>>
>>>
>>> On 21/12/2018 05:06, Paulo Nuin wrote:
>>>> Hi everyone
>>>> In my adventure of trying to build the DB with 2.x I started trying our large data sources (XML files from AceDB dumps) and I am noticing that the process is extremely slow, something like 400% or more. I started with our gene source, which is about 12Gb and usually takes a couple of hours or a little more to load. We are not into more than 10 hours on the build
>>>> <==========---> 80% EXECUTING [10h 31m 39s]
>>>> and I only see
>>>> idle in trans   SELECT nextval('serial’)
>>>> on my Postgres. Is that normal? How can I speed up the process, if there’s a way to speed it up. If the trend continues we will jump from a couple of days of build time, to more than one week.
>>>> Thanks
>>>> Paulo
>>>> _______________________________________________
>>>> dev mailing list
>>>> [hidden email]
>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>
>>> --
>>> sergio contrino                  InterMine, University of Cambridge
>>> https://sergiocontrino.github.io           http://www.intermine.org
>>> _______________________________________________
>>> dev mailing list
>>> [hidden email]
>>> https://lists.intermine.org/mailman/listinfo/dev
>>
>
> _______________________________________________
> dev mailing list
> [hidden email]
> https://lists.intermine.org/mailman/listinfo/dev
>
_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Really slow build process

Paulo Nuin
Hi Julie

Here they are

> On Jan 4, 2019, at 4:51 AM, Julie Sullivan <[hidden email]> wrote:
>
> Hi Paulo,
>
> For the build that failed, can you run two commands for me? In the same console.
>
> > echo $GRADLE_OPTS
>

-server -XX:MaxPermSize=256M -Xmx1700m -XX:+UseParallelGC -Xms1700m -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99



> and
>
> > ./gradlew —stop


OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256M; support was removed in 8.0
No Gradle daemons are running.

Cheers
Paulo

>
> Can you tell me what those two commands produce?
>
> (be sure to use the same console, I think we had issues with AWS and ENV vars before? maybe?)
>
> Thanks!
> Julie
>
> PS. Happy New Year! :)
>
> On 21/12/2018 15:51, Paulo Nuin wrote:
>> Hi Sergio
>> And after 21 hours:
>> FAILURE: Build failed with an exception.
>> * What went wrong:
>> Execution failed for task ':dbmodel:retrieveSingleSource'.
>>> java.lang.OutOfMemoryError: GC overhead limit exceeded
>> Cheers
>> Paulo
>>> On Dec 21, 2018, at 6:02 AM, Paulo Nuin <[hidden email]> wrote:
>>>
>>> Hi Sergio
>>>
>>> It is still going on, and the only major things running on the machine are Java and Postgres, This is how it looks like
>>>
>>> top - 13:01:13 up 187 days, 12:37,  2 users,  load average: 7.36, 7.44, 7.53
>>> Tasks: 240 total,   1 running, 184 sleeping,   0 stopped,   0 zombie
>>> Cpu(s): 94.2%us,  0.2%sy,  0.0%ni,  5.6%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
>>> Mem:  32943180k total, 32615884k used,   327296k free,   709564k buffers
>>> Swap:        0k total,        0k used,        0k free, 24612256k cached
>>>
>>>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>>> 29298 nuin      20   0 6768m 1.5g  28m S 752.3  4.7   8267:28 java
>>>
>>>
>>> Cheers
>>> Paulo
>>>
>>>
>>>> On Dec 21, 2018, at 4:51 AM, sergio contrino <[hidden email]> wrote:
>>>>
>>>> hi paulo,
>>>> it does not look ok and i would consider stopping it if still going on. i would check your memory settings and availability in your machine (i suppose you are running postgres there as well, possibly anything else?).
>>>> if all seems fine, i could run a test here if you share your data file.
>>>> but it is our last work day here until 2nd jan, so it could take us a while to assess this issue.
>>>> thanks (and merry christmas!)
>>>> sergio
>>>>
>>>>
>>>> On 21/12/2018 05:06, Paulo Nuin wrote:
>>>>> Hi everyone
>>>>> In my adventure of trying to build the DB with 2.x I started trying our large data sources (XML files from AceDB dumps) and I am noticing that the process is extremely slow, something like 400% or more. I started with our gene source, which is about 12Gb and usually takes a couple of hours or a little more to load. We are not into more than 10 hours on the build
>>>>> <==========---> 80% EXECUTING [10h 31m 39s]
>>>>> and I only see
>>>>> idle in trans   SELECT nextval('serial’)
>>>>> on my Postgres. Is that normal? How can I speed up the process, if there’s a way to speed it up. If the trend continues we will jump from a couple of days of build time, to more than one week.
>>>>> Thanks
>>>>> Paulo
>>>>> _______________________________________________
>>>>> dev mailing list
>>>>> [hidden email]
>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>
>>>> --
>>>> sergio contrino                  InterMine, University of Cambridge
>>>> https://sergiocontrino.github.io           http://www.intermine.org
>>>> _______________________________________________
>>>> dev mailing list
>>>> [hidden email]
>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>
>> _______________________________________________
>> dev mailing list
>> [hidden email]
>> https://lists.intermine.org/mailman/listinfo/dev

_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Really slow build process

sergio contrino-2
hi paulo!
could you set

export GRADLE_OPTS="-server -Xmx1700m -XX:+UseParallelGC -Xms1700m
-XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99
-Dorg.gradle.daemon=false"

on the terminal where you will run the build? (i would also increase
your Xmx if you have the possibility)

as mentioned before, if you add the dump option just before the failing
source, you can then test this part of the build without having to
rebuild everything at every attempt (see
https://intermine.readthedocs.io/en/latest/database/database-building/build-script/)

please let us know how things proceed. thanks!
sergio





On 08/01/2019 16:40, Paulo Nuin wrote:

> Hi Julie
>
> Here they are
>
>> On Jan 4, 2019, at 4:51 AM, Julie Sullivan <[hidden email]> wrote:
>>
>> Hi Paulo,
>>
>> For the build that failed, can you run two commands for me? In the same console.
>>
>>> echo $GRADLE_OPTS
>>
>
> -server -XX:MaxPermSize=256M -Xmx1700m -XX:+UseParallelGC -Xms1700m -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99
>
>
>
>> and
>>
>>> ./gradlew —stop
>
>
> OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256M; support was removed in 8.0
> No Gradle daemons are running.
>
> Cheers
> Paulo
>
>>
>> Can you tell me what those two commands produce?
>>
>> (be sure to use the same console, I think we had issues with AWS and ENV vars before? maybe?)
>>
>> Thanks!
>> Julie
>>
>> PS. Happy New Year! :)
>>
>> On 21/12/2018 15:51, Paulo Nuin wrote:
>>> Hi Sergio
>>> And after 21 hours:
>>> FAILURE: Build failed with an exception.
>>> * What went wrong:
>>> Execution failed for task ':dbmodel:retrieveSingleSource'.
>>>> java.lang.OutOfMemoryError: GC overhead limit exceeded
>>> Cheers
>>> Paulo
>>>> On Dec 21, 2018, at 6:02 AM, Paulo Nuin <[hidden email]> wrote:
>>>>
>>>> Hi Sergio
>>>>
>>>> It is still going on, and the only major things running on the machine are Java and Postgres, This is how it looks like
>>>>
>>>> top - 13:01:13 up 187 days, 12:37,  2 users,  load average: 7.36, 7.44, 7.53
>>>> Tasks: 240 total,   1 running, 184 sleeping,   0 stopped,   0 zombie
>>>> Cpu(s): 94.2%us,  0.2%sy,  0.0%ni,  5.6%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
>>>> Mem:  32943180k total, 32615884k used,   327296k free,   709564k buffers
>>>> Swap:        0k total,        0k used,        0k free, 24612256k cached
>>>>
>>>>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>>>> 29298 nuin      20   0 6768m 1.5g  28m S 752.3  4.7   8267:28 java
>>>>
>>>>
>>>> Cheers
>>>> Paulo
>>>>
>>>>
>>>>> On Dec 21, 2018, at 4:51 AM, sergio contrino <[hidden email]> wrote:
>>>>>
>>>>> hi paulo,
>>>>> it does not look ok and i would consider stopping it if still going on. i would check your memory settings and availability in your machine (i suppose you are running postgres there as well, possibly anything else?).
>>>>> if all seems fine, i could run a test here if you share your data file.
>>>>> but it is our last work day here until 2nd jan, so it could take us a while to assess this issue.
>>>>> thanks (and merry christmas!)
>>>>> sergio
>>>>>
>>>>>
>>>>> On 21/12/2018 05:06, Paulo Nuin wrote:
>>>>>> Hi everyone
>>>>>> In my adventure of trying to build the DB with 2.x I started trying our large data sources (XML files from AceDB dumps) and I am noticing that the process is extremely slow, something like 400% or more. I started with our gene source, which is about 12Gb and usually takes a couple of hours or a little more to load. We are not into more than 10 hours on the build
>>>>>> <==========---> 80% EXECUTING [10h 31m 39s]
>>>>>> and I only see
>>>>>> idle in trans   SELECT nextval('serial’)
>>>>>> on my Postgres. Is that normal? How can I speed up the process, if there’s a way to speed it up. If the trend continues we will jump from a couple of days of build time, to more than one week.
>>>>>> Thanks
>>>>>> Paulo
>>>>>> _______________________________________________
>>>>>> dev mailing list
>>>>>> [hidden email]
>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>>
>>>>> --
>>>>> sergio contrino                  InterMine, University of Cambridge
>>>>> https://sergiocontrino.github.io           http://www.intermine.org
>>>>> _______________________________________________
>>>>> dev mailing list
>>>>> [hidden email]
>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>
>>> _______________________________________________
>>> dev mailing list
>>> [hidden email]
>>> https://lists.intermine.org/mailman/listinfo/dev
>

--
sergio contrino                  InterMine, University of Cambridge
https://sergiocontrino.github.io           http://www.intermine.org
_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Really slow build process

Paulo Nuin
Hi

I will try, but at the moment I am just testing the build and see if different sources work.

What is the recommended Postgres version?

Cheers
Paulo





> On Jan 8, 2019, at 10:23 AM, sergio contrino <[hidden email]> wrote:
>
> hi paulo!
> could you set
>
> export GRADLE_OPTS="-server -Xmx1700m -XX:+UseParallelGC -Xms1700m -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99 -Dorg.gradle.daemon=false"
>
> on the terminal where you will run the build? (i would also increase your Xmx if you have the possibility)
>
> as mentioned before, if you add the dump option just before the failing source, you can then test this part of the build without having to rebuild everything at every attempt (see https://intermine.readthedocs.io/en/latest/database/database-building/build-script/)
>
> please let us know how things proceed. thanks!
> sergio
>
>
>
>
>
> On 08/01/2019 16:40, Paulo Nuin wrote:
>> Hi Julie
>> Here they are
>>> On Jan 4, 2019, at 4:51 AM, Julie Sullivan <[hidden email]> wrote:
>>>
>>> Hi Paulo,
>>>
>>> For the build that failed, can you run two commands for me? In the same console.
>>>
>>>> echo $GRADLE_OPTS
>>>
>> -server -XX:MaxPermSize=256M -Xmx1700m -XX:+UseParallelGC -Xms1700m -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99
>>> and
>>>
>>>> ./gradlew —stop
>> OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256M; support was removed in 8.0
>> No Gradle daemons are running.
>> Cheers
>> Paulo
>>>
>>> Can you tell me what those two commands produce?
>>>
>>> (be sure to use the same console, I think we had issues with AWS and ENV vars before? maybe?)
>>>
>>> Thanks!
>>> Julie
>>>
>>> PS. Happy New Year! :)
>>>
>>> On 21/12/2018 15:51, Paulo Nuin wrote:
>>>> Hi Sergio
>>>> And after 21 hours:
>>>> FAILURE: Build failed with an exception.
>>>> * What went wrong:
>>>> Execution failed for task ':dbmodel:retrieveSingleSource'.
>>>>> java.lang.OutOfMemoryError: GC overhead limit exceeded
>>>> Cheers
>>>> Paulo
>>>>> On Dec 21, 2018, at 6:02 AM, Paulo Nuin <[hidden email]> wrote:
>>>>>
>>>>> Hi Sergio
>>>>>
>>>>> It is still going on, and the only major things running on the machine are Java and Postgres, This is how it looks like
>>>>>
>>>>> top - 13:01:13 up 187 days, 12:37,  2 users,  load average: 7.36, 7.44, 7.53
>>>>> Tasks: 240 total,   1 running, 184 sleeping,   0 stopped,   0 zombie
>>>>> Cpu(s): 94.2%us,  0.2%sy,  0.0%ni,  5.6%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
>>>>> Mem:  32943180k total, 32615884k used,   327296k free,   709564k buffers
>>>>> Swap:        0k total,        0k used,        0k free, 24612256k cached
>>>>>
>>>>>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>>>>> 29298 nuin      20   0 6768m 1.5g  28m S 752.3  4.7   8267:28 java
>>>>>
>>>>>
>>>>> Cheers
>>>>> Paulo
>>>>>
>>>>>
>>>>>> On Dec 21, 2018, at 4:51 AM, sergio contrino <[hidden email]> wrote:
>>>>>>
>>>>>> hi paulo,
>>>>>> it does not look ok and i would consider stopping it if still going on. i would check your memory settings and availability in your machine (i suppose you are running postgres there as well, possibly anything else?).
>>>>>> if all seems fine, i could run a test here if you share your data file.
>>>>>> but it is our last work day here until 2nd jan, so it could take us a while to assess this issue.
>>>>>> thanks (and merry christmas!)
>>>>>> sergio
>>>>>>
>>>>>>
>>>>>> On 21/12/2018 05:06, Paulo Nuin wrote:
>>>>>>> Hi everyone
>>>>>>> In my adventure of trying to build the DB with 2.x I started trying our large data sources (XML files from AceDB dumps) and I am noticing that the process is extremely slow, something like 400% or more. I started with our gene source, which is about 12Gb and usually takes a couple of hours or a little more to load. We are not into more than 10 hours on the build
>>>>>>> <==========---> 80% EXECUTING [10h 31m 39s]
>>>>>>> and I only see
>>>>>>> idle in trans   SELECT nextval('serial’)
>>>>>>> on my Postgres. Is that normal? How can I speed up the process, if there’s a way to speed it up. If the trend continues we will jump from a couple of days of build time, to more than one week.
>>>>>>> Thanks
>>>>>>> Paulo
>>>>>>> _______________________________________________
>>>>>>> dev mailing list
>>>>>>> [hidden email]
>>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>>>
>>>>>> --
>>>>>> sergio contrino                  InterMine, University of Cambridge
>>>>>> https://sergiocontrino.github.io           http://www.intermine.org
>>>>>> _______________________________________________
>>>>>> dev mailing list
>>>>>> [hidden email]
>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>>
>>>> _______________________________________________
>>>> dev mailing list
>>>> [hidden email]
>>>> https://lists.intermine.org/mailman/listinfo/dev
>
> --
> sergio contrino                  InterMine, University of Cambridge
> https://sergiocontrino.github.io           http://www.intermine.org

_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Really slow build process

Julie Sullivan-2
Less than 2 GB of memory is really low for an InterMine build,
especially if the file you are parsing is 12 GB.

If it's a build machine, I would give as much memory as you can. You
will see a speed increase!


On 08/01/2019 17:36, Paulo Nuin wrote:

> Hi
>
> I will try, but at the moment I am just testing the build and see if different sources work.
>
> What is the recommended Postgres version?
>
> Cheers
> Paulo
>
>
>
>
>
>> On Jan 8, 2019, at 10:23 AM, sergio contrino <[hidden email]> wrote:
>>
>> hi paulo!
>> could you set
>>
>> export GRADLE_OPTS="-server -Xmx1700m -XX:+UseParallelGC -Xms1700m -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99 -Dorg.gradle.daemon=false"
>>
>> on the terminal where you will run the build? (i would also increase your Xmx if you have the possibility)
>>
>> as mentioned before, if you add the dump option just before the failing source, you can then test this part of the build without having to rebuild everything at every attempt (see https://intermine.readthedocs.io/en/latest/database/database-building/build-script/)
>>
>> please let us know how things proceed. thanks!
>> sergio
>>
>>
>>
>>
>>
>> On 08/01/2019 16:40, Paulo Nuin wrote:
>>> Hi Julie
>>> Here they are
>>>> On Jan 4, 2019, at 4:51 AM, Julie Sullivan <[hidden email]> wrote:
>>>>
>>>> Hi Paulo,
>>>>
>>>> For the build that failed, can you run two commands for me? In the same console.
>>>>
>>>>> echo $GRADLE_OPTS
>>>>
>>> -server -XX:MaxPermSize=256M -Xmx1700m -XX:+UseParallelGC -Xms1700m -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99
>>>> and
>>>>
>>>>> ./gradlew —stop
>>> OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256M; support was removed in 8.0
>>> No Gradle daemons are running.
>>> Cheers
>>> Paulo
>>>>
>>>> Can you tell me what those two commands produce?
>>>>
>>>> (be sure to use the same console, I think we had issues with AWS and ENV vars before? maybe?)
>>>>
>>>> Thanks!
>>>> Julie
>>>>
>>>> PS. Happy New Year! :)
>>>>
>>>> On 21/12/2018 15:51, Paulo Nuin wrote:
>>>>> Hi Sergio
>>>>> And after 21 hours:
>>>>> FAILURE: Build failed with an exception.
>>>>> * What went wrong:
>>>>> Execution failed for task ':dbmodel:retrieveSingleSource'.
>>>>>> java.lang.OutOfMemoryError: GC overhead limit exceeded
>>>>> Cheers
>>>>> Paulo
>>>>>> On Dec 21, 2018, at 6:02 AM, Paulo Nuin <[hidden email]> wrote:
>>>>>>
>>>>>> Hi Sergio
>>>>>>
>>>>>> It is still going on, and the only major things running on the machine are Java and Postgres, This is how it looks like
>>>>>>
>>>>>> top - 13:01:13 up 187 days, 12:37,  2 users,  load average: 7.36, 7.44, 7.53
>>>>>> Tasks: 240 total,   1 running, 184 sleeping,   0 stopped,   0 zombie
>>>>>> Cpu(s): 94.2%us,  0.2%sy,  0.0%ni,  5.6%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
>>>>>> Mem:  32943180k total, 32615884k used,   327296k free,   709564k buffers
>>>>>> Swap:        0k total,        0k used,        0k free, 24612256k cached
>>>>>>
>>>>>>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>>>>>> 29298 nuin      20   0 6768m 1.5g  28m S 752.3  4.7   8267:28 java
>>>>>>
>>>>>>
>>>>>> Cheers
>>>>>> Paulo
>>>>>>
>>>>>>
>>>>>>> On Dec 21, 2018, at 4:51 AM, sergio contrino <[hidden email]> wrote:
>>>>>>>
>>>>>>> hi paulo,
>>>>>>> it does not look ok and i would consider stopping it if still going on. i would check your memory settings and availability in your machine (i suppose you are running postgres there as well, possibly anything else?).
>>>>>>> if all seems fine, i could run a test here if you share your data file.
>>>>>>> but it is our last work day here until 2nd jan, so it could take us a while to assess this issue.
>>>>>>> thanks (and merry christmas!)
>>>>>>> sergio
>>>>>>>
>>>>>>>
>>>>>>> On 21/12/2018 05:06, Paulo Nuin wrote:
>>>>>>>> Hi everyone
>>>>>>>> In my adventure of trying to build the DB with 2.x I started trying our large data sources (XML files from AceDB dumps) and I am noticing that the process is extremely slow, something like 400% or more. I started with our gene source, which is about 12Gb and usually takes a couple of hours or a little more to load. We are not into more than 10 hours on the build
>>>>>>>> <==========---> 80% EXECUTING [10h 31m 39s]
>>>>>>>> and I only see
>>>>>>>> idle in trans   SELECT nextval('serial’)
>>>>>>>> on my Postgres. Is that normal? How can I speed up the process, if there’s a way to speed it up. If the trend continues we will jump from a couple of days of build time, to more than one week.
>>>>>>>> Thanks
>>>>>>>> Paulo
>>>>>>>> _______________________________________________
>>>>>>>> dev mailing list
>>>>>>>> [hidden email]
>>>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>>>>
>>>>>>> --
>>>>>>> sergio contrino                  InterMine, University of Cambridge
>>>>>>> https://sergiocontrino.github.io           http://www.intermine.org
>>>>>>> _______________________________________________
>>>>>>> dev mailing list
>>>>>>> [hidden email]
>>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>>>
>>>>> _______________________________________________
>>>>> dev mailing list
>>>>> [hidden email]
>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>
>> --
>> sergio contrino                  InterMine, University of Cambridge
>> https://sergiocontrino.github.io           http://www.intermine.org
>
_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Really slow build process

Paulo Nuin
Hi

This is what I have on a “regular” terminal, I export this when running the build

export GRADLE_OPTS="-server -XX:MaxPermSize=1g -Xmx16g -XX:+UseParallelGC -Xms2g -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99”

And my usual ANT_OPTS is

export ANT_OPTS="-server -XX:MaxPermSize=512M -Xmx12g -XX:+UseParallelGC -Xms2g -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99 -XX:-UseGCOverheadLimit”

Cheers
Paulo



> On Jan 8, 2019, at 10:40 AM, Julie Sullivan <[hidden email]> wrote:
>
> Less than 2 GB of memory is really low for an InterMine build, especially if the file you are parsing is 12 GB.
>
> If it's a build machine, I would give as much memory as you can. You will see a speed increase!
>
>
> On 08/01/2019 17:36, Paulo Nuin wrote:
>> Hi
>> I will try, but at the moment I am just testing the build and see if different sources work.
>> What is the recommended Postgres version?
>> Cheers
>> Paulo
>>> On Jan 8, 2019, at 10:23 AM, sergio contrino <[hidden email]> wrote:
>>>
>>> hi paulo!
>>> could you set
>>>
>>> export GRADLE_OPTS="-server -Xmx1700m -XX:+UseParallelGC -Xms1700m -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99 -Dorg.gradle.daemon=false"
>>>
>>> on the terminal where you will run the build? (i would also increase your Xmx if you have the possibility)
>>>
>>> as mentioned before, if you add the dump option just before the failing source, you can then test this part of the build without having to rebuild everything at every attempt (see https://intermine.readthedocs.io/en/latest/database/database-building/build-script/)
>>>
>>> please let us know how things proceed. thanks!
>>> sergio
>>>
>>>
>>>
>>>
>>>
>>> On 08/01/2019 16:40, Paulo Nuin wrote:
>>>> Hi Julie
>>>> Here they are
>>>>> On Jan 4, 2019, at 4:51 AM, Julie Sullivan <[hidden email]> wrote:
>>>>>
>>>>> Hi Paulo,
>>>>>
>>>>> For the build that failed, can you run two commands for me? In the same console.
>>>>>
>>>>>> echo $GRADLE_OPTS
>>>>>
>>>> -server -XX:MaxPermSize=256M -Xmx1700m -XX:+UseParallelGC -Xms1700m -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99
>>>>> and
>>>>>
>>>>>> ./gradlew —stop
>>>> OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256M; support was removed in 8.0
>>>> No Gradle daemons are running.
>>>> Cheers
>>>> Paulo
>>>>>
>>>>> Can you tell me what those two commands produce?
>>>>>
>>>>> (be sure to use the same console, I think we had issues with AWS and ENV vars before? maybe?)
>>>>>
>>>>> Thanks!
>>>>> Julie
>>>>>
>>>>> PS. Happy New Year! :)
>>>>>
>>>>> On 21/12/2018 15:51, Paulo Nuin wrote:
>>>>>> Hi Sergio
>>>>>> And after 21 hours:
>>>>>> FAILURE: Build failed with an exception.
>>>>>> * What went wrong:
>>>>>> Execution failed for task ':dbmodel:retrieveSingleSource'.
>>>>>>> java.lang.OutOfMemoryError: GC overhead limit exceeded
>>>>>> Cheers
>>>>>> Paulo
>>>>>>> On Dec 21, 2018, at 6:02 AM, Paulo Nuin <[hidden email]> wrote:
>>>>>>>
>>>>>>> Hi Sergio
>>>>>>>
>>>>>>> It is still going on, and the only major things running on the machine are Java and Postgres, This is how it looks like
>>>>>>>
>>>>>>> top - 13:01:13 up 187 days, 12:37,  2 users,  load average: 7.36, 7.44, 7.53
>>>>>>> Tasks: 240 total,   1 running, 184 sleeping,   0 stopped,   0 zombie
>>>>>>> Cpu(s): 94.2%us,  0.2%sy,  0.0%ni,  5.6%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
>>>>>>> Mem:  32943180k total, 32615884k used,   327296k free,   709564k buffers
>>>>>>> Swap:        0k total,        0k used,        0k free, 24612256k cached
>>>>>>>
>>>>>>>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>>>>>>> 29298 nuin      20   0 6768m 1.5g  28m S 752.3  4.7   8267:28 java
>>>>>>>
>>>>>>>
>>>>>>> Cheers
>>>>>>> Paulo
>>>>>>>
>>>>>>>
>>>>>>>> On Dec 21, 2018, at 4:51 AM, sergio contrino <[hidden email]> wrote:
>>>>>>>>
>>>>>>>> hi paulo,
>>>>>>>> it does not look ok and i would consider stopping it if still going on. i would check your memory settings and availability in your machine (i suppose you are running postgres there as well, possibly anything else?).
>>>>>>>> if all seems fine, i could run a test here if you share your data file.
>>>>>>>> but it is our last work day here until 2nd jan, so it could take us a while to assess this issue.
>>>>>>>> thanks (and merry christmas!)
>>>>>>>> sergio
>>>>>>>>
>>>>>>>>
>>>>>>>> On 21/12/2018 05:06, Paulo Nuin wrote:
>>>>>>>>> Hi everyone
>>>>>>>>> In my adventure of trying to build the DB with 2.x I started trying our large data sources (XML files from AceDB dumps) and I am noticing that the process is extremely slow, something like 400% or more. I started with our gene source, which is about 12Gb and usually takes a couple of hours or a little more to load. We are not into more than 10 hours on the build
>>>>>>>>> <==========---> 80% EXECUTING [10h 31m 39s]
>>>>>>>>> and I only see
>>>>>>>>> idle in trans   SELECT nextval('serial’)
>>>>>>>>> on my Postgres. Is that normal? How can I speed up the process, if there’s a way to speed it up. If the trend continues we will jump from a couple of days of build time, to more than one week.
>>>>>>>>> Thanks
>>>>>>>>> Paulo
>>>>>>>>> _______________________________________________
>>>>>>>>> dev mailing list
>>>>>>>>> [hidden email]
>>>>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>>>>>
>>>>>>>> --
>>>>>>>> sergio contrino                  InterMine, University of Cambridge
>>>>>>>> https://sergiocontrino.github.io           http://www.intermine.org
>>>>>>>> _______________________________________________
>>>>>>>> dev mailing list
>>>>>>>> [hidden email]
>>>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>>>>
>>>>>> _______________________________________________
>>>>>> dev mailing list
>>>>>> [hidden email]
>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>
>>> --
>>> sergio contrino                  InterMine, University of Cambridge
>>> https://sergiocontrino.github.io           http://www.intermine.org

_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Really slow build process

sergio contrino-2
hi paul,
from your 2 last mails, it looks like your build is not using the
GRADLE_OPTS setting you are exporting, not sure why.

i don't know your machine setting (how much ram you have, do you use it
only for the build, are you hosting postgres on it, if so what are your
postgres settings regarding memory), but the last setting you sent seems
reasonable.
i could suggest:

- go to the terminal where you are going to run the build. all the below
commands should be run on this terminal

- change your gradle opts (i understand you are on java 8) running the
command:

export GRADLE_OPTS="-server -Xmx16g -XX:+UseParallelGC -Xms2g
-XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99
-Dorg.gradle.daemon=false”

- check this are the setting you get
echo $GRADLE_OPTS

- you could also run (just to have a snapshot of your memory situation)
free -g

if all seems fine
- run the build (please insert the dump as mentioned earlier).

- let us know the result

and then we can assess the situation.
thanks!
sergio



On 08/01/2019 18:36, Paulo Nuin wrote:

> Hi
>
> This is what I have on a “regular” terminal, I export this when running the build
>
> export GRADLE_OPTS="-server -XX:MaxPermSize=1g -Xmx16g -XX:+UseParallelGC -Xms2g -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99”
>
> And my usual ANT_OPTS is
>
> export ANT_OPTS="-server -XX:MaxPermSize=512M -Xmx12g -XX:+UseParallelGC -Xms2g -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99 -XX:-UseGCOverheadLimit”
>
> Cheers
> Paulo
>
>
>
>> On Jan 8, 2019, at 10:40 AM, Julie Sullivan <[hidden email]> wrote:
>>
>> Less than 2 GB of memory is really low for an InterMine build, especially if the file you are parsing is 12 GB.
>>
>> If it's a build machine, I would give as much memory as you can. You will see a speed increase!
>>
>>
>> On 08/01/2019 17:36, Paulo Nuin wrote:
>>> Hi
>>> I will try, but at the moment I am just testing the build and see if different sources work.
>>> What is the recommended Postgres version?
>>> Cheers
>>> Paulo
>>>> On Jan 8, 2019, at 10:23 AM, sergio contrino <[hidden email]> wrote:
>>>>
>>>> hi paulo!
>>>> could you set
>>>>
>>>> export GRADLE_OPTS="-server -Xmx1700m -XX:+UseParallelGC -Xms1700m -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99 -Dorg.gradle.daemon=false"
>>>>
>>>> on the terminal where you will run the build? (i would also increase your Xmx if you have the possibility)
>>>>
>>>> as mentioned before, if you add the dump option just before the failing source, you can then test this part of the build without having to rebuild everything at every attempt (see https://intermine.readthedocs.io/en/latest/database/database-building/build-script/)
>>>>
>>>> please let us know how things proceed. thanks!
>>>> sergio
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 08/01/2019 16:40, Paulo Nuin wrote:
>>>>> Hi Julie
>>>>> Here they are
>>>>>> On Jan 4, 2019, at 4:51 AM, Julie Sullivan <[hidden email]> wrote:
>>>>>>
>>>>>> Hi Paulo,
>>>>>>
>>>>>> For the build that failed, can you run two commands for me? In the same console.
>>>>>>
>>>>>>> echo $GRADLE_OPTS
>>>>>>
>>>>> -server -XX:MaxPermSize=256M -Xmx1700m -XX:+UseParallelGC -Xms1700m -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99
>>>>>> and
>>>>>>
>>>>>>> ./gradlew —stop
>>>>> OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256M; support was removed in 8.0
>>>>> No Gradle daemons are running.
>>>>> Cheers
>>>>> Paulo
>>>>>>
>>>>>> Can you tell me what those two commands produce?
>>>>>>
>>>>>> (be sure to use the same console, I think we had issues with AWS and ENV vars before? maybe?)
>>>>>>
>>>>>> Thanks!
>>>>>> Julie
>>>>>>
>>>>>> PS. Happy New Year! :)
>>>>>>
>>>>>> On 21/12/2018 15:51, Paulo Nuin wrote:
>>>>>>> Hi Sergio
>>>>>>> And after 21 hours:
>>>>>>> FAILURE: Build failed with an exception.
>>>>>>> * What went wrong:
>>>>>>> Execution failed for task ':dbmodel:retrieveSingleSource'.
>>>>>>>> java.lang.OutOfMemoryError: GC overhead limit exceeded
>>>>>>> Cheers
>>>>>>> Paulo
>>>>>>>> On Dec 21, 2018, at 6:02 AM, Paulo Nuin <[hidden email]> wrote:
>>>>>>>>
>>>>>>>> Hi Sergio
>>>>>>>>
>>>>>>>> It is still going on, and the only major things running on the machine are Java and Postgres, This is how it looks like
>>>>>>>>
>>>>>>>> top - 13:01:13 up 187 days, 12:37,  2 users,  load average: 7.36, 7.44, 7.53
>>>>>>>> Tasks: 240 total,   1 running, 184 sleeping,   0 stopped,   0 zombie
>>>>>>>> Cpu(s): 94.2%us,  0.2%sy,  0.0%ni,  5.6%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
>>>>>>>> Mem:  32943180k total, 32615884k used,   327296k free,   709564k buffers
>>>>>>>> Swap:        0k total,        0k used,        0k free, 24612256k cached
>>>>>>>>
>>>>>>>>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>>>>>>>> 29298 nuin      20   0 6768m 1.5g  28m S 752.3  4.7   8267:28 java
>>>>>>>>
>>>>>>>>
>>>>>>>> Cheers
>>>>>>>> Paulo
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Dec 21, 2018, at 4:51 AM, sergio contrino <[hidden email]> wrote:
>>>>>>>>>
>>>>>>>>> hi paulo,
>>>>>>>>> it does not look ok and i would consider stopping it if still going on. i would check your memory settings and availability in your machine (i suppose you are running postgres there as well, possibly anything else?).
>>>>>>>>> if all seems fine, i could run a test here if you share your data file.
>>>>>>>>> but it is our last work day here until 2nd jan, so it could take us a while to assess this issue.
>>>>>>>>> thanks (and merry christmas!)
>>>>>>>>> sergio
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 21/12/2018 05:06, Paulo Nuin wrote:
>>>>>>>>>> Hi everyone
>>>>>>>>>> In my adventure of trying to build the DB with 2.x I started trying our large data sources (XML files from AceDB dumps) and I am noticing that the process is extremely slow, something like 400% or more. I started with our gene source, which is about 12Gb and usually takes a couple of hours or a little more to load. We are not into more than 10 hours on the build
>>>>>>>>>> <==========---> 80% EXECUTING [10h 31m 39s]
>>>>>>>>>> and I only see
>>>>>>>>>> idle in trans   SELECT nextval('serial’)
>>>>>>>>>> on my Postgres. Is that normal? How can I speed up the process, if there’s a way to speed it up. If the trend continues we will jump from a couple of days of build time, to more than one week.
>>>>>>>>>> Thanks
>>>>>>>>>> Paulo
>>>>>>>>>> _______________________________________________
>>>>>>>>>> dev mailing list
>>>>>>>>>> [hidden email]
>>>>>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> sergio contrino                  InterMine, University of Cambridge
>>>>>>>>> https://sergiocontrino.github.io           http://www.intermine.org
>>>>>>>>> _______________________________________________
>>>>>>>>> dev mailing list
>>>>>>>>> [hidden email]
>>>>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> dev mailing list
>>>>>>> [hidden email]
>>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>
>>>> --
>>>> sergio contrino                  InterMine, University of Cambridge
>>>> https://sergiocontrino.github.io           http://www.intermine.org
>

--
sergio contrino                  InterMine, University of Cambridge
https://sergiocontrino.github.io           http://www.intermine.org
_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Really slow build process

Paulo Nuin
Ciao Sergio

I have the exports on my .bashrc and I am sourcing it before every test just to be sure. I had to change a number of things due to heap problems and I am making sure I have the right options set.

Our machines are on AWS and we have about 32 Gb of memory available the the maximum we setting to Java (initially for ant) which is somewhat a tradeoff between performance and keeping other resources working.

We have Postgres in the same machine.

I will try again and let you know.

Thanks

Paulo



> On Jan 9, 2019, at 3:49 AM, sergio contrino <[hidden email]> wrote:
>
> hi paul,
> from your 2 last mails, it looks like your build is not using the GRADLE_OPTS setting you are exporting, not sure why.
>
> i don't know your machine setting (how much ram you have, do you use it only for the build, are you hosting postgres on it, if so what are your postgres settings regarding memory), but the last setting you sent seems reasonable.
> i could suggest:
>
> - go to the terminal where you are going to run the build. all the below commands should be run on this terminal
>
> - change your gradle opts (i understand you are on java 8) running the command:
>
> export GRADLE_OPTS="-server -Xmx16g -XX:+UseParallelGC -Xms2g -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99 -Dorg.gradle.daemon=false”
>
> - check this are the setting you get
> echo $GRADLE_OPTS
>
> - you could also run (just to have a snapshot of your memory situation)
> free -g
>
> if all seems fine
> - run the build (please insert the dump as mentioned earlier).
>
> - let us know the result
>
> and then we can assess the situation.
> thanks!
> sergio
>
>
>
> On 08/01/2019 18:36, Paulo Nuin wrote:
>> Hi
>> This is what I have on a “regular” terminal, I export this when running the build
>> export GRADLE_OPTS="-server -XX:MaxPermSize=1g -Xmx16g -XX:+UseParallelGC -Xms2g -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99”
>> And my usual ANT_OPTS is
>> export ANT_OPTS="-server -XX:MaxPermSize=512M -Xmx12g -XX:+UseParallelGC -Xms2g -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99 -XX:-UseGCOverheadLimit”
>> Cheers
>> Paulo
>>> On Jan 8, 2019, at 10:40 AM, Julie Sullivan <[hidden email]> wrote:
>>>
>>> Less than 2 GB of memory is really low for an InterMine build, especially if the file you are parsing is 12 GB.
>>>
>>> If it's a build machine, I would give as much memory as you can. You will see a speed increase!
>>>
>>>
>>> On 08/01/2019 17:36, Paulo Nuin wrote:
>>>> Hi
>>>> I will try, but at the moment I am just testing the build and see if different sources work.
>>>> What is the recommended Postgres version?
>>>> Cheers
>>>> Paulo
>>>>> On Jan 8, 2019, at 10:23 AM, sergio contrino <[hidden email]> wrote:
>>>>>
>>>>> hi paulo!
>>>>> could you set
>>>>>
>>>>> export GRADLE_OPTS="-server -Xmx1700m -XX:+UseParallelGC -Xms1700m -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99 -Dorg.gradle.daemon=false"
>>>>>
>>>>> on the terminal where you will run the build? (i would also increase your Xmx if you have the possibility)
>>>>>
>>>>> as mentioned before, if you add the dump option just before the failing source, you can then test this part of the build without having to rebuild everything at every attempt (see https://intermine.readthedocs.io/en/latest/database/database-building/build-script/)
>>>>>
>>>>> please let us know how things proceed. thanks!
>>>>> sergio
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 08/01/2019 16:40, Paulo Nuin wrote:
>>>>>> Hi Julie
>>>>>> Here they are
>>>>>>> On Jan 4, 2019, at 4:51 AM, Julie Sullivan <[hidden email]> wrote:
>>>>>>>
>>>>>>> Hi Paulo,
>>>>>>>
>>>>>>> For the build that failed, can you run two commands for me? In the same console.
>>>>>>>
>>>>>>>> echo $GRADLE_OPTS
>>>>>>>
>>>>>> -server -XX:MaxPermSize=256M -Xmx1700m -XX:+UseParallelGC -Xms1700m -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99
>>>>>>> and
>>>>>>>
>>>>>>>> ./gradlew —stop
>>>>>> OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256M; support was removed in 8.0
>>>>>> No Gradle daemons are running.
>>>>>> Cheers
>>>>>> Paulo
>>>>>>>
>>>>>>> Can you tell me what those two commands produce?
>>>>>>>
>>>>>>> (be sure to use the same console, I think we had issues with AWS and ENV vars before? maybe?)
>>>>>>>
>>>>>>> Thanks!
>>>>>>> Julie
>>>>>>>
>>>>>>> PS. Happy New Year! :)
>>>>>>>
>>>>>>> On 21/12/2018 15:51, Paulo Nuin wrote:
>>>>>>>> Hi Sergio
>>>>>>>> And after 21 hours:
>>>>>>>> FAILURE: Build failed with an exception.
>>>>>>>> * What went wrong:
>>>>>>>> Execution failed for task ':dbmodel:retrieveSingleSource'.
>>>>>>>>> java.lang.OutOfMemoryError: GC overhead limit exceeded
>>>>>>>> Cheers
>>>>>>>> Paulo
>>>>>>>>> On Dec 21, 2018, at 6:02 AM, Paulo Nuin <[hidden email]> wrote:
>>>>>>>>>
>>>>>>>>> Hi Sergio
>>>>>>>>>
>>>>>>>>> It is still going on, and the only major things running on the machine are Java and Postgres, This is how it looks like
>>>>>>>>>
>>>>>>>>> top - 13:01:13 up 187 days, 12:37,  2 users,  load average: 7.36, 7.44, 7.53
>>>>>>>>> Tasks: 240 total,   1 running, 184 sleeping,   0 stopped,   0 zombie
>>>>>>>>> Cpu(s): 94.2%us,  0.2%sy,  0.0%ni,  5.6%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
>>>>>>>>> Mem:  32943180k total, 32615884k used,   327296k free,   709564k buffers
>>>>>>>>> Swap:        0k total,        0k used,        0k free, 24612256k cached
>>>>>>>>>
>>>>>>>>>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>>>>>>>>> 29298 nuin      20   0 6768m 1.5g  28m S 752.3  4.7   8267:28 java
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Cheers
>>>>>>>>> Paulo
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On Dec 21, 2018, at 4:51 AM, sergio contrino <[hidden email]> wrote:
>>>>>>>>>>
>>>>>>>>>> hi paulo,
>>>>>>>>>> it does not look ok and i would consider stopping it if still going on. i would check your memory settings and availability in your machine (i suppose you are running postgres there as well, possibly anything else?).
>>>>>>>>>> if all seems fine, i could run a test here if you share your data file.
>>>>>>>>>> but it is our last work day here until 2nd jan, so it could take us a while to assess this issue.
>>>>>>>>>> thanks (and merry christmas!)
>>>>>>>>>> sergio
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 21/12/2018 05:06, Paulo Nuin wrote:
>>>>>>>>>>> Hi everyone
>>>>>>>>>>> In my adventure of trying to build the DB with 2.x I started trying our large data sources (XML files from AceDB dumps) and I am noticing that the process is extremely slow, something like 400% or more. I started with our gene source, which is about 12Gb and usually takes a couple of hours or a little more to load. We are not into more than 10 hours on the build
>>>>>>>>>>> <==========---> 80% EXECUTING [10h 31m 39s]
>>>>>>>>>>> and I only see
>>>>>>>>>>> idle in trans   SELECT nextval('serial’)
>>>>>>>>>>> on my Postgres. Is that normal? How can I speed up the process, if there’s a way to speed it up. If the trend continues we will jump from a couple of days of build time, to more than one week.
>>>>>>>>>>> Thanks
>>>>>>>>>>> Paulo
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> dev mailing list
>>>>>>>>>>> [hidden email]
>>>>>>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> sergio contrino                  InterMine, University of Cambridge
>>>>>>>>>> https://sergiocontrino.github.io           http://www.intermine.org
>>>>>>>>>> _______________________________________________
>>>>>>>>>> dev mailing list
>>>>>>>>>> [hidden email]
>>>>>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> dev mailing list
>>>>>>>> [hidden email]
>>>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>>
>>>>> --
>>>>> sergio contrino                  InterMine, University of Cambridge
>>>>> https://sergiocontrino.github.io           http://www.intermine.org
>
> --
> sergio contrino                  InterMine, University of Cambridge
> https://sergiocontrino.github.io           http://www.intermine.org

_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Really slow build process

Paulo Nuin
Hi

Made sure (again) that GRADLE_OPTS was optima (as reported previously), or very close to optimal, and I am currently at 23 hours to load one XML file and what I can see on my Postgres monitoring is

idle in trans   SELECT nextval('serial’)

There are a couple of things that might be wrong:

- our code is incompatible (our loader was created for our XML files)
- out Postgres is too old (we are at 9.2.24)
- something on InterMine’s code
- we need to add more memory to the process

Ideas?

Thanks

Paulo



> On Jan 9, 2019, at 7:10 AM, Paulo Nuin <[hidden email]> wrote:
>
> Ciao Sergio
>
> I have the exports on my .bashrc and I am sourcing it before every test just to be sure. I had to change a number of things due to heap problems and I am making sure I have the right options set.
>
> Our machines are on AWS and we have about 32 Gb of memory available the the maximum we setting to Java (initially for ant) which is somewhat a tradeoff between performance and keeping other resources working.
>
> We have Postgres in the same machine.
>
> I will try again and let you know.
>
> Thanks
>
> Paulo
>
>
>
>> On Jan 9, 2019, at 3:49 AM, sergio contrino <[hidden email]> wrote:
>>
>> hi paul,
>> from your 2 last mails, it looks like your build is not using the GRADLE_OPTS setting you are exporting, not sure why.
>>
>> i don't know your machine setting (how much ram you have, do you use it only for the build, are you hosting postgres on it, if so what are your postgres settings regarding memory), but the last setting you sent seems reasonable.
>> i could suggest:
>>
>> - go to the terminal where you are going to run the build. all the below commands should be run on this terminal
>>
>> - change your gradle opts (i understand you are on java 8) running the command:
>>
>> export GRADLE_OPTS="-server -Xmx16g -XX:+UseParallelGC -Xms2g -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99 -Dorg.gradle.daemon=false”
>>
>> - check this are the setting you get
>> echo $GRADLE_OPTS
>>
>> - you could also run (just to have a snapshot of your memory situation)
>> free -g
>>
>> if all seems fine
>> - run the build (please insert the dump as mentioned earlier).
>>
>> - let us know the result
>>
>> and then we can assess the situation.
>> thanks!
>> sergio
>>
>>
>>
>> On 08/01/2019 18:36, Paulo Nuin wrote:
>>> Hi
>>> This is what I have on a “regular” terminal, I export this when running the build
>>> export GRADLE_OPTS="-server -XX:MaxPermSize=1g -Xmx16g -XX:+UseParallelGC -Xms2g -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99”
>>> And my usual ANT_OPTS is
>>> export ANT_OPTS="-server -XX:MaxPermSize=512M -Xmx12g -XX:+UseParallelGC -Xms2g -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99 -XX:-UseGCOverheadLimit”
>>> Cheers
>>> Paulo
>>>> On Jan 8, 2019, at 10:40 AM, Julie Sullivan <[hidden email]> wrote:
>>>>
>>>> Less than 2 GB of memory is really low for an InterMine build, especially if the file you are parsing is 12 GB.
>>>>
>>>> If it's a build machine, I would give as much memory as you can. You will see a speed increase!
>>>>
>>>>
>>>> On 08/01/2019 17:36, Paulo Nuin wrote:
>>>>> Hi
>>>>> I will try, but at the moment I am just testing the build and see if different sources work.
>>>>> What is the recommended Postgres version?
>>>>> Cheers
>>>>> Paulo
>>>>>> On Jan 8, 2019, at 10:23 AM, sergio contrino <[hidden email]> wrote:
>>>>>>
>>>>>> hi paulo!
>>>>>> could you set
>>>>>>
>>>>>> export GRADLE_OPTS="-server -Xmx1700m -XX:+UseParallelGC -Xms1700m -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99 -Dorg.gradle.daemon=false"
>>>>>>
>>>>>> on the terminal where you will run the build? (i would also increase your Xmx if you have the possibility)
>>>>>>
>>>>>> as mentioned before, if you add the dump option just before the failing source, you can then test this part of the build without having to rebuild everything at every attempt (see https://intermine.readthedocs.io/en/latest/database/database-building/build-script/)
>>>>>>
>>>>>> please let us know how things proceed. thanks!
>>>>>> sergio
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 08/01/2019 16:40, Paulo Nuin wrote:
>>>>>>> Hi Julie
>>>>>>> Here they are
>>>>>>>> On Jan 4, 2019, at 4:51 AM, Julie Sullivan <[hidden email]> wrote:
>>>>>>>>
>>>>>>>> Hi Paulo,
>>>>>>>>
>>>>>>>> For the build that failed, can you run two commands for me? In the same console.
>>>>>>>>
>>>>>>>>> echo $GRADLE_OPTS
>>>>>>>>
>>>>>>> -server -XX:MaxPermSize=256M -Xmx1700m -XX:+UseParallelGC -Xms1700m -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99
>>>>>>>> and
>>>>>>>>
>>>>>>>>> ./gradlew —stop
>>>>>>> OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256M; support was removed in 8.0
>>>>>>> No Gradle daemons are running.
>>>>>>> Cheers
>>>>>>> Paulo
>>>>>>>>
>>>>>>>> Can you tell me what those two commands produce?
>>>>>>>>
>>>>>>>> (be sure to use the same console, I think we had issues with AWS and ENV vars before? maybe?)
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>> Julie
>>>>>>>>
>>>>>>>> PS. Happy New Year! :)
>>>>>>>>
>>>>>>>> On 21/12/2018 15:51, Paulo Nuin wrote:
>>>>>>>>> Hi Sergio
>>>>>>>>> And after 21 hours:
>>>>>>>>> FAILURE: Build failed with an exception.
>>>>>>>>> * What went wrong:
>>>>>>>>> Execution failed for task ':dbmodel:retrieveSingleSource'.
>>>>>>>>>> java.lang.OutOfMemoryError: GC overhead limit exceeded
>>>>>>>>> Cheers
>>>>>>>>> Paulo
>>>>>>>>>> On Dec 21, 2018, at 6:02 AM, Paulo Nuin <[hidden email]> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi Sergio
>>>>>>>>>>
>>>>>>>>>> It is still going on, and the only major things running on the machine are Java and Postgres, This is how it looks like
>>>>>>>>>>
>>>>>>>>>> top - 13:01:13 up 187 days, 12:37,  2 users,  load average: 7.36, 7.44, 7.53
>>>>>>>>>> Tasks: 240 total,   1 running, 184 sleeping,   0 stopped,   0 zombie
>>>>>>>>>> Cpu(s): 94.2%us,  0.2%sy,  0.0%ni,  5.6%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
>>>>>>>>>> Mem:  32943180k total, 32615884k used,   327296k free,   709564k buffers
>>>>>>>>>> Swap:        0k total,        0k used,        0k free, 24612256k cached
>>>>>>>>>>
>>>>>>>>>> PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>>>>>>>>>> 29298 nuin      20   0 6768m 1.5g  28m S 752.3  4.7   8267:28 java
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Cheers
>>>>>>>>>> Paulo
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> On Dec 21, 2018, at 4:51 AM, sergio contrino <[hidden email]> wrote:
>>>>>>>>>>>
>>>>>>>>>>> hi paulo,
>>>>>>>>>>> it does not look ok and i would consider stopping it if still going on. i would check your memory settings and availability in your machine (i suppose you are running postgres there as well, possibly anything else?).
>>>>>>>>>>> if all seems fine, i could run a test here if you share your data file.
>>>>>>>>>>> but it is our last work day here until 2nd jan, so it could take us a while to assess this issue.
>>>>>>>>>>> thanks (and merry christmas!)
>>>>>>>>>>> sergio
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 21/12/2018 05:06, Paulo Nuin wrote:
>>>>>>>>>>>> Hi everyone
>>>>>>>>>>>> In my adventure of trying to build the DB with 2.x I started trying our large data sources (XML files from AceDB dumps) and I am noticing that the process is extremely slow, something like 400% or more. I started with our gene source, which is about 12Gb and usually takes a couple of hours or a little more to load. We are not into more than 10 hours on the build
>>>>>>>>>>>> <==========---> 80% EXECUTING [10h 31m 39s]
>>>>>>>>>>>> and I only see
>>>>>>>>>>>> idle in trans   SELECT nextval('serial’)
>>>>>>>>>>>> on my Postgres. Is that normal? How can I speed up the process, if there’s a way to speed it up. If the trend continues we will jump from a couple of days of build time, to more than one week.
>>>>>>>>>>>> Thanks
>>>>>>>>>>>> Paulo
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> dev mailing list
>>>>>>>>>>>> [hidden email]
>>>>>>>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> sergio contrino                  InterMine, University of Cambridge
>>>>>>>>>>> https://sergiocontrino.github.io           http://www.intermine.org
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> dev mailing list
>>>>>>>>>>> [hidden email]
>>>>>>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> dev mailing list
>>>>>>>>> [hidden email]
>>>>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>>>
>>>>>> --
>>>>>> sergio contrino                  InterMine, University of Cambridge
>>>>>> https://sergiocontrino.github.io           http://www.intermine.org
>>
>> --
>> sergio contrino                  InterMine, University of Cambridge
>> https://sergiocontrino.github.io           http://www.intermine.org
>

_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Really slow build process

sergio contrino-2
hi paulo,
maybe we should try to test it here. could you send instructions (and file)?
thanks!
sergio

On 10/01/2019 14:38, Paulo Nuin wrote:

> Hi
>
> Made sure (again) that GRADLE_OPTS was optima (as reported previously), or very close to optimal, and I am currently at 23 hours to load one XML file and what I can see on my Postgres monitoring is
>
> idle in trans   SELECT nextval('serial’)
>
> There are a couple of things that might be wrong:
>
> - our code is incompatible (our loader was created for our XML files)
> - out Postgres is too old (we are at 9.2.24)
> - something on InterMine’s code
> - we need to add more memory to the process
>
> Ideas?
>
> Thanks
>
> Paulo
>
>
>
>> On Jan 9, 2019, at 7:10 AM, Paulo Nuin <[hidden email]> wrote:
>>
>> Ciao Sergio
>>
>> I have the exports on my .bashrc and I am sourcing it before every test just to be sure. I had to change a number of things due to heap problems and I am making sure I have the right options set.
>>
>> Our machines are on AWS and we have about 32 Gb of memory available the the maximum we setting to Java (initially for ant) which is somewhat a tradeoff between performance and keeping other resources working.
>>
>> We have Postgres in the same machine.
>>
>> I will try again and let you know.
>>
>> Thanks
>>
>> Paulo
>>
>>
>>
>>> On Jan 9, 2019, at 3:49 AM, sergio contrino <[hidden email]> wrote:
>>>
>>> hi paul,
>>> from your 2 last mails, it looks like your build is not using the GRADLE_OPTS setting you are exporting, not sure why.
>>>
>>> i don't know your machine setting (how much ram you have, do you use it only for the build, are you hosting postgres on it, if so what are your postgres settings regarding memory), but the last setting you sent seems reasonable.
>>> i could suggest:
>>>
>>> - go to the terminal where you are going to run the build. all the below commands should be run on this terminal
>>>
>>> - change your gradle opts (i understand you are on java 8) running the command:
>>>
>>> export GRADLE_OPTS="-server -Xmx16g -XX:+UseParallelGC -Xms2g -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99 -Dorg.gradle.daemon=false”
>>>
>>> - check this are the setting you get
>>> echo $GRADLE_OPTS
>>>
>>> - you could also run (just to have a snapshot of your memory situation)
>>> free -g
>>>
>>> if all seems fine
>>> - run the build (please insert the dump as mentioned earlier).
>>>
>>> - let us know the result
>>>
>>> and then we can assess the situation.
>>> thanks!
>>> sergio
>>>
>>>
>>>
>>> On 08/01/2019 18:36, Paulo Nuin wrote:
>>>> Hi
>>>> This is what I have on a “regular” terminal, I export this when running the build
>>>> export GRADLE_OPTS="-server -XX:MaxPermSize=1g -Xmx16g -XX:+UseParallelGC -Xms2g -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99”
>>>> And my usual ANT_OPTS is
>>>> export ANT_OPTS="-server -XX:MaxPermSize=512M -Xmx12g -XX:+UseParallelGC -Xms2g -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99 -XX:-UseGCOverheadLimit”
>>>> Cheers
>>>> Paulo
>>>>> On Jan 8, 2019, at 10:40 AM, Julie Sullivan <[hidden email]> wrote:
>>>>>
>>>>> Less than 2 GB of memory is really low for an InterMine build, especially if the file you are parsing is 12 GB.
>>>>>
>>>>> If it's a build machine, I would give as much memory as you can. You will see a speed increase!
>>>>>
>>>>>
>>>>> On 08/01/2019 17:36, Paulo Nuin wrote:
>>>>>> Hi
>>>>>> I will try, but at the moment I am just testing the build and see if different sources work.
>>>>>> What is the recommended Postgres version?
>>>>>> Cheers
>>>>>> Paulo
>>>>>>> On Jan 8, 2019, at 10:23 AM, sergio contrino <[hidden email]> wrote:
>>>>>>>
>>>>>>> hi paulo!
>>>>>>> could you set
>>>>>>>
>>>>>>> export GRADLE_OPTS="-server -Xmx1700m -XX:+UseParallelGC -Xms1700m -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99 -Dorg.gradle.daemon=false"
>>>>>>>
>>>>>>> on the terminal where you will run the build? (i would also increase your Xmx if you have the possibility)
>>>>>>>
>>>>>>> as mentioned before, if you add the dump option just before the failing source, you can then test this part of the build without having to rebuild everything at every attempt (see https://intermine.readthedocs.io/en/latest/database/database-building/build-script/)
>>>>>>>
>>>>>>> please let us know how things proceed. thanks!
>>>>>>> sergio
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 08/01/2019 16:40, Paulo Nuin wrote:
>>>>>>>> Hi Julie
>>>>>>>> Here they are
>>>>>>>>> On Jan 4, 2019, at 4:51 AM, Julie Sullivan <[hidden email]> wrote:
>>>>>>>>>
>>>>>>>>> Hi Paulo,
>>>>>>>>>
>>>>>>>>> For the build that failed, can you run two commands for me? In the same console.
>>>>>>>>>
>>>>>>>>>> echo $GRADLE_OPTS
>>>>>>>>>
>>>>>>>> -server -XX:MaxPermSize=256M -Xmx1700m -XX:+UseParallelGC -Xms1700m -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99
>>>>>>>>> and
>>>>>>>>>
>>>>>>>>>> ./gradlew —stop
>>>>>>>> OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256M; support was removed in 8.0
>>>>>>>> No Gradle daemons are running.
>>>>>>>> Cheers
>>>>>>>> Paulo
>>>>>>>>>
>>>>>>>>> Can you tell me what those two commands produce?
>>>>>>>>>
>>>>>>>>> (be sure to use the same console, I think we had issues with AWS and ENV vars before? maybe?)
>>>>>>>>>
>>>>>>>>> Thanks!
>>>>>>>>> Julie
>>>>>>>>>
>>>>>>>>> PS. Happy New Year! :)
>>>>>>>>>
>>>>>>>>> On 21/12/2018 15:51, Paulo Nuin wrote:
>>>>>>>>>> Hi Sergio
>>>>>>>>>> And after 21 hours:
>>>>>>>>>> FAILURE: Build failed with an exception.
>>>>>>>>>> * What went wrong:
>>>>>>>>>> Execution failed for task ':dbmodel:retrieveSingleSource'.
>>>>>>>>>>> java.lang.OutOfMemoryError: GC overhead limit exceeded
>>>>>>>>>> Cheers
>>>>>>>>>> Paulo
>>>>>>>>>>> On Dec 21, 2018, at 6:02 AM, Paulo Nuin <[hidden email]> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi Sergio
>>>>>>>>>>>
>>>>>>>>>>> It is still going on, and the only major things running on the machine are Java and Postgres, This is how it looks like
>>>>>>>>>>>
>>>>>>>>>>> top - 13:01:13 up 187 days, 12:37,  2 users,  load average: 7.36, 7.44, 7.53
>>>>>>>>>>> Tasks: 240 total,   1 running, 184 sleeping,   0 stopped,   0 zombie
>>>>>>>>>>> Cpu(s): 94.2%us,  0.2%sy,  0.0%ni,  5.6%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
>>>>>>>>>>> Mem:  32943180k total, 32615884k used,   327296k free,   709564k buffers
>>>>>>>>>>> Swap:        0k total,        0k used,        0k free, 24612256k cached
>>>>>>>>>>>
>>>>>>>>>>> PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>>>>>>>>>>> 29298 nuin      20   0 6768m 1.5g  28m S 752.3  4.7   8267:28 java
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Cheers
>>>>>>>>>>> Paulo
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> On Dec 21, 2018, at 4:51 AM, sergio contrino <[hidden email]> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> hi paulo,
>>>>>>>>>>>> it does not look ok and i would consider stopping it if still going on. i would check your memory settings and availability in your machine (i suppose you are running postgres there as well, possibly anything else?).
>>>>>>>>>>>> if all seems fine, i could run a test here if you share your data file.
>>>>>>>>>>>> but it is our last work day here until 2nd jan, so it could take us a while to assess this issue.
>>>>>>>>>>>> thanks (and merry christmas!)
>>>>>>>>>>>> sergio
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 21/12/2018 05:06, Paulo Nuin wrote:
>>>>>>>>>>>>> Hi everyone
>>>>>>>>>>>>> In my adventure of trying to build the DB with 2.x I started trying our large data sources (XML files from AceDB dumps) and I am noticing that the process is extremely slow, something like 400% or more. I started with our gene source, which is about 12Gb and usually takes a couple of hours or a little more to load. We are not into more than 10 hours on the build
>>>>>>>>>>>>> <==========---> 80% EXECUTING [10h 31m 39s]
>>>>>>>>>>>>> and I only see
>>>>>>>>>>>>> idle in trans   SELECT nextval('serial’)
>>>>>>>>>>>>> on my Postgres. Is that normal? How can I speed up the process, if there’s a way to speed it up. If the trend continues we will jump from a couple of days of build time, to more than one week.
>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>> Paulo
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> dev mailing list
>>>>>>>>>>>>> [hidden email]
>>>>>>>>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> sergio contrino                  InterMine, University of Cambridge
>>>>>>>>>>>> https://sergiocontrino.github.io           http://www.intermine.org
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> dev mailing list
>>>>>>>>>>>> [hidden email]
>>>>>>>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> dev mailing list
>>>>>>>>>> [hidden email]
>>>>>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>>>>
>>>>>>> --
>>>>>>> sergio contrino                  InterMine, University of Cambridge
>>>>>>> https://sergiocontrino.github.io           http://www.intermine.org
>>>
>>> --
>>> sergio contrino                  InterMine, University of Cambridge
>>> https://sergiocontrino.github.io           http://www.intermine.org
>>
>

--
sergio contrino                  InterMine, University of Cambridge
https://sergiocontrino.github.io           http://www.intermine.org
_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Really slow build process

Paulo Nuin
Hi

I will make a package with the code we have and the file I am testing and I will put on Dropbox. I will probably test a smaller XML before sending things.

Cheers
Paulo



> On Jan 10, 2019, at 8:57 AM, sergio contrino <[hidden email]> wrote:
>
> hi paulo,
> maybe we should try to test it here. could you send instructions (and file)?
> thanks!
> sergio
>
> On 10/01/2019 14:38, Paulo Nuin wrote:
>> Hi
>> Made sure (again) that GRADLE_OPTS was optima (as reported previously), or very close to optimal, and I am currently at 23 hours to load one XML file and what I can see on my Postgres monitoring is
>> idle in trans   SELECT nextval('serial’)
>> There are a couple of things that might be wrong:
>> - our code is incompatible (our loader was created for our XML files)
>> - out Postgres is too old (we are at 9.2.24)
>> - something on InterMine’s code
>> - we need to add more memory to the process
>> Ideas?
>> Thanks
>> Paulo
>>> On Jan 9, 2019, at 7:10 AM, Paulo Nuin <[hidden email]> wrote:
>>>
>>> Ciao Sergio
>>>
>>> I have the exports on my .bashrc and I am sourcing it before every test just to be sure. I had to change a number of things due to heap problems and I am making sure I have the right options set.
>>>
>>> Our machines are on AWS and we have about 32 Gb of memory available the the maximum we setting to Java (initially for ant) which is somewhat a tradeoff between performance and keeping other resources working.
>>>
>>> We have Postgres in the same machine.
>>>
>>> I will try again and let you know.
>>>
>>> Thanks
>>>
>>> Paulo
>>>
>>>
>>>
>>>> On Jan 9, 2019, at 3:49 AM, sergio contrino <[hidden email]> wrote:
>>>>
>>>> hi paul,
>>>> from your 2 last mails, it looks like your build is not using the GRADLE_OPTS setting you are exporting, not sure why.
>>>>
>>>> i don't know your machine setting (how much ram you have, do you use it only for the build, are you hosting postgres on it, if so what are your postgres settings regarding memory), but the last setting you sent seems reasonable.
>>>> i could suggest:
>>>>
>>>> - go to the terminal where you are going to run the build. all the below commands should be run on this terminal
>>>>
>>>> - change your gradle opts (i understand you are on java 8) running the command:
>>>>
>>>> export GRADLE_OPTS="-server -Xmx16g -XX:+UseParallelGC -Xms2g -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99 -Dorg.gradle.daemon=false”
>>>>
>>>> - check this are the setting you get
>>>> echo $GRADLE_OPTS
>>>>
>>>> - you could also run (just to have a snapshot of your memory situation)
>>>> free -g
>>>>
>>>> if all seems fine
>>>> - run the build (please insert the dump as mentioned earlier).
>>>>
>>>> - let us know the result
>>>>
>>>> and then we can assess the situation.
>>>> thanks!
>>>> sergio
>>>>
>>>>
>>>>
>>>> On 08/01/2019 18:36, Paulo Nuin wrote:
>>>>> Hi
>>>>> This is what I have on a “regular” terminal, I export this when running the build
>>>>> export GRADLE_OPTS="-server -XX:MaxPermSize=1g -Xmx16g -XX:+UseParallelGC -Xms2g -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99”
>>>>> And my usual ANT_OPTS is
>>>>> export ANT_OPTS="-server -XX:MaxPermSize=512M -Xmx12g -XX:+UseParallelGC -Xms2g -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99 -XX:-UseGCOverheadLimit”
>>>>> Cheers
>>>>> Paulo
>>>>>> On Jan 8, 2019, at 10:40 AM, Julie Sullivan <[hidden email]> wrote:
>>>>>>
>>>>>> Less than 2 GB of memory is really low for an InterMine build, especially if the file you are parsing is 12 GB.
>>>>>>
>>>>>> If it's a build machine, I would give as much memory as you can. You will see a speed increase!
>>>>>>
>>>>>>
>>>>>> On 08/01/2019 17:36, Paulo Nuin wrote:
>>>>>>> Hi
>>>>>>> I will try, but at the moment I am just testing the build and see if different sources work.
>>>>>>> What is the recommended Postgres version?
>>>>>>> Cheers
>>>>>>> Paulo
>>>>>>>> On Jan 8, 2019, at 10:23 AM, sergio contrino <[hidden email]> wrote:
>>>>>>>>
>>>>>>>> hi paulo!
>>>>>>>> could you set
>>>>>>>>
>>>>>>>> export GRADLE_OPTS="-server -Xmx1700m -XX:+UseParallelGC -Xms1700m -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99 -Dorg.gradle.daemon=false"
>>>>>>>>
>>>>>>>> on the terminal where you will run the build? (i would also increase your Xmx if you have the possibility)
>>>>>>>>
>>>>>>>> as mentioned before, if you add the dump option just before the failing source, you can then test this part of the build without having to rebuild everything at every attempt (see https://intermine.readthedocs.io/en/latest/database/database-building/build-script/)
>>>>>>>>
>>>>>>>> please let us know how things proceed. thanks!
>>>>>>>> sergio
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 08/01/2019 16:40, Paulo Nuin wrote:
>>>>>>>>> Hi Julie
>>>>>>>>> Here they are
>>>>>>>>>> On Jan 4, 2019, at 4:51 AM, Julie Sullivan <[hidden email]> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi Paulo,
>>>>>>>>>>
>>>>>>>>>> For the build that failed, can you run two commands for me? In the same console.
>>>>>>>>>>
>>>>>>>>>>> echo $GRADLE_OPTS
>>>>>>>>>>
>>>>>>>>> -server -XX:MaxPermSize=256M -Xmx1700m -XX:+UseParallelGC -Xms1700m -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99
>>>>>>>>>> and
>>>>>>>>>>
>>>>>>>>>>> ./gradlew —stop
>>>>>>>>> OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256M; support was removed in 8.0
>>>>>>>>> No Gradle daemons are running.
>>>>>>>>> Cheers
>>>>>>>>> Paulo
>>>>>>>>>>
>>>>>>>>>> Can you tell me what those two commands produce?
>>>>>>>>>>
>>>>>>>>>> (be sure to use the same console, I think we had issues with AWS and ENV vars before? maybe?)
>>>>>>>>>>
>>>>>>>>>> Thanks!
>>>>>>>>>> Julie
>>>>>>>>>>
>>>>>>>>>> PS. Happy New Year! :)
>>>>>>>>>>
>>>>>>>>>> On 21/12/2018 15:51, Paulo Nuin wrote:
>>>>>>>>>>> Hi Sergio
>>>>>>>>>>> And after 21 hours:
>>>>>>>>>>> FAILURE: Build failed with an exception.
>>>>>>>>>>> * What went wrong:
>>>>>>>>>>> Execution failed for task ':dbmodel:retrieveSingleSource'.
>>>>>>>>>>>> java.lang.OutOfMemoryError: GC overhead limit exceeded
>>>>>>>>>>> Cheers
>>>>>>>>>>> Paulo
>>>>>>>>>>>> On Dec 21, 2018, at 6:02 AM, Paulo Nuin <[hidden email]> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Sergio
>>>>>>>>>>>>
>>>>>>>>>>>> It is still going on, and the only major things running on the machine are Java and Postgres, This is how it looks like
>>>>>>>>>>>>
>>>>>>>>>>>> top - 13:01:13 up 187 days, 12:37,  2 users,  load average: 7.36, 7.44, 7.53
>>>>>>>>>>>> Tasks: 240 total,   1 running, 184 sleeping,   0 stopped,   0 zombie
>>>>>>>>>>>> Cpu(s): 94.2%us,  0.2%sy,  0.0%ni,  5.6%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
>>>>>>>>>>>> Mem:  32943180k total, 32615884k used,   327296k free,   709564k buffers
>>>>>>>>>>>> Swap:        0k total,        0k used,        0k free, 24612256k cached
>>>>>>>>>>>>
>>>>>>>>>>>> PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>>>>>>>>>>>> 29298 nuin      20   0 6768m 1.5g  28m S 752.3  4.7   8267:28 java
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers
>>>>>>>>>>>> Paulo
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> On Dec 21, 2018, at 4:51 AM, sergio contrino <[hidden email]> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> hi paulo,
>>>>>>>>>>>>> it does not look ok and i would consider stopping it if still going on. i would check your memory settings and availability in your machine (i suppose you are running postgres there as well, possibly anything else?).
>>>>>>>>>>>>> if all seems fine, i could run a test here if you share your data file.
>>>>>>>>>>>>> but it is our last work day here until 2nd jan, so it could take us a while to assess this issue.
>>>>>>>>>>>>> thanks (and merry christmas!)
>>>>>>>>>>>>> sergio
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 21/12/2018 05:06, Paulo Nuin wrote:
>>>>>>>>>>>>>> Hi everyone
>>>>>>>>>>>>>> In my adventure of trying to build the DB with 2.x I started trying our large data sources (XML files from AceDB dumps) and I am noticing that the process is extremely slow, something like 400% or more. I started with our gene source, which is about 12Gb and usually takes a couple of hours or a little more to load. We are not into more than 10 hours on the build
>>>>>>>>>>>>>> <==========---> 80% EXECUTING [10h 31m 39s]
>>>>>>>>>>>>>> and I only see
>>>>>>>>>>>>>> idle in trans   SELECT nextval('serial’)
>>>>>>>>>>>>>> on my Postgres. Is that normal? How can I speed up the process, if there’s a way to speed it up. If the trend continues we will jump from a couple of days of build time, to more than one week.
>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>> Paulo
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> dev mailing list
>>>>>>>>>>>>>> [hidden email]
>>>>>>>>>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> sergio contrino                  InterMine, University of Cambridge
>>>>>>>>>>>>> https://sergiocontrino.github.io           http://www.intermine.org
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> dev mailing list
>>>>>>>>>>>>> [hidden email]
>>>>>>>>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> dev mailing list
>>>>>>>>>>> [hidden email]
>>>>>>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>>>>>
>>>>>>>> --
>>>>>>>> sergio contrino                  InterMine, University of Cambridge
>>>>>>>> https://sergiocontrino.github.io           http://www.intermine.org
>>>>
>>>> --
>>>> sergio contrino                  InterMine, University of Cambridge
>>>> https://sergiocontrino.github.io           http://www.intermine.org
>>>
>
> --
> sergio contrino                  InterMine, University of Cambridge
> https://sergiocontrino.github.io           http://www.intermine.org

_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Really slow build process

sergio contrino-2
thank you paul, you could also put the code on github.
sergio

On 10/01/2019 16:24, Paulo Nuin wrote:

> Hi
>
> I will make a package with the code we have and the file I am testing and I will put on Dropbox. I will probably test a smaller XML before sending things.
>
> Cheers
> Paulo
>
>
>
>> On Jan 10, 2019, at 8:57 AM, sergio contrino <[hidden email]> wrote:
>>
>> hi paulo,
>> maybe we should try to test it here. could you send instructions (and file)?
>> thanks!
>> sergio
>>
>> On 10/01/2019 14:38, Paulo Nuin wrote:
>>> Hi
>>> Made sure (again) that GRADLE_OPTS was optima (as reported previously), or very close to optimal, and I am currently at 23 hours to load one XML file and what I can see on my Postgres monitoring is
>>> idle in trans   SELECT nextval('serial’)
>>> There are a couple of things that might be wrong:
>>> - our code is incompatible (our loader was created for our XML files)
>>> - out Postgres is too old (we are at 9.2.24)
>>> - something on InterMine’s code
>>> - we need to add more memory to the process
>>> Ideas?
>>> Thanks
>>> Paulo
>>>> On Jan 9, 2019, at 7:10 AM, Paulo Nuin <[hidden email]> wrote:
>>>>
>>>> Ciao Sergio
>>>>
>>>> I have the exports on my .bashrc and I am sourcing it before every test just to be sure. I had to change a number of things due to heap problems and I am making sure I have the right options set.
>>>>
>>>> Our machines are on AWS and we have about 32 Gb of memory available the the maximum we setting to Java (initially for ant) which is somewhat a tradeoff between performance and keeping other resources working.
>>>>
>>>> We have Postgres in the same machine.
>>>>
>>>> I will try again and let you know.
>>>>
>>>> Thanks
>>>>
>>>> Paulo
>>>>
>>>>
>>>>
>>>>> On Jan 9, 2019, at 3:49 AM, sergio contrino <[hidden email]> wrote:
>>>>>
>>>>> hi paul,
>>>>> from your 2 last mails, it looks like your build is not using the GRADLE_OPTS setting you are exporting, not sure why.
>>>>>
>>>>> i don't know your machine setting (how much ram you have, do you use it only for the build, are you hosting postgres on it, if so what are your postgres settings regarding memory), but the last setting you sent seems reasonable.
>>>>> i could suggest:
>>>>>
>>>>> - go to the terminal where you are going to run the build. all the below commands should be run on this terminal
>>>>>
>>>>> - change your gradle opts (i understand you are on java 8) running the command:
>>>>>
>>>>> export GRADLE_OPTS="-server -Xmx16g -XX:+UseParallelGC -Xms2g -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99 -Dorg.gradle.daemon=false”
>>>>>
>>>>> - check this are the setting you get
>>>>> echo $GRADLE_OPTS
>>>>>
>>>>> - you could also run (just to have a snapshot of your memory situation)
>>>>> free -g
>>>>>
>>>>> if all seems fine
>>>>> - run the build (please insert the dump as mentioned earlier).
>>>>>
>>>>> - let us know the result
>>>>>
>>>>> and then we can assess the situation.
>>>>> thanks!
>>>>> sergio
>>>>>
>>>>>
>>>>>
>>>>> On 08/01/2019 18:36, Paulo Nuin wrote:
>>>>>> Hi
>>>>>> This is what I have on a “regular” terminal, I export this when running the build
>>>>>> export GRADLE_OPTS="-server -XX:MaxPermSize=1g -Xmx16g -XX:+UseParallelGC -Xms2g -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99”
>>>>>> And my usual ANT_OPTS is
>>>>>> export ANT_OPTS="-server -XX:MaxPermSize=512M -Xmx12g -XX:+UseParallelGC -Xms2g -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99 -XX:-UseGCOverheadLimit”
>>>>>> Cheers
>>>>>> Paulo
>>>>>>> On Jan 8, 2019, at 10:40 AM, Julie Sullivan <[hidden email]> wrote:
>>>>>>>
>>>>>>> Less than 2 GB of memory is really low for an InterMine build, especially if the file you are parsing is 12 GB.
>>>>>>>
>>>>>>> If it's a build machine, I would give as much memory as you can. You will see a speed increase!
>>>>>>>
>>>>>>>
>>>>>>> On 08/01/2019 17:36, Paulo Nuin wrote:
>>>>>>>> Hi
>>>>>>>> I will try, but at the moment I am just testing the build and see if different sources work.
>>>>>>>> What is the recommended Postgres version?
>>>>>>>> Cheers
>>>>>>>> Paulo
>>>>>>>>> On Jan 8, 2019, at 10:23 AM, sergio contrino <[hidden email]> wrote:
>>>>>>>>>
>>>>>>>>> hi paulo!
>>>>>>>>> could you set
>>>>>>>>>
>>>>>>>>> export GRADLE_OPTS="-server -Xmx1700m -XX:+UseParallelGC -Xms1700m -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99 -Dorg.gradle.daemon=false"
>>>>>>>>>
>>>>>>>>> on the terminal where you will run the build? (i would also increase your Xmx if you have the possibility)
>>>>>>>>>
>>>>>>>>> as mentioned before, if you add the dump option just before the failing source, you can then test this part of the build without having to rebuild everything at every attempt (see https://intermine.readthedocs.io/en/latest/database/database-building/build-script/)
>>>>>>>>>
>>>>>>>>> please let us know how things proceed. thanks!
>>>>>>>>> sergio
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 08/01/2019 16:40, Paulo Nuin wrote:
>>>>>>>>>> Hi Julie
>>>>>>>>>> Here they are
>>>>>>>>>>> On Jan 4, 2019, at 4:51 AM, Julie Sullivan <[hidden email]> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi Paulo,
>>>>>>>>>>>
>>>>>>>>>>> For the build that failed, can you run two commands for me? In the same console.
>>>>>>>>>>>
>>>>>>>>>>>> echo $GRADLE_OPTS
>>>>>>>>>>>
>>>>>>>>>> -server -XX:MaxPermSize=256M -Xmx1700m -XX:+UseParallelGC -Xms1700m -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99
>>>>>>>>>>> and
>>>>>>>>>>>
>>>>>>>>>>>> ./gradlew —stop
>>>>>>>>>> OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256M; support was removed in 8.0
>>>>>>>>>> No Gradle daemons are running.
>>>>>>>>>> Cheers
>>>>>>>>>> Paulo
>>>>>>>>>>>
>>>>>>>>>>> Can you tell me what those two commands produce?
>>>>>>>>>>>
>>>>>>>>>>> (be sure to use the same console, I think we had issues with AWS and ENV vars before? maybe?)
>>>>>>>>>>>
>>>>>>>>>>> Thanks!
>>>>>>>>>>> Julie
>>>>>>>>>>>
>>>>>>>>>>> PS. Happy New Year! :)
>>>>>>>>>>>
>>>>>>>>>>> On 21/12/2018 15:51, Paulo Nuin wrote:
>>>>>>>>>>>> Hi Sergio
>>>>>>>>>>>> And after 21 hours:
>>>>>>>>>>>> FAILURE: Build failed with an exception.
>>>>>>>>>>>> * What went wrong:
>>>>>>>>>>>> Execution failed for task ':dbmodel:retrieveSingleSource'.
>>>>>>>>>>>>> java.lang.OutOfMemoryError: GC overhead limit exceeded
>>>>>>>>>>>> Cheers
>>>>>>>>>>>> Paulo
>>>>>>>>>>>>> On Dec 21, 2018, at 6:02 AM, Paulo Nuin <[hidden email]> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Sergio
>>>>>>>>>>>>>
>>>>>>>>>>>>> It is still going on, and the only major things running on the machine are Java and Postgres, This is how it looks like
>>>>>>>>>>>>>
>>>>>>>>>>>>> top - 13:01:13 up 187 days, 12:37,  2 users,  load average: 7.36, 7.44, 7.53
>>>>>>>>>>>>> Tasks: 240 total,   1 running, 184 sleeping,   0 stopped,   0 zombie
>>>>>>>>>>>>> Cpu(s): 94.2%us,  0.2%sy,  0.0%ni,  5.6%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
>>>>>>>>>>>>> Mem:  32943180k total, 32615884k used,   327296k free,   709564k buffers
>>>>>>>>>>>>> Swap:        0k total,        0k used,        0k free, 24612256k cached
>>>>>>>>>>>>>
>>>>>>>>>>>>> PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>>>>>>>>>>>>> 29298 nuin      20   0 6768m 1.5g  28m S 752.3  4.7   8267:28 java
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>> Paulo
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Dec 21, 2018, at 4:51 AM, sergio contrino <[hidden email]> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> hi paulo,
>>>>>>>>>>>>>> it does not look ok and i would consider stopping it if still going on. i would check your memory settings and availability in your machine (i suppose you are running postgres there as well, possibly anything else?).
>>>>>>>>>>>>>> if all seems fine, i could run a test here if you share your data file.
>>>>>>>>>>>>>> but it is our last work day here until 2nd jan, so it could take us a while to assess this issue.
>>>>>>>>>>>>>> thanks (and merry christmas!)
>>>>>>>>>>>>>> sergio
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 21/12/2018 05:06, Paulo Nuin wrote:
>>>>>>>>>>>>>>> Hi everyone
>>>>>>>>>>>>>>> In my adventure of trying to build the DB with 2.x I started trying our large data sources (XML files from AceDB dumps) and I am noticing that the process is extremely slow, something like 400% or more. I started with our gene source, which is about 12Gb and usually takes a couple of hours or a little more to load. We are not into more than 10 hours on the build
>>>>>>>>>>>>>>> <==========---> 80% EXECUTING [10h 31m 39s]
>>>>>>>>>>>>>>> and I only see
>>>>>>>>>>>>>>> idle in trans   SELECT nextval('serial’)
>>>>>>>>>>>>>>> on my Postgres. Is that normal? How can I speed up the process, if there’s a way to speed it up. If the trend continues we will jump from a couple of days of build time, to more than one week.
>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>> Paulo
>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>> dev mailing list
>>>>>>>>>>>>>>> [hidden email]
>>>>>>>>>>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> sergio contrino                  InterMine, University of Cambridge
>>>>>>>>>>>>>> https://sergiocontrino.github.io           http://www.intermine.org
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> dev mailing list
>>>>>>>>>>>>>> [hidden email]
>>>>>>>>>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> dev mailing list
>>>>>>>>>>>> [hidden email]
>>>>>>>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> sergio contrino                  InterMine, University of Cambridge
>>>>>>>>> https://sergiocontrino.github.io           http://www.intermine.org
>>>>>
>>>>> --
>>>>> sergio contrino                  InterMine, University of Cambridge
>>>>> https://sergiocontrino.github.io           http://www.intermine.org
>>>>
>>
>> --
>> sergio contrino                  InterMine, University of Cambridge
>> https://sergiocontrino.github.io           http://www.intermine.org
>

--
sergio contrino                  InterMine, University of Cambridge
https://sergiocontrino.github.io           http://www.intermine.org
_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Really slow build process

Paulo Nuin
I have to merge the repos and it might be faster to add to a tar ball. I also have a a newer version of Postgres on another machine and I will test there before anything.

I let you know anyway.

Cheers
Paulo

> On Jan 10, 2019, at 9:32 AM, sergio contrino <[hidden email]> wrote:
>
> thank you paul, you could also put the code on github.
> sergio
>
> On 10/01/2019 16:24, Paulo Nuin wrote:
>> Hi
>> I will make a package with the code we have and the file I am testing and I will put on Dropbox. I will probably test a smaller XML before sending things.
>> Cheers
>> Paulo
>>> On Jan 10, 2019, at 8:57 AM, sergio contrino <[hidden email]> wrote:
>>>
>>> hi paulo,
>>> maybe we should try to test it here. could you send instructions (and file)?
>>> thanks!
>>> sergio
>>>
>>> On 10/01/2019 14:38, Paulo Nuin wrote:
>>>> Hi
>>>> Made sure (again) that GRADLE_OPTS was optima (as reported previously), or very close to optimal, and I am currently at 23 hours to load one XML file and what I can see on my Postgres monitoring is
>>>> idle in trans   SELECT nextval('serial’)
>>>> There are a couple of things that might be wrong:
>>>> - our code is incompatible (our loader was created for our XML files)
>>>> - out Postgres is too old (we are at 9.2.24)
>>>> - something on InterMine’s code
>>>> - we need to add more memory to the process
>>>> Ideas?
>>>> Thanks
>>>> Paulo
>>>>> On Jan 9, 2019, at 7:10 AM, Paulo Nuin <[hidden email]> wrote:
>>>>>
>>>>> Ciao Sergio
>>>>>
>>>>> I have the exports on my .bashrc and I am sourcing it before every test just to be sure. I had to change a number of things due to heap problems and I am making sure I have the right options set.
>>>>>
>>>>> Our machines are on AWS and we have about 32 Gb of memory available the the maximum we setting to Java (initially for ant) which is somewhat a tradeoff between performance and keeping other resources working.
>>>>>
>>>>> We have Postgres in the same machine.
>>>>>
>>>>> I will try again and let you know.
>>>>>
>>>>> Thanks
>>>>>
>>>>> Paulo
>>>>>
>>>>>
>>>>>
>>>>>> On Jan 9, 2019, at 3:49 AM, sergio contrino <[hidden email]> wrote:
>>>>>>
>>>>>> hi paul,
>>>>>> from your 2 last mails, it looks like your build is not using the GRADLE_OPTS setting you are exporting, not sure why.
>>>>>>
>>>>>> i don't know your machine setting (how much ram you have, do you use it only for the build, are you hosting postgres on it, if so what are your postgres settings regarding memory), but the last setting you sent seems reasonable.
>>>>>> i could suggest:
>>>>>>
>>>>>> - go to the terminal where you are going to run the build. all the below commands should be run on this terminal
>>>>>>
>>>>>> - change your gradle opts (i understand you are on java 8) running the command:
>>>>>>
>>>>>> export GRADLE_OPTS="-server -Xmx16g -XX:+UseParallelGC -Xms2g -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99 -Dorg.gradle.daemon=false”
>>>>>>
>>>>>> - check this are the setting you get
>>>>>> echo $GRADLE_OPTS
>>>>>>
>>>>>> - you could also run (just to have a snapshot of your memory situation)
>>>>>> free -g
>>>>>>
>>>>>> if all seems fine
>>>>>> - run the build (please insert the dump as mentioned earlier).
>>>>>>
>>>>>> - let us know the result
>>>>>>
>>>>>> and then we can assess the situation.
>>>>>> thanks!
>>>>>> sergio
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 08/01/2019 18:36, Paulo Nuin wrote:
>>>>>>> Hi
>>>>>>> This is what I have on a “regular” terminal, I export this when running the build
>>>>>>> export GRADLE_OPTS="-server -XX:MaxPermSize=1g -Xmx16g -XX:+UseParallelGC -Xms2g -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99”
>>>>>>> And my usual ANT_OPTS is
>>>>>>> export ANT_OPTS="-server -XX:MaxPermSize=512M -Xmx12g -XX:+UseParallelGC -Xms2g -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99 -XX:-UseGCOverheadLimit”
>>>>>>> Cheers
>>>>>>> Paulo
>>>>>>>> On Jan 8, 2019, at 10:40 AM, Julie Sullivan <[hidden email]> wrote:
>>>>>>>>
>>>>>>>> Less than 2 GB of memory is really low for an InterMine build, especially if the file you are parsing is 12 GB.
>>>>>>>>
>>>>>>>> If it's a build machine, I would give as much memory as you can. You will see a speed increase!
>>>>>>>>
>>>>>>>>
>>>>>>>> On 08/01/2019 17:36, Paulo Nuin wrote:
>>>>>>>>> Hi
>>>>>>>>> I will try, but at the moment I am just testing the build and see if different sources work.
>>>>>>>>> What is the recommended Postgres version?
>>>>>>>>> Cheers
>>>>>>>>> Paulo
>>>>>>>>>> On Jan 8, 2019, at 10:23 AM, sergio contrino <[hidden email]> wrote:
>>>>>>>>>>
>>>>>>>>>> hi paulo!
>>>>>>>>>> could you set
>>>>>>>>>>
>>>>>>>>>> export GRADLE_OPTS="-server -Xmx1700m -XX:+UseParallelGC -Xms1700m -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99 -Dorg.gradle.daemon=false"
>>>>>>>>>>
>>>>>>>>>> on the terminal where you will run the build? (i would also increase your Xmx if you have the possibility)
>>>>>>>>>>
>>>>>>>>>> as mentioned before, if you add the dump option just before the failing source, you can then test this part of the build without having to rebuild everything at every attempt (see https://intermine.readthedocs.io/en/latest/database/database-building/build-script/)
>>>>>>>>>>
>>>>>>>>>> please let us know how things proceed. thanks!
>>>>>>>>>> sergio
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 08/01/2019 16:40, Paulo Nuin wrote:
>>>>>>>>>>> Hi Julie
>>>>>>>>>>> Here they are
>>>>>>>>>>>> On Jan 4, 2019, at 4:51 AM, Julie Sullivan <[hidden email]> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Paulo,
>>>>>>>>>>>>
>>>>>>>>>>>> For the build that failed, can you run two commands for me? In the same console.
>>>>>>>>>>>>
>>>>>>>>>>>>> echo $GRADLE_OPTS
>>>>>>>>>>>>
>>>>>>>>>>> -server -XX:MaxPermSize=256M -Xmx1700m -XX:+UseParallelGC -Xms1700m -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99
>>>>>>>>>>>> and
>>>>>>>>>>>>
>>>>>>>>>>>>> ./gradlew —stop
>>>>>>>>>>> OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256M; support was removed in 8.0
>>>>>>>>>>> No Gradle daemons are running.
>>>>>>>>>>> Cheers
>>>>>>>>>>> Paulo
>>>>>>>>>>>>
>>>>>>>>>>>> Can you tell me what those two commands produce?
>>>>>>>>>>>>
>>>>>>>>>>>> (be sure to use the same console, I think we had issues with AWS and ENV vars before? maybe?)
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks!
>>>>>>>>>>>> Julie
>>>>>>>>>>>>
>>>>>>>>>>>> PS. Happy New Year! :)
>>>>>>>>>>>>
>>>>>>>>>>>> On 21/12/2018 15:51, Paulo Nuin wrote:
>>>>>>>>>>>>> Hi Sergio
>>>>>>>>>>>>> And after 21 hours:
>>>>>>>>>>>>> FAILURE: Build failed with an exception.
>>>>>>>>>>>>> * What went wrong:
>>>>>>>>>>>>> Execution failed for task ':dbmodel:retrieveSingleSource'.
>>>>>>>>>>>>>> java.lang.OutOfMemoryError: GC overhead limit exceeded
>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>> Paulo
>>>>>>>>>>>>>> On Dec 21, 2018, at 6:02 AM, Paulo Nuin <[hidden email]> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Sergio
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It is still going on, and the only major things running on the machine are Java and Postgres, This is how it looks like
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> top - 13:01:13 up 187 days, 12:37,  2 users,  load average: 7.36, 7.44, 7.53
>>>>>>>>>>>>>> Tasks: 240 total,   1 running, 184 sleeping,   0 stopped,   0 zombie
>>>>>>>>>>>>>> Cpu(s): 94.2%us,  0.2%sy,  0.0%ni,  5.6%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
>>>>>>>>>>>>>> Mem:  32943180k total, 32615884k used,   327296k free,   709564k buffers
>>>>>>>>>>>>>> Swap:        0k total,        0k used,        0k free, 24612256k cached
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>>>>>>>>>>>>>> 29298 nuin      20   0 6768m 1.5g  28m S 752.3  4.7   8267:28 java
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>> Paulo
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Dec 21, 2018, at 4:51 AM, sergio contrino <[hidden email]> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> hi paulo,
>>>>>>>>>>>>>>> it does not look ok and i would consider stopping it if still going on. i would check your memory settings and availability in your machine (i suppose you are running postgres there as well, possibly anything else?).
>>>>>>>>>>>>>>> if all seems fine, i could run a test here if you share your data file.
>>>>>>>>>>>>>>> but it is our last work day here until 2nd jan, so it could take us a while to assess this issue.
>>>>>>>>>>>>>>> thanks (and merry christmas!)
>>>>>>>>>>>>>>> sergio
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 21/12/2018 05:06, Paulo Nuin wrote:
>>>>>>>>>>>>>>>> Hi everyone
>>>>>>>>>>>>>>>> In my adventure of trying to build the DB with 2.x I started trying our large data sources (XML files from AceDB dumps) and I am noticing that the process is extremely slow, something like 400% or more. I started with our gene source, which is about 12Gb and usually takes a couple of hours or a little more to load. We are not into more than 10 hours on the build
>>>>>>>>>>>>>>>> <==========---> 80% EXECUTING [10h 31m 39s]
>>>>>>>>>>>>>>>> and I only see
>>>>>>>>>>>>>>>> idle in trans   SELECT nextval('serial’)
>>>>>>>>>>>>>>>> on my Postgres. Is that normal? How can I speed up the process, if there’s a way to speed it up. If the trend continues we will jump from a couple of days of build time, to more than one week.
>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>> Paulo
>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>> dev mailing list
>>>>>>>>>>>>>>>> [hidden email]
>>>>>>>>>>>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> sergio contrino                  InterMine, University of Cambridge
>>>>>>>>>>>>>>> https://sergiocontrino.github.io           http://www.intermine.org
>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>> dev mailing list
>>>>>>>>>>>>>>> [hidden email]
>>>>>>>>>>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> dev mailing list
>>>>>>>>>>>>> [hidden email]
>>>>>>>>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> sergio contrino                  InterMine, University of Cambridge
>>>>>>>>>> https://sergiocontrino.github.io           http://www.intermine.org
>>>>>>
>>>>>> --
>>>>>> sergio contrino                  InterMine, University of Cambridge
>>>>>> https://sergiocontrino.github.io           http://www.intermine.org
>>>>>
>>>
>>> --
>>> sergio contrino                  InterMine, University of Cambridge
>>> https://sergiocontrino.github.io           http://www.intermine.org
>
> --
> sergio contrino                  InterMine, University of Cambridge
> https://sergiocontrino.github.io           http://www.intermine.org

_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Really slow build process

Paulo Nuin
Hi


So, I did some tests. I setup the same thing locally on an iMacPro with 64 Gb of memory, my GRADLE_OPTS is this


-server -XX:MaxPermSize=1g -Xmx30g -XX:+UseParallelGC -Xms2g -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99

And I tried GO, which is using a native InterMine code on our side. After 12 minutes of processing, I got this

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':dbmodel:retrieveSingleSource'.
> java.lang.OutOfMemoryError: Java heap space

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

* Get more help at https://help.gradle.org

BUILD FAILED in 12m 25s
11 actionable tasks: 7 executed, 4 up-to-date
Expiring Daemon because JVM Tenured space is exhausted
Expiring Daemon because JVM Tenured space is exhausted
Expiring Daemon because JVM Tenured space is exhausted
Expiring Daemon because JVM Tenured space is exhausted
Expiring Daemon because JVM Tenured space is exhausted
Expiring Daemon because JVM Tenured space is exhausted
Expiring Daemon because JVM Tenured space is exhausted
Expiring Daemon because JVM Tenured space is exhausted
Expiring Daemon because JVM Tenured space is exhausted
Expiring Daemon because JVM Tenured space is exhausted

I monitored execution and my load at the time was 1100% (Apple’s Activity Monitor) and I am running Intel Xeons.

How to solve this memory problem?

Regarding, our XML sources, I tried with a smaller file and I got an error, which I will send in a separate email.

Thanks

Paulo



> On Jan 10, 2019, at 9:34 AM, Paulo Nuin <[hidden email]> wrote:
>
> I have to merge the repos and it might be faster to add to a tar ball. I also have a a newer version of Postgres on another machine and I will test there before anything.
>
> I let you know anyway.
>
> Cheers
> Paulo
>
>> On Jan 10, 2019, at 9:32 AM, sergio contrino <[hidden email]> wrote:
>>
>> thank you paul, you could also put the code on github.
>> sergio
>>
>> On 10/01/2019 16:24, Paulo Nuin wrote:
>>> Hi
>>> I will make a package with the code we have and the file I am testing and I will put on Dropbox. I will probably test a smaller XML before sending things.
>>> Cheers
>>> Paulo
>>>> On Jan 10, 2019, at 8:57 AM, sergio contrino <[hidden email]> wrote:
>>>>
>>>> hi paulo,
>>>> maybe we should try to test it here. could you send instructions (and file)?
>>>> thanks!
>>>> sergio
>>>>
>>>> On 10/01/2019 14:38, Paulo Nuin wrote:
>>>>> Hi
>>>>> Made sure (again) that GRADLE_OPTS was optima (as reported previously), or very close to optimal, and I am currently at 23 hours to load one XML file and what I can see on my Postgres monitoring is
>>>>> idle in trans   SELECT nextval('serial’)
>>>>> There are a couple of things that might be wrong:
>>>>> - our code is incompatible (our loader was created for our XML files)
>>>>> - out Postgres is too old (we are at 9.2.24)
>>>>> - something on InterMine’s code
>>>>> - we need to add more memory to the process
>>>>> Ideas?
>>>>> Thanks
>>>>> Paulo
>>>>>> On Jan 9, 2019, at 7:10 AM, Paulo Nuin <[hidden email]> wrote:
>>>>>>
>>>>>> Ciao Sergio
>>>>>>
>>>>>> I have the exports on my .bashrc and I am sourcing it before every test just to be sure. I had to change a number of things due to heap problems and I am making sure I have the right options set.
>>>>>>
>>>>>> Our machines are on AWS and we have about 32 Gb of memory available the the maximum we setting to Java (initially for ant) which is somewhat a tradeoff between performance and keeping other resources working.
>>>>>>
>>>>>> We have Postgres in the same machine.
>>>>>>
>>>>>> I will try again and let you know.
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> Paulo
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On Jan 9, 2019, at 3:49 AM, sergio contrino <[hidden email]> wrote:
>>>>>>>
>>>>>>> hi paul,
>>>>>>> from your 2 last mails, it looks like your build is not using the GRADLE_OPTS setting you are exporting, not sure why.
>>>>>>>
>>>>>>> i don't know your machine setting (how much ram you have, do you use it only for the build, are you hosting postgres on it, if so what are your postgres settings regarding memory), but the last setting you sent seems reasonable.
>>>>>>> i could suggest:
>>>>>>>
>>>>>>> - go to the terminal where you are going to run the build. all the below commands should be run on this terminal
>>>>>>>
>>>>>>> - change your gradle opts (i understand you are on java 8) running the command:
>>>>>>>
>>>>>>> export GRADLE_OPTS="-server -Xmx16g -XX:+UseParallelGC -Xms2g -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99 -Dorg.gradle.daemon=false”
>>>>>>>
>>>>>>> - check this are the setting you get
>>>>>>> echo $GRADLE_OPTS
>>>>>>>
>>>>>>> - you could also run (just to have a snapshot of your memory situation)
>>>>>>> free -g
>>>>>>>
>>>>>>> if all seems fine
>>>>>>> - run the build (please insert the dump as mentioned earlier).
>>>>>>>
>>>>>>> - let us know the result
>>>>>>>
>>>>>>> and then we can assess the situation.
>>>>>>> thanks!
>>>>>>> sergio
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 08/01/2019 18:36, Paulo Nuin wrote:
>>>>>>>> Hi
>>>>>>>> This is what I have on a “regular” terminal, I export this when running the build
>>>>>>>> export GRADLE_OPTS="-server -XX:MaxPermSize=1g -Xmx16g -XX:+UseParallelGC -Xms2g -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99”
>>>>>>>> And my usual ANT_OPTS is
>>>>>>>> export ANT_OPTS="-server -XX:MaxPermSize=512M -Xmx12g -XX:+UseParallelGC -Xms2g -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99 -XX:-UseGCOverheadLimit”
>>>>>>>> Cheers
>>>>>>>> Paulo
>>>>>>>>> On Jan 8, 2019, at 10:40 AM, Julie Sullivan <[hidden email]> wrote:
>>>>>>>>>
>>>>>>>>> Less than 2 GB of memory is really low for an InterMine build, especially if the file you are parsing is 12 GB.
>>>>>>>>>
>>>>>>>>> If it's a build machine, I would give as much memory as you can. You will see a speed increase!
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 08/01/2019 17:36, Paulo Nuin wrote:
>>>>>>>>>> Hi
>>>>>>>>>> I will try, but at the moment I am just testing the build and see if different sources work.
>>>>>>>>>> What is the recommended Postgres version?
>>>>>>>>>> Cheers
>>>>>>>>>> Paulo
>>>>>>>>>>> On Jan 8, 2019, at 10:23 AM, sergio contrino <[hidden email]> wrote:
>>>>>>>>>>>
>>>>>>>>>>> hi paulo!
>>>>>>>>>>> could you set
>>>>>>>>>>>
>>>>>>>>>>> export GRADLE_OPTS="-server -Xmx1700m -XX:+UseParallelGC -Xms1700m -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99 -Dorg.gradle.daemon=false"
>>>>>>>>>>>
>>>>>>>>>>> on the terminal where you will run the build? (i would also increase your Xmx if you have the possibility)
>>>>>>>>>>>
>>>>>>>>>>> as mentioned before, if you add the dump option just before the failing source, you can then test this part of the build without having to rebuild everything at every attempt (see https://intermine.readthedocs.io/en/latest/database/database-building/build-script/)
>>>>>>>>>>>
>>>>>>>>>>> please let us know how things proceed. thanks!
>>>>>>>>>>> sergio
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 08/01/2019 16:40, Paulo Nuin wrote:
>>>>>>>>>>>> Hi Julie
>>>>>>>>>>>> Here they are
>>>>>>>>>>>>> On Jan 4, 2019, at 4:51 AM, Julie Sullivan <[hidden email]> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Paulo,
>>>>>>>>>>>>>
>>>>>>>>>>>>> For the build that failed, can you run two commands for me? In the same console.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> echo $GRADLE_OPTS
>>>>>>>>>>>>>
>>>>>>>>>>>> -server -XX:MaxPermSize=256M -Xmx1700m -XX:+UseParallelGC -Xms1700m -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99
>>>>>>>>>>>>> and
>>>>>>>>>>>>>
>>>>>>>>>>>>>> ./gradlew —stop
>>>>>>>>>>>> OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256M; support was removed in 8.0
>>>>>>>>>>>> No Gradle daemons are running.
>>>>>>>>>>>> Cheers
>>>>>>>>>>>> Paulo
>>>>>>>>>>>>>
>>>>>>>>>>>>> Can you tell me what those two commands produce?
>>>>>>>>>>>>>
>>>>>>>>>>>>> (be sure to use the same console, I think we had issues with AWS and ENV vars before? maybe?)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>> Julie
>>>>>>>>>>>>>
>>>>>>>>>>>>> PS. Happy New Year! :)
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 21/12/2018 15:51, Paulo Nuin wrote:
>>>>>>>>>>>>>> Hi Sergio
>>>>>>>>>>>>>> And after 21 hours:
>>>>>>>>>>>>>> FAILURE: Build failed with an exception.
>>>>>>>>>>>>>> * What went wrong:
>>>>>>>>>>>>>> Execution failed for task ':dbmodel:retrieveSingleSource'.
>>>>>>>>>>>>>>> java.lang.OutOfMemoryError: GC overhead limit exceeded
>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>> Paulo
>>>>>>>>>>>>>>> On Dec 21, 2018, at 6:02 AM, Paulo Nuin <[hidden email]> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Sergio
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It is still going on, and the only major things running on the machine are Java and Postgres, This is how it looks like
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> top - 13:01:13 up 187 days, 12:37,  2 users,  load average: 7.36, 7.44, 7.53
>>>>>>>>>>>>>>> Tasks: 240 total,   1 running, 184 sleeping,   0 stopped,   0 zombie
>>>>>>>>>>>>>>> Cpu(s): 94.2%us,  0.2%sy,  0.0%ni,  5.6%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
>>>>>>>>>>>>>>> Mem:  32943180k total, 32615884k used,   327296k free,   709564k buffers
>>>>>>>>>>>>>>> Swap:        0k total,        0k used,        0k free, 24612256k cached
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>>>>>>>>>>>>>>> 29298 nuin      20   0 6768m 1.5g  28m S 752.3  4.7   8267:28 java
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>> Paulo
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Dec 21, 2018, at 4:51 AM, sergio contrino <[hidden email]> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> hi paulo,
>>>>>>>>>>>>>>>> it does not look ok and i would consider stopping it if still going on. i would check your memory settings and availability in your machine (i suppose you are running postgres there as well, possibly anything else?).
>>>>>>>>>>>>>>>> if all seems fine, i could run a test here if you share your data file.
>>>>>>>>>>>>>>>> but it is our last work day here until 2nd jan, so it could take us a while to assess this issue.
>>>>>>>>>>>>>>>> thanks (and merry christmas!)
>>>>>>>>>>>>>>>> sergio
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 21/12/2018 05:06, Paulo Nuin wrote:
>>>>>>>>>>>>>>>>> Hi everyone
>>>>>>>>>>>>>>>>> In my adventure of trying to build the DB with 2.x I started trying our large data sources (XML files from AceDB dumps) and I am noticing that the process is extremely slow, something like 400% or more. I started with our gene source, which is about 12Gb and usually takes a couple of hours or a little more to load. We are not into more than 10 hours on the build
>>>>>>>>>>>>>>>>> <==========---> 80% EXECUTING [10h 31m 39s]
>>>>>>>>>>>>>>>>> and I only see
>>>>>>>>>>>>>>>>> idle in trans   SELECT nextval('serial’)
>>>>>>>>>>>>>>>>> on my Postgres. Is that normal? How can I speed up the process, if there’s a way to speed it up. If the trend continues we will jump from a couple of days of build time, to more than one week.
>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>> Paulo
>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>> dev mailing list
>>>>>>>>>>>>>>>>> [hidden email]
>>>>>>>>>>>>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> sergio contrino                  InterMine, University of Cambridge
>>>>>>>>>>>>>>>> https://sergiocontrino.github.io           http://www.intermine.org
>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>> dev mailing list
>>>>>>>>>>>>>>>> [hidden email]
>>>>>>>>>>>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> dev mailing list
>>>>>>>>>>>>>> [hidden email]
>>>>>>>>>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> sergio contrino                  InterMine, University of Cambridge
>>>>>>>>>>> https://sergiocontrino.github.io           http://www.intermine.org
>>>>>>>
>>>>>>> --
>>>>>>> sergio contrino                  InterMine, University of Cambridge
>>>>>>> https://sergiocontrino.github.io           http://www.intermine.org
>>>>>>
>>>>
>>>> --
>>>> sergio contrino                  InterMine, University of Cambridge
>>>> https://sergiocontrino.github.io           http://www.intermine.org
>>
>> --
>> sergio contrino                  InterMine, University of Cambridge
>> https://sergiocontrino.github.io           http://www.intermine.org
>

_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Really slow build process

Paulo Nuin
Hi

I carried on with more tests on the iMac Pro. Now, loading the Gene.xml file that failed at 27 hours on AWS. I am currently at 14 hours, with similar output. From the 4 options I gave earlier in the thread

- out Postgres is too old (we are at 9.2.24)  
- we need to add more memory to the process

are not true, as I am loading with around 32 Gb available to the builder and I am using Postgres 10+. So the only options are

- something on InterMine’s code
- our code is incompatible (our loader was created for our XML files)

and I am assuming is the latter, maybe something that our code is doing that is making things slower.

Ideas?

Cheers
Paulo



> On Jan 10, 2019, at 4:54 PM, Paulo Nuin <[hidden email]> wrote:
>
> Hi
>
>
> So, I did some tests. I setup the same thing locally on an iMacPro with 64 Gb of memory, my GRADLE_OPTS is this
>
>
> -server -XX:MaxPermSize=1g -Xmx30g -XX:+UseParallelGC -Xms2g -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99
>
> And I tried GO, which is using a native InterMine code on our side. After 12 minutes of processing, I got this
>
> FAILURE: Build failed with an exception.
>
> * What went wrong:
> Execution failed for task ':dbmodel:retrieveSingleSource'.
>> java.lang.OutOfMemoryError: Java heap space
>
> * Try:
> Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
>
> * Get more help at https://help.gradle.org
>
> BUILD FAILED in 12m 25s
> 11 actionable tasks: 7 executed, 4 up-to-date
> Expiring Daemon because JVM Tenured space is exhausted
> Expiring Daemon because JVM Tenured space is exhausted
> Expiring Daemon because JVM Tenured space is exhausted
> Expiring Daemon because JVM Tenured space is exhausted
> Expiring Daemon because JVM Tenured space is exhausted
> Expiring Daemon because JVM Tenured space is exhausted
> Expiring Daemon because JVM Tenured space is exhausted
> Expiring Daemon because JVM Tenured space is exhausted
> Expiring Daemon because JVM Tenured space is exhausted
> Expiring Daemon because JVM Tenured space is exhausted
>
> I monitored execution and my load at the time was 1100% (Apple’s Activity Monitor) and I am running Intel Xeons.
>
> How to solve this memory problem?
>
> Regarding, our XML sources, I tried with a smaller file and I got an error, which I will send in a separate email.
>
> Thanks
>
> Paulo
>
>
>
>> On Jan 10, 2019, at 9:34 AM, Paulo Nuin <[hidden email]> wrote:
>>
>> I have to merge the repos and it might be faster to add to a tar ball. I also have a a newer version of Postgres on another machine and I will test there before anything.
>>
>> I let you know anyway.
>>
>> Cheers
>> Paulo
>>
>>> On Jan 10, 2019, at 9:32 AM, sergio contrino <[hidden email]> wrote:
>>>
>>> thank you paul, you could also put the code on github.
>>> sergio
>>>
>>> On 10/01/2019 16:24, Paulo Nuin wrote:
>>>> Hi
>>>> I will make a package with the code we have and the file I am testing and I will put on Dropbox. I will probably test a smaller XML before sending things.
>>>> Cheers
>>>> Paulo
>>>>> On Jan 10, 2019, at 8:57 AM, sergio contrino <[hidden email]> wrote:
>>>>>
>>>>> hi paulo,
>>>>> maybe we should try to test it here. could you send instructions (and file)?
>>>>> thanks!
>>>>> sergio
>>>>>
>>>>> On 10/01/2019 14:38, Paulo Nuin wrote:
>>>>>> Hi
>>>>>> Made sure (again) that GRADLE_OPTS was optima (as reported previously), or very close to optimal, and I am currently at 23 hours to load one XML file and what I can see on my Postgres monitoring is
>>>>>> idle in trans   SELECT nextval('serial’)
>>>>>> There are a couple of things that might be wrong:
>>>>>> - our code is incompatible (our loader was created for our XML files)
>>>>>> - out Postgres is too old (we are at 9.2.24)
>>>>>> - something on InterMine’s code
>>>>>> - we need to add more memory to the process
>>>>>> Ideas?
>>>>>> Thanks
>>>>>> Paulo
>>>>>>> On Jan 9, 2019, at 7:10 AM, Paulo Nuin <[hidden email]> wrote:
>>>>>>>
>>>>>>> Ciao Sergio
>>>>>>>
>>>>>>> I have the exports on my .bashrc and I am sourcing it before every test just to be sure. I had to change a number of things due to heap problems and I am making sure I have the right options set.
>>>>>>>
>>>>>>> Our machines are on AWS and we have about 32 Gb of memory available the the maximum we setting to Java (initially for ant) which is somewhat a tradeoff between performance and keeping other resources working.
>>>>>>>
>>>>>>> We have Postgres in the same machine.
>>>>>>>
>>>>>>> I will try again and let you know.
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> Paulo
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> On Jan 9, 2019, at 3:49 AM, sergio contrino <[hidden email]> wrote:
>>>>>>>>
>>>>>>>> hi paul,
>>>>>>>> from your 2 last mails, it looks like your build is not using the GRADLE_OPTS setting you are exporting, not sure why.
>>>>>>>>
>>>>>>>> i don't know your machine setting (how much ram you have, do you use it only for the build, are you hosting postgres on it, if so what are your postgres settings regarding memory), but the last setting you sent seems reasonable.
>>>>>>>> i could suggest:
>>>>>>>>
>>>>>>>> - go to the terminal where you are going to run the build. all the below commands should be run on this terminal
>>>>>>>>
>>>>>>>> - change your gradle opts (i understand you are on java 8) running the command:
>>>>>>>>
>>>>>>>> export GRADLE_OPTS="-server -Xmx16g -XX:+UseParallelGC -Xms2g -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99 -Dorg.gradle.daemon=false”
>>>>>>>>
>>>>>>>> - check this are the setting you get
>>>>>>>> echo $GRADLE_OPTS
>>>>>>>>
>>>>>>>> - you could also run (just to have a snapshot of your memory situation)
>>>>>>>> free -g
>>>>>>>>
>>>>>>>> if all seems fine
>>>>>>>> - run the build (please insert the dump as mentioned earlier).
>>>>>>>>
>>>>>>>> - let us know the result
>>>>>>>>
>>>>>>>> and then we can assess the situation.
>>>>>>>> thanks!
>>>>>>>> sergio
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 08/01/2019 18:36, Paulo Nuin wrote:
>>>>>>>>> Hi
>>>>>>>>> This is what I have on a “regular” terminal, I export this when running the build
>>>>>>>>> export GRADLE_OPTS="-server -XX:MaxPermSize=1g -Xmx16g -XX:+UseParallelGC -Xms2g -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99”
>>>>>>>>> And my usual ANT_OPTS is
>>>>>>>>> export ANT_OPTS="-server -XX:MaxPermSize=512M -Xmx12g -XX:+UseParallelGC -Xms2g -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99 -XX:-UseGCOverheadLimit”
>>>>>>>>> Cheers
>>>>>>>>> Paulo
>>>>>>>>>> On Jan 8, 2019, at 10:40 AM, Julie Sullivan <[hidden email]> wrote:
>>>>>>>>>>
>>>>>>>>>> Less than 2 GB of memory is really low for an InterMine build, especially if the file you are parsing is 12 GB.
>>>>>>>>>>
>>>>>>>>>> If it's a build machine, I would give as much memory as you can. You will see a speed increase!
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 08/01/2019 17:36, Paulo Nuin wrote:
>>>>>>>>>>> Hi
>>>>>>>>>>> I will try, but at the moment I am just testing the build and see if different sources work.
>>>>>>>>>>> What is the recommended Postgres version?
>>>>>>>>>>> Cheers
>>>>>>>>>>> Paulo
>>>>>>>>>>>> On Jan 8, 2019, at 10:23 AM, sergio contrino <[hidden email]> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> hi paulo!
>>>>>>>>>>>> could you set
>>>>>>>>>>>>
>>>>>>>>>>>> export GRADLE_OPTS="-server -Xmx1700m -XX:+UseParallelGC -Xms1700m -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99 -Dorg.gradle.daemon=false"
>>>>>>>>>>>>
>>>>>>>>>>>> on the terminal where you will run the build? (i would also increase your Xmx if you have the possibility)
>>>>>>>>>>>>
>>>>>>>>>>>> as mentioned before, if you add the dump option just before the failing source, you can then test this part of the build without having to rebuild everything at every attempt (see https://intermine.readthedocs.io/en/latest/database/database-building/build-script/)
>>>>>>>>>>>>
>>>>>>>>>>>> please let us know how things proceed. thanks!
>>>>>>>>>>>> sergio
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 08/01/2019 16:40, Paulo Nuin wrote:
>>>>>>>>>>>>> Hi Julie
>>>>>>>>>>>>> Here they are
>>>>>>>>>>>>>> On Jan 4, 2019, at 4:51 AM, Julie Sullivan <[hidden email]> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Paulo,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> For the build that failed, can you run two commands for me? In the same console.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> echo $GRADLE_OPTS
>>>>>>>>>>>>>>
>>>>>>>>>>>>> -server -XX:MaxPermSize=256M -Xmx1700m -XX:+UseParallelGC -Xms1700m -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ./gradlew —stop
>>>>>>>>>>>>> OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256M; support was removed in 8.0
>>>>>>>>>>>>> No Gradle daemons are running.
>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>> Paulo
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Can you tell me what those two commands produce?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> (be sure to use the same console, I think we had issues with AWS and ENV vars before? maybe?)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>> Julie
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> PS. Happy New Year! :)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 21/12/2018 15:51, Paulo Nuin wrote:
>>>>>>>>>>>>>>> Hi Sergio
>>>>>>>>>>>>>>> And after 21 hours:
>>>>>>>>>>>>>>> FAILURE: Build failed with an exception.
>>>>>>>>>>>>>>> * What went wrong:
>>>>>>>>>>>>>>> Execution failed for task ':dbmodel:retrieveSingleSource'.
>>>>>>>>>>>>>>>> java.lang.OutOfMemoryError: GC overhead limit exceeded
>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>> Paulo
>>>>>>>>>>>>>>>> On Dec 21, 2018, at 6:02 AM, Paulo Nuin <[hidden email]> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Sergio
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> It is still going on, and the only major things running on the machine are Java and Postgres, This is how it looks like
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> top - 13:01:13 up 187 days, 12:37,  2 users,  load average: 7.36, 7.44, 7.53
>>>>>>>>>>>>>>>> Tasks: 240 total,   1 running, 184 sleeping,   0 stopped,   0 zombie
>>>>>>>>>>>>>>>> Cpu(s): 94.2%us,  0.2%sy,  0.0%ni,  5.6%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
>>>>>>>>>>>>>>>> Mem:  32943180k total, 32615884k used,   327296k free,   709564k buffers
>>>>>>>>>>>>>>>> Swap:        0k total,        0k used,        0k free, 24612256k cached
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>>>>>>>>>>>>>>>> 29298 nuin      20   0 6768m 1.5g  28m S 752.3  4.7   8267:28 java
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>> Paulo
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Dec 21, 2018, at 4:51 AM, sergio contrino <[hidden email]> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> hi paulo,
>>>>>>>>>>>>>>>>> it does not look ok and i would consider stopping it if still going on. i would check your memory settings and availability in your machine (i suppose you are running postgres there as well, possibly anything else?).
>>>>>>>>>>>>>>>>> if all seems fine, i could run a test here if you share your data file.
>>>>>>>>>>>>>>>>> but it is our last work day here until 2nd jan, so it could take us a while to assess this issue.
>>>>>>>>>>>>>>>>> thanks (and merry christmas!)
>>>>>>>>>>>>>>>>> sergio
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 21/12/2018 05:06, Paulo Nuin wrote:
>>>>>>>>>>>>>>>>>> Hi everyone
>>>>>>>>>>>>>>>>>> In my adventure of trying to build the DB with 2.x I started trying our large data sources (XML files from AceDB dumps) and I am noticing that the process is extremely slow, something like 400% or more. I started with our gene source, which is about 12Gb and usually takes a couple of hours or a little more to load. We are not into more than 10 hours on the build
>>>>>>>>>>>>>>>>>> <==========---> 80% EXECUTING [10h 31m 39s]
>>>>>>>>>>>>>>>>>> and I only see
>>>>>>>>>>>>>>>>>> idle in trans   SELECT nextval('serial’)
>>>>>>>>>>>>>>>>>> on my Postgres. Is that normal? How can I speed up the process, if there’s a way to speed it up. If the trend continues we will jump from a couple of days of build time, to more than one week.
>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>> Paulo
>>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>>> dev mailing list
>>>>>>>>>>>>>>>>>> [hidden email]
>>>>>>>>>>>>>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> sergio contrino                  InterMine, University of Cambridge
>>>>>>>>>>>>>>>>> https://sergiocontrino.github.io           http://www.intermine.org
>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>> dev mailing list
>>>>>>>>>>>>>>>>> [hidden email]
>>>>>>>>>>>>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>> dev mailing list
>>>>>>>>>>>>>>> [hidden email]
>>>>>>>>>>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> sergio contrino                  InterMine, University of Cambridge
>>>>>>>>>>>> https://sergiocontrino.github.io           http://www.intermine.org
>>>>>>>>
>>>>>>>> --
>>>>>>>> sergio contrino                  InterMine, University of Cambridge
>>>>>>>> https://sergiocontrino.github.io           http://www.intermine.org
>>>>>>>
>>>>>
>>>>> --
>>>>> sergio contrino                  InterMine, University of Cambridge
>>>>> https://sergiocontrino.github.io           http://www.intermine.org
>>>
>>> --
>>> sergio contrino                  InterMine, University of Cambridge
>>> https://sergiocontrino.github.io           http://www.intermine.org
>>
>

_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Really slow build process

Julie Sullivan-2
In reply to this post by sergio contrino-2
Hi Paulo,

Can you run echo $GRADLE_OPTS in that same console?

You did that for me a few emails ago and you only had 2 GB allocated to
memory, and you are now still getting an out of memory error. You can
see why we think it's a ENV var issue!

Julie

On 11/01/2019 23:01, Paulo Nuin wrote:

> Hi again
>
> Got an error with my local build, identical to AWS, now with 30Gb devoted to Java
>
> FAILURE: Build failed with an exception.
>
> * What went wrong:
> Execution failed for task ':dbmodel:retrieveSingleSource'.
>> java.lang.OutOfMemoryError: GC overhead limit exceeded
>
> * Try:
> Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
>
> * Get more help at https://help.gradle.org
>
> BUILD FAILED in 22h 3m 46s
> 11 actionable tasks: 7 executed, 4 up-to-date
>
> Cheers
> Paulo
>
>
>
>
>> On Jan 11, 2019, at 11:11 AM, Paulo Nuin <[hidden email]> wrote:
>>
>> Hi Sergio
>>
>>
>>
>>> On Jan 11, 2019, at 10:41 AM, sergio contrino <[hidden email]> wrote:
>>>
>>> thank you for the files paulo.
>>> regarding the code, it would be better if we can refer to a  github repo, so i made these 2 repos from the code you sent
>>> https://github.com/sergiocontrino/wormine
>>> https://github.com/sergiocontrino/wormmine-bio-sources
>>> feel free to point us somewhere else.
>>>
>>
>> These codes are in a repo, but I have to do some merges and checks before updating. I will let you know.
>>
>>
>>> thank you also for the further tests.
>>> is this a new source, or you used to load this mega files before moving to gradle?
>>> if new, since you are using your own xml parser, the problem is probably with your code. in this case we have a xml parser for big files (FullXmlConverter) that you could possibly consider.
>>>
>>
>> The same loader works perfectly with 1.8.5 and earlier versions. Our build time is slightly over 24 hours for our whole dataset, whilst on 2.0 is taking (and failing) to build one source in 24 hours or more.
>>
>> Cheers
>> Paulo
>>
>>
>>
>>> i am away for a week now.
>>> sergio
>>>
>>>
>>> On 11/01/2019 00:23, Paulo Nuin wrote:
>>>> Hi Sergio
>>>> Code and example files are one
>>>> https://www.dropbox.com/sh/qgkheoufhw16bfi/AADjDO87PBAXVS9pj3r7rBsQa?dl=0
>>>> Cheers
>>>> Paulo
>>>>> On Jan 10, 2019, at 9:32 AM, sergio contrino <[hidden email] <mailto:[hidden email]>> wrote:
>>>>>
>>>>> thank you paul, you could also put the code on github.
>>>>> sergio
>>>>>
>>>>> On 10/01/2019 16:24, Paulo Nuin wrote:
>>>>>> Hi
>>>>>> I will make a package with the code we have and the file I am testing and I will put on Dropbox. I will probably test a smaller XML before sending things.
>>>>>> Cheers
>>>>>> Paulo
>>>>>>> On Jan 10, 2019, at 8:57 AM, sergio contrino <[hidden email] <mailto:[hidden email]>> wrote:
>>>>>>>
>>>>>>> hi paulo,
>>>>>>> maybe we should try to test it here. could you send instructions (and file)?
>>>>>>> thanks!
>>>>>>> sergio
>>>>>>>
>>>>>>> On 10/01/2019 14:38, Paulo Nuin wrote:
>>>>>>>> Hi
>>>>>>>> Made sure (again) that GRADLE_OPTS was optima (as reported previously), or very close to optimal, and I am currently at 23 hours to load one XML file and what I can see on my Postgres monitoring is
>>>>>>>> idle in trans   SELECT nextval('serial’)
>>>>>>>> There are a couple of things that might be wrong:
>>>>>>>> - our code is incompatible (our loader was created for our XML files)
>>>>>>>> - out Postgres is too old (we are at 9.2.24)
>>>>>>>> - something on InterMine’s code
>>>>>>>> - we need to add more memory to the process
>>>>>>>> Ideas?
>>>>>>>> Thanks
>>>>>>>> Paulo
>>>>>>>>> On Jan 9, 2019, at 7:10 AM, Paulo Nuin <[hidden email] <mailto:[hidden email]>> wrote:
>>>>>>>>>
>>>>>>>>> Ciao Sergio
>>>>>>>>>
>>>>>>>>> I have the exports on my .bashrc and I am sourcing it before every test just to be sure. I had to change a number of things due to heap problems and I am making sure I have the right options set.
>>>>>>>>>
>>>>>>>>> Our machines are on AWS and we have about 32 Gb of memory available the the maximum we setting to Java (initially for ant) which is somewhat a tradeoff between performance and keeping other resources working.
>>>>>>>>>
>>>>>>>>> We have Postgres in the same machine.
>>>>>>>>>
>>>>>>>>> I will try again and let you know.
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>> Paulo
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On Jan 9, 2019, at 3:49 AM, sergio contrino <[hidden email] <mailto:[hidden email]>> wrote:
>>>>>>>>>>
>>>>>>>>>> hi paul,
>>>>>>>>>> from your 2 last mails, it looks like your build is not using the GRADLE_OPTS setting you are exporting, not sure why.
>>>>>>>>>>
>>>>>>>>>> i don't know your machine setting (how much ram you have, do you use it only for the build, are you hosting postgres on it, if so what are your postgres settings regarding memory), but the last setting you sent seems reasonable.
>>>>>>>>>> i could suggest:
>>>>>>>>>>
>>>>>>>>>> - go to the terminal where you are going to run the build. all the below commands should be run on this terminal
>>>>>>>>>>
>>>>>>>>>> - change your gradle opts (i understand you are on java 8) running the command:
>>>>>>>>>>
>>>>>>>>>> export GRADLE_OPTS="-server -Xmx16g -XX:+UseParallelGC -Xms2g -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99 -Dorg.gradle.daemon=false”
>>>>>>>>>>
>>>>>>>>>> - check this are the setting you get
>>>>>>>>>> echo $GRADLE_OPTS
>>>>>>>>>>
>>>>>>>>>> - you could also run (just to have a snapshot of your memory situation)
>>>>>>>>>> free -g
>>>>>>>>>>
>>>>>>>>>> if all seems fine
>>>>>>>>>> - run the build (please insert the dump as mentioned earlier).
>>>>>>>>>>
>>>>>>>>>> - let us know the result
>>>>>>>>>>
>>>>>>>>>> and then we can assess the situation.
>>>>>>>>>> thanks!
>>>>>>>>>> sergio
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 08/01/2019 18:36, Paulo Nuin wrote:
>>>>>>>>>>> Hi
>>>>>>>>>>> This is what I have on a “regular” terminal, I export this when running the build
>>>>>>>>>>> export GRADLE_OPTS="-server -XX:MaxPermSize=1g -Xmx16g -XX:+UseParallelGC -Xms2g -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99”
>>>>>>>>>>> And my usual ANT_OPTS is
>>>>>>>>>>> export ANT_OPTS="-server -XX:MaxPermSize=512M -Xmx12g -XX:+UseParallelGC -Xms2g -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99 -XX:-UseGCOverheadLimit”
>>>>>>>>>>> Cheers
>>>>>>>>>>> Paulo
>>>>>>>>>>>> On Jan 8, 2019, at 10:40 AM, Julie Sullivan <[hidden email] <mailto:[hidden email]>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Less than 2 GB of memory is really low for an InterMine build, especially if the file you are parsing is 12 GB.
>>>>>>>>>>>>
>>>>>>>>>>>> If it's a build machine, I would give as much memory as you can. You will see a speed increase!
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 08/01/2019 17:36, Paulo Nuin wrote:
>>>>>>>>>>>>> Hi
>>>>>>>>>>>>> I will try, but at the moment I am just testing the build and see if different sources work.
>>>>>>>>>>>>> What is the recommended Postgres version?
>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>> Paulo
>>>>>>>>>>>>>> On Jan 8, 2019, at 10:23 AM, sergio contrino <[hidden email] <mailto:[hidden email]>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> hi paulo!
>>>>>>>>>>>>>> could you set
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> export GRADLE_OPTS="-server -Xmx1700m -XX:+UseParallelGC -Xms1700m -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99 -Dorg.gradle.daemon=false"
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> on the terminal where you will run the build? (i would also increase your Xmx if you have the possibility)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> as mentioned before, if you add the dump option just before the failing source, you can then test this part of the build without having to rebuild everything at every attempt (see https://intermine.readthedocs.io/en/latest/database/database-building/build-script/)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> please let us know how things proceed. thanks!
>>>>>>>>>>>>>> sergio
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 08/01/2019 16:40, Paulo Nuin wrote:
>>>>>>>>>>>>>>> Hi Julie
>>>>>>>>>>>>>>> Here they are
>>>>>>>>>>>>>>>> On Jan 4, 2019, at 4:51 AM, Julie Sullivan <[hidden email] <mailto:[hidden email]>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Paulo,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> For the build that failed, can you run two commands for me? In the same console.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> echo $GRADLE_OPTS
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -server -XX:MaxPermSize=256M -Xmx1700m -XX:+UseParallelGC -Xms1700m -XX:SoftRefLRUPolicyMSPerMB=1 -XX:MaxHeapFreeRatio=99
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ./gradlew —stop
>>>>>>>>>>>>>>> OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256M; support was removed in 8.0
>>>>>>>>>>>>>>> No Gradle daemons are running.
>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>> Paulo
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Can you tell me what those two commands produce?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> (be sure to use the same console, I think we had issues with AWS and ENV vars before? maybe?)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>> Julie
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> PS. Happy New Year! :)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 21/12/2018 15:51, Paulo Nuin wrote:
>>>>>>>>>>>>>>>>> Hi Sergio
>>>>>>>>>>>>>>>>> And after 21 hours:
>>>>>>>>>>>>>>>>> FAILURE: Build failed with an exception.
>>>>>>>>>>>>>>>>> * What went wrong:
>>>>>>>>>>>>>>>>> Execution failed for task ':dbmodel:retrieveSingleSource'.
>>>>>>>>>>>>>>>>>> java.lang.OutOfMemoryError: GC overhead limit exceeded
>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>> Paulo
>>>>>>>>>>>>>>>>>> On Dec 21, 2018, at 6:02 AM, Paulo Nuin <[hidden email] <mailto:[hidden email]>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Sergio
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> It is still going on, and the only major things running on the machine are Java and Postgres, This is how it looks like
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> top - 13:01:13 up 187 days, 12:37,  2 users,  load average: 7.36, 7.44, 7.53
>>>>>>>>>>>>>>>>>> Tasks: 240 total,   1 running, 184 sleeping,   0 stopped,   0 zombie
>>>>>>>>>>>>>>>>>> Cpu(s): 94.2%us,  0.2%sy,  0.0%ni,  5.6%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
>>>>>>>>>>>>>>>>>> Mem:  32943180k total, 32615884k used,   327296k free,   709564k buffers
>>>>>>>>>>>>>>>>>> Swap:        0k total,        0k used,        0k free, 24612256k cached
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>>>>>>>>>>>>>>>>>> 29298 nuin      20   0 6768m 1.5g  28m S 752.3  4.7   8267:28 java
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>> Paulo
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Dec 21, 2018, at 4:51 AM, sergio contrino <[hidden email] <mailto:[hidden email]>> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> hi paulo,
>>>>>>>>>>>>>>>>>>> it does not look ok and i would consider stopping it if still going on. i would check your memory settings and availability in your machine (i suppose you are running postgres there as well, possibly anything else?).
>>>>>>>>>>>>>>>>>>> if all seems fine, i could run a test here if you share your data file.
>>>>>>>>>>>>>>>>>>> but it is our last work day here until 2nd jan, so it could take us a while to assess this issue.
>>>>>>>>>>>>>>>>>>> thanks (and merry christmas!)
>>>>>>>>>>>>>>>>>>> sergio
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On 21/12/2018 05:06, Paulo Nuin wrote:
>>>>>>>>>>>>>>>>>>>> Hi everyone
>>>>>>>>>>>>>>>>>>>> In my adventure of trying to build the DB with 2.x I started trying our large data sources (XML files from AceDB dumps) and I am noticing that the process is extremely slow, something like 400% or more. I started with our gene source, which is about 12Gb and usually takes a couple of hours or a little more to load. We are not into more than 10 hours on the build
>>>>>>>>>>>>>>>>>>>> <==========---> 80% EXECUTING [10h 31m 39s]
>>>>>>>>>>>>>>>>>>>> and I only see
>>>>>>>>>>>>>>>>>>>> idle in trans   SELECT nextval('serial’)
>>>>>>>>>>>>>>>>>>>> on my Postgres. Is that normal? How can I speed up the process, if there’s a way to speed it up. If the trend continues we will jump from a couple of days of build time, to more than one week.
>>>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>> Paulo
>>>>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>>>>> dev mailing list
>>>>>>>>>>>>>>>>>>>> [hidden email] <mailto:[hidden email]>
>>>>>>>>>>>>>>>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> sergio contrino                  InterMine, University of Cambridge
>>>>>>>>>>>>>>>>>>> https://sergiocontrino.github.io http://www.intermine.org
>>>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>>>> dev mailing list
>>>>>>>>>>>>>>>>>>> [hidden email] <mailto:[hidden email]>
>>>>>>>>>>>>>>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>> dev mailing list
>>>>>>>>>>>>>>>>> [hidden email] <mailto:[hidden email]>
>>>>>>>>>>>>>>>>> https://lists.intermine.org/mailman/listinfo/dev
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> sergio contrino                  InterMine, University of Cambridge
>>>>>>>>>>>>>> https://sergiocontrino.github.io http://www.intermine.org
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> sergio contrino                  InterMine, University of Cambridge
>>>>>>>>>> https://sergiocontrino.github.io http://www.intermine.org
>>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> sergio contrino                  InterMine, University of Cambridge
>>>>>>> https://sergiocontrino.github.io http://www.intermine.org
>>>>>
>>>>> --
>>>>> sergio contrino                  InterMine, University of Cambridge
>>>>> https://sergiocontrino.github.io http://www.intermine.org
>>>
>>> --
>>> sergio contrino                  InterMine, University of Cambridge
>>> https://sergiocontrino.github.io           http://www.intermine.org
>>
>
_______________________________________________
dev mailing list
[hidden email]
https://lists.intermine.org/mailman/listinfo/dev
12