scaling apollo

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

scaling apollo

Michael Paulini
Hia,


i was playing around with scaling out Apollo and it seems the maximum
number I can connect at the same time is in the low tens (I also wrapped
some REST calls into sleep loops, as the postgres updates got hiccups
otherwise, as other processes inserted data out of order).

Let's assume I would want to scale it to a few hundred concurrent users
on the same genome, does anyone have any recommendations, except running
multiple instances and merging the results afterwards?

Also has anyone successfully used it with more than say 50 concurrent
users editing models at the same time?

M





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: scaling apollo

nathandunn

On Aug 1, 2017, at 3:47 AM, Michael Paulini <[hidden email]> wrote:

Hia,


i was playing around with scaling out Apollo and it seems the maximum number I can connect at the same time is in the low tens (I also wrapped some REST calls into sleep loops, as the postgres updates got hiccups otherwise, as other processes inserted data out of order).

What type of hiccups are you getting?    

What REST calls are you running into trouble with?  (maybe this is something we can fix)

Can you provide some details of your system (Apollo version, database, production config settings without the password, how you’re running it (memory settings, etc.), ps -ef | grep catalina)?

If you are all hitting the same scaffold region concurrently it may cause problems, though we’d hit a couple a second without a problem.

Let's assume I would want to scale it to a few hundred concurrent users on the same genome, does anyone have any recommendations, except running multiple instances and merging the results afterwards?

A couple of things come to mind:

First, in your apollo-config.groovy, if you started with sample-postgres-apollo-config.groovy for example, there is a production setting here: 


You can mess with the settings for this.   You could also look at pgtune, as well: http://pgtune.leopard.in.ua/

Also has anyone successfully used it with more than say 50 concurrent users editing models at the same time?

I know there are several groups doing this, but data types / load, etc. may cause differences.   

Hopefully they can provide some additional details on what they’ve done.

Nathan

M




This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.






This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: scaling apollo

Michael Paulini
that is what I get when i hit the createUser REST endpoints with a 0.1s sleep in between requests:

2017-08-02 12:39:58,311 [https-openssl-nio-8443-exec-4] ERROR apollo.UserController  - org.springframework.orm.hibernate4.HibernateOptimisticLockingFailureException: Object of class [org.bbop.apollo.Role] with identifier [17]: optimistic locking failed; nested exception is org.hibernate.StaleObjectStateException: Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect) : [org.bbop.apollo.Role#17]
2017-08-02 12:39:58,354 [https-openssl-nio-8443-exec-4] ERROR errors.GrailsExceptionResolver  - StaleObjectStateException occurred when processing request: [POST] /user/createUser
Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect) : [org.bbop.apollo.User#59]. Stacktrace follows:
org.hibernate.StaleObjectStateException: Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect) : [org.bbop.apollo.User#59]
    at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:198)
    at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
    at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
    at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
    at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
    at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
    at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
    at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
    at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

I use the the docker-apollo as base, which is currently postgres 9.6.3 / apollo 7b304aac81f7dab77165f37bf210a6b3cb1b8080 / tomcat 8.5.14 / openjdk-8-jdk

REST endpoint is user/createUser with
where the variables basically are student{[hidden email] / student{i} / test / student{i} ... with i (1..50)
{
      'email'    => email,
      'firstName'=> firstName,
      'lastName' => lastName,
      'newPassword' => newPassword
}

0.5 seconds sleep make it work perfectly and no sleep causes more of the update errors.

The grand plan is to have the tracks on shared storage and run multiple instances of apollo (for each 50ish people), collect the data every X minutes/hours/days and update the track storage.

M


On 01/08/17 17:20, Nathan Dunn wrote:

On Aug 1, 2017, at 3:47 AM, Michael Paulini <[hidden email]> wrote:

Hia,


i was playing around with scaling out Apollo and it seems the maximum number I can connect at the same time is in the low tens (I also wrapped some REST calls into sleep loops, as the postgres updates got hiccups otherwise, as other processes inserted data out of order).

What type of hiccups are you getting?    

What REST calls are you running into trouble with?  (maybe this is something we can fix)

Can you provide some details of your system (Apollo version, database, production config settings without the password, how you’re running it (memory settings, etc.), ps -ef | grep catalina)?

If you are all hitting the same scaffold region concurrently it may cause problems, though we’d hit a couple a second without a problem.

Let's assume I would want to scale it to a few hundred concurrent users on the same genome, does anyone have any recommendations, except running multiple instances and merging the results afterwards?

A couple of things come to mind:

First, in your apollo-config.groovy, if you started with sample-postgres-apollo-config.groovy for example, there is a production setting here: 


You can mess with the settings for this.   You could also look at pgtune, as well: http://pgtune.leopard.in.ua/

Also has anyone successfully used it with more than say 50 concurrent users editing models at the same time?

I know there are several groups doing this, but data types / load, etc. may cause differences.   

Hopefully they can provide some additional details on what they’ve done.

Nathan

M




This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.






This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank. 






This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: scaling apollo

nathandunn

Michael,

I was able to add 100 users this way without issue (running off of master) running locally:

 ./add_users.groovy -inputfile test_user_file.csv -destinationurl http://localhost:8080/apollo/ -username [hidden email] -password demo 




This is using the example groovy script: 


Can you raise an issue here with whatever the current version you were using was:



Its entirely possible that its an issue with running docker on your current system as well, but I’m happy to check it running locally (am on a Mac). 

Nathan


On Aug 2, 2017, at 5:56 AM, Michael Paulini <[hidden email]> wrote:

that is what I get when i hit the createUser REST endpoints with a 0.1s sleep in between requests:

2017-08-02 12:39:58,311 [https-openssl-nio-8443-exec-4] ERROR apollo.UserController  - org.springframework.orm.hibernate4.HibernateOptimisticLockingFailureException: Object of class [org.bbop.apollo.Role] with identifier [17]: optimistic locking failed; nested exception is org.hibernate.StaleObjectStateException: Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect) : [org.bbop.apollo.Role#17]
2017-08-02 12:39:58,354 [https-openssl-nio-8443-exec-4] ERROR errors.GrailsExceptionResolver  - StaleObjectStateException occurred when processing request: [POST] /user/createUser
Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect) : [org.bbop.apollo.User#59]. Stacktrace follows:
org.hibernate.StaleObjectStateException: Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect) : [org.bbop.apollo.User#59]
    at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:198)
    at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
    at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
    at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
    at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
    at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
    at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
    at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
    at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

I use the the docker-apollo as base, which is currently postgres 9.6.3 / apollo 7b304aac81f7dab77165f37bf210a6b3cb1b8080 / tomcat 8.5.14 / openjdk-8-jdk

REST endpoint is user/createUser with
where the variables basically are student{[hidden email] / student{i} / test / student{i} ... with i (1..50)
{
      'email'    => email,
      'firstName'=> firstName,
      'lastName' => lastName,
      'newPassword' => newPassword
}

0.5 seconds sleep make it work perfectly and no sleep causes more of the update errors.

The grand plan is to have the tracks on shared storage and run multiple instances of apollo (for each 50ish people), collect the data every X minutes/hours/days and update the track storage.

M


On 01/08/17 17:20, Nathan Dunn wrote:

On Aug 1, 2017, at 3:47 AM, Michael Paulini <[hidden email]> wrote:

Hia,


i was playing around with scaling out Apollo and it seems the maximum number I can connect at the same time is in the low tens (I also wrapped some REST calls into sleep loops, as the postgres updates got hiccups otherwise, as other processes inserted data out of order).

What type of hiccups are you getting?    

What REST calls are you running into trouble with?  (maybe this is something we can fix)

Can you provide some details of your system (Apollo version, database, production config settings without the password, how you’re running it (memory settings, etc.), ps -ef | grep catalina)?

If you are all hitting the same scaffold region concurrently it may cause problems, though we’d hit a couple a second without a problem.

Let's assume I would want to scale it to a few hundred concurrent users on the same genome, does anyone have any recommendations, except running multiple instances and merging the results afterwards?

A couple of things come to mind:

First, in your apollo-config.groovy, if you started with sample-postgres-apollo-config.groovy for example, there is a production setting here: 


You can mess with the settings for this.   You could also look at pgtune, as well: http://pgtune.leopard.in.ua/

Also has anyone successfully used it with more than say 50 concurrent users editing models at the same time?

I know there are several groups doing this, but data types / load, etc. may cause differences.   

Hopefully they can provide some additional details on what they’ve done.

Nathan

M




This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank. 





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.






This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.


test_user_file.csv.tgz (708 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: scaling apollo

Chris Childers
From my experience testing the Apollo 2 infrastructure, I'm happy with using the web services to make the calls directly, and that includes adding ~500 users, ~60 organisms, and creating and adding people to multiple groups per organisms (thousands of transactions).  It takes some time to get through it all, but the process per transaction is fairly fast. 

Nathan, is the groovy script a wrapper for the web service calls?

Thanks,
Chris


On Wed, Aug 2, 2017 at 8:22 PM, Nathan Dunn <[hidden email]> wrote:

Michael,

I was able to add 100 users this way without issue (running off of master) running locally:

 ./add_users.groovy -inputfile test_user_file.csv -destinationurl http://localhost:8080/apollo/ -username [hidden email] -password demo 




This is using the example groovy script: 


Can you raise an issue here with whatever the current version you were using was:



Its entirely possible that its an issue with running docker on your current system as well, but I’m happy to check it running locally (am on a Mac). 

Nathan


On Aug 2, 2017, at 5:56 AM, Michael Paulini <[hidden email]> wrote:

that is what I get when i hit the createUser REST endpoints with a 0.1s sleep in between requests:

2017-08-02 12:39:58,311 [https-openssl-nio-8443-exec-4] ERROR apollo.UserController  - org.springframework.orm.hibernate4.HibernateOptimisticLockingFailureException: Object of class [org.bbop.apollo.Role] with identifier [17]: optimistic locking failed; nested exception is org.hibernate.StaleObjectStateException: Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect) : [org.bbop.apollo.Role#17]
2017-08-02 12:39:58,354 [https-openssl-nio-8443-exec-4] ERROR errors.GrailsExceptionResolver  - StaleObjectStateException occurred when processing request: [POST] /user/createUser
Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect) : [org.bbop.apollo.User#59]. Stacktrace follows:
org.hibernate.StaleObjectStateException: Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect) : [org.bbop.apollo.User#59]
    at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:198)
    at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
    at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
    at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
    at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
    at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
    at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
    at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
    at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

I use the the docker-apollo as base, which is currently postgres 9.6.3 / apollo 7b304aac81f7dab77165f37bf210a6b3cb1b8080 / tomcat 8.5.14 / openjdk-8-jdk

REST endpoint is user/createUser with
where the variables basically are student{[hidden email] / student{i} / test / student{i} ... with i (1..50)
{
      'email'    => email,
      'firstName'=> firstName,
      'lastName' => lastName,
      'newPassword' => newPassword
}

0.5 seconds sleep make it work perfectly and no sleep causes more of the update errors.

The grand plan is to have the tracks on shared storage and run multiple instances of apollo (for each 50ish people), collect the data every X minutes/hours/days and update the track storage.

M


On 01/08/17 17:20, Nathan Dunn wrote:

On Aug 1, 2017, at 3:47 AM, Michael Paulini <[hidden email]> wrote:

Hia,


i was playing around with scaling out Apollo and it seems the maximum number I can connect at the same time is in the low tens (I also wrapped some REST calls into sleep loops, as the postgres updates got hiccups otherwise, as other processes inserted data out of order).

What type of hiccups are you getting?    

What REST calls are you running into trouble with?  (maybe this is something we can fix)

Can you provide some details of your system (Apollo version, database, production config settings without the password, how you’re running it (memory settings, etc.), ps -ef | grep catalina)?

If you are all hitting the same scaffold region concurrently it may cause problems, though we’d hit a couple a second without a problem.

Let's assume I would want to scale it to a few hundred concurrent users on the same genome, does anyone have any recommendations, except running multiple instances and merging the results afterwards?

A couple of things come to mind:

First, in your apollo-config.groovy, if you started with sample-postgres-apollo-config.groovy for example, there is a production setting here: 


You can mess with the settings for this.   You could also look at pgtune, as well: http://pgtune.leopard.in.ua/

Also has anyone successfully used it with more than say 50 concurrent users editing models at the same time?

I know there are several groups doing this, but data types / load, etc. may cause differences.   

Hopefully they can provide some additional details on what they’ve done.

Nathan

M




This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.




This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank. 





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.






This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.







This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: scaling apollo

nathandunn

Chris, 

Yes, the groovy script wrapper wraps the web services.   Sorry, should have explained that.  It reads in the CSV file and runs a loop over the createUser service. 


Michael,

Thanks for posting the issue.   We have some work to do on docker-apollo in the next 2 weeks and I’ll look at it then.   Very weird that it works for the groovy script (which executes in under a second) and not your ruby one as the scripts look almost identical in execution.

Nathan

On Aug 3, 2017, at 8:30 AM, Chris Childers <[hidden email]> wrote:

From my experience testing the Apollo 2 infrastructure, I'm happy with using the web services to make the calls directly, and that includes adding ~500 users, ~60 organisms, and creating and adding people to multiple groups per organisms (thousands of transactions).  It takes some time to get through it all, but the process per transaction is fairly fast. 

Nathan, is the groovy script a wrapper for the web service calls?

Thanks,
Chris


On Wed, Aug 2, 2017 at 8:22 PM, Nathan Dunn <[hidden email]> wrote:

Michael,

I was able to add 100 users this way without issue (running off of master) running locally:

 ./add_users.groovy -inputfile test_user_file.csv -destinationurl http://localhost:8080/apollo/ -username [hidden email] -password demo 




This is using the example groovy script: 


Can you raise an issue here with whatever the current version you were using was:



Its entirely possible that its an issue with running docker on your current system as well, but I’m happy to check it running locally (am on a Mac). 

Nathan


On Aug 2, 2017, at 5:56 AM, Michael Paulini <[hidden email]> wrote:

that is what I get when i hit the createUser REST endpoints with a 0.1s sleep in between requests:

2017-08-02 12:39:58,311 [https-openssl-nio-8443-exec-4] ERROR apollo.UserController  - org.springframework.orm.hibernate4.HibernateOptimisticLockingFailureException: Object of class [org.bbop.apollo.Role] with identifier [17]: optimistic locking failed; nested exception is org.hibernate.StaleObjectStateException: Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect) : [org.bbop.apollo.Role#17]
2017-08-02 12:39:58,354 [https-openssl-nio-8443-exec-4] ERROR errors.GrailsExceptionResolver  - StaleObjectStateException occurred when processing request: [POST] /user/createUser
Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect) : [org.bbop.apollo.User#59]. Stacktrace follows:
org.hibernate.StaleObjectStateException: Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect) : [org.bbop.apollo.User#59]
    at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:198)
    at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
    at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
    at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
    at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
    at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
    at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
    at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
    at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

I use the the docker-apollo as base, which is currently postgres 9.6.3 / apollo 7b304aac81f7dab77165f37bf210a6b3cb1b8080 / tomcat 8.5.14 / openjdk-8-jdk

REST endpoint is user/createUser with
where the variables basically are student{[hidden email] / student{i} / test / student{i} ... with i (1..50)
{
      'email'    => email,
      'firstName'=> firstName,
      'lastName' => lastName,
      'newPassword' => newPassword
}

0.5 seconds sleep make it work perfectly and no sleep causes more of the update errors.

The grand plan is to have the tracks on shared storage and run multiple instances of apollo (for each 50ish people), collect the data every X minutes/hours/days and update the track storage.

M


On 01/08/17 17:20, Nathan Dunn wrote:

On Aug 1, 2017, at 3:47 AM, Michael Paulini <[hidden email]> wrote:

Hia,


i was playing around with scaling out Apollo and it seems the maximum number I can connect at the same time is in the low tens (I also wrapped some REST calls into sleep loops, as the postgres updates got hiccups otherwise, as other processes inserted data out of order).

What type of hiccups are you getting?    

What REST calls are you running into trouble with?  (maybe this is something we can fix)

Can you provide some details of your system (Apollo version, database, production config settings without the password, how you’re running it (memory settings, etc.), ps -ef | grep catalina)?

If you are all hitting the same scaffold region concurrently it may cause problems, though we’d hit a couple a second without a problem.

Let's assume I would want to scale it to a few hundred concurrent users on the same genome, does anyone have any recommendations, except running multiple instances and merging the results afterwards?

A couple of things come to mind:

First, in your apollo-config.groovy, if you started with sample-postgres-apollo-config.groovy for example, there is a production setting here: 


You can mess with the settings for this.   You could also look at pgtune, as well: http://pgtune.leopard.in.ua/

Also has anyone successfully used it with more than say 50 concurrent users editing models at the same time?

I know there are several groups doing this, but data types / load, etc. may cause differences.   

Hopefully they can provide some additional details on what they’ve done.

Nathan

M




This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.




This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank. 





This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.






This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.






This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.






This list is for the Apollo Annotation Editing Tool. Info at http://genomearchitect.org/
If you wish to unsubscribe from the Apollo List: 1. From the address with which you subscribed to the list, send a message to [hidden email] | 2. In the subject line of your email type: unsubscribe apollo | 3. Leave the message body blank.

Loading...