Re: gmod hardware requirements

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: gmod hardware requirements

Stephen Ficklin-2
Hi Iulia,

I'm CC'ing my response to both the Tripal and Chado mailing list in case others have the same question or want to add additional input.    

The deciding factor for selection of appropriate hardware really depends on the amount of data you'll need to share and store, any analytical tools you might want to make available online and of course your budget.  And unfortunately it's hard to give you a good estimate.  But I can share some experience.  I've worked on setups that include Tripal and therefore Chado, GBrowse, JBrowse and houses transcriptomic, genomic, genetic (markers, variant data, phenotypes), and germplasm data. For one site in particular, the Tripal/Chado databases together are about 250GB in size with another 16GB for GBRowse.  The site has about 5Tb of files for download and also provides some analytical tools.    For this site we use separate web, file and database server.   This way analytical jobs that run on the web server do not impact performance of the database server, and downloading of files doesn't impact the web server.   For the database server we have 256 Gb of RAM to support joins and sorts on large record sets.   If the database can perform these operations in memory it runs much better.  But we find the major bottleneck on the database server is pulling data from the disk.  So, the faster the disks the better performance you'll have (SSD if you can afford it or 15K RPM drives).  We setup our database server to have a separate bank of disks to house the OS and all programs, and a second much larger bank of disks dedicated for database files.  This way other programs and the database don't have to compete for the spin cycles on the drives.  If you can afford to buy enough disks for a RAID 10 configuration then you should get the best performance.   The web server just needs enough RAM and processors/cores for the expected load you'll have.    Also, be sure to consider how you want to mange your backup and disaster recover strategy. 

This example is for what I would consider a large site, but depending on your expected needs you may not need all of what I described.   As Scott mentioned, if your database is small enough and your user base is small you could use a much smaller configuration.

And perhaps others might be able to give more insightful thoughts...

Hope that helps a little,
Stephen

On 1/21/2016 8:48 AM, Iulia Cimpan wrote:
Dear Stephen and Lacey,

We are Research Computing Team at University of Birmingham and we want to install GMOD and components (including Tripal).
We are aware of the documentation http://gmod.org/wiki/Computing_Requirements and also about different requirements to install different components http://gmod.org/wiki/GMOD_Components

I need your help in figuring out the hardware architecture we need for Tripal, in terms of: memory usage, cpu usage for different tasks when using different applications, serial/parallel job executions etc. when about 30-50 users are accessing this entire GMOD install. Can you guide me towards such documentation or groups who already have installed most of your components, such that I get a bit of help ? 

Thank you very much for all your help and I apologise if I'm a bit unclear in my request.

Kind regards,

Iulia Cimpan

Research Software Support Specialist

IT Services, Room 108

Phone: 0121 4144072

Email: [hidden email]


From: Scott Cain [[hidden email]]
Sent: 20 January 2016 16:44
To: Iulia Cimpan; Monica Munoz-Torres; Stephen Ficklin; Lacey Sanderson
Cc: [hidden email]
Subject: Re: gmod hardware requirements

Hi Iulia,

Sorry for the delay; I was at a conference last week so I missed you initial email.

Of course, it can be quite difficult to nail down what sort of hardware you need.  For example, I have a test installation of JBrowse for WormBase (not a lot of users, but quite a bit of data and over 800 tracks) on a AWS t2.nano machine, which has only 1 CPU and 512 MB of ram, and it is quite responsive.  Of course, running a database on that would be not terribly realistic.  The AWS machine that is running http://cloud.gmod.org/ (Again, not a lot of users, but it has GBrowse, JBrowse, Apollo, Tripal and Chado) is an m1.small (1 CPU and 3.75 GB ram) is is generally pretty responsive.  The only thing would not be inclined to do with that machine as is is to hold something like an annotation jamboree where many users are using Apollo at the same time.

For more info on individual components, I'll direct you to Moni Munoz-Torres at Apollo, who does things like workshops where many users are accessing Apollo, so she can give you an idea of what sort of hardware is needed for that, and Stephen Ficklin and Lacey Sanderson who are the lead developers of Tripal and have been involved in many installations of it over the past few years.

Good luck!
Scott


On Thu, Jan 14, 2016 at 10:57 AM, Iulia Cimpan <[hidden email]> wrote:
Dear All,

I am putting together a "GMOD project overview" for my IT team. We want to install many of the tools and databases that GMOD provides.

We are aware of the documentation http://gmod.org/wiki/Computing_Requirements and also about different requirements to install different components http://gmod.org/wiki/GMOD_Components

I need your help in figuring out the hardware architecture we need, in terms of: memory usage, cpu usage for different tasks when using different applications, serial/parallel job executions etc. when about 30-50 users are accessing this entire GMOD install. Can you guide me towards such documentation or groups who already have installed most of your components, such that I get a bit of help ? 

Thank you very much for all your help and I apologise if I'm a bit unclear in my request.

Kind regards,

Iulia Cimpan

Research Software Support Specialist

IT Services, Room 108

Phone: 0121 4144072

Email: [hidden email]




--
------------------------------------------------------------------------
Scott Cain, Ph. D.                                   scott at scottcain dot net
GMOD Coordinator (http://gmod.org/)                     216-392-3087
Ontario Institute for Cancer Research


------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
_______________________________________________
Gmod-schema mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-schema
Reply | Threaded
Open this post in threaded view
|

Re: gmod hardware requirements

Iulia Cimpan
Dear Stephen,

Thank you so much for the details ! I am very grateful for all the information you shared.

Kind regards,

Iulia Cimpan

Research Software Support Specialist

IT Services, Room 108

Phone: 0121 4144072

Email: [hidden email]


From: Stephen Ficklin [[hidden email]]
Sent: 21 January 2016 19:42
To: Iulia Cimpan; Lacey Sanderson; [hidden email]; GMOD Schema/Chado List
Subject: Re: gmod hardware requirements

Hi Iulia,

I'm CC'ing my response to both the Tripal and Chado mailing list in case others have the same question or want to add additional input.    

The deciding factor for selection of appropriate hardware really depends on the amount of data you'll need to share and store, any analytical tools you might want to make available online and of course your budget.  And unfortunately it's hard to give you a good estimate.  But I can share some experience.  I've worked on setups that include Tripal and therefore Chado, GBrowse, JBrowse and houses transcriptomic, genomic, genetic (markers, variant data, phenotypes), and germplasm data. For one site in particular, the Tripal/Chado databases together are about 250GB in size with another 16GB for GBRowse.  The site has about 5Tb of files for download and also provides some analytical tools.    For this site we use separate web, file and database server.   This way analytical jobs that run on the web server do not impact performance of the database server, and downloading of files doesn't impact the web server.   For the database server we have 256 Gb of RAM to support joins and sorts on large record sets.   If the database can perform these operations in memory it runs much better.  But we find the major bottleneck on the database server is pulling data from the disk.  So, the faster the disks the better performance you'll have (SSD if you can afford it or 15K RPM drives).  We setup our database server to have a separate bank of disks to house the OS and all programs, and a second much larger bank of disks dedicated for database files.  This way other programs and the database don't have to compete for the spin cycles on the drives.  If you can afford to buy enough disks for a RAID 10 configuration then you should get the best performance.   The web server just needs enough RAM and processors/cores for the expected load you'll have.    Also, be sure to consider how you want to mange your backup and disaster recover strategy. 

This example is for what I would consider a large site, but depending on your expected needs you may not need all of what I described.   As Scott mentioned, if your database is small enough and your user base is small you could use a much smaller configuration.

And perhaps others might be able to give more insightful thoughts...

Hope that helps a little,
Stephen

On 1/21/2016 8:48 AM, Iulia Cimpan wrote:
Dear Stephen and Lacey,

We are Research Computing Team at University of Birmingham and we want to install GMOD and components (including Tripal).
We are aware of the documentation http://gmod.org/wiki/Computing_Requirements and also about different requirements to install different components http://gmod.org/wiki/GMOD_Components

I need your help in figuring out the hardware architecture we need for Tripal, in terms of: memory usage, cpu usage for different tasks when using different applications, serial/parallel job executions etc. when about 30-50 users are accessing this entire GMOD install. Can you guide me towards such documentation or groups who already have installed most of your components, such that I get a bit of help ? 

Thank you very much for all your help and I apologise if I'm a bit unclear in my request.

Kind regards,

Iulia Cimpan

Research Software Support Specialist

IT Services, Room 108

Phone: 0121 4144072

Email: [hidden email]


From: Scott Cain [[hidden email]]
Sent: 20 January 2016 16:44
To: Iulia Cimpan; Monica Munoz-Torres; Stephen Ficklin; Lacey Sanderson
Cc: [hidden email]
Subject: Re: gmod hardware requirements

Hi Iulia,

Sorry for the delay; I was at a conference last week so I missed you initial email.

Of course, it can be quite difficult to nail down what sort of hardware you need.  For example, I have a test installation of JBrowse for WormBase (not a lot of users, but quite a bit of data and over 800 tracks) on a AWS t2.nano machine, which has only 1 CPU and 512 MB of ram, and it is quite responsive.  Of course, running a database on that would be not terribly realistic.  The AWS machine that is running http://cloud.gmod.org/ (Again, not a lot of users, but it has GBrowse, JBrowse, Apollo, Tripal and Chado) is an m1.small (1 CPU and 3.75 GB ram) is is generally pretty responsive.  The only thing would not be inclined to do with that machine as is is to hold something like an annotation jamboree where many users are using Apollo at the same time.

For more info on individual components, I'll direct you to Moni Munoz-Torres at Apollo, who does things like workshops where many users are accessing Apollo, so she can give you an idea of what sort of hardware is needed for that, and Stephen Ficklin and Lacey Sanderson who are the lead developers of Tripal and have been involved in many installations of it over the past few years.

Good luck!
Scott


On Thu, Jan 14, 2016 at 10:57 AM, Iulia Cimpan <[hidden email]> wrote:
Dear All,

I am putting together a "GMOD project overview" for my IT team. We want to install many of the tools and databases that GMOD provides.

We are aware of the documentation http://gmod.org/wiki/Computing_Requirements and also about different requirements to install different components http://gmod.org/wiki/GMOD_Components

I need your help in figuring out the hardware architecture we need, in terms of: memory usage, cpu usage for different tasks when using different applications, serial/parallel job executions etc. when about 30-50 users are accessing this entire GMOD install. Can you guide me towards such documentation or groups who already have installed most of your components, such that I get a bit of help ? 

Thank you very much for all your help and I apologise if I'm a bit unclear in my request.

Kind regards,

Iulia Cimpan

Research Software Support Specialist

IT Services, Room 108

Phone: 0121 4144072

Email: [hidden email][hidden email]




--
------------------------------------------------------------------------
Scott Cain, Ph. D.                                   scott at scottcain dot net
GMOD Coordinator (http://gmod.org/)                     216-392-3087
Ontario Institute for Cancer Research


------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
_______________________________________________
Gmod-schema mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-schema
Reply | Threaded
Open this post in threaded view
|

Re: gmod hardware requirements

Karl O. Pinc
On Fri, 22 Jan 2016 09:40:39 +0000
Iulia Cimpan <[hidden email]> wrote:

> I need your help in figuring out the hardware architecture we need
> for Tripal, in terms of: memory usage, cpu usage for different tasks
> when using different applications, serial/parallel job executions
> etc. when about 30-50 users are accessing this entire GMOD install.
> Can you guide me towards such documentation or groups who already
> have installed most of your components, such that I get a bit of
> help ?

FWIW, we ran just Chado with 2G of RAM and about 100G of disk.
Did fine, but we did really nothing much but load the reference
genome and poke around.


Karl <[hidden email]>
Free Software:  "You don't pay back, you pay forward."
                 -- Robert A. Heinlein

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
_______________________________________________
Gmod-schema mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-schema
Reply | Threaded
Open this post in threaded view
|

Re: gmod hardware requirements

Iulia Cimpan
Thank you very much Karl ! I appreciate your help.



Kind regards,

Iulia Cimpan

Research Software Support Specialist

IT Services, Room 108

Phone: 0121 4144072

Email: [hidden email]

________________________________________
From: Karl O. Pinc [[hidden email]]
Sent: 23 January 2016 17:50
To: Iulia Cimpan
Cc: Stephen Ficklin; Lacey Sanderson; [hidden email]; GMOD Schema/Chado List
Subject: Re: [Gmod-schema] gmod hardware requirements

On Fri, 22 Jan 2016 09:40:39 +0000
Iulia Cimpan <[hidden email]> wrote:

> I need your help in figuring out the hardware architecture we need
> for Tripal, in terms of: memory usage, cpu usage for different tasks
> when using different applications, serial/parallel job executions
> etc. when about 30-50 users are accessing this entire GMOD install.
> Can you guide me towards such documentation or groups who already
> have installed most of your components, such that I get a bit of
> help ?

FWIW, we ran just Chado with 2G of RAM and about 100G of disk.
Did fine, but we did really nothing much but load the reference
genome and poke around.


Karl <[hidden email]>
Free Software:  "You don't pay back, you pay forward."
                 -- Robert A. Heinlein

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
_______________________________________________
Gmod-schema mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-schema