mod_perl memory

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

mod_perl memory

Scooter Willis-2

I have reviewed about everything I can review regarding the multiple apache2 processes that get spawned when running gbrowse2 using mod_perl. This happens in yeast example as well.

In my genome the memory usage is very high but the machine has 8 gb of memory and 16 cores so use it. From the screen shot you can see that the system monitor indicates lots of memory is allocated. Not sure if each of these instances is a thread and thus reporting the same memory block or if this virutal memory that hasn't been allocated. Not afraid to admit I am not a linux administrator! I have the number of processes to use for rendering set to 16 and have played around with that number where at 4 I don't get as many threads but that could be different based on how I was browsing.

It also appears that when a particular PID goes into pipe_wait it stays in that state but will get CPU increase during an action in gbrowse.  The memory for the various instances will get freed up based on different actions. Hard to tell on the sys_pause instances but when I browse the list resorts based on memory usage so they are doing something.

Performance is good but system monitor is showing memory used as 7.4 out of 7.8 GB used and swap at 3.2 GB.

Just wanted to see if this is expected behavior using gbrowse2 and mod_perl and everything is configured properly considering I am loading all the data into memory vs using database.

Thanks

Scooter
------------------------------------------------------------------------------
This SF.net email is sponsored by Sprint
What will you do first with EVO, the first 4G phone?
Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first
_______________________________________________
Gmod-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-devel

Screenshot-System Monitor.png (233K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: mod_perl memory

Lincoln Stein
Hi Scooter,

It's normal to see multiple apache2 processes when running mod_perl. For performance sake, they will load the in-memory databases when needed and hang onto that memory until the apache process is killed. This ordinarily only happens when you restart apache, but you can configure apache to time out and kill processes after a certain number of requests.

The process lists can be a little deceiving because some of the memory used by the processes are shared with each other. However, if you are seeing most of your free memory being used and a chunk of swap being used as well, I would be a little concerned. I trust that the processes are not growing with each request? This would indicate a memory leak and then I would get really worried.

If you are loading your databases into memory, then you can make a significant reduction in memory used by turning on the render farm support and putting the databases that you wish to use into /etc/gbrowse2/slave_preload.conf. This will force the databases into shared memory.

Lincoln

On Wed, Jun 30, 2010 at 4:45 PM, Scooter Willis <[hidden email]> wrote:

I have reviewed about everything I can review regarding the multiple apache2 processes that get spawned when running gbrowse2 using mod_perl. This happens in yeast example as well.

In my genome the memory usage is very high but the machine has 8 gb of memory and 16 cores so use it. From the screen shot you can see that the system monitor indicates lots of memory is allocated. Not sure if each of these instances is a thread and thus reporting the same memory block or if this virutal memory that hasn't been allocated. Not afraid to admit I am not a linux administrator! I have the number of processes to use for rendering set to 16 and have played around with that number where at 4 I don't get as many threads but that could be different based on how I was browsing.

It also appears that when a particular PID goes into pipe_wait it stays in that state but will get CPU increase during an action in gbrowse.  The memory for the various instances will get freed up based on different actions. Hard to tell on the sys_pause instances but when I browse the list resorts based on memory usage so they are doing something.

Performance is good but system monitor is showing memory used as 7.4 out of 7.8 GB used and swap at 3.2 GB.

Just wanted to see if this is expected behavior using gbrowse2 and mod_perl and everything is configured properly considering I am loading all the data into memory vs using database.

Thanks

Scooter

------------------------------------------------------------------------------
This SF.net email is sponsored by Sprint
What will you do first with EVO, the first 4G phone?
Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first
_______________________________________________
Gmod-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-devel




--
Lincoln D. Stein
Director, Informatics and Biocomputing Platform
Ontario Institute for Cancer Research
101 College St., Suite 800
Toronto, ON, Canada M5G0A3
416 673-8514
Assistant: Renata Musa <[hidden email]>

------------------------------------------------------------------------------
This SF.net email is sponsored by Sprint
What will you do first with EVO, the first 4G phone?
Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first
_______________________________________________
Gmod-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-devel
Reply | Threaded
Open this post in threaded view
|

Re: mod_perl memory

Scooter Willis-2
Lincoln

Thanks for the feedback the slave_preload.conf seems like a good option. Do I need to do anything special when adding the database to the slave_preload.conf ? Do I simply duplicate what I have in my conf. I think the render support was enabled by default.

Hard to tell on the memory usage where I have seen all processes get to 1.6GB and then doing something in the browser caused 18 of the 22 processes to drop to 200 MB each and four remained at 1.6GB. They all seem to find a state of pipe_wait(usr/sbin/apache2), sys_pause(usr/lib/cgi-bin) or hrtimer_nanosleep(usr/lib/cgi-bin) and stay in that state. Clicking around memory is getting freed and currently I have 6 tracks plus DNA view on which drives the work load. I have the number of renders set to 16 but currently have 22 apache2 processes running with 100MB+ of memory allocated.

Scooter
________________________________________
From: Lincoln Stein [[hidden email]]
Sent: Wednesday, June 30, 2010 4:54 PM
To: Scooter Willis
Cc: [hidden email]
Subject: Re: [GMOD-devel] mod_perl memory

Hi Scooter,

It's normal to see multiple apache2 processes when running mod_perl. For performance sake, they will load the in-memory databases when needed and hang onto that memory until the apache process is killed. This ordinarily only happens when you restart apache, but you can configure apache to time out and kill processes after a certain number of requests.

The process lists can be a little deceiving because some of the memory used by the processes are shared with each other. However, if you are seeing most of your free memory being used and a chunk of swap being used as well, I would be a little concerned. I trust that the processes are not growing with each request? This would indicate a memory leak and then I would get really worried.

If you are loading your databases into memory, then you can make a significant reduction in memory used by turning on the render farm support and putting the databases that you wish to use into /etc/gbrowse2/slave_preload.conf. This will force the databases into shared memory.

Lincoln

On Wed, Jun 30, 2010 at 4:45 PM, Scooter Willis <[hidden email]<mailto:[hidden email]>> wrote:

I have reviewed about everything I can review regarding the multiple apache2 processes that get spawned when running gbrowse2 using mod_perl. This happens in yeast example as well.

In my genome the memory usage is very high but the machine has 8 gb of memory and 16 cores so use it. From the screen shot you can see that the system monitor indicates lots of memory is allocated. Not sure if each of these instances is a thread and thus reporting the same memory block or if this virutal memory that hasn't been allocated. Not afraid to admit I am not a linux administrator! I have the number of processes to use for rendering set to 16 and have played around with that number where at 4 I don't get as many threads but that could be different based on how I was browsing.

It also appears that when a particular PID goes into pipe_wait it stays in that state but will get CPU increase during an action in gbrowse.  The memory for the various instances will get freed up based on different actions. Hard to tell on the sys_pause instances but when I browse the list resorts based on memory usage so they are doing something.

Performance is good but system monitor is showing memory used as 7.4 out of 7.8 GB used and swap at 3.2 GB.

Just wanted to see if this is expected behavior using gbrowse2 and mod_perl and everything is configured properly considering I am loading all the data into memory vs using database.

Thanks

Scooter
------------------------------------------------------------------------------
This SF.net email is sponsored by Sprint
What will you do first with EVO, the first 4G phone?
Visit sprint.com/first<http://sprint.com/first> -- http://p.sf.net/sfu/sprint-com-first
_______________________________________________
Gmod-devel mailing list
[hidden email]<mailto:[hidden email]>
https://lists.sourceforge.net/lists/listinfo/gmod-devel




--
Lincoln D. Stein
Director, Informatics and Biocomputing Platform
Ontario Institute for Cancer Research
101 College St., Suite 800
Toronto, ON, Canada M5G0A3
416 673-8514
Assistant: Renata Musa <[hidden email]<mailto:[hidden email]>>

------------------------------------------------------------------------------
This SF.net email is sponsored by Sprint
What will you do first with EVO, the first 4G phone?
Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first
_______________________________________________
Gmod-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-devel