Prevent fastcgi from pre-caching all datasources (patch)

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Prevent fastcgi from pre-caching all datasources (patch)

Keiran Raine
Hi,

We have many data sources and some are quite large so we've found that the pre-caching of all data sources can actually take our server down, this is exacerbated if you use lots of http access BAMs.

For this reason I've made a few modifications that I think are possibly worth of including into the main distribution so that this behaviour can be controlled from the GBrowse.conf file.  The patches are at the bottom of the email.

… I've also been looking at the annoying issue with the mouseover stuff not honouring settings for 'balloon hover = 0' when under fastcgi, I've found how to force it to behave as under straight cgi (force $inline = 1 in RenderPanels->make_map) but I'm looking for a more correct solution at present.

Regards,

Keiran Raine
Principal Bioinformatician
Cancer Genome Project
Wellcome Trust Sanger institute

Tel:+44 (0)1223 834244 Ext: 7703
Office: H104


--- GBrowse-2.55/blib/cgi-bin/gbrowse 2013-08-19 09:16:44.095779174 +0100
+++ GBrowse/usr/lib/cgi-bin/gb2/gbrowse 2013-08-21 22:29:35.305723961 +0100
@@ -25,10 +25,15 @@
     my %sys_env = %ENV;
     my $globals = Bio::Graphics::Browser2->open_globals;
     warn "[$$] FastCGI initializing and loading default data sources...\n";
+   if($globals->preload_datasources) {
     foreach ($globals->data_sources) {
  warn "[$$] Loading $_\n";
  my $source = $globals->create_data_source($_);
  $source->open_database();  # cache default database in memory
+      }
+    }
+    else {
+      warn "[$$] Preload disabled";
     }
 
     while (!$FCGI_DONE) {
=====

--- GBrowse-2.55/lib/Bio/Graphics/Browser2.pm 2013-07-10 15:50:05.000000000 +0100
+++ /gbrowse/GBrowse-2.55/lib/Bio/Graphics/Browser2.pm 2013-08-21 22:25:16.444314277 +0100
@@ -243,6 +243,7 @@
 sub templates_path { shift->config_path('templates_path')  }
 sub moby_path      { shift->config_path('moby_path')       }
 
+sub preload_datasources    { shift->setting(general=>'preload data sources') || 0 }
 sub global_timeout         { shift->setting(general=>'global_timeout')      ||  60   }
 sub remember_settings_time { shift->setting(general=>'expire session')      || '1M'  }
 sub cache_time             { shift->setting(general=>'expire cache')        || '2h'  }
=====

--- GBrowse-2.55/conf/GBrowse.conf 2013-07-10 15:50:05.000000000 +0100
+++ /gbrowse/GBrowse/GBrowse.conf 2013-08-21 22:26:57.267305875 +0100
@@ -77,6 +78,7 @@
 global_timeout         = 60
 search_timeout         = 15
 max_render_processes   = 4   # try double number of CPU/cores
+preload data sources   = 0 # when true & f[ast]cgi loads all datasources at startup, bad if many or have many tracks
 
 # Renderfarm settings
 #include renderfarm.conf

-- The Wellcome Trust Sanger Institute is operated by Genome Rese arch Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

------------------------------------------------------------------------------
Introducing Performance Central, a new site from SourceForge and
AppDynamics. Performance Central is your source for news, insights,
analysis and resources for efficient Application Performance Management.
Visit us today!
http://pubads.g.doubleclick.net/gampad/clk?id=48897511&iu=/4140/ostg.clktrk
_______________________________________________
Gmod-gbrowse mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
Reply | Threaded
Open this post in threaded view
|

Re: Prevent fastcgi from pre-caching all datasources (patch)

Lincoln Stein
With respect to the mouse hover issue, can you provide me with an example that illustrates the error? Are you trying to override the default mouse over behavior by setting mouseover to false?

Lincoln


On Wed, Aug 21, 2013 at 5:42 PM, Keiran Raine <[hidden email]> wrote:
Hi,

We have many data sources and some are quite large so we've found that the pre-caching of all data sources can actually take our server down, this is exacerbated if you use lots of http access BAMs.

For this reason I've made a few modifications that I think are possibly worth of including into the main distribution so that this behaviour can be controlled from the GBrowse.conf file.  The patches are at the bottom of the email.

… I've also been looking at the annoying issue with the mouseover stuff not honouring settings for 'balloon hover = 0' when under fastcgi, I've found how to force it to behave as under straight cgi (force $inline = 1 in RenderPanels->make_map) but I'm looking for a more correct solution at present.

Regards,

Keiran Raine
Principal Bioinformatician
Cancer Genome Project
Wellcome Trust Sanger institute

Tel:<a href="tel:%2B44%20%280%291223%20834244%C2%A0Ext%3A%207703" value="+441223834244" target="_blank">+44 (0)1223 834244 Ext: 7703
Office: H104


--- GBrowse-2.55/blib/cgi-bin/gbrowse 2013-08-19 09:16:44.095779174 +0100
+++ GBrowse/usr/lib/cgi-bin/gb2/gbrowse 2013-08-21 22:29:35.305723961 +0100
@@ -25,10 +25,15 @@
     my %sys_env = %ENV;
     my $globals = Bio::Graphics::Browser2->open_globals;
     warn "[$$] FastCGI initializing and loading default data sources...\n";
+   if($globals->preload_datasources) {
     foreach ($globals->data_sources) {
  warn "[$$] Loading $_\n";
  my $source = $globals->create_data_source($_);
  $source->open_database();  # cache default database in memory
+      }
+    }
+    else {
+      warn "[$$] Preload disabled";
     }
 
     while (!$FCGI_DONE) {
=====

--- GBrowse-2.55/lib/Bio/Graphics/Browser2.pm 2013-07-10 15:50:05.000000000 +0100
+++ /gbrowse/GBrowse-2.55/lib/Bio/Graphics/Browser2.pm 2013-08-21 22:25:16.444314277 +0100
@@ -243,6 +243,7 @@
 sub templates_path { shift->config_path('templates_path')  }
 sub moby_path      { shift->config_path('moby_path')       }
 
+sub preload_datasources    { shift->setting(general=>'preload data sources') || 0 }
 sub global_timeout         { shift->setting(general=>'global_timeout')      ||  60   }
 sub remember_settings_time { shift->setting(general=>'expire session')      || '1M'  }
 sub cache_time             { shift->setting(general=>'expire cache')        || '2h'  }
=====

--- GBrowse-2.55/conf/GBrowse.conf 2013-07-10 15:50:05.000000000 +0100
+++ /gbrowse/GBrowse/GBrowse.conf 2013-08-21 22:26:57.267305875 +0100
@@ -77,6 +78,7 @@
 global_timeout         = 60
 search_timeout         = 15
 max_render_processes   = 4   # try double number of CPU/cores
+preload data sources   = 0 # when true & f[ast]cgi loads all datasources at startup, bad if many or have many tracks
 
 # Renderfarm settings
 #include renderfarm.conf

-- The Wellcome Trust Sanger Institute is operated by Genome Rese arch Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

------------------------------------------------------------------------------
Introducing Performance Central, a new site from SourceForge and
AppDynamics. Performance Central is your source for news, insights,
analysis and resources for efficient Application Performance Management.
Visit us today!
http://pubads.g.doubleclick.net/gampad/clk?id=48897511&iu=/4140/ostg.clktrk
_______________________________________________
Gmod-gbrowse mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse




--
Lincoln D. Stein
Director, Informatics and Biocomputing Platform
Ontario Institute for Cancer Research
101 College St., Suite 800
Toronto, ON, Canada M5G0A3
416 673-8514
Assistant: Renata Musa <[hidden email]>

------------------------------------------------------------------------------
Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more!
Discover the easy way to master current and previous Microsoft technologies
and advance your career. Get an incredible 1,500+ hours of step-by-step
tutorial videos with LearnDevNow. Subscribe today and save!
http://pubads.g.doubleclick.net/gampad/clk?id=58040911&iu=/4140/ostg.clktrk
_______________________________________________
Gmod-gbrowse mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
Reply | Threaded
Open this post in threaded view
|

Re: Prevent fastcgi from pre-caching all datasources (patch)

Lincoln Stein
In reply to this post by Keiran Raine
Hi Keiran,

Would you be willing to submit the preload fixes to GitHub as a patch? This makes it much easier for me to track the provenance of code changes. Little one-liners are OK in emails, but I'd rather track the bigger changes.

Thanks very much for your help.

Lincoln


On Wed, Aug 21, 2013 at 5:42 PM, Keiran Raine <[hidden email]> wrote:
Hi,

We have many data sources and some are quite large so we've found that the pre-caching of all data sources can actually take our server down, this is exacerbated if you use lots of http access BAMs.

For this reason I've made a few modifications that I think are possibly worth of including into the main distribution so that this behaviour can be controlled from the GBrowse.conf file.  The patches are at the bottom of the email.

… I've also been looking at the annoying issue with the mouseover stuff not honouring settings for 'balloon hover = 0' when under fastcgi, I've found how to force it to behave as under straight cgi (force $inline = 1 in RenderPanels->make_map) but I'm looking for a more correct solution at present.

Regards,

Keiran Raine
Principal Bioinformatician
Cancer Genome Project
Wellcome Trust Sanger institute

Tel:<a href="tel:%2B44%20%280%291223%20834244%C2%A0Ext%3A%207703" value="+441223834244" target="_blank">+44 (0)1223 834244 Ext: 7703
Office: H104


--- GBrowse-2.55/blib/cgi-bin/gbrowse 2013-08-19 09:16:44.095779174 +0100
+++ GBrowse/usr/lib/cgi-bin/gb2/gbrowse 2013-08-21 22:29:35.305723961 +0100
@@ -25,10 +25,15 @@
     my %sys_env = %ENV;
     my $globals = Bio::Graphics::Browser2->open_globals;
     warn "[$$] FastCGI initializing and loading default data sources...\n";
+   if($globals->preload_datasources) {
     foreach ($globals->data_sources) {
  warn "[$$] Loading $_\n";
  my $source = $globals->create_data_source($_);
  $source->open_database();  # cache default database in memory
+      }
+    }
+    else {
+      warn "[$$] Preload disabled";
     }
 
     while (!$FCGI_DONE) {
=====

--- GBrowse-2.55/lib/Bio/Graphics/Browser2.pm 2013-07-10 15:50:05.000000000 +0100
+++ /gbrowse/GBrowse-2.55/lib/Bio/Graphics/Browser2.pm 2013-08-21 22:25:16.444314277 +0100
@@ -243,6 +243,7 @@
 sub templates_path { shift->config_path('templates_path')  }
 sub moby_path      { shift->config_path('moby_path')       }
 
+sub preload_datasources    { shift->setting(general=>'preload data sources') || 0 }
 sub global_timeout         { shift->setting(general=>'global_timeout')      ||  60   }
 sub remember_settings_time { shift->setting(general=>'expire session')      || '1M'  }
 sub cache_time             { shift->setting(general=>'expire cache')        || '2h'  }
=====

--- GBrowse-2.55/conf/GBrowse.conf 2013-07-10 15:50:05.000000000 +0100
+++ /gbrowse/GBrowse/GBrowse.conf 2013-08-21 22:26:57.267305875 +0100
@@ -77,6 +78,7 @@
 global_timeout         = 60
 search_timeout         = 15
 max_render_processes   = 4   # try double number of CPU/cores
+preload data sources   = 0 # when true & f[ast]cgi loads all datasources at startup, bad if many or have many tracks
 
 # Renderfarm settings
 #include renderfarm.conf

-- The Wellcome Trust Sanger Institute is operated by Genome Rese arch Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

------------------------------------------------------------------------------
Introducing Performance Central, a new site from SourceForge and
AppDynamics. Performance Central is your source for news, insights,
analysis and resources for efficient Application Performance Management.
Visit us today!
http://pubads.g.doubleclick.net/gampad/clk?id=48897511&iu=/4140/ostg.clktrk
_______________________________________________
Gmod-gbrowse mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse




--
Lincoln D. Stein
Director, Informatics and Biocomputing Platform
Ontario Institute for Cancer Research
101 College St., Suite 800
Toronto, ON, Canada M5G0A3
416 673-8514
Assistant: Renata Musa <[hidden email]>

------------------------------------------------------------------------------
Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more!
Discover the easy way to master current and previous Microsoft technologies
and advance your career. Get an incredible 1,500+ hours of step-by-step
tutorial videos with LearnDevNow. Subscribe today and save!
http://pubads.g.doubleclick.net/gampad/clk?id=58040911&iu=/4140/ostg.clktrk
_______________________________________________
Gmod-gbrowse mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
Reply | Threaded
Open this post in threaded view
|

Re: Prevent fastcgi from pre-caching all datasources (patch)

Keiran Raine
Hi Lincoln,

I was pointed to the repository while you were away, it's already there as pull request #32.

Regards,

Keiran Raine
Principal Bioinformatician
Cancer Genome Project
Wellcome Trust Sanger Institute

Tel:+44 (0)1223 834244 Ext: 7703
Office: H104

On 29 Aug 2013, at 20:21, Lincoln Stein <[hidden email]> wrote:

Hi Keiran,

Would you be willing to submit the preload fixes to GitHub as a patch? This makes it much easier for me to track the provenance of code changes. Little one-liners are OK in emails, but I'd rather track the bigger changes.

Thanks very much for your help.

Lincoln


On Wed, Aug 21, 2013 at 5:42 PM, Keiran Raine <[hidden email]> wrote:
Hi,

We have many data sources and some are quite large so we've found that the pre-caching of all data sources can actually take our server down, this is exacerbated if you use lots of http access BAMs.

For this reason I've made a few modifications that I think are possibly worth of including into the main distribution so that this behaviour can be controlled from the GBrowse.conf file.  The patches are at the bottom of the email.

… I've also been looking at the annoying issue with the mouseover stuff not honouring settings for 'balloon hover = 0' when under fastcgi, I've found how to force it to behave as under straight cgi (force $inline = 1 in RenderPanels->make_map) but I'm looking for a more correct solution at present.

Regards,

Keiran Raine
Principal Bioinformatician
Cancer Genome Project
Wellcome Trust Sanger institute

Tel:<a href="tel:%2B44%20%280%291223%20834244%C2%A0Ext%3A%207703" value="+441223834244" target="_blank">+44 (0)1223 834244 Ext: 7703
Office: H104


--- GBrowse-2.55/blib/cgi-bin/gbrowse 2013-08-19 09:16:44.095779174 +0100
+++ GBrowse/usr/lib/cgi-bin/gb2/gbrowse 2013-08-21 22:29:35.305723961 +0100
@@ -25,10 +25,15 @@
     my %sys_env = %ENV;
     my $globals = Bio::Graphics::Browser2->open_globals;
     warn "[$$] FastCGI initializing and loading default data sources...\n";
+   if($globals->preload_datasources) {
     foreach ($globals->data_sources) {
  warn "[$$] Loading $_\n";
  my $source = $globals->create_data_source($_);
  $source->open_database();  # cache default database in memory
+      }
+    }
+    else {
+      warn "[$$] Preload disabled";
     }
 
     while (!$FCGI_DONE) {
=====

--- GBrowse-2.55/lib/Bio/Graphics/Browser2.pm 2013-07-10 15:50:05.000000000 +0100
+++ /gbrowse/GBrowse-2.55/lib/Bio/Graphics/Browser2.pm 2013-08-21 22:25:16.444314277 +0100
@@ -243,6 +243,7 @@
 sub templates_path { shift->config_path('templates_path')  }
 sub moby_path      { shift->config_path('moby_path')       }
 
+sub preload_datasources    { shift->setting(general=>'preload data sources') || 0 }
 sub global_timeout         { shift->setting(general=>'global_timeout')      ||  60   }
 sub remember_settings_time { shift->setting(general=>'expire session')      || '1M'  }
 sub cache_time             { shift->setting(general=>'expire cache')        || '2h'  }
=====

--- GBrowse-2.55/conf/GBrowse.conf 2013-07-10 15:50:05.000000000 +0100
+++ /gbrowse/GBrowse/GBrowse.conf 2013-08-21 22:26:57.267305875 +0100
@@ -77,6 +78,7 @@
 global_timeout         = 60
 search_timeout         = 15
 max_render_processes   = 4   # try double number of CPU/cores
+preload data sources   = 0 # when true & f[ast]cgi loads all datasources at startup, bad if many or have many tracks
 
 # Renderfarm settings
 #include renderfarm.conf

-- The Wellcome Trust Sanger Institute is operated by Genome Rese arch Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

------------------------------------------------------------------------------
Introducing Performance Central, a new site from SourceForge and
AppDynamics. Performance Central is your source for news, insights,
analysis and resources for efficient Application Performance Management.
Visit us today!
http://pubads.g.doubleclick.net/gampad/clk?id=48897511&iu=/4140/ostg.clktrk
_______________________________________________
Gmod-gbrowse mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse




--
Lincoln D. Stein
Director, Informatics and Biocomputing Platform
Ontario Institute for Cancer Research
101 College St., Suite 800
Toronto, ON, Canada M5G0A3
416 673-8514
Assistant: Renata Musa <[hidden email]>


-- The Wellcome Trust Sanger Institute is operated by Genome Rese arch Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

------------------------------------------------------------------------------
Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more!
Discover the easy way to master current and previous Microsoft technologies
and advance your career. Get an incredible 1,500+ hours of step-by-step
tutorial videos with LearnDevNow. Subscribe today and save!
http://pubads.g.doubleclick.net/gampad/clk?id=58040911&iu=/4140/ostg.clktrk
_______________________________________________
Gmod-gbrowse mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse