How to install biomart in an existing apache environment?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

How to install biomart in an existing apache environment?

Laure
This post was updated on .
CONTENTS DELETED
The author has deleted this message.
Reply | Threaded
Open this post in threaded view
|

Re: How to install biomart in an existing apache environment?

Bob MacCallum
I would run it in a separate back-end apache and use mod_proxy and
ProxyPass directives in your existing front-end apache (some
recompilation may be necessary - configure options "--enable-proxy"
"--enable-proxy-ajp"  - the latter is for a tomcat backend)


On Tue, Nov 30, 2010 at 8:59 AM, Laure QUINTRIC
<[hidden email]> wrote:

> Hello,
>
> I've succeeded installing biomart as a standalone application on an
> apache server, but in order to deploy biomart in our architecture (using
> an existing apache web server), I'd like to know if you have some
> documentation telling how to do it ?
> I mean I have an apache server running other applications than biomart
> and I'd like biomart to be included in this server.
>
> What configuration should I make for httpd.conf?
> Where should I put biomart htdocs and cgi-bin ?
>
> Thanks.
>
> Laure
>
Reply | Threaded
Open this post in threaded view
|

Re: How to install biomart in an existing apache environment?

Dr James A Smith
On 30/11/2010 11:15, Bob MacCallum wrote:
> I would run it in a separate back-end apache and use mod_proxy and
> ProxyPass directives in your existing front-end apache (some
> recompilation may be necessary - configure options "--enable-proxy"
> "--enable-proxy-ajp"  - the latter is for a tomcat backend)
>
I agree with Bob here, unless your biomart is very small I would say
that you can do it - but DON'T you will find that the performance of
your webserver will drop considerably! Also Mart can be very easily
DoSed by legitimate users of the web interface (or more increasingly the
services interface)

This is the experince the Ensembl team had. They actually run mart under
its own dedicated webservers (and they struggle with the huge demands
for memory of a large biomart) - but the server serving mart ONLY serves
mart dynamic content (it doesn't even serve mart static content) which
is handled by the main ensembl servers. The team had to limit the number
of children on the BioMart instances to no more than 15-20 children due
to extreme memory requirements of the BioMart system for a large mart,
otherwise the instances would run out of memory (and these are 16/32 G
machines) - they still do if users generate weird queries - the servers
both backend databases servers and frontend MySQL servers are regularly
DOSed by legitimate users using the web interface. We have also recently
noticed other DOS attacks on martservice URLs where in efficiently
written queries are generating large numbers of small queries for whom
the backend MySQL backs up - and so delays responses to the martservice
request - e.g. requests which look for genes overlapping a region from
within a given list - each return on average 10 bytes of data - but each
request actually takes somewhere between 15 seconds and 2 minutes to
return because of constraints on the MySQL tables....

James
Reply | Threaded
Open this post in threaded view
|

Re: How to install biomart in an existing apache environment?

Leandro Hermida-2
I definitely also agree with the previous two responses, I do the same and run BioMart as documented using it's own back-end specificly compiled apache+mod_perl with a separate already existing front-end apache doing  reverse proxy.

-L

On Tue, Nov 30, 2010 at 12:37 PM, Dr James A Smith <[hidden email]> wrote:
On 30/11/2010 11:15, Bob MacCallum wrote:
I would run it in a separate back-end apache and use mod_proxy and
ProxyPass directives in your existing front-end apache (some
recompilation may be necessary - configure options "--enable-proxy"
"--enable-proxy-ajp"  - the latter is for a tomcat backend)

I agree with Bob here, unless your biomart is very small I would say that you can do it - but DON'T you will find that the performance of your webserver will drop considerably! Also Mart can be very easily DoSed by legitimate users of the web interface (or more increasingly the services interface)

This is the experince the Ensembl team had. They actually run mart under its own dedicated webservers (and they struggle with the huge demands for memory of a large biomart) - but the server serving mart ONLY serves mart dynamic content (it doesn't even serve mart static content) which is handled by the main ensembl servers. The team had to limit the number of children on the BioMart instances to no more than 15-20 children due to extreme memory requirements of the BioMart system for a large mart, otherwise the instances would run out of memory (and these are 16/32 G machines) - they still do if users generate weird queries - the servers both backend databases servers and frontend MySQL servers are regularly DOSed by legitimate users using the web interface. We have also recently noticed other DOS attacks on martservice URLs where in efficiently written queries are generating large numbers of small queries for whom the backend MySQL backs up - and so delays responses to the martservice request - e.g. requests which look for genes overlapping a region from within a given list - each return on average 10 bytes of data - but each request actually takes somewhere between 15 seconds and 2 minutes to return because of constraints on the MySQL tables....

James

Reply | Threaded
Open this post in threaded view
|

Re: How to install biomart in an existing apache environment?

Richard Holland
I also agree with this suggestion - at least as far as BioMart 0.7 is concerned. I understand that big changes are planned to the internals for 0.8 which may help avoid some of the vulnerabilities that currently make it advisable to run BioMart on a separate server, however I don't have any details. I'm looking forward to investigating it when it is released!

cheers,
Richard

On 30 Nov 2010, at 22:19, Leandro Hermida wrote:

> I definitely also agree with the previous two responses, I do the same and run BioMart as documented using it's own back-end specificly compiled apache+mod_perl with a separate already existing front-end apache doing  reverse proxy.
>
> -L
>
> On Tue, Nov 30, 2010 at 12:37 PM, Dr James A Smith <[hidden email]> wrote:
> On 30/11/2010 11:15, Bob MacCallum wrote:
> I would run it in a separate back-end apache and use mod_proxy and
> ProxyPass directives in your existing front-end apache (some
> recompilation may be necessary - configure options "--enable-proxy"
> "--enable-proxy-ajp"  - the latter is for a tomcat backend)
>
> I agree with Bob here, unless your biomart is very small I would say that you can do it - but DON'T you will find that the performance of your webserver will drop considerably! Also Mart can be very easily DoSed by legitimate users of the web interface (or more increasingly the services interface)
>
> This is the experince the Ensembl team had. They actually run mart under its own dedicated webservers (and they struggle with the huge demands for memory of a large biomart) - but the server serving mart ONLY serves mart dynamic content (it doesn't even serve mart static content) which is handled by the main ensembl servers. The team had to limit the number of children on the BioMart instances to no more than 15-20 children due to extreme memory requirements of the BioMart system for a large mart, otherwise the instances would run out of memory (and these are 16/32 G machines) - they still do if users generate weird queries - the servers both backend databases servers and frontend MySQL servers are regularly DOSed by legitimate users using the web interface. We have also recently noticed other DOS attacks on martservice URLs where in efficiently written queries are generating large numbers of small queries for whom the backend MySQL backs up - and so delays responses to th!
 e martservice request - e.g. requests which look for genes overlapping a region from within a given list - each return on average 10 bytes of data - but each request actually takes somewhere between 15 seconds and 2 minutes to return because of constraints on the MySQL tables....
>
> James
>

--
Richard Holland, BSc MBCS
Operations and Delivery Director, Eagle Genomics Ltd
T: +44 (0)1223 654481 ext 3 | E: [hidden email]
http://www.eaglegenomics.com/