[Gmod-ajax] big data best practices

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

[Gmod-ajax] big data best practices

Jillian Rowe
Hi all,

I'm wondering if anyone has some big data best practices. We are finding
our Jbrowse instance is a bit slow to load. We already split VCF and
bamfiles by chromosome. I suppose the next step is to gzip all the other
json data.

Also, is there any way to configure the maximum number of tracks a user
can have available at anytime? What would be really great is if this
could go in with the new heirarchical track selector. So I could say
something like all for track types of '*bam*' (or some xpath type
expression) a user may only have x open at a time.

Best,
Jillian

------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
_______________________________________________
Gmod-ajax mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-ajax
Reply | Threaded
Open this post in threaded view
|

Re: big data best practices

Keiran Raine
Hi Jillian,

Do you actually find that splitting the BAM/VCF by chromosome actually speeds things up other than the initial retrieval of the index file for the track (occurs on first request)?  The whole point of these formats is to only send the data contained in the range requested.  However if the cache use by your browser is to small (or available memory) the index file may get removed very quickly and have to be retrieved again.

One way to get an improvement for BAM is to remove duplicate reads from the files (rather than mark-them).

Regards,

Keiran Raine
Principal Bioinformatician
Cancer Genome Project
Wellcome Trust Sanger Institute

Tel:+44 (0)1223 834244 Ext: 7703
Office: H104

On 29 Dec 2013, at 07:37, Jillian Rowe <[hidden email]> wrote:

Hi all,

I'm wondering if anyone has some big data best practices. We are finding
our Jbrowse instance is a bit slow to load. We already split VCF and
bamfiles by chromosome. I suppose the next step is to gzip all the other
json data.

Also, is there any way to configure the maximum number of tracks a user
can have available at anytime? What would be really great is if this
could go in with the new heirarchical track selector. So I could say
something like all for track types of '*bam*' (or some xpath type
expression) a user may only have x open at a time.

Best,
Jillian

------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
_______________________________________________
Gmod-ajax mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-ajax


-- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
_______________________________________________
Gmod-ajax mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-ajax