[Gmod-ajax] Can trackList.json be gzip'ed?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[Gmod-ajax] Can trackList.json be gzip'ed?

Keiran Raine
Hi,

Many of the inputs for the browser can be compressed for the actual tracks, can we do this for trackList.json too and save it as gzip compressed trackList.jsonz?

Thanks,

Keiran Raine
Principal Bioinformatician
Cancer Genome Project
Wellcome Trust Sanger Institute

Tel:+44 (0)1223 834244 Ext: 7703
Office: H104


-- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

------------------------------------------------------------------------------
Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
Francisco, CA to explore cutting-edge tech and listen to tech luminaries
present their vision of the future. This family event has something for
everyone, including kids. Get more information and register today.
http://sdm.link/attshape
_______________________________________________
Gmod-ajax mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-ajax
Reply | Threaded
Open this post in threaded view
|

Re: Can trackList.json be gzip'ed?

Eric Rasche-3
Apache/nginx should be capable of automatically compressing files before sending them in a way that's completely transparent to the browser/developer/etc.

E.g. https://www.vultr.com/docs/gzip-compression-on-apache-and-nginx

Regards,
Eric

On 08. juli 2016 15:46, Keiran Raine wrote:
Hi,

Many of the inputs for the browser can be compressed for the actual tracks, can we do this for trackList.json too and save it as gzip compressed trackList.jsonz?

Thanks,

Keiran Raine
Principal Bioinformatician
Cancer Genome Project
Wellcome Trust Sanger Institute

Tel:+44 (0)1223 834244 Ext: 7703
Office: H104


-- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.


------------------------------------------------------------------------------
Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
Francisco, CA to explore cutting-edge tech and listen to tech luminaries
present their vision of the future. This family event has something for
everyone, including kids. Get more information and register today.
http://sdm.link/attshape


_______________________________________________
Gmod-ajax mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-ajax

--
Eric Rasche
Programmer II

Center for Phage Technology
Rm 312A, Biochemistry & Biophysics
Texas A&M University
College Station, TX 77843
[hidden email]

------------------------------------------------------------------------------
Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
Francisco, CA to explore cutting-edge tech and listen to tech luminaries
present their vision of the future. This family event has something for
everyone, including kids. Get more information and register today.
http://sdm.link/attshape
_______________________________________________
Gmod-ajax mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-ajax

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Can trackList.json be gzip'ed?

Keiran Raine
Hi Eric,

I have ~600 of these so far, currently they top out at ~2MB, generally in the 2-100K range but we're looking to add more tracks and I expect them all to be 2-3x the size.

What I wanted to know was if JBrowse could receive them gzip so that I can store them compressed (and not have server overhead of compressing them everytime a new user requests it).

Regarsd,

Keiran Raine
Principal Bioinformatician
Cancer Genome Project
Wellcome Trust Sanger Institute

Tel:+44 (0)1223 834244 Ext: 7703
Office: H104

On 8 Jul 2016, at 16:52, Eric Rasche <[hidden email]> wrote:

Apache/nginx should be capable of automatically compressing files before sending them in a way that's completely transparent to the browser/developer/etc.

E.g. https://www.vultr.com/docs/gzip-compression-on-apache-and-nginx

Regards,
Eric

On 08. juli 2016 15:46, Keiran Raine wrote:
Hi,

Many of the inputs for the browser can be compressed for the actual tracks, can we do this for trackList.json too and save it as gzip compressed trackList.jsonz?

Thanks,

Keiran Raine
Principal Bioinformatician
Cancer Genome Project
Wellcome Trust Sanger Institute

Tel:+44 (0)1223 834244 Ext: 7703
Office: H104


-- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.


------------------------------------------------------------------------------
Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
Francisco, CA to explore cutting-edge tech and listen to tech luminaries
present their vision of the future. This family event has something for
everyone, including kids. Get more information and register today.
http://sdm.link/attshape


_______________________________________________
Gmod-ajax mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-ajax

--
Eric Rasche
Programmer II

Center for Phage Technology
Rm 312A, Biochemistry & Biophysics
Texas A&M University
College Station, TX 77843
[hidden email]


-- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

------------------------------------------------------------------------------
Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
Francisco, CA to explore cutting-edge tech and listen to tech luminaries
present their vision of the future. This family event has something for
everyone, including kids. Get more information and register today.
http://sdm.link/attshape
_______________________________________________
Gmod-ajax mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-ajax
Reply | Threaded
Open this post in threaded view
|

Re: Can trackList.json be gzip'ed?

Eric Rasche-3
Hi Keiran,

On 08. juli 2016 16:28, Keiran Raine wrote:
Hi Eric,

I have ~600 of these so far, currently they top out at ~2MB, generally in the 2-100K range but we're looking to add more tracks and I expect them all to be 2-3x the size.

What I wanted to know was if JBrowse could receive them gzip so that I can store them compressed (and not have server overhead of compressing them everytime a new user requests it).
I see.

I looked into this briefly, I was somewhat surprised at the results, I didn't realise how much overhead was added by gzip.
http://www.webperformance.com/library/reports/moddeflate/

It appears there are some tunable settings in Apache which will let you solve this
http://stackoverflow.com/questions/75482/how-can-i-pre-compress-files-with-mod-deflate-in-apache-2-x

And same for nginx. (I imagine the config will be identical, just replacing zopfli with gzip)
http://www.cambus.net/serving-precompressed-content-with-nginx-and-zopfli/

Would that solve your problem? (Anything to avoid in-JS decompression...)

Regarsd,

Keiran Raine
Principal Bioinformatician
Cancer Genome Project
Wellcome Trust Sanger Institute

Tel:+44 (0)1223 834244 Ext: 7703
Office: H104

On 8 Jul 2016, at 16:52, Eric Rasche <[hidden email]> wrote:

Apache/nginx should be capable of automatically compressing files before sending them in a way that's completely transparent to the browser/developer/etc.

E.g. https://www.vultr.com/docs/gzip-compression-on-apache-and-nginx

Regards,
Eric

On 08. juli 2016 15:46, Keiran Raine wrote:
Hi,

Many of the inputs for the browser can be compressed for the actual tracks, can we do this for trackList.json too and save it as gzip compressed trackList.jsonz?

Thanks,

Keiran Raine
Principal Bioinformatician
Cancer Genome Project
Wellcome Trust Sanger Institute

Tel:+44 (0)1223 834244 Ext: 7703
Office: H104


-- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.


------------------------------------------------------------------------------
Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
Francisco, CA to explore cutting-edge tech and listen to tech luminaries
present their vision of the future. This family event has something for
everyone, including kids. Get more information and register today.
http://sdm.link/attshape


_______________________________________________
Gmod-ajax mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-ajax

--
Eric Rasche
Programmer II

Center for Phage Technology
Rm 312A, Biochemistry & Biophysics
Texas A&M University
College Station, TX 77843
[hidden email]


-- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

--
Eric Rasche
Programmer II

Center for Phage Technology
Rm 312A, Biochemistry & Biophysics
Texas A&M University
College Station, TX 77843
[hidden email]

------------------------------------------------------------------------------
Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
Francisco, CA to explore cutting-edge tech and listen to tech luminaries
present their vision of the future. This family event has something for
everyone, including kids. Get more information and register today.
http://sdm.link/attshape
_______________________________________________
Gmod-ajax mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-ajax

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Can trackList.json be gzip'ed?

Keiran Raine
Hi Eric,

I'm not sure what the problem is with JS decompression within the browser.

Each client has the ability to do this as all BAM, VCF tracks are able to be presented in variants of the glib compression and all of these are far larger and heavily used.

The trackList.json file is only downloaded once per session for each datasource.

What do others think?  I note that I've not had any confirmation that it isn't currently possible.

Regards,

Keiran Raine
Principal Bioinformatician
Cancer Genome Project
Wellcome Trust Sanger Institute

Tel:+44 (0)1223 834244 Ext: 7703
Office: H104

On 8 Jul 2016, at 18:03, Eric Rasche <[hidden email]> wrote:

Hi Keiran,

On 08. juli 2016 16:28, Keiran Raine wrote:
Hi Eric,

I have ~600 of these so far, currently they top out at ~2MB, generally in the 2-100K range but we're looking to add more tracks and I expect them all to be 2-3x the size.

What I wanted to know was if JBrowse could receive them gzip so that I can store them compressed (and not have server overhead of compressing them everytime a new user requests it).
I see.

I looked into this briefly, I was somewhat surprised at the results, I didn't realise how much overhead was added by gzip.
http://www.webperformance.com/library/reports/moddeflate/

It appears there are some tunable settings in Apache which will let you solve this
http://stackoverflow.com/questions/75482/how-can-i-pre-compress-files-with-mod-deflate-in-apache-2-x

And same for nginx. (I imagine the config will be identical, just replacing zopfli with gzip)
http://www.cambus.net/serving-precompressed-content-with-nginx-and-zopfli/

Would that solve your problem? (Anything to avoid in-JS decompression...)

Regarsd,

Keiran Raine
Principal Bioinformatician
Cancer Genome Project
Wellcome Trust Sanger Institute

Tel:+44 (0)1223 834244 Ext: 7703
Office: H104

On 8 Jul 2016, at 16:52, Eric Rasche <[hidden email][hidden email]> wrote:

Apache/nginx should be capable of automatically compressing files before sending them in a way that's completely transparent to the browser/developer/etc.

E.g. https://www.vultr.com/docs/gzip-compression-on-apache-and-nginx

Regards,
Eric

On 08. juli 2016 15:46, Keiran Raine wrote:
Hi,

Many of the inputs for the browser can be compressed for the actual tracks, can we do this for trackList.json too and save it as gzip compressed trackList.jsonz?

Thanks,

Keiran Raine
Principal Bioinformatician
Cancer Genome Project
Wellcome Trust Sanger Institute

Tel:+44 (0)1223 834244 Ext: 7703
Office: H104


-- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.


------------------------------------------------------------------------------
Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
Francisco, CA to explore cutting-edge tech and listen to tech luminaries
present their vision of the future. This family event has something for
everyone, including kids. Get more information and register today.
http://sdm.link/attshape


_______________________________________________
Gmod-ajax mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-ajax

--
Eric Rasche
Programmer II

Center for Phage Technology
Rm 312A, Biochemistry & Biophysics
Texas A&M University
College Station, TX 77843
[hidden email]


-- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

--
Eric Rasche
Programmer II

Center for Phage Technology
Rm 312A, Biochemistry & Biophysics
Texas A&M University
College Station, TX 77843
[hidden email]


-- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

------------------------------------------------------------------------------
Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
Francisco, CA to explore cutting-edge tech and listen to tech luminaries
present their vision of the future. This family event has something for
everyone, including kids. Get more information and register today.
http://sdm.link/attshape
_______________________________________________
Gmod-ajax mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-ajax