is possible upload a bam file to mysql database?

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

is possible upload a bam file to mysql database?

dguzman
hi,
my problem is the bam file access velocity.
how do i fix this problem?

best regards



--
View this message in context: http://generic-model-organism-system-database.450254.n5.nabble.com/is-possible-upload-a-bam-file-to-mysql-database-tp5711410.html
Sent from the gmod-gbrowse mailing list archive at Nabble.com.

------------------------------------------------------------------------------
This SF.net email is sponsored by Windows:

Build for Windows Store.

http://p.sf.net/sfu/windows-dev2dev
_______________________________________________
Gmod-gbrowse mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
Reply | Threaded
Open this post in threaded view
|

Re: is possible upload a bam file to mysql database?

Vipin TS

hi,
my problem is the bam file access velocity.
how do i fix this problem?


Please use the Bio::DB::Sam adaptor for connecting BAM files. 

--Vipin
 
best regards



--
View this message in context: http://generic-model-organism-system-database.450254.n5.nabble.com/is-possible-upload-a-bam-file-to-mysql-database-tp5711410.html
Sent from the gmod-gbrowse mailing list archive at Nabble.com.

------------------------------------------------------------------------------
This SF.net email is sponsored by Windows:

Build for Windows Store.

http://p.sf.net/sfu/windows-dev2dev
_______________________________________________
Gmod-gbrowse mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse


------------------------------------------------------------------------------
This SF.net email is sponsored by Windows:

Build for Windows Store.

http://p.sf.net/sfu/windows-dev2dev
_______________________________________________
Gmod-gbrowse mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
Reply | Threaded
Open this post in threaded view
|

Re: is possible upload a bam file to mysql database?

dguzman
Thanks for your answer Vipi,

yes, i use Bio::DB::Sam adaptor

but the DBI::mysql adaptor is supported for this file type??



--
View this message in context: http://generic-model-organism-system-database.450254.n5.nabble.com/is-possible-upload-a-bam-file-to-mysql-database-tp5711410p5711414.html
Sent from the gmod-gbrowse mailing list archive at Nabble.com.

------------------------------------------------------------------------------
This SF.net email is sponsored by Windows:

Build for Windows Store.

http://p.sf.net/sfu/windows-dev2dev
_______________________________________________
Gmod-gbrowse mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
Reply | Threaded
Open this post in threaded view
|

Re: is possible upload a bam file to mysql database?

Scott Cain
Hi,

No, there isn't a way to load a sam file into mysql, and I can't imagine you'd want to.  If you feel that the performance is too slow for a bam file, I am quite sure converting a sam file into GFF (which probably could be done) would be worse.  Depending on what you need from your sam file, you might want to consider converting it to a bigwig file, which probably would have better performance (though I don't know off hand how to do that either; perhaps Timothy Parnell can jump in with suggestions).

Scott

PS: Timothy, I don't know if we ever attend the same meetings, but if we're ever in the same state at the same time, I owe you multiple drinks and/or a good dinner :-)



On Tue, Jun 11, 2013 at 10:04 AM, dguzman <[hidden email]> wrote:
Thanks for your answer Vipi,

yes, i use Bio::DB::Sam adaptor

but the DBI::mysql adaptor is supported for this file type??



--
View this message in context: http://generic-model-organism-system-database.450254.n5.nabble.com/is-possible-upload-a-bam-file-to-mysql-database-tp5711410p5711414.html
Sent from the gmod-gbrowse mailing list archive at Nabble.com.

------------------------------------------------------------------------------
This SF.net email is sponsored by Windows:

Build for Windows Store.

http://p.sf.net/sfu/windows-dev2dev
_______________________________________________
Gmod-gbrowse mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse



--
------------------------------------------------------------------------
Scott Cain, Ph. D.                                   scott at scottcain dot net
GMOD Coordinator (http://gmod.org/)                     216-392-3087
Ontario Institute for Cancer Research

------------------------------------------------------------------------------
This SF.net email is sponsored by Windows:

Build for Windows Store.

http://p.sf.net/sfu/windows-dev2dev
_______________________________________________
Gmod-gbrowse mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
Reply | Threaded
Open this post in threaded view
|

Re: is possible upload a bam file to mysql database?

dguzman
Thanks Scott!!!


best regards



--
View this message in context: http://generic-model-organism-system-database.450254.n5.nabble.com/is-possible-upload-a-bam-file-to-mysql-database-tp5711410p5711416.html
Sent from the gmod-gbrowse mailing list archive at Nabble.com.

------------------------------------------------------------------------------
This SF.net email is sponsored by Windows:

Build for Windows Store.

http://p.sf.net/sfu/windows-dev2dev
_______________________________________________
Gmod-gbrowse mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
Reply | Threaded
Open this post in threaded view
|

Re: is possible upload a bam file to mysql database?

Timothy Parnell
In reply to this post by Scott Cain
Thanks for the complement, Scott! :-)

I agree with Scott, the performance would probably be at least a few fold slower. But you are welcome to do the actual test. There are tools out there to convert Sam alignments to Bed or GFF (BedTools and my own BioToolBox come to mind).

There are couple of things to do to speed things up.

Like Scott said, you can generate a wig (and sequentially bigWig) file of coverage using a variety of methods. Since wig files are just displaying scores, they are considerably faster. Otherwise, the alignments in the Bam file must be iterated and parsed each time to calculate coverage, which takes time. The Bio::DB::Sam module includes a basic utility (bam2bedgraph) to calculate coverage. BedTools also provides a fast coverage option with some options. I have a multi-threaded bam2wig.pl which will calculate coverage any which way under the sun at my BioToolBox project (http://biotoolbox.googlecode.com).

Two, make sure you use semantic zooming properly to switch between coverage and alignments. Alignments are only really useful under 1 or 2 kb, so switch to coverage via bigWig above that zoom level.

Three, place your bam file on the web server and accessed through a local drive. Network (HTTP) access is very convenient, but also slower. You can upload the Bam files as a custom track rather than providing a URL, provided that the bam file is smaller than the upload limit (ask the admin to increase it if necessary). GBrowse should properly generate a bigWig coverage file (if support is installed) and semantic zooming tracks automatically generated.

Beyond that, there's currently not much else to do to make it faster.

Tim

On Jun 11, 2013, at 8:44 AM, Scott Cain <[hidden email]<mailto:[hidden email]>> wrote:

Hi,

No, there isn't a way to load a sam file into mysql, and I can't imagine you'd want to.  If you feel that the performance is too slow for a bam file, I am quite sure converting a sam file into GFF (which probably could be done) would be worse.  Depending on what you need from your sam file, you might want to consider converting it to a bigwig file, which probably would have better performance (though I don't know off hand how to do that either; perhaps Timothy Parnell can jump in with suggestions).

Scott

PS: Timothy, I don't know if we ever attend the same meetings, but if we're ever in the same state at the same time, I owe you multiple drinks and/or a good dinner :-)



On Tue, Jun 11, 2013 at 10:04 AM, dguzman <[hidden email]<mailto:[hidden email]>> wrote:
Thanks for your answer Vipi,

yes, i use Bio::DB::Sam adaptor

but the DBI::mysql adaptor is supported for this file type??



--
View this message in context: http://generic-model-organism-system-database.450254.n5.nabble.com/is-possible-upload-a-bam-file-to-mysql-database-tp5711410p5711414.html
Sent from the gmod-gbrowse mailing list archive at Nabble.com<http://Nabble.com>.

------------------------------------------------------------------------------
This SF.net<http://SF.net> email is sponsored by Windows:

Build for Windows Store.

http://p.sf.net/sfu/windows-dev2dev
_______________________________________________
Gmod-gbrowse mailing list
[hidden email]<mailto:[hidden email]>
https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse



--
------------------------------------------------------------------------
Scott Cain, Ph. D.                                   scott at scottcain dot net
GMOD Coordinator (http://gmod.org/)                     216-392-3087
Ontario Institute for Cancer Research
------------------------------------------------------------------------------
This SF.net<http://SF.net> email is sponsored by Windows:

Build for Windows Store.

http://p.sf.net/sfu/windows-dev2dev_______________________________________________
Gmod-gbrowse mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse


------------------------------------------------------------------------------
This SF.net email is sponsored by Windows:

Build for Windows Store.

http://p.sf.net/sfu/windows-dev2dev
_______________________________________________
Gmod-gbrowse mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
Reply | Threaded
Open this post in threaded view
|

Re: is possible upload a bam file to mysql database?

Jim Hu
I haven't done benchmarks, but subjectively it seems like sometimes bw is slower than bam on E coli sized genomes.

Jim

Sent from my iPad

On Jun 11, 2013, at 10:24 AM, Timothy Parnell <[hidden email]> wrote:

> Thanks for the complement, Scott! :-)
>
> I agree with Scott, the performance would probably be at least a few fold slower. But you are welcome to do the actual test. There are tools out there to convert Sam alignments to Bed or GFF (BedTools and my own BioToolBox come to mind).
>
> There are couple of things to do to speed things up.
>
> Like Scott said, you can generate a wig (and sequentially bigWig) file of coverage using a variety of methods. Since wig files are just displaying scores, they are considerably faster. Otherwise, the alignments in the Bam file must be iterated and parsed each time to calculate coverage, which takes time. The Bio::DB::Sam module includes a basic utility (bam2bedgraph) to calculate coverage. BedTools also provides a fast coverage option with some options. I have a multi-threaded bam2wig.pl which will calculate coverage any which way under the sun at my BioToolBox project (http://biotoolbox.googlecode.com).
>
> Two, make sure you use semantic zooming properly to switch between coverage and alignments. Alignments are only really useful under 1 or 2 kb, so switch to coverage via bigWig above that zoom level.
>
> Three, place your bam file on the web server and accessed through a local drive. Network (HTTP) access is very convenient, but also slower. You can upload the Bam files as a custom track rather than providing a URL, provided that the bam file is smaller than the upload limit (ask the admin to increase it if necessary). GBrowse should properly generate a bigWig coverage file (if support is installed) and semantic zooming tracks automatically generated.
>
> Beyond that, there's currently not much else to do to make it faster.
>
> Tim
>
> On Jun 11, 2013, at 8:44 AM, Scott Cain <[hidden email]<mailto:[hidden email]>> wrote:
>
> Hi,
>
> No, there isn't a way to load a sam file into mysql, and I can't imagine you'd want to.  If you feel that the performance is too slow for a bam file, I am quite sure converting a sam file into GFF (which probably could be done) would be worse.  Depending on what you need from your sam file, you might want to consider converting it to a bigwig file, which probably would have better performance (though I don't know off hand how to do that either; perhaps Timothy Parnell can jump in with suggestions).
>
> Scott
>
> PS: Timothy, I don't know if we ever attend the same meetings, but if we're ever in the same state at the same time, I owe you multiple drinks and/or a good dinner :-)
>
>
>
> On Tue, Jun 11, 2013 at 10:04 AM, dguzman <[hidden email]<mailto:[hidden email]>> wrote:
> Thanks for your answer Vipi,
>
> yes, i use Bio::DB::Sam adaptor
>
> but the DBI::mysql adaptor is supported for this file type??
>
>
>
> --
> View this message in context: http://generic-model-organism-system-database.450254.n5.nabble.com/is-possible-upload-a-bam-file-to-mysql-database-tp5711410p5711414.html
> Sent from the gmod-gbrowse mailing list archive at Nabble.com<http://Nabble.com>.
>
> ------------------------------------------------------------------------------
> This SF.net<http://SF.net> email is sponsored by Windows:
>
> Build for Windows Store.
>
> http://p.sf.net/sfu/windows-dev2dev
> _______________________________________________
> Gmod-gbrowse mailing list
> [hidden email]<mailto:[hidden email]>
> https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
>
>
>
> --
> ------------------------------------------------------------------------
> Scott Cain, Ph. D.                                   scott at scottcain dot net
> GMOD Coordinator (http://gmod.org/)                     216-392-3087
> Ontario Institute for Cancer Research
> ------------------------------------------------------------------------------
> This SF.net<http://SF.net> email is sponsored by Windows:
>
> Build for Windows Store.
>
> http://p.sf.net/sfu/windows-dev2dev_______________________________________________
> Gmod-gbrowse mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
>
>
> ------------------------------------------------------------------------------
> This SF.net email is sponsored by Windows:
>
> Build for Windows Store.
>
> http://p.sf.net/sfu/windows-dev2dev
> _______________________________________________
> Gmod-gbrowse mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse

------------------------------------------------------------------------------
This SF.net email is sponsored by Windows:

Build for Windows Store.

http://p.sf.net/sfu/windows-dev2dev
_______________________________________________
Gmod-gbrowse mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
Reply | Threaded
Open this post in threaded view
|

Re: is possible upload a bam file to mysql database?

Siddhartha Basu
In reply to this post by Timothy Parnell
Hi Timothy,

On Tue, 11 Jun 2013, Timothy Parnell wrote:

> Thanks for the complement, Scott! :-)
>
> I agree with Scott, the performance would probably be at least a few fold slower. But you are welcome to do the actual test. There are tools out there to convert Sam alignments to Bed or GFF (BedTools and my own BioToolBox come to mind).

>
> There are couple of things to do to speed things up.
>
> Like Scott said, you can generate a wig (and sequentially bigWig) file of coverage using a variety of methods.
> Since wig files are just displaying scores, they are considerably faster. Otherwise, the alignments in the Bam file must be iterated and parsed each time to calculate coverage, which takes time. The Bio::DB::Sam module includes a basic utility (bam2bedgraph) to calculate coverage.
> BedTools also provides a fast coverage option with some options. I have a multi-threaded bam2wig.pl which will calculate coverage
> any which way under the sun at my BioToolBox project (http://biotoolbox.googlecode.com).
BioToolBox seems to be an awesome project, written in perl(that i am
comfortable with) and almost each piece is nicely documented.
Currently i am facing slowness in the semantic zooming of the bam tracks
in gbrowse. Looks like the BigWig backend should be an upgrade, would give it a
shot.

thanks,
-siddhartha



>
> Two, make sure you use semantic zooming properly to switch between coverage and alignments. Alignments are only really useful under 1 or 2 kb, so switch to coverage via bigWig above that zoom level.
>
> Three, place your bam file on the web server and accessed through a local drive. Network (HTTP) access is very convenient, but also slower. You can upload the Bam files as a custom track rather than providing a URL, provided that the bam file is smaller than the upload limit (ask the admin to increase it if necessary). GBrowse should properly generate a bigWig coverage file (if support is installed) and semantic zooming tracks automatically generated.
>
> Beyond that, there's currently not much else to do to make it faster.
>
> Tim
>
> On Jun 11, 2013, at 8:44 AM, Scott Cain <[hidden email]<mailto:[hidden email]>> wrote:
>
> Hi,
>
> No, there isn't a way to load a sam file into mysql, and I can't imagine you'd want to.  If you feel that the performance is too slow for a bam file, I am quite sure converting a sam file into GFF (which probably could be done) would be worse.  Depending on what you need from your sam file, you might want to consider converting it to a bigwig file, which probably would have better performance (though I don't know off hand how to do that either; perhaps Timothy Parnell can jump in with suggestions).
>
> Scott
>
> PS: Timothy, I don't know if we ever attend the same meetings, but if we're ever in the same state at the same time, I owe you multiple drinks and/or a good dinner :-)
>
>
>
> On Tue, Jun 11, 2013 at 10:04 AM, dguzman <[hidden email]<mailto:[hidden email]>> wrote:
> Thanks for your answer Vipi,
>
> yes, i use Bio::DB::Sam adaptor
>
> but the DBI::mysql adaptor is supported for this file type??
>
>
>
> --
> View this message in context: http://generic-model-organism-system-database.450254.n5.nabble.com/is-possible-upload-a-bam-file-to-mysql-database-tp5711410p5711414.html
> Sent from the gmod-gbrowse mailing list archive at Nabble.com<http://Nabble.com>.
>
> ------------------------------------------------------------------------------
> This SF.net<http://SF.net> email is sponsored by Windows:
>
> Build for Windows Store.
>
> http://p.sf.net/sfu/windows-dev2dev
> _______________________________________________
> Gmod-gbrowse mailing list
> [hidden email]<mailto:[hidden email]>
> https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
>
>
>
> --
> ------------------------------------------------------------------------
> Scott Cain, Ph. D.                                   scott at scottcain dot net
> GMOD Coordinator (http://gmod.org/)                     216-392-3087
> Ontario Institute for Cancer Research
> ------------------------------------------------------------------------------
> This SF.net<http://SF.net> email is sponsored by Windows:
>
> Build for Windows Store.
>
> http://p.sf.net/sfu/windows-dev2dev_______________________________________________
> Gmod-gbrowse mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
>
>
> ------------------------------------------------------------------------------
> This SF.net email is sponsored by Windows:
>
> Build for Windows Store.
>
> http://p.sf.net/sfu/windows-dev2dev
> _______________________________________________
> Gmod-gbrowse mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse

------------------------------------------------------------------------------
This SF.net email is sponsored by Windows:

Build for Windows Store.

http://p.sf.net/sfu/windows-dev2dev
_______________________________________________
Gmod-gbrowse mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
Reply | Threaded
Open this post in threaded view
|

Re: is possible upload a bam file to mysql database?

Timothy Parnell
Hi Siddhartha,
Which wiggle glyph are you using that is so slow? The wiggle_xyplot and wiggle_whiskers are pretty reasonable, but xyplot would be very slow.

I'm not sure if the BigWig backend is necessarily the bottleneck, since a lot of the actual bw file reading and parsing is done by compiled C code, and the high level Perl interface shouldn't add too much more overhead. I haven't done benchmarking, but I suspect the Bio::Graphics perl modules are the bigger problem. Following them can sometimes be a pretty convoluted path; I'm guessing because they've been added to and extended many times over to work with dozens of different glyphs through the history of the project.

On Jun 11, 2013, at 3:04 PM, Siddhartha Basu <[hidden email]> wrote:

> Hi Timothy,
>
> On Tue, 11 Jun 2013, Timothy Parnell wrote:
>
>> Thanks for the complement, Scott! :-)
>>
>> I agree with Scott, the performance would probably be at least a few fold slower. But you are welcome to do the actual test. There are tools out there to convert Sam alignments to Bed or GFF (BedTools and my own BioToolBox come to mind).
>
>>
>> There are couple of things to do to speed things up.
>>
>> Like Scott said, you can generate a wig (and sequentially bigWig) file of coverage using a variety of methods.
>> Since wig files are just displaying scores, they are considerably faster. Otherwise, the alignments in the Bam file must be iterated and parsed each time to calculate coverage, which takes time. The Bio::DB::Sam module includes a basic utility (bam2bedgraph) to calculate coverage.
>> BedTools also provides a fast coverage option with some options. I have a multi-threaded bam2wig.pl which will calculate coverage
>> any which way under the sun at my BioToolBox project (http://biotoolbox.googlecode.com).
> BioToolBox seems to be an awesome project, written in perl(that i am
> comfortable with) and almost each piece is nicely documented.
> Currently i am facing slowness in the semantic zooming of the bam tracks
> in gbrowse. Looks like the BigWig backend should be an upgrade, would give it a
> shot.
>
> thanks,
> -siddhartha
>
>
>
>>
>> Two, make sure you use semantic zooming properly to switch between coverage and alignments. Alignments are only really useful under 1 or 2 kb, so switch to coverage via bigWig above that zoom level.
>>
>> Three, place your bam file on the web server and accessed through a local drive. Network (HTTP) access is very convenient, but also slower. You can upload the Bam files as a custom track rather than providing a URL, provided that the bam file is smaller than the upload limit (ask the admin to increase it if necessary). GBrowse should properly generate a bigWig coverage file (if support is installed) and semantic zooming tracks automatically generated.
>>
>> Beyond that, there's currently not much else to do to make it faster.
>>
>> Tim
>>
>> On Jun 11, 2013, at 8:44 AM, Scott Cain <[hidden email]<mailto:[hidden email]>> wrote:
>>
>> Hi,
>>
>> No, there isn't a way to load a sam file into mysql, and I can't imagine you'd want to.  If you feel that the performance is too slow for a bam file, I am quite sure converting a sam file into GFF (which probably could be done) would be worse.  Depending on what you need from your sam file, you might want to consider converting it to a bigwig file, which probably would have better performance (though I don't know off hand how to do that either; perhaps Timothy Parnell can jump in with suggestions).
>>
>> Scott
>>
>> PS: Timothy, I don't know if we ever attend the same meetings, but if we're ever in the same state at the same time, I owe you multiple drinks and/or a good dinner :-)
>>
>>
>>
>> On Tue, Jun 11, 2013 at 10:04 AM, dguzman <[hidden email]<mailto:[hidden email]>> wrote:
>> Thanks for your answer Vipi,
>>
>> yes, i use Bio::DB::Sam adaptor
>>
>> but the DBI::mysql adaptor is supported for this file type??
>>
>>
>>
>> --
>> View this message in context: http://generic-model-organism-system-database.450254.n5.nabble.com/is-possible-upload-a-bam-file-to-mysql-database-tp5711410p5711414.html
>> Sent from the gmod-gbrowse mailing list archive at Nabble.com<http://Nabble.com>.
>>
>> ------------------------------------------------------------------------------
>> This SF.net<http://SF.net> email is sponsored by Windows:
>>
>> Build for Windows Store.
>>
>> http://p.sf.net/sfu/windows-dev2dev
>> _______________________________________________
>> Gmod-gbrowse mailing list
>> [hidden email]<mailto:[hidden email]>
>> https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
>>
>>
>>
>> --
>> ------------------------------------------------------------------------
>> Scott Cain, Ph. D.                                   scott at scottcain dot net
>> GMOD Coordinator (http://gmod.org/)                     216-392-3087
>> Ontario Institute for Cancer Research
>> ------------------------------------------------------------------------------
>> This SF.net<http://SF.net> email is sponsored by Windows:
>>
>> Build for Windows Store.
>>
>> http://p.sf.net/sfu/windows-dev2dev_______________________________________________
>> Gmod-gbrowse mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
>>
>>
>> ------------------------------------------------------------------------------
>> This SF.net email is sponsored by Windows:
>>
>> Build for Windows Store.
>>
>> http://p.sf.net/sfu/windows-dev2dev
>> _______________________________________________
>> Gmod-gbrowse mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
>
> ------------------------------------------------------------------------------
> This SF.net email is sponsored by Windows:
>
> Build for Windows Store.
>
> http://p.sf.net/sfu/windows-dev2dev
> _______________________________________________
> Gmod-gbrowse mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse


------------------------------------------------------------------------------
This SF.net email is sponsored by Windows:

Build for Windows Store.

http://p.sf.net/sfu/windows-dev2dev
_______________________________________________
Gmod-gbrowse mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
Reply | Threaded
Open this post in threaded view
|

Re: is possible upload a bam file to mysql database?

Siddhartha Basu
Hi timothy,
At this point, i am actually using the Bio::DB::Bam backend with the
wiggle_xyplot which seems to be sluggish mostly during semantic zooming(~200 bp).
I am having the impression that switching to BigWig
backend probably speed things up. And thinking about your toolbox script
to make the conversion.

thanks,
-siddhartha

On Tue, 11 Jun 2013, Timothy Parnell wrote:

> Hi Siddhartha,
> Which wiggle glyph are you using that is so slow? The wiggle_xyplot and wiggle_whiskers are pretty reasonable, but xyplot would be very slow.
>
> I'm not sure if the BigWig backend is necessarily the bottleneck, since a lot of the actual bw file reading and parsing is done by compiled C code, and the high level Perl interface shouldn't add too much more overhead. I haven't done benchmarking, but I suspect the Bio::Graphics perl modules are the bigger problem. Following them can sometimes be a pretty convoluted path; I'm guessing because they've been added to and extended many times over to work with dozens of different glyphs through the history of the project.
>
> On Jun 11, 2013, at 3:04 PM, Siddhartha Basu <[hidden email]> wrote:
>
> > Hi Timothy,
> >
> > On Tue, 11 Jun 2013, Timothy Parnell wrote:
> >
> >> Thanks for the complement, Scott! :-)
> >>
> >> I agree with Scott, the performance would probably be at least a few fold slower. But you are welcome to do the actual test. There are tools out there to convert Sam alignments to Bed or GFF (BedTools and my own BioToolBox come to mind).
> >
> >>
> >> There are couple of things to do to speed things up.
> >>
> >> Like Scott said, you can generate a wig (and sequentially bigWig) file of coverage using a variety of methods.
> >> Since wig files are just displaying scores, they are considerably faster. Otherwise, the alignments in the Bam file must be iterated and parsed each time to calculate coverage, which takes time. The Bio::DB::Sam module includes a basic utility (bam2bedgraph) to calculate coverage.
> >> BedTools also provides a fast coverage option with some options. I have a multi-threaded bam2wig.pl which will calculate coverage
> >> any which way under the sun at my BioToolBox project (http://biotoolbox.googlecode.com).
> > BioToolBox seems to be an awesome project, written in perl(that i am
> > comfortable with) and almost each piece is nicely documented.
> > Currently i am facing slowness in the semantic zooming of the bam tracks
> > in gbrowse. Looks like the BigWig backend should be an upgrade, would give it a
> > shot.
> >
> > thanks,
> > -siddhartha
> >
> >
> >
> >>
> >> Two, make sure you use semantic zooming properly to switch between coverage and alignments. Alignments are only really useful under 1 or 2 kb, so switch to coverage via bigWig above that zoom level.
> >>
> >> Three, place your bam file on the web server and accessed through a local drive. Network (HTTP) access is very convenient, but also slower. You can upload the Bam files as a custom track rather than providing a URL, provided that the bam file is smaller than the upload limit (ask the admin to increase it if necessary). GBrowse should properly generate a bigWig coverage file (if support is installed) and semantic zooming tracks automatically generated.
> >>
> >> Beyond that, there's currently not much else to do to make it faster.
> >>
> >> Tim
> >>
> >> On Jun 11, 2013, at 8:44 AM, Scott Cain <[hidden email]<mailto:[hidden email]>> wrote:
> >>
> >> Hi,
> >>
> >> No, there isn't a way to load a sam file into mysql, and I can't imagine you'd want to.  If you feel that the performance is too slow for a bam file, I am quite sure converting a sam file into GFF (which probably could be done) would be worse.  Depending on what you need from your sam file, you might want to consider converting it to a bigwig file, which probably would have better performance (though I don't know off hand how to do that either; perhaps Timothy Parnell can jump in with suggestions).
> >>
> >> Scott
> >>
> >> PS: Timothy, I don't know if we ever attend the same meetings, but if we're ever in the same state at the same time, I owe you multiple drinks and/or a good dinner :-)
> >>
> >>
> >>
> >> On Tue, Jun 11, 2013 at 10:04 AM, dguzman <[hidden email]<mailto:[hidden email]>> wrote:
> >> Thanks for your answer Vipi,
> >>
> >> yes, i use Bio::DB::Sam adaptor
> >>
> >> but the DBI::mysql adaptor is supported for this file type??
> >>
> >>
> >>
> >> --
> >> View this message in context: http://generic-model-organism-system-database.450254.n5.nabble.com/is-possible-upload-a-bam-file-to-mysql-database-tp5711410p5711414.html
> >> Sent from the gmod-gbrowse mailing list archive at Nabble.com<http://Nabble.com>.
> >>
> >> ------------------------------------------------------------------------------
> >> This SF.net<http://SF.net> email is sponsored by Windows:
> >>
> >> Build for Windows Store.
> >>
> >> http://p.sf.net/sfu/windows-dev2dev
> >> _______________________________________________
> >> Gmod-gbrowse mailing list
> >> [hidden email]<mailto:[hidden email]>
> >> https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
> >>
> >>
> >>
> >> --
> >> ------------------------------------------------------------------------
> >> Scott Cain, Ph. D.                                   scott at scottcain dot net
> >> GMOD Coordinator (http://gmod.org/)                     216-392-3087
> >> Ontario Institute for Cancer Research
> >> ------------------------------------------------------------------------------
> >> This SF.net<http://SF.net> email is sponsored by Windows:
> >>
> >> Build for Windows Store.
> >>
> >> http://p.sf.net/sfu/windows-dev2dev_______________________________________________
> >> Gmod-gbrowse mailing list
> >> [hidden email]
> >> https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
> >>
> >>
> >> ------------------------------------------------------------------------------
> >> This SF.net email is sponsored by Windows:
> >>
> >> Build for Windows Store.
> >>
> >> http://p.sf.net/sfu/windows-dev2dev
> >> _______________________________________________
> >> Gmod-gbrowse mailing list
> >> [hidden email]
> >> https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
> >
> > ------------------------------------------------------------------------------
> > This SF.net email is sponsored by Windows:
> >
> > Build for Windows Store.
> >
> > http://p.sf.net/sfu/windows-dev2dev
> > _______________________________________________
> > Gmod-gbrowse mailing list
> > [hidden email]
> > https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
>

------------------------------------------------------------------------------
This SF.net email is sponsored by Windows:

Build for Windows Store.

http://p.sf.net/sfu/windows-dev2dev
_______________________________________________
Gmod-gbrowse mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse