SAM/BAM file location

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

SAM/BAM file location

Omar Darwish
Hello all,

Is there is any way I can save my BAM databases other than /var/lib/gbrowse2/databases . As this my file system is full and can not move more data there & could not figure out a way to increase teh space. Any help is appreciated.

 
Thanks!


------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Gmod-gbrowse mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
Reply | Threaded
Open this post in threaded view
|

Re: SAM/BAM file location

Fields, Christopher J
I don't think you are limited to /var/ or anywhere else, the key point is that the data has to be accessible to whatever user the server is running under (www-data, apache2, etc).  I have a local instance where I had BAM data on an external RAID and it worked fine with the proper permissions set.

Also, Bio::DB::Sam can take an ftp or http location as an argument for '-bam':

   https://metacpan.org/module/Bio::DB::Sam#Bio::DB::Sam-Constructor-and-basic-accessors

chris

On Nov 14, 2012, at 10:35 AM, Omar Darwish <[hidden email]>
 wrote:

> Hello all,
>
> Is there is any way I can save my BAM databases other than /var/lib/gbrowse2/databases . As this my file system is full and can not move more data there & could not figure out a way to increase teh space. Any help is appreciated.
>
>  
> Thanks!
>
> ------------------------------------------------------------------------------
> Monitor your physical, virtual and cloud infrastructure from a single
> web console. Get in-depth insight into apps, servers, databases, vmware,
> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
> Pricing starts from $795 for 25 servers or applications!
> http://p.sf.net/sfu/zoho_dev2dev_nov_______________________________________________
> Gmod-gbrowse mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse


------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Gmod-gbrowse mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
Reply | Threaded
Open this post in threaded view
|

Re: SAM/BAM file location

Georg Vogler
In reply to this post by Omar Darwish
Hi, Omar!

I've added this section in my conf file (not the Gbrowse.conf), with arguments pointing to the directory of the bam/bai files:

[name:database]
db_adaptor     = Bio::DB::Sam
db_args        = -bam '/var/www/gbrowse2/yourfile.notx.bam'
                 -bai '/var/www/gbrowse2/yourfile.notx.bam.bai'


I guess you can point to any directory that the web browser has access to.

Best,

Geo

Von: Omar Darwish <[hidden email]>
Datum: Wednesday, November 14, 2012 8:35 AM
An: "Gbrowse (E-mail)" <[hidden email]>, Scott Cain <[hidden email]>
Betreff: [Gmod-gbrowse] SAM/BAM file location

Hello all,

Is there is any way I can save my BAM databases other than /var/lib/gbrowse2/databases . As this my file system is full and can not move more data there & could not figure out a way to increase teh space. Any help is appreciated.

 
Thanks!


------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Gmod-gbrowse mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
Reply | Threaded
Open this post in threaded view
|

Re: SAM/BAM file location

Scott Cain
In reply to this post by Fields, Christopher J
Just to follow up on Chris' suggestion about http: you can even put
the bam files on something like a dropbox public drive and use that
url.  I just tried that with this server:

  http://ec2-54-242-42-142.compute-1.amazonaws.com/cgi-bin/gb2/gbrowse/volvox/?start=21900;stop=31899;ref=ctgA;width=800;version=100;flip=0;grid=on;id=079953d1b81c9dbbcadd73d0eea541ad;l=Reads%1EPairs%1EExampleFeatures

(it won't be around long--it's a AWS micro machine.)  The performance
is quite good, the only caveat is that dropbox will give you an https
url for the file, but Bio::DB::Sam doesn't support https urls, so you
just have to take out the "s".  This is what the database stanza
looked like in my config:

[samdb:database]
db_adaptor     = Bio::DB::Sam
db_args        = -bam http://dl.dropbox.com/u/91697297/volvox.sort.bam
search options = none

Note that fasta files can't be remote--they have to be indexed and
local, though in this instance, I didn't provide any fasta and that's
OK too.

Scott


On Wed, Nov 14, 2012 at 1:10 PM, Fields, Christopher J
<[hidden email]> wrote:

> I don't think you are limited to /var/ or anywhere else, the key point is that the data has to be accessible to whatever user the server is running under (www-data, apache2, etc).  I have a local instance where I had BAM data on an external RAID and it worked fine with the proper permissions set.
>
> Also, Bio::DB::Sam can take an ftp or http location as an argument for '-bam':
>
>    https://metacpan.org/module/Bio::DB::Sam#Bio::DB::Sam-Constructor-and-basic-accessors
>
> chris
>
> On Nov 14, 2012, at 10:35 AM, Omar Darwish <[hidden email]>
>  wrote:
>
>> Hello all,
>>
>> Is there is any way I can save my BAM databases other than /var/lib/gbrowse2/databases . As this my file system is full and can not move more data there & could not figure out a way to increase teh space. Any help is appreciated.
>>
>>
>> Thanks!
>>
>> ------------------------------------------------------------------------------
>> Monitor your physical, virtual and cloud infrastructure from a single
>> web console. Get in-depth insight into apps, servers, databases, vmware,
>> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
>> Pricing starts from $795 for 25 servers or applications!
>> http://p.sf.net/sfu/zoho_dev2dev_nov_______________________________________________
>> Gmod-gbrowse mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
>



--
------------------------------------------------------------------------
Scott Cain, Ph. D.                                   scott at scottcain dot net
GMOD Coordinator (http://gmod.org/)                     216-392-3087
Ontario Institute for Cancer Research

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Gmod-gbrowse mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
Reply | Threaded
Open this post in threaded view
|

Re: SAM/BAM file location

Omar Darwish
Hi Scott,

As iPlant offers only limited space, I came back to your suggestion of storing bam files on public drive (dropbox/mega). It did not work ( Please see the attached httpd error_log)This is my database stanza:

[Sample_DB:database]
db_adaptor     = Bio::DB::Sam
db_args        = -fasta /var/lib/gbrowse2/databases/strawberry/fvesca_v1.1_pseudo.fa
search options = default

Another question, as mega website offers 50G free account I thought it might be better to go with it. But I found that the file address you can get have the file name part encrypted, Will that be a problem?
so the above stanza will look like this:

[Sample_DB:database]
db_adaptor     = Bio::DB::Sam
db_args        = -fasta /var/lib/gbrowse2/databases/strawberry/fvesca_v1.1_pseudo.fa
search options = default


Both cases won't  work! Any suggestions?

Thank you!


On Thu, Nov 15, 2012 at 10:42 AM, Scott Cain <[hidden email]> wrote:
Just to follow up on Chris' suggestion about http: you can even put
the bam files on something like a dropbox public drive and use that
url.  I just tried that with this server:

  http://ec2-54-242-42-142.compute-1.amazonaws.com/cgi-bin/gb2/gbrowse/volvox/?start=21900;stop=31899;ref=ctgA;width=800;version=100;flip=0;grid=on;id=079953d1b81c9dbbcadd73d0eea541ad;l=Reads%1EPairs%1EExampleFeatures

(it won't be around long--it's a AWS micro machine.)  The performance
is quite good, the only caveat is that dropbox will give you an https
url for the file, but Bio::DB::Sam doesn't support https urls, so you
just have to take out the "s".  This is what the database stanza
looked like in my config:

[samdb:database]
db_adaptor     = Bio::DB::Sam
db_args        = -bam http://dl.dropbox.com/u/91697297/volvox.sort.bam
search options = none

Note that fasta files can't be remote--they have to be indexed and
local, though in this instance, I didn't provide any fasta and that's
OK too.

Scott


On Wed, Nov 14, 2012 at 1:10 PM, Fields, Christopher J
<[hidden email]> wrote:
> I don't think you are limited to /var/ or anywhere else, the key point is that the data has to be accessible to whatever user the server is running under (www-data, apache2, etc).  I have a local instance where I had BAM data on an external RAID and it worked fine with the proper permissions set.
>
> Also, Bio::DB::Sam can take an ftp or http location as an argument for '-bam':
>
>    https://metacpan.org/module/Bio::DB::Sam#Bio::DB::Sam-Constructor-and-basic-accessors
>
> chris
>
> On Nov 14, 2012, at 10:35 AM, Omar Darwish <[hidden email]>
>  wrote:
>
>> Hello all,
>>
>> Is there is any way I can save my BAM databases other than /var/lib/gbrowse2/databases . As this my file system is full and can not move more data there & could not figure out a way to increase teh space. Any help is appreciated.
>>
>>
>> Thanks!
>>
>> ------------------------------------------------------------------------------
>> Monitor your physical, virtual and cloud infrastructure from a single
>> web console. Get in-depth insight into apps, servers, databases, vmware,
>> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
>> Pricing starts from $795 for 25 servers or applications!
>> http://p.sf.net/sfu/zoho_dev2dev_nov_______________________________________________
>> Gmod-gbrowse mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
>



--
------------------------------------------------------------------------
Scott Cain, Ph. D.                                   scott at scottcain dot net
GMOD Coordinator (http://gmod.org/)                     <a href="tel:216-392-3087" value="+12163923087">216-392-3087
Ontario Institute for Cancer Research



--
Regards,
Omar

------------------------------------------------------------------------------
The Go Parallel Website, sponsored by Intel - in partnership with Geeknet,
is your hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials, tech docs,
whitepapers, evaluation guides, and opinion stories. Check out the most
recent posts - join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Gmod-gbrowse mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse

error_log.txt (12K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: SAM/BAM file location

Scott Cain
Hi Omar,

I don't know much about mega, so I can't give you much advice there,
but in general, I can say that clicking on the links below should
result in getting the file requested, without going through an
intermediate page (like giving me a window to agree to terms or the
like), since GBrowse/SAMtools can't traverse those pages.

For dropbox, when I clicked on the bam file link, I got a 403 http
error, which is some version of permission denied.  Obviously, the
dropbox link needs to be a public link.

Scott


On Sat, Feb 16, 2013 at 9:14 AM, Omar Darwish <[hidden email]> wrote:

> Hi Scott,
>
> As iPlant offers only limited space, I came back to your suggestion of
> storing bam files on public drive (dropbox/mega). It did not work ( Please
> see the attached httpd error_log)This is my database stanza:
>
> [Sample_DB:database]
> db_adaptor     = Bio::DB::Sam
> db_args        = -fasta
> /var/lib/gbrowse2/databases/strawberry/fvesca_v1.1_pseudo.fa
>                  -bam
> http://dl.dropbox.com/s/uj8by9bcipwr2c4/Sample_DB.sort.bam
> -bai http://dl.dropbox.com/s/yx6zon4w8ryohlb/Sample_DB.sort.bam.bai
> search options = default
>
> Another question, as mega website offers 50G free account I thought it might
> be better to go with it. But I found that the file address you can get have
> the file name part encrypted, Will that be a problem?
> so the above stanza will look like this:
>
> [Sample_DB:database]
> db_adaptor     = Bio::DB::Sam
> db_args        = -fasta
> /var/lib/gbrowse2/databases/strawberry/fvesca_v1.1_pseudo.fa
>                  -bam
> http://mega.co.nz/#!IsAyQBLa!IDnD1Em--hmguYKkekFIghrh7akOl0-Ob5ie6ofg
>  -bai http://mega.co.nz/#!VxJEwJ4Y!JeBUE2TY2OQ_epq4_65yagFWjdgV5OKZDd6QfVY0
> search options = default
>
>
> Both cases won't  work! Any suggestions?
>
> Thank you!
>
>
> On Thu, Nov 15, 2012 at 10:42 AM, Scott Cain <[hidden email]> wrote:
>>
>> Just to follow up on Chris' suggestion about http: you can even put
>> the bam files on something like a dropbox public drive and use that
>> url.  I just tried that with this server:
>>
>>
>> http://ec2-54-242-42-142.compute-1.amazonaws.com/cgi-bin/gb2/gbrowse/volvox/?start=21900;stop=31899;ref=ctgA;width=800;version=100;flip=0;grid=on;id=079953d1b81c9dbbcadd73d0eea541ad;l=Reads%1EPairs%1EExampleFeatures
>>
>> (it won't be around long--it's a AWS micro machine.)  The performance
>> is quite good, the only caveat is that dropbox will give you an https
>> url for the file, but Bio::DB::Sam doesn't support https urls, so you
>> just have to take out the "s".  This is what the database stanza
>> looked like in my config:
>>
>> [samdb:database]
>> db_adaptor     = Bio::DB::Sam
>> db_args        = -bam http://dl.dropbox.com/u/91697297/volvox.sort.bam
>> search options = none
>>
>> Note that fasta files can't be remote--they have to be indexed and
>> local, though in this instance, I didn't provide any fasta and that's
>> OK too.
>>
>> Scott
>>
>>
>> On Wed, Nov 14, 2012 at 1:10 PM, Fields, Christopher J
>> <[hidden email]> wrote:
>> > I don't think you are limited to /var/ or anywhere else, the key point
>> > is that the data has to be accessible to whatever user the server is running
>> > under (www-data, apache2, etc).  I have a local instance where I had BAM
>> > data on an external RAID and it worked fine with the proper permissions set.
>> >
>> > Also, Bio::DB::Sam can take an ftp or http location as an argument for
>> > '-bam':
>> >
>> >
>> > https://metacpan.org/module/Bio::DB::Sam#Bio::DB::Sam-Constructor-and-basic-accessors
>> >
>> > chris
>> >
>> > On Nov 14, 2012, at 10:35 AM, Omar Darwish <[hidden email]>
>> >  wrote:
>> >
>> >> Hello all,
>> >>
>> >> Is there is any way I can save my BAM databases other than
>> >> /var/lib/gbrowse2/databases . As this my file system is full and can not
>> >> move more data there & could not figure out a way to increase teh space. Any
>> >> help is appreciated.
>> >>
>> >>
>> >> Thanks!
>> >>
>> >>
>> >> ------------------------------------------------------------------------------
>> >> Monitor your physical, virtual and cloud infrastructure from a single
>> >> web console. Get in-depth insight into apps, servers, databases,
>> >> vmware,
>> >> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
>> >> Pricing starts from $795 for 25 servers or applications!
>> >>
>> >> http://p.sf.net/sfu/zoho_dev2dev_nov_______________________________________________
>> >> Gmod-gbrowse mailing list
>> >> [hidden email]
>> >> https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
>> >
>>
>>
>>
>> --
>> ------------------------------------------------------------------------
>> Scott Cain, Ph. D.                                   scott at scottcain
>> dot net
>> GMOD Coordinator (http://gmod.org/)                     216-392-3087
>> Ontario Institute for Cancer Research
>
>
>
>
> --
> Regards,
> Omar



--
------------------------------------------------------------------------
Scott Cain, Ph. D.                                   scott at scottcain dot net
GMOD Coordinator (http://gmod.org/)                     216-392-3087
Ontario Institute for Cancer Research

------------------------------------------------------------------------------
The Go Parallel Website, sponsored by Intel - in partnership with Geeknet,
is your hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials, tech docs,
whitepapers, evaluation guides, and opinion stories. Check out the most
recent posts - join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Gmod-gbrowse mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
Reply | Threaded
Open this post in threaded view
|

Re: SAM/BAM file location

Omar Darwish
Hi Scott,

Yes, Mega is different from Dropbox, It goes through intermediate page before downloading the file. Here is the actual link from drop box:

Thanks


On Sat, Feb 16, 2013 at 4:13 PM, Scott Cain <[hidden email]> wrote:
Hi Omar,

I don't know much about mega, so I can't give you much advice there,
but in general, I can say that clicking on the links below should
result in getting the file requested, without going through an
intermediate page (like giving me a window to agree to terms or the
like), since GBrowse/SAMtools can't traverse those pages.

For dropbox, when I clicked on the bam file link, I got a 403 http
error, which is some version of permission denied.  Obviously, the
dropbox link needs to be a public link.

Scott


On Sat, Feb 16, 2013 at 9:14 AM, Omar Darwish <[hidden email]> wrote:
> Hi Scott,
>
> As iPlant offers only limited space, I came back to your suggestion of
> storing bam files on public drive (dropbox/mega). It did not work ( Please
> see the attached httpd error_log)This is my database stanza:
>
> [Sample_DB:database]
> db_adaptor     = Bio::DB::Sam
> db_args        = -fasta
> /var/lib/gbrowse2/databases/strawberry/fvesca_v1.1_pseudo.fa
>                  -bam
> http://dl.dropbox.com/s/uj8by9bcipwr2c4/Sample_DB.sort.bam
> -bai http://dl.dropbox.com/s/yx6zon4w8ryohlb/Sample_DB.sort.bam.bai
> search options = default
>
> Another question, as mega website offers 50G free account I thought it might
> be better to go with it. But I found that the file address you can get have
> the file name part encrypted, Will that be a problem?
> so the above stanza will look like this:
>
> [Sample_DB:database]
> db_adaptor     = Bio::DB::Sam
> db_args        = -fasta
> /var/lib/gbrowse2/databases/strawberry/fvesca_v1.1_pseudo.fa
>                  -bam
> http://mega.co.nz/#!IsAyQBLa!IDnD1Em--hmguYKkekFIghrh7akOl0-Ob5ie6ofg
>  -bai http://mega.co.nz/#!VxJEwJ4Y!JeBUE2TY2OQ_epq4_65yagFWjdgV5OKZDd6QfVY0
> search options = default
>
>
> Both cases won't  work! Any suggestions?
>
> Thank you!
>
>
> On Thu, Nov 15, 2012 at 10:42 AM, Scott Cain <[hidden email]> wrote:
>>
>> Just to follow up on Chris' suggestion about http: you can even put
>> the bam files on something like a dropbox public drive and use that
>> url.  I just tried that with this server:
>>
>>
>> http://ec2-54-242-42-142.compute-1.amazonaws.com/cgi-bin/gb2/gbrowse/volvox/?start=21900;stop=31899;ref=ctgA;width=800;version=100;flip=0;grid=on;id=079953d1b81c9dbbcadd73d0eea541ad;l=Reads%1EPairs%1EExampleFeatures
>>
>> (it won't be around long--it's a AWS micro machine.)  The performance
>> is quite good, the only caveat is that dropbox will give you an https
>> url for the file, but Bio::DB::Sam doesn't support https urls, so you
>> just have to take out the "s".  This is what the database stanza
>> looked like in my config:
>>
>> [samdb:database]
>> db_adaptor     = Bio::DB::Sam
>> db_args        = -bam http://dl.dropbox.com/u/91697297/volvox.sort.bam
>> search options = none
>>
>> Note that fasta files can't be remote--they have to be indexed and
>> local, though in this instance, I didn't provide any fasta and that's
>> OK too.
>>
>> Scott
>>
>>
>> On Wed, Nov 14, 2012 at 1:10 PM, Fields, Christopher J
>> <[hidden email]> wrote:
>> > I don't think you are limited to /var/ or anywhere else, the key point
>> > is that the data has to be accessible to whatever user the server is running
>> > under (www-data, apache2, etc).  I have a local instance where I had BAM
>> > data on an external RAID and it worked fine with the proper permissions set.
>> >
>> > Also, Bio::DB::Sam can take an ftp or http location as an argument for
>> > '-bam':
>> >
>> >
>> > https://metacpan.org/module/Bio::DB::Sam#Bio::DB::Sam-Constructor-and-basic-accessors
>> >
>> > chris
>> >
>> > On Nov 14, 2012, at 10:35 AM, Omar Darwish <[hidden email]>
>> >  wrote:
>> >
>> >> Hello all,
>> >>
>> >> Is there is any way I can save my BAM databases other than
>> >> /var/lib/gbrowse2/databases . As this my file system is full and can not
>> >> move more data there & could not figure out a way to increase teh space. Any
>> >> help is appreciated.
>> >>
>> >>
>> >> Thanks!
>> >>
>> >>
>> >> ------------------------------------------------------------------------------
>> >> Monitor your physical, virtual and cloud infrastructure from a single
>> >> web console. Get in-depth insight into apps, servers, databases,
>> >> vmware,
>> >> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
>> >> Pricing starts from $795 for 25 servers or applications!
>> >>
>> >> http://p.sf.net/sfu/zoho_dev2dev_nov_______________________________________________
>> >> Gmod-gbrowse mailing list
>> >> [hidden email]
>> >> https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
>> >
>>
>>
>>
>> --
>> ------------------------------------------------------------------------
>> Scott Cain, Ph. D.                                   scott at scottcain
>> dot net
>> GMOD Coordinator (http://gmod.org/)                     <a href="tel:216-392-3087" value="+12163923087">216-392-3087
>> Ontario Institute for Cancer Research
>
>
>
>
> --
> Regards,
> Omar



--
------------------------------------------------------------------------
Scott Cain, Ph. D.                                   scott at scottcain dot net
GMOD Coordinator (http://gmod.org/)                     <a href="tel:216-392-3087" value="+12163923087">216-392-3087
Ontario Institute for Cancer Research



--
Regards,
Omar

------------------------------------------------------------------------------
The Go Parallel Website, sponsored by Intel - in partnership with Geeknet,
is your hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials, tech docs,
whitepapers, evaluation guides, and opinion stories. Check out the most
recent posts - join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Gmod-gbrowse mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
Reply | Threaded
Open this post in threaded view
|

Re: SAM/BAM file location

Scott Cain
Hi Omar,

Interesting: if I use curl to fetch your link, I get a small, "you
will be redirected page", but if I fetch a url to a GFF file I've
created (like http://dl.dropbox.com/u/91697297/yeast.gff), I just get
the file.  Perhaps dropbox won't let you just fetch large files
without going through some sort of redirect (which SAMtools probably
can't follow).  I don't know enough about dropbox to know if that's
the case.  When I tested this with dropbox before, it was with a
relatively small BAM file.

Scott


On Sun, Feb 17, 2013 at 11:31 AM, Omar Darwish <[hidden email]> wrote:

> Hi Scott,
>
> Yes, Mega is different from Dropbox, It goes through intermediate page
> before downloading the file. Here is the actual link from drop box:
> https://www.dropbox.com/s/irgefpicin3xpd6/Bowtie_1_1_Aligned.sort.bam
>
> Thanks
>
>
> On Sat, Feb 16, 2013 at 4:13 PM, Scott Cain <[hidden email]> wrote:
>>
>> Hi Omar,
>>
>> I don't know much about mega, so I can't give you much advice there,
>> but in general, I can say that clicking on the links below should
>> result in getting the file requested, without going through an
>> intermediate page (like giving me a window to agree to terms or the
>> like), since GBrowse/SAMtools can't traverse those pages.
>>
>> For dropbox, when I clicked on the bam file link, I got a 403 http
>> error, which is some version of permission denied.  Obviously, the
>> dropbox link needs to be a public link.
>>
>> Scott
>>
>>
>> On Sat, Feb 16, 2013 at 9:14 AM, Omar Darwish <[hidden email]>
>> wrote:
>> > Hi Scott,
>> >
>> > As iPlant offers only limited space, I came back to your suggestion of
>> > storing bam files on public drive (dropbox/mega). It did not work (
>> > Please
>> > see the attached httpd error_log)This is my database stanza:
>> >
>> > [Sample_DB:database]
>> > db_adaptor     = Bio::DB::Sam
>> > db_args        = -fasta
>> > /var/lib/gbrowse2/databases/strawberry/fvesca_v1.1_pseudo.fa
>> >                  -bam
>> > http://dl.dropbox.com/s/uj8by9bcipwr2c4/Sample_DB.sort.bam
>> > -bai http://dl.dropbox.com/s/yx6zon4w8ryohlb/Sample_DB.sort.bam.bai
>> > search options = default
>> >
>> > Another question, as mega website offers 50G free account I thought it
>> > might
>> > be better to go with it. But I found that the file address you can get
>> > have
>> > the file name part encrypted, Will that be a problem?
>> > so the above stanza will look like this:
>> >
>> > [Sample_DB:database]
>> > db_adaptor     = Bio::DB::Sam
>> > db_args        = -fasta
>> > /var/lib/gbrowse2/databases/strawberry/fvesca_v1.1_pseudo.fa
>> >                  -bam
>> > http://mega.co.nz/#!IsAyQBLa!IDnD1Em--hmguYKkekFIghrh7akOl0-Ob5ie6ofg
>> >  -bai
>> > http://mega.co.nz/#!VxJEwJ4Y!JeBUE2TY2OQ_epq4_65yagFWjdgV5OKZDd6QfVY0
>> > search options = default
>> >
>> >
>> > Both cases won't  work! Any suggestions?
>> >
>> > Thank you!
>> >
>> >
>> > On Thu, Nov 15, 2012 at 10:42 AM, Scott Cain <[hidden email]>
>> > wrote:
>> >>
>> >> Just to follow up on Chris' suggestion about http: you can even put
>> >> the bam files on something like a dropbox public drive and use that
>> >> url.  I just tried that with this server:
>> >>
>> >>
>> >>
>> >> http://ec2-54-242-42-142.compute-1.amazonaws.com/cgi-bin/gb2/gbrowse/volvox/?start=21900;stop=31899;ref=ctgA;width=800;version=100;flip=0;grid=on;id=079953d1b81c9dbbcadd73d0eea541ad;l=Reads%1EPairs%1EExampleFeatures
>> >>
>> >> (it won't be around long--it's a AWS micro machine.)  The performance
>> >> is quite good, the only caveat is that dropbox will give you an https
>> >> url for the file, but Bio::DB::Sam doesn't support https urls, so you
>> >> just have to take out the "s".  This is what the database stanza
>> >> looked like in my config:
>> >>
>> >> [samdb:database]
>> >> db_adaptor     = Bio::DB::Sam
>> >> db_args        = -bam http://dl.dropbox.com/u/91697297/volvox.sort.bam
>> >> search options = none
>> >>
>> >> Note that fasta files can't be remote--they have to be indexed and
>> >> local, though in this instance, I didn't provide any fasta and that's
>> >> OK too.
>> >>
>> >> Scott
>> >>
>> >>
>> >> On Wed, Nov 14, 2012 at 1:10 PM, Fields, Christopher J
>> >> <[hidden email]> wrote:
>> >> > I don't think you are limited to /var/ or anywhere else, the key
>> >> > point
>> >> > is that the data has to be accessible to whatever user the server is
>> >> > running
>> >> > under (www-data, apache2, etc).  I have a local instance where I had
>> >> > BAM
>> >> > data on an external RAID and it worked fine with the proper
>> >> > permissions set.
>> >> >
>> >> > Also, Bio::DB::Sam can take an ftp or http location as an argument
>> >> > for
>> >> > '-bam':
>> >> >
>> >> >
>> >> >
>> >> > https://metacpan.org/module/Bio::DB::Sam#Bio::DB::Sam-Constructor-and-basic-accessors
>> >> >
>> >> > chris
>> >> >
>> >> > On Nov 14, 2012, at 10:35 AM, Omar Darwish <[hidden email]>
>> >> >  wrote:
>> >> >
>> >> >> Hello all,
>> >> >>
>> >> >> Is there is any way I can save my BAM databases other than
>> >> >> /var/lib/gbrowse2/databases . As this my file system is full and can
>> >> >> not
>> >> >> move more data there & could not figure out a way to increase teh
>> >> >> space. Any
>> >> >> help is appreciated.
>> >> >>
>> >> >>
>> >> >> Thanks!
>> >> >>
>> >> >>
>> >> >>
>> >> >> ------------------------------------------------------------------------------
>> >> >> Monitor your physical, virtual and cloud infrastructure from a
>> >> >> single
>> >> >> web console. Get in-depth insight into apps, servers, databases,
>> >> >> vmware,
>> >> >> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
>> >> >> Pricing starts from $795 for 25 servers or applications!
>> >> >>
>> >> >>
>> >> >> http://p.sf.net/sfu/zoho_dev2dev_nov_______________________________________________
>> >> >> Gmod-gbrowse mailing list
>> >> >> [hidden email]
>> >> >> https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >>
>> >> ------------------------------------------------------------------------
>> >> Scott Cain, Ph. D.                                   scott at scottcain
>> >> dot net
>> >> GMOD Coordinator (http://gmod.org/)                     216-392-3087
>> >> Ontario Institute for Cancer Research
>> >
>> >
>> >
>> >
>> > --
>> > Regards,
>> > Omar
>>
>>
>>
>> --
>> ------------------------------------------------------------------------
>> Scott Cain, Ph. D.                                   scott at scottcain
>> dot net
>> GMOD Coordinator (http://gmod.org/)                     216-392-3087
>> Ontario Institute for Cancer Research
>
>
>
>
> --
> Regards,
> Omar



--
------------------------------------------------------------------------
Scott Cain, Ph. D.                                   scott at scottcain dot net
GMOD Coordinator (http://gmod.org/)                     216-392-3087
Ontario Institute for Cancer Research

------------------------------------------------------------------------------
The Go Parallel Website, sponsored by Intel - in partnership with Geeknet,
is your hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials, tech docs,
whitepapers, evaluation guides, and opinion stories. Check out the most
recent posts - join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Gmod-gbrowse mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
Reply | Threaded
Open this post in threaded view
|

Re: SAM/BAM file location

Omar Darwish
Okay, Thanks Scott!


On Mon, Feb 18, 2013 at 10:15 AM, Scott Cain <[hidden email]> wrote:
Hi Omar,

Interesting: if I use curl to fetch your link, I get a small, "you
will be redirected page", but if I fetch a url to a GFF file I've
created (like http://dl.dropbox.com/u/91697297/yeast.gff), I just get
the file.  Perhaps dropbox won't let you just fetch large files
without going through some sort of redirect (which SAMtools probably
can't follow).  I don't know enough about dropbox to know if that's
the case.  When I tested this with dropbox before, it was with a
relatively small BAM file.

Scott


On Sun, Feb 17, 2013 at 11:31 AM, Omar Darwish <[hidden email]> wrote:
> Hi Scott,
>
> Yes, Mega is different from Dropbox, It goes through intermediate page
> before downloading the file. Here is the actual link from drop box:
> https://www.dropbox.com/s/irgefpicin3xpd6/Bowtie_1_1_Aligned.sort.bam
>
> Thanks
>
>
> On Sat, Feb 16, 2013 at 4:13 PM, Scott Cain <[hidden email]> wrote:
>>
>> Hi Omar,
>>
>> I don't know much about mega, so I can't give you much advice there,
>> but in general, I can say that clicking on the links below should
>> result in getting the file requested, without going through an
>> intermediate page (like giving me a window to agree to terms or the
>> like), since GBrowse/SAMtools can't traverse those pages.
>>
>> For dropbox, when I clicked on the bam file link, I got a 403 http
>> error, which is some version of permission denied.  Obviously, the
>> dropbox link needs to be a public link.
>>
>> Scott
>>
>>
>> On Sat, Feb 16, 2013 at 9:14 AM, Omar Darwish <[hidden email]>
>> wrote:
>> > Hi Scott,
>> >
>> > As iPlant offers only limited space, I came back to your suggestion of
>> > storing bam files on public drive (dropbox/mega). It did not work (
>> > Please
>> > see the attached httpd error_log)This is my database stanza:
>> >
>> > [Sample_DB:database]
>> > db_adaptor     = Bio::DB::Sam
>> > db_args        = -fasta
>> > /var/lib/gbrowse2/databases/strawberry/fvesca_v1.1_pseudo.fa
>> >                  -bam
>> > http://dl.dropbox.com/s/uj8by9bcipwr2c4/Sample_DB.sort.bam
>> > -bai http://dl.dropbox.com/s/yx6zon4w8ryohlb/Sample_DB.sort.bam.bai
>> > search options = default
>> >
>> > Another question, as mega website offers 50G free account I thought it
>> > might
>> > be better to go with it. But I found that the file address you can get
>> > have
>> > the file name part encrypted, Will that be a problem?
>> > so the above stanza will look like this:
>> >
>> > [Sample_DB:database]
>> > db_adaptor     = Bio::DB::Sam
>> > db_args        = -fasta
>> > /var/lib/gbrowse2/databases/strawberry/fvesca_v1.1_pseudo.fa
>> >                  -bam
>> > http://mega.co.nz/#!IsAyQBLa!IDnD1Em--hmguYKkekFIghrh7akOl0-Ob5ie6ofg
>> >  -bai
>> > http://mega.co.nz/#!VxJEwJ4Y!JeBUE2TY2OQ_epq4_65yagFWjdgV5OKZDd6QfVY0
>> > search options = default
>> >
>> >
>> > Both cases won't  work! Any suggestions?
>> >
>> > Thank you!
>> >
>> >
>> > On Thu, Nov 15, 2012 at 10:42 AM, Scott Cain <[hidden email]>
>> > wrote:
>> >>
>> >> Just to follow up on Chris' suggestion about http: you can even put
>> >> the bam files on something like a dropbox public drive and use that
>> >> url.  I just tried that with this server:
>> >>
>> >>
>> >>
>> >> http://ec2-54-242-42-142.compute-1.amazonaws.com/cgi-bin/gb2/gbrowse/volvox/?start=21900;stop=31899;ref=ctgA;width=800;version=100;flip=0;grid=on;id=079953d1b81c9dbbcadd73d0eea541ad;l=Reads%1EPairs%1EExampleFeatures
>> >>
>> >> (it won't be around long--it's a AWS micro machine.)  The performance
>> >> is quite good, the only caveat is that dropbox will give you an https
>> >> url for the file, but Bio::DB::Sam doesn't support https urls, so you
>> >> just have to take out the "s".  This is what the database stanza
>> >> looked like in my config:
>> >>
>> >> [samdb:database]
>> >> db_adaptor     = Bio::DB::Sam
>> >> db_args        = -bam http://dl.dropbox.com/u/91697297/volvox.sort.bam
>> >> search options = none
>> >>
>> >> Note that fasta files can't be remote--they have to be indexed and
>> >> local, though in this instance, I didn't provide any fasta and that's
>> >> OK too.
>> >>
>> >> Scott
>> >>
>> >>
>> >> On Wed, Nov 14, 2012 at 1:10 PM, Fields, Christopher J
>> >> <[hidden email]> wrote:
>> >> > I don't think you are limited to /var/ or anywhere else, the key
>> >> > point
>> >> > is that the data has to be accessible to whatever user the server is
>> >> > running
>> >> > under (www-data, apache2, etc).  I have a local instance where I had
>> >> > BAM
>> >> > data on an external RAID and it worked fine with the proper
>> >> > permissions set.
>> >> >
>> >> > Also, Bio::DB::Sam can take an ftp or http location as an argument
>> >> > for
>> >> > '-bam':
>> >> >
>> >> >
>> >> >
>> >> > https://metacpan.org/module/Bio::DB::Sam#Bio::DB::Sam-Constructor-and-basic-accessors
>> >> >
>> >> > chris
>> >> >
>> >> > On Nov 14, 2012, at 10:35 AM, Omar Darwish <[hidden email]>
>> >> >  wrote:
>> >> >
>> >> >> Hello all,
>> >> >>
>> >> >> Is there is any way I can save my BAM databases other than
>> >> >> /var/lib/gbrowse2/databases . As this my file system is full and can
>> >> >> not
>> >> >> move more data there & could not figure out a way to increase teh
>> >> >> space. Any
>> >> >> help is appreciated.
>> >> >>
>> >> >>
>> >> >> Thanks!
>> >> >>
>> >> >>
>> >> >>
>> >> >> ------------------------------------------------------------------------------
>> >> >> Monitor your physical, virtual and cloud infrastructure from a
>> >> >> single
>> >> >> web console. Get in-depth insight into apps, servers, databases,
>> >> >> vmware,
>> >> >> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
>> >> >> Pricing starts from $795 for 25 servers or applications!
>> >> >>
>> >> >>
>> >> >> http://p.sf.net/sfu/zoho_dev2dev_nov_______________________________________________
>> >> >> Gmod-gbrowse mailing list
>> >> >> [hidden email]
>> >> >> https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >>
>> >> ------------------------------------------------------------------------
>> >> Scott Cain, Ph. D.                                   scott at scottcain
>> >> dot net
>> >> GMOD Coordinator (http://gmod.org/)                     <a href="tel:216-392-3087" value="+12163923087">216-392-3087
>> >> Ontario Institute for Cancer Research
>> >
>> >
>> >
>> >
>> > --
>> > Regards,
>> > Omar
>>
>>
>>
>> --
>> ------------------------------------------------------------------------
>> Scott Cain, Ph. D.                                   scott at scottcain
>> dot net
>> GMOD Coordinator (http://gmod.org/)                     <a href="tel:216-392-3087" value="+12163923087">216-392-3087
>> Ontario Institute for Cancer Research
>
>
>
>
> --
> Regards,
> Omar



--
------------------------------------------------------------------------
Scott Cain, Ph. D.                                   scott at scottcain dot net
GMOD Coordinator (http://gmod.org/)                     <a href="tel:216-392-3087" value="+12163923087">216-392-3087
Ontario Institute for Cancer Research



--
Regards,
Omar

------------------------------------------------------------------------------
The Go Parallel Website, sponsored by Intel - in partnership with Geeknet,
is your hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials, tech docs,
whitepapers, evaluation guides, and opinion stories. Check out the most
recent posts - join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Gmod-gbrowse mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
Reply | Threaded
Open this post in threaded view
|

Re: SAM/BAM file location

vkrishna
Hi Omar,

Before using `curl` or `wget` to fetch the file from dropbox, try appending "?dl=1" to the download URL like so:
https://www.dropbox.com/s/irgefpicin3xpd6/Bowtie_1_1_Aligned.sort.bam?dl=1

This should invoke a direct download instead of showing an intermediary page where the user can click "Download"

Hope this helps!
Vivek
________________________________________
From: Omar Darwish [[hidden email]]
Sent: Monday, February 18, 2013 1:41 PM
To: Scott Cain
Cc: Gbrowse (E-mail)
Subject: Re: [Gmod-gbrowse] SAM/BAM file location

Okay, Thanks Scott!


On Mon, Feb 18, 2013 at 10:15 AM, Scott Cain <[hidden email]<mailto:[hidden email]>> wrote:
Hi Omar,

Interesting: if I use curl to fetch your link, I get a small, "you
will be redirected page", but if I fetch a url to a GFF file I've
created (like http://dl.dropbox.com/u/91697297/yeast.gff), I just get
the file.  Perhaps dropbox won't let you just fetch large files
without going through some sort of redirect (which SAMtools probably
can't follow).  I don't know enough about dropbox to know if that's
the case.  When I tested this with dropbox before, it was with a
relatively small BAM file.

Scott


On Sun, Feb 17, 2013 at 11:31 AM, Omar Darwish <[hidden email]<mailto:[hidden email]>> wrote:

> Hi Scott,
>
> Yes, Mega is different from Dropbox, It goes through intermediate page
> before downloading the file. Here is the actual link from drop box:
> https://www.dropbox.com/s/irgefpicin3xpd6/Bowtie_1_1_Aligned.sort.bam
>
> Thanks
>
>
> On Sat, Feb 16, 2013 at 4:13 PM, Scott Cain <[hidden email]<mailto:[hidden email]>> wrote:
>>
>> Hi Omar,
>>
>> I don't know much about mega, so I can't give you much advice there,
>> but in general, I can say that clicking on the links below should
>> result in getting the file requested, without going through an
>> intermediate page (like giving me a window to agree to terms or the
>> like), since GBrowse/SAMtools can't traverse those pages.
>>
>> For dropbox, when I clicked on the bam file link, I got a 403 http
>> error, which is some version of permission denied.  Obviously, the
>> dropbox link needs to be a public link.
>>
>> Scott
>>
>>
>> On Sat, Feb 16, 2013 at 9:14 AM, Omar Darwish <[hidden email]<mailto:[hidden email]>>
>> wrote:
>> > Hi Scott,
>> >
>> > As iPlant offers only limited space, I came back to your suggestion of
>> > storing bam files on public drive (dropbox/mega). It did not work (
>> > Please
>> > see the attached httpd error_log)This is my database stanza:
>> >
>> > [Sample_DB:database]
>> > db_adaptor     = Bio::DB::Sam
>> > db_args        = -fasta
>> > /var/lib/gbrowse2/databases/strawberry/fvesca_v1.1_pseudo.fa
>> >                  -bam
>> > http://dl.dropbox.com/s/uj8by9bcipwr2c4/Sample_DB.sort.bam
>> > -bai http://dl.dropbox.com/s/yx6zon4w8ryohlb/Sample_DB.sort.bam.bai
>> > search options = default
>> >
>> > Another question, as mega website offers 50G free account I thought it
>> > might
>> > be better to go with it. But I found that the file address you can get
>> > have
>> > the file name part encrypted, Will that be a problem?
>> > so the above stanza will look like this:
>> >
>> > [Sample_DB:database]
>> > db_adaptor     = Bio::DB::Sam
>> > db_args        = -fasta
>> > /var/lib/gbrowse2/databases/strawberry/fvesca_v1.1_pseudo.fa
>> >                  -bam
>> > http://mega.co.nz/#!IsAyQBLa!IDnD1Em--hmguYKkekFIghrh7akOl0-Ob5ie6ofg
>> >  -bai
>> > http://mega.co.nz/#!VxJEwJ4Y!JeBUE2TY2OQ_epq4_65yagFWjdgV5OKZDd6QfVY0
>> > search options = default
>> >
>> >
>> > Both cases won't  work! Any suggestions?
>> >
>> > Thank you!
>> >
>> >
>> > On Thu, Nov 15, 2012 at 10:42 AM, Scott Cain <[hidden email]<mailto:[hidden email]>>
>> > wrote:
>> >>
>> >> Just to follow up on Chris' suggestion about http: you can even put
>> >> the bam files on something like a dropbox public drive and use that
>> >> url.  I just tried that with this server:
>> >>
>> >>
>> >>
>> >> http://ec2-54-242-42-142.compute-1.amazonaws.com/cgi-bin/gb2/gbrowse/volvox/?start=21900;stop=31899;ref=ctgA;width=800;version=100;flip=0;grid=on;id=079953d1b81c9dbbcadd73d0eea541ad;l=Reads%1EPairs%1EExampleFeatures
>> >>
>> >> (it won't be around long--it's a AWS micro machine.)  The performance
>> >> is quite good, the only caveat is that dropbox will give you an https
>> >> url for the file, but Bio::DB::Sam doesn't support https urls, so you
>> >> just have to take out the "s".  This is what the database stanza
>> >> looked like in my config:
>> >>
>> >> [samdb:database]
>> >> db_adaptor     = Bio::DB::Sam
>> >> db_args        = -bam http://dl.dropbox.com/u/91697297/volvox.sort.bam
>> >> search options = none
>> >>
>> >> Note that fasta files can't be remote--they have to be indexed and
>> >> local, though in this instance, I didn't provide any fasta and that's
>> >> OK too.
>> >>
>> >> Scott
>> >>
>> >>
>> >> On Wed, Nov 14, 2012 at 1:10 PM, Fields, Christopher J
>> >> <[hidden email]<mailto:[hidden email]>> wrote:
>> >> > I don't think you are limited to /var/ or anywhere else, the key
>> >> > point
>> >> > is that the data has to be accessible to whatever user the server is
>> >> > running
>> >> > under (www-data, apache2, etc).  I have a local instance where I had
>> >> > BAM
>> >> > data on an external RAID and it worked fine with the proper
>> >> > permissions set.
>> >> >
>> >> > Also, Bio::DB::Sam can take an ftp or http location as an argument
>> >> > for
>> >> > '-bam':
>> >> >
>> >> >
>> >> >
>> >> > https://metacpan.org/module/Bio::DB::Sam#Bio::DB::Sam-Constructor-and-basic-accessors
>> >> >
>> >> > chris
>> >> >
>> >> > On Nov 14, 2012, at 10:35 AM, Omar Darwish <[hidden email]<mailto:[hidden email]>>
>> >> >  wrote:
>> >> >
>> >> >> Hello all,
>> >> >>
>> >> >> Is there is any way I can save my BAM databases other than
>> >> >> /var/lib/gbrowse2/databases . As this my file system is full and can
>> >> >> not
>> >> >> move more data there & could not figure out a way to increase teh
>> >> >> space. Any
>> >> >> help is appreciated.
>> >> >>
>> >> >>
>> >> >> Thanks!
>> >> >>
>> >> >>
>> >> >>
>> >> >> ------------------------------------------------------------------------------
>> >> >> Monitor your physical, virtual and cloud infrastructure from a
>> >> >> single
>> >> >> web console. Get in-depth insight into apps, servers, databases,
>> >> >> vmware,
>> >> >> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
>> >> >> Pricing starts from $795 for 25 servers or applications!
>> >> >>
>> >> >>
>> >> >> http://p.sf.net/sfu/zoho_dev2dev_nov_______________________________________________
>> >> >> Gmod-gbrowse mailing list
>> >> >> [hidden email]<mailto:[hidden email]>
>> >> >> https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >>
>> >> ------------------------------------------------------------------------
>> >> Scott Cain, Ph. D.                                   scott at scottcain
>> >> dot net
>> >> GMOD Coordinator (http://gmod.org/)                     216-392-3087<tel:216-392-3087>
>> >> Ontario Institute for Cancer Research
>> >
>> >
>> >
>> >
>> > --
>> > Regards,
>> > Omar
>>
>>
>>
>> --
>> ------------------------------------------------------------------------
>> Scott Cain, Ph. D.                                   scott at scottcain
>> dot net
>> GMOD Coordinator (http://gmod.org/)                     216-392-3087<tel:216-392-3087>
>> Ontario Institute for Cancer Research
>
>
>
>
> --
> Regards,
> Omar



--
------------------------------------------------------------------------
Scott Cain, Ph. D.                                   scott at scottcain dot net
GMOD Coordinator (http://gmod.org/)                     216-392-3087<tel:216-392-3087>
Ontario Institute for Cancer Research



--
Regards,
Omar

------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_d2d_feb
_______________________________________________
Gmod-gbrowse mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
Reply | Threaded
Open this post in threaded view
|

Re: SAM/BAM file location

Omar Darwish
Hi Vivek,

Thanks for the suggestion, I tried it but it does seem to work, I am seeing the below errors in my httpd error_log.

[client 127.0.0.1] connect: Permission denied
[client 127.0.0.1] [khttp_connect_file] fail to open file (HTTP code: 0).
[error] [client 127.0.0.1] Could not open database: http://www.dropbox.com/s/8d7v1509yuqa5m4/7_8_Anther_A_filtered_Aligned.sort.bam?dl=1 open: Bad file descriptor at /usr/local/lib64/perl5/Bio/Graphics/Browser2/DataBase.pm line 44


Regards,
Omar


On Mon, Feb 18, 2013 at 8:57 PM, Krishnakumar, Vivek <[hidden email]> wrote:
Hi Omar,

Before using `curl` or `wget` to fetch the file from dropbox, try appending "?dl=1" to the download URL like so:
https://www.dropbox.com/s/irgefpicin3xpd6/Bowtie_1_1_Aligned.sort.bam?dl=1

This should invoke a direct download instead of showing an intermediary page where the user can click "Download"

Hope this helps!
Vivek
________________________________________
From: Omar Darwish [[hidden email]]
Sent: Monday, February 18, 2013 1:41 PM
To: Scott Cain
Cc: Gbrowse (E-mail)
Subject: Re: [Gmod-gbrowse] SAM/BAM file location

Okay, Thanks Scott!


On Mon, Feb 18, 2013 at 10:15 AM, Scott Cain <[hidden email]<mailto:[hidden email]>> wrote:
Hi Omar,

Interesting: if I use curl to fetch your link, I get a small, "you
will be redirected page", but if I fetch a url to a GFF file I've
created (like http://dl.dropbox.com/u/91697297/yeast.gff), I just get
the file.  Perhaps dropbox won't let you just fetch large files
without going through some sort of redirect (which SAMtools probably
can't follow).  I don't know enough about dropbox to know if that's
the case.  When I tested this with dropbox before, it was with a
relatively small BAM file.

Scott


On Sun, Feb 17, 2013 at 11:31 AM, Omar Darwish <[hidden email]<mailto:[hidden email]>> wrote:
> Hi Scott,
>
> Yes, Mega is different from Dropbox, It goes through intermediate page
> before downloading the file. Here is the actual link from drop box:
> https://www.dropbox.com/s/irgefpicin3xpd6/Bowtie_1_1_Aligned.sort.bam
>
> Thanks
>
>
> On Sat, Feb 16, 2013 at 4:13 PM, Scott Cain <[hidden email]<mailto:[hidden email]>> wrote:
>>
>> Hi Omar,
>>
>> I don't know much about mega, so I can't give you much advice there,
>> but in general, I can say that clicking on the links below should
>> result in getting the file requested, without going through an
>> intermediate page (like giving me a window to agree to terms or the
>> like), since GBrowse/SAMtools can't traverse those pages.
>>
>> For dropbox, when I clicked on the bam file link, I got a 403 http
>> error, which is some version of permission denied.  Obviously, the
>> dropbox link needs to be a public link.
>>
>> Scott
>>
>>
>> On Sat, Feb 16, 2013 at 9:14 AM, Omar Darwish <[hidden email]<mailto:[hidden email]>>
>> wrote:
>> > Hi Scott,
>> >
>> > As iPlant offers only limited space, I came back to your suggestion of
>> > storing bam files on public drive (dropbox/mega). It did not work (
>> > Please
>> > see the attached httpd error_log)This is my database stanza:
>> >
>> > [Sample_DB:database]
>> > db_adaptor     = Bio::DB::Sam
>> > db_args        = -fasta
>> > /var/lib/gbrowse2/databases/strawberry/fvesca_v1.1_pseudo.fa
>> >                  -bam
>> > http://dl.dropbox.com/s/uj8by9bcipwr2c4/Sample_DB.sort.bam
>> > -bai http://dl.dropbox.com/s/yx6zon4w8ryohlb/Sample_DB.sort.bam.bai
>> > search options = default
>> >
>> > Another question, as mega website offers 50G free account I thought it
>> > might
>> > be better to go with it. But I found that the file address you can get
>> > have
>> > the file name part encrypted, Will that be a problem?
>> > so the above stanza will look like this:
>> >
>> > [Sample_DB:database]
>> > db_adaptor     = Bio::DB::Sam
>> > db_args        = -fasta
>> > /var/lib/gbrowse2/databases/strawberry/fvesca_v1.1_pseudo.fa
>> >                  -bam
>> > http://mega.co.nz/#!IsAyQBLa!IDnD1Em--hmguYKkekFIghrh7akOl0-Ob5ie6ofg
>> >  -bai
>> > http://mega.co.nz/#!VxJEwJ4Y!JeBUE2TY2OQ_epq4_65yagFWjdgV5OKZDd6QfVY0
>> > search options = default
>> >
>> >
>> > Both cases won't  work! Any suggestions?
>> >
>> > Thank you!
>> >
>> >
>> > On Thu, Nov 15, 2012 at 10:42 AM, Scott Cain <[hidden email]<mailto:[hidden email]>>
>> > wrote:
>> >>
>> >> Just to follow up on Chris' suggestion about http: you can even put
>> >> the bam files on something like a dropbox public drive and use that
>> >> url.  I just tried that with this server:
>> >>
>> >>
>> >>
>> >> http://ec2-54-242-42-142.compute-1.amazonaws.com/cgi-bin/gb2/gbrowse/volvox/?start=21900;stop=31899;ref=ctgA;width=800;version=100;flip=0;grid=on;id=079953d1b81c9dbbcadd73d0eea541ad;l=Reads%1EPairs%1EExampleFeatures
>> >>
>> >> (it won't be around long--it's a AWS micro machine.)  The performance
>> >> is quite good, the only caveat is that dropbox will give you an https
>> >> url for the file, but Bio::DB::Sam doesn't support https urls, so you
>> >> just have to take out the "s".  This is what the database stanza
>> >> looked like in my config:
>> >>
>> >> [samdb:database]
>> >> db_adaptor     = Bio::DB::Sam
>> >> db_args        = -bam http://dl.dropbox.com/u/91697297/volvox.sort.bam
>> >> search options = none
>> >>
>> >> Note that fasta files can't be remote--they have to be indexed and
>> >> local, though in this instance, I didn't provide any fasta and that's
>> >> OK too.
>> >>
>> >> Scott
>> >>
>> >>
>> >> On Wed, Nov 14, 2012 at 1:10 PM, Fields, Christopher J
>> >> <[hidden email]<mailto:[hidden email]>> wrote:
>> >> > I don't think you are limited to /var/ or anywhere else, the key
>> >> > point
>> >> > is that the data has to be accessible to whatever user the server is
>> >> > running
>> >> > under (www-data, apache2, etc).  I have a local instance where I had
>> >> > BAM
>> >> > data on an external RAID and it worked fine with the proper
>> >> > permissions set.
>> >> >
>> >> > Also, Bio::DB::Sam can take an ftp or http location as an argument
>> >> > for
>> >> > '-bam':
>> >> >
>> >> >
>> >> >
>> >> > https://metacpan.org/module/Bio::DB::Sam#Bio::DB::Sam-Constructor-and-basic-accessors
>> >> >
>> >> > chris
>> >> >
>> >> > On Nov 14, 2012, at 10:35 AM, Omar Darwish <[hidden email]<mailto:[hidden email]>>
>> >> >  wrote:
>> >> >
>> >> >> Hello all,
>> >> >>
>> >> >> Is there is any way I can save my BAM databases other than
>> >> >> /var/lib/gbrowse2/databases . As this my file system is full and can
>> >> >> not
>> >> >> move more data there & could not figure out a way to increase teh
>> >> >> space. Any
>> >> >> help is appreciated.
>> >> >>
>> >> >>
>> >> >> Thanks!
>> >> >>
>> >> >>
>> >> >>
>> >> >> ------------------------------------------------------------------------------
>> >> >> Monitor your physical, virtual and cloud infrastructure from a
>> >> >> single
>> >> >> web console. Get in-depth insight into apps, servers, databases,
>> >> >> vmware,
>> >> >> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
>> >> >> Pricing starts from $795 for 25 servers or applications!
>> >> >>
>> >> >>
>> >> >> http://p.sf.net/sfu/zoho_dev2dev_nov_______________________________________________
>> >> >> Gmod-gbrowse mailing list
>> >> >> [hidden email]<mailto:[hidden email]>
>> >> >> https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >>
>> >> ------------------------------------------------------------------------
>> >> Scott Cain, Ph. D.                                   scott at scottcain
>> >> dot net
>> >> GMOD Coordinator (http://gmod.org/)                     <a href="tel:216-392-3087" value="+12163923087">216-392-3087<tel:<a href="tel:216-392-3087" value="+12163923087">216-392-3087>
>> >> Ontario Institute for Cancer Research
>> >
>> >
>> >
>> >
>> > --
>> > Regards,
>> > Omar
>>
>>
>>
>> --
>> ------------------------------------------------------------------------
>> Scott Cain, Ph. D.                                   scott at scottcain
>> dot net
>> GMOD Coordinator (http://gmod.org/)                     <a href="tel:216-392-3087" value="+12163923087">216-392-3087<tel:<a href="tel:216-392-3087" value="+12163923087">216-392-3087>
>> Ontario Institute for Cancer Research
>
>
>
>
> --
> Regards,
> Omar



--
------------------------------------------------------------------------
Scott Cain, Ph. D.                                   scott at scottcain dot net
GMOD Coordinator (http://gmod.org/)                     <a href="tel:216-392-3087" value="+12163923087">216-392-3087<tel:<a href="tel:216-392-3087" value="+12163923087">216-392-3087>
Ontario Institute for Cancer Research



--
Regards,
Omar



--
Regards,
Omar

------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_d2d_feb
_______________________________________________
Gmod-gbrowse mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
Reply | Threaded
Open this post in threaded view
|

Re: SAM/BAM file location

Adam Witney

For me...

wget -c
https://www.dropbox.com/s/irgefpicin3xpd6/Bowtie_1_1_Aligned.sort.bam?dl=1

this brings back an html page, not the bam file. you need to have the
bam file in a location that can be accessed directly by a single URL

Adam


On 20/02/2013 15:30, Omar Darwish wrote:

> Hi Vivek,
>
> Thanks for the suggestion, I tried it but it does seem to work, I am
> seeing the below errors in my httpd error_log.
>
> [client 127.0.0.1] connect: Permission denied
> [client 127.0.0.1] [khttp_connect_file] fail to open file (HTTP code: 0).
> [error] [client 127.0.0.1] Could not open database:
> http://www.dropbox.com/s/8d7v1509yuqa5m4/7_8_Anther_A_filtered_Aligned.sort.bam?dl=1
> open: Bad file descriptor at
> /usr/local/lib64/perl5/Bio/Graphics/Browser2/DataBase.pm line 44
>
>
> Regards,
> Omar
>
>
> On Mon, Feb 18, 2013 at 8:57 PM, Krishnakumar, Vivek <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Hi Omar,
>
>     Before using `curl` or `wget` to fetch the file from dropbox, try
>     appending "?dl=1" to the download URL like so:
>     https://www.dropbox.com/s/irgefpicin3xpd6/Bowtie_1_1_Aligned.sort.bam?dl=1
>
>     This should invoke a direct download instead of showing an
>     intermediary page where the user can click "Download"
>
>     Hope this helps!
>     Vivek
>     ________________________________________
>     From: Omar Darwish [[hidden email] <mailto:[hidden email]>]
>     Sent: Monday, February 18, 2013 1:41 PM
>     To: Scott Cain
>     Cc: Gbrowse (E-mail)
>     Subject: Re: [Gmod-gbrowse] SAM/BAM file location
>
>     Okay, Thanks Scott!
>
>
>     On Mon, Feb 18, 2013 at 10:15 AM, Scott Cain <[hidden email]
>     <mailto:[hidden email]><mailto:[hidden email]
>     <mailto:[hidden email]>>> wrote:
>     Hi Omar,
>
>     Interesting: if I use curl to fetch your link, I get a small, "you
>     will be redirected page", but if I fetch a url to a GFF file I've
>     created (like http://dl.dropbox.com/u/91697297/yeast.gff), I just get
>     the file.  Perhaps dropbox won't let you just fetch large files
>     without going through some sort of redirect (which SAMtools probably
>     can't follow).  I don't know enough about dropbox to know if that's
>     the case.  When I tested this with dropbox before, it was with a
>     relatively small BAM file.
>
>     Scott
>
>
>     On Sun, Feb 17, 2013 at 11:31 AM, Omar Darwish <[hidden email]
>     <mailto:[hidden email]><mailto:[hidden email]
>     <mailto:[hidden email]>>> wrote:
>     > Hi Scott,
>     >
>     > Yes, Mega is different from Dropbox, It goes through intermediate page
>     > before downloading the file. Here is the actual link from drop box:
>     > https://www.dropbox.com/s/irgefpicin3xpd6/Bowtie_1_1_Aligned.sort.bam
>     >
>     > Thanks
>     >
>     >
>     > On Sat, Feb 16, 2013 at 4:13 PM, Scott Cain <[hidden email]
>     <mailto:[hidden email]><mailto:[hidden email]
>     <mailto:[hidden email]>>> wrote:
>     >>
>     >> Hi Omar,
>     >>
>     >> I don't know much about mega, so I can't give you much advice there,
>     >> but in general, I can say that clicking on the links below should
>     >> result in getting the file requested, without going through an
>     >> intermediate page (like giving me a window to agree to terms or the
>     >> like), since GBrowse/SAMtools can't traverse those pages.
>     >>
>     >> For dropbox, when I clicked on the bam file link, I got a 403 http
>     >> error, which is some version of permission denied.  Obviously, the
>     >> dropbox link needs to be a public link.
>     >>
>     >> Scott
>     >>
>     >>
>     >> On Sat, Feb 16, 2013 at 9:14 AM, Omar Darwish
>     <[hidden email]
>     <mailto:[hidden email]><mailto:[hidden email]
>     <mailto:[hidden email]>>>
>     >> wrote:
>     >> > Hi Scott,
>     >> >
>     >> > As iPlant offers only limited space, I came back to your
>     suggestion of
>     >> > storing bam files on public drive (dropbox/mega). It did not work (
>     >> > Please
>     >> > see the attached httpd error_log)This is my database stanza:
>     >> >
>     >> > [Sample_DB:database]
>     >> > db_adaptor     = Bio::DB::Sam
>     >> > db_args        = -fasta
>     >> > /var/lib/gbrowse2/databases/strawberry/fvesca_v1.1_pseudo.fa
>     >> >                  -bam
>     >> > http://dl.dropbox.com/s/uj8by9bcipwr2c4/Sample_DB.sort.bam
>     >> > -bai http://dl.dropbox.com/s/yx6zon4w8ryohlb/Sample_DB.sort.bam.bai
>     >> > search options = default
>     >> >
>     >> > Another question, as mega website offers 50G free account I
>     thought it
>     >> > might
>     >> > be better to go with it. But I found that the file address you
>     can get
>     >> > have
>     >> > the file name part encrypted, Will that be a problem?
>     >> > so the above stanza will look like this:
>     >> >
>     >> > [Sample_DB:database]
>     >> > db_adaptor     = Bio::DB::Sam
>     >> > db_args        = -fasta
>     >> > /var/lib/gbrowse2/databases/strawberry/fvesca_v1.1_pseudo.fa
>     >> >                  -bam
>     >> >
>     http://mega.co.nz/#!IsAyQBLa!IDnD1Em--hmguYKkekFIghrh7akOl0-Ob5ie6ofg
>     >> >  -bai
>     >> >
>     http://mega.co.nz/#!VxJEwJ4Y!JeBUE2TY2OQ_epq4_65yagFWjdgV5OKZDd6QfVY0
>     >> > search options = default
>     >> >
>     >> >
>     >> > Both cases won't  work! Any suggestions?
>     >> >
>     >> > Thank you!
>     >> >
>     >> >
>     >> > On Thu, Nov 15, 2012 at 10:42 AM, Scott Cain
>     <[hidden email]
>     <mailto:[hidden email]><mailto:[hidden email]
>     <mailto:[hidden email]>>>
>     >> > wrote:
>     >> >>
>     >> >> Just to follow up on Chris' suggestion about http: you can
>     even put
>     >> >> the bam files on something like a dropbox public drive and use
>     that
>     >> >> url.  I just tried that with this server:
>     >> >>
>     >> >>
>     >> >>
>     >> >>
>     http://ec2-54-242-42-142.compute-1.amazonaws.com/cgi-bin/gb2/gbrowse/volvox/?start=21900;stop=31899;ref=ctgA;width=800;version=100;flip=0;grid=on;id=079953d1b81c9dbbcadd73d0eea541ad;l=Reads%1EPairs%1EExampleFeatures
>     >> >>
>     >> >> (it won't be around long--it's a AWS micro machine.)  The
>     performance
>     >> >> is quite good, the only caveat is that dropbox will give you
>     an https
>     >> >> url for the file, but Bio::DB::Sam doesn't support https urls,
>     so you
>     >> >> just have to take out the "s".  This is what the database stanza
>     >> >> looked like in my config:
>     >> >>
>     >> >> [samdb:database]
>     >> >> db_adaptor     = Bio::DB::Sam
>     >> >> db_args        = -bam
>     http://dl.dropbox.com/u/91697297/volvox.sort.bam
>     >> >> search options = none
>     >> >>
>     >> >> Note that fasta files can't be remote--they have to be indexed and
>     >> >> local, though in this instance, I didn't provide any fasta and
>     that's
>     >> >> OK too.
>     >> >>
>     >> >> Scott
>     >> >>
>     >> >>
>     >> >> On Wed, Nov 14, 2012 at 1:10 PM, Fields, Christopher J
>     >> >> <[hidden email]
>     <mailto:[hidden email]><mailto:[hidden email]
>     <mailto:[hidden email]>>> wrote:
>     >> >> > I don't think you are limited to /var/ or anywhere else, the key
>     >> >> > point
>     >> >> > is that the data has to be accessible to whatever user the
>     server is
>     >> >> > running
>     >> >> > under (www-data, apache2, etc).  I have a local instance
>     where I had
>     >> >> > BAM
>     >> >> > data on an external RAID and it worked fine with the proper
>     >> >> > permissions set.
>     >> >> >
>     >> >> > Also, Bio::DB::Sam can take an ftp or http location as an
>     argument
>     >> >> > for
>     >> >> > '-bam':
>     >> >> >
>     >> >> >
>     >> >> >
>     >> >> >
>     https://metacpan.org/module/Bio::DB::Sam#Bio::DB::Sam-Constructor-and-basic-accessors
>     >> >> >
>     >> >> > chris
>     >> >> >
>     >> >> > On Nov 14, 2012, at 10:35 AM, Omar Darwish
>     <[hidden email]
>     <mailto:[hidden email]><mailto:[hidden email]
>     <mailto:[hidden email]>>>
>     >> >> >  wrote:
>     >> >> >
>     >> >> >> Hello all,
>     >> >> >>
>     >> >> >> Is there is any way I can save my BAM databases other than
>     >> >> >> /var/lib/gbrowse2/databases . As this my file system is
>     full and can
>     >> >> >> not
>     >> >> >> move more data there & could not figure out a way to
>     increase teh
>     >> >> >> space. Any
>     >> >> >> help is appreciated.
>     >> >> >>
>     >> >> >>
>     >> >> >> Thanks!
>     >> >> >>
>     >> >> >>
>     >> >> >>
>     >> >> >>
>     ------------------------------------------------------------------------------
>     >> >> >> Monitor your physical, virtual and cloud infrastructure from a
>     >> >> >> single
>     >> >> >> web console. Get in-depth insight into apps, servers,
>     databases,
>     >> >> >> vmware,
>     >> >> >> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
>     >> >> >> Pricing starts from $795 for 25 servers or applications!
>     >> >> >>
>     >> >> >>
>     >> >> >>
>     http://p.sf.net/sfu/zoho_dev2dev_nov_______________________________________________
>     >> >> >> Gmod-gbrowse mailing list
>     >> >> >> [hidden email]
>     <mailto:[hidden email]><mailto:[hidden email]
>     <mailto:[hidden email]>>
>     >> >> >> https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
>     >> >> >
>     >> >>
>     >> >>
>     >> >>
>     >> >> --
>     >> >>
>     >> >>
>     ------------------------------------------------------------------------
>     >> >> Scott Cain, Ph. D.                                   scott at
>     scottcain
>     >> >> dot net
>     >> >> GMOD Coordinator (http://gmod.org/)                    
>     216-392-3087 <tel:216-392-3087><tel:216-392-3087 <tel:216-392-3087>>
>     >> >> Ontario Institute for Cancer Research
>     >> >
>     >> >
>     >> >
>     >> >
>     >> > --
>     >> > Regards,
>     >> > Omar
>     >>
>     >>
>     >>
>     >> --
>     >>
>     ------------------------------------------------------------------------
>     >> Scott Cain, Ph. D.                                   scott at
>     scottcain
>     >> dot net
>     >> GMOD Coordinator (http://gmod.org/)                    
>     216-392-3087 <tel:216-392-3087><tel:216-392-3087 <tel:216-392-3087>>
>     >> Ontario Institute for Cancer Research
>     >
>     >
>     >
>     >
>     > --
>     > Regards,
>     > Omar
>
>
>
>     --
>     ------------------------------------------------------------------------
>     Scott Cain, Ph. D.                                   scott at
>     scottcain dot net
>     GMOD Coordinator (http://gmod.org/)                     216-392-3087
>     <tel:216-392-3087><tel:216-392-3087 <tel:216-392-3087>>
>     Ontario Institute for Cancer Research
>
>
>
>     --
>     Regards,
>     Omar
>
>
>
>
> --
> Regards,
> Omar
>
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_d2d_feb
>
>
>
> _______________________________________________
> Gmod-gbrowse mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
>

------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_d2d_feb
_______________________________________________
Gmod-gbrowse mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse