Flagstat on BAM files

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Flagstat on BAM files

ssassi
Hello,
I tried to use NGS: SAM Tools ->flagstat on a BAM files for basic stats, but I got results like you see below. It doesn't seem to be working. Any suggestions?

26584869 in total
0 QC failure
0 duplicates
26584869 mapped (100.00%)
0 paired in sequencing
0 read1
0 read2
0 properly paired (-nan%)
0 with itself and mate mapped
0 singletons (-nan%)
0 with mate mapped to a different chr
0 with mate mapped to a different chr (mapQ>=5)
Thanks
Slim


The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.

___________________________________________________________
The Galaxy User list should be used for the discussion of
Galaxy analysis and other features on the public server
at usegalaxy.org.  Please keep all replies on the list by
using "reply all" in your mail client.  For discussion of
local Galaxy instances and the Galaxy source code, please
use the Galaxy Development list:

  http://lists.bx.psu.edu/listinfo/galaxy-dev

To manage your subscriptions to this and other Galaxy lists,
please use the interface at:

  http://lists.bx.psu.edu/
Reply | Threaded
Open this post in threaded view
|

Re: Flagstat on BAM files

Sean Davis
Hi, Slim.

My guess is that you used an aligner that outputs only aligned reads
(tophat, for example) and that the input was single-ended.  If that is
the case, then what you see below is exactly as expected.  If not,
then you might need to be more specific about how you generated the
BAM file.

Sean

On Tue, Apr 5, 2011 at 12:31 PM, Slim Sassi <[hidden email]> wrote:

> Hello,
> I tried to use NGS: SAM Tools ->flagstat on a BAM files for basic stats, but
> I got results like you see below. It doesn't seem to be working. Any
> suggestions?
> 26584869 in total
>
> 0 QC failure
> 0 duplicates
> 26584869 mapped (100.00%)
> 0 paired in sequencing
> 0 read1
> 0 read2
> 0 properly paired (-nan%)
> 0 with itself and mate mapped
> 0 singletons (-nan%)
> 0 with mate mapped to a different chr
> 0 with mate mapped to a different chr (mapQ>=5)
>
> Thanks
> Slim
>
> The information in this e-mail is intended only for the person to whom it is
> addressed. If you believe this e-mail was sent to you in error and the
> e-mail
> contains patient information, please contact the Partners Compliance
> HelpLine at
> http://www.partners.org/complianceline . If the e-mail was sent to you in
> error
> but does not contain patient information, please contact the sender and
> properly
> dispose of the e-mail.
___________________________________________________________
The Galaxy User list should be used for the discussion of
Galaxy analysis and other features on the public server
at usegalaxy.org.  Please keep all replies on the list by
using "reply all" in your mail client.  For discussion of
local Galaxy instances and the Galaxy source code, please
use the Galaxy Development list:

  http://lists.bx.psu.edu/listinfo/galaxy-dev

To manage your subscriptions to this and other Galaxy lists,
please use the interface at:

  http://lists.bx.psu.edu/
Reply | Threaded
Open this post in threaded view
|

Re: Flagstat on BAM files

ssassi
Sean,
You are correct,  I did use tophat. Can you or anyone suggest a program for BAM/SAM stats where the alignment was done with tophat

Thanks
Slim
On Apr 5, 2011, at 12:51 PM, Sean Davis wrote:

> Hi, Slim.
>
> My guess is that you used an aligner that outputs only aligned reads
> (tophat, for example) and that the input was single-ended.  If that is
> the case, then what you see below is exactly as expected.  If not,
> then you might need to be more specific about how you generated the
> BAM file.
>
> Sean
>
> On Tue, Apr 5, 2011 at 12:31 PM, Slim Sassi <[hidden email]> wrote:
>> Hello,
>> I tried to use NGS: SAM Tools ->flagstat on a BAM files for basic stats, but
>> I got results like you see below. It doesn't seem to be working. Any
>> suggestions?
>> 26584869 in total
>>
>> 0 QC failure
>> 0 duplicates
>> 26584869 mapped (100.00%)
>> 0 paired in sequencing
>> 0 read1
>> 0 read2
>> 0 properly paired (-nan%)
>> 0 with itself and mate mapped
>> 0 singletons (-nan%)
>> 0 with mate mapped to a different chr
>> 0 with mate mapped to a different chr (mapQ>=5)
>>
>> Thanks
>> Slim
>>
>> The information in this e-mail is intended only for the person to whom it is
>> addressed. If you believe this e-mail was sent to you in error and the
>> e-mail
>> contains patient information, please contact the Partners Compliance
>> HelpLine at
>> http://www.partners.org/complianceline . If the e-mail was sent to you in
>> error
>> but does not contain patient information, please contact the sender and
>> properly
>> dispose of the e-mail.


___________________________________________________________
The Galaxy User list should be used for the discussion of
Galaxy analysis and other features on the public server
at usegalaxy.org.  Please keep all replies on the list by
using "reply all" in your mail client.  For discussion of
local Galaxy instances and the Galaxy source code, please
use the Galaxy Development list:

  http://lists.bx.psu.edu/listinfo/galaxy-dev

To manage your subscriptions to this and other Galaxy lists,
please use the interface at:

  http://lists.bx.psu.edu/
Reply | Threaded
Open this post in threaded view
|

Re: Flagstat on BAM files

Sean Davis


On Apr 5, 2011 1:05 PM, "Slim Sassi" <[hidden email]> wrote:
>
> Sean,
> You are correct,  I did use tophat. Can you or anyone suggest a program for BAM/SAM stats where the alignment was done with tophat
>

Slim,

What stats do you want to capture?  The output you gave for flagstats is correct for single-end tophat alignments.  All reads are aligned, none are paired, none are marked as duplicates.

Sean
 
> Thanks
> Slim
> On Apr 5, 2011, at 12:51 PM, Sean Davis wrote:
>
> > Hi, Slim.
> >
> > My guess is that you used an aligner that outputs only aligned reads
> > (tophat, for example) and that the input was single-ended.  If that is
> > the case, then what you see below is exactly as expected.  If not,
> > then you might need to be more specific about how you generated the
> > BAM file.
> >
> > Sean
> >
> > On Tue, Apr 5, 2011 at 12:31 PM, Slim Sassi <[hidden email]> wrote:
> >> Hello,
> >> I tried to use NGS: SAM Tools ->flagstat on a BAM files for basic stats, but
> >> I got results like you see below. It doesn't seem to be working. Any
> >> suggestions?
> >> 26584869 in total
> >>
> >> 0 QC failure
> >> 0 duplicates
> >> 26584869 mapped (100.00%)
> >> 0 paired in sequencing
> >> 0 read1
> >> 0 read2
> >> 0 properly paired (-nan%)
> >> 0 with itself and mate mapped
> >> 0 singletons (-nan%)
> >> 0 with mate mapped to a different chr
> >> 0 with mate mapped to a different chr (mapQ>=5)
> >>
> >> Thanks
> >> Slim
> >>
> >> The information in this e-mail is intended only for the person to whom it is
> >> addressed. If you believe this e-mail was sent to you in error and the
> >> e-mail
> >> contains patient information, please contact the Partners Compliance
> >> HelpLine at
> >> http://www.partners.org/complianceline . If the e-mail was sent to you in
> >> error
> >> but does not contain patient information, please contact the sender and
> >> properly
> >> dispose of the e-mail.
>


___________________________________________________________
The Galaxy User list should be used for the discussion of
Galaxy analysis and other features on the public server
at usegalaxy.org.  Please keep all replies on the list by
using "reply all" in your mail client.  For discussion of
local Galaxy instances and the Galaxy source code, please
use the Galaxy Development list:

  http://lists.bx.psu.edu/listinfo/galaxy-dev

To manage your subscriptions to this and other Galaxy lists,
please use the interface at:

  http://lists.bx.psu.edu/
Reply | Threaded
Open this post in threaded view
|

Stats on BAM files

ssassi
Sean,

I only wanted to start collecting stats with flagstats but knew that I needed something else to get everthing needed. 
I would like to know:
% that didn't pass QC
% mapped
% reads in exons/introns/intergenic regions

and then, knowing that this is more complicated, I wanted to measure bias within transcripts (for example 3' versus 5'). Of course I am assuming that there is a consistent bias.


Thanks
Slim



On Apr 5, 2011, at 1:50 PM, Sean Davis wrote:


On Apr 5, 2011 1:05 PM, "Slim Sassi" <[hidden email]> wrote:
>
> Sean,
> You are correct,  I did use tophat. Can you or anyone suggest a program for BAM/SAM stats where the alignment was done with tophat
>

Slim,

What stats do you want to capture?  The output you gave for flagstats is correct for single-end tophat alignments.  All reads are aligned, none are paired, none are marked as duplicates.

Sean
 
> Thanks
> Slim
> On Apr 5, 2011, at 12:51 PM, Sean Davis wrote:
>
> > Hi, Slim.
> >
> > My guess is that you used an aligner that outputs only aligned reads
> > (tophat, for example) and that the input was single-ended.  If that is
> > the case, then what you see below is exactly as expected.  If not,
> > then you might need to be more specific about how you generated the
> > BAM file.
> >
> > Sean
> >
> > On Tue, Apr 5, 2011 at 12:31 PM, Slim Sassi <[hidden email]> wrote:
> >> Hello,
> >> I tried to use NGS: SAM Tools ->flagstat on a BAM files for basic stats, but
> >> I got results like you see below. It doesn't seem to be working. Any
> >> suggestions?
> >> 26584869 in total
> >>
> >> 0 QC failure
> >> 0 duplicates
> >> 26584869 mapped (100.00%)
> >> 0 paired in sequencing
> >> 0 read1
> >> 0 read2
> >> 0 properly paired (-nan%)
> >> 0 with itself and mate mapped
> >> 0 singletons (-nan%)
> >> 0 with mate mapped to a different chr
> >> 0 with mate mapped to a different chr (mapQ>=5)
> >>
> >> Thanks
> >> Slim
> >>
> >> The information in this e-mail is intended only for the person to whom it is
> >> addressed. If you believe this e-mail was sent to you in error and the
> >> e-mail
> >> contains patient information, please contact the Partners Compliance
> >> HelpLine at
> >> http://www.partners.org/complianceline . If the e-mail was sent to you in
> >> error
> >> but does not contain patient information, please contact the sender and
> >> properly
> >> dispose of the e-mail.
>



___________________________________________________________
The Galaxy User list should be used for the discussion of
Galaxy analysis and other features on the public server
at usegalaxy.org.  Please keep all replies on the list by
using "reply all" in your mail client.  For discussion of
local Galaxy instances and the Galaxy source code, please
use the Galaxy Development list:

  http://lists.bx.psu.edu/listinfo/galaxy-dev

To manage your subscriptions to this and other Galaxy lists,
please use the interface at:

  http://lists.bx.psu.edu/
Reply | Threaded
Open this post in threaded view
|

Re: Stats on BAM files

Sean Davis
On Tue, Apr 5, 2011 at 2:01 PM, Slim Sassi <[hidden email]> wrote:
> Sean,
> I only wanted to start collecting stats with flagstats but knew that I
> needed something else to get everthing needed.
> I would like to know:
> % that didn't pass QC

That information is not in the FASTQ files, so it will not be in the
BAM files, generally.

> % mapped

That information assumes that the aligner writes the the unaligned
reads to the BAM file.  Tophat does not do that.

> % reads in exons/introns/intergenic regions

That is not something that flatstats provides.  However, there are a
number of other tools that could be coerced to give you this type of
information (including Galaxy, probably).

> and then, knowing that this is more complicated, I wanted to measure bias
> within transcripts (for example 3' versus 5'). Of course I am assuming that
> there is a consistent bias.

For this task, you may have to write some code unless someone else on
the list is aware of a package that does this for RNA-seq data.

>
> Thanks
> Slim
>
>
> On Apr 5, 2011, at 1:50 PM, Sean Davis wrote:
>
> On Apr 5, 2011 1:05 PM, "Slim Sassi" <[hidden email]> wrote:
>>
>> Sean,
>> You are correct,  I did use tophat. Can you or anyone suggest a program
>> for BAM/SAM stats where the alignment was done with tophat
>>
>
> Slim,
>
> What stats do you want to capture?  The output you gave for flagstats is
> correct for single-end tophat alignments.  All reads are aligned, none are
> paired, none are marked as duplicates.
>
> Sean
>
>> Thanks
>> Slim
>> On Apr 5, 2011, at 12:51 PM, Sean Davis wrote:
>>
>> > Hi, Slim.
>> >
>> > My guess is that you used an aligner that outputs only aligned reads
>> > (tophat, for example) and that the input was single-ended.  If that is
>> > the case, then what you see below is exactly as expected.  If not,
>> > then you might need to be more specific about how you generated the
>> > BAM file.
>> >
>> > Sean
>> >
>> > On Tue, Apr 5, 2011 at 12:31 PM, Slim Sassi
>> > <[hidden email]> wrote:
>> >> Hello,
>> >> I tried to use NGS: SAM Tools ->flagstat on a BAM files for basic
>> >> stats, but
>> >> I got results like you see below. It doesn't seem to be working. Any
>> >> suggestions?
>> >> 26584869 in total
>> >>
>> >> 0 QC failure
>> >> 0 duplicates
>> >> 26584869 mapped (100.00%)
>> >> 0 paired in sequencing
>> >> 0 read1
>> >> 0 read2
>> >> 0 properly paired (-nan%)
>> >> 0 with itself and mate mapped
>> >> 0 singletons (-nan%)
>> >> 0 with mate mapped to a different chr
>> >> 0 with mate mapped to a different chr (mapQ>=5)
>> >>
>> >> Thanks
>> >> Slim
>> >>
>> >> The information in this e-mail is intended only for the person to whom
>> >> it is
>> >> addressed. If you believe this e-mail was sent to you in error and the
>> >> e-mail
>> >> contains patient information, please contact the Partners Compliance
>> >> HelpLine at
>> >> http://www.partners.org/complianceline . If the e-mail was sent to you
>> >> in
>> >> error
>> >> but does not contain patient information, please contact the sender and
>> >> properly
>> >> dispose of the e-mail.
>>
>
>

___________________________________________________________
The Galaxy User list should be used for the discussion of
Galaxy analysis and other features on the public server
at usegalaxy.org.  Please keep all replies on the list by
using "reply all" in your mail client.  For discussion of
local Galaxy instances and the Galaxy source code, please
use the Galaxy Development list:

  http://lists.bx.psu.edu/listinfo/galaxy-dev

To manage your subscriptions to this and other Galaxy lists,
please use the interface at:

  http://lists.bx.psu.edu/
Reply | Threaded
Open this post in threaded view
|

Re: Flagstat on BAM files

Jen Hillman-Jackson
In reply to this post by ssassi
Hi Slim,

The tool is working correctly as we have it implemented right now, but
there is some room for improvement. We are adding some tuning to our
near-term to do list. Thanks for the suggestion!

Best,

Jen
Galaxy team

On 4/5/11 9:31 AM, Slim Sassi wrote:

> Hello,
> I tried to use NGS: SAM Tools ->flagstat on a BAM files for basic stats,
> but I got results like you see below. It doesn't seem to be working. Any
> suggestions?
>
> 26584869 in total
>
> 0 QC failure
> 0 duplicates
> 26584869 mapped (100.00%)
> 0 paired in sequencing
> 0 read1
> 0 read2
> 0 properly paired (-nan%)
> 0 with itself and mate mapped
> 0 singletons (-nan%)
> 0 with mate mapped to a different chr
> 0 with mate mapped to a different chr (mapQ>=5)
>
> Thanks
> Slim
>
>
> The information in this e-mail is intended only for the person to whom it is
> addressed. If you believe this e-mail was sent to you in error and the
> e-mail
> contains patient information, please contact the Partners Compliance
> HelpLine at
> http://www.partners.org/complianceline . If the e-mail was sent to you
> in error
> but does not contain patient information, please contact the sender and
> properly
> dispose of the e-mail.
>
>
>
> ___________________________________________________________
> The Galaxy User list should be used for the discussion of
> Galaxy analysis and other features on the public server
> at usegalaxy.org.  Please keep all replies on the list by
> using "reply all" in your mail client.  For discussion of
> local Galaxy instances and the Galaxy source code, please
> use the Galaxy Development list:
>
>    http://lists.bx.psu.edu/listinfo/galaxy-dev
>
> To manage your subscriptions to this and other Galaxy lists,
> please use the interface at:
>
>    http://lists.bx.psu.edu/

--
Jennifer Jackson
http://usegalaxy.org
http://galaxyproject.org
___________________________________________________________
The Galaxy User list should be used for the discussion of
Galaxy analysis and other features on the public server
at usegalaxy.org.  Please keep all replies on the list by
using "reply all" in your mail client.  For discussion of
local Galaxy instances and the Galaxy source code, please
use the Galaxy Development list:

  http://lists.bx.psu.edu/listinfo/galaxy-dev

To manage your subscriptions to this and other Galaxy lists,
please use the interface at:

  http://lists.bx.psu.edu/
Jennifer Hillman-Jackson
http://galaxyproject.org
Reply | Threaded
Open this post in threaded view
|

Flagstat crashes in non-linear workflows with TORQUE

Louise-Amélie Schmitt
In reply to this post by Sean Davis
Hello everyone

I observed an issue when flagstat is incorporated in a workflow in which
the BAM file it works on is also used by another program (generate
pileup for instance) and is NOT the input dataset (generated by sam to
bam within the workflow).

I tested it with the local job runner and with TORQUE (with the pbs
scheduler and Maui).

- With the local job runner, it works just fine

- With TORQUE I get the following error message:
pbs_submit failed, PBS error 15033: No free connections

Surprisingly, other non-linear workflows work fine. I only observed this
error with flagstat. Moreover, when flagstat is in a linear workflow, it
works fine too. Ad if it is non-linear but the input dataset is the bam
file flagstat works on, it works fine too.

Please find attached one of the test workflow where I found the error.
The input dataset is a sam file.

Any clue?

Cheers,
LA

___________________________________________________________
The Galaxy User list should be used for the discussion of
Galaxy analysis and other features on the public server
at usegalaxy.org.  Please keep all replies on the list by
using "reply all" in your mail client.  For discussion of
local Galaxy instances and the Galaxy source code, please
use the Galaxy Development list:

  http://lists.bx.psu.edu/listinfo/galaxy-dev

To manage your subscriptions to this and other Galaxy lists,
please use the interface at:

  http://lists.bx.psu.edu/

Galaxy-Workflow-test_flagstat.ga (3K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Flagstat crashes in non-linear workflows with TORQUE

Nate Coraor (nate@bx.psu.edu)
Louise-Amélie Schmitt wrote:

> Hello everyone
>
> I observed an issue when flagstat is incorporated in a workflow in which
> the BAM file it works on is also used by another program (generate
> pileup for instance) and is NOT the input dataset (generated by sam to
> bam within the workflow).
>
> I tested it with the local job runner and with TORQUE (with the pbs
> scheduler and Maui).
>
> - With the local job runner, it works just fine
>
> - With TORQUE I get the following error message:
> pbs_submit failed, PBS error 15033: No free connections

Hi,

This can most likely be fixed by increasing the value of NCONNECTS in
the TORQUE source, in src/include/libpbs.h, and recompiling on your
TORQUE server.  I haven't seen a problem after increasing the value to
20.

--nate

>
> Surprisingly, other non-linear workflows work fine. I only observed this
> error with flagstat. Moreover, when flagstat is in a linear workflow, it
> works fine too. Ad if it is non-linear but the input dataset is the bam
> file flagstat works on, it works fine too.
>
> Please find attached one of the test workflow where I found the error.
> The input dataset is a sam file.
>
> Any clue?
>
> Cheers,
> LA

> {
>     "a_galaxy_workflow": "true",
>     "annotation": "to see if it fails if not forked",
>     "format-version": "0.1",
>     "name": "test flagstat",
>     "steps": {
>         "0": {
>             "annotation": "",
>             "id": 0,
>             "input_connections": {},
>             "inputs": [
>                 {
>                     "description": "",
>                     "name": "Input Dataset"
>                 }
>             ],
>             "name": "Input dataset",
>             "outputs": [],
>             "position": {
>                 "left": 200,
>                 "top": 200
>             },
>             "tool_errors": null,
>             "tool_id": null,
>             "tool_state": "{\"name\": \"Input Dataset\"}",
>             "tool_version": null,
>             "type": "data_input",
>             "user_outputs": []
>         },
>         "1": {
>             "annotation": "",
>             "id": 1,
>             "input_connections": {
>                 "source|input1": {
>                     "id": 0,
>                     "output_name": "output"
>                 }
>             },
>             "inputs": [],
>             "name": "SAM-to-BAM",
>             "outputs": [
>                 {
>                     "name": "output1",
>                     "type": "bam"
>                 }
>             ],
>             "position": {
>                 "left": 274.5,
>                 "top": 307
>             },
>             "tool_errors": null,
>             "tool_id": "sam_to_bam",
>             "tool_state": "{\"source\": \"{\\\"index_source\\\": \\\"cached\\\", \\\"input1\\\": null, \\\"__current_case__\\\": 0}\", \"__page__\": 0}",
>             "tool_version": "1.1.1",
>             "type": "tool",
>             "user_outputs": []
>         },
>         "2": {
>             "annotation": "",
>             "id": 2,
>             "input_connections": {
>                 "input1": {
>                     "id": 1,
>                     "output_name": "output1"
>                 }
>             },
>             "inputs": [],
>             "name": "flagstat",
>             "outputs": [
>                 {
>                     "name": "output1",
>                     "type": "txt"
>                 }
>             ],
>             "position": {
>                 "left": 396.5,
>                 "top": 445
>             },
>             "tool_errors": null,
>             "tool_id": "samtools_flagstat",
>             "tool_state": "{\"__page__\": 0, \"input1\": \"null\"}",
>             "tool_version": "1.0.0",
>             "type": "tool",
>             "user_outputs": []
>         },
>         "3": {
>             "annotation": "",
>             "id": 3,
>             "input_connections": {
>                 "refOrHistory|input1": {
>                     "id": 1,
>                     "output_name": "output1"
>                 }
>             },
>             "inputs": [],
>             "name": "Generate pileup",
>             "outputs": [
>                 {
>                     "name": "output1",
>                     "type": "tabular"
>                 }
>             ],
>             "position": {
>                 "left": 519,
>                 "top": 340
>             },
>             "tool_errors": null,
>             "tool_id": "sam_pileup",
>             "tool_state": "{\"__page__\": 0, \"c\": \"{\\\"consensus\\\": \\\"no\\\", \\\"__current_case__\\\": 0}\", \"indels\": \"\\\"no\\\"\", \"refOrHistory\": \"{\\\"input1\\\": null, \\\"reference\\\": \\\"indexed\\\", \\\"__current_case__\\\": 0}\", \"lastCol\": \"\\\"no\\\"\", \"mapCap\": \"\\\"60\\\"\"}",
>             "tool_version": "1.1.1",
>             "type": "tool",
>             "user_outputs": []
>         }
>     }
> }

> ___________________________________________________________
> The Galaxy User list should be used for the discussion of
> Galaxy analysis and other features on the public server
> at usegalaxy.org.  Please keep all replies on the list by
> using "reply all" in your mail client.  For discussion of
> local Galaxy instances and the Galaxy source code, please
> use the Galaxy Development list:
>
>   http://lists.bx.psu.edu/listinfo/galaxy-dev
>
> To manage your subscriptions to this and other Galaxy lists,
> please use the interface at:
>
>   http://lists.bx.psu.edu/

___________________________________________________________
The Galaxy User list should be used for the discussion of
Galaxy analysis and other features on the public server
at usegalaxy.org.  Please keep all replies on the list by
using "reply all" in your mail client.  For discussion of
local Galaxy instances and the Galaxy source code, please
use the Galaxy Development list:

  http://lists.bx.psu.edu/listinfo/galaxy-dev

To manage your subscriptions to this and other Galaxy lists,
please use the interface at:

  http://lists.bx.psu.edu/
Reply | Threaded
Open this post in threaded view
|

Re: import problem

Jan Haas
In reply to this post by Jen Hillman-Jackson
Hi there,

I'm running a local installation on a rocks-cluster (centOs5) on a mounted NFS (Ext4 formatted). While importing a fastq file of roughly 20GB via FTP I received the following error:
"[Errno28] No Space left on Device"

Any help is appreciated.

Thanks, Jan


___________________________________________________________
The Galaxy User list should be used for the discussion of
Galaxy analysis and other features on the public server
at usegalaxy.org.  Please keep all replies on the list by
using "reply all" in your mail client.  For discussion of
local Galaxy instances and the Galaxy source code, please
use the Galaxy Development list:

  http://lists.bx.psu.edu/listinfo/galaxy-dev

To manage your subscriptions to this and other Galaxy lists,
please use the interface at:

  http://lists.bx.psu.edu/
Reply | Threaded
Open this post in threaded view
|

Re: Flagstat crashes in non-linear workflows with TORQUE

Louise-Amélie Schmitt
In reply to this post by Nate Coraor (nate@bx.psu.edu)
Le 06/04/2011 13:57, Nate Coraor a écrit :

> Louise-Amélie Schmitt wrote:
>> Hello everyone
>>
>> I observed an issue when flagstat is incorporated in a workflow in which
>> the BAM file it works on is also used by another program (generate
>> pileup for instance) and is NOT the input dataset (generated by sam to
>> bam within the workflow).
>>
>> I tested it with the local job runner and with TORQUE (with the pbs
>> scheduler and Maui).
>>
>> - With the local job runner, it works just fine
>>
>> - With TORQUE I get the following error message:
>> pbs_submit failed, PBS error 15033: No free connections
> Hi,
>
> This can most likely be fixed by increasing the value of NCONNECTS in
> the TORQUE source, in src/include/libpbs.h, and recompiling on your
> TORQUE server.  I haven't seen a problem after increasing the value to
> 20.
>
> --nate
>
Hi,

Sorry this is old but I tried recompiling torque after setting the
NCONNECTS to 20 and the issue's still there.

But there's more: It doesn't affect only flagstat but other non-linear
workflows. One of the two jobs that are submitted when their "father"
stopped running triggers the same error:
PBS error 15033: No free connections

And the best part is that it works fine sometimes. But when it crashes,
it's always the same job that crashes.

Does anyone have a clue?

Cheers,
L-A

>> Surprisingly, other non-linear workflows work fine. I only observed this
>> error with flagstat. Moreover, when flagstat is in a linear workflow, it
>> works fine too. Ad if it is non-linear but the input dataset is the bam
>> file flagstat works on, it works fine too.
>>
>> Please find attached one of the test workflow where I found the error.
>> The input dataset is a sam file.
>>
>> Any clue?
>>
>> Cheers,
>> LA
>> {
>>      "a_galaxy_workflow": "true",
>>      "annotation": "to see if it fails if not forked",
>>      "format-version": "0.1",
>>      "name": "test flagstat",
>>      "steps": {
>>          "0": {
>>              "annotation": "",
>>              "id": 0,
>>              "input_connections": {},
>>              "inputs": [
>>                  {
>>                      "description": "",
>>                      "name": "Input Dataset"
>>                  }
>>              ],
>>              "name": "Input dataset",
>>              "outputs": [],
>>              "position": {
>>                  "left": 200,
>>                  "top": 200
>>              },
>>              "tool_errors": null,
>>              "tool_id": null,
>>              "tool_state": "{\"name\": \"Input Dataset\"}",
>>              "tool_version": null,
>>              "type": "data_input",
>>              "user_outputs": []
>>          },
>>          "1": {
>>              "annotation": "",
>>              "id": 1,
>>              "input_connections": {
>>                  "source|input1": {
>>                      "id": 0,
>>                      "output_name": "output"
>>                  }
>>              },
>>              "inputs": [],
>>              "name": "SAM-to-BAM",
>>              "outputs": [
>>                  {
>>                      "name": "output1",
>>                      "type": "bam"
>>                  }
>>              ],
>>              "position": {
>>                  "left": 274.5,
>>                  "top": 307
>>              },
>>              "tool_errors": null,
>>              "tool_id": "sam_to_bam",
>>              "tool_state": "{\"source\": \"{\\\"index_source\\\": \\\"cached\\\", \\\"input1\\\": null, \\\"__current_case__\\\": 0}\", \"__page__\": 0}",
>>              "tool_version": "1.1.1",
>>              "type": "tool",
>>              "user_outputs": []
>>          },
>>          "2": {
>>              "annotation": "",
>>              "id": 2,
>>              "input_connections": {
>>                  "input1": {
>>                      "id": 1,
>>                      "output_name": "output1"
>>                  }
>>              },
>>              "inputs": [],
>>              "name": "flagstat",
>>              "outputs": [
>>                  {
>>                      "name": "output1",
>>                      "type": "txt"
>>                  }
>>              ],
>>              "position": {
>>                  "left": 396.5,
>>                  "top": 445
>>              },
>>              "tool_errors": null,
>>              "tool_id": "samtools_flagstat",
>>              "tool_state": "{\"__page__\": 0, \"input1\": \"null\"}",
>>              "tool_version": "1.0.0",
>>              "type": "tool",
>>              "user_outputs": []
>>          },
>>          "3": {
>>              "annotation": "",
>>              "id": 3,
>>              "input_connections": {
>>                  "refOrHistory|input1": {
>>                      "id": 1,
>>                      "output_name": "output1"
>>                  }
>>              },
>>              "inputs": [],
>>              "name": "Generate pileup",
>>              "outputs": [
>>                  {
>>                      "name": "output1",
>>                      "type": "tabular"
>>                  }
>>              ],
>>              "position": {
>>                  "left": 519,
>>                  "top": 340
>>              },
>>              "tool_errors": null,
>>              "tool_id": "sam_pileup",
>>              "tool_state": "{\"__page__\": 0, \"c\": \"{\\\"consensus\\\": \\\"no\\\", \\\"__current_case__\\\": 0}\", \"indels\": \"\\\"no\\\"\", \"refOrHistory\": \"{\\\"input1\\\": null, \\\"reference\\\": \\\"indexed\\\", \\\"__current_case__\\\": 0}\", \"lastCol\": \"\\\"no\\\"\", \"mapCap\": \"\\\"60\\\"\"}",
>>              "tool_version": "1.1.1",
>>              "type": "tool",
>>              "user_outputs": []
>>          }
>>      }
>> }
>> ___________________________________________________________
>> The Galaxy User list should be used for the discussion of
>> Galaxy analysis and other features on the public server
>> at usegalaxy.org.  Please keep all replies on the list by
>> using "reply all" in your mail client.  For discussion of
>> local Galaxy instances and the Galaxy source code, please
>> use the Galaxy Development list:
>>
>>    http://lists.bx.psu.edu/listinfo/galaxy-dev
>>
>> To manage your subscriptions to this and other Galaxy lists,
>> please use the interface at:
>>
>>    http://lists.bx.psu.edu/

___________________________________________________________
The Galaxy User list should be used for the discussion of
Galaxy analysis and other features on the public server
at usegalaxy.org.  Please keep all replies on the list by
using "reply all" in your mail client.  For discussion of
local Galaxy instances and the Galaxy source code, please
use the Galaxy Development list:

  http://lists.bx.psu.edu/listinfo/galaxy-dev

To manage your subscriptions to this and other Galaxy lists,
please use the interface at:

  http://lists.bx.psu.edu/
Reply | Threaded
Open this post in threaded view
|

Re: Flagstat crashes in non-linear workflows with TORQUE

Nate Coraor (nate@bx.psu.edu)
Louise-Amélie Schmitt wrote:

> Le 06/04/2011 13:57, Nate Coraor a écrit :
> >Louise-Amélie Schmitt wrote:
> >>Hello everyone
> >>
> >>I observed an issue when flagstat is incorporated in a workflow in which
> >>the BAM file it works on is also used by another program (generate
> >>pileup for instance) and is NOT the input dataset (generated by sam to
> >>bam within the workflow).
> >>
> >>I tested it with the local job runner and with TORQUE (with the pbs
> >>scheduler and Maui).
> >>
> >>- With the local job runner, it works just fine
> >>
> >>- With TORQUE I get the following error message:
> >>pbs_submit failed, PBS error 15033: No free connections
> >Hi,
> >
> >This can most likely be fixed by increasing the value of NCONNECTS in
> >the TORQUE source, in src/include/libpbs.h, and recompiling on your
> >TORQUE server.  I haven't seen a problem after increasing the value to
> >20.
> >
> >--nate
> >
> Hi,
>
> Sorry this is old but I tried recompiling torque after setting the
> NCONNECTS to 20 and the issue's still there.
>
> But there's more: It doesn't affect only flagstat but other
> non-linear workflows. One of the two jobs that are submitted when
> their "father" stopped running triggers the same error:
> PBS error 15033: No free connections

Did you recompile both the client and the server?  I am not sure which
needs it, possibly both.

>
> And the best part is that it works fine sometimes. But when it
> crashes, it's always the same job that crashes.
>
> Does anyone have a clue?
>
> Cheers,
> L-A
> >>Surprisingly, other non-linear workflows work fine. I only observed this
> >>error with flagstat. Moreover, when flagstat is in a linear workflow, it
> >>works fine too. Ad if it is non-linear but the input dataset is the bam
> >>file flagstat works on, it works fine too.
> >>
> >>Please find attached one of the test workflow where I found the error.
> >>The input dataset is a sam file.
> >>
> >>Any clue?
> >>
> >>Cheers,
> >>LA
> >>{
> >>     "a_galaxy_workflow": "true",
> >>     "annotation": "to see if it fails if not forked",
> >>     "format-version": "0.1",
> >>     "name": "test flagstat",
> >>     "steps": {
> >>         "0": {
> >>             "annotation": "",
> >>             "id": 0,
> >>             "input_connections": {},
> >>             "inputs": [
> >>                 {
> >>                     "description": "",
> >>                     "name": "Input Dataset"
> >>                 }
> >>             ],
> >>             "name": "Input dataset",
> >>             "outputs": [],
> >>             "position": {
> >>                 "left": 200,
> >>                 "top": 200
> >>             },
> >>             "tool_errors": null,
> >>             "tool_id": null,
> >>             "tool_state": "{\"name\": \"Input Dataset\"}",
> >>             "tool_version": null,
> >>             "type": "data_input",
> >>             "user_outputs": []
> >>         },
> >>         "1": {
> >>             "annotation": "",
> >>             "id": 1,
> >>             "input_connections": {
> >>                 "source|input1": {
> >>                     "id": 0,
> >>                     "output_name": "output"
> >>                 }
> >>             },
> >>             "inputs": [],
> >>             "name": "SAM-to-BAM",
> >>             "outputs": [
> >>                 {
> >>                     "name": "output1",
> >>                     "type": "bam"
> >>                 }
> >>             ],
> >>             "position": {
> >>                 "left": 274.5,
> >>                 "top": 307
> >>             },
> >>             "tool_errors": null,
> >>             "tool_id": "sam_to_bam",
> >>             "tool_state": "{\"source\": \"{\\\"index_source\\\": \\\"cached\\\", \\\"input1\\\": null, \\\"__current_case__\\\": 0}\", \"__page__\": 0}",
> >>             "tool_version": "1.1.1",
> >>             "type": "tool",
> >>             "user_outputs": []
> >>         },
> >>         "2": {
> >>             "annotation": "",
> >>             "id": 2,
> >>             "input_connections": {
> >>                 "input1": {
> >>                     "id": 1,
> >>                     "output_name": "output1"
> >>                 }
> >>             },
> >>             "inputs": [],
> >>             "name": "flagstat",
> >>             "outputs": [
> >>                 {
> >>                     "name": "output1",
> >>                     "type": "txt"
> >>                 }
> >>             ],
> >>             "position": {
> >>                 "left": 396.5,
> >>                 "top": 445
> >>             },
> >>             "tool_errors": null,
> >>             "tool_id": "samtools_flagstat",
> >>             "tool_state": "{\"__page__\": 0, \"input1\": \"null\"}",
> >>             "tool_version": "1.0.0",
> >>             "type": "tool",
> >>             "user_outputs": []
> >>         },
> >>         "3": {
> >>             "annotation": "",
> >>             "id": 3,
> >>             "input_connections": {
> >>                 "refOrHistory|input1": {
> >>                     "id": 1,
> >>                     "output_name": "output1"
> >>                 }
> >>             },
> >>             "inputs": [],
> >>             "name": "Generate pileup",
> >>             "outputs": [
> >>                 {
> >>                     "name": "output1",
> >>                     "type": "tabular"
> >>                 }
> >>             ],
> >>             "position": {
> >>                 "left": 519,
> >>                 "top": 340
> >>             },
> >>             "tool_errors": null,
> >>             "tool_id": "sam_pileup",
> >>             "tool_state": "{\"__page__\": 0, \"c\": \"{\\\"consensus\\\": \\\"no\\\", \\\"__current_case__\\\": 0}\", \"indels\": \"\\\"no\\\"\", \"refOrHistory\": \"{\\\"input1\\\": null, \\\"reference\\\": \\\"indexed\\\", \\\"__current_case__\\\": 0}\", \"lastCol\": \"\\\"no\\\"\", \"mapCap\": \"\\\"60\\\"\"}",
> >>             "tool_version": "1.1.1",
> >>             "type": "tool",
> >>             "user_outputs": []
> >>         }
> >>     }
> >>}
> >>___________________________________________________________
> >>The Galaxy User list should be used for the discussion of
> >>Galaxy analysis and other features on the public server
> >>at usegalaxy.org.  Please keep all replies on the list by
> >>using "reply all" in your mail client.  For discussion of
> >>local Galaxy instances and the Galaxy source code, please
> >>use the Galaxy Development list:
> >>
> >>   http://lists.bx.psu.edu/listinfo/galaxy-dev
> >>
> >>To manage your subscriptions to this and other Galaxy lists,
> >>please use the interface at:
> >>
> >>   http://lists.bx.psu.edu/
>
___________________________________________________________
The Galaxy User list should be used for the discussion of
Galaxy analysis and other features on the public server
at usegalaxy.org.  Please keep all replies on the list by
using "reply all" in your mail client.  For discussion of
local Galaxy instances and the Galaxy source code, please
use the Galaxy Development list:

  http://lists.bx.psu.edu/listinfo/galaxy-dev

To manage your subscriptions to this and other Galaxy lists,
please use the interface at:

  http://lists.bx.psu.edu/