planemo reports test failure when test succeeds

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

planemo reports test failure when test succeeds

Peter van Heusden
Hi there

When testing the latest trimmomatic tool (from https://github.com/fls-bioinformatics-core/galaxy-tools/tree/master/trimmomatic, planemo test reports an error on the last test, but according to the test report the test has succeeded (see attached HTML and JSON). For the test setup you need to set some environment variables:

TRIMMOMATIC_DIR - the location of the trimmomatic jar
TRIMMOMATIC_JAR - the name of the jar file
TRIMMOMATIC_ADAPTERS_DIR - location of adapters

This is with planemo 0.22.2 and a fresh $HOME/.planemo.

Any ideas?

Peter


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

tool_test_output.html (340K) Download Attachment
tool_test_output.json (14K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: planemo reports test failure when test succeeds

Peter Briggs
Hello again Peter

I also see this when using planemo to run the tool tests both on
trimmomatic and also on another of our tools i.e. one or more of the
tests will be reported as 'failed' on the command line when the HTML
report indicates that all tests executed successfully.

Now I think about it, planemo itself must exit with status 0 in these
cases as my tests on Travis-CI also report success, for example see:

https://travis-ci.org/fls-bioinformatics-core/galaxy-tools/jobs/104643315

Scroll to the bottom of the log and the last test is reported as
'failed', however the Travis test overall passes. I hadn't really
thought about it before since the planemo exit status seems to be
correct (i.e. in cases where there are genuine tool test failures, it
will return a non-zero status at the end which causes Travis to
correctly report a failure).

(Btw this referenced above used planemo 0.22.2 but I've seen it in older
versions too.)

I don't think I've ever seen planemo report a 'passed' test as 'failed',
but generally I check the HTML to verify whether the failures are real
or not.

Best wishes

Peter

On 22/01/16 11:40, Peter van Heusden wrote:

> Hi there
>
> When testing the latest trimmomatic tool (from
> https://github.com/fls-bioinformatics-core/galaxy-tools/tree/master/trimmomatic,
> planemo test reports an error on the last test, but according to the
> test report the test has succeeded (see attached HTML and JSON). For the
> test setup you need to set some environment variables:
>
> TRIMMOMATIC_DIR - the location of the trimmomatic jar
> TRIMMOMATIC_JAR - the name of the jar file
> TRIMMOMATIC_ADAPTERS_DIR - location of adapters
>
> This is with planemo 0.22.2 and a fresh $HOME/.planemo.
>
> Any ideas?
>
> Peter
>
>
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>    https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>    http://galaxyproject.org/search/mailinglists/
>

--
Peter Briggs [hidden email]
Bioinformatics Core Facility University of Manchester
B.1083 Michael Smith Bldg Tel: (0161) 2751482
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: planemo reports test failure when test succeeds

Peter van Heusden
Zipho looked at this over here and had some ideas about how some warning output might trigger this spurious failure - I haven't had time to follow up on that theory.

On 4 February 2016 at 14:26, Peter Briggs <[hidden email]> wrote:
Hello again Peter

I also see this when using planemo to run the tool tests both on trimmomatic and also on another of our tools i.e. one or more of the tests will be reported as 'failed' on the command line when the HTML report indicates that all tests executed successfully.

Now I think about it, planemo itself must exit with status 0 in these cases as my tests on Travis-CI also report success, for example see:

https://travis-ci.org/fls-bioinformatics-core/galaxy-tools/jobs/104643315

Scroll to the bottom of the log and the last test is reported as 'failed', however the Travis test overall passes. I hadn't really thought about it before since the planemo exit status seems to be correct (i.e. in cases where there are genuine tool test failures, it will return a non-zero status at the end which causes Travis to correctly report a failure).

(Btw this referenced above used planemo 0.22.2 but I've seen it in older versions too.)

I don't think I've ever seen planemo report a 'passed' test as 'failed', but generally I check the HTML to verify whether the failures are real or not.

Best wishes

Peter


On 22/01/16 11:40, Peter van Heusden wrote:
Hi there

When testing the latest trimmomatic tool (from
https://github.com/fls-bioinformatics-core/galaxy-tools/tree/master/trimmomatic,
planemo test reports an error on the last test, but according to the
test report the test has succeeded (see attached HTML and JSON). For the
test setup you need to set some environment variables:

TRIMMOMATIC_DIR - the location of the trimmomatic jar
TRIMMOMATIC_JAR - the name of the jar file
TRIMMOMATIC_ADAPTERS_DIR - location of adapters

This is with planemo 0.22.2 and a fresh $HOME/.planemo.

Any ideas?

Peter



___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


--
Peter Briggs [hidden email]
Bioinformatics Core Facility University of Manchester
B.1083 Michael Smith Bldg Tel: (0161) 2751482

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/