Job metrics configuration

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Job metrics configuration

Peter van Heusden
Hi there

I'm trying to understand how the job metrics configuration works. Reading through various examples, it looks like it works together with the dynamic destination support.

First on the dynamic destination support, this works, as I understand it, by importing modules (files ending in .py) from the directory set as the rules_module, where rules_module is a parameter for the dynamic job plugin. As in this snipped from the usegalaxy configuration:

        <plugin id="dynamic" type="runner">
            <!-- These live in the virtualenv -->
            <param id="rules_module">usegalaxy.jobs.rules</param>
        </plugin>


So import usegalaxy.jobs.rules, find where this resolves to as a directory, find all .py files in that directory. If rules_module is not specified, the default is galaxy.jobs.rules (under galaxy/lib).

So these .py files are imported and within them are functions that are used in dynamic destinations (those with runner="dynamic") and in turn tools are directed to use these destinations.

Now the job metrics configuration supplies both a user interface and then a set of keys that are made accessible to dynamic runners as key value pairs on the job.parameters attribute (passed in as the third argument, i.e. job). Specifically the job metrics configuration exists under a key '__job_resource'.

Is this all correct? Is there a more straightforward way of moving from job resource specification to job configuration than this?

Thanks,
Peter

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Job metrics configuration

John Chilton-4
So job metrics are completely different than what you described here.
Job metrics are enabled by default and can be configured fairly
directly by modifying config/job_metrics_conf.xml. New plugins are
relatively easy to write and can be added as python files to
lib/galaxy/jobs/metrics/instrumenters/.

What you described is what is used on usegalaxy.org to allow users to
have some input on how jobs are scheduled. I call these job resources
parameters - in job_conf.xml particular tools can be mapped to these
parameters and - yes you are correct dynamic job destination functions
can consume them.

The dynamic job destinations definitely shouldn't be accessing these
through job.parameters keyed on __job_resource. The dynamic
destination function should just take in a parameter named
resource_params and then the function will be passed in.

Dynamic Destinations are documented here:
https://wiki.galaxyproject.org/Admin/Config/Jobs#Dynamic_Destination_Mapping

You are correct job resource parameters have not been documented*,
there is an open issue here:
https://github.com/galaxyproject/galaxy/issues/1266

* It should be noted that while this a frequently requested feature
and used on usegalaxy.org - the feature is somewhat controversial. The
thinking among some is that is exposes a level of detail that Galaxy
is supposed to hide. I don't share this particular concern, but the
implementation does worry me - what it does to persisted job state and
parameter handling is fairly hacky and I'm somewhat surprised we
haven't encountered more problems. This is part of the reason not why
this hasn't been documented, but why I feel less bad that it hasn't
been documented than other stuff I have done that also hasn't been
documented :|.

-John

On Mon, Jan 11, 2016 at 1:56 PM, Peter van Heusden <[hidden email]> wrote:

> Hi there
>
> I'm trying to understand how the job metrics configuration works. Reading
> through various examples, it looks like it works together with the dynamic
> destination support.
>
> First on the dynamic destination support, this works, as I understand it, by
> importing modules (files ending in .py) from the directory set as the
> rules_module, where rules_module is a parameter for the dynamic job plugin.
> As in this snipped from the usegalaxy configuration:
>
>         <plugin id="dynamic" type="runner">
>             <!-- These live in the virtualenv -->
>             <param id="rules_module">usegalaxy.jobs.rules</param>
>         </plugin>
>
>
> So import usegalaxy.jobs.rules, find where this resolves to as a directory,
> find all .py files in that directory. If rules_module is not specified, the
> default is galaxy.jobs.rules (under galaxy/lib).
>
> So these .py files are imported and within them are functions that are used
> in dynamic destinations (those with runner="dynamic") and in turn tools are
> directed to use these destinations.
>
> Now the job metrics configuration supplies both a user interface and then a
> set of keys that are made accessible to dynamic runners as key value pairs
> on the job.parameters attribute (passed in as the third argument, i.e. job).
> Specifically the job metrics configuration exists under a key
> '__job_resource'.
>
> Is this all correct? Is there a more straightforward way of moving from job
> resource specification to job configuration than this?
>
> Thanks,
> Peter
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Job metrics configuration

Peter van Heusden
Hi there John

Thanks for the clarification. My initial aim was, exactly, to expose those parameters that, as you say, some people don't think should be exposed.

My use case is to allow jobs with different native parameters to be scheduled on our clouster from our Galaxy server. Perhaps a better approach is to use dynamic destinations that try and estimate appropriate values for e.g. hvmem, but at present our users are used to setting such parameters themselves.

Peter

On 11 January 2016 at 17:32, John Chilton <[hidden email]> wrote:
So job metrics are completely different than what you described here.
Job metrics are enabled by default and can be configured fairly
directly by modifying config/job_metrics_conf.xml. New plugins are
relatively easy to write and can be added as python files to
lib/galaxy/jobs/metrics/instrumenters/.

What you described is what is used on usegalaxy.org to allow users to
have some input on how jobs are scheduled. I call these job resources
parameters - in job_conf.xml particular tools can be mapped to these
parameters and - yes you are correct dynamic job destination functions
can consume them.

The dynamic job destinations definitely shouldn't be accessing these
through job.parameters keyed on __job_resource. The dynamic
destination function should just take in a parameter named
resource_params and then the function will be passed in.

Dynamic Destinations are documented here:
https://wiki.galaxyproject.org/Admin/Config/Jobs#Dynamic_Destination_Mapping

You are correct job resource parameters have not been documented*,
there is an open issue here:
https://github.com/galaxyproject/galaxy/issues/1266

* It should be noted that while this a frequently requested feature
and used on usegalaxy.org - the feature is somewhat controversial. The
thinking among some is that is exposes a level of detail that Galaxy
is supposed to hide. I don't share this particular concern, but the
implementation does worry me - what it does to persisted job state and
parameter handling is fairly hacky and I'm somewhat surprised we
haven't encountered more problems. This is part of the reason not why
this hasn't been documented, but why I feel less bad that it hasn't
been documented than other stuff I have done that also hasn't been
documented :|.

-John

On Mon, Jan 11, 2016 at 1:56 PM, Peter van Heusden <[hidden email]> wrote:
> Hi there
>
> I'm trying to understand how the job metrics configuration works. Reading
> through various examples, it looks like it works together with the dynamic
> destination support.
>
> First on the dynamic destination support, this works, as I understand it, by
> importing modules (files ending in .py) from the directory set as the
> rules_module, where rules_module is a parameter for the dynamic job plugin.
> As in this snipped from the usegalaxy configuration:
>
>         <plugin id="dynamic" type="runner">
>             <!-- These live in the virtualenv -->
>             <param id="rules_module">usegalaxy.jobs.rules</param>
>         </plugin>
>
>
> So import usegalaxy.jobs.rules, find where this resolves to as a directory,
> find all .py files in that directory. If rules_module is not specified, the
> default is galaxy.jobs.rules (under galaxy/lib).
>
> So these .py files are imported and within them are functions that are used
> in dynamic destinations (those with runner="dynamic") and in turn tools are
> directed to use these destinations.
>
> Now the job metrics configuration supplies both a user interface and then a
> set of keys that are made accessible to dynamic runners as key value pairs
> on the job.parameters attribute (passed in as the third argument, i.e. job).
> Specifically the job metrics configuration exists under a key
> '__job_resource'.
>
> Is this all correct? Is there a more straightforward way of moving from job
> resource specification to job configuration than this?
>
> Thanks,
> Peter
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/