Running command on different queue.

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Running command on different queue.

Schobesberger Richard - S0910595006
Hi all,

I'm currently testing a new component of mine and I stumbled over the following problem:

One command of my component needs a lot of memory and only one computer of the cluster has the necessary memory capable of handling this command.
The component also has an iterator, which needs the rest of the cluster to run efficiently, so I need to specify that this memory intensive command must run on a specific computer.

I tried to specify the <dceSpec> tags in the xml with <executionHost> and <queue> but the command just ignores the settings.
The workflow_monitor tells me the wanted command ran in the <queue>-specified queue but the execution host listed is not in this queue.

So my question is, how can I specify that one command of my component runs on a specific queue or host?

Best regards,

Richard Schobesberger
------------------------------------------------------------------------------
Special Offer-- Download ArcSight Logger for FREE (a $49 USD value)!
Finally, a world-class log management solution at an even better price-free!
Download using promo code Free_Logger_4_Dev2Dev. Offer expires
February 28th, so secure your free ArcSight Logger TODAY!
http://p.sf.net/sfu/arcsight-sfd2d
_______________________________________________
Ergatis-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/ergatis-users
Reply | Threaded
Open this post in threaded view
|

Re: Running command on different queue.

Aaron Gussman
Hi Richard,
  If your command is of <type>RunDistributedCommand</type>, you can
provide options to qsub in a <passthrough> tag.  I have a component
that looks like this:


   <commandSet type="serial">
        <name>cidxxml2idx compute</name>
        <command>
            <type>RunDistributedCommand</type>
            <name>cidxxml2idx</name>
            <state>incomplete</state>
            <executable>$;ENTREZ_BIN$;/cidxxml2idx.pl</executable>
            <param>
                <key>stdout</key>

<value>$;OUTPUT_DIRECTORY$;/$;ITERATOR_NAME$;/g$;GROUP_NUMBER$;/$;I_FILE_BASE$;.$;COMPONENT_NAME$;.stdout</value>
            </param>
            <param>
                <key>stderr</key>

<value>$;OUTPUT_DIRECTORY$;/$;ITERATOR_NAME$;/g$;GROUP_NUMBER$;/$;I_FILE_BASE$;.$;COMPONENT_NAME$;.stderr</value>
            </param>
            <param>
                <key>-db</key>
                <value>$;DB$;</value>
            </param>
            <param>
                <key>-dbinfo</key>
                <value>$;DBINFO$;</value>
            </param>
            <param>
                <key>-build</key>
                <value>$;DBBUILD$;</value>
            </param>
            <param>
                <key>-xml_in</key>
                <value>$;I_FILE_PATH$;</value>
            </param>
            <dceSpec type="sge">
                <passthrough>$;SGE_OPTS$;</passthrough>
            </dceSpec>
        </command>
    </commandSet>


In my component .config I set the value of SGE_OPTS to be things that
I would like to pass through to the qsub command, eg:

$;SGE_OPTS$; = -l h_rt=64800,h_vmem=20G -pe multicore 4 -R y

I think you would want to use -q <queuename>.

If the job is of type RunUnixCommand then the contents of
<passthrough> are effectively ignored.

=Aaron


On Wed, Jan 26, 2011 at 8:53 AM, Schobesberger Richard - S0910595006
<[hidden email]> wrote:

> Hi all,
>
> I'm currently testing a new component of mine and I stumbled over the following problem:
>
> One command of my component needs a lot of memory and only one computer of the cluster has the necessary memory capable of handling this command.
> The component also has an iterator, which needs the rest of the cluster to run efficiently, so I need to specify that this memory intensive command must run on a specific computer.
>
> I tried to specify the <dceSpec> tags in the xml with <executionHost> and <queue> but the command just ignores the settings.
> The workflow_monitor tells me the wanted command ran in the <queue>-specified queue but the execution host listed is not in this queue.
>
> So my question is, how can I specify that one command of my component runs on a specific queue or host?
>
> Best regards,
>
> Richard Schobesberger
> ------------------------------------------------------------------------------
> Special Offer-- Download ArcSight Logger for FREE (a $49 USD value)!
> Finally, a world-class log management solution at an even better price-free!
> Download using promo code Free_Logger_4_Dev2Dev. Offer expires
> February 28th, so secure your free ArcSight Logger TODAY!
> http://p.sf.net/sfu/arcsight-sfd2d
> _______________________________________________
> Ergatis-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/ergatis-users
>

------------------------------------------------------------------------------
Special Offer-- Download ArcSight Logger for FREE (a $49 USD value)!
Finally, a world-class log management solution at an even better price-free!
Download using promo code Free_Logger_4_Dev2Dev. Offer expires
February 28th, so secure your free ArcSight Logger TODAY!
http://p.sf.net/sfu/arcsight-sfd2d
_______________________________________________
Ergatis-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/ergatis-users
Reply | Threaded
Open this post in threaded view
|

Re: Running command on different queue.

Schobesberger Richard - S0910595006
Hi Aaron,

Thanks for the advise! It works now but unfortunately not with the -q command. The -q command was effectively ignored by the engine.
So I used the '-l mem_total=20G' command to distinguish between the different grid nodes.

So thanks again for the $;SGE_OPTS$; and passthrough tip.

Richard
________________________________________
Von: Aaron Gussman [[hidden email]]
Gesendet: Mittwoch, 26. Januar 2011 15:38
An: [hidden email]
Betreff: Re: [Ergatis-users] Running command on different queue.

Hi Richard,
  If your command is of <type>RunDistributedCommand</type>, you can
provide options to qsub in a <passthrough> tag.  I have a component
that looks like this:


   <commandSet type="serial">
        <name>cidxxml2idx compute</name>
        <command>
            <type>RunDistributedCommand</type>
            <name>cidxxml2idx</name>
            <state>incomplete</state>
            <executable>$;ENTREZ_BIN$;/cidxxml2idx.pl</executable>
            <param>
                <key>stdout</key>

<value>$;OUTPUT_DIRECTORY$;/$;ITERATOR_NAME$;/g$;GROUP_NUMBER$;/$;I_FILE_BASE$;.$;COMPONENT_NAME$;.stdout</value>
            </param>
            <param>
                <key>stderr</key>

<value>$;OUTPUT_DIRECTORY$;/$;ITERATOR_NAME$;/g$;GROUP_NUMBER$;/$;I_FILE_BASE$;.$;COMPONENT_NAME$;.stderr</value>
            </param>
            <param>
                <key>-db</key>
                <value>$;DB$;</value>
            </param>
            <param>
                <key>-dbinfo</key>
                <value>$;DBINFO$;</value>
            </param>
            <param>
                <key>-build</key>
                <value>$;DBBUILD$;</value>
            </param>
            <param>
                <key>-xml_in</key>
                <value>$;I_FILE_PATH$;</value>
            </param>
            <dceSpec type="sge">
                <passthrough>$;SGE_OPTS$;</passthrough>
            </dceSpec>
        </command>
    </commandSet>


In my component .config I set the value of SGE_OPTS to be things that
I would like to pass through to the qsub command, eg:

$;SGE_OPTS$; = -l h_rt=64800,h_vmem=20G -pe multicore 4 -R y

I think you would want to use -q <queuename>.

If the job is of type RunUnixCommand then the contents of
<passthrough> are effectively ignored.

=Aaron


On Wed, Jan 26, 2011 at 8:53 AM, Schobesberger Richard - S0910595006
<[hidden email]> wrote:

> Hi all,
>
> I'm currently testing a new component of mine and I stumbled over the following problem:
>
> One command of my component needs a lot of memory and only one computer of the cluster has the necessary memory capable of handling this command.
> The component also has an iterator, which needs the rest of the cluster to run efficiently, so I need to specify that this memory intensive command must run on a specific computer.
>
> I tried to specify the <dceSpec> tags in the xml with <executionHost> and <queue> but the command just ignores the settings.
> The workflow_monitor tells me the wanted command ran in the <queue>-specified queue but the execution host listed is not in this queue.
>
> So my question is, how can I specify that one command of my component runs on a specific queue or host?
>
> Best regards,
>
> Richard Schobesberger
> ------------------------------------------------------------------------------
> Special Offer-- Download ArcSight Logger for FREE (a $49 USD value)!
> Finally, a world-class log management solution at an even better price-free!
> Download using promo code Free_Logger_4_Dev2Dev. Offer expires
> February 28th, so secure your free ArcSight Logger TODAY!
> http://p.sf.net/sfu/arcsight-sfd2d
> _______________________________________________
> Ergatis-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/ergatis-users
>

------------------------------------------------------------------------------
Special Offer-- Download ArcSight Logger for FREE (a $49 USD value)!
Finally, a world-class log management solution at an even better price-free!
Download using promo code Free_Logger_4_Dev2Dev. Offer expires
February 28th, so secure your free ArcSight Logger TODAY!
http://p.sf.net/sfu/arcsight-sfd2d
_______________________________________________
Ergatis-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/ergatis-users

------------------------------------------------------------------------------
The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE:
Pinpoint memory and threading errors before they happen.
Find and fix more than 250 security defects in the development cycle.
Locate bottlenecks in serial and parallel code that limit performance.
http://p.sf.net/sfu/intel-dev2devfeb
_______________________________________________
Ergatis-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/ergatis-users