Does an execution host hav to be a submit host?

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Does an execution host hav to be a submit host?

Schobesberger Richard - S0910595006
Hi all,

I'm currently testing a component with a huge number of groups per iterator. (more groups than cluster nodes)

If I'm correct, every group of the iterator runs a RunWorkflow job on the grid which submits the scripts within the iterator onto the grid.

If the number of groups is greater than slots on the cluster, the scripts within the iterator cannot be started, since they have to wait for there parents to finish.

I tried to remove some execution hosts from the submit host list, which ended in receiving errors from ergatis, stating that the host has no submit rights.

Is there a way to tell ergatis or workflow from which hosts it is able to submit, so that only a part of the grid is used for the RunWorkflow jobs and the remaining hosts are able to perform the child tasks?

Best regards,
Richard
------------------------------------------------------------------------------
Free Software Download: Index, Search & Analyze Logs and other IT data in
Real-Time with Splunk. Collect, index and harness all the fast moving IT data
generated by your applications, servers and devices whether physical, virtual
or in the cloud. Deliver compliance at lower cost and gain new business
insights. http://p.sf.net/sfu/splunk-dev2dev 
_______________________________________________
Ergatis-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/ergatis-users
Reply | Threaded
Open this post in threaded view
|

Re: Does an execution host hav to be a submit host?

Joshua Orvis
Richard -

No, execution hosts do not also need to be submit hosts.  Actually, they often aren't just to prevent that sort of situation you describe.  The only configuration of Ergatis you could do that would *require* the execution hosts to also be submit hosts is if you set the "submit pipeline as jobs" argument on, which would require ONE node of your grid queue to be both a submit and execution host. 

Otherwise, the usual setup would be that your web server runs the RunWorkflow on the overall pipeline XML, spawning the JAVA process that controls your iterator and issues however many jobs are in that iterator (submitted from the web server, which would have to be a submit host.)  Each of these jobs is another RunWorkflow command that processes their assigned group on one node without submitting any other jobs.

It's quite common to have your iterator create more groups than you have nodes on the grid, and this is something the architecture was designed for.

I hope this clears things up.

Joshua



On Mon, Feb 28, 2011 at 7:10 AM, Schobesberger Richard - S0910595006 <[hidden email]> wrote:
Hi all,

I'm currently testing a component with a huge number of groups per iterator. (more groups than cluster nodes)

If I'm correct, every group of the iterator runs a RunWorkflow job on the grid which submits the scripts within the iterator onto the grid.

If the number of groups is greater than slots on the cluster, the scripts within the iterator cannot be started, since they have to wait for there parents to finish.

I tried to remove some execution hosts from the submit host list, which ended in receiving errors from ergatis, stating that the host has no submit rights.

Is there a way to tell ergatis or workflow from which hosts it is able to submit, so that only a part of the grid is used for the RunWorkflow jobs and the remaining hosts are able to perform the child tasks?

Best regards,
Richard
------------------------------------------------------------------------------
Free Software Download: Index, Search & Analyze Logs and other IT data in
Real-Time with Splunk. Collect, index and harness all the fast moving IT data
generated by your applications, servers and devices whether physical, virtual
or in the cloud. Deliver compliance at lower cost and gain new business
insights. http://p.sf.net/sfu/splunk-dev2dev
_______________________________________________
Ergatis-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/ergatis-users


------------------------------------------------------------------------------
Free Software Download: Index, Search & Analyze Logs and other IT data in
Real-Time with Splunk. Collect, index and harness all the fast moving IT data
generated by your applications, servers and devices whether physical, virtual
or in the cloud. Deliver compliance at lower cost and gain new business
insights. http://p.sf.net/sfu/splunk-dev2dev 
_______________________________________________
Ergatis-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/ergatis-users