Global variables for pipeline/template?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Global variables for pipeline/template?

Jason Kwan-2
Hi Guys,

I've made an ergatis template to process the output of CloVR-microbe and create/populate a manatee database on another server with the data  (at the moment a project template, but I will convert it to a global template).  It includes a custom component I made to correct the absolute paths in the bsmls and other housekeeping tasks.  At the moment the user will put in the manatee server address, the mysql user and password into this component.  Is there anyway that these inputs can be put into a variable that can then be used for the corresponding inputs for the subsequent components (initdb and bsml2chado)?  I successfully piped the output of my component (which consist of bsml lists) to the subsequent ones, but the user still has to type the database name etc. multiple times.  I tried using the variable name from the .config file of my component as input for subsequent components, but this didn't work.

Any ideas?

Thanks for your help

Jason
------------------------------------------------------------------------------
RSA(R) Conference 2012
Save $700 by Nov 18
Register now
http://p.sf.net/sfu/rsa-sfdev2dev1
_______________________________________________
Ergatis-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/ergatis-users
Reply | Threaded
Open this post in threaded view
|

Re: Global variables for pipeline/template?

Joshua Orvis
Jason -

There are a few different levels of 'global' here but, while a few of them might solve your problem, I'm not really convinced any of them are exactly what you are looking for here.

The highest level is possibly the software.config, where you can put settings for all components that will be included in all instances of any given component as well as the [common] sections, whose variables will be available to all components in all pipelines.

Next you could have it in the project.config file for that user's project, and the variables will be available to all pipelines within that project.

Those are really the two 'global' areas where variables could be defined at the moment.  There isn't currently a way to define one at pipeline creation time and reference it throughout the entire pipeline.

If this is a component you're using only internally and have developed yourself there are all sorts of things you could do with environmental variables here, however much I dislike them.  Instead of needing the database connection values directly in the XML you could call wrapper scripts from the XML, and within those pull the needed values from the environment and call the underlying command.  Not as clean, certainly, but it would really be the only 'private' way to currently do something like database connection credentials, as all the other methods would put user names, passwords, etc. right into the XML where they might be read by others if you're not careful with permissions.

Joshua


On Mon, Nov 7, 2011 at 9:50 AM, Jason Kwan <[hidden email]> wrote:
Hi Guys,

I've made an ergatis template to process the output of CloVR-microbe and create/populate a manatee database on another server with the data  (at the moment a project template, but I will convert it to a global template).  It includes a custom component I made to correct the absolute paths in the bsmls and other housekeeping tasks.  At the moment the user will put in the manatee server address, the mysql user and password into this component.  Is there anyway that these inputs can be put into a variable that can then be used for the corresponding inputs for the subsequent components (initdb and bsml2chado)?  I successfully piped the output of my component (which consist of bsml lists) to the subsequent ones, but the user still has to type the database name etc. multiple times.  I tried using the variable name from the .config file of my component as input for subsequent components, but this didn't work.

Any ideas?

Thanks for your help

Jason
------------------------------------------------------------------------------
RSA(R) Conference 2012
Save $700 by Nov 18
Register now
http://p.sf.net/sfu/rsa-sfdev2dev1
_______________________________________________
Ergatis-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/ergatis-users


------------------------------------------------------------------------------
RSA(R) Conference 2012
Save $700 by Nov 18
Register now
http://p.sf.net/sfu/rsa-sfdev2dev1
_______________________________________________
Ergatis-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/ergatis-users
Reply | Threaded
Open this post in threaded view
|

Re: Global variables for pipeline/template?

Jason Kwan-2
Hi Joshua,

Thanks for your reply.  I wonder if I just use the export command to make a unix environmental variable, will this be accessible to other (subsequent) components?  I am thinking I would then just put $VAR or something under database name etc. in the template.

Thanks again

Jason

Sent from my iPhone

On Nov 7, 2011, at 11:04 PM, "Joshua Orvis" <[hidden email]> wrote:

Jason -

There are a few different levels of 'global' here but, while a few of them might solve your problem, I'm not really convinced any of them are exactly what you are looking for here.

The highest level is possibly the software.config, where you can put settings for all components that will be included in all instances of any given component as well as the [common] sections, whose variables will be available to all components in all pipelines.

Next you could have it in the project.config file for that user's project, and the variables will be available to all pipelines within that project.

Those are really the two 'global' areas where variables could be defined at the moment.  There isn't currently a way to define one at pipeline creation time and reference it throughout the entire pipeline.

If this is a component you're using only internally and have developed yourself there are all sorts of things you could do with environmental variables here, however much I dislike them.  Instead of needing the database connection values directly in the XML you could call wrapper scripts from the XML, and within those pull the needed values from the environment and call the underlying command.  Not as clean, certainly, but it would really be the only 'private' way to currently do something like database connection credentials, as all the other methods would put user names, passwords, etc. right into the XML where they might be read by others if you're not careful with permissions.

Joshua


On Mon, Nov 7, 2011 at 9:50 AM, Jason Kwan <[hidden email]> wrote:
Hi Guys,

I've made an ergatis template to process the output of CloVR-microbe and create/populate a manatee database on another server with the data  (at the moment a project template, but I will convert it to a global template).  It includes a custom component I made to correct the absolute paths in the bsmls and other housekeeping tasks.  At the moment the user will put in the manatee server address, the mysql user and password into this component.  Is there anyway that these inputs can be put into a variable that can then be used for the corresponding inputs for the subsequent components (initdb and bsml2chado)?  I successfully piped the output of my component (which consist of bsml lists) to the subsequent ones, but the user still has to type the database name etc. multiple times.  I tried using the variable name from the .config file of my component as input for subsequent components, but this didn't work.

Any ideas?

Thanks for your help

Jason
------------------------------------------------------------------------------
RSA(R) Conference 2012
Save $700 by Nov 18
Register now
http://p.sf.net/sfu/rsa-sfdev2dev1
_______________________________________________
Ergatis-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/ergatis-users


------------------------------------------------------------------------------
RSA(R) Conference 2012
Save $700 by Nov 18
Register now
http://p.sf.net/sfu/rsa-sfdev2dev1
_______________________________________________
Ergatis-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/ergatis-users
Reply | Threaded
Open this post in threaded view
|

Re: Global variables for pipeline/template?

Kevin Galens
Jason,

If I understand the problem correctly, you want to streamline the process for configuring a pipeline from a template. There's not a lot of support for this in the interface, but you can create a config file for an entire pipeline. To look at an example of this, see the Prokaryotic_Annotation_Pipeline_Glimmer_Predictions pipeline template in ergatis svn:


Take a look at the pipeline.config file in that template. At the bottom, it contains a [global] section. This section describes variables that can be used across multiple components. Each component can have a section in this file as well. The variables described here are those which will receive the values defined in the [global] section. All other configured variables should still reside in the individual component config files. Basically, the pipeline.config file is for options that will change between each run.

There is no mechanism through the interface to create/run a pipeline using the pipeline.config. There is a relatively simple script in the ergatis svn util directory called run_prok_pipeline.pl:


This script will read a pipeline.layout file, a configured pipeline.config file and a repository root and it will create an instance of a pipeline with those variables configured in the config file. It will optionally take in the ergatis.ini file. If this is provided, the script will automatically call run on the pipeline. 

In general, the script just takes the layout and config file, creates an Ergatis::SavedPipeline object, then calls configure_saved_pipeline on it (which will pull in the variables from the pipeline.config file).

Now, this does absolutely nothing for hiding passwords. So there's that.

Kevin

From: Jason Kwan <[hidden email]>
Reply-To: <[hidden email]>
Date: Tue, 8 Nov 2011 09:11:27 -0700
To: Joshua Orvis <[hidden email]>
Cc: "[hidden email]" <[hidden email]>
Subject: Re: [Ergatis-users] Global variables for pipeline/template?

Hi Joshua,

Thanks for your reply.  I wonder if I just use the export command to make a unix environmental variable, will this be accessible to other (subsequent) components?  I am thinking I would then just put $VAR or something under database name etc. in the template.

Thanks again

Jason

Sent from my iPhone

On Nov 7, 2011, at 11:04 PM, "Joshua Orvis" <[hidden email]> wrote:

Jason -

There are a few different levels of 'global' here but, while a few of them might solve your problem, I'm not really convinced any of them are exactly what you are looking for here.

The highest level is possibly the software.config, where you can put settings for all components that will be included in all instances of any given component as well as the [common] sections, whose variables will be available to all components in all pipelines.

Next you could have it in the project.config file for that user's project, and the variables will be available to all pipelines within that project.

Those are really the two 'global' areas where variables could be defined at the moment.  There isn't currently a way to define one at pipeline creation time and reference it throughout the entire pipeline.

If this is a component you're using only internally and have developed yourself there are all sorts of things you could do with environmental variables here, however much I dislike them.  Instead of needing the database connection values directly in the XML you could call wrapper scripts from the XML, and within those pull the needed values from the environment and call the underlying command.  Not as clean, certainly, but it would really be the only 'private' way to currently do something like database connection credentials, as all the other methods would put user names, passwords, etc. right into the XML where they might be read by others if you're not careful with permissions.

Joshua


On Mon, Nov 7, 2011 at 9:50 AM, Jason Kwan <[hidden email]> wrote:
Hi Guys,

I've made an ergatis template to process the output of CloVR-microbe and create/populate a manatee database on another server with the data  (at the moment a project template, but I will convert it to a global template).  It includes a custom component I made to correct the absolute paths in the bsmls and other housekeeping tasks.  At the moment the user will put in the manatee server address, the mysql user and password into this component.  Is there anyway that these inputs can be put into a variable that can then be used for the corresponding inputs for the subsequent components (initdb and bsml2chado)?  I successfully piped the output of my component (which consist of bsml lists) to the subsequent ones, but the user still has to type the database name etc. multiple times.  I tried using the variable name from the .config file of my component as input for subsequent components, but this didn't work.

Any ideas?

Thanks for your help

Jason
------------------------------------------------------------------------------
RSA(R) Conference 2012
Save $700 by Nov 18
Register now
http://p.sf.net/sfu/rsa-sfdev2dev1
_______________________________________________
Ergatis-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/ergatis-users

------------------------------------------------------------------------------ RSA(R) Conference 2012 Save $700 by Nov 18 Register now http://p.sf.net/sfu/rsa-sfdev2dev1_______________________________________________ Ergatis-users mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/ergatis-users

------------------------------------------------------------------------------
RSA(R) Conference 2012
Save $700 by Nov 18
Register now
http://p.sf.net/sfu/rsa-sfdev2dev1
_______________________________________________
Ergatis-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/ergatis-users
Reply | Threaded
Open this post in threaded view
|

Re: Global variables for pipeline/template?

Jason Kwan-2
Hi Kevin,

Thanks for the info.  I'll do some fiddling around to see what the simplest method will be.  I'm not really too concerned about the password issue as I've got the Ergatis page password protected anyway right now, but I was trying to make it easy to use for the other lab members who need to use it.

Thanks

Jason

On Nov 8, 2011, at 8:35 AM, Kevin Galens wrote:

Jason,

If I understand the problem correctly, you want to streamline the process for configuring a pipeline from a template. There's not a lot of support for this in the interface, but you can create a config file for an entire pipeline. To look at an example of this, see the Prokaryotic_Annotation_Pipeline_Glimmer_Predictions pipeline template in ergatis svn:


Take a look at the pipeline.config file in that template. At the bottom, it contains a [global] section. This section describes variables that can be used across multiple components. Each component can have a section in this file as well. The variables described here are those which will receive the values defined in the [global] section. All other configured variables should still reside in the individual component config files. Basically, the pipeline.config file is for options that will change between each run.

There is no mechanism through the interface to create/run a pipeline using the pipeline.config. There is a relatively simple script in the ergatis svn util directory called run_prok_pipeline.pl:


This script will read a pipeline.layout file, a configured pipeline.config file and a repository root and it will create an instance of a pipeline with those variables configured in the config file. It will optionally take in the ergatis.ini file. If this is provided, the script will automatically call run on the pipeline. 

In general, the script just takes the layout and config file, creates an Ergatis::SavedPipeline object, then calls configure_saved_pipeline on it (which will pull in the variables from the pipeline.config file).

Now, this does absolutely nothing for hiding passwords. So there's that.

Kevin

From: Jason Kwan <[hidden email]>
Reply-To: <[hidden email]>
Date: Tue, 8 Nov 2011 09:11:27 -0700
To: Joshua Orvis <[hidden email]>
Cc: "[hidden email]" <[hidden email]>
Subject: Re: [Ergatis-users] Global variables for pipeline/template?

Hi Joshua,

Thanks for your reply.  I wonder if I just use the export command to make a unix environmental variable, will this be accessible to other (subsequent) components?  I am thinking I would then just put $VAR or something under database name etc. in the template.

Thanks again

Jason

Sent from my iPhone

On Nov 7, 2011, at 11:04 PM, "Joshua Orvis" <[hidden email]> wrote:

Jason -

There are a few different levels of 'global' here but, while a few of them might solve your problem, I'm not really convinced any of them are exactly what you are looking for here.

The highest level is possibly the software.config, where you can put settings for all components that will be included in all instances of any given component as well as the [common] sections, whose variables will be available to all components in all pipelines.

Next you could have it in the project.config file for that user's project, and the variables will be available to all pipelines within that project.

Those are really the two 'global' areas where variables could be defined at the moment.  There isn't currently a way to define one at pipeline creation time and reference it throughout the entire pipeline.

If this is a component you're using only internally and have developed yourself there are all sorts of things you could do with environmental variables here, however much I dislike them.  Instead of needing the database connection values directly in the XML you could call wrapper scripts from the XML, and within those pull the needed values from the environment and call the underlying command.  Not as clean, certainly, but it would really be the only 'private' way to currently do something like database connection credentials, as all the other methods would put user names, passwords, etc. right into the XML where they might be read by others if you're not careful with permissions.

Joshua


On Mon, Nov 7, 2011 at 9:50 AM, Jason Kwan <[hidden email]> wrote:
Hi Guys,

I've made an ergatis template to process the output of CloVR-microbe and create/populate a manatee database on another server with the data  (at the moment a project template, but I will convert it to a global template).  It includes a custom component I made to correct the absolute paths in the bsmls and other housekeeping tasks.  At the moment the user will put in the manatee server address, the mysql user and password into this component.  Is there anyway that these inputs can be put into a variable that can then be used for the corresponding inputs for the subsequent components (initdb and bsml2chado)?  I successfully piped the output of my component (which consist of bsml lists) to the subsequent ones, but the user still has to type the database name etc. multiple times.  I tried using the variable name from the .config file of my component as input for subsequent components, but this didn't work.

Any ideas?

Thanks for your help

Jason
------------------------------------------------------------------------------
RSA(R) Conference 2012
Save $700 by Nov 18
Register now
http://p.sf.net/sfu/rsa-sfdev2dev1
_______________________________________________
Ergatis-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/ergatis-users

------------------------------------------------------------------------------ RSA(R) Conference 2012 Save $700 by Nov 18 Register now http://p.sf.net/sfu/rsa-sfdev2dev1_______________________________________________ Ergatis-users mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/ergatis-users
------------------------------------------------------------------------------
RSA(R) Conference 2012
Save $700 by Nov 18
Register now
http://p.sf.net/sfu/rsa-sfdev2dev1_______________________________________________
Ergatis-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/ergatis-users


------------------------------------------------------------------------------
RSA(R) Conference 2012
Save $700 by Nov 18
Register now
http://p.sf.net/sfu/rsa-sfdev2dev1
_______________________________________________
Ergatis-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/ergatis-users