sharing configuration

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

sharing configuration

Benjamin Hitz

So, I have a plan for sharing configuration/software across several dev machines (desktops) and a production server.
The idea is to use git-svn to "fork" the SVN repository (current 2.17-bugfixes) and then add our config files to the /conf directory and pass them back and forth using git.

If we need to update the source code, we can pull down from the SVN repo, but otherwise we would just use git locally.

However, given somewhat random install situations, I have to know a little about the Gbrowse install internals. Before I start spelunking, I have a general question.
When I run the ./Build install - what does the installer do with the conf directory?

If I have
a) new files
b) new directories
c) modified files.

Without reading the perl module or running some tests, I would guess that it might just "clone" the conf directory recursively and put it in the "installed conf directory" (aka /etc/gbrowse2 by default).
Is it this easy or do I have to set something more complex up?

Ben
--
Ben Hitz
Senior Scientific Programmer ** Saccharomyces Genome Database ** GO Consortium
Stanford University ** [hidden email]




------------------------------------------------------------------------------
Increase Visibility of Your 3D Game App & Earn a Chance To Win $500!
Tap into the largest installed PC base & get more eyes on your game by
optimizing for Intel(R) Graphics Technology. Get started today with the
Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs.
http://p.sf.net/sfu/intelisp-dev2dev
_______________________________________________
Gmod-gbrowse mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
Reply | Threaded
Open this post in threaded view
|

Re: sharing configuration

Keiran Raine
Hi Ben,

Apologies for not answering your actual question, but have you  
considered a much simpler approach?

For keeping your config in sync (and potentially your GBrowse  
installs) you can just use a cron on your servers with:

cp -dru /nfs/someplace/gbrowse/pipeline_conf /gbrowse/.

Our pipeline writes all of our configs to a central nfs location (with  
symlinks for BAM files) but everything is configured to point to a '/
gbrowse/pipeline_conf' location.  It's very fast and reliable, you  
just need to schedule a server/apache restart for after the new  
configs are pulled

If you are using '--prefix' paths when installing your GBrowse perl  
modules there is no reason why this same approach cannot be used to  
drag the full config from your master machine (although you'd need  
some 'scp' like request rather than a simple 'cp').

Does this make sense?

Keiran Raine
Senior Computer Biologist
The Cancer Genome Project
Ext: 7703
[hidden email]





On 30 Nov 2010, at 00:30, Benjamin Hitz wrote:

>
> So, I have a plan for sharing configuration/software across several  
> dev machines (desktops) and a production server.
> The idea is to use git-svn to "fork" the SVN repository (current  
> 2.17-bugfixes) and then add our config files to the /conf directory  
> and pass them back and forth using git.
>
> If we need to update the source code, we can pull down from the SVN  
> repo, but otherwise we would just use git locally.
>
> However, given somewhat random install situations, I have to know a  
> little about the Gbrowse install internals. Before I start  
> spelunking, I have a general question.
> When I run the ./Build install - what does the installer do with the  
> conf directory?
>
> If I have
> a) new files
> b) new directories
> c) modified files.
>
> Without reading the perl module or running some tests, I would guess  
> that it might just "clone" the conf directory recursively and put it  
> in the "installed conf directory" (aka /etc/gbrowse2 by default).
> Is it this easy or do I have to set something more complex up?
>
> Ben
> --
> Ben Hitz
> Senior Scientific Programmer ** Saccharomyces Genome Database ** GO  
> Consortium
> Stanford University ** [hidden email]
>
>
>
>
> ------------------------------------------------------------------------------
> Increase Visibility of Your 3D Game App & Earn a Chance To Win $500!
> Tap into the largest installed PC base & get more eyes on your game by
> optimizing for Intel(R) Graphics Technology. Get started today with  
> the
> Intel(R) Software Partner Program. Five $500 cash prizes are up for  
> grabs.
> http://p.sf.net/sfu/intelisp-dev2dev
> _______________________________________________
> Gmod-gbrowse mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse



--
 The Wellcome Trust Sanger Institute is operated by Genome Research
 Limited, a charity registered in England with number 1021457 and a
 company registered in England with number 2742969, whose registered
 office is 215 Euston Road, London, NW1 2BE.

------------------------------------------------------------------------------
Increase Visibility of Your 3D Game App & Earn a Chance To Win $500!
Tap into the largest installed PC base & get more eyes on your game by
optimizing for Intel(R) Graphics Technology. Get started today with the
Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs.
http://p.sf.net/sfu/intelisp-dev2dev
_______________________________________________
Gmod-gbrowse mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
Reply | Threaded
Open this post in threaded view
|

Re: sharing configuration

Benjamin Hitz
Machines are different architectures and have different install directories, etc...  Also our sysadmins eschew NFS.
Also, having conf files in version control is a plus.

But, yes, this would probably work.

Ben
On Nov 29, 2010, at 11:43 PM, Keiran Raine wrote:

> Hi Ben,
>
> Apologies for not answering your actual question, but have you considered a much simpler approach?
>
> For keeping your config in sync (and potentially your GBrowse installs) you can just use a cron on your servers with:
>
> cp -dru /nfs/someplace/gbrowse/pipeline_conf /gbrowse/.
>
> Our pipeline writes all of our configs to a central nfs location (with symlinks for BAM files) but everything is configured to point to a '/gbrowse/pipeline_conf' location.  It's very fast and reliable, you just need to schedule a server/apache restart for after the new configs are pulled
>
> If you are using '--prefix' paths when installing your GBrowse perl modules there is no reason why this same approach cannot be used to drag the full config from your master machine (although you'd need some 'scp' like request rather than a simple 'cp').
>
> Does this make sense?
>
> Keiran Raine
> Senior Computer Biologist
> The Cancer Genome Project
> Ext: 7703
> [hidden email]
>
>
>
>
>
> On 30 Nov 2010, at 00:30, Benjamin Hitz wrote:
>
>>
>> So, I have a plan for sharing configuration/software across several dev machines (desktops) and a production server.
>> The idea is to use git-svn to "fork" the SVN repository (current 2.17-bugfixes) and then add our config files to the /conf directory and pass them back and forth using git.
>>
>> If we need to update the source code, we can pull down from the SVN repo, but otherwise we would just use git locally.
>>
>> However, given somewhat random install situations, I have to know a little about the Gbrowse install internals. Before I start spelunking, I have a general question.
>> When I run the ./Build install - what does the installer do with the conf directory?
>>
>> If I have
>> a) new files
>> b) new directories
>> c) modified files.
>>
>> Without reading the perl module or running some tests, I would guess that it might just "clone" the conf directory recursively and put it in the "installed conf directory" (aka /etc/gbrowse2 by default).
>> Is it this easy or do I have to set something more complex up?
>>
>> Ben
>> --
>> Ben Hitz
>> Senior Scientific Programmer ** Saccharomyces Genome Database ** GO Consortium
>> Stanford University ** [hidden email]
>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Increase Visibility of Your 3D Game App & Earn a Chance To Win $500!
>> Tap into the largest installed PC base & get more eyes on your game by
>> optimizing for Intel(R) Graphics Technology. Get started today with the
>> Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs.
>> http://p.sf.net/sfu/intelisp-dev2dev
>> _______________________________________________
>> Gmod-gbrowse mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
>
>
>
> --
> The Wellcome Trust Sanger Institute is operated by Genome ResearchLimited, a charity registered in England with number 1021457 and acompany registered in England with number 2742969, whose registeredoffice is 215 Euston Road, London, NW1 2BE.

--
Ben Hitz
Senior Scientific Programmer ** Saccharomyces Genome Database ** GO Consortium
Stanford University ** [hidden email]




------------------------------------------------------------------------------
Increase Visibility of Your 3D Game App & Earn a Chance To Win $500!
Tap into the largest installed PC base & get more eyes on your game by
optimizing for Intel(R) Graphics Technology. Get started today with the
Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs.
http://p.sf.net/sfu/intelisp-dev2dev
_______________________________________________
Gmod-gbrowse mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
Reply | Threaded
Open this post in threaded view
|

Re: sharing configuration

Scott Cain
Hi Ben,

When copying the conf directory, it recursively copies anything in
there, and with the caveat being that conf files are not copied over,
but the new conf file is copied into the conf directory with ".new"
appended.  So if you wanted to use that as a way of distributing
changed conf files you'd have to write another script that would move
all of the .new files over their older versions.

I have to say, I find Keiran's solution quite appealing.  If you use
the --prefix argument, you can have everything installed in the same
place regardless of architecture (for instance, in /gbrowse, as in
Keiran's example).  Even if you didn't use NFS you could use rsync to
keep those directories up to date.

Scott


On Tue, Nov 30, 2010 at 1:35 PM, Benjamin Hitz <[hidden email]> wrote:

> Machines are different architectures and have different install directories, etc...  Also our sysadmins eschew NFS.
> Also, having conf files in version control is a plus.
>
> But, yes, this would probably work.
>
> Ben
> On Nov 29, 2010, at 11:43 PM, Keiran Raine wrote:
>
>> Hi Ben,
>>
>> Apologies for not answering your actual question, but have you considered a much simpler approach?
>>
>> For keeping your config in sync (and potentially your GBrowse installs) you can just use a cron on your servers with:
>>
>> cp -dru /nfs/someplace/gbrowse/pipeline_conf /gbrowse/.
>>
>> Our pipeline writes all of our configs to a central nfs location (with symlinks for BAM files) but everything is configured to point to a '/gbrowse/pipeline_conf' location.  It's very fast and reliable, you just need to schedule a server/apache restart for after the new configs are pulled
>>
>> If you are using '--prefix' paths when installing your GBrowse perl modules there is no reason why this same approach cannot be used to drag the full config from your master machine (although you'd need some 'scp' like request rather than a simple 'cp').
>>
>> Does this make sense?
>>
>> Keiran Raine
>> Senior Computer Biologist
>> The Cancer Genome Project
>> Ext: 7703
>> [hidden email]
>>
>>
>>
>>
>>
>> On 30 Nov 2010, at 00:30, Benjamin Hitz wrote:
>>
>>>
>>> So, I have a plan for sharing configuration/software across several dev machines (desktops) and a production server.
>>> The idea is to use git-svn to "fork" the SVN repository (current 2.17-bugfixes) and then add our config files to the /conf directory and pass them back and forth using git.
>>>
>>> If we need to update the source code, we can pull down from the SVN repo, but otherwise we would just use git locally.
>>>
>>> However, given somewhat random install situations, I have to know a little about the Gbrowse install internals. Before I start spelunking, I have a general question.
>>> When I run the ./Build install - what does the installer do with the conf directory?
>>>
>>> If I have
>>> a) new files
>>> b) new directories
>>> c) modified files.
>>>
>>> Without reading the perl module or running some tests, I would guess that it might just "clone" the conf directory recursively and put it in the "installed conf directory" (aka /etc/gbrowse2 by default).
>>> Is it this easy or do I have to set something more complex up?
>>>
>>> Ben
>>> --
>>> Ben Hitz
>>> Senior Scientific Programmer ** Saccharomyces Genome Database ** GO Consortium
>>> Stanford University ** [hidden email]
>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Increase Visibility of Your 3D Game App & Earn a Chance To Win $500!
>>> Tap into the largest installed PC base & get more eyes on your game by
>>> optimizing for Intel(R) Graphics Technology. Get started today with the
>>> Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs.
>>> http://p.sf.net/sfu/intelisp-dev2dev
>>> _______________________________________________
>>> Gmod-gbrowse mailing list
>>> [hidden email]
>>> https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
>>
>>
>>
>> --
>> The Wellcome Trust Sanger Institute is operated by Genome ResearchLimited, a charity registered in England with number 1021457 and acompany registered in England with number 2742969, whose registeredoffice is 215 Euston Road, London, NW1 2BE.
>
> --
> Ben Hitz
> Senior Scientific Programmer ** Saccharomyces Genome Database ** GO Consortium
> Stanford University ** [hidden email]
>
>
>
>
> ------------------------------------------------------------------------------
> Increase Visibility of Your 3D Game App & Earn a Chance To Win $500!
> Tap into the largest installed PC base & get more eyes on your game by
> optimizing for Intel(R) Graphics Technology. Get started today with the
> Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs.
> http://p.sf.net/sfu/intelisp-dev2dev
> _______________________________________________
> Gmod-gbrowse mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
>



--
------------------------------------------------------------------------
Scott Cain, Ph. D.                                   scott at scottcain dot net
GMOD Coordinator (http://gmod.org/)                     216-392-3087
Ontario Institute for Cancer Research

------------------------------------------------------------------------------
Increase Visibility of Your 3D Game App & Earn a Chance To Win $500!
Tap into the largest installed PC base & get more eyes on your game by
optimizing for Intel(R) Graphics Technology. Get started today with the
Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs.
http://p.sf.net/sfu/intelisp-dev2dev
_______________________________________________
Gmod-gbrowse mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse