Bypassing Data Managers?

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Bypassing Data Managers?

Raymond Wan-2
Dear all,

I'm a bit new at being a Galaxy administrator, so apologies for the
basic question.

I need to install some bowtie and bowtie2 indices.  I have the indices
already but came across this page:

https://wiki.galaxyproject.org/Admin/DataPreparation

which says the steps mentioned on that page have been replaced with
Data Managers.  I'm having problems with installing Data Managers
(accessing ToolShed right now and some errors which I guess I need to
sort out later) and was wondering if there was a way to bypass Data
Managers?

Are Data Managers the only way of installing reference data?  With the
indices already pre-calculated, can I install them without having to
go through re-calculating them within Galaxy?

Thank you!

Ray
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Bypassing Data Managers?

Raymond Wan-2
Dear all,

I have gotten a bit further and was able to install the Data Managers
this time.  Previously, I was having problems accessing the ToolShed
web site and some Python errors in the log file, but fine ... sort of
(I get some intermittent errors accessing the Tool Shed site; maybe
it's a problem on my side).

However, with the FASTA fetch tool installed from the Galaxy Main Tool
Shed, I selected "Local Data" from the Administrator panel.  Under
"Run Data Manager Tools", I now have "Bowtie index", "Reference
Genome", etc.  When I click on any of the options under "Run Data
Manager Tools", though, I get a blank page.  I mean, I get a
user-level menu on the left, history on the right, a menu along the
top.  But the center is blank.  Is there something else I need to do?

If I go to "View Data Manager Jobs", "Reference Genome", though, I get
a message that says there are "There are no jobs for this data
manager.".

If possible, I would like to upload FASTA files myself to our local
Galaxy instance.  For reasons which I'm still working on (and which is
unrelated to Galaxy), download times are very long.  Since I have the
FASTA file already, it might be easier to upload one myself; but if
that's not something Data Managers can do, I guess I can just start
the download and set it aside...

Thank you!

Ray



On Thu, Feb 18, 2016 at 8:36 AM, Raymond Wan <[hidden email]> wrote:

> Dear all,
>
> I'm a bit new at being a Galaxy administrator, so apologies for the
> basic question.
>
> I need to install some bowtie and bowtie2 indices.  I have the indices
> already but came across this page:
>
> https://wiki.galaxyproject.org/Admin/DataPreparation
>
> which says the steps mentioned on that page have been replaced with
> Data Managers.  I'm having problems with installing Data Managers
> (accessing ToolShed right now and some errors which I guess I need to
> sort out later) and was wondering if there was a way to bypass Data
> Managers?
>
> Are Data Managers the only way of installing reference data?  With the
> indices already pre-calculated, can I install them without having to
> go through re-calculating them within Galaxy?
>
> Thank you!
>
> Ray
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Bypassing Data Managers?

Hans-Rudolf Hotz
Hi Ray

If you already have the reference data (as you have written in your
first e-mail), then I suggest you manually add those files (file paths)
to

~/tool-data/bowtie2_indices.loc
~/tool-data/bowtie_indices.loc

This way you prevent duplication of your reference data


Regards, Hans-Rudolf



> Dear all,
>
> I have gotten a bit further and was able to install the Data Managers
> this time.  Previously, I was having problems accessing the ToolShed
> web site and some Python errors in the log file, but fine ... sort of
> (I get some intermittent errors accessing the Tool Shed site; maybe
> it's a problem on my side).
>
> However, with the FASTA fetch tool installed from the Galaxy Main Tool
> Shed, I selected "Local Data" from the Administrator panel.  Under
> "Run Data Manager Tools", I now have "Bowtie index", "Reference
> Genome", etc.  When I click on any of the options under "Run Data
> Manager Tools", though, I get a blank page.  I mean, I get a
> user-level menu on the left, history on the right, a menu along the
> top.  But the center is blank.  Is there something else I need to do?
>
> If I go to "View Data Manager Jobs", "Reference Genome", though, I get
> a message that says there are "There are no jobs for this data
> manager.".
>
> If possible, I would like to upload FASTA files myself to our local
> Galaxy instance.  For reasons which I'm still working on (and which is
> unrelated to Galaxy), download times are very long.  Since I have the
> FASTA file already, it might be easier to upload one myself; but if
> that's not something Data Managers can do, I guess I can just start
> the download and set it aside...
>
> Thank you!
>
> Ray
>
>
>
> On Thu, Feb 18, 2016 at 8:36 AM, Raymond Wan <[hidden email]> wrote:
>> Dear all,
>>
>> I'm a bit new at being a Galaxy administrator, so apologies for the
>> basic question.
>>
>> I need to install some bowtie and bowtie2 indices.  I have the indices
>> already but came across this page:
>>
>> https://wiki.galaxyproject.org/Admin/DataPreparation
>>
>> which says the steps mentioned on that page have been replaced with
>> Data Managers.  I'm having problems with installing Data Managers
>> (accessing ToolShed right now and some errors which I guess I need to
>> sort out later) and was wondering if there was a way to bypass Data
>> Managers?
>>
>> Are Data Managers the only way of installing reference data?  With the
>> indices already pre-calculated, can I install them without having to
>> go through re-calculating them within Galaxy?
>>
>> Thank you!
>>
>> Ray
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>    https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>    http://galaxyproject.org/search/mailinglists/
>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Bypassing Data Managers?

Raymond Wan-2
Hi Hans-Rudolf,

Thank you for getting back to me!

I did take a peak at those files and they seem self-explanatory.  I
think I can do that.  However, I was worried about the message at the
top of:

https://wiki.galaxyproject.org/Admin/DataPreparation

It seemed the method written on that page has now been superseded by
Data Managers?  I was worried editing *.loc files directly with an
editor would break a database and cause more headaches for me.  Is
that not the case or perhaps I can get away with doing it just for the
FASTA files but should still use Data Managers for index generation?

Thank you!

Ray


On Thu, Feb 18, 2016 at 3:54 PM, Hans-Rudolf Hotz <[hidden email]> wrote:

> Hi Ray
>
> If you already have the reference data (as you have written in your first
> e-mail), then I suggest you manually add those files (file paths) to
>
> ~/tool-data/bowtie2_indices.loc
> ~/tool-data/bowtie_indices.loc
>
> This way you prevent duplication of your reference data
>
>
> Regards, Hans-Rudolf
>
>
>
>> Dear all,
>>
>> I have gotten a bit further and was able to install the Data Managers
>> this time.  Previously, I was having problems accessing the ToolShed
>> web site and some Python errors in the log file, but fine ... sort of
>> (I get some intermittent errors accessing the Tool Shed site; maybe
>> it's a problem on my side).
>>
>> However, with the FASTA fetch tool installed from the Galaxy Main Tool
>> Shed, I selected "Local Data" from the Administrator panel.  Under
>> "Run Data Manager Tools", I now have "Bowtie index", "Reference
>> Genome", etc.  When I click on any of the options under "Run Data
>> Manager Tools", though, I get a blank page.  I mean, I get a
>> user-level menu on the left, history on the right, a menu along the
>> top.  But the center is blank.  Is there something else I need to do?
>>
>> If I go to "View Data Manager Jobs", "Reference Genome", though, I get
>> a message that says there are "There are no jobs for this data
>> manager.".
>>
>> If possible, I would like to upload FASTA files myself to our local
>> Galaxy instance.  For reasons which I'm still working on (and which is
>> unrelated to Galaxy), download times are very long.  Since I have the
>> FASTA file already, it might be easier to upload one myself; but if
>> that's not something Data Managers can do, I guess I can just start
>> the download and set it aside...
>>
>> Thank you!
>>
>> Ray
>>
>>
>>
>> On Thu, Feb 18, 2016 at 8:36 AM, Raymond Wan <[hidden email]> wrote:
>>>
>>> Dear all,
>>>
>>> I'm a bit new at being a Galaxy administrator, so apologies for the
>>> basic question.
>>>
>>> I need to install some bowtie and bowtie2 indices.  I have the indices
>>> already but came across this page:
>>>
>>> https://wiki.galaxyproject.org/Admin/DataPreparation
>>>
>>> which says the steps mentioned on that page have been replaced with
>>> Data Managers.  I'm having problems with installing Data Managers
>>> (accessing ToolShed right now and some errors which I guess I need to
>>> sort out later) and was wondering if there was a way to bypass Data
>>> Managers?
>>>
>>> Are Data Managers the only way of installing reference data?  With the
>>> indices already pre-calculated, can I install them without having to
>>> go through re-calculating them within Galaxy?
>>>
>>> Thank you!
>>>
>>> Ray
>>
>> ___________________________________________________________
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>    https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>    http://galaxyproject.org/search/mailinglists/
>>
>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Bypassing Data Managers?

Hans-Rudolf Hotz
Hi Ray

Well, I guess, you can call me 'old-fashioned'  ;)

we still solely rely on manually editing the '*loc' files. The main
reason we do that: Most of our reference data is first prepared (by
others) for people using it on the command line, and only in a second
step we make it available via Galaxy.

Hans-Rudolf


On 02/18/2016 09:00 AM, Raymond Wan wrote:

> Hi Hans-Rudolf,
>
> Thank you for getting back to me!
>
> I did take a peak at those files and they seem self-explanatory.  I
> think I can do that.  However, I was worried about the message at the
> top of:
>
> https://wiki.galaxyproject.org/Admin/DataPreparation
>
> It seemed the method written on that page has now been superseded by
> Data Managers?  I was worried editing *.loc files directly with an
> editor would break a database and cause more headaches for me.  Is
> that not the case or perhaps I can get away with doing it just for the
> FASTA files but should still use Data Managers for index generation?
>
> Thank you!
>
> Ray
>
>
> On Thu, Feb 18, 2016 at 3:54 PM, Hans-Rudolf Hotz <[hidden email]> wrote:
>> Hi Ray
>>
>> If you already have the reference data (as you have written in your first
>> e-mail), then I suggest you manually add those files (file paths) to
>>
>> ~/tool-data/bowtie2_indices.loc
>> ~/tool-data/bowtie_indices.loc
>>
>> This way you prevent duplication of your reference data
>>
>>
>> Regards, Hans-Rudolf
>>
>>
>>
>>> Dear all,
>>>
>>> I have gotten a bit further and was able to install the Data Managers
>>> this time.  Previously, I was having problems accessing the ToolShed
>>> web site and some Python errors in the log file, but fine ... sort of
>>> (I get some intermittent errors accessing the Tool Shed site; maybe
>>> it's a problem on my side).
>>>
>>> However, with the FASTA fetch tool installed from the Galaxy Main Tool
>>> Shed, I selected "Local Data" from the Administrator panel.  Under
>>> "Run Data Manager Tools", I now have "Bowtie index", "Reference
>>> Genome", etc.  When I click on any of the options under "Run Data
>>> Manager Tools", though, I get a blank page.  I mean, I get a
>>> user-level menu on the left, history on the right, a menu along the
>>> top.  But the center is blank.  Is there something else I need to do?
>>>
>>> If I go to "View Data Manager Jobs", "Reference Genome", though, I get
>>> a message that says there are "There are no jobs for this data
>>> manager.".
>>>
>>> If possible, I would like to upload FASTA files myself to our local
>>> Galaxy instance.  For reasons which I'm still working on (and which is
>>> unrelated to Galaxy), download times are very long.  Since I have the
>>> FASTA file already, it might be easier to upload one myself; but if
>>> that's not something Data Managers can do, I guess I can just start
>>> the download and set it aside...
>>>
>>> Thank you!
>>>
>>> Ray
>>>
>>>
>>>
>>> On Thu, Feb 18, 2016 at 8:36 AM, Raymond Wan <[hidden email]> wrote:
>>>>
>>>> Dear all,
>>>>
>>>> I'm a bit new at being a Galaxy administrator, so apologies for the
>>>> basic question.
>>>>
>>>> I need to install some bowtie and bowtie2 indices.  I have the indices
>>>> already but came across this page:
>>>>
>>>> https://wiki.galaxyproject.org/Admin/DataPreparation
>>>>
>>>> which says the steps mentioned on that page have been replaced with
>>>> Data Managers.  I'm having problems with installing Data Managers
>>>> (accessing ToolShed right now and some errors which I guess I need to
>>>> sort out later) and was wondering if there was a way to bypass Data
>>>> Managers?
>>>>
>>>> Are Data Managers the only way of installing reference data?  With the
>>>> indices already pre-calculated, can I install them without having to
>>>> go through re-calculating them within Galaxy?
>>>>
>>>> Thank you!
>>>>
>>>> Ray
>>>
>>> ___________________________________________________________
>>> Please keep all replies on the list by using "reply all"
>>> in your mail client.  To manage your subscriptions to this
>>> and other Galaxy lists, please use the interface at:
>>>     https://lists.galaxyproject.org/
>>>
>>> To search Galaxy mailing lists use the unified search at:
>>>     http://galaxyproject.org/search/mailinglists/
>>>
>>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Bypassing Data Managers?

Peter van Heusden
In reply to this post by Raymond Wan-2
Hi Ray

Data Managers are designed to co-exist with manually editing the .loc files. So don't worry, edit away.

Peter

On 18 February 2016 at 10:00, Raymond Wan <[hidden email]> wrote:
Hi Hans-Rudolf,

Thank you for getting back to me!

I did take a peak at those files and they seem self-explanatory.  I
think I can do that.  However, I was worried about the message at the
top of:

https://wiki.galaxyproject.org/Admin/DataPreparation

It seemed the method written on that page has now been superseded by
Data Managers?  I was worried editing *.loc files directly with an
editor would break a database and cause more headaches for me.  Is
that not the case or perhaps I can get away with doing it just for the
FASTA files but should still use Data Managers for index generation?

Thank you!

Ray


On Thu, Feb 18, 2016 at 3:54 PM, Hans-Rudolf Hotz <[hidden email]> wrote:
> Hi Ray
>
> If you already have the reference data (as you have written in your first
> e-mail), then I suggest you manually add those files (file paths) to
>
> ~/tool-data/bowtie2_indices.loc
> ~/tool-data/bowtie_indices.loc
>
> This way you prevent duplication of your reference data
>
>
> Regards, Hans-Rudolf
>
>
>
>> Dear all,
>>
>> I have gotten a bit further and was able to install the Data Managers
>> this time.  Previously, I was having problems accessing the ToolShed
>> web site and some Python errors in the log file, but fine ... sort of
>> (I get some intermittent errors accessing the Tool Shed site; maybe
>> it's a problem on my side).
>>
>> However, with the FASTA fetch tool installed from the Galaxy Main Tool
>> Shed, I selected "Local Data" from the Administrator panel.  Under
>> "Run Data Manager Tools", I now have "Bowtie index", "Reference
>> Genome", etc.  When I click on any of the options under "Run Data
>> Manager Tools", though, I get a blank page.  I mean, I get a
>> user-level menu on the left, history on the right, a menu along the
>> top.  But the center is blank.  Is there something else I need to do?
>>
>> If I go to "View Data Manager Jobs", "Reference Genome", though, I get
>> a message that says there are "There are no jobs for this data
>> manager.".
>>
>> If possible, I would like to upload FASTA files myself to our local
>> Galaxy instance.  For reasons which I'm still working on (and which is
>> unrelated to Galaxy), download times are very long.  Since I have the
>> FASTA file already, it might be easier to upload one myself; but if
>> that's not something Data Managers can do, I guess I can just start
>> the download and set it aside...
>>
>> Thank you!
>>
>> Ray
>>
>>
>>
>> On Thu, Feb 18, 2016 at 8:36 AM, Raymond Wan <[hidden email]> wrote:
>>>
>>> Dear all,
>>>
>>> I'm a bit new at being a Galaxy administrator, so apologies for the
>>> basic question.
>>>
>>> I need to install some bowtie and bowtie2 indices.  I have the indices
>>> already but came across this page:
>>>
>>> https://wiki.galaxyproject.org/Admin/DataPreparation
>>>
>>> which says the steps mentioned on that page have been replaced with
>>> Data Managers.  I'm having problems with installing Data Managers
>>> (accessing ToolShed right now and some errors which I guess I need to
>>> sort out later) and was wondering if there was a way to bypass Data
>>> Managers?
>>>
>>> Are Data Managers the only way of installing reference data?  With the
>>> indices already pre-calculated, can I install them without having to
>>> go through re-calculating them within Galaxy?
>>>
>>> Thank you!
>>>
>>> Ray
>>
>> ___________________________________________________________
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>    https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>    http://galaxyproject.org/search/mailinglists/
>>
>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Bypassing Data Managers?

Raymond Wan-2
In reply to this post by Hans-Rudolf Hotz
Hi Hans-Rudolf,

Actually, I'm also feeling a little "old-fashioned" myself and would
prefer to edit .loc files, especially given the amount of
documentation out there.  I know Data Managers are a next step up, but
I don't think I'm smart enough to figure out why I was getting a blank
page -- perhaps a problem for another day (that I might ignore unless
it bothers users :-) ).

By the way, I just got it working -- thank you for your help!

Ray


On Thu, Feb 18, 2016 at 4:10 PM, Hans-Rudolf Hotz <[hidden email]> wrote:

> Hi Ray
>
> Well, I guess, you can call me 'old-fashioned'  ;)
>
> we still solely rely on manually editing the '*loc' files. The main reason
> we do that: Most of our reference data is first prepared (by others) for
> people using it on the command line, and only in a second step we make it
> available via Galaxy.
>
> Hans-Rudolf
>
>
>
> On 02/18/2016 09:00 AM, Raymond Wan wrote:
>>
>> Hi Hans-Rudolf,
>>
>> Thank you for getting back to me!
>>
>> I did take a peak at those files and they seem self-explanatory.  I
>> think I can do that.  However, I was worried about the message at the
>> top of:
>>
>> https://wiki.galaxyproject.org/Admin/DataPreparation
>>
>> It seemed the method written on that page has now been superseded by
>> Data Managers?  I was worried editing *.loc files directly with an
>> editor would break a database and cause more headaches for me.  Is
>> that not the case or perhaps I can get away with doing it just for the
>> FASTA files but should still use Data Managers for index generation?
>>
>> Thank you!
>>
>> Ray
>>
>>
>> On Thu, Feb 18, 2016 at 3:54 PM, Hans-Rudolf Hotz <[hidden email]> wrote:
>>>
>>> Hi Ray
>>>
>>> If you already have the reference data (as you have written in your first
>>> e-mail), then I suggest you manually add those files (file paths) to
>>>
>>> ~/tool-data/bowtie2_indices.loc
>>> ~/tool-data/bowtie_indices.loc
>>>
>>> This way you prevent duplication of your reference data
>>>
>>>
>>> Regards, Hans-Rudolf
>>>
>>>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Bypassing Data Managers?

Raymond Wan-2
In reply to this post by Peter van Heusden
Hi Peter,


On Thu, Feb 18, 2016 at 4:13 PM, Peter van Heusden <[hidden email]> wrote:
> Hi Ray
>
> Data Managers are designed to co-exist with manually editing the .loc files.
> So don't worry, edit away.


Thank you for the clarification!  That made me feel better and I
pressed ahead and just got it working.

When I first read that Data Managers involved tables, I had this
sudden fear that it meant the Postgres database and a mistake will
somehow break it.  I guess the tables refer to the various .loc text
files?

Perhaps when I have the luxury of time, I can try to figure out why
the Data Managers weren't working out for me (and also why
transferring data from UCSC can take hours or more from our Galaxy
instance's network).  For now, I'll stick with what works so that the
users are happy...  :-)

Thank you for your help!

Ray
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/