Updating tools on new hg based Galaxy Tool Shed

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Updating tools on new hg based Galaxy Tool Shed

Peter Cock
Hi all,

I just tried updating one of my tools on the new hg based Galaxy Tool
Shed and ran into some issues. I guess I'm the first person to try
this...

First of all, there is no clear "update" action like there used to me.
Could that be restored please?
https://bitbucket.org/galaxy/galaxy-central/issue/585/hg-tool-shed-lacks-update-tool-action

I tried using the "upload files to repository" option, and picked my
tarball. I did not pick a location since I wanted the tar ball
unpacked at the repository root, but this isn't what happened:
https://bitbucket.org/galaxy/galaxy-central/issue/586/hg-tool-shed-upload-files-to-repository

Finally, it seems there is currently no delete files action, so I
can't fix this (delete the misplaced files) via the web interface.
https://bitbucket.org/galaxy/galaxy-central/issue/584/cant-delete-files-in-new-hg-tool-shed

Are you expecting tool authors to work primarily at the hg level?

Regards,

Peter
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/
Reply | Threaded
Open this post in threaded view
|

Re: Updating tools on new hg based Galaxy Tool Shed

Greg Von Kuster
Peter, 

I've added comments to each of these issues in bitbucket, and have pated them inline below as well.

On Jun 16, 2011, at 5:03 AM, Peter Cock wrote:

Hi all,

I just tried updating one of my tools on the new hg based Galaxy Tool
Shed and ran into some issues. I guess I'm the first person to try
this...

First of all, there is no clear "update" action like there used to me.
Could that be restored please?
https://bitbucket.org/galaxy/galaxy-central/issue/585/hg-tool-shed-lacks-update-tool-action

The requested functionality is currently possible using normal mercurial features ( clone -> make changes to the cloned repo -> commit changes to the cloned repo -> push changes to the tool shed repo ).

It is also currently possible to upload a new tarball to any "upload point" (position) in the tool shed repository by selecting the upload point in the built-in repository file browser. - The hierarchy that the uploaded tarball contains will be added to that upload point in the tool shed repository. - Any files in the uploaded tarball that do not currently exist in the tool shed repository will be added. - Any files that exist in the hierarchy of the uploaded tarball that also exist in the same location of the hierarchy of the repository relative to the upload point will be replaced in the repository.

It is on my list to: Delete any files that do not exist in the uploaded tarball's hierarchy but that do existin the same position of the tool shed repository's hierarchy relative to the upload point. This feature will require clear documented communication to the user, which I've not yet had a chance to do, but will soon.



I tried using the "upload files to repository" option, and picked my
tarball. I did not pick a location since I wanted the tar ball
unpacked at the repository root, but this isn't what happened:
https://bitbucket.org/galaxy/galaxy-central/issue/586/hg-tool-shed-upload-files-to-repository

Peter, I believe this is a duplicate of # 584, correct?


Finally, it seems there is currently no delete files action, so I
can't fix this (delete the misplaced files) via the web interface.
https://bitbucket.org/galaxy/galaxy-central/issue/584/cant-delete-files-in-new-hg-tool-shed

You can currently delete specific files by cloning the repository, making any changes you want (including deleting), and committing and pushing the changes using mercurial's command line.

I've added it to my list to provide a feature allowing a user to select a specific file in the repository (hopefully using the built-in file browser) and deleting the file in some way that doesn't require mercurial's command line, but in the meantime, use the above existing feature.



Are you expecting tool authors to work primarily at the hg level?

Regards,

Peter

Greg Von Kuster
Galaxy Development Team




___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/
Reply | Threaded
Open this post in threaded view
|

Re: Updating tools on new hg based Galaxy Tool Shed

Peter Cock
On Thu, Jun 16, 2011 at 2:45 PM, Greg Von Kuster <[hidden email]> wrote:

> Peter,
> I've added comments to each of these issues in bitbucket, and have pated
> them inline below as well.
>> On Jun 16, 2011, at 5:03 AM, Peter Cock wrote:
>>
>> Hi all,
>>
>> I just tried updating one of my tools on the new hg based Galaxy Tool
>> Shed and ran into some issues. I guess I'm the first person to try
>> this...
>>
>> First of all, there is no clear "update" action like there used to me.
>> Could that be restored please?
>> https://bitbucket.org/galaxy/galaxy-central/issue/585/hg-tool-shed-lacks-update-tool-action
>
> The requested functionality is currently possible using normal mercurial
> features ( clone -> make changes to the cloned repo -> commit changes to the
> cloned repo -> push changes to the tool shed repo ).

Right, but not everyone wants to use hg for this.

> It is also currently possible to upload a new tarball to any "upload point"
> (position) in the tool shed repository by selecting the upload point in the
> built-in repository file browser. - The hierarchy that the uploaded tarball
> contains will be added to that upload point in the tool shed repository. -
> Any files in the uploaded tarball that do not currently exist in the tool
> shed repository will be added. - Any files that exist in the hierarchy of
> the uploaded tarball that also exist in the same location of the hierarchy
> of the repository relative to the upload point will be replaced in the
> repository.

Yes, I tried that and its broken (bug 586 below).

>> It is on my list to: Delete any files that do not exist in the uploaded
>> tarball's hierarchy but that do existin the same position of the tool shed
>> repository's hierarchy relative to the upload point. This feature will
>> require clear documented communication to the user, which I've not yet had a
>> chance to do, but will soon.

That would be a change from "upload files" to "replace all the files",
in line with the missing update tool functionality (#585). To me mixing
these two is quite strange.

>> I tried using the "upload files to repository" option, and picked my
>> tarball. I did not pick a location since I wanted the tar ball
>> unpacked at the repository root, but this isn't what happened:
>> https://bitbucket.org/galaxy/galaxy-central/issue/586/hg-tool-shed-upload-files-to-repository
>
> Peter, I believe this is a duplicate of # 584, correct?

Not really, 584 is about deleting files via the web interface.

Perhaps you regard 585 (missing tool update) and 586 (broken upload
at root) as the same basic issue?

>> Finally, it seems there is currently no delete files action, so I
>> can't fix this (delete the misplaced files) via the web interface.
>> https://bitbucket.org/galaxy/galaxy-central/issue/584/cant-delete-files-in-new-hg-tool-shed
>
> You can currently delete specific files by cloning the repository, making
> any changes you want (including deleting), and committing and pushing the
> changes using mercurial's command line.

Again, I'd prefer not to have to do this.

> I've added it to my list to provide a feature allowing a user to select a
> specific file in the repository (hopefully using the built-in file browser)
> and deleting the file in some way that doesn't require mercurial's command
> line, but in the meantime, use the above existing feature.
>
>>
>> Are you expecting tool authors to work primarily at the hg level?
>>

You didn't answer that one ;)

Peter
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/
Reply | Threaded
Open this post in threaded view
|

Re: Updating tools on new hg based Galaxy Tool Shed

Greg Von Kuster
Hi Peter,

On Jun 16, 2011, at 10:10 AM, Peter Cock wrote:

> On Thu, Jun 16, 2011 at 2:45 PM, Greg Von Kuster <[hidden email]> wrote:
>> Peter,
>> I've added comments to each of these issues in bitbucket, and have pated
>> them inline below as well.
>>> On Jun 16, 2011, at 5:03 AM, Peter Cock wrote:
>>>
>>> Hi all,
>>>
>>> I just tried updating one of my tools on the new hg based Galaxy Tool
>>> Shed and ran into some issues. I guess I'm the first person to try
>>> this...
>>>
>>> First of all, there is no clear "update" action like there used to me.
>>> Could that be restored please?
>>> https://bitbucket.org/galaxy/galaxy-central/issue/585/hg-tool-shed-lacks-update-tool-action
>>
>> The requested functionality is currently possible using normal mercurial
>> features ( clone -> make changes to the cloned repo -> commit changes to the
>> cloned repo -> push changes to the tool shed repo ).
>
> Right, but not everyone wants to use hg for this.


I understand, and that is why I am working feverishly to add features that do not force the use of hg form the command line (I've got several features functional, but not everything yet).  The current tool shed requires some use of hg if you want to delete files from the repo, but other than that, hg is not really required.  

The current tool shed also provides some very useful features (e.g., browsing tool code files) that were not available in the old tool shed.  In addition, tool developers are no longer forced to to the extra work that was necessary in the old tool shed to add a tool (i.e., there is no longer a requirement for tool suite config definitions ).  My hope is that the current features make everyone happy enough that they will be able to wait for those messing featuers that are wanted, using the current features in the meantime.


>
>> It is also currently possible to upload a new tarball to any "upload point"
>> (position) in the tool shed repository by selecting the upload point in the
>> built-in repository file browser. - The hierarchy that the uploaded tarball
>> contains will be added to that upload point in the tool shed repository. -
>> Any files in the uploaded tarball that do not currently exist in the tool
>> shed repository will be added. - Any files that exist in the hierarchy of
>> the uploaded tarball that also exist in the same location of the hierarchy
>> of the repository relative to the upload point will be replaced in the
>> repository.
>
> Yes, I tried that and its broken (bug 586 below).


This is because you uploaded a gzipped tarball, which mercurial treats like a single file.  The current tool shed is based on mercurial, so mercurial's behavior is basically the tool shed's behavior.  It is on my list to determine if a file is compressed and if so uncompress it upon upload.  A potential problem with this is that some developers may not want their uploaded file uncompressed.


>
>>> It is on my list to: Delete any files that do not exist in the uploaded
>>> tarball's hierarchy but that do existin the same position of the tool shed
>>> repository's hierarchy relative to the upload point. This feature will
>>> require clear documented communication to the user, which I've not yet had a
>>> chance to do, but will soon.
>
> That would be a change from "upload files" to "replace all the files",
> in line with the missing update tool functionality (#585). To me mixing
> these two is quite strange.


This should become more clear when the feature to delete files in the hierarchy is added.  


>
>>> I tried using the "upload files to repository" option, and picked my
>>> tarball. I did not pick a location since I wanted the tar ball
>>> unpacked at the repository root, but this isn't what happened:
>>> https://bitbucket.org/galaxy/galaxy-central/issue/586/hg-tool-shed-upload-files-to-repository
>>
>> Peter, I believe this is a duplicate of # 584, correct?
>
> Not really, 584 is about deleting files via the web interface.
>
> Perhaps you regard 585 (missing tool update) and 586 (broken upload
> at root) as the same basic issue?


See above explanation.


>
>>> Finally, it seems there is currently no delete files action, so I
>>> can't fix this (delete the misplaced files) via the web interface.
>>> https://bitbucket.org/galaxy/galaxy-central/issue/584/cant-delete-files-in-new-hg-tool-shed
>>
>> You can currently delete specific files by cloning the repository, making
>> any changes you want (including deleting), and committing and pushing the
>> changes using mercurial's command line.
>
> Again, I'd prefer not to have to do this.

Understood...


>
>> I've added it to my list to provide a feature allowing a user to select a
>> specific file in the repository (hopefully using the built-in file browser)
>> and deleting the file in some way that doesn't require mercurial's command
>> line, but in the meantime, use the above existing feature.
>>
>>>
>>> Are you expecting tool authors to work primarily at the hg level?
>>>
>
> You didn't answer that one ;)


Not necessarily, but hg is the basis for uploading and downloading tools.  I'm not sure if it will be possible to completely eliminate the requirement of a tool developer using hg, but we're making every attempt to do so.  Why are you so averse to using hg?


>
> Peter

Greg Von Kuster
Galaxy Development Team
[hidden email]




___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/
Reply | Threaded
Open this post in threaded view
|

Re: Updating tools on new hg based Galaxy Tool Shed

Peter Cock
On Thu, Jun 16, 2011 at 3:26 PM, Greg Von Kuster <[hidden email]> wrote:
> Hi Peter,
>
> I understand, and that is why I am working feverishly to add features
> that do not force the use of hg form the command line (I've got several
> features functional, but not everything yet).  The current tool shed requires
> some use of hg if you want to delete files from the repo, but other than that,
> hg is not really required.

That's great, but I do feel the new hg went live a bit prematurely.

> The current tool shed also provides some very useful features (e.g.,
> browsing tool code files) that were not available in the old tool shed.

Really? I find the browsing the tool code files has regressed.

In old tool shed had a simple non-interactive tree of the files
visible on the main page for a tool or suite, where the individual files
could be clicked on to view/download. It was simple and effective
(at least when there weren't many files which seems typical).

The new tool shed makes you click on tool actions, browse repo,
and then gives an interactive tree which is fully collapsed by default,
forcing lots more clicking to drill down to the files. All that clicking
makes it slow and fiddly to use.

> In addition, tool developers are no longer forced to to the extra work
> that was necessary in the old tool shed to add a tool (i.e., there is no
> longer a requirement for tool suite config definitions ).

For tool suites you have a point, that was a bit of a hassle.

> My hope is that the current features make everyone happy
> enough that they will be able to wait for those messing featuers
> that are wanted, using the current features in the meantime.

It'll get there :)

Another big regression I would prioritise is the complete lack of
any user provided text on a tool's main page. The old tool shed
had the minimal text box where you could summarise the tool,
its requirements, recent changes etc. The new tool shed has
nothing to replace this as far as I can see.

A simple wiki like, ReST, or just plain text details box would do.

Automatic detection of a readme text file in the repo, to be
displayed on the tools' main page would be an elegant solution
(are you familiar with how github.com does this?). It also keeps
the description in the hg repo which is a plus point.

The mock up idea would be another (more complex) way to
tackle this, https://bitbucket.org/galaxy/galaxy-central/issue/565

Peter

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/
Reply | Threaded
Open this post in threaded view
|

Re: Updating tools on new hg based Galaxy Tool Shed

Peter Cock
In reply to this post by Greg Von Kuster
On Thu, Jun 16, 2011 at 3:26 PM, Greg Von Kuster <[hidden email]> wrote:

> Hi Peter,
>
> On Jun 16, 2011, at 10:10 AM, Peter Cock wrote:
>>>>
>>>> Are you expecting tool authors to work primarily at the hg level?
>>>>
>>
>> You didn't answer that one ;)
>
> Not necessarily, but hg is the basis for uploading and downloading tools.
> I'm not sure if it will be possible to completely eliminate the requirement
> of a tool developer using hg, but we're making every attempt to do so.

Good.

>
> Why are you so averse to using hg?
>

Because git suits me better? ;)

Seriously, I have no strong aversion to hg - what puts me off a little is
the need to have one repo per tool or tool-suite, compared to my current
setup where all by tools are in a branch from the main Galaxy repo.
There would be a significant time and effort cost in switching, made
worse by having multiple tools on the Tool Shed.

I'm actually thinking of this (requiring hg knowledge) as a more general
issue, namely a potential impediment to new Tool Shed contributors.

Peter

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/
Reply | Threaded
Open this post in threaded view
|

Re: Updating tools on new hg based Galaxy Tool Shed

Fields, Christopher J
On Jun 16, 2011, at 10:23 AM, Peter Cock wrote:

> On Thu, Jun 16, 2011 at 3:26 PM, Greg Von Kuster <[hidden email]> wrote:
>> Hi Peter,
>>
>> On Jun 16, 2011, at 10:10 AM, Peter Cock wrote:
>>>>>
>>>>> Are you expecting tool authors to work primarily at the hg level?
>>>>>
>>>
>>> You didn't answer that one ;)
>>
>> Not necessarily, but hg is the basis for uploading and downloading tools.
>>  I'm not sure if it will be possible to completely eliminate the requirement
>> of a tool developer using hg, but we're making every attempt to do so.
>
> Good.
>
>>
>> Why are you so averse to using hg?
>>
>
> Because git suits me better? ;)
>
> Seriously, I have no strong aversion to hg - what puts me off a little is
> the need to have one repo per tool or tool-suite, compared to my current
> setup where all by tools are in a branch from the main Galaxy repo.
> There would be a significant time and effort cost in switching, made
> worse by having multiple tools on the Tool Shed.
>
> I'm actually thinking of this (requiring hg knowledge) as a more general
> issue, namely a potential impediment to new Tool Shed contributors.
>
> Peter

I'm of the opposite mind; I'm not sure there is an absolute need to have one's tools be a branch or fork of the main tool shed, though I agree there is also utility in allowing that (having a customized 'main' repo, or optimizing tools already present).  Having completely separate repos for tools seems cleaner, focusing development on those tools alone (not any of the others present in a branch) and pushes maintenance of the tool back to the tool developer themselves (e.g. there is no need for the galaxy devs to 'pull' in changes at any point into a main repo).  This might also allow the galaxy devs to designate a set of 'blessed' or supported tools in various repositories.

Re: git: as Peter knows I'm also primarily a git/github user.  It is feasible at some future point to allow git/github repos.  For instance, one could possibly integrate github usage via this:

http://hg-git.github.com/

Of course, I think it's much more important that any additional vcs integration wait until the new tool shed interface stabilizes somewhat, but (at least from the github perspective) seems like it shouldn't be terribly hard to do.  

chris


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/
Reply | Threaded
Open this post in threaded view
|

Re: Updating tools on new hg based Galaxy Tool Shed

Peter Cock
On Thu, Jun 16, 2011 at 5:40 PM, Chris Fields <[hidden email]> wrote:
>
> I'm of the opposite mind; I'm not sure there is an absolute need to
> have one's tools be a branch or fork of the main tool shed, though
> I agree there is also utility in allowing that (having a customized
> 'main' repo, or optimizing tools already present).

To clarify, I have been using branches on an hg fork of the main
Galaxy repository up until now, then preparing tarballs for upload
to the Galaxy Tool Shed. I could equally have tracked my local
changes under git, svn or cvs. The point is under this model, the
Tool Shed has no connection to whatever repository (or repositories)
the tool author uses on their machine.

Barring teething issues like Bug 584/585/586 tool authors could
continue to use this development model, where the fact that
the Galaxy Tool Shed uses an hg repository internally is an
irrelevant internal implementation detail.

As far as I know, there are no plans to make the Tool Shed work
directly with a full Galaxy hg repo - only mini-repositories, one
for each tool or tool suite.

> Having completely
> separate repos for tools seems cleaner, focusing development on
> those tools alone (not any of the others present in a branch) and
> pushes maintenance of the tool back to the tool developer themselves

Yes, and that is the model the new Galaxy Tool Shed is following.
Although I'm a little hazy on the details of tool suites in the new model
(and there are lots of situations where it makes sense to bundle
several tools).

> (e.g. there is no need for the galaxy devs to 'pull' in changes at any
> point into a main repo).

Well, there still is for general bug fixes, adding new file formats, and
merging any tools which they decide to include in the core set.

> This might also allow the galaxy devs to designate a set of
> 'blessed' or supported tools in various repositories.

Indeed.

> Re: git: as Peter knows I'm also primarily a git/github user.  It is
> feasible at some future point to allow git/github repos.  For instance,
> one could possibly integrate github usage via this:
>
> http://hg-git.github.com/
>
> Of course, I think it's much more important that any additional
> vcs integration wait until the new tool shed interface stabilizes
> somewhat, but (at least from the github perspective) seems like
> it shouldn't be terribly hard to do.

True - if there were sufficient demand from Tool Shed contributors.

Peter

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/