concatenate_files input

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

concatenate_files input

Nicolas Allias
Hi everybody,

First, many thnaks for all your great job !

I'm using concatenate_files, giving a filist as input
$;INPUT_FILE_LIST$; = filelist

For convenience, I'm wondering if there is a possibility to give it a
comma separated list of files, like
$;INPUT_FILES$;= file1,file2,file3

Thanks by advance,
nicolallias

------------------------------------------------------------------------------
Special Offer-- Download ArcSight Logger for FREE (a $49 USD value)!
Finally, a world-class log management solution at an even better price-free!
Download using promo code Free_Logger_4_Dev2Dev. Offer expires
February 28th, so secure your free ArcSight Logger TODAY!
http://p.sf.net/sfu/arcsight-sfd2d
_______________________________________________
Ergatis-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/ergatis-users
Reply | Threaded
Open this post in threaded view
|

Re: concatenate_files input

Joshua Orvis
Nicolas,

Currently comma-separated values are already supported in each of INPUT_FILE_LIST, INPUT_FILE and INPUT_DIRECTORY.  In the past I think they've only been used when people are programmatically creating their config files or editing them by hand, but I think it might work if you're using the interface and use the comma list when defining your input at the top of the pipeline creation form.  This is less than ideal, because there's no way to choose multiple outputs in a visual way, but I think at least that should work already.

Joshua




On Thu, Feb 3, 2011 at 9:09 AM, Nicolas Allias <[hidden email]> wrote:
Hi everybody,

First, many thnaks for all your great job !

I'm using concatenate_files, giving a filist as input
$;INPUT_FILE_LIST$; = filelist

For convenience, I'm wondering if there is a possibility to give it a
comma separated list of files, like
$;INPUT_FILES$;= file1,file2,file3

Thanks by advance,
nicolallias

------------------------------------------------------------------------------
Special Offer-- Download ArcSight Logger for FREE (a $49 USD value)!
Finally, a world-class log management solution at an even better price-free!
Download using promo code Free_Logger_4_Dev2Dev. Offer expires
February 28th, so secure your free ArcSight Logger TODAY!
http://p.sf.net/sfu/arcsight-sfd2d
_______________________________________________
Ergatis-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/ergatis-users


------------------------------------------------------------------------------
Special Offer-- Download ArcSight Logger for FREE (a $49 USD value)!
Finally, a world-class log management solution at an even better price-free!
Download using promo code Free_Logger_4_Dev2Dev. Offer expires
February 28th, so secure your free ArcSight Logger TODAY!
http://p.sf.net/sfu/arcsight-sfd2d
_______________________________________________
Ergatis-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/ergatis-users
Reply | Threaded
Open this post in threaded view
|

Re: concatenate_files input

Nicolas Allias
Hi Joshua,

In fact, it's not well supported.
Giving $;INPUT_FILE_LIST$; = file1,file2 the output file is empty (file1
and file2 both exist and are not empty)

The command executed is (according to the view_component.cgi)
for i in `cat file1,file2`; do cat $i; done >> file.out

launched in a shell, it gives an error, as file1,file2 doesn't exists

Nicolas

Joshua Orvis wrote:

> Nicolas,
>
> Currently comma-separated values are already supported in each of
> INPUT_FILE_LIST, INPUT_FILE and INPUT_DIRECTORY.  In the past I think
> they've only been used when people are programmatically creating their
> config files or editing them by hand, but I think it might work if
> you're using the interface and use the comma list when defining your
> input at the top of the pipeline creation form.  This is less than
> ideal, because there's no way to choose multiple outputs in a visual
> way, but I think at least that should work already.
>
> Joshua
>
>
>
>
> On Thu, Feb 3, 2011 at 9:09 AM, Nicolas Allias
> <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Hi everybody,
>
>     First, many thnaks for all your great job !
>
>     I'm using concatenate_files, giving a filist as input
>     $;INPUT_FILE_LIST$; = filelist
>
>     For convenience, I'm wondering if there is a possibility to give it a
>     comma separated list of files, like
>     $;INPUT_FILES$;= file1,file2,file3
>
>     Thanks by advance,
>     nicolallias
>
>     _______________________________________________
>     Ergatis-users mailing list
>     [hidden email]
>     <mailto:[hidden email]>
>     https://lists.sourceforge.net/lists/listinfo/ergatis-users
>
> _______________________________________________
> Ergatis-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/ergatis-users
>  


------------------------------------------------------------------------------
Special Offer-- Download ArcSight Logger for FREE (a $49 USD value)!
Finally, a world-class log management solution at an even better price-free!
Download using promo code Free_Logger_4_Dev2Dev. Offer expires
February 28th, so secure your free ArcSight Logger TODAY!
http://p.sf.net/sfu/arcsight-sfd2d
_______________________________________________
Ergatis-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/ergatis-users
Reply | Threaded
Open this post in threaded view
|

Re: concatenate_files input

Nicolas Allias
I forgot a precision, I want to give a comma-list of files to
concatenate, not a comma-list of filelist.
The component seems not to be written in other way than receive filelist(s).
In the example, let's say that file1 and file2 are fastq and the result
file.out will be another fastq.

Nicolas Allias wrote:

> Hi Joshua,
>
> In fact, it's not well supported.
> Giving $;INPUT_FILE_LIST$; = file1,file2 the output file is empty (file1
> and file2 both exist and are not empty)
>
> The command executed is (according to the view_component.cgi)
> for i in `cat file1,file2`; do cat $i; done >> file.out
>
> launched in a shell, it gives an error, as file1,file2 doesn't exists
>
> Nicolas
>
> Joshua Orvis wrote:
>  
>> Nicolas,
>>
>> Currently comma-separated values are already supported in each of
>> INPUT_FILE_LIST, INPUT_FILE and INPUT_DIRECTORY.  In the past I think
>> they've only been used when people are programmatically creating their
>> config files or editing them by hand, but I think it might work if
>> you're using the interface and use the comma list when defining your
>> input at the top of the pipeline creation form.  This is less than
>> ideal, because there's no way to choose multiple outputs in a visual
>> way, but I think at least that should work already.
>>
>> Joshua
>>
>>
>>
>>
>> On Thu, Feb 3, 2011 at 9:09 AM, Nicolas Allias
>> <[hidden email]
>> <mailto:[hidden email]>> wrote:
>>
>>     Hi everybody,
>>
>>     First, many thnaks for all your great job !
>>
>>     I'm using concatenate_files, giving a filist as input
>>     $;INPUT_FILE_LIST$; = filelist
>>
>>     For convenience, I'm wondering if there is a possibility to give it a
>>     comma separated list of files, like
>>     $;INPUT_FILES$;= file1,file2,file3
>>
>>     Thanks by advance,
>>     nicolallias
>>
>>     _______________________________________________
>>     Ergatis-users mailing list
>>     [hidden email]
>>     <mailto:[hidden email]>
>>     https://lists.sourceforge.net/lists/listinfo/ergatis-users
>>
>> _______________________________________________
>> Ergatis-users mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/ergatis-users
>>  
> _______________________________________________
> Ergatis-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/ergatis-users
>
>  


------------------------------------------------------------------------------
Special Offer-- Download ArcSight Logger for FREE (a $49 USD value)!
Finally, a world-class log management solution at an even better price-free!
Download using promo code Free_Logger_4_Dev2Dev. Offer expires
February 28th, so secure your free ArcSight Logger TODAY!
http://p.sf.net/sfu/arcsight-sfd2d
_______________________________________________
Ergatis-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/ergatis-users
Reply | Threaded
Open this post in threaded view
|

Re: concatenate_files input

Joshua Orvis
In reply to this post by Nicolas Allias
Nicolas -

I'm sorry, for some reason I completely missed which component you were trying to do this on.  What I said before applies to the iterative components (those that use Ergatis' built-in iterator mechanisms).  I hadn't yet looked at 'concatenate_files' and didn't realize it wasn't this type. 

If you want to modify the component so that both types work feel free to post it, else someone here can try to look at it as soon as we can.

Joshua



On Thu, Feb 3, 2011 at 9:48 AM, Nicolas Allias <[hidden email]> wrote:
Hi Joshua,

In fact, it's not well supported.
Giving $;INPUT_FILE_LIST$; = file1,file2 the output file is empty (file1
and file2 both exist and are not empty)

The command executed is (according to the view_component.cgi)
for i in `cat file1,file2`; do cat $i; done >> file.out

launched in a shell, it gives an error, as file1,file2 doesn't exists

Nicolas

Joshua Orvis wrote:
> Nicolas,
>
> Currently comma-separated values are already supported in each of
> INPUT_FILE_LIST, INPUT_FILE and INPUT_DIRECTORY.  In the past I think
> they've only been used when people are programmatically creating their
> config files or editing them by hand, but I think it might work if
> you're using the interface and use the comma list when defining your
> input at the top of the pipeline creation form.  This is less than
> ideal, because there's no way to choose multiple outputs in a visual
> way, but I think at least that should work already.
>
> Joshua
>
>
>
>
> On Thu, Feb 3, 2011 at 9:09 AM, Nicolas Allias
> <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Hi everybody,
>
>     First, many thnaks for all your great job !
>
>     I'm using concatenate_files, giving a filist as input
>     $;INPUT_FILE_LIST$; = filelist
>
>     For convenience, I'm wondering if there is a possibility to give it a
>     comma separated list of files, like
>     $;INPUT_FILES$;= file1,file2,file3
>
>     Thanks by advance,
>     nicolallias
>
>     _______________________________________________
>     Ergatis-users mailing list
>     [hidden email]
>     <mailto:[hidden email]>
>     https://lists.sourceforge.net/lists/listinfo/ergatis-users
>
> _______________________________________________
> Ergatis-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/ergatis-users
>


------------------------------------------------------------------------------
Special Offer-- Download ArcSight Logger for FREE (a $49 USD value)!
Finally, a world-class log management solution at an even better price-free!
Download using promo code Free_Logger_4_Dev2Dev. Offer expires
February 28th, so secure your free ArcSight Logger TODAY!
http://p.sf.net/sfu/arcsight-sfd2d
_______________________________________________
Ergatis-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/ergatis-users


------------------------------------------------------------------------------
Special Offer-- Download ArcSight Logger for FREE (a $49 USD value)!
Finally, a world-class log management solution at an even better price-free!
Download using promo code Free_Logger_4_Dev2Dev. Offer expires
February 28th, so secure your free ArcSight Logger TODAY!
http://p.sf.net/sfu/arcsight-sfd2d
_______________________________________________
Ergatis-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/ergatis-users
Reply | Threaded
Open this post in threaded view
|

Re: concatenate_files input

Nicolas Allias
Hi all,
Thanks for your time, Joshua, I've written a draft - attached files -
and I'd like some advises to improve it.
Nicolas

Joshua Orvis wrote:

> Nicolas -
>
> I'm sorry, for some reason I completely missed which component you
> were trying to do this on.  What I said before applies to the
> iterative components (those that use Ergatis' built-in iterator
> mechanisms).  I hadn't yet looked at 'concatenate_files' and didn't
> realize it wasn't this type.
>
> If you want to modify the component so that both types work feel free
> to post it, else someone here can try to look at it as soon as we can.
>
> Joshua
>
>
>
> On Thu, Feb 3, 2011 at 9:48 AM, Nicolas Allias
> <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Hi Joshua,
>
>     In fact, it's not well supported.
>     Giving $;INPUT_FILE_LIST$; = file1,file2 the output file is empty
>     (file1
>     and file2 both exist and are not empty)
>
>     The command executed is (according to the view_component.cgi)
>     for i in `cat file1,file2`; do cat $i; done >> file.out
>
>     launched in a shell, it gives an error, as file1,file2 doesn't exists
>
>     Nicolas
>
>     Joshua Orvis wrote:
>     > Nicolas,
>     >
>     > Currently comma-separated values are already supported in each of
>     > INPUT_FILE_LIST, INPUT_FILE and INPUT_DIRECTORY.  In the past I
>     think
>     > they've only been used when people are programmatically creating
>     their
>     > config files or editing them by hand, but I think it might work if
>     > you're using the interface and use the comma list when defining your
>     > input at the top of the pipeline creation form.  This is less than
>     > ideal, because there's no way to choose multiple outputs in a visual
>     > way, but I think at least that should work already.
>     >
>     > Joshua
>     >
>     >
>     >
>     >
>     > On Thu, Feb 3, 2011 at 9:09 AM, Nicolas Allias
>     > <[hidden email]
>     <mailto:[hidden email]>
>     > <mailto:[hidden email]
>     <mailto:[hidden email]>>> wrote:
>     >
>     >     Hi everybody,
>     >
>     >     First, many thnaks for all your great job !
>     >
>     >     I'm using concatenate_files, giving a filist as input
>     >     $;INPUT_FILE_LIST$; = filelist
>     >
>     >     For convenience, I'm wondering if there is a possibility to
>     give it a
>     >     comma separated list of files, like
>     >     $;INPUT_FILES$;= file1,file2,file3
>     >
>     >     Thanks by advance,
>     >     nicolallias
>
> _______________________________________________
> Ergatis-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/ergatis-users
>  


[interface]
classification = utility

[input]
;; One or more (comma separated list) filelists.  Will cat each file in the list(s).
$;INPUT_FILES$;=
;; One or more (comma separated list) files.  Will cat each file.
$;INPUT_FILE$;=

[output]
$;OUTPUT_TOKEN$; = default
$;OUTPUT_DIRECTORY$; = $;REPOSITORY_ROOT$;/output_repository/$;COMPONENT_NAME$;/$;PIPELINEID$;_$;OUTPUT_TOKEN$;
$;OUTPUT_FILE$; = $;OUTPUT_DIRECTORY$;/$;OUTPUT_TOKEN$;.out

[component]
$;COMPONENT_NAME$; = concatenate_files
$;DESCRIPTION$; = Concanate the content of all files in an input file list into a single output file
$;WORKFLOW_REPOSITORY$; = $;REPOSITORY_ROOT$;/workflow/runtime/$;COMPONENT_NAME$;/$;PIPELINEID$;_$;OUTPUT_TOKEN$;
$;PIPELINE_TOKEN$; = unnamed

;The version,revision,tag here is set by an interpolated CVS tag
$;VERSION$; = 0.0
$;RELEASE_TAG$; = $Name$
$;REVISION$; = $Revision: 4680 $

$;TEMPLATE_XML$; = $;DOCS_DIR$;/$;COMPONENT_NAME$;.xml

;the following keys are replaced at runtime by the invocation script
$;COMPONENT_CONFIG$; =
$;COMPONENT_XML$; =
$;PIPELINE_XML$; =
$;PIPELINEID$; =

[include]
$;PROJECT_CONFIG$;=
<?xml version="1.0" encoding="UTF-8"?>

<commandSetRoot xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation='commandSet.xsd'>
    <commandSet type="serial">
        <state>incomplete</state>
        <name>concatenate_files workflow</name>
        <!--Processing-->
        <command>
            <type>RunUnixCommand</type>
            <name>create output directory</name>
            <state>incomplete</state>
            <executable>mkdir</executable>
            <arg>-p -m 777 $;OUTPUT_DIRECTORY$;</arg>
        </command>
        <command>
            <type>RunUnixCommand</type>
            <name>create temp directory</name>
            <state>incomplete</state>
            <executable>mkdir</executable>
            <arg>-p -m 777 $;TMP_DIR$;</arg>
        </command>        
        <command>
            <type>RunUnixCommand</type>
            <name>concatenate_files</name>
            <state>incomplete</state>
            <executable>if [ -n "$;INPUT_FILES$;" ] ; then for i in $(cat $(echo $;INPUT_FILES$; | tr "," " ")); do cat $i; done >> $;OUTPUT_FILE$; ; fi ; if [ -n "$;INPUT_FILE$;" ] ; then cat $(echo $;INPUT_FILE$; | tr "," " ")  >> $;OUTPUT_FILE$; ; fi</executable>
            <param>
                    <key>stderr</key>
                    <value>$;TMP_DIR$;/cat.stderr</value>
                </param>
        </command>
    </commandSet>
</commandSetRoot>
------------------------------------------------------------------------------
The modern datacenter depends on network connectivity to access resources
and provide services. The best practices for maximizing a physical server's
connectivity to a physical network are well understood - see how these
rules translate into the virtual world?
http://p.sf.net/sfu/oracle-sfdevnlfb
_______________________________________________
Ergatis-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/ergatis-users