Galaxy Cleanup

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Galaxy Cleanup

Previti

Hi everybody,

I have a hopefully basic questions regarding temporary output files.

A user was generating >4TB output files via the program bedtools intersectbed. The jobs were cancelled, but I still see the (temporary) output files that were generated.

Does Galaxy clean these up? Does it do this regularly?

Would a quota prevent something like this from happening again or would the jobs have to finish before being "counted"?

Thanks and best regards,

Christopher

--
Dr. Christopher Previti
Genomics and
Proteomics Core Facility
High Throughput Sequencing (W190)
Bioinformatician

German Cancer Research Center (DKFZ)
Foundation under Public Law
Im Neuenheimer Feld 580
69120 Heidelberg
Germany
Room: B2.102 (INF580/TP3)
Phone: +49 6221 42-4661

christopher.previti@...
www.dkfz.de

Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta
VAT-ID No.: DE143293537

Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die Personen bestimmt, an die sie adressiert ist.
Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte Informationen enthalten. Sollten Sie nicht
der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den Absender und löschen Sie die Mitteilung.
Jegliche unbefugte Verwendung der Informationen in dieser Nachricht ist untersagt.



    

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
Reply | Threaded
Open this post in threaded view
|

Re: Galaxy Cleanup

Hans-Rudolf Hotz
Hi Christopher


It depends on the setting of "cleanup_job" ('always', 'onsuccess',
'never'), whether the temporary files are cleaned up or not. Though, to
be honest, I don't know, if this setting as any influence, when you
'delete' a job.

For the quotas, only dataset in the histories are counted

The temporary files are very handy for debugging, so I have 'onsuccess'
on our production server and 'never' on our development server. I just
regularly check the total size of the working directories and do the
clean up manually


Regards, Hans-Rudolf




On 04/27/2018 01:51 PM, Previti wrote:

> Hi everybody,
>
> I have a hopefully basic questions regarding temporary output files.
>
> A user was generating >4TB output files via the program bedtools
> intersectbed. The jobs were cancelled, but I still see the (temporary)
> output files that were generated.
>
> Does Galaxy clean these up? Does it do this regularly?
>
> Would a quota prevent something like this from happening again or would
> the jobs have to finish before being "counted"?
>
> Thanks and best regards,
>
> Christopher
>
> --
> *Dr. Christopher Previti*
> Genomics and Proteomics Core Facility
> High Throughput Sequencing (W190)
> Bioinformatician
>
> German Cancer Research Center (DKFZ)
> Foundation under Public Law
> Im Neuenheimer Feld 580
> 69120 Heidelberg
> Germany
> Room: B2.102 (INF580/TP3)
> Phone: +49 6221 42-4661
>
> [hidden email] <http://www.dkfz.de/>
> www.dkfz.de <http://www.dkfz.de/>
>
> Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta
> VAT-ID No.: DE143293537
>
> Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die
> Personen bestimmt, an die sie adressiert ist.
> Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte
> Informationen enthalten. Sollten Sie nicht
> der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den
> Absender und löschen Sie die Mitteilung.
> Jegliche unbefugte Verwendung der Informationen in dieser Nachricht ist
> untersagt.
>
>
>
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>    https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>    http://galaxyproject.org/search/
>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
Reply | Threaded
Open this post in threaded view
|

Re: Galaxy Cleanup

Nicola Soranzo-2
See also:

https://galaxyproject.github.io/training-material/topics/admin/tutorials/monitoring-maintenance/slides.html

Cheers,
Nicola

On 27/04/18 15:05, Hans-Rudolf Hotz wrote:

> Hi Christopher
>
>
> It depends on the setting of "cleanup_job" ('always', 'onsuccess',
> 'never'), whether the temporary files are cleaned up or not. Though,
> to be honest, I don't know, if this setting as any influence, when you
> 'delete' a job.
>
> For the quotas, only dataset in the histories are counted
>
> The temporary files are very handy for debugging, so I have
> 'onsuccess' on our production server and 'never' on our development
> server. I just regularly check the total size of the working
> directories and do the clean up manually
>
>
> Regards, Hans-Rudolf
>
>
>
>
> On 04/27/2018 01:51 PM, Previti wrote:
>> Hi everybody,
>>
>> I have a hopefully basic questions regarding temporary output files.
>>
>> A user was generating >4TB output files via the program bedtools
>> intersectbed. The jobs were cancelled, but I still see the (temporary)
>> output files that were generated.
>>
>> Does Galaxy clean these up? Does it do this regularly?
>>
>> Would a quota prevent something like this from happening again or would
>> the jobs have to finish before being "counted"?
>>
>> Thanks and best regards,
>>
>> Christopher
>>
>> --
>> *Dr. Christopher Previti*
>> Genomics and Proteomics Core Facility
>> High Throughput Sequencing (W190)
>> Bioinformatician
>>
>> German Cancer Research Center (DKFZ)
>> Foundation under Public Law
>> Im Neuenheimer Feld 580
>> 69120 Heidelberg
>> Germany
>> Room: B2.102 (INF580/TP3)
>> Phone: +49 6221 42-4661
>>
>> [hidden email] <http://www.dkfz.de/>
>> www.dkfz.de <http://www.dkfz.de/>
>>
>> Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta
>> VAT-ID No.: DE143293537
>>
>> Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die
>> Personen bestimmt, an die sie adressiert ist.
>> Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte
>> Informationen enthalten. Sollten Sie nicht
>> der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den
>> Absender und löschen Sie die Mitteilung.
>> Jegliche unbefugte Verwendung der Informationen in dieser Nachricht ist
>> untersagt.
>>
>>
>>
>>
>> ___________________________________________________________
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>    https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>    http://galaxyproject.org/search/
>>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
Reply | Threaded
Open this post in threaded view
|

Re: Galaxy Cleanup

Previti

Thanks! Super useful!

Cheers,

Christopher


On 04/27/2018 04:12 PM, Nicola Soranzo wrote:
See also:

https://galaxyproject.github.io/training-material/topics/admin/tutorials/monitoring-maintenance/slides.html

Cheers,
Nicola

On 27/04/18 15:05, Hans-Rudolf Hotz wrote:
Hi Christopher


It depends on the setting of "cleanup_job" ('always', 'onsuccess', 'never'), whether the temporary files are cleaned up or not. Though, to be honest, I don't know, if this setting as any influence, when you 'delete' a job.

For the quotas, only dataset in the histories are counted

The temporary files are very handy for debugging, so I have 'onsuccess' on our production server and 'never' on our development server. I just regularly check the total size of the working directories and do the clean up manually


Regards, Hans-Rudolf




On 04/27/2018 01:51 PM, Previti wrote:
Hi everybody,

I have a hopefully basic questions regarding temporary output files.

A user was generating >4TB output files via the program bedtools
intersectbed. The jobs were cancelled, but I still see the (temporary)
output files that were generated.

Does Galaxy clean these up? Does it do this regularly?

Would a quota prevent something like this from happening again or would
the jobs have to finish before being "counted"?

Thanks and best regards,

Christopher

-- 
*Dr. Christopher Previti*
Genomics and Proteomics Core Facility
High Throughput Sequencing (W190)
Bioinformatician

German Cancer Research Center (DKFZ)
Foundation under Public Law
Im Neuenheimer Feld 580
69120 Heidelberg
Germany
Room: B2.102 (INF580/TP3)
Phone: +49 6221 42-4661

[hidden email] <http://www.dkfz.de/>
www.dkfz.de <http://www.dkfz.de/>

Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta
VAT-ID No.: DE143293537

Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die
Personen bestimmt, an die sie adressiert ist.
Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte
Informationen enthalten. Sollten Sie nicht
der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den
Absender und löschen Sie die Mitteilung.
Jegliche unbefugte Verwendung der Informationen in dieser Nachricht ist
untersagt.




___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/


--
Dr. Christopher Previti
Genomics and
Proteomics Core Facility
High Throughput Sequencing (W190)
Bioinformatician

German Cancer Research Center (DKFZ)
Foundation under Public Law
Im Neuenheimer Feld 580
69120 Heidelberg
Germany
Room: B2.102 (INF580/TP3)
Phone: +49 6221 42-4661

christopher.previti@...
www.dkfz.de

Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta
VAT-ID No.: DE143293537

Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die Personen bestimmt, an die sie adressiert ist.
Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte Informationen enthalten. Sollten Sie nicht
der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den Absender und löschen Sie die Mitteilung.
Jegliche unbefugte Verwendung der Informationen in dieser Nachricht ist untersagt.



    

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/