job_working_directory: no subdirectories created

classic Classic list List threaded Threaded
1 message Options
D K
Reply | Threaded
Open this post in threaded view
|

job_working_directory: no subdirectories created

D K
Hi Galaxy-devs,

I recently ran into odd behavior when jobs were submitted at the same time to our cluster (jobs wouldn't return correctly, e.g. ). I changed "cleanup_job" in galaxy.ini to "never" to see what was going on in my job_working_directory (/scratch/galaxy/job_working/).

It looks like subdirectories are no longer being created in my parent job_working_directory (e.g. the tool_script.sh along with the outputs are always placed and executed in /scratch/galaxy/job_working). Isn't there usually a bunch of subdirectories created (e.g. 000/XXX, etc...) 

I suspect my issues arise because script execution is happening in the parent directory rather than in temporary ones that get created within the parent directory.

Could this be the issue and if so does anyone know how this could have happened or how I can fix this?

I've also noticed that my environment variables are pointing to the parent directory rather than a subdirectory:

SGE_CWD_PATH=/scratch/galaxy/job_working

As an example, the contents of /scratch/galaxy/job_working look like this (note that there are no usual subdirectories like 000/XXX, etc...):

[galaxy@galaxy job_working]$ ls | more
__instrument_core_epoch_end
__instrument_core_epoch_start
__instrument_core_galaxy_slots
conda-env
conda-metadata-env
galaxy_20771.e
galaxy_20771.o
galaxy_20771.sh
galaxy_20772.e
galaxy_20772.o
galaxy_20772.sh
metadata_in_HistoryDatasetAssociation_46704_NHpqc0
metadata_in_HistoryDatasetAssociation_46705_nUT573
metadata_kwds_HistoryDatasetAssociation_46704_5IADnM
metadata_kwds_HistoryDatasetAssociation_46705__z5cTO
metadata_out_HistoryDatasetAssociation_46704_G8phbW
metadata_out_HistoryDatasetAssociation_46705_qTOxoE
metadata_override_HistoryDatasetAssociation_46704_v0f_Wl
metadata_override_HistoryDatasetAssociation_46705_8pzLX_
metadata_results_HistoryDatasetAssociation_46704_fSFQsP
metadata_results_HistoryDatasetAssociation_46705_r4vUOP
set_metadata_2A6wuD.py
set_metadata_2Dasxn.py
tool_script.sh
working

Any help would be greatly appreciated!!


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/