abnormal termination at worker using supervisord

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

abnormal termination at worker using supervisord

Vipin TS
Dear developers, 

We are experiencing a job dispatching issue on our public galaxy instance (https://galaxy.inf.ethz.ch/). On the compute machine, I am not able to start the workers. The following messages got from the worker log file: 

migrate.versioning.repository DEBUG 2018-01-30 12:23:41,816 Repository lib/galaxy/model/migrate loaded successfully

migrate.versioning.repository DEBUG 2018-01-30 12:23:41,816 Config: OrderedDict([('db_settings', OrderedDict([('__name__', 'db_settings'), ('repository_id', 'Galaxy'), ('version_table', 'migrate_version'), ('required_dbs', '[]')]))])


After this line, it is restarting the process again and stop at the same message. 

I have checked the status of the workers through supervisor few times and it exited.  

-bash-4.2$ supervisorctl status 

galaxy:worker0                   STARTING 


galaxy:worker0                   BACKOFF   Exited too quickly (process log may have details)


galaxy:worker0                   FATAL     Exited too quickly (process log may have details)


my config/galaxy.ini 

[server:worker0]

use  = egg:Paste#http

port = 9999

host = COMPUTE.MACHINE.IP

use_threadpool     = true

threadpool_workers = 5


config/job_conf.xml 

<plugins workers="8">

        <plugin id="condor" type="runner" load="galaxy.jobs.runners.condor:CondorJobRunner" />

</plugins>


<handlers default="handlers">

        <handler id="worker0" tags="handlers"/>

</handlers>


<destinations default="worker0">

        <destination id="worker0" runner="condor">

            <param id="request_cpus">1</param>

        </destination>

</destinations>


config/galaxy_supervisor.conf

[program:galaxy_worker]

command         = /tools/galaxy_ratschlab/.venv/bin/python ./lib/galaxy/main.py -c /tools/galaxy_ratschlab/config/galaxy.ini --server-name=worker%(process_num)s --log-file=/tools/galaxy_ratschlab/worker%(process_num)s.log

directory       = /tools/galaxy_ratschlab

process_name    = worker%(process_num)s

numprocs        = 2

umask           = 022

autostart       = true

autorestart     = true

startsecs       = 20

user            = galaxy

environment     = VIRTUAL_ENV="/tools/galaxy_ratschlab/.venv",PATH="/tools/galaxy_ratschlab/.venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",PYTHONHOME="/tools/galaxy_ratschlab/.venv"


I tried to restart but didn't help, anything wrong with my configuration? Please let me know what you think and need. 

Thanks in advance! 

Vipin 

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/