Quantcast

Maker ERROR

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Maker ERROR

José Mª G. Perez-Silva
Hi, 

I wrote early this day, in reference to a problem of (apparently) space. After I deleted some unnecesary files (despite having plenty of storage left), I killed all the processes, and set 'clean_try=1' as recomended in this post. Before re-running the processes, we checked that there were no limitation over the size of a directory or something similar.

After re-running, at first, all seemed correct, but when I re-checked some time after, I found out a lot of contigs with the status FAILED without folder specification in the '_master_datastore_index.log', looking like:

Contig480 FAILED
Contig496 FAILED
Contig512 FAILED
Contig528 FAILED
Contig544 FAILED
Contig560 FAILED

But checking the 'nohub.out' of every proccess (16 in total, as the machine has 16 cores), I notice that each run is, from time to time, processing the contig correctly. So, after several (a lot) of FAILED contigs, it process one correctly. As said in the previous email, the ERROR dispolayed in the nohup.out is (including the last part of a processed contig at the beguinning):

#--------- command -------------#
Widget::blastx:
/usr/bin/blastall -p blastx -d /data/ge/tmp/maker_VfDQQU/hsap_ensembl%2Efa.mpi.10.6 -i /data/ge/tmp/maker_VfDQQU/0/Contig20.0 -b 10000 -v 10000 -e 1e-06 -z 300 -Y 500000000 -a 4 -U -F T -I T -o /data/ge/round3/cg.maker.output/cg_datastore/56/AC/Contig20//theVoid.Contig20/0/Contig20.0.hsap_ensembl%2Efa.blastx.temp_dir/hsap_ensembl%2Efa.mpi.10.6.blastx
#-------------------------------#
deleted:511 hits
doing blastx of proteins
open3: fork failed: Cannot allocate memory at /home/jmgps/software/maker/bin/../lib/File/NFSLock.pm line 1037.
--> rank=NA, hostname=Planarian2
ERROR: Failed while doing blastx of proteins
ERROR: Chunk failed at level:8, tier_type:3
FAILED CONTIG:Contig20

ERROR: Chunk failed at level:4, tier_type:0
FAILED CONTIG:Contig20

examining contents of the fasta file and run log
ERROR: could not make datastore directory
--> rank=NA, hostname=Planarian2
ERROR: Failed while examining contents of the fasta file and run log
ERROR: Chunk failed at level:0, tier_type:0
FAILED CONTIG:Contig22

examining contents of the fasta file and run log
ERROR: could not make datastore directory
--> rank=NA, hostname=Planarian2
ERROR: Failed while examining contents of the fasta file and run log
ERROR: Chunk failed at level:0, tier_type:0
FAILED CONTIG:Contig24

examining contents of the fasta file and run log
ERROR: could not make datastore directory
--> rank=NA, hostname=Planarian2
ERROR: Failed while examining contents of the fasta file and run log
ERROR: Chunk failed at level:0, tier_type:0
FAILED CONTIG:Contig26

examining contents of the fasta file and run log
ERROR: could not make datastore directory
--> rank=NA, hostname=Planarian2
ERROR: Failed while examining contents of the fasta file and run log
ERROR: Chunk failed at level:0, tier_type:0
FAILED CONTIG:Contig28

I'm totally lost here, I think it is still processing contigs, but the FAILED attemps slow down the whole process, and we are in a hurry due to the maintenance of the machine. And I can't understand the source of the ERROR.

I will be more than happy to provide more details about the problem, if requested.

Thanks a lot for the help!

_______________________________________________
maker-devel mailing list
[hidden email]
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Maker ERROR

Carson Holt-2
Several things.

1. MAKER does a lot of it’s work in a temporary directory (usually /tmp). This directory must be locally mounted and cannot be a network mounted location. If this location is full you can get issues.
2. MAKER needs at least 1GB of RAM per process (2-3GB is safer), so if you don’t have enough RAM you may need to run fewer processes (with MPI multiply whatever you supplied to the mpiexec -n flag by 1GB).
3. If you are launching MAKER multiple times as opposed to launching once via MPI, you will exacerbate the above limitations as well as open up IO limitations. MAKER can and does saturate IO when run multiple times simultaneously (this is especially true for network mounted locations). If you run via MPI you can greatly reduce IO, so make sure you are using MPI and not just launching MAKER multiple times.


If you absolutely have to start multiple jobs, you can reduce IO somewhat by splitting the input fasta into pieces (use fasta_tool). Give a separate piece to each job via maker’s -g flag, and set -base so all results from all jobs get written to the same location. Then each job can avoid multiple file locks that would have been encountered by sharing input. Note that you must rebuild the datastore index using 'maker -dsindex’ when all jobs complete.

—Carson




On Mar 10, 2017, at 10:21 AM, José Mª G. Perez-Silva <[hidden email]> wrote:

Hi, 

I wrote early this day, in reference to a problem of (apparently) space. After I deleted some unnecesary files (despite having plenty of storage left), I killed all the processes, and set 'clean_try=1' as recomended in this post. Before re-running the processes, we checked that there were no limitation over the size of a directory or something similar.

After re-running, at first, all seemed correct, but when I re-checked some time after, I found out a lot of contigs with the status FAILED without folder specification in the '_master_datastore_index.log', looking like:

Contig480 FAILED
Contig496 FAILED
Contig512 FAILED
Contig528 FAILED
Contig544 FAILED
Contig560 FAILED

But checking the 'nohub.out' of every proccess (16 in total, as the machine has 16 cores), I notice that each run is, from time to time, processing the contig correctly. So, after several (a lot) of FAILED contigs, it process one correctly. As said in the previous email, the ERROR dispolayed in the nohup.out is (including the last part of a processed contig at the beguinning):

#--------- command -------------#
Widget::blastx:
/usr/bin/blastall -p blastx -d /data/ge/tmp/maker_VfDQQU/hsap_ensembl%2Efa.mpi.10.6 -i /data/ge/tmp/maker_VfDQQU/0/Contig20.0 -b 10000 -v 10000 -e 1e-06 -z 300 -Y 500000000 -a 4 -U -F T -I T -o /data/ge/round3/cg.maker.output/cg_datastore/56/AC/Contig20//theVoid.Contig20/0/Contig20.0.hsap_ensembl%2Efa.blastx.temp_dir/hsap_ensembl%2Efa.mpi.10.6.blastx
#-------------------------------#
deleted:511 hits
doing blastx of proteins
open3: fork failed: Cannot allocate memory at /home/jmgps/software/maker/bin/../lib/File/NFSLock.pm line 1037.
--> rank=NA, hostname=Planarian2
ERROR: Failed while doing blastx of proteins
ERROR: Chunk failed at level:8, tier_type:3
FAILED CONTIG:Contig20

ERROR: Chunk failed at level:4, tier_type:0
FAILED CONTIG:Contig20

examining contents of the fasta file and run log
ERROR: could not make datastore directory
--> rank=NA, hostname=Planarian2
ERROR: Failed while examining contents of the fasta file and run log
ERROR: Chunk failed at level:0, tier_type:0
FAILED CONTIG:Contig22

examining contents of the fasta file and run log
ERROR: could not make datastore directory
--> rank=NA, hostname=Planarian2
ERROR: Failed while examining contents of the fasta file and run log
ERROR: Chunk failed at level:0, tier_type:0
FAILED CONTIG:Contig24

examining contents of the fasta file and run log
ERROR: could not make datastore directory
--> rank=NA, hostname=Planarian2
ERROR: Failed while examining contents of the fasta file and run log
ERROR: Chunk failed at level:0, tier_type:0
FAILED CONTIG:Contig26

examining contents of the fasta file and run log
ERROR: could not make datastore directory
--> rank=NA, hostname=Planarian2
ERROR: Failed while examining contents of the fasta file and run log
ERROR: Chunk failed at level:0, tier_type:0
FAILED CONTIG:Contig28

I'm totally lost here, I think it is still processing contigs, but the FAILED attemps slow down the whole process, and we are in a hurry due to the maintenance of the machine. And I can't understand the source of the ERROR.

I will be more than happy to provide more details about the problem, if requested.

Thanks a lot for the help!


_______________________________________________
maker-devel mailing list
[hidden email]
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
Loading...