maker with mpi support on example still not done after two days

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

maker with mpi support on example still not done after two days

Taslim, Cenny

Hi Maker developers,

 

Thanks for approving my subscription.

I tried running maker with mpi support on the human fasta file provided in example_01_basic. It’s been running for 2 days and 21 hours. I didn’t think the example require a long time to run.

I’m hoping someone can help me point out the problem.

 

I’m running it with 4 processes:

~/opt/mpich-3.3.1/bin/mpiexec -n 4 ~/opt/maker.4/maker/bin/maker -f 2> maker.error

 

Maker_opts.ctl is the same as opts2.txt

 

These are the log:

STATUS: Parsing control files...

STATUS: Processing and indexing input FASTA files...

STATUS: Setting up database for any GFF3 input...

A data structure will be created for you at:

/maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore

 

To access files for individual sequences use the datastore index:

/maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_master_datastore_index.log

 

STATUS: Now running MAKER...

examining contents of the fasta file and run log

 

 

 

--Next Contig--

 

#---------------------------------------------------------------------

Now starting the contig!!

SeqID: NT_010783.15

Length: 201444

#---------------------------------------------------------------------

 

 

setting up GFF3 output and fasta chunks

doing repeat masking

doing repeat masking

running  repeat masker.

#--------- command -------------#

Widget::RepeatMasker:

cd /gpfs0/scratch/1895302/maker_kXmduG; /opt/maker.4/maker/exe/RepeatMasker/RepeatMasker /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.0.all.rb -species all -dir /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0 -pa 1

#-------------------------------#

running  repeat masker.

#--------- command -------------#

Widget::RepeatMasker:

cd /gpfs0/scratch/1895302/maker_kXmduG; /opt/maker.4/maker/exe/RepeatMasker/RepeatMasker /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.all.rb -species all -dir /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0 -pa 1

#-------------------------------#

doing blastx repeats

formating database...

#--------- command -------------#

Widget::formater:

/opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/1/blastprep/te_proteins%2Efasta.mpi.10.0

#-------------------------------#

doing blastx repeats

formating database...

#--------- command -------------#

Widget::formater:

/opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/2/blastprep/te_proteins%2Efasta.mpi.10.1

#-------------------------------#

running  blast search.

#--------- command -------------#

Widget::blastx:

/opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.0 -query /gpfs0/scratch/1895302/maker_kXmduG/1/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.0.repeatrunner

#-------------------------------#

running  blast search.

#--------- command -------------#

Widget::blastx:

/opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.1 -query /gpfs0/scratch/1895302/maker_kXmduG/2/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.1.repeatrunner

#-------------------------------#

deleted:19 hits

deleted:18 hits

doing blastx repeats

doing blastx repeats

formating database...

#--------- command -------------#

Widget::formater:

/opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/1/blastprep/te_proteins%2Efasta.mpi.10.2

#-------------------------------#

formating database...

#--------- command -------------#

Widget::formater:

/opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/2/blastprep/te_proteins%2Efasta.mpi.10.3

#-------------------------------#

running  blast search.

#--------- command -------------#

Widget::blastx:

/opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.2 -query /gpfs0/scratch/1895302/maker_kXmduG/1/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.2.repeatrunner

#-------------------------------#

running  blast search.

#--------- command -------------#

Widget::blastx:

/opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.3 -query /gpfs0/scratch/1895302/maker_kXmduG/2/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.3.repeatrunner

#-------------------------------#

deleted:9 hits

deleted:9 hits

doing blastx repeats

doing blastx repeats

formating database...

#--------- command -------------#

Widget::formater:

/opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/1/blastprep/te_proteins%2Efasta.mpi.10.5

#-------------------------------#

formating database...

#--------- command -------------#

Widget::formater:

/opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/2/blastprep/te_proteins%2Efasta.mpi.10.4

#-------------------------------#

running  blast search.

#--------- command -------------#

Widget::blastx:

/opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.5 -query /gpfs0/scratch/1895302/maker_kXmduG/1/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.5.repeatrunner

#-------------------------------#

running  blast search.

#--------- command -------------#

Widget::blastx:

/opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.4 -query /gpfs0/scratch/1895302/maker_kXmduG/2/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.4.repeatrunner

#-------------------------------#

deleted:16 hits

deleted:16 hits

doing blastx repeats

doing blastx repeats

formating database...

#--------- command -------------#

Widget::formater:

/opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/1/blastprep/te_proteins%2Efasta.mpi.10.6

#-------------------------------#

formating database...

#--------- command -------------#

Widget::formater:

/opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/2/blastprep/te_proteins%2Efasta.mpi.10.7

#-------------------------------#

running  blast search.

#--------- command -------------#

Widget::blastx:

/opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.6 -query /gpfs0/scratch/1895302/maker_kXmduG/1/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.6.repeatrunner

#-------------------------------#

running  blast search.

#--------- command -------------#

Widget::blastx:

/opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.7 -query /gpfs0/scratch/1895302/maker_kXmduG/2/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.7.repeatrunner

#-------------------------------#

deleted:8 hits

deleted:12 hits

doing blastx repeats

doing blastx repeats

formating database...

#--------- command -------------#

Widget::formater:

/opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/1/blastprep/te_proteins%2Efasta.mpi.10.8

#-------------------------------#

formating database...

#--------- command -------------#

Widget::formater:

/opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/2/blastprep/te_proteins%2Efasta.mpi.10.9

#-------------------------------#

running  blast search.

#--------- command -------------#

Widget::blastx:

/opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.8 -query /gpfs0/scratch/1895302/maker_kXmduG/1/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.8.repeatrunner

#-------------------------------#

running  blast search.

#--------- command -------------#

Widget::blastx:

/opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.9 -query /gpfs0/scratch/1895302/maker_kXmduG/2/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.9.repeatrunner

#-------------------------------#

deleted:5 hits

deleted:7 hits

collecting blastx repeatmasking

processing all repeats

in cluster::shadow_cluster...

...finished clustering.

 

Let me know if you need more information. Thank you in advance for your help.

 

 


_______________________________________________
maker-devel mailing list
[hidden email]
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
Reply | Threaded
Open this post in threaded view
|

Re: maker with mpi support on example still not done after two days

Taslim, Cenny
I figured out that the example job will be finished if I'm not using mpi. i.e. running the job as such is fine:
~/opt/maker.4/maker/bin/maker -f 2> maker.error

Any suggestions? Without mpi, I suspect it will take months to complete as I have human genome with ~3500 contigs

_______________________________________________
maker-devel mailing list
[hidden email]
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
Reply | Threaded
Open this post in threaded view
|

Re: maker with mpi support on example still not done after two days

Carson Holt-2
In reply to this post by Taslim, Cenny
If no additional output is produced, then it is frozen for some reason. My first suggestion would be to reinstall MPICH and then reinstall MAKER using the MPICH you just installed.  Freezing issues are usually related to MPI communication which is handled by the communicator you have installed (i.e. the mpiexec command that launches MAKER).

Alternatively it can be related to the file system. Some less common network mounted file systems do not support do not correctly support hardlinks which can cause programatic file locks to freeze. You can try running with the -nolock flag if that is the case.

—Carson


On Aug 8, 2019, at 7:12 AM, Taslim, Cenny <[hidden email]> wrote:

Hi Maker developers,
 
Thanks for approving my subscription. 
I tried running maker with mpi support on the human fasta file provided in example_01_basic. It’s been running for 2 days and 21 hours. I didn’t think the example require a long time to run.
I’m hoping someone can help me point out the problem.
 
I’m running it with 4 processes:
~/opt/mpich-3.3.1/bin/mpiexec -n 4 ~/opt/maker.4/maker/bin/maker -f 2> maker.error
 
Maker_opts.ctl is the same as opts2.txt
 
These are the log:
STATUS: Parsing control files...
STATUS: Processing and indexing input FASTA files...
STATUS: Setting up database for any GFF3 input...
A data structure will be created for you at:
/maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore
 
To access files for individual sequences use the datastore index:
/maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_master_datastore_index.log
 
STATUS: Now running MAKER...
examining contents of the fasta file and run log
 
 
 
--Next Contig--
 
#---------------------------------------------------------------------
Now starting the contig!!
SeqID: NT_010783.15
Length: 201444
#---------------------------------------------------------------------
 
 
setting up GFF3 output and fasta chunks
doing repeat masking
doing repeat masking
running  repeat masker.
#--------- command -------------#
Widget::RepeatMasker:
cd /gpfs0/scratch/1895302/maker_kXmduG; /opt/maker.4/maker/exe/RepeatMasker/RepeatMasker /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.0.all.rb -species all -dir /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0 -pa 1
#-------------------------------#
running  repeat masker.
#--------- command -------------#
Widget::RepeatMasker:
cd /gpfs0/scratch/1895302/maker_kXmduG; /opt/maker.4/maker/exe/RepeatMasker/RepeatMasker /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.all.rb -species all -dir /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0 -pa 1
#-------------------------------#
doing blastx repeats
formating database...
#--------- command -------------#
Widget::formater:
/opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/1/blastprep/te_proteins%2Efasta.mpi.10.0
#-------------------------------#
doing blastx repeats
formating database...
#--------- command -------------#
Widget::formater:
/opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/2/blastprep/te_proteins%2Efasta.mpi.10.1
#-------------------------------#
running  blast search.
#--------- command -------------#
Widget::blastx:
/opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.0 -query /gpfs0/scratch/1895302/maker_kXmduG/1/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.0.repeatrunner
#-------------------------------#
running  blast search.
#--------- command -------------#
Widget::blastx:
/opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.1 -query /gpfs0/scratch/1895302/maker_kXmduG/2/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.1.repeatrunner
#-------------------------------#
deleted:19 hits
deleted:18 hits
doing blastx repeats
doing blastx repeats
formating database...
#--------- command -------------#
Widget::formater:
/opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/1/blastprep/te_proteins%2Efasta.mpi.10.2
#-------------------------------#
formating database...
#--------- command -------------#
Widget::formater:
/opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/2/blastprep/te_proteins%2Efasta.mpi.10.3
#-------------------------------#
running  blast search.
#--------- command -------------#
Widget::blastx:
/opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.2 -query /gpfs0/scratch/1895302/maker_kXmduG/1/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.2.repeatrunner
#-------------------------------#
running  blast search.
#--------- command -------------#
Widget::blastx:
/opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.3 -query /gpfs0/scratch/1895302/maker_kXmduG/2/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.3.repeatrunner
#-------------------------------#
deleted:9 hits
deleted:9 hits
doing blastx repeats
doing blastx repeats
formating database...
#--------- command -------------#
Widget::formater:
/opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/1/blastprep/te_proteins%2Efasta.mpi.10.5
#-------------------------------#
formating database...
#--------- command -------------#
Widget::formater:
/opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/2/blastprep/te_proteins%2Efasta.mpi.10.4
#-------------------------------#
running  blast search.
#--------- command -------------#
Widget::blastx:
/opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.5 -query /gpfs0/scratch/1895302/maker_kXmduG/1/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.5.repeatrunner
#-------------------------------#
running  blast search.
#--------- command -------------#
Widget::blastx:
/opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.4 -query /gpfs0/scratch/1895302/maker_kXmduG/2/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.4.repeatrunner
#-------------------------------#
deleted:16 hits
deleted:16 hits
doing blastx repeats
doing blastx repeats
formating database...
#--------- command -------------#
Widget::formater:
/opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/1/blastprep/te_proteins%2Efasta.mpi.10.6
#-------------------------------#
formating database...
#--------- command -------------#
Widget::formater:
/opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/2/blastprep/te_proteins%2Efasta.mpi.10.7
#-------------------------------#
running  blast search.
#--------- command -------------#
Widget::blastx:
/opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.6 -query /gpfs0/scratch/1895302/maker_kXmduG/1/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.6.repeatrunner
#-------------------------------#
running  blast search.
#--------- command -------------#
Widget::blastx:
/opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.7 -query /gpfs0/scratch/1895302/maker_kXmduG/2/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.7.repeatrunner
#-------------------------------#
deleted:8 hits
deleted:12 hits
doing blastx repeats
doing blastx repeats
formating database...
#--------- command -------------#
Widget::formater:
/opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/1/blastprep/te_proteins%2Efasta.mpi.10.8
#-------------------------------#
formating database...
#--------- command -------------#
Widget::formater:
/opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/2/blastprep/te_proteins%2Efasta.mpi.10.9
#-------------------------------#
running  blast search.
#--------- command -------------#
Widget::blastx:
/opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.8 -query /gpfs0/scratch/1895302/maker_kXmduG/1/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.8.repeatrunner
#-------------------------------#
running  blast search.
#--------- command -------------#
Widget::blastx:
/opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.9 -query /gpfs0/scratch/1895302/maker_kXmduG/2/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.9.repeatrunner
#-------------------------------#
deleted:5 hits
deleted:7 hits
collecting blastx repeatmasking
processing all repeats
in cluster::shadow_cluster...
...finished clustering.
 
Let me know if you need more information. Thank you in advance for your help.
 
 
_______________________________________________
maker-devel mailing list
[hidden email]
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org


_______________________________________________
maker-devel mailing list
[hidden email]
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
Reply | Threaded
Open this post in threaded view
|

Re: maker with mpi support on example still not done after two days

Carson Holt-2
You can also try OpenMPI instead of MPICH to see if it behaves better (requires MAKER reinstall against OpenMPI libraries).

—Carson


On Aug 14, 2019, at 1:05 PM, Carson Holt <[hidden email]> wrote:

If no additional output is produced, then it is frozen for some reason. My first suggestion would be to reinstall MPICH and then reinstall MAKER using the MPICH you just installed.  Freezing issues are usually related to MPI communication which is handled by the communicator you have installed (i.e. the mpiexec command that launches MAKER).

Alternatively it can be related to the file system. Some less common network mounted file systems do not support do not correctly support hardlinks which can cause programatic file locks to freeze. You can try running with the -nolock flag if that is the case.

—Carson


On Aug 8, 2019, at 7:12 AM, Taslim, Cenny <[hidden email]> wrote:

Hi Maker developers,
 
Thanks for approving my subscription. 
I tried running maker with mpi support on the human fasta file provided in example_01_basic. It’s been running for 2 days and 21 hours. I didn’t think the example require a long time to run.
I’m hoping someone can help me point out the problem.
 
I’m running it with 4 processes:
~/opt/mpich-3.3.1/bin/mpiexec -n 4 ~/opt/maker.4/maker/bin/maker -f 2> maker.error
 
Maker_opts.ctl is the same as opts2.txt
 
These are the log:
STATUS: Parsing control files...
STATUS: Processing and indexing input FASTA files...
STATUS: Setting up database for any GFF3 input...
A data structure will be created for you at:
/maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore
 
To access files for individual sequences use the datastore index:
/maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_master_datastore_index.log
 
STATUS: Now running MAKER...
examining contents of the fasta file and run log
 
 
 
--Next Contig--
 
#---------------------------------------------------------------------
Now starting the contig!!
SeqID: NT_010783.15
Length: 201444
#---------------------------------------------------------------------
 
 
setting up GFF3 output and fasta chunks
doing repeat masking
doing repeat masking
running  repeat masker.
#--------- command -------------#
Widget::RepeatMasker:
cd /gpfs0/scratch/1895302/maker_kXmduG; /opt/maker.4/maker/exe/RepeatMasker/RepeatMasker /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.0.all.rb -species all -dir /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0 -pa 1
#-------------------------------#
running  repeat masker.
#--------- command -------------#
Widget::RepeatMasker:
cd /gpfs0/scratch/1895302/maker_kXmduG; /opt/maker.4/maker/exe/RepeatMasker/RepeatMasker /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.all.rb -species all -dir /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0 -pa 1
#-------------------------------#
doing blastx repeats
formating database...
#--------- command -------------#
Widget::formater:
/opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/1/blastprep/te_proteins%2Efasta.mpi.10.0
#-------------------------------#
doing blastx repeats
formating database...
#--------- command -------------#
Widget::formater:
/opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/2/blastprep/te_proteins%2Efasta.mpi.10.1
#-------------------------------#
running  blast search.
#--------- command -------------#
Widget::blastx:
/opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.0 -query /gpfs0/scratch/1895302/maker_kXmduG/1/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.0.repeatrunner
#-------------------------------#
running  blast search.
#--------- command -------------#
Widget::blastx:
/opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.1 -query /gpfs0/scratch/1895302/maker_kXmduG/2/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.1.repeatrunner
#-------------------------------#
deleted:19 hits
deleted:18 hits
doing blastx repeats
doing blastx repeats
formating database...
#--------- command -------------#
Widget::formater:
/opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/1/blastprep/te_proteins%2Efasta.mpi.10.2
#-------------------------------#
formating database...
#--------- command -------------#
Widget::formater:
/opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/2/blastprep/te_proteins%2Efasta.mpi.10.3
#-------------------------------#
running  blast search.
#--------- command -------------#
Widget::blastx:
/opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.2 -query /gpfs0/scratch/1895302/maker_kXmduG/1/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.2.repeatrunner
#-------------------------------#
running  blast search.
#--------- command -------------#
Widget::blastx:
/opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.3 -query /gpfs0/scratch/1895302/maker_kXmduG/2/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.3.repeatrunner
#-------------------------------#
deleted:9 hits
deleted:9 hits
doing blastx repeats
doing blastx repeats
formating database...
#--------- command -------------#
Widget::formater:
/opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/1/blastprep/te_proteins%2Efasta.mpi.10.5
#-------------------------------#
formating database...
#--------- command -------------#
Widget::formater:
/opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/2/blastprep/te_proteins%2Efasta.mpi.10.4
#-------------------------------#
running  blast search.
#--------- command -------------#
Widget::blastx:
/opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.5 -query /gpfs0/scratch/1895302/maker_kXmduG/1/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.5.repeatrunner
#-------------------------------#
running  blast search.
#--------- command -------------#
Widget::blastx:
/opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.4 -query /gpfs0/scratch/1895302/maker_kXmduG/2/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.4.repeatrunner
#-------------------------------#
deleted:16 hits
deleted:16 hits
doing blastx repeats
doing blastx repeats
formating database...
#--------- command -------------#
Widget::formater:
/opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/1/blastprep/te_proteins%2Efasta.mpi.10.6
#-------------------------------#
formating database...
#--------- command -------------#
Widget::formater:
/opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/2/blastprep/te_proteins%2Efasta.mpi.10.7
#-------------------------------#
running  blast search.
#--------- command -------------#
Widget::blastx:
/opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.6 -query /gpfs0/scratch/1895302/maker_kXmduG/1/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.6.repeatrunner
#-------------------------------#
running  blast search.
#--------- command -------------#
Widget::blastx:
/opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.7 -query /gpfs0/scratch/1895302/maker_kXmduG/2/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.7.repeatrunner
#-------------------------------#
deleted:8 hits
deleted:12 hits
doing blastx repeats
doing blastx repeats
formating database...
#--------- command -------------#
Widget::formater:
/opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/1/blastprep/te_proteins%2Efasta.mpi.10.8
#-------------------------------#
formating database...
#--------- command -------------#
Widget::formater:
/opt/miniconda3/bin/makeblastdb -dbtype prot -in /gpfs0/scratch/1895302/maker_kXmduG/2/blastprep/te_proteins%2Efasta.mpi.10.9
#-------------------------------#
running  blast search.
#--------- command -------------#
Widget::blastx:
/opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.8 -query /gpfs0/scratch/1895302/maker_kXmduG/1/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.8.repeatrunner
#-------------------------------#
running  blast search.
#--------- command -------------#
Widget::blastx:
/opt/miniconda3/bin/blastx -db /gpfs0/scratch/1895302/maker_kXmduG/te_proteins%2Efasta.mpi.10.9 -query /gpfs0/scratch/1895302/maker_kXmduG/2/NT_010783%2E15.1 -num_alignments 10000 -num_descriptions 10000 -evalue 1e-06 -dbsize 300 -searchsp 500000000 -num_threads 1 -seg yes -soft_masking true -lcase_masking -show_gis -out /maker_tutorial/example_01_basic/hsap_contig.maker.output/hsap_contig_datastore/80/99/NT_010783.15//theVoid.NT_010783.15/0/NT_010783%2E15.1.te_proteins%2Efasta.repeatrunner.temp_dir/te_proteins%2Efasta.mpi.10.9.repeatrunner
#-------------------------------#
deleted:5 hits
deleted:7 hits
collecting blastx repeatmasking
processing all repeats
in cluster::shadow_cluster...
...finished clustering.
 
Let me know if you need more information. Thank you in advance for your help.
 
 
_______________________________________________
maker-devel mailing list
[hidden email]
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org



_______________________________________________
maker-devel mailing list
[hidden email]
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org