Help with MPI installation and configuration

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Help with MPI installation and configuration

Tim Fallon
Hi there,

I’ve been troubleshooting the maker 3.0.0 beta plus MPI installation and configuration with our institute’s IT department.  Despite a lot of tries and recompilations from scratch, we haven’t been able to get it to work (non-MPI maker works fine, albeit a bit slow).  I’ve attached our most recent crash log when trying to run maker with MPI.  Any tips?  I’ve never used MPI before maker so not that familiar with troubleshooting it.

See below for my most recent contact with IT on it.  I’ve also attached the most recent log file.  We are trying to execute it through the LSF cluster submission tool, but I don’t think that is related to the problem.  Any ideas?

"
I think one option is to try a build on a host without the full suite of cluster software.  It may actually help, but if nothing else we can say that our problem exists with a (relatively) vanilla ubuntu linux host.

Peter Macfarlane / WIBR IT


Description : Hi Peter,

Hoping to hear back from you on this.  It does seem like we’ve exhausted the reasonable troubleshooting for this maker MPI execution, but maybe worth another shot before I ask the developers?

All the best,
-Tim

On Apr 7, 2017, at 9:11 AM, Tim Fallon <[hidden email]> wrote:

Hi Peter,

I’ve tested with the following command:
bsub -o bsub_maker_with_mpi_log.txt  "/nfs/apps/test/openmpi/bin/mpirun -n 8 /nfs/apps/test/maker/bin/maker”

Unfortunately it still seems to be giving errors and not running, with things like:
"“mca_base_component_repository_open: unable to open mca_patcher_overwrite: /nfs/apps/test/openmpi/lib/openmpi/mca_patcher_overwrite.so: undefined symbol: mca_patcher_base_patch_t_class (ignored)””

It also doesn’t work if I try to run it directly on Tak4.

I’ve attached the relevant error log file.  Any more solutions?

All the best,
-Tim

On Apr 5, 2017, at 3:50 PM, IT Systems Group <[hidden email]> wrote:

I tried rebuilding maker 3 with it's own also-built-from-source openmpi instance.  So with that:

- /nfs/apps/test/maker is now maker 3.00
- /nfs/apps/test/openmpi was used to make the aforementioned maker

I think that all you'll need to do differently here is invoke the version of mpirun in /nfs/apps/test/openmpi/bin
"
All the best,
-Tim

Timothy R. Fallon
PhD candidate
Laboratory of Jing-Ke Weng
Department of Biology
MIT




_______________________________________________
maker-devel mailing list
[hidden email]
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org

bsub_maker_with_mpi_log.txt (22K) Download Attachment
smime.p7s (2K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Help with MPI installation and configuration

Sobel-Sorenson, Chandler - (scar)
Please try with MPICH instead of openmpi... we were having similar
errors with openmpi, but when we switched to mpich it all started
working.  Make sure only the mpich executables are in your PATH, and
that maker is built specifying the mpich package.

Chandler / Systems Administrator
Arizona Genomics Institute
www.genome.arizona.edu



_______________________________________________
maker-devel mailing list
[hidden email]
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Help with MPI installation and configuration

Carson Holt-2
In reply to this post by Tim Fallon
I agree that just using MPICH3 is the easiest solution. But to use OpenMPI you will need to export the environmental variables add the command line flags to mpiexec explained in the ../maker/INSTALL file.

#this must be doen exery time you run it
export LD_PRELOAD=/location/of/openmpi/lib/libmpi.so

# -mca btl ^openib myust be on the command line
mpiexec -mca btl ^openib -n 20 maker

#alternatively add --mca btl vader,tcp,self --mca btl_tcp_if_include ib0 (use ib0 or whatever adapter interface exists on your cluster)
mpiexec --mca btl vader,tcp,self --mca btl_tcp_if_include ib0 -n 20 maker

—Carson

On Apr 17, 2017, at 9:35 AM, Tim Fallon <[hidden email]> wrote:

Hi there,

I’ve been troubleshooting the maker 3.0.0 beta plus MPI installation and configuration with our institute’s IT department.  Despite a lot of tries and recompilations from scratch, we haven’t been able to get it to work (non-MPI maker works fine, albeit a bit slow).  I’ve attached our most recent crash log when trying to run maker with MPI.  Any tips?  I’ve never used MPI before maker so not that familiar with troubleshooting it.

See below for my most recent contact with IT on it.  I’ve also attached the most recent log file.  We are trying to execute it through the LSF cluster submission tool, but I don’t think that is related to the problem.  Any ideas?

"
I think one option is to try a build on a host without the full suite of cluster software.  It may actually help, but if nothing else we can say that our problem exists with a (relatively) vanilla ubuntu linux host.

Peter Macfarlane / WIBR IT


Description : Hi Peter,

Hoping to hear back from you on this.  It does seem like we’ve exhausted the reasonable troubleshooting for this maker MPI execution, but maybe worth another shot before I ask the developers?

All the best,
-Tim

On Apr 7, 2017, at 9:11 AM, Tim Fallon <[hidden email]> wrote:

Hi Peter,

I’ve tested with the following command:
bsub -o bsub_maker_with_mpi_log.txt  "/nfs/apps/test/openmpi/bin/mpirun -n 8 /nfs/apps/test/maker/bin/maker”

Unfortunately it still seems to be giving errors and not running, with things like:
"“mca_base_component_repository_open: unable to open mca_patcher_overwrite: /nfs/apps/test/openmpi/lib/openmpi/mca_patcher_overwrite.so: undefined symbol: mca_patcher_base_patch_t_class (ignored)””

It also doesn’t work if I try to run it directly on Tak4.

I’ve attached the relevant error log file.  Any more solutions?

All the best,
-Tim

On Apr 5, 2017, at 3:50 PM, IT Systems Group <[hidden email]> wrote:

I tried rebuilding maker 3 with it's own also-built-from-source openmpi instance.  So with that:

- /nfs/apps/test/maker is now maker 3.00
- /nfs/apps/test/openmpi was used to make the aforementioned maker

I think that all you'll need to do differently here is invoke the version of mpirun in /nfs/apps/test/openmpi/bin
"
All the best,
-Tim

Timothy R. Fallon
PhD candidate
Laboratory of Jing-Ke Weng
Department of Biology
MIT


<bsub_maker_with_mpi_log.txt>
_______________________________________________
maker-devel mailing list
[hidden email]
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org


_______________________________________________
maker-devel mailing list
[hidden email]
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Help with MPI installation and configuration

Tim Fallon
Hi Carson,

Thanks for the feedback, I’ll look into the maker INSTALL file to ensure things are setup correctly.

Should I be looking into MPICH3 instead?  I’m trying to run maker on a single node (~20 CPUs), rather than distribute it across our cluster.  I assume that using MPICH3 changes the mpiexec / mpirun executable, but not the commands otherwise?  I ask as I may have to configure this myself one day, so any tips appreciated.  In the meantime your comments seem straightforward, will give it a shot.

All the best,
-Tim

On Apr 18, 2017, at 11:10 AM, Carson Holt <[hidden email]> wrote:

I agree that just using MPICH3 is the easiest solution. But to use OpenMPI you will need to export the environmental variables add the command line flags to mpiexec explained in the ../maker/INSTALL file.

#this must be doen exery time you run it
export LD_PRELOAD=/location/of/openmpi/lib/libmpi.so

# -mca btl ^openib myust be on the command line
mpiexec -mca btl ^openib -n 20 maker

#alternatively add --mca btl vader,tcp,self --mca btl_tcp_if_include ib0 (use ib0 or whatever adapter interface exists on your cluster)
mpiexec --mca btl vader,tcp,self --mca btl_tcp_if_include ib0 -n 20 maker

—Carson

On Apr 17, 2017, at 9:35 AM, Tim Fallon <[hidden email]> wrote:

Hi there,

I’ve been troubleshooting the maker 3.0.0 beta plus MPI installation and configuration with our institute’s IT department.  Despite a lot of tries and recompilations from scratch, we haven’t been able to get it to work (non-MPI maker works fine, albeit a bit slow).  I’ve attached our most recent crash log when trying to run maker with MPI.  Any tips?  I’ve never used MPI before maker so not that familiar with troubleshooting it.

See below for my most recent contact with IT on it.  I’ve also attached the most recent log file.  We are trying to execute it through the LSF cluster submission tool, but I don’t think that is related to the problem.  Any ideas?

"
I think one option is to try a build on a host without the full suite of cluster software.  It may actually help, but if nothing else we can say that our problem exists with a (relatively) vanilla ubuntu linux host.

Peter Macfarlane / WIBR IT


Description : Hi Peter,

Hoping to hear back from you on this.  It does seem like we’ve exhausted the reasonable troubleshooting for this maker MPI execution, but maybe worth another shot before I ask the developers?

All the best,
-Tim

On Apr 7, 2017, at 9:11 AM, Tim Fallon <[hidden email]> wrote:

Hi Peter,

I’ve tested with the following command:
bsub -o bsub_maker_with_mpi_log.txt  "/nfs/apps/test/openmpi/bin/mpirun -n 8 /nfs/apps/test/maker/bin/maker”

Unfortunately it still seems to be giving errors and not running, with things like:
"“mca_base_component_repository_open: unable to open mca_patcher_overwrite: /nfs/apps/test/openmpi/lib/openmpi/mca_patcher_overwrite.so: undefined symbol: mca_patcher_base_patch_t_class (ignored)””

It also doesn’t work if I try to run it directly on Tak4.

I’ve attached the relevant error log file.  Any more solutions?

All the best,
-Tim

On Apr 5, 2017, at 3:50 PM, IT Systems Group <[hidden email]> wrote:

I tried rebuilding maker 3 with it's own also-built-from-source openmpi instance.  So with that:

- /nfs/apps/test/maker is now maker 3.00
- /nfs/apps/test/openmpi was used to make the aforementioned maker

I think that all you'll need to do differently here is invoke the version of mpirun in /nfs/apps/test/openmpi/bin
"
All the best,
-Tim

Timothy R. Fallon
PhD candidate
Laboratory of Jing-Ke Weng
Department of Biology
MIT


<bsub_maker_with_mpi_log.txt>
_______________________________________________
maker-devel mailing list
[hidden email]
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org


Timothy R. Fallon
PhD candidate
Laboratory of Jing-Ke Weng
Department of Biology
MIT



_______________________________________________
maker-devel mailing list
[hidden email]
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org

smime.p7s (2K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Help with MPI installation and configuration

Carson Holt-2
MPICH3 works without any special configuration. OpenMPI is built to support infiniband which internally can drive other problems related to how registered memory is used when infiniband libraries are loaded. So you have have to set more parameters.

Both will use shared memory for communications when running on a single machine, but you will still need to set parameters on OpenMPI.

—Carson



On Apr 18, 2017, at 9:19 AM, Tim Fallon <[hidden email]> wrote:

Hi Carson,

Thanks for the feedback, I’ll look into the maker INSTALL file to ensure things are setup correctly.

Should I be looking into MPICH3 instead?  I’m trying to run maker on a single node (~20 CPUs), rather than distribute it across our cluster.  I assume that using MPICH3 changes the mpiexec / mpirun executable, but not the commands otherwise?  I ask as I may have to configure this myself one day, so any tips appreciated.  In the meantime your comments seem straightforward, will give it a shot.

All the best,
-Tim

On Apr 18, 2017, at 11:10 AM, Carson Holt <[hidden email]> wrote:

I agree that just using MPICH3 is the easiest solution. But to use OpenMPI you will need to export the environmental variables add the command line flags to mpiexec explained in the ../maker/INSTALL file.

#this must be doen exery time you run it
export LD_PRELOAD=/location/of/openmpi/lib/libmpi.so

# -mca btl ^openib myust be on the command line
mpiexec -mca btl ^openib -n 20 maker

#alternatively add --mca btl vader,tcp,self --mca btl_tcp_if_include ib0 (use ib0 or whatever adapter interface exists on your cluster)
mpiexec --mca btl vader,tcp,self --mca btl_tcp_if_include ib0 -n 20 maker

—Carson

On Apr 17, 2017, at 9:35 AM, Tim Fallon <[hidden email]> wrote:

Hi there,

I’ve been troubleshooting the maker 3.0.0 beta plus MPI installation and configuration with our institute’s IT department.  Despite a lot of tries and recompilations from scratch, we haven’t been able to get it to work (non-MPI maker works fine, albeit a bit slow).  I’ve attached our most recent crash log when trying to run maker with MPI.  Any tips?  I’ve never used MPI before maker so not that familiar with troubleshooting it.

See below for my most recent contact with IT on it.  I’ve also attached the most recent log file.  We are trying to execute it through the LSF cluster submission tool, but I don’t think that is related to the problem.  Any ideas?

"
I think one option is to try a build on a host without the full suite of cluster software.  It may actually help, but if nothing else we can say that our problem exists with a (relatively) vanilla ubuntu linux host.

Peter Macfarlane / WIBR IT


Description : Hi Peter,

Hoping to hear back from you on this.  It does seem like we’ve exhausted the reasonable troubleshooting for this maker MPI execution, but maybe worth another shot before I ask the developers?

All the best,
-Tim

On Apr 7, 2017, at 9:11 AM, Tim Fallon <[hidden email]> wrote:

Hi Peter,

I’ve tested with the following command:
bsub -o bsub_maker_with_mpi_log.txt  "/nfs/apps/test/openmpi/bin/mpirun -n 8 /nfs/apps/test/maker/bin/maker”

Unfortunately it still seems to be giving errors and not running, with things like:
"“mca_base_component_repository_open: unable to open mca_patcher_overwrite: /nfs/apps/test/openmpi/lib/openmpi/mca_patcher_overwrite.so: undefined symbol: mca_patcher_base_patch_t_class (ignored)””

It also doesn’t work if I try to run it directly on Tak4.

I’ve attached the relevant error log file.  Any more solutions?

All the best,
-Tim

On Apr 5, 2017, at 3:50 PM, IT Systems Group <[hidden email]> wrote:

I tried rebuilding maker 3 with it's own also-built-from-source openmpi instance.  So with that:

- /nfs/apps/test/maker is now maker 3.00
- /nfs/apps/test/openmpi was used to make the aforementioned maker

I think that all you'll need to do differently here is invoke the version of mpirun in /nfs/apps/test/openmpi/bin
"
All the best,
-Tim

Timothy R. Fallon
PhD candidate
Laboratory of Jing-Ke Weng
Department of Biology
MIT


<bsub_maker_with_mpi_log.txt>
_______________________________________________
maker-devel mailing list
[hidden email]
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org


Timothy R. Fallon
PhD candidate
Laboratory of Jing-Ke Weng
Department of Biology
MIT




_______________________________________________
maker-devel mailing list
[hidden email]
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
Loading...