maker with MPI and perl using threads

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

maker with MPI and perl using threads

John Damm Sørensen
Hello,

Recently I assisted one of my customers with problems solving maker
using MPI.

It seems that the main reason for the trouble was maker not initializing
the MPI environment for thread save execution.

In the MPI.pm module you call MPI_Init whereas for a threaded
environment you should call MPI_Init_thread.

I think it would be a good idea to detect whether the users Perl is with
thread support and init MPI accordingly or clearly state that maker is
for unthreaded Perl only.

During the debugging we also found that it was beneficial to have the
latest mxm.c installed:

https://community.mellanox.com/thread/3439


Best Regards

John Damm Sørensen

IT consultant


_______________________________________________
maker-devel mailing list
[hidden email]
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: maker with MPI and perl using threads

Carson Holt-2
MAKER doesn’t use threads. It uses forks. There are several reasons for this, including that Perl already requires you to use hidden fork operations every time you call system() or open(). So trying to get forks, threads, and MPI together is just a mess. So we stuck with just forks and MPI. With MPI flavors with direct infiniband support you can get weird errors because of how forks affect registered memory when the MPI flavor uses OpenIB libraries. There is also another issue with how perl wraps malloc calls that affects registered memory. The best way around both these issues is too disable direct infiniband support, and then set the MPI flavor to use -tcp over the ip-over-infiniband virtual adaptor (usually ib0) or use the ethernet adapter. Sometime last year after a system update we also got an OpenMPI error (only on CentOS6) that referred to a file used for Perl threads (which should not even be in use since we are using forks). We worked around that issue by compile Perl without thread support so that the library couldn't be called.

If you were using OpenMPI and were seeing a reference to Perl threads, then the error you saw may have been related to the latter one I mentioned.

I have tried the MPI_Init_thread option before and had run into issues because of it without it helping any of the previously mentioned fork related issues. But that was some time ago, so I could try it as a solution to the last issue I mentioned rather than installing a no-thread version of perl (if I can ever replicate the error because it went away when we updated from CentOS kernel 6 to kernel 7).

Thanks,
Carson


> On Jun 28, 2017, at 2:54 AM, John Damm Sørensen <[hidden email]> wrote:
>
> Hello,
>
> Recently I assisted one of my customers with problems solving maker using MPI.
>
> It seems that the main reason for the trouble was maker not initializing the MPI environment for thread save execution.
>
> In the MPI.pm module you call MPI_Init whereas for a threaded environment you should call MPI_Init_thread.
>
> I think it would be a good idea to detect whether the users Perl is with thread support and init MPI accordingly or clearly state that maker is for unthreaded Perl only.
>
> During the debugging we also found that it was beneficial to have the latest mxm.c installed:
>
> https://community.mellanox.com/thread/3439
>
>
> Best Regards
>
> John Damm Sørensen
>
> IT consultant
>
>
> _______________________________________________
> maker-devel mailing list
> [hidden email]
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org


_______________________________________________
maker-devel mailing list
[hidden email]
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: maker with MPI and perl using threads

Carson Holt-2
Also when you see calls like threads->new() in the maker script. It is creating a fork and not a thread. It’s a convenience feature activated by the ‘use forks’ line at the top of the script. It allows you to use thread like syntax when working with forks.

—Carson



> On Jun 29, 2017, at 2:43 PM, Carson Holt <[hidden email]> wrote:
>
> MAKER doesn’t use threads. It uses forks. There are several reasons for this, including that Perl already requires you to use hidden fork operations every time you call system() or open(). So trying to get forks, threads, and MPI together is just a mess. So we stuck with just forks and MPI. With MPI flavors with direct infiniband support you can get weird errors because of how forks affect registered memory when the MPI flavor uses OpenIB libraries. There is also another issue with how perl wraps malloc calls that affects registered memory. The best way around both these issues is too disable direct infiniband support, and then set the MPI flavor to use -tcp over the ip-over-infiniband virtual adaptor (usually ib0) or use the ethernet adapter. Sometime last year after a system update we also got an OpenMPI error (only on CentOS6) that referred to a file used for Perl threads (which should not even be in use since we are using forks). We worked around that issue by compile Perl without thread support so that the library couldn't be called.
>
> If you were using OpenMPI and were seeing a reference to Perl threads, then the error you saw may have been related to the latter one I mentioned.
>
> I have tried the MPI_Init_thread option before and had run into issues because of it without it helping any of the previously mentioned fork related issues. But that was some time ago, so I could try it as a solution to the last issue I mentioned rather than installing a no-thread version of perl (if I can ever replicate the error because it went away when we updated from CentOS kernel 6 to kernel 7).
>
> Thanks,
> Carson
>
>
>> On Jun 28, 2017, at 2:54 AM, John Damm Sørensen <[hidden email]> wrote:
>>
>> Hello,
>>
>> Recently I assisted one of my customers with problems solving maker using MPI.
>>
>> It seems that the main reason for the trouble was maker not initializing the MPI environment for thread save execution.
>>
>> In the MPI.pm module you call MPI_Init whereas for a threaded environment you should call MPI_Init_thread.
>>
>> I think it would be a good idea to detect whether the users Perl is with thread support and init MPI accordingly or clearly state that maker is for unthreaded Perl only.
>>
>> During the debugging we also found that it was beneficial to have the latest mxm.c installed:
>>
>> https://community.mellanox.com/thread/3439
>>
>>
>> Best Regards
>>
>> John Damm Sørensen
>>
>> IT consultant
>>
>>
>> _______________________________________________
>> maker-devel mailing list
>> [hidden email]
>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>


_______________________________________________
maker-devel mailing list
[hidden email]
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: maker with MPI and perl using threads

John Damm Sørensen
Thanks Carson,

One thing bothers me. That's this from Perl forks documentation:

module load order: forks first

Since forks overrides core Perl functions, you are *strongly* encouraged
to load the forks module before any other Perl modules. This will insure
the most consistent and stable system behavior. This can be easily done
without affecting existing code, like:

     perl -Mforks  script.pl

But in the maker perlscript the module FindBin that in sturn loads a
bunch of other modules is loaded before forks.

Is that intentionally?

Best

John




Den 29-06-2017 kl. 22:56 skrev Carson Holt:

> Also when you see calls like threads->new() in the maker script. It is creating a fork and not a thread. It’s a convenience feature activated by the ‘use forks’ line at the top of the script. It allows you to use thread like syntax when working with forks.
>
> —Carson
>
>
>
>> On Jun 29, 2017, at 2:43 PM, Carson Holt <[hidden email]> wrote:
>>
>> MAKER doesn’t use threads. It uses forks. There are several reasons for this, including that Perl already requires you to use hidden fork operations every time you call system() or open(). So trying to get forks, threads, and MPI together is just a mess. So we stuck with just forks and MPI. With MPI flavors with direct infiniband support you can get weird errors because of how forks affect registered memory when the MPI flavor uses OpenIB libraries. There is also another issue with how perl wraps malloc calls that affects registered memory. The best way around both these issues is too disable direct infiniband support, and then set the MPI flavor to use -tcp over the ip-over-infiniband virtual adaptor (usually ib0) or use the ethernet adapter. Sometime last year after a system update we also got an OpenMPI error (only on CentOS6) that referred to a file used for Perl threads (which should not even be in use since we are using forks). We worked around that issue by compile Perl without thread support so that the library couldn't be called.
>>
>> If you were using OpenMPI and were seeing a reference to Perl threads, then the error you saw may have been related to the latter one I mentioned.
>>
>> I have tried the MPI_Init_thread option before and had run into issues because of it without it helping any of the previously mentioned fork related issues. But that was some time ago, so I could try it as a solution to the last issue I mentioned rather than installing a no-thread version of perl (if I can ever replicate the error because it went away when we updated from CentOS kernel 6 to kernel 7).
>>
>> Thanks,
>> Carson
>>
>>
>>> On Jun 28, 2017, at 2:54 AM, John Damm Sørensen <[hidden email]> wrote:
>>>
>>> Hello,
>>>
>>> Recently I assisted one of my customers with problems solving maker using MPI.
>>>
>>> It seems that the main reason for the trouble was maker not initializing the MPI environment for thread save execution.
>>>
>>> In the MPI.pm module you call MPI_Init whereas for a threaded environment you should call MPI_Init_thread.
>>>
>>> I think it would be a good idea to detect whether the users Perl is with thread support and init MPI accordingly or clearly state that maker is for unthreaded Perl only.
>>>
>>> During the debugging we also found that it was beneficial to have the latest mxm.c installed:
>>>
>>> https://community.mellanox.com/thread/3439
>>>
>>>
>>> Best Regards
>>>
>>> John Damm Sørensen
>>>
>>> IT consultant
>>>
>>>
>>> _______________________________________________
>>> maker-devel mailing list
>>> [hidden email]
>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org



_______________________________________________
maker-devel mailing list
[hidden email]
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: maker with MPI and perl using threads

Carson Holt-2
FindBin is necessary for library control and is safe to load before forks.

—Carson


> On Jul 1, 2017, at 11:38 AM, John Damm Sørensen <[hidden email]> wrote:
>
> Thanks Carson,
>
> One thing bothers me. That's this from Perl forks documentation:
>
> module load order: forks first
>
> Since forks overrides core Perl functions, you are *strongly* encouraged to load the forks module before any other Perl modules. This will insure the most consistent and stable system behavior. This can be easily done without affecting existing code, like:
>
>    perl -Mforks  script.pl
>
> But in the maker perlscript the module FindBin that in sturn loads a bunch of other modules is loaded before forks.
>
> Is that intentionally?
>
> Best
>
> John
>
>
>
>
> Den 29-06-2017 kl. 22:56 skrev Carson Holt:
>> Also when you see calls like threads->new() in the maker script. It is creating a fork and not a thread. It’s a convenience feature activated by the ‘use forks’ line at the top of the script. It allows you to use thread like syntax when working with forks.
>>
>> —Carson
>>
>>
>>
>>> On Jun 29, 2017, at 2:43 PM, Carson Holt <[hidden email]> wrote:
>>>
>>> MAKER doesn’t use threads. It uses forks. There are several reasons for this, including that Perl already requires you to use hidden fork operations every time you call system() or open(). So trying to get forks, threads, and MPI together is just a mess. So we stuck with just forks and MPI. With MPI flavors with direct infiniband support you can get weird errors because of how forks affect registered memory when the MPI flavor uses OpenIB libraries. There is also another issue with how perl wraps malloc calls that affects registered memory. The best way around both these issues is too disable direct infiniband support, and then set the MPI flavor to use -tcp over the ip-over-infiniband virtual adaptor (usually ib0) or use the ethernet adapter. Sometime last year after a system update we also got an OpenMPI error (only on CentOS6) that referred to a file used for Perl threads (which should not even be in use since we are using forks). We worked around that issue by compile Perl without thread support so that the library couldn't be called.
>>>
>>> If you were using OpenMPI and were seeing a reference to Perl threads, then the error you saw may have been related to the latter one I mentioned.
>>>
>>> I have tried the MPI_Init_thread option before and had run into issues because of it without it helping any of the previously mentioned fork related issues. But that was some time ago, so I could try it as a solution to the last issue I mentioned rather than installing a no-thread version of perl (if I can ever replicate the error because it went away when we updated from CentOS kernel 6 to kernel 7).
>>>
>>> Thanks,
>>> Carson
>>>
>>>
>>>> On Jun 28, 2017, at 2:54 AM, John Damm Sørensen <[hidden email]> wrote:
>>>>
>>>> Hello,
>>>>
>>>> Recently I assisted one of my customers with problems solving maker using MPI.
>>>>
>>>> It seems that the main reason for the trouble was maker not initializing the MPI environment for thread save execution.
>>>>
>>>> In the MPI.pm module you call MPI_Init whereas for a threaded environment you should call MPI_Init_thread.
>>>>
>>>> I think it would be a good idea to detect whether the users Perl is with thread support and init MPI accordingly or clearly state that maker is for unthreaded Perl only.
>>>>
>>>> During the debugging we also found that it was beneficial to have the latest mxm.c installed:
>>>>
>>>> https://community.mellanox.com/thread/3439
>>>>
>>>>
>>>> Best Regards
>>>>
>>>> John Damm Sørensen
>>>>
>>>> IT consultant
>>>>
>>>>
>>>> _______________________________________________
>>>> maker-devel mailing list
>>>> [hidden email]
>>>> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
>
>


_______________________________________________
maker-devel mailing list
[hidden email]
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
Loading...