Hyperthreading

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Hyperthreading

System Admin
We are using maker in a cluster with mpich.  Currently hyperthreading is
on and we use 'mpiexec -n <cores>' to start maker.  Our machinelist file
for mpich specifies the total emulated cores for each node.
With hyperthreading on, we have up to 256 total emulated cores available.

Which is the optimal scenario?
1. Use '-n 256'
2. Use '-n 128' with hyperthreading still on
3. Use '-n 128' with hyperthreading turned off

Thanks

_______________________________________________
maker-devel mailing list
[hidden email]
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Hyperthreading

Carson Holt-2
MAKER is more of a pipeline. It will launch external tools on as many CPUs as you give it with the mpiexec command. I’ve found that many of the tools used get a boost with hyperthreading even though optimizations are not explicitly built into their code. The short answer is you would have to try it both ways. I doubt there will be much more than a 10-15% difference in runtime.

You can pull back to 128 would if you find that you are running low on RAM or have a high IO burden (both of which will double if you go from 128 to 256 even though CPU isn’t really doubling). Also MAKER per job performance plateaus at around 200 processes due to communication overhead. Above that threshold it is often useful to divide datasets into multiple separate jobs that can run simultaneously.

—Carson



> On May 23, 2017, at 1:52 PM, System Admin <[hidden email]> wrote:
>
> We are using maker in a cluster with mpich.  Currently hyperthreading is on and we use 'mpiexec -n <cores>' to start maker.  Our machinelist file for mpich specifies the total emulated cores for each node.
> With hyperthreading on, we have up to 256 total emulated cores available.
>
> Which is the optimal scenario?
> 1. Use '-n 256'
> 2. Use '-n 128' with hyperthreading still on
> 3. Use '-n 128' with hyperthreading turned off
>
> Thanks
>
> _______________________________________________
> maker-devel mailing list
> [hidden email]
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org


_______________________________________________
maker-devel mailing list
[hidden email]
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Hyperthreading

System Admin
Carson Holt wrote on 05/23/2017 01:19 PM:
> You can pull back to 128 would if you find that you are running low on RAM or have a high IO burden (both of which will double if you go from 128 to 256 even though CPU isn’t really doubling). Also MAKER per job performance plateaus at around 200 processes due to communication overhead. Above that threshold it is often useful to divide datasets into multiple separate jobs that can run simultaneously.

Yes, with '-n 192' we found the load on the cluster will initially go up
to 360-380 but then continually decreases until maker is finished.
Memory usage was very low during the processing (under 20%).

_______________________________________________
maker-devel mailing list
[hidden email]
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Hyperthreading

Carson Holt-2
Also make sure cpus in the control file are set to 1 when using MPI. Otherwise it will tell each program it calls to try and use more CPUs per call.

—Carson

> On May 23, 2017, at 2:31 PM, [hidden email] wrote:
>
> Carson Holt wrote on 05/23/2017 01:19 PM:
>> You can pull back to 128 would if you find that you are running low on RAM or have a high IO burden (both of which will double if you go from 128 to 256 even though CPU isn’t really doubling). Also MAKER per job performance plateaus at around 200 processes due to communication overhead. Above that threshold it is often useful to divide datasets into multiple separate jobs that can run simultaneously.
>
> Yes, with '-n 192' we found the load on the cluster will initially go up to 360-380 but then continually decreases until maker is finished. Memory usage was very low during the processing (under 20%).
>
> _______________________________________________
> maker-devel mailing list
> [hidden email]
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org


_______________________________________________
maker-devel mailing list
[hidden email]
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Hyperthreading

Martin MOKREJŠ
In reply to this post by System Admin
[hidden email] wrote:
> Carson Holt wrote on 05/23/2017 01:19 PM:
>> You can pull back to 128 would if you find that you are running low on RAM or have a high IO burden (both of which will double if you go from 128 to 256 even though CPU isn’t really doubling). Also MAKER per job performance plateaus at around 200 processes due to communication overhead. Above that threshold it is often useful to divide datasets into multiple separate jobs that can run simultaneously.
>
> Yes, with '-n 192' we found the load on the cluster will initially go up to 360-380 but then continually decreases until maker is finished. Memory usage was very low during the processing (under 20%).

Hi,
 the high load could be caused by disk IO or other reasons. The only proof is to run top, htop or similar and check that the processes are in *running* state ("R" is displayed in the status column). There could be "S" (sleep) when task is waiting for data input or output and also "D"(disk) coudl be shown when waiting for disk IO (unlike network IO).
Martin

_______________________________________________
maker-devel mailing list
[hidden email]
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Hyperthreading

Martin MOKREJŠ
In reply to this post by System Admin
System Admin wrote:
> We are using maker in a cluster with mpich.  Currently hyperthreading is on and we use 'mpiexec -n <cores>' to start maker.  Our machinelist file for mpich specifies the total emulated cores for each node.
> With hyperthreading on, we have up to 256 total emulated cores available.
>
> Which is the optimal scenario?
> 1. Use '-n 256'
> 2. Use '-n 128' with hyperthreading still on
> 3. Use '-n 128' with hyperthreading turned off

Go for 3. but make sure to disable *hyperthreading* in the kernel of the machines as well. I also disable multicore scheduler (which should again be helping if there are more long-term running processes than physical cores available and if some should probably share a cache). We do not have such jobs, hmmer and blast are mostly accessing data from memory, so the CPU cache is not much relevant for these.
  Hyperthreading only helps if jobs are lousy, waiting for some input/output etc., and in that case *it helps* if another process can be executed on the CPU core (hopefully not having same bottleneck). This is generally a helped in bad situations. You are after good setup, so disable hyperthreading in kernel, load only that many jobs equal to the number of physical CPI cores, and monitor performance. If jobs are starving, resolve the issue.

Martin

_______________________________________________
maker-devel mailing list
[hidden email]
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Hyperthreading

System Admin
In reply to this post by Carson Holt-2
Carson Holt wrote on 05/23/2017 01:38 PM:
> Also make sure cpus in the control file are set to 1 when using MPI.
> Otherwise it will tell each program it calls to try and use more
> CPUs per call.

Yes we are using cpus=1 in the control file
Thanks

_______________________________________________
maker-devel mailing list
[hidden email]
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Hyperthreading

Carson Holt-2
In reply to this post by System Admin
One last thing to check if using CentOS or RedHat. I’ve seen it happen on a handful of clusters where transparent hugepages can create odd load issues and very high sys CPU usage under top (not just with maker but with BWA, GATK, and other programs that can have larger memory footprints). If using CentOS or RedHat, you may want to disable defrag for hugepages.

You do this on CentOS 6 to disable it (the process is similar on CentOS 7 and RedHat but you may have to google it) —>

echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/defrag

—Carson


> On May 23, 2017, at 2:31 PM, [hidden email] wrote:
>
> Carson Holt wrote on 05/23/2017 01:19 PM:
>> You can pull back to 128 would if you find that you are running low on RAM or have a high IO burden (both of which will double if you go from 128 to 256 even though CPU isn’t really doubling). Also MAKER per job performance plateaus at around 200 processes due to communication overhead. Above that threshold it is often useful to divide datasets into multiple separate jobs that can run simultaneously.
>
> Yes, with '-n 192' we found the load on the cluster will initially go up to 360-380 but then continually decreases until maker is finished. Memory usage was very low during the processing (under 20%).
>
> _______________________________________________
> maker-devel mailing list
> [hidden email]
> http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org


_______________________________________________
maker-devel mailing list
[hidden email]
http://box290.bluehost.com/mailman/listinfo/maker-devel_yandell-lab.org
Loading...