[gridengine users] Hybrid PE

Hans-Werner Paulsen hans at MPA-Garching.MPG.DE
Fri Jul 22 13:10:34 UTC 2016

Dear all,
this is with SoGE 8.1.8 and OpenMPI 1.10.2. Our cluster has 24 nodes 
with 20 cores each (40 with HT).
I am looking for information how to setup a PE for hybrid jobs (OpenMP 
jobs running on more than one node). I created a PE with 
"allocation_rule 20", and this seems to work fine with 
OMP_NUM_THREADS=20. Then a user wanted to run a job with 
OMP_NUM_THREADS=5 and used "mpirun -n $((NSLOTS/OMP_NUM_THREADS))", but 
this does not work correctly, on some nodes there are running 20*5=100 
threads, on some nodes there are no running threads. Creating a new PE 
with "allocation_rule 5" has the disadvantage, that one cannot use more 
than 24*5=120 slots.
Best regards,

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4927 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://gridengine.org/pipermail/users/attachments/20160722/626f9bcb/attachment.p7s>

More information about the users mailing list