[gridengine users] problems with $TMPDIR set to nfs mounted file system

[Marlies Hankel]

Dear all,

I found a problem with openMPI jobs when I set the $TMP/$TMPDIR to an 
nfs mounted storage. OpenMPI (version 1.8.4 with intel) writes out an 
error message that it will create some files there which might slow 
things down. And this is exactly what happens. I actually have the 
executables in a D state instead of R and things are ridiculously slow.

The problem does not happen when I use qlogin and I found that 
$TMP/$TMPDIR are not set and so I assume openMPI is using local /tmp 
instead? Also, all is fine when I run directly on the nodes. Also, 
things wehre fien before I switched SGE temp to the nfs file system.

I need the SGE temp directory to be the nfs mounted file system, so I 
cannot change that. But does somebody know how I can tell openMPI to use 
local /tmp?

Out of curiosity, why does a jobs via qsub set $TMPDIR and qlogin does 
not? What is the difference between the two? I have been using the 
qlogin for testing when things have gone wrong via qsub assuming that 
the environment set up would be the same.

Best wishes and thank you in advance for your help

Marlies

-- 

ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms

Please note change of work hours: Monday, Wednesday and Friday

Dr. Marlies Hankel
Research Fellow
High Performance Computing, Quantum Dynamics&  Nanotechnology
Theory and Computational Molecular Sciences Group
Room 229 Australian Institute for Bioengineering and Nanotechnology  (75)
The University of Queensland
Qld 4072, Brisbane
Australia
Tel: +61 (0)7-33463996
Fax: +61 (0)7-334 63992
mobile:+61 (0)404262445
Email: m.hankel at uq.edu.au
http://web.aibn.uq.edu.au/cbn/

ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms

Notice: If you receive this e-mail by mistake, please notify me, and do
not make any use of its contents. I do not waive any privilege,
confidentiality or copyright associated with it. Unless stated
otherwise, this e-mail represents only the views of the Sender and not
the views of The University of Queensland.