[gridengine users] problems with $TMPDIR set to nfs mounted file system

Marlies Hankel m.hankel at uq.edu.au
Wed Apr 29 13:01:01 UTC 2015

Answering my own question:

setting -mca orte_tmpdir_base /tmp solved the problem.



On 04/29/2015 10:43 PM, Marlies Hankel wrote:
> Dear all,
> I found a problem with openMPI jobs when I set the $TMP/$TMPDIR to an 
> nfs mounted storage. OpenMPI (version 1.8.4 with intel) writes out an 
> error message that it will create some files there which might slow 
> things down. And this is exactly what happens. I actually have the 
> executables in a D state instead of R and things are ridiculously slow.
> The problem does not happen when I use qlogin and I found that 
> $TMP/$TMPDIR are not set and so I assume openMPI is using local /tmp 
> instead? Also, all is fine when I run directly on the nodes. Also, 
> things wehre fien before I switched SGE temp to the nfs file system.
> I need the SGE temp directory to be the nfs mounted file system, so I 
> cannot change that. But does somebody know how I can tell openMPI to 
> use local /tmp?
> Out of curiosity, why does a jobs via qsub set $TMPDIR and qlogin does 
> not? What is the difference between the two? I have been using the 
> qlogin for testing when things have gone wrong via qsub assuming that 
> the environment set up would be the same.
> Best wishes and thank you in advance for your help
> Marlies



Please note change of work hours: Monday, Wednesday and Friday

Dr. Marlies Hankel
Research Fellow
High Performance Computing, Quantum Dynamics&  Nanotechnology
Theory and Computational Molecular Sciences Group
Room 229 Australian Institute for Bioengineering and Nanotechnology  (75)
The University of Queensland
Qld 4072, Brisbane
Tel: +61 (0)7-33463996
Fax: +61 (0)7-334 63992
mobile:+61 (0)404262445
Email: m.hankel at uq.edu.au


Notice: If you receive this e-mail by mistake, please notify me, and do
not make any use of its contents. I do not waive any privilege,
confidentiality or copyright associated with it. Unless stated
otherwise, this e-mail represents only the views of the Sender and not
the views of The University of Queensland.

More information about the users mailing list