[gridengine users] problems with $TMPDIR set to nfs mounted file system

Reuti reuti at staff.uni-marburg.de
Wed Apr 29 13:23:41 UTC 2015


> Am 29.04.2015 um 14:43 schrieb Marlies Hankel <m.hankel at uq.edu.au>:
> Dear all,
> I found a problem with openMPI jobs when I set the $TMP/$TMPDIR to an nfs mounted storage. OpenMPI (version 1.8.4 with intel) writes out an error message that it will create some files there which might slow things down. And this is exactly what happens. I actually have the executables in a D state instead of R and things are ridiculously slow.
> The problem does not happen when I use qlogin and I found that $TMP/$TMPDIR are not set and so I assume openMPI is using local /tmp instead? Also, all is fine when I run directly on the nodes. Also, things wehre fien before I switched SGE temp to the nfs file system.
> I need the SGE temp directory to be the nfs mounted file system, so I cannot change that. But does somebody know how I can tell openMPI to use local /tmp?
> Out of curiosity, why does a jobs via qsub set $TMPDIR and qlogin does not? What is the difference between the two? I have been using the qlogin for testing when things have gone wrong via qsub assuming that the environment set up would be the same.

With `qlogin` you get a fresh shell without any set environment. You can get all the variables there when you source in .profile resp. .bash_profile or .bash_login:


MYPARENT=`ps -p $$ -o ppid --no-header`
#MYPARENT=`ps -p $MYPARENT -o ppid --no-header`
#MYPARENT=`ps -p $MYPARENT -o ppid --no-header`
MYSTARTUP=`ps -p $MYPARENT -o command --no-header`

if [ "${MYSTARTUP:0:13}" = "sge_shepherd-" ]; then
   echo "Running inside SGE" 
   echo "Job $MYJOBID"

   set -a
   while read LINE; do export $LINE; done < /var/spool/sge/${HOSTNAME%%.*}/active_jobs/$MYJOBID.1/environment
   unset HISTFILE



The number of calls to set "MYPARENT" depend on the startup ("builtin" vs. "ssh").

-- Reuti

> Best wishes and thank you in advance for your help
> Marlies
> -- 
> ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms
> Please note change of work hours: Monday, Wednesday and Friday
> Dr. Marlies Hankel
> Research Fellow
> High Performance Computing, Quantum Dynamics&  Nanotechnology
> Theory and Computational Molecular Sciences Group
> Room 229 Australian Institute for Bioengineering and Nanotechnology  (75)
> The University of Queensland
> Qld 4072, Brisbane
> Australia
> Tel: +61 (0)7-33463996
> Fax: +61 (0)7-334 63992
> mobile:+61 (0)404262445
> Email: m.hankel at uq.edu.au
> http://web.aibn.uq.edu.au/cbn/
> ccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccmsccms
> Notice: If you receive this e-mail by mistake, please notify me, and do
> not make any use of its contents. I do not waive any privilege,
> confidentiality or copyright associated with it. Unless stated
> otherwise, this e-mail represents only the views of the Sender and not
> the views of The University of Queensland.
> _______________________________________________
> users mailing list
> users at gridengine.org
> https://gridengine.org/mailman/listinfo/users

More information about the users mailing list