[gridengine users] export of environment variables from start_proc_args

[Dave Love]

William Hay <w.hay at ucl.ac.uk> writes:

>>  In the case of OpenMP jobs using the #$ -v and #$ -V prefix would be superfluous as gridengine will start only one process. So the jsv method appears the only way?
> Not sure what you mean about -v and -V.  They control an environment
> variable which controls the number of threads not processes.  In any
> case the jsv has the exact
> same weakness.

I wouldn't call it a weakness.  The client-side JSV is meant to provide
a sensible, overridable default rather than fascism.

> If the user sets the environment variable from after
> the job has started running rather than via grid engine directives
> there
> is no way for grid engine to reach in and modify it (well no sane way
> at least).  I think the best you can do is:
> i)Tell the users "don't do that".
> ii)Ensure if they do "do that" then they rather than other users bear
> the consequences.  OGS recently added support for hwloc which if used
> with cpusets/cgroups should be able to prevent
> them from using cpus not assigned to them.

[The first non-proprietary hwloc support was in SGE, and that has
"slots"-type binding, useful as a default.]  Cgroups are orthogonal to
hwloc, like same as the old PLPA.  Hwloc just regards cgroups as
"administrative restriction".  They aren't relevant without a recent
Linux (specifically not if you're saddled with GridPP), and setting the
runtime variables seems fine for getting non-malicious OpenMP programs to
DTRT.

What you likely want to set for OpenMP jobs is thread affinity
corresponding to the SGE core binding.  For instance, GNU GOMP currently
ignores existing binding so you need GOMP_CPU_AFFINITY corresponding to
the SGE binding.  The upcoming SGE release always sets SGE_BINDING --
not just for the "env" type -- to enable that.  It's convenient to do in
an LD_PRELOAD corresponding to the specific runtime (e.g. via
module(1)), and actually that can set OMP_NUM_THREADS too.

-- 
Community Grid Engine:  http://arc.liv.ac.uk/SGE/