[gridengine users] FW: [SGE-discuss] Can prolog see resource requests?
m.pacey at lancaster.ac.uk
Tue Jul 23 11:03:21 UTC 2013
Thanks for the responses; I now have a working GPU reservation system. It's along the lines I originally planned (ie, a file-based db managed via prolog and epilog scripts) with a couple of the suggestions outlined here:
- I've used a very simple server JSV script to pass the GPU resource request as an environment variable to the job, so that the prolog can see it
- I've switched from setting CUDA_VISIBLE_DEVICES to William's technique of making the reserved devices group-accessible only to the job's SGE group, so that jobs can't undo card reservation post launch.
To answer some of Dave's questions:
- Do we have multiple cards/host?
Yes -we currently have two cards per host, though I've built the scripts to make it easy to cope with differing numbers of cards per host
- Do we need multiple hosts per job?
We have a couple of apps that use MPI, OpenMP and GPUs all-in-one, but I've not seen jobs that need to span hosts. It's a capability the LSF extension offered, so it's something I'm likely to return to after I've looked at parallel job setup.
- How is it done under LSF?
Platform provided us with an application job starter and a load sensor - both are binaries without source, so it's a bit of a black box. There's no trace of anything that looks like an external database.
From: William Hay [mailto:w.hay at ucl.ac.uk]
Sent: 19 July 2013 08:50
Cc: Pacey, Mike; users at gridengine.org
Subject: Re: [SGE-discuss] Can prolog see resource requests?
-----BEGIN PGP SIGNED MESSAGE-----
On 18/07/13 14:09, Pacey, Mike wrote:
> Hi folks,
> I'm looking to build a GPU-card reservation subsystem into our local
> SGE setup, in order to prevent jobs grabbing more GPU cards than they
> request. Univa's RSMAP seems to offer the functionality I'm after, but
> it looks like I can build a simple, GPU-specific approximation with a
> bit of shell scripting.
> I'm starting with the standard recipe of creating a consumable gpu
> resource/complex. The next step would be to reserve free cards on a
> host via a simple central database. The problem I'm having is that
> neither the script, the prolog nor the starter_method seem to have
> access to a job's complex/resource request, so I can't see how many
> cards have been reserved. It's not available as an environment
> variable, and a trawl of $SGE_JOB_SPOOL_DIR/ shows quite a few files
> (including the promising-looking config file), but not the data I'm
> To progress, I'm currently scraping the resource request out of the
> job's own copy of its submission script, which has at least one
> obvious drawback. Am I missing a trick here?
> Regards, Mike.
Among other things our server side JSV sets an environment variable in the job for any resource the epilog/prolog/starter may need to process. Not wholly without drawbacks but slightly fewer than submission script scraping.
We also use this for GPUs. Our prolog/epilog scripts manipulate lockfiles to determine which GPUs to allocate and chgrps the
/dev/nvidia* files to the job's additional group id to control access.
> Dr Mike Pacey, Email:
> M.Pacey at lancaster.ac.uk<mailto:M.Pacey at lancaster.ac.uk> HPC
> Manager, Phone: 01524
> 510659 Information Systems Services, Fax: 01524 594459
> ISS Building, Lancaster University, Lancaster LA1 4WA
More information about the users