[gridengine users] FW: [SGE-discuss] Can prolog see resource requests?

Pacey, Mike m.pacey at lancaster.ac.uk
Tue Jul 23 11:03:21 UTC 2013


Hi all,

Thanks for the responses; I now have a working GPU reservation system. It's along the lines I originally planned (ie, a file-based db managed via prolog and epilog scripts) with a couple of the suggestions outlined here:

- I've used a very simple server JSV script to pass the GPU resource request as an environment variable to the job, so that the prolog can see it

- I've switched from setting CUDA_VISIBLE_DEVICES to William's technique of making the reserved devices group-accessible only to the job's SGE group, so that jobs can't undo card reservation post launch.

To answer some of Dave's questions:

- Do we have multiple cards/host?

Yes -we currently have two cards per host, though I've built the scripts to make it easy to cope with differing numbers of cards per host

- Do we need multiple hosts per job?

We have a couple of apps that use MPI, OpenMP and GPUs all-in-one, but I've not seen  jobs that need to span hosts. It's a capability the LSF extension offered, so it's something I'm likely to return to after I've looked at parallel job setup.

- How is it done under LSF?

Platform provided us with an application job starter and a load sensor - both are binaries without source, so it's a bit of a black box. There's no trace of anything that looks like an external database.

Regards,
Mike.

-----Original Message-----
From: William Hay [mailto:w.hay at ucl.ac.uk] 
Sent: 19 July 2013 08:50
Cc: Pacey, Mike; users at gridengine.org
Subject: Re: [SGE-discuss] Can prolog see resource requests?

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 18/07/13 14:09, Pacey, Mike wrote:
> Hi folks,
> 
> I'm looking to build a GPU-card reservation subsystem into our local 
> SGE setup, in order to prevent jobs grabbing more GPU cards than they 
> request. Univa's RSMAP seems to offer the functionality I'm after, but 
> it looks like I can build a simple, GPU-specific approximation with a 
> bit of shell scripting.
> 
> I'm starting with the standard recipe of creating a consumable gpu 
> resource/complex. The next step would be to reserve free cards on a 
> host via a simple central database. The problem I'm having is that 
> neither the script, the prolog nor the starter_method seem to have 
> access to a job's complex/resource request, so I can't see how many 
> cards have been reserved. It's not available as an environment 
> variable, and a trawl of $SGE_JOB_SPOOL_DIR/ shows quite a few files 
> (including the promising-looking config file), but not the data I'm 
> after.
> 
> To progress, I'm currently scraping the resource request out of the 
> job's own copy of its submission script, which has at least one 
> obvious drawback. Am I missing a trick here?
> 
> Regards, Mike.
> 
Among other things our server side JSV sets an environment variable in the job for any resource the epilog/prolog/starter may need to process.  Not wholly without drawbacks but slightly fewer than submission script scraping.

We also use this for GPUs.  Our prolog/epilog scripts manipulate lockfiles to determine which GPUs to allocate and chgrps the
/dev/nvidia* files to the job's additional group id to control access.

William



> -----
> 
> Dr Mike Pacey,                                          Email:
> M.Pacey at lancaster.ac.uk<mailto:M.Pacey at lancaster.ac.uk> HPC
> Manager,                                            Phone: 01524
> 510659 Information Systems Services,            Fax: 01524 594459 
> ISS Building, Lancaster University, Lancaster LA1 4WA
> 
> 




More information about the users mailing list