[gridengine users] how create and monitor a new consumable?

Reuti reuti at staff.uni-marburg.de
Fri Apr 10 09:40:22 UTC 2015

Hi Marlies,

> Am 10.04.2015 um 06:05 schrieb Marlies Hankel <m.hankel at uq.edu.au>:
> Dear all,
> I am using OGS/Grid Engine 2011.11 as installed under ROCKS 6..1.1.
> We have a global storage which hosts all home directories and which also has a large space which I would like to use as scratch space for jobs. As the storage is faster and so much bigger than the small local disk I would like $TMPDIR to default to /scratch. It will also help with some of our MPI applications that need global scratch space.
> I know I can set the tmp directory to /scratch but I would also like to have the space as a consumable complex that can be requested by the users if scratch space is needed.

This is quite like any consumable with the type being MEMORY calling it "disk" or alike. As it's a global consumable its total available value must be defined in the `qconf -me global`.

Any limits of a file size in "ulimit" would be per process. So there are other means necessary to control eht overall size. There are two ways to control it, but it's not build into SGE:

- looking up the additional group ID for each job controlled by SGE and adding up all open files by checking `lsof`. If the size passes the defined limit, the process could be killed. Already closed file will be missed of course. Or maybe easier:

- For the /scratch directory the upper level permissions can be set to:

node: # ls -dl /scratch
drwxr-xr-t 7 root root 4096 2015-04-10 11:25 /scratch

Hence noone can write at this level, but it allows SGE to create a directory there (as root user) and change the ownership with the usual result that the jobs can write into their dedicated $TMPDIR. A background process could then much easier than before check the size of each of the <job_id_>.<task_id:+1>.<q_name> directories and act accordingly by issuing a `qdel`. For sake of easiness this can be put into the load sensor which outputs the consumed space for the defined complex. It's possible to have a consumable which will also get a value from a load sensor. The tighter restriction will be used, being it the internal book keeping of SGE or teh actual load value.

-- Reuti

> I also would like jobs to be killed if they go over the requested amount.
> I can set up a complex named scratch and make this consumable but how to I make sure that jobs do not go over the requested amount? Or is there already a complex that would do this I could use?
> Thanks in advance
> Marlies
> -- 
> ------------------
> Dr. Marlies Hankel
> Research Fellow, Theory and Computation Group
> Australian Institute for Bioengineering and Nanotechnology (Bldg 75)
> eResearch Analyst, Research Computing Centre and Queensland Cyber Infrastructure Foundation
> The University of Queensland
> Qld 4072, Brisbane, Australia
> Tel: +61 7 334 63996 | Fax: +61 7 334 63992 | mobile:0404262445
> Email: m.hankel at uq.edu.au | www.theory-computation.uq.edu.au
> Notice: If you receive this e-mail by mistake, please notify me,
> and do not make any use of its contents. I do not waive any
> privilege, confidentiality or copyright associated with it. Unless
> stated otherwise, this e-mail represents only the views of the
> Sender and not the views of The University of Queensland.
> _______________________________________________
> users mailing list
> users at gridengine.org
> https://gridengine.org/mailman/listinfo/users

More information about the users mailing list