[gridengine users] resource management and over-subscription?

[Reuti]

Hi Julie,

Am 22.11.2011 um 22:10 schrieb Julie Ashworth:

> hi Reuti and all,
> 
> On 22-11-2011 11.40 +0100, Reuti wrote:
>>> suspend_threshold. But how does SGE choose which jobs get
>>> suspended? 
>> 
>> They are just hanging around there, just stopped. No memory, consumable resource or diskspace will be freed.
> ---end quoted text---
> 
> I have a simple follow-up question. You mentioned that consumable resources
> are freed when a job is suspended. 

sorry for the confusion: the "no" was meant for memory + consumable resource + diskspace (i.e. the $TMPDIR). None of them will be freed.


> I'm using sge6_2u5, and I manually suspended a job with 'qmod', but the 
> consumable resource 'slots' isn't freed. Perhaps 'qmod -s' doesn't free 
> the consumable resource?

Correct. Suspended jobs are still staying there in state "s" and all requested consumable resources are still blocked by the job.


> If so, can you tell me how to do this? I consider
> scripting it (and avoid over-subscription).

This was done in the past by some other users, i.e. increase the number of slots on this particular queue instance for some time and decrease it later again. But in the end it's also an over-subscription. And the problem with the consumable resource is still there.

==

What you can do if your jobs support it: use the checkpointing feature in SGE. A suspension of a job (or queue) will then reschedule the job and free the used resources this way. But:

- your software need to support it outside of SGE already
- SGE doesn't look ahead, means if a new job needs consumable resources like memory used by a running job, it will not trigger the suspension as the new job can't be scheduled (you would have to put the running job on hold, suspend it [this way it can't start instantly again and block again the high priority job], and later on release it again [by some kind of "co-scheduler"]). This way you could stay with one queue, and long running jobs would just have to request a checkpointing environment.

-- Reuti


> Below is output from some
> commands (with usernames modified).
> 
> 
> I have the consumable 'slots' defined:
> 
> # qconf -sc | grep slot
> slots               s          INT         <=    YES         YES        1        1000
> 
> 
> # and a resource quota defined for slots:
> # qconf -srqs max_slots_hosts
> {
>   name         max_slots_hosts
>   description  NONE
>   enabled      TRUE
>   limit        hosts {nx10.priv} to slots=12
>   limit        hosts {nx9.priv,nx11.priv} to slots=24
>   limit        hosts {nx1.priv,nx2.priv,nx3.priv,nx4.priv,nx5.priv,nx6.priv, \
>   nx7.priv,nx8.priv} to slots=7
> }
> 
> 
> I suspended jobs for user1 on nx10:
> 
> $ qhost -j -h nx10
> HOSTNAME                ARCH         NCPU  LOAD  MEMTOT  MEMUSE  SWAPTO  SWAPUS
> -------------------------------------------------------------------------------
> global                  -               -     -       -       -       -       -
> nx10                    lx24-amd64     24  2.10   94.4G   33.1G  240.0G     0.0
>   job-ID  prior   name       user         state submit/start at     queue      master ja-task-ID 
>   ----------------------------------------------------------------------------------------------
>    313758 0.55014 simremrand user1     s     11/11/2011 21:48:14 short.q at nx MASTER        
>    313760 0.55014 simremrand user1     s     11/11/2011 21:51:59 short.q at nx MASTER        
>    314291 0.55014 simrempc   user1     s     11/11/2011 22:00:44 short.q at nx MASTER        
>    314293 0.55014 simrempc   user1     s     11/11/2011 22:25:29 short.q at nx MASTER        
>    314295 0.55014 simrempc   user1     s     11/11/2011 22:31:14 short.q at nx MASTER        
>    314300 0.55014 simrempc   user1     s     11/11/2011 23:30:59 short.q at nx MASTER        
>    314301 0.55014 simrempc   user1     s     11/11/2011 23:40:29 short.q at nx MASTER        
>    314302 0.55014 simrempc   user1     s     11/11/2011 23:43:59 short.q at nx MASTER        
>    314305 0.55014 simrempc   user1     s     11/11/2011 23:48:44 short.q at nx MASTER        
>    314307 0.55014 simrempc   user1     s     11/12/2011 00:02:14 short.q at nx MASTER        
>    338758 0.55027 feat3b_fla user2     r     11/22/2011 12:26:00 short.q at nx MASTER 41
>    338758 0.55027 feat3b_fla user2     r     11/22/2011 12:32:30 short.q at nx MASTER 61
> 
> however, the 'slots' consumable isn't freed when a job is suspended:
> 
> $ qquota -u user1
> resource quota rule limit                filter
> --------------------------------------------------------------------------------
> max_slots_hosts/1  slots=12/12          hosts nx10
> max_slots_hosts/2  slots=24/24          hosts nx9
> max_slots_hosts/2  slots=24/24          hosts nx11
> max_slots_hosts/3  slots=7/7            hosts nx2
> max_slots_hosts/3  slots=7/7            hosts nx4
> max_slots_hosts/3  slots=7/7            hosts nx8
> max_slots_hosts/3  slots=7/7            hosts nx1
> max_slots_hosts/3  slots=7/7            hosts nx5
> max_slots_hosts/3  slots=7/7            hosts nx6
> max_slots_hosts/3  slots=7/7            hosts nx3
> max_slots_users/3  slots=70/70          users user1
> 
> 
> 
> Thanks!
> Best
> Julie
> 
> 
> -- 
> Julie Ashworth <julie.ashworth at berkeley.edu>
> http://www.neuro.berkeley.edu
> PGP Key ID: 0x17F013D2