[gridengine users] Strange SGE PE issue (threaded PE with 999 slots but scheduler thinks the value is 0)

Reuti reuti at staff.uni-marburg.de
Thu Jun 11 21:01:25 UTC 2020


Am 11.06.2020 um 22:44 schrieb Chris Dagdigian:

> 
> The root cause was strange so it's worth documenting here ...
> 
> I had created a new consumable and requestable resource called "gpu" configured like this:
> 
> gpu                 gpu        INT       <=    YES         YES        NONE        0
> 
> And on host A I had set "complex_values gpu=1" and on host B I set "complex_values gpu=2" etc. etc. across the cluster. 
> 
> My mistake was setting the default value of the new complex entry to "NONE" instead of "0" which is what you probably want when the attribute is of type INT
> 
> But this was bizzare;  basically I had a bad default value for a requestable resource and as soon as we set that value down at the execution host level it instantly broke all of our parallel environments.  SGE scheduler was treating my mistake like I had created a requestable resource of type FORCED or something. 

Aha, a couple of days ago I got a request in PM where someone swore that the configuration "h_vmem …  YES YES 0 0" was working fine all the time. Only after my suggestion to add h_vmem on an exechost level to avoid oversubscription all the jobs crashed then, due to no memory being available (as h_vmem = 0 was used this way as an automatically set limit).

Essentially: the default value in a complex definition is ignored, as long as there is nothing to consume from. If it's not ignored, then the type has to match.

-- Reuti


> 
> Strange but resolved now. 
> 
> Regards
> Chris
> 
> 
> 
> 
> Reuti wrote on 6/11/20 4:17 PM:
>> Hi,
>> 
>> Any consumables in place like memory or other resource requests? Any output of `qalter -w v …` or "-w p"?
>> 
>> -- Reuti
>> 
>> 
>> 
>>> Am 11.06.2020 um 20:32 schrieb Chris Dagdigian <dag at sonsorol.org>
>>> :
>>> 
>>> Hi folks,
>>> 
>>> Got a bewildering situation I've never seen before with simple SMP/threaded PE techniques
>>> 
>>> I made a brand new PE called threaded:
>>> 
>>> $ qconf -sp threaded
>>> pe_name            threaded
>>> slots              999
>>> user_lists         NONE
>>> xuser_lists        NONE
>>> start_proc_args    NONE
>>> stop_proc_args     NONE
>>> allocation_rule    $pe_slots
>>> control_slaves     FALSE
>>> job_is_first_task  TRUE
>>> urgency_slots      min
>>> accounting_summary FALSE
>>> qsort_args         NONE
>>> 
>>> 
>>> And I attached that to all.q on an IDLE grid and submitted a job with '-pe threaded 1' argument
>>> 
>>> However all "qstat -j" data is showing this scheduler decision line:
>>> 
>>> cannot run in PE "threaded" because it only offers 0 slots
>>> 
>>> 
>>> I'm sort of lost on how to debug this because I can't figure out how to probe where SGE is keeping track of PE specific slots.  With other stuff I can look at complex_values reported by execution hosts or I can use an "-F" argument to qstat to dump the live state and status of a requestable resource but I don't really have any debug or troubleshooting ideas for "how to figure out why SGE thinks there are 0 slots when the static PE on an idle cluster has. been set to contain 999 slots" 
>>> 
>>> Anyone seen something like this before?  I don't think I've ever seen this particular issue with an SGE parallel environment before ...
>>> 
>>> 
>>> Chris
>>> 
>>> _______________________________________________
>>> users mailing list
>>> 
>>> users at gridengine.org
>>> https://gridengine.org/mailman/listinfo/users
> 




More information about the users mailing list