[gridengine users] RoundRobin scheduling among users

Dan Hyatt dhyatt at dsgmail.wustl.edu
Tue Jan 26 18:30:50 UTC 2016


   I am looking to use this differently.
The problem I am having is that I have users with 200-1000 jobs. I have 
80 servers with almost 1000 cores.
For my normal queue,  I want SGE PE to create up to 4 jobs per server 
until it runs out of servers, then add up to 4 more until all the jobs 
are allocated.  (1 per is fine as long as it will round robin and start 
adding a second job per server, then a third until it runs out of jobs)

Does the allocation rule limit the number of jobs per server PER qsub, 
or total jobs allowed per server?

The problem I am having is that I get 20 jobs per server and overload a 
couple of servers while 80 servers running idle. Each has 10 cores and 
128 GB of RAM so they can handle up to 20 light jobs each.

Also, for the heavy CPU jobs, I want a max of 4 jobs per server, so for 
pe_slots would I just put the integer 4 in there?

Should I create a third PE, lets say "dan" with the desired settings?  
When I tried this before it would throw errors.


Am I correct that I want to change these settings, but I suspect I 
really want to make a custom PE, these are default.

I was looking at http://linux.die.net/man/5/sge_pe  and 
http://www.softpanorama.org/HPC/Grid_engine/parallel_environment.shtml 
but seems to assume I comprehend the details of each.. Such as...can I 
only put one setting for allocation rule per PE and one PE per queue?


[root at blade5-1-1 ~]# qconf -sp make
pe_name            make
slots              999
user_lists         NONE
xuser_lists        NONE
start_proc_args    NONE
stop_proc_args     NONE
allocation_rule    $round_robin
control_slaves     TRUE
job_is_first_task  FALSE
urgency_slots      min
accounting_summary TRUE
qsort_args         NONE

[root at blade5-1-1 ~]# qconf -sp smp
pe_name            smp
slots              999
user_lists         NONE
xuser_lists        NONE
start_proc_args    NONE
stop_proc_args     NONE
allocation_rule    $pe_slots
control_slaves     TRUE
job_is_first_task  TRUE
urgency_slots      min
accounting_summary TRUE
qsort_args         NONE
[root at blade5-1-1 ~]# echo $pe_slots



[root at blade5-1-1 ~]# qconf -sp DAN
pe_name           DAN
slots              999
user_lists         NONE
xuser_lists        NONE
start_proc_args    NONE
stop_proc_args     NONE
allocation_rule    $round_robin
control_slaves     TRUE
job_is_first_task  FALSE
urgency_slots      min
accounting_summary TRUE
qsort_args         NONE

[root at blade5-1-1 ~]# qconf -sp smp
pe_name            smp
slots              999
user_lists         NONE
xuser_lists        NONE
start_proc_args    NONE
stop_proc_args     NONE
allocation_rule    4
control_slaves     TRUE
job_is_first_task  TRUE
urgency_slots      min
accounting_summary TRUE
qsort_args         NONE
[root at blade5-1-1 ~]# echo $pe_slots

>>> Yep, we use functional tickets to accomplish this exact goal. Every user
>>> gets 1000 functional tickets via auto_user_fshare in sge_conf(5), though
>>> your exact number will depend on the number tickets and weights you have
>>> elsewhere in your policy configuration.
>> Also the waiting time should be set to 0, and less importance of the urgency (as the default is to give 1000 per slot in the complex configuration - this means more slots results in being more important):
>>
>> weight_user                       0.900000
>> weight_project                    0.000000
>> weight_department                 0.000000
>> weight_job                        0.100000
>> weight_tickets_functional         1000000
>> weight_tickets_share              0
>> share_override_tickets            TRUE
>> share_functional_shares           TRUE
>> max_functional_jobs_to_schedule   200
>> report_pjob_tickets               TRUE
>> max_pending_tasks_per_job         50
>> halflife_decay_list               none
>> policy_hierarchy                  F
>> weight_ticket                     1.000000
>> weight_waiting_time               0.000000
>> weight_deadline                   3600000.000000
>> weight_urgency                    0.100000
>> weight_priority                   1.000000
>> max_reservation                   32
>> default_duration                  8760:00:00
> We actually do weight waiting time, but at half the value of both
> functional and urgency tickets. We then give big urgency boosts to
> difficult-to-schedule jobs (i.e. lots of memory or CPUs in one spot). It
> took us a while to arrive at a decent mix of short-run / small jobs vs
> long-run / big jobs, and it definitely will be a site-dependent decision.
>




More information about the users mailing list