[gridengine users] suggestions on setting up queues

Chris Dagdigian dag at sonsorol.org
Fri Jan 16 20:51:23 UTC 2015


Queues are just a piece of the puzzle when it comes to handling resource 
allocation on a multi user system, what (if any)  scheduling policies 
and resource quotas are you currently using?

That said you are using the queue methods in a good way. There are 
certain things that can only be really done on a per-queue basis and top 
of the list would be ACL protection and the ability to impose hard or 
soft wallclock limits.

A fairshare-by-user policy with the queue structure you set up would be 
a decent starting point from which you can gather more data and user 
feedback.

Thoughts

  - resource quota would perfectly handle the "only N jobs per user can 
run in the long-job.q cluster queue ..."

  - I've had little success putting wallclock limits on interactive 
queues; there are legit business/scientific reasons in many cases for a 
long running interactive session. You might want to poll the users or 
collect data on this. In a few different environments I've had decent 
success by leaving interactive queue slots unrestricted but putting a 
resource quota around how many slots a single user can consume. It's 
also pretty easy to set up tools that would allow you to dynamically 
adjust the size/count of the interactive slot pool to account for 
changing demand - it's particularly easy when used with SGE hostgroup 
objects.

My $.02





> Stephen Spencer <mailto:spencer at cs.washington.edu>
> January 16, 2015 at 2:50 PM
> Good morning.
>
> With the number of users on our clusters growing, it's becoming less 
> realistic to say "play fair 'cause you're not the only user of the 
> cluster."
>
> I'm looking for suggestions on setting up queues, both the "why" and 
> "how," that will allow more of our users access to the cluster.
>
> What I'm thinking of is a multi-queue approach:
>
>   * some limited number of "interactive" slots (and they'd be
>     time-limited)
>   * a queue for jobs with short time duration - the "express" queue
>   * a queue for jobs that will run longer... but only so many of these
>     per user
>
> Any and all suggestions are welcome.
>
> Thank you!
>
> Best,
> -- 
> Stephen Spencer
> spencer at cs.washington.edu <mailto:spencer at cs.washington.edu>
> _______________________________________________
> users mailing list
> users at gridengine.org
> https://gridengine.org/mailman/listinfo/users



More information about the users mailing list