[gridengine users] coordinating multiple queues...

George Georgalis george at galis.org
Fri Jun 1 21:24:09 UTC 2012


Hi All,

I was wondering if anyone could confirm my plan or provide a better
solution for queue configuration?

First a bit of jobs and workload context. A small group who work together
submit jobs to our grid, these jobs can range from a few GB memory usage to
the max we allow per exec host (30 GB now, 384Gb next year). There are two
patterns of job submissions: short order and batch jobs (and each could
have a big or small memory footprint). When a user so specifies, we would
like their job to run immediately or as soon as there are cycles free on
the grid. Even though we have a backend high performance distributed
filesystem (gluster), our jobs are usually IO bound (we do a lot of one
time, random seeks across all of our big data, so caching is pretty much
always useless).

Administrative caveats:
+ cannot kill/restart jobs to manage queue
+ don't want to swap, ever; because big IO is a big drag
+ we run ge-6.1

Our plan for queue reconfiguration, is:
+ two queues: default.q and first.q
+ h_vmem as a consumable resource
+ set the first.q priority to something high, like 1000
+ nice down jobs sent to the default queue

Our 10 hosts have 8 cores each, so we'll configure slots in each queue as:

slots
 1,[host1.dom=8],[host2.dom=8],[host3.dom=8],[host4.dom=8],[host5.dom=8],[host6.dom=8],[host8.dom=8],[host7.dom=8],[host9.dom=8],[host10.dom=8]

and each host will be configured with (only) a h_vmem limit, in complex
values.

With this config we expect exec hosts to receive jobs from both queues (but
first.q first) until available memory is used up. We would set a default
memory footprint to 1/8 (may experiment with 1/16 or 1/24) available memory
In order to prevent too many threads per core. Since the default jobs come
with a with nice -n10, the jobs from the first.q would get most of the cpu
when both queues might have submit jobs on a single host.

"Run immediately" is not a hard requirement, it is really intended to
address getting new jobs (essentially interactive sessions) going asap and
before multi hour or multi day jobs finish. To that end maybe it would be
better not to configure all slots available to the default queue?

My main question is how to nice down the default queue? Should we make a
wrapper for qsub that prepends nice to the command? Another way? I'm also
curious if there is a better way to establish appropriate priorities for
the queues. And, generally, I welcome your expertise, perhaps there is a
better approach?

Looking forward to your reply!

Regards.
-George


-- 
George Georgalis, (415) 894-2710, http://www.galis.org/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gridengine.org/pipermail/users/attachments/20120601/105afdbf/attachment.html>


More information about the users mailing list