[gridengine users] Fwd: dispatching sge task from an sge task - is that a reasonable practice?

Ben Daniel Pere ben.pere at gmail.com
Thu Feb 25 21:16:49 UTC 2016


Where I work, we have jobs that submit jobs that submit jobs.. this could
potentially cause a deadlock but we're somehow (probably luck) manage to
live with it.. I'm wondering if that's a reasonable practice and if not if
you can suggest a better way to do what we do..

Example:

we have these 3 tasks:

- "analyze.day" job analyzed a day of data and returns some output
- "analyze.month" job sends "analyze.day" jobs for a whole month and
outputs summary
- "analyze.year" job sends  "analyze.month" jobs for a whole year and
outputs summary

usually people run analyze.day everyday on previous day but sometimes they
test their new algorithm on a whole year so they dispatch analyze.year
which dispatched analyze.month which dispatched analyze.day..
We created a "dispatching" queue which is the only queue we allow
submitting jobs from but since both analyze.year and analyze.month need to
run there (both dispatch tasks) we could end up with a dead lock
(theoretically, lots of analyze.year running together taking all
dispatching queue slots and not leaving room for analyze.month tasks which
they will forever wait for), also besides dispatching they also do some
logic so it's a strange animal, this "dispatching" queue..

What's the "correct" practice here?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gridengine.org/pipermail/users/attachments/20160225/6fb868f0/attachment.html>


More information about the users mailing list