[gridengine users] A couple of questions...

William Hay w.hay at ucl.ac.uk
Thu Jun 30 12:15:07 UTC 2011


On 29 June 2011 14:01, Vic <gridengine at beer.org.uk> wrote:
>
>> Can you alter your bsub -> qsub translater include a '-P <project>'
>> string?  That way you can use an RQS to throttle the the number of jobs
>> running at a time.
>
> That's the sort of thing I'm thinking about - but unfortunately, we have
> several users (who might be running several compilations each), so to use
> that sort of throttling would require some nifty scripting to pick up a
> new project per run; I'm not sure how to create that uniqueness yet. But
> this is the sort of ugliness I'm toying with..., :-)

Suggestion:
Create a pool of projects for use.  Put an RQS limit on each of these projects.
Create a file mapping projectname to pid.
A client side jsv that traces its ancestry through the process table.
If a quartus process is there
(if it always occurs at a predictable depth then that would simplify
things) then
get an exclusive lock on the mapping file
1.Check if any project is owned by your quartus pid if so you have
your project release the lock and carry on
2.If not then check for a disused project (quartus pid does not refer
to a running process, no jobs running or queued).  Update
it with your quartus pid release the lock and carry on.

If you have more than one submit node you'll need one pool of accounts
per submit node.


If you have a lot of quartus sessions then you might need a more
sophisticated mapping mechanism than a flatfile.

William


>
>> You also mentioned that this was "well into the iterator".  If you use
>> qsub, you essetially dump all of the jobs into the queue, and let SGE
>> deal with them when it can.  You could also use qrsh or "qsub -now y".
>> These will both block until the job is actually complete.
>
> I can't easily change the invocation sequence; Quartus is separating the
> jobs & dispatching each one to the grid. So if I let them all go without
> dependency settings, a couple of runs will completely swamp the grid for
> several hours, and no-one else can run anything. If I limit each user,
> that user becomes idle as soon as he's committed a run to the grid (won't
> be able to run anything else).
>
> It isn't pretty, is it?
>
> Vic.
>
> _______________________________________________
> users mailing list
> users at gridengine.org
> https://gridengine.org/mailman/listinfo/users
>




More information about the users mailing list