[gridengine users] Complex value consumption in parallel jobs
simon.andrews at babraham.ac.uk
Tue Jan 27 13:02:46 UTC 2015
On 27/01/2015 12:29, "Maria Mierscheid" <reuti at staff.uni-marburg.de> wrote:
>Am 27.01.2015 um 13:03 schrieb Simon Andrews
><simon.andrews at babraham.ac.uk>:
>> I've spent the morning tracking down a scheduling problem on our cluster
>> which arose from a misunderstanding on how complex values and parallel
>> environments interact.
>> In our setup we have configured h_vmem to be consumable so we can
>> based on the memory requirements of the jobs. We also have a parallel
>> environment set up for SMP jobs which allow the user to reserve multiple
>> cores on the same physical machine.
>> This morning we found a load of jobs which couldn't be scheduled despite
>> us appearing to have plenty of memory and cores free. Other jobs with
>> similar memory requirements and numbers of cores were able to be
>> scheduled, but this one set of jobs would only stay queued.
>> We eventually figured out that this was because when we set a pe request
>> and an h_vmem request, that the actual reservation of memory multiplies
>> the h_vmem by the number of cores, so we were actually requesting about
>> 10X the memory we thought we were after. I can see that for MPI type
>> this makes plenty of sense since they are running independently,
>> potentially on different machines. For SMP jobs though we're actually
>> just running different threads so it seems odd to have to make our users
>> calculate a 'memory per core' value, rather than an overall value for
>> Is there therefore any way to configure this behaviour within a pe? I
>> couldn't see anything obvious in the pe or complex config, but this must
>> have been something people have addressed before. For memory it's not
>> bad in that we can at least just divide the allocation, but for
>> like licenses where you only need one for a large SMP job I can't see
>> you could set this up.
>No. It can be configured on a complex level (consumable YES vs. JOB) but
>then you have the opposite problem for MPI jobs. I also had this idea in
>the past to define the multiplication in the PE:
>What you can do: in a JSV you can read the actual requested name of the
>PE and divide the overall request of memory by the number of specified
Thankyou! I hadn't been aware of setting consumable to JOB, and that does
exactly what I need. Since we don't run MPI jobs here this does exactly
what we need, but I can see how this would be problematic for people who
ran a mixture of jobs. I'd thought about messing around with this in a
JSV which might work should we get a wider mix of jobs, but for now this
fixes our problem.
The Babraham Institute, Babraham Research Campus, Cambridge CB22 3AT Registered Charity No. 1053902.
The information transmitted in this email is directed only to the addressee. If you received this in error, please contact the sender and delete this email from your system. The contents of this e-mail are the views of the sender and do not necessarily represent the views of the Babraham Institute. Full conditions at: www.babraham.ac.uk<http://www.babraham.ac.uk/terms>
More information about the users