[gridengine users] Hosts not fully used with fill up

Reuti reuti at staff.uni-marburg.de
Mon Mar 9 13:18:13 UTC 2015


> Am 09.03.2015 um 14:02 schrieb Rafael Arco Arredondo <rafaarco at ugr.es>:
> Hello again,
> Now we have a different situation. I'll try to explain it as best as I
> can. We have two different sets of nodes, A and B, A with 16 cores per
> host and B with 32. We want to put all the nodes in only one cluster
> queue, which has configured two parallel environments as well, pe16 and
> pe32, pe16 with an allocation_rule of 16 and pe32 with 32. The queue has
> the following definition for the number of slots:

Before going into detail: did you attach both PEs to all machines, or was it split too like the slots:

pe_list pe16,[@B=pe32]


-- Reuti

> slots                 16,[@B=32]
> After submitting several jobs with 32 slots to the queue and pe16 as
> parallel environment, we observed that the first job starts running on
> two of the nodes -not only one- of the B set, when it could run in just
> one node. I understand this is because GE allocates the first 16 slots
> as a block, and then it uses the least loaded machine for the next 16
> (which, of course, is not the first machine). Then, the second job used
> 2 nodes of the A set, which is the behavior that we want.
> Of course, the easiest solution is two create two queues, but we don't
> want to do that. Is there any way to avoid the situation that happened
> with the first job? Ideally, we would like to be able to choose the set
> of nodes depending on the parallel environment (that is, if the user
> indicates pe16 then the job is run on set A, and if the user indicates
> pe32 the job is run on set B). I imagine the orthodox way of doing this
> is with two different cluster queues, but I want to make sure it's not
> possible with only one queue.
> We would also like to have more information about how GE selects the
> nodes in this type of situations, i.e. when the allocation rule is a
> fixed number and not fill_up or round_robin. Does it select the least
> loaded node? Or is it the most loaded one? Is there any detailed
> documentation about this? And besides, why does the fill_up rule
> sometimes leave fragmented nodes (that is, when -pe pe16 16 is used and
> 2 nodes are selected, each with 8 slots), even when there are no jobs
> using less than 16 slots? Perhaps because a user submits a job with,
> let's say, 8 slots and it is aborted with error state (Eqw) after
> failing in the prolog (where we control the number of slots be multiple
> of 16), and then the selected slots are kept reserved as if they are
> effectively used?
> I hope everything is clear enough. Thanks again for your help and best
> regards,
> Rafa
> El lun, 22-09-2014 a las 12:13 +0200, Rafael Arco Arredondo escribió:
>> Thanks Reuti, we'll try the fixed allocation rules. We will have to
>> create one or two more parallel environments for the different machine
>> types, but I hope it won't disturb our users much.
>> Rafa
>> El jue, 18-09-2014 a las 15:46 +0200, Reuti escribió:
>>> Am 18.09.2014 um 14:52 schrieb Rafael Arco Arredondo:
>>>> The hosts are free before running the jobs and are all identical in
>>>> terms of available resources.
>>>> Looking a bit deeper into the problem, it seems that sometimes jobs
>>>> requesting only 8 slots are executed on the nodes, overriding the check
>>>> in the prolog (which says the number of slots has to be multiple of 16).
>>> In the prolog? At this point the exechost selection was already done. Do you reschedule the job then?
>>> You can define fixed allocation rules for each type of machine, i.e. like "allocation_rule 8". Then a job requesting a multiple of 8 can select the PE by a wildcard.
>>> qsub -pe baz 32
>>> will get 4 times 8 cores for sure
>>> ===
>>> To make it a little bit flexible:
>>> PE: foo_2
>>> allocation_rule 2
>>> PE: foo_24
>>> allocation_rule 4
>>> PE: foo_248
>>> allocation_rule 8
>>> PE: foo_24816
>>> allocation_rule 16
>>> Submission command:
>>> qsub -pe foo* 16
>>> => gets 16 slots from any of the PEs (but only from one PE once it's selected), it could be 8 times 2 slots, or 4 times 4 slots
>>> If you don't want too many nodes:
>>> qsub -pe foo_248* 16
>>> I want at least 8 per machine, i.e. 2 times 8 cores or one time 16 cores
>>> -- Reuti
>>>> It's like the prolog sometimes wasn't executed...
>>>> El jue, 18-09-2014 a las 14:01 +0200, Winkler, Ursula
>>>> (ursula.winkler at uni-graz.at) escribió:
>>>>> Hi Rafa,
>>>>> Are the jobs scheduled on hosts where already other jobs are running (so that only 8 slots are used on some hosts)? Or are all hosts free? Have all nodes the same resources (i.e. slots, memory,...?) configured?
>>>>> C., U.
>>>>> -----Ursprüngliche Nachricht-----
>>>>> Von: Rafael Arco Arredondo [mailto:rafaarco at ugr.es] 
>>>>> Gesendet: Donnerstag, 18. September 2014 12:23
>>>>> An: Winkler, Ursula (ursula.winkler at uni-graz.at)
>>>>> Cc: users at gridengine.org
>>>>> Betreff: Re: AW: [gridengine users] Hosts not fully used with fill up
>>>>> Thanks Ursula for your reply, but we need a parallel environment for more than one node (we are using it with MPI). Sorry if I wasn't clear enough.
>>>>> I give you an example. We have hosts with 16 slots. Now the user requests 32 slots for a job, but instead of allocating slots on two nodes, sometimes four nodes are used, each with 8 slots. We don't know why this is happening, but it didn't happen with 6.2 with a similar configuration.
>>>>> Cheers,
>>>>> Rafa
>>>>> El jue, 18-09-2014 a las 12:11 +0200, Winkler, Ursula
>>>>> (ursula.winkler at uni-graz.at) escribió:
>>>>>> Hi Rafa,
>>>>>> for such purposes we have configured a separate Parallel Environment with "allocation_rule" "$pe_slots" (instead of "$fill_up"). Jobs scheduled with this rule can run ONLY on one host.
>>>>>> Regards,
>>>>>> Usula
>>>>>> -----Ursprüngliche Nachricht-----
>>>>>> Von: users-bounces at gridengine.org 
>>>>>> [mailto:users-bounces at gridengine.org] Im Auftrag von Rafael Arco 
>>>>>> Arredondo
>>>>>> Gesendet: Donnerstag, 18. September 2014 09:44
>>>>>> An: users at gridengine.org
>>>>>> Betreff: [gridengine users] Hosts not fully used with fill up
>>>>>> Hello everyone,
>>>>>> We are having an issue with the parallel environments and the allocation of slots with the fill up policy.
>>>>>> Although we have configured the resource quotas of the queues not to use more than the number of slots the machine have and we control in the prolog that the jobs be submitted with a number of slots multiple of the number of physical processors, we are observing that sometimes, the slots of a job are split into several nodes, when they should be running in only one node.
>>>>>> We are using Open Grid Scheduler 2011.11p1. This didn't happen in SGE 6.2.
>>>>>> Has anyone experienced the same situation? Any clues of why it is happening?
>>>>>> Thanks in advance,
>>>>>> Rafa
>>>>>> _______________________________________________
>>>>>> users mailing list
>>>>>> users at gridengine.org
>>>>>> https://gridengine.org/mailman/listinfo/users
>>>> _______________________________________________
>>>> users mailing list
>>>> users at gridengine.org
>>>> https://gridengine.org/mailman/listinfo/users
> -- 
> Rafael Arco Arredondo
> Centro de Servicios de Informática y Redes de Comunicaciones
> Campus de Fuentenueva - Edificio Mecenas
> Universidad de Granada
> E-18071 Granada Spain
> Tel: +34 958 241440   Ext:41440   E-mail: rafaarco at ugr.es
> ===============
> "Este mensaje se dirige exclusivamente a su destinatario y puede
> contener información privilegiada o confidencial. Si no es Ud. el
> destinatario indicado, queda notificado de que la utilización,
> divulgación o copia sin autorización está prohibida en virtud de la
> legislación vigente. Si ha recibido este mensaje por error, se ruega lo
> comunique inmediatamente por esta misma vía y proceda a su destrucción.
> This message is intended exclusively for its addressee and may contain
> information that is CONFIDENTIAL and protected by professional
> privilege. If you are not the intended recipient you are hereby notified
> that any dissemination, copy or disclosure of this communication is
> strictly prohibited by law. If this message has been received in error,
> please immediately notify us via e-mail and delete it".
> ================

More information about the users mailing list