[gridengine users] SGE-6.2u5: Problem with scheduling of array tasks of multiple jobs.
e.soyez at science-computing.de
Tue Jun 23 15:48:57 UTC 2015
many thanks for your quick reply! Okay, I need to specify:
>From the scheduler's point of view the jobs are identical. They have no
"-l" resource requirements. And yes, "load" is the only criteria which
restricts access to a node. No nodes in alarm state. Any other ideas?
I'm almost sure it has to do with the "job_load_adjustment", I just cannot
prove it yet.... :-)
On Tue, 23 Jun 2015, William Hay wrote:
> On Tue, 23 Jun 2015 12:33:27 +0000 Erik Soyez <e.soyez at science-computing.de> wrote:
>> we have a very peculiar array job scheduling problem.
>> Cluster: Very heterogeneous SGE cluster with different types of
>> Problem: One array job can use more job slots than two array jobs
>> Example: User A submitted an array job, 64 array tasks were running
>> at the same time. Then user B submitted a similar array job and
>> after a while each of the jobs was running with approx 28 array
>> tasks. According to the users the problem gets worse with each array
>> job. Terminating one of the jobs immediately leads to the normal
>> usage again.
>> Question: Does anybody know if the "job_load_adjustment" of array
>> jobs depends only on number of tasks or if the number of jobs is
>> taken into account as well?
> It should be just array tasks. Is load the only criteria which
> restricts access to a node in your cluster?
> Is there any clue from looking at the actual nodes. Do certain nodes
> go into alarm with fewer jobs?
>> From my point of view limits, quotas, etc. cannot be the cause of the
>> problem because then the total of job slots being used by two jobs
>> could not be less then the job slots being used by one single job.
>> The scheduler configuration though was not made for short array jobs
>> but for longer running parallel jobs on interactively used
> You claim the jobs are similar but that isn't the same as identical
> This might explain the difference. With identical jobs grid engine will
> obtain optimal packing of jobs onto nodes simply by fitting as many
> jobs onto each node as it can. For jobs with differing requirements
> and nodes with different resources there is no simple strategy for
> doing this (indeed it smells like an NP-hard problem to me)
> Imagine two nodes with consumables as shown:
> h_vmem=4G diskspace=30G
> h_vmem=6G diskspace=20G
> Array job 1 requests 2G h_vmem and 15G of diskspace
> Array job 2 requests 3G of h_vmem and 10G of diskspace
> If a task from array job 1 is scheduled to nodeB there isn't room for
> another task of either job on that node(diskspace shortage). If a task
> from array job 2 is scheduled to nodeA there isn't room for another
> task of either job on that node(h_vmem shortage). On the other hand you
> can fit two tasks of array job 1 on node A and two tasks of array job 2
> on nodeB.
Vorstandsvorsitzender/Chairman of the board of management:
Vorstand/Board of Management:
Dr. Bernd Finkbeiner, Dr. Arno Steitz
Vorsitzender des Aufsichtsrats/
Chairman of the Supervisory Board:
Sitz/Registered Office: Tuebingen
Registergericht/Registration Court: Stuttgart
Registernummer/Commercial Register No.: HRB 382196
More information about the users