[gridengine users] Definitions of "complex" and "queue"?

Fritz Ferstl fferstl at univa.com
Tue May 10 06:02:28 UTC 2011


Let me provide some background regarding the goofy usage of the term 
"queue" in Grid Engine. No, it is not an artifact of German engineers 
being involved (trust me, I was the first and the lead ever since). The 
history of Grid Engine started late 1992 when we negotiated a license 
with Florida state university to productize and commercially distribute 
a public domain SW called DQS (Distributed Queuing System).

The notion of a queue as a place where jobs execute is rooted in DQS. 
That's how it was in DQS and we kept it that way for compatibility 
reasons in the follow-on product called CODINE, which was renamed into 
Grid Engine then much much later.

HOWEVER, this is no restriction - it even has some advantages. Classical 
queuing systems have one or more waiting queues. The problem with that 
is that jobs sort of get stuck in those queues once being in there. If 
conditions or configurations in the cluster change dynamically then the 
waiting queues jobs are in may not be best suited anymore. You might be 
able to re-queue a job but that his its issues also.

In Grid Engine all jobs wait in a single real queue - the pending jobs 
list. The resource requests and attributes you submit a job with 
associate it with one or several of the cluster queues which are 
defined. So while queues are really where jobs execute, that 
association, job characteristics to cluster queues, creates logical 
segmentation in the pending jobs list. Some job can go here, others can 
go there. The advantage is that this conserves full flexibility in the 
case of changes until a job really gets started. So if you change a 
queue configuration then new jobs may become eligible for it or others 
will not any longer. This will happen automatically. You don't need to 
re-queue jobs.

I hope this helps.

Cheers,

Fritz

PS: The term "complex" was also introduced not by us Germans. I'm not 
sure anymore whether it is another DQS artifact or something coming from 
the POSIX queuing system standard. Anyhow, the meaning is that it's a 
complex of attributes which is to be defined.

Am 10.05.11 05:37, schrieb Stuart Barkley:
> On Mon, 9 May 2011 at 14:17 -0000, Reuti wrote:
>
>>> I also have always had problems with the usage of "queue" in Grid
>>> Engine.
>>>
>>> To me a queue would be a set of jobs waiting to run, usually with
>>> an ordering (first come first serve, priority, random).
>>>
>>> Grid Engine seems to apply 'queue' to a group of resources with a
>>> common set of configuration/attributes.
>> Yes, "queue" is something to do work in, not to wait. But it maybe
>> already be defined this way in POSIX.
> Really?  To me a queue is where you wait until you get assigned to a
> worker.  You have the email queue (mail waiting to go out or waiting
> to be delivered).  In a network device the packet queue is where
> packets sit until they are able to be processed.
>
> As someone else pointed out, there might be a German/English language
> issue in play here.  What is the German word for this concept
> "something to do work in, not to wait"?
>
> This next bit is quite off topic.  Feel free to ignore...
>
> Using google translate starting from English "queue", most of the
> German translations map back to about the same concept in English.
>
> ** A few double translations (en->de->en) go a little off the mark.
>
> queue (noun) ->  queue ->
>     1. queue
>     2. cue ** (pronounced the same, but very different concept)
>
> (most of the other nouns are strange, but do translate back to queue
> in some form)
>
> queue (verb) ->  anstehen ->
>     1. queue
>     2. hesitate (**)
>     3. queue up
>     4. befit (**)
>     5. stand in line
>     6. be due
>     7. be on the agenda
>
> queue (verb) ->  Schlange stehen ->
>     1. queue
>     2. stand in line
>
> queue (verb) ->  sich anstellen ->
>     1. queue
>     2. behave (**)
>     3. queue up
>     4. act (**)
>     5. line up
>     6. stand in line
>     7. make a fuss (**)
>     8. act up (**)
>
> queue (verb) ->  eine Schlange bilden ->
>     1. queue
>
>>> As a simplified user concept the use of the word 'queue' can
>>> convey a useful model and also be a method to apply differing
>>> defaults to a job ("Submit to the short job queue").
>> No. Specify resource requests in time, memory,... and SGE will
>> select an appropriate queue for you. Submitting to a queue is
>> Torque/PBS style.
> Actually, this is what I prefer (request what you really want), but
> I've been unable to eradicate the concept of distinct queues from our
> users.  The Internet is full of simplified user instructions to submit
> jobs with "-q queuename" (unfortunately they usually don't show the
> implementing SGE configurations).
>
> I dislike the fact that someone created "long", "medium" and "short"
> as SGE queues for our users and told them all they need to do is
> select the correct queue for their job.  I can't change the parameters
> of these without users getting unexpected behavior changes.
>
> I do want to have resources with different wall clock limits so that
> heavy compute users don't grab all of the available resources for
> several weeks.  I think this is/should be another FAQ with a few
> completely worked example configurations.
>
> Do people actually have lots of queues?  Or would just 'interactive'
> and 'batch' work suitably as queues?
>
> I've picked up a lot of good information in the past couple of weeks.
> Now I need to get some time to work with things and try some more
> experiments.
>
> Thanks,
> Stuart

-- 

UnivaFritz Ferstl | CTO and Business Development, EMEA
Univa Corporation <http://www.univa.com/> | The Data Center Optimization 
Company
E-Mail: fferstl at univa.com | Phone: +49.9471.200.195 | Mobile: 
+49.170.819.7390

Where Grid Engine lives



---------------------------------------------------------------------


Notice from Univa Postmaster:


This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. This message has been content scanned by the Univa Mail system.



---------------------------------------------------------------------

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gridengine.org/pipermail/users/attachments/20110510/9e9f4fa9/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Grafik1
Type: image/png
Size: 4331 bytes
Desc: not available
URL: <http://gridengine.org/pipermail/users/attachments/20110510/9e9f4fa9/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Where Grid Engine lives
Type: image/png
Size: 5779 bytes
Desc: not available
URL: <http://gridengine.org/pipermail/users/attachments/20110510/9e9f4fa9/attachment-0001.png>


More information about the users mailing list