[gridengine users] qstat xml output performance

Ed Lauzier elauzier2 at perlstar.com
Wed Jul 31 15:05:19 UTC 2013


 Hi Kevin,

Thanks for the input.
It turns out that qstat is slow anyways even w/o the xml output,
and it takes up a bit of memory in the process also.

120,000 jobs in the system, qstat with xml output consumes 1.5 GB ram
resident and virtual as seen in top. Clearly, qstat is less than optimal
in managing memory when xml output is desired. Without xml output,
qstat consumes approx 700 MB of ram, which is ok....I guess...

The speed to process qstat output w/o xml is still slow, and with xml
is approx 20% slower ( for 120,000 jobs in the system...)

So, there may be no real way to speed the querying process up w/o
an investment in time in looking at the qstat code and how it handles
processing. It may be tied to the scheduler so we may be kinda out
of luck here.

One question is: Will going to Univa Grid Engine and using the new database
spooling help speed things up?

One way to find out. I'll look into obtaining the new code for testing to see
if indeed it is faster in handling qstat requests when 120,000 jobs are
in the system.

Thanks,

Ed


-----Original Message-----
From: Kevin Buckley [mailto:kevin.buckley.ecs.vuw.ac.nz at gmail.com]
Sent: Wednesday, July 31, 2013 02:31 AM
To: 'Ed Lauzier'
Cc: sge-discuss at liverpool.ac.uk, users at gridengine.org
Subject: Re: [gridengine users] qstat xml output performance

On 31 July 2013 15:42, Ed Lauzier  wrote:
 > With 120,000 jobs in the system, it takes 33 seconds to process the
> following output:
>
> qstat -u "*" -F
> "num_proc,mem_free,mem_used,virtual_free,cpu,load_ave,load_short,load_medium,load_long"
> -r -ext -pri -xml > myqstatxmlfile.xml
> ...
> Are there any suggestions to reduce the time to process the qstat xml
> output?
> Do things speed up if you don't try to force the qstat command to limitthe output fields and just do a plainqstat -u "*" -F-r -ext -pri -xml > myqstatxmlfile.xml
> to get everything and then process the XML (with XSLT or grep) to get what youactually want?
> Kevin M. Buckley
> eScience ConsultantSchool of Engineering and Computer Science
> Victoria University of Wellington
> New Zealand
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gridengine.org/pipermail/users/attachments/20130731/432ae380/attachment.html>


More information about the users mailing list