[gridengine users] Anyone using S-GAE reporting app with Univa grid engine?

Tina Friedrich Tina.Friedrich at diamond.ac.uk
Tue Mar 3 11:45:25 UTC 2015


I've been meaning to move to XDMoD for a while now (from UBMoD). Any 
pitfalls to watch out for? Does it still (like UBMoD) index on Job ID 
for SGE (I patched it internally to add a date stamp to that, as my job 
IDs ran over the old 9,999,999 twice in my accounting records...)

I second the 'keep the raw logs'.

Tina

On 02/03/15 21:21, Jesse Becker wrote:
> I spent a bit of time looking at things to replace ARCO--which I found
> more trouble than it's worth.  You *could* get it to work, but it was
> alsways more trouble that it was worth to do so, especially since it's
> simple to parse the accounting and reporting files.
>
> I looked at a few different open source projects to do SGE reporting
> including S-GAE (which looked good).  I went with XDMoD largely because
> the it superceeded UBMoD, the built-in reporting abilities are
> decent, and it supports multiple schedulers.  I also seriously considered
> writing my own, and put some work towards that (also an excuse to play
> with Redis a bit).  But then we'd just have N+1 implementations[1].
>
> I'd be very interested in hearing about S-GAE, since I didn't get around
> to playing much with it.
>
> One thing that we *have* learned is that you should keep all of the
> raw records.  They compress well, and disk space is cheap.  Our UGE
> logs compress about 85% using gzip -9, and is fast.  Other methods
> (xz) get almost 90%, but take about 100 times longer to compress.
> (The specific method doesn't matter, even LZO would do nicely).
>
> This is important, because you can "quickly" re-ingest all of your
> historical records into a new system in case you:
>
>     1) change systems.
>     2) botch an ingest and have to start over
>     3) Have a catestrophic failure of {host,database,hardware}and have
>        to recover.
>
> In the case of XDMoD, "backfilling" records requires a little trickery
> based on how they are processed, but it's nothing too complicated.
>
>
>
>
> [1] Obligatory XKCD:  http://xkcd.com/927/
>
>
>
> On Mon, Mar 02, 2015 at 11:54:26AM -0500, Chris Dagdigian wrote:
>>
>> ooh the  various MoD ("metrics on demand") look pretty interesting.
>> Would love to chat about how people have made XDMoD and other variants
>> work with Grid Engine(s) -- can we get a little thread going on best
>> practices and recommendations for 3rd party reporting/metrics tools?
>> Suspect there would be decent interest in this ...
>>
>> -Chris
>>
>>
>>> Tina Friedrich <mailto:Tina.Friedrich at diamond.ac.uk>
>>> March 2, 2015 at 11:37 AM
>>> Yes, there's an additional field - job_class.
>>>
>>> I'm not using S-GAE, so got nothing for you I'm afraid; I had a
>>> similar problem with UBMoD (which I'm still running), where I had to
>>> make (probably similar) changes to make it work (keep it working,
>>> rather).
>>>
>>> Tina
>>>
>>>
>>>
>>>
>> _______________________________________________
>> users mailing list
>> users at gridengine.org
>> https://gridengine.org/mailman/listinfo/users
>


-- 
Tina Friedrich, Computer Systems Administrator, Diamond Light Source Ltd
Diamond House, Harwell Science and Innovation Campus - 01235 77 8442

-- 
This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. 
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom



More information about the users mailing list