[gridengine users] Epilog to print out usage summary?

Reuti reuti at staff.uni-marburg.de
Fri Jan 23 10:49:32 UTC 2015


Am 23.01.2015 um 09:23 schrieb James Abbott <j.abbott at imperial.ac.uk>:
> 
> Do you rotate your accounting file at all? qacct performance suffers badly when the accounting file gets too big. I  drop a file containing the following into /etc/logrotate.d on the qmaster (replacing the path to the accounting file to it's location on the system...) :
> 
> /opt/sge/current/bss/common/accounting {
>     compress
>     nocreate
>     dateext
>     delaycompress
>     ifempty
>     # keep logs essentually "forever"
>     rotate 5000
>     monthly
> }
> 
> which gives me a date-stamped, gzipped monthly accounting file, so qacct continues to perform in a sensible way.
> 
> James
> 
> On 23/01/15 01:55, David Chin wrote:

Another email I didn't get - and it's also not in my spam folder.

>> Dear GE users:
>> 
>> Is it possible to have an epilog script which prints a summary of resource usage (wallclock time, mem, cpu-seconds)?
>> 
>> I have tried an epilog bash script which calls qacct, but it took a long time to generate that output.

AFAIK the accounting file is written by the qmaster after he got the summary of the shepherd via the sgeexecd. So it should be impossible to access this information in an epilog. What "could" work to access this information in a mail wrapper, as the email from the node is also send after the job left the node already (besides a possible race condition here).

I.e. using the information in the delivered subject of the email to grep for the record in the accounting file (the two cases of a normal job and an array job task need to be distinguished.)

#!/bin/sh
#
# Distinguish between normal jobs and an array job.
#
case `echo "$2" | cut -d " " -f 1` in

      Job) JOB_ID=`echo "$2" | cut -d " " -f 2`
           CONDITION=`echo "$2" | cut -d " " -f 4` ;;

Job-array) JOB_ID=`echo "$2" | cut -d " " -f 3`
           CONDITION=`echo "$2" | cut -d " " -f 5` ;;

        *) ;;

esac
...

Sure, this would mean that a) an email must be requested and b) the mail wrapper must append the information to the output file. I use the job context also for other stuff and safe it along the name of the output/error files in a job prolog in a place where the email wrapper can gather the information and remove it after the email.

-- Reuti


>> Thanks,
>>     Dave Chin
>> 
>> -- 
>> David Chin, Ph.D.
>> david.chin at drexel.edu    Sr. Systems Administrator, URCF, Drexel U.
>> http://www.drexel.edu/research/urcf/        https://linuxfollies.blogspot.com/
>> 215.221.4747 (mobile)                       https://github.com/prehensilecode
>> 
>> 
>> 
>> _______________________________________________
>> users mailing list
>> 
>> users at gridengine.org
>> https://gridengine.org/mailman/listinfo/users
> 
> 
> -- 
> Dr. James Abbott
> Lead Bioinformatician
> Bioinformatics Support Service
> Imperial College, London
> 
> _______________________________________________
> users mailing list
> users at gridengine.org
> https://gridengine.org/mailman/listinfo/users





More information about the users mailing list