[gridengine users] Accessing qacct accounting file from login/compute nodes

Skylar Thompson skylar2 at uw.edu
Thu Feb 21 17:06:26 UTC 2019


We just use the Splunk forwarder to read the reporting file as it's
generated (turns out I mis-spoke - we only forward reporting since it's a
superset of the accounting log), classify it with a custom source type, and
then do automatic field extraction on the Splunk server. Splunk makes this
easy since it's just a colon-delimited file, and we assign names to each
field based on the reporting(5) man page.

Some of the more general dashboards we have:

* Job start delay (difference between submission and start times)
* Average job wallclock time
* Number of successful/failed jobs

On Thu, Feb 21, 2019 at 02:18:45PM +1100, Derrick Lin wrote:
> Hi Skylar,
> 
> Thanks for the info. I will look into the way you suggested.
> 
> BTW, can you tell me more about sending the accounting and reporting files
> to Splunk? We also want to do some dashboards too.
> 
> Cheers,
> 
> On Thu, Feb 21, 2019 at 3:20 AM Skylar Thompson <skylar2 at u.washington.edu>
> wrote:
> 
> > We actually don't have a shared $SGE_ROOT, so that in the event of
> > network/storage trouble the binaries and libraries are still accessible. We
> > do have $SGE_ROOT/$SGE_CELL on network storage via a symlink, so the
> > accounting and reporting files are accessible. We also send the accounting
> > and reporting files to Splunk so we can generate some dashboards for our
> > useres.
> >
> > On Wed, Feb 20, 2019 at 10:12:08AM +0000, Tina Friedrich wrote:
> > > I was about to ask that :)
> > >
> > > $SGE_ROOT ought to be accessible from (the) submit host(s), at least. So
> > > in general, you should be able to access it from there?
> > >
> > > (Note that you can also tell qacct where the accounting file lives - it
> > > assumes a default location, but the file does not have be in that
> > location.)
> > >
> > > Tina
> > >
> > > On 20/02/2019 07:09, Reuti wrote:
> > > > Hi,
> > > >
> > > >> Am 20.02.2019 um 05:31 schrieb Derrick Lin <klin938 at gmail.com>:
> > > >>
> > > >> Hi guys,
> > > >>
> > > >> On our SGE cluster, the accounting file stored on the qmaster node
> > and is not accessible outside. qmaster node is not accessible by any user
> > either.
> > > >>
> > > >> Now we have users request to obtain accounting info via qacct. I am
> > wondering what is the common way to achieve this without giving access to
> > the qmaster node?
> > > >
> > > > You mean, $SGE_ROOT is not shared in your cluster?
> > > >
> > > > -- Reuti
> > > >
> > > >
> > > > _______________________________________________
> > > > users mailing list
> > > > users at gridengine.org
> > > > https://gridengine.org/mailman/listinfo/users
> > > >
> > >
> > > _______________________________________________
> > > users mailing list
> > > users at gridengine.org
> > > https://gridengine.org/mailman/listinfo/users
> >
> > --
> > -- Skylar Thompson (skylar2 at u.washington.edu)
> > -- Genome Sciences Department, System Administrator
> > -- Foege Building S046, (206)-685-7354
> > -- University of Washington School of Medicine
> > _______________________________________________
> > users mailing list
> > users at gridengine.org
> > https://gridengine.org/mailman/listinfo/users
> >

-- 
-- Skylar Thompson (skylar2 at u.washington.edu)
-- Genome Sciences Department, System Administrator
-- Foege Building S046, (206)-685-7354
-- University of Washington School of Medicine


More information about the users mailing list