[gridengine users] Defining and launching jobs programmatically

Reuti reuti at staff.uni-marburg.de
Wed Nov 28 22:40:50 UTC 2012


Hi,

Am 28.11.2012 um 23:02 schrieb Lane Schwartz:

> I'm trying to figure out a way to define and launch jobs programmatically.
> 
> The idea is that I am writing a computationally intensive program in
> some programming language (most typically Java or Scala, but sometimes
> C, C++, Ruby). I would like to be able, within my programming
> language, to define some unit of computation that is then launched as
> an SGE job, with the results returned to the original program.
> 
> It appears that something like this sort of thing exists in one form
> or another (e.g. hadoop, jppf) for various programming languages, but
> as far as I have been able to determine, these frameworks tend to
> require their own grid setup. Meaning, I can set up a hadoop or jppf
> cluster, but that cluster won't interact with my SGE cluster in terms
> of scheduling jobs.
> 
> I am aware of DRMAA, but as far as I can tell it isn't what I am
> looking for. I have successfully used DRMAA to launch (from within
> Java) SGE jobs defined as external scripts. But what I'm looking for
> is a mechanism for defining the job itself as Java code to be
> executed. Or, if I'm writing in C++, as C++ code.
> 
> I won't be surprised if nothing like this exists, but with all of the
> work that so many people have done in grid environments over the
> years, I'm hoping that something like this exists and I just haven't
> heard about it.
> 
> Any pointers or suggestions would be greatly appreciated.

I'm not sure whether you mean something like this which you could also put into a C application:

$ cat offshore.sh 
#!/bin/sh
if [ $# -eq 1 -a "$1" = "SLAVE" ]; then
    echo "Executing slave part."
else
    echo "Executing master part."
    qrsh $0 SLAVE
fi


The problem is of course to get the values back into the application, maybe by some temporary file which you read then in the master part again or catching the stdout.

If it's just to split parts of the application into master plus slaves and MPI isn't appropriate, you could try PVM where the different tasks of the applications can be spawned to another machine in the cluster and send the input and results between the machines. If it's working with PVM on its own, it can be integrated into SGE too. Just submit the master task and the slaves can be spawned by your applications directly (all inside the granted slots for your parallel job then).

-- Reuti




More information about the users mailing list