[gridengine users] Hadoop Integration - how's it going?
ppk at ats.ucla.edu
Mon May 28 16:00:24 UTC 2012
This is how we run hadoop using Grid Engine (for that matter
any scheduler with appropriate alteration)
Basically, run either a prolog or call a script inside the
submission command file itself to parse the output of
PE_HOSTFILE to create hadoop *.site.xml, masters and slaves
files at run time. This methodology is suitable for any
scheduler as it is not dependent on them. If there is
interest I can post the prologue script. Thanks.
On 05/28/2012 06:50 AM, Rayson Ho wrote:
> If you just want to run Hadoop jobs with Grid Engine, then the
> integration (mainly a HDFS monitor) written by DanT in SGE 6.2u5 will
> The on-going discussion is related to running jobs that request
> dynamic allocation - it is going to be more complicated... and we have
> not even defined the interface yet!
> On Mon, May 28, 2012 at 7:35 AM, Vic<gridengine at beer.org.uk> wrote:
>> Hi All.
>> I've just re-read the thread from a few months ago about integrating SGE
>> with Hadoop. This might suddenly have become very useful to me!
>> So what sort of state is it in? Is it the sort of thing I can get my hands
>> dirty with (bearing in mind I'm a SGE neophyte), or will I get my fingers
>> users mailing list
>> users at gridengine.org
> users mailing list
> users at gridengine.org
More information about the users