[gridengine users] Hadoop Integration - how's it going?

[Prakashan Korambath]

This is how we run hadoop using Grid Engine (for that matter 
any scheduler with appropriate alteration)

http://www.ats.ucla.edu/clusters/hoffman2/hadoop/default.htm

Basically, run either a prolog  or call a script inside the 
submission command file itself to parse the output of 
PE_HOSTFILE to create hadoop *.site.xml, masters and slaves 
files at run time.  This methodology is suitable for any 
scheduler as it is not dependent on them. If there is 
interest I can post the prologue script. Thanks.

Prakashan


On 05/28/2012 06:50 AM, Rayson Ho wrote:
> Vic,
>
> If you just want to run Hadoop jobs with Grid Engine, then the
> integration (mainly a HDFS monitor) written by DanT in SGE 6.2u5 will
> work:
>
> https://blogs.oracle.com/templedf/entry/welcome_sun_grid_engine_6
>
> The on-going discussion is related to running jobs that request
> dynamic allocation - it is going to be more complicated... and we have
> not even defined the interface yet!
>
> Rayson
>
>
>
>
> On Mon, May 28, 2012 at 7:35 AM, Vic<gridengine at beer.org.uk>  wrote:
>>
>> Hi All.
>>
>> I've just re-read the thread from a few months ago about integrating SGE
>> with Hadoop. This might suddenly have become very useful to me!
>>
>> So what sort of state is it in? Is it the sort of thing I can get my hands
>> dirty with (bearing in mind I'm a SGE neophyte), or will I get my fingers
>> burnt?
>>
>> Thanks!
>>
>> Vic.
>>
>>
>>
>> _______________________________________________
>> users mailing list
>> users at gridengine.org
>> https://gridengine.org/mailman/listinfo/users
> _______________________________________________
> users mailing list
> users at gridengine.org
> https://gridengine.org/mailman/listinfo/users