[gridengine users] Welcome Home Grid Engine!

Fritz Ferstl fferstl at univa.com
Fri Oct 25 10:43:52 UTC 2013


> Yes and I did not mean to skip and forget all of the other folks who 
> contributed
> to what we know today as Grid Engine.
>
> If you dig far back enough and before it was CODINE, I am sure it started
> with someone writing some home grown code.
Here is an account of the history of the technology 
http://blogs.gridengine.com/content/history-sun-grid-engine and my 
team's 20+ years of involvement.

As Chris writes:
> Have you looked deep into the codebase? The
> learning curve is pretty extreme which can be a major obstacle to long
> term success of an OSS effort. 
This is very true. When we hire a new engineer then we know by 
experience that it will take that new hire at least two years to become 
an effective Grid Engine developer. If it comes to the guts of the 
scheduler or the qmaster then the time required before we entrust major 
changes to them is yet much longer.

Therein certainly lies one of the key reasons why code contributions 
from the community have been so small. In the ~9 years of the Sun 
sponsored open source project (and not taking into account the 
proprietary CODINE/GRD history before) only 3% of the check-ins came 
from the community and that amounted to only 1% of lines of code being 
contributed. The rest came from Sun developers, i.e. my team.

Those 1% are roughly 30,000 lines (added / deleted / changed as reported 
per our open git repo) and within them was one larger feature which some 
of you may be using. It was the array job interdependencies contributed 
by Rising Sun Pictures (RSP). It was a functionality they were requiring 
for their visual effects workflows. We had no time to implement it any 
time soon so we gave them guidance and they did the implementation.

Another larger contribution (almost half of the lines of code 
contributed by the community) came from a former collaboration partner 
of ours back from the GRD times. One main piece he contributed was an 
interface layer in the scheduler which he introduced so he could plug-in 
proprietary code of the company he was working for. I don't know whether 
this interface layer ever got used for anything else (we certainly 
didn't use it and also don't expose it).

A much bigger contribution to open source Grid Engine came later and it 
came from Univa: When we switched from Oracle to Univa we took what was 
left behind by Oracle (an interim and unstable state between 6.2u5 and 
the later Oracle-proprietary 6.2u6) to create our 8.0.0 which we open 
sourced (https://github.com/gridengine/gridengine). This was an effort 
to solidify the code base, to fix the most nagging issues we knew about 
from our work on the Oracle-proprietary 6.2u6 and 6.2u7 releases, to 
make a few focused enhancements like rewriting interactive job support 
which was pretty broken in 6.2u5 and to drop old code "baggage" which we 
knew we wouldn't want to maintain going forward. This resulted in 
 >600,000 lines of code modifications (again: added / deleted / changed 
as per our git repo).

If you are using Son of Grid Engine then this is the basis of what you 
are using and on top come the very respectable efforts of Dave Love who 
is shepherding that version.

How much dedication as well as effort and experience (and thus 
investment) is required to fundamentally evolve the Grid Engine 
technology is probably best exemplified by the very personal account of 
our long-time team member Ernst Bablick about his project introducing 
Job Classes into Univa Grid Engine 8.1 
(http://ernst.bablick.de/blog_files/jc_introduction.html). Not far from 
30,000 lines of code modifications were required for this alone plus all 
the know-how which Ernst has.

This is what I was referring to in my original post in this thread:
> The Grid Engine engineering team always has been a tightly knit group, 
> even prior to the days of joining Sun in 2000 and then throughout all 
> the years at Sun, the one year at Oracle and now since Jan 2011 at 
> Univa. Our dedication and passion is to evolve the Grid Engine 
> technology and help Grid Engine users to apply Grid Engine 
> successfully in their various use cases. 
Thanks,

Fritz


> Joseph Farran <mailto:jfarran at uci.edu>
> 24. Oktober 2013 20:06
> Yes and I did not mean to skip and forget all of the other folks who 
> contributed
> to what we know today as Grid Engine.
>
> If you dig far back enough and before it was CODINE, I am sure it started
> with someone writing some home grown code.
>
> The main point remains however.  Adaptive computing is an excellent 
> example
> of what may likely happen and I hope I am proven wrong.
>
> Also and this is a subjective issue depending on how deep your company 
> pockets
> are, the price for the commercial Grid Engine version is amazingly 
> $high$,
> specially for educational sites where every penny counts.
>
> Joseph
>
>
>
>
>
> _______________________________________________
> users mailing list
> users at gridengine.org
> https://gridengine.org/mailman/listinfo/users
> ChrisDag <mailto:dag at sonsorol.org>
> 24. Oktober 2013 00:38
> Just my $.02 ...
>
> Accidental or intentional that statement trivializes significant efforts
> of lots of people, many of whom were with Grid Engine pre-Sun
> Microsystems back when it was called CODINE. The open source community
> improved and enhanced the product, these people built the damn thing.
>
> Missing from the above characterization is "...hiring almost the entire
> grid engine development team (and now all of the support engineers) and
> continuing to ensure that there is a stable of active people being paid
> to work on it full time ..." Have you looked deep into the codebase? The
> learning curve is pretty extreme which can be a major obstacle to long
> term success of an OSS effort.
>
> The open source forks are doing well - I was worried that they'd be
> 'bugfix only' but there is real enhancement work happening. The major
> risk in my mind is that I suspect the number of serious active
> committers is very small.
>
> I see/deploy numerous Grid Engine systems every year for various people
> and entities. I'd say that maybe 80% of them use OGS or SoGE but the
> remaining 20% use the commercial flavor and are quite happy. Univa's
> roadshows and roadmaps have been impressive enough that I suspect they
> will continue to do well with GE.
>
> I'm personally very glad that both options exist and hope this situation
> continues.
>
> Sorry for being long winded. My TL/DR summary:
>
> - I'm glad both options exist
> - I'm glad all of the various forks are doing well
> - I'm not willing to assume Evil on behalf of Univa. I respect both the
> management and the engineers and the worst they've ever done to me was
> annoy me a few years back when some of their marketing and PR got a
> little over aggressive
>
> -dag
>
>
>
> _______________________________________________
> users mailing list
> users at gridengine.org
> https://gridengine.org/mailman/listinfo/users
> Joseph Farran <mailto:jfarran at uci.edu>
> 23. Oktober 2013 23:57
> Are you kidding me?  NO?
>
> Have you seen what Adaptive Computing did with Moab?    They took Maui 
> added/improved it
> and are now charging a fortune for Moab.
>
> If a company wants to start from scratch with a product fine, but to 
> take a product contributed
> by the community for free and then repackage it with bug fixes and 
> added features, that's not
> good.
>
> We were using Maui and the $price$ for Moab was ridiculously expensive 
> charging by
> node sockets and thus why we ended up with Son of Grid Engine. The 
> price for
> Univa Grid Engine was equally expensive when we initially inquired.
>
> I don't see anything good coming out of this in the long run for us...
>
> Joseph
>
>
>
> _______________________________________________
> users mailing list
> users at gridengine.org
> https://gridengine.org/mailman/listinfo/users
> ChrisDag <mailto:dag at sonsorol.org>
> 22. Oktober 2013 15:20
>
>
> No. You should not make that assumption.
>
>
>
>
>
> _______________________________________________
> users mailing list
> users at gridengine.org
> https://gridengine.org/mailman/listinfo/users
> John Kloss <mailto:john.kloss at gmail.com>
> 22. Oktober 2013 14:39
> Should we assume that, since Univa claims ownership of all copyright 
> and trademarks, _including_ the code under the SISSL, that Univa will 
> be fighting to shutdown the open source versions of Grid Engine (open 
> gridscheduler and Son of Grid Engine)?
>
>   John Kloss II.
>
>
>
> _______________________________________________
> users mailing list
> users at gridengine.org
> https://gridengine.org/mailman/listinfo/users

-- 

UnivaFritz Ferstl | CTO and Business Development, EMEA
Univa Corporation <http://www.univa.com/> | The Data Center Optimization 
Company
E-Mail: fferstl at univa.com | Phone: +49.9471.200.195 | Mobile: 
+49.170.819.7390

Where Grid Engine lives

*Visit us at SC13 at booth #4101*!

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gridengine.org/pipermail/users/attachments/20131025/c7fa19dc/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: compose-unknown-contact.jpg
Type: image/jpeg
Size: 770 bytes
Desc: not available
URL: <http://gridengine.org/pipermail/users/attachments/20131025/c7fa19dc/attachment.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: postbox-contact.jpg
Type: image/jpeg
Size: 1286 bytes
Desc: not available
URL: <http://gridengine.org/pipermail/users/attachments/20131025/c7fa19dc/attachment-0001.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Grafik1
Type: image/png
Size: 4331 bytes
Desc: not available
URL: <http://gridengine.org/pipermail/users/attachments/20131025/c7fa19dc/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Where Grid Engine lives
Type: image/png
Size: 5779 bytes
Desc: not available
URL: <http://gridengine.org/pipermail/users/attachments/20131025/c7fa19dc/attachment-0001.png>


More information about the users mailing list