Adding a new platform support to Hadoop

[Amir Sanjar <v1...@us.ibm.com>]

Is there any specific development, build, and packaging guidelines to add
support for a new hardware platform, in this case PPC64, to hadoop?

Best Regards
Amir Sanjar

Linux System Management Architect and Lead
IBM Senior Software Engineer
Phone# 512-286-8393
Fax#      512-838-8858

Re: Adding a new platform support to Hadoop

[Brian Bockelman <bb...@cse.unl.edu>]

Hi Amir,

The best person to engage is Allen Wittenauer, who has "fought the good fight" on this previously.  It's a bit of an uphill battle, as the primary development platform is definitely Linux / x86-64.  I saw him close a bunch of portability-related tickets a few weeks ago, which made me a little sad.

You didn't say whether you want to only switch the processor, or the processor and the platform (Linux).  I'll talk like you want to do both.

There's three layers of support to think of:
1) All the Java code and surrounding wrappers.  Likely the Java is very portable; the surrounding wrappers have an occasional Linux-ism.  Easy-to-Medium difficulty - more about finding all the little quirks and smoothing them.
2) The native library.  These are very skewed toward linux-specific optimizations, but I think are really required to get working to get Really Great Performance.  Medium-to-hard difficulty - you need a broad POSIX and Linux understanding to convert things.
3) The optimizations in the Java code.  Some performance tunings are made based on Oracle Java on Linux on x86-64.  You need a deep expert in Hadoop, system architectures, and Java to understand what was changed, why it worked better on the target platform, and how the optimization will affect your target platform.  Extremely hard difficulty.

Brian

On Nov 17, 2011, at 9:02 AM, Amir Sanjar wrote:

> Is there any specific development, build, and packaging guidelines to add
> support for a new hardware platform, in this case PPC64, to hadoop?
> 
> Best Regards
> Amir Sanjar
> 
> Linux System Management Architect and Lead
> IBM Senior Software Engineer
> Phone# 512-286-8393
> Fax#      512-838-8858

Re: Adding a new platform support to Hadoop

[Steve Loughran <st...@apache.org>]

On 17/11/11 15:02, Amir Sanjar wrote:
> Is there any specific development, build, and packaging guidelines to add
> support for a new hardware platform, in this case PPC64, to hadoop?
>
> Best Regards
> Amir Sanjar
>
> Linux System Management Architect and Lead
> IBM Senior Software Engineer
> Phone# 512-286-8393
> Fax#      512-838-8858
>

this is something to take up on the -dev lists, not the user lists, 
especially  common-dev@hadoop.apache.org

One problem with any platform is the native code: nobody but you is 
going to build or test it. The only JVM currently recommended is the Sun 
JVM, so again, you will get to test there. This means you are going to 
have to be active testing releases against your target platform. 
Otherwise it will languish in the "not really meant to be used in 
production" category of things.

The apache releases (which are meant to be source distributions anyway; 
the binary artifacts are just an extra), but you will need to work with 
the dev team to make sure the native libraries build properly