Defining Compatibility

[Steve Loughran <st...@apache.org>]

what does it mean to be compatible with Hadoop? And how do products that 
consider themselves compatible with Hadoop say it?

We have plugin schedulers and the like, and all is well, and the Apache 
brand people keep an eye on distributions of the Hadoop code and make 
sure that Apache Hadoop is cleanly distinguished from redistributions of 
binaries by third parties.

But then you get distributions, and you have to define what is meant in 
terms of functionality and compatibility

Presumably, everyone who issues their own release has either explicitly 
or implicitly done a lot more testing than is in the unit test suite, 
testing that exists to stress test the code on large clusters -is there 
stuff there that needs to be added to SVN to help say a build is of 
sufficiently quality to be released?

Then there are the questions about

-things that work with specific versions/releases of Hadoop?
-replacement filesystems ?
-replacement of core parts of the system, like the MapReduce Engine?

IBM have have been talking about "Hadoop on GPFS"
http://www.almaden.ibm.com/storagesystems/projects/hadoop/

If this is running the MR layer, should it say "Apache Hadoop MR engine 
on top of IBM GPFS", or what -and how do you define or assess 
compatibility at this point? Is it up to the vendor to say "works with 
Apache Hadoop", and is running the Terasort client code sufficient to 
say "compatible"?

Similarly, if the MapReduce engine gets swapped out, what then? We in HP 
Labs have been funding some exploratory work at universities in Berlin 
on an engine that does more operations than just map and reduce, but it 
will also handle the existing operations with API compatibility on the 
worker nodes. The goal here is research with an OSS deliverable, but 
while it may support Hadoop jobs, it's not Hadoop.

What to call such things?


-Steve

Re: Defining Compatibility

[Tom White <to...@cloudera.com>]

FWIW the FileSystemContractBaseTest class and the FileContext*BaseTest
classes (and their concrete subclasses) are probably the closest thing
we have to compatibility tests for FileSystem and FileContext
implementations in Hadoop.

Tom

On Mon, Jan 31, 2011 at 7:59 AM, Steve Loughran <st...@apache.org> wrote:
> On 31/01/11 14:32, Chris Douglas wrote:
>>
>> Steve-
>>
>> It's hard to answer without more concrete criteria. Is this a
>> trademark question affecting the marketing of a product? A
>> cross-compatibility taxonomy for users? The minimum criteria to
>> publish a paper/release a product without eye-rolling? The particular
>> compatibility claims made by a system will be nuanced and specific; a
>> runtime that executes MapReduce jobs as they would run in Hadoop can
>> simply make that claim, whether it uses parts of MapReduce, HDFS, or
>> neither.
>
> No, I'm thinking more about what large scale tests are needed to be run
> against the codebase before you can say "it works", and then how to say some
> changes means that it still works.
>
>>
>> For the various distributions "Powered by Apache Hadoop," one would
>> assume that compatibility will vary depending on the featureset and
>> the audience. A distribution that runs MapReduce applications
>> as-written for Apache Hadoop may be incompatible with a user's
>> deployed metrics/monitoring system. Some random script to scrape the
>> UI may not work. The product may only scale to 20 nodes. Whether these
>> are "compatible with Apache Hadoop" is awkward to answer generally,
>> unless we want to define the semantics of that phrase by policy.
>>
>> To put it bluntly, why would we bother to define such a policy? One
>> could assert that a fully-compatible system would implement all the
>> public/stable APIs as defined in HADOOP-5073, but who would that help?
>> And though interoperability is certainly relevant to systems built on
>> top of Hadoop, is there a reason the Apache project needs to be
>> involved in defining the standards for compatibility among them?
>
> Agreed, I'm just thinking about namings and definitions. Even with the
> stable/unstable internal/external split, there's still the question as to
> what the semantics of operations are, both explicit (this operation does X)
> and implicit (and it takes less than Y seconds to do it). It's those
> implicit things that always catch you out (indeed, they are the argument
> points in things like Java and Java EE compatibility test kits)
>

Re: Defining Compatibility

[Steve Loughran <st...@apache.org>]

On 31/01/11 14:32, Chris Douglas wrote:
> Steve-
>
> It's hard to answer without more concrete criteria. Is this a
> trademark question affecting the marketing of a product? A
> cross-compatibility taxonomy for users? The minimum criteria to
> publish a paper/release a product without eye-rolling? The particular
> compatibility claims made by a system will be nuanced and specific; a
> runtime that executes MapReduce jobs as they would run in Hadoop can
> simply make that claim, whether it uses parts of MapReduce, HDFS, or
> neither.

No, I'm thinking more about what large scale tests are needed to be run 
against the codebase before you can say "it works", and then how to say 
some changes means that it still works.

>
> For the various distributions "Powered by Apache Hadoop," one would
> assume that compatibility will vary depending on the featureset and
> the audience. A distribution that runs MapReduce applications
> as-written for Apache Hadoop may be incompatible with a user's
> deployed metrics/monitoring system. Some random script to scrape the
> UI may not work. The product may only scale to 20 nodes. Whether these
> are "compatible with Apache Hadoop" is awkward to answer generally,
> unless we want to define the semantics of that phrase by policy.
>
> To put it bluntly, why would we bother to define such a policy? One
> could assert that a fully-compatible system would implement all the
> public/stable APIs as defined in HADOOP-5073, but who would that help?
> And though interoperability is certainly relevant to systems built on
> top of Hadoop, is there a reason the Apache project needs to be
> involved in defining the standards for compatibility among them?

Agreed, I'm just thinking about namings and definitions. Even with the 
stable/unstable internal/external split, there's still the question as 
to what the semantics of operations are, both explicit (this operation 
does X) and implicit (and it takes less than Y seconds to do it). It's 
those implicit things that always catch you out (indeed, they are the 
argument points in things like Java and Java EE compatibility test kits)

Re: Defining Compatibility

[Chris Douglas <cd...@apache.org>]

Steve-

It's hard to answer without more concrete criteria. Is this a
trademark question affecting the marketing of a product? A
cross-compatibility taxonomy for users? The minimum criteria to
publish a paper/release a product without eye-rolling? The particular
compatibility claims made by a system will be nuanced and specific; a
runtime that executes MapReduce jobs as they would run in Hadoop can
simply make that claim, whether it uses parts of MapReduce, HDFS, or
neither.

For the various distributions "Powered by Apache Hadoop," one would
assume that compatibility will vary depending on the featureset and
the audience. A distribution that runs MapReduce applications
as-written for Apache Hadoop may be incompatible with a user's
deployed metrics/monitoring system. Some random script to scrape the
UI may not work. The product may only scale to 20 nodes. Whether these
are "compatible with Apache Hadoop" is awkward to answer generally,
unless we want to define the semantics of that phrase by policy.

To put it bluntly, why would we bother to define such a policy? One
could assert that a fully-compatible system would implement all the
public/stable APIs as defined in HADOOP-5073, but who would that help?
And though interoperability is certainly relevant to systems built on
top of Hadoop, is there a reason the Apache project needs to be
involved in defining the standards for compatibility among them?

Compatibility matters, but I'm not clear on the objective of this discussion. -C

On Mon, Jan 31, 2011 at 5:18 AM, Steve Loughran <st...@apache.org> wrote:
> what does it mean to be compatible with Hadoop? And how do products that
> consider themselves compatible with Hadoop say it?
>
> We have plugin schedulers and the like, and all is well, and the Apache
> brand people keep an eye on distributions of the Hadoop code and make sure
> that Apache Hadoop is cleanly distinguished from redistributions of binaries
> by third parties.
>
> But then you get distributions, and you have to define what is meant in
> terms of functionality and compatibility
>
> Presumably, everyone who issues their own release has either explicitly or
> implicitly done a lot more testing than is in the unit test suite, testing
> that exists to stress test the code on large clusters -is there stuff there
> that needs to be added to SVN to help say a build is of sufficiently quality
> to be released?
>
> Then there are the questions about
>
> -things that work with specific versions/releases of Hadoop?
> -replacement filesystems ?
> -replacement of core parts of the system, like the MapReduce Engine?
>
> IBM have have been talking about "Hadoop on GPFS"
> http://www.almaden.ibm.com/storagesystems/projects/hadoop/
>
> If this is running the MR layer, should it say "Apache Hadoop MR engine on
> top of IBM GPFS", or what -and how do you define or assess compatibility at
> this point? Is it up to the vendor to say "works with Apache Hadoop", and is
> running the Terasort client code sufficient to say "compatible"?
>
> Similarly, if the MapReduce engine gets swapped out, what then? We in HP
> Labs have been funding some exploratory work at universities in Berlin on an
> engine that does more operations than just map and reduce, but it will also
> handle the existing operations with API compatibility on the worker nodes.
> The goal here is research with an OSS deliverable, but while it may support
> Hadoop jobs, it's not Hadoop.
>
> What to call such things?
>
>
> -Steve
>
>

Re: Defining Compatibility

[Ian Holsman <ha...@holsman.net>]

On Jan 31, 2011, at 8:18 AM, Steve Loughran wrote:

> what does it mean to be compatible with Hadoop? And how do products that consider themselves compatible with Hadoop say it?

I would like to define it in terms of API's and core functionality.

A product (say hive or pig) will run against a set of well defined APIs for a given version.
regardless of who implements the API, it should perform as promised, so switching between distributions or implementations (say HDFS over GFS) should not give the end user any surprises.

Saying that.. HDFS over GFS may expose a superset of APIs that a tool may utilize. 
If the tool requires those APIs it is no longer compatible with Apache Hadoop, and should not be called such.

For example, early versions of HUE required the Thrift API to be present. So it would clearly not be compatible version Apache Hadoop 0.20.

What still perplexes me is what to do when some core functionality (say the append patch) that has a identical API to the end-user is promoted as compatible... 
I classify this change as a 'end user surprise', *BUT* HDFS over GFS would also have similar surprises, where the API is the same, but implemented very differently.

So I'm still not sure if you would classify it as being 0.20 compatible.

> 
> We have plugin schedulers and the like, and all is well, and the Apache brand people keep an eye on distributions of the Hadoop code and make sure that Apache Hadoop is cleanly distinguished from redistributions of binaries by third parties.
> 
> But then you get distributions, and you have to define what is meant in terms of functionality and compatibility
> 
> Presumably, everyone who issues their own release has either explicitly or implicitly done a lot more testing than is in the unit test suite, testing that exists to stress test the code on large clusters -is there stuff there that needs to be added to SVN to help say a build is of sufficiently quality to be released?
> 
> Then there are the questions about
> 
> -things that work with specific versions/releases of Hadoop?
> -replacement filesystems ?
> -replacement of core parts of the system, like the MapReduce Engine?
> 
> IBM have have been talking about "Hadoop on GPFS"
> http://www.almaden.ibm.com/storagesystems/projects/hadoop/
> 
> If this is running the MR layer, should it say "Apache Hadoop MR engine on top of IBM GPFS", or what -and how do you define or assess compatibility at this point? Is it up to the vendor to say "works with Apache Hadoop", and is running the Terasort client code sufficient to say "compatible"?
> 
> Similarly, if the MapReduce engine gets swapped out, what then? We in HP Labs have been funding some exploratory work at universities in Berlin on an engine that does more operations than just map and reduce, but it will also handle the existing operations with API compatibility on the worker nodes. The goal here is research with an OSS deliverable, but while it may support Hadoop jobs, it's not Hadoop.
> 
> What to call such things?
> 
> 
> -Steve
>