You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Sanjay Radia <sr...@yahoo-inc.com> on 2008/10/21 03:50:00 UTC
Hadoop 1.0 Compatibility Discussion.
The Hadoop 1.0 wiki has a section on compatibility.
http://wiki.apache.org/hadoop/Release1.0Requirements
Since the wiki is awkward for discussions, I am continuing the
discussion here.
I or someone will update the wiki when agreements are reached.
Here is the current list of compatibility requirements on the Hadoop
1.0 Wiki for the convenience of this email thread.
--------
What does Hadoop 1.0 mean?
* Standard release numbering: Only bug fixes in 1.x.y releases
and new features in 1.x.0 releases.
* No need for client recompilation when upgrading from 1.x to
1.y, where x <= y
o Can't remove deprecated classes or methods until 2.0
* Old 1.x clients can connect to new 1.y servers, where x <= y
* New FileSystem clients must be able to call old methods when
talking to old servers. This generally will be done by having old
methods continue to use old rpc methods. However, it is legal to have
new implementations of old methods call new rpcs methods, as long as
the library transparently handles the fallback case for old servers.
-----------------
A couple of additional compatibility requirements:
* HDFS metadata and data is preserved across release changes, both
major and minor. That is,
whenever a release is upgraded, the HDFS metadata from the old release
will be converted automatically
as needed.
The above has been followed so far in Hadoop; I am just documenting it
in the 1.0 requirements list.
* In a major release transition [ ie from a release x.y to a
release (x+1).0], a user should be able to read data from the cluster
running the old version. (OR shall we generalize this to: from x.y to
(x+i).z ?)
The motivation: data copying across clusters is a common operation for
many customers
(for example this is routinely at done at Yahoo.). Today, http (or
hftp) provides a guaranteed compatible way of copying data across
versions. Clearly one cannot force a customer to simultaneously
update all its hadoop clusters on to
a new major release. The above documents this requirement; we can
satisfy it via the http/hftp mechanism or some other mechanism.
Question: is one is willing to break applications that operate across
clusters (ie an application that accesses data across clusters that
cross a major release boundary? I asked the operations team at Yahoo
that run our hadoop clusters. We currently do not have any applicaions
that access data across clusters as part of a MR job. The reason
being that Hadoop routinely breaks wire compatibility across releases
and so such apps would be very unreliable. However, the copying of
data across clusters is t is crucial and needs to be supported.
Shall we add a stronger requirement for 1.0: wire compatibility
across major versions? This can be supported by class loading or other
games. Note we can wait to provide this when 2.0 happens. If Hadoop
provided this guarantee then it would allow customers to partition
their data across clusters without risking apps breaking across major
releases due to wire incompatibility issues.
Re: Hadoop 1.0 Compatibility Discussion.
Posted by Sanjay Radia <sr...@yahoo-inc.com>.
On Oct 21, 2008, at 5:23 PM, Konstantin Shvachko wrote:
> Sanjay Radia wrote:
> >> o Can't remove deprecated classes or methods until 2.0
>
> Dhruba Borthakur wrote:
> > 1. APIs that are deprecated in x.y release can be removed in (x+1).
> 0 release.
>
> Current rule is that apis deprecated in M.x.y can be remove in M.(x
> +2).0
> I don't think we want neither relax nor stiffen this requirement.
>
I think we want to strengthen this to : removal of deprecated methods/
classes only on major releases
Isn't this what major and minor releases mean?
I believe that is what customers will expect from a 1.0 release -
stability till 2.0.
Are you worried that maintaining old methods is too much burden
because there will be too many of them?
sanjay
>
>
> > 2. Old 1.x clients can connect to new 1.y servers, where x <= y but
> > the old clients might get reduced functionality or performance. 1.x
> > clients might not be able to connect to 2.z servers.
> >
> > 3. HDFS disk format can change from 1.x to 1.y release and is
> > transparent to user-application. A cluster when rolling back to 1.x
> > from 1,y will revert back to the old disk format.
> >
> >> * In a major release transition [ ie from a release x.y to a
> release
> >> (x+1).0], a user should be able to read data from the cluster
> running the
> >> old version.
> >
> > I think this is a good requirement to have. This will be very useful
> > when we run multiple clusters, especially across data centers
> > (HADOOP-4058 is a use-case).
>
> I don't see anything about compatibility model going from 1.*.* to
> 2.0.0.
> Does that mean we do not provide compatibility between those?
> Does that mean compatibility between 1.*.* and 2.*.* is provided by
> distcp?
> Or another way to ask the same question: will HDFS-1 and HDFS-2 be
> as different as ext2 and ext3?
> I am not saying this is bad just want it to be clarified.
>
> May be we should somehow structure this discussion into sections,
> e.g.:
> - deprecation rules;
> - client/server communication compatibility;
> - inter version data format compatibility;
> = meta-data compatibility
> = block data compatibility
>
> --Konstantin
>
> >> --------
> >> What does Hadoop 1.0 mean?
> >> * Standard release numbering: Only bug fixes in 1.x.y releases
> and new
> >> features in 1.x.0 releases.
> >> * No need for client recompilation when upgrading from 1.x to
> 1.y, where
> >> x <= y
> >> o Can't remove deprecated classes or methods until 2.0
> >> * Old 1.x clients can connect to new 1.y servers, where x <= y
> >> * New FileSystem clients must be able to call old methods when
> talking to
> >> old servers. This generally will be done by having old methods
> continue to
> >> use old rpc methods. However, it is legal to have new
> implementations of old
> >> methods call new rpcs methods, as long as the library
> transparently handles
> >> the fallback case for old servers.
> >> -----------------
> >>
> >> A couple of additional compatibility requirements:
> >>
> >> * HDFS metadata and data is preserved across release changes,
> both major and
> >> minor. That is,
> >> whenever a release is upgraded, the HDFS metadata from the old
> release will
> >> be converted automatically
> >> as needed.
> >>
> >> The above has been followed so far in Hadoop; I am just
> documenting it in
> >> the 1.0 requirements list.
> >>
> >> * In a major release transition [ ie from a release x.y to a
> release
> >> (x+1).0], a user should be able to read data from the cluster
> running the
> >> old version. (OR shall we generalize this to: from x.y to (x
> +i).z ?)
> >>
> >> The motivation: data copying across clusters is a common
> operation for many
> >> customers
> >> (for example this is routinely at done at Yahoo.). Today, http
> (or hftp)
> >> provides a guaranteed compatible way of copying data across
> versions.
> >> Clearly one cannot force a customer to simultaneously update all
> its hadoop
> >> clusters on to
> >> a new major release. The above documents this requirement; we can
> satisfy it
> >> via the http/hftp mechanism or some other mechanism.
> >>
> >> Question: is one is willing to break applications that operate
> across
> >> clusters (ie an application that accesses data across clusters
> that cross a
> >> major release boundary? I asked the operations team at Yahoo that
> run our
> >> hadoop clusters. We currently do not have any applicaions that
> access data
> >> across clusters as part of a MR job. The reason being that
> Hadoop routinely
> >> breaks wire compatibility across releases and so such apps would
> be very
> >> unreliable. However, the copying of data across clusters is t is
> crucial and
> >> needs to be supported.
> >>
> >> Shall we add a stronger requirement for 1.0: wire compatibility
> across
> >> major versions? This can be supported by class loading or other
> games. Note
> >> we can wait to provide this when 2.0 happens. If Hadoop provided
> this
> >> guarantee then it would allow customers to partition their data
> across
> >> clusters without risking apps breaking across major releases due
> to wire
> >> incompatibility issues.
> >>
> >>
> >>
> >
>
Re: Hadoop 1.0 Compatibility Discussion.
Posted by Konstantin Shvachko <sh...@yahoo-inc.com>.
Sanjay Radia wrote:
>> o Can't remove deprecated classes or methods until 2.0
Dhruba Borthakur wrote:
> 1. APIs that are deprecated in x.y release can be removed in (x+1).0 release.
Current rule is that apis deprecated in M.x.y can be remove in M.(x+2).0
I don't think we want neither relax nor stiffen this requirement.
> 2. Old 1.x clients can connect to new 1.y servers, where x <= y but
> the old clients might get reduced functionality or performance. 1.x
> clients might not be able to connect to 2.z servers.
>
> 3. HDFS disk format can change from 1.x to 1.y release and is
> transparent to user-application. A cluster when rolling back to 1.x
> from 1,y will revert back to the old disk format.
>
>> * In a major release transition [ ie from a release x.y to a release
>> (x+1).0], a user should be able to read data from the cluster running the
>> old version.
>
> I think this is a good requirement to have. This will be very useful
> when we run multiple clusters, especially across data centers
> (HADOOP-4058 is a use-case).
I don't see anything about compatibility model going from 1.*.* to 2.0.0.
Does that mean we do not provide compatibility between those?
Does that mean compatibility between 1.*.* and 2.*.* is provided by distcp?
Or another way to ask the same question: will HDFS-1 and HDFS-2 be
as different as ext2 and ext3?
I am not saying this is bad just want it to be clarified.
May be we should somehow structure this discussion into sections, e.g.:
- deprecation rules;
- client/server communication compatibility;
- inter version data format compatibility;
= meta-data compatibility
= block data compatibility
--Konstantin
>> --------
>> What does Hadoop 1.0 mean?
>> * Standard release numbering: Only bug fixes in 1.x.y releases and new
>> features in 1.x.0 releases.
>> * No need for client recompilation when upgrading from 1.x to 1.y, where
>> x <= y
>> o Can't remove deprecated classes or methods until 2.0
>> * Old 1.x clients can connect to new 1.y servers, where x <= y
>> * New FileSystem clients must be able to call old methods when talking to
>> old servers. This generally will be done by having old methods continue to
>> use old rpc methods. However, it is legal to have new implementations of old
>> methods call new rpcs methods, as long as the library transparently handles
>> the fallback case for old servers.
>> -----------------
>>
>> A couple of additional compatibility requirements:
>>
>> * HDFS metadata and data is preserved across release changes, both major and
>> minor. That is,
>> whenever a release is upgraded, the HDFS metadata from the old release will
>> be converted automatically
>> as needed.
>>
>> The above has been followed so far in Hadoop; I am just documenting it in
>> the 1.0 requirements list.
>>
>> * In a major release transition [ ie from a release x.y to a release
>> (x+1).0], a user should be able to read data from the cluster running the
>> old version. (OR shall we generalize this to: from x.y to (x+i).z ?)
>>
>> The motivation: data copying across clusters is a common operation for many
>> customers
>> (for example this is routinely at done at Yahoo.). Today, http (or hftp)
>> provides a guaranteed compatible way of copying data across versions.
>> Clearly one cannot force a customer to simultaneously update all its hadoop
>> clusters on to
>> a new major release. The above documents this requirement; we can satisfy it
>> via the http/hftp mechanism or some other mechanism.
>>
>> Question: is one is willing to break applications that operate across
>> clusters (ie an application that accesses data across clusters that cross a
>> major release boundary? I asked the operations team at Yahoo that run our
>> hadoop clusters. We currently do not have any applicaions that access data
>> across clusters as part of a MR job. The reason being that Hadoop routinely
>> breaks wire compatibility across releases and so such apps would be very
>> unreliable. However, the copying of data across clusters is t is crucial and
>> needs to be supported.
>>
>> Shall we add a stronger requirement for 1.0: wire compatibility across
>> major versions? This can be supported by class loading or other games. Note
>> we can wait to provide this when 2.0 happens. If Hadoop provided this
>> guarantee then it would allow customers to partition their data across
>> clusters without risking apps breaking across major releases due to wire
>> incompatibility issues.
>>
>>
>>
>
Re: Hadoop 1.0 Compatibility Discussion.
Posted by Sanjay Radia <sr...@yahoo-inc.com>.
On Oct 28, 2008, at 2:15 PM, Robert Chansler wrote:
> Editorial: "rollback" should be restricted to exactly the function of
> returning to the state before an upgrade was started. It's OK to
> discuss
> other desirable features along with descriptive names, but "easy
> rollback,"
> "partial rollback," "version rollback" and the like are all confusing.
>
> Substance: In speaking of version numbers, don't confuse desired
> behavior
> (what client can connect with which server) with the details of
> implementation (disk formats did or did not change). We want to avoid
> getting squeezed by the argument that some feature must wait a year
> because
> it modifies the disk format but it too early to change the major
> version
> number.
The protocol version is separate from the disk version.
2.a stronger will have the effect that one may be forced to wait for a
major release.
An alternate way to do this is to stick to 2.a regular and to simply
say that the rollback is a noop for the cases where the change was
"minor" (e.g. only fields got added.). Along the same thoughts: if our
software deals with new fields then we simply won't need to bump the
disk format version number when fields are added (or only bump the
minor version #) and so one will be able to bring up an older hadoop
system on a newer hadoop disk format. BTW there is a subtle issue when
you bring up an older hadoop on a newer disk format where only fields
were added-- the old version of the hadoop is not likely to preserve
the new fields when the on-disk data is updated since the old software
merely ignores the new fields.
sanjay
>
>
>
> On 27 10 08 13:50, "Sanjay Radia" <sr...@yahoo-inc.com> wrote:
>
> > I have merged the various Hadoop 1.0 Compatibility items that have
> > been discussed in this thread and categorized and listed them below.
> >
> >
> >
> > Hadoop 1.0 Compatibility
> > ==================
> >
> > Standard release numbering:
> > - Only bug fixes in dot releases: m.x.y
> > - no changes to API, disk format, protocols or config etc.
> > - new features in major (m.x.0) and minor (m.x.0) releases
> >
> >
> > 1. API Compatibility
> > -------------------------
> > No need for client recompilation when upgrading across minor
> releases
> > (ie. from m.x to m.y, where x <= y)
> > Classes or methods deprecated in m.x can be removed in (m+1).0
> > Note that this is stronger than what we have been doing in Hadoop
> 0.x
> > releases.
> > Motivation. This the industry standard compatibility rules for major
> > and
> > minor releases.
> >
> > 2 Data Compatibility
> > --------------------------
> > 2.a HDFS metadata and data can change across minor or major
> releases ,
> > but such
> > changes are transparent to user application. That is release upgrade
> > must
> > automatically convert the metadata and data as needed. Further, a
> > release
> > upgrade must allow a cluster to roll back to the older version and
> its
> > older
> > disk format.
> > Motivation: Users expect File systems preserve data transparently
> > across
> > releases.
> >
> > 2.a-Stronger
> > HDFS metadata and data can change across minor or major releases,
> but
> > such
> > changes are transparent to user application. That is release upgrade
> > must
> > automatically convert the metadata and data as needed. During
> *minor*
> > releases,
> > disk format changes have to backward and forward compatible; i.e. an
> > older
> > version of Hadoop can be started on a newer version of the disk
> > format. Hence
> > a version roll back is simple, just restart the older version of
> Hadoop.
> > Major releases allow more significant changes to the disk format and
> > have be
> > only backward compatible; however major release upgrade must allow a
> > cluster to
> > roll back to the older version and its older disk format.
> > Motivation: Minor release are very easy to roll back for an admin.
> >
> >
> > 2.a-WeakerAutomaticConversion:
> > Automatic conversion is supported across a small number of releases.
> > If a user
> > wants to jump across multiple releases he may be forced to go
> through
> > a few
> > intermediate release to get to the final desired release.
> >
> > 3. Wire Protocol Compatibility
> > ----------------------------------------
> > We offer no wire compatibility in our 0.x release today.
> > The motivation *isn't* to make a our protocols public. Applications
> > will not
> > call the protocol directly but through a library (in our case
> > FileSystem class
> > and its implementations). Instead the motivation is that customers
> run
> > multiple clusters and have apps that access data across clusters.
> > Customers
> > cannot be expected to update all clusters simultaneously.
> >
> >
> > 3.a Old m.x clients can connect to new m.y servers, where x <= y but
> > the old clients might get reduced functionality or performance. m.x
> > clients might not be able to connect to (m+1).z servers
> >
> > 3.b. New m.y clients must be able to connect to old m.x server,
> where
> > x< y but
> > only for old m.x functionality.
> > Comment: Generally old API methods continue to use old rpc methods.
> > However, it is legal to have new implementations of old API methods
> > call new
> > rpcs methods, as long as the library transparently handles the
> fallback
> > case for old servers.
> >
> > 3.c. At any major release transition [ ie from a release m.x to a
> > release (m+1).0], a user should be able to read data from the
> cluster
> > running the old version. (OR shall we generalize this to: from m.x
> to
> > (m+i).z ?)
> >
> > Motivation: data copying across clusters is a common operation for
> many
> > customers. For example this is routinely at done at Yahoo; another
> use
> > case is
> > HADOOP-4058. Today, http (or hftp) provides a guaranteed compatible
> > way of
> > copying data across versions. Clearly one cannot force a customer to
> > simultaneously update all its Hadoop clusters on to a new major
> > release. The
> > above documents this requirement; we can satisfy it via the http/
> hftp
> > mechanism
> > or some other mechanism.
> >
> > 3.c-Stronger
> > Shall we add a stronger requirement for 1. 0 : wire compatibility
> > across major versions? That is not just for reading but for all
> > operations. This can be supported by class loading or other games.
> > Note we can wait to provide this when 2. 0 happens. If Hadoop
> > provided this guarantee then it would allow customers to partition
> > their data across clusters without risking apps breaking across
> major
> > releases due to wire incompatibility issues.
> >
> > Motivation: Data copying is a compromise. Customers really want to
> run
> > apps across clusters running different versions. ( See item 2 )
> >
> >
> > 4. Intra Hadoop Service Compatibility
> > --------------------------------------------------
> > The HDFS Service has multiple components (NN, DN, Balancer) that
> > communicate
> > amongst themselves. Similarly the MapReduce service has
> > components (JR and TT) that communicate amongst themselves.
> > Currently we require that the all the components of a service have
> the
> > same build version and hence talk the same wire protocols.
> > This build-version checking prevents rolling upgrades. It has the
> > benefit that the admin can ensure that the entire cluster has
> exactly
> > the same build version.
> >
> > 4.a HDFS and MapReduce require that their respective sub-components
> > have the same build version in order to form a cluster.
> > [ie. Maintain the current mechanism.]
> >
> > 4.a-Stronger: Intra-service wire-protocol compatibility
> > [I am listing this here to document it, but I don't think we are
> ready
> > to take
> > this on for Hadoop 1.0. Alternatively, we could require intra -
> service
> > wire
> > compatibility but check for build version till we are ready for
> rolling
> > upgrades]
> >
> > Wire protocols between internal Hadoop components are compatible
> across
> > minor versions.
> > Examples are NN-DN, DN-DN and NN-Balancer, etc.
> > Old m.x components can talk to new m.y components (x<=y)
> > Wire compatibility can break across major versions.
> > Motivation: Allow rolling upgrades.
> >
>
Re: Hadoop 1.0 Compatibility Discussion.
Posted by Robert Chansler <ch...@yahoo-inc.com>.
Editorial: "rollback" should be restricted to exactly the function of
returning to the state before an upgrade was started. It's OK to discuss
other desirable features along with descriptive names, but "easy rollback,"
"partial rollback," "version rollback" and the like are all confusing.
Substance: In speaking of version numbers, don't confuse desired behavior
(what client can connect with which server) with the details of
implementation (disk formats did or did not change). We want to avoid
getting squeezed by the argument that some feature must wait a year because
it modifies the disk format but it too early to change the major version
number.
On 27 10 08 13:50, "Sanjay Radia" <sr...@yahoo-inc.com> wrote:
> I have merged the various Hadoop 1.0 Compatibility items that have
> been discussed in this thread and categorized and listed them below.
>
>
>
> Hadoop 1.0 Compatibility
> ==================
>
> Standard release numbering:
> - Only bug fixes in dot releases: m.x.y
> - no changes to API, disk format, protocols or config etc.
> - new features in major (m.x.0) and minor (m.x.0) releases
>
>
> 1. API Compatibility
> -------------------------
> No need for client recompilation when upgrading across minor releases
> (ie. from m.x to m.y, where x <= y)
> Classes or methods deprecated in m.x can be removed in (m+1).0
> Note that this is stronger than what we have been doing in Hadoop 0.x
> releases.
> Motivation. This the industry standard compatibility rules for major
> and
> minor releases.
>
> 2 Data Compatibility
> --------------------------
> 2.a HDFS metadata and data can change across minor or major releases ,
> but such
> changes are transparent to user application. That is release upgrade
> must
> automatically convert the metadata and data as needed. Further, a
> release
> upgrade must allow a cluster to roll back to the older version and its
> older
> disk format.
> Motivation: Users expect File systems preserve data transparently
> across
> releases.
>
> 2.a-Stronger
> HDFS metadata and data can change across minor or major releases, but
> such
> changes are transparent to user application. That is release upgrade
> must
> automatically convert the metadata and data as needed. During *minor*
> releases,
> disk format changes have to backward and forward compatible; i.e. an
> older
> version of Hadoop can be started on a newer version of the disk
> format. Hence
> a version roll back is simple, just restart the older version of Hadoop.
> Major releases allow more significant changes to the disk format and
> have be
> only backward compatible; however major release upgrade must allow a
> cluster to
> roll back to the older version and its older disk format.
> Motivation: Minor release are very easy to roll back for an admin.
>
>
> 2.a-WeakerAutomaticConversion:
> Automatic conversion is supported across a small number of releases.
> If a user
> wants to jump across multiple releases he may be forced to go through
> a few
> intermediate release to get to the final desired release.
>
> 3. Wire Protocol Compatibility
> ----------------------------------------
> We offer no wire compatibility in our 0.x release today.
> The motivation *isn't* to make a our protocols public. Applications
> will not
> call the protocol directly but through a library (in our case
> FileSystem class
> and its implementations). Instead the motivation is that customers run
> multiple clusters and have apps that access data across clusters.
> Customers
> cannot be expected to update all clusters simultaneously.
>
>
> 3.a Old m.x clients can connect to new m.y servers, where x <= y but
> the old clients might get reduced functionality or performance. m.x
> clients might not be able to connect to (m+1).z servers
>
> 3.b. New m.y clients must be able to connect to old m.x server, where
> x< y but
> only for old m.x functionality.
> Comment: Generally old API methods continue to use old rpc methods.
> However, it is legal to have new implementations of old API methods
> call new
> rpcs methods, as long as the library transparently handles the fallback
> case for old servers.
>
> 3.c. At any major release transition [ ie from a release m.x to a
> release (m+1).0], a user should be able to read data from the cluster
> running the old version. (OR shall we generalize this to: from m.x to
> (m+i).z ?)
>
> Motivation: data copying across clusters is a common operation for many
> customers. For example this is routinely at done at Yahoo; another use
> case is
> HADOOP-4058. Today, http (or hftp) provides a guaranteed compatible
> way of
> copying data across versions. Clearly one cannot force a customer to
> simultaneously update all its Hadoop clusters on to a new major
> release. The
> above documents this requirement; we can satisfy it via the http/hftp
> mechanism
> or some other mechanism.
>
> 3.c-Stronger
> Shall we add a stronger requirement for 1. 0 : wire compatibility
> across major versions? That is not just for reading but for all
> operations. This can be supported by class loading or other games.
> Note we can wait to provide this when 2. 0 happens. If Hadoop
> provided this guarantee then it would allow customers to partition
> their data across clusters without risking apps breaking across major
> releases due to wire incompatibility issues.
>
> Motivation: Data copying is a compromise. Customers really want to run
> apps across clusters running different versions. ( See item 2 )
>
>
> 4. Intra Hadoop Service Compatibility
> --------------------------------------------------
> The HDFS Service has multiple components (NN, DN, Balancer) that
> communicate
> amongst themselves. Similarly the MapReduce service has
> components (JR and TT) that communicate amongst themselves.
> Currently we require that the all the components of a service have the
> same build version and hence talk the same wire protocols.
> This build-version checking prevents rolling upgrades. It has the
> benefit that the admin can ensure that the entire cluster has exactly
> the same build version.
>
> 4.a HDFS and MapReduce require that their respective sub-components
> have the same build version in order to form a cluster.
> [ie. Maintain the current mechanism.]
>
> 4.a-Stronger: Intra-service wire-protocol compatibility
> [I am listing this here to document it, but I don't think we are ready
> to take
> this on for Hadoop 1.0. Alternatively, we could require intra -service
> wire
> compatibility but check for build version till we are ready for rolling
> upgrades]
>
> Wire protocols between internal Hadoop components are compatible across
> minor versions.
> Examples are NN-DN, DN-DN and NN-Balancer, etc.
> Old m.x components can talk to new m.y components (x<=y)
> Wire compatibility can break across major versions.
> Motivation: Allow rolling upgrades.
>
Re: Hadoop 1.0 Compatibility Discussion.
Posted by Sanjay Radia <sr...@yahoo-inc.com>.
I have merged the various Hadoop 1.0 Compatibility items that have
been discussed in this thread and categorized and listed them below.
Hadoop 1.0 Compatibility
==================
Standard release numbering:
- Only bug fixes in dot releases: m.x.y
- no changes to API, disk format, protocols or config etc.
- new features in major (m.x.0) and minor (m.x.0) releases
1. API Compatibility
-------------------------
No need for client recompilation when upgrading across minor releases
(ie. from m.x to m.y, where x <= y)
Classes or methods deprecated in m.x can be removed in (m+1).0
Note that this is stronger than what we have been doing in Hadoop 0.x
releases.
Motivation. This the industry standard compatibility rules for major
and
minor releases.
2 Data Compatibility
--------------------------
2.a HDFS metadata and data can change across minor or major releases ,
but such
changes are transparent to user application. That is release upgrade
must
automatically convert the metadata and data as needed. Further, a
release
upgrade must allow a cluster to roll back to the older version and its
older
disk format.
Motivation: Users expect File systems preserve data transparently
across
releases.
2.a-Stronger
HDFS metadata and data can change across minor or major releases, but
such
changes are transparent to user application. That is release upgrade
must
automatically convert the metadata and data as needed. During *minor*
releases,
disk format changes have to backward and forward compatible; i.e. an
older
version of Hadoop can be started on a newer version of the disk
format. Hence
a version roll back is simple, just restart the older version of Hadoop.
Major releases allow more significant changes to the disk format and
have be
only backward compatible; however major release upgrade must allow a
cluster to
roll back to the older version and its older disk format.
Motivation: Minor release are very easy to roll back for an admin.
2.a-WeakerAutomaticConversion:
Automatic conversion is supported across a small number of releases.
If a user
wants to jump across multiple releases he may be forced to go through
a few
intermediate release to get to the final desired release.
3. Wire Protocol Compatibility
----------------------------------------
We offer no wire compatibility in our 0.x release today.
The motivation *isn't* to make a our protocols public. Applications
will not
call the protocol directly but through a library (in our case
FileSystem class
and its implementations). Instead the motivation is that customers run
multiple clusters and have apps that access data across clusters.
Customers
cannot be expected to update all clusters simultaneously.
3.a Old m.x clients can connect to new m.y servers, where x <= y but
the old clients might get reduced functionality or performance. m.x
clients might not be able to connect to (m+1).z servers
3.b. New m.y clients must be able to connect to old m.x server, where
x< y but
only for old m.x functionality.
Comment: Generally old API methods continue to use old rpc methods.
However, it is legal to have new implementations of old API methods
call new
rpcs methods, as long as the library transparently handles the fallback
case for old servers.
3.c. At any major release transition [ ie from a release m.x to a
release (m+1).0], a user should be able to read data from the cluster
running the old version. (OR shall we generalize this to: from m.x to
(m+i).z ?)
Motivation: data copying across clusters is a common operation for many
customers. For example this is routinely at done at Yahoo; another use
case is
HADOOP-4058. Today, http (or hftp) provides a guaranteed compatible
way of
copying data across versions. Clearly one cannot force a customer to
simultaneously update all its Hadoop clusters on to a new major
release. The
above documents this requirement; we can satisfy it via the http/hftp
mechanism
or some other mechanism.
3.c-Stronger
Shall we add a stronger requirement for 1. 0 : wire compatibility
across major versions? That is not just for reading but for all
operations. This can be supported by class loading or other games.
Note we can wait to provide this when 2. 0 happens. If Hadoop
provided this guarantee then it would allow customers to partition
their data across clusters without risking apps breaking across major
releases due to wire incompatibility issues.
Motivation: Data copying is a compromise. Customers really want to run
apps across clusters running different versions. ( See item 2 )
4. Intra Hadoop Service Compatibility
--------------------------------------------------
The HDFS Service has multiple components (NN, DN, Balancer) that
communicate
amongst themselves. Similarly the MapReduce service has
components (JR and TT) that communicate amongst themselves.
Currently we require that the all the components of a service have the
same build version and hence talk the same wire protocols.
This build-version checking prevents rolling upgrades. It has the
benefit that the admin can ensure that the entire cluster has exactly
the same build version.
4.a HDFS and MapReduce require that their respective sub-components
have the same build version in order to form a cluster.
[ie. Maintain the current mechanism.]
4.a-Stronger: Intra-service wire-protocol compatibility
[I am listing this here to document it, but I don't think we are ready
to take
this on for Hadoop 1.0. Alternatively, we could require intra -service
wire
compatibility but check for build version till we are ready for rolling
upgrades]
Wire protocols between internal Hadoop components are compatible across
minor versions.
Examples are NN-DN, DN-DN and NN-Balancer, etc.
Old m.x components can talk to new m.y components (x<=y)
Wire compatibility can break across major versions.
Motivation: Allow rolling upgrades.
Re: Hadoop 1.0 Compatibility Discussion.
Posted by Sanjay Radia <sr...@yahoo-inc.com>.
On Oct 23, 2008, at 11:02 AM, Dhruba Borthakur wrote:
> Hi Sanjay,
>
> Ok, let me understand the terminology.
> -- Versions 1.0 and versions 2.0 are "major" releases, maybe occur
> once every year
> -- Versions 1.2 and 1.3 are minor releases, maybe occurs once
> every 2 months
> -- Versions 2.4.5 and 2.4.6 are "dot" release.
>
> Is my understanding correct?
>
Yes.
>
>
> >don't you want disk format compatibility to be for both major and
> minor release changes as I stated earlier?
>
> Disk formats can change for both major and minor releases but not for
> dot release. And the hadoop code should handle the required disk
> format upgrades and downgrades transparently. This issue, by itself,
> should have no impact on pre-existing FileSystem APIs.
>
Yes.
How about the following stronger requirement on disk formats:
- disk formats should be forward AND backward compatible for minor
releases (ie 1.2 to 1.3)
this can be done though the versioning techniques used by protocol
buffers, thrift, hessian
No roll back is needed, olver version can ignore the fields
they don't know about.
- disk formats can be changed in a non-backward compatible way for
major releases (eg. 1.9 to 2.) but that a roll back is possible.
So for example adding fields to the fsImage or edits log records is
forward and backward compatible and can be done on any minor release.
But the changes like we did when we added CRCs could only be done on
major releases.
I think this stronger requirement will make Hadoop admin's life a lot
easier.
sanjay
>
>
> Does this make sense?
>
> thanks,
> dhruba
>
> On Wed, Oct 22, 2008 at 9:04 AM, Sanjay Radia <sr...@yahoo-inc.com>
> wrote:
> >
> >
> >
> > On Oct 20, 2008, at 11:44 PM, Dhruba Borthakur wrote:
> >>
> >> 3. HDFS disk format can change from 1.x to 1.y release and is
> >> transparent to user-application. A cluster when rolling back to 1.x
> >> from 1,y will revert back to the old disk format.
> >
> > Dhruba don't you want disk format compatibility to be for both
> major and
> > minor release changes as I stated earlier?
> > I don't think it is acceptable to break data format across major
> releases.
> >
> >
> > Q. do we want disk upgrade to be supported across arbitrary number
> of
> > releases? For example we allowed the crc upgrade for
> > approx 4 releases in the past.
> >
> >
> >
>
Re: Hadoop 1.0 Compatibility Discussion.
Posted by Konstantin Shvachko <sh...@yahoo-inc.com>.
> dot release. And the hadoop code should handle the required disk
> format upgrades and downgrades transparently. This issue, by itself,
If by "downgrades" you mean "rollbacks", then this sounds right to me.
You did not mean that we should support data conversions from
higher versions to lower, right?
I think Sanjay's proposal to weaken the data conversion requirement
makes sense. We should say that direct upgrades between some versions
may not be possible, only via intermediate version(s). But we should
also say what is the minimal distance between such versions.
E.g. "direct upgrades between different major releases may not be
possible".
--Konstantin
Dhruba Borthakur wrote:
> Hi Sanjay,
>
> Ok, let me understand the terminology.
> -- Versions 1.0 and versions 2.0 are "major" releases, maybe occur
> once every year
> -- Versions 1.2 and 1.3 are minor releases, maybe occurs once every 2 months
> -- Versions 2.4.5 and 2.4.6 are "dot" release.
>
> Is my understanding correct?
>
>> don't you want disk format compatibility to be for both major and minor release changes as I stated earlier?
>
> Disk formats can change for both major and minor releases but not for
> dot release. And the hadoop code should handle the required disk
> format upgrades and downgrades transparently. This issue, by itself,
> should have no impact on pre-existing FileSystem APIs.
>
> Does this make sense?
>
> thanks,
> dhruba
>
> On Wed, Oct 22, 2008 at 9:04 AM, Sanjay Radia <sr...@yahoo-inc.com> wrote:
>>
>>
>> On Oct 20, 2008, at 11:44 PM, Dhruba Borthakur wrote:
>>> 3. HDFS disk format can change from 1.x to 1.y release and is
>>> transparent to user-application. A cluster when rolling back to 1.x
>>> from 1,y will revert back to the old disk format.
>> Dhruba don't you want disk format compatibility to be for both major and
>> minor release changes as I stated earlier?
>> I don't think it is acceptable to break data format across major releases.
>>
>>
>> Q. do we want disk upgrade to be supported across arbitrary number of
>> releases? For example we allowed the crc upgrade for
>> approx 4 releases in the past.
>>
>>
>>
>
Re: Hadoop 1.0 Compatibility Discussion.
Posted by Dhruba Borthakur <dh...@gmail.com>.
Hi Sanjay,
Ok, let me understand the terminology.
-- Versions 1.0 and versions 2.0 are "major" releases, maybe occur
once every year
-- Versions 1.2 and 1.3 are minor releases, maybe occurs once every 2 months
-- Versions 2.4.5 and 2.4.6 are "dot" release.
Is my understanding correct?
>don't you want disk format compatibility to be for both major and minor release changes as I stated earlier?
Disk formats can change for both major and minor releases but not for
dot release. And the hadoop code should handle the required disk
format upgrades and downgrades transparently. This issue, by itself,
should have no impact on pre-existing FileSystem APIs.
Does this make sense?
thanks,
dhruba
On Wed, Oct 22, 2008 at 9:04 AM, Sanjay Radia <sr...@yahoo-inc.com> wrote:
>
>
>
> On Oct 20, 2008, at 11:44 PM, Dhruba Borthakur wrote:
>>
>> 3. HDFS disk format can change from 1.x to 1.y release and is
>> transparent to user-application. A cluster when rolling back to 1.x
>> from 1,y will revert back to the old disk format.
>
> Dhruba don't you want disk format compatibility to be for both major and
> minor release changes as I stated earlier?
> I don't think it is acceptable to break data format across major releases.
>
>
> Q. do we want disk upgrade to be supported across arbitrary number of
> releases? For example we allowed the crc upgrade for
> approx 4 releases in the past.
>
>
>
Re: Hadoop 1.0 Compatibility Discussion.
Posted by Sanjay Radia <sr...@yahoo-inc.com>.
On Oct 20, 2008, at 11:44 PM, Dhruba Borthakur wrote:
> 3. HDFS disk format can change from 1.x to 1.y release and is
> transparent to user-application. A cluster when rolling back to 1.x
> from 1,y will revert back to the old disk format.
Dhruba don't you want disk format compatibility to be for both major
and minor release changes as I stated earlier?
I don't think it is acceptable to break data format across major
releases.
Q. do we want disk upgrade to be supported across arbitrary number of
releases? For example we allowed the crc upgrade for
approx 4 releases in the past.
Re: Hadoop 1.0 Compatibility Discussion.
Posted by Sanjay Radia <sr...@yahoo-inc.com>.
On Oct 21, 2008, at 3:31 PM, Dhruba Borthakur wrote:
> Thinking about this some more, it will be beneficial to move the
> client-datanode data transfer communication to a more formalized RPC
> protocol before the 1.0 release
> http://issues.apache.org/jira/browse/HADOOP-4386
>
Either that or
http://issues.apache.org/jira/browse/HADOOP-4008
One of the two are needed before hadoop 1.0.
4008 sticks with the socket level protocol but formalizes the message
exchanges so that they can be versioned.
I think 4008 is much easier to do than 4386 and probably sufficient
for Hadoop 1.0. Hadoop-4386 would be good for Hadoop 2.0
I was planning to add 4008 to the hadoop 1.0 pre-requisites.
sanjay
>
>
> thanks,
> dhruba
>
>
>
> On Mon, Oct 20, 2008 at 11:44 PM, Dhruba Borthakur
> <dh...@gmail.com> wrote:
> > 1. APIs that are deprecated in x.y release can be removed in (x+1).
> 0 release.
> >
> > 2. Old 1.x clients can connect to new 1.y servers, where x <= y but
> > the old clients might get reduced functionality or performance. 1.x
> > clients might not be able to connect to 2.z servers.
> >
> > 3. HDFS disk format can change from 1.x to 1.y release and is
> > transparent to user-application. A cluster when rolling back to 1.x
> > from 1,y will revert back to the old disk format.
> >
> >> * In a major release transition [ ie from a release x.y to a
> release
> >> (x+1).0], a user should be able to read data from the cluster
> running the
> >> old version.
> >
> > I think this is a good requirement to have. This will be very useful
> > when we run multiple clusters, especially across data centers
> > (HADOOP-4058 is a use-case).
> >
> > thanks,
> > dhruba
> >
> >> --------
> >> What does Hadoop 1.0 mean?
> >> * Standard release numbering: Only bug fixes in 1.x.y releases
> and new
> >> features in 1.x.0 releases.
> >> * No need for client recompilation when upgrading from 1.x to
> 1.y, where
> >> x <= y
> >> o Can't remove deprecated classes or methods until 2.0
> >> * Old 1.x clients can connect to new 1.y servers, where x <= y
> >> * New FileSystem clients must be able to call old methods when
> talking to
> >> old servers. This generally will be done by having old methods
> continue to
> >> use old rpc methods. However, it is legal to have new
> implementations of old
> >> methods call new rpcs methods, as long as the library
> transparently handles
> >> the fallback case for old servers.
> >> -----------------
> >>
> >> A couple of additional compatibility requirements:
> >>
> >> * HDFS metadata and data is preserved across release changes,
> both major and
> >> minor. That is,
> >> whenever a release is upgraded, the HDFS metadata from the old
> release will
> >> be converted automatically
> >> as needed.
> >>
> >> The above has been followed so far in Hadoop; I am just
> documenting it in
> >> the 1.0 requirements list.
> >>
> >> * In a major release transition [ ie from a release x.y to a
> release
> >> (x+1).0], a user should be able to read data from the cluster
> running the
> >> old version. (OR shall we generalize this to: from x.y to (x
> +i).z ?)
> >>
> >> The motivation: data copying across clusters is a common
> operation for many
> >> customers
> >> (for example this is routinely at done at Yahoo.). Today, http
> (or hftp)
> >> provides a guaranteed compatible way of copying data across
> versions.
> >> Clearly one cannot force a customer to simultaneously update all
> its hadoop
> >> clusters on to
> >> a new major release. The above documents this requirement; we can
> satisfy it
> >> via the http/hftp mechanism or some other mechanism.
> >>
> >> Question: is one is willing to break applications that operate
> across
> >> clusters (ie an application that accesses data across clusters
> that cross a
> >> major release boundary? I asked the operations team at Yahoo that
> run our
> >> hadoop clusters. We currently do not have any applicaions that
> access data
> >> across clusters as part of a MR job. The reason being that
> Hadoop routinely
> >> breaks wire compatibility across releases and so such apps would
> be very
> >> unreliable. However, the copying of data across clusters is t is
> crucial and
> >> needs to be supported.
> >>
> >> Shall we add a stronger requirement for 1.0: wire compatibility
> across
> >> major versions? This can be supported by class loading or other
> games. Note
> >> we can wait to provide this when 2.0 happens. If Hadoop provided
> this
> >> guarantee then it would allow customers to partition their data
> across
> >> clusters without risking apps breaking across major releases due
> to wire
> >> incompatibility issues.
> >>
> >>
> >>
> >
>
Re: Hadoop 1.0 Compatibility Discussion.
Posted by Dhruba Borthakur <dh...@gmail.com>.
Thinking about this some more, it will be beneficial to move the
client-datanode data transfer communication to a more formalized RPC
protocol before the 1.0 release
http://issues.apache.org/jira/browse/HADOOP-4386
thanks,
dhruba
On Mon, Oct 20, 2008 at 11:44 PM, Dhruba Borthakur <dh...@gmail.com> wrote:
> 1. APIs that are deprecated in x.y release can be removed in (x+1).0 release.
>
> 2. Old 1.x clients can connect to new 1.y servers, where x <= y but
> the old clients might get reduced functionality or performance. 1.x
> clients might not be able to connect to 2.z servers.
>
> 3. HDFS disk format can change from 1.x to 1.y release and is
> transparent to user-application. A cluster when rolling back to 1.x
> from 1,y will revert back to the old disk format.
>
>> * In a major release transition [ ie from a release x.y to a release
>> (x+1).0], a user should be able to read data from the cluster running the
>> old version.
>
> I think this is a good requirement to have. This will be very useful
> when we run multiple clusters, especially across data centers
> (HADOOP-4058 is a use-case).
>
> thanks,
> dhruba
>
>> --------
>> What does Hadoop 1.0 mean?
>> * Standard release numbering: Only bug fixes in 1.x.y releases and new
>> features in 1.x.0 releases.
>> * No need for client recompilation when upgrading from 1.x to 1.y, where
>> x <= y
>> o Can't remove deprecated classes or methods until 2.0
>> * Old 1.x clients can connect to new 1.y servers, where x <= y
>> * New FileSystem clients must be able to call old methods when talking to
>> old servers. This generally will be done by having old methods continue to
>> use old rpc methods. However, it is legal to have new implementations of old
>> methods call new rpcs methods, as long as the library transparently handles
>> the fallback case for old servers.
>> -----------------
>>
>> A couple of additional compatibility requirements:
>>
>> * HDFS metadata and data is preserved across release changes, both major and
>> minor. That is,
>> whenever a release is upgraded, the HDFS metadata from the old release will
>> be converted automatically
>> as needed.
>>
>> The above has been followed so far in Hadoop; I am just documenting it in
>> the 1.0 requirements list.
>>
>> * In a major release transition [ ie from a release x.y to a release
>> (x+1).0], a user should be able to read data from the cluster running the
>> old version. (OR shall we generalize this to: from x.y to (x+i).z ?)
>>
>> The motivation: data copying across clusters is a common operation for many
>> customers
>> (for example this is routinely at done at Yahoo.). Today, http (or hftp)
>> provides a guaranteed compatible way of copying data across versions.
>> Clearly one cannot force a customer to simultaneously update all its hadoop
>> clusters on to
>> a new major release. The above documents this requirement; we can satisfy it
>> via the http/hftp mechanism or some other mechanism.
>>
>> Question: is one is willing to break applications that operate across
>> clusters (ie an application that accesses data across clusters that cross a
>> major release boundary? I asked the operations team at Yahoo that run our
>> hadoop clusters. We currently do not have any applicaions that access data
>> across clusters as part of a MR job. The reason being that Hadoop routinely
>> breaks wire compatibility across releases and so such apps would be very
>> unreliable. However, the copying of data across clusters is t is crucial and
>> needs to be supported.
>>
>> Shall we add a stronger requirement for 1.0: wire compatibility across
>> major versions? This can be supported by class loading or other games. Note
>> we can wait to provide this when 2.0 happens. If Hadoop provided this
>> guarantee then it would allow customers to partition their data across
>> clusters without risking apps breaking across major releases due to wire
>> incompatibility issues.
>>
>>
>>
>
Re: Hadoop 1.0 Compatibility Discussion.
Posted by Dhruba Borthakur <dh...@gmail.com>.
1. APIs that are deprecated in x.y release can be removed in (x+1).0 release.
2. Old 1.x clients can connect to new 1.y servers, where x <= y but
the old clients might get reduced functionality or performance. 1.x
clients might not be able to connect to 2.z servers.
3. HDFS disk format can change from 1.x to 1.y release and is
transparent to user-application. A cluster when rolling back to 1.x
from 1,y will revert back to the old disk format.
> * In a major release transition [ ie from a release x.y to a release
> (x+1).0], a user should be able to read data from the cluster running the
> old version.
I think this is a good requirement to have. This will be very useful
when we run multiple clusters, especially across data centers
(HADOOP-4058 is a use-case).
thanks,
dhruba
> --------
> What does Hadoop 1.0 mean?
> * Standard release numbering: Only bug fixes in 1.x.y releases and new
> features in 1.x.0 releases.
> * No need for client recompilation when upgrading from 1.x to 1.y, where
> x <= y
> o Can't remove deprecated classes or methods until 2.0
> * Old 1.x clients can connect to new 1.y servers, where x <= y
> * New FileSystem clients must be able to call old methods when talking to
> old servers. This generally will be done by having old methods continue to
> use old rpc methods. However, it is legal to have new implementations of old
> methods call new rpcs methods, as long as the library transparently handles
> the fallback case for old servers.
> -----------------
>
> A couple of additional compatibility requirements:
>
> * HDFS metadata and data is preserved across release changes, both major and
> minor. That is,
> whenever a release is upgraded, the HDFS metadata from the old release will
> be converted automatically
> as needed.
>
> The above has been followed so far in Hadoop; I am just documenting it in
> the 1.0 requirements list.
>
> * In a major release transition [ ie from a release x.y to a release
> (x+1).0], a user should be able to read data from the cluster running the
> old version. (OR shall we generalize this to: from x.y to (x+i).z ?)
>
> The motivation: data copying across clusters is a common operation for many
> customers
> (for example this is routinely at done at Yahoo.). Today, http (or hftp)
> provides a guaranteed compatible way of copying data across versions.
> Clearly one cannot force a customer to simultaneously update all its hadoop
> clusters on to
> a new major release. The above documents this requirement; we can satisfy it
> via the http/hftp mechanism or some other mechanism.
>
> Question: is one is willing to break applications that operate across
> clusters (ie an application that accesses data across clusters that cross a
> major release boundary? I asked the operations team at Yahoo that run our
> hadoop clusters. We currently do not have any applicaions that access data
> across clusters as part of a MR job. The reason being that Hadoop routinely
> breaks wire compatibility across releases and so such apps would be very
> unreliable. However, the copying of data across clusters is t is crucial and
> needs to be supported.
>
> Shall we add a stronger requirement for 1.0: wire compatibility across
> major versions? This can be supported by class loading or other games. Note
> we can wait to provide this when 2.0 happens. If Hadoop provided this
> guarantee then it would allow customers to partition their data across
> clusters without risking apps breaking across major releases due to wire
> incompatibility issues.
>
>
>
Re: Hadoop 1.0 Compatibility Discussion.
Posted by Robert Chansler <ch...@yahoo-inc.com>.
I'd endorse Konstantin's suggestion that the only "downgrade" that will be
supported is "rollback," (and roll back works regardless of version).
There should be a time and version rule for allowed upgrades. "Upgrades to
1.x will be supported for 0.x for x>17. Upgrades to X.* will be supported
for (X-1).*, and also (X-2).* if (X-1).0 is less than one year old."
For interoperation between versions, I might conspicuously deprecate
HFTP/HTTP access to files in 1.0 while making a strong demand for
interoperability. "Client applications may read/write data between any two
versions not less than 1.0 that permit upgrade. 1.* need only support
HTTP/HFTP for sufficiently relaxed security regimes. Support for HTTP/HFTP
may be withdrawn in 2.0."
On 20 10 08 18:50, "Sanjay Radia" <sr...@yahoo-inc.com> wrote:
> The Hadoop 1.0 wiki has a section on compatibility.
> http://wiki.apache.org/hadoop/Release1.0Requirements
>
> Since the wiki is awkward for discussions, I am continuing the
> discussion here.
> I or someone will update the wiki when agreements are reached.
>
> Here is the current list of compatibility requirements on the Hadoop
> 1.0 Wiki for the convenience of this email thread.
> --------
> What does Hadoop 1.0 mean?
> * Standard release numbering: Only bug fixes in 1.x.y releases
> and new features in 1.x.0 releases.
> * No need for client recompilation when upgrading from 1.x to
> 1.y, where x <= y
> o Can't remove deprecated classes or methods until 2.0
> * Old 1.x clients can connect to new 1.y servers, where x <= y
> * New FileSystem clients must be able to call old methods when
> talking to old servers. This generally will be done by having old
> methods continue to use old rpc methods. However, it is legal to have
> new implementations of old methods call new rpcs methods, as long as
> the library transparently handles the fallback case for old servers.
> -----------------
>
> A couple of additional compatibility requirements:
>
> * HDFS metadata and data is preserved across release changes, both
> major and minor. That is,
> whenever a release is upgraded, the HDFS metadata from the old release
> will be converted automatically
> as needed.
>
> The above has been followed so far in Hadoop; I am just documenting it
> in the 1.0 requirements list.
>
> * In a major release transition [ ie from a release x.y to a
> release (x+1).0], a user should be able to read data from the cluster
> running the old version. (OR shall we generalize this to: from x.y to
> (x+i).z ?)
>
> The motivation: data copying across clusters is a common operation for
> many customers
> (for example this is routinely at done at Yahoo.). Today, http (or
> hftp) provides a guaranteed compatible way of copying data across
> versions. Clearly one cannot force a customer to simultaneously
> update all its hadoop clusters on to
> a new major release. The above documents this requirement; we can
> satisfy it via the http/hftp mechanism or some other mechanism.
>
> Question: is one is willing to break applications that operate across
> clusters (ie an application that accesses data across clusters that
> cross a major release boundary? I asked the operations team at Yahoo
> that run our hadoop clusters. We currently do not have any applicaions
> that access data across clusters as part of a MR job. The reason
> being that Hadoop routinely breaks wire compatibility across releases
> and so such apps would be very unreliable. However, the copying of
> data across clusters is t is crucial and needs to be supported.
>
> Shall we add a stronger requirement for 1.0: wire compatibility
> across major versions? This can be supported by class loading or other
> games. Note we can wait to provide this when 2.0 happens. If Hadoop
> provided this guarantee then it would allow customers to partition
> their data across clusters without risking apps breaking across major
> releases due to wire incompatibility issues.
>
>