You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Chris Nauroth <cn...@hortonworks.com> on 2014/03/19 21:59:11 UTC

[DISCUSS] Clarification on Compatibility Policy: Upgraded Client + Old Server

I'd like to discuss clarification of part of our compatibility policy.
 Here is a link to the compatibility documentation for release 2.3.0:

http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common/Compatibility.html#Wire_compatibility

For convenience, here are the specific lines in question:

Client-Server compatibility is required to allow users to continue using
the old clients even after upgrading the server (cluster) to a later
version (or vice versa). For example, a Hadoop 2.1.0 client talking to a
Hadoop 2.3.0 cluster.

Client-Server compatibility is also required to allow upgrading individual
components without upgrading others. For example, upgrade HDFS from version
2.1.0 to 2.2.0 without upgrading MapReduce.

Server-Server compatibility is required to allow mixed versions within an
active cluster so the cluster may be upgraded without downtime in a rolling
fashion.

Notice that there is no specific mention of upgrading the client ahead of
the server.  (There is no clause for "upgraded client + old server".)
 Based on my experience, this is a valid use case when a user wants to pick
up a client-side bug fix ahead of the cluster administrator's upgrade
schedule.

Is it our policy to maintain client compatibility with old clusters within
the same major release?  I think many of us have assumed that the answer is
yes and coded our new features accordingly, but it isn't made explicit in
the documentation.  Do we all agree that the answer is yes, or is it
possibly up for debate depending on the change in question?  In RFC 2119
lingo, is it a MUST or a SHOULD?  Either way, I'd like to update the policy
text to make our decision clear.  After we have consensus, I can volunteer
to file an issue and patch the text of the policy.

This discussion started initially in MAPREDUCE-4052, which involved
changing our scripting syntax for MapReduce YARN container submissions.  We
settled the question there by gating the syntax change behind a
configuration option.  By default, it will continue using the existing
syntax currently understood by the pre-2.4.0 NodeManager, thus preserving
compatibility.  We wanted to open the policy question for wider discussion
though.

Thanks, everyone.

Chris Nauroth
Hortonworks
http://hortonworks.com/

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded Client + Old Server

Posted by Karthik Kambatla <ka...@cloudera.com>.
+1 on supporting new clients with old servers of the same major version,
and updating the policy to capture that clearly.


On Wed, Mar 19, 2014 at 1:59 PM, Chris Nauroth <cn...@hortonworks.com>wrote:

> I'd like to discuss clarification of part of our compatibility policy.
>  Here is a link to the compatibility documentation for release 2.3.0:
>
>
> http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common/Compatibility.html#Wire_compatibility
>
> For convenience, here are the specific lines in question:
>
> Client-Server compatibility is required to allow users to continue using
> the old clients even after upgrading the server (cluster) to a later
> version (or vice versa). For example, a Hadoop 2.1.0 client talking to a
> Hadoop 2.3.0 cluster.
>
> Client-Server compatibility is also required to allow upgrading individual
> components without upgrading others. For example, upgrade HDFS from version
> 2.1.0 to 2.2.0 without upgrading MapReduce.
>
> Server-Server compatibility is required to allow mixed versions within an
> active cluster so the cluster may be upgraded without downtime in a rolling
> fashion.
>
> Notice that there is no specific mention of upgrading the client ahead of
> the server.  (There is no clause for "upgraded client + old server".)
>  Based on my experience, this is a valid use case when a user wants to pick
> up a client-side bug fix ahead of the cluster administrator's upgrade
> schedule.
>
> Is it our policy to maintain client compatibility with old clusters within
> the same major release?  I think many of us have assumed that the answer is
> yes and coded our new features accordingly, but it isn't made explicit in
> the documentation.  Do we all agree that the answer is yes, or is it
> possibly up for debate depending on the change in question?  In RFC 2119
> lingo, is it a MUST or a SHOULD?  Either way, I'd like to update the policy
> text to make our decision clear.  After we have consensus, I can volunteer
> to file an issue and patch the text of the policy.
>
> This discussion started initially in MAPREDUCE-4052, which involved
> changing our scripting syntax for MapReduce YARN container submissions.  We
> settled the question there by gating the syntax change behind a
> configuration option.  By default, it will continue using the existing
> syntax currently understood by the pre-2.4.0 NodeManager, thus preserving
> compatibility.  We wanted to open the policy question for wider discussion
> though.
>
> Thanks, everyone.
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded Client + Old Server

Posted by Chris Nauroth <cn...@hortonworks.com>.
Adding back all *-dev lists to make sure everyone is covered.

Chris Nauroth
Hortonworks
http://hortonworks.com/



On Mon, Mar 24, 2014 at 2:02 PM, Chris Nauroth <cn...@hortonworks.com>wrote:

> Thank you, everyone, for the discussion.  There is general agreement, so I
> have filed HADOOP-10423 with a patch to update the compatibility
> documentation.
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
>
> On Thu, Mar 20, 2014 at 11:24 AM, Colin McCabe <cm...@alumni.cmu.edu>wrote:
>
>> +1 for making this guarantee explicit.
>>
>> It also definitely seems like a good idea to test mixed versions in
>> bigtop.
>>
>> HDFS is not immune to "new client, old server" scenarios because the HDFS
>> client gets bundled into a lot of places.
>>
>> Colin
>> On Mar 20, 2014 10:55 AM, "Chris Nauroth" <cn...@hortonworks.com>
>> wrote:
>>
>> > Our use of protobuf helps mitigate a lot of compatibility concerns, but
>> > there still can be situations that require careful coding on our part.
>> >  When adding a new field to a protobuf message, the client might need
>> to do
>> > a null check, even if the server-side implementation in the new version
>> > always populates the field.  When adding a whole new RPC endpoint, the
>> > client might need to consider the possibility that the RPC endpoint
>> isn't
>> > there on an old server, and degrade gracefully after the RPC fails.  The
>> > original issue in MAPREDUCE-4052 concerned the script commands passed
>> in a
>> > YARN container submission, where protobuf doesn't provide any validation
>> > beyond the fact that they're strings.
>> >
>> > Forward compatibility is harder than backward compatibility, and
>> testing is
>> > a big challenge.  Our test suites in the Hadoop repo don't cover this.
>> >  Does anyone know if anything in Bigtop tries to run with mixed
>> versions?
>> >
>> > I agree that we need to make it clear in the language that upgrading
>> client
>> > alone is insufficient to get access to new server-side features,
>> including
>> > new YARN APIs.  Thanks for the suggestions, Steve.
>> >
>> > Chris Nauroth
>> > Hortonworks
>> > http://hortonworks.com/
>> >
>> >
>> >
>> > On Thu, Mar 20, 2014 at 5:53 AM, Steve Loughran <stevel@hortonworks.com
>> > >wrote:
>> >
>> > > I'm clearly supportive of this, though of course the testing costs
>> needed
>> > > to back up the assertion make it more expensive than just a statement.
>> > >
>> > > Two issues
>> > >
>> > > -we'd need to make clear that new cluster features that a client can
>> > invoke
>> > > won't be available. You can't expect snapshot or symlink support
>> running
>> > > against a -2.2.0 cluster, even if the client supports it.
>> > >
>> > > -in YARN, there are no guarantees that an app compiled against later
>> YARN
>> > > APIs will work in old clusters. Because YARN apps upload themselves to
>> > the
>> > > server, and run with their hadoop, hdfs & yarn libraries. We have to
>> do a
>> > > bit of introspection in our code already to support this situation.
>> The
>> > > compatibility doc would need to be clear on that too: "YARN apps that
>> use
>> > > new APIs (including new fields in datastructures) can expect link
>> > > exceptions"
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > On 20 March 2014 04:25, Vinayakumar B <vi...@huawei.com>
>> wrote:
>> > >
>> > > > +1, I agree with your point Chris. It depends on the client
>> application
>> > > > how they using the hdfs jars in their classpath.
>> > > >
>> > > > As implementation already supports the compatibility (through
>> > protobuf),
>> > > > No extra code changes required to support new Client + old server.
>> > > >
>> > > > I feel it will be good to explicitly mention about the
>> compatibility of
>> > > > existing APIs in both versions.
>> > > >
>> > > > Anyway this is not applicable for the new APIs in latest client and
>> > this
>> > > > is understood. We can make it explicit in the document though.
>> > > >
>> > > >
>> > > > Regards,
>> > > > Vinayakumar B
>> > > >
>> > > > -----Original Message-----
>> > > > From: Chris Nauroth [mailto:cnauroth@hortonworks.com]
>> > > > Sent: 20 March 2014 05:36
>> > > > To: common-dev@hadoop.apache.org
>> > > > Cc: mapreduce-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
>> > > > yarn-dev@hadoop.apache.org
>> > > > Subject: Re: [DISCUSS] Clarification on Compatibility Policy:
>> Upgraded
>> > > > Client + Old Server
>> > > >
>> > > > I think this kind of compatibility issue still could surface for
>> HDFS,
>> > > > particularly for custom applications (i.e. something not executed
>> via
>> > > > "hadoop jar" on a cluster node, where the client classes ought to be
>> > > > injected into the classpath automatically).  Running DistCP between
>> 2
>> > > > clusters of different versions could result in a 2.4.0 client
>> calling a
>> > > > 2.3.0 NameNode.  Someone could potentially pick up the 2.4.0 WebHDFS
>> > > > client as a dependency and try to use it to make HTTP calls to a
>> 2.3.0
>> > > HDFS
>> > > > cluster.
>> > > >
>> > > > Chris Nauroth
>> > > > Hortonworks
>> > > > http://hortonworks.com/
>> > > >
>> > > >
>> > > >
>> > > > On Wed, Mar 19, 2014 at 4:28 PM, Vinod Kumar Vavilapalli <
>> > > > vinodkv@apache.org
>> > > > > wrote:
>> > > >
>> > > > > It makes sense only for YARN today where we separated out the
>> > clients.
>> > > > > HDFS is still a monolithic jar so this compatibility issue is
>> kind of
>> > > > > invalid there.
>> > > > >
>> > > > > +vinod
>> > > > >
>> > > > > On Mar 19, 2014, at 1:59 PM, Chris Nauroth <
>> cnauroth@hortonworks.com
>> > >
>> > > > > wrote:
>> > > > >
>> > > > > > I'd like to discuss clarification of part of our compatibility
>> > > policy.
>> > > > > > Here is a link to the compatibility documentation for release
>> > 2.3.0:
>> > > > > >
>> > > > > >
>> > > > >
>> > http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common
>> > > > > /Compatibility.html#Wire_compatibility
>> > > > > >
>> > > > > > For convenience, here are the specific lines in question:
>> > > > > >
>> > > > > > Client-Server compatibility is required to allow users to
>> continue
>> > > > > > using the old clients even after upgrading the server (cluster)
>> to
>> > a
>> > > > > > later version (or vice versa). For example, a Hadoop 2.1.0
>> client
>> > > > > > talking to a Hadoop 2.3.0 cluster.
>> > > > > >
>> > > > > > Client-Server compatibility is also required to allow upgrading
>> > > > > individual
>> > > > > > components without upgrading others. For example, upgrade HDFS
>> from
>> > > > > version
>> > > > > > 2.1.0 to 2.2.0 without upgrading MapReduce.
>> > > > > >
>> > > > > > Server-Server compatibility is required to allow mixed versions
>> > > > > > within an active cluster so the cluster may be upgraded without
>> > > > > > downtime in a
>> > > > > rolling
>> > > > > > fashion.
>> > > > > >
>> > > > > > Notice that there is no specific mention of upgrading the client
>> > > > > > ahead of the server.  (There is no clause for "upgraded client +
>> > old
>> > > > > > server".) Based on my experience, this is a valid use case when
>> a
>> > > > > > user wants to
>> > > > > pick
>> > > > > > up a client-side bug fix ahead of the cluster administrator's
>> > > > > > upgrade schedule.
>> > > > > >
>> > > > > > Is it our policy to maintain client compatibility with old
>> clusters
>> > > > > within
>> > > > > > the same major release?  I think many of us have assumed that
>> the
>> > > > > > answer
>> > > > > is
>> > > > > > yes and coded our new features accordingly, but it isn't made
>> > > > > > explicit in the documentation.  Do we all agree that the answer
>> is
>> > > > > > yes, or is it possibly up for debate depending on the change in
>> > > > > > question?  In RFC 2119 lingo, is it a MUST or a SHOULD?  Either
>> > way,
>> > > > > > I'd like to update the
>> > > > > policy
>> > > > > > text to make our decision clear.  After we have consensus, I can
>> > > > > volunteer
>> > > > > > to file an issue and patch the text of the policy.
>> > > > > >
>> > > > > > This discussion started initially in MAPREDUCE-4052, which
>> involved
>> > > > > > changing our scripting syntax for MapReduce YARN container
>> > > submissions.
>> > > > >  We
>> > > > > > settled the question there by gating the syntax change behind a
>> > > > > > configuration option.  By default, it will continue using the
>> > > > > > existing syntax currently understood by the pre-2.4.0
>> NodeManager,
>> > > > > > thus preserving compatibility.  We wanted to open the policy
>> > > > > > question for wider
>> > > > > discussion
>> > > > > > though.
>> > > > > >
>> > > > > > Thanks, everyone.
>> > > > > >
>> > > > > > Chris Nauroth
>> > > > > > Hortonworks
>> > > > > > http://hortonworks.com/
>> > > > > >
>> > > > > > --
>> > > > > > CONFIDENTIALITY NOTICE
>> > > > > > NOTICE: This message is intended for the use of the individual
>> or
>> > > > > > entity
>> > > > > to
>> > > > > > which it is addressed and may contain information that is
>> > > > > > confidential, privileged and exempt from disclosure under
>> > applicable
>> > > > > > law. If the reader of this message is not the intended
>> recipient,
>> > > > > > you are hereby notified
>> > > > > that
>> > > > > > any printing, copying, dissemination, distribution, disclosure
>> or
>> > > > > > forwarding of this communication is strictly prohibited. If you
>> > have
>> > > > > > received this communication in error, please contact the sender
>> > > > > immediately
>> > > > > > and delete it from your system. Thank You.
>> > > > >
>> > > > >
>> > > > > --
>> > > > > CONFIDENTIALITY NOTICE
>> > > > > NOTICE: This message is intended for the use of the individual or
>> > > > > entity to which it is addressed and may contain information that
>> is
>> > > > > confidential, privileged and exempt from disclosure under
>> applicable
>> > > > > law. If the reader of this message is not the intended recipient,
>> you
>> > > > > are hereby notified that any printing, copying, dissemination,
>> > > > > distribution, disclosure or forwarding of this communication is
>> > > > > strictly prohibited. If you have received this communication in
>> > error,
>> > > > > please contact the sender immediately and delete it from your
>> system.
>> > > > Thank You.
>> > > > >
>> > > >
>> > > > --
>> > > > CONFIDENTIALITY NOTICE
>> > > > NOTICE: This message is intended for the use of the individual or
>> > entity
>> > > > to which it is addressed and may contain information that is
>> > > confidential,
>> > > > privileged and exempt from disclosure under applicable law. If the
>> > reader
>> > > > of this message is not the intended recipient, you are hereby
>> notified
>> > > that
>> > > > any printing, copying, dissemination, distribution, disclosure or
>> > > > forwarding of this communication is strictly prohibited. If you have
>> > > > received this communication in error, please contact the sender
>> > > immediately
>> > > > and delete it from your system. Thank You.
>> > > >
>> > >
>> > > --
>> > > CONFIDENTIALITY NOTICE
>> > > NOTICE: This message is intended for the use of the individual or
>> entity
>> > to
>> > > which it is addressed and may contain information that is
>> confidential,
>> > > privileged and exempt from disclosure under applicable law. If the
>> reader
>> > > of this message is not the intended recipient, you are hereby notified
>> > that
>> > > any printing, copying, dissemination, distribution, disclosure or
>> > > forwarding of this communication is strictly prohibited. If you have
>> > > received this communication in error, please contact the sender
>> > immediately
>> > > and delete it from your system. Thank You.
>> > >
>> >
>> > --
>> > CONFIDENTIALITY NOTICE
>> > NOTICE: This message is intended for the use of the individual or
>> entity to
>> > which it is addressed and may contain information that is confidential,
>> > privileged and exempt from disclosure under applicable law. If the
>> reader
>> > of this message is not the intended recipient, you are hereby notified
>> that
>> > any printing, copying, dissemination, distribution, disclosure or
>> > forwarding of this communication is strictly prohibited. If you have
>> > received this communication in error, please contact the sender
>> immediately
>> > and delete it from your system. Thank You.
>> >
>>
>
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded Client + Old Server

Posted by Chris Nauroth <cn...@hortonworks.com>.
Adding back all *-dev lists to make sure everyone is covered.

Chris Nauroth
Hortonworks
http://hortonworks.com/



On Mon, Mar 24, 2014 at 2:02 PM, Chris Nauroth <cn...@hortonworks.com>wrote:

> Thank you, everyone, for the discussion.  There is general agreement, so I
> have filed HADOOP-10423 with a patch to update the compatibility
> documentation.
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
>
> On Thu, Mar 20, 2014 at 11:24 AM, Colin McCabe <cm...@alumni.cmu.edu>wrote:
>
>> +1 for making this guarantee explicit.
>>
>> It also definitely seems like a good idea to test mixed versions in
>> bigtop.
>>
>> HDFS is not immune to "new client, old server" scenarios because the HDFS
>> client gets bundled into a lot of places.
>>
>> Colin
>> On Mar 20, 2014 10:55 AM, "Chris Nauroth" <cn...@hortonworks.com>
>> wrote:
>>
>> > Our use of protobuf helps mitigate a lot of compatibility concerns, but
>> > there still can be situations that require careful coding on our part.
>> >  When adding a new field to a protobuf message, the client might need
>> to do
>> > a null check, even if the server-side implementation in the new version
>> > always populates the field.  When adding a whole new RPC endpoint, the
>> > client might need to consider the possibility that the RPC endpoint
>> isn't
>> > there on an old server, and degrade gracefully after the RPC fails.  The
>> > original issue in MAPREDUCE-4052 concerned the script commands passed
>> in a
>> > YARN container submission, where protobuf doesn't provide any validation
>> > beyond the fact that they're strings.
>> >
>> > Forward compatibility is harder than backward compatibility, and
>> testing is
>> > a big challenge.  Our test suites in the Hadoop repo don't cover this.
>> >  Does anyone know if anything in Bigtop tries to run with mixed
>> versions?
>> >
>> > I agree that we need to make it clear in the language that upgrading
>> client
>> > alone is insufficient to get access to new server-side features,
>> including
>> > new YARN APIs.  Thanks for the suggestions, Steve.
>> >
>> > Chris Nauroth
>> > Hortonworks
>> > http://hortonworks.com/
>> >
>> >
>> >
>> > On Thu, Mar 20, 2014 at 5:53 AM, Steve Loughran <stevel@hortonworks.com
>> > >wrote:
>> >
>> > > I'm clearly supportive of this, though of course the testing costs
>> needed
>> > > to back up the assertion make it more expensive than just a statement.
>> > >
>> > > Two issues
>> > >
>> > > -we'd need to make clear that new cluster features that a client can
>> > invoke
>> > > won't be available. You can't expect snapshot or symlink support
>> running
>> > > against a -2.2.0 cluster, even if the client supports it.
>> > >
>> > > -in YARN, there are no guarantees that an app compiled against later
>> YARN
>> > > APIs will work in old clusters. Because YARN apps upload themselves to
>> > the
>> > > server, and run with their hadoop, hdfs & yarn libraries. We have to
>> do a
>> > > bit of introspection in our code already to support this situation.
>> The
>> > > compatibility doc would need to be clear on that too: "YARN apps that
>> use
>> > > new APIs (including new fields in datastructures) can expect link
>> > > exceptions"
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > On 20 March 2014 04:25, Vinayakumar B <vi...@huawei.com>
>> wrote:
>> > >
>> > > > +1, I agree with your point Chris. It depends on the client
>> application
>> > > > how they using the hdfs jars in their classpath.
>> > > >
>> > > > As implementation already supports the compatibility (through
>> > protobuf),
>> > > > No extra code changes required to support new Client + old server.
>> > > >
>> > > > I feel it will be good to explicitly mention about the
>> compatibility of
>> > > > existing APIs in both versions.
>> > > >
>> > > > Anyway this is not applicable for the new APIs in latest client and
>> > this
>> > > > is understood. We can make it explicit in the document though.
>> > > >
>> > > >
>> > > > Regards,
>> > > > Vinayakumar B
>> > > >
>> > > > -----Original Message-----
>> > > > From: Chris Nauroth [mailto:cnauroth@hortonworks.com]
>> > > > Sent: 20 March 2014 05:36
>> > > > To: common-dev@hadoop.apache.org
>> > > > Cc: mapreduce-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
>> > > > yarn-dev@hadoop.apache.org
>> > > > Subject: Re: [DISCUSS] Clarification on Compatibility Policy:
>> Upgraded
>> > > > Client + Old Server
>> > > >
>> > > > I think this kind of compatibility issue still could surface for
>> HDFS,
>> > > > particularly for custom applications (i.e. something not executed
>> via
>> > > > "hadoop jar" on a cluster node, where the client classes ought to be
>> > > > injected into the classpath automatically).  Running DistCP between
>> 2
>> > > > clusters of different versions could result in a 2.4.0 client
>> calling a
>> > > > 2.3.0 NameNode.  Someone could potentially pick up the 2.4.0 WebHDFS
>> > > > client as a dependency and try to use it to make HTTP calls to a
>> 2.3.0
>> > > HDFS
>> > > > cluster.
>> > > >
>> > > > Chris Nauroth
>> > > > Hortonworks
>> > > > http://hortonworks.com/
>> > > >
>> > > >
>> > > >
>> > > > On Wed, Mar 19, 2014 at 4:28 PM, Vinod Kumar Vavilapalli <
>> > > > vinodkv@apache.org
>> > > > > wrote:
>> > > >
>> > > > > It makes sense only for YARN today where we separated out the
>> > clients.
>> > > > > HDFS is still a monolithic jar so this compatibility issue is
>> kind of
>> > > > > invalid there.
>> > > > >
>> > > > > +vinod
>> > > > >
>> > > > > On Mar 19, 2014, at 1:59 PM, Chris Nauroth <
>> cnauroth@hortonworks.com
>> > >
>> > > > > wrote:
>> > > > >
>> > > > > > I'd like to discuss clarification of part of our compatibility
>> > > policy.
>> > > > > > Here is a link to the compatibility documentation for release
>> > 2.3.0:
>> > > > > >
>> > > > > >
>> > > > >
>> > http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common
>> > > > > /Compatibility.html#Wire_compatibility
>> > > > > >
>> > > > > > For convenience, here are the specific lines in question:
>> > > > > >
>> > > > > > Client-Server compatibility is required to allow users to
>> continue
>> > > > > > using the old clients even after upgrading the server (cluster)
>> to
>> > a
>> > > > > > later version (or vice versa). For example, a Hadoop 2.1.0
>> client
>> > > > > > talking to a Hadoop 2.3.0 cluster.
>> > > > > >
>> > > > > > Client-Server compatibility is also required to allow upgrading
>> > > > > individual
>> > > > > > components without upgrading others. For example, upgrade HDFS
>> from
>> > > > > version
>> > > > > > 2.1.0 to 2.2.0 without upgrading MapReduce.
>> > > > > >
>> > > > > > Server-Server compatibility is required to allow mixed versions
>> > > > > > within an active cluster so the cluster may be upgraded without
>> > > > > > downtime in a
>> > > > > rolling
>> > > > > > fashion.
>> > > > > >
>> > > > > > Notice that there is no specific mention of upgrading the client
>> > > > > > ahead of the server.  (There is no clause for "upgraded client +
>> > old
>> > > > > > server".) Based on my experience, this is a valid use case when
>> a
>> > > > > > user wants to
>> > > > > pick
>> > > > > > up a client-side bug fix ahead of the cluster administrator's
>> > > > > > upgrade schedule.
>> > > > > >
>> > > > > > Is it our policy to maintain client compatibility with old
>> clusters
>> > > > > within
>> > > > > > the same major release?  I think many of us have assumed that
>> the
>> > > > > > answer
>> > > > > is
>> > > > > > yes and coded our new features accordingly, but it isn't made
>> > > > > > explicit in the documentation.  Do we all agree that the answer
>> is
>> > > > > > yes, or is it possibly up for debate depending on the change in
>> > > > > > question?  In RFC 2119 lingo, is it a MUST or a SHOULD?  Either
>> > way,
>> > > > > > I'd like to update the
>> > > > > policy
>> > > > > > text to make our decision clear.  After we have consensus, I can
>> > > > > volunteer
>> > > > > > to file an issue and patch the text of the policy.
>> > > > > >
>> > > > > > This discussion started initially in MAPREDUCE-4052, which
>> involved
>> > > > > > changing our scripting syntax for MapReduce YARN container
>> > > submissions.
>> > > > >  We
>> > > > > > settled the question there by gating the syntax change behind a
>> > > > > > configuration option.  By default, it will continue using the
>> > > > > > existing syntax currently understood by the pre-2.4.0
>> NodeManager,
>> > > > > > thus preserving compatibility.  We wanted to open the policy
>> > > > > > question for wider
>> > > > > discussion
>> > > > > > though.
>> > > > > >
>> > > > > > Thanks, everyone.
>> > > > > >
>> > > > > > Chris Nauroth
>> > > > > > Hortonworks
>> > > > > > http://hortonworks.com/
>> > > > > >
>> > > > > > --
>> > > > > > CONFIDENTIALITY NOTICE
>> > > > > > NOTICE: This message is intended for the use of the individual
>> or
>> > > > > > entity
>> > > > > to
>> > > > > > which it is addressed and may contain information that is
>> > > > > > confidential, privileged and exempt from disclosure under
>> > applicable
>> > > > > > law. If the reader of this message is not the intended
>> recipient,
>> > > > > > you are hereby notified
>> > > > > that
>> > > > > > any printing, copying, dissemination, distribution, disclosure
>> or
>> > > > > > forwarding of this communication is strictly prohibited. If you
>> > have
>> > > > > > received this communication in error, please contact the sender
>> > > > > immediately
>> > > > > > and delete it from your system. Thank You.
>> > > > >
>> > > > >
>> > > > > --
>> > > > > CONFIDENTIALITY NOTICE
>> > > > > NOTICE: This message is intended for the use of the individual or
>> > > > > entity to which it is addressed and may contain information that
>> is
>> > > > > confidential, privileged and exempt from disclosure under
>> applicable
>> > > > > law. If the reader of this message is not the intended recipient,
>> you
>> > > > > are hereby notified that any printing, copying, dissemination,
>> > > > > distribution, disclosure or forwarding of this communication is
>> > > > > strictly prohibited. If you have received this communication in
>> > error,
>> > > > > please contact the sender immediately and delete it from your
>> system.
>> > > > Thank You.
>> > > > >
>> > > >
>> > > > --
>> > > > CONFIDENTIALITY NOTICE
>> > > > NOTICE: This message is intended for the use of the individual or
>> > entity
>> > > > to which it is addressed and may contain information that is
>> > > confidential,
>> > > > privileged and exempt from disclosure under applicable law. If the
>> > reader
>> > > > of this message is not the intended recipient, you are hereby
>> notified
>> > > that
>> > > > any printing, copying, dissemination, distribution, disclosure or
>> > > > forwarding of this communication is strictly prohibited. If you have
>> > > > received this communication in error, please contact the sender
>> > > immediately
>> > > > and delete it from your system. Thank You.
>> > > >
>> > >
>> > > --
>> > > CONFIDENTIALITY NOTICE
>> > > NOTICE: This message is intended for the use of the individual or
>> entity
>> > to
>> > > which it is addressed and may contain information that is
>> confidential,
>> > > privileged and exempt from disclosure under applicable law. If the
>> reader
>> > > of this message is not the intended recipient, you are hereby notified
>> > that
>> > > any printing, copying, dissemination, distribution, disclosure or
>> > > forwarding of this communication is strictly prohibited. If you have
>> > > received this communication in error, please contact the sender
>> > immediately
>> > > and delete it from your system. Thank You.
>> > >
>> >
>> > --
>> > CONFIDENTIALITY NOTICE
>> > NOTICE: This message is intended for the use of the individual or
>> entity to
>> > which it is addressed and may contain information that is confidential,
>> > privileged and exempt from disclosure under applicable law. If the
>> reader
>> > of this message is not the intended recipient, you are hereby notified
>> that
>> > any printing, copying, dissemination, distribution, disclosure or
>> > forwarding of this communication is strictly prohibited. If you have
>> > received this communication in error, please contact the sender
>> immediately
>> > and delete it from your system. Thank You.
>> >
>>
>
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded Client + Old Server

Posted by Chris Nauroth <cn...@hortonworks.com>.
Adding back all *-dev lists to make sure everyone is covered.

Chris Nauroth
Hortonworks
http://hortonworks.com/



On Mon, Mar 24, 2014 at 2:02 PM, Chris Nauroth <cn...@hortonworks.com>wrote:

> Thank you, everyone, for the discussion.  There is general agreement, so I
> have filed HADOOP-10423 with a patch to update the compatibility
> documentation.
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
>
> On Thu, Mar 20, 2014 at 11:24 AM, Colin McCabe <cm...@alumni.cmu.edu>wrote:
>
>> +1 for making this guarantee explicit.
>>
>> It also definitely seems like a good idea to test mixed versions in
>> bigtop.
>>
>> HDFS is not immune to "new client, old server" scenarios because the HDFS
>> client gets bundled into a lot of places.
>>
>> Colin
>> On Mar 20, 2014 10:55 AM, "Chris Nauroth" <cn...@hortonworks.com>
>> wrote:
>>
>> > Our use of protobuf helps mitigate a lot of compatibility concerns, but
>> > there still can be situations that require careful coding on our part.
>> >  When adding a new field to a protobuf message, the client might need
>> to do
>> > a null check, even if the server-side implementation in the new version
>> > always populates the field.  When adding a whole new RPC endpoint, the
>> > client might need to consider the possibility that the RPC endpoint
>> isn't
>> > there on an old server, and degrade gracefully after the RPC fails.  The
>> > original issue in MAPREDUCE-4052 concerned the script commands passed
>> in a
>> > YARN container submission, where protobuf doesn't provide any validation
>> > beyond the fact that they're strings.
>> >
>> > Forward compatibility is harder than backward compatibility, and
>> testing is
>> > a big challenge.  Our test suites in the Hadoop repo don't cover this.
>> >  Does anyone know if anything in Bigtop tries to run with mixed
>> versions?
>> >
>> > I agree that we need to make it clear in the language that upgrading
>> client
>> > alone is insufficient to get access to new server-side features,
>> including
>> > new YARN APIs.  Thanks for the suggestions, Steve.
>> >
>> > Chris Nauroth
>> > Hortonworks
>> > http://hortonworks.com/
>> >
>> >
>> >
>> > On Thu, Mar 20, 2014 at 5:53 AM, Steve Loughran <stevel@hortonworks.com
>> > >wrote:
>> >
>> > > I'm clearly supportive of this, though of course the testing costs
>> needed
>> > > to back up the assertion make it more expensive than just a statement.
>> > >
>> > > Two issues
>> > >
>> > > -we'd need to make clear that new cluster features that a client can
>> > invoke
>> > > won't be available. You can't expect snapshot or symlink support
>> running
>> > > against a -2.2.0 cluster, even if the client supports it.
>> > >
>> > > -in YARN, there are no guarantees that an app compiled against later
>> YARN
>> > > APIs will work in old clusters. Because YARN apps upload themselves to
>> > the
>> > > server, and run with their hadoop, hdfs & yarn libraries. We have to
>> do a
>> > > bit of introspection in our code already to support this situation.
>> The
>> > > compatibility doc would need to be clear on that too: "YARN apps that
>> use
>> > > new APIs (including new fields in datastructures) can expect link
>> > > exceptions"
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > On 20 March 2014 04:25, Vinayakumar B <vi...@huawei.com>
>> wrote:
>> > >
>> > > > +1, I agree with your point Chris. It depends on the client
>> application
>> > > > how they using the hdfs jars in their classpath.
>> > > >
>> > > > As implementation already supports the compatibility (through
>> > protobuf),
>> > > > No extra code changes required to support new Client + old server.
>> > > >
>> > > > I feel it will be good to explicitly mention about the
>> compatibility of
>> > > > existing APIs in both versions.
>> > > >
>> > > > Anyway this is not applicable for the new APIs in latest client and
>> > this
>> > > > is understood. We can make it explicit in the document though.
>> > > >
>> > > >
>> > > > Regards,
>> > > > Vinayakumar B
>> > > >
>> > > > -----Original Message-----
>> > > > From: Chris Nauroth [mailto:cnauroth@hortonworks.com]
>> > > > Sent: 20 March 2014 05:36
>> > > > To: common-dev@hadoop.apache.org
>> > > > Cc: mapreduce-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
>> > > > yarn-dev@hadoop.apache.org
>> > > > Subject: Re: [DISCUSS] Clarification on Compatibility Policy:
>> Upgraded
>> > > > Client + Old Server
>> > > >
>> > > > I think this kind of compatibility issue still could surface for
>> HDFS,
>> > > > particularly for custom applications (i.e. something not executed
>> via
>> > > > "hadoop jar" on a cluster node, where the client classes ought to be
>> > > > injected into the classpath automatically).  Running DistCP between
>> 2
>> > > > clusters of different versions could result in a 2.4.0 client
>> calling a
>> > > > 2.3.0 NameNode.  Someone could potentially pick up the 2.4.0 WebHDFS
>> > > > client as a dependency and try to use it to make HTTP calls to a
>> 2.3.0
>> > > HDFS
>> > > > cluster.
>> > > >
>> > > > Chris Nauroth
>> > > > Hortonworks
>> > > > http://hortonworks.com/
>> > > >
>> > > >
>> > > >
>> > > > On Wed, Mar 19, 2014 at 4:28 PM, Vinod Kumar Vavilapalli <
>> > > > vinodkv@apache.org
>> > > > > wrote:
>> > > >
>> > > > > It makes sense only for YARN today where we separated out the
>> > clients.
>> > > > > HDFS is still a monolithic jar so this compatibility issue is
>> kind of
>> > > > > invalid there.
>> > > > >
>> > > > > +vinod
>> > > > >
>> > > > > On Mar 19, 2014, at 1:59 PM, Chris Nauroth <
>> cnauroth@hortonworks.com
>> > >
>> > > > > wrote:
>> > > > >
>> > > > > > I'd like to discuss clarification of part of our compatibility
>> > > policy.
>> > > > > > Here is a link to the compatibility documentation for release
>> > 2.3.0:
>> > > > > >
>> > > > > >
>> > > > >
>> > http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common
>> > > > > /Compatibility.html#Wire_compatibility
>> > > > > >
>> > > > > > For convenience, here are the specific lines in question:
>> > > > > >
>> > > > > > Client-Server compatibility is required to allow users to
>> continue
>> > > > > > using the old clients even after upgrading the server (cluster)
>> to
>> > a
>> > > > > > later version (or vice versa). For example, a Hadoop 2.1.0
>> client
>> > > > > > talking to a Hadoop 2.3.0 cluster.
>> > > > > >
>> > > > > > Client-Server compatibility is also required to allow upgrading
>> > > > > individual
>> > > > > > components without upgrading others. For example, upgrade HDFS
>> from
>> > > > > version
>> > > > > > 2.1.0 to 2.2.0 without upgrading MapReduce.
>> > > > > >
>> > > > > > Server-Server compatibility is required to allow mixed versions
>> > > > > > within an active cluster so the cluster may be upgraded without
>> > > > > > downtime in a
>> > > > > rolling
>> > > > > > fashion.
>> > > > > >
>> > > > > > Notice that there is no specific mention of upgrading the client
>> > > > > > ahead of the server.  (There is no clause for "upgraded client +
>> > old
>> > > > > > server".) Based on my experience, this is a valid use case when
>> a
>> > > > > > user wants to
>> > > > > pick
>> > > > > > up a client-side bug fix ahead of the cluster administrator's
>> > > > > > upgrade schedule.
>> > > > > >
>> > > > > > Is it our policy to maintain client compatibility with old
>> clusters
>> > > > > within
>> > > > > > the same major release?  I think many of us have assumed that
>> the
>> > > > > > answer
>> > > > > is
>> > > > > > yes and coded our new features accordingly, but it isn't made
>> > > > > > explicit in the documentation.  Do we all agree that the answer
>> is
>> > > > > > yes, or is it possibly up for debate depending on the change in
>> > > > > > question?  In RFC 2119 lingo, is it a MUST or a SHOULD?  Either
>> > way,
>> > > > > > I'd like to update the
>> > > > > policy
>> > > > > > text to make our decision clear.  After we have consensus, I can
>> > > > > volunteer
>> > > > > > to file an issue and patch the text of the policy.
>> > > > > >
>> > > > > > This discussion started initially in MAPREDUCE-4052, which
>> involved
>> > > > > > changing our scripting syntax for MapReduce YARN container
>> > > submissions.
>> > > > >  We
>> > > > > > settled the question there by gating the syntax change behind a
>> > > > > > configuration option.  By default, it will continue using the
>> > > > > > existing syntax currently understood by the pre-2.4.0
>> NodeManager,
>> > > > > > thus preserving compatibility.  We wanted to open the policy
>> > > > > > question for wider
>> > > > > discussion
>> > > > > > though.
>> > > > > >
>> > > > > > Thanks, everyone.
>> > > > > >
>> > > > > > Chris Nauroth
>> > > > > > Hortonworks
>> > > > > > http://hortonworks.com/
>> > > > > >
>> > > > > > --
>> > > > > > CONFIDENTIALITY NOTICE
>> > > > > > NOTICE: This message is intended for the use of the individual
>> or
>> > > > > > entity
>> > > > > to
>> > > > > > which it is addressed and may contain information that is
>> > > > > > confidential, privileged and exempt from disclosure under
>> > applicable
>> > > > > > law. If the reader of this message is not the intended
>> recipient,
>> > > > > > you are hereby notified
>> > > > > that
>> > > > > > any printing, copying, dissemination, distribution, disclosure
>> or
>> > > > > > forwarding of this communication is strictly prohibited. If you
>> > have
>> > > > > > received this communication in error, please contact the sender
>> > > > > immediately
>> > > > > > and delete it from your system. Thank You.
>> > > > >
>> > > > >
>> > > > > --
>> > > > > CONFIDENTIALITY NOTICE
>> > > > > NOTICE: This message is intended for the use of the individual or
>> > > > > entity to which it is addressed and may contain information that
>> is
>> > > > > confidential, privileged and exempt from disclosure under
>> applicable
>> > > > > law. If the reader of this message is not the intended recipient,
>> you
>> > > > > are hereby notified that any printing, copying, dissemination,
>> > > > > distribution, disclosure or forwarding of this communication is
>> > > > > strictly prohibited. If you have received this communication in
>> > error,
>> > > > > please contact the sender immediately and delete it from your
>> system.
>> > > > Thank You.
>> > > > >
>> > > >
>> > > > --
>> > > > CONFIDENTIALITY NOTICE
>> > > > NOTICE: This message is intended for the use of the individual or
>> > entity
>> > > > to which it is addressed and may contain information that is
>> > > confidential,
>> > > > privileged and exempt from disclosure under applicable law. If the
>> > reader
>> > > > of this message is not the intended recipient, you are hereby
>> notified
>> > > that
>> > > > any printing, copying, dissemination, distribution, disclosure or
>> > > > forwarding of this communication is strictly prohibited. If you have
>> > > > received this communication in error, please contact the sender
>> > > immediately
>> > > > and delete it from your system. Thank You.
>> > > >
>> > >
>> > > --
>> > > CONFIDENTIALITY NOTICE
>> > > NOTICE: This message is intended for the use of the individual or
>> entity
>> > to
>> > > which it is addressed and may contain information that is
>> confidential,
>> > > privileged and exempt from disclosure under applicable law. If the
>> reader
>> > > of this message is not the intended recipient, you are hereby notified
>> > that
>> > > any printing, copying, dissemination, distribution, disclosure or
>> > > forwarding of this communication is strictly prohibited. If you have
>> > > received this communication in error, please contact the sender
>> > immediately
>> > > and delete it from your system. Thank You.
>> > >
>> >
>> > --
>> > CONFIDENTIALITY NOTICE
>> > NOTICE: This message is intended for the use of the individual or
>> entity to
>> > which it is addressed and may contain information that is confidential,
>> > privileged and exempt from disclosure under applicable law. If the
>> reader
>> > of this message is not the intended recipient, you are hereby notified
>> that
>> > any printing, copying, dissemination, distribution, disclosure or
>> > forwarding of this communication is strictly prohibited. If you have
>> > received this communication in error, please contact the sender
>> immediately
>> > and delete it from your system. Thank You.
>> >
>>
>
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded Client + Old Server

Posted by Chris Nauroth <cn...@hortonworks.com>.
Adding back all *-dev lists to make sure everyone is covered.

Chris Nauroth
Hortonworks
http://hortonworks.com/



On Mon, Mar 24, 2014 at 2:02 PM, Chris Nauroth <cn...@hortonworks.com>wrote:

> Thank you, everyone, for the discussion.  There is general agreement, so I
> have filed HADOOP-10423 with a patch to update the compatibility
> documentation.
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
>
> On Thu, Mar 20, 2014 at 11:24 AM, Colin McCabe <cm...@alumni.cmu.edu>wrote:
>
>> +1 for making this guarantee explicit.
>>
>> It also definitely seems like a good idea to test mixed versions in
>> bigtop.
>>
>> HDFS is not immune to "new client, old server" scenarios because the HDFS
>> client gets bundled into a lot of places.
>>
>> Colin
>> On Mar 20, 2014 10:55 AM, "Chris Nauroth" <cn...@hortonworks.com>
>> wrote:
>>
>> > Our use of protobuf helps mitigate a lot of compatibility concerns, but
>> > there still can be situations that require careful coding on our part.
>> >  When adding a new field to a protobuf message, the client might need
>> to do
>> > a null check, even if the server-side implementation in the new version
>> > always populates the field.  When adding a whole new RPC endpoint, the
>> > client might need to consider the possibility that the RPC endpoint
>> isn't
>> > there on an old server, and degrade gracefully after the RPC fails.  The
>> > original issue in MAPREDUCE-4052 concerned the script commands passed
>> in a
>> > YARN container submission, where protobuf doesn't provide any validation
>> > beyond the fact that they're strings.
>> >
>> > Forward compatibility is harder than backward compatibility, and
>> testing is
>> > a big challenge.  Our test suites in the Hadoop repo don't cover this.
>> >  Does anyone know if anything in Bigtop tries to run with mixed
>> versions?
>> >
>> > I agree that we need to make it clear in the language that upgrading
>> client
>> > alone is insufficient to get access to new server-side features,
>> including
>> > new YARN APIs.  Thanks for the suggestions, Steve.
>> >
>> > Chris Nauroth
>> > Hortonworks
>> > http://hortonworks.com/
>> >
>> >
>> >
>> > On Thu, Mar 20, 2014 at 5:53 AM, Steve Loughran <stevel@hortonworks.com
>> > >wrote:
>> >
>> > > I'm clearly supportive of this, though of course the testing costs
>> needed
>> > > to back up the assertion make it more expensive than just a statement.
>> > >
>> > > Two issues
>> > >
>> > > -we'd need to make clear that new cluster features that a client can
>> > invoke
>> > > won't be available. You can't expect snapshot or symlink support
>> running
>> > > against a -2.2.0 cluster, even if the client supports it.
>> > >
>> > > -in YARN, there are no guarantees that an app compiled against later
>> YARN
>> > > APIs will work in old clusters. Because YARN apps upload themselves to
>> > the
>> > > server, and run with their hadoop, hdfs & yarn libraries. We have to
>> do a
>> > > bit of introspection in our code already to support this situation.
>> The
>> > > compatibility doc would need to be clear on that too: "YARN apps that
>> use
>> > > new APIs (including new fields in datastructures) can expect link
>> > > exceptions"
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > On 20 March 2014 04:25, Vinayakumar B <vi...@huawei.com>
>> wrote:
>> > >
>> > > > +1, I agree with your point Chris. It depends on the client
>> application
>> > > > how they using the hdfs jars in their classpath.
>> > > >
>> > > > As implementation already supports the compatibility (through
>> > protobuf),
>> > > > No extra code changes required to support new Client + old server.
>> > > >
>> > > > I feel it will be good to explicitly mention about the
>> compatibility of
>> > > > existing APIs in both versions.
>> > > >
>> > > > Anyway this is not applicable for the new APIs in latest client and
>> > this
>> > > > is understood. We can make it explicit in the document though.
>> > > >
>> > > >
>> > > > Regards,
>> > > > Vinayakumar B
>> > > >
>> > > > -----Original Message-----
>> > > > From: Chris Nauroth [mailto:cnauroth@hortonworks.com]
>> > > > Sent: 20 March 2014 05:36
>> > > > To: common-dev@hadoop.apache.org
>> > > > Cc: mapreduce-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
>> > > > yarn-dev@hadoop.apache.org
>> > > > Subject: Re: [DISCUSS] Clarification on Compatibility Policy:
>> Upgraded
>> > > > Client + Old Server
>> > > >
>> > > > I think this kind of compatibility issue still could surface for
>> HDFS,
>> > > > particularly for custom applications (i.e. something not executed
>> via
>> > > > "hadoop jar" on a cluster node, where the client classes ought to be
>> > > > injected into the classpath automatically).  Running DistCP between
>> 2
>> > > > clusters of different versions could result in a 2.4.0 client
>> calling a
>> > > > 2.3.0 NameNode.  Someone could potentially pick up the 2.4.0 WebHDFS
>> > > > client as a dependency and try to use it to make HTTP calls to a
>> 2.3.0
>> > > HDFS
>> > > > cluster.
>> > > >
>> > > > Chris Nauroth
>> > > > Hortonworks
>> > > > http://hortonworks.com/
>> > > >
>> > > >
>> > > >
>> > > > On Wed, Mar 19, 2014 at 4:28 PM, Vinod Kumar Vavilapalli <
>> > > > vinodkv@apache.org
>> > > > > wrote:
>> > > >
>> > > > > It makes sense only for YARN today where we separated out the
>> > clients.
>> > > > > HDFS is still a monolithic jar so this compatibility issue is
>> kind of
>> > > > > invalid there.
>> > > > >
>> > > > > +vinod
>> > > > >
>> > > > > On Mar 19, 2014, at 1:59 PM, Chris Nauroth <
>> cnauroth@hortonworks.com
>> > >
>> > > > > wrote:
>> > > > >
>> > > > > > I'd like to discuss clarification of part of our compatibility
>> > > policy.
>> > > > > > Here is a link to the compatibility documentation for release
>> > 2.3.0:
>> > > > > >
>> > > > > >
>> > > > >
>> > http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common
>> > > > > /Compatibility.html#Wire_compatibility
>> > > > > >
>> > > > > > For convenience, here are the specific lines in question:
>> > > > > >
>> > > > > > Client-Server compatibility is required to allow users to
>> continue
>> > > > > > using the old clients even after upgrading the server (cluster)
>> to
>> > a
>> > > > > > later version (or vice versa). For example, a Hadoop 2.1.0
>> client
>> > > > > > talking to a Hadoop 2.3.0 cluster.
>> > > > > >
>> > > > > > Client-Server compatibility is also required to allow upgrading
>> > > > > individual
>> > > > > > components without upgrading others. For example, upgrade HDFS
>> from
>> > > > > version
>> > > > > > 2.1.0 to 2.2.0 without upgrading MapReduce.
>> > > > > >
>> > > > > > Server-Server compatibility is required to allow mixed versions
>> > > > > > within an active cluster so the cluster may be upgraded without
>> > > > > > downtime in a
>> > > > > rolling
>> > > > > > fashion.
>> > > > > >
>> > > > > > Notice that there is no specific mention of upgrading the client
>> > > > > > ahead of the server.  (There is no clause for "upgraded client +
>> > old
>> > > > > > server".) Based on my experience, this is a valid use case when
>> a
>> > > > > > user wants to
>> > > > > pick
>> > > > > > up a client-side bug fix ahead of the cluster administrator's
>> > > > > > upgrade schedule.
>> > > > > >
>> > > > > > Is it our policy to maintain client compatibility with old
>> clusters
>> > > > > within
>> > > > > > the same major release?  I think many of us have assumed that
>> the
>> > > > > > answer
>> > > > > is
>> > > > > > yes and coded our new features accordingly, but it isn't made
>> > > > > > explicit in the documentation.  Do we all agree that the answer
>> is
>> > > > > > yes, or is it possibly up for debate depending on the change in
>> > > > > > question?  In RFC 2119 lingo, is it a MUST or a SHOULD?  Either
>> > way,
>> > > > > > I'd like to update the
>> > > > > policy
>> > > > > > text to make our decision clear.  After we have consensus, I can
>> > > > > volunteer
>> > > > > > to file an issue and patch the text of the policy.
>> > > > > >
>> > > > > > This discussion started initially in MAPREDUCE-4052, which
>> involved
>> > > > > > changing our scripting syntax for MapReduce YARN container
>> > > submissions.
>> > > > >  We
>> > > > > > settled the question there by gating the syntax change behind a
>> > > > > > configuration option.  By default, it will continue using the
>> > > > > > existing syntax currently understood by the pre-2.4.0
>> NodeManager,
>> > > > > > thus preserving compatibility.  We wanted to open the policy
>> > > > > > question for wider
>> > > > > discussion
>> > > > > > though.
>> > > > > >
>> > > > > > Thanks, everyone.
>> > > > > >
>> > > > > > Chris Nauroth
>> > > > > > Hortonworks
>> > > > > > http://hortonworks.com/
>> > > > > >
>> > > > > > --
>> > > > > > CONFIDENTIALITY NOTICE
>> > > > > > NOTICE: This message is intended for the use of the individual
>> or
>> > > > > > entity
>> > > > > to
>> > > > > > which it is addressed and may contain information that is
>> > > > > > confidential, privileged and exempt from disclosure under
>> > applicable
>> > > > > > law. If the reader of this message is not the intended
>> recipient,
>> > > > > > you are hereby notified
>> > > > > that
>> > > > > > any printing, copying, dissemination, distribution, disclosure
>> or
>> > > > > > forwarding of this communication is strictly prohibited. If you
>> > have
>> > > > > > received this communication in error, please contact the sender
>> > > > > immediately
>> > > > > > and delete it from your system. Thank You.
>> > > > >
>> > > > >
>> > > > > --
>> > > > > CONFIDENTIALITY NOTICE
>> > > > > NOTICE: This message is intended for the use of the individual or
>> > > > > entity to which it is addressed and may contain information that
>> is
>> > > > > confidential, privileged and exempt from disclosure under
>> applicable
>> > > > > law. If the reader of this message is not the intended recipient,
>> you
>> > > > > are hereby notified that any printing, copying, dissemination,
>> > > > > distribution, disclosure or forwarding of this communication is
>> > > > > strictly prohibited. If you have received this communication in
>> > error,
>> > > > > please contact the sender immediately and delete it from your
>> system.
>> > > > Thank You.
>> > > > >
>> > > >
>> > > > --
>> > > > CONFIDENTIALITY NOTICE
>> > > > NOTICE: This message is intended for the use of the individual or
>> > entity
>> > > > to which it is addressed and may contain information that is
>> > > confidential,
>> > > > privileged and exempt from disclosure under applicable law. If the
>> > reader
>> > > > of this message is not the intended recipient, you are hereby
>> notified
>> > > that
>> > > > any printing, copying, dissemination, distribution, disclosure or
>> > > > forwarding of this communication is strictly prohibited. If you have
>> > > > received this communication in error, please contact the sender
>> > > immediately
>> > > > and delete it from your system. Thank You.
>> > > >
>> > >
>> > > --
>> > > CONFIDENTIALITY NOTICE
>> > > NOTICE: This message is intended for the use of the individual or
>> entity
>> > to
>> > > which it is addressed and may contain information that is
>> confidential,
>> > > privileged and exempt from disclosure under applicable law. If the
>> reader
>> > > of this message is not the intended recipient, you are hereby notified
>> > that
>> > > any printing, copying, dissemination, distribution, disclosure or
>> > > forwarding of this communication is strictly prohibited. If you have
>> > > received this communication in error, please contact the sender
>> > immediately
>> > > and delete it from your system. Thank You.
>> > >
>> >
>> > --
>> > CONFIDENTIALITY NOTICE
>> > NOTICE: This message is intended for the use of the individual or
>> entity to
>> > which it is addressed and may contain information that is confidential,
>> > privileged and exempt from disclosure under applicable law. If the
>> reader
>> > of this message is not the intended recipient, you are hereby notified
>> that
>> > any printing, copying, dissemination, distribution, disclosure or
>> > forwarding of this communication is strictly prohibited. If you have
>> > received this communication in error, please contact the sender
>> immediately
>> > and delete it from your system. Thank You.
>> >
>>
>
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded Client + Old Server

Posted by Chris Nauroth <cn...@hortonworks.com>.
Thank you, everyone, for the discussion.  There is general agreement, so I
have filed HADOOP-10423 with a patch to update the compatibility
documentation.

Chris Nauroth
Hortonworks
http://hortonworks.com/



On Thu, Mar 20, 2014 at 11:24 AM, Colin McCabe <cm...@alumni.cmu.edu>wrote:

> +1 for making this guarantee explicit.
>
> It also definitely seems like a good idea to test mixed versions in bigtop.
>
> HDFS is not immune to "new client, old server" scenarios because the HDFS
> client gets bundled into a lot of places.
>
> Colin
> On Mar 20, 2014 10:55 AM, "Chris Nauroth" <cn...@hortonworks.com>
> wrote:
>
> > Our use of protobuf helps mitigate a lot of compatibility concerns, but
> > there still can be situations that require careful coding on our part.
> >  When adding a new field to a protobuf message, the client might need to
> do
> > a null check, even if the server-side implementation in the new version
> > always populates the field.  When adding a whole new RPC endpoint, the
> > client might need to consider the possibility that the RPC endpoint isn't
> > there on an old server, and degrade gracefully after the RPC fails.  The
> > original issue in MAPREDUCE-4052 concerned the script commands passed in
> a
> > YARN container submission, where protobuf doesn't provide any validation
> > beyond the fact that they're strings.
> >
> > Forward compatibility is harder than backward compatibility, and testing
> is
> > a big challenge.  Our test suites in the Hadoop repo don't cover this.
> >  Does anyone know if anything in Bigtop tries to run with mixed versions?
> >
> > I agree that we need to make it clear in the language that upgrading
> client
> > alone is insufficient to get access to new server-side features,
> including
> > new YARN APIs.  Thanks for the suggestions, Steve.
> >
> > Chris Nauroth
> > Hortonworks
> > http://hortonworks.com/
> >
> >
> >
> > On Thu, Mar 20, 2014 at 5:53 AM, Steve Loughran <stevel@hortonworks.com
> > >wrote:
> >
> > > I'm clearly supportive of this, though of course the testing costs
> needed
> > > to back up the assertion make it more expensive than just a statement.
> > >
> > > Two issues
> > >
> > > -we'd need to make clear that new cluster features that a client can
> > invoke
> > > won't be available. You can't expect snapshot or symlink support
> running
> > > against a -2.2.0 cluster, even if the client supports it.
> > >
> > > -in YARN, there are no guarantees that an app compiled against later
> YARN
> > > APIs will work in old clusters. Because YARN apps upload themselves to
> > the
> > > server, and run with their hadoop, hdfs & yarn libraries. We have to
> do a
> > > bit of introspection in our code already to support this situation. The
> > > compatibility doc would need to be clear on that too: "YARN apps that
> use
> > > new APIs (including new fields in datastructures) can expect link
> > > exceptions"
> > >
> > >
> > >
> > >
> > >
> > > On 20 March 2014 04:25, Vinayakumar B <vi...@huawei.com>
> wrote:
> > >
> > > > +1, I agree with your point Chris. It depends on the client
> application
> > > > how they using the hdfs jars in their classpath.
> > > >
> > > > As implementation already supports the compatibility (through
> > protobuf),
> > > > No extra code changes required to support new Client + old server.
> > > >
> > > > I feel it will be good to explicitly mention about the compatibility
> of
> > > > existing APIs in both versions.
> > > >
> > > > Anyway this is not applicable for the new APIs in latest client and
> > this
> > > > is understood. We can make it explicit in the document though.
> > > >
> > > >
> > > > Regards,
> > > > Vinayakumar B
> > > >
> > > > -----Original Message-----
> > > > From: Chris Nauroth [mailto:cnauroth@hortonworks.com]
> > > > Sent: 20 March 2014 05:36
> > > > To: common-dev@hadoop.apache.org
> > > > Cc: mapreduce-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
> > > > yarn-dev@hadoop.apache.org
> > > > Subject: Re: [DISCUSS] Clarification on Compatibility Policy:
> Upgraded
> > > > Client + Old Server
> > > >
> > > > I think this kind of compatibility issue still could surface for
> HDFS,
> > > > particularly for custom applications (i.e. something not executed via
> > > > "hadoop jar" on a cluster node, where the client classes ought to be
> > > > injected into the classpath automatically).  Running DistCP between 2
> > > > clusters of different versions could result in a 2.4.0 client
> calling a
> > > > 2.3.0 NameNode.  Someone could potentially pick up the 2.4.0 WebHDFS
> > > > client as a dependency and try to use it to make HTTP calls to a
> 2.3.0
> > > HDFS
> > > > cluster.
> > > >
> > > > Chris Nauroth
> > > > Hortonworks
> > > > http://hortonworks.com/
> > > >
> > > >
> > > >
> > > > On Wed, Mar 19, 2014 at 4:28 PM, Vinod Kumar Vavilapalli <
> > > > vinodkv@apache.org
> > > > > wrote:
> > > >
> > > > > It makes sense only for YARN today where we separated out the
> > clients.
> > > > > HDFS is still a monolithic jar so this compatibility issue is kind
> of
> > > > > invalid there.
> > > > >
> > > > > +vinod
> > > > >
> > > > > On Mar 19, 2014, at 1:59 PM, Chris Nauroth <
> cnauroth@hortonworks.com
> > >
> > > > > wrote:
> > > > >
> > > > > > I'd like to discuss clarification of part of our compatibility
> > > policy.
> > > > > > Here is a link to the compatibility documentation for release
> > 2.3.0:
> > > > > >
> > > > > >
> > > > >
> > http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common
> > > > > /Compatibility.html#Wire_compatibility
> > > > > >
> > > > > > For convenience, here are the specific lines in question:
> > > > > >
> > > > > > Client-Server compatibility is required to allow users to
> continue
> > > > > > using the old clients even after upgrading the server (cluster)
> to
> > a
> > > > > > later version (or vice versa). For example, a Hadoop 2.1.0 client
> > > > > > talking to a Hadoop 2.3.0 cluster.
> > > > > >
> > > > > > Client-Server compatibility is also required to allow upgrading
> > > > > individual
> > > > > > components without upgrading others. For example, upgrade HDFS
> from
> > > > > version
> > > > > > 2.1.0 to 2.2.0 without upgrading MapReduce.
> > > > > >
> > > > > > Server-Server compatibility is required to allow mixed versions
> > > > > > within an active cluster so the cluster may be upgraded without
> > > > > > downtime in a
> > > > > rolling
> > > > > > fashion.
> > > > > >
> > > > > > Notice that there is no specific mention of upgrading the client
> > > > > > ahead of the server.  (There is no clause for "upgraded client +
> > old
> > > > > > server".) Based on my experience, this is a valid use case when a
> > > > > > user wants to
> > > > > pick
> > > > > > up a client-side bug fix ahead of the cluster administrator's
> > > > > > upgrade schedule.
> > > > > >
> > > > > > Is it our policy to maintain client compatibility with old
> clusters
> > > > > within
> > > > > > the same major release?  I think many of us have assumed that the
> > > > > > answer
> > > > > is
> > > > > > yes and coded our new features accordingly, but it isn't made
> > > > > > explicit in the documentation.  Do we all agree that the answer
> is
> > > > > > yes, or is it possibly up for debate depending on the change in
> > > > > > question?  In RFC 2119 lingo, is it a MUST or a SHOULD?  Either
> > way,
> > > > > > I'd like to update the
> > > > > policy
> > > > > > text to make our decision clear.  After we have consensus, I can
> > > > > volunteer
> > > > > > to file an issue and patch the text of the policy.
> > > > > >
> > > > > > This discussion started initially in MAPREDUCE-4052, which
> involved
> > > > > > changing our scripting syntax for MapReduce YARN container
> > > submissions.
> > > > >  We
> > > > > > settled the question there by gating the syntax change behind a
> > > > > > configuration option.  By default, it will continue using the
> > > > > > existing syntax currently understood by the pre-2.4.0
> NodeManager,
> > > > > > thus preserving compatibility.  We wanted to open the policy
> > > > > > question for wider
> > > > > discussion
> > > > > > though.
> > > > > >
> > > > > > Thanks, everyone.
> > > > > >
> > > > > > Chris Nauroth
> > > > > > Hortonworks
> > > > > > http://hortonworks.com/
> > > > > >
> > > > > > --
> > > > > > CONFIDENTIALITY NOTICE
> > > > > > NOTICE: This message is intended for the use of the individual or
> > > > > > entity
> > > > > to
> > > > > > which it is addressed and may contain information that is
> > > > > > confidential, privileged and exempt from disclosure under
> > applicable
> > > > > > law. If the reader of this message is not the intended recipient,
> > > > > > you are hereby notified
> > > > > that
> > > > > > any printing, copying, dissemination, distribution, disclosure or
> > > > > > forwarding of this communication is strictly prohibited. If you
> > have
> > > > > > received this communication in error, please contact the sender
> > > > > immediately
> > > > > > and delete it from your system. Thank You.
> > > > >
> > > > >
> > > > > --
> > > > > CONFIDENTIALITY NOTICE
> > > > > NOTICE: This message is intended for the use of the individual or
> > > > > entity to which it is addressed and may contain information that is
> > > > > confidential, privileged and exempt from disclosure under
> applicable
> > > > > law. If the reader of this message is not the intended recipient,
> you
> > > > > are hereby notified that any printing, copying, dissemination,
> > > > > distribution, disclosure or forwarding of this communication is
> > > > > strictly prohibited. If you have received this communication in
> > error,
> > > > > please contact the sender immediately and delete it from your
> system.
> > > > Thank You.
> > > > >
> > > >
> > > > --
> > > > CONFIDENTIALITY NOTICE
> > > > NOTICE: This message is intended for the use of the individual or
> > entity
> > > > to which it is addressed and may contain information that is
> > > confidential,
> > > > privileged and exempt from disclosure under applicable law. If the
> > reader
> > > > of this message is not the intended recipient, you are hereby
> notified
> > > that
> > > > any printing, copying, dissemination, distribution, disclosure or
> > > > forwarding of this communication is strictly prohibited. If you have
> > > > received this communication in error, please contact the sender
> > > immediately
> > > > and delete it from your system. Thank You.
> > > >
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded Client + Old Server

Posted by Colin McCabe <cm...@alumni.cmu.edu>.
+1 for making this guarantee explicit.

It also definitely seems like a good idea to test mixed versions in bigtop.

HDFS is not immune to "new client, old server" scenarios because the HDFS
client gets bundled into a lot of places.

Colin
On Mar 20, 2014 10:55 AM, "Chris Nauroth" <cn...@hortonworks.com> wrote:

> Our use of protobuf helps mitigate a lot of compatibility concerns, but
> there still can be situations that require careful coding on our part.
>  When adding a new field to a protobuf message, the client might need to do
> a null check, even if the server-side implementation in the new version
> always populates the field.  When adding a whole new RPC endpoint, the
> client might need to consider the possibility that the RPC endpoint isn't
> there on an old server, and degrade gracefully after the RPC fails.  The
> original issue in MAPREDUCE-4052 concerned the script commands passed in a
> YARN container submission, where protobuf doesn't provide any validation
> beyond the fact that they're strings.
>
> Forward compatibility is harder than backward compatibility, and testing is
> a big challenge.  Our test suites in the Hadoop repo don't cover this.
>  Does anyone know if anything in Bigtop tries to run with mixed versions?
>
> I agree that we need to make it clear in the language that upgrading client
> alone is insufficient to get access to new server-side features, including
> new YARN APIs.  Thanks for the suggestions, Steve.
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
>
> On Thu, Mar 20, 2014 at 5:53 AM, Steve Loughran <stevel@hortonworks.com
> >wrote:
>
> > I'm clearly supportive of this, though of course the testing costs needed
> > to back up the assertion make it more expensive than just a statement.
> >
> > Two issues
> >
> > -we'd need to make clear that new cluster features that a client can
> invoke
> > won't be available. You can't expect snapshot or symlink support running
> > against a -2.2.0 cluster, even if the client supports it.
> >
> > -in YARN, there are no guarantees that an app compiled against later YARN
> > APIs will work in old clusters. Because YARN apps upload themselves to
> the
> > server, and run with their hadoop, hdfs & yarn libraries. We have to do a
> > bit of introspection in our code already to support this situation. The
> > compatibility doc would need to be clear on that too: "YARN apps that use
> > new APIs (including new fields in datastructures) can expect link
> > exceptions"
> >
> >
> >
> >
> >
> > On 20 March 2014 04:25, Vinayakumar B <vi...@huawei.com> wrote:
> >
> > > +1, I agree with your point Chris. It depends on the client application
> > > how they using the hdfs jars in their classpath.
> > >
> > > As implementation already supports the compatibility (through
> protobuf),
> > > No extra code changes required to support new Client + old server.
> > >
> > > I feel it will be good to explicitly mention about the compatibility of
> > > existing APIs in both versions.
> > >
> > > Anyway this is not applicable for the new APIs in latest client and
> this
> > > is understood. We can make it explicit in the document though.
> > >
> > >
> > > Regards,
> > > Vinayakumar B
> > >
> > > -----Original Message-----
> > > From: Chris Nauroth [mailto:cnauroth@hortonworks.com]
> > > Sent: 20 March 2014 05:36
> > > To: common-dev@hadoop.apache.org
> > > Cc: mapreduce-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
> > > yarn-dev@hadoop.apache.org
> > > Subject: Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded
> > > Client + Old Server
> > >
> > > I think this kind of compatibility issue still could surface for HDFS,
> > > particularly for custom applications (i.e. something not executed via
> > > "hadoop jar" on a cluster node, where the client classes ought to be
> > > injected into the classpath automatically).  Running DistCP between 2
> > > clusters of different versions could result in a 2.4.0 client calling a
> > > 2.3.0 NameNode.  Someone could potentially pick up the 2.4.0 WebHDFS
> > > client as a dependency and try to use it to make HTTP calls to a 2.3.0
> > HDFS
> > > cluster.
> > >
> > > Chris Nauroth
> > > Hortonworks
> > > http://hortonworks.com/
> > >
> > >
> > >
> > > On Wed, Mar 19, 2014 at 4:28 PM, Vinod Kumar Vavilapalli <
> > > vinodkv@apache.org
> > > > wrote:
> > >
> > > > It makes sense only for YARN today where we separated out the
> clients.
> > > > HDFS is still a monolithic jar so this compatibility issue is kind of
> > > > invalid there.
> > > >
> > > > +vinod
> > > >
> > > > On Mar 19, 2014, at 1:59 PM, Chris Nauroth <cnauroth@hortonworks.com
> >
> > > > wrote:
> > > >
> > > > > I'd like to discuss clarification of part of our compatibility
> > policy.
> > > > > Here is a link to the compatibility documentation for release
> 2.3.0:
> > > > >
> > > > >
> > > >
> http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common
> > > > /Compatibility.html#Wire_compatibility
> > > > >
> > > > > For convenience, here are the specific lines in question:
> > > > >
> > > > > Client-Server compatibility is required to allow users to continue
> > > > > using the old clients even after upgrading the server (cluster) to
> a
> > > > > later version (or vice versa). For example, a Hadoop 2.1.0 client
> > > > > talking to a Hadoop 2.3.0 cluster.
> > > > >
> > > > > Client-Server compatibility is also required to allow upgrading
> > > > individual
> > > > > components without upgrading others. For example, upgrade HDFS from
> > > > version
> > > > > 2.1.0 to 2.2.0 without upgrading MapReduce.
> > > > >
> > > > > Server-Server compatibility is required to allow mixed versions
> > > > > within an active cluster so the cluster may be upgraded without
> > > > > downtime in a
> > > > rolling
> > > > > fashion.
> > > > >
> > > > > Notice that there is no specific mention of upgrading the client
> > > > > ahead of the server.  (There is no clause for "upgraded client +
> old
> > > > > server".) Based on my experience, this is a valid use case when a
> > > > > user wants to
> > > > pick
> > > > > up a client-side bug fix ahead of the cluster administrator's
> > > > > upgrade schedule.
> > > > >
> > > > > Is it our policy to maintain client compatibility with old clusters
> > > > within
> > > > > the same major release?  I think many of us have assumed that the
> > > > > answer
> > > > is
> > > > > yes and coded our new features accordingly, but it isn't made
> > > > > explicit in the documentation.  Do we all agree that the answer is
> > > > > yes, or is it possibly up for debate depending on the change in
> > > > > question?  In RFC 2119 lingo, is it a MUST or a SHOULD?  Either
> way,
> > > > > I'd like to update the
> > > > policy
> > > > > text to make our decision clear.  After we have consensus, I can
> > > > volunteer
> > > > > to file an issue and patch the text of the policy.
> > > > >
> > > > > This discussion started initially in MAPREDUCE-4052, which involved
> > > > > changing our scripting syntax for MapReduce YARN container
> > submissions.
> > > >  We
> > > > > settled the question there by gating the syntax change behind a
> > > > > configuration option.  By default, it will continue using the
> > > > > existing syntax currently understood by the pre-2.4.0 NodeManager,
> > > > > thus preserving compatibility.  We wanted to open the policy
> > > > > question for wider
> > > > discussion
> > > > > though.
> > > > >
> > > > > Thanks, everyone.
> > > > >
> > > > > Chris Nauroth
> > > > > Hortonworks
> > > > > http://hortonworks.com/
> > > > >
> > > > > --
> > > > > CONFIDENTIALITY NOTICE
> > > > > NOTICE: This message is intended for the use of the individual or
> > > > > entity
> > > > to
> > > > > which it is addressed and may contain information that is
> > > > > confidential, privileged and exempt from disclosure under
> applicable
> > > > > law. If the reader of this message is not the intended recipient,
> > > > > you are hereby notified
> > > > that
> > > > > any printing, copying, dissemination, distribution, disclosure or
> > > > > forwarding of this communication is strictly prohibited. If you
> have
> > > > > received this communication in error, please contact the sender
> > > > immediately
> > > > > and delete it from your system. Thank You.
> > > >
> > > >
> > > > --
> > > > CONFIDENTIALITY NOTICE
> > > > NOTICE: This message is intended for the use of the individual or
> > > > entity to which it is addressed and may contain information that is
> > > > confidential, privileged and exempt from disclosure under applicable
> > > > law. If the reader of this message is not the intended recipient, you
> > > > are hereby notified that any printing, copying, dissemination,
> > > > distribution, disclosure or forwarding of this communication is
> > > > strictly prohibited. If you have received this communication in
> error,
> > > > please contact the sender immediately and delete it from your system.
> > > Thank You.
> > > >
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > > to which it is addressed and may contain information that is
> > confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded Client + Old Server

Posted by Chris Nauroth <cn...@hortonworks.com>.
Our use of protobuf helps mitigate a lot of compatibility concerns, but
there still can be situations that require careful coding on our part.
 When adding a new field to a protobuf message, the client might need to do
a null check, even if the server-side implementation in the new version
always populates the field.  When adding a whole new RPC endpoint, the
client might need to consider the possibility that the RPC endpoint isn't
there on an old server, and degrade gracefully after the RPC fails.  The
original issue in MAPREDUCE-4052 concerned the script commands passed in a
YARN container submission, where protobuf doesn't provide any validation
beyond the fact that they're strings.

Forward compatibility is harder than backward compatibility, and testing is
a big challenge.  Our test suites in the Hadoop repo don't cover this.
 Does anyone know if anything in Bigtop tries to run with mixed versions?

I agree that we need to make it clear in the language that upgrading client
alone is insufficient to get access to new server-side features, including
new YARN APIs.  Thanks for the suggestions, Steve.

Chris Nauroth
Hortonworks
http://hortonworks.com/



On Thu, Mar 20, 2014 at 5:53 AM, Steve Loughran <st...@hortonworks.com>wrote:

> I'm clearly supportive of this, though of course the testing costs needed
> to back up the assertion make it more expensive than just a statement.
>
> Two issues
>
> -we'd need to make clear that new cluster features that a client can invoke
> won't be available. You can't expect snapshot or symlink support running
> against a -2.2.0 cluster, even if the client supports it.
>
> -in YARN, there are no guarantees that an app compiled against later YARN
> APIs will work in old clusters. Because YARN apps upload themselves to the
> server, and run with their hadoop, hdfs & yarn libraries. We have to do a
> bit of introspection in our code already to support this situation. The
> compatibility doc would need to be clear on that too: "YARN apps that use
> new APIs (including new fields in datastructures) can expect link
> exceptions"
>
>
>
>
>
> On 20 March 2014 04:25, Vinayakumar B <vi...@huawei.com> wrote:
>
> > +1, I agree with your point Chris. It depends on the client application
> > how they using the hdfs jars in their classpath.
> >
> > As implementation already supports the compatibility (through protobuf),
> > No extra code changes required to support new Client + old server.
> >
> > I feel it will be good to explicitly mention about the compatibility of
> > existing APIs in both versions.
> >
> > Anyway this is not applicable for the new APIs in latest client and this
> > is understood. We can make it explicit in the document though.
> >
> >
> > Regards,
> > Vinayakumar B
> >
> > -----Original Message-----
> > From: Chris Nauroth [mailto:cnauroth@hortonworks.com]
> > Sent: 20 March 2014 05:36
> > To: common-dev@hadoop.apache.org
> > Cc: mapreduce-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
> > yarn-dev@hadoop.apache.org
> > Subject: Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded
> > Client + Old Server
> >
> > I think this kind of compatibility issue still could surface for HDFS,
> > particularly for custom applications (i.e. something not executed via
> > "hadoop jar" on a cluster node, where the client classes ought to be
> > injected into the classpath automatically).  Running DistCP between 2
> > clusters of different versions could result in a 2.4.0 client calling a
> > 2.3.0 NameNode.  Someone could potentially pick up the 2.4.0 WebHDFS
> > client as a dependency and try to use it to make HTTP calls to a 2.3.0
> HDFS
> > cluster.
> >
> > Chris Nauroth
> > Hortonworks
> > http://hortonworks.com/
> >
> >
> >
> > On Wed, Mar 19, 2014 at 4:28 PM, Vinod Kumar Vavilapalli <
> > vinodkv@apache.org
> > > wrote:
> >
> > > It makes sense only for YARN today where we separated out the clients.
> > > HDFS is still a monolithic jar so this compatibility issue is kind of
> > > invalid there.
> > >
> > > +vinod
> > >
> > > On Mar 19, 2014, at 1:59 PM, Chris Nauroth <cn...@hortonworks.com>
> > > wrote:
> > >
> > > > I'd like to discuss clarification of part of our compatibility
> policy.
> > > > Here is a link to the compatibility documentation for release 2.3.0:
> > > >
> > > >
> > > http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common
> > > /Compatibility.html#Wire_compatibility
> > > >
> > > > For convenience, here are the specific lines in question:
> > > >
> > > > Client-Server compatibility is required to allow users to continue
> > > > using the old clients even after upgrading the server (cluster) to a
> > > > later version (or vice versa). For example, a Hadoop 2.1.0 client
> > > > talking to a Hadoop 2.3.0 cluster.
> > > >
> > > > Client-Server compatibility is also required to allow upgrading
> > > individual
> > > > components without upgrading others. For example, upgrade HDFS from
> > > version
> > > > 2.1.0 to 2.2.0 without upgrading MapReduce.
> > > >
> > > > Server-Server compatibility is required to allow mixed versions
> > > > within an active cluster so the cluster may be upgraded without
> > > > downtime in a
> > > rolling
> > > > fashion.
> > > >
> > > > Notice that there is no specific mention of upgrading the client
> > > > ahead of the server.  (There is no clause for "upgraded client + old
> > > > server".) Based on my experience, this is a valid use case when a
> > > > user wants to
> > > pick
> > > > up a client-side bug fix ahead of the cluster administrator's
> > > > upgrade schedule.
> > > >
> > > > Is it our policy to maintain client compatibility with old clusters
> > > within
> > > > the same major release?  I think many of us have assumed that the
> > > > answer
> > > is
> > > > yes and coded our new features accordingly, but it isn't made
> > > > explicit in the documentation.  Do we all agree that the answer is
> > > > yes, or is it possibly up for debate depending on the change in
> > > > question?  In RFC 2119 lingo, is it a MUST or a SHOULD?  Either way,
> > > > I'd like to update the
> > > policy
> > > > text to make our decision clear.  After we have consensus, I can
> > > volunteer
> > > > to file an issue and patch the text of the policy.
> > > >
> > > > This discussion started initially in MAPREDUCE-4052, which involved
> > > > changing our scripting syntax for MapReduce YARN container
> submissions.
> > >  We
> > > > settled the question there by gating the syntax change behind a
> > > > configuration option.  By default, it will continue using the
> > > > existing syntax currently understood by the pre-2.4.0 NodeManager,
> > > > thus preserving compatibility.  We wanted to open the policy
> > > > question for wider
> > > discussion
> > > > though.
> > > >
> > > > Thanks, everyone.
> > > >
> > > > Chris Nauroth
> > > > Hortonworks
> > > > http://hortonworks.com/
> > > >
> > > > --
> > > > CONFIDENTIALITY NOTICE
> > > > NOTICE: This message is intended for the use of the individual or
> > > > entity
> > > to
> > > > which it is addressed and may contain information that is
> > > > confidential, privileged and exempt from disclosure under applicable
> > > > law. If the reader of this message is not the intended recipient,
> > > > you are hereby notified
> > > that
> > > > any printing, copying, dissemination, distribution, disclosure or
> > > > forwarding of this communication is strictly prohibited. If you have
> > > > received this communication in error, please contact the sender
> > > immediately
> > > > and delete it from your system. Thank You.
> > >
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> > > entity to which it is addressed and may contain information that is
> > > confidential, privileged and exempt from disclosure under applicable
> > > law. If the reader of this message is not the intended recipient, you
> > > are hereby notified that any printing, copying, dissemination,
> > > distribution, disclosure or forwarding of this communication is
> > > strictly prohibited. If you have received this communication in error,
> > > please contact the sender immediately and delete it from your system.
> > Thank You.
> > >
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> > to which it is addressed and may contain information that is
> confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded Client + Old Server

Posted by Chris Nauroth <cn...@hortonworks.com>.
Our use of protobuf helps mitigate a lot of compatibility concerns, but
there still can be situations that require careful coding on our part.
 When adding a new field to a protobuf message, the client might need to do
a null check, even if the server-side implementation in the new version
always populates the field.  When adding a whole new RPC endpoint, the
client might need to consider the possibility that the RPC endpoint isn't
there on an old server, and degrade gracefully after the RPC fails.  The
original issue in MAPREDUCE-4052 concerned the script commands passed in a
YARN container submission, where protobuf doesn't provide any validation
beyond the fact that they're strings.

Forward compatibility is harder than backward compatibility, and testing is
a big challenge.  Our test suites in the Hadoop repo don't cover this.
 Does anyone know if anything in Bigtop tries to run with mixed versions?

I agree that we need to make it clear in the language that upgrading client
alone is insufficient to get access to new server-side features, including
new YARN APIs.  Thanks for the suggestions, Steve.

Chris Nauroth
Hortonworks
http://hortonworks.com/



On Thu, Mar 20, 2014 at 5:53 AM, Steve Loughran <st...@hortonworks.com>wrote:

> I'm clearly supportive of this, though of course the testing costs needed
> to back up the assertion make it more expensive than just a statement.
>
> Two issues
>
> -we'd need to make clear that new cluster features that a client can invoke
> won't be available. You can't expect snapshot or symlink support running
> against a -2.2.0 cluster, even if the client supports it.
>
> -in YARN, there are no guarantees that an app compiled against later YARN
> APIs will work in old clusters. Because YARN apps upload themselves to the
> server, and run with their hadoop, hdfs & yarn libraries. We have to do a
> bit of introspection in our code already to support this situation. The
> compatibility doc would need to be clear on that too: "YARN apps that use
> new APIs (including new fields in datastructures) can expect link
> exceptions"
>
>
>
>
>
> On 20 March 2014 04:25, Vinayakumar B <vi...@huawei.com> wrote:
>
> > +1, I agree with your point Chris. It depends on the client application
> > how they using the hdfs jars in their classpath.
> >
> > As implementation already supports the compatibility (through protobuf),
> > No extra code changes required to support new Client + old server.
> >
> > I feel it will be good to explicitly mention about the compatibility of
> > existing APIs in both versions.
> >
> > Anyway this is not applicable for the new APIs in latest client and this
> > is understood. We can make it explicit in the document though.
> >
> >
> > Regards,
> > Vinayakumar B
> >
> > -----Original Message-----
> > From: Chris Nauroth [mailto:cnauroth@hortonworks.com]
> > Sent: 20 March 2014 05:36
> > To: common-dev@hadoop.apache.org
> > Cc: mapreduce-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
> > yarn-dev@hadoop.apache.org
> > Subject: Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded
> > Client + Old Server
> >
> > I think this kind of compatibility issue still could surface for HDFS,
> > particularly for custom applications (i.e. something not executed via
> > "hadoop jar" on a cluster node, where the client classes ought to be
> > injected into the classpath automatically).  Running DistCP between 2
> > clusters of different versions could result in a 2.4.0 client calling a
> > 2.3.0 NameNode.  Someone could potentially pick up the 2.4.0 WebHDFS
> > client as a dependency and try to use it to make HTTP calls to a 2.3.0
> HDFS
> > cluster.
> >
> > Chris Nauroth
> > Hortonworks
> > http://hortonworks.com/
> >
> >
> >
> > On Wed, Mar 19, 2014 at 4:28 PM, Vinod Kumar Vavilapalli <
> > vinodkv@apache.org
> > > wrote:
> >
> > > It makes sense only for YARN today where we separated out the clients.
> > > HDFS is still a monolithic jar so this compatibility issue is kind of
> > > invalid there.
> > >
> > > +vinod
> > >
> > > On Mar 19, 2014, at 1:59 PM, Chris Nauroth <cn...@hortonworks.com>
> > > wrote:
> > >
> > > > I'd like to discuss clarification of part of our compatibility
> policy.
> > > > Here is a link to the compatibility documentation for release 2.3.0:
> > > >
> > > >
> > > http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common
> > > /Compatibility.html#Wire_compatibility
> > > >
> > > > For convenience, here are the specific lines in question:
> > > >
> > > > Client-Server compatibility is required to allow users to continue
> > > > using the old clients even after upgrading the server (cluster) to a
> > > > later version (or vice versa). For example, a Hadoop 2.1.0 client
> > > > talking to a Hadoop 2.3.0 cluster.
> > > >
> > > > Client-Server compatibility is also required to allow upgrading
> > > individual
> > > > components without upgrading others. For example, upgrade HDFS from
> > > version
> > > > 2.1.0 to 2.2.0 without upgrading MapReduce.
> > > >
> > > > Server-Server compatibility is required to allow mixed versions
> > > > within an active cluster so the cluster may be upgraded without
> > > > downtime in a
> > > rolling
> > > > fashion.
> > > >
> > > > Notice that there is no specific mention of upgrading the client
> > > > ahead of the server.  (There is no clause for "upgraded client + old
> > > > server".) Based on my experience, this is a valid use case when a
> > > > user wants to
> > > pick
> > > > up a client-side bug fix ahead of the cluster administrator's
> > > > upgrade schedule.
> > > >
> > > > Is it our policy to maintain client compatibility with old clusters
> > > within
> > > > the same major release?  I think many of us have assumed that the
> > > > answer
> > > is
> > > > yes and coded our new features accordingly, but it isn't made
> > > > explicit in the documentation.  Do we all agree that the answer is
> > > > yes, or is it possibly up for debate depending on the change in
> > > > question?  In RFC 2119 lingo, is it a MUST or a SHOULD?  Either way,
> > > > I'd like to update the
> > > policy
> > > > text to make our decision clear.  After we have consensus, I can
> > > volunteer
> > > > to file an issue and patch the text of the policy.
> > > >
> > > > This discussion started initially in MAPREDUCE-4052, which involved
> > > > changing our scripting syntax for MapReduce YARN container
> submissions.
> > >  We
> > > > settled the question there by gating the syntax change behind a
> > > > configuration option.  By default, it will continue using the
> > > > existing syntax currently understood by the pre-2.4.0 NodeManager,
> > > > thus preserving compatibility.  We wanted to open the policy
> > > > question for wider
> > > discussion
> > > > though.
> > > >
> > > > Thanks, everyone.
> > > >
> > > > Chris Nauroth
> > > > Hortonworks
> > > > http://hortonworks.com/
> > > >
> > > > --
> > > > CONFIDENTIALITY NOTICE
> > > > NOTICE: This message is intended for the use of the individual or
> > > > entity
> > > to
> > > > which it is addressed and may contain information that is
> > > > confidential, privileged and exempt from disclosure under applicable
> > > > law. If the reader of this message is not the intended recipient,
> > > > you are hereby notified
> > > that
> > > > any printing, copying, dissemination, distribution, disclosure or
> > > > forwarding of this communication is strictly prohibited. If you have
> > > > received this communication in error, please contact the sender
> > > immediately
> > > > and delete it from your system. Thank You.
> > >
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> > > entity to which it is addressed and may contain information that is
> > > confidential, privileged and exempt from disclosure under applicable
> > > law. If the reader of this message is not the intended recipient, you
> > > are hereby notified that any printing, copying, dissemination,
> > > distribution, disclosure or forwarding of this communication is
> > > strictly prohibited. If you have received this communication in error,
> > > please contact the sender immediately and delete it from your system.
> > Thank You.
> > >
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> > to which it is addressed and may contain information that is
> confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded Client + Old Server

Posted by Chris Nauroth <cn...@hortonworks.com>.
Our use of protobuf helps mitigate a lot of compatibility concerns, but
there still can be situations that require careful coding on our part.
 When adding a new field to a protobuf message, the client might need to do
a null check, even if the server-side implementation in the new version
always populates the field.  When adding a whole new RPC endpoint, the
client might need to consider the possibility that the RPC endpoint isn't
there on an old server, and degrade gracefully after the RPC fails.  The
original issue in MAPREDUCE-4052 concerned the script commands passed in a
YARN container submission, where protobuf doesn't provide any validation
beyond the fact that they're strings.

Forward compatibility is harder than backward compatibility, and testing is
a big challenge.  Our test suites in the Hadoop repo don't cover this.
 Does anyone know if anything in Bigtop tries to run with mixed versions?

I agree that we need to make it clear in the language that upgrading client
alone is insufficient to get access to new server-side features, including
new YARN APIs.  Thanks for the suggestions, Steve.

Chris Nauroth
Hortonworks
http://hortonworks.com/



On Thu, Mar 20, 2014 at 5:53 AM, Steve Loughran <st...@hortonworks.com>wrote:

> I'm clearly supportive of this, though of course the testing costs needed
> to back up the assertion make it more expensive than just a statement.
>
> Two issues
>
> -we'd need to make clear that new cluster features that a client can invoke
> won't be available. You can't expect snapshot or symlink support running
> against a -2.2.0 cluster, even if the client supports it.
>
> -in YARN, there are no guarantees that an app compiled against later YARN
> APIs will work in old clusters. Because YARN apps upload themselves to the
> server, and run with their hadoop, hdfs & yarn libraries. We have to do a
> bit of introspection in our code already to support this situation. The
> compatibility doc would need to be clear on that too: "YARN apps that use
> new APIs (including new fields in datastructures) can expect link
> exceptions"
>
>
>
>
>
> On 20 March 2014 04:25, Vinayakumar B <vi...@huawei.com> wrote:
>
> > +1, I agree with your point Chris. It depends on the client application
> > how they using the hdfs jars in their classpath.
> >
> > As implementation already supports the compatibility (through protobuf),
> > No extra code changes required to support new Client + old server.
> >
> > I feel it will be good to explicitly mention about the compatibility of
> > existing APIs in both versions.
> >
> > Anyway this is not applicable for the new APIs in latest client and this
> > is understood. We can make it explicit in the document though.
> >
> >
> > Regards,
> > Vinayakumar B
> >
> > -----Original Message-----
> > From: Chris Nauroth [mailto:cnauroth@hortonworks.com]
> > Sent: 20 March 2014 05:36
> > To: common-dev@hadoop.apache.org
> > Cc: mapreduce-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
> > yarn-dev@hadoop.apache.org
> > Subject: Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded
> > Client + Old Server
> >
> > I think this kind of compatibility issue still could surface for HDFS,
> > particularly for custom applications (i.e. something not executed via
> > "hadoop jar" on a cluster node, where the client classes ought to be
> > injected into the classpath automatically).  Running DistCP between 2
> > clusters of different versions could result in a 2.4.0 client calling a
> > 2.3.0 NameNode.  Someone could potentially pick up the 2.4.0 WebHDFS
> > client as a dependency and try to use it to make HTTP calls to a 2.3.0
> HDFS
> > cluster.
> >
> > Chris Nauroth
> > Hortonworks
> > http://hortonworks.com/
> >
> >
> >
> > On Wed, Mar 19, 2014 at 4:28 PM, Vinod Kumar Vavilapalli <
> > vinodkv@apache.org
> > > wrote:
> >
> > > It makes sense only for YARN today where we separated out the clients.
> > > HDFS is still a monolithic jar so this compatibility issue is kind of
> > > invalid there.
> > >
> > > +vinod
> > >
> > > On Mar 19, 2014, at 1:59 PM, Chris Nauroth <cn...@hortonworks.com>
> > > wrote:
> > >
> > > > I'd like to discuss clarification of part of our compatibility
> policy.
> > > > Here is a link to the compatibility documentation for release 2.3.0:
> > > >
> > > >
> > > http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common
> > > /Compatibility.html#Wire_compatibility
> > > >
> > > > For convenience, here are the specific lines in question:
> > > >
> > > > Client-Server compatibility is required to allow users to continue
> > > > using the old clients even after upgrading the server (cluster) to a
> > > > later version (or vice versa). For example, a Hadoop 2.1.0 client
> > > > talking to a Hadoop 2.3.0 cluster.
> > > >
> > > > Client-Server compatibility is also required to allow upgrading
> > > individual
> > > > components without upgrading others. For example, upgrade HDFS from
> > > version
> > > > 2.1.0 to 2.2.0 without upgrading MapReduce.
> > > >
> > > > Server-Server compatibility is required to allow mixed versions
> > > > within an active cluster so the cluster may be upgraded without
> > > > downtime in a
> > > rolling
> > > > fashion.
> > > >
> > > > Notice that there is no specific mention of upgrading the client
> > > > ahead of the server.  (There is no clause for "upgraded client + old
> > > > server".) Based on my experience, this is a valid use case when a
> > > > user wants to
> > > pick
> > > > up a client-side bug fix ahead of the cluster administrator's
> > > > upgrade schedule.
> > > >
> > > > Is it our policy to maintain client compatibility with old clusters
> > > within
> > > > the same major release?  I think many of us have assumed that the
> > > > answer
> > > is
> > > > yes and coded our new features accordingly, but it isn't made
> > > > explicit in the documentation.  Do we all agree that the answer is
> > > > yes, or is it possibly up for debate depending on the change in
> > > > question?  In RFC 2119 lingo, is it a MUST or a SHOULD?  Either way,
> > > > I'd like to update the
> > > policy
> > > > text to make our decision clear.  After we have consensus, I can
> > > volunteer
> > > > to file an issue and patch the text of the policy.
> > > >
> > > > This discussion started initially in MAPREDUCE-4052, which involved
> > > > changing our scripting syntax for MapReduce YARN container
> submissions.
> > >  We
> > > > settled the question there by gating the syntax change behind a
> > > > configuration option.  By default, it will continue using the
> > > > existing syntax currently understood by the pre-2.4.0 NodeManager,
> > > > thus preserving compatibility.  We wanted to open the policy
> > > > question for wider
> > > discussion
> > > > though.
> > > >
> > > > Thanks, everyone.
> > > >
> > > > Chris Nauroth
> > > > Hortonworks
> > > > http://hortonworks.com/
> > > >
> > > > --
> > > > CONFIDENTIALITY NOTICE
> > > > NOTICE: This message is intended for the use of the individual or
> > > > entity
> > > to
> > > > which it is addressed and may contain information that is
> > > > confidential, privileged and exempt from disclosure under applicable
> > > > law. If the reader of this message is not the intended recipient,
> > > > you are hereby notified
> > > that
> > > > any printing, copying, dissemination, distribution, disclosure or
> > > > forwarding of this communication is strictly prohibited. If you have
> > > > received this communication in error, please contact the sender
> > > immediately
> > > > and delete it from your system. Thank You.
> > >
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> > > entity to which it is addressed and may contain information that is
> > > confidential, privileged and exempt from disclosure under applicable
> > > law. If the reader of this message is not the intended recipient, you
> > > are hereby notified that any printing, copying, dissemination,
> > > distribution, disclosure or forwarding of this communication is
> > > strictly prohibited. If you have received this communication in error,
> > > please contact the sender immediately and delete it from your system.
> > Thank You.
> > >
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> > to which it is addressed and may contain information that is
> confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded Client + Old Server

Posted by Chris Nauroth <cn...@hortonworks.com>.
Our use of protobuf helps mitigate a lot of compatibility concerns, but
there still can be situations that require careful coding on our part.
 When adding a new field to a protobuf message, the client might need to do
a null check, even if the server-side implementation in the new version
always populates the field.  When adding a whole new RPC endpoint, the
client might need to consider the possibility that the RPC endpoint isn't
there on an old server, and degrade gracefully after the RPC fails.  The
original issue in MAPREDUCE-4052 concerned the script commands passed in a
YARN container submission, where protobuf doesn't provide any validation
beyond the fact that they're strings.

Forward compatibility is harder than backward compatibility, and testing is
a big challenge.  Our test suites in the Hadoop repo don't cover this.
 Does anyone know if anything in Bigtop tries to run with mixed versions?

I agree that we need to make it clear in the language that upgrading client
alone is insufficient to get access to new server-side features, including
new YARN APIs.  Thanks for the suggestions, Steve.

Chris Nauroth
Hortonworks
http://hortonworks.com/



On Thu, Mar 20, 2014 at 5:53 AM, Steve Loughran <st...@hortonworks.com>wrote:

> I'm clearly supportive of this, though of course the testing costs needed
> to back up the assertion make it more expensive than just a statement.
>
> Two issues
>
> -we'd need to make clear that new cluster features that a client can invoke
> won't be available. You can't expect snapshot or symlink support running
> against a -2.2.0 cluster, even if the client supports it.
>
> -in YARN, there are no guarantees that an app compiled against later YARN
> APIs will work in old clusters. Because YARN apps upload themselves to the
> server, and run with their hadoop, hdfs & yarn libraries. We have to do a
> bit of introspection in our code already to support this situation. The
> compatibility doc would need to be clear on that too: "YARN apps that use
> new APIs (including new fields in datastructures) can expect link
> exceptions"
>
>
>
>
>
> On 20 March 2014 04:25, Vinayakumar B <vi...@huawei.com> wrote:
>
> > +1, I agree with your point Chris. It depends on the client application
> > how they using the hdfs jars in their classpath.
> >
> > As implementation already supports the compatibility (through protobuf),
> > No extra code changes required to support new Client + old server.
> >
> > I feel it will be good to explicitly mention about the compatibility of
> > existing APIs in both versions.
> >
> > Anyway this is not applicable for the new APIs in latest client and this
> > is understood. We can make it explicit in the document though.
> >
> >
> > Regards,
> > Vinayakumar B
> >
> > -----Original Message-----
> > From: Chris Nauroth [mailto:cnauroth@hortonworks.com]
> > Sent: 20 March 2014 05:36
> > To: common-dev@hadoop.apache.org
> > Cc: mapreduce-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
> > yarn-dev@hadoop.apache.org
> > Subject: Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded
> > Client + Old Server
> >
> > I think this kind of compatibility issue still could surface for HDFS,
> > particularly for custom applications (i.e. something not executed via
> > "hadoop jar" on a cluster node, where the client classes ought to be
> > injected into the classpath automatically).  Running DistCP between 2
> > clusters of different versions could result in a 2.4.0 client calling a
> > 2.3.0 NameNode.  Someone could potentially pick up the 2.4.0 WebHDFS
> > client as a dependency and try to use it to make HTTP calls to a 2.3.0
> HDFS
> > cluster.
> >
> > Chris Nauroth
> > Hortonworks
> > http://hortonworks.com/
> >
> >
> >
> > On Wed, Mar 19, 2014 at 4:28 PM, Vinod Kumar Vavilapalli <
> > vinodkv@apache.org
> > > wrote:
> >
> > > It makes sense only for YARN today where we separated out the clients.
> > > HDFS is still a monolithic jar so this compatibility issue is kind of
> > > invalid there.
> > >
> > > +vinod
> > >
> > > On Mar 19, 2014, at 1:59 PM, Chris Nauroth <cn...@hortonworks.com>
> > > wrote:
> > >
> > > > I'd like to discuss clarification of part of our compatibility
> policy.
> > > > Here is a link to the compatibility documentation for release 2.3.0:
> > > >
> > > >
> > > http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common
> > > /Compatibility.html#Wire_compatibility
> > > >
> > > > For convenience, here are the specific lines in question:
> > > >
> > > > Client-Server compatibility is required to allow users to continue
> > > > using the old clients even after upgrading the server (cluster) to a
> > > > later version (or vice versa). For example, a Hadoop 2.1.0 client
> > > > talking to a Hadoop 2.3.0 cluster.
> > > >
> > > > Client-Server compatibility is also required to allow upgrading
> > > individual
> > > > components without upgrading others. For example, upgrade HDFS from
> > > version
> > > > 2.1.0 to 2.2.0 without upgrading MapReduce.
> > > >
> > > > Server-Server compatibility is required to allow mixed versions
> > > > within an active cluster so the cluster may be upgraded without
> > > > downtime in a
> > > rolling
> > > > fashion.
> > > >
> > > > Notice that there is no specific mention of upgrading the client
> > > > ahead of the server.  (There is no clause for "upgraded client + old
> > > > server".) Based on my experience, this is a valid use case when a
> > > > user wants to
> > > pick
> > > > up a client-side bug fix ahead of the cluster administrator's
> > > > upgrade schedule.
> > > >
> > > > Is it our policy to maintain client compatibility with old clusters
> > > within
> > > > the same major release?  I think many of us have assumed that the
> > > > answer
> > > is
> > > > yes and coded our new features accordingly, but it isn't made
> > > > explicit in the documentation.  Do we all agree that the answer is
> > > > yes, or is it possibly up for debate depending on the change in
> > > > question?  In RFC 2119 lingo, is it a MUST or a SHOULD?  Either way,
> > > > I'd like to update the
> > > policy
> > > > text to make our decision clear.  After we have consensus, I can
> > > volunteer
> > > > to file an issue and patch the text of the policy.
> > > >
> > > > This discussion started initially in MAPREDUCE-4052, which involved
> > > > changing our scripting syntax for MapReduce YARN container
> submissions.
> > >  We
> > > > settled the question there by gating the syntax change behind a
> > > > configuration option.  By default, it will continue using the
> > > > existing syntax currently understood by the pre-2.4.0 NodeManager,
> > > > thus preserving compatibility.  We wanted to open the policy
> > > > question for wider
> > > discussion
> > > > though.
> > > >
> > > > Thanks, everyone.
> > > >
> > > > Chris Nauroth
> > > > Hortonworks
> > > > http://hortonworks.com/
> > > >
> > > > --
> > > > CONFIDENTIALITY NOTICE
> > > > NOTICE: This message is intended for the use of the individual or
> > > > entity
> > > to
> > > > which it is addressed and may contain information that is
> > > > confidential, privileged and exempt from disclosure under applicable
> > > > law. If the reader of this message is not the intended recipient,
> > > > you are hereby notified
> > > that
> > > > any printing, copying, dissemination, distribution, disclosure or
> > > > forwarding of this communication is strictly prohibited. If you have
> > > > received this communication in error, please contact the sender
> > > immediately
> > > > and delete it from your system. Thank You.
> > >
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> > > entity to which it is addressed and may contain information that is
> > > confidential, privileged and exempt from disclosure under applicable
> > > law. If the reader of this message is not the intended recipient, you
> > > are hereby notified that any printing, copying, dissemination,
> > > distribution, disclosure or forwarding of this communication is
> > > strictly prohibited. If you have received this communication in error,
> > > please contact the sender immediately and delete it from your system.
> > Thank You.
> > >
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> > to which it is addressed and may contain information that is
> confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded Client + Old Server

Posted by Steve Loughran <st...@hortonworks.com>.
I'm clearly supportive of this, though of course the testing costs needed
to back up the assertion make it more expensive than just a statement.

Two issues

-we'd need to make clear that new cluster features that a client can invoke
won't be available. You can't expect snapshot or symlink support running
against a -2.2.0 cluster, even if the client supports it.

-in YARN, there are no guarantees that an app compiled against later YARN
APIs will work in old clusters. Because YARN apps upload themselves to the
server, and run with their hadoop, hdfs & yarn libraries. We have to do a
bit of introspection in our code already to support this situation. The
compatibility doc would need to be clear on that too: "YARN apps that use
new APIs (including new fields in datastructures) can expect link
exceptions"





On 20 March 2014 04:25, Vinayakumar B <vi...@huawei.com> wrote:

> +1, I agree with your point Chris. It depends on the client application
> how they using the hdfs jars in their classpath.
>
> As implementation already supports the compatibility (through protobuf),
> No extra code changes required to support new Client + old server.
>
> I feel it will be good to explicitly mention about the compatibility of
> existing APIs in both versions.
>
> Anyway this is not applicable for the new APIs in latest client and this
> is understood. We can make it explicit in the document though.
>
>
> Regards,
> Vinayakumar B
>
> -----Original Message-----
> From: Chris Nauroth [mailto:cnauroth@hortonworks.com]
> Sent: 20 March 2014 05:36
> To: common-dev@hadoop.apache.org
> Cc: mapreduce-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
> yarn-dev@hadoop.apache.org
> Subject: Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded
> Client + Old Server
>
> I think this kind of compatibility issue still could surface for HDFS,
> particularly for custom applications (i.e. something not executed via
> "hadoop jar" on a cluster node, where the client classes ought to be
> injected into the classpath automatically).  Running DistCP between 2
> clusters of different versions could result in a 2.4.0 client calling a
> 2.3.0 NameNode.  Someone could potentially pick up the 2.4.0 WebHDFS
> client as a dependency and try to use it to make HTTP calls to a 2.3.0 HDFS
> cluster.
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
>
> On Wed, Mar 19, 2014 at 4:28 PM, Vinod Kumar Vavilapalli <
> vinodkv@apache.org
> > wrote:
>
> > It makes sense only for YARN today where we separated out the clients.
> > HDFS is still a monolithic jar so this compatibility issue is kind of
> > invalid there.
> >
> > +vinod
> >
> > On Mar 19, 2014, at 1:59 PM, Chris Nauroth <cn...@hortonworks.com>
> > wrote:
> >
> > > I'd like to discuss clarification of part of our compatibility policy.
> > > Here is a link to the compatibility documentation for release 2.3.0:
> > >
> > >
> > http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common
> > /Compatibility.html#Wire_compatibility
> > >
> > > For convenience, here are the specific lines in question:
> > >
> > > Client-Server compatibility is required to allow users to continue
> > > using the old clients even after upgrading the server (cluster) to a
> > > later version (or vice versa). For example, a Hadoop 2.1.0 client
> > > talking to a Hadoop 2.3.0 cluster.
> > >
> > > Client-Server compatibility is also required to allow upgrading
> > individual
> > > components without upgrading others. For example, upgrade HDFS from
> > version
> > > 2.1.0 to 2.2.0 without upgrading MapReduce.
> > >
> > > Server-Server compatibility is required to allow mixed versions
> > > within an active cluster so the cluster may be upgraded without
> > > downtime in a
> > rolling
> > > fashion.
> > >
> > > Notice that there is no specific mention of upgrading the client
> > > ahead of the server.  (There is no clause for "upgraded client + old
> > > server".) Based on my experience, this is a valid use case when a
> > > user wants to
> > pick
> > > up a client-side bug fix ahead of the cluster administrator's
> > > upgrade schedule.
> > >
> > > Is it our policy to maintain client compatibility with old clusters
> > within
> > > the same major release?  I think many of us have assumed that the
> > > answer
> > is
> > > yes and coded our new features accordingly, but it isn't made
> > > explicit in the documentation.  Do we all agree that the answer is
> > > yes, or is it possibly up for debate depending on the change in
> > > question?  In RFC 2119 lingo, is it a MUST or a SHOULD?  Either way,
> > > I'd like to update the
> > policy
> > > text to make our decision clear.  After we have consensus, I can
> > volunteer
> > > to file an issue and patch the text of the policy.
> > >
> > > This discussion started initially in MAPREDUCE-4052, which involved
> > > changing our scripting syntax for MapReduce YARN container submissions.
> >  We
> > > settled the question there by gating the syntax change behind a
> > > configuration option.  By default, it will continue using the
> > > existing syntax currently understood by the pre-2.4.0 NodeManager,
> > > thus preserving compatibility.  We wanted to open the policy
> > > question for wider
> > discussion
> > > though.
> > >
> > > Thanks, everyone.
> > >
> > > Chris Nauroth
> > > Hortonworks
> > > http://hortonworks.com/
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> > > entity
> > to
> > > which it is addressed and may contain information that is
> > > confidential, privileged and exempt from disclosure under applicable
> > > law. If the reader of this message is not the intended recipient,
> > > you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> >
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or
> > entity to which it is addressed and may contain information that is
> > confidential, privileged and exempt from disclosure under applicable
> > law. If the reader of this message is not the intended recipient, you
> > are hereby notified that any printing, copying, dissemination,
> > distribution, disclosure or forwarding of this communication is
> > strictly prohibited. If you have received this communication in error,
> > please contact the sender immediately and delete it from your system.
> Thank You.
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded Client + Old Server

Posted by Steve Loughran <st...@hortonworks.com>.
I'm clearly supportive of this, though of course the testing costs needed
to back up the assertion make it more expensive than just a statement.

Two issues

-we'd need to make clear that new cluster features that a client can invoke
won't be available. You can't expect snapshot or symlink support running
against a -2.2.0 cluster, even if the client supports it.

-in YARN, there are no guarantees that an app compiled against later YARN
APIs will work in old clusters. Because YARN apps upload themselves to the
server, and run with their hadoop, hdfs & yarn libraries. We have to do a
bit of introspection in our code already to support this situation. The
compatibility doc would need to be clear on that too: "YARN apps that use
new APIs (including new fields in datastructures) can expect link
exceptions"





On 20 March 2014 04:25, Vinayakumar B <vi...@huawei.com> wrote:

> +1, I agree with your point Chris. It depends on the client application
> how they using the hdfs jars in their classpath.
>
> As implementation already supports the compatibility (through protobuf),
> No extra code changes required to support new Client + old server.
>
> I feel it will be good to explicitly mention about the compatibility of
> existing APIs in both versions.
>
> Anyway this is not applicable for the new APIs in latest client and this
> is understood. We can make it explicit in the document though.
>
>
> Regards,
> Vinayakumar B
>
> -----Original Message-----
> From: Chris Nauroth [mailto:cnauroth@hortonworks.com]
> Sent: 20 March 2014 05:36
> To: common-dev@hadoop.apache.org
> Cc: mapreduce-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
> yarn-dev@hadoop.apache.org
> Subject: Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded
> Client + Old Server
>
> I think this kind of compatibility issue still could surface for HDFS,
> particularly for custom applications (i.e. something not executed via
> "hadoop jar" on a cluster node, where the client classes ought to be
> injected into the classpath automatically).  Running DistCP between 2
> clusters of different versions could result in a 2.4.0 client calling a
> 2.3.0 NameNode.  Someone could potentially pick up the 2.4.0 WebHDFS
> client as a dependency and try to use it to make HTTP calls to a 2.3.0 HDFS
> cluster.
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
>
> On Wed, Mar 19, 2014 at 4:28 PM, Vinod Kumar Vavilapalli <
> vinodkv@apache.org
> > wrote:
>
> > It makes sense only for YARN today where we separated out the clients.
> > HDFS is still a monolithic jar so this compatibility issue is kind of
> > invalid there.
> >
> > +vinod
> >
> > On Mar 19, 2014, at 1:59 PM, Chris Nauroth <cn...@hortonworks.com>
> > wrote:
> >
> > > I'd like to discuss clarification of part of our compatibility policy.
> > > Here is a link to the compatibility documentation for release 2.3.0:
> > >
> > >
> > http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common
> > /Compatibility.html#Wire_compatibility
> > >
> > > For convenience, here are the specific lines in question:
> > >
> > > Client-Server compatibility is required to allow users to continue
> > > using the old clients even after upgrading the server (cluster) to a
> > > later version (or vice versa). For example, a Hadoop 2.1.0 client
> > > talking to a Hadoop 2.3.0 cluster.
> > >
> > > Client-Server compatibility is also required to allow upgrading
> > individual
> > > components without upgrading others. For example, upgrade HDFS from
> > version
> > > 2.1.0 to 2.2.0 without upgrading MapReduce.
> > >
> > > Server-Server compatibility is required to allow mixed versions
> > > within an active cluster so the cluster may be upgraded without
> > > downtime in a
> > rolling
> > > fashion.
> > >
> > > Notice that there is no specific mention of upgrading the client
> > > ahead of the server.  (There is no clause for "upgraded client + old
> > > server".) Based on my experience, this is a valid use case when a
> > > user wants to
> > pick
> > > up a client-side bug fix ahead of the cluster administrator's
> > > upgrade schedule.
> > >
> > > Is it our policy to maintain client compatibility with old clusters
> > within
> > > the same major release?  I think many of us have assumed that the
> > > answer
> > is
> > > yes and coded our new features accordingly, but it isn't made
> > > explicit in the documentation.  Do we all agree that the answer is
> > > yes, or is it possibly up for debate depending on the change in
> > > question?  In RFC 2119 lingo, is it a MUST or a SHOULD?  Either way,
> > > I'd like to update the
> > policy
> > > text to make our decision clear.  After we have consensus, I can
> > volunteer
> > > to file an issue and patch the text of the policy.
> > >
> > > This discussion started initially in MAPREDUCE-4052, which involved
> > > changing our scripting syntax for MapReduce YARN container submissions.
> >  We
> > > settled the question there by gating the syntax change behind a
> > > configuration option.  By default, it will continue using the
> > > existing syntax currently understood by the pre-2.4.0 NodeManager,
> > > thus preserving compatibility.  We wanted to open the policy
> > > question for wider
> > discussion
> > > though.
> > >
> > > Thanks, everyone.
> > >
> > > Chris Nauroth
> > > Hortonworks
> > > http://hortonworks.com/
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> > > entity
> > to
> > > which it is addressed and may contain information that is
> > > confidential, privileged and exempt from disclosure under applicable
> > > law. If the reader of this message is not the intended recipient,
> > > you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> >
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or
> > entity to which it is addressed and may contain information that is
> > confidential, privileged and exempt from disclosure under applicable
> > law. If the reader of this message is not the intended recipient, you
> > are hereby notified that any printing, copying, dissemination,
> > distribution, disclosure or forwarding of this communication is
> > strictly prohibited. If you have received this communication in error,
> > please contact the sender immediately and delete it from your system.
> Thank You.
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded Client + Old Server

Posted by Steve Loughran <st...@hortonworks.com>.
I'm clearly supportive of this, though of course the testing costs needed
to back up the assertion make it more expensive than just a statement.

Two issues

-we'd need to make clear that new cluster features that a client can invoke
won't be available. You can't expect snapshot or symlink support running
against a -2.2.0 cluster, even if the client supports it.

-in YARN, there are no guarantees that an app compiled against later YARN
APIs will work in old clusters. Because YARN apps upload themselves to the
server, and run with their hadoop, hdfs & yarn libraries. We have to do a
bit of introspection in our code already to support this situation. The
compatibility doc would need to be clear on that too: "YARN apps that use
new APIs (including new fields in datastructures) can expect link
exceptions"





On 20 March 2014 04:25, Vinayakumar B <vi...@huawei.com> wrote:

> +1, I agree with your point Chris. It depends on the client application
> how they using the hdfs jars in their classpath.
>
> As implementation already supports the compatibility (through protobuf),
> No extra code changes required to support new Client + old server.
>
> I feel it will be good to explicitly mention about the compatibility of
> existing APIs in both versions.
>
> Anyway this is not applicable for the new APIs in latest client and this
> is understood. We can make it explicit in the document though.
>
>
> Regards,
> Vinayakumar B
>
> -----Original Message-----
> From: Chris Nauroth [mailto:cnauroth@hortonworks.com]
> Sent: 20 March 2014 05:36
> To: common-dev@hadoop.apache.org
> Cc: mapreduce-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
> yarn-dev@hadoop.apache.org
> Subject: Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded
> Client + Old Server
>
> I think this kind of compatibility issue still could surface for HDFS,
> particularly for custom applications (i.e. something not executed via
> "hadoop jar" on a cluster node, where the client classes ought to be
> injected into the classpath automatically).  Running DistCP between 2
> clusters of different versions could result in a 2.4.0 client calling a
> 2.3.0 NameNode.  Someone could potentially pick up the 2.4.0 WebHDFS
> client as a dependency and try to use it to make HTTP calls to a 2.3.0 HDFS
> cluster.
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
>
> On Wed, Mar 19, 2014 at 4:28 PM, Vinod Kumar Vavilapalli <
> vinodkv@apache.org
> > wrote:
>
> > It makes sense only for YARN today where we separated out the clients.
> > HDFS is still a monolithic jar so this compatibility issue is kind of
> > invalid there.
> >
> > +vinod
> >
> > On Mar 19, 2014, at 1:59 PM, Chris Nauroth <cn...@hortonworks.com>
> > wrote:
> >
> > > I'd like to discuss clarification of part of our compatibility policy.
> > > Here is a link to the compatibility documentation for release 2.3.0:
> > >
> > >
> > http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common
> > /Compatibility.html#Wire_compatibility
> > >
> > > For convenience, here are the specific lines in question:
> > >
> > > Client-Server compatibility is required to allow users to continue
> > > using the old clients even after upgrading the server (cluster) to a
> > > later version (or vice versa). For example, a Hadoop 2.1.0 client
> > > talking to a Hadoop 2.3.0 cluster.
> > >
> > > Client-Server compatibility is also required to allow upgrading
> > individual
> > > components without upgrading others. For example, upgrade HDFS from
> > version
> > > 2.1.0 to 2.2.0 without upgrading MapReduce.
> > >
> > > Server-Server compatibility is required to allow mixed versions
> > > within an active cluster so the cluster may be upgraded without
> > > downtime in a
> > rolling
> > > fashion.
> > >
> > > Notice that there is no specific mention of upgrading the client
> > > ahead of the server.  (There is no clause for "upgraded client + old
> > > server".) Based on my experience, this is a valid use case when a
> > > user wants to
> > pick
> > > up a client-side bug fix ahead of the cluster administrator's
> > > upgrade schedule.
> > >
> > > Is it our policy to maintain client compatibility with old clusters
> > within
> > > the same major release?  I think many of us have assumed that the
> > > answer
> > is
> > > yes and coded our new features accordingly, but it isn't made
> > > explicit in the documentation.  Do we all agree that the answer is
> > > yes, or is it possibly up for debate depending on the change in
> > > question?  In RFC 2119 lingo, is it a MUST or a SHOULD?  Either way,
> > > I'd like to update the
> > policy
> > > text to make our decision clear.  After we have consensus, I can
> > volunteer
> > > to file an issue and patch the text of the policy.
> > >
> > > This discussion started initially in MAPREDUCE-4052, which involved
> > > changing our scripting syntax for MapReduce YARN container submissions.
> >  We
> > > settled the question there by gating the syntax change behind a
> > > configuration option.  By default, it will continue using the
> > > existing syntax currently understood by the pre-2.4.0 NodeManager,
> > > thus preserving compatibility.  We wanted to open the policy
> > > question for wider
> > discussion
> > > though.
> > >
> > > Thanks, everyone.
> > >
> > > Chris Nauroth
> > > Hortonworks
> > > http://hortonworks.com/
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> > > entity
> > to
> > > which it is addressed and may contain information that is
> > > confidential, privileged and exempt from disclosure under applicable
> > > law. If the reader of this message is not the intended recipient,
> > > you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> >
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or
> > entity to which it is addressed and may contain information that is
> > confidential, privileged and exempt from disclosure under applicable
> > law. If the reader of this message is not the intended recipient, you
> > are hereby notified that any printing, copying, dissemination,
> > distribution, disclosure or forwarding of this communication is
> > strictly prohibited. If you have received this communication in error,
> > please contact the sender immediately and delete it from your system.
> Thank You.
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded Client + Old Server

Posted by Steve Loughran <st...@hortonworks.com>.
I'm clearly supportive of this, though of course the testing costs needed
to back up the assertion make it more expensive than just a statement.

Two issues

-we'd need to make clear that new cluster features that a client can invoke
won't be available. You can't expect snapshot or symlink support running
against a -2.2.0 cluster, even if the client supports it.

-in YARN, there are no guarantees that an app compiled against later YARN
APIs will work in old clusters. Because YARN apps upload themselves to the
server, and run with their hadoop, hdfs & yarn libraries. We have to do a
bit of introspection in our code already to support this situation. The
compatibility doc would need to be clear on that too: "YARN apps that use
new APIs (including new fields in datastructures) can expect link
exceptions"





On 20 March 2014 04:25, Vinayakumar B <vi...@huawei.com> wrote:

> +1, I agree with your point Chris. It depends on the client application
> how they using the hdfs jars in their classpath.
>
> As implementation already supports the compatibility (through protobuf),
> No extra code changes required to support new Client + old server.
>
> I feel it will be good to explicitly mention about the compatibility of
> existing APIs in both versions.
>
> Anyway this is not applicable for the new APIs in latest client and this
> is understood. We can make it explicit in the document though.
>
>
> Regards,
> Vinayakumar B
>
> -----Original Message-----
> From: Chris Nauroth [mailto:cnauroth@hortonworks.com]
> Sent: 20 March 2014 05:36
> To: common-dev@hadoop.apache.org
> Cc: mapreduce-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
> yarn-dev@hadoop.apache.org
> Subject: Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded
> Client + Old Server
>
> I think this kind of compatibility issue still could surface for HDFS,
> particularly for custom applications (i.e. something not executed via
> "hadoop jar" on a cluster node, where the client classes ought to be
> injected into the classpath automatically).  Running DistCP between 2
> clusters of different versions could result in a 2.4.0 client calling a
> 2.3.0 NameNode.  Someone could potentially pick up the 2.4.0 WebHDFS
> client as a dependency and try to use it to make HTTP calls to a 2.3.0 HDFS
> cluster.
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
>
> On Wed, Mar 19, 2014 at 4:28 PM, Vinod Kumar Vavilapalli <
> vinodkv@apache.org
> > wrote:
>
> > It makes sense only for YARN today where we separated out the clients.
> > HDFS is still a monolithic jar so this compatibility issue is kind of
> > invalid there.
> >
> > +vinod
> >
> > On Mar 19, 2014, at 1:59 PM, Chris Nauroth <cn...@hortonworks.com>
> > wrote:
> >
> > > I'd like to discuss clarification of part of our compatibility policy.
> > > Here is a link to the compatibility documentation for release 2.3.0:
> > >
> > >
> > http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common
> > /Compatibility.html#Wire_compatibility
> > >
> > > For convenience, here are the specific lines in question:
> > >
> > > Client-Server compatibility is required to allow users to continue
> > > using the old clients even after upgrading the server (cluster) to a
> > > later version (or vice versa). For example, a Hadoop 2.1.0 client
> > > talking to a Hadoop 2.3.0 cluster.
> > >
> > > Client-Server compatibility is also required to allow upgrading
> > individual
> > > components without upgrading others. For example, upgrade HDFS from
> > version
> > > 2.1.0 to 2.2.0 without upgrading MapReduce.
> > >
> > > Server-Server compatibility is required to allow mixed versions
> > > within an active cluster so the cluster may be upgraded without
> > > downtime in a
> > rolling
> > > fashion.
> > >
> > > Notice that there is no specific mention of upgrading the client
> > > ahead of the server.  (There is no clause for "upgraded client + old
> > > server".) Based on my experience, this is a valid use case when a
> > > user wants to
> > pick
> > > up a client-side bug fix ahead of the cluster administrator's
> > > upgrade schedule.
> > >
> > > Is it our policy to maintain client compatibility with old clusters
> > within
> > > the same major release?  I think many of us have assumed that the
> > > answer
> > is
> > > yes and coded our new features accordingly, but it isn't made
> > > explicit in the documentation.  Do we all agree that the answer is
> > > yes, or is it possibly up for debate depending on the change in
> > > question?  In RFC 2119 lingo, is it a MUST or a SHOULD?  Either way,
> > > I'd like to update the
> > policy
> > > text to make our decision clear.  After we have consensus, I can
> > volunteer
> > > to file an issue and patch the text of the policy.
> > >
> > > This discussion started initially in MAPREDUCE-4052, which involved
> > > changing our scripting syntax for MapReduce YARN container submissions.
> >  We
> > > settled the question there by gating the syntax change behind a
> > > configuration option.  By default, it will continue using the
> > > existing syntax currently understood by the pre-2.4.0 NodeManager,
> > > thus preserving compatibility.  We wanted to open the policy
> > > question for wider
> > discussion
> > > though.
> > >
> > > Thanks, everyone.
> > >
> > > Chris Nauroth
> > > Hortonworks
> > > http://hortonworks.com/
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> > > entity
> > to
> > > which it is addressed and may contain information that is
> > > confidential, privileged and exempt from disclosure under applicable
> > > law. If the reader of this message is not the intended recipient,
> > > you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> >
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or
> > entity to which it is addressed and may contain information that is
> > confidential, privileged and exempt from disclosure under applicable
> > law. If the reader of this message is not the intended recipient, you
> > are hereby notified that any printing, copying, dissemination,
> > distribution, disclosure or forwarding of this communication is
> > strictly prohibited. If you have received this communication in error,
> > please contact the sender immediately and delete it from your system.
> Thank You.
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

RE: [DISCUSS] Clarification on Compatibility Policy: Upgraded Client + Old Server

Posted by Vinayakumar B <vi...@huawei.com>.
+1, I agree with your point Chris. It depends on the client application how they using the hdfs jars in their classpath.

As implementation already supports the compatibility (through protobuf), No extra code changes required to support new Client + old server.

I feel it will be good to explicitly mention about the compatibility of existing APIs in both versions.

Anyway this is not applicable for the new APIs in latest client and this is understood. We can make it explicit in the document though.


Regards,
Vinayakumar B

-----Original Message-----
From: Chris Nauroth [mailto:cnauroth@hortonworks.com] 
Sent: 20 March 2014 05:36
To: common-dev@hadoop.apache.org
Cc: mapreduce-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded Client + Old Server

I think this kind of compatibility issue still could surface for HDFS, particularly for custom applications (i.e. something not executed via "hadoop jar" on a cluster node, where the client classes ought to be injected into the classpath automatically).  Running DistCP between 2 clusters of different versions could result in a 2.4.0 client calling a
2.3.0 NameNode.  Someone could potentially pick up the 2.4.0 WebHDFS client as a dependency and try to use it to make HTTP calls to a 2.3.0 HDFS cluster.

Chris Nauroth
Hortonworks
http://hortonworks.com/



On Wed, Mar 19, 2014 at 4:28 PM, Vinod Kumar Vavilapalli <vinodkv@apache.org
> wrote:

> It makes sense only for YARN today where we separated out the clients.
> HDFS is still a monolithic jar so this compatibility issue is kind of 
> invalid there.
>
> +vinod
>
> On Mar 19, 2014, at 1:59 PM, Chris Nauroth <cn...@hortonworks.com>
> wrote:
>
> > I'd like to discuss clarification of part of our compatibility policy.
> > Here is a link to the compatibility documentation for release 2.3.0:
> >
> >
> http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common
> /Compatibility.html#Wire_compatibility
> >
> > For convenience, here are the specific lines in question:
> >
> > Client-Server compatibility is required to allow users to continue 
> > using the old clients even after upgrading the server (cluster) to a 
> > later version (or vice versa). For example, a Hadoop 2.1.0 client 
> > talking to a Hadoop 2.3.0 cluster.
> >
> > Client-Server compatibility is also required to allow upgrading
> individual
> > components without upgrading others. For example, upgrade HDFS from
> version
> > 2.1.0 to 2.2.0 without upgrading MapReduce.
> >
> > Server-Server compatibility is required to allow mixed versions 
> > within an active cluster so the cluster may be upgraded without 
> > downtime in a
> rolling
> > fashion.
> >
> > Notice that there is no specific mention of upgrading the client 
> > ahead of the server.  (There is no clause for "upgraded client + old 
> > server".) Based on my experience, this is a valid use case when a 
> > user wants to
> pick
> > up a client-side bug fix ahead of the cluster administrator's 
> > upgrade schedule.
> >
> > Is it our policy to maintain client compatibility with old clusters
> within
> > the same major release?  I think many of us have assumed that the 
> > answer
> is
> > yes and coded our new features accordingly, but it isn't made 
> > explicit in the documentation.  Do we all agree that the answer is 
> > yes, or is it possibly up for debate depending on the change in 
> > question?  In RFC 2119 lingo, is it a MUST or a SHOULD?  Either way, 
> > I'd like to update the
> policy
> > text to make our decision clear.  After we have consensus, I can
> volunteer
> > to file an issue and patch the text of the policy.
> >
> > This discussion started initially in MAPREDUCE-4052, which involved 
> > changing our scripting syntax for MapReduce YARN container submissions.
>  We
> > settled the question there by gating the syntax change behind a 
> > configuration option.  By default, it will continue using the 
> > existing syntax currently understood by the pre-2.4.0 NodeManager, 
> > thus preserving compatibility.  We wanted to open the policy 
> > question for wider
> discussion
> > though.
> >
> > Thanks, everyone.
> >
> > Chris Nauroth
> > Hortonworks
> > http://hortonworks.com/
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or 
> > entity
> to
> > which it is addressed and may contain information that is 
> > confidential, privileged and exempt from disclosure under applicable 
> > law. If the reader of this message is not the intended recipient, 
> > you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or 
> > forwarding of this communication is strictly prohibited. If you have 
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or 
> entity to which it is addressed and may contain information that is 
> confidential, privileged and exempt from disclosure under applicable 
> law. If the reader of this message is not the intended recipient, you 
> are hereby notified that any printing, copying, dissemination, 
> distribution, disclosure or forwarding of this communication is 
> strictly prohibited. If you have received this communication in error, 
> please contact the sender immediately and delete it from your system. Thank You.
>

--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.

RE: [DISCUSS] Clarification on Compatibility Policy: Upgraded Client + Old Server

Posted by Vinayakumar B <vi...@huawei.com>.
+1, I agree with your point Chris. It depends on the client application how they using the hdfs jars in their classpath.

As implementation already supports the compatibility (through protobuf), No extra code changes required to support new Client + old server.

I feel it will be good to explicitly mention about the compatibility of existing APIs in both versions.

Anyway this is not applicable for the new APIs in latest client and this is understood. We can make it explicit in the document though.


Regards,
Vinayakumar B

-----Original Message-----
From: Chris Nauroth [mailto:cnauroth@hortonworks.com] 
Sent: 20 March 2014 05:36
To: common-dev@hadoop.apache.org
Cc: mapreduce-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded Client + Old Server

I think this kind of compatibility issue still could surface for HDFS, particularly for custom applications (i.e. something not executed via "hadoop jar" on a cluster node, where the client classes ought to be injected into the classpath automatically).  Running DistCP between 2 clusters of different versions could result in a 2.4.0 client calling a
2.3.0 NameNode.  Someone could potentially pick up the 2.4.0 WebHDFS client as a dependency and try to use it to make HTTP calls to a 2.3.0 HDFS cluster.

Chris Nauroth
Hortonworks
http://hortonworks.com/



On Wed, Mar 19, 2014 at 4:28 PM, Vinod Kumar Vavilapalli <vinodkv@apache.org
> wrote:

> It makes sense only for YARN today where we separated out the clients.
> HDFS is still a monolithic jar so this compatibility issue is kind of 
> invalid there.
>
> +vinod
>
> On Mar 19, 2014, at 1:59 PM, Chris Nauroth <cn...@hortonworks.com>
> wrote:
>
> > I'd like to discuss clarification of part of our compatibility policy.
> > Here is a link to the compatibility documentation for release 2.3.0:
> >
> >
> http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common
> /Compatibility.html#Wire_compatibility
> >
> > For convenience, here are the specific lines in question:
> >
> > Client-Server compatibility is required to allow users to continue 
> > using the old clients even after upgrading the server (cluster) to a 
> > later version (or vice versa). For example, a Hadoop 2.1.0 client 
> > talking to a Hadoop 2.3.0 cluster.
> >
> > Client-Server compatibility is also required to allow upgrading
> individual
> > components without upgrading others. For example, upgrade HDFS from
> version
> > 2.1.0 to 2.2.0 without upgrading MapReduce.
> >
> > Server-Server compatibility is required to allow mixed versions 
> > within an active cluster so the cluster may be upgraded without 
> > downtime in a
> rolling
> > fashion.
> >
> > Notice that there is no specific mention of upgrading the client 
> > ahead of the server.  (There is no clause for "upgraded client + old 
> > server".) Based on my experience, this is a valid use case when a 
> > user wants to
> pick
> > up a client-side bug fix ahead of the cluster administrator's 
> > upgrade schedule.
> >
> > Is it our policy to maintain client compatibility with old clusters
> within
> > the same major release?  I think many of us have assumed that the 
> > answer
> is
> > yes and coded our new features accordingly, but it isn't made 
> > explicit in the documentation.  Do we all agree that the answer is 
> > yes, or is it possibly up for debate depending on the change in 
> > question?  In RFC 2119 lingo, is it a MUST or a SHOULD?  Either way, 
> > I'd like to update the
> policy
> > text to make our decision clear.  After we have consensus, I can
> volunteer
> > to file an issue and patch the text of the policy.
> >
> > This discussion started initially in MAPREDUCE-4052, which involved 
> > changing our scripting syntax for MapReduce YARN container submissions.
>  We
> > settled the question there by gating the syntax change behind a 
> > configuration option.  By default, it will continue using the 
> > existing syntax currently understood by the pre-2.4.0 NodeManager, 
> > thus preserving compatibility.  We wanted to open the policy 
> > question for wider
> discussion
> > though.
> >
> > Thanks, everyone.
> >
> > Chris Nauroth
> > Hortonworks
> > http://hortonworks.com/
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or 
> > entity
> to
> > which it is addressed and may contain information that is 
> > confidential, privileged and exempt from disclosure under applicable 
> > law. If the reader of this message is not the intended recipient, 
> > you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or 
> > forwarding of this communication is strictly prohibited. If you have 
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or 
> entity to which it is addressed and may contain information that is 
> confidential, privileged and exempt from disclosure under applicable 
> law. If the reader of this message is not the intended recipient, you 
> are hereby notified that any printing, copying, dissemination, 
> distribution, disclosure or forwarding of this communication is 
> strictly prohibited. If you have received this communication in error, 
> please contact the sender immediately and delete it from your system. Thank You.
>

--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.

RE: [DISCUSS] Clarification on Compatibility Policy: Upgraded Client + Old Server

Posted by Vinayakumar B <vi...@huawei.com>.
+1, I agree with your point Chris. It depends on the client application how they using the hdfs jars in their classpath.

As implementation already supports the compatibility (through protobuf), No extra code changes required to support new Client + old server.

I feel it will be good to explicitly mention about the compatibility of existing APIs in both versions.

Anyway this is not applicable for the new APIs in latest client and this is understood. We can make it explicit in the document though.


Regards,
Vinayakumar B

-----Original Message-----
From: Chris Nauroth [mailto:cnauroth@hortonworks.com] 
Sent: 20 March 2014 05:36
To: common-dev@hadoop.apache.org
Cc: mapreduce-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded Client + Old Server

I think this kind of compatibility issue still could surface for HDFS, particularly for custom applications (i.e. something not executed via "hadoop jar" on a cluster node, where the client classes ought to be injected into the classpath automatically).  Running DistCP between 2 clusters of different versions could result in a 2.4.0 client calling a
2.3.0 NameNode.  Someone could potentially pick up the 2.4.0 WebHDFS client as a dependency and try to use it to make HTTP calls to a 2.3.0 HDFS cluster.

Chris Nauroth
Hortonworks
http://hortonworks.com/



On Wed, Mar 19, 2014 at 4:28 PM, Vinod Kumar Vavilapalli <vinodkv@apache.org
> wrote:

> It makes sense only for YARN today where we separated out the clients.
> HDFS is still a monolithic jar so this compatibility issue is kind of 
> invalid there.
>
> +vinod
>
> On Mar 19, 2014, at 1:59 PM, Chris Nauroth <cn...@hortonworks.com>
> wrote:
>
> > I'd like to discuss clarification of part of our compatibility policy.
> > Here is a link to the compatibility documentation for release 2.3.0:
> >
> >
> http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common
> /Compatibility.html#Wire_compatibility
> >
> > For convenience, here are the specific lines in question:
> >
> > Client-Server compatibility is required to allow users to continue 
> > using the old clients even after upgrading the server (cluster) to a 
> > later version (or vice versa). For example, a Hadoop 2.1.0 client 
> > talking to a Hadoop 2.3.0 cluster.
> >
> > Client-Server compatibility is also required to allow upgrading
> individual
> > components without upgrading others. For example, upgrade HDFS from
> version
> > 2.1.0 to 2.2.0 without upgrading MapReduce.
> >
> > Server-Server compatibility is required to allow mixed versions 
> > within an active cluster so the cluster may be upgraded without 
> > downtime in a
> rolling
> > fashion.
> >
> > Notice that there is no specific mention of upgrading the client 
> > ahead of the server.  (There is no clause for "upgraded client + old 
> > server".) Based on my experience, this is a valid use case when a 
> > user wants to
> pick
> > up a client-side bug fix ahead of the cluster administrator's 
> > upgrade schedule.
> >
> > Is it our policy to maintain client compatibility with old clusters
> within
> > the same major release?  I think many of us have assumed that the 
> > answer
> is
> > yes and coded our new features accordingly, but it isn't made 
> > explicit in the documentation.  Do we all agree that the answer is 
> > yes, or is it possibly up for debate depending on the change in 
> > question?  In RFC 2119 lingo, is it a MUST or a SHOULD?  Either way, 
> > I'd like to update the
> policy
> > text to make our decision clear.  After we have consensus, I can
> volunteer
> > to file an issue and patch the text of the policy.
> >
> > This discussion started initially in MAPREDUCE-4052, which involved 
> > changing our scripting syntax for MapReduce YARN container submissions.
>  We
> > settled the question there by gating the syntax change behind a 
> > configuration option.  By default, it will continue using the 
> > existing syntax currently understood by the pre-2.4.0 NodeManager, 
> > thus preserving compatibility.  We wanted to open the policy 
> > question for wider
> discussion
> > though.
> >
> > Thanks, everyone.
> >
> > Chris Nauroth
> > Hortonworks
> > http://hortonworks.com/
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or 
> > entity
> to
> > which it is addressed and may contain information that is 
> > confidential, privileged and exempt from disclosure under applicable 
> > law. If the reader of this message is not the intended recipient, 
> > you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or 
> > forwarding of this communication is strictly prohibited. If you have 
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or 
> entity to which it is addressed and may contain information that is 
> confidential, privileged and exempt from disclosure under applicable 
> law. If the reader of this message is not the intended recipient, you 
> are hereby notified that any printing, copying, dissemination, 
> distribution, disclosure or forwarding of this communication is 
> strictly prohibited. If you have received this communication in error, 
> please contact the sender immediately and delete it from your system. Thank You.
>

--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.

Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded Client + Old Server

Posted by Chris Nauroth <cn...@hortonworks.com>.
I think this kind of compatibility issue still could surface for HDFS,
particularly for custom applications (i.e. something not executed via
"hadoop jar" on a cluster node, where the client classes ought to be
injected into the classpath automatically).  Running DistCP between 2
clusters of different versions could result in a 2.4.0 client calling a
2.3.0 NameNode.  Someone could potentially pick up the 2.4.0 WebHDFS client
as a dependency and try to use it to make HTTP calls to a 2.3.0 HDFS
cluster.

Chris Nauroth
Hortonworks
http://hortonworks.com/



On Wed, Mar 19, 2014 at 4:28 PM, Vinod Kumar Vavilapalli <vinodkv@apache.org
> wrote:

> It makes sense only for YARN today where we separated out the clients.
> HDFS is still a monolithic jar so this compatibility issue is kind of
> invalid there.
>
> +vinod
>
> On Mar 19, 2014, at 1:59 PM, Chris Nauroth <cn...@hortonworks.com>
> wrote:
>
> > I'd like to discuss clarification of part of our compatibility policy.
> > Here is a link to the compatibility documentation for release 2.3.0:
> >
> >
> http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common/Compatibility.html#Wire_compatibility
> >
> > For convenience, here are the specific lines in question:
> >
> > Client-Server compatibility is required to allow users to continue using
> > the old clients even after upgrading the server (cluster) to a later
> > version (or vice versa). For example, a Hadoop 2.1.0 client talking to a
> > Hadoop 2.3.0 cluster.
> >
> > Client-Server compatibility is also required to allow upgrading
> individual
> > components without upgrading others. For example, upgrade HDFS from
> version
> > 2.1.0 to 2.2.0 without upgrading MapReduce.
> >
> > Server-Server compatibility is required to allow mixed versions within an
> > active cluster so the cluster may be upgraded without downtime in a
> rolling
> > fashion.
> >
> > Notice that there is no specific mention of upgrading the client ahead of
> > the server.  (There is no clause for "upgraded client + old server".)
> > Based on my experience, this is a valid use case when a user wants to
> pick
> > up a client-side bug fix ahead of the cluster administrator's upgrade
> > schedule.
> >
> > Is it our policy to maintain client compatibility with old clusters
> within
> > the same major release?  I think many of us have assumed that the answer
> is
> > yes and coded our new features accordingly, but it isn't made explicit in
> > the documentation.  Do we all agree that the answer is yes, or is it
> > possibly up for debate depending on the change in question?  In RFC 2119
> > lingo, is it a MUST or a SHOULD?  Either way, I'd like to update the
> policy
> > text to make our decision clear.  After we have consensus, I can
> volunteer
> > to file an issue and patch the text of the policy.
> >
> > This discussion started initially in MAPREDUCE-4052, which involved
> > changing our scripting syntax for MapReduce YARN container submissions.
>  We
> > settled the question there by gating the syntax change behind a
> > configuration option.  By default, it will continue using the existing
> > syntax currently understood by the pre-2.4.0 NodeManager, thus preserving
> > compatibility.  We wanted to open the policy question for wider
> discussion
> > though.
> >
> > Thanks, everyone.
> >
> > Chris Nauroth
> > Hortonworks
> > http://hortonworks.com/
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded Client + Old Server

Posted by Chris Nauroth <cn...@hortonworks.com>.
I think this kind of compatibility issue still could surface for HDFS,
particularly for custom applications (i.e. something not executed via
"hadoop jar" on a cluster node, where the client classes ought to be
injected into the classpath automatically).  Running DistCP between 2
clusters of different versions could result in a 2.4.0 client calling a
2.3.0 NameNode.  Someone could potentially pick up the 2.4.0 WebHDFS client
as a dependency and try to use it to make HTTP calls to a 2.3.0 HDFS
cluster.

Chris Nauroth
Hortonworks
http://hortonworks.com/



On Wed, Mar 19, 2014 at 4:28 PM, Vinod Kumar Vavilapalli <vinodkv@apache.org
> wrote:

> It makes sense only for YARN today where we separated out the clients.
> HDFS is still a monolithic jar so this compatibility issue is kind of
> invalid there.
>
> +vinod
>
> On Mar 19, 2014, at 1:59 PM, Chris Nauroth <cn...@hortonworks.com>
> wrote:
>
> > I'd like to discuss clarification of part of our compatibility policy.
> > Here is a link to the compatibility documentation for release 2.3.0:
> >
> >
> http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common/Compatibility.html#Wire_compatibility
> >
> > For convenience, here are the specific lines in question:
> >
> > Client-Server compatibility is required to allow users to continue using
> > the old clients even after upgrading the server (cluster) to a later
> > version (or vice versa). For example, a Hadoop 2.1.0 client talking to a
> > Hadoop 2.3.0 cluster.
> >
> > Client-Server compatibility is also required to allow upgrading
> individual
> > components without upgrading others. For example, upgrade HDFS from
> version
> > 2.1.0 to 2.2.0 without upgrading MapReduce.
> >
> > Server-Server compatibility is required to allow mixed versions within an
> > active cluster so the cluster may be upgraded without downtime in a
> rolling
> > fashion.
> >
> > Notice that there is no specific mention of upgrading the client ahead of
> > the server.  (There is no clause for "upgraded client + old server".)
> > Based on my experience, this is a valid use case when a user wants to
> pick
> > up a client-side bug fix ahead of the cluster administrator's upgrade
> > schedule.
> >
> > Is it our policy to maintain client compatibility with old clusters
> within
> > the same major release?  I think many of us have assumed that the answer
> is
> > yes and coded our new features accordingly, but it isn't made explicit in
> > the documentation.  Do we all agree that the answer is yes, or is it
> > possibly up for debate depending on the change in question?  In RFC 2119
> > lingo, is it a MUST or a SHOULD?  Either way, I'd like to update the
> policy
> > text to make our decision clear.  After we have consensus, I can
> volunteer
> > to file an issue and patch the text of the policy.
> >
> > This discussion started initially in MAPREDUCE-4052, which involved
> > changing our scripting syntax for MapReduce YARN container submissions.
>  We
> > settled the question there by gating the syntax change behind a
> > configuration option.  By default, it will continue using the existing
> > syntax currently understood by the pre-2.4.0 NodeManager, thus preserving
> > compatibility.  We wanted to open the policy question for wider
> discussion
> > though.
> >
> > Thanks, everyone.
> >
> > Chris Nauroth
> > Hortonworks
> > http://hortonworks.com/
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded Client + Old Server

Posted by Chris Nauroth <cn...@hortonworks.com>.
I think this kind of compatibility issue still could surface for HDFS,
particularly for custom applications (i.e. something not executed via
"hadoop jar" on a cluster node, where the client classes ought to be
injected into the classpath automatically).  Running DistCP between 2
clusters of different versions could result in a 2.4.0 client calling a
2.3.0 NameNode.  Someone could potentially pick up the 2.4.0 WebHDFS client
as a dependency and try to use it to make HTTP calls to a 2.3.0 HDFS
cluster.

Chris Nauroth
Hortonworks
http://hortonworks.com/



On Wed, Mar 19, 2014 at 4:28 PM, Vinod Kumar Vavilapalli <vinodkv@apache.org
> wrote:

> It makes sense only for YARN today where we separated out the clients.
> HDFS is still a monolithic jar so this compatibility issue is kind of
> invalid there.
>
> +vinod
>
> On Mar 19, 2014, at 1:59 PM, Chris Nauroth <cn...@hortonworks.com>
> wrote:
>
> > I'd like to discuss clarification of part of our compatibility policy.
> > Here is a link to the compatibility documentation for release 2.3.0:
> >
> >
> http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common/Compatibility.html#Wire_compatibility
> >
> > For convenience, here are the specific lines in question:
> >
> > Client-Server compatibility is required to allow users to continue using
> > the old clients even after upgrading the server (cluster) to a later
> > version (or vice versa). For example, a Hadoop 2.1.0 client talking to a
> > Hadoop 2.3.0 cluster.
> >
> > Client-Server compatibility is also required to allow upgrading
> individual
> > components without upgrading others. For example, upgrade HDFS from
> version
> > 2.1.0 to 2.2.0 without upgrading MapReduce.
> >
> > Server-Server compatibility is required to allow mixed versions within an
> > active cluster so the cluster may be upgraded without downtime in a
> rolling
> > fashion.
> >
> > Notice that there is no specific mention of upgrading the client ahead of
> > the server.  (There is no clause for "upgraded client + old server".)
> > Based on my experience, this is a valid use case when a user wants to
> pick
> > up a client-side bug fix ahead of the cluster administrator's upgrade
> > schedule.
> >
> > Is it our policy to maintain client compatibility with old clusters
> within
> > the same major release?  I think many of us have assumed that the answer
> is
> > yes and coded our new features accordingly, but it isn't made explicit in
> > the documentation.  Do we all agree that the answer is yes, or is it
> > possibly up for debate depending on the change in question?  In RFC 2119
> > lingo, is it a MUST or a SHOULD?  Either way, I'd like to update the
> policy
> > text to make our decision clear.  After we have consensus, I can
> volunteer
> > to file an issue and patch the text of the policy.
> >
> > This discussion started initially in MAPREDUCE-4052, which involved
> > changing our scripting syntax for MapReduce YARN container submissions.
>  We
> > settled the question there by gating the syntax change behind a
> > configuration option.  By default, it will continue using the existing
> > syntax currently understood by the pre-2.4.0 NodeManager, thus preserving
> > compatibility.  We wanted to open the policy question for wider
> discussion
> > though.
> >
> > Thanks, everyone.
> >
> > Chris Nauroth
> > Hortonworks
> > http://hortonworks.com/
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded Client + Old Server

Posted by Chris Nauroth <cn...@hortonworks.com>.
I think this kind of compatibility issue still could surface for HDFS,
particularly for custom applications (i.e. something not executed via
"hadoop jar" on a cluster node, where the client classes ought to be
injected into the classpath automatically).  Running DistCP between 2
clusters of different versions could result in a 2.4.0 client calling a
2.3.0 NameNode.  Someone could potentially pick up the 2.4.0 WebHDFS client
as a dependency and try to use it to make HTTP calls to a 2.3.0 HDFS
cluster.

Chris Nauroth
Hortonworks
http://hortonworks.com/



On Wed, Mar 19, 2014 at 4:28 PM, Vinod Kumar Vavilapalli <vinodkv@apache.org
> wrote:

> It makes sense only for YARN today where we separated out the clients.
> HDFS is still a monolithic jar so this compatibility issue is kind of
> invalid there.
>
> +vinod
>
> On Mar 19, 2014, at 1:59 PM, Chris Nauroth <cn...@hortonworks.com>
> wrote:
>
> > I'd like to discuss clarification of part of our compatibility policy.
> > Here is a link to the compatibility documentation for release 2.3.0:
> >
> >
> http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common/Compatibility.html#Wire_compatibility
> >
> > For convenience, here are the specific lines in question:
> >
> > Client-Server compatibility is required to allow users to continue using
> > the old clients even after upgrading the server (cluster) to a later
> > version (or vice versa). For example, a Hadoop 2.1.0 client talking to a
> > Hadoop 2.3.0 cluster.
> >
> > Client-Server compatibility is also required to allow upgrading
> individual
> > components without upgrading others. For example, upgrade HDFS from
> version
> > 2.1.0 to 2.2.0 without upgrading MapReduce.
> >
> > Server-Server compatibility is required to allow mixed versions within an
> > active cluster so the cluster may be upgraded without downtime in a
> rolling
> > fashion.
> >
> > Notice that there is no specific mention of upgrading the client ahead of
> > the server.  (There is no clause for "upgraded client + old server".)
> > Based on my experience, this is a valid use case when a user wants to
> pick
> > up a client-side bug fix ahead of the cluster administrator's upgrade
> > schedule.
> >
> > Is it our policy to maintain client compatibility with old clusters
> within
> > the same major release?  I think many of us have assumed that the answer
> is
> > yes and coded our new features accordingly, but it isn't made explicit in
> > the documentation.  Do we all agree that the answer is yes, or is it
> > possibly up for debate depending on the change in question?  In RFC 2119
> > lingo, is it a MUST or a SHOULD?  Either way, I'd like to update the
> policy
> > text to make our decision clear.  After we have consensus, I can
> volunteer
> > to file an issue and patch the text of the policy.
> >
> > This discussion started initially in MAPREDUCE-4052, which involved
> > changing our scripting syntax for MapReduce YARN container submissions.
>  We
> > settled the question there by gating the syntax change behind a
> > configuration option.  By default, it will continue using the existing
> > syntax currently understood by the pre-2.4.0 NodeManager, thus preserving
> > compatibility.  We wanted to open the policy question for wider
> discussion
> > though.
> >
> > Thanks, everyone.
> >
> > Chris Nauroth
> > Hortonworks
> > http://hortonworks.com/
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded Client + Old Server

Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
It makes sense only for YARN today where we separated out the clients. HDFS is still a monolithic jar so this compatibility issue is kind of invalid there.

+vinod

On Mar 19, 2014, at 1:59 PM, Chris Nauroth <cn...@hortonworks.com> wrote:

> I'd like to discuss clarification of part of our compatibility policy.
> Here is a link to the compatibility documentation for release 2.3.0:
> 
> http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common/Compatibility.html#Wire_compatibility
> 
> For convenience, here are the specific lines in question:
> 
> Client-Server compatibility is required to allow users to continue using
> the old clients even after upgrading the server (cluster) to a later
> version (or vice versa). For example, a Hadoop 2.1.0 client talking to a
> Hadoop 2.3.0 cluster.
> 
> Client-Server compatibility is also required to allow upgrading individual
> components without upgrading others. For example, upgrade HDFS from version
> 2.1.0 to 2.2.0 without upgrading MapReduce.
> 
> Server-Server compatibility is required to allow mixed versions within an
> active cluster so the cluster may be upgraded without downtime in a rolling
> fashion.
> 
> Notice that there is no specific mention of upgrading the client ahead of
> the server.  (There is no clause for "upgraded client + old server".)
> Based on my experience, this is a valid use case when a user wants to pick
> up a client-side bug fix ahead of the cluster administrator's upgrade
> schedule.
> 
> Is it our policy to maintain client compatibility with old clusters within
> the same major release?  I think many of us have assumed that the answer is
> yes and coded our new features accordingly, but it isn't made explicit in
> the documentation.  Do we all agree that the answer is yes, or is it
> possibly up for debate depending on the change in question?  In RFC 2119
> lingo, is it a MUST or a SHOULD?  Either way, I'd like to update the policy
> text to make our decision clear.  After we have consensus, I can volunteer
> to file an issue and patch the text of the policy.
> 
> This discussion started initially in MAPREDUCE-4052, which involved
> changing our scripting syntax for MapReduce YARN container submissions.  We
> settled the question there by gating the syntax change behind a
> configuration option.  By default, it will continue using the existing
> syntax currently understood by the pre-2.4.0 NodeManager, thus preserving
> compatibility.  We wanted to open the policy question for wider discussion
> though.
> 
> Thanks, everyone.
> 
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
> 
> -- 
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to 
> which it is addressed and may contain information that is confidential, 
> privileged and exempt from disclosure under applicable law. If the reader 
> of this message is not the intended recipient, you are hereby notified that 
> any printing, copying, dissemination, distribution, disclosure or 
> forwarding of this communication is strictly prohibited. If you have 
> received this communication in error, please contact the sender immediately 
> and delete it from your system. Thank You.


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded Client + Old Server

Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
It makes sense only for YARN today where we separated out the clients. HDFS is still a monolithic jar so this compatibility issue is kind of invalid there.

+vinod

On Mar 19, 2014, at 1:59 PM, Chris Nauroth <cn...@hortonworks.com> wrote:

> I'd like to discuss clarification of part of our compatibility policy.
> Here is a link to the compatibility documentation for release 2.3.0:
> 
> http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common/Compatibility.html#Wire_compatibility
> 
> For convenience, here are the specific lines in question:
> 
> Client-Server compatibility is required to allow users to continue using
> the old clients even after upgrading the server (cluster) to a later
> version (or vice versa). For example, a Hadoop 2.1.0 client talking to a
> Hadoop 2.3.0 cluster.
> 
> Client-Server compatibility is also required to allow upgrading individual
> components without upgrading others. For example, upgrade HDFS from version
> 2.1.0 to 2.2.0 without upgrading MapReduce.
> 
> Server-Server compatibility is required to allow mixed versions within an
> active cluster so the cluster may be upgraded without downtime in a rolling
> fashion.
> 
> Notice that there is no specific mention of upgrading the client ahead of
> the server.  (There is no clause for "upgraded client + old server".)
> Based on my experience, this is a valid use case when a user wants to pick
> up a client-side bug fix ahead of the cluster administrator's upgrade
> schedule.
> 
> Is it our policy to maintain client compatibility with old clusters within
> the same major release?  I think many of us have assumed that the answer is
> yes and coded our new features accordingly, but it isn't made explicit in
> the documentation.  Do we all agree that the answer is yes, or is it
> possibly up for debate depending on the change in question?  In RFC 2119
> lingo, is it a MUST or a SHOULD?  Either way, I'd like to update the policy
> text to make our decision clear.  After we have consensus, I can volunteer
> to file an issue and patch the text of the policy.
> 
> This discussion started initially in MAPREDUCE-4052, which involved
> changing our scripting syntax for MapReduce YARN container submissions.  We
> settled the question there by gating the syntax change behind a
> configuration option.  By default, it will continue using the existing
> syntax currently understood by the pre-2.4.0 NodeManager, thus preserving
> compatibility.  We wanted to open the policy question for wider discussion
> though.
> 
> Thanks, everyone.
> 
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
> 
> -- 
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to 
> which it is addressed and may contain information that is confidential, 
> privileged and exempt from disclosure under applicable law. If the reader 
> of this message is not the intended recipient, you are hereby notified that 
> any printing, copying, dissemination, distribution, disclosure or 
> forwarding of this communication is strictly prohibited. If you have 
> received this communication in error, please contact the sender immediately 
> and delete it from your system. Thank You.


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded Client + Old Server

Posted by Karthik Kambatla <ka...@cloudera.com>.
+1 on supporting new clients with old servers of the same major version,
and updating the policy to capture that clearly.


On Wed, Mar 19, 2014 at 1:59 PM, Chris Nauroth <cn...@hortonworks.com>wrote:

> I'd like to discuss clarification of part of our compatibility policy.
>  Here is a link to the compatibility documentation for release 2.3.0:
>
>
> http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common/Compatibility.html#Wire_compatibility
>
> For convenience, here are the specific lines in question:
>
> Client-Server compatibility is required to allow users to continue using
> the old clients even after upgrading the server (cluster) to a later
> version (or vice versa). For example, a Hadoop 2.1.0 client talking to a
> Hadoop 2.3.0 cluster.
>
> Client-Server compatibility is also required to allow upgrading individual
> components without upgrading others. For example, upgrade HDFS from version
> 2.1.0 to 2.2.0 without upgrading MapReduce.
>
> Server-Server compatibility is required to allow mixed versions within an
> active cluster so the cluster may be upgraded without downtime in a rolling
> fashion.
>
> Notice that there is no specific mention of upgrading the client ahead of
> the server.  (There is no clause for "upgraded client + old server".)
>  Based on my experience, this is a valid use case when a user wants to pick
> up a client-side bug fix ahead of the cluster administrator's upgrade
> schedule.
>
> Is it our policy to maintain client compatibility with old clusters within
> the same major release?  I think many of us have assumed that the answer is
> yes and coded our new features accordingly, but it isn't made explicit in
> the documentation.  Do we all agree that the answer is yes, or is it
> possibly up for debate depending on the change in question?  In RFC 2119
> lingo, is it a MUST or a SHOULD?  Either way, I'd like to update the policy
> text to make our decision clear.  After we have consensus, I can volunteer
> to file an issue and patch the text of the policy.
>
> This discussion started initially in MAPREDUCE-4052, which involved
> changing our scripting syntax for MapReduce YARN container submissions.  We
> settled the question there by gating the syntax change behind a
> configuration option.  By default, it will continue using the existing
> syntax currently understood by the pre-2.4.0 NodeManager, thus preserving
> compatibility.  We wanted to open the policy question for wider discussion
> though.
>
> Thanks, everyone.
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded Client + Old Server

Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
It makes sense only for YARN today where we separated out the clients. HDFS is still a monolithic jar so this compatibility issue is kind of invalid there.

+vinod

On Mar 19, 2014, at 1:59 PM, Chris Nauroth <cn...@hortonworks.com> wrote:

> I'd like to discuss clarification of part of our compatibility policy.
> Here is a link to the compatibility documentation for release 2.3.0:
> 
> http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common/Compatibility.html#Wire_compatibility
> 
> For convenience, here are the specific lines in question:
> 
> Client-Server compatibility is required to allow users to continue using
> the old clients even after upgrading the server (cluster) to a later
> version (or vice versa). For example, a Hadoop 2.1.0 client talking to a
> Hadoop 2.3.0 cluster.
> 
> Client-Server compatibility is also required to allow upgrading individual
> components without upgrading others. For example, upgrade HDFS from version
> 2.1.0 to 2.2.0 without upgrading MapReduce.
> 
> Server-Server compatibility is required to allow mixed versions within an
> active cluster so the cluster may be upgraded without downtime in a rolling
> fashion.
> 
> Notice that there is no specific mention of upgrading the client ahead of
> the server.  (There is no clause for "upgraded client + old server".)
> Based on my experience, this is a valid use case when a user wants to pick
> up a client-side bug fix ahead of the cluster administrator's upgrade
> schedule.
> 
> Is it our policy to maintain client compatibility with old clusters within
> the same major release?  I think many of us have assumed that the answer is
> yes and coded our new features accordingly, but it isn't made explicit in
> the documentation.  Do we all agree that the answer is yes, or is it
> possibly up for debate depending on the change in question?  In RFC 2119
> lingo, is it a MUST or a SHOULD?  Either way, I'd like to update the policy
> text to make our decision clear.  After we have consensus, I can volunteer
> to file an issue and patch the text of the policy.
> 
> This discussion started initially in MAPREDUCE-4052, which involved
> changing our scripting syntax for MapReduce YARN container submissions.  We
> settled the question there by gating the syntax change behind a
> configuration option.  By default, it will continue using the existing
> syntax currently understood by the pre-2.4.0 NodeManager, thus preserving
> compatibility.  We wanted to open the policy question for wider discussion
> though.
> 
> Thanks, everyone.
> 
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
> 
> -- 
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to 
> which it is addressed and may contain information that is confidential, 
> privileged and exempt from disclosure under applicable law. If the reader 
> of this message is not the intended recipient, you are hereby notified that 
> any printing, copying, dissemination, distribution, disclosure or 
> forwarding of this communication is strictly prohibited. If you have 
> received this communication in error, please contact the sender immediately 
> and delete it from your system. Thank You.


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded Client + Old Server

Posted by Karthik Kambatla <ka...@cloudera.com>.
+1 on supporting new clients with old servers of the same major version,
and updating the policy to capture that clearly.


On Wed, Mar 19, 2014 at 1:59 PM, Chris Nauroth <cn...@hortonworks.com>wrote:

> I'd like to discuss clarification of part of our compatibility policy.
>  Here is a link to the compatibility documentation for release 2.3.0:
>
>
> http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common/Compatibility.html#Wire_compatibility
>
> For convenience, here are the specific lines in question:
>
> Client-Server compatibility is required to allow users to continue using
> the old clients even after upgrading the server (cluster) to a later
> version (or vice versa). For example, a Hadoop 2.1.0 client talking to a
> Hadoop 2.3.0 cluster.
>
> Client-Server compatibility is also required to allow upgrading individual
> components without upgrading others. For example, upgrade HDFS from version
> 2.1.0 to 2.2.0 without upgrading MapReduce.
>
> Server-Server compatibility is required to allow mixed versions within an
> active cluster so the cluster may be upgraded without downtime in a rolling
> fashion.
>
> Notice that there is no specific mention of upgrading the client ahead of
> the server.  (There is no clause for "upgraded client + old server".)
>  Based on my experience, this is a valid use case when a user wants to pick
> up a client-side bug fix ahead of the cluster administrator's upgrade
> schedule.
>
> Is it our policy to maintain client compatibility with old clusters within
> the same major release?  I think many of us have assumed that the answer is
> yes and coded our new features accordingly, but it isn't made explicit in
> the documentation.  Do we all agree that the answer is yes, or is it
> possibly up for debate depending on the change in question?  In RFC 2119
> lingo, is it a MUST or a SHOULD?  Either way, I'd like to update the policy
> text to make our decision clear.  After we have consensus, I can volunteer
> to file an issue and patch the text of the policy.
>
> This discussion started initially in MAPREDUCE-4052, which involved
> changing our scripting syntax for MapReduce YARN container submissions.  We
> settled the question there by gating the syntax change behind a
> configuration option.  By default, it will continue using the existing
> syntax currently understood by the pre-2.4.0 NodeManager, thus preserving
> compatibility.  We wanted to open the policy question for wider discussion
> though.
>
> Thanks, everyone.
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded Client + Old Server

Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
It makes sense only for YARN today where we separated out the clients. HDFS is still a monolithic jar so this compatibility issue is kind of invalid there.

+vinod

On Mar 19, 2014, at 1:59 PM, Chris Nauroth <cn...@hortonworks.com> wrote:

> I'd like to discuss clarification of part of our compatibility policy.
> Here is a link to the compatibility documentation for release 2.3.0:
> 
> http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common/Compatibility.html#Wire_compatibility
> 
> For convenience, here are the specific lines in question:
> 
> Client-Server compatibility is required to allow users to continue using
> the old clients even after upgrading the server (cluster) to a later
> version (or vice versa). For example, a Hadoop 2.1.0 client talking to a
> Hadoop 2.3.0 cluster.
> 
> Client-Server compatibility is also required to allow upgrading individual
> components without upgrading others. For example, upgrade HDFS from version
> 2.1.0 to 2.2.0 without upgrading MapReduce.
> 
> Server-Server compatibility is required to allow mixed versions within an
> active cluster so the cluster may be upgraded without downtime in a rolling
> fashion.
> 
> Notice that there is no specific mention of upgrading the client ahead of
> the server.  (There is no clause for "upgraded client + old server".)
> Based on my experience, this is a valid use case when a user wants to pick
> up a client-side bug fix ahead of the cluster administrator's upgrade
> schedule.
> 
> Is it our policy to maintain client compatibility with old clusters within
> the same major release?  I think many of us have assumed that the answer is
> yes and coded our new features accordingly, but it isn't made explicit in
> the documentation.  Do we all agree that the answer is yes, or is it
> possibly up for debate depending on the change in question?  In RFC 2119
> lingo, is it a MUST or a SHOULD?  Either way, I'd like to update the policy
> text to make our decision clear.  After we have consensus, I can volunteer
> to file an issue and patch the text of the policy.
> 
> This discussion started initially in MAPREDUCE-4052, which involved
> changing our scripting syntax for MapReduce YARN container submissions.  We
> settled the question there by gating the syntax change behind a
> configuration option.  By default, it will continue using the existing
> syntax currently understood by the pre-2.4.0 NodeManager, thus preserving
> compatibility.  We wanted to open the policy question for wider discussion
> though.
> 
> Thanks, everyone.
> 
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
> 
> -- 
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to 
> which it is addressed and may contain information that is confidential, 
> privileged and exempt from disclosure under applicable law. If the reader 
> of this message is not the intended recipient, you are hereby notified that 
> any printing, copying, dissemination, distribution, disclosure or 
> forwarding of this communication is strictly prohibited. If you have 
> received this communication in error, please contact the sender immediately 
> and delete it from your system. Thank You.


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.