You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Zhijie Shen <zs...@hortonworks.com> on 2014/07/29 18:26:04 UTC
Hadoop Code of Incompatible Changes
Hi folks,
Recently we have a conversation on YARN-2209 about the incompatible changes
over releases. For those API changes that will break binary compatibility,
source compatibility towards the existing API users, we've already had a
rather clear picture about what we should do. However, YARN-2209 has
introduced another case which I'm not quite sure about, which is kind of *logic
incompatibility*.
In detail, an ApplicationMasterProtocol API is going to throw an exception
which is not expected before. The exception is a sub-class of
YarnException, such that it doesn't need any method signature change, and
won't break any binary/source compatibility. However, the exception is not
expected before, but needs to be treated specially at the AM side. Not
being aware of the newly introduced exception, the existing YARN
applications' AM may not handle it exception properly, and is at the risk
of being broken on a new YARN cluster after this change.
An additional thought around this problem is that the change of what
exception is to throw under what situation may be considered as a *soft *method
signature change, because we're supposed to write this javadoc to tell the
users (though we didn't it well in Hadoop), and users refer to it to guide
how to handle the exception.
In a more generalized form, let's assume we have a method, which behaves as
A, in release 1.0. However, in release 2.0, the method signature has kept
the same, while its behavior is altered from A to B. A and B are different
behaviors. In this case, do we consider it as an incompatible change?
I think it's somewhat a common issue, such that I raise it on the mailing
list. Please share your ideas.
Thanks,
Zhijie
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Hadoop Code of Incompatible Changes
Posted by Steve Loughran <st...@hortonworks.com>.
On 29 July 2014 22:14, Sandy Ryza <sa...@cloudera.com> wrote:
>
> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Compatibility.html
Sandy is correct is a semantic compatibility issue. Back then the notion of
"interface" was defined in the early 1970s by D.L Parnas (see:
http://www.cs.mun.ca/~harold/Courses/Old/CS3718.F08/Diary/p330-parnas.pdf),
he made clear from the outset that an interface was a combination of
(binary signature, functional semantics) -it's just that C++ derivative
languages have only focused on the signature, which is what can be checked
at compile and link times
Policy there is "The behavior of API may be changed to fix incorrect
behavior"
If an AM is now expected to handle new things, then it is a change in the
semantics...though if it was due to a bug fix -things didn't work before-
then it could maybe get in.
With YARN, there's now a (wide) interface between apps and Hadoop, and
right now it is fairly intimate -you do tend to end up using things marked
as private because its how you get stuff like the web proxy/filter and
other things that I can't remember); I think it's going to take a while to
stabilise.
Ideally for this patch, I'd like an AM (including my own) to at least
degrade to its existing behaviour, even if it doesn't pick up any new
features. Is this the case?
-steve
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Hadoop Code of Incompatible Changes
Posted by Steve Loughran <st...@hortonworks.com>.
On 29 July 2014 22:14, Sandy Ryza <sa...@cloudera.com> wrote:
>
> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Compatibility.html
Sandy is correct is a semantic compatibility issue. Back then the notion of
"interface" was defined in the early 1970s by D.L Parnas (see:
http://www.cs.mun.ca/~harold/Courses/Old/CS3718.F08/Diary/p330-parnas.pdf),
he made clear from the outset that an interface was a combination of
(binary signature, functional semantics) -it's just that C++ derivative
languages have only focused on the signature, which is what can be checked
at compile and link times
Policy there is "The behavior of API may be changed to fix incorrect
behavior"
If an AM is now expected to handle new things, then it is a change in the
semantics...though if it was due to a bug fix -things didn't work before-
then it could maybe get in.
With YARN, there's now a (wide) interface between apps and Hadoop, and
right now it is fairly intimate -you do tend to end up using things marked
as private because its how you get stuff like the web proxy/filter and
other things that I can't remember); I think it's going to take a while to
stabilise.
Ideally for this patch, I'd like an AM (including my own) to at least
degrade to its existing behaviour, even if it doesn't pick up any new
features. Is this the case?
-steve
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Hadoop Code of Incompatible Changes
Posted by Sandy Ryza <sa...@cloudera.com>.
Thanks Arpit!
On Tue, Jul 29, 2014 at 2:24 PM, Arpit Agarwal <aa...@hortonworks.com>
wrote:
> I cleared out the wiki page and left a forwarding link to the site docs.
> From a quick scan all the content is included in the site docs.
>
>
> On Tue, Jul 29, 2014 at 2:14 PM, Sandy Ryza <sa...@cloudera.com>
> wrote:
>
> > Eli pointed out to me that this is the up-to-date compatibility guide:
> >
> >
> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Compatibility.html
> >
> > Thanks,
> > Sandy
> >
> >
> > On Tue, Jul 29, 2014 at 9:39 AM, Sandy Ryza <sa...@cloudera.com>
> > wrote:
> >
> > > Hi Zhijie,
> > >
> > > The Hadoop compatibility guide mentions this as "semantic
> compatibility":
> > > http://wiki.apache.org/hadoop/Compatibility
> > >
> > > My interpretation of the section is that we can't change the behavior
> of
> > > public APIs unless we're fixing buggy behavior. If the change could
> > break
> > > an existing application that's behaving reasonably with respect to the
> > old
> > > API, it's an incompatible change.
> > >
> > > -Sandy
> > >
> > >
> > >
> > > On Tue, Jul 29, 2014 at 9:26 AM, Zhijie Shen <zs...@hortonworks.com>
> > > wrote:
> > >
> > >> Hi folks,
> > >>
> > >> Recently we have a conversation on YARN-2209 about the incompatible
> > >> changes
> > >> over releases. For those API changes that will break binary
> > compatibility,
> > >> source compatibility towards the existing API users, we've already
> had a
> > >> rather clear picture about what we should do. However, YARN-2209 has
> > >> introduced another case which I'm not quite sure about, which is kind
> of
> > >> *logic
> > >> incompatibility*.
> > >>
> > >> In detail, an ApplicationMasterProtocol API is going to throw an
> > exception
> > >> which is not expected before. The exception is a sub-class of
> > >> YarnException, such that it doesn't need any method signature change,
> > and
> > >> won't break any binary/source compatibility. However, the exception is
> > not
> > >> expected before, but needs to be treated specially at the AM side. Not
> > >> being aware of the newly introduced exception, the existing YARN
> > >> applications' AM may not handle it exception properly, and is at the
> > risk
> > >> of being broken on a new YARN cluster after this change.
> > >>
> > >> An additional thought around this problem is that the change of what
> > >> exception is to throw under what situation may be considered as a
> *soft
> > >> *method
> > >> signature change, because we're supposed to write this javadoc to tell
> > the
> > >> users (though we didn't it well in Hadoop), and users refer to it to
> > guide
> > >> how to handle the exception.
> > >>
> > >> In a more generalized form, let's assume we have a method, which
> behaves
> > >> as
> > >> A, in release 1.0. However, in release 2.0, the method signature has
> > kept
> > >> the same, while its behavior is altered from A to B. A and B are
> > different
> > >> behaviors. In this case, do we consider it as an incompatible change?
> > >>
> > >> I think it's somewhat a common issue, such that I raise it on the
> > mailing
> > >> list. Please share your ideas.
> > >>
> > >> Thanks,
> > >> Zhijie
> > >>
> > >> --
> > >> Zhijie Shen
> > >> Hortonworks Inc.
> > >> http://hortonworks.com/
> > >>
> > >> --
> > >> CONFIDENTIALITY NOTICE
> > >> NOTICE: This message is intended for the use of the individual or
> entity
> > >> to
> > >> which it is addressed and may contain information that is
> confidential,
> > >> privileged and exempt from disclosure under applicable law. If the
> > reader
> > >> of this message is not the intended recipient, you are hereby notified
> > >> that
> > >> any printing, copying, dissemination, distribution, disclosure or
> > >> forwarding of this communication is strictly prohibited. If you have
> > >> received this communication in error, please contact the sender
> > >> immediately
> > >> and delete it from your system. Thank You.
> > >>
> > >
> > >
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Hadoop Code of Incompatible Changes
Posted by Sandy Ryza <sa...@cloudera.com>.
Thanks Arpit!
On Tue, Jul 29, 2014 at 2:24 PM, Arpit Agarwal <aa...@hortonworks.com>
wrote:
> I cleared out the wiki page and left a forwarding link to the site docs.
> From a quick scan all the content is included in the site docs.
>
>
> On Tue, Jul 29, 2014 at 2:14 PM, Sandy Ryza <sa...@cloudera.com>
> wrote:
>
> > Eli pointed out to me that this is the up-to-date compatibility guide:
> >
> >
> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Compatibility.html
> >
> > Thanks,
> > Sandy
> >
> >
> > On Tue, Jul 29, 2014 at 9:39 AM, Sandy Ryza <sa...@cloudera.com>
> > wrote:
> >
> > > Hi Zhijie,
> > >
> > > The Hadoop compatibility guide mentions this as "semantic
> compatibility":
> > > http://wiki.apache.org/hadoop/Compatibility
> > >
> > > My interpretation of the section is that we can't change the behavior
> of
> > > public APIs unless we're fixing buggy behavior. If the change could
> > break
> > > an existing application that's behaving reasonably with respect to the
> > old
> > > API, it's an incompatible change.
> > >
> > > -Sandy
> > >
> > >
> > >
> > > On Tue, Jul 29, 2014 at 9:26 AM, Zhijie Shen <zs...@hortonworks.com>
> > > wrote:
> > >
> > >> Hi folks,
> > >>
> > >> Recently we have a conversation on YARN-2209 about the incompatible
> > >> changes
> > >> over releases. For those API changes that will break binary
> > compatibility,
> > >> source compatibility towards the existing API users, we've already
> had a
> > >> rather clear picture about what we should do. However, YARN-2209 has
> > >> introduced another case which I'm not quite sure about, which is kind
> of
> > >> *logic
> > >> incompatibility*.
> > >>
> > >> In detail, an ApplicationMasterProtocol API is going to throw an
> > exception
> > >> which is not expected before. The exception is a sub-class of
> > >> YarnException, such that it doesn't need any method signature change,
> > and
> > >> won't break any binary/source compatibility. However, the exception is
> > not
> > >> expected before, but needs to be treated specially at the AM side. Not
> > >> being aware of the newly introduced exception, the existing YARN
> > >> applications' AM may not handle it exception properly, and is at the
> > risk
> > >> of being broken on a new YARN cluster after this change.
> > >>
> > >> An additional thought around this problem is that the change of what
> > >> exception is to throw under what situation may be considered as a
> *soft
> > >> *method
> > >> signature change, because we're supposed to write this javadoc to tell
> > the
> > >> users (though we didn't it well in Hadoop), and users refer to it to
> > guide
> > >> how to handle the exception.
> > >>
> > >> In a more generalized form, let's assume we have a method, which
> behaves
> > >> as
> > >> A, in release 1.0. However, in release 2.0, the method signature has
> > kept
> > >> the same, while its behavior is altered from A to B. A and B are
> > different
> > >> behaviors. In this case, do we consider it as an incompatible change?
> > >>
> > >> I think it's somewhat a common issue, such that I raise it on the
> > mailing
> > >> list. Please share your ideas.
> > >>
> > >> Thanks,
> > >> Zhijie
> > >>
> > >> --
> > >> Zhijie Shen
> > >> Hortonworks Inc.
> > >> http://hortonworks.com/
> > >>
> > >> --
> > >> CONFIDENTIALITY NOTICE
> > >> NOTICE: This message is intended for the use of the individual or
> entity
> > >> to
> > >> which it is addressed and may contain information that is
> confidential,
> > >> privileged and exempt from disclosure under applicable law. If the
> > reader
> > >> of this message is not the intended recipient, you are hereby notified
> > >> that
> > >> any printing, copying, dissemination, distribution, disclosure or
> > >> forwarding of this communication is strictly prohibited. If you have
> > >> received this communication in error, please contact the sender
> > >> immediately
> > >> and delete it from your system. Thank You.
> > >>
> > >
> > >
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Hadoop Code of Incompatible Changes
Posted by Arpit Agarwal <aa...@hortonworks.com>.
I cleared out the wiki page and left a forwarding link to the site docs.
>From a quick scan all the content is included in the site docs.
On Tue, Jul 29, 2014 at 2:14 PM, Sandy Ryza <sa...@cloudera.com> wrote:
> Eli pointed out to me that this is the up-to-date compatibility guide:
>
> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Compatibility.html
>
> Thanks,
> Sandy
>
>
> On Tue, Jul 29, 2014 at 9:39 AM, Sandy Ryza <sa...@cloudera.com>
> wrote:
>
> > Hi Zhijie,
> >
> > The Hadoop compatibility guide mentions this as "semantic compatibility":
> > http://wiki.apache.org/hadoop/Compatibility
> >
> > My interpretation of the section is that we can't change the behavior of
> > public APIs unless we're fixing buggy behavior. If the change could
> break
> > an existing application that's behaving reasonably with respect to the
> old
> > API, it's an incompatible change.
> >
> > -Sandy
> >
> >
> >
> > On Tue, Jul 29, 2014 at 9:26 AM, Zhijie Shen <zs...@hortonworks.com>
> > wrote:
> >
> >> Hi folks,
> >>
> >> Recently we have a conversation on YARN-2209 about the incompatible
> >> changes
> >> over releases. For those API changes that will break binary
> compatibility,
> >> source compatibility towards the existing API users, we've already had a
> >> rather clear picture about what we should do. However, YARN-2209 has
> >> introduced another case which I'm not quite sure about, which is kind of
> >> *logic
> >> incompatibility*.
> >>
> >> In detail, an ApplicationMasterProtocol API is going to throw an
> exception
> >> which is not expected before. The exception is a sub-class of
> >> YarnException, such that it doesn't need any method signature change,
> and
> >> won't break any binary/source compatibility. However, the exception is
> not
> >> expected before, but needs to be treated specially at the AM side. Not
> >> being aware of the newly introduced exception, the existing YARN
> >> applications' AM may not handle it exception properly, and is at the
> risk
> >> of being broken on a new YARN cluster after this change.
> >>
> >> An additional thought around this problem is that the change of what
> >> exception is to throw under what situation may be considered as a *soft
> >> *method
> >> signature change, because we're supposed to write this javadoc to tell
> the
> >> users (though we didn't it well in Hadoop), and users refer to it to
> guide
> >> how to handle the exception.
> >>
> >> In a more generalized form, let's assume we have a method, which behaves
> >> as
> >> A, in release 1.0. However, in release 2.0, the method signature has
> kept
> >> the same, while its behavior is altered from A to B. A and B are
> different
> >> behaviors. In this case, do we consider it as an incompatible change?
> >>
> >> I think it's somewhat a common issue, such that I raise it on the
> mailing
> >> list. Please share your ideas.
> >>
> >> Thanks,
> >> Zhijie
> >>
> >> --
> >> Zhijie Shen
> >> Hortonworks Inc.
> >> http://hortonworks.com/
> >>
> >> --
> >> CONFIDENTIALITY NOTICE
> >> NOTICE: This message is intended for the use of the individual or entity
> >> to
> >> which it is addressed and may contain information that is confidential,
> >> privileged and exempt from disclosure under applicable law. If the
> reader
> >> of this message is not the intended recipient, you are hereby notified
> >> that
> >> any printing, copying, dissemination, distribution, disclosure or
> >> forwarding of this communication is strictly prohibited. If you have
> >> received this communication in error, please contact the sender
> >> immediately
> >> and delete it from your system. Thank You.
> >>
> >
> >
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Hadoop Code of Incompatible Changes
Posted by Arpit Agarwal <aa...@hortonworks.com>.
I cleared out the wiki page and left a forwarding link to the site docs.
>From a quick scan all the content is included in the site docs.
On Tue, Jul 29, 2014 at 2:14 PM, Sandy Ryza <sa...@cloudera.com> wrote:
> Eli pointed out to me that this is the up-to-date compatibility guide:
>
> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Compatibility.html
>
> Thanks,
> Sandy
>
>
> On Tue, Jul 29, 2014 at 9:39 AM, Sandy Ryza <sa...@cloudera.com>
> wrote:
>
> > Hi Zhijie,
> >
> > The Hadoop compatibility guide mentions this as "semantic compatibility":
> > http://wiki.apache.org/hadoop/Compatibility
> >
> > My interpretation of the section is that we can't change the behavior of
> > public APIs unless we're fixing buggy behavior. If the change could
> break
> > an existing application that's behaving reasonably with respect to the
> old
> > API, it's an incompatible change.
> >
> > -Sandy
> >
> >
> >
> > On Tue, Jul 29, 2014 at 9:26 AM, Zhijie Shen <zs...@hortonworks.com>
> > wrote:
> >
> >> Hi folks,
> >>
> >> Recently we have a conversation on YARN-2209 about the incompatible
> >> changes
> >> over releases. For those API changes that will break binary
> compatibility,
> >> source compatibility towards the existing API users, we've already had a
> >> rather clear picture about what we should do. However, YARN-2209 has
> >> introduced another case which I'm not quite sure about, which is kind of
> >> *logic
> >> incompatibility*.
> >>
> >> In detail, an ApplicationMasterProtocol API is going to throw an
> exception
> >> which is not expected before. The exception is a sub-class of
> >> YarnException, such that it doesn't need any method signature change,
> and
> >> won't break any binary/source compatibility. However, the exception is
> not
> >> expected before, but needs to be treated specially at the AM side. Not
> >> being aware of the newly introduced exception, the existing YARN
> >> applications' AM may not handle it exception properly, and is at the
> risk
> >> of being broken on a new YARN cluster after this change.
> >>
> >> An additional thought around this problem is that the change of what
> >> exception is to throw under what situation may be considered as a *soft
> >> *method
> >> signature change, because we're supposed to write this javadoc to tell
> the
> >> users (though we didn't it well in Hadoop), and users refer to it to
> guide
> >> how to handle the exception.
> >>
> >> In a more generalized form, let's assume we have a method, which behaves
> >> as
> >> A, in release 1.0. However, in release 2.0, the method signature has
> kept
> >> the same, while its behavior is altered from A to B. A and B are
> different
> >> behaviors. In this case, do we consider it as an incompatible change?
> >>
> >> I think it's somewhat a common issue, such that I raise it on the
> mailing
> >> list. Please share your ideas.
> >>
> >> Thanks,
> >> Zhijie
> >>
> >> --
> >> Zhijie Shen
> >> Hortonworks Inc.
> >> http://hortonworks.com/
> >>
> >> --
> >> CONFIDENTIALITY NOTICE
> >> NOTICE: This message is intended for the use of the individual or entity
> >> to
> >> which it is addressed and may contain information that is confidential,
> >> privileged and exempt from disclosure under applicable law. If the
> reader
> >> of this message is not the intended recipient, you are hereby notified
> >> that
> >> any printing, copying, dissemination, distribution, disclosure or
> >> forwarding of this communication is strictly prohibited. If you have
> >> received this communication in error, please contact the sender
> >> immediately
> >> and delete it from your system. Thank You.
> >>
> >
> >
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Hadoop Code of Incompatible Changes
Posted by Sandy Ryza <sa...@cloudera.com>.
Eli pointed out to me that this is the up-to-date compatibility guide:
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Compatibility.html
Thanks,
Sandy
On Tue, Jul 29, 2014 at 9:39 AM, Sandy Ryza <sa...@cloudera.com> wrote:
> Hi Zhijie,
>
> The Hadoop compatibility guide mentions this as "semantic compatibility":
> http://wiki.apache.org/hadoop/Compatibility
>
> My interpretation of the section is that we can't change the behavior of
> public APIs unless we're fixing buggy behavior. If the change could break
> an existing application that's behaving reasonably with respect to the old
> API, it's an incompatible change.
>
> -Sandy
>
>
>
> On Tue, Jul 29, 2014 at 9:26 AM, Zhijie Shen <zs...@hortonworks.com>
> wrote:
>
>> Hi folks,
>>
>> Recently we have a conversation on YARN-2209 about the incompatible
>> changes
>> over releases. For those API changes that will break binary compatibility,
>> source compatibility towards the existing API users, we've already had a
>> rather clear picture about what we should do. However, YARN-2209 has
>> introduced another case which I'm not quite sure about, which is kind of
>> *logic
>> incompatibility*.
>>
>> In detail, an ApplicationMasterProtocol API is going to throw an exception
>> which is not expected before. The exception is a sub-class of
>> YarnException, such that it doesn't need any method signature change, and
>> won't break any binary/source compatibility. However, the exception is not
>> expected before, but needs to be treated specially at the AM side. Not
>> being aware of the newly introduced exception, the existing YARN
>> applications' AM may not handle it exception properly, and is at the risk
>> of being broken on a new YARN cluster after this change.
>>
>> An additional thought around this problem is that the change of what
>> exception is to throw under what situation may be considered as a *soft
>> *method
>> signature change, because we're supposed to write this javadoc to tell the
>> users (though we didn't it well in Hadoop), and users refer to it to guide
>> how to handle the exception.
>>
>> In a more generalized form, let's assume we have a method, which behaves
>> as
>> A, in release 1.0. However, in release 2.0, the method signature has kept
>> the same, while its behavior is altered from A to B. A and B are different
>> behaviors. In this case, do we consider it as an incompatible change?
>>
>> I think it's somewhat a common issue, such that I raise it on the mailing
>> list. Please share your ideas.
>>
>> Thanks,
>> Zhijie
>>
>> --
>> Zhijie Shen
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified
>> that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender
>> immediately
>> and delete it from your system. Thank You.
>>
>
>
Re: Hadoop Code of Incompatible Changes
Posted by Sandy Ryza <sa...@cloudera.com>.
Eli pointed out to me that this is the up-to-date compatibility guide:
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Compatibility.html
Thanks,
Sandy
On Tue, Jul 29, 2014 at 9:39 AM, Sandy Ryza <sa...@cloudera.com> wrote:
> Hi Zhijie,
>
> The Hadoop compatibility guide mentions this as "semantic compatibility":
> http://wiki.apache.org/hadoop/Compatibility
>
> My interpretation of the section is that we can't change the behavior of
> public APIs unless we're fixing buggy behavior. If the change could break
> an existing application that's behaving reasonably with respect to the old
> API, it's an incompatible change.
>
> -Sandy
>
>
>
> On Tue, Jul 29, 2014 at 9:26 AM, Zhijie Shen <zs...@hortonworks.com>
> wrote:
>
>> Hi folks,
>>
>> Recently we have a conversation on YARN-2209 about the incompatible
>> changes
>> over releases. For those API changes that will break binary compatibility,
>> source compatibility towards the existing API users, we've already had a
>> rather clear picture about what we should do. However, YARN-2209 has
>> introduced another case which I'm not quite sure about, which is kind of
>> *logic
>> incompatibility*.
>>
>> In detail, an ApplicationMasterProtocol API is going to throw an exception
>> which is not expected before. The exception is a sub-class of
>> YarnException, such that it doesn't need any method signature change, and
>> won't break any binary/source compatibility. However, the exception is not
>> expected before, but needs to be treated specially at the AM side. Not
>> being aware of the newly introduced exception, the existing YARN
>> applications' AM may not handle it exception properly, and is at the risk
>> of being broken on a new YARN cluster after this change.
>>
>> An additional thought around this problem is that the change of what
>> exception is to throw under what situation may be considered as a *soft
>> *method
>> signature change, because we're supposed to write this javadoc to tell the
>> users (though we didn't it well in Hadoop), and users refer to it to guide
>> how to handle the exception.
>>
>> In a more generalized form, let's assume we have a method, which behaves
>> as
>> A, in release 1.0. However, in release 2.0, the method signature has kept
>> the same, while its behavior is altered from A to B. A and B are different
>> behaviors. In this case, do we consider it as an incompatible change?
>>
>> I think it's somewhat a common issue, such that I raise it on the mailing
>> list. Please share your ideas.
>>
>> Thanks,
>> Zhijie
>>
>> --
>> Zhijie Shen
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified
>> that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender
>> immediately
>> and delete it from your system. Thank You.
>>
>
>
Re: Hadoop Code of Incompatible Changes
Posted by Sandy Ryza <sa...@cloudera.com>.
Hi Zhijie,
The Hadoop compatibility guide mentions this as "semantic compatibility":
http://wiki.apache.org/hadoop/Compatibility
My interpretation of the section is that we can't change the behavior of
public APIs unless we're fixing buggy behavior. If the change could break
an existing application that's behaving reasonably with respect to the old
API, it's an incompatible change.
-Sandy
On Tue, Jul 29, 2014 at 9:26 AM, Zhijie Shen <zs...@hortonworks.com> wrote:
> Hi folks,
>
> Recently we have a conversation on YARN-2209 about the incompatible changes
> over releases. For those API changes that will break binary compatibility,
> source compatibility towards the existing API users, we've already had a
> rather clear picture about what we should do. However, YARN-2209 has
> introduced another case which I'm not quite sure about, which is kind of
> *logic
> incompatibility*.
>
> In detail, an ApplicationMasterProtocol API is going to throw an exception
> which is not expected before. The exception is a sub-class of
> YarnException, such that it doesn't need any method signature change, and
> won't break any binary/source compatibility. However, the exception is not
> expected before, but needs to be treated specially at the AM side. Not
> being aware of the newly introduced exception, the existing YARN
> applications' AM may not handle it exception properly, and is at the risk
> of being broken on a new YARN cluster after this change.
>
> An additional thought around this problem is that the change of what
> exception is to throw under what situation may be considered as a *soft
> *method
> signature change, because we're supposed to write this javadoc to tell the
> users (though we didn't it well in Hadoop), and users refer to it to guide
> how to handle the exception.
>
> In a more generalized form, let's assume we have a method, which behaves as
> A, in release 1.0. However, in release 2.0, the method signature has kept
> the same, while its behavior is altered from A to B. A and B are different
> behaviors. In this case, do we consider it as an incompatible change?
>
> I think it's somewhat a common issue, such that I raise it on the mailing
> list. Please share your ideas.
>
> Thanks,
> Zhijie
>
> --
> Zhijie Shen
> Hortonworks Inc.
> http://hortonworks.com/
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Hadoop Code of Incompatible Changes
Posted by Sandy Ryza <sa...@cloudera.com>.
Hi Zhijie,
The Hadoop compatibility guide mentions this as "semantic compatibility":
http://wiki.apache.org/hadoop/Compatibility
My interpretation of the section is that we can't change the behavior of
public APIs unless we're fixing buggy behavior. If the change could break
an existing application that's behaving reasonably with respect to the old
API, it's an incompatible change.
-Sandy
On Tue, Jul 29, 2014 at 9:26 AM, Zhijie Shen <zs...@hortonworks.com> wrote:
> Hi folks,
>
> Recently we have a conversation on YARN-2209 about the incompatible changes
> over releases. For those API changes that will break binary compatibility,
> source compatibility towards the existing API users, we've already had a
> rather clear picture about what we should do. However, YARN-2209 has
> introduced another case which I'm not quite sure about, which is kind of
> *logic
> incompatibility*.
>
> In detail, an ApplicationMasterProtocol API is going to throw an exception
> which is not expected before. The exception is a sub-class of
> YarnException, such that it doesn't need any method signature change, and
> won't break any binary/source compatibility. However, the exception is not
> expected before, but needs to be treated specially at the AM side. Not
> being aware of the newly introduced exception, the existing YARN
> applications' AM may not handle it exception properly, and is at the risk
> of being broken on a new YARN cluster after this change.
>
> An additional thought around this problem is that the change of what
> exception is to throw under what situation may be considered as a *soft
> *method
> signature change, because we're supposed to write this javadoc to tell the
> users (though we didn't it well in Hadoop), and users refer to it to guide
> how to handle the exception.
>
> In a more generalized form, let's assume we have a method, which behaves as
> A, in release 1.0. However, in release 2.0, the method signature has kept
> the same, while its behavior is altered from A to B. A and B are different
> behaviors. In this case, do we consider it as an incompatible change?
>
> I think it's somewhat a common issue, such that I raise it on the mailing
> list. Please share your ideas.
>
> Thanks,
> Zhijie
>
> --
> Zhijie Shen
> Hortonworks Inc.
> http://hortonworks.com/
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>