You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Karthik Kambatla <ka...@cloudera.com> on 2013/04/22 23:00:32 UTC
Compatibility in Apache Hadoop
Hadoop devs,
Eli and I have been working on a writeup on compatibility for developers,
downstream projects and end users. The wiki page -
http://wiki.apache.org/hadoop/Compatibility - is a strawman effort towards
consolidating prior discussions and decisions; including the API
classification effort on HADOOP-5073, Sanjay’s work on Hadoop 1.0
compatibility (HADOOP-5071), and the roadmap page on the Hadoop wiki. The
goal is to identify the the relevant interfaces and concerns for downstream
projects and users. In cases where the community has explicitly agreed upon
a policy (e.g. HADOOP-5073: API classification), call out that policy.
This doc does not intend to propose new policies. The idea is to have one
document that outlines the various compatibility concerns (lots of areas
beyond API compatibility), captures the respective policies that exist, and
if we want to define policies for the items where it’s not clear we have
something to iterate on.
The first draft just lists the types of compatibility. In the next step, we
can add existing policies and subsequently work towards policies for others.
Would love to incorporate your feedback on the first draft.
Thanks
Karthik
Re: Compatibility in Apache Hadoop
Posted by Karthik Kambatla <ka...@cloudera.com>.
On Tue, Apr 23, 2013 at 9:00 AM, Steve Loughran <st...@hortonworks.com>wrote:
> On 22 April 2013 18:32, Eli Collins <el...@cloudera.com> wrote:
>
> > On Mon, Apr 22, 2013 at 5:42 PM, Steve Loughran <st...@hortonworks.com>
> > wrote:
> >
> > >
> > > There's a separate issue that says "we make some guarantee that the
> > > behaviour of a interface remains consistent over versions", which is
> hard
> > > to do without some rigorous definition of what the expected behaviour
> of
> > an
> > > implementation should be.
> >
> >
> > Good point, Steve. I've assumed the semantics of the API had to
> > respect the attribute (eg changing the semantics of FileSystem#close
> > would be an incompatible change, since this is a public/stable API,
> > even if the new semantics are arguably better). But you're right,
> > unless we've actually defined what the semantics of the APIs are it's
> > hard to say if we've materially changed them. How about adding a new
> > section on the page and calling that out explicitly?
> >
>
> +1.
>
> Maybe we should list which bits we consider both well specified and covered
> with tests that verify the implementations in our svn match that
> specification.
>
>
> >
> > In practice I think we'll have to take semantics case by case, clearly
> > define the semantics we care about better in the javadocs (for the
> > major end user-facing classes at least, calling out both intended
> > behavior and behavior that's meant to be undefined) and using
> > individual judgement elsewhere. For example, HDFS-4156 changed
> > DataInputStream#seek to throw an IOE if you seek to a negative offset,
> > instead of succeeding then resulting in an NPE on the next access.
> >
>
> I'd seen that the DFS seek was the best implementation, but hadn't seen the
> cause. The other ones (especially the Buffered one that goes in front of
> most others) is much weaker
>
>
> > That's an incompatible change in terms of semantics, but not semantics
> > intended by the author, or likely semantics programs depend on.
> >
>
> That's a key problem: what do people depend on? A lot of the junit tests
> depended on ordering of methods, after all
>
>
> > However if a change made FileSystem#close three times slower, this
> > perhaps a smaller semantic change (eg doesn't change what exceptions
> > get thrown) but probably much less tolerable for end users.
> >
>
> You know that the blobstores all buffer their data so that
>
> 1. flush() is a no-op
> 2. the write takes place on close()
>
> #1 changes durability expectations, while #2 means the time to close() is
> O(data)*O(latency); P(fail) scales with time and distance, and as lots of
> code swallows exceptions on close, those failures may even miss.
>
> then there's the assumption that rename is atomic, which MapReduce depends
> on.
>
>
> >
> > In any case, even if we get an 80% solution to the semantics issue
> > we'll probably be in good shape for v2 GA if we can sort out the
> > remaining topics. See any other topics missing? Once the overall
> > outline is in shape it make sense to annotate the page with the
> > current policy (if there's already consensus on one), and identifying
> > areas where we need to come up with a policy or are leaving TBD.
> > Currently this is a source of confusion for new developers, some
> > downstream projects and users.
> >
> >
> How about
>
> "semantic compatibility" : we strive to ensure that the behavior of APIs
> remains consistent over versions, though changes for correctness may result
> in changes in behavior That is: if you relied on something which we
> consider to be a bug, it may get fixed.
>
> We are in the process of specifying some APIs more rigorously, enhancing
> our test suites to verify compliance with the specification, effectively
> creating a formal specification for the subset of behaviors that can be
> easily tested. We welcome involvement in this process, from both users and
> implementors of our APIs.
>
+1. Thanks Steve.
Added Semantic compatibility to the wiki.
>
> If you are concerned about compatibility at any level, we strongly
> encourage you follow the Hadoop developer mailing lists, and track on JIRA
> issues that may concern you. You are also strongly advised to verify that
> your code works against beta releases of forthcoming Hadoop versions, as
> that is a time in which identified regressions can be corrected rapidly -if
> you only test when a new final release ships, the time to fix is likely to
> be at least three months. "
>
Added to the introduction as it applies to all compatibility.
Re: Compatibility in Apache Hadoop
Posted by Andrew Purtell <ap...@apache.org>.
API jars only containing interfaces would work. They would be small and
lightweight, and not carry dependencies other than other API modules
presumably.
Some additional details of factoring out the codecs might be debatable
(e.g. native code loading) but I do feel it is a useful motivating example
for such API vs implementation separation.
On Tue, Apr 23, 2013 at 11:50 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> Andrew,
>
> Or with a twist, why not break/consolidate things as follows?
>
> common API
> common IMPL
> hdfs CLIENT IMPL
> hdfs SERVER IMPL
> hdfs TOOLS
> <other filesystems> CLIENT
> yarn API
> yarn CLIENT IMPL
> yarn SERVER IMPL
> yarn TOOLS
> mapred API
> mapred IMPL
> mapred TOOLS
>
> IMO, this would help significantly to reduce dependency hell (like bringing
> servlet, jetty JAR to a hadoop client app).
>
> Thx
>
> On Tue, Apr 23, 2013 at 11:32 AM, Andrew Purtell <apurtell@apache.org
> >wrote:
>
> > At the risk of hijacking this conversation a bit, what do you think of
> the
> > notion of moving interfaces like Seekable and PositionedReadable into a
> new
> > foundational Maven module, perhaps just for such interfaces that define
> and
> > tag support for core semantics, as their details are better defined and
> > documented? I was involved in a discussion today considering factoring
> out
> > the codecs so other ecosystem projects might pull in only codec code.
> > Similar to how hadoop-auth is slender and has a useful servlet filter
> > implementing SPEGNO authentication, and so it is pulled into various
> > places, and can even be used with Hadoop 1. The only thing preventing a
> > clean separation of codecs like this is imports of Seekable and
> > PositionedReadable. But these define behavior, they don't implement it.
> >
> >
> > On Tue, Apr 23, 2013 at 9:00 AM, Steve Loughran <stevel@hortonworks.com
> > >wrote:
> >
> > > On 22 April 2013 18:32, Eli Collins <el...@cloudera.com> wrote:
> > >
> > > > On Mon, Apr 22, 2013 at 5:42 PM, Steve Loughran <
> > stevel@hortonworks.com>
> > > > wrote:
> > > >
> > > > >
> > > > > There's a separate issue that says "we make some guarantee that the
> > > > > behaviour of a interface remains consistent over versions", which
> is
> > > hard
> > > > > to do without some rigorous definition of what the expected
> behaviour
> > > of
> > > > an
> > > > > implementation should be.
> > > >
> > > >
> > > > Good point, Steve. I've assumed the semantics of the API had to
> > > > respect the attribute (eg changing the semantics of FileSystem#close
> > > > would be an incompatible change, since this is a public/stable API,
> > > > even if the new semantics are arguably better). But you're right,
> > > > unless we've actually defined what the semantics of the APIs are it's
> > > > hard to say if we've materially changed them. How about adding a new
> > > > section on the page and calling that out explicitly?
> > > >
> > >
> > > +1.
> > >
> > > Maybe we should list which bits we consider both well specified and
> > covered
> > > with tests that verify the implementations in our svn match that
> > > specification.
> > >
> > >
> > > >
> > > > In practice I think we'll have to take semantics case by case,
> clearly
> > > > define the semantics we care about better in the javadocs (for the
> > > > major end user-facing classes at least, calling out both intended
> > > > behavior and behavior that's meant to be undefined) and using
> > > > individual judgement elsewhere. For example, HDFS-4156 changed
> > > > DataInputStream#seek to throw an IOE if you seek to a negative
> offset,
> > > > instead of succeeding then resulting in an NPE on the next access.
> > > >
> > >
> > > I'd seen that the DFS seek was the best implementation, but hadn't seen
> > the
> > > cause. The other ones (especially the Buffered one that goes in front
> of
> > > most others) is much weaker
> > >
> > >
> > > > That's an incompatible change in terms of semantics, but not
> semantics
> > > > intended by the author, or likely semantics programs depend on.
> > > >
> > >
> > > That's a key problem: what do people depend on? A lot of the junit
> tests
> > > depended on ordering of methods, after all
> > >
> > >
> > > > However if a change made FileSystem#close three times slower, this
> > > > perhaps a smaller semantic change (eg doesn't change what exceptions
> > > > get thrown) but probably much less tolerable for end users.
> > > >
> > >
> > > You know that the blobstores all buffer their data so that
> > >
> > > 1. flush() is a no-op
> > > 2. the write takes place on close()
> > >
> > > #1 changes durability expectations, while #2 means the time to close()
> is
> > > O(data)*O(latency); P(fail) scales with time and distance, and as lots
> of
> > > code swallows exceptions on close, those failures may even miss.
> > >
> > > then there's the assumption that rename is atomic, which MapReduce
> > depends
> > > on.
> > >
> > >
> > > >
> > > > In any case, even if we get an 80% solution to the semantics issue
> > > > we'll probably be in good shape for v2 GA if we can sort out the
> > > > remaining topics. See any other topics missing? Once the overall
> > > > outline is in shape it make sense to annotate the page with the
> > > > current policy (if there's already consensus on one), and identifying
> > > > areas where we need to come up with a policy or are leaving TBD.
> > > > Currently this is a source of confusion for new developers, some
> > > > downstream projects and users.
> > > >
> > > >
> > > How about
> > >
> > > "semantic compatibility" : we strive to ensure that the behavior of
> APIs
> > > remains consistent over versions, though changes for correctness may
> > result
> > > in changes in behavior That is: if you relied on something which we
> > > consider to be a bug, it may get fixed.
> > >
> > > We are in the process of specifying some APIs more rigorously,
> enhancing
> > > our test suites to verify compliance with the specification,
> effectively
> > > creating a formal specification for the subset of behaviors that can be
> > > easily tested. We welcome involvement in this process, from both users
> > and
> > > implementors of our APIs.
> > >
> > > If you are concerned about compatibility at any level, we strongly
> > > encourage you follow the Hadoop developer mailing lists, and track on
> > JIRA
> > > issues that may concern you. You are also strongly advised to verify
> that
> > > your code works against beta releases of forthcoming Hadoop versions,
> as
> > > that is a time in which identified regressions can be corrected rapidly
> > -if
> > > you only test when a new final release ships, the time to fix is likely
> > to
> > > be at least three months. "
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>
>
>
> --
> Alejandro
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: Compatibility in Apache Hadoop
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Andrew,
Or with a twist, why not break/consolidate things as follows?
common API
common IMPL
hdfs CLIENT IMPL
hdfs SERVER IMPL
hdfs TOOLS
<other filesystems> CLIENT
yarn API
yarn CLIENT IMPL
yarn SERVER IMPL
yarn TOOLS
mapred API
mapred IMPL
mapred TOOLS
IMO, this would help significantly to reduce dependency hell (like bringing
servlet, jetty JAR to a hadoop client app).
Thx
On Tue, Apr 23, 2013 at 11:32 AM, Andrew Purtell <ap...@apache.org>wrote:
> At the risk of hijacking this conversation a bit, what do you think of the
> notion of moving interfaces like Seekable and PositionedReadable into a new
> foundational Maven module, perhaps just for such interfaces that define and
> tag support for core semantics, as their details are better defined and
> documented? I was involved in a discussion today considering factoring out
> the codecs so other ecosystem projects might pull in only codec code.
> Similar to how hadoop-auth is slender and has a useful servlet filter
> implementing SPEGNO authentication, and so it is pulled into various
> places, and can even be used with Hadoop 1. The only thing preventing a
> clean separation of codecs like this is imports of Seekable and
> PositionedReadable. But these define behavior, they don't implement it.
>
>
> On Tue, Apr 23, 2013 at 9:00 AM, Steve Loughran <stevel@hortonworks.com
> >wrote:
>
> > On 22 April 2013 18:32, Eli Collins <el...@cloudera.com> wrote:
> >
> > > On Mon, Apr 22, 2013 at 5:42 PM, Steve Loughran <
> stevel@hortonworks.com>
> > > wrote:
> > >
> > > >
> > > > There's a separate issue that says "we make some guarantee that the
> > > > behaviour of a interface remains consistent over versions", which is
> > hard
> > > > to do without some rigorous definition of what the expected behaviour
> > of
> > > an
> > > > implementation should be.
> > >
> > >
> > > Good point, Steve. I've assumed the semantics of the API had to
> > > respect the attribute (eg changing the semantics of FileSystem#close
> > > would be an incompatible change, since this is a public/stable API,
> > > even if the new semantics are arguably better). But you're right,
> > > unless we've actually defined what the semantics of the APIs are it's
> > > hard to say if we've materially changed them. How about adding a new
> > > section on the page and calling that out explicitly?
> > >
> >
> > +1.
> >
> > Maybe we should list which bits we consider both well specified and
> covered
> > with tests that verify the implementations in our svn match that
> > specification.
> >
> >
> > >
> > > In practice I think we'll have to take semantics case by case, clearly
> > > define the semantics we care about better in the javadocs (for the
> > > major end user-facing classes at least, calling out both intended
> > > behavior and behavior that's meant to be undefined) and using
> > > individual judgement elsewhere. For example, HDFS-4156 changed
> > > DataInputStream#seek to throw an IOE if you seek to a negative offset,
> > > instead of succeeding then resulting in an NPE on the next access.
> > >
> >
> > I'd seen that the DFS seek was the best implementation, but hadn't seen
> the
> > cause. The other ones (especially the Buffered one that goes in front of
> > most others) is much weaker
> >
> >
> > > That's an incompatible change in terms of semantics, but not semantics
> > > intended by the author, or likely semantics programs depend on.
> > >
> >
> > That's a key problem: what do people depend on? A lot of the junit tests
> > depended on ordering of methods, after all
> >
> >
> > > However if a change made FileSystem#close three times slower, this
> > > perhaps a smaller semantic change (eg doesn't change what exceptions
> > > get thrown) but probably much less tolerable for end users.
> > >
> >
> > You know that the blobstores all buffer their data so that
> >
> > 1. flush() is a no-op
> > 2. the write takes place on close()
> >
> > #1 changes durability expectations, while #2 means the time to close() is
> > O(data)*O(latency); P(fail) scales with time and distance, and as lots of
> > code swallows exceptions on close, those failures may even miss.
> >
> > then there's the assumption that rename is atomic, which MapReduce
> depends
> > on.
> >
> >
> > >
> > > In any case, even if we get an 80% solution to the semantics issue
> > > we'll probably be in good shape for v2 GA if we can sort out the
> > > remaining topics. See any other topics missing? Once the overall
> > > outline is in shape it make sense to annotate the page with the
> > > current policy (if there's already consensus on one), and identifying
> > > areas where we need to come up with a policy or are leaving TBD.
> > > Currently this is a source of confusion for new developers, some
> > > downstream projects and users.
> > >
> > >
> > How about
> >
> > "semantic compatibility" : we strive to ensure that the behavior of APIs
> > remains consistent over versions, though changes for correctness may
> result
> > in changes in behavior That is: if you relied on something which we
> > consider to be a bug, it may get fixed.
> >
> > We are in the process of specifying some APIs more rigorously, enhancing
> > our test suites to verify compliance with the specification, effectively
> > creating a formal specification for the subset of behaviors that can be
> > easily tested. We welcome involvement in this process, from both users
> and
> > implementors of our APIs.
> >
> > If you are concerned about compatibility at any level, we strongly
> > encourage you follow the Hadoop developer mailing lists, and track on
> JIRA
> > issues that may concern you. You are also strongly advised to verify that
> > your code works against beta releases of forthcoming Hadoop versions, as
> > that is a time in which identified regressions can be corrected rapidly
> -if
> > you only test when a new final release ships, the time to fix is likely
> to
> > be at least three months. "
> >
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
--
Alejandro
Re: Compatibility in Apache Hadoop
Posted by Steve Loughran <st...@hortonworks.com>.
On 23 April 2013 11:32, Andrew Purtell <ap...@apache.org> wrote:
> At the risk of hijacking this conversation a bit, what do you think of the
> notion of moving interfaces like Seekable and PositionedReadable into a new
> foundational Maven module, perhaps just for such interfaces that define and
> tag support for core semantics, as their details are better defined and
> documented? I was involved in a discussion today considering factoring out
> the codecs so other ecosystem projects might pull in only codec code.
> Similar to how hadoop-auth is slender and has a useful servlet filter
> implementing SPEGNO authentication, and so it is pulled into various
> places, and can even be used with Hadoop 1. The only thing preventing a
> clean separation of codecs like this is imports of Seekable and
> PositionedReadable. But these define behavior, they don't implement it.
>
>
I'd be against any more (mandatory) JAR files. Life is complicated enough,
and I would worry about version conflict with a hadoop-fs-api file and any
hadoop-1.x releases (the RemoteIterator stuff is new).
Other people may have other opinions; I can see how it'd make codecs more
portable, and reduce dependencies to their minimum.
-steve
Re: Compatibility in Apache Hadoop
Posted by Andrew Purtell <ap...@apache.org>.
At the risk of hijacking this conversation a bit, what do you think of the
notion of moving interfaces like Seekable and PositionedReadable into a new
foundational Maven module, perhaps just for such interfaces that define and
tag support for core semantics, as their details are better defined and
documented? I was involved in a discussion today considering factoring out
the codecs so other ecosystem projects might pull in only codec code.
Similar to how hadoop-auth is slender and has a useful servlet filter
implementing SPEGNO authentication, and so it is pulled into various
places, and can even be used with Hadoop 1. The only thing preventing a
clean separation of codecs like this is imports of Seekable and
PositionedReadable. But these define behavior, they don't implement it.
On Tue, Apr 23, 2013 at 9:00 AM, Steve Loughran <st...@hortonworks.com>wrote:
> On 22 April 2013 18:32, Eli Collins <el...@cloudera.com> wrote:
>
> > On Mon, Apr 22, 2013 at 5:42 PM, Steve Loughran <st...@hortonworks.com>
> > wrote:
> >
> > >
> > > There's a separate issue that says "we make some guarantee that the
> > > behaviour of a interface remains consistent over versions", which is
> hard
> > > to do without some rigorous definition of what the expected behaviour
> of
> > an
> > > implementation should be.
> >
> >
> > Good point, Steve. I've assumed the semantics of the API had to
> > respect the attribute (eg changing the semantics of FileSystem#close
> > would be an incompatible change, since this is a public/stable API,
> > even if the new semantics are arguably better). But you're right,
> > unless we've actually defined what the semantics of the APIs are it's
> > hard to say if we've materially changed them. How about adding a new
> > section on the page and calling that out explicitly?
> >
>
> +1.
>
> Maybe we should list which bits we consider both well specified and covered
> with tests that verify the implementations in our svn match that
> specification.
>
>
> >
> > In practice I think we'll have to take semantics case by case, clearly
> > define the semantics we care about better in the javadocs (for the
> > major end user-facing classes at least, calling out both intended
> > behavior and behavior that's meant to be undefined) and using
> > individual judgement elsewhere. For example, HDFS-4156 changed
> > DataInputStream#seek to throw an IOE if you seek to a negative offset,
> > instead of succeeding then resulting in an NPE on the next access.
> >
>
> I'd seen that the DFS seek was the best implementation, but hadn't seen the
> cause. The other ones (especially the Buffered one that goes in front of
> most others) is much weaker
>
>
> > That's an incompatible change in terms of semantics, but not semantics
> > intended by the author, or likely semantics programs depend on.
> >
>
> That's a key problem: what do people depend on? A lot of the junit tests
> depended on ordering of methods, after all
>
>
> > However if a change made FileSystem#close three times slower, this
> > perhaps a smaller semantic change (eg doesn't change what exceptions
> > get thrown) but probably much less tolerable for end users.
> >
>
> You know that the blobstores all buffer their data so that
>
> 1. flush() is a no-op
> 2. the write takes place on close()
>
> #1 changes durability expectations, while #2 means the time to close() is
> O(data)*O(latency); P(fail) scales with time and distance, and as lots of
> code swallows exceptions on close, those failures may even miss.
>
> then there's the assumption that rename is atomic, which MapReduce depends
> on.
>
>
> >
> > In any case, even if we get an 80% solution to the semantics issue
> > we'll probably be in good shape for v2 GA if we can sort out the
> > remaining topics. See any other topics missing? Once the overall
> > outline is in shape it make sense to annotate the page with the
> > current policy (if there's already consensus on one), and identifying
> > areas where we need to come up with a policy or are leaving TBD.
> > Currently this is a source of confusion for new developers, some
> > downstream projects and users.
> >
> >
> How about
>
> "semantic compatibility" : we strive to ensure that the behavior of APIs
> remains consistent over versions, though changes for correctness may result
> in changes in behavior That is: if you relied on something which we
> consider to be a bug, it may get fixed.
>
> We are in the process of specifying some APIs more rigorously, enhancing
> our test suites to verify compliance with the specification, effectively
> creating a formal specification for the subset of behaviors that can be
> easily tested. We welcome involvement in this process, from both users and
> implementors of our APIs.
>
> If you are concerned about compatibility at any level, we strongly
> encourage you follow the Hadoop developer mailing lists, and track on JIRA
> issues that may concern you. You are also strongly advised to verify that
> your code works against beta releases of forthcoming Hadoop versions, as
> that is a time in which identified regressions can be corrected rapidly -if
> you only test when a new final release ships, the time to fix is likely to
> be at least three months. "
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: Compatibility in Apache Hadoop
Posted by Steve Loughran <st...@hortonworks.com>.
On 23 April 2013 09:00, Steve Loughran <st...@hortonworks.com> wrote:
>
>
> On 22 April 2013 18:32, Eli Collins <el...@cloudera.com> wrote:
>
>>
>>
>
>> However if a change made FileSystem#close three times slower, this
>> perhaps a smaller semantic change (eg doesn't change what exceptions
>> get thrown) but probably much less tolerable for end users.
>>
>
> You know that the blobstores all buffer their data so that
>
> 1. flush() is a no-op
> 2. the write takes place on close()
>
> #1 changes durability expectations, while #2 means the time to close() is
> O(data)*O(latency); P(fail) scales with time and distance, and as lots of
> code swallows exceptions on close, those failures may even miss.
>
>
for the curious, there are some tests that I plan to get into bigtop that
not only generate various large files, they collect stats on the duration
of operations. On a remote blobstore, its close() that takes most of the
time, even for only a few MB of data
2013-04-23 11:23:21,911 [main] INFO tools.DataGenerator (?:call(?)) -
Generating 100000 lines of data
2013-04-23 11:23:22,122 [main] DEBUG snative.SwiftNativeOutputStream
(SwiftNativeOutputStream.java:uploadOnClose(146)) - Closing write of file
/tmp/data/massive/csv/data-0014.csv;
localfile=target/build/test/output-4786965937321057354.tmp of length 3301583
2013-04-23 11:23:23,437 [main] INFO generate.GenerateManyCSVFilesTest
(?:call(?)) - Total time = 0:02:031; create time=0:00:505; write time
=0:00:210; close time = 0:01:316 partitions=0
Re: Compatibility in Apache Hadoop
Posted by Steve Loughran <st...@hortonworks.com>.
On 22 April 2013 18:32, Eli Collins <el...@cloudera.com> wrote:
> On Mon, Apr 22, 2013 at 5:42 PM, Steve Loughran <st...@hortonworks.com>
> wrote:
>
> >
> > There's a separate issue that says "we make some guarantee that the
> > behaviour of a interface remains consistent over versions", which is hard
> > to do without some rigorous definition of what the expected behaviour of
> an
> > implementation should be.
>
>
> Good point, Steve. I've assumed the semantics of the API had to
> respect the attribute (eg changing the semantics of FileSystem#close
> would be an incompatible change, since this is a public/stable API,
> even if the new semantics are arguably better). But you're right,
> unless we've actually defined what the semantics of the APIs are it's
> hard to say if we've materially changed them. How about adding a new
> section on the page and calling that out explicitly?
>
+1.
Maybe we should list which bits we consider both well specified and covered
with tests that verify the implementations in our svn match that
specification.
>
> In practice I think we'll have to take semantics case by case, clearly
> define the semantics we care about better in the javadocs (for the
> major end user-facing classes at least, calling out both intended
> behavior and behavior that's meant to be undefined) and using
> individual judgement elsewhere. For example, HDFS-4156 changed
> DataInputStream#seek to throw an IOE if you seek to a negative offset,
> instead of succeeding then resulting in an NPE on the next access.
>
I'd seen that the DFS seek was the best implementation, but hadn't seen the
cause. The other ones (especially the Buffered one that goes in front of
most others) is much weaker
> That's an incompatible change in terms of semantics, but not semantics
> intended by the author, or likely semantics programs depend on.
>
That's a key problem: what do people depend on? A lot of the junit tests
depended on ordering of methods, after all
> However if a change made FileSystem#close three times slower, this
> perhaps a smaller semantic change (eg doesn't change what exceptions
> get thrown) but probably much less tolerable for end users.
>
You know that the blobstores all buffer their data so that
1. flush() is a no-op
2. the write takes place on close()
#1 changes durability expectations, while #2 means the time to close() is
O(data)*O(latency); P(fail) scales with time and distance, and as lots of
code swallows exceptions on close, those failures may even miss.
then there's the assumption that rename is atomic, which MapReduce depends
on.
>
> In any case, even if we get an 80% solution to the semantics issue
> we'll probably be in good shape for v2 GA if we can sort out the
> remaining topics. See any other topics missing? Once the overall
> outline is in shape it make sense to annotate the page with the
> current policy (if there's already consensus on one), and identifying
> areas where we need to come up with a policy or are leaving TBD.
> Currently this is a source of confusion for new developers, some
> downstream projects and users.
>
>
How about
"semantic compatibility" : we strive to ensure that the behavior of APIs
remains consistent over versions, though changes for correctness may result
in changes in behavior That is: if you relied on something which we
consider to be a bug, it may get fixed.
We are in the process of specifying some APIs more rigorously, enhancing
our test suites to verify compliance with the specification, effectively
creating a formal specification for the subset of behaviors that can be
easily tested. We welcome involvement in this process, from both users and
implementors of our APIs.
If you are concerned about compatibility at any level, we strongly
encourage you follow the Hadoop developer mailing lists, and track on JIRA
issues that may concern you. You are also strongly advised to verify that
your code works against beta releases of forthcoming Hadoop versions, as
that is a time in which identified regressions can be corrected rapidly -if
you only test when a new final release ships, the time to fix is likely to
be at least three months. "
Fwd: Compatibility in Apache Hadoop
Posted by Eli Collins <el...@cloudera.com>.
On Mon, Apr 22, 2013 at 5:42 PM, Steve Loughran <st...@hortonworks.com> wrote:
>
> On 22 April 2013 14:00, Karthik Kambatla <ka...@cloudera.com> wrote:
>
> > Hadoop devs,
> >
> >
> > This doc does not intend to propose new policies. The idea is to have one
> > document that outlines the various compatibility concerns (lots of areas
> > beyond API compatibility), captures the respective policies that exist, and
> > if we want to define policies for the items where it’s not clear we have
> > something to iterate on.
> >
> > The first draft just lists the types of compatibility. In the next step, we
> > can add existing policies and subsequently work towards policies for
> > others.
> >
> >
> I don't see -yet- a definition of compatible at the API signature level vs
> semantics level.
>
> The @ interface attributes say "these methods are
> internal/external/stable/unstable (there's also @VisibleForTesting,that
> comes out of guava (yes?).
>
> There's a separate issue that says "we make some guarantee that the
> behaviour of a interface remains consistent over versions", which is hard
> to do without some rigorous definition of what the expected behaviour of an
> implementation should be.
Good point, Steve. I've assumed the semantics of the API had to
respect the attribute (eg changing the semantics of FileSystem#close
would be an incompatible change, since this is a public/stable API,
even if the new semantics are arguably better). But you're right,
unless we've actually defined what the semantics of the APIs are it's
hard to say if we've materially changed them. How about adding a new
section on the page and calling that out explicitly?
In practice I think we'll have to take semantics case by case, clearly
define the semantics we care about better in the javadocs (for the
major end user-facing classes at least, calling out both intended
behavior and behavior that's meant to be undefined) and using
individual judgement elsewhere. For example, HDFS-4156 changed
DataInputStream#seek to throw an IOE if you seek to a negative offset,
instead of succeeding then resulting in an NPE on the next access.
That's an incompatible change in terms of semantics, but not semantics
intended by the author, or likely semantics programs depend on.
However if a change made FileSystem#close three times slower, this
perhaps a smaller semantic change (eg doesn't change what exceptions
get thrown) but probably much less tolerable for end users.
In any case, even if we get an 80% solution to the semantics issue
we'll probably be in good shape for v2 GA if we can sort out the
remaining topics. See any other topics missing? Once the overall
outline is in shape it make sense to annotate the page with the
current policy (if there's already consensus on one), and identifying
areas where we need to come up with a policy or are leaving TBD.
Currently this is a source of confusion for new developers, some
downstream projects and users.
Thanks,
Eli
Re: Compatibility in Apache Hadoop
Posted by Steve Loughran <st...@hortonworks.com>.
On 22 April 2013 14:00, Karthik Kambatla <ka...@cloudera.com> wrote:
> Hadoop devs,
>
>
> This doc does not intend to propose new policies. The idea is to have one
> document that outlines the various compatibility concerns (lots of areas
> beyond API compatibility), captures the respective policies that exist, and
> if we want to define policies for the items where it’s not clear we have
> something to iterate on.
>
> The first draft just lists the types of compatibility. In the next step, we
> can add existing policies and subsequently work towards policies for
> others.
>
>
I don't see -yet- a definition of compatible at the API signature level vs
semantics level.
The @ interface attributes say "these methods are
internal/external/stable/unstable (there's also @VisibleForTesting,that
comes out of guava (yes?).
There's a separate issue that says "we make some guarantee that the
behaviour of a interface remains consistent over versions", which is hard
to do without some rigorous definition of what the expected behaviour of an
implementation should be. Even then, there are epiphenomena whose behaviour
isn't a direct part of the specification but which people ended up relying
on. As an example, look how many junit tests have broken in Java7 after
they changed the order in which methods were enumerated. The new ordering
fulfils the java lan specification "no order is guaranteed", but the
pre-java6 implementation offered "same order as you wrote them in the
file". In Hadoop, there's a lot of code that assumes that close() of an
output stream happens within a short period of time, and if you change that
some things break, but nowhere does {{OutputStream.close()}} ever say
"completes within 30s".
Interface semantics are far more important than simple signatures, and they
are the hardest to guarantee -especially when we don't have rigorous enough
definitions of what implementations should really do.
As an example what does the method Seekable.seek() do?
It's header says
/**
* Seek to the given offset from the start of the file.
* The next read() will be from that location. Can't
* seek past the end of the file.
*/
void seek(long pos) throws IOException;
Now, without looking at the source, what do you think the reference
implementation - BufferedFSInputStream - does on any of the sequences:
seek(-1); read()
seek(0); seek(0)
close(); seek(0)
The javadocs don't specify what happens, the implementations don't behave
in ways you'd expect, and that leaves a problem: do you now define the
semantics of seek more rigorously -and fix our implementations -or do you
conclude that some people may expect the current behaviour on negative
seeks (they are ignored, for the curious), and so we can't actually change
them.