You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Jean-Daniel Cryans <jd...@apache.org> on 2013/04/04 22:20:38 UTC
Does compatibility between versions also mean binary compatibility?
tl;dr should two compatible versions be considered both wire and
binary compatible or just the former?
Hey devs,
0.92 is compatible with 0.94, meaning that you can run a client for
either against the other and you can roll restart from 0.92 to 0.94.
What about binary compatibility? Meaning, can you run user code
compiled against 0.92 with 0.94's jars?
Unfortunately, the answer is "no" in this case if you invoke setters
on HColumnDescriptor as you'll get:
java.lang.NoSuchMethodError:
org.apache.hadoop.hbase.HColumnDescriptor.setMaxVersions(I)V
HBASE-5357 "Use builder pattern in HColumnDescriptor" changed the
method signatures by changing "void" to "HColumnDescriptor" so it' not
the same methods anymore.
I don't think we really had talks about binary compatibility before so
this is why I'm raising it up now.
Should "compatible" versions be just wire compatible or both wire and
binary compatible? The latter means we need new tests. I think it
should be both.
What do you guys think?
J-D
Re: Does compatibility between versions also mean binary compatibility?
Posted by Jean-Daniel Cryans <jd...@apache.org>.
On Tue, Apr 9, 2013 at 12:49 AM, Elliott Clark <ec...@apache.org> wrote:
> On Mon, Apr 8, 2013 at 8:02 PM, Enis Söztutar <en...@gmail.com> wrote:
>
> > For normal client-level API's, I am not convinced that we should keep
> > binary compatibility between major versions. It will tie our hands of
> doing
> > some changes between major versions.
> >
>
> I agree with this. Once we go 1.0 I'd be more inclined to agree with
> restricting breaking changes like this. However I think as long as we're
> wire compatible that gets most of what our users really want, while
> allowing flexibility in developing new versions.
>
This is basically the question I'm asking the beginning of this thread: we
said 0.92 and 0.94 were "compatible", so does that mean wire+binary or just
wire? Most seem to think that compatibility meant the former. The problem
here was that when we decided that those versions were compatible, we
didn't qualify it and didn't test for binary compatibility.
I'm good with Enis' major/minor version compatibility specs.
J-D
Re: Does compatibility between versions also mean binary compatibility?
Posted by Elliott Clark <ec...@apache.org>.
On Mon, Apr 8, 2013 at 8:02 PM, Enis Söztutar <en...@gmail.com> wrote:
> For normal client-level API's, I am not convinced that we should keep
> binary compatibility between major versions. It will tie our hands of doing
> some changes between major versions.
>
I agree with this. Once we go 1.0 I'd be more inclined to agree with
restricting breaking changes like this. However I think as long as we're
wire compatible that gets most of what our users really want, while
allowing flexibility in developing new versions.
Re: Does compatibility between versions also mean binary compatibility?
Posted by Enis Söztutar <en...@gmail.com>.
I generally agree that we are not at the point of keeping binary or even
source compatibility of Phoenix's use of RegionScanner's etc. In order to
be very flexible, we are giving coprocessors too much control.
For normal client-level API's, I am not convinced that we should keep
binary compatibility between major versions. It will tie our hands of doing
some changes between major versions. An example is essential column family.
If you add a method to a public interface, you make it source incompatible.
Instead we should support
- Binary and wire compat between minor versions
- Wire compat and rolling restart between major versions. Rolling upgrade
support.
Enis
On Mon, Apr 8, 2013 at 5:40 PM, Andrew Purtell <ap...@apache.org> wrote:
> Yes of course.
>
>
> On Sat, Apr 6, 2013 at 1:11 PM, James Taylor <jt...@salesforce.com>
> wrote:
>
> > That seems reasonable to make an exception for coprocessors on binary
> > compatibility. Can this be explicitly documented if it's not already so
> > folks are sure to know that?
> >
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
Re: Does compatibility between versions also mean binary compatibility?
Posted by Andrew Purtell <ap...@apache.org>.
Yes of course.
On Sat, Apr 6, 2013 at 1:11 PM, James Taylor <jt...@salesforce.com> wrote:
> That seems reasonable to make an exception for coprocessors on binary
> compatibility. Can this be explicitly documented if it's not already so
> folks are sure to know that?
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: Does compatibility between versions also mean binary compatibility?
Posted by James Taylor <jt...@salesforce.com>.
That seems reasonable to make an exception for coprocessors on binary
compatibility. Can this be explicitly documented if it's not already so
folks are sure to know that?
James
On 04/04/2013 05:53 PM, Andrew Purtell wrote:
> Thanks for taking the time for the thoughtful feedback James.
>
>> - between 0.94.3 and 0.94.4, new methods were introduced on
> RegionScanner. Often coprocessors will have their own implementation of
> these so that they can aggregate in the postScannerOpen. Though this broke
> binary compat, it improved scan performance as well. Where does the binary
> compatible line stop?
>> - between 0.94.3 and 0.94.4, the class loading changed for coprocessors.
> If a coprocessor was on the classpath, it didn't matter what the jar path
> was, it would load. In 0.94.4, that was no longer the case - if the jar
> path was invalid, the coprocessor would no longer load. Though this broke
> compatibility, it was a good cleanup for the class loading logic. Call it a
> bug fix or a change in behavior, but either way it was an incompatible
> change. Is a change in behavior that causes incompatibilities still meet
> the binary compatible criteria?
>
> As "owner" of this space, I will claim that no binary compatibility nor
> even interface compatibility should be expected at this time even among
> point releases. This is for two reasons, the first being most important:
>
> 1) The Coprocessor API caters to only a few users currently, so feels
> the pull of the gravity of these major (new) users, e.g. security, Francis
> Liu's work, Phoenix. We can expect this to lessen naturally over time as --
> exactly here -- users like Phoenix become satisfied with the status quo and
> begin to push back. Your input going forward will be valuable.
>
> 2) Coprocessors are an extension surface for internals
>
>
>
> On Thu, Apr 4, 2013 at 4:29 PM, James Taylor <jt...@salesforce.com> wrote:
>
>> The binary compat is a slippery slope. It'd be a bummer if we couldn't
>> take advantage of all the innovation you guys are doing. At the same time,
>> it's tough to require the Phoenix user community, for example, to upgrade
>> their HBase servers to be able to move to the latest version of Phoenix. I
>> don't know what the right answer is, but here are a couple of concrete
>> cases:
>> - between 0.94.3 and 0.94.4, new methods were introduced on RegionScanner.
>> Often coprocessors will have their own implementation of these so that they
>> can aggregate in the postScannerOpen. Though this broke binary compat, it
>> improved scan performance as well. Where does the binary compatible line
>> stop?
>> - between 0.94.3 and 0.94.4, the class loading changed for coprocessors.
>> If a coprocessor was on the classpath, it didn't matter what the jar path
>> was, it would load. In 0.94.4, that was no longer the case - if the jar
>> path was invalid, the coprocessor would no longer load. Though this broke
>> compatibility, it was a good cleanup for the class loading logic. Call it a
>> bug fix or a change in behavior, but either way it was an incompatible
>> change. Is a change in behavior that causes incompatibilities still meet
>> the binary compatible criteria?
>> - between 0.94.4 and 0.94.5, the essential column family feature was
>> introduced. This is an example of one that is binary compatible. We're able
>> to take advantage of the feature and maintain binary compatibility with
>> 0.94.4 (in which case the feature simple wouldn't be available).
>>
>> Maybe if we just explicitly identified compatibility issues, that would be
>> a good start? We'd likely need a way to find them, though.
>>
>> James
>>
>> On 04/04/2013 03:59 PM, lars hofhansl wrote:
>>
>>> I agree we need both, but I'm afraid that ship has sailed.
>>> It's not something we paid a lot of attention to especially being
>>> forward-binary-compatible. I would guess that there will be many more of
>>> these issues.
>>>
>>> Also, we have to qualify this statement somewhere. If you extend
>>> HRegionServer you cannot expect compatibility between releases. Of course
>>> that is silly, but it serves the point I am making.
>>>
>>> For client visible classes (such as in this case) we should make it work,
>>> we identifies issues with Filters and Coprocessors in the past and kept
>>> them binary compatible on a best effort basis.
>>>
>>>
>>> TL;DR: Let's fix this issue, and be wary of more such issues.
>>>
>>>
>>> -- Lars
>>>
>>>
>>>
>>> ______________________________**__
>>> From: Andrew Purtell <ap...@apache.org>
>>> To: "dev@hbase.apache.org" <de...@hbase.apache.org>
>>> Sent: Thursday, April 4, 2013 3:21 PM
>>> Subject: Re: Does compatibility between versions also mean binary
>>> compatibility?
>>> "Compatible" implies both to my understanding of the term, unless
>>> qualified.
>>>
>>> I don't think we should qualify it. This looks like a regression to me.
>>>
>>>
>>> On Thu, Apr 4, 2013 at 1:20 PM, Jean-Daniel Cryans <jdcryans@apache.org
>>>> wrote:
>>> tl;dr should two compatible versions be considered both wire and
>>>> binary compatible or just the former?
>>>>
>>>> Hey devs,
>>>>
>>>> 0.92 is compatible with 0.94, meaning that you can run a client for
>>>> either against the other and you can roll restart from 0.92 to 0.94.
>>>>
>>>> What about binary compatibility? Meaning, can you run user code
>>>> compiled against 0.92 with 0.94's jars?
>>>>
>>>> Unfortunately, the answer is "no" in this case if you invoke setters
>>>> on HColumnDescriptor as you'll get:
>>>>
>>>> java.lang.NoSuchMethodError:
>>>> org.apache.hadoop.hbase.**HColumnDescriptor.**setMaxVersions(I)V
>>>>
>>>> HBASE-5357 "Use builder pattern in HColumnDescriptor" changed the
>>>> method signatures by changing "void" to "HColumnDescriptor" so it' not
>>>> the same methods anymore.
>>>>
>>>> I don't think we really had talks about binary compatibility before so
>>>> this is why I'm raising it up now.
>>>>
>>>> Should "compatible" versions be just wire compatible or both wire and
>>>> binary compatible? The latter means we need new tests. I think it
>>>> should be both.
>>>>
>>>> What do you guys think?
>>>>
>>>> J-D
>>>>
>>>>
>>>
>
Re: Does compatibility between versions also mean binary compatibility?
Posted by Andrew Purtell <ap...@apache.org>.
Thanks for taking the time for the thoughtful feedback James.
> - between 0.94.3 and 0.94.4, new methods were introduced on
RegionScanner. Often coprocessors will have their own implementation of
these so that they can aggregate in the postScannerOpen. Though this broke
binary compat, it improved scan performance as well. Where does the binary
compatible line stop?
> - between 0.94.3 and 0.94.4, the class loading changed for coprocessors.
If a coprocessor was on the classpath, it didn't matter what the jar path
was, it would load. In 0.94.4, that was no longer the case - if the jar
path was invalid, the coprocessor would no longer load. Though this broke
compatibility, it was a good cleanup for the class loading logic. Call it a
bug fix or a change in behavior, but either way it was an incompatible
change. Is a change in behavior that causes incompatibilities still meet
the binary compatible criteria?
As "owner" of this space, I will claim that no binary compatibility nor
even interface compatibility should be expected at this time even among
point releases. This is for two reasons, the first being most important:
1) The Coprocessor API caters to only a few users currently, so feels
the pull of the gravity of these major (new) users, e.g. security, Francis
Liu's work, Phoenix. We can expect this to lessen naturally over time as --
exactly here -- users like Phoenix become satisfied with the status quo and
begin to push back. Your input going forward will be valuable.
2) Coprocessors are an extension surface for internals
On Thu, Apr 4, 2013 at 4:29 PM, James Taylor <jt...@salesforce.com> wrote:
> The binary compat is a slippery slope. It'd be a bummer if we couldn't
> take advantage of all the innovation you guys are doing. At the same time,
> it's tough to require the Phoenix user community, for example, to upgrade
> their HBase servers to be able to move to the latest version of Phoenix. I
> don't know what the right answer is, but here are a couple of concrete
> cases:
> - between 0.94.3 and 0.94.4, new methods were introduced on RegionScanner.
> Often coprocessors will have their own implementation of these so that they
> can aggregate in the postScannerOpen. Though this broke binary compat, it
> improved scan performance as well. Where does the binary compatible line
> stop?
> - between 0.94.3 and 0.94.4, the class loading changed for coprocessors.
> If a coprocessor was on the classpath, it didn't matter what the jar path
> was, it would load. In 0.94.4, that was no longer the case - if the jar
> path was invalid, the coprocessor would no longer load. Though this broke
> compatibility, it was a good cleanup for the class loading logic. Call it a
> bug fix or a change in behavior, but either way it was an incompatible
> change. Is a change in behavior that causes incompatibilities still meet
> the binary compatible criteria?
> - between 0.94.4 and 0.94.5, the essential column family feature was
> introduced. This is an example of one that is binary compatible. We're able
> to take advantage of the feature and maintain binary compatibility with
> 0.94.4 (in which case the feature simple wouldn't be available).
>
> Maybe if we just explicitly identified compatibility issues, that would be
> a good start? We'd likely need a way to find them, though.
>
> James
>
> On 04/04/2013 03:59 PM, lars hofhansl wrote:
>
>> I agree we need both, but I'm afraid that ship has sailed.
>> It's not something we paid a lot of attention to especially being
>> forward-binary-compatible. I would guess that there will be many more of
>> these issues.
>>
>> Also, we have to qualify this statement somewhere. If you extend
>> HRegionServer you cannot expect compatibility between releases. Of course
>> that is silly, but it serves the point I am making.
>>
>> For client visible classes (such as in this case) we should make it work,
>> we identifies issues with Filters and Coprocessors in the past and kept
>> them binary compatible on a best effort basis.
>>
>>
>> TL;DR: Let's fix this issue, and be wary of more such issues.
>>
>>
>> -- Lars
>>
>>
>>
>> ______________________________**__
>> From: Andrew Purtell <ap...@apache.org>
>> To: "dev@hbase.apache.org" <de...@hbase.apache.org>
>> Sent: Thursday, April 4, 2013 3:21 PM
>> Subject: Re: Does compatibility between versions also mean binary
>> compatibility?
>> "Compatible" implies both to my understanding of the term, unless
>> qualified.
>>
>> I don't think we should qualify it. This looks like a regression to me.
>>
>>
>> On Thu, Apr 4, 2013 at 1:20 PM, Jean-Daniel Cryans <jdcryans@apache.org
>> >wrote:
>>
>> tl;dr should two compatible versions be considered both wire and
>>> binary compatible or just the former?
>>>
>>> Hey devs,
>>>
>>> 0.92 is compatible with 0.94, meaning that you can run a client for
>>> either against the other and you can roll restart from 0.92 to 0.94.
>>>
>>> What about binary compatibility? Meaning, can you run user code
>>> compiled against 0.92 with 0.94's jars?
>>>
>>> Unfortunately, the answer is "no" in this case if you invoke setters
>>> on HColumnDescriptor as you'll get:
>>>
>>> java.lang.NoSuchMethodError:
>>> org.apache.hadoop.hbase.**HColumnDescriptor.**setMaxVersions(I)V
>>>
>>> HBASE-5357 "Use builder pattern in HColumnDescriptor" changed the
>>> method signatures by changing "void" to "HColumnDescriptor" so it' not
>>> the same methods anymore.
>>>
>>> I don't think we really had talks about binary compatibility before so
>>> this is why I'm raising it up now.
>>>
>>> Should "compatible" versions be just wire compatible or both wire and
>>> binary compatible? The latter means we need new tests. I think it
>>> should be both.
>>>
>>> What do you guys think?
>>>
>>> J-D
>>>
>>>
>>
>>
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: Does compatibility between versions also mean binary compatibility?
Posted by Elliott Clark <ec...@apache.org>.
>From my point of view the most important thing is that there's no business
downtime when upgrading between versions (excluding the singularity). That
means there needs to be some path to go from (User Code for 0.92 + 0.92
HBase Servers) to (User Code for 0.94 + 0.94 HBase servers) with no down
time.
>From 0.92 to 0.94 that path would be:
Deploy 0.94.x
Roll restart HBase to 0.94
Write new User code for 0.94
Deploy new User Code that was written against 0.94
Keeping the changes to a minimum is great but I would rather have the
freedom to fix/add things that are needed.
On Fri, Apr 5, 2013 at 10:12 AM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> Good points James.
>
> I think that regarding binary compatibility, we need to be at least
> backward compatible.
>
> If I understand your first point correctly, code that compiles against
> 0.94.4 to use the new features added there wouldn't even compile against
> 0.94.3, and I find this reasonable. I'm pretty sure that most here think
> the same way.
>
> Regards,
>
> J-D
>
>
> On Thu, Apr 4, 2013 at 4:29 PM, James Taylor <jt...@salesforce.com>
> wrote:
>
> > The binary compat is a slippery slope. It'd be a bummer if we couldn't
> > take advantage of all the innovation you guys are doing. At the same
> time,
> > it's tough to require the Phoenix user community, for example, to upgrade
> > their HBase servers to be able to move to the latest version of Phoenix.
> I
> > don't know what the right answer is, but here are a couple of concrete
> > cases:
> > - between 0.94.3 and 0.94.4, new methods were introduced on
> RegionScanner.
> > Often coprocessors will have their own implementation of these so that
> they
> > can aggregate in the postScannerOpen. Though this broke binary compat, it
> > improved scan performance as well. Where does the binary compatible line
> > stop?
> > - between 0.94.3 and 0.94.4, the class loading changed for coprocessors.
> > If a coprocessor was on the classpath, it didn't matter what the jar path
> > was, it would load. In 0.94.4, that was no longer the case - if the jar
> > path was invalid, the coprocessor would no longer load. Though this broke
> > compatibility, it was a good cleanup for the class loading logic. Call
> it a
> > bug fix or a change in behavior, but either way it was an incompatible
> > change. Is a change in behavior that causes incompatibilities still meet
> > the binary compatible criteria?
> > - between 0.94.4 and 0.94.5, the essential column family feature was
> > introduced. This is an example of one that is binary compatible. We're
> able
> > to take advantage of the feature and maintain binary compatibility with
> > 0.94.4 (in which case the feature simple wouldn't be available).
> >
> > Maybe if we just explicitly identified compatibility issues, that would
> be
> > a good start? We'd likely need a way to find them, though.
> >
> > James
> >
> >
> > On 04/04/2013 03:59 PM, lars hofhansl wrote:
> >
> >> I agree we need both, but I'm afraid that ship has sailed.
> >> It's not something we paid a lot of attention to especially being
> >> forward-binary-compatible. I would guess that there will be many more of
> >> these issues.
> >>
> >> Also, we have to qualify this statement somewhere. If you extend
> >> HRegionServer you cannot expect compatibility between releases. Of
> course
> >> that is silly, but it serves the point I am making.
> >>
> >> For client visible classes (such as in this case) we should make it
> work,
> >> we identifies issues with Filters and Coprocessors in the past and kept
> >> them binary compatible on a best effort basis.
> >>
> >>
> >> TL;DR: Let's fix this issue, and be wary of more such issues.
> >>
> >>
> >> -- Lars
> >>
> >>
> >>
> >> ______________________________**__
> >> From: Andrew Purtell <ap...@apache.org>
> >> To: "dev@hbase.apache.org" <de...@hbase.apache.org>
> >> Sent: Thursday, April 4, 2013 3:21 PM
> >> Subject: Re: Does compatibility between versions also mean binary
> >> compatibility?
> >> "Compatible" implies both to my understanding of the term, unless
> >> qualified.
> >>
> >> I don't think we should qualify it. This looks like a regression to me.
> >>
> >>
> >> On Thu, Apr 4, 2013 at 1:20 PM, Jean-Daniel Cryans <jdcryans@apache.org
> >> >wrote:
> >>
> >> tl;dr should two compatible versions be considered both wire and
> >>> binary compatible or just the former?
> >>>
> >>> Hey devs,
> >>>
> >>> 0.92 is compatible with 0.94, meaning that you can run a client for
> >>> either against the other and you can roll restart from 0.92 to 0.94.
> >>>
> >>> What about binary compatibility? Meaning, can you run user code
> >>> compiled against 0.92 with 0.94's jars?
> >>>
> >>> Unfortunately, the answer is "no" in this case if you invoke setters
> >>> on HColumnDescriptor as you'll get:
> >>>
> >>> java.lang.NoSuchMethodError:
> >>> org.apache.hadoop.hbase.**HColumnDescriptor.**setMaxVersions(I)V
> >>>
> >>> HBASE-5357 "Use builder pattern in HColumnDescriptor" changed the
> >>> method signatures by changing "void" to "HColumnDescriptor" so it' not
> >>> the same methods anymore.
> >>>
> >>> I don't think we really had talks about binary compatibility before so
> >>> this is why I'm raising it up now.
> >>>
> >>> Should "compatible" versions be just wire compatible or both wire and
> >>> binary compatible? The latter means we need new tests. I think it
> >>> should be both.
> >>>
> >>> What do you guys think?
> >>>
> >>> J-D
> >>>
> >>>
> >>
> >>
> >
>
Re: Does compatibility between versions also mean binary compatibility?
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Good points James.
I think that regarding binary compatibility, we need to be at least
backward compatible.
If I understand your first point correctly, code that compiles against
0.94.4 to use the new features added there wouldn't even compile against
0.94.3, and I find this reasonable. I'm pretty sure that most here think
the same way.
Regards,
J-D
On Thu, Apr 4, 2013 at 4:29 PM, James Taylor <jt...@salesforce.com> wrote:
> The binary compat is a slippery slope. It'd be a bummer if we couldn't
> take advantage of all the innovation you guys are doing. At the same time,
> it's tough to require the Phoenix user community, for example, to upgrade
> their HBase servers to be able to move to the latest version of Phoenix. I
> don't know what the right answer is, but here are a couple of concrete
> cases:
> - between 0.94.3 and 0.94.4, new methods were introduced on RegionScanner.
> Often coprocessors will have their own implementation of these so that they
> can aggregate in the postScannerOpen. Though this broke binary compat, it
> improved scan performance as well. Where does the binary compatible line
> stop?
> - between 0.94.3 and 0.94.4, the class loading changed for coprocessors.
> If a coprocessor was on the classpath, it didn't matter what the jar path
> was, it would load. In 0.94.4, that was no longer the case - if the jar
> path was invalid, the coprocessor would no longer load. Though this broke
> compatibility, it was a good cleanup for the class loading logic. Call it a
> bug fix or a change in behavior, but either way it was an incompatible
> change. Is a change in behavior that causes incompatibilities still meet
> the binary compatible criteria?
> - between 0.94.4 and 0.94.5, the essential column family feature was
> introduced. This is an example of one that is binary compatible. We're able
> to take advantage of the feature and maintain binary compatibility with
> 0.94.4 (in which case the feature simple wouldn't be available).
>
> Maybe if we just explicitly identified compatibility issues, that would be
> a good start? We'd likely need a way to find them, though.
>
> James
>
>
> On 04/04/2013 03:59 PM, lars hofhansl wrote:
>
>> I agree we need both, but I'm afraid that ship has sailed.
>> It's not something we paid a lot of attention to especially being
>> forward-binary-compatible. I would guess that there will be many more of
>> these issues.
>>
>> Also, we have to qualify this statement somewhere. If you extend
>> HRegionServer you cannot expect compatibility between releases. Of course
>> that is silly, but it serves the point I am making.
>>
>> For client visible classes (such as in this case) we should make it work,
>> we identifies issues with Filters and Coprocessors in the past and kept
>> them binary compatible on a best effort basis.
>>
>>
>> TL;DR: Let's fix this issue, and be wary of more such issues.
>>
>>
>> -- Lars
>>
>>
>>
>> ______________________________**__
>> From: Andrew Purtell <ap...@apache.org>
>> To: "dev@hbase.apache.org" <de...@hbase.apache.org>
>> Sent: Thursday, April 4, 2013 3:21 PM
>> Subject: Re: Does compatibility between versions also mean binary
>> compatibility?
>> "Compatible" implies both to my understanding of the term, unless
>> qualified.
>>
>> I don't think we should qualify it. This looks like a regression to me.
>>
>>
>> On Thu, Apr 4, 2013 at 1:20 PM, Jean-Daniel Cryans <jdcryans@apache.org
>> >wrote:
>>
>> tl;dr should two compatible versions be considered both wire and
>>> binary compatible or just the former?
>>>
>>> Hey devs,
>>>
>>> 0.92 is compatible with 0.94, meaning that you can run a client for
>>> either against the other and you can roll restart from 0.92 to 0.94.
>>>
>>> What about binary compatibility? Meaning, can you run user code
>>> compiled against 0.92 with 0.94's jars?
>>>
>>> Unfortunately, the answer is "no" in this case if you invoke setters
>>> on HColumnDescriptor as you'll get:
>>>
>>> java.lang.NoSuchMethodError:
>>> org.apache.hadoop.hbase.**HColumnDescriptor.**setMaxVersions(I)V
>>>
>>> HBASE-5357 "Use builder pattern in HColumnDescriptor" changed the
>>> method signatures by changing "void" to "HColumnDescriptor" so it' not
>>> the same methods anymore.
>>>
>>> I don't think we really had talks about binary compatibility before so
>>> this is why I'm raising it up now.
>>>
>>> Should "compatible" versions be just wire compatible or both wire and
>>> binary compatible? The latter means we need new tests. I think it
>>> should be both.
>>>
>>> What do you guys think?
>>>
>>> J-D
>>>
>>>
>>
>>
>
Re: Does compatibility between versions also mean binary compatibility?
Posted by James Taylor <jt...@salesforce.com>.
The binary compat is a slippery slope. It'd be a bummer if we couldn't
take advantage of all the innovation you guys are doing. At the same
time, it's tough to require the Phoenix user community, for example, to
upgrade their HBase servers to be able to move to the latest version of
Phoenix. I don't know what the right answer is, but here are a couple of
concrete cases:
- between 0.94.3 and 0.94.4, new methods were introduced on
RegionScanner. Often coprocessors will have their own implementation of
these so that they can aggregate in the postScannerOpen. Though this
broke binary compat, it improved scan performance as well. Where does
the binary compatible line stop?
- between 0.94.3 and 0.94.4, the class loading changed for coprocessors.
If a coprocessor was on the classpath, it didn't matter what the jar
path was, it would load. In 0.94.4, that was no longer the case - if the
jar path was invalid, the coprocessor would no longer load. Though this
broke compatibility, it was a good cleanup for the class loading logic.
Call it a bug fix or a change in behavior, but either way it was an
incompatible change. Is a change in behavior that causes
incompatibilities still meet the binary compatible criteria?
- between 0.94.4 and 0.94.5, the essential column family feature was
introduced. This is an example of one that is binary compatible. We're
able to take advantage of the feature and maintain binary compatibility
with 0.94.4 (in which case the feature simple wouldn't be available).
Maybe if we just explicitly identified compatibility issues, that would
be a good start? We'd likely need a way to find them, though.
James
On 04/04/2013 03:59 PM, lars hofhansl wrote:
> I agree we need both, but I'm afraid that ship has sailed.
> It's not something we paid a lot of attention to especially being forward-binary-compatible. I would guess that there will be many more of these issues.
>
> Also, we have to qualify this statement somewhere. If you extend HRegionServer you cannot expect compatibility between releases. Of course that is silly, but it serves the point I am making.
>
> For client visible classes (such as in this case) we should make it work, we identifies issues with Filters and Coprocessors in the past and kept them binary compatible on a best effort basis.
>
>
> TL;DR: Let's fix this issue, and be wary of more such issues.
>
>
> -- Lars
>
>
>
> ________________________________
> From: Andrew Purtell <ap...@apache.org>
> To: "dev@hbase.apache.org" <de...@hbase.apache.org>
> Sent: Thursday, April 4, 2013 3:21 PM
> Subject: Re: Does compatibility between versions also mean binary compatibility?
>
> "Compatible" implies both to my understanding of the term, unless
> qualified.
>
> I don't think we should qualify it. This looks like a regression to me.
>
>
> On Thu, Apr 4, 2013 at 1:20 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
>
>> tl;dr should two compatible versions be considered both wire and
>> binary compatible or just the former?
>>
>> Hey devs,
>>
>> 0.92 is compatible with 0.94, meaning that you can run a client for
>> either against the other and you can roll restart from 0.92 to 0.94.
>>
>> What about binary compatibility? Meaning, can you run user code
>> compiled against 0.92 with 0.94's jars?
>>
>> Unfortunately, the answer is "no" in this case if you invoke setters
>> on HColumnDescriptor as you'll get:
>>
>> java.lang.NoSuchMethodError:
>> org.apache.hadoop.hbase.HColumnDescriptor.setMaxVersions(I)V
>>
>> HBASE-5357 "Use builder pattern in HColumnDescriptor" changed the
>> method signatures by changing "void" to "HColumnDescriptor" so it' not
>> the same methods anymore.
>>
>> I don't think we really had talks about binary compatibility before so
>> this is why I'm raising it up now.
>>
>> Should "compatible" versions be just wire compatible or both wire and
>> binary compatible? The latter means we need new tests. I think it
>> should be both.
>>
>> What do you guys think?
>>
>> J-D
>>
>
>
Re: Does compatibility between versions also mean binary compatibility?
Posted by Stack <st...@duboce.net>.
On Thu, Apr 4, 2013 at 4:06 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> Yes it's already too late for 0.94.
>
> Stack is fixing the Ref Guide about 0.92 and 0.94 being "interface
> compatible": http://hbase.apache.org/book.html#upgrade0.94
I added a note on what we agree binary compatibility means in the
'upgrading' section (Will roll it out to website in morning after adding in
our discussion of how to version JIRA issues on resolve that came up in an
earlier thread...).
St.Ack
Re: Does compatibility between versions also mean binary compatibility?
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Yes it's already too late for 0.94.
Stack is fixing the Ref Guide about 0.92 and 0.94 being "interface
compatible": http://hbase.apache.org/book.html#upgrade0.94
We could add a new column in the release testing matrix to add versions
that we binary-tested, at least for the user-facing methods.
Also, we should have some sort of automated tool to check this. Stack and I
shared a few ideas, I'm opening a jira about it in a moment.
J-D
On Thu, Apr 4, 2013 at 3:59 PM, lars hofhansl <la...@apache.org> wrote:
> I agree we need both, but I'm afraid that ship has sailed.
> It's not something we paid a lot of attention to especially being
> forward-binary-compatible. I would guess that there will be many more of
> these issues.
>
> Also, we have to qualify this statement somewhere. If you extend
> HRegionServer you cannot expect compatibility between releases. Of course
> that is silly, but it serves the point I am making.
>
> For client visible classes (such as in this case) we should make it work,
> we identifies issues with Filters and Coprocessors in the past and kept
> them binary compatible on a best effort basis.
>
>
> TL;DR: Let's fix this issue, and be wary of more such issues.
>
>
> -- Lars
>
>
>
> ________________________________
> From: Andrew Purtell <ap...@apache.org>
> To: "dev@hbase.apache.org" <de...@hbase.apache.org>
> Sent: Thursday, April 4, 2013 3:21 PM
> Subject: Re: Does compatibility between versions also mean binary
> compatibility?
>
> "Compatible" implies both to my understanding of the term, unless
> qualified.
>
> I don't think we should qualify it. This looks like a regression to me.
>
>
> On Thu, Apr 4, 2013 at 1:20 PM, Jean-Daniel Cryans <jdcryans@apache.org
> >wrote:
>
> > tl;dr should two compatible versions be considered both wire and
> > binary compatible or just the former?
> >
> > Hey devs,
> >
> > 0.92 is compatible with 0.94, meaning that you can run a client for
> > either against the other and you can roll restart from 0.92 to 0.94.
> >
> > What about binary compatibility? Meaning, can you run user code
> > compiled against 0.92 with 0.94's jars?
> >
> > Unfortunately, the answer is "no" in this case if you invoke setters
> > on HColumnDescriptor as you'll get:
> >
> > java.lang.NoSuchMethodError:
> > org.apache.hadoop.hbase.HColumnDescriptor.setMaxVersions(I)V
> >
> > HBASE-5357 "Use builder pattern in HColumnDescriptor" changed the
> > method signatures by changing "void" to "HColumnDescriptor" so it' not
> > the same methods anymore.
> >
> > I don't think we really had talks about binary compatibility before so
> > this is why I'm raising it up now.
> >
> > Should "compatible" versions be just wire compatible or both wire and
> > binary compatible? The latter means we need new tests. I think it
> > should be both.
> >
> > What do you guys think?
> >
> > J-D
> >
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
Re: Does compatibility between versions also mean binary compatibility?
Posted by lars hofhansl <la...@apache.org>.
I agree we need both, but I'm afraid that ship has sailed.
It's not something we paid a lot of attention to especially being forward-binary-compatible. I would guess that there will be many more of these issues.
Also, we have to qualify this statement somewhere. If you extend HRegionServer you cannot expect compatibility between releases. Of course that is silly, but it serves the point I am making.
For client visible classes (such as in this case) we should make it work, we identifies issues with Filters and Coprocessors in the past and kept them binary compatible on a best effort basis.
TL;DR: Let's fix this issue, and be wary of more such issues.
-- Lars
________________________________
From: Andrew Purtell <ap...@apache.org>
To: "dev@hbase.apache.org" <de...@hbase.apache.org>
Sent: Thursday, April 4, 2013 3:21 PM
Subject: Re: Does compatibility between versions also mean binary compatibility?
"Compatible" implies both to my understanding of the term, unless
qualified.
I don't think we should qualify it. This looks like a regression to me.
On Thu, Apr 4, 2013 at 1:20 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> tl;dr should two compatible versions be considered both wire and
> binary compatible or just the former?
>
> Hey devs,
>
> 0.92 is compatible with 0.94, meaning that you can run a client for
> either against the other and you can roll restart from 0.92 to 0.94.
>
> What about binary compatibility? Meaning, can you run user code
> compiled against 0.92 with 0.94's jars?
>
> Unfortunately, the answer is "no" in this case if you invoke setters
> on HColumnDescriptor as you'll get:
>
> java.lang.NoSuchMethodError:
> org.apache.hadoop.hbase.HColumnDescriptor.setMaxVersions(I)V
>
> HBASE-5357 "Use builder pattern in HColumnDescriptor" changed the
> method signatures by changing "void" to "HColumnDescriptor" so it' not
> the same methods anymore.
>
> I don't think we really had talks about binary compatibility before so
> this is why I'm raising it up now.
>
> Should "compatible" versions be just wire compatible or both wire and
> binary compatible? The latter means we need new tests. I think it
> should be both.
>
> What do you guys think?
>
> J-D
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: Does compatibility between versions also mean binary compatibility?
Posted by Stack <st...@duboce.net>.
What Andrew and Nick said.
St.Ack
On Thu, Apr 4, 2013 at 3:21 PM, Andrew Purtell <ap...@apache.org> wrote:
> "Compatible" implies both to my understanding of the term, unless
> qualified.
>
> I don't think we should qualify it. This looks like a regression to me.
>
>
> On Thu, Apr 4, 2013 at 1:20 PM, Jean-Daniel Cryans <jdcryans@apache.org
> >wrote:
>
> > tl;dr should two compatible versions be considered both wire and
> > binary compatible or just the former?
> >
> > Hey devs,
> >
> > 0.92 is compatible with 0.94, meaning that you can run a client for
> > either against the other and you can roll restart from 0.92 to 0.94.
> >
> > What about binary compatibility? Meaning, can you run user code
> > compiled against 0.92 with 0.94's jars?
> >
> > Unfortunately, the answer is "no" in this case if you invoke setters
> > on HColumnDescriptor as you'll get:
> >
> > java.lang.NoSuchMethodError:
> > org.apache.hadoop.hbase.HColumnDescriptor.setMaxVersions(I)V
> >
> > HBASE-5357 "Use builder pattern in HColumnDescriptor" changed the
> > method signatures by changing "void" to "HColumnDescriptor" so it' not
> > the same methods anymore.
> >
> > I don't think we really had talks about binary compatibility before so
> > this is why I'm raising it up now.
> >
> > Should "compatible" versions be just wire compatible or both wire and
> > binary compatible? The latter means we need new tests. I think it
> > should be both.
> >
> > What do you guys think?
> >
> > J-D
> >
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
Re: Does compatibility between versions also mean binary compatibility?
Posted by Andrew Purtell <ap...@apache.org>.
"Compatible" implies both to my understanding of the term, unless
qualified.
I don't think we should qualify it. This looks like a regression to me.
On Thu, Apr 4, 2013 at 1:20 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> tl;dr should two compatible versions be considered both wire and
> binary compatible or just the former?
>
> Hey devs,
>
> 0.92 is compatible with 0.94, meaning that you can run a client for
> either against the other and you can roll restart from 0.92 to 0.94.
>
> What about binary compatibility? Meaning, can you run user code
> compiled against 0.92 with 0.94's jars?
>
> Unfortunately, the answer is "no" in this case if you invoke setters
> on HColumnDescriptor as you'll get:
>
> java.lang.NoSuchMethodError:
> org.apache.hadoop.hbase.HColumnDescriptor.setMaxVersions(I)V
>
> HBASE-5357 "Use builder pattern in HColumnDescriptor" changed the
> method signatures by changing "void" to "HColumnDescriptor" so it' not
> the same methods anymore.
>
> I don't think we really had talks about binary compatibility before so
> this is why I'm raising it up now.
>
> Should "compatible" versions be just wire compatible or both wire and
> binary compatible? The latter means we need new tests. I think it
> should be both.
>
> What do you guys think?
>
> J-D
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: Does compatibility between versions also mean binary compatibility?
Posted by Nick Dimiduk <nd...@gmail.com>.
On Thu, Apr 4, 2013 at 1:20 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> tl;dr should two compatible versions be considered both wire and
> binary compatible or just the former?
>
I agree; if the compatibility is not explicitly quantified, it implies both.
-n