You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@phoenix.apache.org by István Tóth <st...@cloudera.com.INVALID> on 2020/01/13 07:25:49 UTC
Re: Moving Phoenix master to Hbase 2.2
Hi!
(Sorry for not replying earlier, I'll need to revisit my mail filters)
I do have a half-baked patch with the lightweight isolation in modules case.
In its current form it's more ugly than complex, but some ugliness is a
small price to pay for not having to maintain multiple branches. (And it
can be beutified later)
I will complete it, and make the PR for that as time permits.
The tephra solution handles the version differences during run (startup)
time, my current solution is compile-time.
Istvan
On Fri, Dec 20, 2019 at 11:05 AM larsh@apache.org <la...@apache.org> wrote:
> Yep that.
> For the record... I do not think this is simple. But it is possible.
>
> On Thursday, December 19, 2019, 8:37:37 PM GMT+1, Andrew Purtell <
> apurtell@apache.org> wrote:
>
> I can't answer for Lars but whenever version incompatibilities come up
> usually only a handful of files are impacted. In the last round, the
> Phoenix access controller, a related file in the same directory, and the
> RPC scheduler. If you cloned these into separate version specific maven
> modules case by case as needed at each round the differences are fairly
> small. On the other hand if you take a principled approach and abstract all
> the things, it will be a huge effort and nobody realistically will want to
> take it on.
>
> On Thu, Dec 19, 2019 at 11:34 AM Geoffrey Jacoby <gj...@gmail.com>
> wrote:
>
> > Lars,
> >
> > I'm curious why you say the differences are easily isolated -- many of
> the
> > core classes of Phoenix either directly inherit HBase classes or
> implement
> > HBase interfaces, and those can vary between minor versions. (See my
> above
> > example of a new coprocessor hook on BaseRegionObserver.)
> >
> > Geoffrey
> >
> > On Thu, Dec 19, 2019 at 10:54 AM larsh@apache.org <la...@apache.org>
> > wrote:
> >
> > > Yep. The differences are pretty minimal - provided they can be
> isolated
> > > easily.
> > > Tephra might be a pretty good model. It supports various versions of
> > HBase
> > > in a single branch and has similar issues as Phoenix (coprocessors,
> etc).
> > > -- Lars
> > > On Thursday, December 19, 2019, 7:07:51 PM GMT+1, Josh Elser <
> > > elserj@apache.org> wrote:
> > >
> > > To clarify, you think that compat modules are better than that
> > > separate-branches model in 4.x?
> > >
> > > On 12/18/19 11:29 AM, larsh@apache.org wrote:
> > > > This is really hard to follow.
> > > >
> > > > I think we should do the same with HBase dependencies in Phoenix that
> > > HBase does with Hadoop dependencies.
> > > >
> > > > That is: We could have a maven module with the specific HBase
> version
> > > dependent code.
> > > > Btw. Tephra does the same... A module for HBase version specific
> code.
> > > > -- Lars
> > > >
> > > > On Tuesday, December 17, 2019, 10:00:31 AM GMT+1, Istvan Toth <
> > > stoty@apache.org> wrote:
> > > >
> > > > What do you think about tying the minor releases to Hbase minor
> > releases
> > > > (not necessarily one-to-one)
> > > >
> > > > for example (provided 5.1 is 2020H1)
> > > >
> > > > 5.0.0 -> HB 2.0
> > > > 5.1.0 -> HB 2.2.2 (and whatever 2.1 is API compatible with it)
> > > > 5.1.x -> HB 2.2.x (treat as maintenance branch, no major new
> features)
> > > > 5.2.0 -> HB 2.3.0 (if released by that time)
> > > > 5.2.x -> HB 2.3.x (treat as maintenance branch, no major new
> features)
> > > > 5.3.0 -> HB 2.3.x (if there is no new major/minor Hbase release)
> > > > master -> latest released HBase version
> > > >
> > > > Alternatively, we could stick with the same HBase version for patch
> > > > releases that we used for the first minor release.
> > > >
> > > > This would limit the number of branches that we have to maintain in
> > > > parallel, while providing maintenance branches for older releases,
> and
> > > > timely-ish Phoenix releases.
> > > >
> > > > The drawback is that users of old HBase versions won't get the latest
> > > > features, on the other hand they can expect more polish.
> > > >
> > > > Istvan
> > > >
> > > > On Thu, Dec 12, 2019 at 8:05 PM Geoffrey Jacoby <gj...@apache.org>
> > > wrote:
> > > >
> > > >> Since HBase 2.0 is EOM'ed, I'm +1 for not worrying about 2.0.x
> > > >> compatibility with the 5.x branch going forward.
> > > >>
> > > >> Given how coupled Phoenix is to the implementation details of HBase
> > > though,
> > > >> I'm not sure trying to abstract those away to keep one Phoenix
> branch
> > > per
> > > >> HBase major version is practical, however. At the least, it would be
> > > really
> > > >> complex.
> > > >>
> > > >> For example, in the new year I plan to return to working on the
> change
> > > data
> > > >> capture and Phoenix-level replication features, both of which depend
> > on
> > > >> WALKey interface changes and a new RegionObserver coprocessor hook
> > > >> introduced in HBASE-22622 and HBASE-22623. This was released in
> HBase
> > > 1.5
> > > >> and will be in the forthcoming HBase 2.3. While the HBase community
> is
> > > >> discussing EOMing 1.3 right now, and maybe 1.4 will go in the medium
> > > term,
> > > >> I don't see all pre-2.3 branch-2's getting deprecated anytime soon.
> > > >>
> > > >> So there will be at least two significant features that can only
> exist
> > > in
> > > >> some but not all of our 4.x and 5.x branches.
> > > >>
> > > >> Geoffrey
> > > >>
> > > >> On Thu, Dec 12, 2019 at 8:21 AM Josh Elser <el...@apache.org>
> wrote:
> > > >>
> > > >>> As much as possible, I'd like to avoid us getting into another
> > > situation
> > > >>> with 5.x where we have multiple branches. My hope was/is that we
> can
> > > >>> keep one Phoenix5 branch that works against an acceptable set of
> > HBase
> > > >>> branches.
> > > >>>
> > > >>> To me, that acceptable set of HBase branches is _a_ 2.1 and 2.2
> > > release.
> > > >>> I don't think we need to support all 2.1.x or 2.2.x, nor do I think
> > we
> > > >>> need to keep trying to maintain 2.0.x as it's already end of
> support
> > by
> > > >>> the HBase community.
> > > >>>
> > > >>> Thanks for updating your PR. I'll add this to my review queue.
> > > >>>
> > > >>> On 12/12/19 1:52 AM, Istvan Toth wrote:
> > > >>>> Hi!
> > > >>>>
> > > >>>> I'd like to start a conversation about supporting HBase 2.2. in
> the
> > > >>>> master branch.
> > > >>>>
> > > >>>> https://issues.apache.org/jira/browse/PHOENIX-5268 has a slightly
> > out
> > > >> of
> > > >>>> date, but functional PR for HBase 2.2 support on master. (Please
> > > review
> > > >>>> and comment if you have the time, I'll try to update the PR in the
> > > next
> > > >>>> few days)
> > > >>>>
> > > >>>> The reason that it is not a straightforward decision to merge it
> is
> > > >> that
> > > >>>> applying that patch breaks compatibility with HBase 2.0.1, the
> > current
> > > >>>> base.
> > > >>>>
> > > >>>> I can see the following outcomes:
> > > >>>>
> > > >>>> - Do nothing
> > > >>>> - Move master to HBase 2.2.2
> > > >>>> - Fork master to Hbase-2.0 and Hbase-2.2 branches
> > > >>>> - Build time compatibility modules
> > > >>>> - Run time compatibility modules
> > > >>>> - Something that I haven't thought of
> > > >>>>
> > > >>>>
> > > >>>> Doing nothing is obviously not a long term solution, as the
> current
> > > >>>> master doesn't work with any of the currently supported HBase
> > > branches,
> > > >>>> but we may postpone the inevitable.
> > > >>>>
> > > >>>> Simply moving master to HBase 2.2 is the most attractive solution
> > from
> > > >> a
> > > >>>> pure developer POV, but there may be other considerations.
> > > >>>>
> > > >>>> Having multiple masters for 2.0 and 2.2 is simple from a code
> > > >>>> perspective, but maintaining two branches is a non-trivial amount
> of
> > > >>>> additional work. (See the 4.x situation)
> > > >>>>
> > > >>>> Moving the HBase version dependent stuff into a separate module,
> and
> > > >>>> choosing at build time is not pretty from a code POV, but saves us
> > the
> > > >>>> hassle of maintaining multiple branches, while maintaining
> > > >> compatibility
> > > >>>> with multiple HBase versions, and can handle future API changes
> as
> > > >> well
> > > >>>> from a single branch. Doing something like this could have saved
> us
> > > the
> > > >>>> effort of maintaining three separate 4.x branches.
> > > >>>>
> > > >>>> I feel that since Phoenix is closely timed to HBase, and requires
> > > >>>> cluster-wide HBase configuration to work anyway, handling the
> > > different
> > > >>>> HBase versions from the same binary/JAR is not worth the effort.
> > > >>>>
> > > >>>> Please share your thoughts!
> > > >>>>
> > > >>>> regards
> > > >>>> Istvan
> > > >>>>
> > > >>>
> > > >>
> > > >
> > > >
> > >
> >
>
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
> - A23, Crosstalk
>
--
*István Tóth* | Sr. Software Engineer
t. (36) 70 283-1788
stoty@cloudera.com <https://www.cloudera.com>
[image: Cloudera] <https://www.cloudera.com/>
[image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
on LinkedIn] <https://www.linkedin.com/company/cloudera>
<https://www.cloudera.com/>
------------------------------
Re: Moving Phoenix master to Hbase 2.2
Posted by Josh Elser <el...@apache.org>.
A solution sounds great! Thanks for working on that.
By compile-time, I assume that means, unless we get lucky with
compatible bytecode, we'd have to have a Phoenix which is compiled
against a specific HBase version?
On 1/13/20 2:25 AM, István Tóth wrote:
> Hi!
>
> (Sorry for not replying earlier, I'll need to revisit my mail filters)
>
> I do have a half-baked patch with the lightweight isolation in modules case.
>
> In its current form it's more ugly than complex, but some ugliness is a
> small price to pay for not having to maintain multiple branches. (And it
> can be beutified later)
>
> I will complete it, and make the PR for that as time permits.
>
> The tephra solution handles the version differences during run (startup)
> time, my current solution is compile-time.
>
> Istvan
>
>
>
>
> On Fri, Dec 20, 2019 at 11:05 AM larsh@apache.org <la...@apache.org> wrote:
>
>> Yep that.
>> For the record... I do not think this is simple. But it is possible.
>>
>> On Thursday, December 19, 2019, 8:37:37 PM GMT+1, Andrew Purtell <
>> apurtell@apache.org> wrote:
>>
>> I can't answer for Lars but whenever version incompatibilities come up
>> usually only a handful of files are impacted. In the last round, the
>> Phoenix access controller, a related file in the same directory, and the
>> RPC scheduler. If you cloned these into separate version specific maven
>> modules case by case as needed at each round the differences are fairly
>> small. On the other hand if you take a principled approach and abstract all
>> the things, it will be a huge effort and nobody realistically will want to
>> take it on.
>>
>> On Thu, Dec 19, 2019 at 11:34 AM Geoffrey Jacoby <gj...@gmail.com>
>> wrote:
>>
>>> Lars,
>>>
>>> I'm curious why you say the differences are easily isolated -- many of
>> the
>>> core classes of Phoenix either directly inherit HBase classes or
>> implement
>>> HBase interfaces, and those can vary between minor versions. (See my
>> above
>>> example of a new coprocessor hook on BaseRegionObserver.)
>>>
>>> Geoffrey
>>>
>>> On Thu, Dec 19, 2019 at 10:54 AM larsh@apache.org <la...@apache.org>
>>> wrote:
>>>
>>>> Yep. The differences are pretty minimal - provided they can be
>> isolated
>>>> easily.
>>>> Tephra might be a pretty good model. It supports various versions of
>>> HBase
>>>> in a single branch and has similar issues as Phoenix (coprocessors,
>> etc).
>>>> -- Lars
>>>> On Thursday, December 19, 2019, 7:07:51 PM GMT+1, Josh Elser <
>>>> elserj@apache.org> wrote:
>>>>
>>>> To clarify, you think that compat modules are better than that
>>>> separate-branches model in 4.x?
>>>>
>>>> On 12/18/19 11:29 AM, larsh@apache.org wrote:
>>>>> This is really hard to follow.
>>>>>
>>>>> I think we should do the same with HBase dependencies in Phoenix that
>>>> HBase does with Hadoop dependencies.
>>>>>
>>>>> That is: We could have a maven module with the specific HBase
>> version
>>>> dependent code.
>>>>> Btw. Tephra does the same... A module for HBase version specific
>> code.
>>>>> -- Lars
>>>>>
>>>>> On Tuesday, December 17, 2019, 10:00:31 AM GMT+1, Istvan Toth <
>>>> stoty@apache.org> wrote:
>>>>>
>>>>> What do you think about tying the minor releases to Hbase minor
>>> releases
>>>>> (not necessarily one-to-one)
>>>>>
>>>>> for example (provided 5.1 is 2020H1)
>>>>>
>>>>> 5.0.0 -> HB 2.0
>>>>> 5.1.0 -> HB 2.2.2 (and whatever 2.1 is API compatible with it)
>>>>> 5.1.x -> HB 2.2.x (treat as maintenance branch, no major new
>> features)
>>>>> 5.2.0 -> HB 2.3.0 (if released by that time)
>>>>> 5.2.x -> HB 2.3.x (treat as maintenance branch, no major new
>> features)
>>>>> 5.3.0 -> HB 2.3.x (if there is no new major/minor Hbase release)
>>>>> master -> latest released HBase version
>>>>>
>>>>> Alternatively, we could stick with the same HBase version for patch
>>>>> releases that we used for the first minor release.
>>>>>
>>>>> This would limit the number of branches that we have to maintain in
>>>>> parallel, while providing maintenance branches for older releases,
>> and
>>>>> timely-ish Phoenix releases.
>>>>>
>>>>> The drawback is that users of old HBase versions won't get the latest
>>>>> features, on the other hand they can expect more polish.
>>>>>
>>>>> Istvan
>>>>>
>>>>> On Thu, Dec 12, 2019 at 8:05 PM Geoffrey Jacoby <gj...@apache.org>
>>>> wrote:
>>>>>
>>>>>> Since HBase 2.0 is EOM'ed, I'm +1 for not worrying about 2.0.x
>>>>>> compatibility with the 5.x branch going forward.
>>>>>>
>>>>>> Given how coupled Phoenix is to the implementation details of HBase
>>>> though,
>>>>>> I'm not sure trying to abstract those away to keep one Phoenix
>> branch
>>>> per
>>>>>> HBase major version is practical, however. At the least, it would be
>>>> really
>>>>>> complex.
>>>>>>
>>>>>> For example, in the new year I plan to return to working on the
>> change
>>>> data
>>>>>> capture and Phoenix-level replication features, both of which depend
>>> on
>>>>>> WALKey interface changes and a new RegionObserver coprocessor hook
>>>>>> introduced in HBASE-22622 and HBASE-22623. This was released in
>> HBase
>>>> 1.5
>>>>>> and will be in the forthcoming HBase 2.3. While the HBase community
>> is
>>>>>> discussing EOMing 1.3 right now, and maybe 1.4 will go in the medium
>>>> term,
>>>>>> I don't see all pre-2.3 branch-2's getting deprecated anytime soon.
>>>>>>
>>>>>> So there will be at least two significant features that can only
>> exist
>>>> in
>>>>>> some but not all of our 4.x and 5.x branches.
>>>>>>
>>>>>> Geoffrey
>>>>>>
>>>>>> On Thu, Dec 12, 2019 at 8:21 AM Josh Elser <el...@apache.org>
>> wrote:
>>>>>>
>>>>>>> As much as possible, I'd like to avoid us getting into another
>>>> situation
>>>>>>> with 5.x where we have multiple branches. My hope was/is that we
>> can
>>>>>>> keep one Phoenix5 branch that works against an acceptable set of
>>> HBase
>>>>>>> branches.
>>>>>>>
>>>>>>> To me, that acceptable set of HBase branches is _a_ 2.1 and 2.2
>>>> release.
>>>>>>> I don't think we need to support all 2.1.x or 2.2.x, nor do I think
>>> we
>>>>>>> need to keep trying to maintain 2.0.x as it's already end of
>> support
>>> by
>>>>>>> the HBase community.
>>>>>>>
>>>>>>> Thanks for updating your PR. I'll add this to my review queue.
>>>>>>>
>>>>>>> On 12/12/19 1:52 AM, Istvan Toth wrote:
>>>>>>>> Hi!
>>>>>>>>
>>>>>>>> I'd like to start a conversation about supporting HBase 2.2. in
>> the
>>>>>>>> master branch.
>>>>>>>>
>>>>>>>> https://issues.apache.org/jira/browse/PHOENIX-5268 has a slightly
>>> out
>>>>>> of
>>>>>>>> date, but functional PR for HBase 2.2 support on master. (Please
>>>> review
>>>>>>>> and comment if you have the time, I'll try to update the PR in the
>>>> next
>>>>>>>> few days)
>>>>>>>>
>>>>>>>> The reason that it is not a straightforward decision to merge it
>> is
>>>>>> that
>>>>>>>> applying that patch breaks compatibility with HBase 2.0.1, the
>>> current
>>>>>>>> base.
>>>>>>>>
>>>>>>>> I can see the following outcomes:
>>>>>>>>
>>>>>>>> - Do nothing
>>>>>>>> - Move master to HBase 2.2.2
>>>>>>>> - Fork master to Hbase-2.0 and Hbase-2.2 branches
>>>>>>>> - Build time compatibility modules
>>>>>>>> - Run time compatibility modules
>>>>>>>> - Something that I haven't thought of
>>>>>>>>
>>>>>>>>
>>>>>>>> Doing nothing is obviously not a long term solution, as the
>> current
>>>>>>>> master doesn't work with any of the currently supported HBase
>>>> branches,
>>>>>>>> but we may postpone the inevitable.
>>>>>>>>
>>>>>>>> Simply moving master to HBase 2.2 is the most attractive solution
>>> from
>>>>>> a
>>>>>>>> pure developer POV, but there may be other considerations.
>>>>>>>>
>>>>>>>> Having multiple masters for 2.0 and 2.2 is simple from a code
>>>>>>>> perspective, but maintaining two branches is a non-trivial amount
>> of
>>>>>>>> additional work. (See the 4.x situation)
>>>>>>>>
>>>>>>>> Moving the HBase version dependent stuff into a separate module,
>> and
>>>>>>>> choosing at build time is not pretty from a code POV, but saves us
>>> the
>>>>>>>> hassle of maintaining multiple branches, while maintaining
>>>>>> compatibility
>>>>>>>> with multiple HBase versions, and can handle future API changes
>> as
>>>>>> well
>>>>>>>> from a single branch. Doing something like this could have saved
>> us
>>>> the
>>>>>>>> effort of maintaining three separate 4.x branches.
>>>>>>>>
>>>>>>>> I feel that since Phoenix is closely timed to HBase, and requires
>>>>>>>> cluster-wide HBase configuration to work anyway, handling the
>>>> different
>>>>>>>> HBase versions from the same binary/JAR is not worth the effort.
>>>>>>>>
>>>>>>>> Please share your thoughts!
>>>>>>>>
>>>>>>>> regards
>>>>>>>> Istvan
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>>
>> --
>> Best regards,
>> Andrew
>>
>> Words like orphans lost among the crosstalk, meaning torn from truth's
>> decrepit hands
>> - A23, Crosstalk
>>
>
>
>