You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Bharath Vissapragada <bh...@apache.org> on 2020/02/04 16:36:22 UTC
[DISCUSS] HBASE-18095 - Remove zookeeper dependency in client codepaths
Hello everyone,
I'd like to kickoff a discuss thread on dev@ to see what folks think about
merging the feature branch for HBASE-18095
<https://issues.apache.org/jira/browse/HBASE-18095> into the master. For
those of you who aren't following this work, over the last few months, a
lot of effort went into a feature branch
<https://github.com/apache/hbase/tree/HBASE-18095/client-locate-meta-no-zookeeper>
to
remove the ZK dependency in the client.
*Please refer to the design doc
<https://docs.google.com/document/d/1JAJdM7eUxg5b417f0xWS4NztKCx1f2b6wZrudPtiXF4/edit>
attached to the parent jira and go through the subtasks for all the
technical details and design considerations*.
*TL;DR*: With this feature, the client connection implementation *does not*
need access to zookeeper to fetch the connection metadata. Instead, a
predefined set of master end points in the configuration are used by
clients to fetch the required metadata.
This new feature is *enabled by default on the feature branch* and passes
the entire nightly test suite (modulo some known flakes not specific to the
branch). At this point, I'm not aware of any performance concerns / feature
gaps compared to original default implementation. The original registry
implementation is still retained and can be used by setting the following
client configuration. This kill switch gives the users more flexibility
since they have a fallback incase they run into any issues.
<property>
<!-- Reverts to the original ZK based connection registry
implementation -->
<name>hbase.client.registry.impl</name>
<value>org.apache.hadoop.hbase.client.ZKConnectionRegistry</value>
</property>
This work is also slated to go into the upcoming releases* 2.3.0* and
*1.6.0*. However, it will be *disabled by default*. Having this work back
ported to those branches enables users to try it out in their environments
and report any feedback.
Please speak up (respond to this email) if there are any objections to
merging this work in the master branch.
Many thanks to Nick Dimiduk, Andrew Purtell and Michael Stack for their
invaluable feedback throughout this work.
- Bharath
Re: [DISCUSS] HBASE-18095 - Remove zookeeper dependency in client codepaths
Posted by Andrew Purtell <an...@gmail.com>.
+1 from me as well
> On Feb 4, 2020, at 9:33 AM, Nick Dimiduk <nd...@apache.org> wrote:
>
> Thanks for the summary, and all your efforts on this enhancement, Bharath.
>
> Let me highlight here on point re: this architecture. Because the Master is
> use in place of ZooKeeper for clients to find their way around the cluster,
> this changes our operational requirements on the Master. With this feature
> enabled, operators will need to keep at least one Master up and responding
> to RPC's in order for clients to locate meta. As with the existing ZK-based
> meta location discovery, the location is cached by the client so as to not
> make excessive calls. There are tests in place that ensure this
> functionality behaves appropriately even if Master leader election somehow
> fails.
>
> As for me, I'm +1 on merging this implementation to master and branch-2.
>
> Thanks,
> Nick
>
>> On Tue, Feb 4, 2020 at 8:37 AM Bharath Vissapragada <bh...@apache.org>
>> wrote:
>>
>> Hello everyone,
>>
>> I'd like to kickoff a discuss thread on dev@ to see what folks think about
>> merging the feature branch for HBASE-18095
>> <https://issues.apache.org/jira/browse/HBASE-18095> into the master. For
>> those of you who aren't following this work, over the last few months, a
>> lot of effort went into a feature branch
>> <
>> https://github.com/apache/hbase/tree/HBASE-18095/client-locate-meta-no-zookeeper
>>>
>> to
>> remove the ZK dependency in the client.
>>
>> *Please refer to the design doc
>> <
>> https://docs.google.com/document/d/1JAJdM7eUxg5b417f0xWS4NztKCx1f2b6wZrudPtiXF4/edit
>>>
>> attached to the parent jira and go through the subtasks for all the
>> technical details and design considerations*.
>>
>> *TL;DR*: With this feature, the client connection implementation *does not*
>> need access to zookeeper to fetch the connection metadata. Instead, a
>> predefined set of master end points in the configuration are used by
>> clients to fetch the required metadata.
>>
>> This new feature is *enabled by default on the feature branch* and passes
>> the entire nightly test suite (modulo some known flakes not specific to the
>> branch). At this point, I'm not aware of any performance concerns / feature
>> gaps compared to original default implementation. The original registry
>> implementation is still retained and can be used by setting the following
>> client configuration. This kill switch gives the users more flexibility
>> since they have a fallback incase they run into any issues.
>>
>> <property>
>> <!-- Reverts to the original ZK based connection registry
>> implementation -->
>> <name>hbase.client.registry.impl</name>
>> <value>org.apache.hadoop.hbase.client.ZKConnectionRegistry</value>
>> </property>
>>
>> This work is also slated to go into the upcoming releases* 2.3.0* and
>> *1.6.0*. However, it will be *disabled by default*. Having this work back
>> ported to those branches enables users to try it out in their environments
>> and report any feedback.
>>
>> Please speak up (respond to this email) if there are any objections to
>> merging this work in the master branch.
>>
>> Many thanks to Nick Dimiduk, Andrew Purtell and Michael Stack for their
>> invaluable feedback throughout this work.
>>
>> - Bharath
>>
Re: [DISCUSS] HBASE-18095 - Remove zookeeper dependency in client codepaths
Posted by Nick Dimiduk <nd...@apache.org>.
Thanks for the summary, and all your efforts on this enhancement, Bharath.
Let me highlight here on point re: this architecture. Because the Master is
use in place of ZooKeeper for clients to find their way around the cluster,
this changes our operational requirements on the Master. With this feature
enabled, operators will need to keep at least one Master up and responding
to RPC's in order for clients to locate meta. As with the existing ZK-based
meta location discovery, the location is cached by the client so as to not
make excessive calls. There are tests in place that ensure this
functionality behaves appropriately even if Master leader election somehow
fails.
As for me, I'm +1 on merging this implementation to master and branch-2.
Thanks,
Nick
On Tue, Feb 4, 2020 at 8:37 AM Bharath Vissapragada <bh...@apache.org>
wrote:
> Hello everyone,
>
> I'd like to kickoff a discuss thread on dev@ to see what folks think about
> merging the feature branch for HBASE-18095
> <https://issues.apache.org/jira/browse/HBASE-18095> into the master. For
> those of you who aren't following this work, over the last few months, a
> lot of effort went into a feature branch
> <
> https://github.com/apache/hbase/tree/HBASE-18095/client-locate-meta-no-zookeeper
> >
> to
> remove the ZK dependency in the client.
>
> *Please refer to the design doc
> <
> https://docs.google.com/document/d/1JAJdM7eUxg5b417f0xWS4NztKCx1f2b6wZrudPtiXF4/edit
> >
> attached to the parent jira and go through the subtasks for all the
> technical details and design considerations*.
>
> *TL;DR*: With this feature, the client connection implementation *does not*
> need access to zookeeper to fetch the connection metadata. Instead, a
> predefined set of master end points in the configuration are used by
> clients to fetch the required metadata.
>
> This new feature is *enabled by default on the feature branch* and passes
> the entire nightly test suite (modulo some known flakes not specific to the
> branch). At this point, I'm not aware of any performance concerns / feature
> gaps compared to original default implementation. The original registry
> implementation is still retained and can be used by setting the following
> client configuration. This kill switch gives the users more flexibility
> since they have a fallback incase they run into any issues.
>
> <property>
> <!-- Reverts to the original ZK based connection registry
> implementation -->
> <name>hbase.client.registry.impl</name>
> <value>org.apache.hadoop.hbase.client.ZKConnectionRegistry</value>
> </property>
>
> This work is also slated to go into the upcoming releases* 2.3.0* and
> *1.6.0*. However, it will be *disabled by default*. Having this work back
> ported to those branches enables users to try it out in their environments
> and report any feedback.
>
> Please speak up (respond to this email) if there are any objections to
> merging this work in the master branch.
>
> Many thanks to Nick Dimiduk, Andrew Purtell and Michael Stack for their
> invaluable feedback throughout this work.
>
> - Bharath
>
Re: [DISCUSS] HBASE-18095 - Remove zookeeper dependency in client codepaths
Posted by Sean Busbey <bu...@apache.org>.
Bump. Bharath?
On Tue, Feb 4, 2020, 12:09 Sean Busbey <bu...@apache.org> wrote:
> I don't see the current design doc in the feature branch (i.e.
> dev-support/design-docs) please include it there.
>
> the current design doc has comments open still, should I assume those
> things haven't been addressed in the branch? or should I assume they
> have but it hasn't been updated yet?
>
> On Tue, Feb 4, 2020 at 10:37 AM Bharath Vissapragada
> <bh...@apache.org> wrote:
> >
> > Hello everyone,
> >
> > I'd like to kickoff a discuss thread on dev@ to see what folks think
> about
> > merging the feature branch for HBASE-18095
> > <https://issues.apache.org/jira/browse/HBASE-18095> into the master. For
> > those of you who aren't following this work, over the last few months, a
> > lot of effort went into a feature branch
> > <
> https://github.com/apache/hbase/tree/HBASE-18095/client-locate-meta-no-zookeeper
> >
> > to
> > remove the ZK dependency in the client.
> >
> > *Please refer to the design doc
> > <
> https://docs.google.com/document/d/1JAJdM7eUxg5b417f0xWS4NztKCx1f2b6wZrudPtiXF4/edit
> >
> > attached to the parent jira and go through the subtasks for all the
> > technical details and design considerations*.
> >
> > *TL;DR*: With this feature, the client connection implementation *does
> not*
> > need access to zookeeper to fetch the connection metadata. Instead, a
> > predefined set of master end points in the configuration are used by
> > clients to fetch the required metadata.
> >
> > This new feature is *enabled by default on the feature branch* and passes
> > the entire nightly test suite (modulo some known flakes not specific to
> the
> > branch). At this point, I'm not aware of any performance concerns /
> feature
> > gaps compared to original default implementation. The original registry
> > implementation is still retained and can be used by setting the following
> > client configuration. This kill switch gives the users more flexibility
> > since they have a fallback incase they run into any issues.
> >
> > <property>
> > <!-- Reverts to the original ZK based connection registry
> > implementation -->
> > <name>hbase.client.registry.impl</name>
> > <value>org.apache.hadoop.hbase.client.ZKConnectionRegistry</value>
> > </property>
> >
> > This work is also slated to go into the upcoming releases* 2.3.0* and
> > *1.6.0*. However, it will be *disabled by default*. Having this work back
> > ported to those branches enables users to try it out in their
> environments
> > and report any feedback.
> >
> > Please speak up (respond to this email) if there are any objections to
> > merging this work in the master branch.
> >
> > Many thanks to Nick Dimiduk, Andrew Purtell and Michael Stack for their
> > invaluable feedback throughout this work.
> >
> > - Bharath
>
Re: [DISCUSS] HBASE-18095 - Remove zookeeper dependency in client codepaths
Posted by Sean Busbey <bu...@apache.org>.
I don't see the current design doc in the feature branch (i.e.
dev-support/design-docs) please include it there.
the current design doc has comments open still, should I assume those
things haven't been addressed in the branch? or should I assume they
have but it hasn't been updated yet?
On Tue, Feb 4, 2020 at 10:37 AM Bharath Vissapragada
<bh...@apache.org> wrote:
>
> Hello everyone,
>
> I'd like to kickoff a discuss thread on dev@ to see what folks think about
> merging the feature branch for HBASE-18095
> <https://issues.apache.org/jira/browse/HBASE-18095> into the master. For
> those of you who aren't following this work, over the last few months, a
> lot of effort went into a feature branch
> <https://github.com/apache/hbase/tree/HBASE-18095/client-locate-meta-no-zookeeper>
> to
> remove the ZK dependency in the client.
>
> *Please refer to the design doc
> <https://docs.google.com/document/d/1JAJdM7eUxg5b417f0xWS4NztKCx1f2b6wZrudPtiXF4/edit>
> attached to the parent jira and go through the subtasks for all the
> technical details and design considerations*.
>
> *TL;DR*: With this feature, the client connection implementation *does not*
> need access to zookeeper to fetch the connection metadata. Instead, a
> predefined set of master end points in the configuration are used by
> clients to fetch the required metadata.
>
> This new feature is *enabled by default on the feature branch* and passes
> the entire nightly test suite (modulo some known flakes not specific to the
> branch). At this point, I'm not aware of any performance concerns / feature
> gaps compared to original default implementation. The original registry
> implementation is still retained and can be used by setting the following
> client configuration. This kill switch gives the users more flexibility
> since they have a fallback incase they run into any issues.
>
> <property>
> <!-- Reverts to the original ZK based connection registry
> implementation -->
> <name>hbase.client.registry.impl</name>
> <value>org.apache.hadoop.hbase.client.ZKConnectionRegistry</value>
> </property>
>
> This work is also slated to go into the upcoming releases* 2.3.0* and
> *1.6.0*. However, it will be *disabled by default*. Having this work back
> ported to those branches enables users to try it out in their environments
> and report any feedback.
>
> Please speak up (respond to this email) if there are any objections to
> merging this work in the master branch.
>
> Many thanks to Nick Dimiduk, Andrew Purtell and Michael Stack for their
> invaluable feedback throughout this work.
>
> - Bharath
Re: [DISCUSS] HBASE-18095 - Remove zookeeper dependency in client codepaths
Posted by Stack <st...@duboce.net>.
On Wed, Feb 19, 2020 at 8:14 PM Bharath Vissapragada <bh...@apache.org>
wrote:
> 'I merged the branch in master. I'll keep an eye on the nightly builds
> incase some issues pop up. If you see any weird issues in tests, especially
> around buggy 'Connection's, let me know. Thanks.
>
>
Hurray!
S
> On Tue, Feb 18, 2020 at 1:56 PM Bharath Vissapragada <bh...@apache.org>
> wrote:
>
> > I got a +1 from Sean. I'll merge the branch EOD tomorrow (Feb 19,
> Pacific)
> > unless anyone objects.
> >
> > On Thu, Feb 13, 2020 at 10:57 AM Bharath Vissapragada <
> bharathv@apache.org>
> > wrote:
> >
> >> Sure Sean. I'll wait for your +1 before merging. I see that you've
> >> already gone through the doc PR, let me address your comments.
> >>
> >> On Thu, Feb 13, 2020 at 6:25 AM Sean Busbey <bu...@apache.org> wrote:
> >>
> >>> Please do not merge until I have a chance to look through it. I can
> start
> >>> that today.
> >>>
> >>> On Wed, Feb 12, 2020, 19:11 Bharath Vissapragada <bh...@apache.org>
> >>> wrote:
> >>>
> >>> > The PR <https://github.com/apache/hbase/pull/1167> to merge the
> >>> feature
> >>> > branch is up for review. I'll wait for it to run all the tests. If
> >>> > everything looks good, I'll merge it by 13 Feb EOD pacific time
> >>> *unless*
> >>> > anyone has any objections. Please speak up here or on the pull
> request.
> >>> >
> >>> > On Thu, Feb 6, 2020 at 10:13 PM Bharath Vissapragada <
> >>> bharathv@apache.org>
> >>> > wrote:
> >>> >
> >>> > > @Sean Ack, let me prepare a PR for the doc.
> >>> > >
> >>> > > @Duo Zhang <zh...@apache.org>: No, the current plan is to ship
> >>> the ZK
> >>> > > dependency as-is with the client so that the users can easily
> switch
> >>> > > between the registries without rebuilding the client. I think we
> can
> >>> get
> >>> > > rid of the dependency (by depreciating first?) once we are 100%
> >>> confident
> >>> > > that there are no known issues with the new registry
> implementation.
> >>> This
> >>> > > might take some time.
> >>> > >
> >>> > > On Thu, Feb 6, 2020 at 7:12 PM 张铎(Duo Zhang) <
> palomino219@gmail.com>
> >>> > > wrote:
> >>> > >
> >>> > >> So we will declare zookeeper as an optional dependency for
> >>> hbase-client,
> >>> > >> and if users want to use the zk based registry, they should
> include
> >>> the
> >>> > zk
> >>> > >> dependency explictly in their pom?
> >>> > >>
> >>> > >> Sean Busbey <bu...@apache.org> 于2020年2月7日周五 上午9:39写道：
> >>> > >>
> >>> > >> > Oh, the response is over in this other subthread. Apologies for
> >>> the
> >>> > bump
> >>> > >> > elsethread. Comments below.
> >>> > >> >
> >>> > >> > On Wed, Feb 5, 2020, 01:37 Bharath Vissapragada <
> >>> bharathv@apache.org>
> >>> > >> > wrote:
> >>> > >> >
> >>> > >> > > Thanks everyone for chiming in. Sean, regarding your comments.
> >>> > >> > >
> >>> > >> > > > I don't see the current design doc in the feature branch
> >>> > >> > > (i.e.dev-support/design-docs) please include it there
> >>> > >> > >
> >>> > >> > > Of course, HBASE-23331 <
> >>> > >> > https://issues.apache.org/jira/browse/HBASE-23331>
> >>> > >> > > is
> >>> > >> > > the subtask for this. The plan is to update the ref guide with
> >>> all
> >>> > the
> >>> > >> > > details once the branch is merged in the master. I'll make
> sure
> >>> to
> >>> > add
> >>> > >> > the
> >>> > >> > > design doc too.
> >>> > >> > >
> >>> > >> > >
> >>> > >> > >
> >>> > >> > Please land proposed doc changes before merge to master. The
> >>> design
> >>> > doc
> >>> > >> > landing as a part of the merge is fine, but ref guide changes
> are
> >>> > >> something
> >>> > >> > that I'm going to want to look at when evaluating the feature
> for
> >>> > >> > suitability for landing.
> >>> > >> >
> >>> > >> >
> >>> > >> >
> >>> > >> > >
> >>> > >> > > > the current design doc has comments open still, should I
> >>> assume
> >>> > >> those
> >>> > >> > > things haven't been addressed in the branch? or should I
> assume
> >>> they
> >>> > >> have
> >>> > >> > > but it hasn't been updated yet?
> >>> > >> > >
> >>> > >> > > I addressed most of them already, forgot to resolve the
> >>> comments.
> >>> > >> There
> >>> > >> > > were some new comments since this email, so I addressed them
> and
> >>> > >> cleaned
> >>> > >> > up
> >>> > >> > > the doc. Thanks for pointing it out.
> >>> > >> > >
> >>> > >> >
> >>> > >> > Awesome. thanks! I'll take another look.
> >>> > >> >
> >>> > >>
> >>> > >
> >>> >
> >>>
> >>
>
Re: [DISCUSS] HBASE-18095 - Remove zookeeper dependency in client codepaths
Posted by Bharath Vissapragada <bh...@apache.org>.
'I merged the branch in master. I'll keep an eye on the nightly builds
incase some issues pop up. If you see any weird issues in tests, especially
around buggy 'Connection's, let me know. Thanks.
On Tue, Feb 18, 2020 at 1:56 PM Bharath Vissapragada <bh...@apache.org>
wrote:
> I got a +1 from Sean. I'll merge the branch EOD tomorrow (Feb 19, Pacific)
> unless anyone objects.
>
> On Thu, Feb 13, 2020 at 10:57 AM Bharath Vissapragada <bh...@apache.org>
> wrote:
>
>> Sure Sean. I'll wait for your +1 before merging. I see that you've
>> already gone through the doc PR, let me address your comments.
>>
>> On Thu, Feb 13, 2020 at 6:25 AM Sean Busbey <bu...@apache.org> wrote:
>>
>>> Please do not merge until I have a chance to look through it. I can start
>>> that today.
>>>
>>> On Wed, Feb 12, 2020, 19:11 Bharath Vissapragada <bh...@apache.org>
>>> wrote:
>>>
>>> > The PR <https://github.com/apache/hbase/pull/1167> to merge the
>>> feature
>>> > branch is up for review. I'll wait for it to run all the tests. If
>>> > everything looks good, I'll merge it by 13 Feb EOD pacific time
>>> *unless*
>>> > anyone has any objections. Please speak up here or on the pull request.
>>> >
>>> > On Thu, Feb 6, 2020 at 10:13 PM Bharath Vissapragada <
>>> bharathv@apache.org>
>>> > wrote:
>>> >
>>> > > @Sean Ack, let me prepare a PR for the doc.
>>> > >
>>> > > @Duo Zhang <zh...@apache.org>: No, the current plan is to ship
>>> the ZK
>>> > > dependency as-is with the client so that the users can easily switch
>>> > > between the registries without rebuilding the client. I think we can
>>> get
>>> > > rid of the dependency (by depreciating first?) once we are 100%
>>> confident
>>> > > that there are no known issues with the new registry implementation.
>>> This
>>> > > might take some time.
>>> > >
>>> > > On Thu, Feb 6, 2020 at 7:12 PM 张铎(Duo Zhang) <pa...@gmail.com>
>>> > > wrote:
>>> > >
>>> > >> So we will declare zookeeper as an optional dependency for
>>> hbase-client,
>>> > >> and if users want to use the zk based registry, they should include
>>> the
>>> > zk
>>> > >> dependency explictly in their pom?
>>> > >>
>>> > >> Sean Busbey <bu...@apache.org> 于2020年2月7日周五 上午9:39写道：
>>> > >>
>>> > >> > Oh, the response is over in this other subthread. Apologies for
>>> the
>>> > bump
>>> > >> > elsethread. Comments below.
>>> > >> >
>>> > >> > On Wed, Feb 5, 2020, 01:37 Bharath Vissapragada <
>>> bharathv@apache.org>
>>> > >> > wrote:
>>> > >> >
>>> > >> > > Thanks everyone for chiming in. Sean, regarding your comments.
>>> > >> > >
>>> > >> > > > I don't see the current design doc in the feature branch
>>> > >> > > (i.e.dev-support/design-docs) please include it there
>>> > >> > >
>>> > >> > > Of course, HBASE-23331 <
>>> > >> > https://issues.apache.org/jira/browse/HBASE-23331>
>>> > >> > > is
>>> > >> > > the subtask for this. The plan is to update the ref guide with
>>> all
>>> > the
>>> > >> > > details once the branch is merged in the master. I'll make sure
>>> to
>>> > add
>>> > >> > the
>>> > >> > > design doc too.
>>> > >> > >
>>> > >> > >
>>> > >> > >
>>> > >> > Please land proposed doc changes before merge to master. The
>>> design
>>> > doc
>>> > >> > landing as a part of the merge is fine, but ref guide changes are
>>> > >> something
>>> > >> > that I'm going to want to look at when evaluating the feature for
>>> > >> > suitability for landing.
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> > >
>>> > >> > > > the current design doc has comments open still, should I
>>> assume
>>> > >> those
>>> > >> > > things haven't been addressed in the branch? or should I assume
>>> they
>>> > >> have
>>> > >> > > but it hasn't been updated yet?
>>> > >> > >
>>> > >> > > I addressed most of them already, forgot to resolve the
>>> comments.
>>> > >> There
>>> > >> > > were some new comments since this email, so I addressed them and
>>> > >> cleaned
>>> > >> > up
>>> > >> > > the doc. Thanks for pointing it out.
>>> > >> > >
>>> > >> >
>>> > >> > Awesome. thanks! I'll take another look.
>>> > >> >
>>> > >>
>>> > >
>>> >
>>>
>>
Re: [DISCUSS] HBASE-18095 - Remove zookeeper dependency in client codepaths
Posted by Bharath Vissapragada <bh...@apache.org>.
I got a +1 from Sean. I'll merge the branch EOD tomorrow (Feb 19, Pacific)
unless anyone objects.
On Thu, Feb 13, 2020 at 10:57 AM Bharath Vissapragada <bh...@apache.org>
wrote:
> Sure Sean. I'll wait for your +1 before merging. I see that you've already
> gone through the doc PR, let me address your comments.
>
> On Thu, Feb 13, 2020 at 6:25 AM Sean Busbey <bu...@apache.org> wrote:
>
>> Please do not merge until I have a chance to look through it. I can start
>> that today.
>>
>> On Wed, Feb 12, 2020, 19:11 Bharath Vissapragada <bh...@apache.org>
>> wrote:
>>
>> > The PR <https://github.com/apache/hbase/pull/1167> to merge the feature
>> > branch is up for review. I'll wait for it to run all the tests. If
>> > everything looks good, I'll merge it by 13 Feb EOD pacific time *unless*
>> > anyone has any objections. Please speak up here or on the pull request.
>> >
>> > On Thu, Feb 6, 2020 at 10:13 PM Bharath Vissapragada <
>> bharathv@apache.org>
>> > wrote:
>> >
>> > > @Sean Ack, let me prepare a PR for the doc.
>> > >
>> > > @Duo Zhang <zh...@apache.org>: No, the current plan is to ship
>> the ZK
>> > > dependency as-is with the client so that the users can easily switch
>> > > between the registries without rebuilding the client. I think we can
>> get
>> > > rid of the dependency (by depreciating first?) once we are 100%
>> confident
>> > > that there are no known issues with the new registry implementation.
>> This
>> > > might take some time.
>> > >
>> > > On Thu, Feb 6, 2020 at 7:12 PM 张铎(Duo Zhang) <pa...@gmail.com>
>> > > wrote:
>> > >
>> > >> So we will declare zookeeper as an optional dependency for
>> hbase-client,
>> > >> and if users want to use the zk based registry, they should include
>> the
>> > zk
>> > >> dependency explictly in their pom?
>> > >>
>> > >> Sean Busbey <bu...@apache.org> 于2020年2月7日周五 上午9:39写道：
>> > >>
>> > >> > Oh, the response is over in this other subthread. Apologies for the
>> > bump
>> > >> > elsethread. Comments below.
>> > >> >
>> > >> > On Wed, Feb 5, 2020, 01:37 Bharath Vissapragada <
>> bharathv@apache.org>
>> > >> > wrote:
>> > >> >
>> > >> > > Thanks everyone for chiming in. Sean, regarding your comments.
>> > >> > >
>> > >> > > > I don't see the current design doc in the feature branch
>> > >> > > (i.e.dev-support/design-docs) please include it there
>> > >> > >
>> > >> > > Of course, HBASE-23331 <
>> > >> > https://issues.apache.org/jira/browse/HBASE-23331>
>> > >> > > is
>> > >> > > the subtask for this. The plan is to update the ref guide with
>> all
>> > the
>> > >> > > details once the branch is merged in the master. I'll make sure
>> to
>> > add
>> > >> > the
>> > >> > > design doc too.
>> > >> > >
>> > >> > >
>> > >> > >
>> > >> > Please land proposed doc changes before merge to master. The design
>> > doc
>> > >> > landing as a part of the merge is fine, but ref guide changes are
>> > >> something
>> > >> > that I'm going to want to look at when evaluating the feature for
>> > >> > suitability for landing.
>> > >> >
>> > >> >
>> > >> >
>> > >> > >
>> > >> > > > the current design doc has comments open still, should I assume
>> > >> those
>> > >> > > things haven't been addressed in the branch? or should I assume
>> they
>> > >> have
>> > >> > > but it hasn't been updated yet?
>> > >> > >
>> > >> > > I addressed most of them already, forgot to resolve the comments.
>> > >> There
>> > >> > > were some new comments since this email, so I addressed them and
>> > >> cleaned
>> > >> > up
>> > >> > > the doc. Thanks for pointing it out.
>> > >> > >
>> > >> >
>> > >> > Awesome. thanks! I'll take another look.
>> > >> >
>> > >>
>> > >
>> >
>>
>
Re: [DISCUSS] HBASE-18095 - Remove zookeeper dependency in client codepaths
Posted by Bharath Vissapragada <bh...@apache.org>.
Sure Sean. I'll wait for your +1 before merging. I see that you've already
gone through the doc PR, let me address your comments.
On Thu, Feb 13, 2020 at 6:25 AM Sean Busbey <bu...@apache.org> wrote:
> Please do not merge until I have a chance to look through it. I can start
> that today.
>
> On Wed, Feb 12, 2020, 19:11 Bharath Vissapragada <bh...@apache.org>
> wrote:
>
> > The PR <https://github.com/apache/hbase/pull/1167> to merge the feature
> > branch is up for review. I'll wait for it to run all the tests. If
> > everything looks good, I'll merge it by 13 Feb EOD pacific time *unless*
> > anyone has any objections. Please speak up here or on the pull request.
> >
> > On Thu, Feb 6, 2020 at 10:13 PM Bharath Vissapragada <
> bharathv@apache.org>
> > wrote:
> >
> > > @Sean Ack, let me prepare a PR for the doc.
> > >
> > > @Duo Zhang <zh...@apache.org>: No, the current plan is to ship the
> ZK
> > > dependency as-is with the client so that the users can easily switch
> > > between the registries without rebuilding the client. I think we can
> get
> > > rid of the dependency (by depreciating first?) once we are 100%
> confident
> > > that there are no known issues with the new registry implementation.
> This
> > > might take some time.
> > >
> > > On Thu, Feb 6, 2020 at 7:12 PM 张铎(Duo Zhang) <pa...@gmail.com>
> > > wrote:
> > >
> > >> So we will declare zookeeper as an optional dependency for
> hbase-client,
> > >> and if users want to use the zk based registry, they should include
> the
> > zk
> > >> dependency explictly in their pom?
> > >>
> > >> Sean Busbey <bu...@apache.org> 于2020年2月7日周五 上午9:39写道：
> > >>
> > >> > Oh, the response is over in this other subthread. Apologies for the
> > bump
> > >> > elsethread. Comments below.
> > >> >
> > >> > On Wed, Feb 5, 2020, 01:37 Bharath Vissapragada <
> bharathv@apache.org>
> > >> > wrote:
> > >> >
> > >> > > Thanks everyone for chiming in. Sean, regarding your comments.
> > >> > >
> > >> > > > I don't see the current design doc in the feature branch
> > >> > > (i.e.dev-support/design-docs) please include it there
> > >> > >
> > >> > > Of course, HBASE-23331 <
> > >> > https://issues.apache.org/jira/browse/HBASE-23331>
> > >> > > is
> > >> > > the subtask for this. The plan is to update the ref guide with all
> > the
> > >> > > details once the branch is merged in the master. I'll make sure to
> > add
> > >> > the
> > >> > > design doc too.
> > >> > >
> > >> > >
> > >> > >
> > >> > Please land proposed doc changes before merge to master. The design
> > doc
> > >> > landing as a part of the merge is fine, but ref guide changes are
> > >> something
> > >> > that I'm going to want to look at when evaluating the feature for
> > >> > suitability for landing.
> > >> >
> > >> >
> > >> >
> > >> > >
> > >> > > > the current design doc has comments open still, should I assume
> > >> those
> > >> > > things haven't been addressed in the branch? or should I assume
> they
> > >> have
> > >> > > but it hasn't been updated yet?
> > >> > >
> > >> > > I addressed most of them already, forgot to resolve the comments.
> > >> There
> > >> > > were some new comments since this email, so I addressed them and
> > >> cleaned
> > >> > up
> > >> > > the doc. Thanks for pointing it out.
> > >> > >
> > >> >
> > >> > Awesome. thanks! I'll take another look.
> > >> >
> > >>
> > >
> >
>
Re: [DISCUSS] HBASE-18095 - Remove zookeeper dependency in client codepaths
Posted by Sean Busbey <bu...@apache.org>.
Please do not merge until I have a chance to look through it. I can start
that today.
On Wed, Feb 12, 2020, 19:11 Bharath Vissapragada <bh...@apache.org>
wrote:
> The PR <https://github.com/apache/hbase/pull/1167> to merge the feature
> branch is up for review. I'll wait for it to run all the tests. If
> everything looks good, I'll merge it by 13 Feb EOD pacific time *unless*
> anyone has any objections. Please speak up here or on the pull request.
>
> On Thu, Feb 6, 2020 at 10:13 PM Bharath Vissapragada <bh...@apache.org>
> wrote:
>
> > @Sean Ack, let me prepare a PR for the doc.
> >
> > @Duo Zhang <zh...@apache.org>: No, the current plan is to ship the ZK
> > dependency as-is with the client so that the users can easily switch
> > between the registries without rebuilding the client. I think we can get
> > rid of the dependency (by depreciating first?) once we are 100% confident
> > that there are no known issues with the new registry implementation. This
> > might take some time.
> >
> > On Thu, Feb 6, 2020 at 7:12 PM 张铎(Duo Zhang) <pa...@gmail.com>
> > wrote:
> >
> >> So we will declare zookeeper as an optional dependency for hbase-client,
> >> and if users want to use the zk based registry, they should include the
> zk
> >> dependency explictly in their pom?
> >>
> >> Sean Busbey <bu...@apache.org> 于2020年2月7日周五 上午9:39写道：
> >>
> >> > Oh, the response is over in this other subthread. Apologies for the
> bump
> >> > elsethread. Comments below.
> >> >
> >> > On Wed, Feb 5, 2020, 01:37 Bharath Vissapragada <bh...@apache.org>
> >> > wrote:
> >> >
> >> > > Thanks everyone for chiming in. Sean, regarding your comments.
> >> > >
> >> > > > I don't see the current design doc in the feature branch
> >> > > (i.e.dev-support/design-docs) please include it there
> >> > >
> >> > > Of course, HBASE-23331 <
> >> > https://issues.apache.org/jira/browse/HBASE-23331>
> >> > > is
> >> > > the subtask for this. The plan is to update the ref guide with all
> the
> >> > > details once the branch is merged in the master. I'll make sure to
> add
> >> > the
> >> > > design doc too.
> >> > >
> >> > >
> >> > >
> >> > Please land proposed doc changes before merge to master. The design
> doc
> >> > landing as a part of the merge is fine, but ref guide changes are
> >> something
> >> > that I'm going to want to look at when evaluating the feature for
> >> > suitability for landing.
> >> >
> >> >
> >> >
> >> > >
> >> > > > the current design doc has comments open still, should I assume
> >> those
> >> > > things haven't been addressed in the branch? or should I assume they
> >> have
> >> > > but it hasn't been updated yet?
> >> > >
> >> > > I addressed most of them already, forgot to resolve the comments.
> >> There
> >> > > were some new comments since this email, so I addressed them and
> >> cleaned
> >> > up
> >> > > the doc. Thanks for pointing it out.
> >> > >
> >> >
> >> > Awesome. thanks! I'll take another look.
> >> >
> >>
> >
>
Re: [DISCUSS] HBASE-18095 - Remove zookeeper dependency in client codepaths
Posted by Bharath Vissapragada <bh...@apache.org>.
The PR <https://github.com/apache/hbase/pull/1167> to merge the feature
branch is up for review. I'll wait for it to run all the tests. If
everything looks good, I'll merge it by 13 Feb EOD pacific time *unless*
anyone has any objections. Please speak up here or on the pull request.
On Thu, Feb 6, 2020 at 10:13 PM Bharath Vissapragada <bh...@apache.org>
wrote:
> @Sean Ack, let me prepare a PR for the doc.
>
> @Duo Zhang <zh...@apache.org>: No, the current plan is to ship the ZK
> dependency as-is with the client so that the users can easily switch
> between the registries without rebuilding the client. I think we can get
> rid of the dependency (by depreciating first?) once we are 100% confident
> that there are no known issues with the new registry implementation. This
> might take some time.
>
> On Thu, Feb 6, 2020 at 7:12 PM 张铎(Duo Zhang) <pa...@gmail.com>
> wrote:
>
>> So we will declare zookeeper as an optional dependency for hbase-client,
>> and if users want to use the zk based registry, they should include the zk
>> dependency explictly in their pom?
>>
>> Sean Busbey <bu...@apache.org> 于2020年2月7日周五 上午9:39写道：
>>
>> > Oh, the response is over in this other subthread. Apologies for the bump
>> > elsethread. Comments below.
>> >
>> > On Wed, Feb 5, 2020, 01:37 Bharath Vissapragada <bh...@apache.org>
>> > wrote:
>> >
>> > > Thanks everyone for chiming in. Sean, regarding your comments.
>> > >
>> > > > I don't see the current design doc in the feature branch
>> > > (i.e.dev-support/design-docs) please include it there
>> > >
>> > > Of course, HBASE-23331 <
>> > https://issues.apache.org/jira/browse/HBASE-23331>
>> > > is
>> > > the subtask for this. The plan is to update the ref guide with all the
>> > > details once the branch is merged in the master. I'll make sure to add
>> > the
>> > > design doc too.
>> > >
>> > >
>> > >
>> > Please land proposed doc changes before merge to master. The design doc
>> > landing as a part of the merge is fine, but ref guide changes are
>> something
>> > that I'm going to want to look at when evaluating the feature for
>> > suitability for landing.
>> >
>> >
>> >
>> > >
>> > > > the current design doc has comments open still, should I assume
>> those
>> > > things haven't been addressed in the branch? or should I assume they
>> have
>> > > but it hasn't been updated yet?
>> > >
>> > > I addressed most of them already, forgot to resolve the comments.
>> There
>> > > were some new comments since this email, so I addressed them and
>> cleaned
>> > up
>> > > the doc. Thanks for pointing it out.
>> > >
>> >
>> > Awesome. thanks! I'll take another look.
>> >
>>
>
Re: [DISCUSS] HBASE-18095 - Remove zookeeper dependency in client codepaths
Posted by Bharath Vissapragada <bh...@apache.org>.
@Sean Ack, let me prepare a PR for the doc.
@Duo Zhang <zh...@apache.org>: No, the current plan is to ship the ZK
dependency as-is with the client so that the users can easily switch
between the registries without rebuilding the client. I think we can get
rid of the dependency (by depreciating first?) once we are 100% confident
that there are no known issues with the new registry implementation. This
might take some time.
On Thu, Feb 6, 2020 at 7:12 PM 张铎(Duo Zhang) <pa...@gmail.com> wrote:
> So we will declare zookeeper as an optional dependency for hbase-client,
> and if users want to use the zk based registry, they should include the zk
> dependency explictly in their pom?
>
> Sean Busbey <bu...@apache.org> 于2020年2月7日周五 上午9:39写道：
>
> > Oh, the response is over in this other subthread. Apologies for the bump
> > elsethread. Comments below.
> >
> > On Wed, Feb 5, 2020, 01:37 Bharath Vissapragada <bh...@apache.org>
> > wrote:
> >
> > > Thanks everyone for chiming in. Sean, regarding your comments.
> > >
> > > > I don't see the current design doc in the feature branch
> > > (i.e.dev-support/design-docs) please include it there
> > >
> > > Of course, HBASE-23331 <
> > https://issues.apache.org/jira/browse/HBASE-23331>
> > > is
> > > the subtask for this. The plan is to update the ref guide with all the
> > > details once the branch is merged in the master. I'll make sure to add
> > the
> > > design doc too.
> > >
> > >
> > >
> > Please land proposed doc changes before merge to master. The design doc
> > landing as a part of the merge is fine, but ref guide changes are
> something
> > that I'm going to want to look at when evaluating the feature for
> > suitability for landing.
> >
> >
> >
> > >
> > > > the current design doc has comments open still, should I assume those
> > > things haven't been addressed in the branch? or should I assume they
> have
> > > but it hasn't been updated yet?
> > >
> > > I addressed most of them already, forgot to resolve the comments. There
> > > were some new comments since this email, so I addressed them and
> cleaned
> > up
> > > the doc. Thanks for pointing it out.
> > >
> >
> > Awesome. thanks! I'll take another look.
> >
>
Re: [DISCUSS] HBASE-18095 - Remove zookeeper dependency in client codepaths
Posted by "张铎 (Duo Zhang)" <pa...@gmail.com>.
So we will declare zookeeper as an optional dependency for hbase-client,
and if users want to use the zk based registry, they should include the zk
dependency explictly in their pom?
Sean Busbey <bu...@apache.org> 于2020年2月7日周五 上午9:39写道：
> Oh, the response is over in this other subthread. Apologies for the bump
> elsethread. Comments below.
>
> On Wed, Feb 5, 2020, 01:37 Bharath Vissapragada <bh...@apache.org>
> wrote:
>
> > Thanks everyone for chiming in. Sean, regarding your comments.
> >
> > > I don't see the current design doc in the feature branch
> > (i.e.dev-support/design-docs) please include it there
> >
> > Of course, HBASE-23331 <
> https://issues.apache.org/jira/browse/HBASE-23331>
> > is
> > the subtask for this. The plan is to update the ref guide with all the
> > details once the branch is merged in the master. I'll make sure to add
> the
> > design doc too.
> >
> >
> >
> Please land proposed doc changes before merge to master. The design doc
> landing as a part of the merge is fine, but ref guide changes are something
> that I'm going to want to look at when evaluating the feature for
> suitability for landing.
>
>
>
> >
> > > the current design doc has comments open still, should I assume those
> > things haven't been addressed in the branch? or should I assume they have
> > but it hasn't been updated yet?
> >
> > I addressed most of them already, forgot to resolve the comments. There
> > were some new comments since this email, so I addressed them and cleaned
> up
> > the doc. Thanks for pointing it out.
> >
>
> Awesome. thanks! I'll take another look.
>
Re: [DISCUSS] HBASE-18095 - Remove zookeeper dependency in client codepaths
Posted by Sean Busbey <bu...@apache.org>.
Oh, the response is over in this other subthread. Apologies for the bump
elsethread. Comments below.
On Wed, Feb 5, 2020, 01:37 Bharath Vissapragada <bh...@apache.org> wrote:
> Thanks everyone for chiming in. Sean, regarding your comments.
>
> > I don't see the current design doc in the feature branch
> (i.e.dev-support/design-docs) please include it there
>
> Of course, HBASE-23331 <https://issues.apache.org/jira/browse/HBASE-23331>
> is
> the subtask for this. The plan is to update the ref guide with all the
> details once the branch is merged in the master. I'll make sure to add the
> design doc too.
>
>
>
Please land proposed doc changes before merge to master. The design doc
landing as a part of the merge is fine, but ref guide changes are something
that I'm going to want to look at when evaluating the feature for
suitability for landing.
>
> > the current design doc has comments open still, should I assume those
> things haven't been addressed in the branch? or should I assume they have
> but it hasn't been updated yet?
>
> I addressed most of them already, forgot to resolve the comments. There
> were some new comments since this email, so I addressed them and cleaned up
> the doc. Thanks for pointing it out.
>
Awesome. thanks! I'll take another look.
Re: [DISCUSS] HBASE-18095 - Remove zookeeper dependency in client codepaths
Posted by Zach York <zy...@gmail.com>.
I have reviewed the doc and it looks like a good feature!
+1
On Thu, Feb 6, 2020 at 11:47 AM Wellington Chevreuil <
wellington.chevreuil@gmail.com> wrote:
> Thanks for clarifying on this, Bharath and Andrew. Sorry for the late
> reply, +1 for adding it into branch-2 as well as non-default registry.
>
> Em qui., 6 de fev. de 2020 às 00:59, Bharath Vissapragada <
> bharathv@apache.org> escreveu:
>
> > Whatever Andrew said. Wellington, I also addressed your comments in the
> > doc. Thanks for taking your time and going through it. Appreciate it.
> >
> > On Wed, Feb 5, 2020 at 2:29 PM Andrew Purtell <an...@gmail.com>
> > wrote:
> >
> > > Which registry to use is a client side configuration so old clients are
> > > unaffected. (At some future time an operator might want to foreclose on
> > > direct ZK access with network or host ACLs but of course this is
> totally
> > up
> > > to them and they are in complete control.)
> > >
> > > > On Feb 5, 2020, at 2:12 PM, Wellington Chevreuil <
> > > wellington.chevreuil@gmail.com> wrote:
> > > >
> > > > Thanks for the detailed summary, Bharath. I'm +1 for master.
> > > >
> > > > Just additional question I have, it wasn't clear for me on the
> > > doc/summary:
> > > > does it consider fallback to ZK based registry, in case of clients
> > > running
> > > > old versions on clusters with this feature enabled?
> > > >
> > > >> Em qua., 5 de fev. de 2020 às 07:37, Bharath Vissapragada <
> > > >> bharathv@apache.org> escreveu:
> > > >>
> > > >> Thanks everyone for chiming in. Sean, regarding your comments.
> > > >>
> > > >>> I don't see the current design doc in the feature branch
> > > >> (i.e.dev-support/design-docs) please include it there
> > > >>
> > > >> Of course, HBASE-23331 <
> > > https://issues.apache.org/jira/browse/HBASE-23331>
> > > >> is
> > > >> the subtask for this. The plan is to update the ref guide with all
> the
> > > >> details once the branch is merged in the master. I'll make sure to
> add
> > > the
> > > >> design doc too.
> > > >>
> > > >>> the current design doc has comments open still, should I assume
> those
> > > >> things haven't been addressed in the branch? or should I assume they
> > > have
> > > >> but it hasn't been updated yet?
> > > >>
> > > >> I addressed most of them already, forgot to resolve the comments.
> > There
> > > >> were some new comments since this email, so I addressed them and
> > > cleaned up
> > > >> the doc. Thanks for pointing it out.
> > > >>
> > > >>> On Tue, Feb 4, 2020 at 10:15 PM Stack <st...@duboce.net> wrote:
> > > >>>
> > > >>> I'm +1 for merge to master with it default enabled and to branch-2
> > with
> > > >> it
> > > >>> off by default.
> > > >>>
> > > >>> Nice work Bharath.
> > > >>>
> > > >>> S
> > > >>>
> > > >>> On Tue, Feb 4, 2020 at 8:37 AM Bharath Vissapragada <
> > > bharathv@apache.org
> > > >>>
> > > >>> wrote:
> > > >>>
> > > >>>> Hello everyone,
> > > >>>>
> > > >>>> I'd like to kickoff a discuss thread on dev@ to see what folks
> > think
> > > >>> about
> > > >>>> merging the feature branch for HBASE-18095
> > > >>>> <https://issues.apache.org/jira/browse/HBASE-18095> into the
> > master.
> > > >> For
> > > >>>> those of you who aren't following this work, over the last few
> > months,
> > > >> a
> > > >>>> lot of effort went into a feature branch
> > > >>>> <
> > > >>>>
> > > >>>
> > > >>
> > >
> >
> https://github.com/apache/hbase/tree/HBASE-18095/client-locate-meta-no-zookeeper
> > > >>>>>
> > > >>>> to
> > > >>>> remove the ZK dependency in the client.
> > > >>>>
> > > >>>> *Please refer to the design doc
> > > >>>> <
> > > >>>>
> > > >>>
> > > >>
> > >
> >
> https://docs.google.com/document/d/1JAJdM7eUxg5b417f0xWS4NztKCx1f2b6wZrudPtiXF4/edit
> > > >>>>>
> > > >>>> attached to the parent jira and go through the subtasks for all
> the
> > > >>>> technical details and design considerations*.
> > > >>>>
> > > >>>> *TL;DR*: With this feature, the client connection implementation
> > *does
> > > >>> not*
> > > >>>> need access to zookeeper to fetch the connection metadata.
> Instead,
> > a
> > > >>>> predefined set of master end points in the configuration are used
> by
> > > >>>> clients to fetch the required metadata.
> > > >>>>
> > > >>>> This new feature is *enabled by default on the feature branch* and
> > > >> passes
> > > >>>> the entire nightly test suite (modulo some known flakes not
> specific
> > > to
> > > >>> the
> > > >>>> branch). At this point, I'm not aware of any performance concerns
> /
> > > >>> feature
> > > >>>> gaps compared to original default implementation. The original
> > > registry
> > > >>>> implementation is still retained and can be used by setting the
> > > >> following
> > > >>>> client configuration. This kill switch gives the users more
> > > flexibility
> > > >>>> since they have a fallback incase they run into any issues.
> > > >>>>
> > > >>>> <property>
> > > >>>> <!-- Reverts to the original ZK based connection registry
> > > >>>> implementation -->
> > > >>>> <name>hbase.client.registry.impl</name>
> > > >>>>
> > <value>org.apache.hadoop.hbase.client.ZKConnectionRegistry</value>
> > > >>>> </property>
> > > >>>>
> > > >>>> This work is also slated to go into the upcoming releases* 2.3.0*
> > and
> > > >>>> *1.6.0*. However, it will be *disabled by default*. Having this
> work
> > > >> back
> > > >>>> ported to those branches enables users to try it out in their
> > > >>> environments
> > > >>>> and report any feedback.
> > > >>>>
> > > >>>> Please speak up (respond to this email) if there are any
> objections
> > to
> > > >>>> merging this work in the master branch.
> > > >>>>
> > > >>>> Many thanks to Nick Dimiduk, Andrew Purtell and Michael Stack for
> > > their
> > > >>>> invaluable feedback throughout this work.
> > > >>>>
> > > >>>> - Bharath
> > > >>>>
> > > >>>
> > > >>
> > >
> >
>
Re: [DISCUSS] HBASE-18095 - Remove zookeeper dependency in client codepaths
Posted by Wellington Chevreuil <we...@gmail.com>.
Thanks for clarifying on this, Bharath and Andrew. Sorry for the late
reply, +1 for adding it into branch-2 as well as non-default registry.
Em qui., 6 de fev. de 2020 às 00:59, Bharath Vissapragada <
bharathv@apache.org> escreveu:
> Whatever Andrew said. Wellington, I also addressed your comments in the
> doc. Thanks for taking your time and going through it. Appreciate it.
>
> On Wed, Feb 5, 2020 at 2:29 PM Andrew Purtell <an...@gmail.com>
> wrote:
>
> > Which registry to use is a client side configuration so old clients are
> > unaffected. (At some future time an operator might want to foreclose on
> > direct ZK access with network or host ACLs but of course this is totally
> up
> > to them and they are in complete control.)
> >
> > > On Feb 5, 2020, at 2:12 PM, Wellington Chevreuil <
> > wellington.chevreuil@gmail.com> wrote:
> > >
> > > Thanks for the detailed summary, Bharath. I'm +1 for master.
> > >
> > > Just additional question I have, it wasn't clear for me on the
> > doc/summary:
> > > does it consider fallback to ZK based registry, in case of clients
> > running
> > > old versions on clusters with this feature enabled?
> > >
> > >> Em qua., 5 de fev. de 2020 às 07:37, Bharath Vissapragada <
> > >> bharathv@apache.org> escreveu:
> > >>
> > >> Thanks everyone for chiming in. Sean, regarding your comments.
> > >>
> > >>> I don't see the current design doc in the feature branch
> > >> (i.e.dev-support/design-docs) please include it there
> > >>
> > >> Of course, HBASE-23331 <
> > https://issues.apache.org/jira/browse/HBASE-23331>
> > >> is
> > >> the subtask for this. The plan is to update the ref guide with all the
> > >> details once the branch is merged in the master. I'll make sure to add
> > the
> > >> design doc too.
> > >>
> > >>> the current design doc has comments open still, should I assume those
> > >> things haven't been addressed in the branch? or should I assume they
> > have
> > >> but it hasn't been updated yet?
> > >>
> > >> I addressed most of them already, forgot to resolve the comments.
> There
> > >> were some new comments since this email, so I addressed them and
> > cleaned up
> > >> the doc. Thanks for pointing it out.
> > >>
> > >>> On Tue, Feb 4, 2020 at 10:15 PM Stack <st...@duboce.net> wrote:
> > >>>
> > >>> I'm +1 for merge to master with it default enabled and to branch-2
> with
> > >> it
> > >>> off by default.
> > >>>
> > >>> Nice work Bharath.
> > >>>
> > >>> S
> > >>>
> > >>> On Tue, Feb 4, 2020 at 8:37 AM Bharath Vissapragada <
> > bharathv@apache.org
> > >>>
> > >>> wrote:
> > >>>
> > >>>> Hello everyone,
> > >>>>
> > >>>> I'd like to kickoff a discuss thread on dev@ to see what folks
> think
> > >>> about
> > >>>> merging the feature branch for HBASE-18095
> > >>>> <https://issues.apache.org/jira/browse/HBASE-18095> into the
> master.
> > >> For
> > >>>> those of you who aren't following this work, over the last few
> months,
> > >> a
> > >>>> lot of effort went into a feature branch
> > >>>> <
> > >>>>
> > >>>
> > >>
> >
> https://github.com/apache/hbase/tree/HBASE-18095/client-locate-meta-no-zookeeper
> > >>>>>
> > >>>> to
> > >>>> remove the ZK dependency in the client.
> > >>>>
> > >>>> *Please refer to the design doc
> > >>>> <
> > >>>>
> > >>>
> > >>
> >
> https://docs.google.com/document/d/1JAJdM7eUxg5b417f0xWS4NztKCx1f2b6wZrudPtiXF4/edit
> > >>>>>
> > >>>> attached to the parent jira and go through the subtasks for all the
> > >>>> technical details and design considerations*.
> > >>>>
> > >>>> *TL;DR*: With this feature, the client connection implementation
> *does
> > >>> not*
> > >>>> need access to zookeeper to fetch the connection metadata. Instead,
> a
> > >>>> predefined set of master end points in the configuration are used by
> > >>>> clients to fetch the required metadata.
> > >>>>
> > >>>> This new feature is *enabled by default on the feature branch* and
> > >> passes
> > >>>> the entire nightly test suite (modulo some known flakes not specific
> > to
> > >>> the
> > >>>> branch). At this point, I'm not aware of any performance concerns /
> > >>> feature
> > >>>> gaps compared to original default implementation. The original
> > registry
> > >>>> implementation is still retained and can be used by setting the
> > >> following
> > >>>> client configuration. This kill switch gives the users more
> > flexibility
> > >>>> since they have a fallback incase they run into any issues.
> > >>>>
> > >>>> <property>
> > >>>> <!-- Reverts to the original ZK based connection registry
> > >>>> implementation -->
> > >>>> <name>hbase.client.registry.impl</name>
> > >>>>
> <value>org.apache.hadoop.hbase.client.ZKConnectionRegistry</value>
> > >>>> </property>
> > >>>>
> > >>>> This work is also slated to go into the upcoming releases* 2.3.0*
> and
> > >>>> *1.6.0*. However, it will be *disabled by default*. Having this work
> > >> back
> > >>>> ported to those branches enables users to try it out in their
> > >>> environments
> > >>>> and report any feedback.
> > >>>>
> > >>>> Please speak up (respond to this email) if there are any objections
> to
> > >>>> merging this work in the master branch.
> > >>>>
> > >>>> Many thanks to Nick Dimiduk, Andrew Purtell and Michael Stack for
> > their
> > >>>> invaluable feedback throughout this work.
> > >>>>
> > >>>> - Bharath
> > >>>>
> > >>>
> > >>
> >
>
Re: [DISCUSS] HBASE-18095 - Remove zookeeper dependency in client codepaths
Posted by Bharath Vissapragada <bh...@apache.org>.
Whatever Andrew said. Wellington, I also addressed your comments in the
doc. Thanks for taking your time and going through it. Appreciate it.
On Wed, Feb 5, 2020 at 2:29 PM Andrew Purtell <an...@gmail.com>
wrote:
> Which registry to use is a client side configuration so old clients are
> unaffected. (At some future time an operator might want to foreclose on
> direct ZK access with network or host ACLs but of course this is totally up
> to them and they are in complete control.)
>
> > On Feb 5, 2020, at 2:12 PM, Wellington Chevreuil <
> wellington.chevreuil@gmail.com> wrote:
> >
> > Thanks for the detailed summary, Bharath. I'm +1 for master.
> >
> > Just additional question I have, it wasn't clear for me on the
> doc/summary:
> > does it consider fallback to ZK based registry, in case of clients
> running
> > old versions on clusters with this feature enabled?
> >
> >> Em qua., 5 de fev. de 2020 às 07:37, Bharath Vissapragada <
> >> bharathv@apache.org> escreveu:
> >>
> >> Thanks everyone for chiming in. Sean, regarding your comments.
> >>
> >>> I don't see the current design doc in the feature branch
> >> (i.e.dev-support/design-docs) please include it there
> >>
> >> Of course, HBASE-23331 <
> https://issues.apache.org/jira/browse/HBASE-23331>
> >> is
> >> the subtask for this. The plan is to update the ref guide with all the
> >> details once the branch is merged in the master. I'll make sure to add
> the
> >> design doc too.
> >>
> >>> the current design doc has comments open still, should I assume those
> >> things haven't been addressed in the branch? or should I assume they
> have
> >> but it hasn't been updated yet?
> >>
> >> I addressed most of them already, forgot to resolve the comments. There
> >> were some new comments since this email, so I addressed them and
> cleaned up
> >> the doc. Thanks for pointing it out.
> >>
> >>> On Tue, Feb 4, 2020 at 10:15 PM Stack <st...@duboce.net> wrote:
> >>>
> >>> I'm +1 for merge to master with it default enabled and to branch-2 with
> >> it
> >>> off by default.
> >>>
> >>> Nice work Bharath.
> >>>
> >>> S
> >>>
> >>> On Tue, Feb 4, 2020 at 8:37 AM Bharath Vissapragada <
> bharathv@apache.org
> >>>
> >>> wrote:
> >>>
> >>>> Hello everyone,
> >>>>
> >>>> I'd like to kickoff a discuss thread on dev@ to see what folks think
> >>> about
> >>>> merging the feature branch for HBASE-18095
> >>>> <https://issues.apache.org/jira/browse/HBASE-18095> into the master.
> >> For
> >>>> those of you who aren't following this work, over the last few months,
> >> a
> >>>> lot of effort went into a feature branch
> >>>> <
> >>>>
> >>>
> >>
> https://github.com/apache/hbase/tree/HBASE-18095/client-locate-meta-no-zookeeper
> >>>>>
> >>>> to
> >>>> remove the ZK dependency in the client.
> >>>>
> >>>> *Please refer to the design doc
> >>>> <
> >>>>
> >>>
> >>
> https://docs.google.com/document/d/1JAJdM7eUxg5b417f0xWS4NztKCx1f2b6wZrudPtiXF4/edit
> >>>>>
> >>>> attached to the parent jira and go through the subtasks for all the
> >>>> technical details and design considerations*.
> >>>>
> >>>> *TL;DR*: With this feature, the client connection implementation *does
> >>> not*
> >>>> need access to zookeeper to fetch the connection metadata. Instead, a
> >>>> predefined set of master end points in the configuration are used by
> >>>> clients to fetch the required metadata.
> >>>>
> >>>> This new feature is *enabled by default on the feature branch* and
> >> passes
> >>>> the entire nightly test suite (modulo some known flakes not specific
> to
> >>> the
> >>>> branch). At this point, I'm not aware of any performance concerns /
> >>> feature
> >>>> gaps compared to original default implementation. The original
> registry
> >>>> implementation is still retained and can be used by setting the
> >> following
> >>>> client configuration. This kill switch gives the users more
> flexibility
> >>>> since they have a fallback incase they run into any issues.
> >>>>
> >>>> <property>
> >>>> <!-- Reverts to the original ZK based connection registry
> >>>> implementation -->
> >>>> <name>hbase.client.registry.impl</name>
> >>>> <value>org.apache.hadoop.hbase.client.ZKConnectionRegistry</value>
> >>>> </property>
> >>>>
> >>>> This work is also slated to go into the upcoming releases* 2.3.0* and
> >>>> *1.6.0*. However, it will be *disabled by default*. Having this work
> >> back
> >>>> ported to those branches enables users to try it out in their
> >>> environments
> >>>> and report any feedback.
> >>>>
> >>>> Please speak up (respond to this email) if there are any objections to
> >>>> merging this work in the master branch.
> >>>>
> >>>> Many thanks to Nick Dimiduk, Andrew Purtell and Michael Stack for
> their
> >>>> invaluable feedback throughout this work.
> >>>>
> >>>> - Bharath
> >>>>
> >>>
> >>
>
Re: [DISCUSS] HBASE-18095 - Remove zookeeper dependency in client codepaths
Posted by Andrew Purtell <an...@gmail.com>.
Which registry to use is a client side configuration so old clients are unaffected. (At some future time an operator might want to foreclose on direct ZK access with network or host ACLs but of course this is totally up to them and they are in complete control.)
> On Feb 5, 2020, at 2:12 PM, Wellington Chevreuil <we...@gmail.com> wrote:
>
> Thanks for the detailed summary, Bharath. I'm +1 for master.
>
> Just additional question I have, it wasn't clear for me on the doc/summary:
> does it consider fallback to ZK based registry, in case of clients running
> old versions on clusters with this feature enabled?
>
>> Em qua., 5 de fev. de 2020 às 07:37, Bharath Vissapragada <
>> bharathv@apache.org> escreveu:
>>
>> Thanks everyone for chiming in. Sean, regarding your comments.
>>
>>> I don't see the current design doc in the feature branch
>> (i.e.dev-support/design-docs) please include it there
>>
>> Of course, HBASE-23331 <https://issues.apache.org/jira/browse/HBASE-23331>
>> is
>> the subtask for this. The plan is to update the ref guide with all the
>> details once the branch is merged in the master. I'll make sure to add the
>> design doc too.
>>
>>> the current design doc has comments open still, should I assume those
>> things haven't been addressed in the branch? or should I assume they have
>> but it hasn't been updated yet?
>>
>> I addressed most of them already, forgot to resolve the comments. There
>> were some new comments since this email, so I addressed them and cleaned up
>> the doc. Thanks for pointing it out.
>>
>>> On Tue, Feb 4, 2020 at 10:15 PM Stack <st...@duboce.net> wrote:
>>>
>>> I'm +1 for merge to master with it default enabled and to branch-2 with
>> it
>>> off by default.
>>>
>>> Nice work Bharath.
>>>
>>> S
>>>
>>> On Tue, Feb 4, 2020 at 8:37 AM Bharath Vissapragada <bharathv@apache.org
>>>
>>> wrote:
>>>
>>>> Hello everyone,
>>>>
>>>> I'd like to kickoff a discuss thread on dev@ to see what folks think
>>> about
>>>> merging the feature branch for HBASE-18095
>>>> <https://issues.apache.org/jira/browse/HBASE-18095> into the master.
>> For
>>>> those of you who aren't following this work, over the last few months,
>> a
>>>> lot of effort went into a feature branch
>>>> <
>>>>
>>>
>> https://github.com/apache/hbase/tree/HBASE-18095/client-locate-meta-no-zookeeper
>>>>>
>>>> to
>>>> remove the ZK dependency in the client.
>>>>
>>>> *Please refer to the design doc
>>>> <
>>>>
>>>
>> https://docs.google.com/document/d/1JAJdM7eUxg5b417f0xWS4NztKCx1f2b6wZrudPtiXF4/edit
>>>>>
>>>> attached to the parent jira and go through the subtasks for all the
>>>> technical details and design considerations*.
>>>>
>>>> *TL;DR*: With this feature, the client connection implementation *does
>>> not*
>>>> need access to zookeeper to fetch the connection metadata. Instead, a
>>>> predefined set of master end points in the configuration are used by
>>>> clients to fetch the required metadata.
>>>>
>>>> This new feature is *enabled by default on the feature branch* and
>> passes
>>>> the entire nightly test suite (modulo some known flakes not specific to
>>> the
>>>> branch). At this point, I'm not aware of any performance concerns /
>>> feature
>>>> gaps compared to original default implementation. The original registry
>>>> implementation is still retained and can be used by setting the
>> following
>>>> client configuration. This kill switch gives the users more flexibility
>>>> since they have a fallback incase they run into any issues.
>>>>
>>>> <property>
>>>> <!-- Reverts to the original ZK based connection registry
>>>> implementation -->
>>>> <name>hbase.client.registry.impl</name>
>>>> <value>org.apache.hadoop.hbase.client.ZKConnectionRegistry</value>
>>>> </property>
>>>>
>>>> This work is also slated to go into the upcoming releases* 2.3.0* and
>>>> *1.6.0*. However, it will be *disabled by default*. Having this work
>> back
>>>> ported to those branches enables users to try it out in their
>>> environments
>>>> and report any feedback.
>>>>
>>>> Please speak up (respond to this email) if there are any objections to
>>>> merging this work in the master branch.
>>>>
>>>> Many thanks to Nick Dimiduk, Andrew Purtell and Michael Stack for their
>>>> invaluable feedback throughout this work.
>>>>
>>>> - Bharath
>>>>
>>>
>>
Re: [DISCUSS] HBASE-18095 - Remove zookeeper dependency in client codepaths
Posted by Wellington Chevreuil <we...@gmail.com>.
Thanks for the detailed summary, Bharath. I'm +1 for master.
Just additional question I have, it wasn't clear for me on the doc/summary:
does it consider fallback to ZK based registry, in case of clients running
old versions on clusters with this feature enabled?
Em qua., 5 de fev. de 2020 às 07:37, Bharath Vissapragada <
bharathv@apache.org> escreveu:
> Thanks everyone for chiming in. Sean, regarding your comments.
>
> > I don't see the current design doc in the feature branch
> (i.e.dev-support/design-docs) please include it there
>
> Of course, HBASE-23331 <https://issues.apache.org/jira/browse/HBASE-23331>
> is
> the subtask for this. The plan is to update the ref guide with all the
> details once the branch is merged in the master. I'll make sure to add the
> design doc too.
>
> > the current design doc has comments open still, should I assume those
> things haven't been addressed in the branch? or should I assume they have
> but it hasn't been updated yet?
>
> I addressed most of them already, forgot to resolve the comments. There
> were some new comments since this email, so I addressed them and cleaned up
> the doc. Thanks for pointing it out.
>
> On Tue, Feb 4, 2020 at 10:15 PM Stack <st...@duboce.net> wrote:
>
> > I'm +1 for merge to master with it default enabled and to branch-2 with
> it
> > off by default.
> >
> > Nice work Bharath.
> >
> > S
> >
> > On Tue, Feb 4, 2020 at 8:37 AM Bharath Vissapragada <bharathv@apache.org
> >
> > wrote:
> >
> > > Hello everyone,
> > >
> > > I'd like to kickoff a discuss thread on dev@ to see what folks think
> > about
> > > merging the feature branch for HBASE-18095
> > > <https://issues.apache.org/jira/browse/HBASE-18095> into the master.
> For
> > > those of you who aren't following this work, over the last few months,
> a
> > > lot of effort went into a feature branch
> > > <
> > >
> >
> https://github.com/apache/hbase/tree/HBASE-18095/client-locate-meta-no-zookeeper
> > > >
> > > to
> > > remove the ZK dependency in the client.
> > >
> > > *Please refer to the design doc
> > > <
> > >
> >
> https://docs.google.com/document/d/1JAJdM7eUxg5b417f0xWS4NztKCx1f2b6wZrudPtiXF4/edit
> > > >
> > > attached to the parent jira and go through the subtasks for all the
> > > technical details and design considerations*.
> > >
> > > *TL;DR*: With this feature, the client connection implementation *does
> > not*
> > > need access to zookeeper to fetch the connection metadata. Instead, a
> > > predefined set of master end points in the configuration are used by
> > > clients to fetch the required metadata.
> > >
> > > This new feature is *enabled by default on the feature branch* and
> passes
> > > the entire nightly test suite (modulo some known flakes not specific to
> > the
> > > branch). At this point, I'm not aware of any performance concerns /
> > feature
> > > gaps compared to original default implementation. The original registry
> > > implementation is still retained and can be used by setting the
> following
> > > client configuration. This kill switch gives the users more flexibility
> > > since they have a fallback incase they run into any issues.
> > >
> > > <property>
> > > <!-- Reverts to the original ZK based connection registry
> > > implementation -->
> > > <name>hbase.client.registry.impl</name>
> > > <value>org.apache.hadoop.hbase.client.ZKConnectionRegistry</value>
> > > </property>
> > >
> > > This work is also slated to go into the upcoming releases* 2.3.0* and
> > > *1.6.0*. However, it will be *disabled by default*. Having this work
> back
> > > ported to those branches enables users to try it out in their
> > environments
> > > and report any feedback.
> > >
> > > Please speak up (respond to this email) if there are any objections to
> > > merging this work in the master branch.
> > >
> > > Many thanks to Nick Dimiduk, Andrew Purtell and Michael Stack for their
> > > invaluable feedback throughout this work.
> > >
> > > - Bharath
> > >
> >
>
Re: [DISCUSS] HBASE-18095 - Remove zookeeper dependency in client codepaths
Posted by Bharath Vissapragada <bh...@apache.org>.
Thanks everyone for chiming in. Sean, regarding your comments.
> I don't see the current design doc in the feature branch
(i.e.dev-support/design-docs) please include it there
Of course, HBASE-23331 <https://issues.apache.org/jira/browse/HBASE-23331> is
the subtask for this. The plan is to update the ref guide with all the
details once the branch is merged in the master. I'll make sure to add the
design doc too.
> the current design doc has comments open still, should I assume those
things haven't been addressed in the branch? or should I assume they have
but it hasn't been updated yet?
I addressed most of them already, forgot to resolve the comments. There
were some new comments since this email, so I addressed them and cleaned up
the doc. Thanks for pointing it out.
On Tue, Feb 4, 2020 at 10:15 PM Stack <st...@duboce.net> wrote:
> I'm +1 for merge to master with it default enabled and to branch-2 with it
> off by default.
>
> Nice work Bharath.
>
> S
>
> On Tue, Feb 4, 2020 at 8:37 AM Bharath Vissapragada <bh...@apache.org>
> wrote:
>
> > Hello everyone,
> >
> > I'd like to kickoff a discuss thread on dev@ to see what folks think
> about
> > merging the feature branch for HBASE-18095
> > <https://issues.apache.org/jira/browse/HBASE-18095> into the master. For
> > those of you who aren't following this work, over the last few months, a
> > lot of effort went into a feature branch
> > <
> >
> https://github.com/apache/hbase/tree/HBASE-18095/client-locate-meta-no-zookeeper
> > >
> > to
> > remove the ZK dependency in the client.
> >
> > *Please refer to the design doc
> > <
> >
> https://docs.google.com/document/d/1JAJdM7eUxg5b417f0xWS4NztKCx1f2b6wZrudPtiXF4/edit
> > >
> > attached to the parent jira and go through the subtasks for all the
> > technical details and design considerations*.
> >
> > *TL;DR*: With this feature, the client connection implementation *does
> not*
> > need access to zookeeper to fetch the connection metadata. Instead, a
> > predefined set of master end points in the configuration are used by
> > clients to fetch the required metadata.
> >
> > This new feature is *enabled by default on the feature branch* and passes
> > the entire nightly test suite (modulo some known flakes not specific to
> the
> > branch). At this point, I'm not aware of any performance concerns /
> feature
> > gaps compared to original default implementation. The original registry
> > implementation is still retained and can be used by setting the following
> > client configuration. This kill switch gives the users more flexibility
> > since they have a fallback incase they run into any issues.
> >
> > <property>
> > <!-- Reverts to the original ZK based connection registry
> > implementation -->
> > <name>hbase.client.registry.impl</name>
> > <value>org.apache.hadoop.hbase.client.ZKConnectionRegistry</value>
> > </property>
> >
> > This work is also slated to go into the upcoming releases* 2.3.0* and
> > *1.6.0*. However, it will be *disabled by default*. Having this work back
> > ported to those branches enables users to try it out in their
> environments
> > and report any feedback.
> >
> > Please speak up (respond to this email) if there are any objections to
> > merging this work in the master branch.
> >
> > Many thanks to Nick Dimiduk, Andrew Purtell and Michael Stack for their
> > invaluable feedback throughout this work.
> >
> > - Bharath
> >
>
Re: [DISCUSS] HBASE-18095 - Remove zookeeper dependency in client codepaths
Posted by Stack <st...@duboce.net>.
I'm +1 for merge to master with it default enabled and to branch-2 with it
off by default.
Nice work Bharath.
S
On Tue, Feb 4, 2020 at 8:37 AM Bharath Vissapragada <bh...@apache.org>
wrote:
> Hello everyone,
>
> I'd like to kickoff a discuss thread on dev@ to see what folks think about
> merging the feature branch for HBASE-18095
> <https://issues.apache.org/jira/browse/HBASE-18095> into the master. For
> those of you who aren't following this work, over the last few months, a
> lot of effort went into a feature branch
> <
> https://github.com/apache/hbase/tree/HBASE-18095/client-locate-meta-no-zookeeper
> >
> to
> remove the ZK dependency in the client.
>
> *Please refer to the design doc
> <
> https://docs.google.com/document/d/1JAJdM7eUxg5b417f0xWS4NztKCx1f2b6wZrudPtiXF4/edit
> >
> attached to the parent jira and go through the subtasks for all the
> technical details and design considerations*.
>
> *TL;DR*: With this feature, the client connection implementation *does not*
> need access to zookeeper to fetch the connection metadata. Instead, a
> predefined set of master end points in the configuration are used by
> clients to fetch the required metadata.
>
> This new feature is *enabled by default on the feature branch* and passes
> the entire nightly test suite (modulo some known flakes not specific to the
> branch). At this point, I'm not aware of any performance concerns / feature
> gaps compared to original default implementation. The original registry
> implementation is still retained and can be used by setting the following
> client configuration. This kill switch gives the users more flexibility
> since they have a fallback incase they run into any issues.
>
> <property>
> <!-- Reverts to the original ZK based connection registry
> implementation -->
> <name>hbase.client.registry.impl</name>
> <value>org.apache.hadoop.hbase.client.ZKConnectionRegistry</value>
> </property>
>
> This work is also slated to go into the upcoming releases* 2.3.0* and
> *1.6.0*. However, it will be *disabled by default*. Having this work back
> ported to those branches enables users to try it out in their environments
> and report any feedback.
>
> Please speak up (respond to this email) if there are any objections to
> merging this work in the master branch.
>
> Many thanks to Nick Dimiduk, Andrew Purtell and Michael Stack for their
> invaluable feedback throughout this work.
>
> - Bharath
>