You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Andrew Purtell <ap...@apache.org> on 2014/01/17 23:53:15 UTC
Fwd: Where are we in ZOOKEEPER-1416
What is going on with this thread over on dev@zookeeper? Bringing it to the
attention of people over here.
---------- Forwarded message ----------
From: Ted Dunning <te...@gmail.com>
Date: Fri, Jan 17, 2014 at 2:41 PM
Subject: Re: Where are we in ZOOKEEPER-1416
To: "dev@zookeeper.apache.org" <de...@zookeeper.apache.org>
My reference here is to the comments a ways up thread. Kishore and I
clearly agree completely that idempotency and dealing with the state as it
is right now are the keys to correct design.
On Fri, Jan 17, 2014 at 2:14 PM, Ted Dunning <te...@gmail.com> wrote:
>
> That comment indicates a lack of understanding of ZK, not a bug in ZK.
>
> You don't lose state transitions if you read new state at the same time
> you set the new watch.
>
> Likewise, it is simply a product of bad design to have a problem with
> asynchronous notification. Changes on other machines *are* asynchronous
so
> anybody who can't handle that is inherently denying reality. If you want
> to inject the notifications into a sequential view of an event stream,
that
> is trivial to do.
>
> Systems that depend on transition notification are generally not as robust
> as systems that depend on current state. Building a cluster manager works
> better if the master is notified that a change has happened, but then
> simply deals with the situation as it stands.
>
> As an analog, imagine that you have a system that shows a number x and a
> second system that is supposed to show an echo of that number.
>
> Design A is notified of changes to x in the form of deltas. If there is
> ever an error in handling events, the echo will be off forever. The error
> that causes the delta to be dropped could be notification or a coding
error
> or a misunderstanding of how parallel systems work. For instance, the
> InterruptedException might not be handled right.
>
> Design B is notified of changes to x and whenever a change happens, the
> second system simply goes and reads the new state. Errors will be quickly
> corrected.
>
> It sounds like the original poster is trying to build something like
> Design A when they should be building Design B.
>
>
>
>
>
> On Fri, Jan 17, 2014 at 12:34 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> HBASE-5487 is also related.
>>
>> The discussion there is very long. Below is an excerpt from Honghua:
>>
>> too many tricky scenarios/bugs due to ZK watch is one-time(which can
>> result
>> in missed state transition) and the notification/process is
>> asyncronous(which can lead to delayed/non-update-to-date state in master
>> memory).
>>
>> Cheers
>>
>>
>> On Fri, Jan 17, 2014 at 11:25 AM, Ted Yu <yu...@gmail.com> wrote:
>>
>> > Hi, Flavio:
>> > HBASE-8365 is one such case.
>> >
>> > Let me search around for other related discussion.
>> >
>> >
>> > On Fri, Jan 17, 2014 at 11:17 AM, Flavio Junqueira <
>> fpjunqueira@yahoo.com>wrote:
>> >
>> >> Hi Ted,
>> >>
>> >> Can you provide more detail on how the precise deltas could make it
>> more
>> >> robust?
>> >>
>> >> -Flavio
>> >>
>> >> -----Original Message-----
>> >> From: "Ted Yu" <yu...@gmail.com>
>> >> Sent: 17/01/2014 17:25
>> >> To: "dev@zookeeper.apache.org" <de...@zookeeper.apache.org>
>> >> Subject: Re: Where are we in ZOOKEEPER-1416
>> >>
>> >> Having the ability to know exact deltas would help make HBase region
>> >> assignment more robust.
>> >>
>> >> Cheers
>> >>
>> >>
>> >>
>> >> On Fri, Jan 17, 2014 at 9:13 AM, kishore g <g....@gmail.com>
>> wrote:
>> >>
>> >> > I agree with you, I like the side effect and in fact I would prefer
>> to
>> >> have
>> >> > one notification for all changes under a parent node.
>> >> >
>> >> > However, Hao is probably asking for ability to know exact deltas.
>> >> >
>> >> >
>> >> > On Fri, Jan 17, 2014 at 8:15 AM, FPJ <fp...@yahoo.com> wrote:
>> >> >
>> >> > > We don't need to have a mapping between every change and a
>> >> notification.
>> >> > If
>> >> > > there are 2+ changes between notifications, you'll be able to
>> observe
>> >> it
>> >> > by
>> >> > > reading the ZK state. In fact, one nice side-effect is that we
>> reduce
>> >> the
>> >> > > number of notifications when there are many concurrent changes.
>> >> > >
>> >> > > The only situation I can see it being necessary is the one in
>> which we
>> >> > need
>> >> > > to know precisely the changes and we haven't cached a previous
>> >> version of
>> >> > > the state.
>> >> > >
>> >> > > -Flavio
>> >> > >
>> >> > > > -----Original Message-----
>> >> > > > From: kishore g [mailto:g.kishore@gmail.com]
>> >> > > > Sent: 17 January 2014 16:06
>> >> > > > To: dev@zookeeper.apache.org
>> >> > > > Subject: Re: Where are we in ZOOKEEPER-1416
>> >> > > >
>> >> > > > I think Hao is pointing out that there is no way to see every
>> change
>> >> > > > (delta) that happened to a znode. Consider 2 changes A,B in
quick
>> >> > > > succession. When client gets notified of A and before setting
the
>> >> watch
>> >> > > the
>> >> > > > change B has occurred on the server side. This means the client
>> >> cannot
>> >> > > know
>> >> > > > the delta A. Client can only read the state after change B is
>> >> applied.
>> >> > > >
>> >> > > > Implementing the concept of Persistent watcher guarantees that
>> >> client
>> >> > if
>> >> > > > notified after every change.
>> >> > > >
>> >> > > > This is a nice to have feature but I dont understand the
>> >> requirement in
>> >> > > Hbase
>> >> > > > where this is needed. Hao, can you shed more light on how this
>> >> would be
>> >> > > > useful for HBase (to act like state machine)
>> >> > > >
>> >> > > > thanks,
>> >> > > > Kishore G
>> >> > > >
>> >> > > >
>> >> > > >
>> >> > > >
>> >> > > >
>> >> > > >
>> >> > > >
>> >> > > >
>> >> > > > On Fri, Jan 17, 2014 at 5:18 AM, FPJ <fp...@yahoo.com>
>> wrote:
>> >> > > >
>> >> > > > > But you don't really miss events, you'll see them when you
read
>> >> the
>> >> > ZK
>> >> > > > > state. If you follow the pattern I described, you're supposed
>> to
>> >> > > > > observe all changes. Perhaps I'm missing some concrete use
case
>> >> you
>> >> > > > > have mind.
>> >> > > > >
>> >> > > > > -Flavio
>> >> > > > >
>> >> > > > > > -----Original Message-----
>> >> > > > > > From: 陈迪豪 [mailto:chendihao@xiaomi.com]
>> >> > > > > > Sent: 17 January 2014 13:03
>> >> > > > > > To: dev@zookeeper.apache.org
>> >> > > > > > Subject: RE: Where are we in ZOOKEEPER-1416
>> >> > > > > >
>> >> > > > > > No, it's not complicated. But for the people who don't
>> >> understand
>> >> > zk
>> >> > > > > deeply,
>> >> > > > > > they would easily ignore the fact that they would miss
>> events in
>> >> > > > > > some
>> >> > > > > way.
>> >> > > > > > Moreover, I think providing persistent watch is good for
>> >> developers
>> >> > > > > > to
>> >> > > > > build
>> >> > > > > > the "state-machine" application. Actually, HBase suffer from
>> >> > missing
>> >> > > > > > the intermediate state when use zk to store the data.
>> >> > > > > >
>> >> > > > > > If the feature is implemented, I would like to see the patch
>> and
>> >> > > > > > consider
>> >> > > > > if it
>> >> > > > > > can be used for us.
>> >> > > > > >
>> >> > > > > > ________________________________________
>> >> > > > > > From: Flavio Junqueira [fpjunqueira@yahoo.com]
>> >> > > > > > Sent: Friday, January 17, 2014 8:12 PM
>> >> > > > > > To: dev@zookeeper.apache.org
>> >> > > > > > Subject: RE: Where are we in ZOOKEEPER-1416
>> >> > > > > >
>> >> > > > > > My take is that persistent subscriptions add complexity and
>> are
>> >> not
>> >> > > > > strictly
>> >> > > > > > necessary. You can follow this pattern of setting a watch,
>> >> reading
>> >> > > > > > the
>> >> > > > > state
>> >> > > > > > upon a notification and setting a new watch. Why do you feel
>> >> that's
>> >> > > > > > complicated?
>> >> > > > > >
>> >> > > > > > -Flavio
>> >> > > > > >
>> >> > > > > > -----Original Message-----
>> >> > > > > > From: 陈迪豪 [mailto:chendihao@xiaomi.com]
>> >> > > > > > Sent: Friday, January 17, 2014 3:13 AM
>> >> > > > > > To: dev@zookeeper.apache.org
>> >> > > > > > Subject: Where are we in ZOOKEEPER-1416
>> >> > > > > >
>> >> > > > > >
>> >> > > > > >
>> >> > > > > > Persistent watch and implementing the feature to act like
>> "state
>> >> > > > machine"
>> >> > > > > > which is mentioned in
>> >> > > > > > ZOOKEEPER-153<
>> https://issues.apache.org/jira/browse/ZOOKEEPER-
>> >> > > > 153>,
>> >> > > > > > would be great for ZooKeeper user. I think HBase would like
>> to
>> >> know
>> >> > > > > > all
>> >> > > > > the
>> >> > > > > > change in zk rather than missing kind of events.
>> >> > > > > >
>> >> > > > > > So, would we continue developing these features? It's also a
>> >> little
>> >> > > > > > complicated to develop with zk and I think there're lots of
>> >> things
>> >> > > > > > to
>> >> > > > > improve.
>> >> > > > >
>> >> > > > >
>> >> > > > >
>> >> > >
>> >> > >
>> >> >
>> >>
>> >
>> >
>>
>
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: Where are we in ZOOKEEPER-1416
Posted by Ted Yu <yu...@gmail.com>.
Pardon me, Honghua.
I should have given proper context.
Will pay attention in the future.
Cheers
On Jan 18, 2014, at 4:05 PM, Ted Dunning <te...@gmail.com> wrote:
> On Sat, Jan 18, 2014 at 8:58 AM, Stack <st...@duboce.net> wrote:
>
>>> On Fri, Jan 17, 2014 at 2:14 PM, Ted Dunning <te...@gmail.com>
>>> wrote:
>>>
>>>>
>>>> That comment indicates a lack of understanding of ZK, not a bug in ZK.
>>
>>
>> Mighty Ted Dunning, I'd just like to say that the quote is badly out of
>> context and misrepresents the clueful gentleman quoted (The comment
>> actually leads off with the root problem being HBase's USE of ZK, NOT ZK).
>
> Fair correction. I don't think I was the one who removed the context, but
> I should have been more sensitive to its lack.
>
> Corrections always welcome. Corrections from Stack pretty much always
> accepted without question.
>
>
>>>>>> Having the ability to know exact deltas would help make HBase region
>>>>>> assignment more robust.
>>
>> Nor am I sure what this is about. Those actually working on making HBase
>> region assignment more 'robust' have not asked for the above; i.e. have ZK
>> make fundamental changes in the way it operates, though it is working as
>> advertised, just to 'fix' a downstream project's misuse.
>
> See previous general principle. This is much more compatible with what I
> remember of the strategies being used in the HBase Master.
Re: Where are we in ZOOKEEPER-1416
Posted by Ted Dunning <te...@gmail.com>.
On Sat, Jan 18, 2014 at 8:58 AM, Stack <st...@duboce.net> wrote:
> > On Fri, Jan 17, 2014 at 2:14 PM, Ted Dunning <te...@gmail.com>
> > wrote:
> >
> > >
> > > That comment indicates a lack of understanding of ZK, not a bug in ZK.
> > >
>
>
> Mighty Ted Dunning, I'd just like to say that the quote is badly out of
> context and misrepresents the clueful gentleman quoted (The comment
> actually leads off with the root problem being HBase's USE of ZK, NOT ZK).
>
Fair correction. I don't think I was the one who removed the context, but
I should have been more sensitive to its lack.
Corrections always welcome. Corrections from Stack pretty much always
accepted without question.
> >> >> Having the ability to know exact deltas would help make HBase region
> >> >> assignment more robust.
>
> Nor am I sure what this is about. Those actually working on making HBase
> region assignment more 'robust' have not asked for the above; i.e. have ZK
> make fundamental changes in the way it operates, though it is working as
> advertised, just to 'fix' a downstream project's misuse.
>
See previous general principle. This is much more compatible with what I
remember of the strategies being used in the HBase Master.
Re: Where are we in ZOOKEEPER-1416
Posted by Stack <st...@duboce.net>.
>
>
> On Fri, Jan 17, 2014 at 2:14 PM, Ted Dunning <te...@gmail.com>
> wrote:
>
> >
> > That comment indicates a lack of understanding of ZK, not a bug in ZK.
> >
Mighty Ted Dunning, I'd just like to say that the quote is badly out of
context and misrepresents the clueful gentleman quoted (The comment
actually leads off with the root problem being HBase's USE of ZK, NOT ZK).
>> >> Having the ability to know exact deltas would help make HBase region
>> >> assignment more robust.
Nor am I sure what this is about. Those actually working on making HBase
region assignment more 'robust' have not asked for the above; i.e. have ZK
make fundamental changes in the way it operates, though it is working as
advertised, just to 'fix' a downstream project's misuse.
Go 49ers!
St.Ack
Re: Where are we in ZOOKEEPER-1416
Posted by Stack <st...@duboce.net>.
On Fri, Jan 17, 2014 at 2:53 PM, Andrew Purtell <ap...@apache.org> wrote:
> What is going on with this thread over on dev@zookeeper? Bringing it to
> the
> attention of people over here.
>
>
>
Ted?
St.Ack
> ---------- Forwarded message ----------
> From: Ted Dunning <te...@gmail.com>
> Date: Fri, Jan 17, 2014 at 2:41 PM
> Subject: Re: Where are we in ZOOKEEPER-1416
> To: "dev@zookeeper.apache.org" <de...@zookeeper.apache.org>
>
>
> My reference here is to the comments a ways up thread. Kishore and I
> clearly agree completely that idempotency and dealing with the state as it
> is right now are the keys to correct design.
>
>
> On Fri, Jan 17, 2014 at 2:14 PM, Ted Dunning <te...@gmail.com>
> wrote:
>
> >
> > That comment indicates a lack of understanding of ZK, not a bug in ZK.
> >
> > You don't lose state transitions if you read new state at the same time
> > you set the new watch.
> >
> > Likewise, it is simply a product of bad design to have a problem with
> > asynchronous notification. Changes on other machines *are* asynchronous
> so
> > anybody who can't handle that is inherently denying reality. If you want
> > to inject the notifications into a sequential view of an event stream,
> that
> > is trivial to do.
> >
> > Systems that depend on transition notification are generally not as
> robust
> > as systems that depend on current state. Building a cluster manager
> works
> > better if the master is notified that a change has happened, but then
> > simply deals with the situation as it stands.
> >
> > As an analog, imagine that you have a system that shows a number x and a
> > second system that is supposed to show an echo of that number.
> >
> > Design A is notified of changes to x in the form of deltas. If there is
> > ever an error in handling events, the echo will be off forever. The
> error
> > that causes the delta to be dropped could be notification or a coding
> error
> > or a misunderstanding of how parallel systems work. For instance, the
> > InterruptedException might not be handled right.
> >
> > Design B is notified of changes to x and whenever a change happens, the
> > second system simply goes and reads the new state. Errors will be
> quickly
> > corrected.
> >
> > It sounds like the original poster is trying to build something like
> > Design A when they should be building Design B.
> >
> >
> >
> >
> >
> > On Fri, Jan 17, 2014 at 12:34 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> >> HBASE-5487 is also related.
> >>
> >> The discussion there is very long. Below is an excerpt from Honghua:
> >>
> >> too many tricky scenarios/bugs due to ZK watch is one-time(which can
> >> result
> >> in missed state transition) and the notification/process is
> >> asyncronous(which can lead to delayed/non-update-to-date state in master
> >> memory).
> >>
> >> Cheers
> >>
> >>
> >> On Fri, Jan 17, 2014 at 11:25 AM, Ted Yu <yu...@gmail.com> wrote:
> >>
> >> > Hi, Flavio:
> >> > HBASE-8365 is one such case.
> >> >
> >> > Let me search around for other related discussion.
> >> >
> >> >
> >> > On Fri, Jan 17, 2014 at 11:17 AM, Flavio Junqueira <
> >> fpjunqueira@yahoo.com>wrote:
> >> >
> >> >> Hi Ted,
> >> >>
> >> >> Can you provide more detail on how the precise deltas could make it
> >> more
> >> >> robust?
> >> >>
> >> >> -Flavio
> >> >>
> >> >> -----Original Message-----
> >> >> From: "Ted Yu" <yu...@gmail.com>
> >> >> Sent: 17/01/2014 17:25
> >> >> To: "dev@zookeeper.apache.org" <de...@zookeeper.apache.org>
> >> >> Subject: Re: Where are we in ZOOKEEPER-1416
> >> >>
> >> >> Having the ability to know exact deltas would help make HBase region
> >> >> assignment more robust.
> >> >>
> >> >> Cheers
> >> >>
> >> >>
> >> >>
> >> >> On Fri, Jan 17, 2014 at 9:13 AM, kishore g <g....@gmail.com>
> >> wrote:
> >> >>
> >> >> > I agree with you, I like the side effect and in fact I would prefer
> >> to
> >> >> have
> >> >> > one notification for all changes under a parent node.
> >> >> >
> >> >> > However, Hao is probably asking for ability to know exact deltas.
> >> >> >
> >> >> >
> >> >> > On Fri, Jan 17, 2014 at 8:15 AM, FPJ <fp...@yahoo.com>
> wrote:
> >> >> >
> >> >> > > We don't need to have a mapping between every change and a
> >> >> notification.
> >> >> > If
> >> >> > > there are 2+ changes between notifications, you'll be able to
> >> observe
> >> >> it
> >> >> > by
> >> >> > > reading the ZK state. In fact, one nice side-effect is that we
> >> reduce
> >> >> the
> >> >> > > number of notifications when there are many concurrent changes.
> >> >> > >
> >> >> > > The only situation I can see it being necessary is the one in
> >> which we
> >> >> > need
> >> >> > > to know precisely the changes and we haven't cached a previous
> >> >> version of
> >> >> > > the state.
> >> >> > >
> >> >> > > -Flavio
> >> >> > >
> >> >> > > > -----Original Message-----
> >> >> > > > From: kishore g [mailto:g.kishore@gmail.com]
> >> >> > > > Sent: 17 January 2014 16:06
> >> >> > > > To: dev@zookeeper.apache.org
> >> >> > > > Subject: Re: Where are we in ZOOKEEPER-1416
> >> >> > > >
> >> >> > > > I think Hao is pointing out that there is no way to see every
> >> change
> >> >> > > > (delta) that happened to a znode. Consider 2 changes A,B in
> quick
> >> >> > > > succession. When client gets notified of A and before setting
> the
> >> >> watch
> >> >> > > the
> >> >> > > > change B has occurred on the server side. This means the client
> >> >> cannot
> >> >> > > know
> >> >> > > > the delta A. Client can only read the state after change B is
> >> >> applied.
> >> >> > > >
> >> >> > > > Implementing the concept of Persistent watcher guarantees that
> >> >> client
> >> >> > if
> >> >> > > > notified after every change.
> >> >> > > >
> >> >> > > > This is a nice to have feature but I dont understand the
> >> >> requirement in
> >> >> > > Hbase
> >> >> > > > where this is needed. Hao, can you shed more light on how this
> >> >> would be
> >> >> > > > useful for HBase (to act like state machine)
> >> >> > > >
> >> >> > > > thanks,
> >> >> > > > Kishore G
> >> >> > > >
> >> >> > > >
> >> >> > > >
> >> >> > > >
> >> >> > > >
> >> >> > > >
> >> >> > > >
> >> >> > > >
> >> >> > > > On Fri, Jan 17, 2014 at 5:18 AM, FPJ <fp...@yahoo.com>
> >> wrote:
> >> >> > > >
> >> >> > > > > But you don't really miss events, you'll see them when you
> read
> >> >> the
> >> >> > ZK
> >> >> > > > > state. If you follow the pattern I described, you're supposed
> >> to
> >> >> > > > > observe all changes. Perhaps I'm missing some concrete use
> case
> >> >> you
> >> >> > > > > have mind.
> >> >> > > > >
> >> >> > > > > -Flavio
> >> >> > > > >
> >> >> > > > > > -----Original Message-----
> >> >> > > > > > From: 陈迪豪 [mailto:chendihao@xiaomi.com]
> >> >> > > > > > Sent: 17 January 2014 13:03
> >> >> > > > > > To: dev@zookeeper.apache.org
> >> >> > > > > > Subject: RE: Where are we in ZOOKEEPER-1416
> >> >> > > > > >
> >> >> > > > > > No, it's not complicated. But for the people who don't
> >> >> understand
> >> >> > zk
> >> >> > > > > deeply,
> >> >> > > > > > they would easily ignore the fact that they would miss
> >> events in
> >> >> > > > > > some
> >> >> > > > > way.
> >> >> > > > > > Moreover, I think providing persistent watch is good for
> >> >> developers
> >> >> > > > > > to
> >> >> > > > > build
> >> >> > > > > > the "state-machine" application. Actually, HBase suffer
> from
> >> >> > missing
> >> >> > > > > > the intermediate state when use zk to store the data.
> >> >> > > > > >
> >> >> > > > > > If the feature is implemented, I would like to see the
> patch
> >> and
> >> >> > > > > > consider
> >> >> > > > > if it
> >> >> > > > > > can be used for us.
> >> >> > > > > >
> >> >> > > > > > ________________________________________
> >> >> > > > > > From: Flavio Junqueira [fpjunqueira@yahoo.com]
> >> >> > > > > > Sent: Friday, January 17, 2014 8:12 PM
> >> >> > > > > > To: dev@zookeeper.apache.org
> >> >> > > > > > Subject: RE: Where are we in ZOOKEEPER-1416
> >> >> > > > > >
> >> >> > > > > > My take is that persistent subscriptions add complexity and
> >> are
> >> >> not
> >> >> > > > > strictly
> >> >> > > > > > necessary. You can follow this pattern of setting a watch,
> >> >> reading
> >> >> > > > > > the
> >> >> > > > > state
> >> >> > > > > > upon a notification and setting a new watch. Why do you
> feel
> >> >> that's
> >> >> > > > > > complicated?
> >> >> > > > > >
> >> >> > > > > > -Flavio
> >> >> > > > > >
> >> >> > > > > > -----Original Message-----
> >> >> > > > > > From: 陈迪豪 [mailto:chendihao@xiaomi.com]
> >> >> > > > > > Sent: Friday, January 17, 2014 3:13 AM
> >> >> > > > > > To: dev@zookeeper.apache.org
> >> >> > > > > > Subject: Where are we in ZOOKEEPER-1416
> >> >> > > > > >
> >> >> > > > > >
> >> >> > > > > >
> >> >> > > > > > Persistent watch and implementing the feature to act like
> >> "state
> >> >> > > > machine"
> >> >> > > > > > which is mentioned in
> >> >> > > > > > ZOOKEEPER-153<
> >> https://issues.apache.org/jira/browse/ZOOKEEPER-
> >> >> > > > 153>,
> >> >> > > > > > would be great for ZooKeeper user. I think HBase would like
> >> to
> >> >> know
> >> >> > > > > > all
> >> >> > > > > the
> >> >> > > > > > change in zk rather than missing kind of events.
> >> >> > > > > >
> >> >> > > > > > So, would we continue developing these features? It's also
> a
> >> >> little
> >> >> > > > > > complicated to develop with zk and I think there're lots of
> >> >> things
> >> >> > > > > > to
> >> >> > > > > improve.
> >> >> > > > >
> >> >> > > > >
> >> >> > > > >
> >> >> > >
> >> >> > >
> >> >> >
> >> >>
> >> >
> >> >
> >>
> >
> >
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>