You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by Michael Han <ha...@apache.org> on 2019/08/02 18:03:35 UTC
KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum
Folks,
Some of you might already see this. Comments?
https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum
What caught my eyes are:
*Worse still, although ZooKeeper is the store of record, the state in
ZooKeeper often doesn't match the state that is held in memory in the
controller. For example, when a partition leader changes its ISR in ZK,
the controller will typically not learn about these changes for many
seconds. There is no generic way for the controller to follow the
ZooKeeper event log. Although the controller can set one-shot watches, the
number of watches is limited for performance reasons. When a watch
triggers, it doesn't tell the controller the current state-- only that the
state has changed. By the time the controller re-reads the znode and sets
up a new watch, the state may have changed from what it was when the watch
originally fired. If there is no watch set, the controller may not learn
about the change at all. In some cases, restarting the controller is the
only way to resolve the discrepancy.*
I've seen some similar zookeeper use cases that ended up like what's
described here. How can ZooKeeper solve this? It seems to me that the only
solution is to provide linearizable read on watched operations. Thoughts?
Michael.
Re: KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum
Posted by Norbert Kalmar <nk...@apache.org>.
Very useful thread on how watchers work in ZooKeeper and why they work that
way!
I talked with a Kafka dev about this. He said there are only talks for now
about removing ZooKeeper, not going to happen soon, but some people
definitely want it done. They even want their own Leader election, so not
just the message/subscribe model. I think this is very dangerous. I like
Ted's saying that ZooKeeper is not just battle tested, but battle scarred.
Anyway, my takeaway has been said before:
Kafka is using watchers as if it were meant to be a messaging service,
which it is not. (The already linked ZOOKEEPER-153 is what they want, as
Patrick stated).
And anyone interested on watchers in ZooKeeper - read this thread! :)
Cheers!
Norbert
On Sat, Aug 10, 2019 at 3:48 AM Ted Dunning <te...@gmail.com> wrote:
> On Wed, Aug 7, 2019 at 5:55 PM Michael Han <ha...@apache.org> wrote:
>
> > Related discussions:
> > https://lists.apache.org/list.html?dev@kafka.apache.org:lte=1M:KIP-500
> >
> > >> Why would this lead to any discrepancy? It seems to me that the
> > controller, will read an even newer state in such a scenario.
> >
> > I think in this case the Kafka side of expectation is not just be able to
> > get latest state, but also not miss any of the state changes.
> >
>
>
> The core of the problem is that if you need to see all the changes, then it
> is messages, not state.
>
Re: KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum
Posted by Ted Dunning <te...@gmail.com>.
On Wed, Aug 7, 2019 at 5:55 PM Michael Han <ha...@apache.org> wrote:
> Related discussions:
> https://lists.apache.org/list.html?dev@kafka.apache.org:lte=1M:KIP-500
>
> >> Why would this lead to any discrepancy? It seems to me that the
> controller, will read an even newer state in such a scenario.
>
> I think in this case the Kafka side of expectation is not just be able to
> get latest state, but also not miss any of the state changes.
>
The core of the problem is that if you need to see all the changes, then it
is messages, not state.
Re: KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum
Posted by Michael Han <ha...@apache.org>.
Related discussions:
https://lists.apache.org/list.html?dev@kafka.apache.org:lte=1M:KIP-500
>> Why would this lead to any discrepancy? It seems to me that the
controller, will read an even newer state in such a scenario.
I think in this case the Kafka side of expectation is not just be able to
get latest state, but also not miss any of the state changes. See the Kafka
thread ^ which said: *"Treating metadata as a log avoids a lot of the
complex failure corner cases we have seen where a broker misses a single
update sent from the controller, but gets subsequent updates."*
We don't support this in ZK today - clients will always be able to get
latest state, but client might miss watcher events between the first
watcher fired and the reset of the watcher on client side (and any changes
happening in this period will not get notified). The subscribe API can
solve this.
One thing that the proposal only briefly mentioned but I think is a key
differentiator is scalability: managing metadata as event logs will provide
better scalability, as ZK is inherently limited by memory. One can shard a
cluster to work around the limits, but that creates other problems
(consistency, operation, etc). Having an on disk storage backend for ZK's
data tree might solve this.
On Wed, Aug 7, 2019 at 12:25 PM Ted Dunning <te...@gmail.com> wrote:
> On Wed, Aug 7, 2019 at 11:31 AM Karolos Antoniadis <ka...@gmail.com>
> wrote:
>
> > In the paragraph that Michael mentioned, among others it is written: "For
> > example, when a partition leader changes its ISR in ZK, the controller
> will
> > typically not learn about these changes for many seconds." Why would it
> > take "many seconds"?
>
>
> I think that this is conflating the situation with partition from or of the
> ZK cluster with simple hand-offs.
>
> In the case of a partition leader crashing, it will be several seconds
> before the rest of the world hears about the event.
>
>
> > Sending a watch event to the controller should be
> > pretty fast.
> >
>
> Absolutely. If the ZK cluster has its act together. And if the cause of the
> watch is detected quickly. And if you don't have a watch storm happening
> due to huge numbers of listeners.
>
> But none of those problems are really helped by moving the consensus
> algorithms into a library.
>
> Also, in the same paragraph, Colin states "By the time the controller
> > re-reads the znode and sets up a new watch, the state may have changed
> from
> > what it was when the watch originally fired. [...] only way to resolve
> the
> > discrepancy." Why would this lead to any discrepancy? It seems to me that
> > the controller, will read an even newer state in such a scenario.
> >
>
> You are correct and this has always been one of the selling points of ZK.
> The way that you can reset the watch as part of the read operation means
> that you can guarantee never to lose anything and if you are slow to
> respond, you always get data that is as up-to-date as possible. Load
> shedding tricks like that are really helpful. Getting notifications of
> every change is actually disastrous in many cases, partly because of the
> number of notifications and partly because the notifications can become
> very heavy-weight with the data they have to carry.
>
> Putting this into a library doesn't help at all, of course.
>
>
>
> >
> > Also, another argument mentioned in original KIP-500 proposal had to do
> > with speeding up the failover of a controller: "Because the controllers
> > will now all track the latest state, controller failover will not
> require a
> > lengthy reloading period where we transfer all the state to the new
> > controller." But this does not seem to be a problem with ZK per se and
> > could be solved by keeping a broker as a standby controller (briefly
> > mentioned here https://www.slideshare.net/ConfluentInc/a-deep-dive-into-
> > kafka-controller
> > <
> https://www.slideshare.net/ConfluentInc/a-deep-dive-into-kafka-controller>
> > as future work.)
> >
>
> Also, the state still has to move. Using an in-process library doesn't
> change that at all. It could move via ZK or it could move as part of quorum
> decisions or via some sort of follow-the-leader protocol. But it has to
> move. Whoever is leader has to write it out to the network and whoever is
> follower has to read it in. Whether the data is written/read directly or
> via ZK isn't really a big deal.
>
>
> >
>
Re: KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum
Posted by Ted Dunning <te...@gmail.com>.
On Wed, Aug 7, 2019 at 11:31 AM Karolos Antoniadis <ka...@gmail.com>
wrote:
> In the paragraph that Michael mentioned, among others it is written: "For
> example, when a partition leader changes its ISR in ZK, the controller will
> typically not learn about these changes for many seconds." Why would it
> take "many seconds"?
I think that this is conflating the situation with partition from or of the
ZK cluster with simple hand-offs.
In the case of a partition leader crashing, it will be several seconds
before the rest of the world hears about the event.
> Sending a watch event to the controller should be
> pretty fast.
>
Absolutely. If the ZK cluster has its act together. And if the cause of the
watch is detected quickly. And if you don't have a watch storm happening
due to huge numbers of listeners.
But none of those problems are really helped by moving the consensus
algorithms into a library.
Also, in the same paragraph, Colin states "By the time the controller
> re-reads the znode and sets up a new watch, the state may have changed from
> what it was when the watch originally fired. [...] only way to resolve the
> discrepancy." Why would this lead to any discrepancy? It seems to me that
> the controller, will read an even newer state in such a scenario.
>
You are correct and this has always been one of the selling points of ZK.
The way that you can reset the watch as part of the read operation means
that you can guarantee never to lose anything and if you are slow to
respond, you always get data that is as up-to-date as possible. Load
shedding tricks like that are really helpful. Getting notifications of
every change is actually disastrous in many cases, partly because of the
number of notifications and partly because the notifications can become
very heavy-weight with the data they have to carry.
Putting this into a library doesn't help at all, of course.
>
> Also, another argument mentioned in original KIP-500 proposal had to do
> with speeding up the failover of a controller: "Because the controllers
> will now all track the latest state, controller failover will not require a
> lengthy reloading period where we transfer all the state to the new
> controller." But this does not seem to be a problem with ZK per se and
> could be solved by keeping a broker as a standby controller (briefly
> mentioned here https://www.slideshare.net/ConfluentInc/a-deep-dive-into-
> kafka-controller
> <https://www.slideshare.net/ConfluentInc/a-deep-dive-into-kafka-controller>
> as future work.)
>
Also, the state still has to move. Using an in-process library doesn't
change that at all. It could move via ZK or it could move as part of quorum
decisions or via some sort of follow-the-leader protocol. But it has to
move. Whoever is leader has to write it out to the network and whoever is
follower has to read it in. Whether the data is written/read directly or
via ZK isn't really a big deal.
>
Re: KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum
Posted by Karolos Antoniadis <ka...@gmail.com>.
In the paragraph that Michael mentioned, among others it is written: "For
example, when a partition leader changes its ISR in ZK, the controller will
typically not learn about these changes for many seconds." Why would it
take "many seconds"? Sending a watch event to the controller should be
pretty fast.
Also, in the same paragraph, Colin states "By the time the controller
re-reads the znode and sets up a new watch, the state may have changed from
what it was when the watch originally fired. [...] only way to resolve the
discrepancy." Why would this lead to any discrepancy? It seems to me that
the controller, will read an even newer state in such a scenario.
Also, another argument mentioned in original KIP-500 proposal had to do
with speeding up the failover of a controller: "Because the controllers
will now all track the latest state, controller failover will not require a
lengthy reloading period where we transfer all the state to the new
controller." But this does not seem to be a problem with ZK per se and
could be solved by keeping a broker as a standby controller (briefly
mentioned here https://www.slideshare.net/ConfluentInc/a-deep-dive-into-
kafka-controller as future work.)
By the way, in regards to the deployment & configuration issue, Colin
provides some more arguments in Kafka's dev mailing list (https://lists.
apache.org/list.html?dev@kafka.apache.org:lte=1M:KIP-500) if anybody is
interested in having a look.
On Fri, 2 Aug 2019 at 16:01, Michael Han <ha...@apache.org> wrote:
> Very well said, thank you Ted!
>
> >> I would still opt for quorum outside rather than quorum as a library.
>
> One observation on out side quorum vs library: for Raft, cockroach db and
> TiDB both choose the library approach instead of depending on etcd, though
> they all share the etcd's Raft implementation. ZooKeeper could be used in a
> similar approach if we can abstract ZAB and provides a nice SMR interface
> on top of it.
>
> On Fri, Aug 2, 2019 at 12:44 PM Ted Dunning <te...@gmail.com> wrote:
>
> > The core issue in these situations in my experience is that having the
> > quorum as a separate service can be a pain point. This misunderstanding
> > about how watches work and why they don't provide the data is just a
> > symptom of this. Having an integrated quorum is very attractive from the
> > point of view of management and tighter integration with the record of
> > state.
> >
> > If I had it all to do over again, though, I think I would still opt for
> > quorum outside rather than quorum as a library. There are management
> > burdens, but many of those management burdens are implicit in the fact
> that
> > managing the state of the system is different from managing the system or
> > doing the stuff the system does. Pulling the quorum system into the
> > do-stuff system doesn't actually make life all that much easier even if
> it
> > does simplify the installer.
> >
> > The countervailing risk that you are likely to get a quorum system wrong
> is
> > really significant. Having a battle-tested (some might say
> battle-scarred)
> > system like ZK is quite a virtue since you can have a different level of
> > confidence in it than something you whipped up last week.
> >
> >
> >
> > On Fri, Aug 2, 2019 at 11:49 AM Patrick Hunt <ph...@apache.org> wrote:
> >
> > > Michael I think you are describing subscribe - this?
> > > https://issues.apache.org/jira/browse/ZOOKEEPER-153
> > > wasn't there some work done to keep tlogs around for a while? Or am I
> > miss
> > > remembering? (fb folks?)
> > >
> > > I'll also add that we haven't done any benchmarking in quite some time.
> > It
> > > would be interesting to collect a few of these use cases from the
> > > community, esp downstreams, and evaluate performance, see if we can
> > > address.
> > >
> > > Patrick
> > >
> > > On Fri, Aug 2, 2019 at 11:03 AM Michael Han <ha...@apache.org> wrote:
> > >
> > > > Folks,
> > > >
> > > > Some of you might already see this. Comments?
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum
> > > >
> > > >
> > > > What caught my eyes are:
> > > >
> > > > *Worse still, although ZooKeeper is the store of record, the state in
> > > > ZooKeeper often doesn't match the state that is held in memory in the
> > > > controller. For example, when a partition leader changes its ISR in
> > ZK,
> > > > the controller will typically not learn about these changes for many
> > > > seconds. There is no generic way for the controller to follow the
> > > > ZooKeeper event log. Although the controller can set one-shot
> watches,
> > > the
> > > > number of watches is limited for performance reasons. When a watch
> > > > triggers, it doesn't tell the controller the current state-- only
> that
> > > the
> > > > state has changed. By the time the controller re-reads the znode and
> > > sets
> > > > up a new watch, the state may have changed from what it was when the
> > > watch
> > > > originally fired. If there is no watch set, the controller may not
> > learn
> > > > about the change at all. In some cases, restarting the controller is
> > the
> > > > only way to resolve the discrepancy.*
> > > >
> > > > I've seen some similar zookeeper use cases that ended up like what's
> > > > described here. How can ZooKeeper solve this? It seems to me that the
> > > only
> > > > solution is to provide linearizable read on watched operations.
> > Thoughts?
> > > >
> > > > Michael.
> > > >
> > >
> >
>
Re: KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum
Posted by Michael Han <ha...@apache.org>.
Very well said, thank you Ted!
>> I would still opt for quorum outside rather than quorum as a library.
One observation on out side quorum vs library: for Raft, cockroach db and
TiDB both choose the library approach instead of depending on etcd, though
they all share the etcd's Raft implementation. ZooKeeper could be used in a
similar approach if we can abstract ZAB and provides a nice SMR interface
on top of it.
On Fri, Aug 2, 2019 at 12:44 PM Ted Dunning <te...@gmail.com> wrote:
> The core issue in these situations in my experience is that having the
> quorum as a separate service can be a pain point. This misunderstanding
> about how watches work and why they don't provide the data is just a
> symptom of this. Having an integrated quorum is very attractive from the
> point of view of management and tighter integration with the record of
> state.
>
> If I had it all to do over again, though, I think I would still opt for
> quorum outside rather than quorum as a library. There are management
> burdens, but many of those management burdens are implicit in the fact that
> managing the state of the system is different from managing the system or
> doing the stuff the system does. Pulling the quorum system into the
> do-stuff system doesn't actually make life all that much easier even if it
> does simplify the installer.
>
> The countervailing risk that you are likely to get a quorum system wrong is
> really significant. Having a battle-tested (some might say battle-scarred)
> system like ZK is quite a virtue since you can have a different level of
> confidence in it than something you whipped up last week.
>
>
>
> On Fri, Aug 2, 2019 at 11:49 AM Patrick Hunt <ph...@apache.org> wrote:
>
> > Michael I think you are describing subscribe - this?
> > https://issues.apache.org/jira/browse/ZOOKEEPER-153
> > wasn't there some work done to keep tlogs around for a while? Or am I
> miss
> > remembering? (fb folks?)
> >
> > I'll also add that we haven't done any benchmarking in quite some time.
> It
> > would be interesting to collect a few of these use cases from the
> > community, esp downstreams, and evaluate performance, see if we can
> > address.
> >
> > Patrick
> >
> > On Fri, Aug 2, 2019 at 11:03 AM Michael Han <ha...@apache.org> wrote:
> >
> > > Folks,
> > >
> > > Some of you might already see this. Comments?
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum
> > >
> > >
> > > What caught my eyes are:
> > >
> > > *Worse still, although ZooKeeper is the store of record, the state in
> > > ZooKeeper often doesn't match the state that is held in memory in the
> > > controller. For example, when a partition leader changes its ISR in
> ZK,
> > > the controller will typically not learn about these changes for many
> > > seconds. There is no generic way for the controller to follow the
> > > ZooKeeper event log. Although the controller can set one-shot watches,
> > the
> > > number of watches is limited for performance reasons. When a watch
> > > triggers, it doesn't tell the controller the current state-- only that
> > the
> > > state has changed. By the time the controller re-reads the znode and
> > sets
> > > up a new watch, the state may have changed from what it was when the
> > watch
> > > originally fired. If there is no watch set, the controller may not
> learn
> > > about the change at all. In some cases, restarting the controller is
> the
> > > only way to resolve the discrepancy.*
> > >
> > > I've seen some similar zookeeper use cases that ended up like what's
> > > described here. How can ZooKeeper solve this? It seems to me that the
> > only
> > > solution is to provide linearizable read on watched operations.
> Thoughts?
> > >
> > > Michael.
> > >
> >
>
Re: KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum
Posted by Ted Dunning <te...@gmail.com>.
The core issue in these situations in my experience is that having the
quorum as a separate service can be a pain point. This misunderstanding
about how watches work and why they don't provide the data is just a
symptom of this. Having an integrated quorum is very attractive from the
point of view of management and tighter integration with the record of
state.
If I had it all to do over again, though, I think I would still opt for
quorum outside rather than quorum as a library. There are management
burdens, but many of those management burdens are implicit in the fact that
managing the state of the system is different from managing the system or
doing the stuff the system does. Pulling the quorum system into the
do-stuff system doesn't actually make life all that much easier even if it
does simplify the installer.
The countervailing risk that you are likely to get a quorum system wrong is
really significant. Having a battle-tested (some might say battle-scarred)
system like ZK is quite a virtue since you can have a different level of
confidence in it than something you whipped up last week.
On Fri, Aug 2, 2019 at 11:49 AM Patrick Hunt <ph...@apache.org> wrote:
> Michael I think you are describing subscribe - this?
> https://issues.apache.org/jira/browse/ZOOKEEPER-153
> wasn't there some work done to keep tlogs around for a while? Or am I miss
> remembering? (fb folks?)
>
> I'll also add that we haven't done any benchmarking in quite some time. It
> would be interesting to collect a few of these use cases from the
> community, esp downstreams, and evaluate performance, see if we can
> address.
>
> Patrick
>
> On Fri, Aug 2, 2019 at 11:03 AM Michael Han <ha...@apache.org> wrote:
>
> > Folks,
> >
> > Some of you might already see this. Comments?
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum
> >
> >
> > What caught my eyes are:
> >
> > *Worse still, although ZooKeeper is the store of record, the state in
> > ZooKeeper often doesn't match the state that is held in memory in the
> > controller. For example, when a partition leader changes its ISR in ZK,
> > the controller will typically not learn about these changes for many
> > seconds. There is no generic way for the controller to follow the
> > ZooKeeper event log. Although the controller can set one-shot watches,
> the
> > number of watches is limited for performance reasons. When a watch
> > triggers, it doesn't tell the controller the current state-- only that
> the
> > state has changed. By the time the controller re-reads the znode and
> sets
> > up a new watch, the state may have changed from what it was when the
> watch
> > originally fired. If there is no watch set, the controller may not learn
> > about the change at all. In some cases, restarting the controller is the
> > only way to resolve the discrepancy.*
> >
> > I've seen some similar zookeeper use cases that ended up like what's
> > described here. How can ZooKeeper solve this? It seems to me that the
> only
> > solution is to provide linearizable read on watched operations. Thoughts?
> >
> > Michael.
> >
>
Re: KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum
Posted by Michael Han <ha...@apache.org>.
Hi Pat,
Yes, ZOOKEEPER-153 could help this case. The gist of the issue is reliable
change notification with data. The linearizable read I had in mind alone
might not solve this as it's missing the reliably capturing change
notification part.
>> I'll also add that we haven't done any benchmarking in quite some time.
I think this is a very good point. The existing public benchmarks are
either targeted old version, or not optimally set up. This creates a gap
between current scalability and performance of ZK and the existing (usually
negative) public perception. With many improvements on scale / perf in last
2 years the status quo is very different now.
On Fri, Aug 2, 2019 at 11:49 AM Patrick Hunt <ph...@apache.org> wrote:
> Michael I think you are describing subscribe - this?
> https://issues.apache.org/jira/browse/ZOOKEEPER-153
> wasn't there some work done to keep tlogs around for a while? Or am I miss
> remembering? (fb folks?)
>
> I'll also add that we haven't done any benchmarking in quite some time. It
> would be interesting to collect a few of these use cases from the
> community, esp downstreams, and evaluate performance, see if we can
> address.
>
> Patrick
>
> On Fri, Aug 2, 2019 at 11:03 AM Michael Han <ha...@apache.org> wrote:
>
> > Folks,
> >
> > Some of you might already see this. Comments?
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum
> >
> >
> > What caught my eyes are:
> >
> > *Worse still, although ZooKeeper is the store of record, the state in
> > ZooKeeper often doesn't match the state that is held in memory in the
> > controller. For example, when a partition leader changes its ISR in ZK,
> > the controller will typically not learn about these changes for many
> > seconds. There is no generic way for the controller to follow the
> > ZooKeeper event log. Although the controller can set one-shot watches,
> the
> > number of watches is limited for performance reasons. When a watch
> > triggers, it doesn't tell the controller the current state-- only that
> the
> > state has changed. By the time the controller re-reads the znode and
> sets
> > up a new watch, the state may have changed from what it was when the
> watch
> > originally fired. If there is no watch set, the controller may not learn
> > about the change at all. In some cases, restarting the controller is the
> > only way to resolve the discrepancy.*
> >
> > I've seen some similar zookeeper use cases that ended up like what's
> > described here. How can ZooKeeper solve this? It seems to me that the
> only
> > solution is to provide linearizable read on watched operations. Thoughts?
> >
> > Michael.
> >
>
Re: KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum
Posted by Andor Molnar <an...@apache.org>.
Sure, well said. Don't get me wrong when I'm kidding Ted: I strongly
believe that Kafka is misusing ZooKeeper for their event processing,
although I don't know the details, just read the linked post on their
wiki. Details might be important here, because at first glance it
seemed to me that they expect a functionality in ZooKeeper which they
suppose to provide.
Back to the subscriber topic: I'm planning to resurrect the patch that
Jordan is referring to.
I think it would be a great new feature in 3.6 or 4.0.
Andor
-----Original Message-----
From: Ted Dunning <te...@gmail.com>
Reply-To: dev@zookeeper.apache.org
To: dev@zookeeper.apache.org
Subject: Re: KIP-500: Replace ZooKeeper with a Self-Managed Metadata
Quorum
Date: Tue, 13 Aug 2019 12:00:36 -0700
On Tue, Aug 13, 2019 at 10:41 AM Andor Molnar <an...@apache.org> wrote:
> Hmmm... and what should Kafka do if it wants to see all events?
> UseRabbitMQ? :)
Kafka should not be using ZK to store messages. And synchronization
ofconfiguration doesn't require message passing.
Read the original ZK paper [1] to understand that coordination and
reliablestate synchronization in ZK is not message passing. The point
is aboutproviding primitives to allow a read cache of roughly coherent
values withgood bounds on the semantics. Kafka should use ZK to
coordinate who ismaster and should use its own mechanisms to pass
messages. A key point isthe performance that can be gained by using a
cache coherence strategyrather than an "all updates" strategy.
Look at the Omega scheduler[2] paper to see the motivation behind
thesmall-steps style of scheduling and resource allocation.
Seriously, it you are doing things like designating where replicas go
andwho has the baton, it is critical to fast forward to things as they
are nowrather than things as they were. When the load hits the fan, it
isimportant to be able to ignore the water under the bridge.
This sort of topic arises in lots of other places. People complain that
itis hard to get synchronized and atomic updates to multiple files in
adistributed file system. Moral: don't use a file system when
asynchronization tool like ZK is needed.
People complain about message passing systems that they are slow in
termsof total write speed (can't write 100's of GB/s) and don't allow
native andefficient reading of state from a large data structure.
Moral: don't use amessage store when you really want a file store.
People complain the ZK's watches only give an update to latest state
sothey have to put messages into independent znodes and performance and
scalegoes to crap. Moral: don't use a synchronization system to store
messages.
Getting architectural basics close to write goes a long way.
[1]
https://www.usenix.org/legacy/event/usenix10/tech/full_papers/Hunt.pdf
[2] https://ai.google/research/pubs/pub41684
Re: KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum
Posted by Ted Dunning <te...@gmail.com>.
On Tue, Aug 13, 2019 at 10:41 AM Andor Molnar <an...@apache.org> wrote:
> Hmmm... and what should Kafka do if it wants to see all events? Use
> RabbitMQ? :)
Kafka should not be using ZK to store messages. And synchronization of
configuration doesn't require message passing.
Read the original ZK paper [1] to understand that coordination and reliable
state synchronization in ZK is not message passing. The point is about
providing primitives to allow a read cache of roughly coherent values with
good bounds on the semantics. Kafka should use ZK to coordinate who is
master and should use its own mechanisms to pass messages. A key point is
the performance that can be gained by using a cache coherence strategy
rather than an "all updates" strategy.
Look at the Omega scheduler[2] paper to see the motivation behind the
small-steps style of scheduling and resource allocation.
Seriously, it you are doing things like designating where replicas go and
who has the baton, it is critical to fast forward to things as they are now
rather than things as they were. When the load hits the fan, it is
important to be able to ignore the water under the bridge.
This sort of topic arises in lots of other places. People complain that it
is hard to get synchronized and atomic updates to multiple files in a
distributed file system. Moral: don't use a file system when a
synchronization tool like ZK is needed.
People complain about message passing systems that they are slow in terms
of total write speed (can't write 100's of GB/s) and don't allow native and
efficient reading of state from a large data structure. Moral: don't use a
message store when you really want a file store.
People complain the ZK's watches only give an update to latest state so
they have to put messages into independent znodes and performance and scale
goes to crap. Moral: don't use a synchronization system to store messages.
Getting architectural basics close to write goes a long way.
[1] https://www.usenix.org/legacy/event/usenix10/tech/full_papers/Hunt.pdf
[2] https://ai.google/research/pubs/pub41684
Re: KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum
Posted by Andor Molnar <an...@apache.org>.
Hmmm... and what should Kafka do if it wants to see all events? Use
RabbitMQ? :)
Andor
-----Original Message-----From: Ted Dunning <te...@gmail.com>Repl
y-To: dev@zookeeper.apache.orgTo: dev@zookeeper.apache.orgSubject: Re:
KIP-500: Replace ZooKeeper with a Self-Managed Metadata QuorumDate:
Tue, 13 Aug 2019 09:10:04 -0700
If you want to see all events, use Kafka.
On Tue, Aug 13, 2019 at 8:20 AM Jordan Zimmerman <
jordan@jordanzimmerman.com>wrote:
> Also see https://issues.apache.org/jira/browse/ZOOKEEPER-1416 <
> https://issues.apache.org/jira/browse/ZOOKEEPER-1416>
> There are many use cases where a client wants to see all events from
> agiven parent path down. The semantics of setting one-time watches on
> asingle node in ZK are cumbersome for these use cases. FWIW I had a
> workingPR a few years ago but it's fallen far behind 3.6 now.
> -Jordan
> > On Aug 13, 2019, at 8:18 AM, Andor Molnar <an...@apache.org> wrote:
> > Subscriber APIhttps://issues.apache.org/jira/browse/ZOOKEEPER-153
> >
> > Is it supposed to be something like a generic Observer API on the
> > client
> side?
> > Observers essentially consume ordered updates of ZAB, so we would
> > need
> to provide a way for users to implement their own “observers”. They
> shouldbe able to filter for path to be more convenient.
> > Andor
> >
> >
> > > On 2019. Aug 2., at 20:48, Patrick Hunt <ph...@apache.org> wrote:
> > > Michael I think you are describing subscribe - this?
> > > https://issues.apache.org/jira/browse/ZOOKEEPER-153
> > > wasn't there some work done to keep tlogs around for a while? Or
> > > am I
> miss
> > > remembering? (fb folks?)
> > > I'll also add that we haven't done any benchmarking in quite some
> > > time.
> It
> > > would be interesting to collect a few of these use cases from
> > > thecommunity, esp downstreams, and evaluate performance, see if
> > > we can
> address.
> > > Patrick
> > > On Fri, Aug 2, 2019 at 11:03 AM Michael Han <ha...@apache.org>
> > > wrote:
> > > > Folks,
> > > > Some of you might already see this. Comments?
> > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum
> > > > What caught my eyes are:
> > > > *Worse still, although ZooKeeper is the store of record, the
> > > > state inZooKeeper often doesn't match the state that is held in
> > > > memory in thecontroller. For example, when a partition leader
> > > > changes its ISR in
> ZK,
> > > > the controller will typically not learn about these changes for
> > > > manyseconds. There is no generic way for the controller to
> > > > follow theZooKeeper event log. Although the controller can set
> > > > one-shot
> watches, the
> > > > number of watches is limited for performance reasons. When a
> > > > watchtriggers, it doesn't tell the controller the current
> > > > state-- only that
> the
> > > > state has changed. By the time the controller re-reads the
> > > > znode and
> sets
> > > > up a new watch, the state may have changed from what it was
> > > > when the
> watch
> > > > originally fired. If there is no watch set, the controller may
> > > > not
> learn
> > > > about the change at all. In some cases, restarting the
> > > > controller is
> the
> > > > only way to resolve the discrepancy.*
> > > > I've seen some similar zookeeper use cases that ended up like
> > > > what'sdescribed here. How can ZooKeeper solve this? It seems to
> > > > me that the
> only
> > > > solution is to provide linearizable read on watched operations.
> Thoughts?
> > > > Michael.
Re: KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum
Posted by Patrick Hunt <ph...@apache.org>.
On Tue, Aug 13, 2019 at 9:10 AM Ted Dunning <te...@gmail.com> wrote:
> If you want to see all events, use Kafka.
>
>
heh - I was thinking the same thing. ;-)
Patrick
>
>
> On Tue, Aug 13, 2019 at 8:20 AM Jordan Zimmerman <
> jordan@jordanzimmerman.com>
> wrote:
>
> > Also see https://issues.apache.org/jira/browse/ZOOKEEPER-1416 <
> > https://issues.apache.org/jira/browse/ZOOKEEPER-1416>
> >
> > There are many use cases where a client wants to see all events from a
> > given parent path down. The semantics of setting one-time watches on a
> > single node in ZK are cumbersome for these use cases. FWIW I had a
> working
> > PR a few years ago but it's fallen far behind 3.6 now.
> >
> > -Jordan
> >
> > > On Aug 13, 2019, at 8:18 AM, Andor Molnar <an...@apache.org> wrote:
> > >
> > > Subscriber API
> > > https://issues.apache.org/jira/browse/ZOOKEEPER-153
> > >
> > > Is it supposed to be something like a generic Observer API on the
> client
> > side?
> > > Observers essentially consume ordered updates of ZAB, so we would need
> > to provide a way for users to implement their own “observers”. They
> should
> > be able to filter for path to be more convenient.
> > >
> > > Andor
> > >
> > >
> > >
> > >> On 2019. Aug 2., at 20:48, Patrick Hunt <ph...@apache.org> wrote:
> > >>
> > >> Michael I think you are describing subscribe - this?
> > >> https://issues.apache.org/jira/browse/ZOOKEEPER-153
> > >> wasn't there some work done to keep tlogs around for a while? Or am I
> > miss
> > >> remembering? (fb folks?)
> > >>
> > >> I'll also add that we haven't done any benchmarking in quite some
> time.
> > It
> > >> would be interesting to collect a few of these use cases from the
> > >> community, esp downstreams, and evaluate performance, see if we can
> > address.
> > >>
> > >> Patrick
> > >>
> > >> On Fri, Aug 2, 2019 at 11:03 AM Michael Han <ha...@apache.org> wrote:
> > >>
> > >>> Folks,
> > >>>
> > >>> Some of you might already see this. Comments?
> > >>>
> > >>>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum
> > >>>
> > >>>
> > >>> What caught my eyes are:
> > >>>
> > >>> *Worse still, although ZooKeeper is the store of record, the state in
> > >>> ZooKeeper often doesn't match the state that is held in memory in the
> > >>> controller. For example, when a partition leader changes its ISR in
> > ZK,
> > >>> the controller will typically not learn about these changes for many
> > >>> seconds. There is no generic way for the controller to follow the
> > >>> ZooKeeper event log. Although the controller can set one-shot
> > watches, the
> > >>> number of watches is limited for performance reasons. When a watch
> > >>> triggers, it doesn't tell the controller the current state-- only
> that
> > the
> > >>> state has changed. By the time the controller re-reads the znode and
> > sets
> > >>> up a new watch, the state may have changed from what it was when the
> > watch
> > >>> originally fired. If there is no watch set, the controller may not
> > learn
> > >>> about the change at all. In some cases, restarting the controller is
> > the
> > >>> only way to resolve the discrepancy.*
> > >>>
> > >>> I've seen some similar zookeeper use cases that ended up like what's
> > >>> described here. How can ZooKeeper solve this? It seems to me that the
> > only
> > >>> solution is to provide linearizable read on watched operations.
> > Thoughts?
> > >>>
> > >>> Michael.
> > >>>
> > >
> >
> >
>
Re: KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum
Posted by Ted Dunning <te...@gmail.com>.
If you want to see all events, use Kafka.
On Tue, Aug 13, 2019 at 8:20 AM Jordan Zimmerman <jo...@jordanzimmerman.com>
wrote:
> Also see https://issues.apache.org/jira/browse/ZOOKEEPER-1416 <
> https://issues.apache.org/jira/browse/ZOOKEEPER-1416>
>
> There are many use cases where a client wants to see all events from a
> given parent path down. The semantics of setting one-time watches on a
> single node in ZK are cumbersome for these use cases. FWIW I had a working
> PR a few years ago but it's fallen far behind 3.6 now.
>
> -Jordan
>
> > On Aug 13, 2019, at 8:18 AM, Andor Molnar <an...@apache.org> wrote:
> >
> > Subscriber API
> > https://issues.apache.org/jira/browse/ZOOKEEPER-153
> >
> > Is it supposed to be something like a generic Observer API on the client
> side?
> > Observers essentially consume ordered updates of ZAB, so we would need
> to provide a way for users to implement their own “observers”. They should
> be able to filter for path to be more convenient.
> >
> > Andor
> >
> >
> >
> >> On 2019. Aug 2., at 20:48, Patrick Hunt <ph...@apache.org> wrote:
> >>
> >> Michael I think you are describing subscribe - this?
> >> https://issues.apache.org/jira/browse/ZOOKEEPER-153
> >> wasn't there some work done to keep tlogs around for a while? Or am I
> miss
> >> remembering? (fb folks?)
> >>
> >> I'll also add that we haven't done any benchmarking in quite some time.
> It
> >> would be interesting to collect a few of these use cases from the
> >> community, esp downstreams, and evaluate performance, see if we can
> address.
> >>
> >> Patrick
> >>
> >> On Fri, Aug 2, 2019 at 11:03 AM Michael Han <ha...@apache.org> wrote:
> >>
> >>> Folks,
> >>>
> >>> Some of you might already see this. Comments?
> >>>
> >>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum
> >>>
> >>>
> >>> What caught my eyes are:
> >>>
> >>> *Worse still, although ZooKeeper is the store of record, the state in
> >>> ZooKeeper often doesn't match the state that is held in memory in the
> >>> controller. For example, when a partition leader changes its ISR in
> ZK,
> >>> the controller will typically not learn about these changes for many
> >>> seconds. There is no generic way for the controller to follow the
> >>> ZooKeeper event log. Although the controller can set one-shot
> watches, the
> >>> number of watches is limited for performance reasons. When a watch
> >>> triggers, it doesn't tell the controller the current state-- only that
> the
> >>> state has changed. By the time the controller re-reads the znode and
> sets
> >>> up a new watch, the state may have changed from what it was when the
> watch
> >>> originally fired. If there is no watch set, the controller may not
> learn
> >>> about the change at all. In some cases, restarting the controller is
> the
> >>> only way to resolve the discrepancy.*
> >>>
> >>> I've seen some similar zookeeper use cases that ended up like what's
> >>> described here. How can ZooKeeper solve this? It seems to me that the
> only
> >>> solution is to provide linearizable read on watched operations.
> Thoughts?
> >>>
> >>> Michael.
> >>>
> >
>
>
Re: KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
This is the amount of code a client must write to achieve this in ZooKeeper: https://github.com/apache/curator/blob/master/curator-recipes/src/main/java/org/apache/curator/framework/recipes/cache/TreeCache.java <https://github.com/apache/curator/blob/master/curator-recipes/src/main/java/org/apache/curator/framework/recipes/cache/TreeCache.java> - note that this class takes advantage of Curator as well. Comparing this use case to Kafka is mistaken. While some users might want durable and complete events, most really only want a simple way to follow everything that happens from a given parent downward. That this is effectively impossible to do in ZooKeeper (other than using Curator) is a large hole IMO.
-Jordan
> On Aug 13, 2019, at 8:20 AM, Jordan Zimmerman <jo...@jordanzimmerman.com> wrote:
>
> Also see https://issues.apache.org/jira/browse/ZOOKEEPER-1416 <https://issues.apache.org/jira/browse/ZOOKEEPER-1416>
>
> There are many use cases where a client wants to see all events from a given parent path down. The semantics of setting one-time watches on a single node in ZK are cumbersome for these use cases. FWIW I had a working PR a few years ago but it's fallen far behind 3.6 now.
>
> -Jordan
>
>> On Aug 13, 2019, at 8:18 AM, Andor Molnar <andor@apache.org <ma...@apache.org>> wrote:
>>
>> Subscriber API
>> https://issues.apache.org/jira/browse/ZOOKEEPER-153 <https://issues.apache.org/jira/browse/ZOOKEEPER-153>
>>
>> Is it supposed to be something like a generic Observer API on the client side?
>> Observers essentially consume ordered updates of ZAB, so we would need to provide a way for users to implement their own “observers”. They should be able to filter for path to be more convenient.
>>
>> Andor
>>
>>
>>
>>> On 2019. Aug 2., at 20:48, Patrick Hunt <ph...@apache.org> wrote:
>>>
>>> Michael I think you are describing subscribe - this?
>>> https://issues.apache.org/jira/browse/ZOOKEEPER-153
>>> wasn't there some work done to keep tlogs around for a while? Or am I miss
>>> remembering? (fb folks?)
>>>
>>> I'll also add that we haven't done any benchmarking in quite some time. It
>>> would be interesting to collect a few of these use cases from the
>>> community, esp downstreams, and evaluate performance, see if we can address.
>>>
>>> Patrick
>>>
>>> On Fri, Aug 2, 2019 at 11:03 AM Michael Han <ha...@apache.org> wrote:
>>>
>>>> Folks,
>>>>
>>>> Some of you might already see this. Comments?
>>>>
>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum
>>>>
>>>>
>>>> What caught my eyes are:
>>>>
>>>> *Worse still, although ZooKeeper is the store of record, the state in
>>>> ZooKeeper often doesn't match the state that is held in memory in the
>>>> controller. For example, when a partition leader changes its ISR in ZK,
>>>> the controller will typically not learn about these changes for many
>>>> seconds. There is no generic way for the controller to follow the
>>>> ZooKeeper event log. Although the controller can set one-shot watches, the
>>>> number of watches is limited for performance reasons. When a watch
>>>> triggers, it doesn't tell the controller the current state-- only that the
>>>> state has changed. By the time the controller re-reads the znode and sets
>>>> up a new watch, the state may have changed from what it was when the watch
>>>> originally fired. If there is no watch set, the controller may not learn
>>>> about the change at all. In some cases, restarting the controller is the
>>>> only way to resolve the discrepancy.*
>>>>
>>>> I've seen some similar zookeeper use cases that ended up like what's
>>>> described here. How can ZooKeeper solve this? It seems to me that the only
>>>> solution is to provide linearizable read on watched operations. Thoughts?
>>>>
>>>> Michael.
>>>>
>>
>
Re: KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
Also see https://issues.apache.org/jira/browse/ZOOKEEPER-1416 <https://issues.apache.org/jira/browse/ZOOKEEPER-1416>
There are many use cases where a client wants to see all events from a given parent path down. The semantics of setting one-time watches on a single node in ZK are cumbersome for these use cases. FWIW I had a working PR a few years ago but it's fallen far behind 3.6 now.
-Jordan
> On Aug 13, 2019, at 8:18 AM, Andor Molnar <an...@apache.org> wrote:
>
> Subscriber API
> https://issues.apache.org/jira/browse/ZOOKEEPER-153
>
> Is it supposed to be something like a generic Observer API on the client side?
> Observers essentially consume ordered updates of ZAB, so we would need to provide a way for users to implement their own “observers”. They should be able to filter for path to be more convenient.
>
> Andor
>
>
>
>> On 2019. Aug 2., at 20:48, Patrick Hunt <ph...@apache.org> wrote:
>>
>> Michael I think you are describing subscribe - this?
>> https://issues.apache.org/jira/browse/ZOOKEEPER-153
>> wasn't there some work done to keep tlogs around for a while? Or am I miss
>> remembering? (fb folks?)
>>
>> I'll also add that we haven't done any benchmarking in quite some time. It
>> would be interesting to collect a few of these use cases from the
>> community, esp downstreams, and evaluate performance, see if we can address.
>>
>> Patrick
>>
>> On Fri, Aug 2, 2019 at 11:03 AM Michael Han <ha...@apache.org> wrote:
>>
>>> Folks,
>>>
>>> Some of you might already see this. Comments?
>>>
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum
>>>
>>>
>>> What caught my eyes are:
>>>
>>> *Worse still, although ZooKeeper is the store of record, the state in
>>> ZooKeeper often doesn't match the state that is held in memory in the
>>> controller. For example, when a partition leader changes its ISR in ZK,
>>> the controller will typically not learn about these changes for many
>>> seconds. There is no generic way for the controller to follow the
>>> ZooKeeper event log. Although the controller can set one-shot watches, the
>>> number of watches is limited for performance reasons. When a watch
>>> triggers, it doesn't tell the controller the current state-- only that the
>>> state has changed. By the time the controller re-reads the znode and sets
>>> up a new watch, the state may have changed from what it was when the watch
>>> originally fired. If there is no watch set, the controller may not learn
>>> about the change at all. In some cases, restarting the controller is the
>>> only way to resolve the discrepancy.*
>>>
>>> I've seen some similar zookeeper use cases that ended up like what's
>>> described here. How can ZooKeeper solve this? It seems to me that the only
>>> solution is to provide linearizable read on watched operations. Thoughts?
>>>
>>> Michael.
>>>
>
Re: KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum
Posted by Andor Molnar <an...@apache.org>.
Subscriber API
https://issues.apache.org/jira/browse/ZOOKEEPER-153
Is it supposed to be something like a generic Observer API on the client side?
Observers essentially consume ordered updates of ZAB, so we would need to provide a way for users to implement their own “observers”. They should be able to filter for path to be more convenient.
Andor
> On 2019. Aug 2., at 20:48, Patrick Hunt <ph...@apache.org> wrote:
>
> Michael I think you are describing subscribe - this?
> https://issues.apache.org/jira/browse/ZOOKEEPER-153
> wasn't there some work done to keep tlogs around for a while? Or am I miss
> remembering? (fb folks?)
>
> I'll also add that we haven't done any benchmarking in quite some time. It
> would be interesting to collect a few of these use cases from the
> community, esp downstreams, and evaluate performance, see if we can address.
>
> Patrick
>
> On Fri, Aug 2, 2019 at 11:03 AM Michael Han <ha...@apache.org> wrote:
>
>> Folks,
>>
>> Some of you might already see this. Comments?
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum
>>
>>
>> What caught my eyes are:
>>
>> *Worse still, although ZooKeeper is the store of record, the state in
>> ZooKeeper often doesn't match the state that is held in memory in the
>> controller. For example, when a partition leader changes its ISR in ZK,
>> the controller will typically not learn about these changes for many
>> seconds. There is no generic way for the controller to follow the
>> ZooKeeper event log. Although the controller can set one-shot watches, the
>> number of watches is limited for performance reasons. When a watch
>> triggers, it doesn't tell the controller the current state-- only that the
>> state has changed. By the time the controller re-reads the znode and sets
>> up a new watch, the state may have changed from what it was when the watch
>> originally fired. If there is no watch set, the controller may not learn
>> about the change at all. In some cases, restarting the controller is the
>> only way to resolve the discrepancy.*
>>
>> I've seen some similar zookeeper use cases that ended up like what's
>> described here. How can ZooKeeper solve this? It seems to me that the only
>> solution is to provide linearizable read on watched operations. Thoughts?
>>
>> Michael.
>>
Re: KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum
Posted by Patrick Hunt <ph...@apache.org>.
Michael I think you are describing subscribe - this?
https://issues.apache.org/jira/browse/ZOOKEEPER-153
wasn't there some work done to keep tlogs around for a while? Or am I miss
remembering? (fb folks?)
I'll also add that we haven't done any benchmarking in quite some time. It
would be interesting to collect a few of these use cases from the
community, esp downstreams, and evaluate performance, see if we can address.
Patrick
On Fri, Aug 2, 2019 at 11:03 AM Michael Han <ha...@apache.org> wrote:
> Folks,
>
> Some of you might already see this. Comments?
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum
>
>
> What caught my eyes are:
>
> *Worse still, although ZooKeeper is the store of record, the state in
> ZooKeeper often doesn't match the state that is held in memory in the
> controller. For example, when a partition leader changes its ISR in ZK,
> the controller will typically not learn about these changes for many
> seconds. There is no generic way for the controller to follow the
> ZooKeeper event log. Although the controller can set one-shot watches, the
> number of watches is limited for performance reasons. When a watch
> triggers, it doesn't tell the controller the current state-- only that the
> state has changed. By the time the controller re-reads the znode and sets
> up a new watch, the state may have changed from what it was when the watch
> originally fired. If there is no watch set, the controller may not learn
> about the change at all. In some cases, restarting the controller is the
> only way to resolve the discrepancy.*
>
> I've seen some similar zookeeper use cases that ended up like what's
> described here. How can ZooKeeper solve this? It seems to me that the only
> solution is to provide linearizable read on watched operations. Thoughts?
>
> Michael.
>