You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@curator.apache.org by David Smiley <ds...@apache.org> on 2020/05/01 05:43:24 UTC
Dealing with eventual consistency
Hello,
I'm trying to come to grips with the ramifications of ZooKeeper's eventual
consistency model and what mechanisms exists to help.
Imagine two machines that use ZK, one of which stores data in it then tells
the other machine to do something that will require it to read what the
first machine wrote.
ZK's docs warn about this:
https://github.com/apache/zookeeper/blob/master/zookeeper-docs/src/main/resources/markdown/zookeeperProgrammers.md#ch_zkGuarantees
.. and refer to a sync() method to help:
https://github.com/apache/zookeeper/blob/2e14a29cc6e58d9561e80b737a3168fbb1f752b4/zookeeper-server/src/main/java/org/apache/zookeeper/ZooKeeper.java#L3057
But I'm worried about over-using sync() because I imagine it's not free.
For the scenario I have in mind, the vast majority of the time, the second
machine will see the latest state because lots of time passes between the
write and the read. Imagine this is some immutable configuration data that
is set once then used a lot by many ZK clients thereafter many thousands of
times for days on end.
I'm aware a znode or perhaps pzxid (for a path of data) could be passed to
the second machine... but for the system in question, this would be really
ugly and I want to consider alternatives. Besides, the data is organized
into a tree that could have arbitrary nesting, so it's not clear to me that
there's a single version for this any way.
Scott Blum told me about how there's an increasing "zxid" for all state
change in ZK. I can see this on ZK's ClientCnxn.getLastZxid(). If I were
to pass that zxid to the additional machines (ZK clients) from the first
for basically all interactions (not too ugly for the system), how would the
receiving machine use this to get in sync? I'm guessing it could read its
own connection zxid and if it's out of date than call sync()? Does that
make sense? Is there another strategy to be recommended? Does Curator
facilitate this in any way?
Thanks in advance! I already searched this list for answers.
~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley
Re: Dealing with eventual consistency
Posted by Scott Blum <dr...@gmail.com>.
FWIW, I dug through the docs, a bit of the source code, and some
internets. The amount of super clear information on this subject is
vanishingly small. :/
On Fri, May 1, 2020 at 1:43 AM David Smiley <ds...@apache.org> wrote:
> Hello,
>
> I'm trying to come to grips with the ramifications of ZooKeeper's eventual
> consistency model and what mechanisms exists to help.
>
> Imagine two machines that use ZK, one of which stores data in it then
> tells the other machine to do something that will require it to read what
> the first machine wrote.
>
> ZK's docs warn about this:
>
> https://github.com/apache/zookeeper/blob/master/zookeeper-docs/src/main/resources/markdown/zookeeperProgrammers.md#ch_zkGuarantees
> .. and refer to a sync() method to help:
>
> https://github.com/apache/zookeeper/blob/2e14a29cc6e58d9561e80b737a3168fbb1f752b4/zookeeper-server/src/main/java/org/apache/zookeeper/ZooKeeper.java#L3057
>
> But I'm worried about over-using sync() because I imagine it's not free.
> For the scenario I have in mind, the vast majority of the time, the second
> machine will see the latest state because lots of time passes between the
> write and the read. Imagine this is some immutable configuration data that
> is set once then used a lot by many ZK clients thereafter many thousands of
> times for days on end.
>
> I'm aware a znode or perhaps pzxid (for a path of data) could be passed to
> the second machine... but for the system in question, this would be really
> ugly and I want to consider alternatives. Besides, the data is organized
> into a tree that could have arbitrary nesting, so it's not clear to me that
> there's a single version for this any way.
>
> Scott Blum told me about how there's an increasing "zxid" for all state
> change in ZK. I can see this on ZK's ClientCnxn.getLastZxid(). If I were
> to pass that zxid to the additional machines (ZK clients) from the first
> for basically all interactions (not too ugly for the system), how would the
> receiving machine use this to get in sync? I'm guessing it could read its
> own connection zxid and if it's out of date than call sync()? Does that
> make sense? Is there another strategy to be recommended? Does Curator
> facilitate this in any way?
>
> Thanks in advance! I already searched this list for answers.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
Re: Dealing with eventual consistency
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
That part is true I'm not sure how much use it is though. If there are multiple writers you can't know what the latest version is - there may be other servers writing that have been seen yet. But, then, I probably don't completely understand the use case.
-JZ
> On May 2, 2020, at 9:35 AM, Scott Blum <dr...@gmail.com> wrote:
>
> I don't follow... I'm saying that if server A wants to be absolutely sure a write is visible to server B, it can grab the mxid of the write, send it to server B, and server B can ensure that its LastZxid >= that value before doing the read.
Re: Dealing with eventual consistency
Posted by Scott Blum <dr...@gmail.com>.
I don't follow... I'm saying that if server A wants to be absolutely sure a
write is visible to server B, it can grab the mxid of the write, send it to
server B, and server B can ensure that its LastZxid >= that value before
doing the read.
Re: Dealing with eventual consistency
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
You can still miss a pending write on the client. Just because you read N from the ZNode doesn't mean that that's it's real value on the leader. The only way in ZooKeeper to be certain is to do a write as those always take place on the leader.
-Jordan
> On May 1, 2020, at 5:51 PM, Scott Blum <dr...@gmail.com> wrote:
>
> On Fri, May 1, 2020 at 6:00 PM David Smiley <david.w.smiley@gmail.com <ma...@gmail.com>> wrote:
> I don't think I'll use the "LastZxid" trick because we update parts of the ZK tree with high frequency but not this one yet the Zxid would still soar upwards.
>
> I don't follow.. if server A writes a node, recording the mzxid associated with that write, and passes it along to server B, then server B just needs to be sure its LastZxid is >= the one that server A wrote. Doesn't matter if server B's LastZxid is the same, 10 ahead, or 1000 ahead.
Re: Dealing with eventual consistency
Posted by Scott Blum <dr...@gmail.com>.
On Fri, May 1, 2020 at 6:00 PM David Smiley <da...@gmail.com>
wrote:
> I don't think I'll use the "LastZxid" trick because we update parts of the
> ZK tree with high frequency but not this one yet the Zxid would still soar
> upwards.
>
I don't follow.. if server A writes a node, recording the mzxid associated
with that write, and passes it along to server B, then server B just needs
to be sure its LastZxid is >= the one that server A wrote. Doesn't matter
if server B's LastZxid is the same, 10 ahead, or 1000 ahead.
Re: Dealing with eventual consistency
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
You can also know whether or not you have a recent version of a ZNode by using the version. Read the node and its Stat, save that (Curator's Cache has this) and then update the node using the version. All updates occur on the current ZooKeeper leader so you get an exception if the version doesn't match.
Other than that, ZK is an eventually consistent system as you know.
-JZ
> On May 1, 2020, at 5:00 PM, David Smiley <da...@gmail.com> wrote:
>
> Thanks for pointing out that conversation RE sync() "Consistency Guarantees". It's a shame sync() has that deficiency of not actually getting a quorum, thus there's still an edge case.
>
> I don't think I'll use the "LastZxid" trick because we update parts of the ZK tree with high frequency but not this one yet the Zxid would still soar upwards.
>
> The Cache stuff is really nifty but still leaves an eventual consistency issue. Machine A need to get machine B to do something predicated on the cached thing having a version >= some value that Machine A knows.
>
> I think I must just deal with the fact of passing on a version / xid even though I said for the system in question it's ugly. I need to make it look pretty :-)
>
> ~ David
>
>
> On Fri, May 1, 2020 at 11:46 AM Jordan Zimmerman <jordan@jordanzimmerman.com <ma...@jordanzimmerman.com>> wrote:
>> I'm aware a znode or perhaps pzxid (for a path of data) could be passed to the second machine.
>
> It's available to the client. See: https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/ClientCnxn.java#L810 <https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/ClientCnxn.java#L810>
>
> You'd create your own ZooKeeper subclass - "cnxn" is a protected field. So, something like:
>
> public class MyZooKeeper extends ZooKeeper {
> ...
>
> public long getLastZxid() {
> return cnxn.getLastZxid();
> }
> }
>
>> But I'm worried about over-using sync() because I imagine it's not free.
>
> TBH - I've never understood the sync() method. I always thought it was useless but Alex Shraer wrote some ways that it can be useful. See: https://mail-archives.apache.org/mod_mbox/zookeeper-dev/201908.mbox/thread?2 <https://mail-archives.apache.org/mod_mbox/zookeeper-dev/201908.mbox/thread?2> (search for "Consistency Guarantees").
>
>> Imagine this is some immutable configuration data that is set once then used a lot by many ZK clients thereafter many thousands of times for days on end.
>
> Note: this is how Facebook uses ZooKeeper. It's the backbone of their config system. They have 10s of thousands (something like that) of read-only observers that sit in front of their main ZK ensemble.
>
>> Does Curator facilitate this in any way?
>
> This is what the various Cache recipes are for (Scott's TreeCache or the upcoming CuratorCache). These recipes take care of pulling down the latest versions of ZNodes for you.
>
> -Jordan
>
>> On May 1, 2020, at 12:43 AM, David Smiley <dsmiley@apache.org <ma...@apache.org>> wrote:
>>
>> Hello,
>>
>> I'm trying to come to grips with the ramifications of ZooKeeper's eventual consistency model and what mechanisms exists to help.
>>
>> Imagine two machines that use ZK, one of which stores data in it then tells the other machine to do something that will require it to read what the first machine wrote.
>>
>> ZK's docs warn about this:
>> https://github.com/apache/zookeeper/blob/master/zookeeper-docs/src/main/resources/markdown/zookeeperProgrammers.md#ch_zkGuarantees <https://github.com/apache/zookeeper/blob/master/zookeeper-docs/src/main/resources/markdown/zookeeperProgrammers.md#ch_zkGuarantees>
>> .. and refer to a sync() method to help:
>> https://github.com/apache/zookeeper/blob/2e14a29cc6e58d9561e80b737a3168fbb1f752b4/zookeeper-server/src/main/java/org/apache/zookeeper/ZooKeeper.java#L3057 <https://github.com/apache/zookeeper/blob/2e14a29cc6e58d9561e80b737a3168fbb1f752b4/zookeeper-server/src/main/java/org/apache/zookeeper/ZooKeeper.java#L3057>
>>
>> But I'm worried about over-using sync() because I imagine it's not free. For the scenario I have in mind, the vast majority of the time, the second machine will see the latest state because lots of time passes between the write and the read. Imagine this is some immutable configuration data that is set once then used a lot by many ZK clients thereafter many thousands of times for days on end.
>>
>> I'm aware a znode or perhaps pzxid (for a path of data) could be passed to the second machine... but for the system in question, this would be really ugly and I want to consider alternatives. Besides, the data is organized into a tree that could have arbitrary nesting, so it's not clear to me that there's a single version for this any way.
>>
>> Scott Blum told me about how there's an increasing "zxid" for all state change in ZK. I can see this on ZK's ClientCnxn.getLastZxid(). If I were to pass that zxid to the additional machines (ZK clients) from the first for basically all interactions (not too ugly for the system), how would the receiving machine use this to get in sync? I'm guessing it could read its own connection zxid and if it's out of date than call sync()? Does that make sense? Is there another strategy to be recommended? Does Curator facilitate this in any way?
>>
>> Thanks in advance! I already searched this list for answers.
>>
>> ~ David Smiley
>> Apache Lucene/Solr Search Developer
>> http://www.linkedin.com/in/davidwsmiley <http://www.linkedin.com/in/davidwsmiley>
Re: Dealing with eventual consistency
Posted by David Smiley <da...@gmail.com>.
Thanks for pointing out that conversation RE sync() "Consistency
Guarantees". It's a shame sync() has that deficiency of not actually
getting a quorum, thus there's still an edge case.
I don't think I'll use the "LastZxid" trick because we update parts of the
ZK tree with high frequency but not this one yet the Zxid would still soar
upwards.
The Cache stuff is really nifty but still leaves an eventual consistency
issue. Machine A need to get machine B to do something predicated on the
cached thing having a version >= some value that Machine A knows.
I think I must just deal with the fact of passing on a version / xid even
though I said for the system in question it's ugly. I need to make it look
pretty :-)
~ David
On Fri, May 1, 2020 at 11:46 AM Jordan Zimmerman <jo...@jordanzimmerman.com>
wrote:
> I'm aware a znode or perhaps pzxid (for a path of data) could be passed to
> the second machine.
>
>
> It's available to the client. See:
> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/ClientCnxn.java#L810
>
>
> You'd create your own ZooKeeper subclass - "cnxn" is a protected field.
> So, something like:
>
> public class MyZooKeeper extends ZooKeeper {
> ...
>
> public long getLastZxid() {
> return cnxn.getLastZxid();
> }
> }
>
> But I'm worried about over-using sync() because I imagine it's not free.
>
>
> TBH - I've never understood the sync() method. I always thought it was
> useless but Alex Shraer wrote some ways that it can be useful. See:
> https://mail-archives.apache.org/mod_mbox/zookeeper-dev/201908.mbox/thread?2
> (search for "Consistency Guarantees").
>
> Imagine this is some immutable configuration data that is set once then
> used a lot by many ZK clients thereafter many thousands of times for days
> on end.
>
>
> Note: this is how Facebook uses ZooKeeper. It's the backbone of their
> config system. They have 10s of thousands (something like that) of
> read-only observers that sit in front of their main ZK ensemble.
>
> Does Curator facilitate this in any way?
>
>
> This is what the various Cache recipes are for (Scott's TreeCache or the
> upcoming CuratorCache). These recipes take care of pulling down the latest
> versions of ZNodes for you.
>
> -Jordan
>
> On May 1, 2020, at 12:43 AM, David Smiley <ds...@apache.org> wrote:
>
> Hello,
>
> I'm trying to come to grips with the ramifications of ZooKeeper's eventual
> consistency model and what mechanisms exists to help.
>
> Imagine two machines that use ZK, one of which stores data in it then
> tells the other machine to do something that will require it to read what
> the first machine wrote.
>
> ZK's docs warn about this:
>
> https://github.com/apache/zookeeper/blob/master/zookeeper-docs/src/main/resources/markdown/zookeeperProgrammers.md#ch_zkGuarantees
> .. and refer to a sync() method to help:
>
> https://github.com/apache/zookeeper/blob/2e14a29cc6e58d9561e80b737a3168fbb1f752b4/zookeeper-server/src/main/java/org/apache/zookeeper/ZooKeeper.java#L3057
>
> But I'm worried about over-using sync() because I imagine it's not free.
> For the scenario I have in mind, the vast majority of the time, the second
> machine will see the latest state because lots of time passes between the
> write and the read. Imagine this is some immutable configuration data that
> is set once then used a lot by many ZK clients thereafter many thousands of
> times for days on end.
>
> I'm aware a znode or perhaps pzxid (for a path of data) could be passed to
> the second machine... but for the system in question, this would be really
> ugly and I want to consider alternatives. Besides, the data is organized
> into a tree that could have arbitrary nesting, so it's not clear to me that
> there's a single version for this any way.
>
> Scott Blum told me about how there's an increasing "zxid" for all state
> change in ZK. I can see this on ZK's ClientCnxn.getLastZxid(). If I were
> to pass that zxid to the additional machines (ZK clients) from the first
> for basically all interactions (not too ugly for the system), how would the
> receiving machine use this to get in sync? I'm guessing it could read its
> own connection zxid and if it's out of date than call sync()? Does that
> make sense? Is there another strategy to be recommended? Does Curator
> facilitate this in any way?
>
> Thanks in advance! I already searched this list for answers.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
>
Re: Dealing with eventual consistency
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
> I'm aware a znode or perhaps pzxid (for a path of data) could be passed to the second machine.
It's available to the client. See: https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/ClientCnxn.java#L810
You'd create your own ZooKeeper subclass - "cnxn" is a protected field. So, something like:
public class MyZooKeeper extends ZooKeeper {
...
public long getLastZxid() {
return cnxn.getLastZxid();
}
}
> But I'm worried about over-using sync() because I imagine it's not free.
TBH - I've never understood the sync() method. I always thought it was useless but Alex Shraer wrote some ways that it can be useful. See: https://mail-archives.apache.org/mod_mbox/zookeeper-dev/201908.mbox/thread?2 (search for "Consistency Guarantees").
> Imagine this is some immutable configuration data that is set once then used a lot by many ZK clients thereafter many thousands of times for days on end.
Note: this is how Facebook uses ZooKeeper. It's the backbone of their config system. They have 10s of thousands (something like that) of read-only observers that sit in front of their main ZK ensemble.
> Does Curator facilitate this in any way?
This is what the various Cache recipes are for (Scott's TreeCache or the upcoming CuratorCache). These recipes take care of pulling down the latest versions of ZNodes for you.
-Jordan
> On May 1, 2020, at 12:43 AM, David Smiley <ds...@apache.org> wrote:
>
> Hello,
>
> I'm trying to come to grips with the ramifications of ZooKeeper's eventual consistency model and what mechanisms exists to help.
>
> Imagine two machines that use ZK, one of which stores data in it then tells the other machine to do something that will require it to read what the first machine wrote.
>
> ZK's docs warn about this:
> https://github.com/apache/zookeeper/blob/master/zookeeper-docs/src/main/resources/markdown/zookeeperProgrammers.md#ch_zkGuarantees <https://github.com/apache/zookeeper/blob/master/zookeeper-docs/src/main/resources/markdown/zookeeperProgrammers.md#ch_zkGuarantees>
> .. and refer to a sync() method to help:
> https://github.com/apache/zookeeper/blob/2e14a29cc6e58d9561e80b737a3168fbb1f752b4/zookeeper-server/src/main/java/org/apache/zookeeper/ZooKeeper.java#L3057 <https://github.com/apache/zookeeper/blob/2e14a29cc6e58d9561e80b737a3168fbb1f752b4/zookeeper-server/src/main/java/org/apache/zookeeper/ZooKeeper.java#L3057>
>
> But I'm worried about over-using sync() because I imagine it's not free. For the scenario I have in mind, the vast majority of the time, the second machine will see the latest state because lots of time passes between the write and the read. Imagine this is some immutable configuration data that is set once then used a lot by many ZK clients thereafter many thousands of times for days on end.
>
> I'm aware a znode or perhaps pzxid (for a path of data) could be passed to the second machine... but for the system in question, this would be really ugly and I want to consider alternatives. Besides, the data is organized into a tree that could have arbitrary nesting, so it's not clear to me that there's a single version for this any way.
>
> Scott Blum told me about how there's an increasing "zxid" for all state change in ZK. I can see this on ZK's ClientCnxn.getLastZxid(). If I were to pass that zxid to the additional machines (ZK clients) from the first for basically all interactions (not too ugly for the system), how would the receiving machine use this to get in sync? I'm guessing it could read its own connection zxid and if it's out of date than call sync()? Does that make sense? Is there another strategy to be recommended? Does Curator facilitate this in any way?
>
> Thanks in advance! I already searched this list for answers.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley <http://www.linkedin.com/in/davidwsmiley>