Re: zookeeper client session write-read consistency

[Chris Nauroth <cn...@hortonworks.com>]

Hello Wayne,

No, this cannot happen.  ZooKeeper guarantees that a client session will
never "go back in time" and read state older than state that the client
session already read.  This is accomplished by exchanging a transaction
ID, called the "zxid", between the server and client.  On every response
from the server, it tells the client its last known committed zxid.  The
client remembers this zxid.  If at any point a client needs to reestablish
its session, there is a guarantee enforced that the session reestablishes
with a server that has a zxid equal to or greater than the client's last
observed zxid.

For more details, please see pages 9-10 of "ZooKeeper: Wait-free
coordination for Internet-scale systems".

http://static.cs.brown.edu/courses/cs227/archives/2012/papers/replication/h
unt.pdf


Quoting that paper:

ZooKeeper servers process requests from clients in
FIFO order. Responses include the zxid that the response
is relative to. Even heartbeat messages during intervals
of no activity include the last zxid seen by the server that
the client is connected to. If the client connects to a new
server, that new server ensures that its view of the ZooKeeper
data is at least as recent as the view of the client
by checking the last zxid of the client against its last zxid.
If the client has a more recent view than the server, the
server does not reestablish the session with the client until
the server has caught up. The client is guaranteed to
be able to find another server that has a recent view of the
system since the client only sees changes that have been
replicated to a majority of the ZooKeeper servers. This
behavior is important to guarantee durability.


--Chris Nauroth




On 2/27/16, 4:34 PM, "wayne" <fl...@gmail.com> wrote:

>Thanks German for the awesome answer!
>
>I have an interesting followup question. My question is how does Zookeeper
>guarantee that within a client session, a subsequent read always observes
>a
>previous write?
>
>More specifically, consider the following set up (to simplify the
>discussion
>assuming every operation is synchronous):
>
>I have Zookeeper server s1, s2 and s3. s1 is the ZAB master. I have a
>client
>c, which connects to s3 (its local replica) and establish a session with
>s3.
>
>(1) c issues a write, which goes to s3, s3 identifies that it is not the
>master, s3 forwards the request to s1. Then a network partition happens
>between s3 and s1, so s1 is only able to replicate the write to itself and
>s2. (The write succeeds because the majority agree to commit the write).
>Then the network partition heals itself, so s1 returns success to s3,
>which
>in turn returns success to the client.
>
>(2) c issues a read, the reads goes to s3, s3 serves the read locally,
>which
>does not reflect the write in (1).
>
>Could this ever happen? Thanks!
>
>
>
>--
>View this message in context:
>http://zookeeper-user.578899.n2.nabble.com/zookeeper-client-session-write-
>read-consistency-tp7579330p7582086.html
>Sent from the zookeeper-user mailing list archive at Nabble.com.
>

Re: zookeeper client session write-read consistency

[Alexander Shraer <sh...@gmail.com>]

I'm not sure, maybe someone else can clarify.

I think that ZOOKEEPER-22 (which is still open) leaves the client
in a state where it does not know whether its write completed or not, even
when it reconnects.
My guess is that the client zxid counter is not updated in this case
(connection drops while write is processed),
so it may reconnect to a server that didn't see the write, even if the
write has in fact completed.
I'm not sure how/whether curator is handling this.

On Mon, Mar 7, 2016 at 7:57 AM, Vitalii Tymchyshyn <vi...@tym.im> wrote:

> How about a reconnect to another server between the write and the next
> read?
> Curious about both cases maintaining the session and establishing a new one
> with some higher level retrier like Curator.
>
> Пн, 7 бер. 2016 07:50 користувач Alexander Shraer <sh...@gmail.com>
> пише:
>
> > The server to which the client is connected will buffer the read until
> the
> > write is executed and applied to its state, so the read will necessarily
> > return a value at least as recent as the one written by the write in your
> > example. ZK guarantees that async operations are executed in order of
> > invocation.
> > On Mar 6, 2016 23:57, "wayne" <fl...@gmail.com> wrote:
> >
> > Thanks Chris! I appreciate the answer a lot!
> >
> > What you said made perfect sense in the case that request are sent
> > synchronously (which was my assumption :)). What if the requests are sent
> > asynchronously? e.g. If I call AsyncWrite, AsyncRead within a session,
> when
> > the AsyncRead is executed, the previous AsyncWrite's result might not
> have
> > been returned to the client yet, then there is no way for the client to
> > know
> > the previous AsyncWrite's zxid, correct? In that case, could the
> situation
> > I
> > mentioned in my previous post happen?
> >
> >
> >
> > --
> > View this message in context:
> >
> >
> http://zookeeper-user.578899.n2.nabble.com/zookeeper-client-session-write-read-consistency-tp7579330p7582099.html
> > Sent from the zookeeper-user mailing list archive at Nabble.com.
> >
>

Re: zookeeper client session write-read consistency

[Vitalii Tymchyshyn <vi...@tym.im>]

How about a reconnect to another server between the write and the next read?
Curious about both cases maintaining the session and establishing a new one
with some higher level retrier like Curator.

Пн, 7 бер. 2016 07:50 користувач Alexander Shraer <sh...@gmail.com> пише:

> The server to which the client is connected will buffer the read until the
> write is executed and applied to its state, so the read will necessarily
> return a value at least as recent as the one written by the write in your
> example. ZK guarantees that async operations are executed in order of
> invocation.
> On Mar 6, 2016 23:57, "wayne" <fl...@gmail.com> wrote:
>
> Thanks Chris! I appreciate the answer a lot!
>
> What you said made perfect sense in the case that request are sent
> synchronously (which was my assumption :)). What if the requests are sent
> asynchronously? e.g. If I call AsyncWrite, AsyncRead within a session, when
> the AsyncRead is executed, the previous AsyncWrite's result might not have
> been returned to the client yet, then there is no way for the client to
> know
> the previous AsyncWrite's zxid, correct? In that case, could the situation
> I
> mentioned in my previous post happen?
>
>
>
> --
> View this message in context:
>
> http://zookeeper-user.578899.n2.nabble.com/zookeeper-client-session-write-read-consistency-tp7579330p7582099.html
> Sent from the zookeeper-user mailing list archive at Nabble.com.
>

Re: zookeeper client session write-read consistency

[Alexander Shraer <sh...@gmail.com>]

The server to which the client is connected will buffer the read until the
write is executed and applied to its state, so the read will necessarily
return a value at least as recent as the one written by the write in your
example. ZK guarantees that async operations are executed in order of
invocation.
On Mar 6, 2016 23:57, "wayne" <fl...@gmail.com> wrote:

Thanks Chris! I appreciate the answer a lot!

What you said made perfect sense in the case that request are sent
synchronously (which was my assumption :)). What if the requests are sent
asynchronously? e.g. If I call AsyncWrite, AsyncRead within a session, when
the AsyncRead is executed, the previous AsyncWrite's result might not have
been returned to the client yet, then there is no way for the client to know
the previous AsyncWrite's zxid, correct? In that case, could the situation I
mentioned in my previous post happen?



--
View this message in context:
http://zookeeper-user.578899.n2.nabble.com/zookeeper-client-session-write-read-consistency-tp7579330p7582099.html
Sent from the zookeeper-user mailing list archive at Nabble.com.

Re: zookeeper client session write-read consistency

[wayne <fl...@gmail.com>]

Thanks Chris! I appreciate the answer a lot!

What you said made perfect sense in the case that request are sent
synchronously (which was my assumption :)). What if the requests are sent
asynchronously? e.g. If I call AsyncWrite, AsyncRead within a session, when
the AsyncRead is executed, the previous AsyncWrite's result might not have
been returned to the client yet, then there is no way for the client to know
the previous AsyncWrite's zxid, correct? In that case, could the situation I
mentioned in my previous post happen?



--
View this message in context: http://zookeeper-user.578899.n2.nabble.com/zookeeper-client-session-write-read-consistency-tp7579330p7582099.html
Sent from the zookeeper-user mailing list archive at Nabble.com.