You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by Neha Narkhede <ne...@gmail.com> on 2013/08/01 03:41:46 UTC
Zookeeper 3.3.4 guarantees across sessions from the same client ?
The behavior we saw on one of our zookeeper clients is as follows. The
session expires on the client, it assumes the ephemeral nodes are deleted,
so it establishes a new session with zookeeper and tries to re-create the
ephemeral nodes. However, when it tries to re-create the ephemeral node,
zookeeper throws back a NodeExists error code. Now this is legitimate
during a session disconnect event (since zkclient automatically retries the
operation and raises a NodeExists error). Also by design, Kafka doesn't
have multiple clients create the same ephemeral node, so Kafka server
assumes the NodeExists is normal. However, after a few seconds zookeeper
deletes that ephemeral node. So from the client's perspective, even though
the client has a new valid session, its ephemeral node is gone.
After poking at the transaction and log4j logs, I saw that the NodeExists
was because the zookeeper leader had retained the ephemeral node from the
previous expired session. It turns out that it notified the client of the
session expiration before actually deleting the ephemeral node. It is also
worth noting that the previous session was expired due to a long fsync
operation on the zookeeper leader. After it returned from the fsync, it had
a whole bunch of sessions to expire.
In this case, it seems that zookeeper should not notify the client that the
session is expired until the ephemeral node information is actually gone.
Or maybe I'm not clear on what the guarantees from zookeeper are, across
sessions from the same client.
Thanks,
Neha
Re: Zookeeper 3.3.4 guarantees across sessions from the same client ?
Posted by Neha Narkhede <ne...@gmail.com>.
Hi Flavio,
Thanks a bunch for checking on that and confirming the behavior.
I looked at the code in PrepRequestProcessor where zookeeper throws
NodeExists. It only checks that the ephemeral node exists, but doesn't
check if the ephemeral owner session is marked as "expired". I was
wondering if it is a feasible solution to detect that the owner is a
session that is getting expired, so instead of returning NodeExists, let
the write go through and just replace the ephemeral owner with the new
session id. However, this may have other side-effects that I'm not aware
of. Curious to know what you think of this fix.
In any case, we can handle it in Kafka by writing the timestamp as part of
the ephemeral path. On NodeExists, we can read the path back and retry the
operation if the timestamp doesn't match.
Thanks,
Neha
On Thu, Aug 1, 2013 at 2:07 AM, FPJ <fp...@yahoo.com> wrote:
> Hi Neha,
>
> Unless we expire a session and delete ephemerals atomically, there are only
> two options I see:
>
> 1- Delete right before expiring the session
> 2- Delete right after expiring the session
>
> Because of timing, we can have the following. With the first, a client
> might
> observe the delete before the session actually expires, which violates our
> contract. With the second, you may observe an ephemeral znode after the
> session has expired as you have. I would say that the second option is
> correct as long as the ephemerals are eventually deleted, but it does have
> the side-effect you're mentioning.
>
> -Flavio
>
> > -----Original Message-----
> > From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> > Sent: 01 August 2013 02:42
> > To: dev@zookeeper.apache.org
> > Subject: Zookeeper 3.3.4 guarantees across sessions from the same client
> ?
> >
> > The behavior we saw on one of our zookeeper clients is as follows. The
> > session expires on the client, it assumes the ephemeral nodes are
> deleted,
> > so it establishes a new session with zookeeper and tries to re-create the
> > ephemeral nodes. However, when it tries to re-create the ephemeral node,
> > zookeeper throws back a NodeExists error code. Now this is legitimate
> during
> > a session disconnect event (since zkclient automatically retries the
> operation
> > and raises a NodeExists error). Also by design, Kafka doesn't have
> multiple
> > clients create the same ephemeral node, so Kafka server assumes the
> > NodeExists is normal. However, after a few seconds zookeeper deletes that
> > ephemeral node. So from the client's perspective, even though the client
> has
> > a new valid session, its ephemeral node is gone.
> >
> > After poking at the transaction and log4j logs, I saw that the NodeExists
> was
> > because the zookeeper leader had retained the ephemeral node from the
> > previous expired session. It turns out that it notified the client of the
> session
> > expiration before actually deleting the ephemeral node. It is also worth
> > noting that the previous session was expired due to a long fsync
> operation
> > on the zookeeper leader. After it returned from the fsync, it had a whole
> > bunch of sessions to expire.
> >
> > In this case, it seems that zookeeper should not notify the client that
> the
> > session is expired until the ephemeral node information is actually gone.
> > Or maybe I'm not clear on what the guarantees from zookeeper are, across
> > sessions from the same client.
> >
> > Thanks,
> >
> > Neha
>
>
RE: Zookeeper 3.3.4 guarantees across sessions from the same client ?
Posted by FPJ <fp...@yahoo.com>.
Hi Neha,
Unless we expire a session and delete ephemerals atomically, there are only
two options I see:
1- Delete right before expiring the session
2- Delete right after expiring the session
Because of timing, we can have the following. With the first, a client might
observe the delete before the session actually expires, which violates our
contract. With the second, you may observe an ephemeral znode after the
session has expired as you have. I would say that the second option is
correct as long as the ephemerals are eventually deleted, but it does have
the side-effect you're mentioning.
-Flavio
> -----Original Message-----
> From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> Sent: 01 August 2013 02:42
> To: dev@zookeeper.apache.org
> Subject: Zookeeper 3.3.4 guarantees across sessions from the same client ?
>
> The behavior we saw on one of our zookeeper clients is as follows. The
> session expires on the client, it assumes the ephemeral nodes are deleted,
> so it establishes a new session with zookeeper and tries to re-create the
> ephemeral nodes. However, when it tries to re-create the ephemeral node,
> zookeeper throws back a NodeExists error code. Now this is legitimate
during
> a session disconnect event (since zkclient automatically retries the
operation
> and raises a NodeExists error). Also by design, Kafka doesn't have
multiple
> clients create the same ephemeral node, so Kafka server assumes the
> NodeExists is normal. However, after a few seconds zookeeper deletes that
> ephemeral node. So from the client's perspective, even though the client
has
> a new valid session, its ephemeral node is gone.
>
> After poking at the transaction and log4j logs, I saw that the NodeExists
was
> because the zookeeper leader had retained the ephemeral node from the
> previous expired session. It turns out that it notified the client of the
session
> expiration before actually deleting the ephemeral node. It is also worth
> noting that the previous session was expired due to a long fsync
operation
> on the zookeeper leader. After it returned from the fsync, it had a whole
> bunch of sessions to expire.
>
> In this case, it seems that zookeeper should not notify the client that
the
> session is expired until the ephemeral node information is actually gone.
> Or maybe I'm not clear on what the guarantees from zookeeper are, across
> sessions from the same client.
>
> Thanks,
>
> Neha
Re: Zookeeper 3.3.4 guarantees across sessions from the same client ?
Posted by Neha Narkhede <ne...@gmail.com>.
I haven't had the chance to reproduce this on zookeeper 3.4, but I was able
to reproduce this on 3.3.4. Filed
https://issues.apache.org/jira/browse/ZOOKEEPER-1740. I will try to see if
the fix works.
Thanks,
Neha
Re: Zookeeper 3.3.4 guarantees across sessions from the same client ?
Posted by Flavio Junqueira <fp...@yahoo.com>.
I was actually thinking that if the create comes after the deletes, it should already observe that the znode has been deleted, so there is no point in adding the check. I'm also wondering if the scenario you have observed has been resolved in the 3.4 branch. If you can reproduce it, then it would be good to check.
-Flavio
On Aug 3, 2013, at 1:22 AM, Flavio Junqueira <fp...@yahoo.com> wrote:
> Hi Neha,
>
> Your proposal seems fine to me. When we close a session in PrepRequestProcessor, we line up all ephemerals of the session to be deleted, so as long as the create comes after those deletes, I don't see a problem with checking if the session is closing. Do you want to create a jira so that we can work on it and perhaps other can comment as well?
>
> -Flavio
>
> On Aug 1, 2013, at 7:02 PM, wrote:
>
>> Hi Neha,
>>
>> I'm assuming that the Unless we expire a session and delete ephemerals
>> atomically, there are only two options I see:
>>
>> 1- Delete right before expiring the session
>> 2- Delete right after expiring the session
>>
>> Because of timing, we can have the following. With the first, a client
>> might observe the delete before the session actually expires, which
>> violates our contract. With the second, you may observe an ephemeral znode
>> after the session has expired as you have. I would say that the second
>> option is correct as long as the ephemerals are eventually deleted, but it
>> does have the side-effect you're mentioning.
>>
>> -Flavio
>>
>>> -----Original Message-----
>>> From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
>>> Sent: 01 August 2013 02:42
>>> To: dev@zookeeper.apache.org
>>> Subject: Zookeeper 3.3.4 guarantees across sessions from the same client
>> ?
>>>
>>> The behavior we saw on one of our zookeeper clients is as follows. The
>>> session expires on the client, it assumes the ephemeral nodes are
>> deleted,
>>> so it establishes a new session with zookeeper and tries to re-create
>> the
>>> ephemeral nodes. However, when it tries to re-create the ephemeral node,
>>> zookeeper throws back a NodeExists error code. Now this is legitimate
>> during
>>> a session disconnect event (since zkclient automatically retries the
>> operation
>>> and raises a NodeExists error). Also by design, Kafka doesn't have
>> multiple
>>> clients create the same ephemeral node, so Kafka server assumes the
>>> NodeExists is normal. However, after a few seconds zookeeper deletes
>> that
>>> ephemeral node. So from the client's perspective, even though the client
>> has
>>> a new valid session, its ephemeral node is gone.
>>>
>>> After poking at the transaction and log4j logs, I saw that the
>> NodeExists was
>>> because the zookeeper leader had retained the ephemeral node from the
>>> previous expired session. It turns out that it notified the client of
>> the session
>>> expiration before actually deleting the ephemeral node. It is also worth
>>> noting that the previous session was expired due to a long fsync
>> operation
>>> on the zookeeper leader. After it returned from the fsync, it had a
>> whole
>>> bunch of sessions to expire.
>>>
>>> In this case, it seems that zookeeper should not notify the client that
>> the
>>> session is expired until the ephemeral node information is actually
>> gone.
>>> Or maybe I'm not clear on what the guarantees from zookeeper are, across
>>> sessions from the same client.
>>>
>>> Thanks,
>>>
>>> Neha
>
Re: Zookeeper 3.3.4 guarantees across sessions from the same client ?
Posted by Flavio Junqueira <fp...@yahoo.com>.
Hi Neha,
Your proposal seems fine to me. When we close a session in PrepRequestProcessor, we line up all ephemerals of the session to be deleted, so as long as the create comes after those deletes, I don't see a problem with checking if the session is closing. Do you want to create a jira so that we can work on it and perhaps other can comment as well?
-Flavio
On Aug 1, 2013, at 7:02 PM, wrote:
> Hi Neha,
>
> I'm assuming that the Unless we expire a session and delete ephemerals
> atomically, there are only two options I see:
>
> 1- Delete right before expiring the session
> 2- Delete right after expiring the session
>
> Because of timing, we can have the following. With the first, a client
> might observe the delete before the session actually expires, which
> violates our contract. With the second, you may observe an ephemeral znode
> after the session has expired as you have. I would say that the second
> option is correct as long as the ephemerals are eventually deleted, but it
> does have the side-effect you're mentioning.
>
> -Flavio
>
>> -----Original Message-----
>> From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
>> Sent: 01 August 2013 02:42
>> To: dev@zookeeper.apache.org
>> Subject: Zookeeper 3.3.4 guarantees across sessions from the same client
> ?
>>
>> The behavior we saw on one of our zookeeper clients is as follows. The
>> session expires on the client, it assumes the ephemeral nodes are
> deleted,
>> so it establishes a new session with zookeeper and tries to re-create
> the
>> ephemeral nodes. However, when it tries to re-create the ephemeral node,
>> zookeeper throws back a NodeExists error code. Now this is legitimate
> during
>> a session disconnect event (since zkclient automatically retries the
> operation
>> and raises a NodeExists error). Also by design, Kafka doesn't have
> multiple
>> clients create the same ephemeral node, so Kafka server assumes the
>> NodeExists is normal. However, after a few seconds zookeeper deletes
> that
>> ephemeral node. So from the client's perspective, even though the client
> has
>> a new valid session, its ephemeral node is gone.
>>
>> After poking at the transaction and log4j logs, I saw that the
> NodeExists was
>> because the zookeeper leader had retained the ephemeral node from the
>> previous expired session. It turns out that it notified the client of
> the session
>> expiration before actually deleting the ephemeral node. It is also worth
>> noting that the previous session was expired due to a long fsync
> operation
>> on the zookeeper leader. After it returned from the fsync, it had a
> whole
>> bunch of sessions to expire.
>>
>> In this case, it seems that zookeeper should not notify the client that
> the
>> session is expired until the ephemeral node information is actually
> gone.
>> Or maybe I'm not clear on what the guarantees from zookeeper are, across
>> sessions from the same client.
>>
>> Thanks,
>>
>> Neha