ZK Client won't time out when quorum irrevocably goes away

[Ryan Rawson <ry...@gmail.com>]

Hey all,

We had an issue here at SU, where we moved a ZK cluster from under a
series of running clients (HBase master & regionservers)... but the
clients never handled the situation well, and we had to do a bunch of
process kills.

One log had lines like this:

2011-02-03 04:25:21,992 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server zookeeper/10.10.21.10:2181
2011-02-03 04:25:33,992 INFO org.apache.zookeeper.ClientCnxn: Client
session timed out, have not heard from server in 13423ms for sessionid
0x42d4ff0a1034fe3, closing socket connection and attempting reconnect
2011-02-03 04:25:34,170 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server zookeeper/10.10.21.12:2181
2011-02-03 04:25:46,168 INFO org.apache.zookeeper.ClientCnxn: Client
session timed out, have not heard from server in 12075ms for sessionid
0x42d4ff0a1034fe3, closing socket connection and attempting reconnect
2011-02-03 04:25:47,058 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server zookeeper/10.10.21.11:2181
2011-02-03 04:25:59,056 INFO org.apache.zookeeper.ClientCnxn: Client
session timed out, have not heard from server in 12787ms for sessionid
0x42d4ff0a1034fe3, closing socket connection and attempting reconnect

The problem was we _moved_ the machines, and renumbered the IPs,
therefore the OLD ip was no longer pingable, and apparently from this
log message the socket connect just timed out. It might be that
timeouts are handled differently than connection refused, I haven't
done that digging yet.

The result was the client never realized that it's session was
actually timed out, and the HBase processes continued to run. Kill -9
and a restart fixed it.

Thoughts?
-ryan

Re: ZK Client won't time out when quorum irrevocably goes away

[Patrick Hunt <ph...@apache.org>]

Hi Ted, I just sent a followup before seeing this which says similar:

>>I supposed that the client could close it's session if it sees that
>>the disconnect happened long enough ago (the session timeout + some
>>safety factor). But this really is a special case (and 338 should be
>>implemented to address).

Sounds reasonable to me. Enter a jira.

Patrick

On Thu, Feb 3, 2011 at 4:08 PM, Ted Dunning <te...@gmail.com> wrote:
> Patrick,
>
> Is it really impossible for the client to say that soooo much time has
> passed in disconnected state that the session MUST have expired by now?
>
> I have heard this assertion before and it always irked me a bit, but Ryan's
> scenario is a great thought experiment (well, though experiment for US, not
> for him).  Why can't those clients decide the session is expired after 3
> days when the timeout is 3 minutes?
>
> On Thu, Feb 3, 2011 at 4:01 PM, Patrick Hunt <ph...@apache.org> wrote:
>
>> On Thu, Feb 3, 2011 at 2:57 PM, Ryan Rawson <ry...@gmail.com> wrote:
>> > The result was the client never realized that it's session was
>> > actually timed out, and the HBase processes continued to run. Kill -9
>> > and a restart fixed it.
>>
>> Hi Ryan,
>>
>> there are two issues at play here, session timeout and session
>> expiration. Correct me if I'm wrong but I think you meant to say "the
>> client never realized that it's session was actually _expired_". Which
>> is correct behavior. Clients can only determine if a session is
>> expired once they reconnect to the cluster. Session timeout on the
>> other hand happens when the server heartbeat is not received by the
>> client w/in the session timeout period. Clients who are disconnected
>> from the cluster will attempt to reconnect back to the cluster until
>> they are successful. When a client is disconnected the client's
>> watchers will be notified about the disconnect. (same for expiration).
>>
>> See questions 1 & 2 here in the faq, specifically "Example state
>> transitions" in question 2:
>> https://cwiki.apache.org/confluence/display/ZOOKEEPER/FAQ
>> Your clients were stuck btw steps 4 and 5 (which they will never reach
>> in your scenario).
>>
>> Does that help?
>>
>> Patrick
>>
>

Re: ZK Client won't time out when quorum irrevocably goes away

[Ryan Rawson <ry...@gmail.com>]

In our case the zk client had been silently (well lots of log messages)
failing for 7 hours our so. So not exactly academic, but an unusual
situation to be sure.
On Feb 3, 2011 4:08 PM, "Ted Dunning" <te...@gmail.com> wrote:
> Patrick,
>
> Is it really impossible for the client to say that soooo much time has
> passed in disconnected state that the session MUST have expired by now?
>
> I have heard this assertion before and it always irked me a bit, but
Ryan's
> scenario is a great thought experiment (well, though experiment for US,
not
> for him). Why can't those clients decide the session is expired after 3
> days when the timeout is 3 minutes?
>
> On Thu, Feb 3, 2011 at 4:01 PM, Patrick Hunt <ph...@apache.org> wrote:
>
>> On Thu, Feb 3, 2011 at 2:57 PM, Ryan Rawson <ry...@gmail.com> wrote:
>> > The result was the client never realized that it's session was
>> > actually timed out, and the HBase processes continued to run. Kill -9
>> > and a restart fixed it.
>>
>> Hi Ryan,
>>
>> there are two issues at play here, session timeout and session
>> expiration. Correct me if I'm wrong but I think you meant to say "the
>> client never realized that it's session was actually _expired_". Which
>> is correct behavior. Clients can only determine if a session is
>> expired once they reconnect to the cluster. Session timeout on the
>> other hand happens when the server heartbeat is not received by the
>> client w/in the session timeout period. Clients who are disconnected
>> from the cluster will attempt to reconnect back to the cluster until
>> they are successful. When a client is disconnected the client's
>> watchers will be notified about the disconnect. (same for expiration).
>>
>> See questions 1 & 2 here in the faq, specifically "Example state
>> transitions" in question 2:
>> https://cwiki.apache.org/confluence/display/ZOOKEEPER/FAQ
>> Your clients were stuck btw steps 4 and 5 (which they will never reach
>> in your scenario).
>>
>> Does that help?
>>
>> Patrick
>>

Re: ZK Client won't time out when quorum irrevocably goes away

[Ted Dunning <te...@gmail.com>]

Patrick,

Is it really impossible for the client to say that soooo much time has
passed in disconnected state that the session MUST have expired by now?

I have heard this assertion before and it always irked me a bit, but Ryan's
scenario is a great thought experiment (well, though experiment for US, not
for him).  Why can't those clients decide the session is expired after 3
days when the timeout is 3 minutes?

On Thu, Feb 3, 2011 at 4:01 PM, Patrick Hunt <ph...@apache.org> wrote:

> On Thu, Feb 3, 2011 at 2:57 PM, Ryan Rawson <ry...@gmail.com> wrote:
> > The result was the client never realized that it's session was
> > actually timed out, and the HBase processes continued to run. Kill -9
> > and a restart fixed it.
>
> Hi Ryan,
>
> there are two issues at play here, session timeout and session
> expiration. Correct me if I'm wrong but I think you meant to say "the
> client never realized that it's session was actually _expired_". Which
> is correct behavior. Clients can only determine if a session is
> expired once they reconnect to the cluster. Session timeout on the
> other hand happens when the server heartbeat is not received by the
> client w/in the session timeout period. Clients who are disconnected
> from the cluster will attempt to reconnect back to the cluster until
> they are successful. When a client is disconnected the client's
> watchers will be notified about the disconnect. (same for expiration).
>
> See questions 1 & 2 here in the faq, specifically "Example state
> transitions" in question 2:
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/FAQ
> Your clients were stuck btw steps 4 and 5 (which they will never reach
> in your scenario).
>
> Does that help?
>
> Patrick
>

Re: ZK Client won't time out when quorum irrevocably goes away

[Ted Dunning <te...@gmail.com>]

Ryan... just as an aside, you can move a ZK cluster a step at a time by
adding new nodes and then decommissioning old nodes.
Your clients may get confused if they try to reconnect and never really
learned about the whole quorum, but you can do quite a lot with rolling
restarts as far as getting the ZK state itself walked over to new machines.

On Thu, Feb 3, 2011 at 4:06 PM, Ryan Rawson <ry...@gmail.com> wrote:

> Yes thanks, I'm a little loose with the language not realizing there
> are specific states, etc.
>
> So in our scenario here, the quorum has moved, the clients will never
> (a) reconnect ever again and (b) not be able to find the new quorum
> location because IP addresses are cached.  If either:
> - The client refreshed from DNS (although the JVM seems to have a DNS
> cache which has hosed us as well)
> - The client expires the session
>
> We might have been in a better situation.
>
> Reading the FAQ, it seems like the onus might be on the client to
> check for session disconnect and compare it against the negotiated
> session timeout to determine "oh hey we havent talked to ZK in a
> while, lets quit".  Is that an expected client task?
>
> Thanks for the quick reply!
> -ryan
>
> On Thu, Feb 3, 2011 at 4:01 PM, Patrick Hunt <ph...@apache.org> wrote:
> > On Thu, Feb 3, 2011 at 2:57 PM, Ryan Rawson <ry...@gmail.com> wrote:
> >> The result was the client never realized that it's session was
> >> actually timed out, and the HBase processes continued to run. Kill -9
> >> and a restart fixed it.
> >
> > Hi Ryan,
> >
> > there are two issues at play here, session timeout and session
> > expiration. Correct me if I'm wrong but I think you meant to say "the
> > client never realized that it's session was actually _expired_". Which
> > is correct behavior. Clients can only determine if a session is
> > expired once they reconnect to the cluster. Session timeout on the
> > other hand happens when the server heartbeat is not received by the
> > client w/in the session timeout period. Clients who are disconnected
> > from the cluster will attempt to reconnect back to the cluster until
> > they are successful. When a client is disconnected the client's
> > watchers will be notified about the disconnect. (same for expiration).
> >
> > See questions 1 & 2 here in the faq, specifically "Example state
> > transitions" in question 2:
> > https://cwiki.apache.org/confluence/display/ZOOKEEPER/FAQ
> > Your clients were stuck btw steps 4 and 5 (which they will never reach
> > in your scenario).
> >
> > Does that help?
> >
> > Patrick
> >
>

Re: ZK Client won't time out when quorum irrevocably goes away

[Patrick Hunt <ph...@apache.org>]

On Thu, Feb 3, 2011 at 4:06 PM, Ryan Rawson <ry...@gmail.com> wrote:
> Yes thanks, I'm a little loose with the language not realizing there
> are specific states, etc.
>

No worries, I just wanted to make sure I understood the situation correctly.

> So in our scenario here, the quorum has moved, the clients will never
> (a) reconnect ever again and (b) not be able to find the new quorum
> location because IP addresses are cached.  If either:

That's correct. You've hit this issue:
https://issues.apache.org/jira/browse/ZOOKEEPER-338

> - The client refreshed from DNS (although the JVM seems to have a DNS
> cache which has hosed us as well)
> - The client expires the session
>
> We might have been in a better situation.

Perhaps. Did you re-create the ZK database or did you re-use the
existing datastore? (ie moved them from the old to the new server)

>
> Reading the FAQ, it seems like the onus might be on the client to
> check for session disconnect and compare it against the negotiated
> session timeout to determine "oh hey we havent talked to ZK in a
> while, lets quit".  Is that an expected client task?

No, the expectation is that the client will eventually reconnect and
see the session as expired. That's the typical case with network
partitioning or the cluster being brought down, etc...

This is a special case due to ZOOKEEPER-338. For this scenario the
only "official" recourse at this time is to restart the client. (but
depending on your answer to my previous question you might need to do
that anyway)

I supposed that the client could close it's session if it sees that
the disconnect happened long enough ago (the session timeout + some
safety factor). But this really is a special case (and 338 should be
implemented to address).

Patrick

>
> Thanks for the quick reply!
> -ryan
>
> On Thu, Feb 3, 2011 at 4:01 PM, Patrick Hunt <ph...@apache.org> wrote:
>> On Thu, Feb 3, 2011 at 2:57 PM, Ryan Rawson <ry...@gmail.com> wrote:
>>> The result was the client never realized that it's session was
>>> actually timed out, and the HBase processes continued to run. Kill -9
>>> and a restart fixed it.
>>
>> Hi Ryan,
>>
>> there are two issues at play here, session timeout and session
>> expiration. Correct me if I'm wrong but I think you meant to say "the
>> client never realized that it's session was actually _expired_". Which
>> is correct behavior. Clients can only determine if a session is
>> expired once they reconnect to the cluster. Session timeout on the
>> other hand happens when the server heartbeat is not received by the
>> client w/in the session timeout period. Clients who are disconnected
>> from the cluster will attempt to reconnect back to the cluster until
>> they are successful. When a client is disconnected the client's
>> watchers will be notified about the disconnect. (same for expiration).
>>
>> See questions 1 & 2 here in the faq, specifically "Example state
>> transitions" in question 2:
>> https://cwiki.apache.org/confluence/display/ZOOKEEPER/FAQ
>> Your clients were stuck btw steps 4 and 5 (which they will never reach
>> in your scenario).
>>
>> Does that help?
>>
>> Patrick
>>
>

Re: ZK Client won't time out when quorum irrevocably goes away

[Ryan Rawson <ry...@gmail.com>]

Yes thanks, I'm a little loose with the language not realizing there
are specific states, etc.

So in our scenario here, the quorum has moved, the clients will never
(a) reconnect ever again and (b) not be able to find the new quorum
location because IP addresses are cached.  If either:
- The client refreshed from DNS (although the JVM seems to have a DNS
cache which has hosed us as well)
- The client expires the session

We might have been in a better situation.

Reading the FAQ, it seems like the onus might be on the client to
check for session disconnect and compare it against the negotiated
session timeout to determine "oh hey we havent talked to ZK in a
while, lets quit".  Is that an expected client task?

Thanks for the quick reply!
-ryan

On Thu, Feb 3, 2011 at 4:01 PM, Patrick Hunt <ph...@apache.org> wrote:
> On Thu, Feb 3, 2011 at 2:57 PM, Ryan Rawson <ry...@gmail.com> wrote:
>> The result was the client never realized that it's session was
>> actually timed out, and the HBase processes continued to run. Kill -9
>> and a restart fixed it.
>
> Hi Ryan,
>
> there are two issues at play here, session timeout and session
> expiration. Correct me if I'm wrong but I think you meant to say "the
> client never realized that it's session was actually _expired_". Which
> is correct behavior. Clients can only determine if a session is
> expired once they reconnect to the cluster. Session timeout on the
> other hand happens when the server heartbeat is not received by the
> client w/in the session timeout period. Clients who are disconnected
> from the cluster will attempt to reconnect back to the cluster until
> they are successful. When a client is disconnected the client's
> watchers will be notified about the disconnect. (same for expiration).
>
> See questions 1 & 2 here in the faq, specifically "Example state
> transitions" in question 2:
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/FAQ
> Your clients were stuck btw steps 4 and 5 (which they will never reach
> in your scenario).
>
> Does that help?
>
> Patrick
>

Re: ZK Client won't time out when quorum irrevocably goes away

[Patrick Hunt <ph...@apache.org>]

On Thu, Feb 3, 2011 at 2:57 PM, Ryan Rawson <ry...@gmail.com> wrote:
> The result was the client never realized that it's session was
> actually timed out, and the HBase processes continued to run. Kill -9
> and a restart fixed it.

Hi Ryan,

there are two issues at play here, session timeout and session
expiration. Correct me if I'm wrong but I think you meant to say "the
client never realized that it's session was actually _expired_". Which
is correct behavior. Clients can only determine if a session is
expired once they reconnect to the cluster. Session timeout on the
other hand happens when the server heartbeat is not received by the
client w/in the session timeout period. Clients who are disconnected
from the cluster will attempt to reconnect back to the cluster until
they are successful. When a client is disconnected the client's
watchers will be notified about the disconnect. (same for expiration).

See questions 1 & 2 here in the faq, specifically "Example state
transitions" in question 2:
https://cwiki.apache.org/confluence/display/ZOOKEEPER/FAQ
Your clients were stuck btw steps 4 and 5 (which they will never reach
in your scenario).

Does that help?

Patrick