[jira] [Commented] (CURATOR-644) CLONE - Race conditions in LeaderLatch after reconnecting to ensemble

["Ken Huang (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CURATOR-644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17558226#comment-17558226 ] 

Ken Huang commented on CURATOR-644:
-----------------------------------

We observed the same issue happening in curator version 4.2.0.

In one of our service host, within a day, we found that:
 # There were 14000 errors from org.apache.curator.framework.recipes.leader.LeaderLatch on “Can't find our node. Resetting. Index: -1”
 # There were over a million errors from org.apache.curator.framework.imps.CuratorFrameworkImpl on “Background operation retry gave up” with stack trace included org.apache.zookeeper.KeeperException.ConnectionLossException

We found this issue is very similar to the one reported 2 years ago. It was closed as the owner of the ticket said
```4.2.0 fixes a serious bug that might be the source of the problem (I'm not sure thought).```

But we are using curator 4.2.0 and this issue still occurs. Therefore cloning the previous issue and report here. Can someone please take a look? Thanks

> CLONE - Race conditions in LeaderLatch after reconnecting to ensemble
> ---------------------------------------------------------------------
>
>                 Key: CURATOR-644
>                 URL: https://issues.apache.org/jira/browse/CURATOR-644
>             Project: Apache Curator
>          Issue Type: Bug
>    Affects Versions: 4.1.0
>            Reporter: Ken Huang
>            Assignee: Jordan Zimmerman
>            Priority: Minor
>
> We use LeaderLatch in a lot of places in our system and when ZooKeeper ensemble is unstable and clients are reconnecting to logs are full of messages like the following:
> {{[2017-08-31 19:18:34,562][ERROR][org.apache.curator.framework.recipes.leader.LeaderLatch] Can't find our node. Resetting. Index: -1 {}}}
> According to the [implementation|https://github.com/apache/curator/blob/4251fe328908e5fca37af034fabc190aa452c73f/curator-recipes/src/main/java/org/apache/curator/framework/recipes/leader/LeaderLatch.java#L529-L536], this can happen in two cases:
>  * When internal state `ourPath` is null
>  * When the list of latches does not have the expected one.
> I believe we hit the first condition because of races that occur after client reconnects to ZooKeeper.
>  * Client reconnects to ZooKeeper and LeaderLatch gets the event and calls reset method which set the internal state (`ourPath`) to null, removes old latch and creates a new one. This happens in thread "Curator-ConnectionStateManager-0".
>  * Almost simultaneously, LeaderLatch gets another even NodeDeleted ([here|https://github.com/apache/curator/blob/4251fe328908e5fca37af034fabc190aa452c73f/curator-recipes/src/main/java/org/apache/curator/framework/recipes/leader/LeaderLatch.java#L543-L554]) and tries to re-read the list of latches and check leadership. This happens in the thread "main-EventThread".
> Therefore, sometimes there is a situation when method `checkLeadership` is called when `ourPath` is null.
> Below is an approximate diagram of what happens:
> !51868597-65791000-231c-11e9-9bfa-1def62bc3ea1.png|width=1261,height=150!



--
This message was sent by Atlassian Jira
(v8.20.7#820007)