[jira] [Commented] (KAFKA-1155) Kafka server can miss zookeeper watches during long zkclient callbacks

["Guozhang Wang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879274#comment-13879274 ] 

Guozhang Wang commented on KAFKA-1155:
--------------------------------------

Following the timestamp approach, the kafka controller needs to store the registry timestamp for each broker in its in-memory cache, and whenever the watcher fires, it needs to read each broker znode to compare and update its timestamps. If the timestamps are different, treat this broker as onBrokerFailure followed by onBrokerStartup.

This would need a ZK read on each broker path on each watcher fires. To optimize this, we can use the

public List<String> getChildren(String path, Watcher watcher, Stat stat) throws KeeperException, InterruptedException

of Zookeeper, in which the returned stat has the metadata such as version id which will be updated when the children list has changed. We can check on this value on watcher firing, and only reads broker path for timestamp when the stat metadata has changed. This requires to change the zkclient implementation though.

> Kafka server can miss zookeeper watches during long zkclient callbacks
> ----------------------------------------------------------------------
>
>                 Key: KAFKA-1155
>                 URL: https://issues.apache.org/jira/browse/KAFKA-1155
>             Project: Kafka
>          Issue Type: Bug
>          Components: controller
>    Affects Versions: 0.8.0, 0.8.1
>            Reporter: Neha Narkhede
>            Assignee: Neha Narkhede
>            Priority: Critical
>
> On getting a zookeeper watch, zkclient invokes the blocking user callback and only re-registers the watch after the callback returns. This leaves a possibly large window of time when Kafka has not registered for watches on the desired zookeeper paths and hence can miss important state changes (on the controller). In any case, it is worth noting that even though zookeeper has a read-and-set-watch API, there can always be a window of time between the watch being fired, the callback and the read-and-set-watch API call. Due to the zkclient wrapper, it is difficult to handle this properly in the Kafka code unless we directly use the zookeeper client. One way of getting around this issue is to use timestamps on the paths and when a watch fires, check if the timestamp in zk is different from the one in the callback handler.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)