[jira] [Commented] (KAFKA-265) Add a queue of zookeeper notifications in the zookeeper consumer to reduce the number of rebalancing attempts

["Neha Narkhede (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204981#comment-13204981 ] 

Neha Narkhede commented on KAFKA-265:
-------------------------------------

If the queue is full, the ZKclient listener can hang temporarily. This is not ideal, since ZKClient will not be able to deliver more events until a rebalance operation is completed and the queue is cleared. In practice, this might not be a big issue, but can be easily avoided.

I think there is an alternative solution to this problem, one that will 

1. avoid maintaining this queue 
2. reduce memory consumption in the consumer
3. avoid adding another config option 

How about using just using a boolean variable that will indicate at least one rebalancing operation request ? The watcher thread can use a Condition to wait if the boolean variable is false. The ZK listener can merely set the boolean to true and signal the Condition, so that the watcher thread can proceed with a rebalancing operation.
                
> Add a queue of zookeeper notifications in the zookeeper consumer to reduce the number of rebalancing attempts
> -------------------------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-265
>                 URL: https://issues.apache.org/jira/browse/KAFKA-265
>             Project: Kafka
>          Issue Type: Sub-task
>          Components: core
>    Affects Versions: 0.7
>            Reporter: Neha Narkhede
>            Assignee: Jun Rao
>             Fix For: 0.7.1
>
>         Attachments: kafka-265.patch
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> The correct fix for KAFKA-262 and other known issues with the current consumer rebalancing approach, is to get rid of the cache in the zookeeper consumer. 
> The side-effect of that fix, though, is the large number of zookeeper notifications that will trigger a full rebalance operation on the consumer. 
> Ideally, the zookeeper notifications can be batched and only one rebalance operation can be triggered for several such ZK notifications. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira