[jira] [Comment Edited] (KAFKA-388) Add a highly available consumer co-ordinator to a Kafka cluster

["Guozhang Wang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13409695#comment-13409695 ] 

Guozhang Wang edited comment on KAFKA-388 at 7/9/12 6:12 PM:
-------------------------------------------------------------

Patch_v1 attached.

The coordinator keeps the following static information in cache:

interestGroup: Map[String, Set[String]] // For each topic, the set of groups that are interested in it, hence will be involved
// in rebalancing once the partitions for that topic has changed

Then on startup, it does the following:

1. Set watchers for topic-broker changes (ChildChanges) on /brokers/topics/[topic], and initialize the interestGroup for each topic

2. Read the consumers' intended interests for topicCounts within each topic, and as long as one consumer are interested in some topic A, put the group as A's interested group

3. Set watchers for group member changes (ChildChanges) on /consumers/[group]/ids for each group

4. Set watchers for group changes (ChildChanges) on /consumers (Do we really care if a whole group is gone?)

5. Set StateChange watcher for itself for trying to re-elect as the coordinator

In addition, it keeps a underRebalance AtomicBoolean for each group indicating if the group is under rebalance. This is used to batch multiple rebalance request for the same group


                
      was (Author: guozhang):
    The coordinator keeps the following static information in cache:

interestGroup: Map[String, Set[String]] // For each topic, the set of groups that are interested in it, hence will be involved
// in rebalancing once the partitions for that topic has changed

Then on startup, it does the following:

1. Set watchers for topic-broker changes (ChildChanges) on /brokers/topics/[topic], and initialize the interestGroup for each topic

2. Read the consumers' intended interests for topicCounts within each topic, and as long as one consumer are interested in some topic A, put the group as A's interested group

3. Set watchers for group member changes (ChildChanges) on /consumers/[group]/ids for each group

4. Set watchers for group changes (ChildChanges) on /consumers (Do we really care if a whole group is gone?)

5. Set StateChange watcher for itself for trying to re-elect as the coordinator

In addition, it keeps a underRebalance AtomicBoolean for each group indicating if the group is under rebalance. This is used to batch multiple rebalance request for the same group


                  
> Add a highly available consumer co-ordinator to a Kafka cluster
> ---------------------------------------------------------------
>
>                 Key: KAFKA-388
>                 URL: https://issues.apache.org/jira/browse/KAFKA-388
>             Project: Kafka
>          Issue Type: Sub-task
>    Affects Versions: 0.7
>            Reporter: Neha Narkhede
>            Assignee: Neha Narkhede
>              Labels: patch
>         Attachments: kafka-388_v1.patch
>
>
> This JIRA will add a highly available co-ordinator for consumer rebalancing. Detailed design of the co-ordinator leader election and startup procedure is documented here - https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Detailed+Consumer+Coordinator+Design

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira