You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Neha Narkhede (Created) (JIRA)" <ji...@apache.org> on 2012/02/06 22:54:59 UTC
[jira] [Created] (KAFKA-264) Change the consumer side load
balancing and distributed co-ordination to use a consumer co-ordinator
Change the consumer side load balancing and distributed co-ordination to use a consumer co-ordinator
----------------------------------------------------------------------------------------------------
Key: KAFKA-264
URL: https://issues.apache.org/jira/browse/KAFKA-264
Project: Kafka
Issue Type: New Feature
Components: core
Affects Versions: 0.7
Reporter: Neha Narkhede
Fix For: 0.7.1
A high level design for the zookeeper consumer is here - https://cwiki.apache.org/confluence/display/KAFKA/Consumer+co-ordinator
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-264) Change the consumer side load
balancing and distributed co-ordination to use a consumer co-ordinator
Posted by "Guozhang Wang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Guozhang Wang updated KAFKA-264:
--------------------------------
Attachment: KAFKA-264.v1.patch
Please refer to this wiki page for the detailed description of the implementation:
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Detailed+Consumer+Coordinator+Design#KafkaDetailedConsumerCoordinatorDesign-11.Implementation
> Change the consumer side load balancing and distributed co-ordination to use a consumer co-ordinator
> ----------------------------------------------------------------------------------------------------
>
> Key: KAFKA-264
> URL: https://issues.apache.org/jira/browse/KAFKA-264
> Project: Kafka
> Issue Type: New Feature
> Components: core
> Affects Versions: 0.7
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Fix For: 0.8
>
> Attachments: KAFKA-264.v1.patch
>
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> A high level design for the zookeeper consumer is here - https://cwiki.apache.org/confluence/display/KAFKA/Consumer+co-ordinator
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-264) Change the consumer side load
balancing and distributed co-ordination to use a consumer co-ordinator
Posted by "Neha Narkhede (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201697#comment-13201697 ]
Neha Narkhede commented on KAFKA-264:
-------------------------------------
If a process goes down, the ephemeral node for /consumers/[group]/leader will get deleted from Zookeeper. Each consumer process listens on that path, and triggers a leader election. However, these are very low level details of the implementation, which I intended to chalk out after the high level design looked good.
You can read more about Zookeeper based leader election here - http://zookeeper.apache.org/doc/trunk/recipes.html#sc_leaderElection
> Change the consumer side load balancing and distributed co-ordination to use a consumer co-ordinator
> ----------------------------------------------------------------------------------------------------
>
> Key: KAFKA-264
> URL: https://issues.apache.org/jira/browse/KAFKA-264
> Project: Kafka
> Issue Type: New Feature
> Components: core
> Affects Versions: 0.7
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Fix For: 0.7.1
>
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> A high level design for the zookeeper consumer is here - https://cwiki.apache.org/confluence/display/KAFKA/Consumer+co-ordinator
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (KAFKA-264) Change the consumer side
load balancing and distributed co-ordination to use a consumer co-ordinator
Posted by "Prashanth Menon (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201657#comment-13201657 ]
Prashanth Menon edited comment on KAFKA-264 at 2/6/12 10:04 PM:
----------------------------------------------------------------
Interesting stuff here. Just a question that logically follows from having a coordinator: what happens if the coordinating process goes down or becomes unavailable?
was (Author: prashanth.menon):
Interesting stuff here. Just a question that logically follows from having a coordinator: what happens if the coordinating process goes does?
> Change the consumer side load balancing and distributed co-ordination to use a consumer co-ordinator
> ----------------------------------------------------------------------------------------------------
>
> Key: KAFKA-264
> URL: https://issues.apache.org/jira/browse/KAFKA-264
> Project: Kafka
> Issue Type: New Feature
> Components: core
> Affects Versions: 0.7
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Fix For: 0.7.1
>
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> A high level design for the zookeeper consumer is here - https://cwiki.apache.org/confluence/display/KAFKA/Consumer+co-ordinator
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-264) Change the consumer side load
balancing and distributed co-ordination to use a consumer co-ordinator
Posted by "Neha Narkhede (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201672#comment-13201672 ]
Neha Narkhede commented on KAFKA-264:
-------------------------------------
The consumer co-ordinator itself is highly available, since it is elected from amongst the available consumer processes in a group. That can be done using the standard recipe for leader election using Zookeeper
> Change the consumer side load balancing and distributed co-ordination to use a consumer co-ordinator
> ----------------------------------------------------------------------------------------------------
>
> Key: KAFKA-264
> URL: https://issues.apache.org/jira/browse/KAFKA-264
> Project: Kafka
> Issue Type: New Feature
> Components: core
> Affects Versions: 0.7
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Fix For: 0.7.1
>
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> A high level design for the zookeeper consumer is here - https://cwiki.apache.org/confluence/display/KAFKA/Consumer+co-ordinator
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-264) Change the consumer side load
balancing and distributed co-ordination to use a consumer co-ordinator
Posted by "Prashanth Menon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201657#comment-13201657 ]
Prashanth Menon commented on KAFKA-264:
---------------------------------------
Interesting stuff here. Just a question that logically follows from having a coordinator: what happens if the coordinating process goes does?
> Change the consumer side load balancing and distributed co-ordination to use a consumer co-ordinator
> ----------------------------------------------------------------------------------------------------
>
> Key: KAFKA-264
> URL: https://issues.apache.org/jira/browse/KAFKA-264
> Project: Kafka
> Issue Type: New Feature
> Components: core
> Affects Versions: 0.7
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Fix For: 0.7.1
>
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> A high level design for the zookeeper consumer is here - https://cwiki.apache.org/confluence/display/KAFKA/Consumer+co-ordinator
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-264) Change the consumer side load
balancing and distributed co-ordination to use a consumer co-ordinator
Posted by "Prashanth Menon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201684#comment-13201684 ]
Prashanth Menon commented on KAFKA-264:
---------------------------------------
Hmm, forgive me if my questions are trivial :)
What I mean is, the election of the leader can be handled by ZK, but what happens if the thread/process/machine with the coordinator goes down? Does every consumer group listen on the consumer-leader path in ZK and issue a new round of coordinator election, ala how it'll be done for replication leaders?
> Change the consumer side load balancing and distributed co-ordination to use a consumer co-ordinator
> ----------------------------------------------------------------------------------------------------
>
> Key: KAFKA-264
> URL: https://issues.apache.org/jira/browse/KAFKA-264
> Project: Kafka
> Issue Type: New Feature
> Components: core
> Affects Versions: 0.7
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Fix For: 0.7.1
>
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> A high level design for the zookeeper consumer is here - https://cwiki.apache.org/confluence/display/KAFKA/Consumer+co-ordinator
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-264) Change the consumer side load
balancing and distributed co-ordination to use a consumer co-ordinator
Posted by "Prashanth Menon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201725#comment-13201725 ]
Prashanth Menon commented on KAFKA-264:
---------------------------------------
Yup, got it, makes sense. I misinterpreted the wiki and thought election was only done once and at startup, hence my queston. I'll probably take another look when I get some spare time :-) Thanks, Neha.
> Change the consumer side load balancing and distributed co-ordination to use a consumer co-ordinator
> ----------------------------------------------------------------------------------------------------
>
> Key: KAFKA-264
> URL: https://issues.apache.org/jira/browse/KAFKA-264
> Project: Kafka
> Issue Type: New Feature
> Components: core
> Affects Versions: 0.7
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Fix For: 0.7.1
>
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> A high level design for the zookeeper consumer is here - https://cwiki.apache.org/confluence/display/KAFKA/Consumer+co-ordinator
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-264) Change the consumer side load
balancing and distributed co-ordination to use a consumer co-ordinator
Posted by "Joe Stein (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joe Stein updated KAFKA-264:
----------------------------
Fix Version/s: (was: 0.7.1)
0.8
> Change the consumer side load balancing and distributed co-ordination to use a consumer co-ordinator
> ----------------------------------------------------------------------------------------------------
>
> Key: KAFKA-264
> URL: https://issues.apache.org/jira/browse/KAFKA-264
> Project: Kafka
> Issue Type: New Feature
> Components: core
> Affects Versions: 0.7
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Fix For: 0.8
>
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> A high level design for the zookeeper consumer is here - https://cwiki.apache.org/confluence/display/KAFKA/Consumer+co-ordinator
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (KAFKA-264) Change the consumer side load
balancing and distributed co-ordination to use a consumer co-ordinator
Posted by "Neha Narkhede (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Neha Narkhede reassigned KAFKA-264:
-----------------------------------
Assignee: Neha Narkhede
> Change the consumer side load balancing and distributed co-ordination to use a consumer co-ordinator
> ----------------------------------------------------------------------------------------------------
>
> Key: KAFKA-264
> URL: https://issues.apache.org/jira/browse/KAFKA-264
> Project: Kafka
> Issue Type: New Feature
> Components: core
> Affects Versions: 0.7
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Fix For: 0.7.1
>
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> A high level design for the zookeeper consumer is here - https://cwiki.apache.org/confluence/display/KAFKA/Consumer+co-ordinator
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira