[jira] [Commented] (KAFKA-167) Move partition assignment to the broker

["nicu marasoiu (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14059796#comment-14059796 ] 

nicu marasoiu commented on KAFKA-167:
-------------------------------------

Hi,

So I would suggest this: for each consumer group, one broker can be elected as partition/replica assignment HA coordinator using a standard election mechanism on zookeeper already available in Curator.

I think it is nicer to distribute as much as possible, and, while recognizing that the algorithm is more deterministic, stable and convergent when using a centralized coordinator, there is no reason to pick the same coordinator for different consumer groups.

Apart from the mail discussion thread, can you please indicate any discussion on design, implementation, so that I can try to pick a small part and code something for it?

Thanks,
Nicu Marasoiu

> Move partition assignment to the broker
> ---------------------------------------
>
>                 Key: KAFKA-167
>                 URL: https://issues.apache.org/jira/browse/KAFKA-167
>             Project: Kafka
>          Issue Type: New Feature
>            Reporter: Jay Kreps
>
> Currently partitions are assigned to consumers for consumption via a co-ordination algorithm by the consumers. This means there is effectively no master broker node, which is nice in a way, but makes the consumer logic complex to implement. It would be good to move the co-ordination to the brokers. This would make implementation of a consumer client much much easier, since only minimal zk interaction would be required. There are a number of details that would  have to be worked out to finalize the protocol. Discussion is here:
> http://mail-archives.apache.org/mod_mbox/incubator-kafka-dev/201109.mbox/%3cCAFbh0Q2ADjXcbCYqGh8TB4jXBAs3Hq5RLxk9o9w76we-bWanVw@mail.gmail.com%3e



--
This message was sent by Atlassian JIRA
(v6.2#6252)