The motive to detach new consumer and zookeeper

[Li Tao <ah...@gmail.com>]

Hi guys,

As we know, the new consumer of 0.9 kafka does not need to communicate with
zookeeper.
What is the motive / rational behind this disign? I have some
guesses(correct me if i am wrong):

1. Reduce the complexity of consumer. It's hard to maintain the connections
and states of kafka servers and zookeeper servers.

2. Improve performance. Too many consumers querying and writting to
zookeeper will decrease the performance as a whole.

Thanks.

Re: The motive to detach new consumer and zookeeper

[Jason Gustafson <ja...@confluent.io>]

Hi Li,

I think reducing the client's complexity and improving performance were two
of the main reasons for the change. The rebalance protocol on top of
Zookeeper was difficult to implement correctly, and I think a number of
Kafka clients never actually got it working. Removing it as a dependence
also makes a lot of sense with the introduction of security in 0.9.
Zookeeper is easier to protect since it only needs to be accessed by the
brokers and the clients are easier to audit since all requests go through
Kafka. The full rationale is documented here:
https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design.

-Jason

On Sat, Dec 5, 2015 at 1:07 AM, Li Tao <ah...@gmail.com> wrote:

> Hi guys,
>
> As we know, the new consumer of 0.9 kafka does not need to communicate with
> zookeeper.
> What is the motive / rational behind this disign? I have some
> guesses(correct me if i am wrong):
>
> 1. Reduce the complexity of consumer. It's hard to maintain the connections
> and states of kafka servers and zookeeper servers.
>
> 2. Improve performance. Too many consumers querying and writting to
> zookeeper will decrease the performance as a whole.
>
> Thanks.
>