About kafka 0.8 producer auto detect broker

[gj1989lh <gj...@163.com>]

Hi,
We know in Kafka 0.7, we can specify zk.connect. And with zookeeper, the producer can dynamically detect broker. But in Kafka 0.8, we can't specify zk.connect for producer. How does the producer in Kafka 0.8 auto detect broker? 
I have done two experiments.
In first one, I configure the broker.list as broker1 and broker2, and set num.partitions=4 for all brokers. Then, I start broker 1 and broker 3. Without create topic1, I produce message directly, there are 4 partitions generated. Two in broker 1 and two in broker 3. How can the producer detect broker 3?
In second one, I configure the broker.list as broker 1 and broker2, and set num.partitions=4 for all brokers. Then I start broker 3 only. Without create topic2, I produce message directly. As a result ,there is no partition created for topic 2, and the producer report ConnectException, it says fetching topic metadata for topics [Set(topic 2)] from broker 3 failed (kafka.client.ClientUtils$)


Can I guess that the auto detecting ability for producer is weakened in Kafka 0.8? Why you get rid of zookeeper from producer in Kafka 0.8?


Many thanks and best regards!

Re: About kafka 0.8 producer auto detect broker

[Neha Narkhede <ne...@gmail.com>]

> Can I guess that the auto detecting ability for producer is weakened in
> Kafka 0.8? Why you get rid of zookeeper from producer in Kafka 0.8?


In 0.8, we got rid of zookeeper from the producer and replaced its
functionality with a metadata API. On startup and whenever a request fails,
the producer refreshes its view of the cluster by sending a metadata
request to any of the brokers. So if a broker goes down or a new broker
comes up, the leader for some partitions might change and the producer will
know since its requests to the older leaders will fail.

By removing the dependency of zookeeper from the producer, we have in fact
helped scale the number of producers you can have in a cluster since now
each of them don't need to have a connection to zookeeper and deal with
session expirations when there are minor glitches on the zookeeper cluster.
Another goal of this change was also to enable writing non-java clients
easier.

Thanks,
Neha