Kafka 0.10 quickstart: consumer fails when “primary” broker is brought down

[Steven Peh <St...@macquarie.com>]

I’ve posted this on Stackoverflow as well: http://stackoverflow.com/questions/41756435/kafka-0-10-quickstart-consumer-fails-when-primary-broker-is-brought-down



So I'm trying the kafka quickstart<https://kafka.apache.org/quickstart> as per the main documentation. Got the multi-cluster example all setup and test per the instructions and it works. For example, bringing down one broker and the producer and consumer can still send and receive.



However, as per the example, we setup 3 brokers and we bring down broker 2 (with broker id = 1). Now if I bring up all brokers again, but I bring down broker 1 (with broker id = 0), the consumer just hangs. This only happens with broker 1 (id = 0), does not happen with broker 2 or 3. I'm testing this on Windows 7.

Is there something special here with broker 1? Looking at the config they are exactly the same between all 3 brokers except the id, port number and log file location.

I thought it is just a problem with the provided console consumer which doesn't take a broker list, so I wrote a simple java consumer as per their documentation using the default setup but specify the list of brokers in the "bootstrap.servers" property, but no dice, still get the same problem.

The moment I startup broker 1 (broker id = 0), the consumers will just resume working. This isn't a highly available/fault tolerant behavior for the consumer... any help on how to setup a HA/fault tolerant consumer?

Producers doesn't seem to have an issue.


Cheers.
Steven Peh
Technology Division | Macquarie Group Services Australia Pty Limited
50 Martin Place, Sydney, NSW, 2000, Australia
T +61 2 8237 5156 I M +61 438 444 839 I E steven.peh@macquarie.com<ma...@macquarie.com>
www.macquarie.com<http://www.macquarie.com/>


This email and any attachment is confidential. If you are not the intended recipient, please delete this message. Macquarie does not guarantee the integrity of any emails or attachments. For important disclosures and information about the incorporation and regulated status of Macquarie Group entities please see: www.macquarie.com/disclosures

Re: Kafka 0.10 quickstart: consumer fails when “primary” broker is brought down

[Hans Jespersen <ha...@confluent.io>]

The topic created in the quickstart only has a replication factor of 1 and a single partition. 

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
So the one topic partition is on broker 1. When broker 1 is up, you can consume the data but when it’s down, there is not a replica on any of the other brokers.

Try creating a new topic with replication-factor 3 and you should get better availability in the event of one or even two broker failures.

-hans


-- 
/**
 * Hans Jespersen, Principal Systems Engineer, Confluent Inc.
 * hans@confluent.io (650)924-2670
 */




> On Jan 19, 2017, at 9:24 PM, Steven Peh <St...@macquarie.com> wrote:
> 
> I’ve posted this on Stackoverflow as well: http://stackoverflow.com/questions/41756435/kafka-0-10-quickstart-consumer-fails-when-primary-broker-is-brought-down
> 
> 
> 
> So I'm trying the kafka quickstart<https://kafka.apache.org/quickstart> as per the main documentation. Got the multi-cluster example all setup and test per the instructions and it works. For example, bringing down one broker and the producer and consumer can still send and receive.
> 
> 
> 
> However, as per the example, we setup 3 brokers and we bring down broker 2 (with broker id = 1). Now if I bring up all brokers again, but I bring down broker 1 (with broker id = 0), the consumer just hangs. This only happens with broker 1 (id = 0), does not happen with broker 2 or 3. I'm testing this on Windows 7.
> 
> Is there something special here with broker 1? Looking at the config they are exactly the same between all 3 brokers except the id, port number and log file location.
> 
> I thought it is just a problem with the provided console consumer which doesn't take a broker list, so I wrote a simple java consumer as per their documentation using the default setup but specify the list of brokers in the "bootstrap.servers" property, but no dice, still get the same problem.
> 
> The moment I startup broker 1 (broker id = 0), the consumers will just resume working. This isn't a highly available/fault tolerant behavior for the consumer... any help on how to setup a HA/fault tolerant consumer?
> 
> Producers doesn't seem to have an issue.
> 
> 
> Cheers.
> Steven Peh
> Technology Division | Macquarie Group Services Australia Pty Limited
> 50 Martin Place, Sydney, NSW, 2000, Australia
> T +61 2 8237 5156 I M +61 438 444 839 I E steven.peh@macquarie.com<ma...@macquarie.com>
> www.macquarie.com<http://www.macquarie.com/>
> 
> 
> This email and any attachment is confidential. If you are not the intended recipient, please delete this message. Macquarie does not guarantee the integrity of any emails or attachments. For important disclosures and information about the incorporation and regulated status of Macquarie Group entities please see: www.macquarie.com/disclosures