You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@kafka.apache.org by Florin Trofin <ft...@adobe.com> on 2013/07/13 02:55:48 UTC

ConsumerRebalanceFailedException when broker unavailable

Hello,

I'm going through a series of tests with Kafka 0.8 to see what the failure cases are. One simple scenario is this:

1. Launch Zookeeper
2. Launch my consumer (written in scala) but don't launch a Kafka broker (I am not using replication in my configuration).

What happens in this particular scenario is that my consumer hits the createMessageStreams which internally makes 4 failed attempts and then it raises a ConsumerRebalanceFailedException:

kafka.common.ConsumerRebalanceFailedException: zfilter_vm-florin-ubuntu-d-12-1373675347482-ff93b841 can't rebalance after 4 retries
	at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:397)
	at kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:681)
	at kafka.consumer.ZookeeperConsumerConnector.consume(ZookeeperConsumerConnector.scala:205)
	at kafka.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:133)
	at kafka.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:126)


I was expecting a more specific exception - something like NoBrokerAvailable instead of this. I can add a catch clause for this exception and retry the operation in a loop until the broker comes online but I dislike the approach because I am making an assumption here - that ConsumerRebalanceFailedException means "no broker available". Is there a better way to do this? Do you have a pattern for consumers for when brokers go offline and come back up?

Thanks a lot!

Florin

Re: ConsumerRebalanceFailedException when broker unavailable

Posted by Joel Koshy <jj...@gmail.com>.

I think this is addressed in KAFKA-969

On Fri, Jul 12, 2013 at 5:55 PM, Florin Trofin <ft...@adobe.com> wrote:
> Hello,
>
> I'm going through a series of tests with Kafka 0.8 to see what the failure cases are. One simple scenario is this:
>
> 1. Launch Zookeeper
> 2. Launch my consumer (written in scala) but don't launch a Kafka broker (I am not using replication in my configuration).
>
> What happens in this particular scenario is that my consumer hits the createMessageStreams which internally makes 4 failed attempts and then it raises a ConsumerRebalanceFailedException:
>
> kafka.common.ConsumerRebalanceFailedException: zfilter_vm-florin-ubuntu-d-12-1373675347482-ff93b841 can't rebalance after 4 retries
>         at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:397)
>         at kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:681)
>         at kafka.consumer.ZookeeperConsumerConnector.consume(ZookeeperConsumerConnector.scala:205)
>         at kafka.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:133)
>         at kafka.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:126)
>
>
> I was expecting a more specific exception - something like NoBrokerAvailable instead of this. I can add a catch clause for this exception and retry the operation in a loop until the broker comes online but I dislike the approach because I am making an assumption here - that ConsumerRebalanceFailedException means "no broker available". Is there a better way to do this? Do you have a pattern for consumers for when brokers go offline and come back up?
>
> Thanks a lot!
>
> Florin