Re: ConsumerRebalanceFailedException

[Neha Narkhede <ne...@gmail.com>]

This actually doesn't look like the entire log. Could you attach your
entire log in pastebin.com or something similar? I'm looking for something
like this in your log - "Rebalancing attempt failed... "

Thanks,
Neha


On Fri, Feb 28, 2014 at 12:44 PM, Yu, Libo <li...@citi.com> wrote:

> Here is a snapshot of our logs. We've known that our three brokers somehow
> were offline and that caused the exception.
>
> key LOAN.SMC.134096888 takes 2 ms
> [06/02/14 08:54:06:006 AM EST] 198  INFO consumer.kafkaConsumerImpl:
> commitOddLots happen, topicName = credit.cache.smc.debt
> .topic, selector = null
> [06/02/14 08:54:28:028 AM EST] 215  INFO mkconsumer.MKConsumer: Put to
> table Credit.SMC.LOAN info put gdm entity Credit.SMC.LOAN,
>  key LOAN.SMC.134096887 takes 3 ms
> [06/02/14 08:54:46:046 AM EST] 198  INFO consumer.kafkaConsumerImpl:
> commitOddLots happen, topicName = credit.cache.smc.debt
> .topic, selector = null
> [06/02/14 08:56:03:003 AM EST] 102 ERROR producer.SyncProducer: Producer
> connection to host7:11934 unsucce
> ssful
> java.nio.channels.ClosedByInterruptException
>         at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
>         at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:634)
>         at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
>         at kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
>         at
> kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161)
>         at
> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:68)
>         at kafka.producer.SyncProducer.send(SyncProducer.scala:112)
>         at
> kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:53)
>         at
> kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:88)
>         at
> kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:64)
>         at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> [06/02/14 08:56:03:003 AM EST] 102 ERROR producer.SyncProducer: Producer
> connection to host9:11934 unsucce
> ssful
> java.nio.channels.ClosedByInterruptException
>         at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
>         at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:634)
>         at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
>         at kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
>         at
> kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161)
>         at
> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:68)
>         at kafka.producer.SyncProducer.send(SyncProducer.scala:112)
>         at
> kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:53)
>         at
> kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:88)
>         at
> kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:64)
>         at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> [06/02/14 08:56:20:020 AM EST] 102 ERROR
> consumer.ZookeeperConsumerConnector:
> [mluser_mlusergroup_host8-1391315687926-95bc2ef2], error during
> syncedRebalance
> kafka.common.ConsumerRebalanceFailedException:
> mluser_mlusergroup_host8-1391315687926-95bc2ef2 can't rebalance after 4
> retries
>         at
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:397)
>         at
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anon$1.run(ZookeeperConsumerConnector.scala:326)
> [06/02/14 08:56:20:020 AM EST] 102 ERROR
> consumer.ZookeeperConsumerConnector:
> [mluser_mlusergroup_host8-1391315695110-e8e36bd0], error during
> syncedRebalance
> kafka.common.ConsumerRebalanceFailedException:
> mluser_mlusergroup_host8-1391315695110-e8e36bd0 can't rebalance after 4
> retries
>         at
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:397)
>         at
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anon$1.run(ZookeeperConsumerConnector.scala:326)
> [06/02/14 08:56:20:020 AM EST] 102 ERROR
> consumer.ZookeeperConsumerConnector:
> [mluser_mlusergroup_host8-1391315699203-94c60ea8], error during
> syncedRebalance
> kafka.common.ConsumerRebalanceFailedException:
> mluser_mlusergroup_host8-1391315699203-94c60ea8 can't rebalance after 4
> retries
>         at
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:397)
>         at
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anon$1.run(ZookeeperConsumerConnector.scala:326)
> [06/02/14 08:59:18:018 AM EST] 215  INFO mkconsumer.MKConsumer: Put to
> table Credit.SMC.MUNI info put gdm entity Credit.SMC.MUNI, key
> MUNI.SMC.134103391 takes 4 ms
> [06/02/14 08:59:18:018 AM EST] 215  INFO mkconsumer.MKConsumer: Put to
> table Credit.SMC.
>
> Regards,
>
> Libo
>
>
> -----Original Message-----
> From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> Sent: Tuesday, February 25, 2014 4:03 PM
> To: users@kafka.apache.org
> Subject: Re: ConsumerRebalanceFailedException
>
> Could you send around the consumer log when it throws
> ConsumerRebalanceFailedException. It should state the reason for the failed
> rebalance attempts.
>
> Thanks,
> Neha
>
>
> On Tue, Feb 25, 2014 at 12:01 PM, Yu, Libo <li...@citi.com> wrote:
>
> > Hi all,
> >
> > I tried to reproduce this exception. In case one, when no broker was
> > running, I launched all consumers and got this exception. In case two,
> > while the consumers and brokers were running, I shutdown all brokers
> > one by one and did not see this exception. I wonder why in case two
> > this exception did not occur. Thanks.
> >
> >
> > Regards,
> >
> > Libo
> >
> >
>