You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Yu, Libo " <li...@citi.com> on 2014/02/25 21:01:48 UTC
ConsumerRebalanceFailedException
Hi all,
I tried to reproduce this exception. In case one, when no broker was running, I launched all consumers and
got this exception. In case two, while the consumers and brokers were running, I shutdown all brokers one by
one and did not see this exception. I wonder why in case two this exception did not occur. Thanks.
Regards,
Libo
Re: ConsumerRebalanceFailedException
Posted by Neha Narkhede <ne...@gmail.com>.
This actually doesn't look like the entire log. Could you attach your
entire log in pastebin.com or something similar? I'm looking for something
like this in your log - "Rebalancing attempt failed... "
Thanks,
Neha
On Fri, Feb 28, 2014 at 12:44 PM, Yu, Libo <li...@citi.com> wrote:
> Here is a snapshot of our logs. We've known that our three brokers somehow
> were offline and that caused the exception.
>
> key LOAN.SMC.134096888 takes 2 ms
> [06/02/14 08:54:06:006 AM EST] 198 INFO consumer.kafkaConsumerImpl:
> commitOddLots happen, topicName = credit.cache.smc.debt
> .topic, selector = null
> [06/02/14 08:54:28:028 AM EST] 215 INFO mkconsumer.MKConsumer: Put to
> table Credit.SMC.LOAN info put gdm entity Credit.SMC.LOAN,
> key LOAN.SMC.134096887 takes 3 ms
> [06/02/14 08:54:46:046 AM EST] 198 INFO consumer.kafkaConsumerImpl:
> commitOddLots happen, topicName = credit.cache.smc.debt
> .topic, selector = null
> [06/02/14 08:56:03:003 AM EST] 102 ERROR producer.SyncProducer: Producer
> connection to host7:11934 unsucce
> ssful
> java.nio.channels.ClosedByInterruptException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:634)
> at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> at kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> at
> kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161)
> at
> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:68)
> at kafka.producer.SyncProducer.send(SyncProducer.scala:112)
> at
> kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:53)
> at
> kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:88)
> at
> kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:64)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> [06/02/14 08:56:03:003 AM EST] 102 ERROR producer.SyncProducer: Producer
> connection to host9:11934 unsucce
> ssful
> java.nio.channels.ClosedByInterruptException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:634)
> at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> at kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> at
> kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161)
> at
> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:68)
> at kafka.producer.SyncProducer.send(SyncProducer.scala:112)
> at
> kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:53)
> at
> kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:88)
> at
> kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:64)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> [06/02/14 08:56:20:020 AM EST] 102 ERROR
> consumer.ZookeeperConsumerConnector:
> [mluser_mlusergroup_host8-1391315687926-95bc2ef2], error during
> syncedRebalance
> kafka.common.ConsumerRebalanceFailedException:
> mluser_mlusergroup_host8-1391315687926-95bc2ef2 can't rebalance after 4
> retries
> at
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:397)
> at
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anon$1.run(ZookeeperConsumerConnector.scala:326)
> [06/02/14 08:56:20:020 AM EST] 102 ERROR
> consumer.ZookeeperConsumerConnector:
> [mluser_mlusergroup_host8-1391315695110-e8e36bd0], error during
> syncedRebalance
> kafka.common.ConsumerRebalanceFailedException:
> mluser_mlusergroup_host8-1391315695110-e8e36bd0 can't rebalance after 4
> retries
> at
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:397)
> at
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anon$1.run(ZookeeperConsumerConnector.scala:326)
> [06/02/14 08:56:20:020 AM EST] 102 ERROR
> consumer.ZookeeperConsumerConnector:
> [mluser_mlusergroup_host8-1391315699203-94c60ea8], error during
> syncedRebalance
> kafka.common.ConsumerRebalanceFailedException:
> mluser_mlusergroup_host8-1391315699203-94c60ea8 can't rebalance after 4
> retries
> at
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:397)
> at
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anon$1.run(ZookeeperConsumerConnector.scala:326)
> [06/02/14 08:59:18:018 AM EST] 215 INFO mkconsumer.MKConsumer: Put to
> table Credit.SMC.MUNI info put gdm entity Credit.SMC.MUNI, key
> MUNI.SMC.134103391 takes 4 ms
> [06/02/14 08:59:18:018 AM EST] 215 INFO mkconsumer.MKConsumer: Put to
> table Credit.SMC.
>
> Regards,
>
> Libo
>
>
> -----Original Message-----
> From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
> Sent: Tuesday, February 25, 2014 4:03 PM
> To: users@kafka.apache.org
> Subject: Re: ConsumerRebalanceFailedException
>
> Could you send around the consumer log when it throws
> ConsumerRebalanceFailedException. It should state the reason for the failed
> rebalance attempts.
>
> Thanks,
> Neha
>
>
> On Tue, Feb 25, 2014 at 12:01 PM, Yu, Libo <li...@citi.com> wrote:
>
> > Hi all,
> >
> > I tried to reproduce this exception. In case one, when no broker was
> > running, I launched all consumers and got this exception. In case two,
> > while the consumers and brokers were running, I shutdown all brokers
> > one by one and did not see this exception. I wonder why in case two
> > this exception did not occur. Thanks.
> >
> >
> > Regards,
> >
> > Libo
> >
> >
>
RE: ConsumerRebalanceFailedException
Posted by "Yu, Libo " <li...@citi.com>.
Here is a snapshot of our logs. We've known that our three brokers somehow were offline and that caused the exception.
key LOAN.SMC.134096888 takes 2 ms
[06/02/14 08:54:06:006 AM EST] 198 INFO consumer.kafkaConsumerImpl: commitOddLots happen, topicName = credit.cache.smc.debt
.topic, selector = null
[06/02/14 08:54:28:028 AM EST] 215 INFO mkconsumer.MKConsumer: Put to table Credit.SMC.LOAN info put gdm entity Credit.SMC.LOAN,
key LOAN.SMC.134096887 takes 3 ms
[06/02/14 08:54:46:046 AM EST] 198 INFO consumer.kafkaConsumerImpl: commitOddLots happen, topicName = credit.cache.smc.debt
.topic, selector = null
[06/02/14 08:56:03:003 AM EST] 102 ERROR producer.SyncProducer: Producer connection to host7:11934 unsucce
ssful
java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:634)
at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
at kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
at kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:68)
at kafka.producer.SyncProducer.send(SyncProducer.scala:112)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:53)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:88)
at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:64)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
[06/02/14 08:56:03:003 AM EST] 102 ERROR producer.SyncProducer: Producer connection to host9:11934 unsucce
ssful
java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:634)
at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
at kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
at kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:68)
at kafka.producer.SyncProducer.send(SyncProducer.scala:112)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:53)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:88)
at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:64)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
[06/02/14 08:56:20:020 AM EST] 102 ERROR consumer.ZookeeperConsumerConnector: [mluser_mlusergroup_host8-1391315687926-95bc2ef2], error during syncedRebalance
kafka.common.ConsumerRebalanceFailedException: mluser_mlusergroup_host8-1391315687926-95bc2ef2 can't rebalance after 4 retries
at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:397)
at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anon$1.run(ZookeeperConsumerConnector.scala:326)
[06/02/14 08:56:20:020 AM EST] 102 ERROR consumer.ZookeeperConsumerConnector: [mluser_mlusergroup_host8-1391315695110-e8e36bd0], error during syncedRebalance
kafka.common.ConsumerRebalanceFailedException: mluser_mlusergroup_host8-1391315695110-e8e36bd0 can't rebalance after 4 retries
at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:397)
at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anon$1.run(ZookeeperConsumerConnector.scala:326)
[06/02/14 08:56:20:020 AM EST] 102 ERROR consumer.ZookeeperConsumerConnector: [mluser_mlusergroup_host8-1391315699203-94c60ea8], error during syncedRebalance
kafka.common.ConsumerRebalanceFailedException: mluser_mlusergroup_host8-1391315699203-94c60ea8 can't rebalance after 4 retries
at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:397)
at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anon$1.run(ZookeeperConsumerConnector.scala:326)
[06/02/14 08:59:18:018 AM EST] 215 INFO mkconsumer.MKConsumer: Put to table Credit.SMC.MUNI info put gdm entity Credit.SMC.MUNI, key MUNI.SMC.134103391 takes 4 ms
[06/02/14 08:59:18:018 AM EST] 215 INFO mkconsumer.MKConsumer: Put to table Credit.SMC.
Regards,
Libo
-----Original Message-----
From: Neha Narkhede [mailto:neha.narkhede@gmail.com]
Sent: Tuesday, February 25, 2014 4:03 PM
To: users@kafka.apache.org
Subject: Re: ConsumerRebalanceFailedException
Could you send around the consumer log when it throws ConsumerRebalanceFailedException. It should state the reason for the failed rebalance attempts.
Thanks,
Neha
On Tue, Feb 25, 2014 at 12:01 PM, Yu, Libo <li...@citi.com> wrote:
> Hi all,
>
> I tried to reproduce this exception. In case one, when no broker was
> running, I launched all consumers and got this exception. In case two,
> while the consumers and brokers were running, I shutdown all brokers
> one by one and did not see this exception. I wonder why in case two
> this exception did not occur. Thanks.
>
>
> Regards,
>
> Libo
>
>
Re: ConsumerRebalanceFailedException
Posted by Neha Narkhede <ne...@gmail.com>.
Could you send around the consumer log when it throws
ConsumerRebalanceFailedException. It should state the reason for the failed
rebalance attempts.
Thanks,
Neha
On Tue, Feb 25, 2014 at 12:01 PM, Yu, Libo <li...@citi.com> wrote:
> Hi all,
>
> I tried to reproduce this exception. In case one, when no broker was
> running, I launched all consumers and
> got this exception. In case two, while the consumers and brokers were
> running, I shutdown all brokers one by
> one and did not see this exception. I wonder why in case two this
> exception did not occur. Thanks.
>
>
> Regards,
>
> Libo
>
>