You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Kashyap Mhaisekar <ka...@gmail.com> on 2014/04/03 04:56:37 UTC
Kafka Rebalance exceptions in Storm
Hi,
I consistently get the following exception when I submit a storm topology
using StormSubmitter. I am using 0.8.2 Storm.
The spout I use is a Kafka High Level consumer thats custom written.
When I submit the same topology using LocalCluster, it works fine. Am I
missing anything
*kafka.common.ConsumerRebalanceFailedException:
Sample_Kafka_client_abcd_com-1396492916541-2f28228c can't rebalance after 4
retries*
* at
kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:432)
~[stormjar.jar:na]*
* at
kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:722)
~[stormjar.jar:na]*
* at
kafka.consumer.ZookeeperConsumerConnector.consume(ZookeeperConsumerConnector.scala:212)
~[stormjar.jar:na]*
* at
kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:80)
~[stormjar.jar:na]*
* at
kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:92)
~[stormjar.jar:na]*
*2014-04-02 21:41:58 b.s.util [INFO] Halting process: ("Worker died")*
Thanks in advance
Regards,
Kashyap
Re: Kafka Rebalance exceptions in Storm
Posted by Kashyap Mhaisekar <ka...@gmail.com>.
Thanks. How did you manage the offset persistence in this case?
On Thursday, April 3, 2014, padma priya chitturi <pa...@gmail.com>
wrote:
> The same exception i encountered when tried to integrate storm spout with
> kafka high level consumer. This was resolved when i used SimpleConsumer.
>
>
> On Thu, Apr 3, 2014 at 8:26 AM, Kashyap Mhaisekar <kashyap.m@gmail.com<javascript:_e(%7B%7D,'cvml','kashyap.m@gmail.com');>
> > wrote:
>
>> Hi,
>> I consistently get the following exception when I submit a storm topology
>> using StormSubmitter. I am using 0.8.2 Storm.
>> The spout I use is a Kafka High Level consumer thats custom written.
>>
>> When I submit the same topology using LocalCluster, it works fine. Am I
>> missing anything
>>
>> *kafka.common.ConsumerRebalanceFailedException:
>> Sample_Kafka_client_abcd_com-1396492916541-2f28228c can't rebalance after 4
>> retries*
>> * at
>> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:432)
>> ~[stormjar.jar:na]*
>> * at
>> kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:722)
>> ~[stormjar.jar:na]*
>> * at
>> kafka.consumer.ZookeeperConsumerConnector.consume(ZookeeperConsumerConnector.scala:212)
>> ~[stormjar.jar:na]*
>> * at
>> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:80)
>> ~[stormjar.jar:na]*
>> * at
>> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:92)
>> ~[stormjar.jar:na]*
>>
>>
>> *2014-04-02 21:41:58 b.s.util [INFO] Halting process: ("Worker died") *
>>
>> Thanks in advance
>> Regards,
>> Kashyap
>>
>
>
Re: Kafka Rebalance exceptions in Storm
Posted by padma priya chitturi <pa...@gmail.com>.
The same exception i encountered when tried to integrate storm spout with
kafka high level consumer. This was resolved when i used SimpleConsumer.
On Thu, Apr 3, 2014 at 8:26 AM, Kashyap Mhaisekar <ka...@gmail.com>wrote:
> Hi,
> I consistently get the following exception when I submit a storm topology
> using StormSubmitter. I am using 0.8.2 Storm.
> The spout I use is a Kafka High Level consumer thats custom written.
>
> When I submit the same topology using LocalCluster, it works fine. Am I
> missing anything
>
> *kafka.common.ConsumerRebalanceFailedException:
> Sample_Kafka_client_abcd_com-1396492916541-2f28228c can't rebalance after 4
> retries*
> * at
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:432)
> ~[stormjar.jar:na]*
> * at
> kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:722)
> ~[stormjar.jar:na]*
> * at
> kafka.consumer.ZookeeperConsumerConnector.consume(ZookeeperConsumerConnector.scala:212)
> ~[stormjar.jar:na]*
> * at
> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:80)
> ~[stormjar.jar:na]*
> * at
> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:92)
> ~[stormjar.jar:na]*
>
>
> *2014-04-02 21:41:58 b.s.util [INFO] Halting process: ("Worker died") *
>
> Thanks in advance
> Regards,
> Kashyap
>