You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Sathyakumar Seshachalam <sa...@trimble.com> on 2016/07/01 08:36:22 UTC
Coordinator lost for consumer groups
Hi,
I have 3 Kafka nodes (running 0.9.0) that all had active consumers and
producers.
Now all these had uncompacted __consumer_offsets group that grew to 1.8 TB.
So I restarted these nodes with a log.cleaner.enabled to save some space.
Since then consumers have stalled.
When I do a ./kafka-consumer-groups.sh kafka-groups.sh --zookeeper
10.211.16.215 --group groupX --describe,
I get
"Could not fetch offset from kafka for group GroupX partition [GroupX, 0]
due to kafka.common.NotCoordinatorForConsumerException". Note that the
compaction is still in progress. And I get this for most consumer groups.
Any clues how to fix this ?
Regards,
Sathya
Re: Coordinator lost for consumer groups
Posted by Ben Stopford <be...@confluent.io>.
You might try increasing the log.cleaner.dedupe.buffer.size. This should increase the deduplication yield for each scan.
If you haven’t seen them there are some notes on log compaction here: https://cwiki.apache.org/confluence/display/KAFKA/Log+Compaction <https://cwiki.apache.org/confluence/display/KAFKA/Log+Compaction>
> On 1 Jul 2016, at 10:10, Sathyakumar Seshachalam <sa...@trimble.com> wrote:
>
> The problem still persists. But note that I was running old consumer (Zk
> based) to describe consumers. Running ./kafka-consumer-groups.sh
> kafka-groups.sh --bootstrap-server 10.211.16.215 --group groupX --describe,
> I get the below error. So none of the consumer groups seem to have a
> coordinator now.
> Error while executing consumer group command This is not the correct
> coordinator for this group.
> org.apache.kafka.common.errors.NotCoordinatorForGroupException: This is not
> the correct coordinator for this group.
>
> On Fri, Jul 1, 2016 at 2:15 PM, Sathyakumar Seshachalam <
> sathyakumar_seshachalam@trimble.com> wrote:
>
>> And I am willing to suspend log compaction and restart the brokers, but am
>> worried if that will leave the system in a recoverable state or If I just
>> have to wait it out.
>>
>> On Fri, Jul 1, 2016 at 2:06 PM, Sathyakumar Seshachalam <
>> sathyakumar_seshachalam@trimble.com> wrote:
>>
>>> Hi,
>>>
>>> I have 3 Kafka nodes (running 0.9.0) that all had active consumers and
>>> producers.
>>>
>>> Now all these had uncompacted __consumer_offsets group that grew to 1.8
>>> TB. So I restarted these nodes with a log.cleaner.enabled to save some
>>> space. Since then consumers have stalled.
>>>
>>> When I do a ./kafka-consumer-groups.sh kafka-groups.sh --zookeeper
>>> 10.211.16.215 --group groupX --describe,
>>>
>>> I get
>>> "Could not fetch offset from kafka for group GroupX partition [GroupX, 0]
>>> due to kafka.common.NotCoordinatorForConsumerException". Note that the
>>> compaction is still in progress. And I get this for most consumer groups.
>>>
>>> Any clues how to fix this ?
>>>
>>> Regards,
>>> Sathya
>>>
>>>
>>>
>>>
>>>
>>
>>
Re: Coordinator lost for consumer groups
Posted by Sathyakumar Seshachalam <sa...@trimble.com>.
The problem still persists. But note that I was running old consumer (Zk
based) to describe consumers. Running ./kafka-consumer-groups.sh
kafka-groups.sh --bootstrap-server 10.211.16.215 --group groupX --describe,
I get the below error. So none of the consumer groups seem to have a
coordinator now.
Error while executing consumer group command This is not the correct
coordinator for this group.
org.apache.kafka.common.errors.NotCoordinatorForGroupException: This is not
the correct coordinator for this group.
On Fri, Jul 1, 2016 at 2:15 PM, Sathyakumar Seshachalam <
sathyakumar_seshachalam@trimble.com> wrote:
> And I am willing to suspend log compaction and restart the brokers, but am
> worried if that will leave the system in a recoverable state or If I just
> have to wait it out.
>
> On Fri, Jul 1, 2016 at 2:06 PM, Sathyakumar Seshachalam <
> sathyakumar_seshachalam@trimble.com> wrote:
>
>> Hi,
>>
>> I have 3 Kafka nodes (running 0.9.0) that all had active consumers and
>> producers.
>>
>> Now all these had uncompacted __consumer_offsets group that grew to 1.8
>> TB. So I restarted these nodes with a log.cleaner.enabled to save some
>> space. Since then consumers have stalled.
>>
>> When I do a ./kafka-consumer-groups.sh kafka-groups.sh --zookeeper
>> 10.211.16.215 --group groupX --describe,
>>
>> I get
>> "Could not fetch offset from kafka for group GroupX partition [GroupX, 0]
>> due to kafka.common.NotCoordinatorForConsumerException". Note that the
>> compaction is still in progress. And I get this for most consumer groups.
>>
>> Any clues how to fix this ?
>>
>> Regards,
>> Sathya
>>
>>
>>
>>
>>
>
>
Re: Coordinator lost for consumer groups
Posted by Sathyakumar Seshachalam <sa...@trimble.com>.
And I am willing to suspend log compaction and restart the brokers, but am
worried if that will leave the system in a recoverable state or If I just
have to wait it out.
On Fri, Jul 1, 2016 at 2:06 PM, Sathyakumar Seshachalam <
sathyakumar_seshachalam@trimble.com> wrote:
> Hi,
>
> I have 3 Kafka nodes (running 0.9.0) that all had active consumers and
> producers.
>
> Now all these had uncompacted __consumer_offsets group that grew to 1.8
> TB. So I restarted these nodes with a log.cleaner.enabled to save some
> space. Since then consumers have stalled.
>
> When I do a ./kafka-consumer-groups.sh kafka-groups.sh --zookeeper
> 10.211.16.215 --group groupX --describe,
>
> I get
> "Could not fetch offset from kafka for group GroupX partition [GroupX, 0]
> due to kafka.common.NotCoordinatorForConsumerException". Note that the
> compaction is still in progress. And I get this for most consumer groups.
>
> Any clues how to fix this ?
>
> Regards,
> Sathya
>
>
>
>
>