Failed to fetch offset for partition; There are unstable offsets that need to be cleared

[Domantas Petrauskas <dp...@stackstate.com>]

Hello,

I am getting this error in a Kafka consumer, which causes the consumer fail to consuming messages:

INFO o.a.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-ConsumerGroup-1, groupId= ConsumerGroup] The following partitions still have unstable offsets which are not cleared on the broker side: [topic-6], this could be either transactional offsets waiting for completion, or normal offsets waiting for replication after appending to local log DEBUG o.a.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-ConsumerGroup-1, groupId=ConsumerGroup] Failed to fetch offset for partition topic-6: There are unstable offsets that need to be cleared

I tried to find some details on how to fix this issue, but could not find anything, so I tried to restart Kafka, restart the application with the consumer, reset the offset, but that did not help. Then I tried to delete and re-create the consumer group and that helped, which is obviously not ideal. What could be the cause, how to troubleshoot this issue?

Best regards,

Domantas Petrauskas