You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Sergey Ivanov (Jira)" <ji...@apache.org> on 2022/02/02 18:32:00 UTC
[jira] [Comment Edited] (KAFKA-13639) NotEnoughReplicasException for __consumer_offsets topic due to out of order offset
[ https://issues.apache.org/jira/browse/KAFKA-13639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17486005#comment-17486005 ]
Sergey Ivanov edited comment on KAFKA-13639 at 2/2/22, 6:31 PM:
----------------------------------------------------------------
We found mail thread which describe the similar issue:
[https://lists.apache.org/thread/2l1snn1jrzd5p2d0n55vs3dg5jr7f35v]
and the same in
[https://stackoverflow.com/questions/64514851/apache-kafka-kafka-common-offsetsoutoforderexception-when-reassigning-consume]
when __consumer_offsets topic's partition was corrupted due to shutdown Kafka cluster (with working clients).
They said that issue was resolved when Kafka compacts topics segments (with our retention is 1Gb or 7 day). We couldn't wait that time so we speeded up this process and set the following properties for the topic:
{code:java}
./bin/kafka-topics.sh --bootstrap-server localhost:9092 --alter --topic __consumer_offsets --config retention.ms=10000
{code}
Then we restarted Kafka cluster, and the issue gone, partition 36 has been replicated.
We faced this issue at lease one more time and it doesn't go without our manually retention changes. So looks like this is not random +bug.+
Can anyone help with investigation? We can provide more logs if necessary.
was (Author: mrmigles):
We found mail thread which describe the similar issue:
[https://lists.apache.org/thread/2l1snn1jrzd5p2d0n55vs3dg5jr7f35v]
and the same in
[https://stackoverflow.com/questions/64514851/apache-kafka-kafka-common-offsetsoutoforderexception-when-reassigning-consume]
when __consumer_offsets topic's partition was corrupted due to shutdown Kafka cluster (with working clients).
They said that issue was resolved when Kafka compacts topics segments (with our retention is 1Gb or 7 day). We couldn't wait that time so we speeded up this process and set the following properties for the topic:
{code:java}
./bin/kafka-topics.sh --bootstrap-server localhost:9092 --alter --topic __consumer_offsets --config retention.ms=10000
{code}
Then we restarted Kafka cluster, and {*}the issue gone{*}, partition 36 has been replicated.
We faced this issue at lease one more time and it doesn't go without our manually retention changes. So looks like this is not random bug.
Can anyone help with investigation? We can provide more logs if necessary.
> NotEnoughReplicasException for __consumer_offsets topic due to out of order offset
> ----------------------------------------------------------------------------------
>
> Key: KAFKA-13639
> URL: https://issues.apache.org/jira/browse/KAFKA-13639
> Project: Kafka
> Issue Type: Bug
> Components: core, log
> Affects Versions: 2.6.2
> Reporter: Sergey Ivanov
> Priority: Major
>
> Hello,
> We faced a strange issue with Kafka during testing failover scenarios: this assumes forces shutdown nodes where Kafka pods are placed (Kafka is deployed to Kubernetes), and then return these nodes.
> After this Kafka pods are started normally but +some+ consumers could not connect to it with errors:
>
> {code:java}
> [2022-01-27T14:35:09.051][level=DEBUG][class=kafka_client:utils.go:120]: Failed to sync group mae_processor: [15] Group Coordinator Not Available: the broker returns this error code for group coordinator requests, offset commits, and most group management requests if the offsets topic has not yet been created, or if the group coordinator is not active{code}
>
>
> It looked like there were issues with ___consumer_offsets_ topic. In logs of brokers we found this error:
> {code:java}
> [2022-01-27T14:56:00,233][INFO][category=kafka.coordinator.group.GroupCoordinator] [GroupCoordinator 1]: Group mae_processor with generation 329 is now empty (__consumer_offsets-36)
> [2022-01-27T14:56:00,233][ERROR][category=kafka.server.ReplicaManager] [ReplicaManager broker=1] Error processing append operation on partition __consumer_offsets-36
> org.apache.kafka.common.errors.NotEnoughReplicasException: The size of the current ISR Set(1) is insufficient to satisfy the min.isr requirement of 2 for partition __consumer_offsets-36
> [2022-01-27T14:56:00,233][WARN][category=kafka.coordinator.group.GroupCoordinator] [GroupCoordinator 1]: Failed to write empty metadata for group mae_processor: The coordinator is not available.
> {code}
> If we check partitions of __consumer_offsets it really has one partition with insufficient ISR:
> {code:java}
> topic "__consumer_offsets" with 50 partitions:
> partition 0, leader 1, replicas: 1,3,2, isrs: 1,2,3
> ...
> partition 35, leader 3, replicas: 3,1,2, isrs: 1,2,3
> partition 36, leader 1, replicas: 1,3,2, isrs: 1
> partition 37, leader 2, replicas: 2,1,3, isrs: 1,2,3
> ....
> partition 49, leader 2, replicas: 2,1,3, isrs: 1,2,3{code}
> We wait some time but the issue didn't go, we still had one partition with insufficient ISR.
> First of all we [thought |https://stackoverflow.com/questions/51491152/fixing-under-replicated-partitions-in-kafka/53540963#53540963]this is issue with Kafka-ZooKeeper coordinations, so we restarted ZooKeeper cluster and brokers 2 and 3, which didn't have ISR. +But it didn't help.+
> We also tried to manually ellect leader for this partition with kafka-leader-election.sh (in hope it will help). +But it didn't help too.+
> In logs we also found an issue:
> {code:java}
> [2022-01-27T16:17:29,531][ERROR][category=kafka.server.ReplicaFetcherThread] [ReplicaFetcher replicaId=2, leaderId=1, fetcherId=0] Unexpected error occurred while processing data for partition __consumer_offsets-36 at offset 19536
> kafka.common.OffsetsOutOfOrderException: Out of order offsets found in append to __consumer_offsets-36: List(19536, 19536, 19537, 19538, 19539, 19540, 19541, 19542, 19543, 19544, 19545, 19546, 19547, 19548, 19549, 19550, 19551, 19552, 19553, 19554, 19555, 19556, 19557, 19558, 19559, 19560, 19561)
> at kafka.log.Log.$anonfun$append$2(Log.scala:1126)
> at kafka.log.Log.append(Log.scala:2349)
> at kafka.log.Log.appendAsFollower(Log.scala:1036)
> at
> [2022-01-27T16:17:29,531][WARN][category=kafka.server.ReplicaFetcherThread] [ReplicaFetcher replicaId=2, leaderId=1, fetcherId=0] Partition __consumer_offsets-36 marked as failed
> {code}
> This looks like root cause, right? Can force shutdown Kafka process lead to this issue?
> Looks like a bug, moreover, shall Kafka handle case of corrupting data (if it's the root cause of issue above)?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)