[jira] [Commented] (KAFKA-9350) IllegalStateException when materializing transactional offset commits

["zhangzhisheng (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-9350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17675543#comment-17675543 ] 

zhangzhisheng commented on KAFKA-9350:
--------------------------------------

 

kafka version: 2.13_2.7.2

I've found the same error in a 2.7.2 broker

 
{code:java}

// code placeholder
[2023-01-11 20:06:09,167] ERROR Uncaught exception in scheduled task 'handleTxnCompletion-1701009' (kafka.utils.KafkaScheduler)
java.lang.IllegalStateException: Trying to complete a transactional offset commit for producerId 1701009 and groupId prd-feature-result even though the offset commit record itself hasn't been appended to the log.
        at kafka.coordinator.group.GroupMetadata.$anonfun$completePendingTxnOffsetCommit$2(GroupMetadata.scala:667)
        at kafka.utils.Implicits$MapExtensionMethods$.$anonfun$forKeyValue$1(Implicits.scala:62)
        at scala.collection.MapOps.foreachEntry(Map.scala:221)
        at scala.collection.MapOps.foreachEntry$(Map.scala:217)
        at scala.collection.AbstractMap.foreachEntry(Map.scala:381)
        at kafka.coordinator.group.GroupMetadata.$anonfun$completePendingTxnOffsetCommit$1(GroupMetadata.scala:664)
        at kafka.coordinator.group.GroupMetadata.completePendingTxnOffsetCommit(GroupMetadata.scala:663)
        at kafka.coordinator.group.GroupMetadataManager.$anonfun$handleTxnCompletion$2(GroupMetadataManager.scala:889)
        at kafka.coordinator.group.GroupMetadata.inLock(GroupMetadata.scala:228) {code}
 

 

> IllegalStateException when materializing transactional offset commits
> ---------------------------------------------------------------------
>
>                 Key: KAFKA-9350
>                 URL: https://issues.apache.org/jira/browse/KAFKA-9350
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 2.4.0
>            Reporter: Jason Gustafson
>            Assignee: Jason Gustafson
>            Priority: Major
>
> We have caught this exception a few times in the log:
> {code}
> java.lang.IllegalStateException: Trying to complete a transactional offset commit for producerId 16031 and groupId foo even though the offset commit record itself hasn't been appended to the log.
> 	at kafka.coordinator.group.GroupMetadata.$anonfun$completePendingTxnOffsetCommit$2(GroupMetadata.scala:595)
> 	at scala.collection.mutable.HashMap.$anonfun$foreach$1(HashMap.scala:149)
> 	at scala.collection.mutable.HashTable.foreachEntry(HashTable.scala:237)
> 	at scala.collection.mutable.HashTable.foreachEntry$(HashTable.scala:230)
> 	at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:44)
> 	at scala.collection.mutable.HashMap.foreach(HashMap.scala:149)
> 	at kafka.coordinator.group.GroupMetadata.$anonfun$completePendingTxnOffsetCommit$1(GroupMetadata.scala:592)
> 	at kafka.coordinator.group.GroupMetadata.$anonfun$completePendingTxnOffsetCommit$1$adapted(GroupMetadata.scala:591)
> 	at scala.Option.foreach(Option.scala:274)
> 	at kafka.coordinator.group.GroupMetadata.completePendingTxnOffsetCommit(GroupMetadata.scala:591)
> 	at kafka.coordinator.group.GroupMetadataManager.$anonfun$handleTxnCompletion$2(GroupMetadataManager.scala:828)
> 	at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:253)
> 	at kafka.coordinator.group.GroupMetadata.inLock(GroupMetadata.scala:209)
> 	at kafka.coordinator.group.GroupMetadataManager.$anonfun$handleTxnCompletion$1(GroupMetadataManager.scala:827)
> 	at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
> 	at kafka.coordinator.group.GroupMetadataManager.handleTxnCompletion(GroupMetadataManager.scala:824)
> 	at kafka.coordinator.group.GroupMetadataManager.$anonfun$scheduleHandleTxnCompletion$1(GroupMetadataManager.scala:819)
> 	at kafka.utils.KafkaScheduler.$anonfun$schedule$2(KafkaScheduler.scala:116)
> 	at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:65)
> 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> 	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> 	at java.base/java.lang.Thread.run(Thread.java:834)
> {code}
> It seems that the end transaction marker callback is getting triggered before the offset commit callback. This is puzzling because transaction completion should be tied to a successful TxnOffsetCommit response which depends on completion of the offset commit callback. So it's possible either that there is some case we're missing in the broker or there is some bug in the client. I looked through the logic on both sides and there is no obvious problem.
> In any case, it probably makes sense to let the broker behave more defensively since there is no guarantee that a client won't send EndTxn before receiving a successful TxnOffsetCommit response.
> Note the impact of this bug would tend to not be noticed because usually there is a subsequent offset commit which succeeds. However, in the worst case, it can violate EOS guarantees because it could cause the consumer to revert to a previously committed offset.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)