[jira] [Created] (KAFKA-17831) Transaction coordinators returning COORDINATOR_LOAD_IN_PROGRESS until leader changes or brokers are restarted after network instability

["Kay Hartmann (Jira)" <ji...@apache.org>]

Kay Hartmann created KAFKA-17831:
------------------------------------

             Summary: Transaction coordinators returning COORDINATOR_LOAD_IN_PROGRESS until leader changes or brokers are restarted after network instability
                 Key: KAFKA-17831
                 URL: https://issues.apache.org/jira/browse/KAFKA-17831
             Project: Kafka
          Issue Type: Bug
          Components: core
    Affects Versions: 3.7.1, 3.6.1
            Reporter: Kay Hartmann


After experiencing a (heavy) network outage/instability, our brokers arrived in a state where some producers were not able to perform transactions, but the brokers continued to respond to those producers with  `COORDINATOR_LOAD_IN_PROGRESS`. We were able to see corresponding DEBUG logs in the brokers:
{code:java}
2024-08-06 15:22:01,178 DEBUG [TransactionCoordinator id=11] Returning COORDINATOR_LOAD_IN_PROGRESS error code to client for my-client's AddPartitions request (kafka.coordinator.transaction.TransactionCoordinator) [data-plane-kafka-request-handler-5] {code}
This did not occur for all transactions, but for a subset of transactional ids with the same hash that would go through the same transaction coordinator/partition leader for the corresponding `__transaction_state` partition. We were able to resolve this the first time by shifting partition leaders for the transaction topic around and the second time by simply restarting brokers.

 

This lead us to believe that it has to be some kind of dirty in-memory state transaction coordinators have for a `__transaction_state` partition. We found two cases ([#1|https://github.com/apache/kafka/blob/3.6.1/core/src/main/scala/kafka/coordinator/transaction/TransactionStateManager.scala#L319], [#2|https://github.com/apache/kafka/blob/3.6.1/core/src/main/scala/kafka/coordinator/transaction/TransactionStateManager.scala#L376]) in which the TransactionStateManager returns `COORDINATOR_LOAD_IN_PROGRESS`. In both cases `loadingPartitions` has some state that signals that the TransactionStateManager is still occupied with initializing transactional data for that `__transaction_state` partition.

We believe that the network outage caused partition leaders to be shifted around continuously between their replicas and somehow this lead to outdated data in `loadingPartitions` that wasn't cleaned up. I had a look at the [method|https://github.com/apache/kafka/blob/3.6.1/core/src/main/scala/kafka/coordinator/transaction/TransactionStateManager.scala#L518] where it is updated and cleaned, but wasn't able to identify a case in which there could be a failure to clean.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)