[jira] [Created] (KAFKA-21181) Transaction coordinator could return INVALID_TXN_STATE from stale cache after losing __transaction_state leadership

["Ritika Reddy (Jira)" <ji...@apache.org>]

Ritika Reddy created KAFKA-21181:
------------------------------------

             Summary: Transaction coordinator could return INVALID_TXN_STATE from stale cache after losing __transaction_state leadership
                 Key: KAFKA-21181
                 URL: https://issues.apache.org/jira/browse/KAFKA-21181
             Project: Kafka
          Issue Type: Bug
    Affects Versions: 4.3.0, 4.2.0
            Reporter: Ritika Reddy
            Assignee: Ritika Reddy
             Fix For: 4.3.0, 4.2.0


When a broker loses leadership of a __transaction_state partition, its TransactionStateManager keeps the partition's cache entry until TransactionCoordinator.onResignation runs. In BrokerMetadataPublisher.onMetadataUpdate, that happens after replicaManager.applyDelta and after the group coordinator update. So there's a window in which the broker is already a follower for the partition but still answers coordinator requests from its old cache.

TransactionStateManager.getAndMaybeAddTransactionState (trunk lines 400-421) only checks loadingPartitions and transactionMetadataCache; it never checks that the broker still leads the partition. Answers that require a log write are safe, because the acks=-1 append fails with NOT_LEADER_OR_FOLLOWER, which is mapped to NOT_COORDINATOR. Answers decided purely from the cache are not.

With TV2 this surfaces as a fatal error for the producer:
1. Transaction N commits on coordinator A. The cache on A holds COMPLETE_COMMIT at epoch E, and the client now has epoch E.
2. Leadership of the __transaction_state partition moves to broker B. B loads the partition, and the partition leaders' implicit AddPartitionsToTxn for transaction N+1 go to B, which moves the transaction to ONGOING.
3. The client hasn't had any reason to rediscover the coordinator: under TV2 its first coordinator request in the transaction is EndTxn. So it sends EndTxn(COMMIT, E) to A.
4. A is already a follower but hasn't resigned yet. It answers from the cache: COMPLETE_COMMIT + COMMIT at the current epoch is not a retry, so it returns INVALID_TXN_STATE without writing to the log:
TransactionalId: <id>'s state is COMPLETE_COMMIT, but received transaction marker result to send: COMMIT
5. The producer treats INVALID_TXN_STATE as fatal. The transaction on B stays ONGOING until it times out.

The correct response is NOT_COORDINATOR, which would make the client rediscover B and commit. We've seen gaps of several seconds between the broker becoming follower and resigning, and in one case ~37 s, with no broker pause involved.

Other cache-only answers are exposed the same way, for example PRODUCER_FENCED / INVALID_PRODUCER_ID_MAPPING, the verify-only AddPartitionsToTxn path, and the "partitions already added" shortcut.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)