[jira] [Created] (FLINK-40689) Kafka sink checkpoint recovery fails after producer epoch rollover with transaction V2

["Aleksandr Savonin (Jira)" <ji...@apache.org>]

Aleksandr Savonin created FLINK-40689:
-----------------------------------------

             Summary: Kafka sink checkpoint recovery fails after producer epoch rollover with transaction V2
                 Key: FLINK-40689
                 URL: https://issues.apache.org/jira/browse/FLINK-40689
             Project: Flink
          Issue Type: Bug
          Components: Connectors / Kafka
            Reporter: Aleksandr Savonin


Kafka sink checkpoint recovery can repeatedly fail with InvalidPidMappingException after a Kafka V2 transaction completes at producer epoch 32766.
The failure occurs when:
1. Flink checkpoints a pending committable containing the producer ID and epoch.
2. Kafka commits the transaction and rotates the producer ID, resetting the epoch to zero.
3. Flink restores the checkpoint and retries the commit using the saved identity.

The connector restores the producer ID and epoch but does not restore the transaction protocol. A fresh recovery producer therefore constructs an old-protocol EndTxn request. Kafka rejects the previous producer ID instead of applying V2’s idempotent rollover retry handling.
Recovery fails, and subsequent restarts can restore the same checkpoint and fail again. The reproduced transaction’s records are already committed and the observed impact is job availability.
Expected behavior: Recovery should successfully acknowledge an already-committed transaction, including after producer ID rollover.

Reproduction: Verified with Flink 2.2.1, Kafka client 4.2.0, and Kafka broker 4.2.0. The integration test advances actual broker epochs, verifies committed record visibility, and restores the same checkpoint into three fresh committer harnesses. Kafka 3.9 provides the V1 control.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)