[jira] [Created] (FLINK-35990) Lingering Transactions with FlinkKafkaProducer after failures & scale-down

["Peter Larsen (Jira)" <ji...@apache.org>]

Peter Larsen created FLINK-35990:
------------------------------------

             Summary: Lingering Transactions with FlinkKafkaProducer after failures & scale-down
                 Key: FLINK-35990
                 URL: https://issues.apache.org/jira/browse/FLINK-35990
             Project: Flink
          Issue Type: Bug
          Components: Connectors / Kafka
    Affects Versions: 1.17.2, 1.14.3
            Reporter: Peter Larsen


Hi! I’ve recently hit some issues with lingering transactions not getting aborted by FlinkKafkaProducer on 1.14.3. The failure seems to be triggered by a failed restart from a checkpoint, then restarting with lower parallelism. I made a test that I think reproduces the issue and pushed it up to a fork [here|https://github.com/peterdlarsen/flink/compare/peterdlarsen:c0027e5...peterdlarsen:b4c4750]. I also reproduced on a local cluster with 1.14.3 and am happy to share more details if that’s useful!

I’m assuming migrating to KafkaSink is the recommended remediation as opposed to fixing, but wanted to report in case it’s helpful to anyone else.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)