[jira] [Created] (FLINK-31914) Failing to close FlinkKafkaInternalProducer created in KafkaWriter with exactly-once semantic results in memory leak

["datariver (Jira)" <ji...@apache.org>]

datariver created FLINK-31914:
---------------------------------

             Summary: Failing to close FlinkKafkaInternalProducer created in KafkaWriter with exactly-once semantic results in memory leak
                 Key: FLINK-31914
                 URL: https://issues.apache.org/jira/browse/FLINK-31914
             Project: Flink
          Issue Type: Bug
    Affects Versions: 1.15.0
            Reporter: datariver
         Attachments: image-2023-04-24-16-11-22-251.png

Hi [~arvid] , If Exactly-Once writing is enabled, Kafka's transactional writing will be used. KafkaWriter will create FlinkKafkaInternalProducer in the initialization and snapshotState methods, but there is no place to close it. As Checkpoints increase, Producers will continue to accumulate. Each Producer maintains a Buffer, which will cause memory leaks and Job OOM.
By dumping an in-memory instance of Task Manager, you can see that there are a lot of Producers:

!image-2023-04-24-16-11-22-251.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)