[jira] [Commented] (FLINK-31914) Failing to close FlinkKafkaInternalProducer created in KafkaWriter with exactly-once semantic results in memory leak

["chouc (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-31914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17716059#comment-17716059 ] 

chouc commented on FLINK-31914:
-------------------------------

{color:#000000}The FlinkKafkaInternalProducer is registered in Closer and closer is closed when Flink Writer invoke close.

{color}

> Failing to close FlinkKafkaInternalProducer created in KafkaWriter with exactly-once semantic results in memory leak
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-31914
>                 URL: https://issues.apache.org/jira/browse/FLINK-31914
>             Project: Flink
>          Issue Type: Bug
>          Components: Connectors / Kafka
>    Affects Versions: 1.15.0, 1.16.0, 1.17.0
>            Reporter: datariver
>            Priority: Major
>         Attachments: image-2023-04-24-16-11-22-251.png
>
>
> Hi [~arvid] , If Exactly-Once writing is enabled, Kafka's transactional writing will be used. KafkaWriter will create FlinkKafkaInternalProducer in the initialization and snapshotState methods, but there is no place to close it. As Checkpoints increase, Producers will continue to accumulate. Each Producer maintains a Buffer, which will cause memory leaks and Job OOM.
> By dumping an in-memory instance of Task Manager, you can see that there are a lot of Producers:
> !image-2023-04-24-16-11-22-251.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)