[jira] [Commented] (FLINK-34693) Memory leak in KafkaWriter

["Sylwester Lachiewicz (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-34693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18113945#comment-18113945 ] 

Sylwester Lachiewicz commented on FLINK-34693:
----------------------------------------------

Triage finding (2026-09-10, connector main at 48ddbf1f):

The retained objects in the attached heap dump ({{java.lang.Object[9019]}}, 1.54 GB) sit in {{KafkaWriter.producerCloseables}}, an append-only deque that is only drained by {{close()}}. A producer that leaves the pool through a committable and never comes back stays referenced there for the life of the writer, which is what happens on 3.x whenever writer and committer are not chained, because {{KafkaCommittableSerializer}} carries only the transactional id, producer id and epoch. This is the mechanism Robert Metzger described on [FLINK-29492|https://issues.apache.org/jira/browse/FLINK-29492].

Per line: absent on main, 5.0 and 4.0, where [FLINK-37282|https://issues.apache.org/jira/browse/FLINK-37282] ([df353e9d|https://github.com/apache/flink-connector-kafka/commit/df353e9d2543eb5babb325a4f9895fe89623a53f], [f4015d17|https://github.com/apache/flink-connector-kafka/commit/f4015d17bd204b8decaf8d5337de011107e808e4]) replaced the deque with a producer pool and [FLINK-37613|https://issues.apache.org/jira/browse/FLINK-37613] closed the remaining failure path. Present unchanged on the live v3.4 branch ([KafkaWriter.java|https://github.com/apache/flink-connector-kafka/blob/v3.4/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/sink/KafkaWriter.java], field plus three add and closeAll sites).

Recommendation: keep open for a v3.4-only fix and link FLINK-29492, FLINK-37282 and FLINK-37613. FLINK-37282 does not cherry-pick (31 files, +2637/-689 in the sink package between 3.4.0 and 4.0.0). A scoped change that stops tracking a producer in {{producerCloseables}} once it is handed to a committable, and closes the intermediate producers created by the transactional id gap-fill loop in {{getTransactionalProducer()}}, is in the range of 50 to 100 lines. I can prepare that if a 3.4.x release is still planned.

_This comment was created with AI assistance._

> Memory leak in KafkaWriter
> --------------------------
>
>                 Key: FLINK-34693
>                 URL: https://issues.apache.org/jira/browse/FLINK-34693
>             Project: Flink
>          Issue Type: Bug
>          Components: Connectors / Kafka
>    Affects Versions: kafka-3.1.0, kafka-3.0.2
>            Reporter: Stefan Richter
>            Priority: Blocker
>         Attachments: image-2024-03-15-10-30-08-280.png
>
>
> KafkaWriter is keeping objects in Dequeue of closeables ({{{}producerCloseables{}}}) that are never removed so that the can be GC’ed.
> From heap dump:!04599375-f923-4d1a-8d68-9e17e54b363c#media-blob-url=true&id=9d1e022e-8762-45b3-877b-d298ec956078&collection=&contextId=870337&height=306&width=2284&alt=!  !image-2024-03-15-10-30-08-280.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)