You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flink.apache.org by Hang Ruan <ru...@gmail.com> on 2023/02/17 11:19:17 UTC

Should the sink writer and committer always be chained together

Hi all,

I see that some users are finding the way to disable the chain of sink
writer and committer [1]. This makes me think about the bug FLINK-29492
[2], which is caused by the operator chaining disabled. When operator
chaining disabled, the kafka committer can not return the producers to the
kafka writer. The `closer` in kafka writer will register a new producer
every time the method `getOrCreateTransactionalProducer` is invoked. This
will cause the OOM error.

I wonder if we should assume the writer and committer will always be
chained together. Or should we force them to be chained together?

If the writer and committer are allowed not to be chained, how do the
writers notice that the committer is not chained in order to fix the
FLINK-29492 [2]?

Best,
Hang

[1] https://lists.apache.org/thread/v06c9pq6c119bdwhord71rtvb2wybx1h
[2] https://issues.apache.org/jira/browse/FLINK-29492