You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@flink.apache.org by "Zhipeng Zhang (Jira)" <ji...@apache.org> on 2023/04/24 06:17:00 UTC

[jira] [Created] (FLINK-31901) AbstractBroadcastWrapperOperator should not block checkpoint barriers when processing cached records

Zhipeng Zhang created FLINK-31901:
-------------------------------------

             Summary: AbstractBroadcastWrapperOperator should not block checkpoint barriers when processing cached records
                 Key: FLINK-31901
                 URL: https://issues.apache.org/jira/browse/FLINK-31901
             Project: Flink
          Issue Type: Improvement
          Components: Library / Machine Learning
            Reporter: Zhipeng Zhang
             Fix For: ml-2.3.0


Currently `BroadcastUtils#withBroadcast` tries to caches the non-broadcast input until the broadcast inputs are all processed. After the broadcast variables are ready, we first process the cached records and then continue to process the newly arrived records.

 

Processing cached elements is invoked via `Input#processElement` and `Input#processWatermark`.  However, processing cached element may take a long time since there may be many cached records, which could potentially block the checkpoint barrier.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)