You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@flink.apache.org by "Prabhu Joseph (Jira)" <ji...@apache.org> on 2023/12/06 09:04:00 UTC

[jira] [Updated] (FLINK-33753) ContinuousFileReaderOperator consume records as mini batch

     [ https://issues.apache.org/jira/browse/FLINK-33753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Prabhu Joseph updated FLINK-33753:
----------------------------------
    Affects Version/s: 1.16.0
                           (was: 1.18.0)

> ContinuousFileReaderOperator consume records as mini batch
> ----------------------------------------------------------
>
>                 Key: FLINK-33753
>                 URL: https://issues.apache.org/jira/browse/FLINK-33753
>             Project: Flink
>          Issue Type: Improvement
>    Affects Versions: 1.16.0
>            Reporter: Prabhu Joseph
>            Priority: Major
>
> The ContinuousFileReaderOperator reads and collects the records from a split in a loop. If the split size is large, then the loop will take more time, and then the mailbox executor won't have a chance to process the checkpoint barrier. This leads to checkpoint timing out. ContinuousFileReaderOperator could be improved to consume the records in a mini batch, similar to Hudi's StreamReadOperator (https://issues.apache.org/jira/browse/HUDI-2485).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)