You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by "Ruguo Yu (Jira)" <ji...@apache.org> on 2023/12/11 07:07:00 UTC

[jira] [Updated] (HUDI-7209) Add configuration to skip not exists file in streaming read

     [ https://issues.apache.org/jira/browse/HUDI-7209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ruguo Yu updated HUDI-7209:
---------------------------
    Fix Version/s: 1.0.0

> Add configuration to skip not exists file in streaming read
> -----------------------------------------------------------
>
>                 Key: HUDI-7209
>                 URL: https://issues.apache.org/jira/browse/HUDI-7209
>             Project: Apache Hudi
>          Issue Type: Improvement
>          Components: flink
>            Reporter: Ruguo Yu
>            Priority: Minor
>             Fix For: 1.0.0
>
>
> In `streaming reading`, if there are a large number of files in metada, especially archive files that are very old, then it is IO-intensive to determine whether the file exists during the file traversal process. In extreme cases, flink checkpoint may not be completed.
> <img width="1074" alt="image" src="https://github.com/apache/hudi/assets/13013780/f25cda8d-e75c-4380-b660-8ad347c4a6ca">
> Another potential problem is that if deleted files are skipped by default, is there a problem of missing data and the user is not aware of it?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)