You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Zhenqiu Huang (Jira)" <ji...@apache.org> on 2025/01/17 03:07:00 UTC

[jira] [Assigned] (HUDI-8881) Potentail dataloss in Flink hudi sink

     [ https://issues.apache.org/jira/browse/HUDI-8881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Zhenqiu Huang reassigned HUDI-8881:
-----------------------------------

    Assignee: Zhenqiu Huang

> Potentail dataloss in Flink hudi sink
> -------------------------------------
>
>                 Key: HUDI-8881
>                 URL: https://issues.apache.org/jira/browse/HUDI-8881
>             Project: Apache Hudi
>          Issue Type: Improvement
>          Components: flink
>            Reporter: Zhenqiu Huang
>            Assignee: Zhenqiu Huang
>            Priority: Critical
>
>  Let's say checkpoint A is completed, then coodinator start to hudi commit. If there is a write error, then the table will be rolledback and flink job will fail after the exception is thrown. When the job restart from failure, what will happen? From my understanding, the time between kafka offset in checkpoint A -1 and offset in checkpoint A will lose.
> we should check whether WriteMetadataEvent contains write failure once the event is received, if any rollback immediately and throw exception to prevent checkpoint to complete when there is any write failure.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)