You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2021/09/15 08:11:00 UTC

[jira] [Updated] (HUDI-2435) Tuning clustering job handle errors

     [ https://issues.apache.org/jira/browse/HUDI-2435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated HUDI-2435:
---------------------------------
    Labels: pull-request-available  (was: )

> Tuning clustering job handle errors
> -----------------------------------
>
>                 Key: HUDI-2435
>                 URL: https://issues.apache.org/jira/browse/HUDI-2435
>             Project: Apache Hudi
>          Issue Type: Task
>            Reporter: Yue Zhang
>            Priority: Major
>              Labels: pull-request-available
>
> Before clustering job/ async clustering finished, hudi will perform errors check using JavaRDD<WriteStatus> writeResponse
> It is a collect spark action, when executor is crashed by accident and the cache of JavaRDD<WriteStatus> is lost, so that this collect action will trigger a complete compute and create unexpected marker files or data files.
> We should use Option<HoodieCommitMetadata> commitMetadata to do handle errors action instead.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)