You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by "Yue Zhang (Jira)" <ji...@apache.org> on 2021/09/15 08:03:00 UTC

[jira] [Created] (HUDI-2435) Tuning clustering job handle errors

Yue Zhang created HUDI-2435:
-------------------------------

             Summary: Tuning clustering job handle errors
                 Key: HUDI-2435
                 URL: https://issues.apache.org/jira/browse/HUDI-2435
             Project: Apache Hudi
          Issue Type: Task
            Reporter: Yue Zhang


Before clustering job/ async clustering finished, hudi will perform errors check using JavaRDD<WriteStatus> writeResponse

It is a collect spark action, when executor is crashed by accident and the cache of JavaRDD<WriteStatus> is lost, so that this collect action will trigger a complete compute and create unexpected marker files or data files.

We should use Option<HoodieCommitMetadata> commitMetadata to do handle errors action instead.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)