You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by "Prashant Wason (Jira)" <ji...@apache.org> on 2021/06/15 06:00:00 UTC

[jira] [Created] (HUDI-2013) Fallback to file listing may lead to data loss

Prashant Wason created HUDI-2013:
------------------------------------

             Summary: Fallback to file listing may lead to data loss
                 Key: HUDI-2013
                 URL: https://issues.apache.org/jira/browse/HUDI-2013
             Project: Apache Hudi
          Issue Type: Bug
            Reporter: Prashant Wason
            Assignee: Prashant Wason


When fallback to file listing mode is enabled (hoodie.metadata.fallback.enable, default is true), then if listing from the metadata table leads to an exception the normal file-system listing used.

Metadata table listing may fail if the table is inconsistent or due to bugs. Falling back to file listing has the following downsides:
 # It masks the issue as the commit does not fail (only an exception is logged).
 # By the time the issue is discovered, logs may have been lost
 # There is no guarantee that all the commits wrote/updated the correct files.

Since listing from metadata table is per-partition, the issue is further complicated when listing for some partitions succeeds (file-list retrieved from metadata table) and fails for other partitions (file list retrieved from filesystem).

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)