You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by "Prashant Wason (Jira)" <ji...@apache.org> on 2021/02/10 23:03:00 UTC

[jira] [Created] (HUDI-1611) Allow directories to be filtered during the bootstrap of the metadata table

Prashant Wason created HUDI-1611:
------------------------------------

             Summary: Allow directories to be filtered during the bootstrap of the metadata table
                 Key: HUDI-1611
                 URL: https://issues.apache.org/jira/browse/HUDI-1611
             Project: Apache Hudi
          Issue Type: Sub-task
            Reporter: Prashant Wason
            Assignee: Prashant Wason


During the bootstrap of the Metadata Table, all the directories which contain the partition metadata directory are assumed to be partitions and are added to the metadata table.

In our HDFS clusters, we have directories like .backup, .temp which are used by various teams for non-hoodie purposes (e.g. .backup may be keeping a snapshot of the dataset). During bootstrap, Metadata Table ends up containing all those paths also as partitions.

In this patch, I would like to introduce a configuration for HoodieMetadataConfig to filter out some directories based on a regular expression string. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)