[jira] [Created] (HUDI-4773) Adding filter mode to Clustering to filter for recent files

["sivabalan narayanan (Jira)" <ji...@apache.org>]

sivabalan narayanan created HUDI-4773:
-----------------------------------------

             Summary: Adding filter mode to Clustering to filter for recent files
                 Key: HUDI-4773
                 URL: https://issues.apache.org/jira/browse/HUDI-4773
             Project: Apache Hudi
          Issue Type: Improvement
          Components: clustering
            Reporter: sivabalan narayanan


We have partition aware clustering strategy and recent partitions based strategy as well for clustering. This plays out well if partitioning is based on dates. but what incase partitioning is based on some other random field. 

 

So, we might need another clustering filtering strategy to consider only those file groups which got touched in the last N commits. 

For eg, if a user configures clustering to run every 5 commits, every time clustering runs, it will consider only the file groups touched in the last 5 commits. This will avoid triggering repeated clustering for already clustered file groups as well and clustering will be very fast only delta file groups are considered. 

 

 

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)