[I] Fix incremental cleaning for clean based on LATEST_FILE_VERSIONS [hudi]

["hudi-bot (via GitHub)" <gi...@apache.org>]

hudi-bot opened a new issue, #15442:
URL: https://github.com/apache/hudi/issues/15442

   clean based on LATEST_FILE_VERSIONS can be improved further since incremental clean is not enabled. lets see if we can improvise. 
   
    
   
   context from author:
   
    
   
    
   
   Currently incremental cleaning is run for both KEEP_LATEST_COMMITS, KEEP_LATEST_BY_HOURS
   policies. It is not run when KEEP_LATEST_FILE_VERSIONS.
   
   This can lead to not cleaning files. This PR fixes this problem by enabling incremental cleaning for KEEP_LATEST_FILE_VERSIONS only.
   
   Here is the scenario of the problem:
   
   Say we have 3 committed files in partition-A and we add a new commit in partition-B, and we trigger cleaning for the first time (full partition scan):
    {{partition-A/
   commit-0.parquet
   commit-1.parquet
   commit-2.parquet
   partition-B/
   commit-3.parquet}}
   In the case say we have KEEP_LATEST_COMMITS with CLEANER_COMMITS_RETAINED=3, the cleaner will remove the commit-0.parquet to keep 3 commits.
   For the next cleaning, incremental cleaning will trigger, and won't consider partition-A/ until a new commit change it. In case no later commit changes partition-A then commit-1.parquet will stay forever. However it should be removed by the cleaner.
   
   Now if in case of KEEP_LATEST_FILE_VERSIONS, the cleaner will only keep commit-2.parquet. Then it makes sense that incremental cleaning won't consider partition-A until it is changed. Because there is only one commit.
   
   This is why incremental cleaning should only be enabled with KEEP_LATEST_FILE_VERSIONS
   
   Hope this is clear enough
   
    
   
   ## JIRA info
   
   - Link: https://issues.apache.org/jira/browse/HUDI-4878
   - Type: Improvement


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Fix incremental cleaning for clean based on LATEST_FILE_VERSIONS [hudi]

["hudi-bot (via GitHub)" <gi...@apache.org>]

hudi-bot commented on issue #15442:
URL: https://github.com/apache/hudi/issues/15442#issuecomment-3630117248

   ### Linked PR(s)
   
   - https://github.com/apache/hudi/pull/6498


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org