You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by "Bibhu Pala (Jira)" <ji...@apache.org> on 2024/09/02 05:57:00 UTC

[jira] [Assigned] (HUDI-6352) KEEP_LATEST_BY_HOURS should consider modified time instead of commit time while setting earliestCommitToRetain value

     [ https://issues.apache.org/jira/browse/HUDI-6352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Bibhu Pala reassigned HUDI-6352:
--------------------------------

    Assignee: Bibhu Pala

> KEEP_LATEST_BY_HOURS should consider modified time instead of commit time while setting earliestCommitToRetain value
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: HUDI-6352
>                 URL: https://issues.apache.org/jira/browse/HUDI-6352
>             Project: Apache Hudi
>          Issue Type: Bug
>          Components: cleaning, table-service
>            Reporter: Surya Prasanna Yalla
>            Assignee: Bibhu Pala
>            Priority: Major
>              Labels: pull-request-available
>
> In CleanPlanner, KEEP_LATEST_BY_HOURS is setting earliestCommitToRetain value by consider timestamp directly, this will introduce bug if there are out of order commits where commit with lower timestamp is completed much later than commits with higher timestamps.
> This policy's implementation needs to be revisit.
> It should basically store the timestamp until which it cleaned let this be t1. Next cleaner instant should consider all the partitions and files that are modified from the point of t1 until (currentime-x) hours. Whichever files are not valid they should be removed.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)