You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Purushotham Pushpavanthar (Jira)" <ji...@apache.org> on 2023/01/16 07:13:00 UTC
[jira] [Commented] (HUDI-2511) Aggressive archival configs compared to cleaner configs make cleaning moot
[ https://issues.apache.org/jira/browse/HUDI-2511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17677191#comment-17677191 ]
Purushotham Pushpavanthar commented on HUDI-2511:
-------------------------------------------------
[~shivnarayan] this issue is still appearing in Hudi 0.11.1. I'm using `hoodie.cleaner.policy=KEEP_LATEST_BY_HOURS`. Though, we have validation checks here `org.apache.hudi.config.HoodieWriteConfig.Builder#validate()` but it only takes care of validating `hoodie.cleaner.commits.retained` against `min and max commits`. We might need extra checks for the above policy.
> Aggressive archival configs compared to cleaner configs make cleaning moot
> --------------------------------------------------------------------------
>
> Key: HUDI-2511
> URL: https://issues.apache.org/jira/browse/HUDI-2511
> Project: Apache Hudi
> Issue Type: Improvement
> Reporter: sivabalan narayanan
> Assignee: sivabalan narayanan
> Priority: Blocker
> Labels: core-flow-ds, sev:high, user-support-issues
>
> if hoodie.keep.max.commits <= hoodie.cleaner.commits.retained, then cleaner will never kick in only. Bcoz, by then archival will kick in and will move entries from active to archived.
> We need to revisit this and either throw exception or make cleaner also look into archived commits.
> Related issue: [https://github.com/apache/hudi/issues/3739]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)