You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "nsivabalan (via GitHub)" <gi...@apache.org> on 2023/04/11 00:51:53 UTC
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8422: [HUDI-6056] Validate archival configs alignment with cleaner configs with policy based on hours
nsivabalan commented on code in PR #8422:
URL: https://github.com/apache/hudi/pull/8422#discussion_r1162197182
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -3030,24 +3030,26 @@ private void validate() {
.equals(HoodieFailedWritesCleaningPolicy.EAGER.name()), "To enable optimistic concurrency control, set hoodie.cleaner.policy.failed.writes=LAZY");
}
- HoodieCleaningPolicy.valueOf(writeConfig.getString(CLEANER_POLICY));
- // Ensure minInstantsToKeep > cleanerCommitsRetained, otherwise we will archive some
- // commit instant on timeline, that still has not been cleaned. Could miss some data via incr pull
- int minInstantsToKeep = Integer.parseInt(writeConfig.getStringOrDefault(HoodieArchivalConfig.MIN_COMMITS_TO_KEEP));
- int maxInstantsToKeep = Integer.parseInt(writeConfig.getStringOrDefault(HoodieArchivalConfig.MAX_COMMITS_TO_KEEP));
- int cleanerCommitsRetained =
- Integer.parseInt(writeConfig.getStringOrDefault(HoodieCleanConfig.CLEANER_COMMITS_RETAINED));
- checkArgument(maxInstantsToKeep > minInstantsToKeep,
- String.format(
- "Increase %s=%d to be greater than %s=%d.",
- HoodieArchivalConfig.MAX_COMMITS_TO_KEEP.key(), maxInstantsToKeep,
- HoodieArchivalConfig.MIN_COMMITS_TO_KEEP.key(), minInstantsToKeep));
- checkArgument(minInstantsToKeep > cleanerCommitsRetained,
- String.format(
- "Increase %s=%d to be greater than %s=%d. Otherwise, there is risk of incremental pull "
- + "missing data from few instants.",
- HoodieArchivalConfig.MIN_COMMITS_TO_KEEP.key(), minInstantsToKeep,
- HoodieCleanConfig.CLEANER_COMMITS_RETAINED.key(), cleanerCommitsRetained));
+ HoodieCleaningPolicy cleaningPolicy = HoodieCleaningPolicy.valueOf(writeConfig.getString(CLEANER_POLICY));
+ if (cleaningPolicy == HoodieCleaningPolicy.KEEP_LATEST_COMMITS) {
Review Comment:
NTR(note to reviewer) : no changes except that this validation is done only incase of KEEP LATEST COMMITS cleaner policy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org