You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "lokeshj1703 (via GitHub)" <gi...@apache.org> on 2024/06/12 13:11:00 UTC
[PR] [ENG-10413] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
lokeshj1703 opened a new pull request, #11440:
URL: https://github.com/apache/hudi/pull/11440
### Change Logs
The PR adds changes such that:
Cleaner will also track additional metadata called "EarliestCommitToNotArchive". Whose value will either refer to first savepoint if first savepoint < earliest commit to retain. Or it will refer to "earliestCommitToRetain".
Archival will guard itself against "EarliestCommitToNotArchive" from latest completed clean.
### Impact
NA
### Risk level (write none, low medium or high below)
low
### Documentation Update
NA
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11440:
URL: https://github.com/apache/hudi/pull/11440#issuecomment-2233938345
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24366",
"triggerID" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"triggerType" : "PUSH"
}, {
"hash" : "4bb072faa37b2ea398144fbbc24deff966153cfa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24471",
"triggerID" : "4bb072faa37b2ea398144fbbc24deff966153cfa",
"triggerType" : "PUSH"
}, {
"hash" : "7e8e5336ab1f079d93ccaa0981e70798169532f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24807",
"triggerID" : "7e8e5336ab1f079d93ccaa0981e70798169532f1",
"triggerType" : "PUSH"
}, {
"hash" : "15b2428c2dd7d7980fd98b78643811652e08fd35",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24823",
"triggerID" : "15b2428c2dd7d7980fd98b78643811652e08fd35",
"triggerType" : "PUSH"
}, {
"hash" : "0ccf2a1257697222f488723c7ae3fde9cd7d6b8d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24861",
"triggerID" : "0ccf2a1257697222f488723c7ae3fde9cd7d6b8d",
"triggerType" : "PUSH"
}, {
"hash" : "228d9a1568699f5fa9c7372af6159b78c9189256",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24866",
"triggerID" : "228d9a1568699f5fa9c7372af6159b78c9189256",
"triggerType" : "PUSH"
}, {
"hash" : "56188d58f0289b2c6abcf6153bd995746b7f8307",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "56188d58f0289b2c6abcf6153bd995746b7f8307",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 228d9a1568699f5fa9c7372af6159b78c9189256 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24866)
* 56188d58f0289b2c6abcf6153bd995746b7f8307 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #11440:
URL: https://github.com/apache/hudi/pull/11440#discussion_r1642831308
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/timeline/HoodieTimelineArchiver.java:
##########
@@ -362,6 +367,25 @@ private boolean deleteArchivedInstants(List<ActiveAction> activeActions, HoodieE
return true;
}
+ private Option<String> getEarliestCommitToNotArchive(HoodieActiveTimeline activeTimeline, HoodieTableMetaClient metaClient) throws IOException {
+ // if clean policy is based on file versions, earliest commit to not archive may not be set. So, we have to explicitly check the savepoint timeline
+ // and guard against the first one.
+ String earliestInstantToNotArchive = table.getCompletedSavepointTimeline().firstInstant().map(HoodieInstant::getTimestamp).orElse(null);
+ if (config.getCleanerPolicy() != HoodieCleaningPolicy.KEEP_LATEST_FILE_VERSIONS) {
+ Option<HoodieInstant> cleanInstantOpt =
+ activeTimeline.getCleanerTimeline().filterCompletedInstants().lastInstant();
+ if (cleanInstantOpt.isPresent()) {
+ HoodieInstant cleanInstant = cleanInstantOpt.get();
+ String cleanerEarliestInstantToNotArchive = CleanerUtils.getCleanerPlan(metaClient, cleanInstant.isRequested()
+ ? cleanInstant
+ : HoodieTimeline.getCleanRequestedInstant(cleanInstant.getTimestamp()))
+ .getExtraMetadata().getOrDefault(CleanPlanner.EARLIEST_COMMIT_TO_NOT_ARCHIVE, null);
Review Comment:
getExtraMetadata() can return null and will NPE here. Check if the map exists before doing getOrDefault
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11440:
URL: https://github.com/apache/hudi/pull/11440#issuecomment-2229097599
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24366",
"triggerID" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"triggerType" : "PUSH"
}, {
"hash" : "4bb072faa37b2ea398144fbbc24deff966153cfa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24471",
"triggerID" : "4bb072faa37b2ea398144fbbc24deff966153cfa",
"triggerType" : "PUSH"
}, {
"hash" : "7e8e5336ab1f079d93ccaa0981e70798169532f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24807",
"triggerID" : "7e8e5336ab1f079d93ccaa0981e70798169532f1",
"triggerType" : "PUSH"
}, {
"hash" : "15b2428c2dd7d7980fd98b78643811652e08fd35",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24823",
"triggerID" : "15b2428c2dd7d7980fd98b78643811652e08fd35",
"triggerType" : "PUSH"
}, {
"hash" : "0ccf2a1257697222f488723c7ae3fde9cd7d6b8d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0ccf2a1257697222f488723c7ae3fde9cd7d6b8d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 15b2428c2dd7d7980fd98b78643811652e08fd35 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24823)
* 0ccf2a1257697222f488723c7ae3fde9cd7d6b8d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #11440:
URL: https://github.com/apache/hudi/pull/11440
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #11440:
URL: https://github.com/apache/hudi/pull/11440#discussion_r1674895597
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanPlanActionExecutor.java:
##########
@@ -149,17 +150,24 @@ HoodieCleanerPlan requestClean(HoodieEngineContext context) {
.map(x -> new HoodieActionInstant(x.getTimestamp(), x.getAction(), x.getState().name())).orElse(null),
planner.getLastCompletedCommitTimestamp(),
config.getCleanerPolicy().name(), Collections.emptyMap(),
- CleanPlanner.LATEST_CLEAN_PLAN_VERSION, cleanOps, partitionsToDelete, prepareExtraMetadata(planner.getSavepointedTimestamps()));
+ CleanPlanner.LATEST_CLEAN_PLAN_VERSION, cleanOps, partitionsToDelete, prepareExtraMetadata(planner.getSavepointedTimestamps(), planner.getEarliestCommitToNotArchive()));
} catch (IOException e) {
throw new HoodieIOException("Failed to schedule clean operation", e);
}
}
- private Map<String, String> prepareExtraMetadata(List<String> savepointedTimestamps) {
- if (savepointedTimestamps.isEmpty()) {
+ private Map<String, String> prepareExtraMetadata(List<String> savepointedTimestamps, Option<String> earliestCommitToNotArchive) {
+ if (savepointedTimestamps.isEmpty() && !earliestCommitToNotArchive.isPresent()) {
return Collections.emptyMap();
} else {
- return Collections.singletonMap(SAVEPOINTED_TIMESTAMPS, savepointedTimestamps.stream().collect(Collectors.joining(",")));
+ Map<String, String> extraMetadata = new HashMap<>();
+ if (!savepointedTimestamps.isEmpty()) {
+ extraMetadata.put(SAVEPOINTED_TIMESTAMPS, savepointedTimestamps.stream().collect(Collectors.joining(",")));
+ }
+ if (earliestCommitToNotArchive.isPresent()) {
+ extraMetadata.put(EARLIEST_COMMIT_TO_NOT_ARCHIVE, earliestCommitToNotArchive.get());
Review Comment:
Can we put the name more straight-forward like `SKIPPED_SAVEPOINTED_COMMITS`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #11440:
URL: https://github.com/apache/hudi/pull/11440#discussion_r1681475904
##########
hudi-common/src/main/java/org/apache/hudi/common/util/ClusteringUtils.java:
##########
@@ -376,9 +381,75 @@ public static Option<HoodieInstant> getEarliestInstantToRetainForClustering(
oldestInstantToRetain = replaceOrClusterTimeline.firstInstant();
}
}
+ oldestInstantToRetain = Option.ofNullable(
+ HoodieTimeline.minTimestampInstant(oldestInstantToRetain.orElse(null), earliestSavepointInClean.orElse(null)));
return oldestInstantToRetain;
}
+ public static Option<HoodieInstant> getEarliestSavepointInClean(HoodieActiveTimeline activeTimeline, HoodieTableMetaClient metaClient, HoodieCleaningPolicy cleanerPolicy,
+ Option<HoodieInstant> cleanInstantOpt, boolean shouldArchiveBeyondSavepoint) throws IOException {
+ // EarliestSavepoint in clean is required to block archival when savepoint is deleted.
+ // This ensures that archival is blocked until clean has cleaned up files retained due to savepoint.
+ // If this guard is not present, the archival of commits can lead to duplicates. Here is a scenario
+ // illustrating the same. This scenario considers a case where EarliestSavepoint guard is not present
+ // c1.dc - f1 (c1 deltacommit creates file with id f1)
+ // c2.dc - f2 (c2 deltacommit creates file with id f2)
+ // c2.sp - Savepoint at c2
+ // c3.rc (replacing f2 -> f3) (Replace commit replacing file id f2 with f3)
+ // c4.dc
+ //
+ // Lets say Incremental cleaner moved past the c3.rc without cleaning f2 since savepoint is created at c2.
+ // Archival is blocked at c2 since there is a savepoint at c2.
+ // Lets say the savepoint at c2 is now deleted, Archival would archive c3.rc since it is unblocked now.
+ // Since c3 is archived and f2 has not been cleaned, the table view would be considering f2 as a valid
+ // file id. This causes duplicates.
+ if (shouldArchiveBeyondSavepoint) {
+ // When archive beyond savepoint is enabled, we do not block the archival based on cleaner earliestSavepoint
+ return Option.empty();
+ }
+ // explicitly check the savepoint timeline and guard against the first one
+ Option<String> firstSavepointOpt = activeTimeline.getSavePointTimeline().filterCompletedInstants().firstInstant().map(HoodieInstant::getTimestamp);
+ String earliestSavepointTs = firstSavepointOpt.orElse(null);
+ Option<String> cleanerEarliestSavepoint = Option.empty();
+ Option<HoodieActionInstant> cleanerEarliestToRetain = Option.empty();
+
+ if (cleanerPolicy != HoodieCleaningPolicy.KEEP_LATEST_FILE_VERSIONS && cleanInstantOpt.isPresent()) {
Review Comment:
@nsivabalan Is specific handling for `KEEP_LATEST_FILE_VERSIONS` still required or can we keep logic same for all cleaner policies? It seems like SAVEPOINTED_TIMESTAMPS would always be created.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11440:
URL: https://github.com/apache/hudi/pull/11440#issuecomment-2221276113
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24366",
"triggerID" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"triggerType" : "PUSH"
}, {
"hash" : "4bb072faa37b2ea398144fbbc24deff966153cfa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24471",
"triggerID" : "4bb072faa37b2ea398144fbbc24deff966153cfa",
"triggerType" : "PUSH"
}, {
"hash" : "7e8e5336ab1f079d93ccaa0981e70798169532f1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24807",
"triggerID" : "7e8e5336ab1f079d93ccaa0981e70798169532f1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7e8e5336ab1f079d93ccaa0981e70798169532f1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24807)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11440:
URL: https://github.com/apache/hudi/pull/11440#issuecomment-2230535145
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24366",
"triggerID" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"triggerType" : "PUSH"
}, {
"hash" : "4bb072faa37b2ea398144fbbc24deff966153cfa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24471",
"triggerID" : "4bb072faa37b2ea398144fbbc24deff966153cfa",
"triggerType" : "PUSH"
}, {
"hash" : "7e8e5336ab1f079d93ccaa0981e70798169532f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24807",
"triggerID" : "7e8e5336ab1f079d93ccaa0981e70798169532f1",
"triggerType" : "PUSH"
}, {
"hash" : "15b2428c2dd7d7980fd98b78643811652e08fd35",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24823",
"triggerID" : "15b2428c2dd7d7980fd98b78643811652e08fd35",
"triggerType" : "PUSH"
}, {
"hash" : "0ccf2a1257697222f488723c7ae3fde9cd7d6b8d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24861",
"triggerID" : "0ccf2a1257697222f488723c7ae3fde9cd7d6b8d",
"triggerType" : "PUSH"
}, {
"hash" : "228d9a1568699f5fa9c7372af6159b78c9189256",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "228d9a1568699f5fa9c7372af6159b78c9189256",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0ccf2a1257697222f488723c7ae3fde9cd7d6b8d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24861)
* 228d9a1568699f5fa9c7372af6159b78c9189256 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11440:
URL: https://github.com/apache/hudi/pull/11440#issuecomment-2237118823
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24366",
"triggerID" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"triggerType" : "PUSH"
}, {
"hash" : "4bb072faa37b2ea398144fbbc24deff966153cfa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24471",
"triggerID" : "4bb072faa37b2ea398144fbbc24deff966153cfa",
"triggerType" : "PUSH"
}, {
"hash" : "7e8e5336ab1f079d93ccaa0981e70798169532f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24807",
"triggerID" : "7e8e5336ab1f079d93ccaa0981e70798169532f1",
"triggerType" : "PUSH"
}, {
"hash" : "15b2428c2dd7d7980fd98b78643811652e08fd35",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24823",
"triggerID" : "15b2428c2dd7d7980fd98b78643811652e08fd35",
"triggerType" : "PUSH"
}, {
"hash" : "0ccf2a1257697222f488723c7ae3fde9cd7d6b8d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24861",
"triggerID" : "0ccf2a1257697222f488723c7ae3fde9cd7d6b8d",
"triggerType" : "PUSH"
}, {
"hash" : "228d9a1568699f5fa9c7372af6159b78c9189256",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24866",
"triggerID" : "228d9a1568699f5fa9c7372af6159b78c9189256",
"triggerType" : "PUSH"
}, {
"hash" : "56188d58f0289b2c6abcf6153bd995746b7f8307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24881",
"triggerID" : "56188d58f0289b2c6abcf6153bd995746b7f8307",
"triggerType" : "PUSH"
}, {
"hash" : "8769cd80906eb931df92e33211beee8d4d9a7497",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24898",
"triggerID" : "8769cd80906eb931df92e33211beee8d4d9a7497",
"triggerType" : "PUSH"
}, {
"hash" : "e644864df1edaf92ad2eeb6b77d1a5468528398c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24902",
"triggerID" : "e644864df1edaf92ad2eeb6b77d1a5468528398c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8769cd80906eb931df92e33211beee8d4d9a7497 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24898)
* e644864df1edaf92ad2eeb6b77d1a5468528398c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24902)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11440:
URL: https://github.com/apache/hudi/pull/11440#issuecomment-2163084989
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24366",
"triggerID" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 22d7e90f753b9cd7374312c8ff73851453cf9736 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24366)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11440:
URL: https://github.com/apache/hudi/pull/11440#issuecomment-2236688545
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24366",
"triggerID" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"triggerType" : "PUSH"
}, {
"hash" : "4bb072faa37b2ea398144fbbc24deff966153cfa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24471",
"triggerID" : "4bb072faa37b2ea398144fbbc24deff966153cfa",
"triggerType" : "PUSH"
}, {
"hash" : "7e8e5336ab1f079d93ccaa0981e70798169532f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24807",
"triggerID" : "7e8e5336ab1f079d93ccaa0981e70798169532f1",
"triggerType" : "PUSH"
}, {
"hash" : "15b2428c2dd7d7980fd98b78643811652e08fd35",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24823",
"triggerID" : "15b2428c2dd7d7980fd98b78643811652e08fd35",
"triggerType" : "PUSH"
}, {
"hash" : "0ccf2a1257697222f488723c7ae3fde9cd7d6b8d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24861",
"triggerID" : "0ccf2a1257697222f488723c7ae3fde9cd7d6b8d",
"triggerType" : "PUSH"
}, {
"hash" : "228d9a1568699f5fa9c7372af6159b78c9189256",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24866",
"triggerID" : "228d9a1568699f5fa9c7372af6159b78c9189256",
"triggerType" : "PUSH"
}, {
"hash" : "56188d58f0289b2c6abcf6153bd995746b7f8307",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24881",
"triggerID" : "56188d58f0289b2c6abcf6153bd995746b7f8307",
"triggerType" : "PUSH"
}, {
"hash" : "8769cd80906eb931df92e33211beee8d4d9a7497",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24898",
"triggerID" : "8769cd80906eb931df92e33211beee8d4d9a7497",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 56188d58f0289b2c6abcf6153bd995746b7f8307 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24881)
* 8769cd80906eb931df92e33211beee8d4d9a7497 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24898)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #11440:
URL: https://github.com/apache/hudi/pull/11440#issuecomment-2237812627
hey @lokeshj1703 : can you update the PR desc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #11440:
URL: https://github.com/apache/hudi/pull/11440#discussion_r1683621408
##########
hudi-common/src/main/java/org/apache/hudi/common/util/ClusteringUtils.java:
##########
@@ -333,28 +335,31 @@ public static List<HoodieInstant> getPendingClusteringInstantTimes(HoodieTableMe
* Returns the earliest instant to retain.
* Make sure the clustering instant won't be archived before cleaned, and the earliest inflight clustering instant has a previous commit.
*
- * @param activeTimeline The active timeline
- * @param metaClient The meta client
+ * @param activeTimeline The active timeline
+ * @param metaClient The meta client
+ * @param cleanerPolicy The hoodie cleaning policy
+ * @param shouldArchiveBeyondSavepoint Is archival enabled to run beyond savepoint
* @return the earliest instant to retain for clustering
*/
public static Option<HoodieInstant> getEarliestInstantToRetainForClustering(
- HoodieActiveTimeline activeTimeline, HoodieTableMetaClient metaClient) throws IOException {
+ HoodieActiveTimeline activeTimeline, HoodieTableMetaClient metaClient, HoodieCleaningPolicy cleanerPolicy, boolean shouldArchiveBeyondSavepoint) throws IOException {
Option<HoodieInstant> oldestInstantToRetain = Option.empty();
HoodieTimeline replaceOrClusterTimeline = activeTimeline.getTimelineOfActions(CollectionUtils.createSet(HoodieTimeline.REPLACE_COMMIT_ACTION, HoodieTimeline.CLUSTERING_ACTION));
+ Option<HoodieInstant> cleanInstantOpt =
Review Comment:
Why moves the `cleanInstantOpt` out from the check `!replaceOrClusterTimeline.empty()` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11440:
URL: https://github.com/apache/hudi/pull/11440#issuecomment-2221109430
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24366",
"triggerID" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"triggerType" : "PUSH"
}, {
"hash" : "4bb072faa37b2ea398144fbbc24deff966153cfa",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24471",
"triggerID" : "4bb072faa37b2ea398144fbbc24deff966153cfa",
"triggerType" : "PUSH"
}, {
"hash" : "7e8e5336ab1f079d93ccaa0981e70798169532f1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24807",
"triggerID" : "7e8e5336ab1f079d93ccaa0981e70798169532f1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4bb072faa37b2ea398144fbbc24deff966153cfa Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24471)
* 7e8e5336ab1f079d93ccaa0981e70798169532f1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24807)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #11440:
URL: https://github.com/apache/hudi/pull/11440#issuecomment-2221624233
@danny0405 @lokeshj1703 : are we aligned on any optimization we need on top of the patch? for eg, we were discussing about replace commit timeline right? did you guys sync up on that. if not, can we sync up and make forward progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11440:
URL: https://github.com/apache/hudi/pull/11440#issuecomment-2229174194
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24366",
"triggerID" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"triggerType" : "PUSH"
}, {
"hash" : "4bb072faa37b2ea398144fbbc24deff966153cfa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24471",
"triggerID" : "4bb072faa37b2ea398144fbbc24deff966153cfa",
"triggerType" : "PUSH"
}, {
"hash" : "7e8e5336ab1f079d93ccaa0981e70798169532f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24807",
"triggerID" : "7e8e5336ab1f079d93ccaa0981e70798169532f1",
"triggerType" : "PUSH"
}, {
"hash" : "15b2428c2dd7d7980fd98b78643811652e08fd35",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24823",
"triggerID" : "15b2428c2dd7d7980fd98b78643811652e08fd35",
"triggerType" : "PUSH"
}, {
"hash" : "0ccf2a1257697222f488723c7ae3fde9cd7d6b8d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24861",
"triggerID" : "0ccf2a1257697222f488723c7ae3fde9cd7d6b8d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0ccf2a1257697222f488723c7ae3fde9cd7d6b8d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24861)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11440:
URL: https://github.com/apache/hudi/pull/11440#issuecomment-2230613582
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24366",
"triggerID" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"triggerType" : "PUSH"
}, {
"hash" : "4bb072faa37b2ea398144fbbc24deff966153cfa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24471",
"triggerID" : "4bb072faa37b2ea398144fbbc24deff966153cfa",
"triggerType" : "PUSH"
}, {
"hash" : "7e8e5336ab1f079d93ccaa0981e70798169532f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24807",
"triggerID" : "7e8e5336ab1f079d93ccaa0981e70798169532f1",
"triggerType" : "PUSH"
}, {
"hash" : "15b2428c2dd7d7980fd98b78643811652e08fd35",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24823",
"triggerID" : "15b2428c2dd7d7980fd98b78643811652e08fd35",
"triggerType" : "PUSH"
}, {
"hash" : "0ccf2a1257697222f488723c7ae3fde9cd7d6b8d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24861",
"triggerID" : "0ccf2a1257697222f488723c7ae3fde9cd7d6b8d",
"triggerType" : "PUSH"
}, {
"hash" : "228d9a1568699f5fa9c7372af6159b78c9189256",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24866",
"triggerID" : "228d9a1568699f5fa9c7372af6159b78c9189256",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 228d9a1568699f5fa9c7372af6159b78c9189256 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24866)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11440:
URL: https://github.com/apache/hudi/pull/11440#issuecomment-2234158809
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24366",
"triggerID" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"triggerType" : "PUSH"
}, {
"hash" : "4bb072faa37b2ea398144fbbc24deff966153cfa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24471",
"triggerID" : "4bb072faa37b2ea398144fbbc24deff966153cfa",
"triggerType" : "PUSH"
}, {
"hash" : "7e8e5336ab1f079d93ccaa0981e70798169532f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24807",
"triggerID" : "7e8e5336ab1f079d93ccaa0981e70798169532f1",
"triggerType" : "PUSH"
}, {
"hash" : "15b2428c2dd7d7980fd98b78643811652e08fd35",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24823",
"triggerID" : "15b2428c2dd7d7980fd98b78643811652e08fd35",
"triggerType" : "PUSH"
}, {
"hash" : "0ccf2a1257697222f488723c7ae3fde9cd7d6b8d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24861",
"triggerID" : "0ccf2a1257697222f488723c7ae3fde9cd7d6b8d",
"triggerType" : "PUSH"
}, {
"hash" : "228d9a1568699f5fa9c7372af6159b78c9189256",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24866",
"triggerID" : "228d9a1568699f5fa9c7372af6159b78c9189256",
"triggerType" : "PUSH"
}, {
"hash" : "56188d58f0289b2c6abcf6153bd995746b7f8307",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24881",
"triggerID" : "56188d58f0289b2c6abcf6153bd995746b7f8307",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 56188d58f0289b2c6abcf6153bd995746b7f8307 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24881)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11440:
URL: https://github.com/apache/hudi/pull/11440#issuecomment-2222595823
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24366",
"triggerID" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"triggerType" : "PUSH"
}, {
"hash" : "4bb072faa37b2ea398144fbbc24deff966153cfa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24471",
"triggerID" : "4bb072faa37b2ea398144fbbc24deff966153cfa",
"triggerType" : "PUSH"
}, {
"hash" : "7e8e5336ab1f079d93ccaa0981e70798169532f1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24807",
"triggerID" : "7e8e5336ab1f079d93ccaa0981e70798169532f1",
"triggerType" : "PUSH"
}, {
"hash" : "15b2428c2dd7d7980fd98b78643811652e08fd35",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24823",
"triggerID" : "15b2428c2dd7d7980fd98b78643811652e08fd35",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7e8e5336ab1f079d93ccaa0981e70798169532f1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24807)
* 15b2428c2dd7d7980fd98b78643811652e08fd35 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24823)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11440:
URL: https://github.com/apache/hudi/pull/11440#issuecomment-2237186474
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24366",
"triggerID" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"triggerType" : "PUSH"
}, {
"hash" : "4bb072faa37b2ea398144fbbc24deff966153cfa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24471",
"triggerID" : "4bb072faa37b2ea398144fbbc24deff966153cfa",
"triggerType" : "PUSH"
}, {
"hash" : "7e8e5336ab1f079d93ccaa0981e70798169532f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24807",
"triggerID" : "7e8e5336ab1f079d93ccaa0981e70798169532f1",
"triggerType" : "PUSH"
}, {
"hash" : "15b2428c2dd7d7980fd98b78643811652e08fd35",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24823",
"triggerID" : "15b2428c2dd7d7980fd98b78643811652e08fd35",
"triggerType" : "PUSH"
}, {
"hash" : "0ccf2a1257697222f488723c7ae3fde9cd7d6b8d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24861",
"triggerID" : "0ccf2a1257697222f488723c7ae3fde9cd7d6b8d",
"triggerType" : "PUSH"
}, {
"hash" : "228d9a1568699f5fa9c7372af6159b78c9189256",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24866",
"triggerID" : "228d9a1568699f5fa9c7372af6159b78c9189256",
"triggerType" : "PUSH"
}, {
"hash" : "56188d58f0289b2c6abcf6153bd995746b7f8307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24881",
"triggerID" : "56188d58f0289b2c6abcf6153bd995746b7f8307",
"triggerType" : "PUSH"
}, {
"hash" : "8769cd80906eb931df92e33211beee8d4d9a7497",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24898",
"triggerID" : "8769cd80906eb931df92e33211beee8d4d9a7497",
"triggerType" : "PUSH"
}, {
"hash" : "e644864df1edaf92ad2eeb6b77d1a5468528398c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24902",
"triggerID" : "e644864df1edaf92ad2eeb6b77d1a5468528398c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e644864df1edaf92ad2eeb6b77d1a5468528398c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24902)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11440:
URL: https://github.com/apache/hudi/pull/11440#issuecomment-2238212778
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24366",
"triggerID" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"triggerType" : "PUSH"
}, {
"hash" : "4bb072faa37b2ea398144fbbc24deff966153cfa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24471",
"triggerID" : "4bb072faa37b2ea398144fbbc24deff966153cfa",
"triggerType" : "PUSH"
}, {
"hash" : "7e8e5336ab1f079d93ccaa0981e70798169532f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24807",
"triggerID" : "7e8e5336ab1f079d93ccaa0981e70798169532f1",
"triggerType" : "PUSH"
}, {
"hash" : "15b2428c2dd7d7980fd98b78643811652e08fd35",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24823",
"triggerID" : "15b2428c2dd7d7980fd98b78643811652e08fd35",
"triggerType" : "PUSH"
}, {
"hash" : "0ccf2a1257697222f488723c7ae3fde9cd7d6b8d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24861",
"triggerID" : "0ccf2a1257697222f488723c7ae3fde9cd7d6b8d",
"triggerType" : "PUSH"
}, {
"hash" : "228d9a1568699f5fa9c7372af6159b78c9189256",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24866",
"triggerID" : "228d9a1568699f5fa9c7372af6159b78c9189256",
"triggerType" : "PUSH"
}, {
"hash" : "56188d58f0289b2c6abcf6153bd995746b7f8307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24881",
"triggerID" : "56188d58f0289b2c6abcf6153bd995746b7f8307",
"triggerType" : "PUSH"
}, {
"hash" : "8769cd80906eb931df92e33211beee8d4d9a7497",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24898",
"triggerID" : "8769cd80906eb931df92e33211beee8d4d9a7497",
"triggerType" : "PUSH"
}, {
"hash" : "e644864df1edaf92ad2eeb6b77d1a5468528398c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24902",
"triggerID" : "e644864df1edaf92ad2eeb6b77d1a5468528398c",
"triggerType" : "PUSH"
}, {
"hash" : "f4c05a461c3603f87ea19fc2e11aca69ad75fad3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f4c05a461c3603f87ea19fc2e11aca69ad75fad3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e644864df1edaf92ad2eeb6b77d1a5468528398c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24902)
* f4c05a461c3603f87ea19fc2e11aca69ad75fad3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #11440:
URL: https://github.com/apache/hudi/pull/11440#discussion_r1683651944
##########
hudi-common/src/main/java/org/apache/hudi/common/util/ClusteringUtils.java:
##########
@@ -365,20 +370,56 @@ public static Option<HoodieInstant> getEarliestInstantToRetainForClustering(
// TODO: This case has to be handled. HUDI-6352
retainLowerBound = cleanInstant.getTimestamp();
}
-
- oldestInstantToRetain = replaceOrClusterTimeline.filter(instant ->
- HoodieTimeline.compareTimestamps(
- instant.getTimestamp(),
- HoodieTimeline.GREATER_THAN_OR_EQUALS,
- retainLowerBound))
- .firstInstant();
+ oldestInstantToRetain = replaceOrClusterTimeline.findInstantsAfterOrEquals(retainLowerBound).firstInstant();
} else {
oldestInstantToRetain = replaceOrClusterTimeline.firstInstant();
}
}
+ if (!shouldArchiveBeyondSavepoint) {
+ // explicitly check the savepoint timeline and guard against the first one
+ Option<HoodieInstant> firstSavepointOpt = activeTimeline.getSavePointTimeline().filterCompletedInstants().firstInstant();
+ oldestInstantToRetain = Option.ofNullable(HoodieTimeline.minTimestampInstant(oldestInstantToRetain.orElse(null), firstSavepointOpt.orElse(null)));
+ }
return oldestInstantToRetain;
}
+ public static Option<String> getEarliestReplacedSavepointInClean(HoodieActiveTimeline activeTimeline, HoodieCleaningPolicy cleanerPolicy,
Review Comment:
`getEarliestCommitToNotArchive` is not a good name because all the logic in archiver is to compute the "earliest commit to not archive".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #11440:
URL: https://github.com/apache/hudi/pull/11440#issuecomment-2204767885
> Whose value will either refer to first savepoint if first savepoint < earliest commit to retain.
Does this mean we can not archive beyond any savepoint now?
> Last completed clean instant is required in the timeline to fetch earliest commit to not archive.
I don't think this is reasonable in high level, there are some scenarios that people will never enable cleaning service, or the cleaning just got stuck and the halt of archival would impact the performance of ingestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #11440:
URL: https://github.com/apache/hudi/pull/11440#issuecomment-2235123582
@lokeshj1703 Thanks for the contribution, I have reviewed and created a patch:
[7779.patch.zip](https://github.com/user-attachments/files/16276593/7779.patch.zip)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11440:
URL: https://github.com/apache/hudi/pull/11440#issuecomment-2238522152
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24366",
"triggerID" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"triggerType" : "PUSH"
}, {
"hash" : "4bb072faa37b2ea398144fbbc24deff966153cfa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24471",
"triggerID" : "4bb072faa37b2ea398144fbbc24deff966153cfa",
"triggerType" : "PUSH"
}, {
"hash" : "7e8e5336ab1f079d93ccaa0981e70798169532f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24807",
"triggerID" : "7e8e5336ab1f079d93ccaa0981e70798169532f1",
"triggerType" : "PUSH"
}, {
"hash" : "15b2428c2dd7d7980fd98b78643811652e08fd35",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24823",
"triggerID" : "15b2428c2dd7d7980fd98b78643811652e08fd35",
"triggerType" : "PUSH"
}, {
"hash" : "0ccf2a1257697222f488723c7ae3fde9cd7d6b8d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24861",
"triggerID" : "0ccf2a1257697222f488723c7ae3fde9cd7d6b8d",
"triggerType" : "PUSH"
}, {
"hash" : "228d9a1568699f5fa9c7372af6159b78c9189256",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24866",
"triggerID" : "228d9a1568699f5fa9c7372af6159b78c9189256",
"triggerType" : "PUSH"
}, {
"hash" : "56188d58f0289b2c6abcf6153bd995746b7f8307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24881",
"triggerID" : "56188d58f0289b2c6abcf6153bd995746b7f8307",
"triggerType" : "PUSH"
}, {
"hash" : "8769cd80906eb931df92e33211beee8d4d9a7497",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24898",
"triggerID" : "8769cd80906eb931df92e33211beee8d4d9a7497",
"triggerType" : "PUSH"
}, {
"hash" : "e644864df1edaf92ad2eeb6b77d1a5468528398c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24902",
"triggerID" : "e644864df1edaf92ad2eeb6b77d1a5468528398c",
"triggerType" : "PUSH"
}, {
"hash" : "f4c05a461c3603f87ea19fc2e11aca69ad75fad3",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24907",
"triggerID" : "f4c05a461c3603f87ea19fc2e11aca69ad75fad3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f4c05a461c3603f87ea19fc2e11aca69ad75fad3 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24907)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #11440:
URL: https://github.com/apache/hudi/pull/11440#discussion_r1674898261
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanPlanner.java:
##########
@@ -553,13 +552,62 @@ public Pair<Boolean, List<CleanFileInfo>> getDeletePaths(String partitionPath, O
* Returns the earliest commit to retain based on cleaning policy.
*/
public Option<HoodieInstant> getEarliestCommitToRetain() {
- return CleanerUtils.getEarliestCommitToRetain(
- hoodieTable.getMetaClient().getActiveTimeline().getCommitsAndCompactionTimeline(),
- config.getCleanerPolicy(),
- config.getCleanerCommitsRetained(),
- Instant.now(),
- config.getCleanerHoursRetained(),
- hoodieTable.getMetaClient().getTableConfig().getTimelineTimezone());
+ if (!earliestCommitToRetain.isPresent()) {
+ earliestCommitToRetain = CleanerUtils.getEarliestCommitToRetain(
+ hoodieTable.getMetaClient().getActiveTimeline().getCommitsAndCompactionTimeline(),
+ config.getCleanerPolicy(),
+ config.getCleanerCommitsRetained(),
+ Instant.now(),
+ config.getCleanerHoursRetained(),
+ hoodieTable.getMetaClient().getTableConfig().getTimelineTimezone());
+ }
+ return earliestCommitToRetain;
+ }
+
+ /**
+ * Returns earliest commit to not archive to assist with guarding archival.
+ * @return
+ */
+ public Option<String> getEarliestCommitToNotArchive() {
+ // compute only if not set
+ if (!earliestCommitToNotArchiveOpt.isPresent() && config.getCleanerPolicy() != HoodieCleaningPolicy.KEEP_LATEST_FILE_VERSIONS) { // earliest commit to retain and earliest commit to not archive
+ // makes sense only when clean policy is num commits based or hours based.
+ Option<HoodieInstant> earliestToRetain = getEarliestCommitToRetain();
+ String earliestCommitToNotArchive = null;
+ if (!savepointedTimestamps.isEmpty()) { // if savepoints are found.
+ // find the first savepoint timestamp
+ String firstSavepoint = savepointedTimestamps.stream().sorted().findFirst().get();
+ earliestCommitToNotArchive = firstSavepoint;
+
+ // earliest commit to not archive = min (earliest commit to retain, earliest commit to not archive)
+ if (earliestToRetain.isPresent()) {
+ String earliestToRetainTimestamp = earliestToRetain.get().getTimestamp();
+ // When earliestToRetainTimestamp is greater than first savepoint timestamp but there are no
+ // replace commits between the first savepoint and the earliestToRetainTimestamp, we can set the
+ // earliestCommitToNotArchive to earliestToRetainTimestamp
+ if (HoodieTimeline.compareTimestamps(earliestToRetainTimestamp, HoodieTimeline.GREATER_THAN, firstSavepoint)) {
+ HoodieTimeline replaceTimeline = hoodieTable.getActiveTimeline().getCompletedReplaceTimeline().findInstantsInClosedRange(firstSavepoint, earliestToRetainTimestamp);
+ earliestCommitToNotArchive = replaceTimeline.empty() ? earliestToRetainTimestamp : firstSavepoint;
+ LOG.info(String.format("Setting Earliest Commit to Not Archive to %s since replace timeline is empty, earliestCommitToRetain: %s, total list of savepointed timestamps : %s",
+ earliestCommitToNotArchive, earliestToRetain, Arrays.toString(savepointedTimestamps.toArray())));
+ } else {
+ earliestCommitToNotArchive = earliestToRetainTimestamp;
+ LOG.info(String.format("Setting Earliest Commit to Not Archive to %s, earliestCommitToRetain: %s, total list of savepointed timestamps : %s", earliestCommitToNotArchive,
+ earliestToRetain, Arrays.toString(savepointedTimestamps.toArray())));
+ }
+ }
+ } else {
+ // no savepoints found.
+ // lets set the value regardless. Would help us differentiate older versions of hudi where this will not be set vs latest version where this will be set to either earliest savepoint
+ // or the earliest commit to retain.
+ if (earliestToRetain.isPresent()) {
+ LOG.info(String.format("Setting Earliest Commit to Not Archive same as Earliest commit to retain to %s", earliestToRetain.get().getTimestamp()));
+ earliestCommitToNotArchive = earliestToRetain.get().getTimestamp();
+ }
+ }
+ earliestCommitToNotArchiveOpt = Option.ofNullable(earliestCommitToNotArchive);
Review Comment:
Can we use a separate variable like `skippedReplacedSavepoints` apart from the `earliestToRetainTimestamp` ? And let the archiver to decide which one to use.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #11440:
URL: https://github.com/apache/hudi/pull/11440#issuecomment-2165026195
@nsivabalan Is this the fix we talked before for savepoint deletion? Do we introduce cyclic dependency for clean and archiving.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11440:
URL: https://github.com/apache/hudi/pull/11440#issuecomment-2163201030
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24366",
"triggerID" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 22d7e90f753b9cd7374312c8ff73851453cf9736 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24366)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11440:
URL: https://github.com/apache/hudi/pull/11440#issuecomment-2236654713
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24366",
"triggerID" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"triggerType" : "PUSH"
}, {
"hash" : "4bb072faa37b2ea398144fbbc24deff966153cfa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24471",
"triggerID" : "4bb072faa37b2ea398144fbbc24deff966153cfa",
"triggerType" : "PUSH"
}, {
"hash" : "7e8e5336ab1f079d93ccaa0981e70798169532f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24807",
"triggerID" : "7e8e5336ab1f079d93ccaa0981e70798169532f1",
"triggerType" : "PUSH"
}, {
"hash" : "15b2428c2dd7d7980fd98b78643811652e08fd35",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24823",
"triggerID" : "15b2428c2dd7d7980fd98b78643811652e08fd35",
"triggerType" : "PUSH"
}, {
"hash" : "0ccf2a1257697222f488723c7ae3fde9cd7d6b8d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24861",
"triggerID" : "0ccf2a1257697222f488723c7ae3fde9cd7d6b8d",
"triggerType" : "PUSH"
}, {
"hash" : "228d9a1568699f5fa9c7372af6159b78c9189256",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24866",
"triggerID" : "228d9a1568699f5fa9c7372af6159b78c9189256",
"triggerType" : "PUSH"
}, {
"hash" : "56188d58f0289b2c6abcf6153bd995746b7f8307",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24881",
"triggerID" : "56188d58f0289b2c6abcf6153bd995746b7f8307",
"triggerType" : "PUSH"
}, {
"hash" : "8769cd80906eb931df92e33211beee8d4d9a7497",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8769cd80906eb931df92e33211beee8d4d9a7497",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 56188d58f0289b2c6abcf6153bd995746b7f8307 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24881)
* 8769cd80906eb931df92e33211beee8d4d9a7497 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11440:
URL: https://github.com/apache/hudi/pull/11440#issuecomment-2236822194
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24366",
"triggerID" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"triggerType" : "PUSH"
}, {
"hash" : "4bb072faa37b2ea398144fbbc24deff966153cfa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24471",
"triggerID" : "4bb072faa37b2ea398144fbbc24deff966153cfa",
"triggerType" : "PUSH"
}, {
"hash" : "7e8e5336ab1f079d93ccaa0981e70798169532f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24807",
"triggerID" : "7e8e5336ab1f079d93ccaa0981e70798169532f1",
"triggerType" : "PUSH"
}, {
"hash" : "15b2428c2dd7d7980fd98b78643811652e08fd35",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24823",
"triggerID" : "15b2428c2dd7d7980fd98b78643811652e08fd35",
"triggerType" : "PUSH"
}, {
"hash" : "0ccf2a1257697222f488723c7ae3fde9cd7d6b8d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24861",
"triggerID" : "0ccf2a1257697222f488723c7ae3fde9cd7d6b8d",
"triggerType" : "PUSH"
}, {
"hash" : "228d9a1568699f5fa9c7372af6159b78c9189256",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24866",
"triggerID" : "228d9a1568699f5fa9c7372af6159b78c9189256",
"triggerType" : "PUSH"
}, {
"hash" : "56188d58f0289b2c6abcf6153bd995746b7f8307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24881",
"triggerID" : "56188d58f0289b2c6abcf6153bd995746b7f8307",
"triggerType" : "PUSH"
}, {
"hash" : "8769cd80906eb931df92e33211beee8d4d9a7497",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24898",
"triggerID" : "8769cd80906eb931df92e33211beee8d4d9a7497",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8769cd80906eb931df92e33211beee8d4d9a7497 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24898)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #11440:
URL: https://github.com/apache/hudi/pull/11440#discussion_r1680216558
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanPlanActionExecutor.java:
##########
@@ -149,17 +150,24 @@ HoodieCleanerPlan requestClean(HoodieEngineContext context) {
.map(x -> new HoodieActionInstant(x.getTimestamp(), x.getAction(), x.getState().name())).orElse(null),
planner.getLastCompletedCommitTimestamp(),
config.getCleanerPolicy().name(), Collections.emptyMap(),
- CleanPlanner.LATEST_CLEAN_PLAN_VERSION, cleanOps, partitionsToDelete, prepareExtraMetadata(planner.getSavepointedTimestamps()));
+ CleanPlanner.LATEST_CLEAN_PLAN_VERSION, cleanOps, partitionsToDelete, prepareExtraMetadata(planner.getSavepointedTimestamps(), planner.getEarliestSavepoint()));
} catch (IOException e) {
throw new HoodieIOException("Failed to schedule clean operation", e);
}
}
- private Map<String, String> prepareExtraMetadata(List<String> savepointedTimestamps) {
- if (savepointedTimestamps.isEmpty()) {
+ private Map<String, String> prepareExtraMetadata(List<String> savepointedTimestamps, Option<String> earliestSavepoint) {
+ if (savepointedTimestamps.isEmpty() && !earliestSavepoint.isPresent()) {
return Collections.emptyMap();
} else {
- return Collections.singletonMap(SAVEPOINTED_TIMESTAMPS, savepointedTimestamps.stream().collect(Collectors.joining(",")));
+ Map<String, String> extraMetadata = new HashMap<>();
+ if (!savepointedTimestamps.isEmpty()) {
+ extraMetadata.put(SAVEPOINTED_TIMESTAMPS, savepointedTimestamps.stream().collect(Collectors.joining(",")));
Review Comment:
Because we already remember the savepointed timestamps(let's name it as `s1`) and `earliestCommitToRetain` in the clean metadata, in archiver,
let's just check all the instants(name it as `s2`) within the range [min(s1), earliestCommitToRetain], if `s2` contains any replace_commits, then guard the archival with min(s1), otherwise the archiver is good to go.
And there is no need to put any extra metadata in the clean metadata actually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #11440:
URL: https://github.com/apache/hudi/pull/11440#discussion_r1683633218
##########
hudi-common/src/main/java/org/apache/hudi/common/util/ClusteringUtils.java:
##########
@@ -365,20 +370,56 @@ public static Option<HoodieInstant> getEarliestInstantToRetainForClustering(
// TODO: This case has to be handled. HUDI-6352
retainLowerBound = cleanInstant.getTimestamp();
}
-
- oldestInstantToRetain = replaceOrClusterTimeline.filter(instant ->
- HoodieTimeline.compareTimestamps(
- instant.getTimestamp(),
- HoodieTimeline.GREATER_THAN_OR_EQUALS,
- retainLowerBound))
- .firstInstant();
+ oldestInstantToRetain = replaceOrClusterTimeline.findInstantsAfterOrEquals(retainLowerBound).firstInstant();
} else {
oldestInstantToRetain = replaceOrClusterTimeline.firstInstant();
}
}
+ if (!shouldArchiveBeyondSavepoint) {
+ // explicitly check the savepoint timeline and guard against the first one
+ Option<HoodieInstant> firstSavepointOpt = activeTimeline.getSavePointTimeline().filterCompletedInstants().firstInstant();
+ oldestInstantToRetain = Option.ofNullable(HoodieTimeline.minTimestampInstant(oldestInstantToRetain.orElse(null), firstSavepointOpt.orElse(null)));
+ }
return oldestInstantToRetain;
}
+ public static Option<String> getEarliestReplacedSavepointInClean(HoodieActiveTimeline activeTimeline, HoodieCleaningPolicy cleanerPolicy,
Review Comment:
we can name this method as
getEarliestCommitToNotArchive. and keep this method in HoodieTimelineArchiver.
either we can compute and pass the value to ClusteringUtils.getEarliestInstantToRetainForClustering or make a function as an arg.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "nbalajee (via GitHub)" <gi...@apache.org>.
nbalajee commented on PR #11440:
URL: https://github.com/apache/hudi/pull/11440#issuecomment-2215519646
Let's call out that for replacecommit (clustering operation with replace fileIDs), archiving a replacecommit before the replaced files are cleaned, would result in a data quality issue for the query engines. Queries would include the replaced fileIDs (considering them as valid, as they are archived), which may manifest as duplicates in query results.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11440:
URL: https://github.com/apache/hudi/pull/11440#issuecomment-2222704418
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24366",
"triggerID" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"triggerType" : "PUSH"
}, {
"hash" : "4bb072faa37b2ea398144fbbc24deff966153cfa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24471",
"triggerID" : "4bb072faa37b2ea398144fbbc24deff966153cfa",
"triggerType" : "PUSH"
}, {
"hash" : "7e8e5336ab1f079d93ccaa0981e70798169532f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24807",
"triggerID" : "7e8e5336ab1f079d93ccaa0981e70798169532f1",
"triggerType" : "PUSH"
}, {
"hash" : "15b2428c2dd7d7980fd98b78643811652e08fd35",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24823",
"triggerID" : "15b2428c2dd7d7980fd98b78643811652e08fd35",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 15b2428c2dd7d7980fd98b78643811652e08fd35 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24823)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on PR #11440:
URL: https://github.com/apache/hudi/pull/11440#issuecomment-2229085411
@danny0405 I have updated the PR to address your comments. Can you please check?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #11440:
URL: https://github.com/apache/hudi/pull/11440#discussion_r1674893075
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/timeline/HoodieTimelineArchiver.java:
##########
@@ -264,20 +268,46 @@ private List<HoodieInstant> getCommitInstantsToArchive() throws IOException {
.map(Option::get)
.min(HoodieInstant.COMPARATOR);
- // Step2: We cannot archive any commits which are made after the first savepoint present,
+ // Step2: We cannot archive any commits which are later than EarliestCommitToNotArchive.
// unless HoodieArchivalConfig#ARCHIVE_BEYOND_SAVEPOINT is enabled.
- Option<HoodieInstant> firstSavepoint = table.getCompletedSavepointTimeline().firstInstant();
+
+ // EarliestCommitToNotArchive is required to block archival when savepoint is deleted.
+ // This ensures that archival is blocked until clean has cleaned up files retained due to savepoint.
+ // Since EarliestCommitToNotArchive is advanced by cleaner, it is a way for cleaner to notify the archival
+ // that cleanup has finished and archival can advance further.
+ // If this guard is not present, the archival of commits can lead to duplicates. Here is a scenario
+ // illustrating the same. This scenario considers a case where EarliestCommitToNotArchive guard is not present
+ // c1.dc - f1 (c1 deltacommit creates file with id f1)
+ // c2.dc - f2 (c2 deltacommit creates file with id f2)
+ // c2.sp - Savepoint at c2
+ // c3.rc (replacing f2 -> f3) (Replace commit replacing file id f2 with f3)
+ // c4.dc
+ //
+ // Lets say Incremental cleaner moved past the c3.rc without cleaning f2 since savepoint is created at c2.
+ // Archival is blocked at c2 since there is a savepoint at c2.
+ // Lets say the savepoint at c2 is now deleted, Archival would archive c3.rc since it is unblocked now.
+ // Since c3 is archived and f2 has not been cleaned, the table view would be considering f2 as a valid
+ // file id. This causes duplicates.
Review Comment:
Can we put the logic inside `getEarliestInstantToRetainForClustering` to make the logic around replace_commit more concentrated and there is no need to go through the replace_commit timeline multiple times?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #11440:
URL: https://github.com/apache/hudi/pull/11440#discussion_r1674895597
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanPlanActionExecutor.java:
##########
@@ -149,17 +150,24 @@ HoodieCleanerPlan requestClean(HoodieEngineContext context) {
.map(x -> new HoodieActionInstant(x.getTimestamp(), x.getAction(), x.getState().name())).orElse(null),
planner.getLastCompletedCommitTimestamp(),
config.getCleanerPolicy().name(), Collections.emptyMap(),
- CleanPlanner.LATEST_CLEAN_PLAN_VERSION, cleanOps, partitionsToDelete, prepareExtraMetadata(planner.getSavepointedTimestamps()));
+ CleanPlanner.LATEST_CLEAN_PLAN_VERSION, cleanOps, partitionsToDelete, prepareExtraMetadata(planner.getSavepointedTimestamps(), planner.getEarliestCommitToNotArchive()));
} catch (IOException e) {
throw new HoodieIOException("Failed to schedule clean operation", e);
}
}
- private Map<String, String> prepareExtraMetadata(List<String> savepointedTimestamps) {
- if (savepointedTimestamps.isEmpty()) {
+ private Map<String, String> prepareExtraMetadata(List<String> savepointedTimestamps, Option<String> earliestCommitToNotArchive) {
+ if (savepointedTimestamps.isEmpty() && !earliestCommitToNotArchive.isPresent()) {
return Collections.emptyMap();
} else {
- return Collections.singletonMap(SAVEPOINTED_TIMESTAMPS, savepointedTimestamps.stream().collect(Collectors.joining(",")));
+ Map<String, String> extraMetadata = new HashMap<>();
+ if (!savepointedTimestamps.isEmpty()) {
+ extraMetadata.put(SAVEPOINTED_TIMESTAMPS, savepointedTimestamps.stream().collect(Collectors.joining(",")));
+ }
+ if (earliestCommitToNotArchive.isPresent()) {
+ extraMetadata.put(EARLIEST_COMMIT_TO_NOT_ARCHIVE, earliestCommitToNotArchive.get());
Review Comment:
Can we put the name more straight-forward like `SKIPPED_REPLACED_SAVEPOINTS`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #11440:
URL: https://github.com/apache/hudi/pull/11440#discussion_r1674899742
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieTimeline.java:
##########
@@ -437,6 +437,15 @@ static boolean compareTimestamps(String commit1, BiPredicate<String, String> pre
return predicateToApply.test(commit1, commit2);
}
+ static String minTimestamp(String commit1, String commit2) {
+ if (StringUtils.isNullOrEmpty(commit1)) {
+ return commit2;
+ } else if (StringUtils.isNullOrEmpty(commit2)) {
+ return commit1;
+ }
+ return GREATER_THAN.test(commit1, commit2) ? commit2 : commit1;
+ }
+
/**
Review Comment:
Should we reuse `minInstant` instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11440:
URL: https://github.com/apache/hudi/pull/11440#issuecomment-2222511800
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24366",
"triggerID" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"triggerType" : "PUSH"
}, {
"hash" : "4bb072faa37b2ea398144fbbc24deff966153cfa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24471",
"triggerID" : "4bb072faa37b2ea398144fbbc24deff966153cfa",
"triggerType" : "PUSH"
}, {
"hash" : "7e8e5336ab1f079d93ccaa0981e70798169532f1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24807",
"triggerID" : "7e8e5336ab1f079d93ccaa0981e70798169532f1",
"triggerType" : "PUSH"
}, {
"hash" : "15b2428c2dd7d7980fd98b78643811652e08fd35",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "15b2428c2dd7d7980fd98b78643811652e08fd35",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7e8e5336ab1f079d93ccaa0981e70798169532f1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24807)
* 15b2428c2dd7d7980fd98b78643811652e08fd35 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on PR #11440:
URL: https://github.com/apache/hudi/pull/11440#issuecomment-2221025516
I have explained the scenario in the comments. PTAL @nbalajee @danny0405
I have also fixed the earliestCommitToNotArchive logic to take into account last clean instant and removed separate handling in the cleaner timeline archival. The case where earliestCommitToNotArchive > last clean instant will not be possible since this timestamp is generated by cleaner but still added a check to consider the minimum of last clean instant and earliestCommitToNotArchive.
cc @nsivabalan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on PR #11440:
URL: https://github.com/apache/hudi/pull/11440#issuecomment-2222500128
I have added the optimisation. @danny0405 @nsivabalan PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11440:
URL: https://github.com/apache/hudi/pull/11440#issuecomment-2233960417
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24366",
"triggerID" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"triggerType" : "PUSH"
}, {
"hash" : "4bb072faa37b2ea398144fbbc24deff966153cfa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24471",
"triggerID" : "4bb072faa37b2ea398144fbbc24deff966153cfa",
"triggerType" : "PUSH"
}, {
"hash" : "7e8e5336ab1f079d93ccaa0981e70798169532f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24807",
"triggerID" : "7e8e5336ab1f079d93ccaa0981e70798169532f1",
"triggerType" : "PUSH"
}, {
"hash" : "15b2428c2dd7d7980fd98b78643811652e08fd35",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24823",
"triggerID" : "15b2428c2dd7d7980fd98b78643811652e08fd35",
"triggerType" : "PUSH"
}, {
"hash" : "0ccf2a1257697222f488723c7ae3fde9cd7d6b8d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24861",
"triggerID" : "0ccf2a1257697222f488723c7ae3fde9cd7d6b8d",
"triggerType" : "PUSH"
}, {
"hash" : "228d9a1568699f5fa9c7372af6159b78c9189256",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24866",
"triggerID" : "228d9a1568699f5fa9c7372af6159b78c9189256",
"triggerType" : "PUSH"
}, {
"hash" : "56188d58f0289b2c6abcf6153bd995746b7f8307",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24881",
"triggerID" : "56188d58f0289b2c6abcf6153bd995746b7f8307",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 228d9a1568699f5fa9c7372af6159b78c9189256 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24866)
* 56188d58f0289b2c6abcf6153bd995746b7f8307 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24881)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11440:
URL: https://github.com/apache/hudi/pull/11440#issuecomment-2163065510
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 22d7e90f753b9cd7374312c8ff73851453cf9736 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #11440:
URL: https://github.com/apache/hudi/pull/11440#discussion_r1680216558
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanPlanActionExecutor.java:
##########
@@ -149,17 +150,24 @@ HoodieCleanerPlan requestClean(HoodieEngineContext context) {
.map(x -> new HoodieActionInstant(x.getTimestamp(), x.getAction(), x.getState().name())).orElse(null),
planner.getLastCompletedCommitTimestamp(),
config.getCleanerPolicy().name(), Collections.emptyMap(),
- CleanPlanner.LATEST_CLEAN_PLAN_VERSION, cleanOps, partitionsToDelete, prepareExtraMetadata(planner.getSavepointedTimestamps()));
+ CleanPlanner.LATEST_CLEAN_PLAN_VERSION, cleanOps, partitionsToDelete, prepareExtraMetadata(planner.getSavepointedTimestamps(), planner.getEarliestSavepoint()));
} catch (IOException e) {
throw new HoodieIOException("Failed to schedule clean operation", e);
}
}
- private Map<String, String> prepareExtraMetadata(List<String> savepointedTimestamps) {
- if (savepointedTimestamps.isEmpty()) {
+ private Map<String, String> prepareExtraMetadata(List<String> savepointedTimestamps, Option<String> earliestSavepoint) {
+ if (savepointedTimestamps.isEmpty() && !earliestSavepoint.isPresent()) {
return Collections.emptyMap();
} else {
- return Collections.singletonMap(SAVEPOINTED_TIMESTAMPS, savepointedTimestamps.stream().collect(Collectors.joining(",")));
+ Map<String, String> extraMetadata = new HashMap<>();
+ if (!savepointedTimestamps.isEmpty()) {
+ extraMetadata.put(SAVEPOINTED_TIMESTAMPS, savepointedTimestamps.stream().collect(Collectors.joining(",")));
Review Comment:
Because we already remember the savepointed timestamps(let's name it as `s1`) and `earliestCommitToRetain` in the clean metadata, in archiver,
let's just check all the instants(name it as `s2`) within the range [min(s1), earliestCommitToRetain], if `s2` contains any replace_commits, then guard the archival with min(s1), otherwise we are good to go.
So there is no need to put any extra metadata in the clean metadata actually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11440:
URL: https://github.com/apache/hudi/pull/11440#issuecomment-2221096095
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24366",
"triggerID" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"triggerType" : "PUSH"
}, {
"hash" : "4bb072faa37b2ea398144fbbc24deff966153cfa",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24471",
"triggerID" : "4bb072faa37b2ea398144fbbc24deff966153cfa",
"triggerType" : "PUSH"
}, {
"hash" : "7e8e5336ab1f079d93ccaa0981e70798169532f1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7e8e5336ab1f079d93ccaa0981e70798169532f1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4bb072faa37b2ea398144fbbc24deff966153cfa Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24471)
* 7e8e5336ab1f079d93ccaa0981e70798169532f1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #11440:
URL: https://github.com/apache/hudi/pull/11440#issuecomment-2221755217
> did you guys sync up on that. if not, can we sync up and make forward progress.
I thought we have synced up and reached concensus in the last stand-up meeting, no?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11440:
URL: https://github.com/apache/hudi/pull/11440#issuecomment-2237106014
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24366",
"triggerID" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"triggerType" : "PUSH"
}, {
"hash" : "4bb072faa37b2ea398144fbbc24deff966153cfa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24471",
"triggerID" : "4bb072faa37b2ea398144fbbc24deff966153cfa",
"triggerType" : "PUSH"
}, {
"hash" : "7e8e5336ab1f079d93ccaa0981e70798169532f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24807",
"triggerID" : "7e8e5336ab1f079d93ccaa0981e70798169532f1",
"triggerType" : "PUSH"
}, {
"hash" : "15b2428c2dd7d7980fd98b78643811652e08fd35",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24823",
"triggerID" : "15b2428c2dd7d7980fd98b78643811652e08fd35",
"triggerType" : "PUSH"
}, {
"hash" : "0ccf2a1257697222f488723c7ae3fde9cd7d6b8d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24861",
"triggerID" : "0ccf2a1257697222f488723c7ae3fde9cd7d6b8d",
"triggerType" : "PUSH"
}, {
"hash" : "228d9a1568699f5fa9c7372af6159b78c9189256",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24866",
"triggerID" : "228d9a1568699f5fa9c7372af6159b78c9189256",
"triggerType" : "PUSH"
}, {
"hash" : "56188d58f0289b2c6abcf6153bd995746b7f8307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24881",
"triggerID" : "56188d58f0289b2c6abcf6153bd995746b7f8307",
"triggerType" : "PUSH"
}, {
"hash" : "8769cd80906eb931df92e33211beee8d4d9a7497",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24898",
"triggerID" : "8769cd80906eb931df92e33211beee8d4d9a7497",
"triggerType" : "PUSH"
}, {
"hash" : "e644864df1edaf92ad2eeb6b77d1a5468528398c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e644864df1edaf92ad2eeb6b77d1a5468528398c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8769cd80906eb931df92e33211beee8d4d9a7497 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24898)
* e644864df1edaf92ad2eeb6b77d1a5468528398c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #11440:
URL: https://github.com/apache/hudi/pull/11440#discussion_r1683622608
##########
hudi-common/src/main/java/org/apache/hudi/common/util/ClusteringUtils.java:
##########
@@ -365,20 +370,56 @@ public static Option<HoodieInstant> getEarliestInstantToRetainForClustering(
// TODO: This case has to be handled. HUDI-6352
retainLowerBound = cleanInstant.getTimestamp();
}
-
- oldestInstantToRetain = replaceOrClusterTimeline.filter(instant ->
- HoodieTimeline.compareTimestamps(
- instant.getTimestamp(),
- HoodieTimeline.GREATER_THAN_OR_EQUALS,
- retainLowerBound))
- .firstInstant();
+ oldestInstantToRetain = replaceOrClusterTimeline.findInstantsAfterOrEquals(retainLowerBound).firstInstant();
} else {
oldestInstantToRetain = replaceOrClusterTimeline.firstInstant();
}
}
+ if (!shouldArchiveBeyondSavepoint) {
Review Comment:
It looks like this chagne is not necessary if we do not make changes to lines L276 ~ L280 of `HoodieTimelineArchiver: `.
##########
hudi-common/src/main/java/org/apache/hudi/common/util/ClusteringUtils.java:
##########
@@ -365,20 +370,56 @@ public static Option<HoodieInstant> getEarliestInstantToRetainForClustering(
// TODO: This case has to be handled. HUDI-6352
retainLowerBound = cleanInstant.getTimestamp();
}
-
- oldestInstantToRetain = replaceOrClusterTimeline.filter(instant ->
- HoodieTimeline.compareTimestamps(
- instant.getTimestamp(),
- HoodieTimeline.GREATER_THAN_OR_EQUALS,
- retainLowerBound))
- .firstInstant();
+ oldestInstantToRetain = replaceOrClusterTimeline.findInstantsAfterOrEquals(retainLowerBound).firstInstant();
} else {
oldestInstantToRetain = replaceOrClusterTimeline.firstInstant();
}
}
+ if (!shouldArchiveBeyondSavepoint) {
Review Comment:
It looks like this change is not necessary if we do not make changes to lines L276 ~ L280 of `HoodieTimelineArchiver: `.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #11440:
URL: https://github.com/apache/hudi/pull/11440#discussion_r1681470626
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanPlanActionExecutor.java:
##########
@@ -149,17 +150,24 @@ HoodieCleanerPlan requestClean(HoodieEngineContext context) {
.map(x -> new HoodieActionInstant(x.getTimestamp(), x.getAction(), x.getState().name())).orElse(null),
planner.getLastCompletedCommitTimestamp(),
config.getCleanerPolicy().name(), Collections.emptyMap(),
- CleanPlanner.LATEST_CLEAN_PLAN_VERSION, cleanOps, partitionsToDelete, prepareExtraMetadata(planner.getSavepointedTimestamps()));
+ CleanPlanner.LATEST_CLEAN_PLAN_VERSION, cleanOps, partitionsToDelete, prepareExtraMetadata(planner.getSavepointedTimestamps(), planner.getEarliestSavepoint()));
} catch (IOException e) {
throw new HoodieIOException("Failed to schedule clean operation", e);
}
}
- private Map<String, String> prepareExtraMetadata(List<String> savepointedTimestamps) {
- if (savepointedTimestamps.isEmpty()) {
+ private Map<String, String> prepareExtraMetadata(List<String> savepointedTimestamps, Option<String> earliestSavepoint) {
+ if (savepointedTimestamps.isEmpty() && !earliestSavepoint.isPresent()) {
return Collections.emptyMap();
} else {
- return Collections.singletonMap(SAVEPOINTED_TIMESTAMPS, savepointedTimestamps.stream().collect(Collectors.joining(",")));
+ Map<String, String> extraMetadata = new HashMap<>();
+ if (!savepointedTimestamps.isEmpty()) {
+ extraMetadata.put(SAVEPOINTED_TIMESTAMPS, savepointedTimestamps.stream().collect(Collectors.joining(",")));
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #11440:
URL: https://github.com/apache/hudi/pull/11440#discussion_r1683797553
##########
hudi-common/src/main/java/org/apache/hudi/common/util/ClusteringUtils.java:
##########
@@ -365,20 +370,56 @@ public static Option<HoodieInstant> getEarliestInstantToRetainForClustering(
// TODO: This case has to be handled. HUDI-6352
retainLowerBound = cleanInstant.getTimestamp();
}
-
- oldestInstantToRetain = replaceOrClusterTimeline.filter(instant ->
- HoodieTimeline.compareTimestamps(
- instant.getTimestamp(),
- HoodieTimeline.GREATER_THAN_OR_EQUALS,
- retainLowerBound))
- .firstInstant();
+ oldestInstantToRetain = replaceOrClusterTimeline.findInstantsAfterOrEquals(retainLowerBound).firstInstant();
} else {
oldestInstantToRetain = replaceOrClusterTimeline.firstInstant();
}
}
+ if (!shouldArchiveBeyondSavepoint) {
Review Comment:
Addressed
##########
hudi-common/src/main/java/org/apache/hudi/common/util/ClusteringUtils.java:
##########
@@ -333,28 +335,31 @@ public static List<HoodieInstant> getPendingClusteringInstantTimes(HoodieTableMe
* Returns the earliest instant to retain.
* Make sure the clustering instant won't be archived before cleaned, and the earliest inflight clustering instant has a previous commit.
*
- * @param activeTimeline The active timeline
- * @param metaClient The meta client
+ * @param activeTimeline The active timeline
+ * @param metaClient The meta client
+ * @param cleanerPolicy The hoodie cleaning policy
+ * @param shouldArchiveBeyondSavepoint Is archival enabled to run beyond savepoint
* @return the earliest instant to retain for clustering
*/
public static Option<HoodieInstant> getEarliestInstantToRetainForClustering(
- HoodieActiveTimeline activeTimeline, HoodieTableMetaClient metaClient) throws IOException {
+ HoodieActiveTimeline activeTimeline, HoodieTableMetaClient metaClient, HoodieCleaningPolicy cleanerPolicy, boolean shouldArchiveBeyondSavepoint) throws IOException {
Option<HoodieInstant> oldestInstantToRetain = Option.empty();
HoodieTimeline replaceOrClusterTimeline = activeTimeline.getTimelineOfActions(CollectionUtils.createSet(HoodieTimeline.REPLACE_COMMIT_ACTION, HoodieTimeline.CLUSTERING_ACTION));
+ Option<HoodieInstant> cleanInstantOpt =
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7779] Guard archival on savepoint removal until cleaner is able to clean it up [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11440:
URL: https://github.com/apache/hudi/pull/11440#issuecomment-2238310021
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24366",
"triggerID" : "22d7e90f753b9cd7374312c8ff73851453cf9736",
"triggerType" : "PUSH"
}, {
"hash" : "4bb072faa37b2ea398144fbbc24deff966153cfa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24471",
"triggerID" : "4bb072faa37b2ea398144fbbc24deff966153cfa",
"triggerType" : "PUSH"
}, {
"hash" : "7e8e5336ab1f079d93ccaa0981e70798169532f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24807",
"triggerID" : "7e8e5336ab1f079d93ccaa0981e70798169532f1",
"triggerType" : "PUSH"
}, {
"hash" : "15b2428c2dd7d7980fd98b78643811652e08fd35",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24823",
"triggerID" : "15b2428c2dd7d7980fd98b78643811652e08fd35",
"triggerType" : "PUSH"
}, {
"hash" : "0ccf2a1257697222f488723c7ae3fde9cd7d6b8d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24861",
"triggerID" : "0ccf2a1257697222f488723c7ae3fde9cd7d6b8d",
"triggerType" : "PUSH"
}, {
"hash" : "228d9a1568699f5fa9c7372af6159b78c9189256",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24866",
"triggerID" : "228d9a1568699f5fa9c7372af6159b78c9189256",
"triggerType" : "PUSH"
}, {
"hash" : "56188d58f0289b2c6abcf6153bd995746b7f8307",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24881",
"triggerID" : "56188d58f0289b2c6abcf6153bd995746b7f8307",
"triggerType" : "PUSH"
}, {
"hash" : "8769cd80906eb931df92e33211beee8d4d9a7497",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24898",
"triggerID" : "8769cd80906eb931df92e33211beee8d4d9a7497",
"triggerType" : "PUSH"
}, {
"hash" : "e644864df1edaf92ad2eeb6b77d1a5468528398c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24902",
"triggerID" : "e644864df1edaf92ad2eeb6b77d1a5468528398c",
"triggerType" : "PUSH"
}, {
"hash" : "f4c05a461c3603f87ea19fc2e11aca69ad75fad3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24907",
"triggerID" : "f4c05a461c3603f87ea19fc2e11aca69ad75fad3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e644864df1edaf92ad2eeb6b77d1a5468528398c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24902)
* f4c05a461c3603f87ea19fc2e11aca69ad75fad3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24907)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org