You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "the-other-tim-brown (via GitHub)" <gi...@apache.org> on 2025/07/31 14:30:11 UTC
[PR] [HUDI-9675] Add safety check to Savepoint when earliest commit retained is not set in cleaner output [hudi]
the-other-tim-brown opened a new pull request, #13660:
URL: https://github.com/apache/hudi/pull/13660
### Change Logs
In the case of cleaner policy `KEEP_LATEST_FILE_VERSIONS`, the `earliestCommitToRetain` is never set in the cleaner plan or clean metadata. The savepoint operation currently relies on this field to determine whether the instant is valid for a savepoint. If that value is not set, the code will assume all instants are valid which is not accurate.
To handle the case where the `KEEP_LATEST_FILE_VERSIONS` policy is used, we instead ensure that the instant provided to the savepoint is no older than the last commit before the last clean operation. Since the clean metadata does not include the number of file versions it kept, this is the only commit that we can guarantee is fully intact with this cleaner policy.
### Impact
Adds more safety when creating savepoints
### Risk level (write none, low medium or high below)
None
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9675] Add safety check to Savepoint when earliest commit retained is not set in cleaner output [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13660:
URL: https://github.com/apache/hudi/pull/13660#discussion_r2246833693
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/savepoint/SavepointActionExecutor.java:
##########
@@ -145,10 +132,41 @@ public HoodieSavepointMetadata execute() {
.saveAsComplete(
true, instantGenerator.createNewInstant(HoodieInstant.State.INFLIGHT, HoodieTimeline.SAVEPOINT_ACTION, instantTime), Option.of(metadata),
savepointCompletedInstant -> table.getMetaClient().getTableFormat().savepoint(savepointCompletedInstant, table.getContext(), table.getMetaClient(), table.getViewManager()));
- LOG.info("Savepoint " + instantTime + " created");
+ LOG.info("Savepoint {} created", instantTime);
return metadata;
} catch (HoodieIOException e) {
throw new HoodieSavepointException("Failed to savepoint " + instantTime, e);
}
}
+
+ @VisibleForTesting
+ String getLastCommitRetained() {
+ Option<HoodieInstant> cleanInstant = table.getCleanTimeline().lastInstant();
+ // if there are no clean instants, we can use the first completed commit
+ if (cleanInstant.isEmpty()) {
+ return table.getCompletedCommitsTimeline().firstInstant().get().requestedTime();
+ }
+ return cleanInstant.map(instant -> {
+ try {
+ if (instant.isCompleted()) {
+ return table.getActiveTimeline().readCleanMetadata(instant)
+ .getEarliestCommitToRetain();
+ } else {
+ // clean is pending or inflight
+ return table.getActiveTimeline().readCleanerPlan(
+ instantGenerator.createNewInstant(REQUESTED, instant.getAction(), instant.requestedTime()))
+ .getEarliestInstantToRetain().getTimestamp();
+ }
+ } catch (IOException e) {
+ throw new HoodieSavepointException("Failed to savepoint " + instantTime, e);
+ }
+ }).orElseGet(() ->
+ // If there is no earliest commit to retain, but there are clean instants on the timeline, we must assume KEEP_LATEST_FILE_VERSIONS policy is used.
Review Comment:
> we must assume KEEP_LATEST_FILE_VERSIONS policy is used.
This is not true, `getEarliestInstantToRetain` is only there for incremental cleaning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9675] Add safety check to Savepoint when earliest commit retained is not set in cleaner output [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13660:
URL: https://github.com/apache/hudi/pull/13660#discussion_r2250261839
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/savepoint/SavepointActionExecutor.java:
##########
@@ -145,10 +132,41 @@ public HoodieSavepointMetadata execute() {
.saveAsComplete(
true, instantGenerator.createNewInstant(HoodieInstant.State.INFLIGHT, HoodieTimeline.SAVEPOINT_ACTION, instantTime), Option.of(metadata),
savepointCompletedInstant -> table.getMetaClient().getTableFormat().savepoint(savepointCompletedInstant, table.getContext(), table.getMetaClient(), table.getViewManager()));
- LOG.info("Savepoint " + instantTime + " created");
+ LOG.info("Savepoint {} created", instantTime);
return metadata;
} catch (HoodieIOException e) {
throw new HoodieSavepointException("Failed to savepoint " + instantTime, e);
}
}
+
+ @VisibleForTesting
+ String getLastCommitRetained() {
+ Option<HoodieInstant> cleanInstant = table.getCleanTimeline().lastInstant();
+ // if there are no clean instants, we can use the first completed commit
+ if (cleanInstant.isEmpty()) {
+ return table.getCompletedCommitsTimeline().firstInstant().get().requestedTime();
+ }
+ return cleanInstant.map(instant -> {
+ try {
+ if (instant.isCompleted()) {
+ return table.getActiveTimeline().readCleanMetadata(instant)
+ .getEarliestCommitToRetain();
+ } else {
+ // clean is pending or inflight
+ return table.getActiveTimeline().readCleanerPlan(
+ instantGenerator.createNewInstant(REQUESTED, instant.getAction(), instant.requestedTime()))
+ .getEarliestInstantToRetain().getTimestamp();
+ }
+ } catch (IOException e) {
+ throw new HoodieSavepointException("Failed to savepoint " + instantTime, e);
+ }
+ }).orElseGet(() ->
+ // If there is no earliest commit to retain, but there are clean instants on the timeline, we must assume KEEP_LATEST_FILE_VERSIONS policy is used.
+ // In that case, the last commit before the clean instant is the last commit guaranteed to be retained.
+ table.getActiveTimeline().getWriteTimeline().filterCompletedInstants()
+ .filter(instant -> compareTimestamps(instant.requestedTime(), LESSER_THAN_OR_EQUALS, cleanInstant.get().requestedTime()))
Review Comment:
Updated to use the clean instant instead of throwing an exception
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9675] Add safety check to Savepoint when earliest commit retained is not set in cleaner output [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13660:
URL: https://github.com/apache/hudi/pull/13660#discussion_r2249088638
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/savepoint/SavepointActionExecutor.java:
##########
@@ -145,10 +132,41 @@ public HoodieSavepointMetadata execute() {
.saveAsComplete(
true, instantGenerator.createNewInstant(HoodieInstant.State.INFLIGHT, HoodieTimeline.SAVEPOINT_ACTION, instantTime), Option.of(metadata),
savepointCompletedInstant -> table.getMetaClient().getTableFormat().savepoint(savepointCompletedInstant, table.getContext(), table.getMetaClient(), table.getViewManager()));
- LOG.info("Savepoint " + instantTime + " created");
+ LOG.info("Savepoint {} created", instantTime);
return metadata;
} catch (HoodieIOException e) {
throw new HoodieSavepointException("Failed to savepoint " + instantTime, e);
}
}
+
+ @VisibleForTesting
+ String getLastCommitRetained() {
+ Option<HoodieInstant> cleanInstant = table.getCleanTimeline().lastInstant();
+ // if there are no clean instants, we can use the first completed commit
+ if (cleanInstant.isEmpty()) {
+ return table.getCompletedCommitsTimeline().firstInstant().get().requestedTime();
+ }
+ return cleanInstant.map(instant -> {
+ try {
+ if (instant.isCompleted()) {
+ return table.getActiveTimeline().readCleanMetadata(instant)
+ .getEarliestCommitToRetain();
+ } else {
+ // clean is pending or inflight
+ return table.getActiveTimeline().readCleanerPlan(
+ instantGenerator.createNewInstant(REQUESTED, instant.getAction(), instant.requestedTime()))
+ .getEarliestInstantToRetain().getTimestamp();
+ }
+ } catch (IOException e) {
+ throw new HoodieSavepointException("Failed to savepoint " + instantTime, e);
+ }
+ }).orElseGet(() ->
+ // If there is no earliest commit to retain, but there are clean instants on the timeline, we must assume KEEP_LATEST_FILE_VERSIONS policy is used.
+ // In that case, the last commit before the clean instant is the last commit guaranteed to be retained.
+ table.getActiveTimeline().getWriteTimeline().filterCompletedInstants()
Review Comment:
We already have validation check in L82.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9675] Add safety check to Savepoint when earliest commit retained is not set in cleaner output [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13660:
URL: https://github.com/apache/hudi/pull/13660#issuecomment-3140233881
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7889741d56a1cec46822ac747eb9159854dc0f17",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7403",
"triggerID" : "7889741d56a1cec46822ac747eb9159854dc0f17",
"triggerType" : "PUSH"
}, {
"hash" : "f72a8e99a5d269b117d2e317a4d374dff8895270",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f72a8e99a5d269b117d2e317a4d374dff8895270",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7889741d56a1cec46822ac747eb9159854dc0f17 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7403)
* f72a8e99a5d269b117d2e317a4d374dff8895270 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9675] Add safety check to Savepoint when earliest commit retained is not set in cleaner output [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13660:
URL: https://github.com/apache/hudi/pull/13660#discussion_r2250261651
##########
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/table/action/savepoint/TestSavepointActionExecutor.java:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.savepoint;
+
+import org.apache.hudi.avro.model.HoodieActionInstant;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.common.engine.HoodieLocalEngineContext;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.InstantGenerator;
+import org.apache.hudi.common.table.timeline.versioning.v2.InstantComparatorV2;
+import org.apache.hudi.common.testutils.MockHoodieTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.storage.hadoop.HadoopStorageConfiguration;
+import org.apache.hudi.table.HoodieTable;
+
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.util.stream.Stream;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.mockito.Mockito.RETURNS_DEEP_STUBS;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+class TestSavepointActionExecutor {
+ private final HoodieTable mockTable = mock(HoodieTable.class, RETURNS_DEEP_STUBS);
+ private final HoodieLocalEngineContext engineContext = new HoodieLocalEngineContext(new HadoopStorageConfiguration(false));
+ private final String instantTime = "20240121101012345";
+
+ @Test
+ void testLastCommitRetained_noCleanCommits() {
+ String expectedInstant = "20240101101012345";
+ when(mockTable.getCompletedCommitsTimeline().firstInstant()).thenReturn(Option.of(new HoodieInstant(HoodieInstant.State.COMPLETED, HoodieTimeline.COMMIT_ACTION, expectedInstant,
+ InstantComparatorV2.REQUESTED_TIME_BASED_COMPARATOR)));
+ when(mockTable.getCleanTimeline().lastInstant()).thenReturn(Option.empty());
+
+ SavepointActionExecutor executor = new SavepointActionExecutor(engineContext, null, mockTable, instantTime, "user", "comment");
+ assertEquals(expectedInstant, executor.getLastCommitRetained());
+ }
+
+ @Test
+ void testLastCommitRetained_completedClean() throws IOException {
+ String expectedInstant = "20240101101012345";
+ HoodieInstant cleanInstant = new HoodieInstant(HoodieInstant.State.COMPLETED, HoodieTimeline.CLEAN_ACTION, "20240111101012345",
+ InstantComparatorV2.REQUESTED_TIME_BASED_COMPARATOR);
+ when(mockTable.getCompletedCommitsTimeline().firstInstant()).thenReturn(Option.of(new HoodieInstant(HoodieInstant.State.COMPLETED, HoodieTimeline.COMMIT_ACTION, expectedInstant,
+ InstantComparatorV2.REQUESTED_TIME_BASED_COMPARATOR)));
+ when(mockTable.getCleanTimeline().lastInstant()).thenReturn(Option.of(cleanInstant));
+
+ HoodieCleanMetadata cleanMetadata = new HoodieCleanMetadata();
+ cleanMetadata.setEarliestCommitToRetain(expectedInstant);
+ when(mockTable.getActiveTimeline().readCleanMetadata(cleanInstant)).thenReturn(cleanMetadata);
+
+ SavepointActionExecutor executor = new SavepointActionExecutor(engineContext, null, mockTable, instantTime, "user", "comment");
+ assertEquals(expectedInstant, executor.getLastCommitRetained());
+ }
+
+ @Test
+ void testLastCommitRetained_requestedClean() throws IOException {
+ String expectedInstant = "20240101101012345";
+ String cleanRequested = "20240111101012345";
+ HoodieInstant cleanInstant = new HoodieInstant(HoodieInstant.State.REQUESTED, HoodieTimeline.CLEAN_ACTION, cleanRequested,
+ InstantComparatorV2.REQUESTED_TIME_BASED_COMPARATOR);
+ InstantGenerator instantGenerator = mock(InstantGenerator.class);
+ when(mockTable.getInstantGenerator()).thenReturn(instantGenerator);
+ when(instantGenerator.createNewInstant(HoodieInstant.State.REQUESTED, HoodieTimeline.CLEAN_ACTION, cleanRequested))
+ .thenReturn(cleanInstant);
+
+ when(mockTable.getCompletedCommitsTimeline().firstInstant()).thenReturn(Option.of(new HoodieInstant(HoodieInstant.State.COMPLETED, HoodieTimeline.COMMIT_ACTION, expectedInstant,
+ InstantComparatorV2.REQUESTED_TIME_BASED_COMPARATOR)));
+ when(mockTable.getCleanTimeline().lastInstant()).thenReturn(Option.of(cleanInstant));
+
+ HoodieCleanerPlan cleanerPlan = new HoodieCleanerPlan();
+ HoodieActionInstant lastInstantToRetain = HoodieActionInstant.newBuilder().setAction(HoodieTimeline.COMMIT_ACTION)
+ .setState(HoodieInstant.State.COMPLETED.name()).setTimestamp(expectedInstant).build();
+ cleanerPlan.setEarliestInstantToRetain(lastInstantToRetain);
+ when(mockTable.getActiveTimeline().readCleanerPlan(cleanInstant)).thenReturn(cleanerPlan);
+
+ SavepointActionExecutor executor = new SavepointActionExecutor(engineContext, null, mockTable, instantTime, "user", "comment");
+ assertEquals(expectedInstant, executor.getLastCommitRetained());
+ }
+
+ @Test
+ void testLastCommitRetained_completedCleanWithoutEarliestInstantRetained() throws IOException {
+ String expectedInstant = "20240110101012345";
+ HoodieInstant cleanInstant = new HoodieInstant(HoodieInstant.State.COMPLETED, HoodieTimeline.CLEAN_ACTION, "20240111101012345",
+ InstantComparatorV2.REQUESTED_TIME_BASED_COMPARATOR);
+ when(mockTable.getCompletedCommitsTimeline().firstInstant()).thenReturn(Option.of(new HoodieInstant(HoodieInstant.State.COMPLETED, HoodieTimeline.COMMIT_ACTION, expectedInstant,
+ InstantComparatorV2.REQUESTED_TIME_BASED_COMPARATOR)));
+ when(mockTable.getCleanTimeline().lastInstant()).thenReturn(Option.of(cleanInstant));
+
+ HoodieCleanMetadata cleanMetadata = new HoodieCleanMetadata();
+ when(mockTable.getActiveTimeline().readCleanMetadata(cleanInstant)).thenReturn(cleanMetadata);
+
+ MockHoodieTimeline mockTimeline = new MockHoodieTimeline(Stream.of("20240108101012345", "20240109101012345", expectedInstant), Stream.empty());
Review Comment:
Added an instant greater than the clean
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9675] Add safety check to Savepoint when earliest commit retained is not set in cleaner output [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13660:
URL: https://github.com/apache/hudi/pull/13660#discussion_r2246840455
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/savepoint/SavepointActionExecutor.java:
##########
@@ -145,10 +132,41 @@ public HoodieSavepointMetadata execute() {
.saveAsComplete(
true, instantGenerator.createNewInstant(HoodieInstant.State.INFLIGHT, HoodieTimeline.SAVEPOINT_ACTION, instantTime), Option.of(metadata),
savepointCompletedInstant -> table.getMetaClient().getTableFormat().savepoint(savepointCompletedInstant, table.getContext(), table.getMetaClient(), table.getViewManager()));
- LOG.info("Savepoint " + instantTime + " created");
+ LOG.info("Savepoint {} created", instantTime);
return metadata;
} catch (HoodieIOException e) {
throw new HoodieSavepointException("Failed to savepoint " + instantTime, e);
}
}
+
+ @VisibleForTesting
+ String getLastCommitRetained() {
+ Option<HoodieInstant> cleanInstant = table.getCleanTimeline().lastInstant();
+ // if there are no clean instants, we can use the first completed commit
+ if (cleanInstant.isEmpty()) {
+ return table.getCompletedCommitsTimeline().firstInstant().get().requestedTime();
+ }
+ return cleanInstant.map(instant -> {
+ try {
+ if (instant.isCompleted()) {
+ return table.getActiveTimeline().readCleanMetadata(instant)
+ .getEarliestCommitToRetain();
+ } else {
+ // clean is pending or inflight
+ return table.getActiveTimeline().readCleanerPlan(
+ instantGenerator.createNewInstant(REQUESTED, instant.getAction(), instant.requestedTime()))
+ .getEarliestInstantToRetain().getTimestamp();
+ }
+ } catch (IOException e) {
+ throw new HoodieSavepointException("Failed to savepoint " + instantTime, e);
+ }
+ }).orElseGet(() ->
+ // If there is no earliest commit to retain, but there are clean instants on the timeline, we must assume KEEP_LATEST_FILE_VERSIONS policy is used.
+ // In that case, the last commit before the clean instant is the last commit guaranteed to be retained.
+ table.getActiveTimeline().getWriteTimeline().filterCompletedInstants()
+ .filter(instant -> compareTimestamps(instant.requestedTime(), LESSER_THAN_OR_EQUALS, cleanInstant.get().requestedTime()))
Review Comment:
This is not consistent with the comment above "Cannot allow savepoint time on a commit that could have been cleaned", should it be `GREATER_THAN` instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9675] Add safety check to Savepoint when earliest commit retained is not set in cleaner output [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13660:
URL: https://github.com/apache/hudi/pull/13660#issuecomment-3149007778
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7889741d56a1cec46822ac747eb9159854dc0f17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7403",
"triggerID" : "7889741d56a1cec46822ac747eb9159854dc0f17",
"triggerType" : "PUSH"
}, {
"hash" : "f72a8e99a5d269b117d2e317a4d374dff8895270",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7407",
"triggerID" : "f72a8e99a5d269b117d2e317a4d374dff8895270",
"triggerType" : "PUSH"
}, {
"hash" : "0fc30596ccd89b8931ec13f456158778d01a456c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7496",
"triggerID" : "0fc30596ccd89b8931ec13f456158778d01a456c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fc30596ccd89b8931ec13f456158778d01a456c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7496)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9675] Add safety check to Savepoint when earliest commit retained is not set in cleaner output [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13660:
URL: https://github.com/apache/hudi/pull/13660#issuecomment-3148916276
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7889741d56a1cec46822ac747eb9159854dc0f17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7403",
"triggerID" : "7889741d56a1cec46822ac747eb9159854dc0f17",
"triggerType" : "PUSH"
}, {
"hash" : "f72a8e99a5d269b117d2e317a4d374dff8895270",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7407",
"triggerID" : "f72a8e99a5d269b117d2e317a4d374dff8895270",
"triggerType" : "PUSH"
}, {
"hash" : "0fc30596ccd89b8931ec13f456158778d01a456c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7496",
"triggerID" : "0fc30596ccd89b8931ec13f456158778d01a456c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f72a8e99a5d269b117d2e317a4d374dff8895270 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7407)
* 0fc30596ccd89b8931ec13f456158778d01a456c Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7496)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9675] Add safety check to Savepoint when earliest commit retained is not set in cleaner output [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13660:
URL: https://github.com/apache/hudi/pull/13660#issuecomment-3140921248
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7889741d56a1cec46822ac747eb9159854dc0f17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7403",
"triggerID" : "7889741d56a1cec46822ac747eb9159854dc0f17",
"triggerType" : "PUSH"
}, {
"hash" : "f72a8e99a5d269b117d2e317a4d374dff8895270",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7407",
"triggerID" : "f72a8e99a5d269b117d2e317a4d374dff8895270",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f72a8e99a5d269b117d2e317a4d374dff8895270 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7407)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9675] Add safety check to Savepoint when earliest commit retained is not set in cleaner output [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13660:
URL: https://github.com/apache/hudi/pull/13660#discussion_r2248274732
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/savepoint/SavepointActionExecutor.java:
##########
@@ -145,10 +132,41 @@ public HoodieSavepointMetadata execute() {
.saveAsComplete(
true, instantGenerator.createNewInstant(HoodieInstant.State.INFLIGHT, HoodieTimeline.SAVEPOINT_ACTION, instantTime), Option.of(metadata),
savepointCompletedInstant -> table.getMetaClient().getTableFormat().savepoint(savepointCompletedInstant, table.getContext(), table.getMetaClient(), table.getViewManager()));
- LOG.info("Savepoint " + instantTime + " created");
+ LOG.info("Savepoint {} created", instantTime);
return metadata;
} catch (HoodieIOException e) {
throw new HoodieSavepointException("Failed to savepoint " + instantTime, e);
}
}
+
+ @VisibleForTesting
+ String getLastCommitRetained() {
+ Option<HoodieInstant> cleanInstant = table.getCleanTimeline().lastInstant();
+ // if there are no clean instants, we can use the first completed commit
+ if (cleanInstant.isEmpty()) {
+ return table.getCompletedCommitsTimeline().firstInstant().get().requestedTime();
+ }
+ return cleanInstant.map(instant -> {
+ try {
+ if (instant.isCompleted()) {
+ return table.getActiveTimeline().readCleanMetadata(instant)
+ .getEarliestCommitToRetain();
+ } else {
+ // clean is pending or inflight
+ return table.getActiveTimeline().readCleanerPlan(
+ instantGenerator.createNewInstant(REQUESTED, instant.getAction(), instant.requestedTime()))
+ .getEarliestInstantToRetain().getTimestamp();
+ }
+ } catch (IOException e) {
+ throw new HoodieSavepointException("Failed to savepoint " + instantTime, e);
+ }
+ }).orElseGet(() ->
+ // If there is no earliest commit to retain, but there are clean instants on the timeline, we must assume KEEP_LATEST_FILE_VERSIONS policy is used.
+ // In that case, the last commit before the clean instant is the last commit guaranteed to be retained.
+ table.getActiveTimeline().getWriteTimeline().filterCompletedInstants()
+ .filter(instant -> compareTimestamps(instant.requestedTime(), LESSER_THAN_OR_EQUALS, cleanInstant.get().requestedTime()))
Review Comment:
This is searching for the last commit before the clean because that is the last known valid point in time where all data files are guaranteed to be present since the clean will not touch the files in the current view of the table when it runs its plan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9675] Add safety check to Savepoint when earliest commit retained is not set in cleaner output [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13660:
URL: https://github.com/apache/hudi/pull/13660#discussion_r2249099542
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/savepoint/SavepointActionExecutor.java:
##########
@@ -145,10 +132,41 @@ public HoodieSavepointMetadata execute() {
.saveAsComplete(
true, instantGenerator.createNewInstant(HoodieInstant.State.INFLIGHT, HoodieTimeline.SAVEPOINT_ACTION, instantTime), Option.of(metadata),
savepointCompletedInstant -> table.getMetaClient().getTableFormat().savepoint(savepointCompletedInstant, table.getContext(), table.getMetaClient(), table.getViewManager()));
- LOG.info("Savepoint " + instantTime + " created");
+ LOG.info("Savepoint {} created", instantTime);
return metadata;
} catch (HoodieIOException e) {
throw new HoodieSavepointException("Failed to savepoint " + instantTime, e);
}
}
+
+ @VisibleForTesting
+ String getLastCommitRetained() {
+ Option<HoodieInstant> cleanInstant = table.getCleanTimeline().lastInstant();
+ // if there are no clean instants, we can use the first completed commit
+ if (cleanInstant.isEmpty()) {
+ return table.getCompletedCommitsTimeline().firstInstant().get().requestedTime();
+ }
+ return cleanInstant.map(instant -> {
+ try {
+ if (instant.isCompleted()) {
+ return table.getActiveTimeline().readCleanMetadata(instant)
+ .getEarliestCommitToRetain();
+ } else {
+ // clean is pending or inflight
+ return table.getActiveTimeline().readCleanerPlan(
+ instantGenerator.createNewInstant(REQUESTED, instant.getAction(), instant.requestedTime()))
+ .getEarliestInstantToRetain().getTimestamp();
+ }
+ } catch (IOException e) {
+ throw new HoodieSavepointException("Failed to savepoint " + instantTime, e);
+ }
+ }).orElseGet(() ->
+ // If there is no earliest commit to retain, but there are clean instants on the timeline, we must assume KEEP_LATEST_FILE_VERSIONS policy is used.
+ // In that case, the last commit before the clean instant is the last commit guaranteed to be retained.
+ table.getActiveTimeline().getWriteTimeline().filterCompletedInstants()
+ .filter(instant -> compareTimestamps(instant.requestedTime(), LESSER_THAN_OR_EQUALS, cleanInstant.get().requestedTime()))
Review Comment:
Not really, you can filter the `getCompletedCommitsTimeline()` to find the commits that is `<=` the latest clean instant, if there is return it otherwise return the clean instant.
```java
return getCompletedCommitsTimeline().getInstantStream.filter(instant -> instant <= cleanInstant).lastInstant.orElse(cleanInstant);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9675] Add safety check to Savepoint when earliest commit retained is not set in cleaner output [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13660:
URL: https://github.com/apache/hudi/pull/13660#discussion_r2249098752
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/savepoint/SavepointActionExecutor.java:
##########
@@ -145,10 +132,41 @@ public HoodieSavepointMetadata execute() {
.saveAsComplete(
true, instantGenerator.createNewInstant(HoodieInstant.State.INFLIGHT, HoodieTimeline.SAVEPOINT_ACTION, instantTime), Option.of(metadata),
savepointCompletedInstant -> table.getMetaClient().getTableFormat().savepoint(savepointCompletedInstant, table.getContext(), table.getMetaClient(), table.getViewManager()));
- LOG.info("Savepoint " + instantTime + " created");
+ LOG.info("Savepoint {} created", instantTime);
return metadata;
} catch (HoodieIOException e) {
throw new HoodieSavepointException("Failed to savepoint " + instantTime, e);
}
}
+
+ @VisibleForTesting
+ String getLastCommitRetained() {
+ Option<HoodieInstant> cleanInstant = table.getCleanTimeline().lastInstant();
+ // if there are no clean instants, we can use the first completed commit
+ if (cleanInstant.isEmpty()) {
+ return table.getCompletedCommitsTimeline().firstInstant().get().requestedTime();
+ }
+ return cleanInstant.map(instant -> {
+ try {
+ if (instant.isCompleted()) {
+ return table.getActiveTimeline().readCleanMetadata(instant)
+ .getEarliestCommitToRetain();
+ } else {
+ // clean is pending or inflight
+ return table.getActiveTimeline().readCleanerPlan(
+ instantGenerator.createNewInstant(REQUESTED, instant.getAction(), instant.requestedTime()))
+ .getEarliestInstantToRetain().getTimestamp();
+ }
+ } catch (IOException e) {
+ throw new HoodieSavepointException("Failed to savepoint " + instantTime, e);
+ }
+ }).orElseGet(() ->
+ // If there is no earliest commit to retain, but there are clean instants on the timeline, we must assume KEEP_LATEST_FILE_VERSIONS policy is used.
Review Comment:
The logic in this patch is correct as a best effort I think, that is to find "the last commit finished before the clean instant".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9675] Add safety check to Savepoint when earliest commit retained is not set in cleaner output [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13660:
URL: https://github.com/apache/hudi/pull/13660#issuecomment-3140220547
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7889741d56a1cec46822ac747eb9159854dc0f17",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7403",
"triggerID" : "7889741d56a1cec46822ac747eb9159854dc0f17",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7889741d56a1cec46822ac747eb9159854dc0f17 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7403)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9675] Add safety check to Savepoint when earliest commit retained is not set in cleaner output [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #13660:
URL: https://github.com/apache/hudi/pull/13660
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9675] Add safety check to Savepoint when earliest commit retained is not set in cleaner output [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13660:
URL: https://github.com/apache/hudi/pull/13660#issuecomment-3140214017
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7889741d56a1cec46822ac747eb9159854dc0f17",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7889741d56a1cec46822ac747eb9159854dc0f17",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7889741d56a1cec46822ac747eb9159854dc0f17 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9675] Add safety check to Savepoint when earliest commit retained is not set in cleaner output [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13660:
URL: https://github.com/apache/hudi/pull/13660#discussion_r2249092664
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/savepoint/SavepointActionExecutor.java:
##########
@@ -145,10 +132,41 @@ public HoodieSavepointMetadata execute() {
.saveAsComplete(
true, instantGenerator.createNewInstant(HoodieInstant.State.INFLIGHT, HoodieTimeline.SAVEPOINT_ACTION, instantTime), Option.of(metadata),
savepointCompletedInstant -> table.getMetaClient().getTableFormat().savepoint(savepointCompletedInstant, table.getContext(), table.getMetaClient(), table.getViewManager()));
- LOG.info("Savepoint " + instantTime + " created");
+ LOG.info("Savepoint {} created", instantTime);
return metadata;
} catch (HoodieIOException e) {
throw new HoodieSavepointException("Failed to savepoint " + instantTime, e);
}
}
+
+ @VisibleForTesting
+ String getLastCommitRetained() {
+ Option<HoodieInstant> cleanInstant = table.getCleanTimeline().lastInstant();
+ // if there are no clean instants, we can use the first completed commit
+ if (cleanInstant.isEmpty()) {
+ return table.getCompletedCommitsTimeline().firstInstant().get().requestedTime();
+ }
+ return cleanInstant.map(instant -> {
+ try {
+ if (instant.isCompleted()) {
+ return table.getActiveTimeline().readCleanMetadata(instant)
+ .getEarliestCommitToRetain();
+ } else {
+ // clean is pending or inflight
+ return table.getActiveTimeline().readCleanerPlan(
+ instantGenerator.createNewInstant(REQUESTED, instant.getAction(), instant.requestedTime()))
+ .getEarliestInstantToRetain().getTimestamp();
+ }
+ } catch (IOException e) {
+ throw new HoodieSavepointException("Failed to savepoint " + instantTime, e);
+ }
+ }).orElseGet(() ->
+ // If there is no earliest commit to retain, but there are clean instants on the timeline, we must assume KEEP_LATEST_FILE_VERSIONS policy is used.
+ // In that case, the last commit before the clean instant is the last commit guaranteed to be retained.
+ table.getActiveTimeline().getWriteTimeline().filterCompletedInstants()
+ .filter(instant -> compareTimestamps(instant.requestedTime(), LESSER_THAN_OR_EQUALS, cleanInstant.get().requestedTime()))
Review Comment:
I am looking for the last commit before the latest clean instant, not the one after it. Is there any good utility for that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9675] Add safety check to Savepoint when earliest commit retained is not set in cleaner output [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13660:
URL: https://github.com/apache/hudi/pull/13660#discussion_r2249062634
##########
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/table/action/savepoint/TestSavepointActionExecutor.java:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.savepoint;
+
+import org.apache.hudi.avro.model.HoodieActionInstant;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.common.engine.HoodieLocalEngineContext;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.InstantGenerator;
+import org.apache.hudi.common.table.timeline.versioning.v2.InstantComparatorV2;
+import org.apache.hudi.common.testutils.MockHoodieTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.storage.hadoop.HadoopStorageConfiguration;
+import org.apache.hudi.table.HoodieTable;
+
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.util.stream.Stream;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.mockito.Mockito.RETURNS_DEEP_STUBS;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+class TestSavepointActionExecutor {
+ private final HoodieTable mockTable = mock(HoodieTable.class, RETURNS_DEEP_STUBS);
+ private final HoodieLocalEngineContext engineContext = new HoodieLocalEngineContext(new HadoopStorageConfiguration(false));
+ private final String instantTime = "20240121101012345";
+
+ @Test
+ void testLastCommitRetained_noCleanCommits() {
+ String expectedInstant = "20240101101012345";
+ when(mockTable.getCompletedCommitsTimeline().firstInstant()).thenReturn(Option.of(new HoodieInstant(HoodieInstant.State.COMPLETED, HoodieTimeline.COMMIT_ACTION, expectedInstant,
+ InstantComparatorV2.REQUESTED_TIME_BASED_COMPARATOR)));
+ when(mockTable.getCleanTimeline().lastInstant()).thenReturn(Option.empty());
+
+ SavepointActionExecutor executor = new SavepointActionExecutor(engineContext, null, mockTable, instantTime, "user", "comment");
+ assertEquals(expectedInstant, executor.getLastCommitRetained());
+ }
+
+ @Test
+ void testLastCommitRetained_completedClean() throws IOException {
+ String expectedInstant = "20240101101012345";
+ HoodieInstant cleanInstant = new HoodieInstant(HoodieInstant.State.COMPLETED, HoodieTimeline.CLEAN_ACTION, "20240111101012345",
+ InstantComparatorV2.REQUESTED_TIME_BASED_COMPARATOR);
+ when(mockTable.getCompletedCommitsTimeline().firstInstant()).thenReturn(Option.of(new HoodieInstant(HoodieInstant.State.COMPLETED, HoodieTimeline.COMMIT_ACTION, expectedInstant,
+ InstantComparatorV2.REQUESTED_TIME_BASED_COMPARATOR)));
+ when(mockTable.getCleanTimeline().lastInstant()).thenReturn(Option.of(cleanInstant));
+
+ HoodieCleanMetadata cleanMetadata = new HoodieCleanMetadata();
+ cleanMetadata.setEarliestCommitToRetain(expectedInstant);
+ when(mockTable.getActiveTimeline().readCleanMetadata(cleanInstant)).thenReturn(cleanMetadata);
+
+ SavepointActionExecutor executor = new SavepointActionExecutor(engineContext, null, mockTable, instantTime, "user", "comment");
+ assertEquals(expectedInstant, executor.getLastCommitRetained());
+ }
+
+ @Test
+ void testLastCommitRetained_requestedClean() throws IOException {
+ String expectedInstant = "20240101101012345";
+ String cleanRequested = "20240111101012345";
+ HoodieInstant cleanInstant = new HoodieInstant(HoodieInstant.State.REQUESTED, HoodieTimeline.CLEAN_ACTION, cleanRequested,
+ InstantComparatorV2.REQUESTED_TIME_BASED_COMPARATOR);
+ InstantGenerator instantGenerator = mock(InstantGenerator.class);
+ when(mockTable.getInstantGenerator()).thenReturn(instantGenerator);
+ when(instantGenerator.createNewInstant(HoodieInstant.State.REQUESTED, HoodieTimeline.CLEAN_ACTION, cleanRequested))
+ .thenReturn(cleanInstant);
+
+ when(mockTable.getCompletedCommitsTimeline().firstInstant()).thenReturn(Option.of(new HoodieInstant(HoodieInstant.State.COMPLETED, HoodieTimeline.COMMIT_ACTION, expectedInstant,
+ InstantComparatorV2.REQUESTED_TIME_BASED_COMPARATOR)));
+ when(mockTable.getCleanTimeline().lastInstant()).thenReturn(Option.of(cleanInstant));
+
+ HoodieCleanerPlan cleanerPlan = new HoodieCleanerPlan();
+ HoodieActionInstant lastInstantToRetain = HoodieActionInstant.newBuilder().setAction(HoodieTimeline.COMMIT_ACTION)
+ .setState(HoodieInstant.State.COMPLETED.name()).setTimestamp(expectedInstant).build();
+ cleanerPlan.setEarliestInstantToRetain(lastInstantToRetain);
+ when(mockTable.getActiveTimeline().readCleanerPlan(cleanInstant)).thenReturn(cleanerPlan);
+
+ SavepointActionExecutor executor = new SavepointActionExecutor(engineContext, null, mockTable, instantTime, "user", "comment");
+ assertEquals(expectedInstant, executor.getLastCommitRetained());
+ }
+
+ @Test
+ void testLastCommitRetained_completedCleanWithoutEarliestInstantRetained() throws IOException {
+ String expectedInstant = "20240110101012345";
+ HoodieInstant cleanInstant = new HoodieInstant(HoodieInstant.State.COMPLETED, HoodieTimeline.CLEAN_ACTION, "20240111101012345",
+ InstantComparatorV2.REQUESTED_TIME_BASED_COMPARATOR);
+ when(mockTable.getCompletedCommitsTimeline().firstInstant()).thenReturn(Option.of(new HoodieInstant(HoodieInstant.State.COMPLETED, HoodieTimeline.COMMIT_ACTION, expectedInstant,
+ InstantComparatorV2.REQUESTED_TIME_BASED_COMPARATOR)));
+ when(mockTable.getCleanTimeline().lastInstant()).thenReturn(Option.of(cleanInstant));
+
+ HoodieCleanMetadata cleanMetadata = new HoodieCleanMetadata();
+ when(mockTable.getActiveTimeline().readCleanMetadata(cleanInstant)).thenReturn(cleanMetadata);
+
+ MockHoodieTimeline mockTimeline = new MockHoodieTimeline(Stream.of("20240108101012345", "20240109101012345", expectedInstant), Stream.empty());
Review Comment:
can we add more completed instants greater than latest completed clean in this test as well.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/savepoint/SavepointActionExecutor.java:
##########
@@ -145,10 +132,41 @@ public HoodieSavepointMetadata execute() {
.saveAsComplete(
true, instantGenerator.createNewInstant(HoodieInstant.State.INFLIGHT, HoodieTimeline.SAVEPOINT_ACTION, instantTime), Option.of(metadata),
savepointCompletedInstant -> table.getMetaClient().getTableFormat().savepoint(savepointCompletedInstant, table.getContext(), table.getMetaClient(), table.getViewManager()));
- LOG.info("Savepoint " + instantTime + " created");
+ LOG.info("Savepoint {} created", instantTime);
return metadata;
} catch (HoodieIOException e) {
throw new HoodieSavepointException("Failed to savepoint " + instantTime, e);
}
}
+
+ @VisibleForTesting
+ String getLastCommitRetained() {
+ Option<HoodieInstant> cleanInstant = table.getCleanTimeline().lastInstant();
+ // if there are no clean instants, we can use the first completed commit
+ if (cleanInstant.isEmpty()) {
+ return table.getCompletedCommitsTimeline().firstInstant().get().requestedTime();
+ }
+ return cleanInstant.map(instant -> {
+ try {
+ if (instant.isCompleted()) {
+ return table.getActiveTimeline().readCleanMetadata(instant)
+ .getEarliestCommitToRetain();
+ } else {
+ // clean is pending or inflight
+ return table.getActiveTimeline().readCleanerPlan(
+ instantGenerator.createNewInstant(REQUESTED, instant.getAction(), instant.requestedTime()))
+ .getEarliestInstantToRetain().getTimestamp();
+ }
+ } catch (IOException e) {
+ throw new HoodieSavepointException("Failed to savepoint " + instantTime, e);
+ }
+ }).orElseGet(() ->
+ // If there is no earliest commit to retain, but there are clean instants on the timeline, we must assume KEEP_LATEST_FILE_VERSIONS policy is used.
+ // In that case, the last commit before the clean instant is the last commit guaranteed to be retained.
+ table.getActiveTimeline().getWriteTimeline().filterCompletedInstants()
Review Comment:
if earliest commit to retain is not found, can we add a warn log to recommend users to switch to KEEP_LATEST_COMMITS or KEEP_LATEST_BY_HOURS if they are looking to savepoint earlier commits.
we can still proceed w/ whats best we could do for current savepoint trigger.
but user can switch the clean policy from next time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9675] Add safety check to Savepoint when earliest commit retained is not set in cleaner output [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13660:
URL: https://github.com/apache/hudi/pull/13660#issuecomment-3140240833
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7889741d56a1cec46822ac747eb9159854dc0f17",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7403",
"triggerID" : "7889741d56a1cec46822ac747eb9159854dc0f17",
"triggerType" : "PUSH"
}, {
"hash" : "f72a8e99a5d269b117d2e317a4d374dff8895270",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f72a8e99a5d269b117d2e317a4d374dff8895270",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7889741d56a1cec46822ac747eb9159854dc0f17 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7403)
* f72a8e99a5d269b117d2e317a4d374dff8895270 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9675] Add safety check to Savepoint when earliest commit retained is not set in cleaner output [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13660:
URL: https://github.com/apache/hudi/pull/13660#issuecomment-3140429339
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7889741d56a1cec46822ac747eb9159854dc0f17",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7403",
"triggerID" : "7889741d56a1cec46822ac747eb9159854dc0f17",
"triggerType" : "PUSH"
}, {
"hash" : "f72a8e99a5d269b117d2e317a4d374dff8895270",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7407",
"triggerID" : "f72a8e99a5d269b117d2e317a4d374dff8895270",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7889741d56a1cec46822ac747eb9159854dc0f17 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7403)
* f72a8e99a5d269b117d2e317a4d374dff8895270 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7407)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9675] Add safety check to Savepoint when earliest commit retained is not set in cleaner output [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13660:
URL: https://github.com/apache/hudi/pull/13660#discussion_r2248269285
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/savepoint/SavepointActionExecutor.java:
##########
@@ -145,10 +132,41 @@ public HoodieSavepointMetadata execute() {
.saveAsComplete(
true, instantGenerator.createNewInstant(HoodieInstant.State.INFLIGHT, HoodieTimeline.SAVEPOINT_ACTION, instantTime), Option.of(metadata),
savepointCompletedInstant -> table.getMetaClient().getTableFormat().savepoint(savepointCompletedInstant, table.getContext(), table.getMetaClient(), table.getViewManager()));
- LOG.info("Savepoint " + instantTime + " created");
+ LOG.info("Savepoint {} created", instantTime);
return metadata;
} catch (HoodieIOException e) {
throw new HoodieSavepointException("Failed to savepoint " + instantTime, e);
}
}
+
+ @VisibleForTesting
+ String getLastCommitRetained() {
+ Option<HoodieInstant> cleanInstant = table.getCleanTimeline().lastInstant();
+ // if there are no clean instants, we can use the first completed commit
+ if (cleanInstant.isEmpty()) {
+ return table.getCompletedCommitsTimeline().firstInstant().get().requestedTime();
+ }
+ return cleanInstant.map(instant -> {
+ try {
+ if (instant.isCompleted()) {
+ return table.getActiveTimeline().readCleanMetadata(instant)
+ .getEarliestCommitToRetain();
+ } else {
+ // clean is pending or inflight
+ return table.getActiveTimeline().readCleanerPlan(
+ instantGenerator.createNewInstant(REQUESTED, instant.getAction(), instant.requestedTime()))
+ .getEarliestInstantToRetain().getTimestamp();
+ }
+ } catch (IOException e) {
+ throw new HoodieSavepointException("Failed to savepoint " + instantTime, e);
+ }
+ }).orElseGet(() ->
+ // If there is no earliest commit to retain, but there are clean instants on the timeline, we must assume KEEP_LATEST_FILE_VERSIONS policy is used.
Review Comment:
If the cleaning is not incremental and this is not set, how can we determine if the savepoint instant is valid? Is there some other cleaner helper method we can reuse?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9675] Add safety check to Savepoint when earliest commit retained is not set in cleaner output [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13660:
URL: https://github.com/apache/hudi/pull/13660#discussion_r2249090253
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/savepoint/SavepointActionExecutor.java:
##########
@@ -145,10 +132,41 @@ public HoodieSavepointMetadata execute() {
.saveAsComplete(
true, instantGenerator.createNewInstant(HoodieInstant.State.INFLIGHT, HoodieTimeline.SAVEPOINT_ACTION, instantTime), Option.of(metadata),
savepointCompletedInstant -> table.getMetaClient().getTableFormat().savepoint(savepointCompletedInstant, table.getContext(), table.getMetaClient(), table.getViewManager()));
- LOG.info("Savepoint " + instantTime + " created");
+ LOG.info("Savepoint {} created", instantTime);
return metadata;
} catch (HoodieIOException e) {
throw new HoodieSavepointException("Failed to savepoint " + instantTime, e);
}
}
+
+ @VisibleForTesting
+ String getLastCommitRetained() {
+ Option<HoodieInstant> cleanInstant = table.getCleanTimeline().lastInstant();
+ // if there are no clean instants, we can use the first completed commit
+ if (cleanInstant.isEmpty()) {
+ return table.getCompletedCommitsTimeline().firstInstant().get().requestedTime();
+ }
+ return cleanInstant.map(instant -> {
+ try {
+ if (instant.isCompleted()) {
+ return table.getActiveTimeline().readCleanMetadata(instant)
+ .getEarliestCommitToRetain();
+ } else {
+ // clean is pending or inflight
+ return table.getActiveTimeline().readCleanerPlan(
+ instantGenerator.createNewInstant(REQUESTED, instant.getAction(), instant.requestedTime()))
+ .getEarliestInstantToRetain().getTimestamp();
+ }
+ } catch (IOException e) {
+ throw new HoodieSavepointException("Failed to savepoint " + instantTime, e);
+ }
+ }).orElseGet(() ->
+ // If there is no earliest commit to retain, but there are clean instants on the timeline, we must assume KEEP_LATEST_FILE_VERSIONS policy is used.
+ // In that case, the last commit before the clean instant is the last commit guaranteed to be retained.
+ table.getActiveTimeline().getWriteTimeline().filterCompletedInstants()
+ .filter(instant -> compareTimestamps(instant.requestedTime(), LESSER_THAN_OR_EQUALS, cleanInstant.get().requestedTime()))
Review Comment:
yeah, the `GREATER_THAN` check is still in line82 so we should be good, actually here you want to find the smallest instant that is greater than the latest clean instant. You can reuse the `getCompletedCommitsTimeline()` in line 147 and get the last instant of it, if is greater than then last clean instant, return it otherwise just reuturn the clean instant.
```java
HoodieInstant lastCommit = getCompletedCommitsTimeline().lastInstant().orElse(null);
return lastCommit != null && lastCommit.greaterThan(cleanInstant)? lastCommit : cleanInstant;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9675] Add safety check to Savepoint when earliest commit retained is not set in cleaner output [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13660:
URL: https://github.com/apache/hudi/pull/13660#issuecomment-3148914043
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7889741d56a1cec46822ac747eb9159854dc0f17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7403",
"triggerID" : "7889741d56a1cec46822ac747eb9159854dc0f17",
"triggerType" : "PUSH"
}, {
"hash" : "f72a8e99a5d269b117d2e317a4d374dff8895270",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7407",
"triggerID" : "f72a8e99a5d269b117d2e317a4d374dff8895270",
"triggerType" : "PUSH"
}, {
"hash" : "0fc30596ccd89b8931ec13f456158778d01a456c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc30596ccd89b8931ec13f456158778d01a456c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f72a8e99a5d269b117d2e317a4d374dff8895270 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7407)
* 0fc30596ccd89b8931ec13f456158778d01a456c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org