You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "stream2000 (via GitHub)" <gi...@apache.org> on 2023/02/02 11:41:35 UTC
[GitHub] [hudi] stream2000 opened a new pull request, #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
stream2000 opened a new pull request, #7826:
URL: https://github.com/apache/hudi/pull/7826
### Change Logs
There are 2 steps to getInstantsToRollback for lazy clean
1. get all inflight instants excluding compaction and clusteringÂ
2. check whether the heartbeat for the instant is expired. if true, add it to the result of instantsToRollback
This approach will cause a problem. If the instant is completed after step 1, the heartbeat file of the instant will be deleted.
Then the instant will be counted as an expired instant, and included in the result of instantsToRollback, which is a fatal error.
We fixed it by adding an extra step 3. After checking whether the heartbeat is expired, we reload the active timeline and check if the expired instant is an inflight instant.
### Impact
fix lazy clean schedule rollback on completed instant
### Risk level (write none, low medium or high below)
low
### Documentation Update
None
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Change Logs and Impact were stated clearly
- [x] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #7826:
URL: https://github.com/apache/hudi/pull/7826#discussion_r1146292616
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieTableServiceClient.java:
##########
@@ -707,20 +709,37 @@ protected List<String> getInstantsToRollback(HoodieTableMetaClient metaClient, H
}
}).collect(Collectors.toList());
} else if (cleaningPolicy.isLazy()) {
- return inflightInstantsStream.filter(instant -> {
- try {
- return heartbeatClient.isHeartbeatExpired(instant.getTimestamp());
- } catch (IOException io) {
- throw new HoodieException("Failed to check heartbeat for instant " + instant, io);
- }
- }).map(HoodieInstant::getTimestamp).collect(Collectors.toList());
+ return getInstantsToRollbackForLazyCleanPolicy(metaClient, inflightInstantsStream);
} else if (cleaningPolicy.isNever()) {
return Collections.emptyList();
} else {
throw new IllegalArgumentException("Invalid Failed Writes Cleaning Policy " + config.getFailedWritesCleanPolicy());
}
}
+ @VisibleForTesting
+ public List<String> getInstantsToRollbackForLazyCleanPolicy(HoodieTableMetaClient metaClient,
+ Stream<HoodieInstant> inflightInstantsStream) {
+ // Get expired instants, must store them into list before double-checking
+ List<String> expiredInstants = inflightInstantsStream.filter(instant -> {
+ try {
+ // An instant transformed from inflight to completed have no heartbeat file and will be detected as expired instant here
+ return heartbeatClient.isHeartbeatExpired(instant.getTimestamp());
+ } catch (IOException io) {
+ throw new HoodieException("Failed to check heartbeat for instant " + instant, io);
+ }
+ }).map(HoodieInstant::getTimestamp).collect(Collectors.toList());
+
+ if (!expiredInstants.isEmpty()) {
Review Comment:
@nsivabalan Hi, all comments are addressed. Now we will check the existence of commit file instead of reloading the timeline when doing the double check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1530790896
a related PR https://github.com/apache/hudi/pull/7469
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #7826:
URL: https://github.com/apache/hudi/pull/7826#discussion_r1186918090
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/TestHoodieClientMultiWriter.java:
##########
@@ -545,6 +546,116 @@ public void testMultiWriterWithAsyncTableServicesWithConflict(HoodieTableType ta
assertTrue(validInstants.containsAll(completedInstants));
}
+ @ParameterizedTest
+ @EnumSource(value = HoodieTableType.class, names = {"MERGE_ON_READ"})
+ public void testMultiWriterWithAsyncLazyCleanRollback(HoodieTableType tableType) throws Exception {
+ // create inserts X 1
+ if (tableType == HoodieTableType.MERGE_ON_READ) {
+ setUpMORTestTable();
+ }
+ // Disabling embedded timeline server, it doesn't work with multiwriter
+ HoodieWriteConfig.Builder writeConfigBuilder = getConfigBuilder()
+ .withCleanConfig(HoodieCleanConfig.newBuilder()
+ .withAutoClean(false)
+ .withAsyncClean(true)
+ .withFailedWritesCleaningPolicy(HoodieFailedWritesCleaningPolicy.LAZY).build())
+ .withCompactionConfig(HoodieCompactionConfig.newBuilder()
+ .withInlineCompaction(false)
+ .withMaxNumDeltaCommitsBeforeCompaction(2).build())
+ .withEmbeddedTimelineServerEnabled(false)
+ // Timeline-server-based markers are not used for multi-writer tests
+ .withMarkersType(MarkerType.DIRECT.name())
+ .withFileSystemViewConfig(FileSystemViewStorageConfig.newBuilder().withStorageType(
+ FileSystemViewStorageType.MEMORY).build())
+ .withWriteConcurrencyMode(WriteConcurrencyMode.OPTIMISTIC_CONCURRENCY_CONTROL)
+ // Set the config so that heartbeat will expire in 1 second without update
+ .withLockConfig(HoodieLockConfig.newBuilder().withLockProvider(InProcessLockProvider.class)
+ .build()).withAutoCommit(false).withProperties(lockProperties);
+ Set<String> validInstants = new HashSet<>();
+ // Create the first commit with inserts
+ HoodieWriteConfig cfg = writeConfigBuilder.build();
+ SparkRDDWriteClient client = getHoodieWriteClient(cfg);
+ createCommitWithInserts(cfg, client, "000", "001", 200, true);
+ validInstants.add("001");
+
+ // Three clients running actions in parallel
+ final int threadCount = 3;
+ final ExecutorService executor = Executors.newFixedThreadPool(threadCount);
+
+ final SparkRDDWriteClient client1 = getHoodieWriteClient(cfg);
+ final SparkRDDWriteClient client2 = getHoodieWriteClient(cfg);
+ final String commitTime2 = "002";
+ final String commitTime3 = "003";
+ AtomicReference<Object> writeStatus1 = new AtomicReference<>(null);
+ AtomicReference<Object> writeStatus2 = new AtomicReference<>(null);
+
+ // Create upserts, schedule cleaning, schedule compaction in parallel
+ Future future1 = executor.submit(() -> {
+ final int numRecords = 100;
+ assertDoesNotThrow(() -> {
+ writeStatus1.set(createCommitWithInserts(cfg, client1, "001", commitTime2, numRecords, false));
+ });
+ });
+ // Create upserts, schedule cleaning, schedule compaction in parallel
+ Future future2 = executor.submit(() -> {
+ final int numRecords = 100;
+ assertDoesNotThrow(() -> {
+ writeStatus2.set(createCommitWithInserts(cfg, client2, "001", commitTime3, numRecords, false));
+ });
+ });
+
+ future1.get();
+ future2.get();
+
+ HoodieTableMetaClient tableMetaClient = client.getTableServiceClient().createMetaClient(true);
+
+ // Get inflight instant stream before commit
+ List<HoodieInstant> inflightInstants = client
+ .getTableServiceClient()
+ .getInflightTimelineExcludeCompactionAndClustering(tableMetaClient)
+ .getReverseOrderedInstants()
+ .collect(Collectors.toList());
+
+ // Commit the instants and get instants to rollback in parallel
+ future1 = executor.submit(() -> {
+ long start = System.currentTimeMillis();
+ LOG.info(String.format("Start to commit instant %s", commitTime2));
+ client1.commit(commitTime2, writeStatus1.get());
+ validInstants.add(commitTime2);
+ LOG.info(String.format("commit the instant cost %d ms", System.currentTimeMillis() - start));
+ });
+
+ future2 = executor.submit(() -> {
+ long start = System.currentTimeMillis();
+ LOG.info(String.format("Start to commit instant %s", commitTime3));
+ client2.commit(commitTime3, writeStatus2.get());
+ validInstants.add(commitTime3);
+ LOG.info(String.format("commit the instant %s cost %d ms", commitTime3, System.currentTimeMillis() - start));
+ });
+
+ Future future3 = executor.submit(() -> {
+ // Sleep to let the writer complete the instants
+ assertDoesNotThrow(() -> Thread.sleep(6000));
Review Comment:
we should try to use CountDownLatch to deterministically proceed here.
for eg, lets define a countdownlatch w/ count 2. and count them down at L 624 and 633. And here at L 638, we just need to await for the countdown latch to get to 0 and proceed. also, lets do a timed await and not wait indefinitely.
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/TestHoodieClientMultiWriter.java:
##########
@@ -545,6 +546,116 @@ public void testMultiWriterWithAsyncTableServicesWithConflict(HoodieTableType ta
assertTrue(validInstants.containsAll(completedInstants));
}
+ @ParameterizedTest
+ @EnumSource(value = HoodieTableType.class, names = {"MERGE_ON_READ"})
+ public void testMultiWriterWithAsyncLazyCleanRollback(HoodieTableType tableType) throws Exception {
Review Comment:
why not both table types ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1552322372
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1415246582
For multi-writer senarios, we suggest to set up the lazy cleaning strategy, did you set up that for both Flink and Spark conf already?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1552341440
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0ce6917cfa857601ae31780553fd685291a0bc47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17147",
"triggerID" : "0ce6917cfa857601ae31780553fd685291a0bc47",
"triggerType" : "PUSH"
}, {
"hash" : "0ce6917cfa857601ae31780553fd685291a0bc47",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17163",
"triggerID" : "1479155691",
"triggerType" : "MANUAL"
}, {
"hash" : "0ce6917cfa857601ae31780553fd685291a0bc47",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17163",
"triggerID" : "1477699232",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1477699232",
"triggerType" : "MANUAL"
}, {
"hash" : "0ce6917cfa857601ae31780553fd685291a0bc47",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17163",
"triggerID" : "1552322372",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] LinMingQiang commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "LinMingQiang (via GitHub)" <gi...@apache.org>.
LinMingQiang commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1537969149
I think you can determine the reason why the heartbeat file does not exist in `isHeartbeatExpired`, If the heartbeat file does not exist, determine whether the current instant has been completed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #7826:
URL: https://github.com/apache/hudi/pull/7826#discussion_r1098259478
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieTableServiceClient.java:
##########
@@ -707,20 +709,33 @@ protected List<String> getInstantsToRollback(HoodieTableMetaClient metaClient, H
}
}).collect(Collectors.toList());
} else if (cleaningPolicy.isLazy()) {
- return inflightInstantsStream.filter(instant -> {
- try {
- return heartbeatClient.isHeartbeatExpired(instant.getTimestamp());
- } catch (IOException io) {
- throw new HoodieException("Failed to check heartbeat for instant " + instant, io);
- }
- }).map(HoodieInstant::getTimestamp).collect(Collectors.toList());
+ return getInstantsToRollbackForLazyCleanPolicy(metaClient, inflightInstantsStream);
} else if (cleaningPolicy.isNever()) {
return Collections.emptyList();
} else {
throw new IllegalArgumentException("Invalid Failed Writes Cleaning Policy " + config.getFailedWritesCleanPolicy());
}
}
+ @VisibleForTesting
+ public List<String> getInstantsToRollbackForLazyCleanPolicy(HoodieTableMetaClient metaClient,
+ Stream<HoodieInstant> inflightInstantsStream) {
+ // Get expired instants, must store them into list before double-checking
+ List<String> expiredInstants = inflightInstantsStream.filter(instant -> {
+ try {
+ // An instant transformed from inflight to completed have no heartbeat file and will be detected as expired instant here
+ return heartbeatClient.isHeartbeatExpired(instant.getTimestamp());
+ } catch (IOException io) {
+ throw new HoodieException("Failed to check heartbeat for instant " + instant, io);
+ }
+ }).map(HoodieInstant::getTimestamp).collect(Collectors.toList());
+
+ // Double check whether the heartbeat-expired instant is an inflight instant
+ metaClient.reloadActiveTimeline();
Review Comment:
Only heartbeat-expired instants will be rollback. If the instant is finished after `metaClient.reloadActiveTimeline`, the heartbeat is not expired when checking heartbeat, so it won't be rollback. That why we must check heartbeat first and then reload the timeline to do the double check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1413946486
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14873",
"triggerID" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"triggerType" : "PUSH"
}, {
"hash" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14873)
* 0346edf996c9c8a3eb44aa6dd4bcf258578d3f72 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1475775778
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14873",
"triggerID" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"triggerType" : "PUSH"
}, {
"hash" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14875",
"triggerID" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"triggerType" : "PUSH"
}, {
"hash" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15074",
"triggerID" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"triggerType" : "PUSH"
}, {
"hash" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15805",
"triggerID" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* db7cd7aa1f23a635c3a4b0b71ea136231a76f520 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15074)
* 6832b7787f8363cb09e672d8675d8bc68bad3a2c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15805)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1534064726
> a related PR #7469
The iusse that this PR is trying to solve not only happens in multi-writer scenarios but also in single writer with async lazy clean
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1551838077
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0ce6917cfa857601ae31780553fd685291a0bc47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17147",
"triggerID" : "0ce6917cfa857601ae31780553fd685291a0bc47",
"triggerType" : "PUSH"
}, {
"hash" : "0000",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1477699232",
"triggerType" : "MANUAL"
}, {
"hash" : "0ce6917cfa857601ae31780553fd685291a0bc47",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17163",
"triggerID" : "1479155691",
"triggerType" : "MANUAL"
}, {
"hash" : "0ce6917cfa857601ae31780553fd685291a0bc47",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17163",
"triggerID" : "1477699232",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 0000 Unknown: [CANCELED](TBD)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1550845588
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14873",
"triggerID" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"triggerType" : "PUSH"
}, {
"hash" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14875",
"triggerID" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"triggerType" : "PUSH"
}, {
"hash" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15074",
"triggerID" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"triggerType" : "PUSH"
}, {
"hash" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15805",
"triggerID" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"triggerType" : "PUSH"
}, {
"hash" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15828",
"triggerID" : "1477699232",
"triggerType" : "MANUAL"
}, {
"hash" : "ed708d668d1521a10b4baa62d5c79aaefdd326f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15835",
"triggerID" : "ed708d668d1521a10b4baa62d5c79aaefdd326f1",
"triggerType" : "PUSH"
}, {
"hash" : "b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15842",
"triggerID" : "b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566",
"triggerType" : "PUSH"
}, {
"hash" : "b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15851",
"triggerID" : "1479155691",
"triggerType" : "MANUAL"
}, {
"hash" : "e76f56f4223f346ab0a7cb17ae67beb75d89fa67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16814",
"triggerID" : "e76f56f4223f346ab0a7cb17ae67beb75d89fa67",
"triggerType" : "PUSH"
}, {
"hash" : "ac053c8d3f59609327fca976670c8720ca9455e1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16933",
"triggerID" : "ac053c8d3f59609327fca976670c8720ca9455e1",
"triggerType" : "PUSH"
}, {
"hash" : "d7a06415cfecd30371d2282b2ebe270acfa98ac4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d7a06415cfecd30371d2282b2ebe270acfa98ac4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ac053c8d3f59609327fca976670c8720ca9455e1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16933)
* d7a06415cfecd30371d2282b2ebe270acfa98ac4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #7826:
URL: https://github.com/apache/hudi/pull/7826#discussion_r1183190244
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieTableServiceClient.java:
##########
@@ -707,20 +709,34 @@ protected List<String> getInstantsToRollback(HoodieTableMetaClient metaClient, H
}
}).collect(Collectors.toList());
} else if (cleaningPolicy.isLazy()) {
- return inflightInstantsStream.filter(instant -> {
- try {
- return heartbeatClient.isHeartbeatExpired(instant.getTimestamp());
- } catch (IOException io) {
- throw new HoodieException("Failed to check heartbeat for instant " + instant, io);
- }
- }).map(HoodieInstant::getTimestamp).collect(Collectors.toList());
+ return getInstantsToRollbackForLazyCleanPolicy(metaClient, inflightInstantsStream);
} else if (cleaningPolicy.isNever()) {
return Collections.emptyList();
} else {
throw new IllegalArgumentException("Invalid Failed Writes Cleaning Policy " + config.getFailedWritesCleanPolicy());
}
}
+ @VisibleForTesting
+ public List<String> getInstantsToRollbackForLazyCleanPolicy(HoodieTableMetaClient metaClient,
+ Stream<HoodieInstant> inflightInstantsStream) {
+ // Get expired instants, must store them into list before double-checking
+ List<HoodieInstant> expiredInstants = inflightInstantsStream.filter(instant -> {
+ try {
+ // An instant transformed from inflight to completed have no heartbeat file and will be detected as expired instant here
+ return heartbeatClient.isHeartbeatExpired(instant.getTimestamp());
+ } catch (IOException io) {
+ throw new HoodieException("Failed to check heartbeat for instant " + instant, io);
+ }
+ }).collect(Collectors.toList());
+
+ // Only return instants that haven't been completed by other writers
+ return expiredInstants.stream()
+ .filter(instant -> !metaClient.getActiveTimeline().isCompletedCommitFileExists(instant))
Review Comment:
Agreed. Lazy clean is usually an async operation and reloading the timeline will not cost too much time if archiving works as expected. We don't need to introduce a new api`isCompletedCommitFileExists` here just to reduce the cost of reloading the timeline in lazy clean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1478975524
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14873",
"triggerID" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"triggerType" : "PUSH"
}, {
"hash" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14875",
"triggerID" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"triggerType" : "PUSH"
}, {
"hash" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15074",
"triggerID" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"triggerType" : "PUSH"
}, {
"hash" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15805",
"triggerID" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"triggerType" : "PUSH"
}, {
"hash" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15828",
"triggerID" : "1477699232",
"triggerType" : "MANUAL"
}, {
"hash" : "ed708d668d1521a10b4baa62d5c79aaefdd326f1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15835",
"triggerID" : "ed708d668d1521a10b4baa62d5c79aaefdd326f1",
"triggerType" : "PUSH"
}, {
"hash" : "b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ed708d668d1521a10b4baa62d5c79aaefdd326f1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15835)
* b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1414187422
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14873",
"triggerID" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"triggerType" : "PUSH"
}, {
"hash" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14875",
"triggerID" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0346edf996c9c8a3eb44aa6dd4bcf258578d3f72 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14875)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1550922565
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14873",
"triggerID" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"triggerType" : "PUSH"
}, {
"hash" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14875",
"triggerID" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"triggerType" : "PUSH"
}, {
"hash" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15074",
"triggerID" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"triggerType" : "PUSH"
}, {
"hash" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15805",
"triggerID" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"triggerType" : "PUSH"
}, {
"hash" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15828",
"triggerID" : "1477699232",
"triggerType" : "MANUAL"
}, {
"hash" : "ed708d668d1521a10b4baa62d5c79aaefdd326f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15835",
"triggerID" : "ed708d668d1521a10b4baa62d5c79aaefdd326f1",
"triggerType" : "PUSH"
}, {
"hash" : "b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15842",
"triggerID" : "b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566",
"triggerType" : "PUSH"
}, {
"hash" : "b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15851",
"triggerID" : "1479155691",
"triggerType" : "MANUAL"
}, {
"hash" : "e76f56f4223f346ab0a7cb17ae67beb75d89fa67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16814",
"triggerID" : "e76f56f4223f346ab0a7cb17ae67beb75d89fa67",
"triggerType" : "PUSH"
}, {
"hash" : "ac053c8d3f59609327fca976670c8720ca9455e1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16933",
"triggerID" : "ac053c8d3f59609327fca976670c8720ca9455e1",
"triggerType" : "PUSH"
}, {
"hash" : "d7a06415cfecd30371d2282b2ebe270acfa98ac4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17142",
"triggerID" : "d7a06415cfecd30371d2282b2ebe270acfa98ac4",
"triggerType" : "PUSH"
}, {
"hash" : "0ce6917cfa857601ae31780553fd685291a0bc47",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0ce6917cfa857601ae31780553fd685291a0bc47",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ac053c8d3f59609327fca976670c8720ca9455e1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16933)
* d7a06415cfecd30371d2282b2ebe270acfa98ac4 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17142)
* 0ce6917cfa857601ae31780553fd685291a0bc47 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1550854787
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14873",
"triggerID" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"triggerType" : "PUSH"
}, {
"hash" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14875",
"triggerID" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"triggerType" : "PUSH"
}, {
"hash" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15074",
"triggerID" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"triggerType" : "PUSH"
}, {
"hash" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15805",
"triggerID" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"triggerType" : "PUSH"
}, {
"hash" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15828",
"triggerID" : "1477699232",
"triggerType" : "MANUAL"
}, {
"hash" : "ed708d668d1521a10b4baa62d5c79aaefdd326f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15835",
"triggerID" : "ed708d668d1521a10b4baa62d5c79aaefdd326f1",
"triggerType" : "PUSH"
}, {
"hash" : "b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15842",
"triggerID" : "b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566",
"triggerType" : "PUSH"
}, {
"hash" : "b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15851",
"triggerID" : "1479155691",
"triggerType" : "MANUAL"
}, {
"hash" : "e76f56f4223f346ab0a7cb17ae67beb75d89fa67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16814",
"triggerID" : "e76f56f4223f346ab0a7cb17ae67beb75d89fa67",
"triggerType" : "PUSH"
}, {
"hash" : "ac053c8d3f59609327fca976670c8720ca9455e1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16933",
"triggerID" : "ac053c8d3f59609327fca976670c8720ca9455e1",
"triggerType" : "PUSH"
}, {
"hash" : "d7a06415cfecd30371d2282b2ebe270acfa98ac4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17142",
"triggerID" : "d7a06415cfecd30371d2282b2ebe270acfa98ac4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ac053c8d3f59609327fca976670c8720ca9455e1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16933)
* d7a06415cfecd30371d2282b2ebe270acfa98ac4 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17142)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1479010116
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14873",
"triggerID" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"triggerType" : "PUSH"
}, {
"hash" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14875",
"triggerID" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"triggerType" : "PUSH"
}, {
"hash" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15074",
"triggerID" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"triggerType" : "PUSH"
}, {
"hash" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15805",
"triggerID" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"triggerType" : "PUSH"
}, {
"hash" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15828",
"triggerID" : "1477699232",
"triggerType" : "MANUAL"
}, {
"hash" : "ed708d668d1521a10b4baa62d5c79aaefdd326f1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15835",
"triggerID" : "ed708d668d1521a10b4baa62d5c79aaefdd326f1",
"triggerType" : "PUSH"
}, {
"hash" : "b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15842",
"triggerID" : "b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ed708d668d1521a10b4baa62d5c79aaefdd326f1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15835)
* b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15842)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1477701562
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14873",
"triggerID" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"triggerType" : "PUSH"
}, {
"hash" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14875",
"triggerID" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"triggerType" : "PUSH"
}, {
"hash" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15074",
"triggerID" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"triggerType" : "PUSH"
}, {
"hash" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15805",
"triggerID" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"triggerType" : "PUSH"
}, {
"hash" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15828",
"triggerID" : "1477699232",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 6832b7787f8363cb09e672d8675d8bc68bad3a2c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15805) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15828)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #7826:
URL: https://github.com/apache/hudi/pull/7826#discussion_r1102210796
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieTableServiceClient.java:
##########
@@ -707,20 +709,33 @@ protected List<String> getInstantsToRollback(HoodieTableMetaClient metaClient, H
}
}).collect(Collectors.toList());
} else if (cleaningPolicy.isLazy()) {
- return inflightInstantsStream.filter(instant -> {
- try {
- return heartbeatClient.isHeartbeatExpired(instant.getTimestamp());
- } catch (IOException io) {
- throw new HoodieException("Failed to check heartbeat for instant " + instant, io);
- }
- }).map(HoodieInstant::getTimestamp).collect(Collectors.toList());
+ return getInstantsToRollbackForLazyCleanPolicy(metaClient, inflightInstantsStream);
} else if (cleaningPolicy.isNever()) {
return Collections.emptyList();
} else {
throw new IllegalArgumentException("Invalid Failed Writes Cleaning Policy " + config.getFailedWritesCleanPolicy());
}
}
+ @VisibleForTesting
+ public List<String> getInstantsToRollbackForLazyCleanPolicy(HoodieTableMetaClient metaClient,
+ Stream<HoodieInstant> inflightInstantsStream) {
+ // Get expired instants, must store them into list before double-checking
+ List<String> expiredInstants = inflightInstantsStream.filter(instant -> {
+ try {
+ // An instant transformed from inflight to completed have no heartbeat file and will be detected as expired instant here
+ return heartbeatClient.isHeartbeatExpired(instant.getTimestamp());
+ } catch (IOException io) {
+ throw new HoodieException("Failed to check heartbeat for instant " + instant, io);
+ }
+ }).map(HoodieInstant::getTimestamp).collect(Collectors.toList());
+
+ // Double check whether the heartbeat-expired instant is an inflight instant
+ metaClient.reloadActiveTimeline();
Review Comment:
> active timeline reload is an expensive operation. we should try to minimize. can you check if exporedInstants is not empty and reload only during such cases ? if infact you can return right away if its empty.
Nice Suggestion. Already fixed it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1534066550
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14873",
"triggerID" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"triggerType" : "PUSH"
}, {
"hash" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14875",
"triggerID" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"triggerType" : "PUSH"
}, {
"hash" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15074",
"triggerID" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"triggerType" : "PUSH"
}, {
"hash" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15805",
"triggerID" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"triggerType" : "PUSH"
}, {
"hash" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15828",
"triggerID" : "1477699232",
"triggerType" : "MANUAL"
}, {
"hash" : "ed708d668d1521a10b4baa62d5c79aaefdd326f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15835",
"triggerID" : "ed708d668d1521a10b4baa62d5c79aaefdd326f1",
"triggerType" : "PUSH"
}, {
"hash" : "b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15842",
"triggerID" : "b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566",
"triggerType" : "PUSH"
}, {
"hash" : "b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15851",
"triggerID" : "1479155691",
"triggerType" : "MANUAL"
}, {
"hash" : "e76f56f4223f346ab0a7cb17ae67beb75d89fa67",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e76f56f4223f346ab0a7cb17ae67beb75d89fa67",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15842) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15851)
* e76f56f4223f346ab0a7cb17ae67beb75d89fa67 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1538000227
> I think you can determine the reason why the heartbeat file does not exist in `isHeartbeatExpired`, If the heartbeat file does not exist, determine whether the current instant has been completed.
@nsivabalan What do you think? We can also solve the problem by adding an extra step at `HoodieHeartbeatUtils#isHeartbeatExpired`. When we find that the heartbeat file is missing, also check whether the completed commit file is present so that the instants what `HoodieHeartbeatUtils#isHeartbeatExpired` returned are indeed expired instants.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1529103489
sorry, dropped the ball. reviewing it again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a diff in pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "leesf (via GitHub)" <gi...@apache.org>.
leesf commented on code in PR #7826:
URL: https://github.com/apache/hudi/pull/7826#discussion_r1098254914
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieTableServiceClient.java:
##########
@@ -707,20 +709,33 @@ protected List<String> getInstantsToRollback(HoodieTableMetaClient metaClient, H
}
}).collect(Collectors.toList());
} else if (cleaningPolicy.isLazy()) {
- return inflightInstantsStream.filter(instant -> {
- try {
- return heartbeatClient.isHeartbeatExpired(instant.getTimestamp());
- } catch (IOException io) {
- throw new HoodieException("Failed to check heartbeat for instant " + instant, io);
- }
- }).map(HoodieInstant::getTimestamp).collect(Collectors.toList());
+ return getInstantsToRollbackForLazyCleanPolicy(metaClient, inflightInstantsStream);
} else if (cleaningPolicy.isNever()) {
return Collections.emptyList();
} else {
throw new IllegalArgumentException("Invalid Failed Writes Cleaning Policy " + config.getFailedWritesCleanPolicy());
}
}
+ @VisibleForTesting
+ public List<String> getInstantsToRollbackForLazyCleanPolicy(HoodieTableMetaClient metaClient,
+ Stream<HoodieInstant> inflightInstantsStream) {
+ // Get expired instants, must store them into list before double-checking
+ List<String> expiredInstants = inflightInstantsStream.filter(instant -> {
+ try {
+ // An instant transformed from inflight to completed have no heartbeat file and will be detected as expired instant here
+ return heartbeatClient.isHeartbeatExpired(instant.getTimestamp());
+ } catch (IOException io) {
+ throw new HoodieException("Failed to check heartbeat for instant " + instant, io);
+ }
+ }).map(HoodieInstant::getTimestamp).collect(Collectors.toList());
+
+ // Double check whether the heartbeat-expired instant is an inflight instant
+ metaClient.reloadActiveTimeline();
Review Comment:
even reload active timeline, there is still possibility to rollback completed instant if the instant completed after ` metaClient.reloadActiveTimeline`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1413958629
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14873",
"triggerID" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"triggerType" : "PUSH"
}, {
"hash" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14875",
"triggerID" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14873)
* 0346edf996c9c8a3eb44aa6dd4bcf258578d3f72 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14875)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1413934311
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14873",
"triggerID" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14873)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #7826:
URL: https://github.com/apache/hudi/pull/7826#discussion_r1108374607
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieTableServiceClient.java:
##########
@@ -707,20 +709,33 @@ protected List<String> getInstantsToRollback(HoodieTableMetaClient metaClient, H
}
}).collect(Collectors.toList());
} else if (cleaningPolicy.isLazy()) {
- return inflightInstantsStream.filter(instant -> {
- try {
- return heartbeatClient.isHeartbeatExpired(instant.getTimestamp());
- } catch (IOException io) {
- throw new HoodieException("Failed to check heartbeat for instant " + instant, io);
- }
- }).map(HoodieInstant::getTimestamp).collect(Collectors.toList());
+ return getInstantsToRollbackForLazyCleanPolicy(metaClient, inflightInstantsStream);
} else if (cleaningPolicy.isNever()) {
return Collections.emptyList();
} else {
throw new IllegalArgumentException("Invalid Failed Writes Cleaning Policy " + config.getFailedWritesCleanPolicy());
}
}
+ @VisibleForTesting
+ public List<String> getInstantsToRollbackForLazyCleanPolicy(HoodieTableMetaClient metaClient,
+ Stream<HoodieInstant> inflightInstantsStream) {
+ // Get expired instants, must store them into list before double-checking
+ List<String> expiredInstants = inflightInstantsStream.filter(instant -> {
+ try {
+ // An instant transformed from inflight to completed have no heartbeat file and will be detected as expired instant here
+ return heartbeatClient.isHeartbeatExpired(instant.getTimestamp());
+ } catch (IOException io) {
+ throw new HoodieException("Failed to check heartbeat for instant " + instant, io);
+ }
+ }).map(HoodieInstant::getTimestamp).collect(Collectors.toList());
+
+ // Double check whether the heartbeat-expired instant is an inflight instant
+ metaClient.reloadActiveTimeline();
Review Comment:
> > Yes，we use a separate spark job to run table services and the instant in completed by the flink writer
>
> But from the code, the compaction and clustering instants are excluded, can you elaborate a little more, what kind of table service the Spark offline job takes over?
We use spark offline job to run clean/archive, and lazy clean will rollback failed instants
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1432416236
@nsivabalan Hi, all comments have been addressed, could you please help to review the changes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1479130033
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14873",
"triggerID" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"triggerType" : "PUSH"
}, {
"hash" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14875",
"triggerID" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"triggerType" : "PUSH"
}, {
"hash" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15074",
"triggerID" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"triggerType" : "PUSH"
}, {
"hash" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15805",
"triggerID" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"triggerType" : "PUSH"
}, {
"hash" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15828",
"triggerID" : "1477699232",
"triggerType" : "MANUAL"
}, {
"hash" : "ed708d668d1521a10b4baa62d5c79aaefdd326f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15835",
"triggerID" : "ed708d668d1521a10b4baa62d5c79aaefdd326f1",
"triggerType" : "PUSH"
}, {
"hash" : "b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15842",
"triggerID" : "b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15842)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1415208883
> I'm wondering how this senario happens:
>
> > If the instant is completed after step 1,
>
> Is here another writer there?
Yes，we use a separate spark job to run table services and the instant in completed by the flink writer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1425295693
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14873",
"triggerID" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"triggerType" : "PUSH"
}, {
"hash" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14875",
"triggerID" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"triggerType" : "PUSH"
}, {
"hash" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15074",
"triggerID" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* db7cd7aa1f23a635c3a4b0b71ea136231a76f520 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15074)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1475766502
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14873",
"triggerID" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"triggerType" : "PUSH"
}, {
"hash" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14875",
"triggerID" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"triggerType" : "PUSH"
}, {
"hash" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15074",
"triggerID" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"triggerType" : "PUSH"
}, {
"hash" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* db7cd7aa1f23a635c3a4b0b71ea136231a76f520 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15074)
* 6832b7787f8363cb09e672d8675d8bc68bad3a2c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #7826:
URL: https://github.com/apache/hudi/pull/7826#discussion_r1187056760
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/TestHoodieClientMultiWriter.java:
##########
@@ -545,6 +546,116 @@ public void testMultiWriterWithAsyncTableServicesWithConflict(HoodieTableType ta
assertTrue(validInstants.containsAll(completedInstants));
}
+ @ParameterizedTest
+ @EnumSource(value = HoodieTableType.class, names = {"MERGE_ON_READ"})
+ public void testMultiWriterWithAsyncLazyCleanRollback(HoodieTableType tableType) throws Exception {
Review Comment:
added COPY_ON_WRITE
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #7826:
URL: https://github.com/apache/hudi/pull/7826#discussion_r1195279932
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieTableServiceClient.java:
##########
@@ -759,20 +759,37 @@ protected List<String> getInstantsToRollback(HoodieTableMetaClient metaClient, H
}
}).collect(Collectors.toList());
} else if (cleaningPolicy.isLazy()) {
- return inflightInstantsStream.filter(instant -> {
- try {
- return heartbeatClient.isHeartbeatExpired(instant.getTimestamp());
- } catch (IOException io) {
- throw new HoodieException("Failed to check heartbeat for instant " + instant, io);
- }
- }).map(HoodieInstant::getTimestamp).collect(Collectors.toList());
+ return getInstantsToRollbackForLazyCleanPolicy(metaClient, inflightInstantsStream);
} else if (cleaningPolicy.isNever()) {
return Collections.emptyList();
} else {
throw new IllegalArgumentException("Invalid Failed Writes Cleaning Policy " + config.getFailedWritesCleanPolicy());
}
}
+ protected List<String> getInstantsToRollbackForLazyCleanPolicy(HoodieTableMetaClient metaClient,
+ Stream<HoodieInstant> inflightInstantsStream) {
+ // Get expired instants, must store them into list before double-checking
+ List<String> expiredInstants = inflightInstantsStream.filter(instant -> {
+ try {
+ // An instant transformed from inflight to completed have no heartbeat file and will be detected as expired instant here
+ return heartbeatClient.isHeartbeatExpired(instant.getTimestamp());
+ } catch (IOException io) {
+ throw new HoodieException("Failed to check heartbeat for instant " + instant, io);
+ }
+ }).map(HoodieInstant::getTimestamp).collect(Collectors.toList());
+
+ if (!expiredInstants.isEmpty()) {
+ LOG.info("Found expired instants to rollback for lazy clean: " + String.join(",", expiredInstants));
+ // Only return instants that haven't been completed by other writers
+ metaClient.reloadActiveTimeline();
+ HoodieTimeline latestInflightTimeline = getInflightTimelineExcludeCompactionAndClustering(metaClient);
Review Comment:
"refreshed" is more accurate
```suggestion
HoodieTimeline refreshedInflightTimeline = getInflightTimelineExcludeCompactionAndClustering(metaClient);
```
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieTableServiceClient.java:
##########
@@ -759,20 +759,37 @@ protected List<String> getInstantsToRollback(HoodieTableMetaClient metaClient, H
}
}).collect(Collectors.toList());
} else if (cleaningPolicy.isLazy()) {
- return inflightInstantsStream.filter(instant -> {
- try {
- return heartbeatClient.isHeartbeatExpired(instant.getTimestamp());
- } catch (IOException io) {
- throw new HoodieException("Failed to check heartbeat for instant " + instant, io);
- }
- }).map(HoodieInstant::getTimestamp).collect(Collectors.toList());
+ return getInstantsToRollbackForLazyCleanPolicy(metaClient, inflightInstantsStream);
} else if (cleaningPolicy.isNever()) {
return Collections.emptyList();
} else {
throw new IllegalArgumentException("Invalid Failed Writes Cleaning Policy " + config.getFailedWritesCleanPolicy());
}
}
+ protected List<String> getInstantsToRollbackForLazyCleanPolicy(HoodieTableMetaClient metaClient,
Review Comment:
`protected` implies potentially this can be overridden by subclasses, while in fact this is not meant for overriding. We should avoid this access modifier misuse. If you want to test it, use visiblefortesing and make it package private
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/TestHoodieClientMultiWriter.java:
##########
@@ -545,6 +546,120 @@ public void testMultiWriterWithAsyncTableServicesWithConflict(HoodieTableType ta
assertTrue(validInstants.containsAll(completedInstants));
}
+ @ParameterizedTest
+ @EnumSource(value = HoodieTableType.class, names = {"MERGE_ON_READ", "COPY_ON_WRITE"})
+ public void testMultiWriterWithAsyncLazyCleanRollback(HoodieTableType tableType) throws Exception {
+ // create inserts X 1
+ if (tableType == HoodieTableType.MERGE_ON_READ) {
+ setUpMORTestTable();
+ }
+ // Disabling embedded timeline server, it doesn't work with multiwriter
+ HoodieWriteConfig.Builder writeConfigBuilder = getConfigBuilder()
+ .withCleanConfig(HoodieCleanConfig.newBuilder()
+ .withAutoClean(false)
+ .withAsyncClean(true)
+ .withFailedWritesCleaningPolicy(HoodieFailedWritesCleaningPolicy.LAZY).build())
+ .withCompactionConfig(HoodieCompactionConfig.newBuilder()
+ .withInlineCompaction(false)
+ .withMaxNumDeltaCommitsBeforeCompaction(2).build())
+ .withEmbeddedTimelineServerEnabled(false)
+ // Timeline-server-based markers are not used for multi-writer tests
+ .withMarkersType(MarkerType.DIRECT.name())
+ .withFileSystemViewConfig(FileSystemViewStorageConfig.newBuilder().withStorageType(
+ FileSystemViewStorageType.MEMORY).build())
+ .withWriteConcurrencyMode(WriteConcurrencyMode.OPTIMISTIC_CONCURRENCY_CONTROL)
+ // Set the config so that heartbeat will expire in 1 second without update
+ .withLockConfig(HoodieLockConfig.newBuilder().withLockProvider(InProcessLockProvider.class)
+ .build()).withAutoCommit(false).withProperties(lockProperties);
+ Set<String> validInstants = new HashSet<>();
+ // Create the first commit with inserts
+ HoodieWriteConfig cfg = writeConfigBuilder.build();
+ SparkRDDWriteClient client = getHoodieWriteClient(cfg);
+ createCommitWithInserts(cfg, client, "000", "001", 200, true);
+ validInstants.add("001");
+
+ // Three clients running actions in parallel
+ final int threadCount = 3;
+ final ExecutorService executor = Executors.newFixedThreadPool(threadCount);
+
+ final SparkRDDWriteClient client1 = getHoodieWriteClient(cfg);
+ final SparkRDDWriteClient client2 = getHoodieWriteClient(cfg);
+ final String commitTime2 = "002";
+ final String commitTime3 = "003";
+ AtomicReference<Object> writeStatus1 = new AtomicReference<>(null);
+ AtomicReference<Object> writeStatus2 = new AtomicReference<>(null);
+
+ Future future1 = executor.submit(() -> {
+ final int numRecords = 100;
+ assertDoesNotThrow(() -> {
+ writeStatus1.set(createCommitWithInserts(cfg, client1, "001", commitTime2, numRecords, false));
+ });
+ });
+ Future future2 = executor.submit(() -> {
+ final int numRecords = 100;
+ assertDoesNotThrow(() -> {
+ writeStatus2.set(createCommitWithInserts(cfg, client2, "001", commitTime3, numRecords, false));
+ });
+ });
+
+ future1.get();
+ future2.get();
+
+ final CountDownLatch commitCountDownLatch = new CountDownLatch(2);
+ HoodieTableMetaClient tableMetaClient = client.getTableServiceClient().createMetaClient(true);
+
+ // Get inflight instant stream before commit
+ List<HoodieInstant> inflightInstants = client
+ .getTableServiceClient()
+ .getInflightTimelineExcludeCompactionAndClustering(tableMetaClient)
+ .getReverseOrderedInstants()
+ .collect(Collectors.toList());
+
+ // Commit the instants and get instants to rollback in parallel
+ future1 = executor.submit(() -> {
+ long start = System.currentTimeMillis();
+ LOG.info(String.format("Start to commit instant %s", commitTime2));
+ client1.commit(commitTime2, writeStatus1.get());
+ commitCountDownLatch.countDown();
+ validInstants.add(commitTime2);
+ LOG.info(String.format("commit the instant cost %d ms", System.currentTimeMillis() - start));
+ });
+
+ future2 = executor.submit(() -> {
+ long start = System.currentTimeMillis();
+ LOG.info(String.format("Start to commit instant %s", commitTime3));
+ client2.commit(commitTime3, writeStatus2.get());
+ commitCountDownLatch.countDown();
+ validInstants.add(commitTime3);
+ LOG.info(String.format("commit the instant %s cost %d ms", commitTime3, System.currentTimeMillis() - start));
+ });
+
+ Future future3 = executor.submit(() -> {
+ try {
+ commitCountDownLatch.await(30000, TimeUnit.MILLISECONDS);
+ } catch (InterruptedException e) {
+ //
+ }
+ LOG.info("Start to get instants to rollback");
+ List<String> instantsToRollback =
+ client.getTableServiceClient().getInstantsToRollbackForLazyCleanPolicy(tableMetaClient, inflightInstants.stream());
+ // No instants will be rollback though some instants may be detected as `expired`
+ assertTrue(instantsToRollback.isEmpty());
Review Comment:
why not make commit2 inflight so we have 1 instant to return here while having commit3 completed and not included? this will give a better coverage
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/TestHoodieClientMultiWriter.java:
##########
@@ -545,6 +546,120 @@ public void testMultiWriterWithAsyncTableServicesWithConflict(HoodieTableType ta
assertTrue(validInstants.containsAll(completedInstants));
}
+ @ParameterizedTest
+ @EnumSource(value = HoodieTableType.class, names = {"MERGE_ON_READ", "COPY_ON_WRITE"})
+ public void testMultiWriterWithAsyncLazyCleanRollback(HoodieTableType tableType) throws Exception {
+ // create inserts X 1
+ if (tableType == HoodieTableType.MERGE_ON_READ) {
+ setUpMORTestTable();
+ }
+ // Disabling embedded timeline server, it doesn't work with multiwriter
+ HoodieWriteConfig.Builder writeConfigBuilder = getConfigBuilder()
+ .withCleanConfig(HoodieCleanConfig.newBuilder()
+ .withAutoClean(false)
+ .withAsyncClean(true)
+ .withFailedWritesCleaningPolicy(HoodieFailedWritesCleaningPolicy.LAZY).build())
+ .withCompactionConfig(HoodieCompactionConfig.newBuilder()
+ .withInlineCompaction(false)
+ .withMaxNumDeltaCommitsBeforeCompaction(2).build())
+ .withEmbeddedTimelineServerEnabled(false)
+ // Timeline-server-based markers are not used for multi-writer tests
+ .withMarkersType(MarkerType.DIRECT.name())
+ .withFileSystemViewConfig(FileSystemViewStorageConfig.newBuilder().withStorageType(
+ FileSystemViewStorageType.MEMORY).build())
+ .withWriteConcurrencyMode(WriteConcurrencyMode.OPTIMISTIC_CONCURRENCY_CONTROL)
+ // Set the config so that heartbeat will expire in 1 second without update
+ .withLockConfig(HoodieLockConfig.newBuilder().withLockProvider(InProcessLockProvider.class)
+ .build()).withAutoCommit(false).withProperties(lockProperties);
+ Set<String> validInstants = new HashSet<>();
+ // Create the first commit with inserts
+ HoodieWriteConfig cfg = writeConfigBuilder.build();
+ SparkRDDWriteClient client = getHoodieWriteClient(cfg);
+ createCommitWithInserts(cfg, client, "000", "001", 200, true);
+ validInstants.add("001");
+
+ // Three clients running actions in parallel
+ final int threadCount = 3;
+ final ExecutorService executor = Executors.newFixedThreadPool(threadCount);
+
+ final SparkRDDWriteClient client1 = getHoodieWriteClient(cfg);
+ final SparkRDDWriteClient client2 = getHoodieWriteClient(cfg);
+ final String commitTime2 = "002";
+ final String commitTime3 = "003";
+ AtomicReference<Object> writeStatus1 = new AtomicReference<>(null);
+ AtomicReference<Object> writeStatus2 = new AtomicReference<>(null);
+
+ Future future1 = executor.submit(() -> {
+ final int numRecords = 100;
+ assertDoesNotThrow(() -> {
+ writeStatus1.set(createCommitWithInserts(cfg, client1, "001", commitTime2, numRecords, false));
+ });
+ });
+ Future future2 = executor.submit(() -> {
+ final int numRecords = 100;
+ assertDoesNotThrow(() -> {
+ writeStatus2.set(createCommitWithInserts(cfg, client2, "001", commitTime3, numRecords, false));
+ });
+ });
+
+ future1.get();
+ future2.get();
+
+ final CountDownLatch commitCountDownLatch = new CountDownLatch(2);
+ HoodieTableMetaClient tableMetaClient = client.getTableServiceClient().createMetaClient(true);
+
+ // Get inflight instant stream before commit
+ List<HoodieInstant> inflightInstants = client
+ .getTableServiceClient()
+ .getInflightTimelineExcludeCompactionAndClustering(tableMetaClient)
+ .getReverseOrderedInstants()
+ .collect(Collectors.toList());
+
+ // Commit the instants and get instants to rollback in parallel
+ future1 = executor.submit(() -> {
+ long start = System.currentTimeMillis();
+ LOG.info(String.format("Start to commit instant %s", commitTime2));
+ client1.commit(commitTime2, writeStatus1.get());
+ commitCountDownLatch.countDown();
+ validInstants.add(commitTime2);
+ LOG.info(String.format("commit the instant cost %d ms", System.currentTimeMillis() - start));
+ });
+
+ future2 = executor.submit(() -> {
+ long start = System.currentTimeMillis();
+ LOG.info(String.format("Start to commit instant %s", commitTime3));
+ client2.commit(commitTime3, writeStatus2.get());
+ commitCountDownLatch.countDown();
+ validInstants.add(commitTime3);
+ LOG.info(String.format("commit the instant %s cost %d ms", commitTime3, System.currentTimeMillis() - start));
+ });
+
+ Future future3 = executor.submit(() -> {
+ try {
+ commitCountDownLatch.await(30000, TimeUnit.MILLISECONDS);
+ } catch (InterruptedException e) {
+ //
+ }
+ LOG.info("Start to get instants to rollback");
+ List<String> instantsToRollback =
+ client.getTableServiceClient().getInstantsToRollbackForLazyCleanPolicy(tableMetaClient, inflightInstants.stream());
+ // No instants will be rollback though some instants may be detected as `expired`
+ assertTrue(instantsToRollback.isEmpty());
+ });
+
+ future1.get();
+ future2.get();
+ future3.get();
+
+ validInstants.addAll(
+ metaClient.reloadActiveTimeline().getCompletedReplaceTimeline()
+ .filterCompletedInstants().getInstantsAsStream().map(HoodieInstant::getTimestamp).collect(Collectors.toSet()));
+ Set<String> completedInstants = metaClient.reloadActiveTimeline().getCommitsTimeline()
+ .filterCompletedInstants().getInstantsAsStream().map(HoodieInstant::getTimestamp)
+ .collect(Collectors.toSet());
+ assertTrue(validInstants.containsAll(completedInstants));
Review Comment:
what is the purpose of this assertion here? looks redundant
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieTableServiceClient.java:
##########
@@ -759,20 +759,37 @@ protected List<String> getInstantsToRollback(HoodieTableMetaClient metaClient, H
}
}).collect(Collectors.toList());
} else if (cleaningPolicy.isLazy()) {
- return inflightInstantsStream.filter(instant -> {
- try {
- return heartbeatClient.isHeartbeatExpired(instant.getTimestamp());
- } catch (IOException io) {
- throw new HoodieException("Failed to check heartbeat for instant " + instant, io);
- }
- }).map(HoodieInstant::getTimestamp).collect(Collectors.toList());
+ return getInstantsToRollbackForLazyCleanPolicy(metaClient, inflightInstantsStream);
} else if (cleaningPolicy.isNever()) {
return Collections.emptyList();
} else {
throw new IllegalArgumentException("Invalid Failed Writes Cleaning Policy " + config.getFailedWritesCleanPolicy());
}
}
+ protected List<String> getInstantsToRollbackForLazyCleanPolicy(HoodieTableMetaClient metaClient,
+ Stream<HoodieInstant> inflightInstantsStream) {
+ // Get expired instants, must store them into list before double-checking
+ List<String> expiredInstants = inflightInstantsStream.filter(instant -> {
+ try {
+ // An instant transformed from inflight to completed have no heartbeat file and will be detected as expired instant here
+ return heartbeatClient.isHeartbeatExpired(instant.getTimestamp());
+ } catch (IOException io) {
+ throw new HoodieException("Failed to check heartbeat for instant " + instant, io);
+ }
+ }).map(HoodieInstant::getTimestamp).collect(Collectors.toList());
+
+ if (!expiredInstants.isEmpty()) {
+ LOG.info("Found expired instants to rollback for lazy clean: " + String.join(",", expiredInstants));
Review Comment:
this log is misleading as you may filter it later. it should log the final result as you want to log info here
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/TestHoodieClientMultiWriter.java:
##########
@@ -545,6 +546,120 @@ public void testMultiWriterWithAsyncTableServicesWithConflict(HoodieTableType ta
assertTrue(validInstants.containsAll(completedInstants));
}
+ @ParameterizedTest
+ @EnumSource(value = HoodieTableType.class, names = {"MERGE_ON_READ", "COPY_ON_WRITE"})
+ public void testMultiWriterWithAsyncLazyCleanRollback(HoodieTableType tableType) throws Exception {
+ // create inserts X 1
+ if (tableType == HoodieTableType.MERGE_ON_READ) {
+ setUpMORTestTable();
+ }
+ // Disabling embedded timeline server, it doesn't work with multiwriter
+ HoodieWriteConfig.Builder writeConfigBuilder = getConfigBuilder()
+ .withCleanConfig(HoodieCleanConfig.newBuilder()
+ .withAutoClean(false)
+ .withAsyncClean(true)
+ .withFailedWritesCleaningPolicy(HoodieFailedWritesCleaningPolicy.LAZY).build())
+ .withCompactionConfig(HoodieCompactionConfig.newBuilder()
+ .withInlineCompaction(false)
+ .withMaxNumDeltaCommitsBeforeCompaction(2).build())
+ .withEmbeddedTimelineServerEnabled(false)
+ // Timeline-server-based markers are not used for multi-writer tests
+ .withMarkersType(MarkerType.DIRECT.name())
+ .withFileSystemViewConfig(FileSystemViewStorageConfig.newBuilder().withStorageType(
+ FileSystemViewStorageType.MEMORY).build())
+ .withWriteConcurrencyMode(WriteConcurrencyMode.OPTIMISTIC_CONCURRENCY_CONTROL)
+ // Set the config so that heartbeat will expire in 1 second without update
+ .withLockConfig(HoodieLockConfig.newBuilder().withLockProvider(InProcessLockProvider.class)
+ .build()).withAutoCommit(false).withProperties(lockProperties);
+ Set<String> validInstants = new HashSet<>();
+ // Create the first commit with inserts
+ HoodieWriteConfig cfg = writeConfigBuilder.build();
+ SparkRDDWriteClient client = getHoodieWriteClient(cfg);
+ createCommitWithInserts(cfg, client, "000", "001", 200, true);
+ validInstants.add("001");
+
+ // Three clients running actions in parallel
+ final int threadCount = 3;
+ final ExecutorService executor = Executors.newFixedThreadPool(threadCount);
+
+ final SparkRDDWriteClient client1 = getHoodieWriteClient(cfg);
+ final SparkRDDWriteClient client2 = getHoodieWriteClient(cfg);
+ final String commitTime2 = "002";
+ final String commitTime3 = "003";
+ AtomicReference<Object> writeStatus1 = new AtomicReference<>(null);
+ AtomicReference<Object> writeStatus2 = new AtomicReference<>(null);
+
+ Future future1 = executor.submit(() -> {
+ final int numRecords = 100;
+ assertDoesNotThrow(() -> {
+ writeStatus1.set(createCommitWithInserts(cfg, client1, "001", commitTime2, numRecords, false));
+ });
+ });
+ Future future2 = executor.submit(() -> {
+ final int numRecords = 100;
+ assertDoesNotThrow(() -> {
+ writeStatus2.set(createCommitWithInserts(cfg, client2, "001", commitTime3, numRecords, false));
+ });
+ });
+
+ future1.get();
+ future2.get();
+
+ final CountDownLatch commitCountDownLatch = new CountDownLatch(2);
+ HoodieTableMetaClient tableMetaClient = client.getTableServiceClient().createMetaClient(true);
+
+ // Get inflight instant stream before commit
+ List<HoodieInstant> inflightInstants = client
+ .getTableServiceClient()
+ .getInflightTimelineExcludeCompactionAndClustering(tableMetaClient)
+ .getReverseOrderedInstants()
+ .collect(Collectors.toList());
+
+ // Commit the instants and get instants to rollback in parallel
+ future1 = executor.submit(() -> {
+ long start = System.currentTimeMillis();
+ LOG.info(String.format("Start to commit instant %s", commitTime2));
+ client1.commit(commitTime2, writeStatus1.get());
+ commitCountDownLatch.countDown();
+ validInstants.add(commitTime2);
+ LOG.info(String.format("commit the instant cost %d ms", System.currentTimeMillis() - start));
Review Comment:
this log is noisy
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/TestHoodieClientMultiWriter.java:
##########
@@ -545,6 +546,120 @@ public void testMultiWriterWithAsyncTableServicesWithConflict(HoodieTableType ta
assertTrue(validInstants.containsAll(completedInstants));
}
+ @ParameterizedTest
+ @EnumSource(value = HoodieTableType.class, names = {"MERGE_ON_READ", "COPY_ON_WRITE"})
+ public void testMultiWriterWithAsyncLazyCleanRollback(HoodieTableType tableType) throws Exception {
+ // create inserts X 1
+ if (tableType == HoodieTableType.MERGE_ON_READ) {
+ setUpMORTestTable();
+ }
+ // Disabling embedded timeline server, it doesn't work with multiwriter
+ HoodieWriteConfig.Builder writeConfigBuilder = getConfigBuilder()
+ .withCleanConfig(HoodieCleanConfig.newBuilder()
+ .withAutoClean(false)
+ .withAsyncClean(true)
+ .withFailedWritesCleaningPolicy(HoodieFailedWritesCleaningPolicy.LAZY).build())
+ .withCompactionConfig(HoodieCompactionConfig.newBuilder()
+ .withInlineCompaction(false)
+ .withMaxNumDeltaCommitsBeforeCompaction(2).build())
+ .withEmbeddedTimelineServerEnabled(false)
+ // Timeline-server-based markers are not used for multi-writer tests
+ .withMarkersType(MarkerType.DIRECT.name())
+ .withFileSystemViewConfig(FileSystemViewStorageConfig.newBuilder().withStorageType(
+ FileSystemViewStorageType.MEMORY).build())
+ .withWriteConcurrencyMode(WriteConcurrencyMode.OPTIMISTIC_CONCURRENCY_CONTROL)
+ // Set the config so that heartbeat will expire in 1 second without update
+ .withLockConfig(HoodieLockConfig.newBuilder().withLockProvider(InProcessLockProvider.class)
+ .build()).withAutoCommit(false).withProperties(lockProperties);
+ Set<String> validInstants = new HashSet<>();
+ // Create the first commit with inserts
+ HoodieWriteConfig cfg = writeConfigBuilder.build();
+ SparkRDDWriteClient client = getHoodieWriteClient(cfg);
+ createCommitWithInserts(cfg, client, "000", "001", 200, true);
+ validInstants.add("001");
+
+ // Three clients running actions in parallel
+ final int threadCount = 3;
+ final ExecutorService executor = Executors.newFixedThreadPool(threadCount);
+
+ final SparkRDDWriteClient client1 = getHoodieWriteClient(cfg);
+ final SparkRDDWriteClient client2 = getHoodieWriteClient(cfg);
+ final String commitTime2 = "002";
+ final String commitTime3 = "003";
+ AtomicReference<Object> writeStatus1 = new AtomicReference<>(null);
+ AtomicReference<Object> writeStatus2 = new AtomicReference<>(null);
+
+ Future future1 = executor.submit(() -> {
+ final int numRecords = 100;
+ assertDoesNotThrow(() -> {
+ writeStatus1.set(createCommitWithInserts(cfg, client1, "001", commitTime2, numRecords, false));
+ });
+ });
+ Future future2 = executor.submit(() -> {
+ final int numRecords = 100;
+ assertDoesNotThrow(() -> {
+ writeStatus2.set(createCommitWithInserts(cfg, client2, "001", commitTime3, numRecords, false));
+ });
+ });
+
+ future1.get();
+ future2.get();
+
+ final CountDownLatch commitCountDownLatch = new CountDownLatch(2);
+ HoodieTableMetaClient tableMetaClient = client.getTableServiceClient().createMetaClient(true);
+
+ // Get inflight instant stream before commit
+ List<HoodieInstant> inflightInstants = client
+ .getTableServiceClient()
+ .getInflightTimelineExcludeCompactionAndClustering(tableMetaClient)
+ .getReverseOrderedInstants()
+ .collect(Collectors.toList());
+
+ // Commit the instants and get instants to rollback in parallel
+ future1 = executor.submit(() -> {
+ long start = System.currentTimeMillis();
+ LOG.info(String.format("Start to commit instant %s", commitTime2));
+ client1.commit(commitTime2, writeStatus1.get());
+ commitCountDownLatch.countDown();
+ validInstants.add(commitTime2);
+ LOG.info(String.format("commit the instant cost %d ms", System.currentTimeMillis() - start));
+ });
+
+ future2 = executor.submit(() -> {
+ long start = System.currentTimeMillis();
+ LOG.info(String.format("Start to commit instant %s", commitTime3));
+ client2.commit(commitTime3, writeStatus2.get());
+ commitCountDownLatch.countDown();
+ validInstants.add(commitTime3);
+ LOG.info(String.format("commit the instant %s cost %d ms", commitTime3, System.currentTimeMillis() - start));
Review Comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1551549721
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14873",
"triggerID" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"triggerType" : "PUSH"
}, {
"hash" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14875",
"triggerID" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"triggerType" : "PUSH"
}, {
"hash" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15074",
"triggerID" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"triggerType" : "PUSH"
}, {
"hash" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15805",
"triggerID" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"triggerType" : "PUSH"
}, {
"hash" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15828",
"triggerID" : "1477699232",
"triggerType" : "MANUAL"
}, {
"hash" : "ed708d668d1521a10b4baa62d5c79aaefdd326f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15835",
"triggerID" : "ed708d668d1521a10b4baa62d5c79aaefdd326f1",
"triggerType" : "PUSH"
}, {
"hash" : "b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15842",
"triggerID" : "b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566",
"triggerType" : "PUSH"
}, {
"hash" : "b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15851",
"triggerID" : "1479155691",
"triggerType" : "MANUAL"
}, {
"hash" : "e76f56f4223f346ab0a7cb17ae67beb75d89fa67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16814",
"triggerID" : "e76f56f4223f346ab0a7cb17ae67beb75d89fa67",
"triggerType" : "PUSH"
}, {
"hash" : "ac053c8d3f59609327fca976670c8720ca9455e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16933",
"triggerID" : "ac053c8d3f59609327fca976670c8720ca9455e1",
"triggerType" : "PUSH"
}, {
"hash" : "d7a06415cfecd30371d2282b2ebe270acfa98ac4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17142",
"triggerID" : "d7a06415cfecd30371d2282b2ebe270acfa98ac4",
"triggerType" : "PUSH"
}, {
"hash" : "0ce6917cfa857601ae31780553fd685291a0bc47",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17147",
"triggerID" : "0ce6917cfa857601ae31780553fd685291a0bc47",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0ce6917cfa857601ae31780553fd685291a0bc47 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17147)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1534180237
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14873",
"triggerID" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"triggerType" : "PUSH"
}, {
"hash" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14875",
"triggerID" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"triggerType" : "PUSH"
}, {
"hash" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15074",
"triggerID" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"triggerType" : "PUSH"
}, {
"hash" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15805",
"triggerID" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"triggerType" : "PUSH"
}, {
"hash" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15828",
"triggerID" : "1477699232",
"triggerType" : "MANUAL"
}, {
"hash" : "ed708d668d1521a10b4baa62d5c79aaefdd326f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15835",
"triggerID" : "ed708d668d1521a10b4baa62d5c79aaefdd326f1",
"triggerType" : "PUSH"
}, {
"hash" : "b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15842",
"triggerID" : "b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566",
"triggerType" : "PUSH"
}, {
"hash" : "b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15851",
"triggerID" : "1479155691",
"triggerType" : "MANUAL"
}, {
"hash" : "e76f56f4223f346ab0a7cb17ae67beb75d89fa67",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16814",
"triggerID" : "e76f56f4223f346ab0a7cb17ae67beb75d89fa67",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e76f56f4223f346ab0a7cb17ae67beb75d89fa67 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16814)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #7826:
URL: https://github.com/apache/hudi/pull/7826#discussion_r1101898172
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieTableServiceClient.java:
##########
@@ -707,20 +709,33 @@ protected List<String> getInstantsToRollback(HoodieTableMetaClient metaClient, H
}
}).collect(Collectors.toList());
} else if (cleaningPolicy.isLazy()) {
- return inflightInstantsStream.filter(instant -> {
- try {
- return heartbeatClient.isHeartbeatExpired(instant.getTimestamp());
- } catch (IOException io) {
- throw new HoodieException("Failed to check heartbeat for instant " + instant, io);
- }
- }).map(HoodieInstant::getTimestamp).collect(Collectors.toList());
+ return getInstantsToRollbackForLazyCleanPolicy(metaClient, inflightInstantsStream);
} else if (cleaningPolicy.isNever()) {
return Collections.emptyList();
} else {
throw new IllegalArgumentException("Invalid Failed Writes Cleaning Policy " + config.getFailedWritesCleanPolicy());
}
}
+ @VisibleForTesting
+ public List<String> getInstantsToRollbackForLazyCleanPolicy(HoodieTableMetaClient metaClient,
+ Stream<HoodieInstant> inflightInstantsStream) {
+ // Get expired instants, must store them into list before double-checking
+ List<String> expiredInstants = inflightInstantsStream.filter(instant -> {
+ try {
+ // An instant transformed from inflight to completed have no heartbeat file and will be detected as expired instant here
+ return heartbeatClient.isHeartbeatExpired(instant.getTimestamp());
+ } catch (IOException io) {
+ throw new HoodieException("Failed to check heartbeat for instant " + instant, io);
+ }
+ }).map(HoodieInstant::getTimestamp).collect(Collectors.toList());
+
+ // Double check whether the heartbeat-expired instant is an inflight instant
+ metaClient.reloadActiveTimeline();
Review Comment:
active timeline reload is an expensive operation. we should try to minimize.
can you check if exporedInstants is not empty and reload only during such cases ?
if infact you can return right away if its empty.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1477981749
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14873",
"triggerID" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"triggerType" : "PUSH"
}, {
"hash" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14875",
"triggerID" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"triggerType" : "PUSH"
}, {
"hash" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15074",
"triggerID" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"triggerType" : "PUSH"
}, {
"hash" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15805",
"triggerID" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"triggerType" : "PUSH"
}, {
"hash" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15828",
"triggerID" : "1477699232",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 6832b7787f8363cb09e672d8675d8bc68bad3a2c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15805) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15828)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1478286948
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14873",
"triggerID" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"triggerType" : "PUSH"
}, {
"hash" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14875",
"triggerID" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"triggerType" : "PUSH"
}, {
"hash" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15074",
"triggerID" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"triggerType" : "PUSH"
}, {
"hash" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15805",
"triggerID" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"triggerType" : "PUSH"
}, {
"hash" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15828",
"triggerID" : "1477699232",
"triggerType" : "MANUAL"
}, {
"hash" : "ed708d668d1521a10b4baa62d5c79aaefdd326f1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15835",
"triggerID" : "ed708d668d1521a10b4baa62d5c79aaefdd326f1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6832b7787f8363cb09e672d8675d8bc68bad3a2c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15805) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15828)
* ed708d668d1521a10b4baa62d5c79aaefdd326f1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15835)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1479155691
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1537819683
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14873",
"triggerID" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"triggerType" : "PUSH"
}, {
"hash" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14875",
"triggerID" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"triggerType" : "PUSH"
}, {
"hash" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15074",
"triggerID" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"triggerType" : "PUSH"
}, {
"hash" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15805",
"triggerID" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"triggerType" : "PUSH"
}, {
"hash" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15828",
"triggerID" : "1477699232",
"triggerType" : "MANUAL"
}, {
"hash" : "ed708d668d1521a10b4baa62d5c79aaefdd326f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15835",
"triggerID" : "ed708d668d1521a10b4baa62d5c79aaefdd326f1",
"triggerType" : "PUSH"
}, {
"hash" : "b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15842",
"triggerID" : "b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566",
"triggerType" : "PUSH"
}, {
"hash" : "b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15851",
"triggerID" : "1479155691",
"triggerType" : "MANUAL"
}, {
"hash" : "e76f56f4223f346ab0a7cb17ae67beb75d89fa67",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16814",
"triggerID" : "e76f56f4223f346ab0a7cb17ae67beb75d89fa67",
"triggerType" : "PUSH"
}, {
"hash" : "ac053c8d3f59609327fca976670c8720ca9455e1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16933",
"triggerID" : "ac053c8d3f59609327fca976670c8720ca9455e1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e76f56f4223f346ab0a7cb17ae67beb75d89fa67 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16814)
* ac053c8d3f59609327fca976670c8720ca9455e1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16933)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #7826:
URL: https://github.com/apache/hudi/pull/7826#discussion_r1196008996
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/TestHoodieClientMultiWriter.java:
##########
@@ -545,6 +546,120 @@ public void testMultiWriterWithAsyncTableServicesWithConflict(HoodieTableType ta
assertTrue(validInstants.containsAll(completedInstants));
}
+ @ParameterizedTest
+ @EnumSource(value = HoodieTableType.class, names = {"MERGE_ON_READ", "COPY_ON_WRITE"})
+ public void testMultiWriterWithAsyncLazyCleanRollback(HoodieTableType tableType) throws Exception {
+ // create inserts X 1
+ if (tableType == HoodieTableType.MERGE_ON_READ) {
+ setUpMORTestTable();
+ }
+ // Disabling embedded timeline server, it doesn't work with multiwriter
+ HoodieWriteConfig.Builder writeConfigBuilder = getConfigBuilder()
+ .withCleanConfig(HoodieCleanConfig.newBuilder()
+ .withAutoClean(false)
+ .withAsyncClean(true)
+ .withFailedWritesCleaningPolicy(HoodieFailedWritesCleaningPolicy.LAZY).build())
+ .withCompactionConfig(HoodieCompactionConfig.newBuilder()
+ .withInlineCompaction(false)
+ .withMaxNumDeltaCommitsBeforeCompaction(2).build())
+ .withEmbeddedTimelineServerEnabled(false)
+ // Timeline-server-based markers are not used for multi-writer tests
+ .withMarkersType(MarkerType.DIRECT.name())
+ .withFileSystemViewConfig(FileSystemViewStorageConfig.newBuilder().withStorageType(
+ FileSystemViewStorageType.MEMORY).build())
+ .withWriteConcurrencyMode(WriteConcurrencyMode.OPTIMISTIC_CONCURRENCY_CONTROL)
+ // Set the config so that heartbeat will expire in 1 second without update
+ .withLockConfig(HoodieLockConfig.newBuilder().withLockProvider(InProcessLockProvider.class)
+ .build()).withAutoCommit(false).withProperties(lockProperties);
+ Set<String> validInstants = new HashSet<>();
+ // Create the first commit with inserts
+ HoodieWriteConfig cfg = writeConfigBuilder.build();
+ SparkRDDWriteClient client = getHoodieWriteClient(cfg);
+ createCommitWithInserts(cfg, client, "000", "001", 200, true);
+ validInstants.add("001");
+
+ // Three clients running actions in parallel
+ final int threadCount = 3;
+ final ExecutorService executor = Executors.newFixedThreadPool(threadCount);
+
+ final SparkRDDWriteClient client1 = getHoodieWriteClient(cfg);
+ final SparkRDDWriteClient client2 = getHoodieWriteClient(cfg);
+ final String commitTime2 = "002";
+ final String commitTime3 = "003";
+ AtomicReference<Object> writeStatus1 = new AtomicReference<>(null);
+ AtomicReference<Object> writeStatus2 = new AtomicReference<>(null);
+
+ Future future1 = executor.submit(() -> {
+ final int numRecords = 100;
+ assertDoesNotThrow(() -> {
+ writeStatus1.set(createCommitWithInserts(cfg, client1, "001", commitTime2, numRecords, false));
+ });
+ });
+ Future future2 = executor.submit(() -> {
+ final int numRecords = 100;
+ assertDoesNotThrow(() -> {
+ writeStatus2.set(createCommitWithInserts(cfg, client2, "001", commitTime3, numRecords, false));
+ });
+ });
+
+ future1.get();
+ future2.get();
+
+ final CountDownLatch commitCountDownLatch = new CountDownLatch(2);
+ HoodieTableMetaClient tableMetaClient = client.getTableServiceClient().createMetaClient(true);
+
+ // Get inflight instant stream before commit
+ List<HoodieInstant> inflightInstants = client
+ .getTableServiceClient()
+ .getInflightTimelineExcludeCompactionAndClustering(tableMetaClient)
+ .getReverseOrderedInstants()
+ .collect(Collectors.toList());
+
+ // Commit the instants and get instants to rollback in parallel
+ future1 = executor.submit(() -> {
+ long start = System.currentTimeMillis();
+ LOG.info(String.format("Start to commit instant %s", commitTime2));
+ client1.commit(commitTime2, writeStatus1.get());
+ commitCountDownLatch.countDown();
+ validInstants.add(commitTime2);
+ LOG.info(String.format("commit the instant cost %d ms", System.currentTimeMillis() - start));
+ });
+
+ future2 = executor.submit(() -> {
+ long start = System.currentTimeMillis();
+ LOG.info(String.format("Start to commit instant %s", commitTime3));
+ client2.commit(commitTime3, writeStatus2.get());
+ commitCountDownLatch.countDown();
+ validInstants.add(commitTime3);
+ LOG.info(String.format("commit the instant %s cost %d ms", commitTime3, System.currentTimeMillis() - start));
+ });
+
+ Future future3 = executor.submit(() -> {
+ try {
+ commitCountDownLatch.await(30000, TimeUnit.MILLISECONDS);
+ } catch (InterruptedException e) {
+ //
+ }
+ LOG.info("Start to get instants to rollback");
+ List<String> instantsToRollback =
+ client.getTableServiceClient().getInstantsToRollbackForLazyCleanPolicy(tableMetaClient, inflightInstants.stream());
+ // No instants will be rollback though some instants may be detected as `expired`
+ assertTrue(instantsToRollback.isEmpty());
+ });
+
+ future1.get();
+ future2.get();
+ future3.get();
+
+ validInstants.addAll(
+ metaClient.reloadActiveTimeline().getCompletedReplaceTimeline()
+ .filterCompletedInstants().getInstantsAsStream().map(HoodieInstant::getTimestamp).collect(Collectors.toSet()));
+ Set<String> completedInstants = metaClient.reloadActiveTimeline().getCommitsTimeline()
+ .filterCompletedInstants().getInstantsAsStream().map(HoodieInstant::getTimestamp)
+ .collect(Collectors.toSet());
+ assertTrue(validInstants.containsAll(completedInstants));
Review Comment:
removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #7826:
URL: https://github.com/apache/hudi/pull/7826#discussion_r1196010098
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/TestHoodieClientMultiWriter.java:
##########
@@ -545,6 +546,120 @@ public void testMultiWriterWithAsyncTableServicesWithConflict(HoodieTableType ta
assertTrue(validInstants.containsAll(completedInstants));
}
+ @ParameterizedTest
+ @EnumSource(value = HoodieTableType.class, names = {"MERGE_ON_READ", "COPY_ON_WRITE"})
+ public void testMultiWriterWithAsyncLazyCleanRollback(HoodieTableType tableType) throws Exception {
+ // create inserts X 1
+ if (tableType == HoodieTableType.MERGE_ON_READ) {
+ setUpMORTestTable();
+ }
+ // Disabling embedded timeline server, it doesn't work with multiwriter
+ HoodieWriteConfig.Builder writeConfigBuilder = getConfigBuilder()
+ .withCleanConfig(HoodieCleanConfig.newBuilder()
+ .withAutoClean(false)
+ .withAsyncClean(true)
+ .withFailedWritesCleaningPolicy(HoodieFailedWritesCleaningPolicy.LAZY).build())
+ .withCompactionConfig(HoodieCompactionConfig.newBuilder()
+ .withInlineCompaction(false)
+ .withMaxNumDeltaCommitsBeforeCompaction(2).build())
+ .withEmbeddedTimelineServerEnabled(false)
+ // Timeline-server-based markers are not used for multi-writer tests
+ .withMarkersType(MarkerType.DIRECT.name())
+ .withFileSystemViewConfig(FileSystemViewStorageConfig.newBuilder().withStorageType(
+ FileSystemViewStorageType.MEMORY).build())
+ .withWriteConcurrencyMode(WriteConcurrencyMode.OPTIMISTIC_CONCURRENCY_CONTROL)
+ // Set the config so that heartbeat will expire in 1 second without update
+ .withLockConfig(HoodieLockConfig.newBuilder().withLockProvider(InProcessLockProvider.class)
+ .build()).withAutoCommit(false).withProperties(lockProperties);
+ Set<String> validInstants = new HashSet<>();
+ // Create the first commit with inserts
+ HoodieWriteConfig cfg = writeConfigBuilder.build();
+ SparkRDDWriteClient client = getHoodieWriteClient(cfg);
+ createCommitWithInserts(cfg, client, "000", "001", 200, true);
+ validInstants.add("001");
+
+ // Three clients running actions in parallel
+ final int threadCount = 3;
+ final ExecutorService executor = Executors.newFixedThreadPool(threadCount);
+
+ final SparkRDDWriteClient client1 = getHoodieWriteClient(cfg);
+ final SparkRDDWriteClient client2 = getHoodieWriteClient(cfg);
+ final String commitTime2 = "002";
+ final String commitTime3 = "003";
+ AtomicReference<Object> writeStatus1 = new AtomicReference<>(null);
+ AtomicReference<Object> writeStatus2 = new AtomicReference<>(null);
+
+ Future future1 = executor.submit(() -> {
+ final int numRecords = 100;
+ assertDoesNotThrow(() -> {
+ writeStatus1.set(createCommitWithInserts(cfg, client1, "001", commitTime2, numRecords, false));
+ });
+ });
+ Future future2 = executor.submit(() -> {
+ final int numRecords = 100;
+ assertDoesNotThrow(() -> {
+ writeStatus2.set(createCommitWithInserts(cfg, client2, "001", commitTime3, numRecords, false));
+ });
+ });
+
+ future1.get();
+ future2.get();
+
+ final CountDownLatch commitCountDownLatch = new CountDownLatch(2);
+ HoodieTableMetaClient tableMetaClient = client.getTableServiceClient().createMetaClient(true);
+
+ // Get inflight instant stream before commit
+ List<HoodieInstant> inflightInstants = client
+ .getTableServiceClient()
+ .getInflightTimelineExcludeCompactionAndClustering(tableMetaClient)
+ .getReverseOrderedInstants()
+ .collect(Collectors.toList());
+
+ // Commit the instants and get instants to rollback in parallel
+ future1 = executor.submit(() -> {
+ long start = System.currentTimeMillis();
+ LOG.info(String.format("Start to commit instant %s", commitTime2));
+ client1.commit(commitTime2, writeStatus1.get());
+ commitCountDownLatch.countDown();
+ validInstants.add(commitTime2);
+ LOG.info(String.format("commit the instant cost %d ms", System.currentTimeMillis() - start));
Review Comment:
for debug, removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1551827719
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0ce6917cfa857601ae31780553fd685291a0bc47",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17147",
"triggerID" : "0ce6917cfa857601ae31780553fd685291a0bc47",
"triggerType" : "PUSH"
}, {
"hash" : "0ce6917cfa857601ae31780553fd685291a0bc47",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1477699232",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1477699232",
"triggerType" : "MANUAL"
}, {
"hash" : "0ce6917cfa857601ae31780553fd685291a0bc47",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17163",
"triggerID" : "1479155691",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 0ce6917cfa857601ae31780553fd685291a0bc47 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17147) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17163)
* 0000 Unknown: [CANCELED](TBD)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1413685140
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1413693517
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14873",
"triggerID" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14873)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1415276571
> For multi-writer scenarios, we suggest to set up the lazy cleaning strategy, did you set up that for both Flink and Spark conf already?
Yes, we already setup the policy in both engine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #7826:
URL: https://github.com/apache/hudi/pull/7826#discussion_r1108271487
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieTableServiceClient.java:
##########
@@ -707,20 +709,33 @@ protected List<String> getInstantsToRollback(HoodieTableMetaClient metaClient, H
}
}).collect(Collectors.toList());
} else if (cleaningPolicy.isLazy()) {
- return inflightInstantsStream.filter(instant -> {
- try {
- return heartbeatClient.isHeartbeatExpired(instant.getTimestamp());
- } catch (IOException io) {
- throw new HoodieException("Failed to check heartbeat for instant " + instant, io);
- }
- }).map(HoodieInstant::getTimestamp).collect(Collectors.toList());
+ return getInstantsToRollbackForLazyCleanPolicy(metaClient, inflightInstantsStream);
} else if (cleaningPolicy.isNever()) {
return Collections.emptyList();
} else {
throw new IllegalArgumentException("Invalid Failed Writes Cleaning Policy " + config.getFailedWritesCleanPolicy());
}
}
+ @VisibleForTesting
+ public List<String> getInstantsToRollbackForLazyCleanPolicy(HoodieTableMetaClient metaClient,
+ Stream<HoodieInstant> inflightInstantsStream) {
+ // Get expired instants, must store them into list before double-checking
+ List<String> expiredInstants = inflightInstantsStream.filter(instant -> {
+ try {
+ // An instant transformed from inflight to completed have no heartbeat file and will be detected as expired instant here
+ return heartbeatClient.isHeartbeatExpired(instant.getTimestamp());
+ } catch (IOException io) {
+ throw new HoodieException("Failed to check heartbeat for instant " + instant, io);
+ }
+ }).map(HoodieInstant::getTimestamp).collect(Collectors.toList());
+
+ // Double check whether the heartbeat-expired instant is an inflight instant
+ metaClient.reloadActiveTimeline();
Review Comment:
> Yes，we use a separate spark job to run table services and the instant in completed by the flink writer
But from the code, the compaction and clustering instants are excluded, can you elaborate a little more, what kind of table service the Spark offline job takes over?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #7826:
URL: https://github.com/apache/hudi/pull/7826#discussion_r1139797072
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieTableServiceClient.java:
##########
@@ -707,20 +709,37 @@ protected List<String> getInstantsToRollback(HoodieTableMetaClient metaClient, H
}
}).collect(Collectors.toList());
} else if (cleaningPolicy.isLazy()) {
- return inflightInstantsStream.filter(instant -> {
- try {
- return heartbeatClient.isHeartbeatExpired(instant.getTimestamp());
- } catch (IOException io) {
- throw new HoodieException("Failed to check heartbeat for instant " + instant, io);
- }
- }).map(HoodieInstant::getTimestamp).collect(Collectors.toList());
+ return getInstantsToRollbackForLazyCleanPolicy(metaClient, inflightInstantsStream);
} else if (cleaningPolicy.isNever()) {
return Collections.emptyList();
} else {
throw new IllegalArgumentException("Invalid Failed Writes Cleaning Policy " + config.getFailedWritesCleanPolicy());
}
}
+ @VisibleForTesting
+ public List<String> getInstantsToRollbackForLazyCleanPolicy(HoodieTableMetaClient metaClient,
+ Stream<HoodieInstant> inflightInstantsStream) {
+ // Get expired instants, must store them into list before double-checking
+ List<String> expiredInstants = inflightInstantsStream.filter(instant -> {
+ try {
+ // An instant transformed from inflight to completed have no heartbeat file and will be detected as expired instant here
+ return heartbeatClient.isHeartbeatExpired(instant.getTimestamp());
+ } catch (IOException io) {
+ throw new HoodieException("Failed to check heartbeat for instant " + instant, io);
+ }
+ }).map(HoodieInstant::getTimestamp).collect(Collectors.toList());
+
+ if (!expiredInstants.isEmpty()) {
Review Comment:
generally reloading entire timeline is a costly operation. I am trying to see if there is any other elegant way to fix this.
Ideally the heart beat file should always be generated by the regular writer. and hence only condition when the heart beat file goes missing is, the writer completed the commit and deleted it. and if thats a safe assumption, if we encounter heart beat file not found, we should ignore the commit of interest.
Only catch here is. if a user switches from non lazy to lazy clean, those may not have heart beats.
Can we introduce locking for entire getInstantsToRollback. and so that we can guarantee that if a writer is concurrently completing a commit, we can deduce that when compared to a failed commit in previous launch when writer had eager failed policy.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieTableServiceClient.java:
##########
@@ -707,20 +709,37 @@ protected List<String> getInstantsToRollback(HoodieTableMetaClient metaClient, H
}
}).collect(Collectors.toList());
} else if (cleaningPolicy.isLazy()) {
- return inflightInstantsStream.filter(instant -> {
- try {
- return heartbeatClient.isHeartbeatExpired(instant.getTimestamp());
- } catch (IOException io) {
- throw new HoodieException("Failed to check heartbeat for instant " + instant, io);
- }
- }).map(HoodieInstant::getTimestamp).collect(Collectors.toList());
+ return getInstantsToRollbackForLazyCleanPolicy(metaClient, inflightInstantsStream);
} else if (cleaningPolicy.isNever()) {
return Collections.emptyList();
} else {
throw new IllegalArgumentException("Invalid Failed Writes Cleaning Policy " + config.getFailedWritesCleanPolicy());
}
}
+ @VisibleForTesting
+ public List<String> getInstantsToRollbackForLazyCleanPolicy(HoodieTableMetaClient metaClient,
+ Stream<HoodieInstant> inflightInstantsStream) {
+ // Get expired instants, must store them into list before double-checking
+ List<String> expiredInstants = inflightInstantsStream.filter(instant -> {
+ try {
+ // An instant transformed from inflight to completed have no heartbeat file and will be detected as expired instant here
+ return heartbeatClient.isHeartbeatExpired(instant.getTimestamp());
+ } catch (IOException io) {
+ throw new HoodieException("Failed to check heartbeat for instant " + instant, io);
+ }
+ }).map(HoodieInstant::getTimestamp).collect(Collectors.toList());
+
+ if (!expiredInstants.isEmpty()) {
Review Comment:
or we can also try checking for existence of completed commit file in ".hoodie" after we get hold of list of commits to rollback. Those which seem to have a completed instant, we can ignore them from the list.
let me know what do you think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1537811250
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14873",
"triggerID" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"triggerType" : "PUSH"
}, {
"hash" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14875",
"triggerID" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"triggerType" : "PUSH"
}, {
"hash" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15074",
"triggerID" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"triggerType" : "PUSH"
}, {
"hash" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15805",
"triggerID" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"triggerType" : "PUSH"
}, {
"hash" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15828",
"triggerID" : "1477699232",
"triggerType" : "MANUAL"
}, {
"hash" : "ed708d668d1521a10b4baa62d5c79aaefdd326f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15835",
"triggerID" : "ed708d668d1521a10b4baa62d5c79aaefdd326f1",
"triggerType" : "PUSH"
}, {
"hash" : "b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15842",
"triggerID" : "b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566",
"triggerType" : "PUSH"
}, {
"hash" : "b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15851",
"triggerID" : "1479155691",
"triggerType" : "MANUAL"
}, {
"hash" : "e76f56f4223f346ab0a7cb17ae67beb75d89fa67",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16814",
"triggerID" : "e76f56f4223f346ab0a7cb17ae67beb75d89fa67",
"triggerType" : "PUSH"
}, {
"hash" : "ac053c8d3f59609327fca976670c8720ca9455e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ac053c8d3f59609327fca976670c8720ca9455e1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e76f56f4223f346ab0a7cb17ae67beb75d89fa67 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16814)
* ac053c8d3f59609327fca976670c8720ca9455e1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1550934661
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14873",
"triggerID" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"triggerType" : "PUSH"
}, {
"hash" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14875",
"triggerID" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"triggerType" : "PUSH"
}, {
"hash" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15074",
"triggerID" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"triggerType" : "PUSH"
}, {
"hash" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15805",
"triggerID" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"triggerType" : "PUSH"
}, {
"hash" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15828",
"triggerID" : "1477699232",
"triggerType" : "MANUAL"
}, {
"hash" : "ed708d668d1521a10b4baa62d5c79aaefdd326f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15835",
"triggerID" : "ed708d668d1521a10b4baa62d5c79aaefdd326f1",
"triggerType" : "PUSH"
}, {
"hash" : "b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15842",
"triggerID" : "b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566",
"triggerType" : "PUSH"
}, {
"hash" : "b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15851",
"triggerID" : "1479155691",
"triggerType" : "MANUAL"
}, {
"hash" : "e76f56f4223f346ab0a7cb17ae67beb75d89fa67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16814",
"triggerID" : "e76f56f4223f346ab0a7cb17ae67beb75d89fa67",
"triggerType" : "PUSH"
}, {
"hash" : "ac053c8d3f59609327fca976670c8720ca9455e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16933",
"triggerID" : "ac053c8d3f59609327fca976670c8720ca9455e1",
"triggerType" : "PUSH"
}, {
"hash" : "d7a06415cfecd30371d2282b2ebe270acfa98ac4",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17142",
"triggerID" : "d7a06415cfecd30371d2282b2ebe270acfa98ac4",
"triggerType" : "PUSH"
}, {
"hash" : "0ce6917cfa857601ae31780553fd685291a0bc47",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17147",
"triggerID" : "0ce6917cfa857601ae31780553fd685291a0bc47",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d7a06415cfecd30371d2282b2ebe270acfa98ac4 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17142)
* 0ce6917cfa857601ae31780553fd685291a0bc47 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17147)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1552595634
<img width="699" alt="Screenshot 2023-05-18 at 3 05 25 PM" src="https://github.com/apache/hudi/assets/2701446/6ed2748b-0d75-4d35-af37-b59e682d2067">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1476484890
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14873",
"triggerID" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"triggerType" : "PUSH"
}, {
"hash" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14875",
"triggerID" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"triggerType" : "PUSH"
}, {
"hash" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15074",
"triggerID" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"triggerType" : "PUSH"
}, {
"hash" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15805",
"triggerID" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6832b7787f8363cb09e672d8675d8bc68bad3a2c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15805)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1498420260
@nsivabalan Gently ping, could you help to review this pr again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1415193513
I'm wondering how this senario happens:
> If the instant is completed after step 1,
Is here another writer there?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1425090312
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14873",
"triggerID" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"triggerType" : "PUSH"
}, {
"hash" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14875",
"triggerID" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"triggerType" : "PUSH"
}, {
"hash" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0346edf996c9c8a3eb44aa6dd4bcf258578d3f72 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14875)
* db7cd7aa1f23a635c3a4b0b71ea136231a76f520 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "leesf (via GitHub)" <gi...@apache.org>.
leesf commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1448109215
@nsivabalan do you have any comments? or we can merge this MR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "leesf (via GitHub)" <gi...@apache.org>.
leesf commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1461098677
@nsivabalan gently ping
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #7826:
URL: https://github.com/apache/hudi/pull/7826#discussion_r1181278240
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/functional/TestHoodieSparkMergeOnReadTableRollback.java:
##########
@@ -912,6 +914,62 @@ public void testLazyRollbackOfFailedCommit(boolean rollbackUsingMarkers) throws
assertEquals(numLogFilesAfterRollback, numLogFilesAfterCompaction);
}
+ @Test
+ public void testGetInstantToRollbackForLazyCleanPolicy() throws Exception {
+ Properties properties = new Properties();
+ properties.setProperty(HoodieTableConfig.BASE_FILE_FORMAT.key(), HoodieTableConfig.BASE_FILE_FORMAT.defaultValue().toString());
+ HoodieTableMetaClient metaClient = getHoodieMetaClient(HoodieTableType.MERGE_ON_READ, properties);
+
+ HoodieWriteConfig cfg = getWriteConfig(true, false);
+ HoodieWriteConfig autoCommitFalseCfg = getWriteConfig(false, false);
+ autoCommitFalseCfg.setValue(CLIENT_HEARTBEAT_INTERVAL_IN_MS.key(), "2000");
+ autoCommitFalseCfg.setValue(CLIENT_HEARTBEAT_NUM_TOLERABLE_MISSES.key(), "2");
+ HoodieTestDataGenerator dataGen = new HoodieTestDataGenerator();
+
+ SparkRDDWriteClient client = getHoodieWriteClient(cfg);
+ // Commit 1
+ insertRecords(client, dataGen, "001");
+
+ // Trigger an inflight commit 2
+ SparkRDDWriteClient autoCommitFalseClient = getHoodieWriteClient(autoCommitFalseCfg);
+ String commitTime2 = "002";
+ autoCommitFalseClient.startCommitWithTime(commitTime2);
+ List<HoodieRecord> records = dataGen.generateInserts(commitTime2, 20);
+ JavaRDD<HoodieRecord> writeRecords = jsc().parallelize(records, 1);
+ JavaRDD<WriteStatus> statuses = autoCommitFalseClient.upsert(writeRecords, commitTime2);
+
+ // Stop updating heartbeat
+ autoCommitFalseClient.getHeartbeatClient().getHeartbeat(commitTime2).getTimer().cancel();
+ // Sleep to make the heartbeat expired. In production env, heartbeat is expired because of commit
+ Thread.sleep(4000);
Review Comment:
can we reduce this to 2 secs. bcoz, the heart beat expiry is set to 2 secs right. just trying to reduce the total test run time.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieTableServiceClient.java:
##########
@@ -707,20 +709,34 @@ protected List<String> getInstantsToRollback(HoodieTableMetaClient metaClient, H
}
}).collect(Collectors.toList());
} else if (cleaningPolicy.isLazy()) {
- return inflightInstantsStream.filter(instant -> {
- try {
- return heartbeatClient.isHeartbeatExpired(instant.getTimestamp());
- } catch (IOException io) {
- throw new HoodieException("Failed to check heartbeat for instant " + instant, io);
- }
- }).map(HoodieInstant::getTimestamp).collect(Collectors.toList());
+ return getInstantsToRollbackForLazyCleanPolicy(metaClient, inflightInstantsStream);
} else if (cleaningPolicy.isNever()) {
return Collections.emptyList();
} else {
throw new IllegalArgumentException("Invalid Failed Writes Cleaning Policy " + config.getFailedWritesCleanPolicy());
}
}
+ @VisibleForTesting
Review Comment:
is it not possible to test it at the write client layer. would like to avoid using "VisibleForTesting" as much as possible.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieTableServiceClient.java:
##########
@@ -707,20 +709,34 @@ protected List<String> getInstantsToRollback(HoodieTableMetaClient metaClient, H
}
}).collect(Collectors.toList());
} else if (cleaningPolicy.isLazy()) {
- return inflightInstantsStream.filter(instant -> {
- try {
- return heartbeatClient.isHeartbeatExpired(instant.getTimestamp());
- } catch (IOException io) {
- throw new HoodieException("Failed to check heartbeat for instant " + instant, io);
- }
- }).map(HoodieInstant::getTimestamp).collect(Collectors.toList());
+ return getInstantsToRollbackForLazyCleanPolicy(metaClient, inflightInstantsStream);
} else if (cleaningPolicy.isNever()) {
return Collections.emptyList();
} else {
throw new IllegalArgumentException("Invalid Failed Writes Cleaning Policy " + config.getFailedWritesCleanPolicy());
}
}
+ @VisibleForTesting
+ public List<String> getInstantsToRollbackForLazyCleanPolicy(HoodieTableMetaClient metaClient,
+ Stream<HoodieInstant> inflightInstantsStream) {
+ // Get expired instants, must store them into list before double-checking
+ List<HoodieInstant> expiredInstants = inflightInstantsStream.filter(instant -> {
+ try {
+ // An instant transformed from inflight to completed have no heartbeat file and will be detected as expired instant here
+ return heartbeatClient.isHeartbeatExpired(instant.getTimestamp());
+ } catch (IOException io) {
+ throw new HoodieException("Failed to check heartbeat for instant " + instant, io);
+ }
+ }).collect(Collectors.toList());
+
+ // Only return instants that haven't been completed by other writers
+ return expiredInstants.stream()
+ .filter(instant -> !metaClient.getActiveTimeline().isCompletedCommitFileExists(instant))
Review Comment:
may be, better option is to reload entire timeline and check if the commit is completed.
we don't want to make such adhoc calls. every call should go via proper timeline calls as much as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1552336252
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0ce6917cfa857601ae31780553fd685291a0bc47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17147",
"triggerID" : "0ce6917cfa857601ae31780553fd685291a0bc47",
"triggerType" : "PUSH"
}, {
"hash" : "0ce6917cfa857601ae31780553fd685291a0bc47",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17163",
"triggerID" : "1479155691",
"triggerType" : "MANUAL"
}, {
"hash" : "0ce6917cfa857601ae31780553fd685291a0bc47",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17163",
"triggerID" : "1477699232",
"triggerType" : "MANUAL"
}, {
"hash" : "0ce6917cfa857601ae31780553fd685291a0bc47",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1552322372",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1477699232",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #7826:
URL: https://github.com/apache/hudi/pull/7826#discussion_r1196042139
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieTableServiceClient.java:
##########
@@ -759,20 +759,37 @@ protected List<String> getInstantsToRollback(HoodieTableMetaClient metaClient, H
}
}).collect(Collectors.toList());
} else if (cleaningPolicy.isLazy()) {
- return inflightInstantsStream.filter(instant -> {
- try {
- return heartbeatClient.isHeartbeatExpired(instant.getTimestamp());
- } catch (IOException io) {
- throw new HoodieException("Failed to check heartbeat for instant " + instant, io);
- }
- }).map(HoodieInstant::getTimestamp).collect(Collectors.toList());
+ return getInstantsToRollbackForLazyCleanPolicy(metaClient, inflightInstantsStream);
} else if (cleaningPolicy.isNever()) {
return Collections.emptyList();
} else {
throw new IllegalArgumentException("Invalid Failed Writes Cleaning Policy " + config.getFailedWritesCleanPolicy());
}
}
+ protected List<String> getInstantsToRollbackForLazyCleanPolicy(HoodieTableMetaClient metaClient,
Review Comment:
Found that we can use `getInstantsToRollback` directly to test the fix if we create the metaClient and load timeline before we commit the inflight instant.
All comments are addressed, thanks for the review~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1479839037
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14873",
"triggerID" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"triggerType" : "PUSH"
}, {
"hash" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14875",
"triggerID" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"triggerType" : "PUSH"
}, {
"hash" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15074",
"triggerID" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"triggerType" : "PUSH"
}, {
"hash" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15805",
"triggerID" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"triggerType" : "PUSH"
}, {
"hash" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15828",
"triggerID" : "1477699232",
"triggerType" : "MANUAL"
}, {
"hash" : "ed708d668d1521a10b4baa62d5c79aaefdd326f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15835",
"triggerID" : "ed708d668d1521a10b4baa62d5c79aaefdd326f1",
"triggerType" : "PUSH"
}, {
"hash" : "b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15842",
"triggerID" : "b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566",
"triggerType" : "PUSH"
}, {
"hash" : "b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15851",
"triggerID" : "1479155691",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15842) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15851)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1478563821
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14873",
"triggerID" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"triggerType" : "PUSH"
}, {
"hash" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14875",
"triggerID" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"triggerType" : "PUSH"
}, {
"hash" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15074",
"triggerID" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"triggerType" : "PUSH"
}, {
"hash" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15805",
"triggerID" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"triggerType" : "PUSH"
}, {
"hash" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15828",
"triggerID" : "1477699232",
"triggerType" : "MANUAL"
}, {
"hash" : "ed708d668d1521a10b4baa62d5c79aaefdd326f1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15835",
"triggerID" : "ed708d668d1521a10b4baa62d5c79aaefdd326f1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ed708d668d1521a10b4baa62d5c79aaefdd326f1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15835)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1477699232
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1538052478
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14873",
"triggerID" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"triggerType" : "PUSH"
}, {
"hash" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14875",
"triggerID" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"triggerType" : "PUSH"
}, {
"hash" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15074",
"triggerID" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"triggerType" : "PUSH"
}, {
"hash" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15805",
"triggerID" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"triggerType" : "PUSH"
}, {
"hash" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15828",
"triggerID" : "1477699232",
"triggerType" : "MANUAL"
}, {
"hash" : "ed708d668d1521a10b4baa62d5c79aaefdd326f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15835",
"triggerID" : "ed708d668d1521a10b4baa62d5c79aaefdd326f1",
"triggerType" : "PUSH"
}, {
"hash" : "b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15842",
"triggerID" : "b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566",
"triggerType" : "PUSH"
}, {
"hash" : "b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15851",
"triggerID" : "1479155691",
"triggerType" : "MANUAL"
}, {
"hash" : "e76f56f4223f346ab0a7cb17ae67beb75d89fa67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16814",
"triggerID" : "e76f56f4223f346ab0a7cb17ae67beb75d89fa67",
"triggerType" : "PUSH"
}, {
"hash" : "ac053c8d3f59609327fca976670c8720ca9455e1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16933",
"triggerID" : "ac053c8d3f59609327fca976670c8720ca9455e1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ac053c8d3f59609327fca976670c8720ca9455e1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16933)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1552372300
waiting for https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=17147&view=results
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan merged pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan merged PR #7826:
URL: https://github.com/apache/hudi/pull/7826
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #7826:
URL: https://github.com/apache/hudi/pull/7826#discussion_r1184519230
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieTableServiceClient.java:
##########
@@ -707,20 +709,34 @@ protected List<String> getInstantsToRollback(HoodieTableMetaClient metaClient, H
}
}).collect(Collectors.toList());
} else if (cleaningPolicy.isLazy()) {
- return inflightInstantsStream.filter(instant -> {
- try {
- return heartbeatClient.isHeartbeatExpired(instant.getTimestamp());
- } catch (IOException io) {
- throw new HoodieException("Failed to check heartbeat for instant " + instant, io);
- }
- }).map(HoodieInstant::getTimestamp).collect(Collectors.toList());
+ return getInstantsToRollbackForLazyCleanPolicy(metaClient, inflightInstantsStream);
} else if (cleaningPolicy.isNever()) {
return Collections.emptyList();
} else {
throw new IllegalArgumentException("Invalid Failed Writes Cleaning Policy " + config.getFailedWritesCleanPolicy());
}
}
+ @VisibleForTesting
Review Comment:
I move the test to `TestHoodieClientMultiWriter` and now still need to change the access modifier to protected. It's a little bit hard to test through calling `BaseHoodieTableServiceClient.rollbackFailedWrites` directly because the execution sequence is hard to control.
So now the test will still get inflight instants before commit and then call `getInstantsToRollbackForLazyCleanPolicy` to check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #7826:
URL: https://github.com/apache/hudi/pull/7826#discussion_r1187063352
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/TestHoodieClientMultiWriter.java:
##########
@@ -545,6 +546,116 @@ public void testMultiWriterWithAsyncTableServicesWithConflict(HoodieTableType ta
assertTrue(validInstants.containsAll(completedInstants));
}
+ @ParameterizedTest
+ @EnumSource(value = HoodieTableType.class, names = {"MERGE_ON_READ"})
+ public void testMultiWriterWithAsyncLazyCleanRollback(HoodieTableType tableType) throws Exception {
+ // create inserts X 1
+ if (tableType == HoodieTableType.MERGE_ON_READ) {
+ setUpMORTestTable();
+ }
+ // Disabling embedded timeline server, it doesn't work with multiwriter
+ HoodieWriteConfig.Builder writeConfigBuilder = getConfigBuilder()
+ .withCleanConfig(HoodieCleanConfig.newBuilder()
+ .withAutoClean(false)
+ .withAsyncClean(true)
+ .withFailedWritesCleaningPolicy(HoodieFailedWritesCleaningPolicy.LAZY).build())
+ .withCompactionConfig(HoodieCompactionConfig.newBuilder()
+ .withInlineCompaction(false)
+ .withMaxNumDeltaCommitsBeforeCompaction(2).build())
+ .withEmbeddedTimelineServerEnabled(false)
+ // Timeline-server-based markers are not used for multi-writer tests
+ .withMarkersType(MarkerType.DIRECT.name())
+ .withFileSystemViewConfig(FileSystemViewStorageConfig.newBuilder().withStorageType(
+ FileSystemViewStorageType.MEMORY).build())
+ .withWriteConcurrencyMode(WriteConcurrencyMode.OPTIMISTIC_CONCURRENCY_CONTROL)
+ // Set the config so that heartbeat will expire in 1 second without update
+ .withLockConfig(HoodieLockConfig.newBuilder().withLockProvider(InProcessLockProvider.class)
+ .build()).withAutoCommit(false).withProperties(lockProperties);
+ Set<String> validInstants = new HashSet<>();
+ // Create the first commit with inserts
+ HoodieWriteConfig cfg = writeConfigBuilder.build();
+ SparkRDDWriteClient client = getHoodieWriteClient(cfg);
+ createCommitWithInserts(cfg, client, "000", "001", 200, true);
+ validInstants.add("001");
+
+ // Three clients running actions in parallel
+ final int threadCount = 3;
+ final ExecutorService executor = Executors.newFixedThreadPool(threadCount);
+
+ final SparkRDDWriteClient client1 = getHoodieWriteClient(cfg);
+ final SparkRDDWriteClient client2 = getHoodieWriteClient(cfg);
+ final String commitTime2 = "002";
+ final String commitTime3 = "003";
+ AtomicReference<Object> writeStatus1 = new AtomicReference<>(null);
+ AtomicReference<Object> writeStatus2 = new AtomicReference<>(null);
+
+ // Create upserts, schedule cleaning, schedule compaction in parallel
+ Future future1 = executor.submit(() -> {
+ final int numRecords = 100;
+ assertDoesNotThrow(() -> {
+ writeStatus1.set(createCommitWithInserts(cfg, client1, "001", commitTime2, numRecords, false));
+ });
+ });
+ // Create upserts, schedule cleaning, schedule compaction in parallel
+ Future future2 = executor.submit(() -> {
+ final int numRecords = 100;
+ assertDoesNotThrow(() -> {
+ writeStatus2.set(createCommitWithInserts(cfg, client2, "001", commitTime3, numRecords, false));
+ });
+ });
+
+ future1.get();
+ future2.get();
+
+ HoodieTableMetaClient tableMetaClient = client.getTableServiceClient().createMetaClient(true);
+
+ // Get inflight instant stream before commit
+ List<HoodieInstant> inflightInstants = client
+ .getTableServiceClient()
+ .getInflightTimelineExcludeCompactionAndClustering(tableMetaClient)
+ .getReverseOrderedInstants()
+ .collect(Collectors.toList());
+
+ // Commit the instants and get instants to rollback in parallel
+ future1 = executor.submit(() -> {
+ long start = System.currentTimeMillis();
+ LOG.info(String.format("Start to commit instant %s", commitTime2));
+ client1.commit(commitTime2, writeStatus1.get());
+ validInstants.add(commitTime2);
+ LOG.info(String.format("commit the instant cost %d ms", System.currentTimeMillis() - start));
+ });
+
+ future2 = executor.submit(() -> {
+ long start = System.currentTimeMillis();
+ LOG.info(String.format("Start to commit instant %s", commitTime3));
+ client2.commit(commitTime3, writeStatus2.get());
+ validInstants.add(commitTime3);
+ LOG.info(String.format("commit the instant %s cost %d ms", commitTime3, System.currentTimeMillis() - start));
+ });
+
+ Future future3 = executor.submit(() -> {
+ // Sleep to let the writer complete the instants
+ assertDoesNotThrow(() -> Thread.sleep(6000));
Review Comment:
Changed, thanks for your suggestion! In fact, we can do these operations sequentially(without introducing multi-thread). But I think a multi-thread test can show the case that we want to fix in a better way so we can keep using multi-thread.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] LinMingQiang commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "LinMingQiang (via GitHub)" <gi...@apache.org>.
LinMingQiang commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1538026347
This situation is relatively rare, we dont need to check every time, and this should be a special case for determining whether instants has expired.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1534073143
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14873",
"triggerID" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"triggerType" : "PUSH"
}, {
"hash" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14875",
"triggerID" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"triggerType" : "PUSH"
}, {
"hash" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15074",
"triggerID" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"triggerType" : "PUSH"
}, {
"hash" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15805",
"triggerID" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"triggerType" : "PUSH"
}, {
"hash" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15828",
"triggerID" : "1477699232",
"triggerType" : "MANUAL"
}, {
"hash" : "ed708d668d1521a10b4baa62d5c79aaefdd326f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15835",
"triggerID" : "ed708d668d1521a10b4baa62d5c79aaefdd326f1",
"triggerType" : "PUSH"
}, {
"hash" : "b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15842",
"triggerID" : "b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566",
"triggerType" : "PUSH"
}, {
"hash" : "b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15851",
"triggerID" : "1479155691",
"triggerType" : "MANUAL"
}, {
"hash" : "e76f56f4223f346ab0a7cb17ae67beb75d89fa67",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16814",
"triggerID" : "e76f56f4223f346ab0a7cb17ae67beb75d89fa67",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15842) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15851)
* e76f56f4223f346ab0a7cb17ae67beb75d89fa67 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16814)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1551817593
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0ce6917cfa857601ae31780553fd685291a0bc47",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0ce6917cfa857601ae31780553fd685291a0bc47",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0ce6917cfa857601ae31780553fd685291a0bc47 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #7826:
URL: https://github.com/apache/hudi/pull/7826#discussion_r1196009711
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/TestHoodieClientMultiWriter.java:
##########
@@ -545,6 +546,120 @@ public void testMultiWriterWithAsyncTableServicesWithConflict(HoodieTableType ta
assertTrue(validInstants.containsAll(completedInstants));
}
+ @ParameterizedTest
+ @EnumSource(value = HoodieTableType.class, names = {"MERGE_ON_READ", "COPY_ON_WRITE"})
+ public void testMultiWriterWithAsyncLazyCleanRollback(HoodieTableType tableType) throws Exception {
+ // create inserts X 1
+ if (tableType == HoodieTableType.MERGE_ON_READ) {
+ setUpMORTestTable();
+ }
+ // Disabling embedded timeline server, it doesn't work with multiwriter
+ HoodieWriteConfig.Builder writeConfigBuilder = getConfigBuilder()
+ .withCleanConfig(HoodieCleanConfig.newBuilder()
+ .withAutoClean(false)
+ .withAsyncClean(true)
+ .withFailedWritesCleaningPolicy(HoodieFailedWritesCleaningPolicy.LAZY).build())
+ .withCompactionConfig(HoodieCompactionConfig.newBuilder()
+ .withInlineCompaction(false)
+ .withMaxNumDeltaCommitsBeforeCompaction(2).build())
+ .withEmbeddedTimelineServerEnabled(false)
+ // Timeline-server-based markers are not used for multi-writer tests
+ .withMarkersType(MarkerType.DIRECT.name())
+ .withFileSystemViewConfig(FileSystemViewStorageConfig.newBuilder().withStorageType(
+ FileSystemViewStorageType.MEMORY).build())
+ .withWriteConcurrencyMode(WriteConcurrencyMode.OPTIMISTIC_CONCURRENCY_CONTROL)
+ // Set the config so that heartbeat will expire in 1 second without update
+ .withLockConfig(HoodieLockConfig.newBuilder().withLockProvider(InProcessLockProvider.class)
+ .build()).withAutoCommit(false).withProperties(lockProperties);
+ Set<String> validInstants = new HashSet<>();
+ // Create the first commit with inserts
+ HoodieWriteConfig cfg = writeConfigBuilder.build();
+ SparkRDDWriteClient client = getHoodieWriteClient(cfg);
+ createCommitWithInserts(cfg, client, "000", "001", 200, true);
+ validInstants.add("001");
+
+ // Three clients running actions in parallel
+ final int threadCount = 3;
+ final ExecutorService executor = Executors.newFixedThreadPool(threadCount);
+
+ final SparkRDDWriteClient client1 = getHoodieWriteClient(cfg);
+ final SparkRDDWriteClient client2 = getHoodieWriteClient(cfg);
+ final String commitTime2 = "002";
+ final String commitTime3 = "003";
+ AtomicReference<Object> writeStatus1 = new AtomicReference<>(null);
+ AtomicReference<Object> writeStatus2 = new AtomicReference<>(null);
+
+ Future future1 = executor.submit(() -> {
+ final int numRecords = 100;
+ assertDoesNotThrow(() -> {
+ writeStatus1.set(createCommitWithInserts(cfg, client1, "001", commitTime2, numRecords, false));
+ });
+ });
+ Future future2 = executor.submit(() -> {
+ final int numRecords = 100;
+ assertDoesNotThrow(() -> {
+ writeStatus2.set(createCommitWithInserts(cfg, client2, "001", commitTime3, numRecords, false));
+ });
+ });
+
+ future1.get();
+ future2.get();
+
+ final CountDownLatch commitCountDownLatch = new CountDownLatch(2);
+ HoodieTableMetaClient tableMetaClient = client.getTableServiceClient().createMetaClient(true);
+
+ // Get inflight instant stream before commit
+ List<HoodieInstant> inflightInstants = client
+ .getTableServiceClient()
+ .getInflightTimelineExcludeCompactionAndClustering(tableMetaClient)
+ .getReverseOrderedInstants()
+ .collect(Collectors.toList());
+
+ // Commit the instants and get instants to rollback in parallel
+ future1 = executor.submit(() -> {
+ long start = System.currentTimeMillis();
+ LOG.info(String.format("Start to commit instant %s", commitTime2));
+ client1.commit(commitTime2, writeStatus1.get());
+ commitCountDownLatch.countDown();
+ validInstants.add(commitTime2);
+ LOG.info(String.format("commit the instant cost %d ms", System.currentTimeMillis() - start));
+ });
+
+ future2 = executor.submit(() -> {
+ long start = System.currentTimeMillis();
+ LOG.info(String.format("Start to commit instant %s", commitTime3));
+ client2.commit(commitTime3, writeStatus2.get());
+ commitCountDownLatch.countDown();
+ validInstants.add(commitTime3);
+ LOG.info(String.format("commit the instant %s cost %d ms", commitTime3, System.currentTimeMillis() - start));
+ });
+
+ Future future3 = executor.submit(() -> {
+ try {
+ commitCountDownLatch.await(30000, TimeUnit.MILLISECONDS);
+ } catch (InterruptedException e) {
+ //
+ }
+ LOG.info("Start to get instants to rollback");
+ List<String> instantsToRollback =
+ client.getTableServiceClient().getInstantsToRollbackForLazyCleanPolicy(tableMetaClient, inflightInstants.stream());
+ // No instants will be rollback though some instants may be detected as `expired`
+ assertTrue(instantsToRollback.isEmpty());
Review Comment:
Nice suggestions! will modify it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1425115994
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14873",
"triggerID" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"triggerType" : "PUSH"
}, {
"hash" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14875",
"triggerID" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"triggerType" : "PUSH"
}, {
"hash" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15074",
"triggerID" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0346edf996c9c8a3eb44aa6dd4bcf258578d3f72 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14875)
* db7cd7aa1f23a635c3a4b0b71ea136231a76f520 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15074)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1479158680
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14873",
"triggerID" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"triggerType" : "PUSH"
}, {
"hash" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14875",
"triggerID" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"triggerType" : "PUSH"
}, {
"hash" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15074",
"triggerID" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"triggerType" : "PUSH"
}, {
"hash" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15805",
"triggerID" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"triggerType" : "PUSH"
}, {
"hash" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15828",
"triggerID" : "1477699232",
"triggerType" : "MANUAL"
}, {
"hash" : "ed708d668d1521a10b4baa62d5c79aaefdd326f1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15835",
"triggerID" : "ed708d668d1521a10b4baa62d5c79aaefdd326f1",
"triggerType" : "PUSH"
}, {
"hash" : "b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15842",
"triggerID" : "b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566",
"triggerType" : "PUSH"
}, {
"hash" : "b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15851",
"triggerID" : "1479155691",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* b74d73f66e53a4cbd6b6048c4d07e19c1b9ad566 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15842) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15851)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7826:
URL: https://github.com/apache/hudi/pull/7826#issuecomment-1478268519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14873",
"triggerID" : "cc13523fe6fe8ea3a76a2b5feee39ea16abc04bf",
"triggerType" : "PUSH"
}, {
"hash" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14875",
"triggerID" : "0346edf996c9c8a3eb44aa6dd4bcf258578d3f72",
"triggerType" : "PUSH"
}, {
"hash" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15074",
"triggerID" : "db7cd7aa1f23a635c3a4b0b71ea136231a76f520",
"triggerType" : "PUSH"
}, {
"hash" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15805",
"triggerID" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"triggerType" : "PUSH"
}, {
"hash" : "6832b7787f8363cb09e672d8675d8bc68bad3a2c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15828",
"triggerID" : "1477699232",
"triggerType" : "MANUAL"
}, {
"hash" : "ed708d668d1521a10b4baa62d5c79aaefdd326f1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ed708d668d1521a10b4baa62d5c79aaefdd326f1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6832b7787f8363cb09e672d8675d8bc68bad3a2c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15805) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15828)
* ed708d668d1521a10b4baa62d5c79aaefdd326f1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #7826: [HUDI-5675] fix lazy clean schedule rollback on completed instant
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #7826:
URL: https://github.com/apache/hudi/pull/7826#discussion_r1141679698
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieTableServiceClient.java:
##########
@@ -707,20 +709,37 @@ protected List<String> getInstantsToRollback(HoodieTableMetaClient metaClient, H
}
}).collect(Collectors.toList());
} else if (cleaningPolicy.isLazy()) {
- return inflightInstantsStream.filter(instant -> {
- try {
- return heartbeatClient.isHeartbeatExpired(instant.getTimestamp());
- } catch (IOException io) {
- throw new HoodieException("Failed to check heartbeat for instant " + instant, io);
- }
- }).map(HoodieInstant::getTimestamp).collect(Collectors.toList());
+ return getInstantsToRollbackForLazyCleanPolicy(metaClient, inflightInstantsStream);
} else if (cleaningPolicy.isNever()) {
return Collections.emptyList();
} else {
throw new IllegalArgumentException("Invalid Failed Writes Cleaning Policy " + config.getFailedWritesCleanPolicy());
}
}
+ @VisibleForTesting
+ public List<String> getInstantsToRollbackForLazyCleanPolicy(HoodieTableMetaClient metaClient,
+ Stream<HoodieInstant> inflightInstantsStream) {
+ // Get expired instants, must store them into list before double-checking
+ List<String> expiredInstants = inflightInstantsStream.filter(instant -> {
+ try {
+ // An instant transformed from inflight to completed have no heartbeat file and will be detected as expired instant here
+ return heartbeatClient.isHeartbeatExpired(instant.getTimestamp());
+ } catch (IOException io) {
+ throw new HoodieException("Failed to check heartbeat for instant " + instant, io);
+ }
+ }).map(HoodieInstant::getTimestamp).collect(Collectors.toList());
+
+ if (!expiredInstants.isEmpty()) {
Review Comment:
I think checking for existence of completed commit file in ".hoodie" is much better than reloading the whole timeline, which is very lightweight while introducing a lock for entire getInstantsToRollback may be kind of complicated. I will push a new commit that only check the existence of commit file instead of reloading the timeline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org