You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "vamsikarnika (via GitHub)" <gi...@apache.org> on 2026/09/08 09:04:10 UTC
[PR] fix(heartbeat): don't fail a durable commit when heartbeat cleanup races a refresh [hudi]
vamsikarnika opened a new pull request, #19867:
URL: https://github.com/apache/hudi/pull/19867
### Describe the issue this Pull Request addresses
A commit that has already been made durable can be reported back to the caller as a **failed** write,
purely because of heartbeat file cleanup.
`stop()` cancels the heartbeat scheduler with `shutdownNow()` and immediately deletes the heartbeat
file, without waiting for a refresh that is already in flight. The refresh recreates the file, and on
storage that enforces preconditions it also changes the object generation, so the generation-matched
delete is rejected. Observed in production on GCS as `412 conditionNotMet`, retried four times by the
connector and then given up on:
```
Precondition not met while deleting '.../.hoodie/.heartbeat/<instant>' at generation <gen>.
Attempt 1. Retrying: {"code":412, ... "reason":"conditionNotMet"}
```
`HoodieHadoopStorage.delete` reports a failed delete by throwing `HoodieIOException` when the object
still exists afterwards, which is exactly the outcome here. That exception is unchecked, so it escapes
the `IOException`-only catch in `WriterHeartbeatUtils.deleteHeartbeatFile`, propagates out of
`BaseHoodieWriteClient.postCommit` and aborts `commitStats` — after `commit()` has already completed.
The write is on storage, but the caller is told the round failed.
### Summary and Changelog
Users no longer see a successful commit reported as a failure because of heartbeat cleanup.
- `HoodieHeartbeatClient.stopHeartbeatScheduler`: `shutdown()` plus `awaitTermination()` before the
caller deletes the heartbeat file, so a refresh in flight completes first and the delete then reads
a settled object. `shutdownNow()` interrupts rather than waits, which is what left the window open.
The await is deliberately **not** placed in `shutdownHeartbeatScheduler`, because the missed-refresh
path in `updateHeartbeat` calls that from the scheduler thread and must not await itself. The bound
is `heartbeatWriteTimeoutMs + heartbeatIntervalInMs`, since an in-flight tick can be parked on the
bounded write.
- `WriterHeartbeatUtils.deleteHeartbeatFile`: also catch `HoodieIOException`. By the time `postCommit`
runs the cleanup the commit is durable, and no caller consumes the return value
(`HoodieHeartbeatClient.stop`, `BaseHoodieTableServiceClient` x2 all discard it), so a failed cleanup
of a transient lease marker must never be fatal. The existing `log.error` is kept, so a leaked file
is still visible.
The two changes are complementary rather than redundant: a write that exceeds `heartbeatWriteTimeoutMs`
is deliberately abandoned by `writeHeartbeatFile`, and an abandoned write can still land after the
delete. The await closes the ordering race for refreshes that complete normally; the widened catch is
what keeps the abandoned-write tail non-fatal.
Test: `TestHoodieHeartbeatClient#testStopWaitsForInFlightHeartbeatRefresh` gates a scheduled refresh
mid-write, asserts `stop()` parks rather than deleting, and asserts no refresh is recorded after the
delete. It fails on current master (the stopper never parks) and passes with this change.
### Impact
No public API change. `deleteHeartbeatFile` keeps its signature and its documented "returns whether
the file was deleted" contract; this makes the implementation honour that contract instead of sometimes
throwing. `stop()` may now block briefly behind an in-flight heartbeat refresh, bounded as described
above.
### Risk Level
low
Confined to the heartbeat classes. No new locks and nothing held across storage IO. The widened catch is
on a path whose result every caller discards. Verified with JDK 11: module compiles, 0 checkstyle
violations, and all 9 tests in `TestHoodieHeartbeatClient` pass, including the two existing
scheduler-liveness tests (`testSlowHeartbeatWriteDoesNotBlockScheduler`,
`testScheduledHeartbeatRetriesAfterWriteFailure`). The new test was confirmed to fail against
unmodified master.
### Documentation Update
none — no new config, no default changed, no user-facing behaviour beyond the removal of a spurious
failure.
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(heartbeat): don't fail a durable commit when heartbeat cleanup races a refresh [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan merged PR #19867:
URL: https://github.com/apache/hudi/pull/19867
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(heartbeat): don't fail a durable commit when heartbeat cleanup races a refresh [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #19867:
URL: https://github.com/apache/hudi/pull/19867#discussion_r4066828110
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/heartbeat/HoodieHeartbeatClient.java:
##########
@@ -196,15 +196,34 @@ private boolean isHeartbeatStarted(Heartbeat heartbeat) {
private void stopHeartbeatScheduler(Heartbeat heartbeat) {
log.info("Stopping heartbeat for instant {}", heartbeat.getInstantTime());
shutdownHeartbeatScheduler(heartbeat);
+ // Callers delete the heartbeat file next. A refresh landing after that delete recreates the
+ // file, and on storage that enforces preconditions it also changes the object generation, so a
+ // generation-matched delete is rejected (e.g. GCS 412 conditionNotMet).
+ awaitHeartbeatSchedulerTermination(heartbeat);
heartbeat.setHeartbeatStopped(true);
log.info("Stopped heartbeat for instant {}", heartbeat.getInstantTime());
}
+ /** Stops further refreshes without waiting; safe to call from the scheduler thread itself. */
private void shutdownHeartbeatScheduler(Heartbeat heartbeat) {
if (heartbeat.getScheduledFuture() != null) {
heartbeat.getScheduledFuture().cancel(false);
}
- heartbeat.getHeartbeatScheduler().shutdownNow();
+ heartbeat.getHeartbeatScheduler().shutdown();
+ }
+
+ private void awaitHeartbeatSchedulerTermination(Heartbeat heartbeat) {
+ // An in-flight tick can be parked on the bounded write, so allow for that plus one interval.
+ long timeoutMs = this.heartbeatWriteTimeoutMs + this.heartbeatIntervalInMs;
+ try {
+ if (!heartbeat.getHeartbeatScheduler().awaitTermination(timeoutMs, TimeUnit.MILLISECONDS)) {
+ log.warn("Timed out after {} ms awaiting an in-flight heartbeat refresh for instant {}",
Review Comment:
nvm. deletion of file is strictly not required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(heartbeat): don't fail a durable commit when heartbeat cleanup races a refresh [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19867:
URL: https://github.com/apache/hudi/pull/19867#issuecomment-5586538545
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7d287404bf06498470055db19bf52f39c5d8f36",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16898",
"triggerID" : "f7d287404bf06498470055db19bf52f39c5d8f36",
"triggerType" : "PUSH"
}, {
"hash" : "c524f772dfe05bf63efdabdb792cdcfb17a7bec6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16913",
"triggerID" : "c524f772dfe05bf63efdabdb792cdcfb17a7bec6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f7d287404bf06498470055db19bf52f39c5d8f36 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16898)
* c524f772dfe05bf63efdabdb792cdcfb17a7bec6 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16913)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(heartbeat): don't fail a durable commit when heartbeat cleanup races a refresh [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19867:
URL: https://github.com/apache/hudi/pull/19867#discussion_r3958654759
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/heartbeat/WriterHeartbeatUtils.java:
##########
@@ -58,8 +59,10 @@ public static boolean deleteHeartbeatFile(HoodieStorage storage,
} else {
log.info("Deleted the heartbeat for instant {}", instantTime);
}
- } catch (IOException io) {
- log.error("Unable to delete heartbeat for instant {}", instantTime, io);
+ } catch (IOException | HoodieIOException e) {
Review Comment:
🤖 nit: since `HoodieIOException` is unchecked, catching it alongside the checked `IOException` here reads a bit oddly — might be worth a short comment (or a second catch block) explaining it's deliberately covering both the wrapped IOException path and the unchecked storage rejection.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/heartbeat/HoodieHeartbeatClient.java:
##########
@@ -196,15 +196,34 @@ private boolean isHeartbeatStarted(Heartbeat heartbeat) {
private void stopHeartbeatScheduler(Heartbeat heartbeat) {
log.info("Stopping heartbeat for instant {}", heartbeat.getInstantTime());
shutdownHeartbeatScheduler(heartbeat);
+ // Callers delete the heartbeat file next. A refresh landing after that delete recreates the
+ // file, and on storage that enforces preconditions it also changes the object generation, so a
+ // generation-matched delete is rejected (e.g. GCS 412 conditionNotMet).
+ awaitHeartbeatSchedulerTermination(heartbeat);
heartbeat.setHeartbeatStopped(true);
log.info("Stopped heartbeat for instant {}", heartbeat.getInstantTime());
}
+ /** Stops further refreshes without waiting; safe to call from the scheduler thread itself. */
private void shutdownHeartbeatScheduler(Heartbeat heartbeat) {
if (heartbeat.getScheduledFuture() != null) {
heartbeat.getScheduledFuture().cancel(false);
}
- heartbeat.getHeartbeatScheduler().shutdownNow();
+ heartbeat.getHeartbeatScheduler().shutdown();
Review Comment:
🤖 Since the tick is now allowed to run to completion, have you considered that `stop()` has already done `instantToHeartbeatMap.remove(instantTime)` before it awaits? The in-flight `updateHeartbeat` then hits `instantToHeartbeatMap.get(instantTime)` → null → NPE at `heartbeat.getLastHeartbeatTime()`, which `HeartbeatTask` logs at ERROR as "will retry on next tick" on the success path of a durable commit. A `heartbeat == null` early-return in `updateHeartbeat` would keep that quiet.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/heartbeat/HoodieHeartbeatClient.java:
##########
@@ -196,15 +196,34 @@ private boolean isHeartbeatStarted(Heartbeat heartbeat) {
private void stopHeartbeatScheduler(Heartbeat heartbeat) {
log.info("Stopping heartbeat for instant {}", heartbeat.getInstantTime());
shutdownHeartbeatScheduler(heartbeat);
+ // Callers delete the heartbeat file next. A refresh landing after that delete recreates the
+ // file, and on storage that enforces preconditions it also changes the object generation, so a
+ // generation-matched delete is rejected (e.g. GCS 412 conditionNotMet).
+ awaitHeartbeatSchedulerTermination(heartbeat);
heartbeat.setHeartbeatStopped(true);
log.info("Stopped heartbeat for instant {}", heartbeat.getInstantTime());
}
+ /** Stops further refreshes without waiting; safe to call from the scheduler thread itself. */
private void shutdownHeartbeatScheduler(Heartbeat heartbeat) {
if (heartbeat.getScheduledFuture() != null) {
heartbeat.getScheduledFuture().cancel(false);
}
- heartbeat.getHeartbeatScheduler().shutdownNow();
+ heartbeat.getHeartbeatScheduler().shutdown();
+ }
+
+ private void awaitHeartbeatSchedulerTermination(Heartbeat heartbeat) {
+ // An in-flight tick can be parked on the bounded write, so allow for that plus one interval.
+ long timeoutMs = this.heartbeatWriteTimeoutMs + this.heartbeatIntervalInMs;
+ try {
+ if (!heartbeat.getHeartbeatScheduler().awaitTermination(timeoutMs, TimeUnit.MILLISECONDS)) {
Review Comment:
🤖 Confirmed from the PR-head source: `close()` and `getHeartbeatWriteExecutor()` both lock `this`, and `close()` now sits in `awaitTermination` while holding it. Worth noting the window is narrow — a tick only contends for the monitor on the `getHeartbeatWriteExecutor()` call, so once it's parked in `future.get` there's no contention — and the production postCommit path goes through `stop()`, which isn't synchronized, so the fix works as intended there. Narrowing `close()`'s `synchronized` to just the executor teardown (after `stopHeartbeatTimers()` returns) seems like the smallest change; otherwise a tick that loses the race lazily recreates a cached pool that nothing ever shuts down.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/heartbeat/HoodieHeartbeatClient.java:
##########
@@ -196,15 +196,34 @@ private boolean isHeartbeatStarted(Heartbeat heartbeat) {
private void stopHeartbeatScheduler(Heartbeat heartbeat) {
log.info("Stopping heartbeat for instant {}", heartbeat.getInstantTime());
shutdownHeartbeatScheduler(heartbeat);
+ // Callers delete the heartbeat file next. A refresh landing after that delete recreates the
+ // file, and on storage that enforces preconditions it also changes the object generation, so a
+ // generation-matched delete is rejected (e.g. GCS 412 conditionNotMet).
+ awaitHeartbeatSchedulerTermination(heartbeat);
Review Comment:
🤖 The PR description does call this tail out explicitly ("an abandoned write can still land after the delete") and leans on the widened catch plus the log for it. One thing to flag on the suggested alternative: after `future.cancel(true)`, `future.get()` returns immediately with `CancellationException` rather than waiting for the task body, so awaiting that future wouldn't actually close the window — you'd need a separate completion signal from the write task. Given nothing sweeps `.heartbeat/` for completed instants (only `stop()` and `rollbackFailedWrites` delete), the orphan is permanent but inert, since `getInstantsToRollback` only looks at inflight instants.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(heartbeat): don't fail a durable commit when heartbeat cleanup races a refresh [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19867:
URL: https://github.com/apache/hudi/pull/19867#issuecomment-5588577957
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7d287404bf06498470055db19bf52f39c5d8f36",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16898",
"triggerID" : "f7d287404bf06498470055db19bf52f39c5d8f36",
"triggerType" : "PUSH"
}, {
"hash" : "c524f772dfe05bf63efdabdb792cdcfb17a7bec6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16913",
"triggerID" : "c524f772dfe05bf63efdabdb792cdcfb17a7bec6",
"triggerType" : "PUSH"
}, {
"hash" : "db229bccb452483d8558f98d0dfeb4d5b53dc8bb",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16915",
"triggerID" : "db229bccb452483d8558f98d0dfeb4d5b53dc8bb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* db229bccb452483d8558f98d0dfeb4d5b53dc8bb Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16915)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(heartbeat): don't fail a durable commit when heartbeat cleanup races a refresh [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19867:
URL: https://github.com/apache/hudi/pull/19867#issuecomment-5588117861
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7d287404bf06498470055db19bf52f39c5d8f36",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16898",
"triggerID" : "f7d287404bf06498470055db19bf52f39c5d8f36",
"triggerType" : "PUSH"
}, {
"hash" : "c524f772dfe05bf63efdabdb792cdcfb17a7bec6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16913",
"triggerID" : "c524f772dfe05bf63efdabdb792cdcfb17a7bec6",
"triggerType" : "PUSH"
}, {
"hash" : "db229bccb452483d8558f98d0dfeb4d5b53dc8bb",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16915",
"triggerID" : "db229bccb452483d8558f98d0dfeb4d5b53dc8bb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c524f772dfe05bf63efdabdb792cdcfb17a7bec6 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16913)
* db229bccb452483d8558f98d0dfeb4d5b53dc8bb Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16915)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(heartbeat): don't fail a durable commit when heartbeat cleanup races a refresh [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19867:
URL: https://github.com/apache/hudi/pull/19867#issuecomment-5582492054
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7d287404bf06498470055db19bf52f39c5d8f36",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16898",
"triggerID" : "f7d287404bf06498470055db19bf52f39c5d8f36",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f7d287404bf06498470055db19bf52f39c5d8f36 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16898)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(heartbeat): don't fail a durable commit when heartbeat cleanup races a refresh [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19867:
URL: https://github.com/apache/hudi/pull/19867#issuecomment-5582394899
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7d287404bf06498470055db19bf52f39c5d8f36",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f7d287404bf06498470055db19bf52f39c5d8f36",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f7d287404bf06498470055db19bf52f39c5d8f36 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(heartbeat): don't fail a durable commit when heartbeat cleanup races a refresh [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19867:
URL: https://github.com/apache/hudi/pull/19867#issuecomment-5584442192
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7d287404bf06498470055db19bf52f39c5d8f36",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16898",
"triggerID" : "f7d287404bf06498470055db19bf52f39c5d8f36",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f7d287404bf06498470055db19bf52f39c5d8f36 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16898)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(heartbeat): don't fail a durable commit when heartbeat cleanup races a refresh [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #19867:
URL: https://github.com/apache/hudi/pull/19867#discussion_r4050377299
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/heartbeat/HoodieHeartbeatClient.java:
##########
@@ -196,15 +196,34 @@ private boolean isHeartbeatStarted(Heartbeat heartbeat) {
private void stopHeartbeatScheduler(Heartbeat heartbeat) {
log.info("Stopping heartbeat for instant {}", heartbeat.getInstantTime());
shutdownHeartbeatScheduler(heartbeat);
+ // Callers delete the heartbeat file next. A refresh landing after that delete recreates the
+ // file, and on storage that enforces preconditions it also changes the object generation, so a
+ // generation-matched delete is rejected (e.g. GCS 412 conditionNotMet).
+ awaitHeartbeatSchedulerTermination(heartbeat);
heartbeat.setHeartbeatStopped(true);
log.info("Stopped heartbeat for instant {}", heartbeat.getInstantTime());
}
+ /** Stops further refreshes without waiting; safe to call from the scheduler thread itself. */
private void shutdownHeartbeatScheduler(Heartbeat heartbeat) {
if (heartbeat.getScheduledFuture() != null) {
heartbeat.getScheduledFuture().cancel(false);
}
- heartbeat.getHeartbeatScheduler().shutdownNow();
+ heartbeat.getHeartbeatScheduler().shutdown();
+ }
+
+ private void awaitHeartbeatSchedulerTermination(Heartbeat heartbeat) {
+ // An in-flight tick can be parked on the bounded write, so allow for that plus one interval.
+ long timeoutMs = this.heartbeatWriteTimeoutMs + this.heartbeatIntervalInMs;
+ try {
+ if (!heartbeat.getHeartbeatScheduler().awaitTermination(timeoutMs, TimeUnit.MILLISECONDS)) {
+ log.warn("Timed out after {} ms awaiting an in-flight heartbeat refresh for instant {}",
Review Comment:
shouldn't we throw here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(heartbeat): don't fail a durable commit when heartbeat cleanup races a refresh [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19867:
URL: https://github.com/apache/hudi/pull/19867#issuecomment-5586718745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7d287404bf06498470055db19bf52f39c5d8f36",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16898",
"triggerID" : "f7d287404bf06498470055db19bf52f39c5d8f36",
"triggerType" : "PUSH"
}, {
"hash" : "c524f772dfe05bf63efdabdb792cdcfb17a7bec6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16913",
"triggerID" : "c524f772dfe05bf63efdabdb792cdcfb17a7bec6",
"triggerType" : "PUSH"
}, {
"hash" : "db229bccb452483d8558f98d0dfeb4d5b53dc8bb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "db229bccb452483d8558f98d0dfeb4d5b53dc8bb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f7d287404bf06498470055db19bf52f39c5d8f36 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16898)
* c524f772dfe05bf63efdabdb792cdcfb17a7bec6 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16913)
* db229bccb452483d8558f98d0dfeb4d5b53dc8bb UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(heartbeat): don't fail a durable commit when heartbeat cleanup races a refresh [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19867:
URL: https://github.com/apache/hudi/pull/19867#discussion_r3956292898
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/heartbeat/HoodieHeartbeatClient.java:
##########
@@ -196,15 +196,34 @@ private boolean isHeartbeatStarted(Heartbeat heartbeat) {
private void stopHeartbeatScheduler(Heartbeat heartbeat) {
log.info("Stopping heartbeat for instant {}", heartbeat.getInstantTime());
shutdownHeartbeatScheduler(heartbeat);
+ // Callers delete the heartbeat file next. A refresh landing after that delete recreates the
+ // file, and on storage that enforces preconditions it also changes the object generation, so a
+ // generation-matched delete is rejected (e.g. GCS 412 conditionNotMet).
+ awaitHeartbeatSchedulerTermination(heartbeat);
heartbeat.setHeartbeatStopped(true);
log.info("Stopped heartbeat for instant {}", heartbeat.getInstantTime());
}
+ /** Stops further refreshes without waiting; safe to call from the scheduler thread itself. */
private void shutdownHeartbeatScheduler(Heartbeat heartbeat) {
if (heartbeat.getScheduledFuture() != null) {
heartbeat.getScheduledFuture().cancel(false);
}
- heartbeat.getHeartbeatScheduler().shutdownNow();
+ heartbeat.getHeartbeatScheduler().shutdown();
+ }
+
+ private void awaitHeartbeatSchedulerTermination(Heartbeat heartbeat) {
+ // An in-flight tick can be parked on the bounded write, so allow for that plus one interval.
+ long timeoutMs = this.heartbeatWriteTimeoutMs + this.heartbeatIntervalInMs;
+ try {
+ if (!heartbeat.getHeartbeatScheduler().awaitTermination(timeoutMs, TimeUnit.MILLISECONDS)) {
Review Comment:
🤖 `close()` is `synchronized` and so is `getHeartbeatWriteExecutor()`, both on `this`. If a tick has just entered `updateHeartbeat` when `close()` starts, it blocks on the monitor while `close()` sits here in `awaitTermination` holding it — so `close()` stalls for the full timeout (2×interval, ~120s at defaults), and after it nulls `heartbeatWriteExecutor` the tick lazily creates a fresh pool. Could the await be done outside the monitor, or the executor guarded by a separate lock / created eagerly?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/heartbeat/HoodieHeartbeatClient.java:
##########
@@ -196,15 +196,34 @@ private boolean isHeartbeatStarted(Heartbeat heartbeat) {
private void stopHeartbeatScheduler(Heartbeat heartbeat) {
log.info("Stopping heartbeat for instant {}", heartbeat.getInstantTime());
shutdownHeartbeatScheduler(heartbeat);
+ // Callers delete the heartbeat file next. A refresh landing after that delete recreates the
+ // file, and on storage that enforces preconditions it also changes the object generation, so a
+ // generation-matched delete is rejected (e.g. GCS 412 conditionNotMet).
+ awaitHeartbeatSchedulerTermination(heartbeat);
Review Comment:
🤖 When the bounded write itself times out, `future.cancel(true)` interrupts the write thread but a storage upload already on the wire may still land; the scheduler thread returns, this await succeeds, and the delete runs before that late create. Is that residual window intended to be covered only by the new `HoodieIOException` catch (leaving an orphan heartbeat file), or would it be worth also awaiting the outstanding write future here?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/client/heartbeat/TestHoodieHeartbeatClient.java:
##########
@@ -165,6 +166,91 @@ public void testScheduledHeartbeatRetriesAfterWriteFailure() {
}
}
+ /**
+ * stop() must not delete the heartbeat file while a scheduled refresh is still writing it. A
+ * refresh landing after the delete recreates the file, and on storage that enforces preconditions
+ * it changes the object generation so a generation-matched delete is rejected (e.g. GCS 412).
+ */
+ @Test
+ public void testStopWaitsForInFlightHeartbeatRefresh() throws Exception {
+ CountDownLatch refreshEntered = new CountDownLatch(1);
+ CountDownLatch releaseRefresh = new CountDownLatch(1);
+ OrderRecordingStorage storage = new OrderRecordingStorage(
+ (FileSystem) metaClient.getStorage().getFileSystem(), refreshEntered, releaseRefresh);
+ HoodieHeartbeatClient client = new HoodieHeartbeatClient(
+ storage, metaClient.getBasePath().toString(), heartBeatInterval, numTolerableMisses);
Review Comment:
🤖 With `heartBeatInterval = 1000L` the write timeout is also 1s. If thread start + Awaitility's default 100ms poll delay/interval push the gap between `refreshEntered` and `releaseRefresh.countDown()` past ~1s on a loaded CI box, `future.get` times out, `awaitTermination` returns immediately, the delete runs before the gated create lands, and the last-event assertion fails spuriously. Would a larger interval for this test (e.g. 5000L like `testNumHeartbeatsGenerated`) be safer?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(heartbeat): don't fail a durable commit when heartbeat cleanup races a refresh [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19867:
URL: https://github.com/apache/hudi/pull/19867#issuecomment-5583624657
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19867?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `38.46154%` with `8 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 78.36%. Comparing base ([`7567809`](https://app.codecov.io/gh/apache/hudi/commit/75678093b3eb6af46b8dcfa1c1e41ccc384a9afd?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`f7d2874`](https://app.codecov.io/gh/apache/hudi/commit/f7d287404bf06498470055db19bf52f39c5d8f36?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19867?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...e/hudi/client/heartbeat/HoodieHeartbeatClient.java](https://app.codecov.io/gh/apache/hudi/pull/19867?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fclient%2Fheartbeat%2FHoodieHeartbeatClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9oZWFydGJlYXQvSG9vZGllSGVhcnRiZWF0Q2xpZW50LmphdmE=) | 45.45% | [5 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19867?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...he/hudi/client/heartbeat/WriterHeartbeatUtils.java](https://app.codecov.io/gh/apache/hudi/pull/19867?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fclient%2Fheartbeat%2FWriterHeartbeatUtils.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9oZWFydGJlYXQvV3JpdGVySGVhcnRiZWF0VXRpbHMuamF2YQ==) | 0.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19867?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19867 +/- ##
============================================
- Coverage 78.36% 78.36% -0.01%
- Complexity 33966 33967 +1
============================================
Files 2543 2543
Lines 142085 142095 +10
Branches 17253 17254 +1
============================================
+ Hits 111342 111346 +4
- Misses 23033 23038 +5
- Partials 7710 7711 +1
```
| [Components](https://app.codecov.io/gh/apache/hudi/pull/19867/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hudi-common](https://app.codecov.io/gh/apache/hudi/pull/19867/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.75% <ø> (ø)` | |
| [hudi-client](https://app.codecov.io/gh/apache/hudi/pull/19867/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.20% <38.46%> (-0.02%)` | :arrow_down: |
| [hudi-flink](https://app.codecov.io/gh/apache/hudi/pull/19867/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `85.57% <ø> (ø)` | |
| [hudi-spark-datasource](https://app.codecov.io/gh/apache/hudi/pull/19867/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `73.26% <ø> (ø)` | |
| [hudi-utilities](https://app.codecov.io/gh/apache/hudi/pull/19867/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `74.54% <ø> (ø)` | |
| [hudi-cli](https://app.codecov.io/gh/apache/hudi/pull/19867/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `15.13% <ø> (ø)` | |
| [hudi-hadoop](https://app.codecov.io/gh/apache/hudi/pull/19867/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.80% <ø> (ø)` | |
| [hudi-sync](https://app.codecov.io/gh/apache/hudi/pull/19867/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `75.68% <ø> (ø)` | |
| [hudi-io](https://app.codecov.io/gh/apache/hudi/pull/19867/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.19% <ø> (ø)` | |
| [hudi-timeline-service](https://app.codecov.io/gh/apache/hudi/pull/19867/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.44% <ø> (ø)` | |
| [hudi-cloud](https://app.codecov.io/gh/apache/hudi/pull/19867/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `65.81% <ø> (ø)` | |
| [hudi-kafka-connect](https://app.codecov.io/gh/apache/hudi/pull/19867/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.20% <ø> (ø)` | |
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/19867/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19867/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.10% <38.46%> (-0.02%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19867?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...he/hudi/client/heartbeat/WriterHeartbeatUtils.java](https://app.codecov.io/gh/apache/hudi/pull/19867?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fclient%2Fheartbeat%2FWriterHeartbeatUtils.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9oZWFydGJlYXQvV3JpdGVySGVhcnRiZWF0VXRpbHMuamF2YQ==) | `60.86% <0.00%> (ø)` | |
| [...e/hudi/client/heartbeat/HoodieHeartbeatClient.java](https://app.codecov.io/gh/apache/hudi/pull/19867?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fclient%2Fheartbeat%2FHoodieHeartbeatClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9oZWFydGJlYXQvSG9vZGllSGVhcnRiZWF0Q2xpZW50LmphdmE=) | `82.92% <45.45%> (-3.80%)` | :arrow_down: |
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(heartbeat): don't fail a durable commit when heartbeat cleanup races a refresh [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19867:
URL: https://github.com/apache/hudi/pull/19867#issuecomment-5586446790
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7d287404bf06498470055db19bf52f39c5d8f36",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16898",
"triggerID" : "f7d287404bf06498470055db19bf52f39c5d8f36",
"triggerType" : "PUSH"
}, {
"hash" : "c524f772dfe05bf63efdabdb792cdcfb17a7bec6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c524f772dfe05bf63efdabdb792cdcfb17a7bec6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f7d287404bf06498470055db19bf52f39c5d8f36 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16898)
* c524f772dfe05bf63efdabdb792cdcfb17a7bec6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(heartbeat): don't fail a durable commit when heartbeat cleanup races a refresh [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19867:
URL: https://github.com/apache/hudi/pull/19867#issuecomment-5586808456
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7d287404bf06498470055db19bf52f39c5d8f36",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16898",
"triggerID" : "f7d287404bf06498470055db19bf52f39c5d8f36",
"triggerType" : "PUSH"
}, {
"hash" : "c524f772dfe05bf63efdabdb792cdcfb17a7bec6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16913",
"triggerID" : "c524f772dfe05bf63efdabdb792cdcfb17a7bec6",
"triggerType" : "PUSH"
}, {
"hash" : "db229bccb452483d8558f98d0dfeb4d5b53dc8bb",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16915",
"triggerID" : "db229bccb452483d8558f98d0dfeb4d5b53dc8bb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f7d287404bf06498470055db19bf52f39c5d8f36 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16898)
* c524f772dfe05bf63efdabdb792cdcfb17a7bec6 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16913)
* db229bccb452483d8558f98d0dfeb4d5b53dc8bb Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16915)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org