You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "wombatu-kun (via GitHub)" <gi...@apache.org> on 2026/06/16 03:54:07 UTC
[PR] [MINOR] Wait for ZK connection in lock provider to de-flake direct-marker detection test [hudi]
wombatu-kun opened a new pull request, #19014:
URL: https://github.com/apache/hudi/pull/19014
### Describe the issue this Pull Request addresses
`TestSimpleTransactionDirectMarkerBasedDetectionStrategyWithZKLockProvider` is flaky on Azure CI: one parameterization has hung for ~7 minutes (422s) and failed the job with `CuratorConnectionLossException` thrown from `BaseZookeeperBasedLockProvider.<init>` during `createPathIfNotExists`. The root cause is amplification of a single transient ZooKeeper `ConnectionLoss` by two nested retry layers - the Curator `BoundedExponentialBackoffRetry` inside the lock provider constructor and `LockManager`'s `RetryHelper` around provider creation - because the test inherits the production-default retry counts (client retries 50, Curator retries 15) instead of bounding them like the sibling `TestZookeeperBasedLockProvider`.
### Summary and Changelog
- `BaseZookeeperBasedLockProvider`: after `curatorFrameworkClient.start()` and before the first `exists`/`create`, block until the client is connected (bounded by the existing `ZK_CONNECTION_TIMEOUT_MS`). The first ZooKeeper operation no longer races asynchronous connection setup, so a healthy server connects cleanly and a genuinely unreachable one fails deterministically after one connection timeout instead of throwing `ConnectionLoss` into the retry storm.
- `TestSimpleTransactionDirectMarkerBasedDetectionStrategyWithZKLockProvider`: bound the lock-acquire retries and ZooKeeper timeouts, mirroring the non-flaky `TestZookeeperBasedLockProvider`, so a transient failure fails fast (seconds) instead of being amplified into a multi-minute hang. The `@AfterEach` now closes the embedded `TestingServer` in a `finally` so it is not leaked if resource cleanup or directory deletion throws.
### Impact
ZooKeeper-based lock acquisition waits for the client connection before its first operation. Early conflict detection and any `ZookeeperBasedLockProvider` user now fail fast and deterministically against an unreachable ZooKeeper instead of retrying a `ConnectionLoss` for minutes. No public API or configuration change.
### Risk Level
low
The change reuses the already-configured `ZK_CONNECTION_TIMEOUT_MS` and adds no new config. Verified locally: the touched modules compile with 0 checkstyle violations, and `TestZookeeperBasedLockProvider` (8/8) passes - it constructs `BaseZookeeperBasedLockProvider` against a real embedded ZooKeeper and exercises the new connect-wait path.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Wait for ZK connection in lock provider to de-flake direct-marker detection test [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19014:
URL: https://github.com/apache/hudi/pull/19014#issuecomment-4714786234
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5351879610e7b13fa0f258726cde629a6dad7559",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14726",
"triggerID" : "5351879610e7b13fa0f258726cde629a6dad7559",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5351879610e7b13fa0f258726cde629a6dad7559 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14726)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Wait for ZK connection in lock provider to de-flake direct-marker detection test [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19014:
URL: https://github.com/apache/hudi/pull/19014#issuecomment-4715656904
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5351879610e7b13fa0f258726cde629a6dad7559",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14726",
"triggerID" : "5351879610e7b13fa0f258726cde629a6dad7559",
"triggerType" : "PUSH"
}, {
"hash" : "1225b1e4678d4e525370a114aad7f1d6ac05c0ad",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14728",
"triggerID" : "1225b1e4678d4e525370a114aad7f1d6ac05c0ad",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1225b1e4678d4e525370a114aad7f1d6ac05c0ad Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14728)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Wait for ZK connection in lock provider to de-flake direct-marker detection test [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19014:
URL: https://github.com/apache/hudi/pull/19014#issuecomment-4714963565
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5351879610e7b13fa0f258726cde629a6dad7559",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14726",
"triggerID" : "5351879610e7b13fa0f258726cde629a6dad7559",
"triggerType" : "PUSH"
}, {
"hash" : "1225b1e4678d4e525370a114aad7f1d6ac05c0ad",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1225b1e4678d4e525370a114aad7f1d6ac05c0ad",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5351879610e7b13fa0f258726cde629a6dad7559 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14726)
* 1225b1e4678d4e525370a114aad7f1d6ac05c0ad UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Wait for ZK connection in lock provider to de-flake direct-marker detection test [hudi]
Posted by "wombatu-kun (via GitHub)" <gi...@apache.org>.
wombatu-kun commented on code in PR #19014:
URL: https://github.com/apache/hudi/pull/19014#discussion_r3418095200
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/BaseZookeeperBasedLockProvider.java:
##########
@@ -78,6 +78,15 @@ public BaseZookeeperBasedLockProvider(final LockConfiguration lockConfiguration,
.connectionTimeoutMs(ConfigUtils.getIntWithAltKeys(lockConfiguration.getConfig(), ZK_CONNECTION_TIMEOUT_MS))
.build();
this.curatorFrameworkClient.start();
+ try {
+ int connectionTimeoutMs = ConfigUtils.getIntWithAltKeys(lockConfiguration.getConfig(), ZK_CONNECTION_TIMEOUT_MS);
+ if (!this.curatorFrameworkClient.blockUntilConnected(connectionTimeoutMs, TimeUnit.MILLISECONDS)) {
+ throw new HoodieLockException("Failed to connect to ZooKeeper within " + connectionTimeoutMs + " ms");
+ }
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
Review Comment:
Done 1225b1e4678d
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Wait for ZK connection in lock provider to de-flake direct-marker detection test [hudi]
Posted by "wombatu-kun (via GitHub)" <gi...@apache.org>.
wombatu-kun commented on code in PR #19014:
URL: https://github.com/apache/hudi/pull/19014#discussion_r3418096404
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/BaseZookeeperBasedLockProvider.java:
##########
@@ -78,6 +78,15 @@ public BaseZookeeperBasedLockProvider(final LockConfiguration lockConfiguration,
.connectionTimeoutMs(ConfigUtils.getIntWithAltKeys(lockConfiguration.getConfig(), ZK_CONNECTION_TIMEOUT_MS))
.build();
this.curatorFrameworkClient.start();
+ try {
+ int connectionTimeoutMs = ConfigUtils.getIntWithAltKeys(lockConfiguration.getConfig(), ZK_CONNECTION_TIMEOUT_MS);
Review Comment:
Done 1225b1e4678d
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Wait for ZK connection in lock provider to de-flake direct-marker detection test [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19014:
URL: https://github.com/apache/hudi/pull/19014#issuecomment-4715164217
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5351879610e7b13fa0f258726cde629a6dad7559",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14726",
"triggerID" : "5351879610e7b13fa0f258726cde629a6dad7559",
"triggerType" : "PUSH"
}, {
"hash" : "1225b1e4678d4e525370a114aad7f1d6ac05c0ad",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14728",
"triggerID" : "1225b1e4678d4e525370a114aad7f1d6ac05c0ad",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5351879610e7b13fa0f258726cde629a6dad7559 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14726)
* 1225b1e4678d4e525370a114aad7f1d6ac05c0ad Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14728)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Wait for ZK connection in lock provider to de-flake direct-marker detection test [hudi]
Posted by "wombatu-kun (via GitHub)" <gi...@apache.org>.
wombatu-kun merged PR #19014:
URL: https://github.com/apache/hudi/pull/19014
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Wait for ZK connection in lock provider to de-flake direct-marker detection test [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19014:
URL: https://github.com/apache/hudi/pull/19014#issuecomment-4715297998
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5351879610e7b13fa0f258726cde629a6dad7559",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14726",
"triggerID" : "5351879610e7b13fa0f258726cde629a6dad7559",
"triggerType" : "PUSH"
}, {
"hash" : "1225b1e4678d4e525370a114aad7f1d6ac05c0ad",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14728",
"triggerID" : "1225b1e4678d4e525370a114aad7f1d6ac05c0ad",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5351879610e7b13fa0f258726cde629a6dad7559 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14726)
* 1225b1e4678d4e525370a114aad7f1d6ac05c0ad Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14728)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Wait for ZK connection in lock provider to de-flake direct-marker detection test [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19014:
URL: https://github.com/apache/hudi/pull/19014#discussion_r3418016722
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/BaseZookeeperBasedLockProvider.java:
##########
@@ -78,6 +78,15 @@ public BaseZookeeperBasedLockProvider(final LockConfiguration lockConfiguration,
.connectionTimeoutMs(ConfigUtils.getIntWithAltKeys(lockConfiguration.getConfig(), ZK_CONNECTION_TIMEOUT_MS))
.build();
this.curatorFrameworkClient.start();
+ try {
+ int connectionTimeoutMs = ConfigUtils.getIntWithAltKeys(lockConfiguration.getConfig(), ZK_CONNECTION_TIMEOUT_MS);
+ if (!this.curatorFrameworkClient.blockUntilConnected(connectionTimeoutMs, TimeUnit.MILLISECONDS)) {
+ throw new HoodieLockException("Failed to connect to ZooKeeper within " + connectionTimeoutMs + " ms");
+ }
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
Review Comment:
🤖 If `blockUntilConnected` returns false (or we get interrupted) we throw before reaching `createPathIfNotExists`, but the `curatorFrameworkClient` has already been `start()`ed and is never closed — the Curator background threads leak for every failed construction. Could you wrap this block (and ideally `createPathIfNotExists` too) in a try/catch that calls `curatorFrameworkClient.close()` before rethrowing? Otherwise the deterministic fast-failure path this PR enables becomes a thread leak on every unreachable-ZK retry.
<sub><i>- AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/BaseZookeeperBasedLockProvider.java:
##########
@@ -78,6 +78,15 @@ public BaseZookeeperBasedLockProvider(final LockConfiguration lockConfiguration,
.connectionTimeoutMs(ConfigUtils.getIntWithAltKeys(lockConfiguration.getConfig(), ZK_CONNECTION_TIMEOUT_MS))
.build();
this.curatorFrameworkClient.start();
+ try {
+ int connectionTimeoutMs = ConfigUtils.getIntWithAltKeys(lockConfiguration.getConfig(), ZK_CONNECTION_TIMEOUT_MS);
Review Comment:
🤖 nit: `ZK_CONNECTION_TIMEOUT_MS` is already fetched from config just above in the builder — could you hoist it to a local variable before the builder call and reuse it here? Avoids the silent duplication and makes it obvious the two uses are the same value.
<sub><i>- AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Wait for ZK connection in lock provider to de-flake direct-marker detection test [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19014:
URL: https://github.com/apache/hudi/pull/19014#issuecomment-4714767417
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5351879610e7b13fa0f258726cde629a6dad7559",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5351879610e7b13fa0f258726cde629a6dad7559",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5351879610e7b13fa0f258726cde629a6dad7559 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org