You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "skywalker0618 (via GitHub)" <gi...@apache.org> on 2026/07/17 23:11:59 UTC
[I] Hudi-hive-sync tool leak connections to hive metastore [hudi]
skywalker0618 opened a new issue, #19321:
URL: https://github.com/apache/hudi/issues/19321
### Bug Description
**What happened:**
`HoodieHiveSyncClient` (hiveql sync mode) leaks one Hive Metastore connection on every sync cycle whenever the underlying metastore client is rebuilt by `RetryingMetaStoreClient`. For long-running jobs that sync on every commit (Flink streaming, Spark structured streaming, DeltaStreamer continuous mode), the metastore's open-connection count grows monotonically until HMS hits its connection limit.
### Root cause
`HoodieHiveSyncClient` holds its metastore handle in a single field:
private IMetaStoreClient client; // a RetryingMetaStoreClient proxy
`RetryingMetaStoreClient` is a dynamic proxy: when a call throws a transient `TException`/`TTransportException` (e.g. `"SASL authentication not complete"`, or any dropped/reset connection), it **transparently tears down the current underlying `IMetaStoreClient`, builds a fresh one, and retries the call.** The retry succeeds, so the sync itself looks healthy in the logs.
The problem is in teardown. `close()` only calls `Hive.closeCurrent()`:
public void close() {
...
if (client != null) {
Hive.closeCurrent(); // closes the MSC bound to the thread-local Hive singleton
client = null;
}
}
`Hive.closeCurrent()` closes the metastore client referenced by the thread-local `Hive` singleton. But once `RetryingMetaStoreClient` has rebuilt the connection, the **freshly-built** underlying MSC is reachable only through the proxy chain on this `client` field — the `Hive` singleton still points at the **older** MSC instance. So `Hive.closeCurrent()`:
- closes the *older* MSC (already dead/replaced), and
- **orphans the retry-created MSC**, whose live socket to HMS is never closed.
Result: one leaked TCP connection to the metastore per cycle in which a retry occurred.
### Why it's easy to miss
- The sync succeeds (retry masks the transient failure), so there's no error surfaced.
- The leak is only visible as a slowly climbing HMS `current connections` / active-thrift-connection count over hours/days, eventually manifesting as HMS connection exhaustion or `Too many open files` on the client.
**What you expected:**
`close()` should reliably release the metastore connection that is actually live on `this.client`, regardless of whether `RetryingMetaStoreClient` has swapped the underlying instance out from under the `Hive` singleton.
Fix: close the proxied `IMetaStoreClient` **directly** before falling back to `Hive.closeCurrent()`. `client.close()` walks the proxy and closes the currently-bound real MSC by identity, so the retry-created connection is always released; `Hive.closeCurrent()` stays as a belt-and-suspenders cleanup for the singleton path. Wrap the direct close in its own try/catch so a transient close failure still lets the fallback run.
if (client != null) {
try {
client.close(); // closes the currently-bound MSC (incl. retry-rebuilt one)
} catch (Exception e) {
LOG.warn("Failed to close IMetaStoreClient proxy directly; "
+ "Hive.closeCurrent() will run anyway", e);
}
Hive.closeCurrent(); // fallback for the thread-local Hive singleton
client = null;
}
Expected outcome: HMS open-connection count stays bounded across sync cycles (returns to baseline between cycles) instead of growing by one per retry cycle.
**Steps to reproduce:**
1.
2.
3.
### Environment
- Hudi: 0.14.x
- Sync mode: `hoodie.datasource.hive_sync.mode=hiveql` (uses `IMetaStoreClient` + `HiveQueryDDLExecutor`)
- Any deployment where the metastore client periodically retries (transient SASL/connection resets), which is common under Kerberos/SASL.
- Highest impact: long-running jobs that sync every commit (Flink streaming, DeltaStreamer continuous mode).
### Logs and Stack Trace
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Hudi-hive-sync tool leak connections to hive metastore [hudi]
Posted by "skywalker0618 (via GitHub)" <gi...@apache.org>.
skywalker0618 commented on issue #19321:
URL: https://github.com/apache/hudi/issues/19321#issuecomment-5008844360
@danny0405 @HuangZhenQiu How do you feel about this proposal? If it sounds good I can go ahead to create PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Hudi-hive-sync tool leak connections to hive metastore [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #19321:
URL: https://github.com/apache/hudi/issues/19321#issuecomment-5018240456
sure, you can fire a PR is the bug still exists on master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Hudi-hive-sync tool leak connections to hive metastore [hudi]
Posted by "skywalker0618 (via GitHub)" <gi...@apache.org>.
skywalker0618 commented on issue #19321:
URL: https://github.com/apache/hudi/issues/19321#issuecomment-5028503238
Hi @danny0405 do you mind taking a look at https://github.com/apache/hudi/pull/19331 which fixes this issue? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Hudi-hive-sync tool leak connections to hive metastore [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 closed issue #19321: Hudi-hive-sync tool leak connections to hive metastore
URL: https://github.com/apache/hudi/issues/19321
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org