You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "warren288 (via GitHub)" <gi...@apache.org> on 2026/09/15 03:42:16 UTC
[I] [Bug] initialize-transaction-coordinator-metadata hangs indefinitely on ZK ConnectionLoss due to non-daemon executor thread leak in ZKMetadataStore constructor [pulsar]
warren288 opened a new issue, #26582:
URL: https://github.com/apache/pulsar/issues/26582
### Search before reporting
- [x] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Read release policy
- [x] I understand that [unsupported versions](https://pulsar.apache.org/contribute/release-policy/#supported-versions) don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.
### User environment
Environment where reproduced:
- Apache Pulsar 3.0.11
- JDK: OpenJDK 21.0.5+11 (Temurin)
- OS: Linux
- zookeeper.client.secure=true with ClientCnxnSocketNetty (TLS-enabled ZK connection)
This bug exists in Apache Pulsar 3.0.11 (and all 3.x/4.x versions, as the ZKMetadataStore constructor and AbstractMetadataStore executor pattern have not changed).
### Issue Description
When PulsarTransactionCoordinatorMetadataSetup#main() fails to connect to ZooKeeper within the session timeout (default 30s), the process hangs indefinitely after the exception is printed instead of exiting with a non-zero status code.
Root cause
The bug is a two-layer resource leak in ZKMetadataStore construction:
Layer 1: Executor not cleaned up on construction failure
ZKMetadataStore constructor (ZKMetadataStore.java:87-117) calls super(metadataStoreConfig) first, which creates a ScheduledThreadPoolExecutor wit a non-daemon DefaultThreadFactory in AbstractMetadataStore.java:99-101:
this.executor = new ScheduledThreadPoolExecutor(1,
new DefaultThreadFactory(
StringUtils.isNotBlank(metadataStoreName) ? metadataStoreName : getClass().getSimpleName()));
Netty's DefaultThreadFactory(String) single-argument constructor sets daemon = false, so the worker thread configuration-metadata-store-1-1 is a non-daemon thread.
When PulsarZooKeeperClient.Builder.build() subsequently fails at waitForConnection() (PulsarZooKeeperClient.java:259), the exception propagates t the catch (Throwable t) block in ZKMetadataStore.java:114-116:
} catch (Throwable t) {
throw new MetadataStoreException(t); // executor not cleaned up
}
The catch block does not call super.close() or executor.shutdowor and its non-daemon worker thread are leaked.
Layer 2: try-with-resources does not call close() on failed ini
PulsarTransactionCoordinatorMetadataSetup.main() (PulsarTransacadataSetup.java:98-101) uses try-with-resources:
try (MetadataStoreExtended configStore = PulsarClusterMetadataSadataStore(...)) {
// ...
}
Per JLS §14.20.3, if the resource initialization expression thrclose() is not called. Since initConfigMetadataStore() → MetadataStoreExtended.create() → new ZKMetadataStore(...) throws, the configStore variable is never assigned, and AbstractMetadataStore.close() (which calls executor.shutdownNow()) is never invoked.
Result: The non-daemon thread configuration-metadata-store-1-1 ED_WAITING state. After main() throws and returns, the JVM creates the DestroyJavaVM thread, but cannot exit because a nonstill alive. The process hangs indefinitely.
Triggering condition
The issue is intermittent. It triggers whenever the ZooKeeper client fails to receive a SyncConnected event within the session timeout (ZooKeeperWatcherBase.waitForConnection(), default 30s). This can happen due to:
- Transient ZK server unavailability (leader election, GC pause reached)
- Network jitter or latency spikes
- TLS handshake timeout (when zookeeper.client.secure=true is set)
- System resource contention during JVM startup (e.g., javaagen
When the ZK connection succeeds within the timeout, the command completes normally and the process exits. When it fails, the process hangs.
### Error messages
```text
org.apache.pulsar.metadata.api.MetadataStoreException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.pulsar.metadata.impl.ZKMetadataStore.<init>(ZKMetadataStore.java:114)
at org.apache.pulsar.metadata.api.extended.MetadataStoreExtended.create(MetadataStoreExtended.java:48)
at org.apache.pulsar.PulsarClusterMetadataSetup.initConfigMetadataStore(PulsarClusterMetadataSetup.java:508)
at org.apache.pulsar.PulsarTransactionCoordinatorMetadataSetup.main(PulsarTransactionCoordinatorMetadataSetup.java:98)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase.waitForConnection(ZooKeeperWatcherBase.java:170)
at org.apache.pulsar.metadata.impl.PulsarZooKeeperClient$Builder.build(PulsarZooKeeperClient.java:259)
at org.apache.pulsar.metadata.impl.ZKMetadataStore.<init>(ZKMetadataStore.java:101)
... 3 more
After the exception is printed, the process never exits. kill -3 thread dumps show:
"configuration-metadata-store-1-1" #43 prio=5 os_prio=0 cpu=1546.81ms elapsed=852.87s
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:269)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(DelayedWorkQueue.java:1188)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1002)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1147)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:636)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
"DestroyJavaVM" #50 prio=5 os_prio=0 cpu=0.26ms elapsed=852.87s
java.lang.Thread.State: RUNNABLE
Three consecutive kill -3 dumps (at elapsed 853s, 935s, 1113s) consistently show configuration-metadata-store-1-1 as the only non-daemon user thread keeping the JVM alive.
```
### Reproducing the issue
# 1. Point to a ZK server that is temporarily unreachable or slow to respond
# (e.g., stop ZK, add firewall rule, or saturate ZK connections)
pulsar initialize-transaction-coordinator-metadata \
-cs <zk_address>:<port> \
-c <cluster_name>
# 2. Observe: ConnectionLoss exception is printed, but the process does not exit
# 3. Verify with thread dump:
kill -3 <pid>
# Look for:
# - "DestroyJavaVM" thread in RUNNABLE state (main() has returned)
# - "configuration-metadata-store-1-1" as the only non-daemon user thread
# in TIMED_WAITING (parking) on ScheduledThreadPoolExecutor$DelayedWorkQueue.take
### Additional information
_No response_
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] initialize-transaction-coordinator-metadata hangs indefinitely on ZK ConnectionLoss due to non-daemon executor thread leak in ZKMetadataStore constructor [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat closed issue #26582: [Bug] initialize-transaction-coordinator-metadata hangs indefinitely on ZK ConnectionLoss due to non-daemon executor thread leak in ZKMetadataStore constructor
URL: https://github.com/apache/pulsar/issues/26582
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] initialize-transaction-coordinator-metadata hangs indefinitely on ZK ConnectionLoss due to non-daemon executor thread leak in ZKMetadataStore constructor [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #26582:
URL: https://github.com/apache/pulsar/issues/26582#issuecomment-5680190886
I checked the code and this looks like a real issue. The same failure path is still present in the current checkout.
One detail worth adding: creating the executor alone doesn’t start the thread. `AbstractBatchedMetadataStore` schedules the periodic batch flush before the ZooKeeper connection is established. I checked that this also happens in 3.0.11.
If the connection fails, the constructor throws without cleaning up the superclass resources. Try-with-resources is working as expected here: it cannot close a resource whose constructor failed. The constructor needs to handle that cleanup.
The executor layout has changed in the current code, so the leaked thread would now be named `…-batch-flusher…`, but the underlying issue remains.
This is based on source inspection; I haven’t reproduced the hang locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org