You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "tiny-rain (via GitHub)" <gi...@apache.org> on 2023/04/14 11:53:58 UTC
[GitHub] [pulsar] tiny-rain opened a new issue, #20105: [Bug] ZKMetadataStore runs into deadlock
tiny-rain opened a new issue, #20105:
URL: https://github.com/apache/pulsar/issues/20105
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Version
Pulsar version: 2.11
OS: MacOs Monterey
### Minimal reproduce step
1. make zk connection lost of zkc in ZKMetadataStore
2. wait connection check thread in ZkSessionWather begin to run.
### What did you expect to see?
- **thread1:** In the session watch callback of ZKMetadataStore, it will firstly lock the pending request queue in ClientCnxn class,and then call zkSessionWather.process, which need acquire lock of zkSessionWatcher.
- **Thread2:** While the schedule connection check thread in ZkSessionWather begin running, execute zookeeper.exists in ZkSessionWather.checkConnectionStatus(), in this way, the lock of zkSessionWatcher is hold by connectionStatusCheck thread, and waiting lock of pendingrequest queue in ClientCnxn, which was already holed by the session wather callback thread.
Risk: all the metadata request are blocked.
### What did you see instead?
Java stack information for the threads listed above:
===================================================
"metadata-store-40-1":
at org.apache.pulsar.metadata.impl.ZKSessionWatcher.process(ZKSessionWatcher.java:120)
- waiting to lock <0x000000070096de18> (a org.apache.pulsar.metadata.impl.ZKSessionWatcher)
at org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$new$0(ZKMetadataStore.java:101)
at org.apache.pulsar.metadata.impl.ZKMetadataStore$$Lambda$818/0x0000000801684680.run(Unknown Source)
at java.util.concurrent.Executors$RunnableAdapter.call(java.base@17.0.6/Executors.java:539)
at java.util.concurrent.FutureTask.run$$$capture(java.base@17.0.6/FutureTask.java:264)
at java.util.concurrent.FutureTask.run(java.base@17.0.6/FutureTask.java)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@17.0.6/ScheduledThreadPoolExecutor.java:304)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.6/ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.6/ThreadPoolExecutor.java:635)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(java.base@17.0.6/Thread.java:833)
"metadata-store-zk-session-watcher-41-1":
at org.apache.zookeeper.ClientCnxn$EventThread.queuePacket(ClientCnxn.java:527)
- waiting to lock <0x0000000706564568> (a java.util.concurrent.LinkedBlockingQueue)
at org.apache.zookeeper.ClientCnxn.finishPacket(ClientCnxn.java:771)
at org.apache.zookeeper.ClientCnxn.conLossPacket(ClientCnxn.java:808)
at org.apache.zookeeper.ClientCnxn.queuePacket(ClientCnxn.java:1676)
- locked <0x0000000700d27530> (a org.apache.zookeeper.ZooKeeper$States)
at org.apache.zookeeper.ClientCnxn.queuePacket(ClientCnxn.java:1645)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1919)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1930)
at org.apache.pulsar.metadata.impl.PulsarZooKeeperClient$16.zkRun(PulsarZooKeeperClient.java:926)
at org.apache.pulsar.metadata.impl.PulsarZooKeeperClient$ZkRetryRunnable.run(PulsarZooKeeperClient.java:390)
at org.apache.pulsar.metadata.impl.PulsarZooKeeperClient.exists(PulsarZooKeeperClient.java:936)
at org.apache.pulsar.metadata.impl.ZKSessionWatcher.checkConnectionStatus(ZKSessionWatcher.java:86)
- locked <0x000000070096de18> (a org.apache.pulsar.metadata.impl.ZKSessionWatcher)
at org.apache.pulsar.metadata.impl.ZKSessionWatcher$$Lambda$328/0x00000008013d3470.run(Unknown Source)
at org.apache.pulsar.common.util.Runnables$CatchingAndLoggingRunnable.run(Runnables.java:54)
at java.util.concurrent.Executors$RunnableAdapter.call(java.base@17.0.6/Executors.java:539)
at java.util.concurrent.FutureTask.runAndReset$$$capture(java.base@17.0.6/FutureTask.java:305)
at java.util.concurrent.FutureTask.runAndReset(java.base@17.0.6/FutureTask.java)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@17.0.6/ScheduledThreadPoolExecutor.java:305)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.6/ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.6/ThreadPoolExecutor.java:635)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(java.base@17.0.6/Thread.java:833)
"ZKC-connect-executor-0-SendThread(localhost:2181)":
at org.apache.pulsar.metadata.impl.ZKSessionWatcher.setSessionInvalid(ZKSessionWatcher.java:125)
- waiting to lock <0x000000070096de18> (a org.apache.pulsar.metadata.impl.ZKSessionWatcher)
at org.apache.pulsar.metadata.impl.ZKMetadataStore$$Lambda$823/0x00000008016866c8.accept(Unknown Source)
at java.util.Optional.ifPresent(java.base@17.0.6/Optional.java:178)
at org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$receivedSessionEvent$3(ZKMetadataStore.java:146)
at org.apache.pulsar.metadata.impl.ZKMetadataStore$$Lambda$821/0x0000000801685dd8.processResult(Unknown Source)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:726)
at org.apache.zookeeper.ClientCnxn$EventThread.queuePacket(ClientCnxn.java:531)
- locked <0x0000000706564568> (a java.util.concurrent.LinkedBlockingQueue)
at org.apache.zookeeper.ClientCnxn.finishPacket(ClientCnxn.java:771)
at org.apache.zookeeper.ClientCnxn.conLossPacket(ClientCnxn.java:808)
at org.apache.zookeeper.ClientCnxn.access$2500(ClientCnxn.java:106)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1399)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanAndNotifyState(ClientCnxn.java:1327)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1301)
Found 1 deadlock.
### Anything else?
_No response_
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] nodece commented on issue #20105: [Bug] ZKMetadataStore runs into deadlock
Posted by "nodece (via GitHub)" <gi...@apache.org>.
nodece commented on issue #20105:
URL: https://github.com/apache/pulsar/issues/20105#issuecomment-1533492051
Closed by #20122.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] nodece closed issue #20105: [Bug] ZKMetadataStore runs into deadlock
Posted by "nodece (via GitHub)" <gi...@apache.org>.
nodece closed issue #20105: [Bug] ZKMetadataStore runs into deadlock
URL: https://github.com/apache/pulsar/issues/20105
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] tiny-rain commented on issue #20105: [Bug] ZKMetadataStore runs into deadlock
Posted by "tiny-rain (via GitHub)" <gi...@apache.org>.
tiny-rain commented on issue #20105:
URL: https://github.com/apache/pulsar/issues/20105#issuecomment-1508391757
`Java stack information for the threads listed above:
===================================================
"metadata-store-40-1":
at org.apache.pulsar.metadata.impl.ZKSessionWatcher.process(ZKSessionWatcher.java:120)
- waiting to lock <0x000000070096de18> (a org.apache.pulsar.metadata.impl.ZKSessionWatcher)
at org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$new$0(ZKMetadataStore.java:101)
at org.apache.pulsar.metadata.impl.ZKMetadataStore$$Lambda$818/0x0000000801684680.run(Unknown Source)
at java.util.concurrent.Executors$RunnableAdapter.call(java.base@17.0.6/Executors.java:539)
at java.util.concurrent.FutureTask.run$$$capture(java.base@17.0.6/FutureTask.java:264)
at java.util.concurrent.FutureTask.run(java.base@17.0.6/FutureTask.java)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@17.0.6/ScheduledThreadPoolExecutor.java:304)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.6/ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.6/ThreadPoolExecutor.java:635)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(java.base@17.0.6/Thread.java:833)
"metadata-store-zk-session-watcher-41-1":
at org.apache.zookeeper.ClientCnxn$EventThread.queuePacket(ClientCnxn.java:527)
- waiting to lock <0x0000000706564568> (a java.util.concurrent.LinkedBlockingQueue)
at org.apache.zookeeper.ClientCnxn.finishPacket(ClientCnxn.java:771)
at org.apache.zookeeper.ClientCnxn.conLossPacket(ClientCnxn.java:808)
at org.apache.zookeeper.ClientCnxn.queuePacket(ClientCnxn.java:1676)
- locked <0x0000000700d27530> (a org.apache.zookeeper.ZooKeeper$States)
at org.apache.zookeeper.ClientCnxn.queuePacket(ClientCnxn.java:1645)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1919)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1930)
at org.apache.pulsar.metadata.impl.PulsarZooKeeperClient$16.zkRun(PulsarZooKeeperClient.java:926)
at org.apache.pulsar.metadata.impl.PulsarZooKeeperClient$ZkRetryRunnable.run(PulsarZooKeeperClient.java:390)
at org.apache.pulsar.metadata.impl.PulsarZooKeeperClient.exists(PulsarZooKeeperClient.java:936)
at org.apache.pulsar.metadata.impl.ZKSessionWatcher.checkConnectionStatus(ZKSessionWatcher.java:86)
- locked <0x000000070096de18> (a org.apache.pulsar.metadata.impl.ZKSessionWatcher)
at org.apache.pulsar.metadata.impl.ZKSessionWatcher$$Lambda$328/0x00000008013d3470.run(Unknown Source)
at org.apache.pulsar.common.util.Runnables$CatchingAndLoggingRunnable.run(Runnables.java:54)
at java.util.concurrent.Executors$RunnableAdapter.call(java.base@17.0.6/Executors.java:539)
at java.util.concurrent.FutureTask.runAndReset$$$capture(java.base@17.0.6/FutureTask.java:305)
at java.util.concurrent.FutureTask.runAndReset(java.base@17.0.6/FutureTask.java)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@17.0.6/ScheduledThreadPoolExecutor.java:305)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.6/ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.6/ThreadPoolExecutor.java:635)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(java.base@17.0.6/Thread.java:833)
"ZKC-connect-executor-0-SendThread(localhost:2181)":
at org.apache.pulsar.metadata.impl.ZKSessionWatcher.setSessionInvalid(ZKSessionWatcher.java:125)
- waiting to lock <0x000000070096de18> (a org.apache.pulsar.metadata.impl.ZKSessionWatcher)
at org.apache.pulsar.metadata.impl.ZKMetadataStore$$Lambda$823/0x00000008016866c8.accept(Unknown Source)
at java.util.Optional.ifPresent(java.base@17.0.6/Optional.java:178)
at org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$receivedSessionEvent$3(ZKMetadataStore.java:146)
at org.apache.pulsar.metadata.impl.ZKMetadataStore$$Lambda$821/0x0000000801685dd8.processResult(Unknown Source)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:726)
at org.apache.zookeeper.ClientCnxn$EventThread.queuePacket(ClientCnxn.java:531)
- locked <0x0000000706564568> (a java.util.concurrent.LinkedBlockingQueue)
at org.apache.zookeeper.ClientCnxn.finishPacket(ClientCnxn.java:771)
at org.apache.zookeeper.ClientCnxn.conLossPacket(ClientCnxn.java:808)
at org.apache.zookeeper.ClientCnxn.access$2500(ClientCnxn.java:106)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1399)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanAndNotifyState(ClientCnxn.java:1327)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1301)
Found 1 deadlock.
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org