You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/12/12 10:25:37 UTC
[GitHub] [pulsar] merlimat opened a new pull request #12484: [Branch-2.7] Fixed deadlock on metadata cache missing while doing che…
merlimat opened a new pull request #12484:
URL: https://github.com/apache/pulsar/pull/12484
### Motivation
After the changes in #12340, there were still a couple of places making blocking calls. These calls occupy all the ordered scheduler threads preventing the callbacks to complete, until the 30 seconds timeout expire.
```
"bookkeeper-ml-scheduler-OrderedScheduler-7-0" #50 prio=5 os_prio=0 tid=0x00007f2d40050000 nid=0xe5 waiting on condition [0x00007f2d998d0000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00007f38940080e0> (a java.util.concurrent.CompletableFuture$Signaller)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1709)
at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1788)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
at org.apache.pulsar.zookeeper.ZooKeeperDataCache.get(ZooKeeperDataCache.java:97)
at org.apache.pulsar.broker.service.persistent.PersistentTopic.checkReplication(PersistentTopic.java:1152)
at org.apache.pulsar.broker.service.BrokerService$3.openLedgerComplete(BrokerService.java:1107)
at org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl.lambda$asyncOpen$8(ManagedLedgerFactoryImpl.java:425)
at org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl$$Lambda$581/978469035.accept(Unknown Source)
at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:670)
at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:646)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
at org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl$2.initializeComplete(ManagedLedgerFactoryImpl.java:397)
at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl$3$1.operationComplete(ManagedLedgerImpl.java:498)
at org.apache.bookkeeper.mledger.impl.ManagedCursorImpl$1.operationComplete(ManagedCursorImpl.java:316)
at org.apache.bookkeeper.mledger.impl.ManagedCursorImpl$1.operationComplete(ManagedCursorImpl.java:289)
at org.apache.bookkeeper.mledger.impl.MetaStoreImpl.lambda$asyncGetCursorInfo$11(MetaStoreImpl.java:170)
at org.apache.bookkeeper.mledger.impl.MetaStoreImpl$$Lambda$679/542144696.accept(Unknown Source)
at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:670)
at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:646)
at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
at org.apache.bookkeeper.common.util.OrderedExecutor$TimedRunnable.run(OrderedExecutor.java:203)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
"pulsar-ordered-OrderedExecutor-0-0" #13 prio=5 os_prio=0 tid=0x00007f3f73dac800 nid=0xc1 waiting on condition [0x00007f2de07e1000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00007f38940388f8> (a java.util.concurrent.CompletableFuture$Signaller)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1709)
at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1788)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
at org.apache.pulsar.zookeeper.ZooKeeperDataCache.get(ZooKeeperDataCache.java:97)
at org.apache.pulsar.broker.service.BrokerService.lambda$getManagedLedgerConfig$43(BrokerService.java:1199)
at org.apache.pulsar.broker.service.BrokerService$$Lambda$455/163843091.run(Unknown Source)
at org.apache.bookkeeper.mledger.util.SafeRun$2.safeRun(SafeRun.java:49)
at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
```
Instead converted the code to use `getAsync()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] hangc0276 commented on pull request #12484: [Branch-2.7] Fixed deadlock on metadata cache missing while doing che…
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on pull request #12484:
URL: https://github.com/apache/pulsar/pull/12484#issuecomment-958598901
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #12484: [Branch-2.7] Fixed deadlock on metadata cache missing while doing che…
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on pull request #12484:
URL: https://github.com/apache/pulsar/pull/12484#issuecomment-991834998
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui closed pull request #12484: [Branch-2.7] Fixed deadlock on metadata cache missing while doing che…
Posted by GitBox <gi...@apache.org>.
codelipenghui closed pull request #12484:
URL: https://github.com/apache/pulsar/pull/12484
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] hangc0276 commented on pull request #12484: [Branch-2.7] Fixed deadlock on metadata cache missing while doing che…
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on pull request #12484:
URL: https://github.com/apache/pulsar/pull/12484#issuecomment-958598901
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #12484: [Branch-2.7] Fixed deadlock on metadata cache missing while doing che…
Posted by GitBox <gi...@apache.org>.
lhotari commented on pull request #12484:
URL: https://github.com/apache/pulsar/pull/12484#issuecomment-1034951243
I rebased the changes. Let's see what the test failures are.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] hangc0276 commented on pull request #12484: [Branch-2.7] Fixed deadlock on metadata cache missing while doing che…
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on pull request #12484:
URL: https://github.com/apache/pulsar/pull/12484#issuecomment-958598901
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #12484: [Branch-2.7] Fixed deadlock on metadata cache missing while doing che…
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12484:
URL: https://github.com/apache/pulsar/pull/12484#issuecomment-1066259863
The pr had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #12484: [Branch-2.7] Fixed deadlock on metadata cache missing while doing che…
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on pull request #12484:
URL: https://github.com/apache/pulsar/pull/12484#issuecomment-991834998
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #12484: [Branch-2.7] Fixed deadlock on metadata cache missing while doing che…
Posted by GitBox <gi...@apache.org>.
lhotari commented on pull request #12484:
URL: https://github.com/apache/pulsar/pull/12484#issuecomment-1036106896
There are too many failures that I'm not confident to pick this in 2.7.5 release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] 315157973 commented on pull request #12484: [Branch-2.7] Fixed deadlock on metadata cache missing while doing che…
Posted by GitBox <gi...@apache.org>.
315157973 commented on pull request #12484:
URL: https://github.com/apache/pulsar/pull/12484#issuecomment-953543704
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #12484: [Branch-2.7] Fixed deadlock on metadata cache missing while doing che…
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on pull request #12484:
URL: https://github.com/apache/pulsar/pull/12484#issuecomment-953591022
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org