You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2019/04/08 11:13:34 UTC
[GitHub] [pulsar] massakam opened a new pull request #4003: [broker] Fix
deadlock during GC of non-persistent topic
massakam opened a new pull request #4003: [broker] Fix deadlock during GC of non-persistent topic
URL: https://github.com/apache/pulsar/pull/4003
### Motivation
A deadlock can occur when non-persistent topics are deleted by GC. The cause is that thread `pulsar-inactivity-monitor-x` iterates over the topics map and tries to remove the element from the map.
```
"pulsar-inactivity-monitor-23-1" #65 prio=5 os_prio=0 tid=0x00007f64c5a50800 nid=0x723a waiting on condition [0x00007f6429bbc000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c13a3af8> (a org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section)
at java.util.concurrent.locks.StampedLock.acquireWrite(StampedLock.java:1119)
at java.util.concurrent.locks.StampedLock.writeLock(StampedLock.java:354)
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.remove(ConcurrentOpenHashMap.java:305)
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.access$200(ConcurrentOpenHashMap.java:181)
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.remove(ConcurrentOpenHashMap.java:136)
at org.apache.pulsar.broker.service.BrokerService.removeTopicFromCache(BrokerService.java:1018)
at org.apache.pulsar.broker.service.nonpersistent.NonPersistentTopic.lambda$42(NonPersistentTopic.java:469)
at org.apache.pulsar.broker.service.nonpersistent.NonPersistentTopic$$Lambda$575/1212530958.accept(Unknown Source)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:778)
at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2140)
at org.apache.pulsar.broker.service.nonpersistent.NonPersistentTopic.lambda$11(NonPersistentTopic.java:463)
at org.apache.pulsar.broker.service.nonpersistent.NonPersistentTopic$$Lambda$574/1385461747.accept(Unknown Source)
at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:656)
at java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:669)
at java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:1997)
at org.apache.pulsar.broker.service.nonpersistent.NonPersistentTopic.delete(NonPersistentTopic.java:446)
at org.apache.pulsar.broker.service.nonpersistent.NonPersistentTopic.lambda$34(NonPersistentTopic.java:920)
at org.apache.pulsar.broker.service.nonpersistent.NonPersistentTopic$$Lambda$573/2109183578.apply(Unknown Source)
at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:981)
at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2124)
at org.apache.pulsar.broker.service.nonpersistent.NonPersistentTopic.checkGC(NonPersistentTopic.java:920)
at org.apache.pulsar.broker.service.BrokerService.lambda$18(BrokerService.java:863)
at org.apache.pulsar.broker.service.BrokerService$$Lambda$312/1012814229.accept(Unknown Source)
at org.apache.pulsar.broker.service.BrokerService.lambda$20(BrokerService.java:893)
at org.apache.pulsar.broker.service.BrokerService$$Lambda$313/945930950.accept(Unknown Source)
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:386)
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:160)
at org.apache.pulsar.broker.service.BrokerService.forEachTopic(BrokerService.java:890)
at org.apache.pulsar.broker.service.BrokerService.checkGC(BrokerService.java:863)
at org.apache.pulsar.broker.service.BrokerService.lambda$1(BrokerService.java:334)
at org.apache.pulsar.broker.service.BrokerService$$Lambda$87/1991724700.run(Unknown Source)
at org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:32)
at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
```
The read lock is not released until ConcurrentOpenHashMap#forEach completes. If the read lock is not released, the write lock can not be acquired and ConcurrentOpenHashMap#remove operation is blocked. Therefore, forEach will never complete.
### Modifications
As with https://github.com/apache/pulsar/pull/1660, remove topics from the topic map in a different thread.
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services