You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2017/12/02 07:58:02 UTC

[GitHub] rdhabalia opened a new pull request #934: Avoid dead-lock on broker service when broker failed to load managed-ledger

rdhabalia opened a new pull request #934: Avoid dead-lock on broker service when broker failed to load managed-ledger
URL: https://github.com/apache/incubator-pulsar/pull/934
 
 
   ### Motivation
   
   Recently we have seen that sometimes broker get stuck with dead-lock when it fails to load managed-ledger. From [thread-dump](https://gist.github.com/rdhabalia/2d64abdcdf6ce23e5d54d617daf47d99) it seems that when a thread tries to create a topic in concurrentOpenHashMap and if the thread fails to create topic due to failure in loading managed-ledger then same thread tries to remove topic from concurrentOpenHashMap which causes the deadlock and all io and web threads result in deadlock.
   ```
   java.lang.Thread.State: WAITING (parking)
           at sun.misc.Unsafe.park(Native Method)
           - parking to wait for  <0x000000029e30d960> (a org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section)
           at java.util.concurrent.locks.StampedLock.acquireWrite(StampedLock.java:1119)
           at java.util.concurrent.locks.StampedLock.writeLock(StampedLock.java:354)
           at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.remove(ConcurrentOpenHashMap.java:305)
           at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.access$200(ConcurrentOpenHashMap.java:181)
           at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.remove(ConcurrentOpenHashMap.java:143)
           at org.apache.pulsar.broker.service.BrokerService$3.openLedgerFailed(BrokerService.java:597)
           at org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl.lambda$asyncOpen$7(ManagedLedgerFactoryImpl.java:260)
           at org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl$$Lambda$255/449424498.apply(Unknown Source)
           at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870)
           at java.util.concurrent.CompletableFuture.uniExceptionallyStage(CompletableFuture.java:884)
           at java.util.concurrent.CompletableFuture.exceptionally(CompletableFuture.java:2196)
           at org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl.asyncOpen(ManagedLedgerFactoryImpl.java:259)
           at org.apache.pulsar.broker.service.BrokerService.lambda$10(BrokerService.java:568)
           at org.apache.pulsar.broker.service.BrokerService$$Lambda$247/1679793959.accept(Unknown Source)
           at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:656)
           at java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:669)
           at java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:1997)
           at org.apache.pulsar.broker.service.BrokerService.createPersistentTopic(BrokerService.java:565)
           at org.apache.pulsar.broker.service.BrokerService.createPersistentTopic(BrokerService.java:536)
           at org.apache.pulsar.broker.service.BrokerService.lambda$4(BrokerService.java:423)
           at org.apache.pulsar.broker.service.BrokerService$$Lambda$244/1271565658.apply(Unknown Source)
           at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.put(ConcurrentOpenHashMap.java:275)
           at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.computeIfAbsent(ConcurrentOpenHashMap.java:130)
           at org.apache.pulsar.broker.service.BrokerService.getTopic(BrokerService.java:422)
           at org.apache.pulsar.broker.service.ServerCnx.lambda$5(ServerCnx.java:481)
           at org.apache.pulsar.broker.service.ServerCnx$$Lambda$294/982643452.apply(Unknown Source)
           at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
           at java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:614)
           at java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:1983)
           at org.apache.pulsar.broker.service.ServerCnx.handleProducer(ServerCnx.java:448)
   ```
   
   ### Modifications
   
   - Clean up topic map in different thread.
   
   ### Result
   
   - It will prevent possible dead-lock when broker fails to cleanup failed topic.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services