You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2019/09/30 23:02:48 UTC
[GitHub] [pulsar] rdhabalia opened a new pull request #5298: [pulsar-broker]
Fix: invalidate cache on zk-cache timeout
rdhabalia opened a new pull request #5298: [pulsar-broker] Fix: invalidate cache on zk-cache timeout
URL: https://github.com/apache/pulsar/pull/5298
### Motivation
Right now, while creating producer/consumer, broker tries to authorize them by fetching namespace policies using zkCache. But sometimes, zkCache times-out while getting zk-node into cache and zk-cache keeps pending key for 5 mins. Because of that client can't create producer/consumer on that topic for 5 mins.
Fix: broker should invalidate key if it receives timeout while accessing zk-cache so, subsequent requests can successfully fetch it
```
01:07:14.847 [pulsar-ordered-OrderedExecutor-2-0] WARN org.apache.pulsar.broker.service.BrokerService - Got exception when reading persistence policy for persistent://prop/cluster/ns/topic: null
java.util.concurrent.TimeoutException: null
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021) ~[?:?]
at org.apache.pulsar.zookeeper.ZooKeeperDataCache.get(ZooKeeperDataCache.java:95) ~[pulsar-zookeeper-utils-2.4.3-yahoo.jar:2.4.3-yahoo]
at org.apache.pulsar.broker.service.BrokerService.lambda$getManagedLedgerConfig$119(BrokerService.java:723) ~[pulsar-broker-2.4.3-yahoo.jar:2.4.3-yahoo]
at org.apache.bookkeeper.mledger.util.SafeRun$2.safeRun(SafeRun.java:49) [managed-ledger-original-2.4.3-yahoo.jar:2.4.3-yahoo]
at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [bookkeeper-common-4.9.0.jar:4.9.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.32.Final.jar:4.1.32.Final]
at java.lang.Thread.run(Thread.java:834) [?:?]
01:07
:
:
01:07:14.848 [bookkeeper-ml-workers-OrderedExecutor-3-0] WARN org.apache.pulsar.broker.service.persistent.DispatchRateLimiter - Failed to get message-rate for persistent://prop/cluster/ns/topic
java.util.concurrent.TimeoutException: null
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021) ~[?:?]
at org.apache.pulsar.broker.service.persistent.DispatchRateLimiter.getPolicies(DispatchRateLimiter.java:254) ~[pulsar-broker-2.4.3-yahoo.jar:2.4.3-yahoo]
at org.apache.pulsar.broker.service.persistent.DispatchRateLimiter.isDispatchRateNeeded(DispatchRateLimiter.java:155) ~[pulsar-broker-2.4.3-yahoo.jar:2.4.3-yahoo]
at org.apache.pulsar.broker.service.persistent.PersistentTopic.initializeDispatchRateLimiterIfNeeded(PersistentTopic.java:248) ~[pulsar-broker-2.4.3-yahoo.jar:2.4.3-yahoo]
at org.apache.pulsar.broker.service.persistent.PersistentTopic.<init>(PersistentTopic.java:199) ~[pulsar-broker-2.4.3-yahoo.jar:2.4.3-yahoo]
at org.apache.pulsar.broker.service.BrokerService$3.openLedgerComplete(BrokerService.java:656) ~[pulsar-broker-2.4.3-yahoo.jar:2.4.3-yahoo]
at org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl.lambda$asyncOpen$98(ManagedLedgerFactoryImpl.java:328) ~[managed-ledger-original-2.4.3-yahoo.jar:2.4.3-yahoo
]
at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:714) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) ~[?:?]
at org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl$2.initializeComplete(ManagedLedgerFactoryImpl.java:316) ~[managed-ledger-original-2.4.3-yahoo.jar:2.4.3-yahoo]
at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl$3$1.operationComplete(ManagedLedgerImpl.java:467) ~[managed-ledger-original-2.4.3-yahoo.jar:2.4.3-yahoo]
at org.apache.bookkeeper.mledger.impl.ManagedCursorImpl$1.operationComplete(ManagedCursorImpl.java:273) ~[managed-ledger-original-2.4.3-yahoo.jar:2.4.3-yahoo]
at org.apache.bookkeeper.mledger.impl.ManagedCursorImpl$1.operationComplete(ManagedCursorImpl.java:246) ~[managed-ledger-original-2.4.3-yahoo.jar:2.4.3-yahoo]
at org.apache.bookkeeper.mledger.impl.MetaStoreImplZookeeper.lambda$null$120(MetaStoreImplZookeeper.java:241) ~[managed-ledger-original-2.4.3-yahoo.jar:2.4.3-yahoo]
at org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:32) [managed-ledger-original-2.4.3-yahoo.jar:2.4.3-yahoo]
at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [bookkeeper-common-4.9.0.jar:4.9.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.32.Final.jar:4.1.32.Final]
at java.lang.Thread.run(Thread.java:834) [?:?]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services