You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/11/12 09:34:06 UTC
[GitHub] [pulsar] Technoboy- opened a new pull request #12773: Fix TopicPoliciesCacheNotInitException issue.
Technoboy- opened a new pull request #12773:
URL: https://github.com/apache/pulsar/pull/12773
### Motivation
Sometimes, we may get `TopicPoliciesCacheNotInitException` with below stack trace:
```
15:45:47.020 [pulsar-web-41-3] INFO org.eclipse.jetty.server.RequestLog - 10.0.0.42 - - [10/Nov/2021:15:45:47 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/1.19+" 1
15:45:51.221 [pulsar-2-15] ERROR org.apache.pulsar.broker.admin.impl.PersistentTopicsBase - [null] Failed to perform getRetention on topic persistent://public/default/UpdateNodeCharts
java.lang.RuntimeException: org.apache.pulsar.broker.service.BrokerServiceException$TopicPoliciesCacheNotInitException: Topic policies cache have not init.
at org.apache.pulsar.broker.service.TopicPoliciesService.lambda$getTopicPoliciesAsyncWithRetry$0(TopicPoliciesService.java:84) ~[io.streamnative-pulsar-broker-2.8.1.21.jar:2.8.1.21]
at org.apache.pulsar.client.util.RetryUtil.executeWithRetry(RetryUtil.java:50) ~[io.streamnative-pulsar-client-original-2.8.1.21.jar:2.8.1.21]
at org.apache.pulsar.client.util.RetryUtil.lambda$executeWithRetry$1(RetryUtil.java:63) ~[io.streamnative-pulsar-client-original-2.8.1.21.jar:2.8.1.21]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.68.Final.jar:4.1.68.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]
```
This is because : https://github.com/apache/pulsar/blob/c3da1452a444c9599cb85562a3faa82ddfdecec8/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesService.java#L298-L312
when `reader.readNextAsync()` throws exceptions, the msg will be null which will throw NPE without any catch block.
### Modifications
- Close reader when `reader.readNextAsync()` throw exceptions.
- Remove reducant field `policyCacheInitMap` . we can use `readerCaches` instead.
- Add retry logic when `getTopicPolicies`.
### Documentation
- [ x ] `no-need-doc`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- commented on pull request #12773: Fix TopicPoliciesCacheNotInitException issue.
Posted by GitBox <gi...@apache.org>.
Technoboy- commented on pull request #12773:
URL: https://github.com/apache/pulsar/pull/12773#issuecomment-967112955
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- commented on pull request #12773: Fix TopicPoliciesCacheNotInitException issue.
Posted by GitBox <gi...@apache.org>.
Technoboy- commented on pull request #12773:
URL: https://github.com/apache/pulsar/pull/12773#issuecomment-968276329
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- removed a comment on pull request #12773: Fix TopicPoliciesCacheNotInitException issue.
Posted by GitBox <gi...@apache.org>.
Technoboy- removed a comment on pull request #12773:
URL: https://github.com/apache/pulsar/pull/12773#issuecomment-968276329
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- commented on pull request #12773: Fix TopicPoliciesCacheNotInitException issue.
Posted by GitBox <gi...@apache.org>.
Technoboy- commented on pull request #12773:
URL: https://github.com/apache/pulsar/pull/12773#issuecomment-968618996
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui merged pull request #12773: Fix TopicPoliciesCacheNotInitException issue.
Posted by GitBox <gi...@apache.org>.
codelipenghui merged pull request #12773:
URL: https://github.com/apache/pulsar/pull/12773
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- commented on pull request #12773: Fix TopicPoliciesCacheNotInitException issue.
Posted by GitBox <gi...@apache.org>.
Technoboy- commented on pull request #12773:
URL: https://github.com/apache/pulsar/pull/12773#issuecomment-970090725
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- commented on pull request #12773: Fix TopicPoliciesCacheNotInitException issue.
Posted by GitBox <gi...@apache.org>.
Technoboy- commented on pull request #12773:
URL: https://github.com/apache/pulsar/pull/12773#issuecomment-968422949
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- removed a comment on pull request #12773: Fix TopicPoliciesCacheNotInitException issue.
Posted by GitBox <gi...@apache.org>.
Technoboy- removed a comment on pull request #12773:
URL: https://github.com/apache/pulsar/pull/12773#issuecomment-968289870
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- commented on pull request #12773: Fix TopicPoliciesCacheNotInitException issue.
Posted by GitBox <gi...@apache.org>.
Technoboy- commented on pull request #12773:
URL: https://github.com/apache/pulsar/pull/12773#issuecomment-970258481
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on a change in pull request #12773: Fix TopicPoliciesCacheNotInitException issue.
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on a change in pull request #12773:
URL: https://github.com/apache/pulsar/pull/12773#discussion_r748370933
##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesService.java
##########
@@ -287,13 +275,34 @@ public boolean test(NamespaceBundle namespaceBundle) {
});
}
+ private synchronized void prepareInitPoliciesCache(NamespaceName namespace, CompletableFuture<Void> result) {
+ if (readerCaches.get(namespace) == null) {
+ SystemTopicClient<PulsarEvent> systemTopicClient = namespaceEventsSystemTopicFactory
+ .createTopicPoliciesSystemTopicClient(namespace);
+ CompletableFuture<SystemTopicClient.Reader<PulsarEvent>> readerCompletableFuture =
+ systemTopicClient.newReaderAsync();
+ readerCaches.put(namespace, readerCompletableFuture);
+ readerCompletableFuture.whenComplete((reader, ex) -> {
+ if (ex != null) {
+ log.error("[{}] Failed to create reader on __change_events topic", namespace, ex);
+ result.completeExceptionally(ex);
Review comment:
Should remove the failed reader future？
##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesService.java
##########
@@ -287,13 +275,34 @@ public boolean test(NamespaceBundle namespaceBundle) {
});
}
+ private synchronized void prepareInitPoliciesCache(NamespaceName namespace, CompletableFuture<Void> result) {
+ if (readerCaches.get(namespace) == null) {
+ SystemTopicClient<PulsarEvent> systemTopicClient = namespaceEventsSystemTopicFactory
+ .createTopicPoliciesSystemTopicClient(namespace);
+ CompletableFuture<SystemTopicClient.Reader<PulsarEvent>> readerCompletableFuture =
+ systemTopicClient.newReaderAsync();
+ readerCaches.put(namespace, readerCompletableFuture);
+ readerCompletableFuture.whenComplete((reader, ex) -> {
+ if (ex != null) {
+ log.error("[{}] Failed to create reader on __change_events topic", namespace, ex);
+ result.completeExceptionally(ex);
+ } else {
+ initPolicesCache(reader, result);
+ result.thenRun(() -> readMorePolicies(reader));
+ }
+ });
+ }
+ }
+
private void initPolicesCache(SystemTopicClient.Reader<PulsarEvent> reader, CompletableFuture<Void> future) {
reader.hasMoreEventsAsync().whenComplete((hasMore, ex) -> {
if (ex != null) {
log.error("[{}] Failed to check the move events for the system topic",
reader.getSystemTopic().getTopicName(), ex);
future.completeExceptionally(ex);
readerCaches.remove(reader.getSystemTopic().getTopicName().getNamespaceObject());
+ reader.closeAsync();
Review comment:
Same as the above comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- commented on pull request #12773: Fix TopicPoliciesCacheNotInitException issue.
Posted by GitBox <gi...@apache.org>.
Technoboy- commented on pull request #12773:
URL: https://github.com/apache/pulsar/pull/12773#issuecomment-968289870
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- commented on pull request #12773: Fix TopicPoliciesCacheNotInitException issue.
Posted by GitBox <gi...@apache.org>.
Technoboy- commented on pull request #12773:
URL: https://github.com/apache/pulsar/pull/12773#issuecomment-970214995
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org