You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/06/02 07:08:36 UTC
[GitHub] [pulsar] truong-hua opened a new issue, #15891: "Topic policies cache have not init" randomly happen on different topics in the same namespace
truong-hua opened a new issue, #15891:
URL: https://github.com/apache/pulsar/issues/15891
**Describe the bug**
I think my bug is related to this issue https://github.com/apache/pulsar/issues/10231 but actually it's not because our Pulsar has disabled the `brokerDeleteInactiveTopicsEnabled` in broker level. And the issue "Topic policies cache have not init" happened on both namespaces with "inactiveTopicPolicies" enabled and disabled.
We are all using partition topics.
Our full adjusted configs:
```
allowAutoTopicCreationType=partitioned
brokerDeleteInactivePartitionedTopicMetadataEnabled=true
brokerDeleteInactiveTopicsEnabled=false
clusterName=mycluster
configurationMetadataStoreUrl=zookeeper:2181
forceDeleteNamespaceAllowed=true
forceDeleteTenantAllowed=true
functionsWorkerEnabled=false
managedLedgerDefaultAckQuorum=2
managedLedgerDefaultEnsembleSize=3
managedLedgerDefaultWriteQuorum=3
metadataStoreUrl=zookeeper:2181
numHttpServerThreads=32
numIOThreads=8
systemTopicEnabled=true
topicLevelPoliciesEnabled=true
```
**To Reproduce**
Steps to reproduce the behavior:
I have no idea of how to reproduce this issue but it happens on our production environment and not in staging. May be it's related to production data
**Expected behavior**
We can set topic policies properly without exception that:
```
Message: Topic policies cache have not init.
Stacktrace:
org.apache.pulsar.broker.service.BrokerServiceException$TopicPoliciesCacheNotInitException: Topic policies cache have not init.
at org.apache.pulsar.broker.service.SystemTopicBasedTopicPoliciesService.getTopicPolicies(SystemTopicBasedTopicPoliciesService.java:207)
at org.apache.pulsar.broker.service.TopicPoliciesService.lambda$getTopicPoliciesAsyncWithRetry$0(TopicPoliciesService.java:100)
at org.apache.pulsar.client.util.RetryUtil.executeWithRetry(RetryUtil.java:48)
at org.apache.pulsar.client.util.RetryUtil.lambda$null$1(RetryUtil.java:57)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:829)
```
**Desktop (please complete the following information):**
- OS: Ubuntu with Pulsar 2.10.0
**Additional context**
Currently is there any Rest API for manually init the Topic Policies Cache or any way to check it via command line and manually init it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dragonls commented on issue #15891: "Topic policies cache have not init" randomly happen on different topics in the same namespace
Posted by GitBox <gi...@apache.org>.
dragonls commented on issue #15891:
URL: https://github.com/apache/pulsar/issues/15891#issuecomment-1175021533
>
It also happens to us just the same error as describe above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dragonls commented on issue #15891: "Topic policies cache have not init" randomly happen on different topics in the same namespace
Posted by GitBox <gi...@apache.org>.
dragonls commented on issue #15891:
URL: https://github.com/apache/pulsar/issues/15891#issuecomment-1175022168
> It also happens to us when we try to determine a topic compaction threshold.
>
> We use Pulsar 2.8.3.
>
> Error when issuing command to get compaction threshold:
>
> ```
> root@rev-pulsar-toolset-0:/pulsar/bin# ./pulsar-admin topics get-compaction-threshold persistent://dbus/test/dummy-objects
> HTTP 500 Server Error
>
> Reason: HTTP 500 Server Error
> ```
>
> Error seen on broker:
>
> ```
> 10:02:50.919 [pulsar-io-4-4] INFO org.apache.pulsar.broker.service.ServerCnx - [PersistentTopic{topic=persistent://dbus/dgt-time-series-point/status}][rev-pulsar-4-4116] Closed producer on cnx /10.72.168.226:33920. producerId=8040
> 10:02:51.016 [pulsar-2-1] ERROR org.apache.pulsar.broker.admin.impl.PersistentTopicsBase - [null] Failed to perform getCompactionThreshold on topic persistent://dbus/test/dummy-objects
> org.apache.pulsar.broker.service.BrokerServiceException$TopicPoliciesCacheNotInitException: Topic policies cache have not init.
> at org.apache.pulsar.broker.service.SystemTopicBasedTopicPoliciesService.getTopicPolicies(SystemTopicBasedTopicPoliciesService.java:185) ~[org.apache.pulsar-pulsar-broker-2.8.3.jar:2.8.3]
> at org.apache.pulsar.broker.service.TopicPoliciesService.lambda$getTopicPoliciesAsyncWithRetry$0(TopicPoliciesService.java:83) ~[org.apache.pulsar-pulsar-broker-2.8.3.jar:2.8.3]
> at org.apache.pulsar.client.util.RetryUtil.executeWithRetry(RetryUtil.java:49) ~[org.apache.pulsar-pulsar-client-original-2.8.3.jar:2.8.3]
> at org.apache.pulsar.client.util.RetryUtil.lambda$executeWithRetry$1(RetryUtil.java:58) ~[org.apache.pulsar-pulsar-client-original-2.8.3.jar:2.8.3]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
> at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
> at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
> at java.lang.Thread.run(Thread.java:829) [?:?]
> ```
It also happens to us just the same error as describe above. Also Pulsar 2.8.3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] shaniMaayan commented on issue #15891: "Topic policies cache have not init" randomly happen on different topics in the same namespace
Posted by GitBox <gi...@apache.org>.
shaniMaayan commented on issue #15891:
URL: https://github.com/apache/pulsar/issues/15891#issuecomment-1154155160
it happened to me and my team as well, we are currently testing topic policies(so topic policies are enabled) , this error pops up randomly for us (probably related to tests), the way around we found is to unload namespace, we also did not find a way to reproduce
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] fmiguelez commented on issue #15891: "Topic policies cache have not init" randomly happen on different topics in the same namespace
Posted by "fmiguelez (via GitHub)" <gi...@apache.org>.
fmiguelez commented on issue #15891:
URL: https://github.com/apache/pulsar/issues/15891#issuecomment-1731242916
> @Raven888888 @diegosalvi @truong-hua We have found the root cause and pushed out the fix #17562 Thanks for reporting the issue.
@Technoboy-
A similar issue is happening in 2.8.4. I see that fix was backported to 2.8 branch to be scheduled for 2.8.5 but such version was never released. Is branch 2.8.x unmaintained? Could version 2.8.5 be released? We are in an urgent situation here as it is too risky to update to 2.9.5 in our production systems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- commented on issue #15891: "Topic policies cache have not init" randomly happen on different topics in the same namespace
Posted by GitBox <gi...@apache.org>.
Technoboy- commented on issue #15891:
URL: https://github.com/apache/pulsar/issues/15891#issuecomment-1247607420
@Raven888888 @diegosalvi @truong-hua
We have found the root cause and pushed out the fix https://github.com/apache/pulsar/pull/17562
Thanks for reporting the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] omnilight commented on issue #15891: "Topic policies cache have not init" randomly happen on different topics in the same namespace
Posted by GitBox <gi...@apache.org>.
omnilight commented on issue #15891:
URL: https://github.com/apache/pulsar/issues/15891#issuecomment-1175456066
Pulsar 2.9.2, same issue. #16331 won't help us because we have `is_allow_auto_update_schema` enabled
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dragonls commented on issue #15891: "Topic policies cache have not init" randomly happen on different topics in the same namespace
Posted by GitBox <gi...@apache.org>.
dragonls commented on issue #15891:
URL: https://github.com/apache/pulsar/issues/15891#issuecomment-1178612099
> @dragonls thank for that, is it safe to manually update the znode. Do we have any document about metadata stored in the zk?
I'm not sure whether it is safe, but for my case, the topic has beed terminated and no ledgers left, which make the topic can not be loaded. I also reproduce my case in test env and do not see anything else wrong.
I do not see any document about metadata stored in the zk.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Raven888888 commented on issue #15891: "Topic policies cache have not init" randomly happen on different topics in the same namespace
Posted by GitBox <gi...@apache.org>.
Raven888888 commented on issue #15891:
URL: https://github.com/apache/pulsar/issues/15891#issuecomment-1243871294
Same the same issue today. Out of nowhere broker unable to create topic due to "Topic policy cache have not init".
I have enabled systemTopic and topic level policy, been using the same config for almost a month and suddenly this pops up in production.
Delete topic, unload topic, restart broker, delete `__change_events` topic, all did not help.
Had to urgently pull down production pulsar cluster and disable systemTopic and topic level policy. And it works after.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Raven888888 commented on issue #15891: "Topic policies cache have not init" randomly happen on different topics in the same namespace
Posted by GitBox <gi...@apache.org>.
Raven888888 commented on issue #15891:
URL: https://github.com/apache/pulsar/issues/15891#issuecomment-1247648039
Awesome to hear that @Technoboy- appreciate it
Hope to use back topic level policy in the next release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] fmiguelez commented on issue #15891: "Topic policies cache have not init" randomly happen on different topics in the same namespace
Posted by GitBox <gi...@apache.org>.
fmiguelez commented on issue #15891:
URL: https://github.com/apache/pulsar/issues/15891#issuecomment-1160243339
It also happens to us when we try to determine a topic compaction threshold.
We use Pulsar 2.8.3.
Error when issuing command to get compaction threshold:
```
root@rev-pulsar-toolset-0:/pulsar/bin# ./pulsar-admin topics get-compaction-threshold persistent://dbus/test/dummy-objects
HTTP 500 Server Error
Reason: HTTP 500 Server Error
```
Error seen on broker:
```
10:02:50.919 [pulsar-io-4-4] INFO org.apache.pulsar.broker.service.ServerCnx - [PersistentTopic{topic=persistent://dbus/dgt-time-series-point/status}][rev-pulsar-4-4116] Closed producer on cnx /10.72.168.226:33920. producerId=8040
10:02:51.016 [pulsar-2-1] ERROR org.apache.pulsar.broker.admin.impl.PersistentTopicsBase - [null] Failed to perform getCompactionThreshold on topic persistent://dbus/test/dummy-objects
org.apache.pulsar.broker.service.BrokerServiceException$TopicPoliciesCacheNotInitException: Topic policies cache have not init.
at org.apache.pulsar.broker.service.SystemTopicBasedTopicPoliciesService.getTopicPolicies(SystemTopicBasedTopicPoliciesService.java:185) ~[org.apache.pulsar-pulsar-broker-2.8.3.jar:2.8.3]
at org.apache.pulsar.broker.service.TopicPoliciesService.lambda$getTopicPoliciesAsyncWithRetry$0(TopicPoliciesService.java:83) ~[org.apache.pulsar-pulsar-broker-2.8.3.jar:2.8.3]
at org.apache.pulsar.client.util.RetryUtil.executeWithRetry(RetryUtil.java:49) ~[org.apache.pulsar-pulsar-client-original-2.8.3.jar:2.8.3]
at org.apache.pulsar.client.util.RetryUtil.lambda$executeWithRetry$1(RetryUtil.java:58) ~[org.apache.pulsar-pulsar-client-original-2.8.3.jar:2.8.3]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Raven888888 commented on issue #15891: "Topic policies cache have not init" randomly happen on different topics in the same namespace
Posted by GitBox <gi...@apache.org>.
Raven888888 commented on issue #15891:
URL: https://github.com/apache/pulsar/issues/15891#issuecomment-1244898748
Same the same issue today. Out of nowhere broker unable to create topic due to "Topic policy cache have not init".
I have enabled systemTopic and topic level policy, been using the same config for almost a month and suddenly this pops up in production.
Delete topic, unload topic, restart broker, delete `__change_events` topic, all did not help.
Had to urgently pull down production pulsar cluster and disable systemTopic and topic level policy. And it works after.
Pulsar 2.10.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- closed issue #15891: "Topic policies cache have not init" randomly happen on different topics in the same namespace
Posted by GitBox <gi...@apache.org>.
Technoboy- closed issue #15891: "Topic policies cache have not init" randomly happen on different topics in the same namespace
URL: https://github.com/apache/pulsar/issues/15891
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dragonls commented on issue #15891: "Topic policies cache have not init" randomly happen on different topics in the same namespace
Posted by GitBox <gi...@apache.org>.
dragonls commented on issue #15891:
URL: https://github.com/apache/pulsar/issues/15891#issuecomment-1178512034
I solved my problem anyway.
I tracked the code and found that "Topic policies cache have not init" most likely because something wrong with the `__change_events` topic. The broker failed to consume the system topic. In this scene of mine, the system topic can not be loaded.
I checked the system topic and found that one partition (`__change_events-partition-1`) is `terminated` and the ledger had already beed deleted. I tried deleting the `__change_events-partition-1` but failed. Finally I delete the znode `/managed-ledgers/tenant/namespace/persistent/__change_events-partition-1` and restart the broker which is the owner of `__change_events-partition-1`, the broker created the partition (`allowAutoTopicCreation=true`) and then everything back to normal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- commented on issue #15891: "Topic policies cache have not init" randomly happen on different topics in the same namespace
Posted by GitBox <gi...@apache.org>.
Technoboy- commented on issue #15891:
URL: https://github.com/apache/pulsar/issues/15891#issuecomment-1193466315
> I solved my problem anyway.
>
> I tracked the code and found that "Topic policies cache have not init" most likely because something wrong with the `__change_events` topic. The broker failed to consume the system topic. In this scene of mine, the system topic can not be loaded.
>
> I checked the system topic and found that one partition (`__change_events-partition-1`) is `terminated` and the ledger had already beed deleted. I tried deleting the `__change_events-partition-1` but failed. Finally I delete the znode `/managed-ledgers/tenant/namespace/persistent/__change_events-partition-1` and restart the broker which is the owner of `__change_events-partition-1`, the broker auto created the partition (`allowAutoTopicCreation=true`) and then restart all brokers. Everything back to normal.
What do you mean the partition is `terminated` ? Do you mean that the partition state is `terminated` which can't produce any messages but can consume ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] arnarg commented on issue #15891: "Topic policies cache have not init" randomly happen on different topics in the same namespace
Posted by GitBox <gi...@apache.org>.
arnarg commented on issue #15891:
URL: https://github.com/apache/pulsar/issues/15891#issuecomment-1172249945
Do you have `schemaValidationEnforced` turned on on a namespace level? I'm seeing the following in logs just before receiving the `Topic policies cache have not init.` exception.
```
2022-07-01T11:25:15,816+0000 [pulsar-io-4-5] WARN org.apache.pulsar.client.impl.PulsarClientImpl - [persistent://showcase/user/__change_events] Failed to get create topic reader
java.util.concurrent.CompletionException: org.apache.pulsar.client.api.PulsarClientException$IncompatibleSchemaException: {"errorMsg":"Schema not found and schema auto updating is disabled.","reqId":4307185312891752491, "remote":"dev01-neuron-pulsar-broker-0.dev01-neuron-pulsar-broker.dev01-neuron-pulsar.svc.cluster.local/10.130.2.163:6650", "local":"/10.130.2.163:56456"}
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331) ~[?:?]
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346) ~[?:?]
at java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:777) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) [?:?]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) [?:?]
at org.apache.pulsar.client.impl.ClientCnx.handleError(ClientCnx.java:715) [org.apache.pulsar-pulsar-client-original-2.9.2.jar:2.9.2]
at org.apache.pulsar.common.protocol.PulsarDecoder.channelRead(PulsarDecoder.java:177) [org.apache.pulsar-pulsar-common-2.9.2.jar:2.9.2]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327) [io.netty-netty-codec-4.1.74.Final.jar:4.1.74.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299) [io.netty-netty-codec-4.1.74.Final.jar:4.1.74.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795) [io.netty-netty-transport-classes-epoll-4.1.74.Final.jar:4.1.74.Final]
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) [io.netty-netty-transport-classes-epoll-4.1.74.Final.jar:4.1.74.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) [io.netty-netty-transport-classes-epoll-4.1.74.Final.jar:4.1.74.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: org.apache.pulsar.client.api.PulsarClientException$IncompatibleSchemaException: {"errorMsg":"Schema not found and schema auto updating is disabled.","reqId":4307185312891752491, "remote":"dev01-neuron-pulsar-broker-0.dev01-neuron-pulsar-broker.dev01-neuron-pulsar.svc.cluster.local/10.130.2.163:6650", "local":"/10.130.2.163:56456"}
at org.apache.pulsar.client.impl.ClientCnx.getPulsarClientException(ClientCnx.java:1170) ~[org.apache.pulsar-pulsar-client-original-2.9.2.jar:2.9.2]
at org.apache.pulsar.client.impl.ClientCnx.handleError(ClientCnx.java:716) ~[org.apache.pulsar-pulsar-client-original-2.9.2.jar:2.9.2]
... 20 more
```
And to me it suggest that the reader that the broker creates on this internal `__change_events` topic can't work in a namespace with `schemaValidationEnforced` enabled since it doesn't specify any schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] truong-hua commented on issue #15891: "Topic policies cache have not init" randomly happen on different topics in the same namespace
Posted by GitBox <gi...@apache.org>.
truong-hua commented on issue #15891:
URL: https://github.com/apache/pulsar/issues/15891#issuecomment-1144657607
> Could you provide some broker logs when this error happens?
Dear @Technoboy- ,
The broker logs has no more information than the exception
```
2022-06-02T09:36:44,454+0000 [pulsar-2-2] ERROR org.apache.pulsar.broker.admin.impl.PersistentTopicsBase - [null] Failed to perform setSubscriptionDispatchRate on topic persistent://public/ticdc/inbox_message
org.apache.pulsar.broker.service.BrokerServiceException$TopicPoliciesCacheNotInitException: Topic policies cache have not init.
at org.apache.pulsar.broker.service.SystemTopicBasedTopicPoliciesService.getTopicPolicies(SystemTopicBasedTopicPoliciesService.java:207) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0]
at org.apache.pulsar.broker.service.TopicPoliciesService.lambda$getTopicPoliciesAsyncWithRetry$0(TopicPoliciesService.java:100) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0]
at org.apache.pulsar.client.util.RetryUtil.executeWithRetry(RetryUtil.java:48) ~[org.apache.pulsar-pulsar-client-original-2.10.0.jar:2.10.0]
at org.apache.pulsar.client.util.RetryUtil.lambda$null$1(RetryUtil.java:57) ~[org.apache.pulsar-pulsar-client-original-2.10.0.jar:2.10.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]
2022-06-02T09:36:44,457+0000 [pulsar-2-2] INFO org.eclipse.jetty.server.RequestLog - 10.0.0.134 - - [02/Jun/2022:09:36:29 +0000] "POST /admin/v2/persistent/public/ticdc/inbox_message/subscriptionDispatchRate?authoritative=false HTTP/1.1" 500 1277 "-" "curl/7.68.0" 14901
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- commented on issue #15891: "Topic policies cache have not init" randomly happen on different topics in the same namespace
Posted by GitBox <gi...@apache.org>.
Technoboy- commented on issue #15891:
URL: https://github.com/apache/pulsar/issues/15891#issuecomment-1146544285
>
> > Could you provide some broker logs when this error happens?
>
> Dear @Technoboy- ,
>
> The broker logs has no more information than the exception
>
> ```
> 2022-06-02T09:36:44,454+0000 [pulsar-2-2] ERROR org.apache.pulsar.broker.admin.impl.PersistentTopicsBase - [null] Failed to perform setSubscriptionDispatchRate on topic persistent://public/ticdc/inbox_message
>
> org.apache.pulsar.broker.service.BrokerServiceException$TopicPoliciesCacheNotInitException: Topic policies cache have not init.
>
> at org.apache.pulsar.broker.service.SystemTopicBasedTopicPoliciesService.getTopicPolicies(SystemTopicBasedTopicPoliciesService.java:207) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0]
>
> at org.apache.pulsar.broker.service.TopicPoliciesService.lambda$getTopicPoliciesAsyncWithRetry$0(TopicPoliciesService.java:100) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0]
>
> at org.apache.pulsar.client.util.RetryUtil.executeWithRetry(RetryUtil.java:48) ~[org.apache.pulsar-pulsar-client-original-2.10.0.jar:2.10.0]
>
> at org.apache.pulsar.client.util.RetryUtil.lambda$null$1(RetryUtil.java:57) ~[org.apache.pulsar-pulsar-client-original-2.10.0.jar:2.10.0]
>
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
>
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
>
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
>
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
>
> at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
>
> at java.lang.Thread.run(Thread.java:829) [?:?]
>
> 2022-06-02T09:36:44,457+0000 [pulsar-2-2] INFO org.eclipse.jetty.server.RequestLog - 10.0.0.134 - - [02/Jun/2022:09:36:29 +0000] "POST /admin/v2/persistent/public/ticdc/inbox_message/subscriptionDispatchRate?authoritative=false HTTP/1.1" 500 1277 "-" "curl/7.68.0" 14901
> ```
Can you search for any other error logs in this broker?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- commented on issue #15891: "Topic policies cache have not init" randomly happen on different topics in the same namespace
Posted by GitBox <gi...@apache.org>.
Technoboy- commented on issue #15891:
URL: https://github.com/apache/pulsar/issues/15891#issuecomment-1144620256
Could you provide some broker logs when this error happens?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] truong-hua commented on issue #15891: "Topic policies cache have not init" randomly happen on different topics in the same namespace
Posted by GitBox <gi...@apache.org>.
truong-hua commented on issue #15891:
URL: https://github.com/apache/pulsar/issues/15891#issuecomment-1178544721
@dragonls thank for that, is it safe to manually update the znode. Do we have any document about metadata stored in the zk?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dragonls commented on issue #15891: "Topic policies cache have not init" randomly happen on different topics in the same namespace
Posted by GitBox <gi...@apache.org>.
dragonls commented on issue #15891:
URL: https://github.com/apache/pulsar/issues/15891#issuecomment-1194904884
Yes, someone may have teminated the topic via pulsar admin.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] diegosalvi commented on issue #15891: "Topic policies cache have not init" randomly happen on different topics in the same namespace
Posted by GitBox <gi...@apache.org>.
diegosalvi commented on issue #15891:
URL: https://github.com/apache/pulsar/issues/15891#issuecomment-1201080250
I've the same issue in a production environment. It happened just after Pulsar upgrade from 2.6.3 to 2.9.2.
Strange enough I didn't see it happen on staging environment with the same configuration. Real difference between environments is traffic volume: streams much more used in production, some streams on staging aren't really used for long time...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org