[GitHub] [pulsar] f-ld opened a new issue #4718: failed to get Partitioned metadata : Policies not found for mytenant/mynamespace namespace

[GitBox <gi...@apache.org>]

f-ld opened a new issue #4718: failed to get Partitioned metadata : Policies not found for mytenant/mynamespace namespace
URL: https://github.com/apache/pulsar/issues/4718
 
 
   **Describe the bug**
   We have 4 pulsar cluster running on kubernetes with geo-replication.
   All worked well. Before going to prod, we did some load testing. And during one of them (last one at time of writing), pulsar brokers started to loop on exceptions like:
   ```
   13:30:24.053 [pulsar-io-21-3] WARN  org.apache.pulsar.client.impl.BinaryProtoLookupService - [non-persistent://mytenant/mynamespace/mytopic-98f088294f0c7a509ab4b1a5412b79308a75d50e] failed to get Partitioned metadata : Policies not found for session/user_events namespace
   java.util.concurrent.CompletionException: org.apache.pulsar.client.api.PulsarClientException$BrokerMetadataException: Policies not found for mytenant/mynamespace
   ```
   Note : we have a lot of non-persistent / not replicated topics in that namespace (23k). Too many?
   
   **To Reproduce**
   Well, difficult. We got this "by chance" during a load test and I cannot find out what caused it
   
   During my investigation so far I tried many things, including:
   ```
   pulsar-admin namespaces policies mytenant/mynamespace
   ```
   that returns information without any problem. But I really want to understand what happened and not only restart the cluster and start from fresh.
   
   Everything looks fine (according to logs):
   - zookeeper
   - global zookeeper (used for geo replication)
   - bookkeeper (I checked even if we do not persist those topics)
   
   Only brokers in all regions do log a lot (well, maybe not that much per topic, but with so many topics it is quite a lot).
   
   Any clue on how to investigate / fix it (in case it happens in prod once we do that move) ?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services