[GitHub] [pulsar] devinbost edited a comment on issue #6054: Catastrophic frequent random topic freezes, especially on high-traffic topics.

[GitBox <gi...@apache.org>]

devinbost edited a comment on issue #6054:
URL: https://github.com/apache/pulsar/issues/6054#issuecomment-817004087


   I've confirmed that permits are positive (and appear healthy) on ALL the functions' ConsumerImpl objects when the _brokers_ reported **negative** permits for the topics in this location:
   
   - `broker.service.Consumer`
   
   Interestingly the `totalAvailablePermits` on `broker.service.persistent.PersistentDispatcherMultipleConsumers` == 0 for each topic.
   
   I was a little surprised that `PersistentDispatcherMultipleConsumers` was used instead of `PersistentDispatcherSingleActiveConsumer` for the topics because each topic only had a single subscription with a single consumer, but heap dumps don't lie. 
   
   We also saw the negative permits on partitioned topics and non-partitioned topics. 
   
   We've also confirmed that after a very large amount of time, the messages will eventually flow through. It's like stop-and-go traffic. A few messages get through and then permits go back to negative, there's a pause, and then a few more messages get through, and the process repeats. So, messages that would typically take seconds to clear the pipe take hours to clear. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org