[GitHub] [pulsar] addisonj commented on issue #14082: Pulsar messages present in backlog, but not getting dispatched to Consumers

[GitBox <gi...@apache.org>]

addisonj commented on issue #14082:
URL: https://github.com/apache/pulsar/issues/14082#issuecomment-1029181907


   @anant-ahuja This appears to be a problem with your consumers not acknowledging all messages.
   
   To explain, if you look at the `unackedMessages` and `availablePermits` together indicate this problem. To explain, see this selection from the topic stats:
   
   ```
    {
    ...
           "availablePermits" : 0,
           "unackedMessages" : 3,
   ...
           "connectedSince" : "2022-02-03T00:11:00.17Z",
           "address" : "/10.32.8.9:59458"
         }
   ```
   
   unackedMessages indicates that Pulsar has dispatched messages to this consumer, but the consumer has not acknowledged them. unackedMessages themselves are not an issue, but, the avaiablePermits is zero which indicates to Pulsar that it cannot send any more messages.
   
   The way Pulsar works is that clients dictate when messages can be sent. The client does this by sending the broker a "flow" command to tell the broker it can send up to some number of messages (refereed to as permits). When a client library processes messages and is ready for more, it sends another flow command, which gives the broker more permits to send more messages.
   
   You never have to worry about sending flow messges yourself, instead, the pulsar client library does it automatically based on when you acknowledge messages. However, if your application fails to acknowledge too many messages, this can cause issues with the client not sending any more flow commands.
   
   As far as how to fix this, here are a few options:
   
   1. Find the code paths that are resulting in unprocessed messages (usually related to error handling or deadlocks in process) and fix those by adding error handling or a timeout of process, then either nack the messages or just ack the message (depending on your processing semantics)
   2. Use Pulsar consumer "ackTimeout" feature which will send a negative acknowledgement (causing redelivery) if processing doesn't happen within some period of time
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org