[GitHub] [pulsar] codelipenghui commented on issue #8975: Message TTL expires 1 message at a time

[GitBox <gi...@apache.org>]

codelipenghui commented on issue #8975:
URL: https://github.com/apache/pulsar/issues/8975#issuecomment-753248674


   @eolivelli Can this problem be reproduced stably? I have tried the following steps:
   
   1. Build branch-2.6
   2. Start a standalone
   3. Start a perf producer and create a subscription
   4. After a while, stop the producer and check the topic stats, there are more than 5000 backlogs for the subscription
   ```
   {
     "msgRateIn" : 0.0,
     "msgThroughputIn" : 0.0,
     "msgRateOut" : 0.0,
     "msgThroughputOut" : 0.0,
     "bytesInCounter" : 13840614971,
     "msgInCounter" : 13360521,
     "bytesOutCounter" : 0,
     "msgOutCounter" : 0,
     "averageMsgSize" : 0.0,
     "msgChunkPublished" : false,
     "storageSize" : 13840614971,
     "backlogSize" : 2450361989,
     "publishers" : [ ],
     "subscriptions" : {
       "test" : {
         "msgRateOut" : 0.0,
         "msgThroughputOut" : 0.0,
         "bytesOutCounter" : 0,
         "msgOutCounter" : 0,
         "msgRateRedeliver" : 0.0,
         "chuckedMessageRate" : 0,
         "msgBacklog" : 4389,
         "msgBacklogNoDelayed" : 4389,
         "blockedSubscriptionOnUnackedMsgs" : false,
         "msgDelayed" : 0,
         "unackedMessages" : 0,
         "type" : "Exclusive",
         "msgRateExpired" : 94.68975601131787,
         "lastExpireTimestamp" : 1609471808999,
         "lastConsumedFlowTimestamp" : 1609471702688,
         "lastConsumedTimestamp" : 0,
         "lastAckedTimestamp" : 0,
         "consumers" : [ ],
         "isDurable" : true,
         "isReplicated" : false,
         "consumersAfterMarkDeletePosition" : { },
         "nonContiguousDeletedMessagesRanges" : 0,
         "nonContiguousDeletedMessagesRangesSerializedSize" : 0
       }
     },
     "replication" : { },
     "deduplicationStatus" : "Disabled",
     "nonContiguousDeletedMessagesRanges" : 0,
     "nonContiguousDeletedMessagesRangesSerializedSize" : 0
   }
   ```
   
   And then I set the TTL for the namespace `bin/pulsar-admin namespaces set-message-ttl -ttl 30 public/default`
   
   Wait for a while and check the logs from the broker:
   
   ```
   11:32:07.826 [pulsar-msg-expiry-monitor-54-1] INFO  org.apache.pulsar.broker.service.persistent.PersistentMessageExpiryMonitor - [persistent://public/default/bob][test] Starting message expiry check, ttl= 30 seconds
   11:32:07.835 [bookkeeper-ml-workers-OrderedExecutor-1-0] INFO  org.apache.pulsar.broker.service.persistent.PersistentMessageExpiryMonitor - [persistent://public/default/bob][test] Expiring all messages until position 10:24790
   ```
   
   This means the message expiring task started. And then check the topic stats again
   
   ```
   {
     "msgRateIn" : 0.0,
     "msgThroughputIn" : 0.0,
     "msgRateOut" : 0.0,
     "msgThroughputOut" : 0.0,
     "bytesInCounter" : 13840614971,
     "msgInCounter" : 13360521,
     "bytesOutCounter" : 0,
     "msgOutCounter" : 0,
     "averageMsgSize" : 0.0,
     "msgChunkPublished" : false,
     "storageSize" : 13840614971,
     "backlogSize" : 0,
     "publishers" : [ ],
     "subscriptions" : {
       "test" : {
         "msgRateOut" : 0.0,
         "msgThroughputOut" : 0.0,
         "bytesOutCounter" : 0,
         "msgOutCounter" : 0,
         "msgRateRedeliver" : 0.0,
         "chuckedMessageRate" : 0,
         "msgBacklog" : 0,
         "msgBacklogNoDelayed" : 0,
         "blockedSubscriptionOnUnackedMsgs" : false,
         "msgDelayed" : 0,
         "unackedMessages" : 0,
         "type" : "Exclusive",
         "msgRateExpired" : 39.47333315318755,
         "lastExpireTimestamp" : 1609471927826,
         "lastConsumedFlowTimestamp" : 1609471702688,
         "lastConsumedTimestamp" : 0,
         "lastAckedTimestamp" : 0,
         "consumers" : [ ],
         "isDurable" : true,
         "isReplicated" : false,
         "consumersAfterMarkDeletePosition" : { },
         "nonContiguousDeletedMessagesRanges" : 0,
         "nonContiguousDeletedMessagesRangesSerializedSize" : 0
       }
     },
     "replication" : { },
     "deduplicationStatus" : "Disabled",
     "nonContiguousDeletedMessagesRanges" : 0,
     "nonContiguousDeletedMessagesRangesSerializedSize" : 0
   }
   ```
   
   Seems it works. Could you please provide more details about the issue? such as topic stats, ttl policy, and the broker logs. This will help us to locate the problem.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org