[GitHub] [pulsar] cdbartholomew opened a new issue #5579: [broker] Every time PersistentMessageExpiryMonitor runs, it deletes a non-expired message when using TTL

[GitBox <gi...@apache.org>]

cdbartholomew opened a new issue #5579: [broker] Every time PersistentMessageExpiryMonitor runs, it deletes a non-expired message when using TTL
URL: https://github.com/apache/pulsar/issues/5579
 
 
   **Describe the bug**
   
   I have a topic in a namespace with messageTtl set to 24 hours. The topic has a subscription without a connected consumer. I publish a handful of messages to the topic.  Within 5 minutes the oldest message in the topic is deleted. Every 5 minutes after that, one more message gets deleted from the topic even though none have expired yet.
   
   I configured the expiry monitor to run every minute. I published 4 messages into the topic. You can see from these logs, that every minute it is expiring one message:
   
   ```21:55:39.997 [pulsar-msg-expiry-monitor-25-1] INFO  org.apache.pulsar.broker.service.persistent.PersistentMessageExpiryMonitor - [persistent://support-kafkaesque-io/local-useast2-aws/tc1-messages][tc2-consume] Starting message expiry check, ttl= 172800 seconds
   21:55:40.017 [bookkeeper-ml-workers-OrderedExecutor-6-0] INFO  org.apache.pulsar.broker.service.persistent.PersistentMessageExpiryMonitor - [persistent://support-kafkaesque-io/local-useast2-aws/tc1-messages][tc2-consume] Expiring all messages until position 292:15
   21:56:39.995 [pulsar-msg-expiry-monitor-25-1] INFO  org.apache.pulsar.broker.service.persistent.PersistentMessageExpiryMonitor - [persistent://support-kafkaesque-io/local-useast2-aws/tc1-messages][tc2-consume] Starting message expiry check, ttl= 172800 seconds
   21:56:39.999 [bookkeeper-ml-workers-OrderedExecutor-6-0] INFO  org.apache.pulsar.broker.service.persistent.PersistentMessageExpiryMonitor - [persistent://support-kafkaesque-io/local-useast2-aws/tc1-messages][tc2-consume] Expiring all messages until position 292:16
   21:57:39.995 [pulsar-msg-expiry-monitor-25-1] INFO  org.apache.pulsar.broker.service.persistent.PersistentMessageExpiryMonitor - [persistent://support-kafkaesque-io/local-useast2-aws/tc1-messages][tc2-consume] Starting message expiry check, ttl= 172800 seconds
   21:57:39.997 [bookkeeper-ml-workers-OrderedExecutor-6-0] INFO  org.apache.pulsar.broker.service.persistent.PersistentMessageExpiryMonitor - [persistent://support-kafkaesque-io/local-useast2-aws/tc1-messages][tc2-consume] Expiring all messages until position 292:17
   21:58:39.995 [pulsar-msg-expiry-monitor-25-1] INFO  org.apache.pulsar.broker.service.persistent.PersistentMessageExpiryMonitor - [persistent://support-kafkaesque-io/local-useast2-aws/tc1-messages][tc2-consume] Starting message expiry check, ttl= 172800 seconds
   21:58:39.997 [bookkeeper-ml-workers-OrderedExecutor-6-0] INFO  org.apache.pulsar.broker.service.persistent.PersistentMessageExpiryMonitor - [persistent://support-kafkaesque-io/local-useast2-aws/tc1-messages][tc2-consume] Expiring all messages until position 292:18
   ```
   
   My topic that started with 4 messages is now empty. None of these messages have actually expired so none should have been deleted.
   
   **To Reproduce**
   Steps to reproduce the behavior:
   
   1. Configure namespace with a TTL, for example, 24 hours.
   2. Create a topic in that namespace and add a subscription. Make sure there are no consumers so that messages are in the backlog.
   3. Publish messages to the topic. Check that they are all in the backlog.
   4. Every time the expiry monitor wakes up (5 minutes by default), it deletes one message from the backlog until they are all gone.
   
   **Expected behavior**
   None of the messages should be deleted from the backlog until they actually expire.
   
   **Additional context**
   Looking at the PersistentMessageExpiryMonitor class, it searches for expired messages and is expecting that if no expired messages are found, the search returns `null` . However, the search appears to be returning the oldest position in the backlog when none have expired, not `null`. Since a position is returned, the code expires messages up to that position. 
   
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services