You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "hanmz (via GitHub)" <gi...@apache.org> on 2024/07/14 12:21:25 UTC

[I] [Bug] [broker]Sometime after reset cursor, unable to reconsume all messages in topic when ttl is less than retention time. [pulsar]

hanmz opened a new issue, #23030:
URL: https://github.com/apache/pulsar/issues/23030

   ### Search before asking
   
   - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
   
   
   ### Read release policy
   
   - [X] I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.
   
   
   ### Version
   
   ALL
   
   ### Minimal reproduce step
   
   1. Create a namespaces, set ttl time to be less than retention time, for example ttl=60s, retention time=60min.
   2. Continuously send a large number of messages to the topic.
   3. Create a new subscription and start consumption from the latest location. At this time, it can be consumed in time without any backlog.
   4.  We adjust the dispatch rate to a smaller threshold(For avoid consuming all backlog messages too quickly). Then we reset cursor to the earliest location.
   5. We found that consumption will start from the earliest position, but after a period of time(in messageExpiryCheckIntervalInMinutes), the intermediate messages will be skipped and go directly to the ttl time position.
   
   ### What did you expect to see?
   
    Reconsume all messages from reset position in topic.
   
   ### What did you see instead?
   
   Some messages before ttl cannot be consumed (skip directly)
   
   ### Anything else?
   
   The reason for this problem is that ttl checks that the consumption position is before ttl, and resets the consume read position to the position of ttl.
   
   ### Are you willing to submit a PR?
   
   - [ ] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [broker]Sometime after reset cursor, unable to reconsume all messages in topic when ttl is less than retention time. [pulsar]

Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat closed issue #23030: [Bug] [broker]Sometime after reset cursor, unable to reconsume all messages in topic when ttl is less than retention time.
URL: https://github.com/apache/pulsar/issues/23030


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [broker]Sometime after reset cursor, unable to reconsume all messages in topic when ttl is less than retention time. [pulsar]

Posted by "hanmz (via GitHub)" <gi...@apache.org>.
hanmz commented on issue #23030:
URL: https://github.com/apache/pulsar/issues/23030#issuecomment-2242249062

   > it seems this is expected behavior
   
   This is normal behavior, but it can easily cause misunderstandings


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [broker]Sometime after reset cursor, unable to reconsume all messages in topic when ttl is less than retention time. [pulsar]

Posted by "dao-jun (via GitHub)" <gi...@apache.org>.
dao-jun commented on issue #23030:
URL: https://github.com/apache/pulsar/issues/23030#issuecomment-2228803824

   it seems this is expected behavior


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [broker]Sometime after reset cursor, unable to reconsume all messages in topic when ttl is less than retention time. [pulsar]

Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on issue #23030:
URL: https://github.com/apache/pulsar/issues/23030#issuecomment-6380994995

   Thanks @hanmz for the report. As discussed in the thread, this is how message TTL works: on each expiry check, `PersistentMessageExpiryMonitor` acknowledges every message on the subscription that is older than the TTL, wherever the cursor was reset to. Retention only keeps the data available for a new reset. To replay messages older than the TTL, remove or raise the TTL on the namespace/topic, or replay them before the next expiry check. Closing as working as designed. Please reopen if you see messages newer than the TTL being skipped on a currently supported release.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org