You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "KannarFr (via GitHub)" <gi...@apache.org> on 2024/04/04 18:02:12 UTC
[I] [Bug] Consumer Stops Receiving Messages with Large Backlogs Post-Processing in v3.2.2: [pulsar]
KannarFr opened a new issue, #22435:
URL: https://github.com/apache/pulsar/issues/22435
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Read release policy
- [X] I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.
### Version
3.2.2
### Minimal reproduce step
Open consumer on a subscription using very large backlog by passing "old" MessageId.
### What did you expect to see?
Consume all messages and wait for upcoming messages when the backlog is consumed.
### What did you see instead?
Initially, everything functions correctly—the process begins with a backlog of (let's say) 100,000 messages, which gradually decreases to 0 as we approach the present ("now"). However, once the backlog is fully processed, the consumer unexpectedly stops receiving new messages, leading to an increase in the backlog again. The consumer is still connected. This behavior is consistently reproducible with my topics that have a substantial amount of data.
I've also noticed that when the starting point (since) is relatively close to the current time (now), this problem does not occur.
### Anything else?
We do not know if this bug was introduced by v3.2.2. We didn't see it before. We are currently rollbacking brokers to 3.2.1 to confirm this.
Otherwise, it may be related to https://github.com/apache/pulsar/pull/22191.
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Consumer Stops Receiving Messages with Large Backlogs Post-Processing [pulsar]
Posted by "KannarFr (via GitHub)" <gi...@apache.org>.
KannarFr commented on issue #22435:
URL: https://github.com/apache/pulsar/issues/22435#issuecomment-2039865545
I confirm that I can reproduce the issue using v3.2.1 brokers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Consumer Stops Receiving Messages with Large Backlogs Post-Processing [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #22435:
URL: https://github.com/apache/pulsar/issues/22435#issuecomment-2040991765
Trying to understand the logic of #22191 and the changes regarding "backlogged" cursors before that, PRs #19343, #6766, #4066, #162 . The change made in #4066 to inactive cursors in the checkBackloggedCursor method seems suspicious, and might be the reason why a hack such as #9789 was needed.
In any case, it seems that #22191 should be reverted as the first step. However, the proper fix seems to be to sort out various issues in this area.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Consumer Stops Receiving Messages with Large Backlogs Post-Processing [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #22435:
URL: https://github.com/apache/pulsar/issues/22435#issuecomment-2041819604
> If revert, the OOM issue will be there.
Great work on the fix #22454, @Technoboy- . It looks like reverting #22191 changes to `addWaitingCursor` is something that should be done there since changes to `addWaitingCursor` weren't relevant after all. The other changes made in #22454 will fix the OOM that #22191 attempted to fix. That's great!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Consumer Stops Receiving Messages with Large Backlogs Post-Processing [pulsar]
Posted by "KannarFr (via GitHub)" <gi...@apache.org>.
KannarFr commented on issue #22435:
URL: https://github.com/apache/pulsar/issues/22435#issuecomment-2038009404
I confirm bumping the client to v3.2.2 doesn't fix the issue.
I also confirm using a `Durable` consumer does not produce the issue. So it might be related to https://github.com/apache/pulsar/pull/22191. I'll rollback to v3.2.1 tomorrow to confirm.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Consumer Stops Receiving Messages with Large Backlogs Post-Processing [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #22435:
URL: https://github.com/apache/pulsar/issues/22435#issuecomment-2040994799
One of the problems is a possible race condition here:
https://github.com/apache/pulsar/blob/902728ef6590233b87c14d2528590ad7e6fdcc12/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java#L977-L979
In ManagedLedger, tasks are executed on 2 threads: the executor thread and the scheduler thread.
Reminds me of this old comment on an experimental PR: https://github.com/apache/pulsar/pull/11387/files#r693112234
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Consumer Stops Receiving Messages with Large Backlogs Post-Processing [pulsar]
Posted by "Technoboy- (via GitHub)" <gi...@apache.org>.
Technoboy- closed issue #22435: [Bug] Consumer Stops Receiving Messages with Large Backlogs Post-Processing
URL: https://github.com/apache/pulsar/issues/22435
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Consumer Stops Receiving Messages with Large Backlogs Post-Processing [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #22435:
URL: https://github.com/apache/pulsar/issues/22435#issuecomment-2041077698
One of the problems is a possible race condition here:
https://github.com/apache/pulsar/blob/902728ef6590233b87c14d2528590ad7e6fdcc12/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java#L977-L979
In ManagedLedger, tasks are executed on 2 threads: the executor thread and the scheduler thread.
Reminds me of this old comment on an experimental PR: https://github.com/apache/pulsar/pull/11387/files#r693112234
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Consumer Stops Receiving Messages with Large Backlogs Post-Processing [pulsar]
Posted by "Technoboy- (via GitHub)" <gi...@apache.org>.
Technoboy- commented on issue #22435:
URL: https://github.com/apache/pulsar/issues/22435#issuecomment-2041461730
> In any case, it seems that https://github.com/apache/pulsar/pull/22191 should be reverted as the first step
If revert, the OOM issue will be there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org