You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@rocketmq.apache.org by "humkum (via GitHub)" <gi...@apache.org> on 2024/12/26 12:18:55 UTC
[I] [Bug] The consumeFromWhere configuration didn't work in some cases. [rocketmq]
humkum opened a new issue, #9083:
URL: https://github.com/apache/rocketmq/issues/9083
### Before Creating the Bug Report
- [X] I found a bug, not just asking a question, which should be created in [GitHub Discussions](https://github.com/apache/rocketmq/discussions).
- [X] I have searched the [GitHub Issues](https://github.com/apache/rocketmq/issues) and [GitHub Discussions](https://github.com/apache/rocketmq/discussions) of this repository and believe that this is not a duplicate.
- [X] I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ.
### Runtime platform environment
Ubuntu 18.04
### RocketMQ version
5.3
### JDK Version
Open_JDK 1.8.0_202
### Describe the Bug
PushConsumer and LitePullConsumer's consumeFromWhere configuration didn't work in follow situation:
1. New created topic
2. New created consumerGroup
3. There are some messages in topic's queue, which are not expired
4. The value of (maxOffset - 0) not over the size the broker thought is in memory.
### Steps to Reproduce
1. Create a new Topic
2. Send some messages to the topic
3. Create a new Push/LitePull Consumer, and start to consume message from the topic created, with the consumeFromWhere default configured as ConsumeFromWhere.CONSUME_FROM_LAST_OFFSET
4. We would find that the consumer didn't consume from the last offset, it would consume from the earliest offset 0.
### What Did You Expect to See?
The consumer consume from the last offset or consume from the timestamp user configured.
### What Did You See Instead?
The consumer consume from the earliest offset 0.
### Additional Context
Root cause as follows:
When new created consumer first consume from a topic, it would compute the offset from where to pull message. Because the brokers don't have it's consume offset, so the normal offset result should be computed as the consumeFromWhere's value, ConsumeFromWhere.CONSUME_FROM_LAST_OFFSET, ConsumeFromWhere.CONSUME_FROM_EARLIEST_OFFSET,ConsumeFromWhere.CONSUME_FROM_TIMSTAMP, etc.
In computePullFromWhereWithException function: the consumer would query offset from the brokers to check if the brokers had the consumer offset. If not, it would set the pull offset to min/max offset or the offset corresponding to the timestamp configured, as follows:
But in broker side, it the consumer start pull message for the first time, that there's no consume offset for it. If the min offset equals to 0, and the value of (max offset - min offset) is considered as in memory, the read offset would returned as 0. So the consumer would consume from 0, even it set consumeFromWhere as from last offset or from timestamp. As follows:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] The consumeFromWhere configuration didn't work in some cases. [rocketmq]
Posted by "humkum (via GitHub)" <gi...@apache.org>.
humkum commented on issue #9083:
URL: https://github.com/apache/rocketmq/issues/9083#issuecomment-2563417202
> This is a protection measure. When a broker scales out, the registered topic route might become visible to Producers first, causing some messages to be sent. If the Consumer starts consuming from the latest offset at this time, these messages would be skipped.
I get it, thanks a lot!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] The consumeFromWhere configuration didn't work in some cases. [rocketmq]
Posted by "humkum (via GitHub)" <gi...@apache.org>.
humkum closed issue #9083: [Bug] The consumeFromWhere configuration didn't work in some cases.
URL: https://github.com/apache/rocketmq/issues/9083
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] The consumeFromWhere configuration didn't work in some cases. [rocketmq]
Posted by "qianye1001 (via GitHub)" <gi...@apache.org>.
qianye1001 commented on issue #9083:
URL: https://github.com/apache/rocketmq/issues/9083#issuecomment-2563404677
This is a protection measure. When a broker scales out, the registered topic route might become visible to Producers first, causing some messages to be sent. If the Consumer starts consuming from the latest offset at this time, these messages would be skipped.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org