You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by GitBox <gi...@apache.org> on 2022/08/28 02:41:15 UTC
[GitHub] [rocketmq] shengminw opened a new issue, #4909: Fix CONSUME_FROM_LAST_OFFSET mode may pull data from 0L
shengminw opened a new issue, #4909:
URL: https://github.com/apache/rocketmq/issues/4909
## Problem View
When we set **CONSUME_FROM_LAST_OFFSET** to pull data, if it is the first time to start the consumer group, data may be pulled from the 0L. This is not what we expected.
I built a demo to reproduce this situation.
First sent the data using the producer：
Set consumer mode as **CONSUME_FROM_LAST_OFFSET**.
At the first startup, the consumption situation obtained should theoretically not pull the previous message, but the message is still pulled from the beginning:
## Location the problem
When a consumer group needs to pull messages, it obtains the initial location to be consumed according to the startup mode.
In cluster mode, the offset is read from the broker side. (cluster mode has the above problem)
In broadcast mode, the offset of the local record is read, and -1 is returned if it is not read.
In broker, in **ConsumerManageProcessor#queryConsumerOffset**, the broker will query the consumeoffset by consumegroup, topic and queueId.
● If the offsetTable records the consumption information of this consumer group under this topic and this queueId, return this recorded consumeoffset
● If there is no record in the offsetTable, the minimum offset in the consumequeue will be obtained. (When the minimum offset is less than or equal to 0 && offset 0 is in memory, it returns to 0) -> This leads that when consumequeue does not clean up expired files, the minimum offset is 0 and it is in memory , 0 will be returned, and the client receives an offset of 0, so it starts to consume from the starting position.
## Solution
When querying consumeoffset, set ZeroIfNotFound false in the requestHeader.
So, when the previous record is not obtained, it will no longer return 0.
## Another Question
The above solution can solve this problem, but I still have some doubts about the original implementation method
In **ConsumerManageProcessor#queryConsumerOffset**，why we need to set offset as 0L, when minOffset <= 0 and offset 0L is in memory. Can we return response just setCode as ResponseCode.QUERY_NOT_FOUND?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] RongtongJin closed issue #4909: Fix CONSUME_FROM_LAST_OFFSET mode may pull data from 0L
Posted by GitBox <gi...@apache.org>.
RongtongJin closed issue #4909: Fix CONSUME_FROM_LAST_OFFSET mode may pull data from 0L
URL: https://github.com/apache/rocketmq/issues/4909
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] zhanghaou commented on issue #4909: Fix CONSUME_FROM_LAST_OFFSET mode may pull data from 0L
Posted by "zhanghaou (via GitHub)" <gi...@apache.org>.
zhanghaou commented on issue #4909:
URL: https://github.com/apache/rocketmq/issues/4909#issuecomment-1436263645
>
Does it related to topic auto create?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Fix CONSUME_FROM_LAST_OFFSET mode may pull data from 0L [rocketmq]
Posted by "qixiaobo (via GitHub)" <gi...@apache.org>.
qixiaobo commented on issue #4909:
URL: https://github.com/apache/rocketmq/issues/4909#issuecomment-3128361551
参考 https://www.zhihu.com/tardis/zm/art/163248992?source_id=1003
所以总结来说就是：对于很多集群，一个topic不呆个几天，很可能都被broker误认为是扩容的topic。这就是为什么很多人在测试环境测试总是发现CONSUME_FROM_LAST_OFFSET不生效，事实上CONSUME_FROM_TIMESTAMP也不生效，而最后感觉都是CONSUME_FROM_FIRST_OFFSET的策略，实际上并不是CONSUME_FROM_FIRST_OFFSET生效了，而是broker强行建议消费者从头开始消费了。
上面这个bug条件可能还是有点绕，而且具备非常大的不确定性！的确比较坑。
以下总结几个常见的CONSUME_FROM_LAST_OFFSET不生效bug场景：
1.刚新建不久的一个topic，里面有一些消息曾经发送过，起一个新的consumer group去消费，CONSUME_FROM_LAST_OFFSET可能不生效
2.某些测试环境，或某些累计消息量很小的灰度集群，新起的consumer group 去监听任意一个有消息的topic，CONSUME_FROM_LAST_OFFSET可能不生效。
3.新建一个集群，topic从老集群做了迁移，数据量即使很大，也会认为和#1场景一样。CONSUME_FROM_LAST_OFFSET可能不生效。
虽然两个条件的不确定性很大，但是对于第一个条件是否minOffset <= 0 ，我们是可以通过控制台/命令行去确认的，如下图是我一套测试环境的topic，都一个月了，minOffset其实还是0，证明消息量并不够，一直没被清理，那么针对这个topic，很可能我起一个新消费者也是要消费历史消息的。之所以说很可能，是因为还得看这个消息还在不在broker的page cache，这就很“随缘“了。
祁晓波:解决建议
那针对这些场景我们的确需要跳过，那怎么办？https://zhuanlan.zhihu.com/p/26119361 一文中给过一些建议，简单说就是
代码中进行时间判断做跳过
消费者组启动消费前前先重置一下消费进度，建议这种情况下手动创建消费者组，并把consumeEnable关闭让他不能消费，再调整进度，再重新开启consumeEnable
结语：
做个总结：
1. 对于已经存在的消费者组+topic+queue的订阅关系，无论如何都是遵循历史进度进行消费
2. 对于新的消费者组+topic+queue关系，在正常情况下，遵循客户端配置的策略
3. 对于特殊的场景被broker认为queue是新queue的情况下，一律从头开始消费（令可杀错不放过）
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] zhanghaou commented on issue #4909: Fix CONSUME_FROM_LAST_OFFSET mode may pull data from 0L
Posted by "zhanghaou (via GitHub)" <gi...@apache.org>.
zhanghaou commented on issue #4909:
URL: https://github.com/apache/rocketmq/issues/4909#issuecomment-1432340710
@duhenglucky @RongtongJin I'm also confused about this question. Can you explain it?
---------------------------------------------------------------------------------------
Another Question
The above solution can solve this problem, but I still have some doubts about the original implementation method
In ConsumerManageProcessor#queryConsumerOffset，why we need to set offset as 0L, when minOffset <= 0 and offset 0L is in memory. Can we return response just setCode as ResponseCode.QUERY_NOT_FOUND?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Fix CONSUME_FROM_LAST_OFFSET mode may pull data from 0L [rocketmq]
Posted by "qixiaobo (via GitHub)" <gi...@apache.org>.
qixiaobo commented on issue #4909:
URL: https://github.com/apache/rocketmq/issues/4909#issuecomment-3128357132
The same problem we met
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org