You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by GitBox <gi...@apache.org> on 2022/08/28 02:41:15 UTC

[GitHub] [rocketmq] shengminw opened a new issue, #4909: Fix CONSUME_FROM_LAST_OFFSET mode may pull data from 0L

shengminw opened a new issue, #4909:
URL: https://github.com/apache/rocketmq/issues/4909

   ## Problem View
   When we set **CONSUME_FROM_LAST_OFFSET** to pull data, if it is the first time to start the consumer group, data may be pulled from the 0L. This is not what we expected.
   I built a demo to reproduce this situation.
   First sent the data using the producer：
   ![image](https://user-images.githubusercontent.com/107378995/187054813-dfd00802-bb5c-4d17-b2ea-aea2a563cda1.png)
   
   Set consumer mode as **CONSUME_FROM_LAST_OFFSET**.
   ![image](https://user-images.githubusercontent.com/107378995/187054823-193f540e-8669-474b-a520-2d057400fe0d.png)
   
   At the first startup, the consumption situation obtained should theoretically not pull the previous message, but the message is still pulled from the beginning:
   ![image](https://user-images.githubusercontent.com/107378995/187054840-bba0f765-22e0-400a-986d-1d5eb5e233ed.png)
   
   
   ## Location the problem
   When a consumer group needs to pull messages, it obtains the initial location to be consumed according to the startup mode.
   ![image](https://user-images.githubusercontent.com/107378995/187054864-7c409a2e-a9ae-43a1-b3b6-bf5f39e687fd.png)
   
   In cluster mode, the offset is read from the broker side. (cluster mode has the above problem)
   
   In broadcast mode, the offset of the local record is read, and -1 is returned if it is not read.
   
   In broker, in  **ConsumerManageProcessor#queryConsumerOffset**, the broker will query the consumeoffset by consumegroup, topic and queueId.
   
   ● If the offsetTable records the consumption information of this consumer group under this topic and this queueId, return this recorded consumeoffset
   ● If there is no record in the offsetTable, the minimum offset in the consumequeue will be obtained. (When the minimum offset is less than or equal to 0 && offset 0 is in memory, it returns to 0) -> This leads that when consumequeue does not clean up expired files, the minimum offset is 0 and it is in memory , 0 will be returned, and the client receives an offset of 0, so it starts to consume from the starting position.
   ![image](https://user-images.githubusercontent.com/107378995/187054882-ed370b1b-1263-483f-af62-60fe0013e9b2.png)
   
   
   ## Solution
   When querying consumeoffset, set ZeroIfNotFound false in the requestHeader. 
   So, when the previous record is not obtained, it will no longer return 0.
   ![image](https://user-images.githubusercontent.com/107378995/187054887-01f72caa-7b8b-4870-a698-1cc0b90bf223.png)
   
   
   ## Another Question
   
   The above solution can solve this problem, but I still have some doubts about the original implementation method
   
   In  **ConsumerManageProcessor#queryConsumerOffset**，why we need to set offset as 0L, when minOffset <= 0 and offset 0L is in memory. Can we return response just setCode as ResponseCode.QUERY_NOT_FOUND?
   
   ![image](https://user-images.githubusercontent.com/107378995/187054939-96d3e099-be62-401c-ac47-a8819dfa817e.png)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] RongtongJin closed issue #4909: Fix CONSUME_FROM_LAST_OFFSET mode may pull data from 0L

Posted by GitBox <gi...@apache.org>.
RongtongJin closed issue #4909: Fix CONSUME_FROM_LAST_OFFSET mode may pull data from 0L
URL: https://github.com/apache/rocketmq/issues/4909


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] zhanghaou commented on issue #4909: Fix CONSUME_FROM_LAST_OFFSET mode may pull data from 0L

Posted by "zhanghaou (via GitHub)" <gi...@apache.org>.
zhanghaou commented on issue #4909:
URL: https://github.com/apache/rocketmq/issues/4909#issuecomment-1436263645

   > 
   
   Does it related to topic auto create? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] zhanghaou commented on issue #4909: Fix CONSUME_FROM_LAST_OFFSET mode may pull data from 0L

Posted by "zhanghaou (via GitHub)" <gi...@apache.org>.
zhanghaou commented on issue #4909:
URL: https://github.com/apache/rocketmq/issues/4909#issuecomment-1432340710

   @duhenglucky @RongtongJin I'm also confused about this question.  Can you explain it?
   
   ---------------------------------------------------------------------------------------
   
   Another Question
   The above solution can solve this problem, but I still have some doubts about the original implementation method
   
   In ConsumerManageProcessor#queryConsumerOffset，why we need to set offset as 0L, when minOffset <= 0 and offset 0L is in memory. Can we return response just setCode as ResponseCode.QUERY_NOT_FOUND?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org