You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by GitBox <gi...@apache.org> on 2021/10/29 08:36:57 UTC
[GitHub] [rocketmq] cserwen opened a new pull request #3444: [ISSUE #3443] commit the right consumption offset for popReviewService
cserwen opened a new pull request #3444:
URL: https://github.com/apache/rocketmq/pull/3444
**Make sure set the target branch to `develop`**
## What is the purpose of the change
#3443
## Brief changelog
After fixing it.
```
2021-10-29 15:51:14 INFO PopReviveService_0 - start revive topic=rmq_sys_REVIVE_LOG_cluster, reviveQueueId=0
2021-10-29 15:51:14 INFO PopReviveService_0 - reviveQueueId=0, old offset is 142765
2021-10-29 15:51:14 WARN PopReviveService_0 - offset illegal. GetMessageStatus=OFFSET_OVERFLOW_ONE, topic=rmq_sys_REVIVE_LOG_cluster, groupId=CID_RMQ_SYS_REVIVE_GROUP, requestOffset=142766
2021-10-29 15:51:14 INFO PopReviveService_0 - reviveQueueId=0, reach tail,offset 142766
2021-10-29 15:51:14 INFO PopReviveService_0 - reviveQueueId=0,ck listSize=0
2021-10-29 15:51:14 INFO PopReviveService_0 - reviveQueueId=0,revive finish,old offset is 142765, new offset is 142765, ckDelay=0
2021-10-29 15:51:14 INFO PopReviveService_0 - reviveQueueId=0,has no new msg ,take a rest 2
2021-10-29 15:51:17 INFO PopReviveService_0 - start revive topic=rmq_sys_REVIVE_LOG_cluster, reviveQueueId=0
2021-10-29 15:51:17 INFO PopReviveService_0 - reviveQueueId=0, old offset is 142765
2021-10-29 15:51:17 WARN PopReviveService_0 - offset illegal. GetMessageStatus=OFFSET_OVERFLOW_ONE, topic=rmq_sys_REVIVE_LOG_cluster, groupId=CID_RMQ_SYS_REVIVE_GROUP, requestOffset=142766
2021-10-29 15:51:17 INFO PopReviveService_0 - reviveQueueId=0, reach tail,offset 142766
2021-10-29 15:51:17 INFO PopReviveService_0 - reviveQueueId=0,ck listSize=0
2021-10-29 15:51:17 INFO PopReviveService_0 - reviveQueueId=0,revive finish,old offset is 142765, new offset is 142765, ckDelay=0
2021-10-29 15:51:17 INFO PopReviveService_0 - reviveQueueId=0,has no new msg ,take a rest 3
2021-10-29 15:51:21 INFO PopReviveService_0 - start revive topic=rmq_sys_REVIVE_LOG_cluster, reviveQueueId=0
2021-10-29 15:51:21 INFO PopReviveService_0 - reviveQueueId=0, old offset is 142765
2021-10-29 15:51:21 WARN PopReviveService_0 - offset illegal. GetMessageStatus=OFFSET_OVERFLOW_ONE, topic=rmq_sys_REVIVE_LOG_cluster, groupId=CID_RMQ_SYS_REVIVE_GROUP, requestOffset=142766
2021-10-29 15:51:21 INFO PopReviveService_0 - reviveQueueId=0, reach tail,offset 142766
2021-10-29 15:51:21 INFO PopReviveService_0 - reviveQueueId=0,ck listSize=0
2021-10-29 15:51:21 INFO PopReviveService_0 - reviveQueueId=0,revive finish,old offset is 142765, new offset is 142765, ckDelay=0
2021-10-29 15:51:21 INFO PopReviveService_0 - reviveQueueId=0,has no new msg ,take a rest 3
```
## Verifying this change
XXXX
Follow this checklist to help us incorporate your contribution quickly and easily. Notice, `it would be helpful if you could finish the following 5 checklist(the last one is not necessary)before request the community to review your PR`.
- [x] Make sure there is a [Github issue](https://github.com/apache/rocketmq/issues) filed for the change (usually before you start working on it). Trivial changes like typos do not require a Github issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
- [x] Format the pull request title like `[ISSUE #123] Fix UnknownException when host config not exist`. Each commit in the pull request should have a meaningful subject line and body.
- [x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
- [x] Write necessary unit-test(over 80% coverage) to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in [test module](https://github.com/apache/rocketmq/tree/master/test).
- [x] Run `mvn -B clean apache-rat:check findbugs:findbugs checkstyle:checkstyle` to make sure basic checks pass. Run `mvn clean install -DskipITs` to make sure unit-test pass. Run `mvn clean test-compile failsafe:integration-test` to make sure integration-test pass.
- [ ] If this contribution is large, please file an [Apache Individual Contributor License Agreement](http://www.apache.org/licenses/#clas).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] cserwen closed pull request #3444: [ISSUE #3443] commit the right consumption offset for popReviewService
Posted by GitBox <gi...@apache.org>.
cserwen closed pull request #3444:
URL: https://github.com/apache/rocketmq/pull/3444
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] odbozhou commented on pull request #3444: [ISSUE #3443] commit the right consumption offset for popReviewService
Posted by GitBox <gi...@apache.org>.
odbozhou commented on pull request #3444:
URL: https://github.com/apache/rocketmq/pull/3444#issuecomment-957136997
> > In order to prevent the message from being lost. Revive topic queue Offset update needs to pass ck msg. Revive topic contains both ck msg and ack msg. If the offset is updated directly by the number of Revive topic queue messages, there is a possibility of message loss. For example, some ck msgs are not confirmed by ack msg. , And without sending the retry topic, the offset of the receive topic is updated directly. There is a possibility of repetition of ack msg due to the network or other reasons. The message corresponding to ack has been acked, and ack will be consumed repeatedly, but it has no effect. These ack msg will not be consumed repeatedly when the new ck msg is updated. NS
>
> @odbozhou Thanks for the explanation, but I have two questions:
>
> * The offset was updated after putting messages into retry topic. Is it possible to fail to put msgs?
> * Even if the above example happens, updating the offset by the ck msgs and updating it by the number of Revive topic queue messages are the same, because the ck msgs will be consumed only once. So I think updating the consume offset by the number of msgs directly will be better.
- The ck message is not necessarily consumed only once. As shown in the figure, if break at this step, some ck has not been confirmed, and no retry topic has been sent. However, the consumption location of the receive topic queue is updated by the number of msgs. The unacknowledged ck message may lose the message, which does not meet the semantics of at least once.
- The msg in the receive topic has both ck and ack。
and it cannot only mean the number of ck.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] cserwen commented on pull request #3444: [ISSUE #3443] commit the right consumption offset for popReviewService
Posted by GitBox <gi...@apache.org>.
cserwen commented on pull request #3444:
URL: https://github.com/apache/rocketmq/pull/3444#issuecomment-956069134
> In order to prevent the message from being lost. Revive topic queue Offset update needs to pass ck msg. Revive topic contains both ck msg and ack msg. If the offset is updated directly by the number of Revive topic queue messages, there is a possibility of message loss. For example, some ck msgs are not confirmed by ack msg. , And without sending the retry topic, the offset of the receive topic is updated directly. There is a possibility of repetition of ack msg due to the network or other reasons. The message corresponding to ack has been acked, and ack will be consumed repeatedly, but it has no effect. These ack msg will not be consumed repeatedly when the new ck msg is updated. NS
@odbozhou
Thanks for the explanation, but I have two questions:
- The offset was updated after putting messages into retry topic. Is it possible to fail to put msgs?
- Even if the above example happens, updating the offset by the ck msgs and updating it by the number of Revive topic queue messages are the same, because the ck msgs will be consumed only once. So I think updating the consume offset by the number of msgs directly will be better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] cserwen commented on pull request #3444: [ISSUE #3443] commit the right consumption offset for popReviewService
Posted by GitBox <gi...@apache.org>.
cserwen commented on pull request #3444:
URL: https://github.com/apache/rocketmq/pull/3444#issuecomment-957169713
@odbozhou OK. I got it, but it is really confusing. :confused:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] cserwen closed pull request #3444: [ISSUE #3443] commit the right consumption offset for popReviewService
Posted by GitBox <gi...@apache.org>.
cserwen closed pull request #3444:
URL: https://github.com/apache/rocketmq/pull/3444
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] odbozhou commented on pull request #3444: [ISSUE #3443] commit the right consumption offset for popReviewService
Posted by GitBox <gi...@apache.org>.
odbozhou commented on pull request #3444:
URL: https://github.com/apache/rocketmq/pull/3444#issuecomment-957136997
> > In order to prevent the message from being lost. Revive topic queue Offset update needs to pass ck msg. Revive topic contains both ck msg and ack msg. If the offset is updated directly by the number of Revive topic queue messages, there is a possibility of message loss. For example, some ck msgs are not confirmed by ack msg. , And without sending the retry topic, the offset of the receive topic is updated directly. There is a possibility of repetition of ack msg due to the network or other reasons. The message corresponding to ack has been acked, and ack will be consumed repeatedly, but it has no effect. These ack msg will not be consumed repeatedly when the new ck msg is updated. NS
>
> @odbozhou Thanks for the explanation, but I have two questions:
>
> * The offset was updated after putting messages into retry topic. Is it possible to fail to put msgs?
> * Even if the above example happens, updating the offset by the ck msgs and updating it by the number of Revive topic queue messages are the same, because the ck msgs will be consumed only once. So I think updating the consume offset by the number of msgs directly will be better.
- The ck message is not necessarily consumed only once. As shown in the figure, if break at this step, some ck has not been confirmed, and no retry topic has been sent. However, the consumption location of the receive topic queue is updated by the number of msgs. The unacknowledged ck message may lose the message, which does not meet the semantics of at least once.
- The msg in the receive topic has both ck and ack。
and it cannot only mean the number of ck.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] cserwen commented on pull request #3444: [ISSUE #3443] commit the right consumption offset for popReviewService
Posted by GitBox <gi...@apache.org>.
cserwen commented on pull request #3444:
URL: https://github.com/apache/rocketmq/pull/3444#issuecomment-957169713
@odbozhou OK. I got it, but it is really confusing. :confused:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] odbozhou commented on pull request #3444: [ISSUE #3443] commit the right consumption offset for popReviewService
Posted by GitBox <gi...@apache.org>.
odbozhou commented on pull request #3444:
URL: https://github.com/apache/rocketmq/pull/3444#issuecomment-957136997
> > In order to prevent the message from being lost. Revive topic queue Offset update needs to pass ck msg. Revive topic contains both ck msg and ack msg. If the offset is updated directly by the number of Revive topic queue messages, there is a possibility of message loss. For example, some ck msgs are not confirmed by ack msg. , And without sending the retry topic, the offset of the receive topic is updated directly. There is a possibility of repetition of ack msg due to the network or other reasons. The message corresponding to ack has been acked, and ack will be consumed repeatedly, but it has no effect. These ack msg will not be consumed repeatedly when the new ck msg is updated. NS
>
> @odbozhou Thanks for the explanation, but I have two questions:
>
> * The offset was updated after putting messages into retry topic. Is it possible to fail to put msgs?
> * Even if the above example happens, updating the offset by the ck msgs and updating it by the number of Revive topic queue messages are the same, because the ck msgs will be consumed only once. So I think updating the consume offset by the number of msgs directly will be better.
- The ck message is not necessarily consumed only once. As shown in the figure, if break at this step, some ck has not been confirmed, and no retry topic has been sent. However, the consumption location of the receive topic queue is updated by the number of msgs. The unacknowledged ck message may lose the message, which does not meet the semantics of at least once.
- The msg in the receive topic has both ck and ack。
and it cannot only mean the number of ck.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] cserwen commented on pull request #3444: [ISSUE #3443] commit the right consumption offset for popReviewService
Posted by GitBox <gi...@apache.org>.
cserwen commented on pull request #3444:
URL: https://github.com/apache/rocketmq/pull/3444#issuecomment-957169713
@odbozhou OK. I got it, but it is really confusing. :confused:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] cserwen closed pull request #3444: [ISSUE #3443] commit the right consumption offset for popReviewService
Posted by GitBox <gi...@apache.org>.
cserwen closed pull request #3444:
URL: https://github.com/apache/rocketmq/pull/3444
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org