You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by GitBox <gi...@apache.org> on 2020/04/21 09:18:59 UTC
[GitHub] [rocketmq] panzhi33 opened a new pull request #1955: get real and effective min offset in queue
panzhi33 opened a new pull request #1955:
URL: https://github.com/apache/rocketmq/pull/1955
## What is the purpose of the change
Can obtain the number of real and effective dead letter topic messages。because the message corresponding to the smallest position of the consume queue has been deleted，therefore, you need to use this displacement point to determine whether the message exists.
## Brief changelog
are some refactoring methods and calling ready-made methods
## Verifying this change
XXXX
Follow this checklist to help us incorporate your contribution quickly and easily. Notice, `it would be helpful if you could finish the following 5 checklist(the last one is not necessary)before request the community to review your PR`.
- [x] Make sure there is a [Github issue](https://github.com/apache/rocketmq/issues) filed for the change (usually before you start working on it). Trivial changes like typos do not require a Github issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
- [x] Format the pull request title like `[ISSUE #123] Fix UnknownException when host config not exist`. Each commit in the pull request should have a meaningful subject line and body.
- [x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
- [x] Write necessary unit-test(over 80% coverage) to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in [test module](https://github.com/apache/rocketmq/tree/master/test).
- [x] Run `mvn -B clean apache-rat:check findbugs:findbugs checkstyle:checkstyle` to make sure basic checks pass. Run `mvn clean install -DskipITs` to make sure unit-test pass. Run `mvn clean test-compile failsafe:integration-test` to make sure integration-test pass.
- [ ] If this contribution is large, please file an [Apache Individual Contributor License Agreement](http://www.apache.org/licenses/#clas).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] panzhi33 removed a comment on pull request #1955: [ISSUE 1960]get real and effective min offset in queue
Posted by GitBox <gi...@apache.org>.
panzhi33 removed a comment on pull request #1955:
URL: https://github.com/apache/rocketmq/pull/1955#issuecomment-619024392
sorry，because before we need to query the content of the dead letter queue message，did‘t consider this place does not need。I am modified to compare 'this.commitLog.getMinOffset()'
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] coveralls edited a comment on pull request #1955: [ISSUE 1960]get real and effective min offset in queue
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #1955:
URL: https://github.com/apache/rocketmq/pull/1955#issuecomment-617622411
[](https://coveralls.io/builds/30318419)
Coverage increased (+0.03%) to 50.947% when pulling **bdfbc61b89b6c7eb38fbe7834d0d6e1b4cab3647 on panzhi33:develop** into **1706cecff0993efdb00a079068f972d2e9a3699f on apache:develop**.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] panzhi33 commented on pull request #1955: [ISSUE 1960]get real and effective min offset in queue
Posted by GitBox <gi...@apache.org>.
panzhi33 commented on pull request #1955:
URL: https://github.com/apache/rocketmq/pull/1955#issuecomment-618765172
> Hi, @panzhi33 Plz submit related issues to link this PR
ok. fix #1960
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] coveralls edited a comment on issue #1955: get real and effective min offset in queue
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on issue #1955:
URL: https://github.com/apache/rocketmq/pull/1955#issuecomment-617622411
[](https://coveralls.io/builds/30255834)
Coverage decreased (-0.1%) to 50.792% when pulling **907a887d8a1ca09ee7e00fb9aadcaa0ad435f11e on panzhi33:develop** into **1706cecff0993efdb00a079068f972d2e9a3699f on apache:develop**.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] RongtongJin commented on a change in pull request #1955: [ISSUE 1960]get real and effective min offset in queue
Posted by GitBox <gi...@apache.org>.
RongtongJin commented on a change in pull request #1955:
URL: https://github.com/apache/rocketmq/pull/1955#discussion_r414501880
##########
File path: store/src/main/java/org/apache/rocketmq/store/DefaultMessageStore.java
##########
@@ -663,9 +663,27 @@ public long getMaxOffsetInQueue(String topic, int queueId) {
}
public long getMinOffsetInQueue(String topic, int queueId) {
+ return getMinOffsetInQueue(topic, queueId,false);
+ }
+
+ public long getMinOffsetInQueue(String topic, int queueId, boolean realOffset) {
ConsumeQueue logic = this.findConsumeQueue(topic, queueId);
if (logic != null) {
- return logic.getMinOffsetInQueue();
+ long consumeQueueOffset = logic.getMinOffsetInQueue();
+ MessageExt msgExt = null;
+ long commitLogOffset = 0L;
+ if (realOffset) {
+ for (;consumeQueueOffset <= getMaxOffsetInQueue(topic,queueId);consumeQueueOffset++) {
+ commitLogOffset = getCommitLogOffsetInQueue(topic, queueId, consumeQueueOffset);
+ msgExt = lookMessageByOffset(commitLogOffset);
+ if (null != msgExt) {
+ return consumeQueueOffset;
+ }
+ }
+ return consumeQueueOffset;
Review comment:
IMO, this is a very CPU intensive operation. It is possible to get a lot of cold data and decode it, which will affect the stability of the broker.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] coveralls edited a comment on issue #1955: get real and effective min offset in queue
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on issue #1955:
URL: https://github.com/apache/rocketmq/pull/1955#issuecomment-617622411
[](https://coveralls.io/builds/30265602)
Coverage increased (+0.004%) to 50.921% when pulling **3fa9e10fc56f37a7b51987180b74ed513558cba5 on panzhi33:develop** into **1706cecff0993efdb00a079068f972d2e9a3699f on apache:develop**.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] RongtongJin commented on issue #1955: get real and effective min offset in queue
Posted by GitBox <gi...@apache.org>.
RongtongJin commented on issue #1955:
URL: https://github.com/apache/rocketmq/pull/1955#issuecomment-618399880
Hi, @panzhi33 Plz submit related issues to link this PR
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] panzhi33 commented on a change in pull request #1955: [ISSUE 1960]get real and effective min offset in queue
Posted by GitBox <gi...@apache.org>.
panzhi33 commented on a change in pull request #1955:
URL: https://github.com/apache/rocketmq/pull/1955#discussion_r414593841
##########
File path: store/src/main/java/org/apache/rocketmq/store/DefaultMessageStore.java
##########
@@ -663,9 +663,27 @@ public long getMaxOffsetInQueue(String topic, int queueId) {
}
public long getMinOffsetInQueue(String topic, int queueId) {
+ return getMinOffsetInQueue(topic, queueId,false);
+ }
+
+ public long getMinOffsetInQueue(String topic, int queueId, boolean realOffset) {
ConsumeQueue logic = this.findConsumeQueue(topic, queueId);
if (logic != null) {
- return logic.getMinOffsetInQueue();
+ long consumeQueueOffset = logic.getMinOffsetInQueue();
+ MessageExt msgExt = null;
+ long commitLogOffset = 0L;
+ if (realOffset) {
+ for (;consumeQueueOffset <= getMaxOffsetInQueue(topic,queueId);consumeQueueOffset++) {
+ commitLogOffset = getCommitLogOffsetInQueue(topic, queueId, consumeQueueOffset);
+ msgExt = lookMessageByOffset(commitLogOffset);
+ if (null != msgExt) {
+ return consumeQueueOffset;
+ }
+ }
+ return consumeQueueOffset;
Review comment:
sorry，because before we need to query the content of the dead letter queue message，did‘t consider this place does not need。I am modified to compare 'this.commitLog.getMinOffset()'
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] panzhi33 commented on pull request #1955: [ISSUE 1960]get real and effective min offset in queue
Posted by GitBox <gi...@apache.org>.
panzhi33 commented on pull request #1955:
URL: https://github.com/apache/rocketmq/pull/1955#issuecomment-619024392
sorry，because before we need to query the content of the dead letter queue message，did‘t consider this place does not need。I am modified to compare 'this.commitLog.getMinOffset()'
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] coveralls commented on issue #1955: get real and effective min offset in queue
Posted by GitBox <gi...@apache.org>.
coveralls commented on issue #1955:
URL: https://github.com/apache/rocketmq/pull/1955#issuecomment-617622411
[](https://coveralls.io/builds/30250721)
Coverage decreased (-0.06%) to 50.852% when pulling **c432037c8c5cca108588bfcaaa297bd7ba89595f on panzhi33:develop** into **1706cecff0993efdb00a079068f972d2e9a3699f on apache:develop**.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org