You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@rocketmq.apache.org by "echooymxq (via GitHub)" <gi...@apache.org> on 2023/04/20 07:32:48 UTC
[GitHub] [rocketmq] echooymxq opened a new pull request, #6623: [ISSUE #6598] Fix pop cause out memory error when message store not w…
echooymxq opened a new pull request, #6623:
URL: https://github.com/apache/rocketmq/pull/6623
…riteable.
**Make sure set the target branch to `develop`**
## What is the purpose of the change
close #6598
<!--
If this PR fixes a GitHub issue, please add `fixes #<XXX>` or `closes #<XXX>`. Please refer to the documentation for more information:
https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue
-->
fix #<xxx> <!-- <xxx> replace with issue id -->
## Brief changelog
XX
## Verifying this change
XXXX
Follow this checklist to help us incorporate your contribution quickly and easily. Notice, `it would be helpful if you could finish the following 5 checklist(the last one is not necessary)before request the community to review your PR`.
- [x] Make sure there is a [Github issue](https://github.com/apache/rocketmq/issues) filed for the change (usually before you start working on it). Trivial changes like typos do not require a Github issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
- [x] Format the pull request title like `[ISSUE #123] Fix UnknownException when host config not exist`. Each commit in the pull request should have a meaningful subject line and body.
- [x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
- [x] Write necessary unit-test(over 80% coverage) to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in [test module](https://github.com/apache/rocketmq/tree/master/test).
- [x] Run `mvn -B clean apache-rat:check findbugs:findbugs checkstyle:checkstyle` to make sure basic checks pass. Run `mvn clean install -DskipITs` to make sure unit-test pass. Run `mvn clean test-compile failsafe:integration-test` to make sure integration-test pass.
- [ ] If this contribution is large, please file an [Apache Individual Contributor License Agreement](http://www.apache.org/licenses/#clas).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] echooymxq commented on a diff in pull request #6623: [ISSUE #6598] Fix pop cause out memory error when message store not writeable
Posted by "echooymxq (via GitHub)" <gi...@apache.org>.
echooymxq commented on code in PR #6623:
URL: https://github.com/apache/rocketmq/pull/6623#discussion_r1174745036
##########
broker/src/main/java/org/apache/rocketmq/broker/processor/PopBufferMergeService.java:
##########
@@ -402,9 +402,14 @@ private boolean checkQueueOk(PopCheckPointWrapper pointWrapper) {
* @param nextBeginOffset
* @return
*/
- public void addCkJustOffset(PopCheckPoint point, int reviveQueueId, long reviveQueueOffset, long nextBeginOffset) {
+ public boolean addCkJustOffset(PopCheckPoint point, int reviveQueueId, long reviveQueueOffset, long nextBeginOffset) {
PopCheckPointWrapper pointWrapper = new PopCheckPointWrapper(reviveQueueId, reviveQueueOffset, point, nextBeginOffset, true);
+ if (this.counter.get() > brokerController.getBrokerConfig().getPopCkMaxBufferSize()) {
Review Comment:
Yeah, if the buffer full, because the limit condition, the add ck to buffer will be failed, but the `addCkJustOffset ` will add buffer success. So i add the condition to here to avoid it can always add success.
what's more if the memory ack not enable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] drpmma commented on a diff in pull request #6623: [ISSUE #6598] Fix pop cause out memory error when message store not writeable
Posted by "drpmma (via GitHub)" <gi...@apache.org>.
drpmma commented on code in PR #6623:
URL: https://github.com/apache/rocketmq/pull/6623#discussion_r1174546518
##########
broker/src/main/java/org/apache/rocketmq/broker/processor/PopMessageProcessor.java:
##########
@@ -602,6 +602,28 @@ private CompletableFuture<Long> popMsgFromQueue(boolean isRetry, GetMessageResul
return atomicRestNum.get();
}
if (!result.getMessageMapedList().isEmpty()) {
+ if (isOrder) {
+ this.brokerController.getConsumerOrderInfoManager().update(isRetry, topic,
+ requestHeader.getConsumerGroup(),
+ queueId, popTime, requestHeader.getInvisibleTime(), result.getMessageQueueOffset(),
+ orderCountInfo);
+ this.brokerController.getConsumerOffsetManager().commitOffset(channel.remoteAddress().toString(),
+ requestHeader.getConsumerGroup(), topic, queueId, finalOffset);
+ } else {
+ boolean success = appendCheckPoint(requestHeader, topic, reviveQid, queueId, finalOffset, result, popTime, this.brokerController.getBrokerConfig().getBrokerName());
+ if (!success) {
+ // If append checkPoint error, will not return these messages.
+ for (SelectMappedBufferResult mapedBuffer : result.getMessageMapedList()) {
Review Comment:
mappedBuffer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] codecov-commenter commented on pull request #6623: [ISSUE #6598] Fix pop cause out memory error when message store not writeable
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #6623:
URL: https://github.com/apache/rocketmq/pull/6623#issuecomment-1519028792
## [Codecov](https://codecov.io/gh/apache/rocketmq/pull/6623?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#6623](https://codecov.io/gh/apache/rocketmq/pull/6623?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8410785) into [develop](https://codecov.io/gh/apache/rocketmq/commit/ea8b9d9f7a96a83d12006fd89c90356db658ef2e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (ea8b9d9) will **decrease** coverage by `0.02%`.
> The diff coverage is `25.92%`.
```diff
@@ Coverage Diff @@
## develop #6623 +/- ##
=============================================
- Coverage 43.11% 43.09% -0.02%
+ Complexity 8997 8991 -6
=============================================
Files 1107 1107
Lines 78287 78297 +10
Branches 10202 10206 +4
=============================================
- Hits 33755 33744 -11
- Misses 40314 40331 +17
- Partials 4218 4222 +4
```
| [Impacted Files](https://codecov.io/gh/apache/rocketmq/pull/6623?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...cketmq/broker/processor/PopBufferMergeService.java](https://codecov.io/gh/apache/rocketmq/pull/6623?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9icm9rZXIvcHJvY2Vzc29yL1BvcEJ1ZmZlck1lcmdlU2VydmljZS5qYXZh) | `33.10% <0.00%> (-0.24%)` | :arrow_down: |
| [...rocketmq/broker/processor/PopMessageProcessor.java](https://codecov.io/gh/apache/rocketmq/pull/6623?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9icm9rZXIvcHJvY2Vzc29yL1BvcE1lc3NhZ2VQcm9jZXNzb3IuamF2YQ==) | `36.88% <30.43%> (-0.74%)` | :arrow_down: |
... and [14 files with indirect coverage changes](https://codecov.io/gh/apache/rocketmq/pull/6623/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] drpmma commented on a diff in pull request #6623: [ISSUE #6598] Fix pop cause out memory error when message store not writeable
Posted by "drpmma (via GitHub)" <gi...@apache.org>.
drpmma commented on code in PR #6623:
URL: https://github.com/apache/rocketmq/pull/6623#discussion_r1174728921
##########
broker/src/main/java/org/apache/rocketmq/broker/processor/PopBufferMergeService.java:
##########
@@ -402,9 +402,14 @@ private boolean checkQueueOk(PopCheckPointWrapper pointWrapper) {
* @param nextBeginOffset
* @return
*/
- public void addCkJustOffset(PopCheckPoint point, int reviveQueueId, long reviveQueueOffset, long nextBeginOffset) {
+ public boolean addCkJustOffset(PopCheckPoint point, int reviveQueueId, long reviveQueueOffset, long nextBeginOffset) {
PopCheckPointWrapper pointWrapper = new PopCheckPointWrapper(reviveQueueId, reviveQueueOffset, point, nextBeginOffset, true);
+ if (this.counter.get() > brokerController.getBrokerConfig().getPopCkMaxBufferSize()) {
Review Comment:
The `addCkJustOffset` will only be invoked when add ck to buffer failed.
So this condition will always be true and this method will always return false.
See addCk
https://github.com/apache/rocketmq/blob/8410785ef9d0bd2d2387bb5a706d84a4971356be/broker/src/main/java/org/apache/rocketmq/broker/processor/PopBufferMergeService.java#L447-L467
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE #6598] Fix pop cause out memory error when message store not writeable [rocketmq]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #6623: [ISSUE #6598] Fix pop cause out memory error when message store not writeable
URL: https://github.com/apache/rocketmq/pull/6623
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] drpmma commented on a diff in pull request #6623: [ISSUE #6598] Fix pop cause out memory error when message store not writeable
Posted by "drpmma (via GitHub)" <gi...@apache.org>.
drpmma commented on code in PR #6623:
URL: https://github.com/apache/rocketmq/pull/6623#discussion_r1174546471
##########
broker/src/main/java/org/apache/rocketmq/broker/processor/PopMessageProcessor.java:
##########
@@ -602,6 +602,28 @@ private CompletableFuture<Long> popMsgFromQueue(boolean isRetry, GetMessageResul
return atomicRestNum.get();
}
if (!result.getMessageMapedList().isEmpty()) {
+ if (isOrder) {
+ this.brokerController.getConsumerOrderInfoManager().update(isRetry, topic,
+ requestHeader.getConsumerGroup(),
+ queueId, popTime, requestHeader.getInvisibleTime(), result.getMessageQueueOffset(),
+ orderCountInfo);
+ this.brokerController.getConsumerOffsetManager().commitOffset(channel.remoteAddress().toString(),
+ requestHeader.getConsumerGroup(), topic, queueId, finalOffset);
+ } else {
+ boolean success = appendCheckPoint(requestHeader, topic, reviveQid, queueId, finalOffset, result, popTime, this.brokerController.getBrokerConfig().getBrokerName());
+ if (!success) {
+ // If append checkPoint error, will not return these messages.
+ for (SelectMappedBufferResult mapedBuffer : result.getMessageMapedList()) {
Review Comment:
mappedBuffer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE #6598] Fix pop cause out memory error when message store not writeable [rocketmq]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6623:
URL: https://github.com/apache/rocketmq/pull/6623#issuecomment-2106064894
This PR is stale because it has been open for 365 days with no activity. It will be closed in 3 days if no further activity occurs. If you wish not to mark it as stale, please leave a comment in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] drpmma commented on a diff in pull request #6623: [ISSUE #6598] Fix pop cause out memory error when message store not writeable
Posted by "drpmma (via GitHub)" <gi...@apache.org>.
drpmma commented on code in PR #6623:
URL: https://github.com/apache/rocketmq/pull/6623#discussion_r1174728921
##########
broker/src/main/java/org/apache/rocketmq/broker/processor/PopBufferMergeService.java:
##########
@@ -402,9 +402,14 @@ private boolean checkQueueOk(PopCheckPointWrapper pointWrapper) {
* @param nextBeginOffset
* @return
*/
- public void addCkJustOffset(PopCheckPoint point, int reviveQueueId, long reviveQueueOffset, long nextBeginOffset) {
+ public boolean addCkJustOffset(PopCheckPoint point, int reviveQueueId, long reviveQueueOffset, long nextBeginOffset) {
PopCheckPointWrapper pointWrapper = new PopCheckPointWrapper(reviveQueueId, reviveQueueOffset, point, nextBeginOffset, true);
+ if (this.counter.get() > brokerController.getBrokerConfig().getPopCkMaxBufferSize()) {
Review Comment:
The
The `addCkJustOffset` will only be invoked when add ck to buffer failed.
So this condition will always be true and return false.
See addCk
https://github.com/apache/rocketmq/blob/8410785ef9d0bd2d2387bb5a706d84a4971356be/broker/src/main/java/org/apache/rocketmq/broker/processor/PopBufferMergeService.java#L447-L467
##########
broker/src/main/java/org/apache/rocketmq/broker/processor/PopBufferMergeService.java:
##########
@@ -402,9 +402,14 @@ private boolean checkQueueOk(PopCheckPointWrapper pointWrapper) {
* @param nextBeginOffset
* @return
*/
- public void addCkJustOffset(PopCheckPoint point, int reviveQueueId, long reviveQueueOffset, long nextBeginOffset) {
+ public boolean addCkJustOffset(PopCheckPoint point, int reviveQueueId, long reviveQueueOffset, long nextBeginOffset) {
PopCheckPointWrapper pointWrapper = new PopCheckPointWrapper(reviveQueueId, reviveQueueOffset, point, nextBeginOffset, true);
+ if (this.counter.get() > brokerController.getBrokerConfig().getPopCkMaxBufferSize()) {
Review Comment:
The `addCkJustOffset` will only be invoked when add ck to buffer failed.
So this condition will always be true and return false.
See addCk
https://github.com/apache/rocketmq/blob/8410785ef9d0bd2d2387bb5a706d84a4971356be/broker/src/main/java/org/apache/rocketmq/broker/processor/PopBufferMergeService.java#L447-L467
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] echooymxq commented on a diff in pull request #6623: [ISSUE #6598] Fix pop cause out memory error when message store not writeable
Posted by "echooymxq (via GitHub)" <gi...@apache.org>.
echooymxq commented on code in PR #6623:
URL: https://github.com/apache/rocketmq/pull/6623#discussion_r1174745036
##########
broker/src/main/java/org/apache/rocketmq/broker/processor/PopBufferMergeService.java:
##########
@@ -402,9 +402,14 @@ private boolean checkQueueOk(PopCheckPointWrapper pointWrapper) {
* @param nextBeginOffset
* @return
*/
- public void addCkJustOffset(PopCheckPoint point, int reviveQueueId, long reviveQueueOffset, long nextBeginOffset) {
+ public boolean addCkJustOffset(PopCheckPoint point, int reviveQueueId, long reviveQueueOffset, long nextBeginOffset) {
PopCheckPointWrapper pointWrapper = new PopCheckPointWrapper(reviveQueueId, reviveQueueOffset, point, nextBeginOffset, true);
+ if (this.counter.get() > brokerController.getBrokerConfig().getPopCkMaxBufferSize()) {
Review Comment:
Yeah, if the buffer full, because the limit condition, the add ck to buffer will be failed, but the `addCkJustOffset ` will add buffer success.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE #6598] Fix pop cause out memory error when message store not writeable [rocketmq]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6623:
URL: https://github.com/apache/rocketmq/pull/6623#issuecomment-2113677474
This PR was closed because it has been inactive for 3 days since being marked as stale.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] lizhimins commented on a diff in pull request #6623: [ISSUE #6598] Fix pop cause out memory error when message store not writeable
Posted by "lizhimins (via GitHub)" <gi...@apache.org>.
lizhimins commented on code in PR #6623:
URL: https://github.com/apache/rocketmq/pull/6623#discussion_r1192146299
##########
broker/src/main/java/org/apache/rocketmq/broker/processor/PopMessageProcessor.java:
##########
@@ -602,6 +602,28 @@ private CompletableFuture<Long> popMsgFromQueue(boolean isRetry, GetMessageResul
return atomicRestNum.get();
}
if (!result.getMessageMapedList().isEmpty()) {
+ if (isOrder) {
+ this.brokerController.getConsumerOrderInfoManager().update(isRetry, topic,
+ requestHeader.getConsumerGroup(),
+ queueId, popTime, requestHeader.getInvisibleTime(), result.getMessageQueueOffset(),
+ orderCountInfo);
+ this.brokerController.getConsumerOffsetManager().commitOffset(channel.remoteAddress().toString(),
+ requestHeader.getConsumerGroup(), topic, queueId, finalOffset);
+ } else {
+ boolean success = appendCheckPoint(requestHeader, topic, reviveQid, queueId, finalOffset, result, popTime, this.brokerController.getBrokerConfig().getBrokerName());
+ if (!success) {
+ // If append checkPoint error, will not return these messages.
+ for (SelectMappedBufferResult mapedBuffer : result.getMessageMapedList()) {
+ mapedBuffer.release();
+ }
+ atomicRestNum.set(result.getMaxOffset() - atomicOffset.get() + atomicRestNum.get());
Review Comment:
Suggest adding comments here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] echooymxq commented on a diff in pull request #6623: [ISSUE #6598] Fix pop cause out memory error when message store not writeable
Posted by "echooymxq (via GitHub)" <gi...@apache.org>.
echooymxq commented on code in PR #6623:
URL: https://github.com/apache/rocketmq/pull/6623#discussion_r1174745036
##########
broker/src/main/java/org/apache/rocketmq/broker/processor/PopBufferMergeService.java:
##########
@@ -402,9 +402,14 @@ private boolean checkQueueOk(PopCheckPointWrapper pointWrapper) {
* @param nextBeginOffset
* @return
*/
- public void addCkJustOffset(PopCheckPoint point, int reviveQueueId, long reviveQueueOffset, long nextBeginOffset) {
+ public boolean addCkJustOffset(PopCheckPoint point, int reviveQueueId, long reviveQueueOffset, long nextBeginOffset) {
PopCheckPointWrapper pointWrapper = new PopCheckPointWrapper(reviveQueueId, reviveQueueOffset, point, nextBeginOffset, true);
+ if (this.counter.get() > brokerController.getBrokerConfig().getPopCkMaxBufferSize()) {
Review Comment:
Yeah, if the buffer full, because the limit condition, the add ck to buffer will be failed, but the `addCkJustOffset ` will add buffer success. So i add the condition to here to avoid it can always add success.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org