You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@rocketmq.apache.org by "ai-yang (via GitHub)" <gi...@apache.org> on 2026/08/02 13:30:36 UTC
[PR] [ISSUE #10755] Fix ConsumeQueueExt truncation cleanup [rocketmq]
ai-yang opened a new pull request, #10758:
URL: https://github.com/apache/rocketmq/pull/10758
### Which Issue(s) This PR Fixes
Fixes #10755
### Brief Description
`ConsumeQueue.truncateDirtyLogicFiles()` truncated the main consume queue but did not trim `ConsumeQueueExt` to the last retained extension address. Ext units from the discarded CQ tail therefore remained readable, and later appends plus reload/recovery could preserve those orphaned units permanently.
This change aligns CQExt truncation with the successfully retained CQ state:
- normal truncation exits locate the last readable retained Ext reference after `minLogicOffset` and truncate to it;
- when no Ext reference is retained, all Ext mapped files are removed and the address cursor is reset only after deletion succeeds;
- partial mapped-file deletion enters a pending state that blocks address reuse until cleanup is retried successfully;
- CQ physical-delete failures and `deleteFile=false` do not mutate CQExt independently;
- recovery of an empty CQ cleans up orphaned Ext files left by an interrupted truncation.
### How Did You Test This Change?
- Unmodified `develop`: the deterministic truncate/reload regression failed in 5/5 isolated JDK 8 Maven processes.
- Fixed targeted suite: 20 isolated Maven/JVM processes at 9/9 each (180/180 total), covering partial truncation, reload, full clear, failed deletion/retry, address reuse, `minLogicOffset`, and recovery.
- Full `store -am test`: common 241/241, remoting 174/174, and store 322 tests with 4 skips, 0 failures, and 0 errors.
- Maven validate and Checkstyle: 0 violations.
- SpotBugs: 0 bugs/errors across all four reactor modules.
- `git diff --check`: passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE #10755] Fix ConsumeQueueExt truncation cleanup [rocketmq]
Posted by "ai-yang (via GitHub)" <gi...@apache.org>.
ai-yang commented on PR #10758:
URL: https://github.com/apache/rocketmq/pull/10758#issuecomment-5158667299
Clarification on the automated review warning: `truncateAllPending` is reset to `false` after all Ext mapped files are deleted and the flush/commit positions are reset in `ConsumeQueueExt.truncateAll()`. It intentionally remains `true` only when deletion is incomplete, preventing address reuse until a later retry succeeds. The held-buffer regression covers block → release/retry → address reuse → reload.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE #10755] Fix ConsumeQueueExt truncation cleanup [rocketmq]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #10758:
URL: https://github.com/apache/rocketmq/pull/10758#issuecomment-5302627595
## [Codecov](https://app.codecov.io/gh/apache/rocketmq/pull/10758?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `91.56627%` with `7 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 48.55%. Comparing base ([`a06836d`](https://app.codecov.io/gh/apache/rocketmq/commit/a06836dd564e5e43115493f775626cf98d51d10e?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`82bc882`](https://app.codecov.io/gh/apache/rocketmq/commit/82bc88273bb4a7f277735621bbc45a70f4d875e3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 13 commits behind head on develop.
| [Files with missing lines](https://app.codecov.io/gh/apache/rocketmq/pull/10758?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...n/java/org/apache/rocketmq/store/ConsumeQueue.java](https://app.codecov.io/gh/apache/rocketmq/pull/10758?src=pr&el=tree&filepath=store%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Frocketmq%2Fstore%2FConsumeQueue.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3N0b3JlL0NvbnN1bWVRdWV1ZS5qYXZh) | 92.59% | [0 Missing and 4 partials :warning: ](https://app.codecov.io/gh/apache/rocketmq/pull/10758?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ava/org/apache/rocketmq/store/ConsumeQueueExt.java](https://app.codecov.io/gh/apache/rocketmq/pull/10758?src=pr&el=tree&filepath=store%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Frocketmq%2Fstore%2FConsumeQueueExt.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3N0b3JlL0NvbnN1bWVRdWV1ZUV4dC5qYXZh) | 89.65% | [1 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/rocketmq/pull/10758?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## develop #10758 +/- ##
=============================================
+ Coverage 48.29% 48.55% +0.26%
- Complexity 13502 13684 +182
=============================================
Files 1380 1381 +1
Lines 101091 101517 +426
Branches 13101 13199 +98
=============================================
+ Hits 48824 49294 +470
+ Misses 46305 46208 -97
- Partials 5962 6015 +53
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/rocketmq/pull/10758?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE #10755] Fix ConsumeQueueExt truncation cleanup [rocketmq]
Posted by "ai-yang (via GitHub)" <gi...@apache.org>.
ai-yang commented on PR #10758:
URL: https://github.com/apache/rocketmq/pull/10758#issuecomment-5441509131
Update after refreshing this PR against the latest `develop`:
- rebased onto `e348efa66b08eb645ee123706ea6492fa9a3ad35` (RocketMQ 5.5.1);
- squashed the two PR commits into one signed-off commit (`00e1c9767`);
- reran the complete `ConsumeQueueTest`: 21/21 passed;
- Checkstyle, SpotBugs, and `git diff --check` passed in the affected reactor build.
The full CI matrix has been retriggered by the force-push.
@RongtongJin @guyinyou, could one of you please take a human review when available, particularly for the CQ/CQExt truncation and recovery invariants? Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE #10755] Fix ConsumeQueueExt truncation cleanup [rocketmq]
Posted by "RockteMQ-AI (via GitHub)" <gi...@apache.org>.
RockteMQ-AI commented on PR #10758:
URL: https://github.com/apache/rocketmq/pull/10758#issuecomment-5161427177
Thanks for the clarification, @ai-yang. You're right — I can see that `truncateAllPending` is correctly reset to `false` at the end of the successful path in `truncateAll()` (after `setFlushedWhere(0)` and `setCommittedWhere(0)`). The early return on the failure path (files not fully deleted) intentionally keeps it `true` to gate new `put()` calls until a retry succeeds. My original warning was incorrect on this point; the design is sound. 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE #10755] Fix ConsumeQueueExt truncation cleanup [rocketmq]
Posted by "ai-yang (via GitHub)" <gi...@apache.org>.
ai-yang commented on PR #10758:
URL: https://github.com/apache/rocketmq/pull/10758#issuecomment-5463264767
All 10 CI checks are green on the refreshed single signed-off commit. @RongtongJin @guyinyou, could you please take a human review when convenient? Review-size guide: production is +124/-39 lines, while +629 lines are recovery/truncation regressions. The key point is keeping main ConsumeQueue and ConsumeQueueExt truncation boundaries consistent when the main CQ becomes empty.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org