You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@rocketmq.apache.org by "ai-yang (via GitHub)" <gi...@apache.org> on 2026/08/15 14:01:44 UTC
[PR] [ISSUE #10936] Avoid temporary buffers during batch message encoding [rocketmq]
ai-yang opened a new pull request, #10938:
URL: https://github.com/apache/rocketmq/pull/10938
### Which Issue(s) This PR Fixes
- Fixes #10936
- Related to #2931, #2932, #10442, #10444, and #10588
### Brief Description
`MessageDecoder.encodeMessages` currently encodes every message into its own complete `byte[]`, retains those arrays, then allocates a final result and copies every message again.
This change removes those temporary full-message buffers:
- the first pass caches each body reference and the serialized property bytes while computing the exact aggregate size;
- one final `ByteBuffer` is allocated;
- the second pass writes every message directly into that buffer;
- single-message and batch encoding share internal size and field-writing helpers;
- a byte-for-byte compatibility test covers an empty body, Unicode properties, a 4 KiB body, edge flag values and an empty list;
- an independent fixed golden vector verifies the wire field layout without relying on the shared helper as its oracle.
The public API and encoded wire format are unchanged.
### How Did You Test This Change?
#### Correctness
The new tests compare `encodeMessages(messages)` with the concatenation of `encodeMessage(message)` for every input and verify an independent fixed wire vector. The complete `common` module suite passes:
```text
Tests run: 245, Failures: 0, Errors: 0, Skipped: 0
BUILD SUCCESS
```
Command:
```bash
mvn -o -Dmaven.repo.local=/developer/wangrui/.m2/repository \
-Dspotbugs-plugin.version=4.2.2 -Dspotbugs.skip=true \
-Dcheckstyle.skip=true -Drat.skip=true \
-pl common test
```
Relevant downstream tests also pass:
```text
AppendCallbackTest + BatchPutMessageTest + AppendPropCRCTest:
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0
ProduceAccumulatorTest:
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
```
#### JMH benchmark
JMH 1.36 configuration:
- throughput mode with GC profiler;
- one thread pinned to CPU 5;
- 3 forks;
- 5 warm-up and 7 measurement iterations per fork, 500 ms each;
- fixed `-Xms512m -Xmx512m` heap;
- OpenJDK 11.0.31, Ubuntu 22.04.4, Intel Xeon Gold 6133.
The benchmark's `baseline()` method reconstructs the previous aggregation algorithm in the same binary (per-message full encodings followed by a final copy), so the table labels it as a reconstructed baseline.
| Scenario | Reconstructed baseline ops/s | This PR ops/s | Change | Baseline B/op | This PR B/op | Change |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| 10 x 128 B | 340,743 +/- 6,348 | 381,842 +/- 9,236 | +12.06% | 6,760 | 4,552 | -32.66% |
| 100 x 128 B | 32,247 +/- 1,099 | 36,246 +/- 242 | +12.40% | 67,152 | 45,192 | -32.70% |
| 10 x 1 KiB | 179,223 +/- 3,518 | 240,718 +/- 10,326 | +34.31% | 24,680 | 13,512 | -45.25% |
| 100 x 1 KiB | 16,954 +/- 594 | 22,411 +/- 594 | +32.19% | 246,352 | 134,792 | -45.28% |
Values are means plus or minus JMH's 99.9% confidence-interval half-width. The baseline and optimized throughput confidence intervals do not overlap in any scenario.
The 100 x 1 KiB case was repeated on OpenJDK 8u492:
```text
Throughput: 14,472.346 +/- 329.526 -> 18,954.609 +/- 113.257 ops/s (+30.97%)
Allocation: 288,568 -> 173,832 B/op (-39.76%)
```
Its throughput confidence intervals are also disjoint.
I also benchmarked the actual production method from separate baseline and optimized checkouts in A-B-A order for 100 x 1 KiB:
| Production checkout | Throughput ops/s | 99.9% CI | Allocation B/op |
| --- | ---: | ---: | ---: |
| Baseline A1 | 17,352.526 +/- 400.418 | [16,952.108, 17,752.943] | 246,352.044 |
| This PR | 22,600.219 +/- 880.650 | [21,719.569, 23,480.870] | 134,792.035 |
| Baseline A2 | 16,876.707 +/- 671.661 | [16,205.046, 17,548.368] | 246,352.045 |
This is +30.24%/+33.91% throughput versus the two baseline runs and -45.28% allocation. The confidence intervals are disjoint.
Benchmark source, methodology, checksums, and all raw per-fork/per-iteration JSON results: https://gist.github.com/ai-yang/c3e3f65351054263dec6f7b8d67ad3cd
`git diff --check` also passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE #10936] Avoid temporary buffers during batch message encoding [rocketmq]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #10938:
URL: https://github.com/apache/rocketmq/pull/10938#issuecomment-5302696919
## [Codecov](https://app.codecov.io/gh/apache/rocketmq/pull/10938?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 48.49%. Comparing base ([`293f588`](https://app.codecov.io/gh/apache/rocketmq/commit/293f5885719fc4aa3619446a1900f58ccfcfdd29?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`d66b4f8`](https://app.codecov.io/gh/apache/rocketmq/commit/d66b4f88d8195f2ddcd60e0e8cfed1ca36b34e80?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## develop #10938 +/- ##
=============================================
- Coverage 48.62% 48.49% -0.13%
+ Complexity 13692 13652 -40
=============================================
Files 1381 1381
Lines 101464 101472 +8
Branches 13187 13187
=============================================
- Hits 49337 49210 -127
- Misses 46142 46250 +108
- Partials 5985 6012 +27
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/rocketmq/pull/10938?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE #10936] Avoid temporary buffers during batch message encoding [rocketmq]
Posted by "ai-yang (via GitHub)" <gi...@apache.org>.
ai-yang commented on PR #10938:
URL: https://github.com/apache/rocketmq/pull/10938#issuecomment-5441504735
Update after refreshing this PR against the latest `develop`:
- rebased onto `e348efa66b08eb645ee123706ea6492fa9a3ad35` (RocketMQ 5.5.1);
- squashed the implementation and follow-up test change into one signed-off commit (`2c793417c`);
- reran the complete `common` test suite: 245/245 passed;
- reran `ReceiptHandleGroupTest`, which was the only failing test in the previous Ubuntu CI job: 7/7 passed;
- `git diff --check` passed.
The previous `ReceiptHandleGroupTest` failure is not reproducible on the refreshed head and is outside this PR's changed files. The force-push has retriggered the full CI matrix.
@RongtongJin, since you have recent context in `MessageDecoder`, could you please take a human review when available, particularly for wire compatibility and the shared sizing/writing helpers? Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org