You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@rocketmq.apache.org by "wang-jiahua (via GitHub)" <gi...@apache.org> on 2026/06/17 15:14:18 UTC
[PR] [ISSUE #10527] Reduce auxiliary component allocation via AtomicLong, string caches, StringBuilder reuse, and dirty flag [rocketmq]
wang-jiahua opened a new pull request, #10528:
URL: https://github.com/apache/rocketmq/pull/10528
<!-- Please make sure the target branch is right. In most case, the target branch should be `develop`. -->
### Which Issue(s) This PR Fixes
<!-- Please ensure that the related issue has already been created, and [link this pull request to that issue using keywords](<https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword>) to ensure automatic closure. -->
Fixes #10527
### Brief Description
Reduce allocation in auxiliary store components through five independent optimizations:
1. **`QueueOffsetOperator`** — Replace `ConcurrentMap<String, Long>` with `ConcurrentMap<String, AtomicLong>`. Eliminates `Long` boxing on every queue offset update.
2. **`BrokerStatsManager`** — Cache `buildStatsKey`/`topicQueueKey`/`consumerOffset` string results. Change `Integer` parameters to `int` in `incQueue*` methods to eliminate autoboxing.
3. **`IndexService`** — Reuse `StringBuilder` via ThreadLocal instead of allocating per-call.
4. **`TimerWheel`** — Add volatile `dirty` flag to skip flush when no changes occurred since last flush.
5. **`AppendMessageResult`** — Add constructor with pre-computed fields to avoid redundant allocation.
### How Did You Test This Change?
1. **Unit tests**: All 15 `BrokerStatsManagerTest` tests pass (including `testOnTopicDeleted` which exercises all `incQueue*` methods).
2. **Compilation**: `store` module compiles cleanly on JDK 21.
3. **Commercial compatibility**: No commercial classes extend `QueueOffsetOperator`, `BrokerStatsManager`, `IndexService`, `TimerWheel`, or `AppendMessageResult`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE #10527] Reduce auxiliary component allocation via AtomicLong, string caches, StringBuilder reuse, and dirty flag [rocketmq]
Posted by "wang-jiahua (via GitHub)" <gi...@apache.org>.
wang-jiahua commented on code in PR #10528:
URL: https://github.com/apache/rocketmq/pull/10528#discussion_r3878053987
##########
store/src/main/java/org/apache/rocketmq/store/index/IndexService.java:
##########
@@ -215,10 +216,12 @@ public QueryOffsetResult queryOffset(String topic, String key, int maxNum, long
}
private String buildKey(final String topic, final String key) {
- return topic + "#" + key;
+ reusableKeyBuilder.setLength(0);
Review Comment:
Confirmed — `buildKey` is reachable from `queryOffset` (query thread pools) and from concurrent dispatch when `enableBuildConsumeQueueConcurrently` is on, so a shared instance StringBuilder can interleave. Switched to `ThreadLocal<StringBuilder>` in 43be17b98, which keeps the allocation win on the hot build path and is safe on every thread.
##########
store/src/main/java/org/apache/rocketmq/store/timer/TimerWheel.java:
##########
@@ -128,17 +130,25 @@ public void flush() {
if (mappedByteBuffer == null) {
return;
}
+ if (!dirty) {
Review Comment:
The per-thread-buffer scenario doesn't apply here: `localBuffer` holds `byteBuffer.duplicate()`, and duplicates share the single underlying direct buffer — the ThreadLocal only isolates position/limit cursors, so there is exactly one copy of the wheel data and no thread can flush stale/uninitialized content. That said, this comment made me re-examine the flag ordering and there was a real lost-update window: clearing `dirty` after the diff could swallow a concurrent `putSlot`'s mark, leaving the last write unflushed until the next putSlot. Fixed in 43be17b98 by clearing the flag before diffing — any interleaving now causes at most one extra flush, never a missed one.
##########
store/src/main/java/org/apache/rocketmq/store/queue/QueueOffsetOperator.java:
##########
@@ -123,10 +132,18 @@ public void setLmqTopicQueueTable(ConcurrentMap<String, Long> lmqTopicQueueTable
}
public ConcurrentMap<String, Long> getTopicQueueTable() {
- return topicQueueTable;
+ ConcurrentMap<String, Long> snapshot = new ConcurrentHashMap<>(this.topicQueueTable.size() * 2);
Review Comment:
Agreed, and on reflection the risk is worse than a doc issue: callers (including extensions that reach this via reflection, e.g. recovery flows pre-populating offsets) may rely on mutating the returned map and would break silently with a snapshot. I reverted the QueueOffsetOperator changes entirely in 43be17b98 — `getTopicQueueTable()` returns the live map again and the AtomicLong conversion is dropped from this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE #10527] Reduce auxiliary component allocation via AtomicLong, string caches, StringBuilder reuse, and dirty flag [rocketmq]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #10528:
URL: https://github.com/apache/rocketmq/pull/10528#issuecomment-4732648291
## [Codecov](https://app.codecov.io/gh/apache/rocketmq/pull/10528?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `59.31034%` with `59 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 48.07%. Comparing base ([`59a70d9`](https://app.codecov.io/gh/apache/rocketmq/commit/59a70d9b80a348a2e3fcaa91cd0c52e1c998b308?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`958a954`](https://app.codecov.io/gh/apache/rocketmq/commit/958a954b082722045eddc702e41f33e55b1bfdec?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 4 commits behind head on develop.
| [Files with missing lines](https://app.codecov.io/gh/apache/rocketmq/pull/10528?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...pache/rocketmq/store/stats/BrokerStatsManager.java](https://app.codecov.io/gh/apache/rocketmq/pull/10528?src=pr&el=tree&filepath=store%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Frocketmq%2Fstore%2Fstats%2FBrokerStatsManager.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3N0b3JlL3N0YXRzL0Jyb2tlclN0YXRzTWFuYWdlci5qYXZh) | 55.42% | [22 Missing and 15 partials :warning: ](https://app.codecov.io/gh/apache/rocketmq/pull/10528?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...org/apache/rocketmq/store/AppendMessageResult.java](https://app.codecov.io/gh/apache/rocketmq/pull/10528?src=pr&el=tree&filepath=store%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Frocketmq%2Fstore%2FAppendMessageResult.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3N0b3JlL0FwcGVuZE1lc3NhZ2VSZXN1bHQuamF2YQ==) | 0.00% | [10 Missing :warning: ](https://app.codecov.io/gh/apache/rocketmq/pull/10528?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/rocketmq/store/queue/QueueOffsetOperator.java](https://app.codecov.io/gh/apache/rocketmq/pull/10528?src=pr&el=tree&filepath=store%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Frocketmq%2Fstore%2Fqueue%2FQueueOffsetOperator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3N0b3JlL3F1ZXVlL1F1ZXVlT2Zmc2V0T3BlcmF0b3IuamF2YQ==) | 74.19% | [7 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/rocketmq/pull/10528?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...va/org/apache/rocketmq/store/timer/TimerWheel.java](https://app.codecov.io/gh/apache/rocketmq/pull/10528?src=pr&el=tree&filepath=store%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Frocketmq%2Fstore%2Ftimer%2FTimerWheel.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3JvY2tldG1xL3N0b3JlL3RpbWVyL1RpbWVyV2hlZWwuamF2YQ==) | 75.00% | [3 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/rocketmq/pull/10528?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## develop #10528 +/- ##
=============================================
- Coverage 48.19% 48.07% -0.12%
+ Complexity 13394 13381 -13
=============================================
Files 1377 1377
Lines 100730 100850 +120
Branches 13012 13042 +30
=============================================
- Hits 48542 48487 -55
- Misses 46254 46392 +138
- Partials 5934 5971 +37
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/rocketmq/pull/10528?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE #10527] Reduce auxiliary component allocation via AtomicLong, string caches, StringBuilder reuse, and dirty flag [rocketmq]
Posted by "RockteMQ-AI (via GitHub)" <gi...@apache.org>.
RockteMQ-AI commented on code in PR #10528:
URL: https://github.com/apache/rocketmq/pull/10528#discussion_r3867553364
##########
store/src/main/java/org/apache/rocketmq/store/index/IndexService.java:
##########
@@ -215,10 +216,12 @@ public QueryOffsetResult queryOffset(String topic, String key, int maxNum, long
}
private String buildKey(final String topic, final String key) {
- return topic + "#" + key;
+ reusableKeyBuilder.setLength(0);
Review Comment:
**[Critical] Thread-safety bug**: `reusableKeyBuilder` is an instance field shared across threads, but `buildKey()` is called from `queryOffset()` which can be invoked concurrently by multiple admin/consumer query threads. StringBuilder is NOT thread-safe — concurrent calls to `setLength(0)` and `append()` will interleave, producing corrupted keys like `topic1#key2` instead of `topic1#key1`.
This will cause index lookups to return wrong results or miss entries entirely.
**Fix**: Use `ThreadLocal<StringBuilder>` instead of a shared instance field, or revert to string concatenation (the JIT compiler optimizes it well).
##########
store/src/main/java/org/apache/rocketmq/store/timer/TimerWheel.java:
##########
@@ -128,17 +130,25 @@ public void flush() {
if (mappedByteBuffer == null) {
return;
}
+ if (!dirty) {
Review Comment:
**[Critical] Thread-safety bug**: The `dirty` flag is instance-level (shared across threads), but `localBuffer` is `ThreadLocal<ByteBuffer>` (per-thread). This creates a race condition:
1. Thread A calls `putSlot()` → sets `dirty = true`
2. Thread B calls `flush()` → sees `dirty = true` → flushes Thread B's local buffer
3. Thread B's buffer may be uninitialized (all zeros) or contain stale data
4. Flushing zeros over the mapped file corrupts the timer wheel
**Fix**: Make `dirty` a `ThreadLocal<Boolean>` to match the per-thread buffer, or use a synchronized block around flush operations.
##########
store/src/main/java/org/apache/rocketmq/store/queue/QueueOffsetOperator.java:
##########
@@ -123,10 +132,18 @@ public void setLmqTopicQueueTable(ConcurrentMap<String, Long> lmqTopicQueueTable
}
public ConcurrentMap<String, Long> getTopicQueueTable() {
- return topicQueueTable;
+ ConcurrentMap<String, Long> snapshot = new ConcurrentHashMap<>(this.topicQueueTable.size() * 2);
Review Comment:
**[Warning] Behavioral change**: `getTopicQueueTable()` now returns a snapshot copy instead of the live map. While this is safer for thread safety, it's a behavioral change that could affect callers who expected to modify the map directly.
If any code relies on modifying the returned map and seeing those changes reflected in the internal state, it will break silently.
**Recommendation**: Document this change in the method's Javadoc, or consider whether callers actually need live access.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE #10527] Reduce auxiliary component allocation via AtomicLong, string caches, StringBuilder reuse, and dirty flag [rocketmq]
Posted by "wang-jiahua (via GitHub)" <gi...@apache.org>.
wang-jiahua commented on code in PR #10528:
URL: https://github.com/apache/rocketmq/pull/10528#discussion_r3429505721
##########
store/src/main/java/org/apache/rocketmq/store/index/IndexService.java:
##########
@@ -49,6 +49,7 @@ public class IndexService implements CommitLogDispatchStore {
private final String storePath;
private final ArrayList<IndexFile> indexFileList = new ArrayList<>();
private final ReadWriteLock readWriteLock = new ReentrantReadWriteLock();
+ private final StringBuilder reusableKeyBuilder = new StringBuilder(128);
Review Comment:
@/tmp/reply-s4-idx.txt
##########
store/src/main/java/org/apache/rocketmq/store/timer/TimerWheel.java:
##########
@@ -55,6 +55,8 @@ protected ByteBuffer initialValue() {
};
private final int wheelLength;
+ private volatile boolean dirty;
Review Comment:
@/tmp/reply-s4-timer.txt
##########
store/src/main/java/org/apache/rocketmq/store/stats/BrokerStatsManager.java:
##########
@@ -464,47 +470,92 @@ public void incGroupAckNums(final String group, final String topic, final int in
}
public String buildStatsKey(String topic, String group) {
- StringBuilder strBuilder;
if (topic != null && group != null) {
- strBuilder = new StringBuilder(topic.length() + group.length() + 1);
- } else {
- strBuilder = new StringBuilder();
+ ConcurrentHashMap<String, String> groupMap = statsKeyByGroupCache.get(topic);
+ if (groupMap != null) {
+ String cached = groupMap.get(group);
+ if (cached != null) {
+ return cached;
+ }
+ }
+ String key = topic + "@" + group;
+ statsKeyByGroupCache.computeIfAbsent(topic, t -> new ConcurrentHashMap<>()).put(group, key);
+ return key;
}
- strBuilder.append(topic).append("@").append(group);
- return strBuilder.toString();
+ return topic + "@" + group;
}
public String buildStatsKey(String topic, int queueId) {
- StringBuilder strBuilder;
- if (topic != null) {
- strBuilder = new StringBuilder(topic.length() + 5);
- } else {
- strBuilder = new StringBuilder();
+ String[] keys = statsKeyByQueueCache.get(topic);
+ if (keys != null && queueId >= 0 && queueId < keys.length) {
+ String cached = keys[queueId];
+ if (cached != null) {
+ return cached;
+ }
}
- strBuilder.append(topic).append("@").append(queueId);
- return strBuilder.toString();
+ String key = topic + "@" + queueId;
Review Comment:
@/tmp/reply-s4-null.txt
##########
store/src/main/java/org/apache/rocketmq/store/stats/BrokerStatsManager.java:
##########
@@ -380,32 +386,32 @@ public void onGroupDeleted(final String group) {
this.momentStatsItemSetFallTime.delValueBySuffixKey(group, "@");
}
- public void incQueuePutNums(final String topic, final Integer queueId) {
+ public void incQueuePutNums(final String topic, final int queueId) {
Review Comment:
@/tmp/reply-s4-int.txt
##########
store/src/main/java/org/apache/rocketmq/store/stats/BrokerStatsManager.java:
##########
@@ -141,6 +142,11 @@ public class BrokerStatsManager {
private ScheduledExecutorService cleanResourceExecutor;
private final HashMap<String, StatsItemSet> statsTable = new HashMap<>();
+ private final ConcurrentHashMap<String, String[]> latencyKeyCache = new ConcurrentHashMap<>();
+ private final ConcurrentHashMap<String, String[]> statsKeyByQueueCache = new ConcurrentHashMap<>();
+ private final ConcurrentHashMap<String, ConcurrentHashMap<String, String>> statsKeyByGroupCache = new ConcurrentHashMap<>();
+ private final ConcurrentHashMap<String, ConcurrentHashMap<String, String[]>> statsKeyByTopicQueueGroupCache = new ConcurrentHashMap<>();
+ private final ConcurrentHashMap<String, ConcurrentHashMap<String, String[]>> statsKeyByQueueTopicGroupCache = new ConcurrentHashMap<>();
Review Comment:
@/tmp/reply-s4-cache.txt
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE #10527] Reduce auxiliary component allocation via AtomicLong, string caches, StringBuilder reuse, and dirty flag [rocketmq]
Posted by "wang-jiahua (via GitHub)" <gi...@apache.org>.
wang-jiahua commented on PR #10528:
URL: https://github.com/apache/rocketmq/pull/10528#issuecomment-4766521642
## Server Benchmark Results (2026-06-22)
**Environment**: 8C30G broker, Dragonwell JDK 21, 1KB msg, 128 queues, sync mode, 256 threads, 20s warmup + 30s collect
### Producer-only (steady state, lines 3-5)
| Test | TPS | Avg RT | GC | vs Baseline |
|------|-----|--------|----|-------------|
| Baseline | 217k | 1.18ms | 354 | — |
| #10528 | 224k | 1.14ms | 332 | **+3.2% TPS, -3.4% RT** ✅ |
Auxiliary component optimizations (QueueOffsetOperator AtomicLong, IndexService StringBuilder reuse, TimerWheel dirty flag) reduce boxing and string allocation on the put message path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE #10527] Reduce auxiliary component allocation via AtomicLong, string caches, StringBuilder reuse, and dirty flag [rocketmq]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #10528:
URL: https://github.com/apache/rocketmq/pull/10528#discussion_r3429301864
##########
store/src/main/java/org/apache/rocketmq/store/index/IndexService.java:
##########
@@ -49,6 +49,7 @@ public class IndexService implements CommitLogDispatchStore {
private final String storePath;
private final ArrayList<IndexFile> indexFileList = new ArrayList<>();
private final ReadWriteLock readWriteLock = new ReentrantReadWriteLock();
+ private final StringBuilder reusableKeyBuilder = new StringBuilder(128);
Review Comment:
Using a single instance `StringBuilder` (`reusableKeyBuilder`) makes `buildKey(...)` not thread-safe. If `IndexService` methods are called concurrently, keys can be corrupted due to races on the shared builder. Prefer a local `StringBuilder`, a `ThreadLocal<StringBuilder>`, or guard access (e.g., synchronize) around builder mutation.
##########
store/src/main/java/org/apache/rocketmq/store/index/IndexService.java:
##########
@@ -215,10 +216,12 @@ public QueryOffsetResult queryOffset(String topic, String key, int maxNum, long
}
private String buildKey(final String topic, final String key) {
- return topic + "#" + key;
+ reusableKeyBuilder.setLength(0);
+ return reusableKeyBuilder.append(topic).append('#').append(key).toString();
}
private String buildKey(final String topic, final String key, final String indexType) {
- return topic + "#" + indexType + "#" + key;
+ reusableKeyBuilder.setLength(0);
+ return reusableKeyBuilder.append(topic).append('#').append(indexType).append('#').append(key).toString();
Review Comment:
Using a single instance `StringBuilder` (`reusableKeyBuilder`) makes `buildKey(...)` not thread-safe. If `IndexService` methods are called concurrently, keys can be corrupted due to races on the shared builder. Prefer a local `StringBuilder`, a `ThreadLocal<StringBuilder>`, or guard access (e.g., synchronize) around builder mutation.
##########
store/src/main/java/org/apache/rocketmq/store/stats/BrokerStatsManager.java:
##########
@@ -464,47 +470,92 @@ public void incGroupAckNums(final String group, final String topic, final int in
}
public String buildStatsKey(String topic, String group) {
- StringBuilder strBuilder;
if (topic != null && group != null) {
- strBuilder = new StringBuilder(topic.length() + group.length() + 1);
- } else {
- strBuilder = new StringBuilder();
+ ConcurrentHashMap<String, String> groupMap = statsKeyByGroupCache.get(topic);
+ if (groupMap != null) {
+ String cached = groupMap.get(group);
+ if (cached != null) {
+ return cached;
+ }
+ }
+ String key = topic + "@" + group;
+ statsKeyByGroupCache.computeIfAbsent(topic, t -> new ConcurrentHashMap<>()).put(group, key);
+ return key;
}
- strBuilder.append(topic).append("@").append(group);
- return strBuilder.toString();
+ return topic + "@" + group;
}
public String buildStatsKey(String topic, int queueId) {
- StringBuilder strBuilder;
- if (topic != null) {
- strBuilder = new StringBuilder(topic.length() + 5);
- } else {
- strBuilder = new StringBuilder();
+ String[] keys = statsKeyByQueueCache.get(topic);
+ if (keys != null && queueId >= 0 && queueId < keys.length) {
+ String cached = keys[queueId];
+ if (cached != null) {
+ return cached;
+ }
}
- strBuilder.append(topic).append("@").append(queueId);
- return strBuilder.toString();
+ String key = topic + "@" + queueId;
Review Comment:
`ConcurrentHashMap` does not permit `null` keys. `buildStatsKey(String, int)` and `getLatencyKey(...)` now call `...Cache.get(topic)` without guarding `topic != null`, which can throw `NullPointerException` where the previous implementation tolerated `topic == null`. Add an early return for `topic == null` before accessing the cache (or skip caching in that case).
##########
store/src/main/java/org/apache/rocketmq/store/timer/TimerWheel.java:
##########
@@ -301,6 +310,7 @@ public void putSlot(long timeMs, long firstPos, long lastPos, int num, int magic
localBuffer.get().putLong(lastPos);
localBuffer.get().putInt(num);
localBuffer.get().putInt(magic);
+ dirty = true;
Review Comment:
The `dirty` flag can lose updates under concurrency: during `flush()`, another thread can set `dirty = true` after the `if (!dirty)` check but before `dirty = false` at the end, causing the write to be missed and the wheel state not persisted. Use an atomic state transition (e.g., `AtomicBoolean.compareAndSet(true, false)` at flush start) or synchronize/serialize modifications and flush so `dirty` cannot be cleared while writes are in-flight.
##########
store/src/main/java/org/apache/rocketmq/store/timer/TimerWheel.java:
##########
@@ -313,11 +323,13 @@ public void reviseSlot(long timeMs, long firstPos, long lastPos, boolean force)
} else {
if (IGNORE != firstPos) {
localBuffer.get().putLong(firstPos);
+ dirty = true;
Review Comment:
The `dirty` flag can lose updates under concurrency: during `flush()`, another thread can set `dirty = true` after the `if (!dirty)` check but before `dirty = false` at the end, causing the write to be missed and the wheel state not persisted. Use an atomic state transition (e.g., `AtomicBoolean.compareAndSet(true, false)` at flush start) or synchronize/serialize modifications and flush so `dirty` cannot be cleared while writes are in-flight.
##########
store/src/main/java/org/apache/rocketmq/store/stats/BrokerStatsManager.java:
##########
@@ -380,32 +386,32 @@ public void onGroupDeleted(final String group) {
this.momentStatsItemSetFallTime.delValueBySuffixKey(group, "@");
}
- public void incQueuePutNums(final String topic, final Integer queueId) {
+ public void incQueuePutNums(final String topic, final int queueId) {
if (enableQueueStat) {
this.statsTable.get(Stats.QUEUE_PUT_NUMS).addValue(buildStatsKey(topic, queueId), 1, 1);
}
}
- public void incQueuePutNums(final String topic, final Integer queueId, int num, int times) {
+ public void incQueuePutNums(final String topic, final int queueId, int num, int times) {
if (enableQueueStat) {
this.statsTable.get(Stats.QUEUE_PUT_NUMS).addValue(buildStatsKey(topic, queueId), num, times);
}
}
- public void incQueuePutSize(final String topic, final Integer queueId, final int size) {
+ public void incQueuePutSize(final String topic, final int queueId, final int size) {
if (enableQueueStat) {
this.statsTable.get(Stats.QUEUE_PUT_SIZE).addValue(buildStatsKey(topic, queueId), size, 1);
}
}
- public void incQueueGetNums(final String group, final String topic, final Integer queueId, final int incValue) {
+ public void incQueueGetNums(final String group, final String topic, final int queueId, final int incValue) {
Review Comment:
Changing public method parameters from `Integer` to primitive `int` is a source/binary compatibility change for callers compiled against the old signature, and also removes the ability to pass `null` (previously would produce a string key like `topic@null`). If this is a public API, consider keeping overloads with `Integer` (possibly deprecated) or performing the migration in a compatibility-friendly way.
##########
store/src/main/java/org/apache/rocketmq/store/timer/TimerWheel.java:
##########
@@ -313,11 +323,13 @@ public void reviseSlot(long timeMs, long firstPos, long lastPos, boolean force)
} else {
if (IGNORE != firstPos) {
localBuffer.get().putLong(firstPos);
+ dirty = true;
} else {
localBuffer.get().getLong();
}
if (IGNORE != lastPos) {
localBuffer.get().putLong(lastPos);
+ dirty = true;
Review Comment:
The `dirty` flag can lose updates under concurrency: during `flush()`, another thread can set `dirty = true` after the `if (!dirty)` check but before `dirty = false` at the end, causing the write to be missed and the wheel state not persisted. Use an atomic state transition (e.g., `AtomicBoolean.compareAndSet(true, false)` at flush start) or synchronize/serialize modifications and flush so `dirty` cannot be cleared while writes are in-flight.
##########
store/src/main/java/org/apache/rocketmq/store/timer/TimerWheel.java:
##########
@@ -55,6 +55,8 @@ protected ByteBuffer initialValue() {
};
private final int wheelLength;
+ private volatile boolean dirty;
Review Comment:
The `dirty` flag can lose updates under concurrency: during `flush()`, another thread can set `dirty = true` after the `if (!dirty)` check but before `dirty = false` at the end, causing the write to be missed and the wheel state not persisted. Use an atomic state transition (e.g., `AtomicBoolean.compareAndSet(true, false)` at flush start) or synchronize/serialize modifications and flush so `dirty` cannot be cleared while writes are in-flight.
##########
store/src/main/java/org/apache/rocketmq/store/timer/TimerWheel.java:
##########
@@ -128,17 +130,25 @@ public void flush() {
if (mappedByteBuffer == null) {
return;
}
+ if (!dirty) {
+ return;
+ }
ByteBuffer bf = localBuffer.get();
- bf.position(0);
- bf.limit(wheelLength);
- mappedByteBuffer.position(0);
- mappedByteBuffer.limit(wheelLength);
- for (int i = 0; i < wheelLength; i++) {
- if (bf.get(i) != mappedByteBuffer.get(i)) {
- mappedByteBuffer.put(i, bf.get(i));
+ int longAligned = wheelLength & ~7;
Review Comment:
The `dirty` flag can lose updates under concurrency: during `flush()`, another thread can set `dirty = true` after the `if (!dirty)` check but before `dirty = false` at the end, causing the write to be missed and the wheel state not persisted. Use an atomic state transition (e.g., `AtomicBoolean.compareAndSet(true, false)` at flush start) or synchronize/serialize modifications and flush so `dirty` cannot be cleared while writes are in-flight.
##########
store/src/main/java/org/apache/rocketmq/store/timer/TimerWheel.java:
##########
@@ -287,11 +297,10 @@ public int getSlotIndex(long timeMs) {
public void putSlot(long timeMs, long firstPos, long lastPos) {
localBuffer.get().position(getSlotIndex(timeMs) * Slot.SIZE);
- // To be compatible with previous version.
- // The previous version's precision is fixed at 1000ms and it store timeMs / 1000 in slot.
localBuffer.get().putLong(timeMs / precisionMs);
localBuffer.get().putLong(firstPos);
localBuffer.get().putLong(lastPos);
+ dirty = true;
Review Comment:
The `dirty` flag can lose updates under concurrency: during `flush()`, another thread can set `dirty = true` after the `if (!dirty)` check but before `dirty = false` at the end, causing the write to be missed and the wheel state not persisted. Use an atomic state transition (e.g., `AtomicBoolean.compareAndSet(true, false)` at flush start) or synchronize/serialize modifications and flush so `dirty` cannot be cleared while writes are in-flight.
##########
store/src/main/java/org/apache/rocketmq/store/timer/TimerWheel.java:
##########
@@ -128,17 +130,25 @@ public void flush() {
if (mappedByteBuffer == null) {
return;
}
+ if (!dirty) {
+ return;
+ }
ByteBuffer bf = localBuffer.get();
- bf.position(0);
- bf.limit(wheelLength);
- mappedByteBuffer.position(0);
- mappedByteBuffer.limit(wheelLength);
- for (int i = 0; i < wheelLength; i++) {
- if (bf.get(i) != mappedByteBuffer.get(i)) {
- mappedByteBuffer.put(i, bf.get(i));
+ int longAligned = wheelLength & ~7;
+ for (int i = 0; i < longAligned; i += 8) {
+ long local = bf.getLong(i);
+ if (local != mappedByteBuffer.getLong(i)) {
+ mappedByteBuffer.putLong(i, local);
+ }
+ }
+ for (int i = longAligned; i < wheelLength; i++) {
+ byte b = bf.get(i);
+ if (b != mappedByteBuffer.get(i)) {
+ mappedByteBuffer.put(i, b);
}
}
this.mappedByteBuffer.force();
+ dirty = false;
Review Comment:
The `dirty` flag can lose updates under concurrency: during `flush()`, another thread can set `dirty = true` after the `if (!dirty)` check but before `dirty = false` at the end, causing the write to be missed and the wheel state not persisted. Use an atomic state transition (e.g., `AtomicBoolean.compareAndSet(true, false)` at flush start) or synchronize/serialize modifications and flush so `dirty` cannot be cleared while writes are in-flight.
##########
store/src/main/java/org/apache/rocketmq/store/stats/BrokerStatsManager.java:
##########
@@ -380,32 +386,32 @@ public void onGroupDeleted(final String group) {
this.momentStatsItemSetFallTime.delValueBySuffixKey(group, "@");
}
- public void incQueuePutNums(final String topic, final Integer queueId) {
+ public void incQueuePutNums(final String topic, final int queueId) {
if (enableQueueStat) {
this.statsTable.get(Stats.QUEUE_PUT_NUMS).addValue(buildStatsKey(topic, queueId), 1, 1);
}
}
- public void incQueuePutNums(final String topic, final Integer queueId, int num, int times) {
+ public void incQueuePutNums(final String topic, final int queueId, int num, int times) {
if (enableQueueStat) {
this.statsTable.get(Stats.QUEUE_PUT_NUMS).addValue(buildStatsKey(topic, queueId), num, times);
}
}
- public void incQueuePutSize(final String topic, final Integer queueId, final int size) {
+ public void incQueuePutSize(final String topic, final int queueId, final int size) {
if (enableQueueStat) {
this.statsTable.get(Stats.QUEUE_PUT_SIZE).addValue(buildStatsKey(topic, queueId), size, 1);
}
}
- public void incQueueGetNums(final String group, final String topic, final Integer queueId, final int incValue) {
+ public void incQueueGetNums(final String group, final String topic, final int queueId, final int incValue) {
if (enableQueueStat) {
final String statsKey = buildStatsKey(topic, queueId, group);
this.statsTable.get(Stats.QUEUE_GET_NUMS).addValue(statsKey, incValue, 1);
}
}
- public void incQueueGetSize(final String group, final String topic, final Integer queueId, final int incValue) {
+ public void incQueueGetSize(final String group, final String topic, final int queueId, final int incValue) {
Review Comment:
Changing public method parameters from `Integer` to primitive `int` is a source/binary compatibility change for callers compiled against the old signature, and also removes the ability to pass `null` (previously would produce a string key like `topic@null`). If this is a public API, consider keeping overloads with `Integer` (possibly deprecated) or performing the migration in a compatibility-friendly way.
##########
store/src/main/java/org/apache/rocketmq/store/stats/BrokerStatsManager.java:
##########
@@ -523,14 +574,30 @@ public void incGroupGetLatency(final String group, final String topic, final int
}
public void incTopicPutLatency(final String topic, final int queueId, final int incValue) {
- StringBuilder statsKey;
+ String statsKey = getLatencyKey(topic, queueId);
+ this.statsTable.get(Stats.TOPIC_PUT_LATENCY).addValue(statsKey, incValue, 1);
+ }
+
+ private String getLatencyKey(final String topic, final int queueId) {
+ String[] keys = latencyKeyCache.get(topic);
+ if (keys != null && queueId < keys.length) {
+ String k = keys[queueId];
+ if (k != null) {
+ return k;
+ }
+ }
Review Comment:
`ConcurrentHashMap` does not permit `null` keys. `buildStatsKey(String, int)` and `getLatencyKey(...)` now call `...Cache.get(topic)` without guarding `topic != null`, which can throw `NullPointerException` where the previous implementation tolerated `topic == null`. Add an early return for `topic == null` before accessing the cache (or skip caching in that case).
##########
store/src/main/java/org/apache/rocketmq/store/stats/BrokerStatsManager.java:
##########
@@ -380,32 +386,32 @@ public void onGroupDeleted(final String group) {
this.momentStatsItemSetFallTime.delValueBySuffixKey(group, "@");
}
- public void incQueuePutNums(final String topic, final Integer queueId) {
+ public void incQueuePutNums(final String topic, final int queueId) {
Review Comment:
Changing public method parameters from `Integer` to primitive `int` is a source/binary compatibility change for callers compiled against the old signature, and also removes the ability to pass `null` (previously would produce a string key like `topic@null`). If this is a public API, consider keeping overloads with `Integer` (possibly deprecated) or performing the migration in a compatibility-friendly way.
##########
store/src/main/java/org/apache/rocketmq/store/stats/BrokerStatsManager.java:
##########
@@ -380,32 +386,32 @@ public void onGroupDeleted(final String group) {
this.momentStatsItemSetFallTime.delValueBySuffixKey(group, "@");
}
- public void incQueuePutNums(final String topic, final Integer queueId) {
+ public void incQueuePutNums(final String topic, final int queueId) {
if (enableQueueStat) {
this.statsTable.get(Stats.QUEUE_PUT_NUMS).addValue(buildStatsKey(topic, queueId), 1, 1);
}
}
- public void incQueuePutNums(final String topic, final Integer queueId, int num, int times) {
+ public void incQueuePutNums(final String topic, final int queueId, int num, int times) {
if (enableQueueStat) {
this.statsTable.get(Stats.QUEUE_PUT_NUMS).addValue(buildStatsKey(topic, queueId), num, times);
}
}
- public void incQueuePutSize(final String topic, final Integer queueId, final int size) {
+ public void incQueuePutSize(final String topic, final int queueId, final int size) {
Review Comment:
Changing public method parameters from `Integer` to primitive `int` is a source/binary compatibility change for callers compiled against the old signature, and also removes the ability to pass `null` (previously would produce a string key like `topic@null`). If this is a public API, consider keeping overloads with `Integer` (possibly deprecated) or performing the migration in a compatibility-friendly way.
##########
store/src/main/java/org/apache/rocketmq/store/stats/BrokerStatsManager.java:
##########
@@ -141,6 +142,11 @@ public class BrokerStatsManager {
private ScheduledExecutorService cleanResourceExecutor;
private final HashMap<String, StatsItemSet> statsTable = new HashMap<>();
+ private final ConcurrentHashMap<String, String[]> latencyKeyCache = new ConcurrentHashMap<>();
+ private final ConcurrentHashMap<String, String[]> statsKeyByQueueCache = new ConcurrentHashMap<>();
+ private final ConcurrentHashMap<String, ConcurrentHashMap<String, String>> statsKeyByGroupCache = new ConcurrentHashMap<>();
+ private final ConcurrentHashMap<String, ConcurrentHashMap<String, String[]>> statsKeyByTopicQueueGroupCache = new ConcurrentHashMap<>();
+ private final ConcurrentHashMap<String, ConcurrentHashMap<String, String[]>> statsKeyByQueueTopicGroupCache = new ConcurrentHashMap<>();
Review Comment:
These caches are unbounded and appear to retain entries indefinitely per topic/group/queue, which can lead to memory growth in long-running brokers with high churn or many unique groups/topics. Consider clearing relevant cache entries in existing lifecycle hooks (e.g., on topic/group deletion) and/or introducing size bounds/eviction to prevent unbounded retention.
##########
store/src/main/java/org/apache/rocketmq/store/stats/BrokerStatsManager.java:
##########
@@ -380,32 +386,32 @@ public void onGroupDeleted(final String group) {
this.momentStatsItemSetFallTime.delValueBySuffixKey(group, "@");
}
- public void incQueuePutNums(final String topic, final Integer queueId) {
+ public void incQueuePutNums(final String topic, final int queueId) {
if (enableQueueStat) {
this.statsTable.get(Stats.QUEUE_PUT_NUMS).addValue(buildStatsKey(topic, queueId), 1, 1);
}
}
- public void incQueuePutNums(final String topic, final Integer queueId, int num, int times) {
+ public void incQueuePutNums(final String topic, final int queueId, int num, int times) {
Review Comment:
Changing public method parameters from `Integer` to primitive `int` is a source/binary compatibility change for callers compiled against the old signature, and also removes the ability to pass `null` (previously would produce a string key like `topic@null`). If this is a public API, consider keeping overloads with `Integer` (possibly deprecated) or performing the migration in a compatibility-friendly way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org