You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@rocketmq.apache.org by "Aias00 (via GitHub)" <gi...@apache.org> on 2026/07/10 04:58:50 UTC
[PR] [ISSUE #10559] Support wildcard pattern matching for LiteTopic subscriptions [rocketmq]
Aias00 opened a new pull request, #10607:
URL: https://github.com/apache/rocketmq/pull/10607
## Which issue does this PR close?
Closes #10559.
## What's the purpose of this PR
Add NATS-style wildcard pattern matching so a wildcard LiteTopic consumer group can subscribe to a **subset** of lite-topics under its parent topic (instead of always receiving all of them), using `__` as the segment separator. Full backward compatibility is preserved: wildcard groups without patterns behave identically to before.
## Brief changelog
- **`LitePatternMatcher` (new, `common`)** — the matching engine: `validate`, `matches` (recursive, no regex), `matchesAny`, `expand`. `*` matches exactly one `__`-delimited segment; `**` matches one or more trailing segments (like NATS `>`).
- **`LiteSubscription`** — add in-memory `wildcardPatterns` set (rebuilt on reconnect, same as `liteTopicSet`).
- **`LiteSubscriptionRegistry[Impl]`** — a wildcard group with non-empty validated patterns enters **pattern mode**: eagerly expand against `collectByParentTopic` and register real lmqNames. Empty patterns keep legacy receive-all behavior (synthetic `topic@group` key) unchanged. `getAllSubscriber` always merges the synthetic-key clients for any wildcard group, so mixed pattern/legacy groups deliver to both. New `reexpandWildcardPatterns` picks up lite-topics created after the initial subscription.
- **`LiteSubscriptionCtlProcessor`** — route wildcard `COMPLETE_ADD` through `toWildcardPatterns` (patterns ride the existing `liteTopicSet` — **no wire protocol change**). Non-empty but all-invalid pattern input is **rejected** instead of silently widening to receive-all.
- **`LiteEventDispatcher.doFullDispatchForWildcardGroup`** — replace the O(N) `forEachLiteTopic` scan with O(M) `collectByParentTopic`; re-expand patterns for new LMQs and clear backlog per pattern-mode client.
### Pattern semantics
| Pattern | Matches | Does NOT match |
|---|---|---|
| `pay__refund` | `pay__refund` | `pay__refund__notify`, `notify__refund` |
| `pay__*` | `pay__refund`, `pay__success` | `pay__refund__notify` |
| `*__refund` | `pay__refund`, `notify__refund` | `pay__refund__notify` |
| `pay__*__notify` | `pay__refund__notify`, `pay__success__notify` | `pay__refund` |
| `**` | all | — |
| `pay__**` | `pay__refund`, `pay__refund__notify` | `notify__refund` |
## How to test / verify
- New `LitePatternMatcherTest` (common): all six issue rows (match + non-match), `validate` accept/reject cases, `expand`/`matchesAny` semantics — 18 tests.
- Updated registry/dispatcher/processor tests, split into legacy + pattern-mode variants; added mixed-group delivery, all-invalid-pattern rejection, and lazy re-expansion coverage.
- `mvn -pl common,broker -am test` — lite suite green (138 tests, 0 failures); checkstyle clean on common + broker.
- Performance sanity: `matches` ≈ 1µs/call (target <5µs); `expand(1k candidates × 5 patterns)` ≈ 360µs (target <10ms).
## Scope
Broker + common only, matching the issue's "Key Changes" table. Patterns are carried via the existing `LiteSubscriptionDTO.liteTopicSet`; the broker distinguishes them via `isWildcardGroup()`. A consumer/proxy API (`subscribeLite`) and a true LMQ-creation listener for sub-second new-topic pickup are intentionally deferred to follow-up PRs.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE #10559] Support wildcard pattern matching for LiteTopic subscriptions [rocketmq]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #10607:
URL: https://github.com/apache/rocketmq/pull/10607#discussion_r3556479562
##########
broker/src/main/java/org/apache/rocketmq/broker/lite/LiteSubscriptionRegistryImpl.java:
##########
@@ -124,8 +126,27 @@ public void removePartialSubscription(String clientId, String group, String topi
@Override
public void addCompleteSubscription(String clientId, String group, String topic, Set<String> lmqNameAll, long version) {
+ addCompleteSubscription(clientId, group, topic, lmqNameAll, Collections.emptySet(), version);
+ }
+
+ @Override
+ public void addCompleteSubscription(String clientId, String group, String topic, Set<String> lmqNameAll,
+ Set<String> wildcardPatterns, long version) {
Set<String> lmqNameNew;
- if (LiteMetadataUtil.isWildcardGroup(group, brokerController)) {
+ boolean isWildcardGroup = LiteMetadataUtil.isWildcardGroup(group, brokerController);
+ boolean isPatternMode = isWildcardGroup && wildcardPatterns != null && !wildcardPatterns.isEmpty();
+
+ LiteSubscription thisSub = getOrCreateLiteSubscription(clientId, group, topic);
+ if (isPatternMode) {
+ // Persist the patterns (authoritative intent) and eagerly expand against existing
+ // lite-topics under the parent topic. The expanded lmqNames are registered as a normal
+ // subscription; the topic@group synthetic key is NOT used for pattern-mode groups.
+ thisSub.setWildcardPatterns(wildcardPatterns);
+ lmqNameNew = expandWildcardPatterns(topic, wildcardPatterns);
+ markWildcardGroup(topic, group);
+ } else if (isWildcardGroup) {
+ // Legacy wildcard group: receive all lite-topics under the parent topic via the
+ // synthetic topic@group key.
lmqNameNew = Collections.singleton(mockLmqNameForWildcardGroup(topic, group));
markWildcardGroup(topic, group);
Review Comment:
`addCompleteSubscription` only calls `setWildcardPatterns(...)` in pattern-mode. If a client previously subscribed in pattern mode and later re-subscribes in legacy mode (empty patterns) or as a non-wildcard group, the old `wildcardPatterns` remain on the `LiteSubscription`. This can misclassify the client as pattern-mode (e.g., `doFullDispatchForWildcardGroup` will treat it as pattern-mode) and can also cause `reexpandWildcardPatterns` to re-register matched LMQs unexpectedly.
##########
broker/src/main/java/org/apache/rocketmq/broker/processor/LiteSubscriptionCtlProcessor.java:
##########
@@ -94,8 +95,16 @@ public RemotingCommand processRequest(ChannelHandlerContext ctx, RemotingCommand
case COMPLETE_ADD:
checkConsumeEnable(group);
this.liteSubscriptionRegistry.updateClientChannel(clientId, ctx.channel());
- this.liteSubscriptionRegistry.addCompleteSubscription(clientId, group, topic, lmqNameSet,
- entry.getVersion());
+ if (LiteMetadataUtil.isWildcardGroup(group, brokerController)) {
+ // For a wildcard group, liteTopicSet carries pattern strings (not literal
+ // lmqNames); pass them through verbatim and let the registry expand them.
+ Set<String> wildcardPatterns = toWildcardPatterns(entry);
+ this.liteSubscriptionRegistry.addCompleteSubscription(clientId, group, topic, lmqNameSet,
+ wildcardPatterns, entry.getVersion());
+ } else {
+ this.liteSubscriptionRegistry.addCompleteSubscription(clientId, group, topic, lmqNameSet,
+ entry.getVersion());
+ }
Review Comment:
In the wildcard `COMPLETE_ADD` path, `lmqNameSet` is derived by converting the incoming `liteTopicSet` entries with `LiteUtil.toLmqName(...)`. But for wildcard groups, `liteTopicSet` carries wildcard pattern strings and should not be treated as literal child topic names. Passing a pattern-derived `lmqNameSet` into the registry is confusing and makes it easy to accidentally start relying on invalid LMQ names in future changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE #10559] Support wildcard pattern matching for LiteTopic subscriptions [rocketmq]
Posted by "Aias00 (via GitHub)" <gi...@apache.org>.
Aias00 closed pull request #10607: [ISSUE #10559] Support wildcard pattern matching for LiteTopic subscriptions
URL: https://github.com/apache/rocketmq/pull/10607
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE #10559] Support wildcard pattern matching for LiteTopic subscriptions [rocketmq]
Posted by "f1amingo (via GitHub)" <gi...@apache.org>.
f1amingo commented on PR #10607:
URL: https://github.com/apache/rocketmq/pull/10607#issuecomment-5002218235
Thanks for this proposal. I'd like to raise a design-level concern.
**Scenario-architecture mismatch**
LiteTopic was designed (per RIP-83) as a session-scoped ephemeral channel — AI sessions, IoT debugging, game rooms — with a single queue, sequential pull, no retry, and ~200 QPS ceiling. The PR builds its examples around business event routing (pay__refund, notify__**), which implicitly targets thousands of QPS, parallel consumption, and retry guarantees. These diverge on throughput, consumption model, and reliability — not as implementation issues, but as a fundamental design mismatch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE #10559] Support wildcard pattern matching for LiteTopic subscriptions [rocketmq]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #10607:
URL: https://github.com/apache/rocketmq/pull/10607#issuecomment-4932286488
## [Codecov](https://app.codecov.io/gh/apache/rocketmq/pull/10607?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `80.45977%` with `34 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 48.24%. Comparing base ([`0e4ccf1`](https://app.codecov.io/gh/apache/rocketmq/commit/0e4ccf1b669161453c9381af92652ae2cb9fa4e7?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`4d51ad9`](https://app.codecov.io/gh/apache/rocketmq/commit/4d51ad9997a2168571fab8a294c17dedd816066e?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/rocketmq/pull/10607?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...etmq/broker/lite/LiteSubscriptionRegistryImpl.java](https://app.codecov.io/gh/apache/rocketmq/pull/10607?src=pr&el=tree&filepath=broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Frocketmq%2Fbroker%2Flite%2FLiteSubscriptionRegistryImpl.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-YnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9icm9rZXIvbGl0ZS9MaXRlU3Vic2NyaXB0aW9uUmVnaXN0cnlJbXBsLmphdmE=) | 75.86% | [4 Missing and 10 partials :warning: ](https://app.codecov.io/gh/apache/rocketmq/pull/10607?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../apache/rocketmq/common/lite/LiteSubscription.java](https://app.codecov.io/gh/apache/rocketmq/pull/10607?src=pr&el=tree&filepath=common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Frocketmq%2Fcommon%2Flite%2FLiteSubscription.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9jb21tb24vbGl0ZS9MaXRlU3Vic2NyaXB0aW9uLmphdmE=) | 0.00% | [7 Missing :warning: ](https://app.codecov.io/gh/apache/rocketmq/pull/10607?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...pache/rocketmq/common/lite/LitePatternMatcher.java](https://app.codecov.io/gh/apache/rocketmq/pull/10607?src=pr&el=tree&filepath=common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Frocketmq%2Fcommon%2Flite%2FLitePatternMatcher.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9jb21tb24vbGl0ZS9MaXRlUGF0dGVybk1hdGNoZXIuamF2YQ==) | 89.65% | [2 Missing and 4 partials :warning: ](https://app.codecov.io/gh/apache/rocketmq/pull/10607?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/rocketmq/broker/lite/LiteEventDispatcher.java](https://app.codecov.io/gh/apache/rocketmq/pull/10607?src=pr&el=tree&filepath=broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Frocketmq%2Fbroker%2Flite%2FLiteEventDispatcher.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-YnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9icm9rZXIvbGl0ZS9MaXRlRXZlbnREaXNwYXRjaGVyLmphdmE=) | 82.75% | [3 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/rocketmq/pull/10607?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...broker/processor/LiteSubscriptionCtlProcessor.java](https://app.codecov.io/gh/apache/rocketmq/pull/10607?src=pr&el=tree&filepath=broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Frocketmq%2Fbroker%2Fprocessor%2FLiteSubscriptionCtlProcessor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-YnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9icm9rZXIvcHJvY2Vzc29yL0xpdGVTdWJzY3JpcHRpb25DdGxQcm9jZXNzb3IuamF2YQ==) | 90.90% | [1 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/rocketmq/pull/10607?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## develop #10607 +/- ##
=============================================
- Coverage 48.27% 48.24% -0.04%
- Complexity 13436 13484 +48
=============================================
Files 1378 1379 +1
Lines 100820 100972 +152
Branches 13041 13089 +48
=============================================
+ Hits 48670 48711 +41
- Misses 46208 46285 +77
- Partials 5942 5976 +34
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/rocketmq/pull/10607?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ISSUE #10559] Support wildcard pattern matching for LiteTopic subscriptions [rocketmq]
Posted by "Aias00 (via GitHub)" <gi...@apache.org>.
Aias00 commented on PR #10607:
URL: https://github.com/apache/rocketmq/pull/10607#issuecomment-5015815242
> Thanks for this proposal. I'd like to raise a design-level concern. **Scenario-architecture mismatch** LiteTopic was designed (per RIP-83) as a session-scoped ephemeral channel — AI sessions, IoT debugging, game rooms — with a single queue, sequential pull, no retry, and ~200 QPS ceiling. The PR builds its examples around business event routing (pay__refund, notify__**), which implicitly targets thousands of QPS, parallel consumption, and retry guarantees. These diverge on throughput, consumption model, and reliability — not as implementation issues, but as a fundamental design mismatch.
thx, i understand, i will close this pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org