[I] [Enhancement] Support Wildcard Pattern Matching for LiteTopic Consumer Subscriptions [rocketmq]

["leizhiyuan (via GitHub)" <gi...@apache.org>]

leizhiyuan opened a new issue, #10559:
URL: https://github.com/apache/rocketmq/issues/10559

   ### Before Creating the Enhancement Request
   
   - [x] I have confirmed that this should be classified as an enhancement rather than a bug/feature.
   
   
   ### Summary
   
   Support wildcard pattern matching for LiteTopic consumer subscriptions, enabling hierarchical topic organization and fine-grained subscriber-side filtering.
   
   ### Motivation
   
   PR #10204 introduced Lite wildcard groups, but the current implementation has several limitations:
   
   1. **No consumer-side pattern filtering** — A wildcard consumer group receives all LiteTopics under the parent topic. Consumers cannot express which subset they want; they always get everything.
   
   2. **Full LMQ scan on every dispatch cycle** — `doFullDispatchForWildcardGroup` calls `forEachLiteTopic()` which scans the entire ConsumeQueue table (O(N) over all LMQs), not just the topics under the bound parent topic.
   
   3. **No hierarchical topic organization** — LiteTopic names are flat strings with no way to express logical grouping (e.g., "all payment-related subtopics").
   
   This makes it difficult to build systems where different consumers within the same group handle different subsets of events.
   
   ### Describe the Solution You'd Like
   
   **Wildcard pattern matching** for LiteTopic consumer subscriptions, inspired by NATS subject matching.
   
   #### Key Design Decisions
   
   **Separator: `__` (double underscore)**
   - `.` is already used in RocketMQ topic names (e.g., `%LMQ%$parent$child`) and in ConsumeQueue file paths
   - `__` is visually distinctive, rarely appears in topic names, and avoids conflicts
   
   **Pattern Tokens**
   
   | Token | Meaning | Position | NATS Equivalent |
   |-------|---------|----------|-----------------|
   | `*`   | Matches exactly one `__`-delimited segment | Anywhere | `*` |
   | `**`  | Matches one or more segments | Must be at the end | `>` |
   
   **Examples** (parent topic = `order_events`)
   
   | Pattern | Matches | Does Not Match |
   |---------|---------|----------------|
   | `pay__refund` | `pay__refund` | `pay__refund__notify`, `notify__refund` |
   | `pay__*` | `pay__refund`, `pay__success` | `pay__refund__notify` |
   | `*__refund` | `pay__refund`, `notify__refund` | `pay__refund__notify` |
   | `pay__*__notify` | `pay__refund__notify`, `pay__success__notify` | `pay__refund` |
   | `**` | `pay`, `pay__refund`, `pay__refund__notify` | (matches all) |
   | `pay__**` | `pay__refund`, `pay__refund__notify` | `notify__refund` |
   
   #### Implementation Plan
   
   **Core matching engine:** `LitePatternMatcher` in `common` module, providing three static methods:
   - `matches(pattern, liteTopic)` — recursive segment-by-segment matching, no regex
   - `expand(pattern, candidates)` — expand a pattern against a set of candidate topics
   - `validate(pattern)` — validate pattern syntax
   
   **Two-phase subscription expansion:**
   1. **Eager expansion** at `COMPLETE_ADD` time — expand all patterns against existing LiteTopics under the parent topic, register matched topics to `liteTopic2Group`
   2. **Lazy re-expansion** when new LiteTopics are created via `onRegister()` — check if the new topic matches any client's patterns, add to subscription if so
   
   **Dispatch optimization:** Replace full LMQ scan (`forEachLiteTopic`, O(N)) with targeted iteration via `collectByParentTopic` (O(M), where M = topics under the parent topic).
   
   **Multi-pattern support (v1):** `liteTopicSet` is a `Set<String>`, so multiple patterns can be sent in a single `COMPLETE_ADD` request. The union of all matched topics becomes the client's effective subscription.
   
   **Protocol compatibility:** Reuse `LiteSubscriptionDTO.liteTopicSet` to carry pattern strings for wildcard groups. No wire protocol changes needed — the broker distinguishes via `isWildcardGroup()`.
   
   #### Consumer API Example
   
   ```java
   DefaultLitePullConsumer consumer = new DefaultLitePullConsumer("order_consumer_group");
   consumer.setLiteBindTopic("order_events");
   consumer.subscribeLite("pay__*");       // single-level payment events
   consumer.subscribeLite("notify__**");   // all notification events (multi-level)
   ```
   
   #### Key Changes
   
   | Component | Change |
   |-----------|--------|
   | `LitePatternMatcher` (new) | Core matching engine in `common` |
   | `LiteSubscription` | + `wildcardPatterns: Set<String>` |
   | `LiteSubscriptionRegistryImpl` | Multi-pattern expansion in `addCompleteSubscription` |
   | `LiteEventDispatcher` | Lazy pattern re-expansion in `onRegister`; dispatch optimization |
   
   ### Describe Alternatives You've Considered
   
   | Alternative | Why Rejected |
   |-------------|-------------|
   | Store compiled Trie per wildcard group | Over-engineering for v1; `String.split` + recursive matching is fast enough for < 10K topics |
   | Support `**` in the middle of a pattern (e.g., `order__**__refund`) | NATS does not support this; greedy vs. non-greedy makes matching ambiguous; deferred to v2 |
   | Use `.` as separator | Conflicts with existing topic naming conventions in RocketMQ |
   | Per-message pattern matching (no eager expansion) | Would require checking patterns on every `doDispatch` call; eager expansion is more efficient |
   | Add a dedicated `pattern` field to `LiteSubscriptionDTO` | Unnecessary; reusing `liteTopicSet` is simpler and avoids wire protocol changes |
   
   ### Additional Context
   
   - Inspired by NATS subject-based addressing and MQTT topic filters (with `+` / `#`), but using `__` separator to avoid conflicts with RocketMQ's existing naming conventions
   - Aligned with FlowMQ's "unified topic routing" philosophy — one topic namespace, multiple protocol bindings, unified wildcard matching
   - Performance target: `matches()` < 5μs per call, `expand()` < 10ms for 1,000 candidates
   - Full backward compatibility: wildcard groups without patterns continue to receive all LiteTopics under the parent topic (existing behavior)
   - Registers via `LiteSubscriptionCtlRequestBody` without any wire protocol changes
   - Reuses the existing `collectByParentTopic` API for dispatch optimization


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Enhancement] Support Wildcard Pattern Matching for LiteTopic Consumer Subscriptions [rocketmq]

["f1amingo (via GitHub)" <gi...@apache.org>]

f1amingo commented on issue #10559:
URL: https://github.com/apache/rocketmq/issues/10559#issuecomment-5001416193

   Thanks for this proposal. I'd like to raise a design-level concern.
   ### Scenario-architecture mismatch
   LiteTopic was designed (per RIP-83) as a session-scoped ephemeral channel — AI sessions, IoT debugging, game rooms — with a single queue, sequential pull, no retry, and ~200 QPS ceiling. The PR builds its examples around business event routing (pay__refund, notify__**), which implicitly targets thousands of QPS, parallel consumption, and retry guarantees. These diverge on throughput, consumption model, and reliability — not as implementation issues, but as a fundamental design mismatch.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Enhancement] Support Wildcard Pattern Matching for LiteTopic Consumer Subscriptions [rocketmq]

["RockteMQ-AI (via GitHub)" <gi...@apache.org>]

RockteMQ-AI commented on issue #10559:
URL: https://github.com/apache/rocketmq/issues/10559#issuecomment-4841659511

   **Issue Evaluation**
   
   Category: `type/enhancement` | Status: **Evaluated**
   
   **Feasibility:** Feasible
   **Scope:** Broker dispatch layer (LiteTopic/LMQ), consumer subscription protocol, common module (new `LitePatternMatcher`)
   **Compatibility:** Backward-compatible — existing exact-match subscriptions continue to work; pattern matching is opt-in.
   
   **Assessment:**
   
   This is a well-structured proposal that addresses real limitations in the current LiteTopic wildcard implementation (introduced in PR #10204). The three identified gaps — no consumer-side filtering, full LMQ scan overhead, and flat topic naming — are legitimate pain points for hierarchical event architectures.
   
   Key observations:
   - The `__` separator choice is pragmatic, avoiding conflicts with existing `.` usage in topic names and ConsumeQueue paths.
   - The NATS-inspired `*` / `**` pattern semantics are well-understood and have proven production viability in other messaging systems.
   - The two-phase expansion approach (eager at `COMPLETE_ADD`, lazy at dispatch) is a reasonable design that balances freshness with performance.
   - The `LitePatternMatcher` in `common` with recursive segment matching (no regex) is the right performance-conscious approach.
   
   **Potential concerns for discussion:**
   1. **Pattern index maintenance** — The trie-based index for fast pattern→topic resolution adds memory overhead per broker. Consider the upper bound when thousands of patterns are active.
   2. **Cross-module coordination** — If `rocketmq-clients` also needs pattern expression support (for proxy-mode consumption), the pattern syntax should be formalized as a protocol-level contract, not just a broker-internal format.
   3. **Validation edge cases** — Patterns like `**__something` (non-terminal `**`) should be explicitly rejected; the `validate()` method needs clear rules.
   
   Overall, this is a solid enhancement with a clear implementation path. Recommend proceeding to spec/PR phase after community discussion on the concerns above.
   
   ---
   *Automated evaluation by RockteMQ-AI*


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org