You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@rocketmq.apache.org by "imjyz (via GitHub)" <gi...@apache.org> on 2026/07/28 13:24:46 UTC

[I] [Enhancement] Expose TopicRouteData and queue-aware receive APIs in SimpleConsumer [rocketmq-clients]

imjyz opened a new issue, #1315:
URL: https://github.com/apache/rocketmq-clients/issues/1315

   ### Before Creating the Enhancement Request
   
   - [x] I have confirmed that this should be classified as an enhancement rather than a bug/feature.
   
   
   ### Programming Language of the Client
   
   Java
   
   ### Summary
   
   The current SimpleConsumer only provides generic receive methods and selects the topic and message queue internally.
   当前 SimpleConsumer 只提供通用 receive 方法,Topic 和 MessageQueue 均由 SDK 内部选择。
   
   ```java
   CompletableFuture<List<MessageView>> receiveAsync(
       int maxMessageNum,
       Duration invisibleDuration);
   ```
   
   Because callers cannot obtain the topic route or control the target queue, they cannot determine how many concurrent receive requests are required to guarantee deterministic low latency.
   由于调用方无法获取 Topic 路由,也无法控制 Receive 对应的 Queue,因此无法确定需要维持多少个并发 Receive 请求才能保证确定性的低延时。
   
   ### Motivation
   
   Even if a topic has four queues and the caller continuously maintains four concurrent receive requests, those requests are not guaranteed to cover the four queues individually.
   即使 Topic 有 4 个 Queue,并持续维持 4 个并发 Receive 请求,也不能保证这些请求分别覆盖 4 个 Queue。
   
   Increasing receive concurrency only reduces the probability of an uncovered queue while increasing empty polling traffic. It cannot provide deterministic queue coverage or latency.
   继续增加 Receive 并发只能降低 Queue 未覆盖的概率,同时增加空轮询流量,无法提供确定性的 Queue 覆盖和消费延时。
   
   Callers need to know the topic route and explicitly select the target queue so that they can maintain one independent receive chain per queue.
   调用方需要获取 Topic 路由并明确指定目标 Queue,从而为每个 Queue 维护独立的 Receive 链路。
   
   ### Describe the Solution You'd Like
   
   The Java client already provides TopicRouteData and MessageQueueImpl. We propose exposing the following APIs through SimpleConsumer:
   Java 客户端已经存在 TopicRouteData 和 MessageQueueImpl,建议通过 SimpleConsumer 暴露以下接口:
   
   ```java
   TopicRouteData getTopicRouteData(String topic)
       throws ClientException;
   
   CompletableFuture<List<MessageView>> receiveAsync(
       MessageQueueImpl messageQueue,
       int maxMessageNum,
       Duration invisibleDuration);
   ```
   
   A synchronous queue-aware receive overload can also be provided.
   同时可以提供对应的同步 receive 重载。
   
   With these APIs, callers can discover all queues for a topic and maintain one independent receive chain per queue, providing deterministic queue coverage and low-latency consumption without blindly increasing receive concurrency.
   通过这些接口,调用方可以获取 Topic 下的全部 Queue,并为每个 Queue 维护独立 Receive 链路,从而获得确定性的 Queue 覆盖和低延时消费,而不需要盲目增加 Receive 并发数。
   
   ### Describe Alternatives You've Considered
   
   \-
   
   ### Additional Context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Enhancement] Expose TopicRouteData and queue-aware receive APIs in SimpleConsumer [rocketmq-clients]

Posted by "RockteMQ-AI (via GitHub)" <gi...@apache.org>.
RockteMQ-AI commented on issue #1315:
URL: https://github.com/apache/rocketmq-clients/issues/1315#issuecomment-5394978396

   **Issue Evaluation**
   
   Category: `enhancement` | Status: **Evaluated**
   
   **Feasibility:** Feasible
   **Scope:** Java SimpleConsumer API
   **Compatibility:** Additive API — new overloaded methods, existing methods unchanged.
   
   Reasonable enhancement. Exposing `TopicRouteData` and queue-aware receive methods gives applications finer control over message consumption, enabling use cases like targeted consumption from specific queues or custom load balancing strategies.
   
   Key considerations:
   - The new API should clearly document the contract (e.g., what happens if the specified queue becomes unavailable).
   - Consider whether `TopicRouteData` should be a stable public type or an internal detail exposed through a wrapper.
   - Ensure the existing receive methods continue to work as before (no behavioral changes).
   
   ---
   *Automated evaluation by github-manager*
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Enhancement] Expose TopicRouteData and queue-aware receive APIs in SimpleConsumer [rocketmq-clients]

Posted by "RockteMQ-AI (via GitHub)" <gi...@apache.org>.
RockteMQ-AI commented on issue #1315:
URL: https://github.com/apache/rocketmq-clients/issues/1315#issuecomment-5370361476

   Hi @imjyz, thanks for the suggestion!
   
   This has been classified as an **enhancement/feature request** and noted for the maintainers to review.
   
   If you have a concrete implementation proposal or would like to contribute a PR, the community would greatly appreciate it!
   
   ---
   🤖 *Automated issue analysis by github-manager*


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Enhancement] Expose TopicRouteData and queue-aware receive APIs in SimpleConsumer [rocketmq-clients]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #1315:
URL: https://github.com/apache/rocketmq-clients/issues/1315#issuecomment-5851389841

   This issue was closed because it has been inactive for 3 days since being marked as stale.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Enhancement] Expose TopicRouteData and queue-aware receive APIs in SimpleConsumer [rocketmq-clients]

Posted by "RockteMQ-AI (via GitHub)" <gi...@apache.org>.
RockteMQ-AI commented on issue #1315:
URL: https://github.com/apache/rocketmq-clients/issues/1315#issuecomment-5227425288

   **Feedback Timeout**\n\nNo feedback has been received on the proposed spec within 72 hours. The issue will remain open for community review, but the automated fix workflow is being paused.\n\nIf you'd like to proceed with this fix, please comment with:\n- `/approve` - Generate a PR for this fix\n- `/revise` - Request changes to the spec\n- `/reject` - Cancel this fix\n\nAlternatively, community members can submit their own PR to address this issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Enhancement] Expose TopicRouteData and queue-aware receive APIs in SimpleConsumer [rocketmq-clients]

Posted by "RockteMQ-AI (via GitHub)" <gi...@apache.org>.
RockteMQ-AI commented on issue #1315:
URL: https://github.com/apache/rocketmq-clients/issues/1315#issuecomment-5364741367

   **Issue Evaluation**
   
   Category: `enhancement` | Status: **Evaluated**
   
   **Feasibility:** feasible
   **Scope:** Java gRPC client
   **Compatibility:** Additive API change, no breaking changes.
   
   This is a reasonable enhancement. Exposing `TopicRouteData` and queue-aware receive APIs in the SimpleConsumer would enable advanced use cases like:
   - Custom queue-level load balancing
   - Manual partition assignment
   - Queue-level monitoring and diagnostics
   
   The current SimpleConsumer abstracts away queue details, which is convenient for simple cases but limiting for advanced users who need finer control.
   
   ---
   *Automated evaluation by RockteMQ-AI*
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Enhancement] Expose TopicRouteData and queue-aware receive APIs in SimpleConsumer [rocketmq-clients]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #1315:
URL: https://github.com/apache/rocketmq-clients/issues/1315#issuecomment-5805368620

   This issue is stale because it has been open for 30 days with no activity. It will be closed in 3 days if no further activity occurs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Enhancement] Expose TopicRouteData and queue-aware receive APIs in SimpleConsumer [rocketmq-clients]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #1315: [Enhancement] Expose TopicRouteData and queue-aware receive APIs in SimpleConsumer
URL: https://github.com/apache/rocketmq-clients/issues/1315


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Enhancement] Expose TopicRouteData and queue-aware receive APIs in SimpleConsumer [rocketmq-clients]

Posted by "RockteMQ-AI (via GitHub)" <gi...@apache.org>.
RockteMQ-AI commented on issue #1315:
URL: https://github.com/apache/rocketmq-clients/issues/1315#issuecomment-5231197915

   **Issue Evaluation**
   
   Category: `enhancement` | Status: **Evaluated**
   
   **Feasibility:** Feasible but needs discussion — exposing internal routing data changes the API surface.
   
   **Scope:** SimpleConsumer API. Requires exposing `TopicRouteData` and adding queue-aware receive APIs.
   
   **Compatibility:** Additive API change. Must ensure the exposed routing data is stable and does not leak internal implementation details.
   
   This enhancement could be useful for advanced consumers needing fine-grained control. The design should be reviewed to avoid tight coupling between client API and internal routing mechanics.
   
   ---
   *Automated evaluation by RockteMQ-AI*


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Enhancement] Expose TopicRouteData and queue-aware receive APIs in SimpleConsumer [rocketmq-clients]

Posted by "RockteMQ-AI (via GitHub)" <gi...@apache.org>.
RockteMQ-AI commented on issue #1315:
URL: https://github.com/apache/rocketmq-clients/issues/1315#issuecomment-5231195913

   **Issue Evaluation**
   
   Category: `enhancement` | Status: **Evaluated**
   
   **Feasibility:** Feasible but needs discussion — exposing internal routing data changes the API surface.
   
   **Scope:** SimpleConsumer API. Requires exposing `TopicRouteData` and adding queue-aware receive APIs.
   
   **Compatibility:** Additive API change. Must ensure the exposed routing data is stable and does not leak internal implementation details that may change.
   
   This enhancement could be useful for advanced consumers that need fine-grained control over message receiving. The design should be reviewed to ensure it does not create tight coupling between the client API and internal routing mechanics.
   
   ---
   *Automated evaluation by RockteMQ-AI*


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Enhancement] Expose TopicRouteData and queue-aware receive APIs in SimpleConsumer [rocketmq-clients]

Posted by "RockteMQ-AI (via GitHub)" <gi...@apache.org>.
RockteMQ-AI commented on issue #1315:
URL: https://github.com/apache/rocketmq-clients/issues/1315#issuecomment-5105894375

   **Issue Evaluation**
   
   Category: `type/enhancement` | Status: **Evaluated — Needs Discussion**
   
   Thank you for the detailed proposal. The motivation is valid — the current `SimpleConsumer.receive()` internally round-robins across queues via `SubscriptionLoadBalancer`, so callers cannot deterministically cover all queues without knowing the route topology.
   
   **Feasibility:** Technically feasible. The internal building blocks already exist:
   - `TopicRouteData` and `MessageQueueImpl` in `java/client/.../route/`
   - Route fetching via `ClientImpl.getRouteData(topic)`
   - Queue-level receive via `wrapReceiveMessageRequest` / `receiveMessage` in `SimpleConsumerImpl`
   
   **Scope of changes:**
   - A new public `MessageQueue` interface in `client-apis` (cannot expose `MessageQueueImpl` directly — it leaks internal types like `Broker`, `Permission`, protobuf)
   - Two `default` methods on `SimpleConsumer`: e.g. `getMessageQueues(String topic)` and `receiveAsync(MessageQueue, int, Duration)`
   - Implementation in `SimpleConsumerImpl` (~15-30 lines reusing existing internals)
   - Tests + example
   
   **Key concerns that need community discussion before implementation:**
   
   1. **Cross-language API contract** — `client-apis` is designed to mirror the shared rocketmq-apis spec across Java/C++/C#/Go/Rust clients. Adding queue-awareness to `SimpleConsumer` in Java alone would diverge from the multi-language spec. This should be discussed on the dev mailing list first.
   
   2. **Semantic shift** — `SimpleConsumer` intentionally hides queue details (server-side load balancing via consumer group). Queue-pinned receive can skew consumption distribution and interacts unexpectedly with shared consumer groups.
   
   3. **Staleness** — Routes change dynamically (queue migration, permission changes). A caller-held `MessageQueue` reference can become stale; error semantics need to be defined.
   
   4. **Alternative approach** — Consider exposing only the queue *count* (or a "desired concurrency" hint), which addresses the stated motivation without leaking queue identity or changing the core API contract.
   
   **Recommendation:** This is a meaningful enhancement, but given its impact on the public API surface and cross-language consistency, it would benefit from discussion on the [dev mailing list](dev@rocketmq.apache.org) before a PR is opened.
   
   ---
   *Automated evaluation by github-manager-bot*
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org