You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@rocketmq.apache.org by "qianye1001 (via GitHub)" <gi...@apache.org> on 2026/08/21 03:07:31 UTC
[I] [Bug] Java PushConsumer waits the full cache backoff after cache has drained [rocketmq-clients]
qianye1001 opened a new issue, #1340:
URL: https://github.com/apache/rocketmq-clients/issues/1340
### Before Creating the Issue
- [x] I have searched the existing issues and did not find a matching open issue.
### Programming Language
Java
### Version
Current `master`
### What happened
The Java gRPC PushConsumer divides its total message-count and byte cache limits across all assigned `ProcessQueue` instances:
```text
per-queue count threshold = maxCacheMessageCount / processQueueCount
per-queue byte threshold = maxCacheMessageSizeInBytes / processQueueCount
```
When a client is assigned many queues, each queue can therefore receive only a small cache quota. Once either quota is full, `ProcessQueueImpl.receiveMessage` schedules the next receive attempt after a fixed one-second delay.
ACK/NACK completion removes messages from the local cache, but cache eviction does not wake the paused receive loop. Even if consumption drains the queue almost immediately, that queue still waits for the one-second timer.
With server-side backlog, each receive can return immediately, so the client may repeatedly enter this cycle:
```text
many assigned queues
-> small per-queue quota
-> cache full
-> fixed one-second receive gap
-> lower receive throughput
-> consumer lag grows
-> subsequent receives fill the cache immediately
```
This can make receive request frequency and consumption throughput fall while lag continues to grow.
Issue #1196 proposed a per-queue cache option. PR #1214 increased the default total message cache from 1024 to 4096, which mitigates the frequency of cache-full events but does not remove the fixed recovery gap.
### Expected behavior
A cache-full queue should remain paused while it is near the high watermark, but should resume promptly after ACK/NACK completion drains both cache dimensions to a low watermark. The existing one-second task should remain only as a fallback.
### Proposed fix
- Represent each cache-full pause with a distinct token that retains the not-yet-sent receive `attemptId`.
- Store the active token in an `AtomicReference`.
- After cache eviction, resume when both cached message count and cached bytes are at or below 20% of their current per-queue thresholds (`threshold / 5L`; naturally zero for thresholds below five).
- Use token-identity CAS so concurrent ACKs and the fallback timer can schedule only one resume, and a stale timer cannot affect a newer pause (ABA).
- Resume through the consumer scheduler and reuse the paused attempt's `attemptId`.
- Do not resume a dropped queue or a stopped consumer.
- Keep the existing one-second timer as a liveness fallback.
### Scope
This issue concerns the Java gRPC `PushConsumer` / `ProcessQueueImpl` receive loop. `SimpleConsumer` uses a separate receive path and is not affected by this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Java PushConsumer waits the full cache backoff after cache has drained [rocketmq-clients]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #1340: [Bug] Java PushConsumer waits the full cache backoff after cache has drained
URL: https://github.com/apache/rocketmq-clients/issues/1340
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Java PushConsumer waits the full cache backoff after cache has drained [rocketmq-clients]
Posted by "RockteMQ-AI (via GitHub)" <gi...@apache.org>.
RockteMQ-AI commented on issue #1340:
URL: https://github.com/apache/rocketmq-clients/issues/1340#issuecomment-5655082602
**Issue Evaluation**
Category: `type/bug` | Status: **Confirmed**
The issue is valid. After the cache drains (e.g., all messages processed quickly), the `ProcessQueue` still waits the full backoff duration before the next receive call, even though the cache is empty and could accept more messages immediately.
The fix should detect when the cache has drained and reset the backoff timer, allowing the next receive to proceed without waiting. This is a latency optimization that improves throughput in bursty workloads.
---
*Automated evaluation by github-manager*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Java PushConsumer waits the full cache backoff after cache has drained [rocketmq-clients]
Posted by "RockteMQ-AI (via GitHub)" <gi...@apache.org>.
RockteMQ-AI commented on issue #1340:
URL: https://github.com/apache/rocketmq-clients/issues/1340#issuecomment-5394950880
**Issue Evaluation**
Category: `bug` | Status: **Confirmed**
Well-analyzed throughput degradation. The fixed 1-second backoff after cache-full pause is wasteful when consumption drains the cache quickly — especially with many assigned queues and small per-queue quotas.
**Root Cause:** `ProcessQueueImpl.receiveMessage` uses a fixed delay timer after cache-full pause; cache eviction (ACK/NACK) does not signal the paused receive loop to resume immediately.
**Impact:** Consumer throughput degrades as queue assignment grows, despite available cache capacity.
**Severity:** Medium
**Suggested approach:** Add a cache-drain signal that the ACK/NACK path can trigger to wake the paused receive loop immediately, bypassing the fixed delay.
---
*Automated evaluation by github-manager*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Java PushConsumer waits the full cache backoff after cache has drained [rocketmq-clients]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #1340:
URL: https://github.com/apache/rocketmq-clients/issues/1340#issuecomment-6181647777
This issue is stale because it has been open for 30 days with no activity. It will be closed in 3 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Java PushConsumer waits the full cache backoff after cache has drained [rocketmq-clients]
Posted by "RockteMQ-AI (via GitHub)" <gi...@apache.org>.
RockteMQ-AI commented on issue #1340:
URL: https://github.com/apache/rocketmq-clients/issues/1340#issuecomment-5440297933
**Issue Evaluation**
Category: bug | Status: **Confirmed**
This is a valid issue. When the PushConsumer cache drains and new messages arrive, the consumer waits the full cache backoff period before resuming, causing unnecessary latency. The backoff should be reset or shortened when the cache transitions from full to empty.
**Root Cause:** Cache backoff timer not being cancelled/reset when cache drains.
**Impact:** Increased message delivery latency after burst consumption patterns.
**Severity:** Medium — affects consumption responsiveness.
---
*Automated evaluation by github-manager*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Java PushConsumer waits the full cache backoff after cache has drained [rocketmq-clients]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #1340:
URL: https://github.com/apache/rocketmq-clients/issues/1340#issuecomment-6257238386
This issue was closed because it has been inactive for 3 days since being marked as stale.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Java PushConsumer waits the full cache backoff after cache has drained [rocketmq-clients]
Posted by "RockteMQ-AI (via GitHub)" <gi...@apache.org>.
RockteMQ-AI commented on issue #1340:
URL: https://github.com/apache/rocketmq-clients/issues/1340#issuecomment-5370300851
Hi @qianye1001, thanks for reporting this!
I've analyzed this issue against the codebase and can confirm this appears to be a **real bug**.
**Root Cause:** `ProcessQueueImpl.receiveMessage(String)` schedules a fixed 1-second backoff via `receiveMessageLater(RECEIVING_BACKOFF_DELAY_WHEN_CACHE_IS_FULL, attemptId)` whenever `isCacheFull()` is true, but `evictCache(MessageViewImpl)` only removes messages from the local cache after ACK/NACK completion and never resumes the paused receive loop. Therefore, even if consumption drains the cache immediately, the queue still waits the full one-second timer before issuing the next receive request.
**Affected Files:** java/client/src/main/java/org/apache/rocketmq/client/java/impl/consumer/ProcessQueueImpl.java, java/client/src/main/java/org/apache/rocketmq/client/java/impl/consumer/PushConsumerImpl.java
**Analysis:** The per-queue thresholds are computed by dividing total cache limits by the number of assigned queues (`PushConsumerImpl.cacheMessageCountThresholdPerQueue()` / `cacheMessageBytesThresholdPerQueue()`). With many queues the quota per queue is small, so the cache fills quickly. Once full, the receive loop pauses for `RECEIVING_BACKOFF_DELAY_WHEN_CACHE_IS_FULL` (1s). Because cache eviction (`eraseMessage`, `discardMessage`, etc.) does not signal or reschedule the receive loop, the queue cannot resume early, causing repeated one-second gaps and reduced throughput when server-side backlog is high. This matches the issue description exactly.
I'll prepare a fix spec and work on a PR. The community is welcome to provide feedback on the approach before implementation.
---
🤖 *Automated issue analysis by github-manager*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Java PushConsumer waits the full cache backoff after cache has drained [rocketmq-clients]
Posted by "RockteMQ-AI (via GitHub)" <gi...@apache.org>.
RockteMQ-AI commented on issue #1340:
URL: https://github.com/apache/rocketmq-clients/issues/1340#issuecomment-5364729188
**Issue Evaluation**
Category: `bug` | Status: **Confirmed**
This is a valid performance issue. When a `ProcessQueue` drains its cache below the per-queue threshold, the consumer still waits the full `cacheBackoffMs` before the next receive cycle, because the backoff timer is not reset upon cache drain. This introduces unnecessary latency in asymmetric queue-load scenarios.
**Root Cause:** The backoff scheduling in `PushConsumerImpl` does not account for cache-drain events — the next receive is always delayed by the full backoff interval regardless of current cache utilization.
**Impact:** Increased message delivery latency when queue assignments are imbalanced across consumers.
**Severity:** medium
A fix should reset or shorten the backoff delay when a `ProcessQueue` transitions from above-threshold to below-threshold.
---
*Automated evaluation by RockteMQ-AI*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Java PushConsumer waits the full cache backoff after cache has drained [rocketmq-clients]
Posted by "RockteMQ-AI (via GitHub)" <gi...@apache.org>.
RockteMQ-AI commented on issue #1340:
URL: https://github.com/apache/rocketmq-clients/issues/1340#issuecomment-5556535939
**Issue Evaluation**
Category: `bug` | Status: **Confirmed**
Verified against `java/client/src/main/java/org/apache/rocketmq/client/java/impl/consumer/ProcessQueueImpl.java`:
- **Line 218-220**: When `isCacheFull()` returns true, `receiveMessageLater(RECEIVING_BACKOFF_DELAY_WHEN_CACHE_IS_FULL, attemptId)` is called with a fixed delay.
- **Line 188-193**: `receiveMessageLater` schedules via `scheduler.schedule(...)` with the full delay duration.
- **Line 145**: `cacheFullNanoTime` tracks when cache became full, but the backoff delay is not reduced even when the cache drains before the delay expires.
**Root Cause:** The cache-full backoff uses a fixed delay (`RECEIVING_BACKOFF_DELAY_WHEN_CACHE_IS_FULL`) regardless of whether the cache drains during the wait. The consumer waits the full backoff period even when messages could be received sooner.
**Impact:** Medium — unnecessary latency spikes when the cache drains quickly but the consumer is still in backoff.
**Severity:** Medium — performance degradation, not correctness.
**Fix direction:** Check if cache is still full before applying the full backoff, or use a shorter initial delay with exponential backoff that resets when cache drains.
An automated fix proposal will be generated. Reply `/approve` to proceed with PR generation.
---
*Automated evaluation by RockteMQ-AI*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org