You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@rocketmq.apache.org by "messere1 (via GitHub)" <gi...@apache.org> on 2026/10/10 09:46:29 UTC
[I] [Bug] Exclusive eviction tombstones of clients that never sync again are never reclaimed [rocketmq]
messere1 opened a new issue, #11354:
URL: https://github.com/apache/rocketmq/issues/11354
### Before Creating the Bug Report
- [x] I found a bug, not just asking a question, which should be created in [GitHub Discussions](https://github.com/apache/rocketmq/discussions).
- [x] I have searched the [GitHub Issues](https://github.com/apache/rocketmq/issues) and [GitHub Discussions](https://github.com/apache/rocketmq/discussions) of this repository and believe that this is not a duplicate.
- [x] I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ.
### Runtime platform environment
OS: Windows 11 / Linux (broker module, lite topic path)
### RocketMQ version
branch: develop
Git commit id: 9ea2ccdc2b41445aee5e177e14e501d606f5a1ed (2026-10-08)
### JDK Version
Compiler: Temurin JDK 17.0.19
### Describe the Bug
In exclusive mode, when clientB claims a liteTopic that clientA still holds, `LiteSubscriptionRegistryImpl.excludeClientByLmqName` evicts clientA and records a tombstone so clientA's stale pulls are rejected until its next full subscription sync:
```java
exclusiveEvictionTombstones.add(clientGroup.clientId, lmqName);
```
A tombstone can be cleared in three ways:
1. the evicted client re-claims the liteTopic itself (`addPartialSubscription` calls `exclusiveEvictionTombstones.remove`);
2. the evicted client performs a full sync (`addCompleteSubscription` calls `removeStale` for liteTopics no longer in its set);
3. `removeCompleteSubscription` calls `removeAllOf` — invoked either by an explicit `COMPLETE_REMOVE` or by the expiry cleaner `cleanupExpiredSubscriptions`.
However, when the eviction took the client's **last** liteTopic, its `LiteSubscription` is dropped from `client2Subscription` immediately (the same "remove client if no more liteTopic" branch). If that client then terminates without sending `COMPLETE_REMOVE` (crash, kill, instance replaced by a new clientId), none of the three paths can ever fire:
- paths 1 and 2 need a request from the client, which never arrives;
- path 3 via the expiry cleaner cannot fire because `cleanupExpiredSubscriptions` only walks `client2Subscription`, which no longer contains the evicted clientId.
The tombstone entry therefore stays in `ExclusiveEvictionTombstones` until the broker restarts. The expiry cleaner even logs its size every check interval (`ExclusiveEvictionTombstones size: {}`), so on a broker with regular exclusive takeovers this is a grow-only counter.
Impact: under exclusive-mode client churn (every takeover evicts the previous holder; autoscaling/rolling restarts produce a stream of clients that lose their last liteTopic and then go away), the tombstone set grows unboundedly — an O(number of takeovers) memory leak of `clientId$lmqName` keys held for the broker's lifetime.
This is the same reachability gap as the evicted client's channel entry (issue #11352): the eviction path drops the subscription directly, making every cleaner that only walks `client2Subscription` blind to the remaining per-client state.
### Steps to Reproduce
1. Create a subscription group with `liteSubExclusive=true` bound to a LITE topic.
2. Start clientA and subscribe (PARTIAL_ADD) to liteTopic L — its only liteTopic.
3. Start clientB and subscribe to the same liteTopic L. clientA is evicted; since L was its last liteTopic, clientA is removed from `client2Subscription`, and a tombstone `(clientA, L)` is recorded.
4. clientA now terminates without sending `COMPLETE_REMOVE`.
5. Run the expiry cleaner (`cleanupExpiredSubscriptions`) any number of times: it walks `client2Subscription`, does not find clientA, and never clears the tombstone. `hasExclusiveEvictionTombstone(clientA, L)` keeps returning true forever.
Unit-level reproduction: in `LiteSubscriptionRegistryImplTest`, perform the takeover of step 3, then call `registry.cleanupExpiredSubscriptions(...)` — `registry.hasExclusiveEvictionTombstone("clientA", "lmq1")` is still true afterwards even though clientA has had no subscription, no channel and no requests for arbitrarily long.
### What Did You Expect to See?
Tombstones of clients that will never sync again should be bounded by the same horizon the broker already uses to consider a client gone: `liteSubscriptionCheckTimeoutMills`. A tombstone older than the check timeout belongs to a client that either re-subscribed (paths 1/2 already cleared it) or is gone for good, so the expiry cleaner should discard it.
### What Did You See Instead?
Tombstones of evicted clients that never sync again are never reclaimed; the set grows without bound for the lifetime of the broker process.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Exclusive eviction tombstones of clients that never sync again are never reclaimed [rocketmq]
Posted by "RockteMQ-AI (via GitHub)" <gi...@apache.org>.
RockteMQ-AI commented on issue #11354:
URL: https://github.com/apache/rocketmq/issues/11354#issuecomment-6096386301
**Issue Evaluation**
Category: `type/bug` | Status: **Confirmed**
The reported issue has been verified against the current codebase (`develop` branch, commit `9ea2ccdc2b41`).
**Root Cause:** In `LiteSubscriptionRegistryImpl.excludeClientByLmqName`, when an evicted client's last liteTopic is taken, the client is immediately removed from `client2Subscription` (lines 345-347). Its tombstone entry in `exclusiveEvictionTombstones` persists, but none of the 3 cleanup paths can fire if the client terminates without sending `COMPLETE_REMOVE`:
- Path 1 (`addPartialSubscription` → `remove`) — requires client request
- Path 2 (`addCompleteSubscription` → `removeStale`) — requires client request
- Path 3 (`removeCompleteSubscription` → `removeAllOf` via `cleanupExpiredSubscriptions`) — only walks `client2Subscription`, which no longer contains the evicted clientId
**Impact:** Under exclusive-mode client churn (autoscaling, rolling restarts), the tombstone set grows unboundedly — an O(number of takeovers) memory leak of `clientId$lmqName` keys held for the broker's lifetime.
**Severity:** Medium — requires sustained exclusive-mode churn to become significant, but is a grow-only leak with no recovery path.
**Precondition confirmed:** The leak only occurs when eviction empties the client's lmq set; if other liteTopics remain, `client2Subscription` (and thus expiry cleanup) still covers it.
An automated fix proposal will be generated. Reply `/approve` to proceed with PR generation.
---
*Automated evaluation by github-manager-bot*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Exclusive eviction tombstones of clients that never sync again are never reclaimed [rocketmq]
Posted by "RockteMQ-AI (via GitHub)" <gi...@apache.org>.
RockteMQ-AI commented on issue #11354:
URL: https://github.com/apache/rocketmq/issues/11354#issuecomment-6110394221
**Issue Evaluation**
Category: `type/bug` | Status: **Confirmed**
The tombstone reclaim gap is well-analyzed. The three cleanup paths all require either client action or presence in `client2Subscription` — once the evicted client's last liteTopic is removed and the client goes away without sending `COMPLETE_REMOVE`, the tombstone entry becomes unreachable for cleanup.
**Root Cause:** `excludeClientByLmqName` removes the `LiteSubscription` from `client2Subscription` when the evicted client's last liteTopic is taken, making the tombstone entry orphaned. The expiry cleaner `cleanupExpiredSubscriptions` only walks `client2Subscription`, so it cannot see the orphaned tombstone.
**Impact:** O(number of takeovers) memory leak of `clientId$lmqName` keys in `ExclusiveEvictionTombstones` over the broker's lifetime. On brokers with regular exclusive-mode client churn (autoscaling, rolling restarts), this is a grow-only counter.
**Severity:** Medium — memory leak under specific churn conditions, not immediate data loss.
An automated fix proposal can be generated. Reply `/approve` to proceed with PR generation.
---
*Automated evaluation by RocketMQ-AI*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org