You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "dragonls (via GitHub)" <gi...@apache.org> on 2026/10/09 07:46:50 UTC
Re: [I] [Bug] Shared subscription stalls forever: havePendingRead=true while the cursor has no read (pendingReadOps=0, waitingReadOp=false) — the #26174 invariant, unguarded on PersistentDispatcherMultipleConsumers [pulsar]
dragonls commented on issue #26454:
URL: https://github.com/apache/pulsar/issues/26454#issuecomment-6076739128
We hit the same state on one of our brokers and captured a **heap dump of the topic owner broker before unloading**. Findings below;
### Environment
- Broker: internal fork based on **branch-3.0 (3.0.14)**. It already contains the backports of #24551, #24741, #24515 and #24569, so none of those prevent this. The dispatcher logic involved (`readMoreEntries`, `cancelPendingRead`, `readEntriesComplete/Failed`, `checkAndUnblockIfStuck`) is the same as the 4.0.12 code quoted in the issue description.
- Topic: persistent, partitioned; only one partition affected. Subscription: **Shared**, durable. Two other subscriptions on the same partition were healthy.
- Effective broker config, read from `ServiceConfiguration` in the dump:
```
unblockStuckSubscriptionEnabled=false # <-- note: detector disabled
dispatcherDispatchMessagesInSubscriptionThread=true
managedLedgerReadEntryTimeoutSeconds=0
managedLedgerNewEntriesCheckDelayInMillis=10
managedLedgerMaxReadsInFlightSizeInMB=0 # InflightReadsLimiter disabled
dispatcherMaxReadBatchSize=100
maxUnackedMessagesPerSubscription=0
maxUnackedMessagesPerConsumer=0
subscriptionRedeliveryTrackerEnabled=true
autoSkipNonRecoverableData=true
ttlDurationDefaultInSeconds=3600
messageExpiryCheckIntervalInMinutes=5
transactions disabled (TransactionBufferDisable)
```
### State in the heap dump
**Dispatcher** (`PersistentDispatcherMultipleConsumers`):
| field | value |
|---|---|
| `havePendingRead` | **true** |
| `havePendingReplayRead` | false |
| `shouldRewindBeforeReadingOrReplaying` | **true** |
| `sendInProgress` | false |
| `totalAvailablePermits` | 9192 |
| consumers | 2 (permits 8192 and 1000, `unackedMessages=0`, `pendingAcks` empty, not blocked) |
| `blockedDispatcherOnUnackedMsgs` | 0 |
| `isClosed` | 0 |
| `redeliveryMessages` / delayed tracker | empty / none |
| `readBatchSize` | 100 (max, i.e. no recent read failures) |
| `readFailureBackoff.next` | 15000 (initial value) |
**Cursor** (`ManagedCursorImpl`):
| field | value |
|---|---|
| `state` | Open |
| `waitingReadOp` | **null** |
| `pendingReadOps` | **0** |
| `markDeletePosition` / `readPosition` | `102651931:19384` / `102651931:19385` |
| `registeredToWaitingCursors` | false |
| ML `lastConfirmedEntry` | `102664377:10214` (ML state `LedgerOpened`), so a large backlog is readable |
This is the same inconsistent triple as in the description (`havePendingRead=true`, `pendingReadOps=0`, no waiting op). In our case it is the only `PersistentDispatcherMultipleConsumers` on the broker in this state. Query used:
```
SELECT toString(d.name) FROM org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers d
WHERE d.havePendingRead = true and d.cursor.waitingReadOp = null and d.cursor.pendingReadOps = 0
```
### Additional observations that may narrow it down
1. **The `unblockStuckSubscriptionEnabled` detector is not involved.** It is disabled on this broker (`false`), and the stall still happened. So `checkAndUnblockIfStuck` racing the normal read path is not a precondition.
2. **No read exists anywhere in the heap, not even a stale one.**
- None of the live `OpReadEntry` instances with `cursor != null` (40 in the heap) belongs to this cursor.
- All recycled `OpReadEntry` instances (`cursor == null`) are referenced only by `Recycler$DefaultHandle` (plus the static `WAITING_READ_OP_FOR_CLOSED_CURSOR` instance). None is held by a BK/entry-cache callback chain, a `ReadEntryCallbackWrapper`, `PendingReadsManager` or an executor task.
- `InflightReadsLimiter` is disabled, so no queued handle.
- In the thread stacks captured with the dump, no thread is on a read or dispatch path. The only `org.apache.bookkeeper.mledger` / `org.apache.pulsar.broker.service` frames are one publish in progress (`ServerCnx.handleSend` → `Producer.publishMessage`, `OpAddEntry.addComplete`).
So this is not a BK read that never returned. Either the read completion was consumed without clearing the flag, or the read was dropped before or without reaching the cursor.
3. **`shouldRewindBeforeReadingOrReplaying=true` gives a lower bound on when the flag got stuck.** `addConsumer()` sets this flag only when a consumer joins an *empty* consumer list while `havePendingRead || havePendingReplayRead` is true. `readEntriesComplete()` clears it only on a `Normal` completion, and `readEntriesFailed()` clears it on any failure. Since it is still `true`, **no Normal read has completed and no read has failed since the last time a consumer joined an empty subscription.**
- Combined with point 4, the stuck state predates the C++ consumer's connection.
- That means at some earlier point all consumers were removed while the flag was already (or about to become) stuck. `clearComponentsAfterRemovedAllConsumers()` → `cancelPendingRead()` then had nothing to cancel (`cursor.cancelPendingReadRequest()` returns false when there is no waiting op), so the flag stayed `true`.
- This matches the "`cancelPendingRead()` early return" path from the description. It also shows that once stuck, new consumers can never recover the subscription: reconnecting consumers, seeking or resetting the cursor do not touch `havePendingRead`.
4. **Timeline (UTC):**
- `03:22:52`: last (and only) Flow from the C++ consumer. Its `lastConsumedTimestamp` is `0`, so it never received a single message on that connection.
- `06:34:52`: our extra Java consumer connected and sent Flow. `lastConsumedTimestamp=0`, nothing delivered either.
- Meanwhile message TTL expiry kept advancing the mark-delete position (`lastExpireTimestamp` 06:34:16), which is why `readPosition` = `markDelete + 1`. The dispatcher itself issued no reads.
- Heap dump taken a few minutes after 06:34; afterwards the topic was unloaded to recover.
5. Other things we ruled out from the dump: permits (9192 available), unacked limits (both 0), dispatch rate limiter (no throttling state relevant since the read path short-circuits on `havePendingRead`), delayed delivery (no tracker), transaction `maxReadPosition` (`TransactionBufferDisable`), offloaded ledger (ledger `102651931` has no offload context), `InflightReadsLimiter` (disabled).
### Regarding the questions in the description
- Q1/Q2: from our data, an epoch guard alone would only help if a stale completion is what clears or strands the flag. A safety net in `checkAndUnblockIfStuck` would recover our case too: `havePendingRead && !cursor.hasPendingReadRequest() && cursor.getPendingReadOps() == 0` with an unchanged read position across consecutive checks → reset the flag and `readMoreEntries()`. Another option is resetting the flag in `addConsumer()` when the consumer list is empty and the cursor has no pending or waiting read, instead of setting `shouldRewindBeforeReadingOrReplaying`.
- Q4: `managedLedgerReadEntryTimeoutSeconds` would not help here; there is no read at the cursor or BK level to time out.
If there are specific fields or objects you'd like us to inspect (e.g. other subscriptions/cursors on the same ML, ledger cache handles, executor queues), let us know and we'll query the dump.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org