You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "kaminski-dev (via GitHub)" <gi...@apache.org> on 2026/08/18 07:39:28 UTC
[D] Transactions: the pending-ack replay thread can get stuck forever after a lost ledger read [pulsar]
GitHub user kaminski-dev created a discussion: Transactions: the pending-ack replay thread can get stuck forever after a lost ledger read
In case others have seen it: I ran into transaction pending-ack replay loop. On one broker, a pulsar-transaction-executor-N-1 thread was stuck forever inside MLPendingAckStore$PendingAckReplay.run(), sitting on Thread.sleep(1) (MLPendingAckStore.java:431), and it never recovered on its own.
Versions: I confirmed this on 4.0.10. The same replay code is identical on branch-4.1 (4.1.x) and on branch-4.2 (4.2.x): the Thread.sleep(1) replay loop, the single-thread executors that are shared by hash, and the consumer that waits on pendingAckHandleFuture(). All line numbers below are from the 4.0.10 tag (pulsar-broker and managed-ledger).
My observation: These replay executors are single-threaded, and many subscriptions share the same thread by hashing. When transactions are enabled, every persistent subscription builds a PendingAckHandle — even consumers that never use transactions — and PersistentSubscription.addConsumerInternal waits for pendingAckHandleFuture() before the consumer can be added. So, if one replay is stuck, every other subscription on that same thread cannot create consumers. It can even turn into a full broker restart if the health-check subscription happens to land on the stuck thread.
How it gets stuck: the replay loop (MLPendingAckStore.java:406-439) keeps running until a read either succeeds or fails. A read failure sets isReadable = false (:470), and fillQueue() (:474-483) only starts a new read when outstandingReadsRequests == 0. So, if a read is sent but its callback never comes back:
- outstandingReadsRequests stays at 1, so no new read is ever started;
- the entry queue stays empty, so entryQueue.poll() returns null;
- isReadable stays true, so the loop never exits.
The result is an endless 1ms sleep loop. Normally a watchdog would rescue this: ManagedLedgerImpl.checkReadTimeout() (ManagedLedgerImpl.java:4531) calls lastReadCallback.readFailed(TimeoutException) after a timeout. But that watchdog is turned off when managedLedgerReadEntryTimeoutSeconds=0, so nothing ever completes the read.
Possible trigger: the read gets lost because the ledger it is reading is deleted underneath it — and this does not need a rare bookie garbage-collection race. New ledgers are created with timestamp=0 (ManagedLedgerImpl.java:618 on init, :1640 on rollover; the real timestamp is only written when the ledger is closed, :1846). And hasLedgerRetentionExpired(retentionTimeMs, 0) (:2770-2772) is always true for these system ledgers, which run with retention 0. So, an empty or just-rolled-over pending-ack ledger becomes eligible for deletion right away in internalTrimLedgers (:2967). In other words, the normal create-and-delete of empty ledgers is enough to race the replay read and make its callback disappear.
Related issues I found:
- #14852 (closed) fixed only the CursorAlreadyClosedException case — where the read failure callback does fire — not the "callback never comes back" case.
- #23506 confirms that without a read timeout, managed-ledger reads can hang forever.
- #24401 fixed the transaction buffer replay by giving it its own executor (already in 4.0.10), but the pending-ack replay (MLPendingAckStore.PendingAckReplay) still has no such isolation.
A couple of directions that might help: giving PendingAckReplay its own time limit, independent of managedLedgerReadEntryTimeoutSeconds, so it can never loop forever on Thread.sleep(1) (:431); and moving pending-ack replay onto its own executor, the same way #24401 did for the transaction buffer, so one stuck replay can't block unrelated subscriptions.
I have a small, repeatable local reproduction (one replay thread, small ledgers, read timeout set to 0) that leaves the executor stuck while the broker otherwise keeps running. Happy to share the reproduction and thread dumps.
Has anyone seen this behavior? Does this analysis seem plausible? Could this explain the broker pod restarts, and is there a recommended configuration to avoid it besides setting managedLedgerReadEntryTimeoutSeconds to a non-zero value?
GitHub link: https://github.com/apache/pulsar/discussions/26364
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org
Re: [D] Transactions: the pending-ack replay thread can get stuck forever after a lost ledger read [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
GitHub user lhotari added a comment to the discussion: Transactions: the pending-ack replay thread can get stuck forever after a lost ledger read
Filed this as **#26368** and opened **#26369** with a fix.
### What the PR does
* Ports the #13739 `else` branch into `MLPendingAckStore.FillEntryQueueCallback.fillQueue()`: when the
cursor has nothing left to read and the queue is drained, the replay ends instead of sleeping forever.
That is the state where the loop guard and the read gate disagree permanently, and it is the one I
could reproduce deterministically.
* Re-checks `cursor.isClosed()` inside the loop, so closing the subscription ends a replay whose read
completion never arrives.
* Restores the interrupt flag and ends the replay via `replayFailed()` instead of continuing, so the
task is cancellable and a stuck replay no longer survives `ExecutorProvider.shutdownNow()`.
* Releases entries left in the queue on every abnormal exit — a read can still land while the replay is
ending, and those buffers were never released.
* `PendingAckHandleImpl.exceptionHandleFuture()` no longer retries once the handle is closing. The retry
path resets the state to `None` before scheduling `init()`, which defeats the `checkIfClose()` guard
and reopens the pending-ack store of a subscription that is going away. That was reachable before, but
the new in-loop close check would have made it happen on every unload during a replay.
Three tests, each verified to fail when its corresponding change is reverted.
### What it does not fix
Being explicit, because it matters for whether this actually covers your incident:
* If what you hit was the **"read was issued and its callback never came back"** variant — your original
model, `outstandingReadsRequests == 1` — the loop still waits, because there is genuinely nothing to
decide on. What changes is that it is no longer *unrecoverable*: unloading the topic now ends it, and
it no longer outlives broker shutdown. Preventing that state belongs in managed-ledger, not here.
* A terminal read failure such as `LedgerNotExistException` under the default
`autoSkipNonRecoverableData=false` still leaves `isReadable == true` and re-issues the same doomed
read. Fixing that is entangled with something I did not want to change quietly: every
`isReadable = false` exit currently falls through to `replayComplete()`, and
`TransactionTest.testEndTPRecoveringWhenManagerLedgerDisReadable` explicitly asserts that a fenced
managed ledger or a closed cursor leaves the handle `Ready`. I tried the obvious fix, that test caught
it, and I backed it out — it is a deliberate old decision and reversing it deserves its own PR and its
own discussion.
* `TopicTransactionBuffer` has the mirror-image gap: it has the `else` escape (that is where #13739
landed) but no `cursor.isClosed()` check at all and a bare `//no-op` interrupt handler. Its abort path
closes the topic, so it needs its own change and test.
All three are written up in #26368.
### Still useful from you
The fix addresses the state I could prove. It would be good to know it is the state you actually hit,
and your reproduction is the fastest way to settle that — please attach it to #26368 if you can share
it. Failing that, the heap-dump values of `outstandingReadsRequests` / `isReadable` /
`readPosition` / `currentLoadPosition` / `lastConfirmedEntry` from the stuck broker would tell us
directly, as would whether `"...stat reply fail"` ERROR lines were repeating and whether the broker had
restarted shortly before.
Review on the PR is welcome, particularly on the `Ready`-versus-failed question above.
---
<sub>*Assisted-by: Claude (Opus 5, Fable) and OpenAI Codex (gpt-5.6-sol). The code paths, line numbers
and commit references were reviewed by me against the source before posting.*</sub>
GitHub link: https://github.com/apache/pulsar/discussions/26364#discussioncomment-18065763
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org
Re: [D] Transactions: the pending-ack replay thread can get stuck forever after a lost ledger read [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
GitHub user lhotari added a comment to the discussion: Transactions: the pending-ack replay thread can get stuck forever after a lost ledger read
Thanks for the unusually careful write-up — the stack frame, the line numbers and the co-hashing
analysis are all accurate, and this is a real bug. I dug into it and I think the conclusion is:
**you found a genuine defect, the impact is wider than you describe, but the mechanism is different
from the one you propose — and, importantly, the workaround you're asking about does not work.**
## The short answer to your question
`managedLedgerReadEntryTimeoutSeconds` will **not** get you out of this, so please don't rely on it.
`checkReadTimeout()` delivers `createManagedLedgerException(BKException.Code.TimeoutException)`.
`TimeoutException` is not in `isBkErrorNotRecoverable(...)` (that switch only covers
`NoSuchLedgerExists*` and `NoSuchEntry`), so it returns a **plain** `ManagedLedgerException`
(`ManagedLedgerImpl.java` `createManagedLedgerException(int)`). That reaches
`MLPendingAckStore.FillEntryQueueCallback.readEntriesFailed`, and — see below — a plain
`ManagedLedgerException` matches none of the conditions that stop the replay. It just decrements
`outstandingReadsRequests` and lets the loop issue the read again.
It's still reasonable defence-in-depth, but framing it as the mitigation would be misleading.
## The thing I think you'll find most interesting
`TopicTransactionBuffer` has the *same* recovery loop with the *same* `FillEntryQueueCallback` class
name — and it terminates, because its `fillQueue()` has an `else` branch that `MLPendingAckStore`
lacks:
```java
// TopicTransactionBuffer.FillEntryQueueCallback.fillQueue()
if (cursor.hasMoreEntries()) {
outstandingReadsRequests.incrementAndGet();
cursor.asyncReadEntries(...);
} else {
if (entryQueue.size() == 0) {
isReadable = false; // <-- loop exits
}
}
```
```java
// MLPendingAckStore.FillEntryQueueCallback.fillQueue() -- v4.0.10 :474-483
if (cursor.hasMoreEntries()) {
outstandingReadsRequests.incrementAndGet();
readAsync(NUMBER_OF_PER_READ_ENTRY, this);
}
// no else branch
return isReadable;
```
That escape was added by **#13739** (`7dee63ed707`, merged 2022-01-18). Modification #3 of its commit
message reads:
> if `cursor.hasMoreEntries() == false && entryQueue.size() == 0`, return false and stop recovering.
> - If the cursor is cleared when transaction is recovering, there will no entries can be read, but
> `currentLoadPosition < lastConfirmedEntry`.
So the project diagnosed this exact stall in the sibling class four years ago, fixed it there, and
never ported it to `MLPendingAckStore`. That's both the strongest evidence your report is real and
the natural shape of the fix.
## Why I think the mechanism is different
The loop guard and the read gate measure different things:
- `lastConfirmedEntry` is a **construction-time snapshot** of the ML's LAC (it's `final`), and
`currentLoadPosition` is seeded from the cursor's **mark-delete** position;
- but `fillQueue()`'s gate is `cursor.hasMoreEntries()`, driven by the cursor's **read** position.
So "there is still work to do" and "there is still something to read" can disagree permanently, with
**`outstandingReadsRequests == 0`** — no lost callback needed anywhere. That state issues no read and
logs nothing, which matches a silent hang better than a dropped callback does.
Two smaller corrections, both in your favour in the sense that the bug is worse than stated:
- "A read failure sets `isReadable = false` (:470)" — v4.0.10 `:470` is the field *declaration*. The
assignment is `:506`, and it's conditional. Parenthesised, the guard is
`(autoSkipNonRecoverableData && NonRecoverableLedgerException) || ManagedLedgerFencedException ||
CursorAlreadyClosedException`. With the default `autoSkipNonRecoverableData=false`, a definitive
`LedgerNotExistException` leaves `isReadable == true` — so the loop re-issues the same doomed read
~1000×/s forever, one ERROR line per iteration. Your deleted-ledger scenario lands here.
- The trim race as described doesn't hold: `internalTrimLedgers` only iterates
`ledgers.headMap(slowestReaderLedgerId, false)`, bounded by the slowest **durable cursor's
mark-delete** position, and empty ledgers are removed inline in `ledgerClosed()` before the trimmer
ever sees them. Also worth knowing: BookKeeper's own `readEntryTimeout` is on by default (30s, from
`bookkeeperClientTimeoutInSeconds`), so a silent bookie-side loss is hard to achieve regardless of
the ML-level setting.
Everything else you wrote holds: the co-hashed single-thread executors, `addConsumerInternal` waiting
on `pendingAckHandleFuture()` with no timeout, and a `PendingAckHandle` for every persistent
subscription. The shape is identical on 4.0/4.1/4.2/master and back through 3.x to 2.10.0.
## What would settle it
The thread dump can't distinguish the cases — they all park on the same `Thread.sleep(1)` frame. If
you still have the broker (or can reproduce):
1. From a heap dump: `outstandingReadsRequests`, `isReadable`, and the cursor's `readPosition`
vs `currentLoadPosition` vs `lastConfirmedEntry`. `1` means a genuinely lost callback; `0` means
one of the two states above.
2. Were `"MLPendingAckStore of topic [...] stat reply fail!"` ERROR lines repeating? A high repeat
count points at the failing-read case; total silence points at the no-more-entries case.
3. Had that broker restarted shortly before the hang?
4. Did message TTL, `skip`, `clear-backlog`, or a topic unload coincident with a commit/abort touch
that subscription?
5. Is `transactionPendingAckBatchedWriteEnabled` on?
6. Did it recur after unloading the topic?
And yes please — **your reproduction would be very welcome.** My analysis is code-level only; I
haven't run a repro, and yours would pin down which of the states you actually hit.
## Next steps
This deserves a tracking issue rather than a Q&A thread — happy to open one, or feel free to file it
and I'll link my notes. My current thinking on the fix is: port the #13739 `else` branch, and pair it
with routing aborted replays to `replayFailed` rather than `replayComplete` (today every
`isReadable=false` exit falls through to `replayComplete()`, which would mark the handle Ready with
partial pending-ack state — so widening the exit conditions without that change trades a hang for
silent incorrect transaction state). Re-checking `cursor.isClosed()` inside the loop and restoring
the interrupt are cheap additions that make unload a reliable remedy.
---
<sub>*Assisted-by: Claude (Opus 5, Fable) and OpenAI Codex (gpt-5.6-sol). The code paths, line
numbers and commit references above were reviewed by me against the source before posting; the
analysis is code-level only — I have not run a reproduction.*</sub>
GitHub link: https://github.com/apache/pulsar/discussions/26364#discussioncomment-18063974
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org
Re: [D] Transactions: the pending-ack replay thread can get stuck forever after a lost ledger read [pulsar]
Posted by "kaminski-dev (via GitHub)" <gi...@apache.org>.
GitHub user kaminski-dev edited a comment on the discussion: Transactions: the pending-ack replay thread can get stuck forever after a lost ledger read
Thanks, @lhotari for the quick response and even the PR on that!
We agree that managedLedgerReadEntryTimeoutSeconds is not a fix for this replay loop. A managed-ledger timeout becomes a plain ManagedLedgerException; MLPendingAckStore.readEntriesFailed does not set isReadable=false for it, so replay retries rather than terminating.
Our proposed empty-ledger trim race was also incorrect. As you pointed out, internalTrimLedgers is bounded by the slowest durable cursor's mark-delete position.
We have now compared two environments and appear to have observed both replay failure modes.
**SIT: silent replay stall**
In the original SIT incident window, "MLPendingAckStore ... stat reply fail!" appeared only twice, both as CursorAlreadyClosedException on broker-2. There were no LedgerNotExist, NoSuchLedger, or NonRecoverable errors from the transaction executor, and no error storm on the repeatedly affected broker-1.
The stuck executor was silent. This strongly favours the outstandingReadsRequests == 0 / cursor.hasMoreEntries() == false state addressed by #26369, rather than our original lost-callback hypothesis or a repeatedly failing ledger read. Without the heap dump, however, we cannot prove the value of outstandingReadsRequests. As of now we did not have a heap dump which we can provide.
**Answers for the SIT incidents:**
1. We do not have the heap values for outstandingReadsRequests, isReadable, or the cursor positions.
2. No repeating "stat reply fail!" errors were present around the silent stalls.
3. Several affected brokers had restarted shortly beforehand. In four of six retained shutdown dumps, the broker had started approximately 5–18 minutes earlier.
4. We found no correlated skip, clear-backlog, explicit expiry request, or identifiable commit/abort operation. Topic unload messages occurred during broker shutdown and therefore appear to be consequences rather than proven triggers.
5. We could not confirm transactionPendingAckBatchedWriteEnabled from the runtime logs. There is no override, so it is the v4.0.10 default of false.
6. Broker shutdown unloaded topics, but the replay executor still failed to terminate. Because the shutdown dump does not identify the owning topic, we cannot prove whether unloading that specific topic had been attempted earlier.
**UAT: repeating read-failure loop**
On UAT, which uses the same broker configuration, we observed the other case.
On 2026-07-26, "MLPendingAckStore ... stat reply fail!" repeated at very high frequency on all four brokers. In a 15-minute window, the approximate counts were:
- broker-0: 47,616
- broker-1: 1,450
- broker-2: 59,358
- broker-3: 31,149
Every sampled error was:
ManagedLedgerException:
Bookie handle is not available error code: -8
Affected pending-ack stores included:
< internal-tenant >/< internal-namespace >/< internal-topic >-v1
< sub-name >
and:
< internal-tenant >/< internal-namespace >/< internal-topic >-v1
< sub-name >
This is clearly the repeatedly failing-read case. It is not evidence of a lost callback because the callbacks were arriving with failures and replay was immediately issuing another read.
**Answers for the UAT incident:**
1. Again, we do not have the heap values.
2. Yes, "stat reply fail!" repeated thousands of times with plain ManagedLedgerException and BookKeeper error -8.
3. Yes. The cluster was in a broker restart loop. For example, broker-0 started at 06:28:37Z, and failures were repeating by 06:28:49Z.
4. We found no correlated skip, clear-backlog, explicit expiry request, or identifiable commit/abort operation for the affected subscriptions. There was extensive shutdown/unload/reload activity, together with BookKeeper unavailability.
5. The deployment configuration matches SIT, but the batched pending-ack value was not independently visible in the logs.
6. The affected topics were unloaded and subsequently recovered. Both subscriptions later logged pending ack state reply success!, and no further "stat reply fail!" errors appeared after approximately 06:30Z.
A separate silent PendingAckReplay shutdown stall occurred again in UAT on 2026-08-05 without a corresponding "stat reply fail!" storm. That reinforces the conclusion that we have encountered both states at different times: the silent no-more-entries stall and the repeatedly failing-read loop.
In the attachments you will find the test which I run to "prove" my thesis and the output of the test itself.
I will try to get the heap dump, but if anything else is needed please let me know as I am happy to help here in the investigations.
[test_output.log](https://github.com/user-attachments/files/31186134/test_output.log)
[DeletedPendingAckLedgerTest.java](https://github.com/user-attachments/files/31186141/DeletedPendingAckLedgerTest.java)
GitHub link: https://github.com/apache/pulsar/discussions/26364#discussioncomment-18068314
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org
Re: [D] Transactions: the pending-ack replay thread can get stuck forever after a lost ledger read [pulsar]
Posted by "kaminski-dev (via GitHub)" <gi...@apache.org>.
GitHub user kaminski-dev added a comment to the discussion: Transactions: the pending-ack replay thread can get stuck forever after a lost ledger read
Thanks, @lhotari for the quick response and even the PR on that!
We agree that managedLedgerReadEntryTimeoutSeconds is not a fix for this replay loop. A managed-ledger timeout becomes a plain ManagedLedgerException; MLPendingAckStore.readEntriesFailed does not set isReadable=false for it, so replay retries rather than terminating.
Our proposed empty-ledger trim race was also incorrect. As you pointed out, internalTrimLedgers is bounded by the slowest durable cursor's mark-delete position.
We have now compared two environments and appear to have observed both replay failure modes.
**SIT: silent replay stall**
In the original SIT incident window, "MLPendingAckStore ... stat reply fail!" appeared only twice, both as CursorAlreadyClosedException on broker-2. There were no LedgerNotExist, NoSuchLedger, or NonRecoverable errors from the transaction executor, and no error storm on the repeatedly affected broker-1.
The stuck executor was silent. This strongly favours the outstandingReadsRequests == 0 / cursor.hasMoreEntries() == false state addressed by #26369, rather than our original lost-callback hypothesis or a repeatedly failing ledger read. Without the heap dump, however, we cannot prove the value of outstandingReadsRequests. As of now we did not have a heap dump which we can provide.
**Answers for the SIT incidents:**
1. We do not have the heap values for outstandingReadsRequests, isReadable, or the cursor positions.
2. No repeating "stat reply fail!" errors were present around the silent stalls.
3. Several affected brokers had restarted shortly beforehand. In four of six retained shutdown dumps, the broker had started approximately 5–18 minutes earlier.
4. We found no correlated skip, clear-backlog, explicit expiry request, or identifiable commit/abort operation. Topic unload messages occurred during broker shutdown and therefore appear to be consequences rather than proven triggers.
5. We could not confirm transactionPendingAckBatchedWriteEnabled from the runtime logs. There is no override, so it is the v4.0.10 default of false.
6. Broker shutdown unloaded topics, but the replay executor still failed to terminate. Because the shutdown dump does not identify the owning topic, we cannot prove whether unloading that specific topic had been attempted earlier.
**UAT: repeating read-failure loop**
On UAT, which uses the same broker configuration, we observed the other case.
On 2026-07-26, "MLPendingAckStore ... stat reply fail!" repeated at very high frequency on all four brokers. In a 15-minute window, the approximate counts were:
- broker-0: 47,616
- broker-1: 1,450
- broker-2: 59,358
- broker-3: 31,149
Every sampled error was:
ManagedLedgerException:
Bookie handle is not available error code: -8
Affected pending-ack stores included:
<internal-tenant>/<internal-namespace>/<internal-topic>-v1
<sub-name>
and:
<internal-tenant>/<internal-namespace>/<internal-topic>-v1
<sub-name>
This is clearly the repeatedly failing-read case. It is not evidence of a lost callback because the callbacks were arriving with failures and replay was immediately issuing another read.
**Answers for the UAT incident:**
1. Again, we do not have the heap values.
2. Yes, "stat reply fail!" repeated thousands of times with plain ManagedLedgerException and BookKeeper error -8.
3. Yes. The cluster was in a broker restart loop. For example, broker-0 started at 06:28:37Z, and failures were repeating by 06:28:49Z.
4. We found no correlated skip, clear-backlog, explicit expiry request, or identifiable commit/abort operation for the affected subscriptions. There was extensive shutdown/unload/reload activity, together with BookKeeper unavailability.
5. The deployment configuration matches SIT, but the batched pending-ack value was not independently visible in the logs.
6. The affected topics were unloaded and subsequently recovered. Both subscriptions later logged pending ack state reply success!, and no further "stat reply fail!" errors appeared after approximately 06:30Z.
A separate silent PendingAckReplay shutdown stall occurred again in UAT on 2026-08-05 without a corresponding "stat reply fail!" storm. That reinforces the conclusion that we have encountered both states at different times: the silent no-more-entries stall and the repeatedly failing-read loop.
In the attachments you will find the test which I run to "prove" my thesis and the output of the test itself.
I will try to get the heap dump, but if anything else is needed please let me know as I am happy to help here in the investigations.
[test_output.log](https://github.com/user-attachments/files/31186134/test_output.log)
[DeletedPendingAckLedgerTest.java](https://github.com/user-attachments/files/31186141/DeletedPendingAckLedgerTest.java)
GitHub link: https://github.com/apache/pulsar/discussions/26364#discussioncomment-18068314
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org
Re: [D] Transactions: the pending-ack replay thread can get stuck forever after a lost ledger read [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
GitHub user lhotari added a comment to the discussion: Transactions: the pending-ack replay thread can get stuck forever after a lost ledger read
This is excellent — thank you. The two environments plus the reproduction settle a lot.
### Your reproduction is the *other* state, and #26369 does not fix it
I traced it against the PR head. Deleting the ledger gives
`BKNoSuchLedgerExistsOnMetadataServerException`, which `isBkErrorNotRecoverable` classifies as
non-recoverable, so `createManagedLedgerException` returns `LedgerNotExistException`. With
`autoSkipNonRecoverableData=false`, `OpReadEntry.internalReadEntriesFailed` skips its
advance-past-the-bad-ledger branch (that branch is gated on auto-skip) and fails the read **without
moving the read position**. In `readEntriesFailed` that exception matches none of the three cases that
clear `isReadable`, so the outstanding counter drops back to zero and `fillQueue()` issues the identical
read again.
Both of #26369's new exits are unreachable there: the completion branch requires
`hasMoreEntries() == false`, and it stays true forever because the read position never advances; the
in-loop close check requires a closed cursor, and the cursor is open. **So your reproduction would still
block the lane with the PR applied.** I did not add it to #26369 — a test that passes identically before
and after would be verifying nothing.
I filed it instead: **#26374**, with your reproduction and the UAT numbers as the evidence. Your test's
mock seams are the right skeleton for that fix's tests, so it was far from wasted — it is what pinned
the classification.
### Why that half was never fixed
Worth knowing, because it explains the shape: #12700's stated motivation was *exactly* your scenario —
*"if any ledger was deleted from bookkeeper, or ManagerLedger was fenced, MLPendingAckStore will not
stop recovering and continue to report the exception"*. It added the guard **and** the test that pins
it. But that test sets `autoSkipNonRecoverableData(true)`, so the deleted-ledger case was only ever
fixed for the non-default configuration. The default was left spinning, and that is what you hit.
That also means the fix does not need a design discussion, which I had assumed it might: the pinned test
covers only auto-skip-enabled, fenced, and closed-cursor. The two exception classes that actually cause
your storms are pinned by nothing, so they can be handled without touching the existing contract. Your
`-8` case (`BookieHandleNotAvailable`) is retryable and should back off — which also releases the replay
thread between attempts, curing the starvation; the deleted-ledger case is not retryable and should fail
the handle so it is visible.
### On the SIT stalls
Your reading matches mine. Silent, no error storm, brokers restarted 5–18 minutes earlier — that is the
no-more-entries state #26369 fixes, and the recent restarts fit its most likely cause. It is not proof
without the heap values, but I would not hold up the fix for them. If you do get a heap dump, the four
positions are still the decisive measurement.
One detail worth flagging from your point 6: *"broker shutdown unloaded topics, but the replay executor
still failed to terminate"*. That is consistent with the old code on both counts — the interrupt was
swallowed, so `shutdownNow()` could not stop it, and `cursor.isClosed()` was only checked once before
the loop, so unload could not either. Both are fixed in #26369, so that specific symptom should go away
regardless of which state caused the stall.
### Also in #26369 since you last looked
A reviewer spotted that a read still in flight when the replay ends could deliver entries afterwards
that nothing ever released. Fixed, with the caveat that the obvious fix is racy — the queue is
single-consumer, so a late callback must release its own entries rather than drain — plus a test.
Nothing further needed from you unless the heap dump turns up. Thanks for the rigour on this one; the
two-environment comparison is what made the classification unambiguous.
---
<sub>*Assisted-by: Claude (Opus 5, Fable) and OpenAI Codex (gpt-5.6-sol). Code paths, line numbers and
commit references were reviewed by me against the source before posting.*</sub>
GitHub link: https://github.com/apache/pulsar/discussions/26364#discussioncomment-18071245
----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org