You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "void-ptr974 (via GitHub)" <gi...@apache.org> on 2026/10/18 02:59:42 UTC

[PR] [fix][broker] Fix managed-ledger scheduler thread stalls when estimating offline backlog [pulsar]

void-ptr974 opened a new pull request, #26972:
URL: https://github.com/apache/pulsar/pull/26972

   Related: #26817 also changes offline-backlog handle ownership and removes the inner callback wait in its current head. This PR targets current `master` with an asynchronous estimator used by REST and explicit timeout/failure publication semantics. The overlapping offline-backlog changes will need reconciliation if both PRs proceed; #26817's broader initialization and recovery cleanup is outside this change.
   
   ### Motivation
   
   An offline backlog query can stall a shared managed-ledger metadata callback worker and return before cursor backlog reads finish, even when the metadata service and BookKeeper are healthy. This affects estimation on a cache miss or expired cache entry when the managed ledger and cursor lists are non-empty; it does not require a large backlog or concurrent requests.
   
   `calculateCursorBacklogs` executes `cursorCounter.await()` inside the `getCursors` callback. Both that callback and the cursor metadata callbacks needed to decrement the latch are dispatched through `executor.chooseThread(ledgerName)`. The current callback therefore waits for work queued behind itself on the same single-threaded worker.
   
   The REST endpoint uses `accurate=false`, so the callback worker remains blocked until its 60-second wait expires. The caller has a separate 60-second wait and can return before cursor reads complete. In a local reproduction using real ZooKeeper and BookKeeper, estimation took 60.022 seconds and returned `messageBacklog=0` for three unacknowledged entries; independently queued operations on the same worker were delayed for 60.006 seconds. The underlying `accurate=true` API has an unbounded wait and the same self-deadlock. That case was observed in the blocked state and then interrupted for test cleanup.
   
   The impact extends beyond the queried topic. Other ledgers assigned to the same factory scheduler worker also depend on that worker for metadata completion callbacks. A separate local fixture confirmed that another topic's cursor metadata callback and managed-ledger open remained pending while a topic on a different worker and raw metadata-store reads continued to complete. This is a shared callback-executor stall, rather than evidence that the metadata backend or every broker I/O thread has stopped.
   
   Statistics can also be returned and cached while late callbacks continue to modify them. Concurrent cursor callbacks update a shared backlog counter and ordinary map without a single aggregation point. Source inspection found the same await/dispatch pattern in the locally available `v4.0.12` and `v4.2.3` tag sources; those release tags were not runtime-tested here.
   
   ### Modifications
   
   - Compose managed-ledger metadata, cursor metadata, and BookKeeper reads through futures, allowing the ordered callback worker to return instead of waiting for its own queued callbacks.
   - Collect independent cursor results and aggregate them once, after successful completion, before publishing statistics.
   - Apply a 60-second asynchronous timeout to non-accurate estimation. Timeout and metadata, ledger-read, parsing, or handle-close failures complete exceptionally without publishing partially collected statistics. Late I/O completions retain private results and still release their handles.
   - Connect the backlog REST continuation to the asynchronous estimator and cache results only after successful completion.
   - Close read handles opened by the query on success and failure, and check a failed cursor-ledger open before dereferencing its handle.
   - Keep the blocking BookKeeperAdmin fallback for accurate estimates off metadata and BookKeeper callback threads by using the supplied executor. Synchronous callers perform this fallback on their own calling thread.
   - Preserve existing synchronous signatures and the accumulation behavior of `calculateCursorBacklogs`. Add a default asynchronous adapter so existing `ManagedLedgerFactory` implementations do not need a new abstract method.
   
   The REST path and response schema remain unchanged. One observable behavior change is intentional: a timed-out or failed estimate now returns an error instead of a successful response containing statistics collected before all required reads completed. Timeout currently follows the existing generic REST error mapping. Non-accurate mode remains approximate: an unknown cursor LAC is represented by a cursor backlog of `-1` and excluded from the total.
   
   ### Verifying this change
   
   - [ ] Make sure that the change passes the CI checks.
   
   This change added tests and can be verified as follows:
   
   - The two liveness regression cases (`accurate=true` and `false`) fail against the unpatched estimator on latest `master` (`e9225d6c4b7`) with a timeout at the actual callback wait. Both pass with this change.
   - Seventeen managed-ledger regression invocations cover callback-thread liveness, concurrent completion of mixed metadata/BK cursors, timeout followed by late completion, empty ledger/cursor lists, metadata and cursor-ledger failures, unknown LAC handling, handle closure, and synchronous accumulation compatibility.
   - Five default-adapter/helper tests cover legacy implementations, executor dispatch, failure propagation, validation, and snapshot publication.
   - The existing real-BookKeeper backlog test now creates and queries the same persistence path and checks the actual backlog count instead of only checking for a non-null result.
   - Three REST unit tests cover pending estimates, success-only caching, failure without caching, and cache hits.
   
   Local verification passed: 26 targeted test invocations, without automatic retries, and `quickCheck`.
   
   ```bash
   ./gradlew :managed-ledger:test \
     --tests org.apache.bookkeeper.mledger.impl.ManagedLedgerOfflineBacklogTest \
     --tests org.apache.bookkeeper.mledger.ManagedLedgerFactoryBacklogTest \
     --tests org.apache.bookkeeper.mledger.impl.ManagedLedgerBkTest.testOfflineTopicBacklog \
     -PtestRetryCount=0
   ./gradlew :pulsar-broker:test \
     --tests org.apache.pulsar.broker.admin.impl.PersistentTopicsBacklogTest \
     -PtestRetryCount=0
   ./gradlew quickCheck
   ```
   
   The real-backend reproduction calls the estimator used by REST directly. The cross-topic diagnostic uses an in-memory metadata store and mock BookKeeper. Neither is an HTTP end-to-end load test; no full CI or release-version runtime matrix is claimed.
   
   ### Does this pull request potentially affect one of the following parts:
   
   - [ ] Dependencies (add or upgrade a dependency)
   - [x] The public API
   - [ ] The schema
   - [ ] The default values of configurations
   - [x] The threading model
   - [ ] The binary protocol
   - [x] The REST endpoints
   - [ ] The admin CLI options
   - [ ] The metrics
   - [ ] Anything that affects deployment
   
   The public API addition is a default asynchronous method; existing synchronous signatures remain available. The threading changes are confined to offline backlog estimation and its REST continuation. Metadata callback routing and ordering are unchanged. REST failures now propagate instead of caching a partially collected successful result.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [fix][broker] Fix managed-ledger scheduler thread stalls when estimating offline backlog [pulsar]

Posted by "codelipenghui (via GitHub)" <gi...@apache.org>.
codelipenghui merged PR #26972:
URL: https://github.com/apache/pulsar/pull/26972


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [fix][broker] Fix managed-ledger scheduler thread stalls when estimating offline backlog [pulsar]

Posted by "void-ptr974 (via GitHub)" <gi...@apache.org>.
void-ptr974 commented on PR #26972:
URL: https://github.com/apache/pulsar/pull/26972#issuecomment-6245007779

   Related: #26817 also changes offline-backlog handle ownership and removes the inner callback wait in its current head. This PR targets current `master` with an asynchronous estimator used by REST and explicit timeout/failure publication semantics. The overlapping offline-backlog changes will need reconciliation if both PRs proceed; #26817's broader initialization and recovery cleanup is outside this change.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org