[PR] [improve][offload] Add queued latency metrics for offloader executors [pulsar]

["BewareMyPower (via GitHub)" <gi...@apache.org>]

BewareMyPower opened a new pull request, #25321:
URL: https://github.com/apache/pulsar/pull/25321

   ### Motivation
   
   `BlobStoreManagedLedgerOffloader` uses `OrderedScheduler` instances for both offload/write work and read work, but those executors currently do not expose an offloader-specific queued-latency metric.
   
   The jcloud offloader runs blocking blob-store operations on these executors:
   - `scheduler` handles synchronous offload and delete work.
   - `readExecutor` handles blocking offloaded-ledger open/read/close work.
   
   When these tasks queue up behind other blocking work, it is hard to tell from the existing offloader metrics whether the executors themselves are saturated.
   
   ### Modifications
   
   This change adds two new offloader metrics:
   - `brk_ledgeroffloader_offload_executor_queue_latency`
   - `brk_ledgeroffloader_read_offload_executor_queue_latency`
   
   The implementation:
   - Extends `LedgerOffloaderStats` and `LedgerOffloaderStatsImpl` with the two queue-latency summaries.
   - Adds a small wrapper utility to record executor queue wait time when a task actually starts running.
   - Instruments the jcloud offloader paths that submit blocking work to `scheduler` and `readExecutor`.
   - Instruments the offloaded read-handle implementations so queued latency is also tracked for read and close tasks after the handle is opened.
   - Extends existing tests to assert the new metrics are recorded.
   
   The queue-latency metrics use the same topic/namespace label behavior as the existing ledger offloader metrics.
   
   ### Verifying this change
   
   - [ ] Make sure that the change passes the CI checks.
   
   This change added tests and can be verified as follows:
   
   - `./mvnw -pl managed-ledger -Dmaven.test.skip=true install`
   - `./mvnw -pl tiered-storage/jcloud -Dtest=BlobStoreManagedLedgerOffloaderTest,BlobStoreBackedReadHandleImplTest -Dsurefire.failIfNoSpecifiedTests=false test`
   
   Notes:
   - The jcloud targeted test run passed locally: 26 tests, 0 failures.
   - The updated `pulsar-broker` test class was not run locally because this checkout currently has unrelated compile failures in `pulsar-broker` / `pulsar-metadata`.
   
   ### Does this pull request potentially affect one of the following parts:
   
   - [ ] Dependencies (add or upgrade a dependency)
   - [ ] The public API
   - [ ] The schema
   - [ ] The default values of configurations
   - [ ] The threading model
   - [ ] The binary protocol
   - [ ] The REST endpoints
   - [ ] The admin CLI options
   - [x] The metrics
   - [ ] Anything that affects deployment
   
   ### Documentation
   
   - [ ] `doc`
   - [ ] `doc-required`
   - [x] `doc-not-needed`
   - [ ] `doc-complete`
   
   ### Matching PR in forked repository
   
   PR in forked repository: N/A


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [improve][offload] PIP-461: Add queued latency metrics for offloader executors [pulsar]

["merlimat (via GitHub)" <gi...@apache.org>]

merlimat commented on PR #25321:
URL: https://github.com/apache/pulsar/pull/25321#issuecomment-6182171289

   Hi @BewareMyPower, thanks for working on the offloader executor metrics, and sorry it never got a proper review. The PIP-461 proposal (#25322) was discussed on dev@ but never went to a vote, and it has now been closed, so this implementation can't move forward as it is. We're going through the backlog of open PRs (https://lists.apache.org/thread/cgy7w0qgnq8s7j0ykyzgrn3hj4nb67wf), so I'm closing this one. If you'd like to continue, feel free to start a [VOTE] on dev@ and reopen both PRs rebased on current master.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [improve][offload] Add queued latency metrics for offloader executors [pulsar]

["BewareMyPower (via GitHub)" <gi...@apache.org>]

BewareMyPower commented on PR #25321:
URL: https://github.com/apache/pulsar/pull/25321#issuecomment-4060462077

   I will write a proposal for it since it adds new metrics


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [improve][offload] PIP-461: Add queued latency metrics for offloader executors [pulsar]

["merlimat (via GitHub)" <gi...@apache.org>]

merlimat closed pull request #25321: [improve][offload] PIP-461: Add queued latency metrics for offloader executors
URL: https://github.com/apache/pulsar/pull/25321


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org