You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "ethqunzhong (via GitHub)" <gi...@apache.org> on 2025/10/13 03:52:10 UTC
[PR] [improve][offload]use dedicated bookkeeper-ml-offload-scheduler to avoid any potential … [pulsar]
ethqunzhong opened a new pull request, #24839:
URL: https://github.com/apache/pulsar/pull/24839
This PR is for running tests for upstream PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][offload]use dedicated bookkeeper-ml-offload-scheduler to avoid any potential … [pulsar]
Posted by "ethqunzhong (via GitHub)" <gi...@apache.org>.
ethqunzhong commented on PR #24839:
URL: https://github.com/apache/pulsar/pull/24839#issuecomment-3400702462
> Offloader already have a separate threadpool, the configuration key is `managedLedgerOffloadMaxThreads`, and it's default value is 2. Why add a new threadpool?
Strictly speaking, the thread pool configured by `managedLedgerOffloadMaxThreads`(use at `FSOffloader.scheduler`) is used in the offloader instance to perform cold storage operations for the sequential reading and writing of specific ledgers. The thread pool newly added in this modification is used for scheduling the offload operations of all ManagedLedgers at the Broker level—a capability that was previously handled by `ManagedLedgerImpl#scheduledExecutor`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][offload]use dedicated bookkeeper-ml-offload-scheduler to avoid block core service [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat closed pull request #24839: [improve][offload]use dedicated bookkeeper-ml-offload-scheduler to avoid block core service
URL: https://github.com/apache/pulsar/pull/24839
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][offload]use dedicated bookkeeper-ml-offload-scheduler to avoid block core service [pulsar]
Posted by "dao-jun (via GitHub)" <gi...@apache.org>.
dao-jun commented on PR #24839:
URL: https://github.com/apache/pulsar/pull/24839#issuecomment-3404347765
Maybe add a Backoff to https://github.com/apache/pulsar/blob/master/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java#L3702-L3704 is better? To avoid frequent retries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][offload]use dedicated bookkeeper-ml-offload-scheduler to avoid any potential … [pulsar]
Posted by "ethqunzhong (via GitHub)" <gi...@apache.org>.
ethqunzhong closed pull request #24839: [improve][offload]use dedicated bookkeeper-ml-offload-scheduler to avoid any potential …
URL: https://github.com/apache/pulsar/pull/24839
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][offload]use dedicated bookkeeper-ml-offload-scheduler to avoid any potential … [pulsar]
Posted by "dao-jun (via GitHub)" <gi...@apache.org>.
dao-jun commented on PR #24839:
URL: https://github.com/apache/pulsar/pull/24839#issuecomment-3400117013
Offloader already have a separate threadpool, the configuration key is `managedLedgerOffloadMaxThreads`, and it's default value is 2. Why add a new threadpool?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][offload]use dedicated bookkeeper-ml-offload-scheduler to avoid block core service [pulsar]
Posted by "hangc0276 (via GitHub)" <gi...@apache.org>.
hangc0276 commented on PR #24839:
URL: https://github.com/apache/pulsar/pull/24839#issuecomment-4100562285
@ethqunzhong, could you provide more stack to show how the ordered-scheduler threads run into the Hadoop retry?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][offload]use dedicated bookkeeper-ml-offload-scheduler to avoid any potential … [pulsar]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #24839:
URL: https://github.com/apache/pulsar/pull/24839#issuecomment-3395769613
@ethqunzhong Please add the following content to your PR description and select a checkbox:
```
- [ ] `doc` <!-- Your PR contains doc changes -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][offload]use dedicated bookkeeper-ml-offload-scheduler to avoid block core service [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on PR #24839:
URL: https://github.com/apache/pulsar/pull/24839#issuecomment-6182185800
@ethqunzhong thanks for working on this. The repeated automatic offload retries that kept the managed ledger scheduler busy have since been reduced on master by #25793, which keeps at most one automatic offload in flight per managed ledger and merges repeated triggers instead of retrying every 100ms. We're going through the backlog of open PRs (https://lists.apache.org/thread/cgy7w0qgnq8s7j0ykyzgrn3hj4nb67wf), so I'm closing this one. If you still see the scheduler blocked on current master, please open an issue with the stack trace hangc0276 asked for.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org