You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/02/16 17:28:30 UTC
[GitHub] [pulsar] lhotari opened a new issue #14329: PIP-142 Increase default numHttpServerThreads value to 200 to prevent Admin API unavailability
lhotari opened a new issue #14329:
URL: https://github.com/apache/pulsar/issues/14329
<!---
Instructions for creating a PIP using this issue template:
1. The author(s) of the proposal will create a GitHub issue ticket using this template.
(Optionally, it can be helpful to send a note discussing the proposal to
dev@pulsar.apache.org mailing list before submitting this GitHub issue. This discussion can
help developers gauge interest in the proposed changes before formalizing the proposal.)
2. The author(s) will send a note to the dev@pulsar.apache.org mailing list
to start the discussion, using subject prefix `[PIP] xxx`. To determine the appropriate PIP
number `xxx`, inspect the mailing list (https://lists.apache.org/list.html?dev@pulsar.apache.org)
for the most recent PIP. Add 1 to that PIP's number to get your PIP's number.
3. Based on the discussion and feedback, some changes might be applied by
the author(s) to the text of the proposal.
4. Once some consensus is reached, there will be a vote to formally approve
the proposal. The vote will be held on the dev@pulsar.apache.org mailing list. Everyone
is welcome to vote on the proposal, though it will considered to be binding
only the vote of PMC members. It will be required to have a lazy majority of
at least 3 binding +1s votes. The vote should stay open for at least 48 hours.
5. When the vote is closed, if the outcome is positive, the state of the
proposal is updated and the Pull Requests associated with this proposal can
start to get merged into the master branch.
-->
## Motivation
Since Pulsar Admin API uses the blocking servlet API, all Jetty threads might be occupied and this causes unavailability on the Pulsar Admin API. The default value for the maximum number of threads for Jetty is too low in Pulsar. That is the root cause of many problems where Pulsar Admin API is unavailable when all threads are in use.
## Additional context
- Examples of previous issues where Jetty threads have been occupied and caused problems: #13666 #4756 #10619
- Mailing list thread about "make async" changes: https://lists.apache.org/thread/tn7rt59cd1k724l4ytfcmzx1w2sbtw7l
## Implementation
- Jetty defaults to 200 maximum threads, to prevent thread pool starvation. Make Pulsar use the same default value by setting `numHttpServerThreads=200`.
- Update the documentation for `numHttpServerThreads`
- The PR is already in place: https://github.com/apache/pulsar/pull/14320
- Set Jetty selectors and acceptors parameters to `-1` so that Jetty automatically chooses optimal values based on available cores. The rationale is explained in the Q&A below.
- A separate PR will be made for this change.
## Q&A
### Q: What's the reason of setting the default value to 200? If the node just have one core, what will happen?
These are threads. Jetty defaults to 200 maximum threads, to prevent thread pool starvation. This is recommended when using blocking Servlet API. The problem is that Pulsar uses the blocking servlet API and doesn't have a sufficient number of threads which are needed and recommended.
The value 200 doesn't mean that there will be 200 threads to start with. This is the maximum size for the thread pool. When the value is more than 8, Jetty will start with 8 initial threads and add more threads to the pool when all threads are occupied.
### Q: Do we need to take the number of system cores into consideration for the maximum threads of the thread pool?
No. Jetty is different from Netty in this aspect. In Netty, everything should be asynchronous and "thou shall never block". In Jetty, the maximum number of threads for the thread pool should be set to 50-500 threads and blocking operations are fine.
The recommendation for the thread pool is explained in Jetty documentation https://www.eclipse.org/jetty/documentation/jetty-9/index.html#_thread_pool
> Thread Pool
> Configure with goal of limiting memory usage maximum available. Typically this is >50 and <500
However, there are separate settings which should take the number of available processors (cores) into account in Jetty.
http port acceptor and selector count:
https://github.com/apache/pulsar/blob/b540523b474e4194e30c1acab65dfafdd11d3210/pulsar-broker/src/main/java/org/apache/pulsar/broker/web/WebService.java#L88
https port acceptor and selector count:
https://github.com/apache/pulsar/blob/b540523b474e4194e30c1acab65dfafdd11d3210/pulsar-broker/src/main/java/org/apache/pulsar/broker/web/WebService.java#L125
Jetty [documentantion for acceptors](https://www.eclipse.org/jetty/documentation/jetty-9/index.html#_acceptors):
> Acceptors
> The standard rule of thumb for the number of Accepters to configure is one per CPU on a given machine.
Jetty [documentation for selectors](https://www.eclipse.org/jetty/javadoc/jetty-9/org/eclipse/jetty/server/ServerConnector.html):
> Selectors
> The default number of selectors is equal to half of the number of processors available to the JVM, which should allow optimal performance even if all the connections used are performing significant non-blocking work in the callback tasks.
The settings in jetty are the "acceptor" and "selector" thread count settings. These have been fixed to 1 in Pulsar.
The `acceptors` and `selectors` settings should be both set to -1. Jetty would pick the recommended count based on cores in that case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #14329: PIP-142 Increase default numHttpServerThreads value to 200 to prevent Admin API unavailability
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #14329:
URL: https://github.com/apache/pulsar/issues/14329#issuecomment-1073147962
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] frankjkelly commented on issue #14329: PIP-142 Increase default numHttpServerThreads value to 200 to prevent Admin API unavailability
Posted by GitBox <gi...@apache.org>.
frankjkelly commented on issue #14329:
URL: https://github.com/apache/pulsar/issues/14329#issuecomment-1073983064
Bump
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org