You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by "Keith Wall (JIRA)" <ji...@apache.org> on 2015/08/07 19:34:46 UTC
[jira] [Comment Edited] (QPID-6683) Jetty thread seemingly spins if
many HTTP ports are defined on hosts with large numbers of core
[ https://issues.apache.org/jira/browse/QPID-6683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14662121#comment-14662121 ]
Keith Wall edited comment on QPID-6683 at 8/7/15 5:34 PM:
----------------------------------------------------------
The issue is that Qpid is constraining the Jetty *Server's* thread pool to 50 threads, and this is too few when many Jetty connectors are in use.
Here are the details:
# SelectChannelConnector#_accepts is defaults to Math.max(1,(Runtime.getRuntime().availableProcessors()+3)/4 acceptors. Qpid does not overriding this value.
# SelectorManager (SelectorManager#doStart) creates x number of "Selector" threads, where x is the SelectorManager#_selectSets. _selectSets itself is defaulted from SelectChannelConnector#_accepts
# AbstractConnector#doStart creates x number of "Acceptor" threads, where x is the SelectChannelConnector#_accepts.
So, in the falling case, on the 80 core machine, Jetty desires to create 2 * 20 acceptor + 2 * 20 selectors = 80 threads, but this exceeds the configured default 50 (from context variable "jetty.threadPool.maxThreads"). Increasing "jetty.threadPool.maxThreads" to 80 gets rid of the "insufficient threads configured" but the spinning "Dispatched Idle.." message remains. Jetty needs to be able schedule at least one other task (the Idle tick in SelectorManager#dispatch line 708) into its pool. With 81 threads all *seems* okay, but it is hard to be sure without a full inspection of the Jetty code (I can't find any docs that help). It is probably best to add some wriggle room just in case :)
It seems to me we should:
# Have a Jetty ThreadPool per Connector (rather than per Server)
# Size the ThreadPool with maximum and minimum thread pool sizes coming from HttpPort attributes (backed by context vars). Default sizes 5 and 50 respectively.
# Size SelectChannelConnector#_accepts to be (maximumThreadPool - wriggle room) / 2. Let wriggle room be say, 10.
# Context variables on HttpManagementConfiguration will be removed. This is an edge case, so I propose no upgrader change.
was (Author: k-wall):
The issue is that Qpid is constraining the Jetty *Server's* thread pool to 50 threads, and this is too few when many Jetty connectors are in use.
Here are the details:
# SelectChannelConnector#_accepts is defaults to Math.max(1,(Runtime.getRuntime().availableProcessors()+3)/4 acceptors. Qpid does not overriding this value.
# SelectorManager (SelectorManager#doStart) creates x number of "Selector" threads, where x is the SelectorManager#_selectSets. _selectSets itself is defaulted from SelectChannelConnector#_accepts
# AbstractConnector#doStart creates x number of "Acceptor" threads, where x is the SelectChannelConnector#_accepts.
So, in the falling case, on the 80 core machine, Jetty desires to create 2 * 20 acceptor + 2 * 20 selectors = 80 threads, but this exceeds the configured default 50 (from context variable "jetty.threadPool.maxThreads"). Increasing "jetty.threadPool.maxThreads" to 80 gets rid of the "insufficient threads configured" but the spinning "Dispatched Idle.." message remains. Jetty needs to be able schedule at least one other task (the Idle tick in SelectorManager#dispatch line 708) into its pool. With 81 threads all *seems* okay, but it is hard to be sure without a full inspection of the Jetty code (I can't find any docs that help). It is problem best to add some wriggle room just in case :)
It seems to me we should:
# Have a Jetty ThreadPool per Connector (rather than per Server)
# Size the ThreadPool with maximum and minimum thread pool sizes coming from HttpPort attributes (backed by context vars). Default sizes 5 and 50 respectively.
# Size SelectChannelConnector#_accepts to be (maximumThreadPool - wriggle room) / 2. Let wriggle room be say, 10.
# Context variables on HttpManagementConfiguration will be removed. This is an edge case, so I propose no upgrader change.
> Jetty thread seemingly spins if many HTTP ports are defined on hosts with large numbers of core
> -----------------------------------------------------------------------------------------------
>
> Key: QPID-6683
> URL: https://issues.apache.org/jira/browse/QPID-6683
> Project: Qpid
> Issue Type: Bug
> Components: Java Broker
> Affects Versions: 0.32
> Reporter: Keith Wall
> Assignee: Keith Wall
> Fix For: qpid-java-6.0
>
>
> If I configure many HTTP management ports on a box with a large number of cores (80), I find that one of the two ports will be non-functional (produce a timeout in the browser/blank page). Turning on Jetty logs reveals a Jetty WARN during startup, during the attempt to configure the second port:
> {noformat}
> 2015-08-07 10:30:21,963 WARN [Broker-Config] (o.e.j.s.AbstractConnector) - insufficient threads configured for @0.0.0.0:8080
> {noformat}
> then evidence of a spinning Jetty thread once the connect
> {noformat}
> 2015-08-07 11:35:32,737 DEBUG [HttpManagement-226 Selector2] (o.e.j.u.t.QueuedThreadPool) - Dispatched Idle-org.eclipse.jetty.io.nio.SelectorManager$SelectSet$1@37ca01b0 to stopped HttpManagement{5<=0<=50/50,1000}
> 2015-08-07 11:35:32,737 DEBUG [HttpManagement-233 Selector9] (o.e.j.u.t.QueuedThreadPool) - Dispatched Idle-org.eclipse.jetty.io.nio.SelectorManager$SelectSet$1@2f34e66d to stopped HttpManagement{5<=0<=50/50,1000}
> 2015-08-07 11:35:32,737 DEBUG [HttpManagement-228 Selector4] (o.e.j.u.t.QueuedThreadPool) - Dispatched Idle-org.eclipse.jetty.io.nio.SelectorManager$SelectSet$1@3a041651 to stopped HttpManagement{5<=0<=50/50,1000}
> 2015-08-07 11:35:32,737 DEBUG [HttpManagement-229 Selector5] (o.e.j.u.t.QueuedThreadPool) - Dispatched Idle-org.eclipse.jetty.io.nio.SelectorManager$SelectSet$1@3a041651 to stopped HttpManagement{5<=0<=50/50,1000}
> 2015-08-07 11:35:32,738 DEBUG [HttpManagement-230 Selector6] (o.e.j.u.t.QueuedThreadPool) - Dispatched Idle-org.eclipse.jetty.io.nio.SelectorManager$SelectSet$1@62553d98 to stopped HttpManagement{5<=0<=50/50,1000}
> 2015-08-07 11:35:32,738 DEBUG [HttpManagement-225 Selector1] (o.e.j.u.t.QueuedThreadPool) - Dispatched Idle-org.eclipse.jetty.io.nio.SelectorManager$SelectSet$1@4a3ef897 to stopped HttpManagement{5<=0<=50/50,1000}
> 2015-08-07 11:35:32,740 DEBUG [HttpManagement-232 Selector8] (o.e.j.u.t.QueuedThreadPool) - Dispatched Idle-org.eclipse.jetty.io.nio.SelectorManager$SelectSet$1@6c32a795 to stopped HttpManagement{5<=0<=50/50,1000}
> 2015-08-07 11:35:32,742 DEBUG [HttpManagement-227 Selector3] (o.e.j.u.t.QueuedThreadPool) - Dispatched Idle-org.eclipse.jetty.io.nio.SelectorManager$SelectSet$1@71985eb1 to stopped HttpManagement{5<=0<=50/50,1000}
> 2015-08-07 11:35:32,745 DEBUG [HttpManagement-231 Selector7] (o.e.j.u.t.QueuedThreadPool) - Dispatched Idle-org.eclipse.jetty.io.nio.SelectorManager$SelectSet$1@4a70f0ba to stopped HttpManagement{5<=0<=50/50,1000}
> {noformat}
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org