You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tomcat.apache.org by Mark Thomas <ma...@apache.org> on 2011/05/03 22:02:01 UTC
BIO performance issues
All,
Summary
-------
While trying to align the documentation for maxConnections with the
actual implementation a couple of performance issue have been identified
with the BIO connector. All bar one of these have been fixed. A fix
needs to be agreed for the remaining issue, ideally before the next
7.0.x release.
Background
----------
New connections to the BIO connector are placed in a queue. When a
thread is available, it takes the next connection of the queue and
processes it. When the connection has been processed if the connection
is kept-alive it is placed back on the queue. If there is no keep-alive
the connection is closed.
Scenario
--------
This ended up being very long, so I moved it to the end. The exact
pattern of delays will vary depending on timeouts, request frequency
etc. but the scenario shows an example of how delays can occur. The
short version is that requests with data to process (particularly new
connections) tend to get delayed in the queue waiting for a thread to
process them when the threads are all tied up processing keep-alive
connections.
Root cause
----------
The underlying cause of all of the performance issues observed is when
the threads are tied up doing HTTP keep-alive when there is no data
process but there are other connections in the queue that do have data
that could be processed.
Solution A
----------
NIO is designed to handle this using a poller. That isn't available to
BIO so I attempted to simulate it. That generated excessive CPU load so
I do not think simulated polling is the tight solution.
Solution B
----------
Return to the Tomcat 6 implementation where maxConnections == maxThreads.
Additional clean-up
-------------------
maxConnections is unnecessary in APR since pollerSize performs the same
function.
Summary
-------
The proposed changes are:
a) restore disabling keep-alive when threads used >= 75% of maxThreads
b) remove maxConnections and associated code from the APR connector
c) remove the configuration options for maxConnections from the BIO
connector
d) use maxThreads instead of maxConnections for the BIO connector
e) update the docs
Mark
Scenario
--------
BIO connector, default configuration
- maxThreads=200
- maxConnections=10000
- keepAliveTimeout=connectionTimeout=20000 (20s)
When there are less than maxThreads current connections, no issues are
observed.
Consider the following scenario:
- Start from zero load.
- Clients make 1 request every 5 seconds and use HTTP keep-alive
- Clients ramp up from 0 to 1000 at the rate of 100 new clients every
second. I'll call each block of 100 clients group A, B, C, ..., J.
- Assume request processing takes 0s just to keep the numbers simple.
I've tried several ways to express this. This is verbose, but hopefully
clear. There may be some mistakes in the detail but the overall result
is consistent with observations of some JMeter test runs.
Time: 0s
Activity: None
Queue: None
Threads: None
Group A: Inactive
Group B: Inactive
Group C: Inactive
Group D: Inactive
Group E: Inactive
Group F: Inactive
Group G: Inactive
Group H: Inactive
Group I: Inactive
Group J: Inactive
Time: 1s
Activity: Group A clients connect and are added to the queue
100 threads are allocated to process the A connections
The A connections are processed and returned to the queue
100 threads are allocated to process the A connections
Queue: None
Threads: A
Group A: Allocated threads. In HTTP keep-alive for 0s
Group B: Inactive
Group C: Inactive
Group D: Inactive
Group E: Inactive
Group F: Inactive
Group G: Inactive
Group H: Inactive
Group I: Inactive
Group J: Inactive
Time: 2s
Activity: Group B clients connect and are added to the queue
100 threads are allocated to process the B connections
The B connections are processed and returned to the queue
100 threads are allocated to process the B connections
Queue: None
Threads: A, B
Group A: Allocated threads. In HTTP keep-alive for 1s
Group B: Allocated threads. In HTTP keep-alive for 0s
Group C: Inactive
Group D: Inactive
Group E: Inactive
Group F: Inactive
Group G: Inactive
Group H: Inactive
Group I: Inactive
Group J: Inactive
Time: 3s
Activity: Group C clients connect and are added to the queue
Queue: C
Threads: A, B
Group A: Allocated threads. In HTTP keep-alive for 2s
Group B: Allocated threads. In HTTP keep-alive for 1s
Group C: Queued for 0s (no threads available)
Group D: Inactive
Group E: Inactive
Group F: Inactive
Group G: Inactive
Group H: Inactive
Group I: Inactive
Group J: Inactive
Time: 4s
Activity: Group D clients connect and are added to the queue
Queue: C, D
Threads: A, B
Group A: Allocated threads. In HTTP keep-alive for 3s
Group B: Allocated threads. In HTTP keep-alive for 2s
Group C: Queued for 1s
Group D: Queued for 0s
Group E: Inactive
Group F: Inactive
Group G: Inactive
Group H: Inactive
Group I: Inactive
Group J: Inactive
Time: 5s
Activity: Group E clients connect and are added to the queue
Queue: C, D, E
Threads: A, B
Group A: Allocated threads. In HTTP keep-alive for 4s
Group B: Allocated threads. In HTTP keep-alive for 3s
Group C: Queued for 2s
Group D: Queued for 1s
Group E: Queued for 0s
Group F: Inactive
Group G: Inactive
Group H: Inactive
Group I: Inactive
Group J: Inactive
Time: 6s
Activity: Group F clients connect and are added to the queue
A connections send requests
A connections are processed and returned to the queue
C connections are processed and returned to the queue
D connections are processed and returned to the queue
E connections are processed and returned to the queue
F connections are processed and returned to the queue
A connections are allocated threads but are in keep-alive
Queue: C, D, E, F
Threads: A, B
Group A: Allocated threads. In HTTP keep-alive for 0s
Group B: Allocated threads. In HTTP keep-alive for 4s
Group C: Queued. In HTTP keep-alive for 0s
Group D: Queued. In HTTP keep-alive for 0s
Group E: Queued. In HTTP keep-alive for 0s
Group F: Queued. In HTTP keep-alive for 0s
Group G: Inactive
Group H: Inactive
Group I: Inactive
Group J: Inactive
Time: 7s
Activity: Group G clients connect and are added to the queue
B connections send requests
B connections are processed and returned to the queue
C connections are allocated threads but are in keep-alive
Queue: D, E, F, G, B
Threads: C, A
Group A: Allocated threads. In HTTP keep-alive for 1s
Group B: Queued. In HTTP keep-alive for 0s
Group C: Allocated threads. In HTTP keep-alive for 1s
Group D: Queued. In HTTP keep-alive for 1s
Group E: Queued. In HTTP keep-alive for 1s
Group F: Queued. In HTTP keep-alive for 1s
Group G: Queued for 0s
Group H: Inactive
Group I: Inactive
Group J: Inactive
Time: 8s
Activity: Group H clients connect and are added to the queue
Queue: D, E, F, G, B, H
Threads: C, A
Group A: Allocated threads. In HTTP keep-alive for 2s
Group B: Queued. In HTTP keep-alive for 1s
Group C: Allocated threads. In HTTP keep-alive for 2s
Group D: Queued. In HTTP keep-alive for 2s
Group E: Queued. In HTTP keep-alive for 2s
Group F: Queued. In HTTP keep-alive for 2s
Group G: Queued for 1s
Group H: Queued for 0s
Group I: Inactive
Group J: Inactive
Time: 9s
Activity: Group I clients connect and are added to the queue
Queue: D, E, F, G, B, H, I
Threads: C, A
Group A: Allocated threads. In HTTP keep-alive for 3s
Group B: Queued. In HTTP keep-alive for for 2s
Group C: Allocated threads. In HTTP keep-alive for 3s
Group D: Queued. In HTTP keep-alive for 3s
Group E: Queued. In HTTP keep-alive for 3s
Group F: Queued. In HTTP keep-alive for 3s
Group G: Queued for 2s
Group H: Queued for 1s
Group I: Queued for 0s
Group J: Inactive
Time: 10s
Activity: Group J clients connect and are added to the queue
Queue: D, E, F, G, B, H, I, J
Threads: C, A
Group A: Allocated threads. In HTTP keep-alive for 4s
Group B: Queued. In HTTP keep-alive for for 3s
Group C: Allocated threads. In HTTP keep-alive for 4s
Group D: Queued. In HTTP keep-alive for 4s
Group E: Queued. In HTTP keep-alive for 4s
Group F: Queued. In HTTP keep-alive for 4s
Group G: Queued for 3s
Group H: Queued for 2s
Group I: Queued for 1s
Group J: Queued for 0s
Time: 11s
Activity: C connections send requests
C connections are processed and returned to the queue
A connections send requests
A connections are processed and returned to the queue
D connections send requests
D connections are processed and returned to the queue
E connections send requests
E connections are processed and returned to the queue
F connections send requests
F connections are processed and returned to the queue
G connections are processed and returned to the queue
B connections are allocated threads but are in keep-alive
H connections are processed and returned to the queue
I connections are processed and returned to the queue
J connections are processed and returned to the queue
C connections are allocated threads but are in keep-alive
Queue: A, D, E, F, G, H, I, J
Threads: B, C
Group A: Queued. In HTTP keep-alive for 0s
Group B: Allocated threads. In HTTP keep-alive for 4s
Group C: Allocated threads. In HTTP keep-alive for 0s
Group D: Queued. In HTTP keep-alive for 0s
Group E: Queued. In HTTP keep-alive for 0s
Group F: Queued. In HTTP keep-alive for 0s
Group G: Queued. In HTTP keep-alive for 0s
Group H: Queued. In HTTP keep-alive for 0s
Group I: Queued. In HTTP keep-alive for 0s
Group J: Queued. In HTTP keep-alive for 0s
The issues are:
At time 6s:
- 3s delay processing C first requests
- 2s delay processing D first requests
- 1s delay processing E first requests
At time 11s:
- 4s delay processing G first requests
- 3s delay processing H first requests
- 2s delay processing I first requests
- 1s delay processing J first requests
The delays are particularly bad for new connections which must wait
behind all the other requests in the queue.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: BIO performance issues
Posted by Mark Thomas <ma...@apache.org>.
On 04/05/2011 15:20, Rainer Jung wrote:
> I hope the following is not too long and confusing ...
Not at all.
> On 03.05.2011 22:02, Mark Thomas wrote:
>> Solution A
>> ----------
>> NIO is designed to handle this using a poller. That isn't available to
>> BIO so I attempted to simulate it. That generated excessive CPU load so
>> I do not think simulated polling is the tight solution.
>
> I expect generating the SocketTimeoutException is expensive, because the
> JVM has to generate the stack information. The rate of the Exception
> when handling mostly keep-alive (extreme case) is your "poll" timeout
> times the number of threads, e.g. 100ms timeout times 200 threads is
> 2000 exceptions per second. Even if there is another reason for the high
> CPU load, I expect it to be roughly proportional to the poll rate.
<snip/>
> Do we see a cpu time and thread blocking time efficient way of handling
> many keep-alive connections? I don't see any API, that would help here.
> Of course one could try to build a hyprid "blocking for normal
> processing but non-blocking for keep-alive" thing, but since we already
> have NIO I would also support recommending NIO for keep-alive.
The only API I see is NIO and that is what the NIO connector is for. I
don't see an efficient way to do this with the BIO API.
> Switching the default from BIO to NIO is a big change, but only after we
> switch will we find the last buglets and problems arising under rare
> conditions. So if we want to switch, we should do it very soon. Doing it
> late in the TC 7 cycle would be bad.
I think we are beyond the point where we can change the default HTTP
connector to NIO for Tomcat 7 (as much as part of me really, really
wants to).
> Lastly: APR uses server to server connections, as does HTTP when using a
> reverse proxy in front of Tomcat. In those cases we have much fewer
> connections with a higher rate of requests per connection. There
> maxThreads == maxConnections is fine (and even the 75% rule could be
> switched off).
It is. The 75% limit only applies to the HTTP BIO connector.
> So for this scenario it would be nice to not drop BIO, at
> least until the major TC version after the default switched to NIO.
We need a AJP-NIO connector first ;). Maybe an exercise for the upcoming
Apache Retreat?
The one huge advantage of AJP-NIO would be no need for aligning thread
counts on httpd and Tomcat or messing about with timeouts etc to ensure
connections <= maxThreads. As long as the *active* connections from
httpd <= maxThreads, all would be fine.
>> Solution B
>> ----------
>> Return to the Tomcat 6 implementation where maxConnections == maxThreads.
>>
>> Additional clean-up
>> -------------------
>> maxConnections is unnecessary in APR since pollerSize performs the same
>> function.
>>
>> Summary
>> -------
>> The proposed changes are:
>> a) restore disabling keep-alive when threads used>= 75% of maxThreads
>> b) remove maxConnections and associated code from the APR connector
>> c) remove the configuration options for maxConnections from the BIO
>> connector
>> d) use maxThreads instead of maxConnections for the BIO connector
>> e) update the docs
>
> I agree (especially after your additional clarifications in reply to
> Konstantin).
Great. I think we are heading towards a consensus.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: BIO performance issues
Posted by Rainer Jung <ra...@kippdata.de>.
I hope the following is not too long and confusing ...
On 03.05.2011 22:02, Mark Thomas wrote:
> Scenario
> --------
> This ended up being very long, so I moved it to the end. The exact
> pattern of delays will vary depending on timeouts, request frequency
> etc. but the scenario shows an example of how delays can occur. The
> short version is that requests with data to process (particularly new
> connections) tend to get delayed in the queue waiting for a thread to
> process them when the threads are all tied up processing keep-alive
> connections.
>
> Root cause
> ----------
> The underlying cause of all of the performance issues observed is when
> the threads are tied up doing HTTP keep-alive when there is no data
> process but there are other connections in the queue that do have data
> that could be processed.
>
> Solution A
> ----------
> NIO is designed to handle this using a poller. That isn't available to
> BIO so I attempted to simulate it. That generated excessive CPU load so
> I do not think simulated polling is the tight solution.
I expect generating the SocketTimeoutException is expensive, because the
JVM has to generate the stack information. The rate of the Exception
when handling mostly keep-alive (extreme case) is your "poll" timeout
times the number of threads, e.g. 100ms timeout times 200 threads is
2000 exceptions per second. Even if there is another reason for the high
CPU load, I expect it to be roughly proportional to the poll rate. In a
saturated system with lots of keep-alive you will have:
pollRate = 1 / pollTimeout * maxThreads
(e.g. 1 / 0.1s * 200 = 2000/s)
averageWaitBeforePoll = maxConnection / pollRate / 2
(e.g. 10000 / 2000 / 2 / s = 1.5s)
So we see, that in your case though we already have a high poll event
rate, we end up with every connection only being polled every 2.5
seconds, which is too much of request latency. If we want to reduce this
latency, we would need to increse the rate. But then CPU gets even
worse. Or we need to reduce maxConnections.
Let us try a different sizing:
maxThreads 200 , maxConnections 1000 (less overcommitment, but still
very different from 200), pollTimeout 200ms.
rate = 1000, half of the previous rate due to the doubled timeout.
averageWaitBeforePoll = 0.5 seconds.
Although this is an improvement, we still have a high poll rate and even
0.5 seconds average wait time for new connections isn't nice.
The tradeoff is: To be CPU effective, we have to reduce the poll rate.
Assuming a fixed thread and connection count, this automatically leads
to longer averageWaitBeforePoll, i.e. request latency. There seems to be
no sweet spot for sizing the system.
If we do not find an efficient way (in terms of CPU and blocking time of
threads) to handle the keep-alive connections, then I don't expect a
solution to the problem - except for disabling keep-alive or not
accepting much more connections than we have threads. At the end that's
the 75% threads busy then disable keep-alive solution. One could throw
in some "reduce keep-alive timeout under load" feature, but I doubt it
will help much more than the simply solution.
Do we see a cpu time and thread blocking time efficient way of handling
many keep-alive connections? I don't see any API, that would help here.
Of course one could try to build a hyprid "blocking for normal
processing but non-blocking for keep-alive" thing, but since we already
have NIO I would also support recommending NIO for keep-alive.
Switching the default from BIO to NIO is a big change, but only after we
switch will we find the last buglets and problems arising under rare
conditions. So if we want to switch, we should do it very soon. Doing it
late in the TC 7 cycle would be bad.
Lastly: APR uses server to server connections, as does HTTP when using a
reverse proxy in front of Tomcat. In those cases we have much fewer
connections with a higher rate of requests per connection. There
maxThreads == maxConnections is fine (and even the 75% rule could be
switched off). So for this scenario it would be nice to not drop BIO, at
least until the major TC version after the default switched to NIO.
> Solution B
> ----------
> Return to the Tomcat 6 implementation where maxConnections == maxThreads.
>
> Additional clean-up
> -------------------
> maxConnections is unnecessary in APR since pollerSize performs the same
> function.
>
> Summary
> -------
> The proposed changes are:
> a) restore disabling keep-alive when threads used>= 75% of maxThreads
> b) remove maxConnections and associated code from the APR connector
> c) remove the configuration options for maxConnections from the BIO
> connector
> d) use maxThreads instead of maxConnections for the BIO connector
> e) update the docs
I agree (especially after your additional clarifications in reply to
Konstantin).
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: BIO performance issues
Posted by Mark Thomas <ma...@apache.org>.
igaz <th...@nomagicsoftware.com> wrote:
>Is this thread/issue still *active*?
No. The issues were resolved at the time as described in the thread.
BIO is still available in Tomcat 8 but NIO is now the default.
> I have a few ideas (and I worked on a
>similar issue awhile back).
>But this thread was last active 2 years ago and there was even discussion of
>deprecating BIO in tomcat 8, so I don't want to waste time on something
>that is dormant or has been superceded
The BIO connector is still present and there are no plans (at the moment) to drop it. That is probably a discussion for Tomcat 9.
In the meantime, if you have ideas on how BIO (or any other connector) performance may be improved do let the list know.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: BIO performance issues
Posted by igaz <th...@nomagicsoftware.com>.
Is this thread/issue still *active*? I have a few ideas (and I worked on a
similar issue awhile back).
But this thread was last active 2 years ago and there was even discussion of
deprecating BIO in tomcat 8, so I don't want to waste time on something that
is dormant or has been superceded
--
View this message in context: http://tomcat.10.n6.nabble.com/BIO-performance-issues-tp2277428p4996545.html
Sent from the Tomcat - Dev mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: BIO performance issues
Posted by Tim Funk <fu...@apache.org>.
Solution B and disabling keepalive when a threshold is reached looks to be
the way to go. If people have issue with that, that's why the other
connectors exist.
-Tim
On Tue, May 3, 2011 at 4:02 PM, Mark Thomas <ma...@apache.org> wrote:
<SNIP>
>
> Solution B
> ----------
> Return to the Tomcat 6 implementation where maxConnections == maxThreads.
>
> Additional clean-up
> -------------------
> maxConnections is unnecessary in APR since pollerSize performs the same
> function.
>
> Summary
> -------
> The proposed changes are:
> a) restore disabling keep-alive when threads used >= 75% of maxThreads
> b) remove maxConnections and associated code from the APR connector
> c) remove the configuration options for maxConnections from the BIO
> connector
> d) use maxThreads instead of maxConnections for the BIO connector
> e) update the docs
>
>
>
Re: BIO performance issues
Posted by Mark Thomas <ma...@apache.org>.
On 04/05/2011 14:03, Konstantin Kolinko wrote:
> 2011/5/4 Mark Thomas <ma...@apache.org>:
>> On 04/05/2011 12:04, Konstantin Kolinko wrote:
>>> I do not quite get what happens in 6s. My first understanding from
>>> reading the scenario was that the threads at 2s are busy because we
>>> need them to process keep-alive connections. Then I see that at 6s
>>> keep-alive connections are in the Queue.
>>>
>>> What prevents those connections that exist at "2s" from being placed
>>> in the Queue?
>>
>> The keep-alive state is as viewed by the client. At 6s the requests in
>> the queue are open connections that have already processed one requests
>> and therefore are in HTTP keep-alive. However, there are no free threads
>> to process them. If any data was to turn up on those connections they
>> would sit in the queue until a thread was available.
>>
>> The difference between "Allocated threads. In HTTP keep-alive Xs" and
>> "Queued. In HTTP keep-alive for Xs" is that in the allocated case, data
>> will be processed as soon as it arrives.
>
> Maybe they can be put into the queue, if there is no data after small
> timeout? In your scenario they were waiting for 3s.
That is simulated polling. It works, but with the associated increase in
CPU usage.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: BIO performance issues
Posted by Konstantin Kolinko <kn...@gmail.com>.
2011/5/4 Mark Thomas <ma...@apache.org>:
> On 04/05/2011 12:04, Konstantin Kolinko wrote:
>> I do not quite get what happens in 6s. My first understanding from
>> reading the scenario was that the threads at 2s are busy because we
>> need them to process keep-alive connections. Then I see that at 6s
>> keep-alive connections are in the Queue.
>>
>> What prevents those connections that exist at "2s" from being placed
>> in the Queue?
>
> The keep-alive state is as viewed by the client. At 6s the requests in
> the queue are open connections that have already processed one requests
> and therefore are in HTTP keep-alive. However, there are no free threads
> to process them. If any data was to turn up on those connections they
> would sit in the queue until a thread was available.
>
> The difference between "Allocated threads. In HTTP keep-alive Xs" and
> "Queued. In HTTP keep-alive for Xs" is that in the allocated case, data
> will be processed as soon as it arrives.
Maybe they can be put into the queue, if there is no data after small
timeout? In your scenario they were waiting for 3s.
> In the queued case, data will
> not be processed a) all the connections ahead of it in the queue have
> been processed and b) a thread is available to process it.
>
>> What is "simulated polling" that you tried to implement? Was it trying
>> to read data from each of keep-alive sockets in turn with a short
>> timeout?
> Exactly.
>
>> Where increased CPU usage comes from?
> Because of the short timeout, threads that would normally be blocked are
> generating SocketTimeoutExceptions and looping through the
> Http11Processor.process method every 100ms.
>
>>> The proposed changes are:
>>> a) restore disabling keep-alive when threads used >= 75% of maxThreads
>>
>> Anyway, if the problem persists, I'll be +1 for a), as it should
>> prevent us from falling in this state.
>
> That certainly would deal with the immediate issue. I didn't fully
> understand Filip's changes when I removed the code that disabled keep-alive.
>
>(...)
Best regards,
Konstantin Kolinko
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: BIO performance issues
Posted by Mark Thomas <ma...@apache.org>.
On 04/05/2011 12:04, Konstantin Kolinko wrote:
> I do not quite get what happens in 6s. My first understanding from
> reading the scenario was that the threads at 2s are busy because we
> need them to process keep-alive connections. Then I see that at 6s
> keep-alive connections are in the Queue.
>
> What prevents those connections that exist at "2s" from being placed
> in the Queue?
The keep-alive state is as viewed by the client. At 6s the requests in
the queue are open connections that have already processed one requests
and therefore are in HTTP keep-alive. However, there are no free threads
to process them. If any data was to turn up on those connections they
would sit in the queue until a thread was available.
The difference between "Allocated threads. In HTTP keep-alive Xs" and
"Queued. In HTTP keep-alive for Xs" is that in the allocated case, data
will be processed as soon as it arrives. In the queued case, data will
not be processed a) all the connections ahead of it in the queue have
been processed and b) a thread is available to process it.
> What is "simulated polling" that you tried to implement? Was it trying
> to read data from each of keep-alive sockets in turn with a short
> timeout?
Exactly.
> Where increased CPU usage comes from?
Because of the short timeout, threads that would normally be blocked are
generating SocketTimeoutExceptions and looping through the
Http11Processor.process method every 100ms.
>> The proposed changes are:
>> a) restore disabling keep-alive when threads used >= 75% of maxThreads
>
> Anyway, if the problem persists, I'll be +1 for a), as it should
> prevent us from falling in this state.
That certainly would deal with the immediate issue. I didn't fully
understand Filip's changes when I removed the code that disabled keep-alive.
> +0 for b)..e), as I do not quite get them. An alternative could be to
> change the default value of maxConnections and calculate it from
> maxThreads.
With the change described in a), maxConnections is effectively an
duplication of acceptCount for BIO. Changes c) and d) are intended to
remove the duplication so acceptCount functions as it does in Tomcat 6.
maxConnections duplicates pollerSize for APR. Change b) is intended to
remove the duplication.
Change e) will align the docs with whatever changes are made. Currently
maxConnections is documented as only applying to BIO whereas it applies
to BIO, NIO and APR.
> I wonder, maybe it is time to use the NIO connector as the default one
> in Tomcat 7.
I was beginning to think the along the same lines although I was
actually thinking remove BIO entirely for Tomcat 8.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: BIO performance issues
Posted by Konstantin Kolinko <kn...@gmail.com>.
2011/5/4 Mark Thomas <ma...@apache.org>:
> Time: 2s
> Activity: Group B clients connect and are added to the queue
> 100 threads are allocated to process the B connections
> The B connections are processed and returned to the queue
> 100 threads are allocated to process the B connections
> Queue: None
> Threads: A, B
> Group A: Allocated threads. In HTTP keep-alive for 1s
> Group B: Allocated threads. In HTTP keep-alive for 0s
> Group C: Inactive
> Group D: Inactive
> Group E: Inactive
> Group F: Inactive
> Group G: Inactive
> Group H: Inactive
> Group I: Inactive
> Group J: Inactive
>
> Time: 6s
> Activity: Group F clients connect and are added to the queue
> A connections send requests
> A connections are processed and returned to the queue
> C connections are processed and returned to the queue
> D connections are processed and returned to the queue
> E connections are processed and returned to the queue
> F connections are processed and returned to the queue
> A connections are allocated threads but are in keep-alive
> Queue: C, D, E, F
> Threads: A, B
> Group A: Allocated threads. In HTTP keep-alive for 0s
> Group B: Allocated threads. In HTTP keep-alive for 4s
> Group C: Queued. In HTTP keep-alive for 0s
> Group D: Queued. In HTTP keep-alive for 0s
> Group E: Queued. In HTTP keep-alive for 0s
> Group F: Queued. In HTTP keep-alive for 0s
> Group G: Inactive
> Group H: Inactive
> Group I: Inactive
> Group J: Inactive
I do not quite get what happens in 6s. My first understanding from
reading the scenario was that the threads at 2s are busy because we
need them to process keep-alive connections. Then I see that at 6s
keep-alive connections are in the Queue.
What prevents those connections that exist at "2s" from being placed
in the Queue?
What is "simulated polling" that you tried to implement? Was it trying
to read data from each of keep-alive sockets in turn with a short
timeout? Where increased CPU usage comes from?
> The proposed changes are:
> a) restore disabling keep-alive when threads used >= 75% of maxThreads
Anyway, if the problem persists, I'll be +1 for a), as it should
prevent us from falling in this state.
+0 for b)..e), as I do not quite get them. An alternative could be to
change the default value of maxConnections and calculate it from
maxThreads.
I wonder, maybe it is time to use the NIO connector as the default one
in Tomcat 7.
Best regards,
Konstantin Kolinko
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: BIO performance issues
Posted by Filip Hanik - Dev Lists <de...@hanik.com>.
On 5/4/2011 9:54 AM, Mark Thomas wrote:
> On 04/05/2011 16:17, Filip Hanik - Dev Lists wrote:
>> On 5/3/2011 2:02 PM, Mark Thomas wrote:
>> In a similar fashion, we can also craft a test run that will yield a
>> substantial improvement over the old implementation in throughput.
>> So there is a test case to prove every scenario.
> Could you outline what a test case looks like. It would help with the
> general understanding of what problem maxConnections is trying to solve.
ok, we have an acceptor thread(AT) that does ServerSocket.accept() in an endless loop.
In the previous implementation, the AT would accept the socket, then wait for a thread to become available to handle the connection.
New incoming connections would then be handled by the backlog in the operating system. The old implementation was extremely unfair in how it
handled the requests, some requests could get handled right away, while others could wait for long period of times (this is with the old
impl). As you may be familiar, a client's connection may "die" in the backlog at which point the client has to attempt a new connection.
if you really want a simple test case, then do
maxThreads=200
clients=200
keepalive=on
In the old impl, keep alive would be turned off and performance would suffer, even though the system has plenty of resources to handle it.
while this test case is very narrow and simple, it's the other extreme of the use case you presented.
The new implementation, queue based, was as a result to be able to disconnect a thread from a socket due to the new async requirements.
Previously, a thread was married to a socket for as long as the socket was alive.
Anyway, with the new implementation, just like with NIO, there is no longer a stopper on the acceptor thread (AT) it will happily keep
accepting connections until you run out of buffer space or port numbers.
This presents a DoS risk, this risk has existed in NIO for a while. So maxConnections has been put in place to stop accepting connections,
and push back new connections into the backlog.
so maxConnections exists to stop the acceptor thread from taking in more than it can handle.
>> Here is what I propose, and you'll see that it's pretty much inline with
>> what you suggest.
> Yep. That works for me. I do have some additional questions around
> maxConnections - mainly so I can get the docs right.
>
>>> c) remove the configuration options for maxConnections from the BIO
>>> connector
>> I think you still misunderstand why maxConnections is there, at some
>> point you need to push back on the TCP stack.
> Some more detail on exactly the purpose of maxConnections would be
> useful. The purposes I can see are:
> - limiting connections since the addition of the queue means they are
> not limited by maxThreads
correct, a system with
maxThreads=200 should be able to handle connections=500 with keep alive on and perform very well.
> - fair (order received) processing of connections?
correct. almost no clients use pipelined requests, so chances that there is data on a new finished request is very slim. It is more probable
that there is data on a request that was finished earlier in the cycle.
I hope that explains it. And by default, with the config options/defaults I suggested, you'll get the exact behavior of the old connector,
but can still benefit from the new connector logic
> - ?
>
> Mark
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: dev-help@tomcat.apache.org
>
>
>
> -----
> No virus found in this message.
> Checked by AVG - www.avg.com
> Version: 10.0.1321 / Virus Database: 1500/3615 - Release Date: 05/04/11
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: BIO performance issues
Posted by Mark Thomas <ma...@apache.org>.
On 04/05/2011 16:17, Filip Hanik - Dev Lists wrote:
> On 5/3/2011 2:02 PM, Mark Thomas wrote:
> In a similar fashion, we can also craft a test run that will yield a
> substantial improvement over the old implementation in throughput.
> So there is a test case to prove every scenario.
Could you outline what a test case looks like. It would help with the
general understanding of what problem maxConnections is trying to solve.
> Here is what I propose, and you'll see that it's pretty much inline with
> what you suggest.
Yep. That works for me. I do have some additional questions around
maxConnections - mainly so I can get the docs right.
>> c) remove the configuration options for maxConnections from the BIO
>> connector
> I think you still misunderstand why maxConnections is there, at some
> point you need to push back on the TCP stack.
Some more detail on exactly the purpose of maxConnections would be
useful. The purposes I can see are:
- limiting connections since the addition of the queue means they are
not limited by maxThreads
- fair (order received) processing of connections?
- ?
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: BIO performance issues
Posted by Filip Hanik - Dev Lists <de...@hanik.com>.
fan-tastic! Thank you!
On 5/5/2011 9:56 AM, Mark Thomas wrote:
> On 04/05/2011 16:17, Filip Hanik - Dev Lists wrote:
>> Here is what I propose, and you'll see that it's pretty much inline with
>> what you suggest.
>>
>> a) add in a flag that lets keep alive be turned off during concurrency
>> disableKeepAlivePercentage="75"
>> this would be the default value. and would achieve exactly what you want
>> in your a)
>>
>>> a) restore disabling keep-alive when threads used>= 75% of maxThreads
>>> b) remove maxConnections and associated code from the APR connector
>> b)
>> or you can simply make pollerSize==maxConnections
>> meaning that
>> setPollerSize(5000) is the same as setMaxConnections(5000)
>> that way you retain the same properties across connectors
>>
>>> c) remove the configuration options for maxConnections from the BIO
>>> connector
>> I think you still misunderstand why maxConnections is there, at some
>> point you need to push back on the TCP stack.
>> and still be able to control when to accept new connections.
>> Keep this setting in there, exactly as it is.
>>
>>> d) use maxThreads instead of maxConnections for the BIO connector
>> if you feel so strongly on this, simple set maxConnections=200 as the
>> default value for BIO connector.
> All now implemented in trunk.
>
> Mark
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: dev-help@tomcat.apache.org
>
>
>
> -----
> No virus found in this message.
> Checked by AVG - www.avg.com
> Version: 10.0.1321 / Virus Database: 1500/3617 - Release Date: 05/05/11
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: BIO performance issues
Posted by Mark Thomas <ma...@apache.org>.
On 04/05/2011 16:17, Filip Hanik - Dev Lists wrote:
> Here is what I propose, and you'll see that it's pretty much inline with
> what you suggest.
>
> a) add in a flag that lets keep alive be turned off during concurrency
> disableKeepAlivePercentage="75"
> this would be the default value. and would achieve exactly what you want
> in your a)
>
>> a) restore disabling keep-alive when threads used>= 75% of maxThreads
>
>> b) remove maxConnections and associated code from the APR connector
> b)
> or you can simply make pollerSize==maxConnections
> meaning that
> setPollerSize(5000) is the same as setMaxConnections(5000)
> that way you retain the same properties across connectors
>
>> c) remove the configuration options for maxConnections from the BIO
>> connector
> I think you still misunderstand why maxConnections is there, at some
> point you need to push back on the TCP stack.
> and still be able to control when to accept new connections.
> Keep this setting in there, exactly as it is.
>
>> d) use maxThreads instead of maxConnections for the BIO connector
> if you feel so strongly on this, simple set maxConnections=200 as the
> default value for BIO connector.
All now implemented in trunk.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: BIO performance issues
Posted by Filip Hanik - Dev Lists <de...@hanik.com>.
On 5/3/2011 2:02 PM, Mark Thomas wrote:
> All,
>
> Summary
> -------
> While trying to align the documentation for maxConnections with the
> actual implementation a couple of performance issue have been identified
> with the BIO connector. All bar one of these have been fixed. A fix
> needs to be agreed for the remaining issue, ideally before the next
> 7.0.x release.
>
> Background
> ----------
> New connections to the BIO connector are placed in a queue. When a
> thread is available, it takes the next connection of the queue and
> processes it. When the connection has been processed if the connection
> is kept-alive it is placed back on the queue. If there is no keep-alive
> the connection is closed.
>
> Scenario
> --------
> This ended up being very long, so I moved it to the end. The exact
> pattern of delays will vary depending on timeouts, request frequency
> etc. but the scenario shows an example of how delays can occur. The
> short version is that requests with data to process (particularly new
> connections) tend to get delayed in the queue waiting for a thread to
> process them when the threads are all tied up processing keep-alive
> connections.
>
> Root cause
> ----------
> The underlying cause of all of the performance issues observed is when
> the threads are tied up doing HTTP keep-alive when there is no data
> process but there are other connections in the queue that do have data
> that could be processed.
>
> Solution A
> ----------
> NIO is designed to handle this using a poller. That isn't available to
> BIO so I attempted to simulate it. That generated excessive CPU load so
> I do not think simulated polling is the tight solution.
>
> Solution B
> ----------
> Return to the Tomcat 6 implementation where maxConnections == maxThreads.
>
> Additional clean-up
> -------------------
> maxConnections is unnecessary in APR since pollerSize performs the same
> function.
I'd -1 to Solution A and Solution B, mostly because the very descriptive example below shows a scenario that can be configured around.
I would say that
- maxThreads=200
- maxConnections=10000
- connectionTimeout=20000
- keepAliveTimeout=20000
is a gross misconfiguration for the BIO connector if you really expect this type of concurrency. the 20 second timeout stems from our 10Mbps
network days. I can equally prove that the old implementation has no fairness what so ever. As some connections will stay on the backlog
until they expire while some get accepted.
In a similar fashion, we can also craft a test run that will yield a substantial improvement over the old implementation in throughput.
So there is a test case to prove every scenario.
Here is what I propose, and you'll see that it's pretty much inline with what you suggest.
a) add in a flag that lets keep alive be turned off during concurrency
disableKeepAlivePercentage="75"
this would be the default value. and would achieve exactly what you want in your a)
> a) restore disabling keep-alive when threads used>= 75% of maxThreads
> b) remove maxConnections and associated code from the APR connector
b)
or you can simply make pollerSize==maxConnections
meaning that
setPollerSize(5000) is the same as setMaxConnections(5000)
that way you retain the same properties across connectors
> c) remove the configuration options for maxConnections from the BIO
> connector
I think you still misunderstand why maxConnections is there, at some point you need to push back on the TCP stack.
and still be able to control when to accept new connections.
Keep this setting in there, exactly as it is.
> d) use maxThreads instead of maxConnections for the BIO connector
if you feel so strongly on this, simple set maxConnections=200 as the default value for BIO connector.
so if the new implementation has these default values for BIO
maxThreads=200
maxConnections=200
disableKeepAlivePercentage=75
this will grant you the exact behavior of the old implementation, while giving others the benefit of configuring for better performance
using BIO
best
Filip
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
RE: BIO performance issues
Posted by David Dabbs <dm...@gmail.com>.
> -----Original Message-----
> From: Mark Thomas [mailto:markt@apache.org]
> Sent: Tuesday, May 03, 2011 3:02 PM
> To: Tomcat Developers List
> Subject: BIO performance issues
>
> All,
>
> Summary
> -------
> While trying to align the documentation for maxConnections with the
> actual implementation a couple of performance issue have been
> identified
> with the BIO connector. All bar one of these have been fixed. A fix
> needs to be agreed for the remaining issue, ideally before the next
> 7.0.x release.
>
. . .snip . . .
>
> At time 11s:
> - 4s delay processing G first requests
> - 3s delay processing H first requests
> - 2s delay processing I first requests
> - 1s delay processing J first requests
>
> The delays are particularly bad for new connections which must wait
> behind all the other requests in the queue.
>
>
The AJP connector isn't mentioned in your write-up (many thanks), but I just
want to confirm that this issue doesn't happen when using AJP, native or
non-native.
Thanks,
David
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org