You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Paul P Wolf <Pa...@deutschebahn.com> on 2021/05/18 11:44:04 UTC
maxConnections behaving unexpected - no connection gets ever refused
Hi,
I am trying to run a spring boot application with an embedded tomcat. In a scenario, where there is a lot of load on the service, I would like tomcat to refuse connections or return something like a 503 Service Unavailable. From what I understand, you could have this behaviour by setting maxConnections and and any additional connections get refused. At least this was how the old blocking io acceptor worked, from what I understand.
The documentation says "The maximum number of connections that the server will accept and process at any given time. When this number has been reached, the server will accept, but not process, one further connection. This additional connection be blocked until [...]".
However the documentation doesn't really state what happens if maxConnection+2 connections are reached.
I tried to run my application with following settings (embedded-tomcat 9.0.45):
maxConnections=3
acceptCount=2
maxThreads=1
processorCache=1
I created an endpoint, which just sleeps for a few seconds and returns a string. When I create 50 separate connection via curl instances to call that service I see the following behaviour with the NIO Acceptor:
* 6 http connections are accepted immediately (maxThreads + acceptCount + maxConnections)
* 44 http connections aren't established just, but neither are they refused. I will call them "blocked", but different from the specification those are 44 blocked connections and not just 1
* once the first request finishes, the latest (blocked) requests gets a connection and is being processed (not a request from the accept-queue or one of the other already established connections)
* when there are no further blocked requests, the requests still get processed in last in first out order
* I see some timeouts after a while with this setup, depending on what timeouts I set on curl. The requests without an established connection timeout with "connect-timeout" parameter and the ones with established connections depending on the "max-time" parameter.
Now I have a lot of questions and I suspect some bugs, but I wanted to ask on this list first:
1. Is there a way to refuse connections, instead of blocking them?
2. I can't control the connect-timeout of the clients - is there a way to control it on the server side? (I tried to change keepAliveTimeout and connectionTimeout without success)
3. Why doesn't the specification state what happens if many more requests are being made than maxConnections allows for? (esp. with the behaviour being different among acceptors)
4. The processing order is troublesome - early requests get starved by following ones. Why is this lifo instead of fifo?
5. The number of accepted connections is unintuitively. Why are active connections with requests either being processed or being in the accept queue not counted as active connections (actual connections = maxConnections + acceptCount + maxThreads)?
6. What is the difference between requests in the accept queue and requests outside the queue, but still within the maxConnections limit?
Any insight would be appreciated. If there is anything in here, that I should report as bug, please let me know. I am happy to file one.
Best regards,
Paul
________________________________
Pflichtangaben anzeigen<http://www.deutschebahn.com/pflichtangaben/20210430>
N?here Informationen zur Datenverarbeitung im DB-Konzern finden Sie hier: http://www.deutschebahn.com/de/konzern/datenschutz
Re: AW: AW: AW: maxConnections behaving unexpected - no connection
gets ever refused
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Paul,
On 5/19/21 08:32, Paul P Wolf wrote:
> sorry, my message was misformated, so here again with hopefully better formatting:
>
>> The clients timeout because they spend more than timeout in the
>> acceptCount/backlog queue waiting for Tomcat to call Socket.accept()
>
> Ok, so you are stating, that clients timeout while their requests are in the acceptCount/backlog. This is not what I am seeing. See below.
>
>> So we have:
>> maxThreads=4
>> maxConnections=10
>> acceptCount=20
>>
>> and a request processing time of 1 second.
>>
>> I'd guess that the OS is using a much large accept count. Let's model it.
>>
>> 0s - 50 connections in acceptCount
>> 1s - 39 connections in acceptCount, 11 connections maintained by Tomcat,
>> 4 requests processing
>> 2s - 35 connections in acceptCount, 11 connections maintained by Tomcat,
>> 4 requests processing, 4 completed requests 3s - 31 connections
>> in acceptCount, 11 connections maintained by Tomcat,
>> 4 requests processing, 8 completed requests 4s - 27 connections
>> in acceptCount, 11 connections maintained by Tomcat,
>> 4 requests processing, 12 completed requests 5s - 23 connections
>> in acceptCount, 11 connections maintained by Tomcat,
>> 4 requests processing, 16 completed requests 6s - 19 connections
>> timed out, 11 connections maintained by Tomcat,
>> 4 requests processing, 20 completed requests 7s - 19 connections
>> timed out, 7 connections maintained by Tomcat,
>> 4 requests processing, 24 completed requests 8s - 19 connections
>> timed out, 3 connections maintained by Tomcat,
>> 3 requests processing, 28 completed requests 9s - 19 connections
>> timed out, 0 connections maintained by Tomcat,
>> 0 requests processing, 31 completed requests
>>
>> That seems to match what you observed. That suggests the OS is using
>> an acceptCount of at least 50.
>
> I can see what you mean, but this is not what I see, because your assumption of a 1s processing time is wrong. The processing time of each request is 10s (thanks to a 10s sleep, which blocks the Thread).
>
> So here is what I see instead (note I don't guess the response time, but do actually see/measure it):
>
> 0s - 4 requests processing, 11 connections maintained, 20 connections in acceptCount, 19 not in acceptCount
>
> 5s - 4 requests processing, 11 connections maintained, 20 connections in acceptCount, 0 not in acceptCount, 19 timed-out
>
> 10s - 4 requests processing, 11 connections maintained, 16 connections in acceptCount, 0 not in acceptCount, 19 timed-out, 4 requests processed
>
> 20s - 4 requests processing, 11 connections maintained, 12 connections in acceptCount, 0 not in acceptCount, 19 timed-out, 8 requests processed
>
> 30s - 4 requests processing, 11 connections maintained, 8 connections in acceptCount, 0 not in acceptCount, 19 timed-out, 12 requests processed
>
> 40s - 4 requests processing, 11 connections maintained, 4 connections in acceptCount, 0 not in acceptCount, 19 timed-out, 16 requests processed
>
> 50s - 4 requests processing, 11 connections maintained, 0 connections in acceptCount, 0 not in acceptCount, 19 timed-out, 20 requests processed
>
> 60s - 4 requests processing, 7 connections maintained, 0 connections in acceptCount, 0 not in acceptCount, 19 timed-out, 24 requests processed
>
> 70s - 3 requests processing, 3 connections maintained, 0 connections in acceptCount, 0 not in acceptCount, 19 timed-out, 28 requests processed
>
> 80s - 0 requests processing, 0 connections maintained, 0 connections in acceptCount, 0 not in acceptCount, 19 timed-out, 31 requests processed
>
> The interesting thing to note is, that contrary to your statement, I don't see timeouts for requests in the acceptCount/backlog.
>
> To verify my interpretation, I increased the acceptCount and I saw that the time outs decrease by the same number. I decreased the acceptCount and saw that the time outs increase.
>
> The behaviour seems 100% predictable to me... just not as expected based on the documentation.
>
>> Not up to Tomcat. Tomcat can only call Socket.accept() and does so
>> under the control of maxConnections.
>>
>> Connection refused == acceptCount/backlog full (or no listening socket).
>>
>> Connection refusal is entirely under the control of the OS and will be
>> driven largely by the actual value of acceptCount/backlog.
> So if it is not up to Tomcat, then the documentation "Any further simultaneous requests will receive "connection refused" errors" is clearly wrong, isn't it?
I realize we've moved-on a bit from here, but I wanted to make sure we
were all using the same language about timeouts and such. Showing your
curl command-line would be helpful. Under certain situations, curl will
retry connections, etc. and knowing exactly what you are handing to curl
would be helpful.
I would certainly expect that if you have maxConnections=10 and
acceptCount=10 and your service just sleeps for 10 seconds and you have
a 5-second client timeout, then:
* 11 connections are immediately accepted by Tomcat, allocated a thread
and immediately start sleeping (ESTABLISHED)
* 10 connections are immediately put into the TCP stack's backlog
(ESTABLISHED, to my surprise!)
* 9 connections are refused (connect error)
After 5 seconds:
* All 21 remaining clients time-out
If the above is not what is happening, then I would suspect:
1. The OS is using a bigger backlog than requested
2. curl is retrying
3. We are misunderstanding how curl's settings affect connect/read timeouts
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: AW: AW: AW: AW: maxConnections behaving unexpected - no
connection gets ever refused
Posted by Mark Thomas <ma...@apache.org>.
On 20/05/2021 10:58, Mark Thomas wrote:
<snip/>
> I'll get docs updated. The website will update after the next round of
> releases (due in the first week or so of June).
https://ci.apache.org/projects/tomcat/tomcat10/docs/config/http.html
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: AW: AW: AW: AW: maxConnections behaving unexpected - no
connection gets ever refused
Posted by Mark Thomas <ma...@apache.org>.
On 20/05/2021 07:24, Paul P Wolf wrote:
>> OK, that looks like clients with a connection timeout of 5s.
> Correct. I think I forgot to mention, that I set the connection timeout of the curl instances to 5s. I didn't set the max timeout
>
>>>> Connection refusal is entirely under the control of the OS and will
>>>> be driven largely by the actual value of acceptCount/backlog.
>>> So if it is not up to Tomcat, then the documentation "Any further
>> simultaneous requests will receive "connection refused" errors" is clearly
>> wrong, isn't it?
>>
>> That statement in the docs ignores acceptCount so it certainly needs
>> updating to reflect that.
> I am not sure what you mean by "ignores acceptCount" as it directly relates to it.
> Source: https://tomcat.apache.org/tomcat-9.0-doc/config/http.html
> Longer quote:
> "Each incoming request requires a thread for the duration of that request. If more simultaneous requests are received than can be handled by the currently available request processing threads, additional threads will be created up to the configured maximum (the value of the maxThreads attribute). If still more simultaneous requests are received, they are stacked up inside the server socket created by the Connector, up to the configured maximum (the value of the acceptCount attribute). Any further simultaneous requests will receive "connection refused" errors, until resources are available to process them."
It uses acceptCount where it should use maxConnections and doesn't
describe acceptCount at all.
>> I think if we replaced "connection refused" with "connection refused or
>> connection timeout" that should cover the possible variations.
> While this may seem like a minor change, I would really appreciate it. This minor detail would have saved us a lot of time, effort and money. This gets compounded by sites like Stackoverflow perpetuating bad assumptions about how Tomcat behaves in such situations.
>
> Again: Thank you very much for all the information and help you provided. I would appreciate it, if you could update the documentation to reflect that tomcat does not always refuse connections in these situation (from what I understood, tomcat doesn't handle these situations at all, but the OS does instead).
I'll get docs updated. The website will update after the next round of
releases (due in the first week or so of June).
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
AW: AW: AW: AW: maxConnections behaving unexpected - no connection
gets ever refused
Posted by Paul P Wolf <Pa...@deutschebahn.com>.
> OK, that looks like clients with a connection timeout of 5s.
Correct. I think I forgot to mention, that I set the connection timeout of the curl instances to 5s. I didn't set the max timeout
> >> Connection refusal is entirely under the control of the OS and will
> >> be driven largely by the actual value of acceptCount/backlog.
> > So if it is not up to Tomcat, then the documentation "Any further
> simultaneous requests will receive "connection refused" errors" is clearly
> wrong, isn't it?
>
> That statement in the docs ignores acceptCount so it certainly needs
> updating to reflect that.
I am not sure what you mean by "ignores acceptCount" as it directly relates to it.
Source: https://tomcat.apache.org/tomcat-9.0-doc/config/http.html
Longer quote:
"Each incoming request requires a thread for the duration of that request. If more simultaneous requests are received than can be handled by the currently available request processing threads, additional threads will be created up to the configured maximum (the value of the maxThreads attribute). If still more simultaneous requests are received, they are stacked up inside the server socket created by the Connector, up to the configured maximum (the value of the acceptCount attribute). Any further simultaneous requests will receive "connection refused" errors, until resources are available to process them."
> I think if we replaced "connection refused" with "connection refused or
> connection timeout" that should cover the possible variations.
While this may seem like a minor change, I would really appreciate it. This minor detail would have saved us a lot of time, effort and money. This gets compounded by sites like Stackoverflow perpetuating bad assumptions about how Tomcat behaves in such situations.
Again: Thank you very much for all the information and help you provided. I would appreciate it, if you could update the documentation to reflect that tomcat does not always refuse connections in these situation (from what I understood, tomcat doesn't handle these situations at all, but the OS does instead).
________________________________
Pflichtangaben anzeigen<http://www.deutschebahn.com/pflichtangaben/20210430>
Nähere Informationen zur Datenverarbeitung im DB-Konzern finden Sie hier: http://www.deutschebahn.com/de/konzern/datenschutz
Re: AW: AW: AW: maxConnections behaving unexpected - no connection
gets ever refused
Posted by Mark Thomas <ma...@apache.org>.
On 19/05/2021 13:32, Paul P Wolf wrote:
<snip/>
>> So we have:
>> maxThreads=4
>> maxConnections=10
>> acceptCount=20
<snip/>
>The processing time of each request is 10s (thanks to a 10s sleep, which blocks the Thread).
>
> So here is what I see instead (note I don't guess the response time, but do actually see/measure it):
>
> 0s - 4 requests processing, 11 connections maintained, 20 connections in acceptCount, 19 not in acceptCount
>
> 5s - 4 requests processing, 11 connections maintained, 20 connections in acceptCount, 0 not in acceptCount, 19 timed-out
OK, that looks like clients with a connection timeout of 5s.
> 10s - 4 requests processing, 11 connections maintained, 16 connections in acceptCount, 0 not in acceptCount, 19 timed-out, 4 requests processed
>
> 20s - 4 requests processing, 11 connections maintained, 12 connections in acceptCount, 0 not in acceptCount, 19 timed-out, 8 requests processed
>
> 30s - 4 requests processing, 11 connections maintained, 8 connections in acceptCount, 0 not in acceptCount, 19 timed-out, 12 requests processed
>
> 40s - 4 requests processing, 11 connections maintained, 4 connections in acceptCount, 0 not in acceptCount, 19 timed-out, 16 requests processed
>
> 50s - 4 requests processing, 11 connections maintained, 0 connections in acceptCount, 0 not in acceptCount, 19 timed-out, 20 requests processed
>
> 60s - 4 requests processing, 7 connections maintained, 0 connections in acceptCount, 0 not in acceptCount, 19 timed-out, 24 requests processed
>
> 70s - 3 requests processing, 3 connections maintained, 0 connections in acceptCount, 0 not in acceptCount, 19 timed-out, 28 requests processed
>
> 80s - 0 requests processing, 0 connections maintained, 0 connections in acceptCount, 0 not in acceptCount, 19 timed-out, 31 requests processed
>
> The interesting thing to note is, that contrary to your statement, I don't see timeouts for requests in the acceptCount/backlog.
Two possibilities there.
1. The write timeout is > ~70s.
2. The write is buffered in the network stack (more likely in the client
I think) and the read timeout is > ~70s.
2 strikes me as more likely than 1.
With a request body bigger than the total amount of buffering in the
client and server network stacks (typically much larger than you might
expect) I think you'll see different results, namely twrite timeouts for
client connections still in the acceptCount - assuming the write timeout
is less than ~70s.
> To verify my interpretation, I increased the acceptCount and I saw that the time outs decrease by the same number. I decreased the acceptCount and saw that the time outs increase.
I agree that is a strong indication that your interpretation is correct.
Further, I don't see anything to suggest your interpretation is incorrect.
> The behaviour seems 100% predictable to me... just not as expected based on the documentation.
I was going to ask which bit of the documentation but you cover that below.
>> Not up to Tomcat. Tomcat can only call Socket.accept() and does so
>> under the control of maxConnections.
>>
>> Connection refused == acceptCount/backlog full (or no listening socket).
>>
>> Connection refusal is entirely under the control of the OS and will be
>> driven largely by the actual value of acceptCount/backlog.
> So if it is not up to Tomcat, then the documentation "Any further simultaneous requests will receive "connection refused" errors" is clearly wrong, isn't it?
That statement in the docs ignores acceptCount so it certainly needs
updating to reflect that.
Beyond that I suspect we are getting into variation in behaviour between
operating systems. There might also be a local host vs remote host
aspect but I don't think that is a factor here. The bigger factors will
be the timeouts configured on the client, the network stack buffering
behaviour and how the OS handles new connections when the acceptCount is
full.
I think if we replaced "connection refused" with "connection refused or
connection timeout" that should cover the possible variations.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
AW: AW: AW: maxConnections behaving unexpected - no connection gets
ever refused
Posted by Paul P Wolf <Pa...@deutschebahn.com>.
sorry, my message was misformated, so here again with hopefully better formatting:
> The clients timeout because they spend more than timeout in the
> acceptCount/backlog queue waiting for Tomcat to call Socket.accept()
Ok, so you are stating, that clients timeout while their requests are in the acceptCount/backlog. This is not what I am seeing. See below.
> So we have:
> maxThreads=4
> maxConnections=10
> acceptCount=20
>
> and a request processing time of 1 second.
>
> I'd guess that the OS is using a much large accept count. Let's model it.
>
> 0s - 50 connections in acceptCount
> 1s - 39 connections in acceptCount, 11 connections maintained by Tomcat,
> 4 requests processing
> 2s - 35 connections in acceptCount, 11 connections maintained by Tomcat,
> 4 requests processing, 4 completed requests 3s - 31 connections
> in acceptCount, 11 connections maintained by Tomcat,
> 4 requests processing, 8 completed requests 4s - 27 connections
> in acceptCount, 11 connections maintained by Tomcat,
> 4 requests processing, 12 completed requests 5s - 23 connections
> in acceptCount, 11 connections maintained by Tomcat,
> 4 requests processing, 16 completed requests 6s - 19 connections
> timed out, 11 connections maintained by Tomcat,
> 4 requests processing, 20 completed requests 7s - 19 connections
> timed out, 7 connections maintained by Tomcat,
> 4 requests processing, 24 completed requests 8s - 19 connections
> timed out, 3 connections maintained by Tomcat,
> 3 requests processing, 28 completed requests 9s - 19 connections
> timed out, 0 connections maintained by Tomcat,
> 0 requests processing, 31 completed requests
>
> That seems to match what you observed. That suggests the OS is using
> an acceptCount of at least 50.
I can see what you mean, but this is not what I see, because your assumption of a 1s processing time is wrong. The processing time of each request is 10s (thanks to a 10s sleep, which blocks the Thread).
So here is what I see instead (note I don't guess the response time, but do actually see/measure it):
0s - 4 requests processing, 11 connections maintained, 20 connections in acceptCount, 19 not in acceptCount
5s - 4 requests processing, 11 connections maintained, 20 connections in acceptCount, 0 not in acceptCount, 19 timed-out
10s - 4 requests processing, 11 connections maintained, 16 connections in acceptCount, 0 not in acceptCount, 19 timed-out, 4 requests processed
20s - 4 requests processing, 11 connections maintained, 12 connections in acceptCount, 0 not in acceptCount, 19 timed-out, 8 requests processed
30s - 4 requests processing, 11 connections maintained, 8 connections in acceptCount, 0 not in acceptCount, 19 timed-out, 12 requests processed
40s - 4 requests processing, 11 connections maintained, 4 connections in acceptCount, 0 not in acceptCount, 19 timed-out, 16 requests processed
50s - 4 requests processing, 11 connections maintained, 0 connections in acceptCount, 0 not in acceptCount, 19 timed-out, 20 requests processed
60s - 4 requests processing, 7 connections maintained, 0 connections in acceptCount, 0 not in acceptCount, 19 timed-out, 24 requests processed
70s - 3 requests processing, 3 connections maintained, 0 connections in acceptCount, 0 not in acceptCount, 19 timed-out, 28 requests processed
80s - 0 requests processing, 0 connections maintained, 0 connections in acceptCount, 0 not in acceptCount, 19 timed-out, 31 requests processed
The interesting thing to note is, that contrary to your statement, I don't see timeouts for requests in the acceptCount/backlog.
To verify my interpretation, I increased the acceptCount and I saw that the time outs decrease by the same number. I decreased the acceptCount and saw that the time outs increase.
The behaviour seems 100% predictable to me... just not as expected based on the documentation.
> Not up to Tomcat. Tomcat can only call Socket.accept() and does so
> under the control of maxConnections.
>
> Connection refused == acceptCount/backlog full (or no listening socket).
>
> Connection refusal is entirely under the control of the OS and will be
> driven largely by the actual value of acceptCount/backlog.
So if it is not up to Tomcat, then the documentation "Any further simultaneous requests will receive "connection refused" errors" is clearly wrong, isn't it?
________________________________
Pflichtangaben anzeigen<http://www.deutschebahn.com/pflichtangaben/20210430>
Nähere Informationen zur Datenverarbeitung im DB-Konzern finden Sie hier: http://www.deutschebahn.com/de/konzern/datenschutz
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
AW: AW: AW: maxConnections behaving unexpected - no connection gets
ever refused
Posted by Paul P Wolf <Pa...@deutschebahn.com>.
> The clients timeout because they spend more than timeout in the
> acceptCount/backlog queue waiting for Tomcat to call Socket.accept()
Ok, so you are stating, that clients timeout while their requests are in the acceptCount/backlog. This is not what I am seeing. See below.
> So we have:
> maxThreads=4
> maxConnections=10
> acceptCount=20
>
> and a request processing time of 1 second.
>
> I'd guess that the OS is using a much large accept count. Let's model it.
>
> 0s - 50 connections in acceptCount
> 1s - 39 connections in acceptCount, 11 connections maintained by Tomcat,
> 4 requests processing
> 2s - 35 connections in acceptCount, 11 connections maintained by Tomcat,
> 4 requests processing, 4 completed requests 3s - 31 connections in
> acceptCount, 11 connections maintained by Tomcat,
> 4 requests processing, 8 completed requests 4s - 27 connections in
> acceptCount, 11 connections maintained by Tomcat,
> 4 requests processing, 12 completed requests 5s - 23 connections in
> acceptCount, 11 connections maintained by Tomcat,
> 4 requests processing, 16 completed requests 6s - 19 connections timed
> out, 11 connections maintained by Tomcat,
> 4 requests processing, 20 completed requests 7s - 19 connections timed
> out, 7 connections maintained by Tomcat,
> 4 requests processing, 24 completed requests 8s - 19 connections timed
> out, 3 connections maintained by Tomcat,
> 3 requests processing, 28 completed requests 9s - 19 connections timed
> out, 0 connections maintained by Tomcat,
> 0 requests processing, 31 completed requests
>
> That seems to match what you observed. That suggests the OS is using an
> acceptCount of at least 50.
I can see what you mean, but this is not what I see, because your assumption of a 1s processing time is wrong. The processing time of each request is 10s (thanks to a 10s sleep, which blocks the Thread).
So here is what I see instead (note I don't guess the response time, but do actually see/measure it):
0s - 4 requests processing, 11 connections maintained, 20 connections in acceptCount, 19 not in acceptCount
5s - 4 requests processing, 11 connections maintained, 20 connections in acceptCount, 0 not in acceptCount, 19 timed-out
10s - 4 requests processing, 11 connections maintained, 16 connections in acceptCount, 0 not in acceptCount, 19 timed-out, 4 requests processed
20s - 4 requests processing, 11 connections maintained, 12 connections in acceptCount, 0 not in acceptCount, 19 timed-out, 8 requests processed
30s - 4 requests processing, 11 connections maintained, 8 connections in acceptCount, 0 not in acceptCount, 19 timed-out, 12 requests processed
40s - 4 requests processing, 11 connections maintained, 4 connections in acceptCount, 0 not in acceptCount, 19 timed-out, 16 requests processed
50s - 4 requests processing, 11 connections maintained, 0 connections in acceptCount, 0 not in acceptCount, 19 timed-out, 20 requests processed
60s - 4 requests processing, 7 connections maintained, 0 connections in acceptCount, 0 not in acceptCount, 19 timed-out, 24 requests processed
70s - 3 requests processing, 3 connections maintained, 0 connections in acceptCount, 0 not in acceptCount, 19 timed-out, 28 requests processed
80s - 0 requests processing, 0 connections maintained, 0 connections in acceptCount, 0 not in acceptCount, 19 timed-out, 31 requests processed
The interesting thing to note is, that contrary to your statement, I don't see timeouts for requests in the acceptCount/backlog.
To verify my interpretation, I increased the acceptCount and I saw that the time outs decrease by the same number. I decreased the acceptCount and saw that the time outs increase.
The behaviour seems 100% predictable to me... just not as expected based on the documentation.
> Not up to Tomcat. Tomcat can only call Socket.accept() and does so under
> the control of maxConnections.
>
> Connection refused == acceptCount/backlog full (or no listening socket).
>
> Connection refusal is entirely under the control of the OS and will be driven
> largely by the actual value of acceptCount/backlog.
So if it is not up to Tomcat, then the documentation "Any further simultaneous requests will receive "connection refused" errors" is clearly wrong, isn't it?
________________________________
Pflichtangaben anzeigen<http://www.deutschebahn.com/pflichtangaben/20210430>
Nähere Informationen zur Datenverarbeitung im DB-Konzern finden Sie hier: http://www.deutschebahn.com/de/konzern/datenschutz
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: AW: AW: maxConnections behaving unexpected - no connection gets
ever refused
Posted by Mark Thomas <ma...@apache.org>.
On 19/05/2021 12:24, Paul P Wolf wrote:
> Thank you Thomas. I carefully read your explanation. It makes sense to me and is completely different from what I understood up until this point. With this new understanding, the problem still persists. Please let me rephrase my issues in the light of what I just learned.
>
> To summarize:
> - thread limit defines how many requests can be processed concurrently.
Yes, via maxThreads.
> - maxConnections defines how many connections are accepted by tomcat via socket.accept() and can be monitored by tomcat. this does not include the connections/requests being currently processed in an active thread.
Not quite. This does include connections currently being processed in an
active thread.
> - acceptCount is an OS backlag, which is not monitored by tomcat and the OS may decides to override the value.
Correct.
> - if all threads, maxConnections and acceptCount backlog are full, further requests get refused by the OS
Threads don't matter here. Just maxConnections and acceptCount/backlog
> Now my still persisting issues:
>> Say Tomcat can process 2000 requests a second and the typical client timeout
>> is 5s, then an acceptCount/backlog of anything up to 10000 should be OK but
>> above that some clients will time out because Tomcat won't be able to clear
>> the all backlog before the unprocessed client connections timeout.
> If there are more requests than there is space in the backlog and the maxConnections is reached, why would you expect client timeouts instead of refused connections? Timeouts are what I see, but not what I expect, when I read "Any further simultaneous requests will receive "connection refused" errors".
Let me expand on the point I was trying to make. Using the 2000 req/s
number above, a client timeout of 5s, an acceptCount of 20000 and no
keep-alive I'd expect to see something close to the following:
0s - 20000 connections in acceptCount
1s - 18000 connections in acceptCount, 2000 completed requests
2s - 16000 connections in acceptCount, 4000 completed requests
3s - 14000 connections in acceptCount, 6000 completed requests
4s - 12000 connections in acceptCount, 8000 completed requests
5s - 10000 connections in acceptCount, 10000 completed requests
>5s - 10000 client timeouts, 10000 completed requests
The clients timeout because they spend more than timeout in the
acceptCount/backlog queue waiting for Tomcat to call Socket.accept()
> Different question around the same issue: What would need to happen, so that there would be refused connections instead of client timeouts?
Same scenario as above but with an acceptCount of 5000
0s - 5000 connections in acceptCount, 15000 refused connections
1s - 3000 connections in acceptCount, 15000 refused connections,
2000 completed requests
2s - 1000 connections in acceptCount, 15000 refused connections,
4000 completed requests
3s - 15000 refused connections, 5000 completed requests
>> Your numbers are too close together. If you use numbers that are further
>> apart, the behaviour should be more obvious. Something like:
>> maxThreads=4
>> maxConnections=10
>> acceptCount=20
>
> What do you mean by "numbers are too close together"? Why would that be an issue? What would be far enough? Is there any documentation? The processing speed shouldn't be an issue, as the endpoints sleep for 10s.
My point was that with values a 3, 2 and 1 and the off-by-one behaviour
of maxConnections it is harder to match up observed numbers with
configuration values. If the configuration values are further apart it
should be easier to match observations, and changes in observations,
which configuration values and changes in configuration values.
Keep in mind that my numbers above assume things happen instantly
whereas in reality there is always going to be an ordering. The observed
numbers can be slightly different from what you expect sometimes. If
your configuration values are only 1 apart it will be hard to be sure
what you are seeing.
> Regardless, I tried your suggested configuration and nothing really changed: I see 31 successful requests and 19 timed out after 5 seconds. Still not a single refused connection. And considering the numbers, the OS acknowledged the configured acceptCount number.
So we have:
maxThreads=4
maxConnections=10
acceptCount=20
and a request processing time of 1 second.
I'd guess that the OS is using a much large accept count. Let's model it.
0s - 50 connections in acceptCount
1s - 39 connections in acceptCount, 11 connections maintained by Tomcat,
4 requests processing
2s - 35 connections in acceptCount, 11 connections maintained by Tomcat,
4 requests processing, 4 completed requests
3s - 31 connections in acceptCount, 11 connections maintained by Tomcat,
4 requests processing, 8 completed requests
4s - 27 connections in acceptCount, 11 connections maintained by Tomcat,
4 requests processing, 12 completed requests
5s - 23 connections in acceptCount, 11 connections maintained by Tomcat,
4 requests processing, 16 completed requests
6s - 19 connections timed out, 11 connections maintained by Tomcat,
4 requests processing, 20 completed requests
7s - 19 connections timed out, 7 connections maintained by Tomcat,
4 requests processing, 24 completed requests
8s - 19 connections timed out, 3 connections maintained by Tomcat,
3 requests processing, 28 completed requests
9s - 19 connections timed out, 0 connections maintained by Tomcat,
0 requests processing, 31 completed requests
That seems to match what you observed. That suggests the OS is using an
acceptCount of at least 50.
> Same question as before: what needs to change to make Tomcat refuse connections? This still seems like a bug to me.
Not up to Tomcat. Tomcat can only call Socket.accept() and does so under
the control of maxConnections.
Connection refused == acceptCount/backlog full (or no listening socket).
Connection refusal is entirely under the control of the OS and will be
driven largely by the actual value of acceptCount/backlog.
Other factors will complicate this further.
Clients may automatically retry connections. Browsers especially. I
often use a custom HTTP client (or just telnet) when testing Tomcat to
ensure that I can control exactly what the client is doing.
Once the client has sent the request, it will try to read the response.
If there is a large traffic spike and Tomcat takes a while to clear the
acceptCount queue you can see clients timing out waiting for the
response. You don't normally see this in browsers as they have a fairly
long read timeout to handler slow web sites.
HTH,
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
AW: AW: maxConnections behaving unexpected - no connection gets ever
refused
Posted by Paul P Wolf <Pa...@deutschebahn.com>.
Thank you Thomas. I carefully read your explanation. It makes sense to me and is completely different from what I understood up until this point. With this new understanding, the problem still persists. Please let me rephrase my issues in the light of what I just learned.
To summarize:
- thread limit defines how many requests can be processed concurrently.
- maxConnections defines how many connections are accepted by tomcat via socket.accept() and can be monitored by tomcat. this does not include the connections/requests being currently processed in an active thread.
- acceptCount is an OS backlag, which is not monitored by tomcat and the OS may decides to override the value.
- if all threads, maxConnections and acceptCount backlog are full, further requests get refused by the OS
Now my still persisting issues:
> Say Tomcat can process 2000 requests a second and the typical client timeout
> is 5s, then an acceptCount/backlog of anything up to 10000 should be OK but
> above that some clients will time out because Tomcat won't be able to clear
> the all backlog before the unprocessed client connections timeout.
If there are more requests than there is space in the backlog and the maxConnections is reached, why would you expect client timeouts instead of refused connections? Timeouts are what I see, but not what I expect, when I read "Any further simultaneous requests will receive "connection refused" errors".
Different question around the same issue: What would need to happen, so that there would be refused connections instead of client timeouts?
> Your numbers are too close together. If you use numbers that are further
> apart, the behaviour should be more obvious. Something like:
> maxThreads=4
> maxConnections=10
> acceptCount=20
What do you mean by "numbers are too close together"? Why would that be an issue? What would be far enough? Is there any documentation? The processing speed shouldn't be an issue, as the endpoints sleep for 10s.
Regardless, I tried your suggested configuration and nothing really changed: I see 31 successful requests and 19 timed out after 5 seconds. Still not a single refused connection. And considering the numbers, the OS acknowledged the configured acceptCount number.
Same question as before: what needs to change to make Tomcat refuse connections? This still seems like a bug to me.
________________________________
Pflichtangaben anzeigen<http://www.deutschebahn.com/pflichtangaben/20210430>
Nähere Informationen zur Datenverarbeitung im DB-Konzern finden Sie hier: http://www.deutschebahn.com/de/konzern/datenschutz
Re: AW: maxConnections behaving unexpected - no connection gets ever
refused
Posted by Mark Thomas <ma...@apache.org>.
On 19/05/2021 09:28, Paul P Wolf wrote:
<snip/>
> In regards to point 5 and 6, let me try to point out my issues with the documentation and your explanations:
> - "Each incoming request requires a thread for the duration of that request. If more simultaneous requests are received than can be handled by the currently available request processing threads, additional threads will be created up to the configured maximum (the value of the maxThreads attribute)." So far the documentation sounds good to me.
Ack.
> - "If still more simultaneous requests are received, they are stacked up inside the server socket created by the Connector, up to the configured maximum (the value of the acceptCount attribute)." This is what I meant with "accept queue" or what you call "Tomcat's own queue". Would you usually configure the acceptCount to be less than maxConnections or is it completely unrelated to maxConnections? I would intuitively assume you would set it to less, but now I am not sure anymore.
maxConnections is the maximum number of concurrent connections Tomcat is
prepared to establish via a call to Socket.accept(). Once that limit is
reached Tomcat will not call Socket.accept() again until one or more
established connections is closed. Tomcat will then call Socket.accept()
until maxConnections is reached or Socket.accept() blocks waiting for an
incoming connection.
The acceptCount/backlog is maintained by the OS and is invisible to
Tomcat. It is the queue of connections that have been received by the
network stack and are waiting for Tomcat to call Socket.accept() to
start processing them.
maxConnections and acceptCount are entirely unrelated.
The correct setting for maxConnections is driven largely by the maximum
concurrent load you want (essentially maxThreads), whether you are using
keep-alive and the percentage of time kept-alive connections are idle vs
active.
For example if you had a system that could support 200 concurrently
processing requests, those connections use keep-alive had active
requests for 10% of the time then a good value for maxConnections would
be around 200/10% = 2000
The correct setting for acceptCount is driven by the maximum spike in
new connections you want to accept. The higher the value, the higher the
spike in new connection attempts you'll be able to support but those new
connections will then be queued in the acceptCount/backlog queue waiting
for Tomcat to process them. You need to consider how quickly Tomcat can
clear that backlog vs typical client timeouts.
Say Tomcat can process 2000 requests a second and the typical client
timeout is 5s, then an acceptCount/backlog of anything up to 10000
should be OK but above that some clients will time out because Tomcat
won't be able to clear the all backlog before the unprocessed client
connections timeout.
> - "Any further simultaneous requests will receive "connection refused" errors, until resources are available to process them." Now if that is the case, than what does maxConnection have to do with anything at all?
See above. That is describing what happens when maxConnections has been
reached AND the acceptCount/backlog queue is full.
> Also I just don't see any connections being refused, but instead they linger in SYN-SENT state. You say, that acceptCount can be overridden by the OS, but why would that take away tomcats possibility of monitoring those connections, as "they are stacked up inside the server socket created by the Connector, up to the configured maximum (the value of the acceptCount attribute)"?
No they are not. You are confusing the connections where Tomcat has
called Socket.accept() but not passed the socket to a thread for
processing with connections still in the acceptCount/backlog queue.
> The way I see it, tomcat should always be able to monitor those connections or never at. Are there separate acceptCounts for Tomcat/OS/TCP stack?
Tomcat has full visibility of the number of connections where
Socket.accept() has been called. Tomcat has no visibility of the number
of connections in the acceptCount/backlog queue.
> - "Note that once the [maxConnection] limit has been reached, the operating system may still accept connections based on the acceptCount setting". Here again my confusion rises: Does this only apply/make sense if the OS overrides the acceptCount?
There is always an acceptCount. The issue is whether the OS follows the
setting provided by Tomcat or does its own thing anyway.
> If so, would Tomcat still be able to monitor those connections in the server socket created by the Connector or not?
See above. Tomcat tracks connections where Socket.accept() has been
called. Tomcat cannot track connection requests in the
acceptCount/backlog queue.
> If the OS doesn't override the acceptCount, is Tomcat then able to monitor the connections?
Whether the OS follows the acceptCount setting provided by Tomcat or not
has no impact on Tomcat's inability to monitor the acceptCount/backlog
queue.
> Furthermore: The documentation implies that if acceptCount < maxConnections, than it will only allow maxConnections, but as you pointed out yourself earlier, this doesn't seem to be the case, but instead it's "maxConnections (3) + acceptCount (2) + 1".
Which text implies that?
> What suggests to me that the OS doesn't override the acceptCount in my case is, that the ESTABLISHED connections always correlate to my acceptCount setting, but still connections don't get refused. This seems like a bug to me.
Your numbers are too close together. If you use numbers that are further
apart, the behaviour should be more obvious. Something like:
maxThreads=4
maxConnections=10
acceptCount=20
> - "When you say "accept queue", I think "TCP stack backlog" which is 100% in the OS and not visible to Tomcat.
Correct.
> Tomcat itself has its own queue of incoming connections (ones that have been accept()ed but no thread has yet been assigned).#
Correct.
> I suppose you mean this Tomcat-queue when you say "requests outside the queue, but still within the maxConnections limit"?"" - with accept queue I mean the internal one. How do you know what goes into the TCP stack backlog and what goes into the internal queue? What I mean by "requests outside the queue, but still within the maxConnections limit" is the following: Imagine your acceptCount is 5, but you accept more, because maxConnection 13. So maxConnection - acceptCount = 8. What happens with those requests? Do they land in the TCP stack backlog?
A new connection first gets added to the acceptCount/backlog queue. When
Tomcat calls Socket.accept() it is removed from the acceptCount/backlog
queue and added to the current connection count. It is then added to
the Poller. Ignoring writes, when there is data to read, the Poller
passes the connection to the thread pool. Once the data has been read,
the connection is returned to the Poller (or closed). When the
connection is closed the current connection count is reduced.
> As discussed before, I actually see 19 ESTABLISHED connections with a configuration like that. So do actually 13 land in the backlog? To me this seems like a (documentation) bug.
That looks right to me, assuming that the OS is putting the connections
into the established state as soon as they land in the acceptQueue /
backlog. You have maxConnections(13) + acceptCount(5) + 1 = 19
established connections.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
AW: maxConnections behaving unexpected - no connection gets ever
refused
Posted by Paul P Wolf <Pa...@deutschebahn.com>.
> Paul,
Thanks for the reply. I am not really much further with my main issue, but I hope this reply provides more information to you, so you can either clear up my confusion or see how tomcat doesn't work as intended in my case.
> On 5/18/21 07:44, Paul P Wolf wrote:
> > Hi,
> >
> > I am trying to run a spring boot application with an embedded tomcat. In a
> scenario, where there is a lot of load on the service, I would like tomcat to
> refuse connections or return something like a 503 Service Unavailable. From
> what I understand, you could have this behaviour by setting maxConnections
> and and any additional connections get refused. At least this was how the old
> blocking io acceptor worked, from what I understand.
> >
> > The documentation says "The maximum number of connections that the
> server will accept and process at any given time. When this number has been
> reached, the server will accept, but not process, one further connection. This
> additional connection be blocked until [...]".
> > However the documentation doesn't really state what happens if
> maxConnection+2 connections are reached.
> >
> > I tried to run my application with following settings (embedded-tomcat
> 9.0.45):
> > maxConnections=3
> > acceptCount=2
> > maxThreads=1
> > processorCache=1
> >
> > I created an endpoint, which just sleeps for a few seconds and returns
> > a string. When I create 50 separate connection via curl instances to
> > call that service I see the following behaviour with the NIO Acceptor:
> >
> > * 6 http connections are accepted immediately (maxThreads +
> > acceptCount + maxConnections)
> The maxThreads setting should not be relevant, here. maxConnections
> counts the total connections without regard to how many of them are
> actually having work done (which requires a thread from maxThreads).
>
> So accepting 6 connections really means:
>
> maxConnections (3) + acceptCount (2) + 1 (because maxConnections says it
> will accept 1+maxConnections, which is a little confusing IMO).
>
I tried some different settings to check this and your interpretation seems to be correct. Thanks!
> > * 44 http connections aren't established just, but neither are they
> > refused. I will call them "blocked", but different from the
> > specification those are 44 blocked connections and not just 1
> What is the TCP state of the first 6 connections? What about the other 44?
> What is the difference between "accepted" and "blocked", and how are you
> telling them apart?
I tell them apart by inferring from curl's behaviour. I have a service endpoint sleep for 10s. I start 50 curls in parallel with a 5s connect-timeout. 6 connections go through (after a total of 60 seconds), while the other 44 fail after the 5s timeout. Makes sense?
By that I infer that 44 connections are in the SYN-SENT state and the other 6 are in the ESTABLISHED state, after the curl instances have started.
> > * once the first request finishes, the latest (blocked) requests
> > gets a connection and is being processed (not a request from the
> > accept-queue or one of the other already established connections)
> So the queue is behaving "unfairly"?
>
> > * when there are no further blocked requests, the requests still
> > get processed in last in first out order
> More unfairness.
>
> You didn't post your <Connector> configuration (which is pretty critical for
> trying to think about these things), but I suspect you aren't using an
> <Executor>, which may ensure fairness. (Older Tomcats used an internal
> thread pool which was NOT an "Executor" but later Tomcat should always be
> using an Executor, which was intended to enforce fairness. Hmm.)
Correct, it behaves unfairly. As stated I am using spring boot and an embedded tomcat. From why I understand there is no (obvious) way to provide traditional tomcat configuration files. However I used an TomcatConnectorCustomizer to inspect the connector and there seems to be no Executor configured. Would you say then, that there is a problem with the default spring boot configuration not using an Executor?
> > * I see some timeouts after a while with this setup, depending on
> > what timeouts I set on curl. The requests without an established
> > connection timeout with "connect-timeout" parameter and the ones with
> > established connections depending on the "max-time" parameter.
> When you get a timeout, what kind of timeout are you encountering (on the
> client side)? Is this a "connect timeout" or a "read timeout"?
I encounter both, depending on the configuration. My point here was that I could use those timeouts to infer the tcp state as stated above.
> > Now I have a lot of questions and I suspect some bugs, but I wanted to ask
> on this list first:
> >
> > 1. Is there a way to refuse connections, instead of blocking them?
>
> maxConnections + acceptCount *should* be doing this. Remember that
> acceptCount is just a suggestion to the OS. The OS is free to always accept
> 65535 connections into its TCP/IP stack whether you say so or not. Likewise, if
> you are connecting through other network devices (e.g.
> load-balancer, whatever), there may be different rules in different places.
> I'm assuming you are connecting directly to Tomcat on localhost, but you
> didn't specify.
You are correct in assuming that I tested everything on localhost. So do I understand you correctly? On one hand tomcat would behave as intended by me, but on the other hand, I can't really configure the acceptCount? How would I figure out what the actual accept count of my OS is? If acceptCount turns out to be greater than 16 or so that would be bad for my situation. However I don't really think, that the OS overrides the acceptCount, because I can see the effect of changing the parameter by observing the number of ESTABLISHED connections. So this rather seems like a bug to me considering those assumptions.
> > 2. I can't control the connect-timeout of the clients - is there a
> > way to control it on the server side? (I tried to change
> > keepAliveTimeout and connectionTimeout without success)
> It's impossible for the server to instruct the client to time-out. The situation
> here is that the client can't reach the server therefore the server can't
> communicate anything to the client.
>
> If you can't change the connect-timeout of the clients, do you know what the
> effective timeout actually is?
Only in my local test setup
> > 3. Why doesn't the specification state what happens if many more
> > requests are being made than maxConnections allows for? (esp. with the
> > behaviour being different among acceptors)
> Which specification? The Tomcat documentation explains what happens with
> 0-maxConnections, then maxConnections + 1, then maxConnection + N
> where N > 1, so ... what are you actually asking?
I am refering to http://tomcat.apache.org/tomcat-9.0-doc/config/http.html. I didn't find an explanation before, but now I see "[...] Any further simultaneous requests will receive "connection refused" errors, until resources are available to process them.". I will try to summarize my understanding and problems with the configuration parameters bellow
> > 4. The processing order is troublesome - early requests get starved
> > by following ones. Why is this lifo instead of fifo?
> Please try explicitly configuring <Executor> and see if that changes anything.
as explained above, there is not <Executor> config in spring boot, but it seems no executor is set by default!?
> > 5. The number of accepted connections is unintuitively. Why are
> > active connections with requests either being processed or being in
> > the accept queue not counted as active connections (actual connections
> > = maxConnections + acceptCount + maxThreads)?
> Tomcat cannot see the OS's TCP backlog (the "accept queue") so nothing can
> be counted there. The only thing Tomcat can see is up to maxConnections + 1
> at which point the acceptor thread will stop accepting incoming connections
> (and those client connections will queue-up in the TCP stack's backlog).
>
> > 6. What is the difference between requests in the accept queue and
> > requests outside the queue, but still within the maxConnections limit?
> Let's be clear about nomenclature. When you say "accept queue", I think
> "TCP stack backlog" which is 100% in the OS and not visible to Tomcat.
> Tomcat itself has its own queue of incoming connections (ones that have
> been accept()ed but no thread has yet been assigned). I suppose you mean
> this Tomcat-queue when you say "requests outside the queue, but still
> within the maxConnections limit"?
>
> The difference is that the former (backlog) is not visible to Tomcat at all (or
> any server that you might be using, this is not Tomcat- or
> Java-specific) and the latter (having been accept()ed by Tomcat) are being
> managed by Tomcat.
In regards to point 5 and 6, let me try to point out my issues with the documentation and your explanations:
- "Each incoming request requires a thread for the duration of that request. If more simultaneous requests are received than can be handled by the currently available request processing threads, additional threads will be created up to the configured maximum (the value of the maxThreads attribute)." So far the documentation sounds good to me.
- "If still more simultaneous requests are received, they are stacked up inside the server socket created by the Connector, up to the configured maximum (the value of the acceptCount attribute)." This is what I meant with "accept queue" or what you call "Tomcat's own queue". Would you usually configure the acceptCount to be less than maxConnections or is it completely unrelated to maxConnections? I would intuitively assume you would set it to less, but now I am not sure anymore.
- "Any further simultaneous requests will receive "connection refused" errors, until resources are available to process them." Now if that is the case, than what does maxConnection have to do with anything at all? Also I just don't see any connections being refused, but instead they linger in SYN-SENT state. You say, that acceptCount can be overridden by the OS, but why would that take away tomcats possibility of monitoring those connections, as "they are stacked up inside the server socket created by the Connector, up to the configured maximum (the value of the acceptCount attribute)"? The way I see it, tomcat should always be able to monitor those connections or never at. Are there separate acceptCounts for Tomcat/OS/TCP stack?
- "Note that once the [maxConnection] limit has been reached, the operating system may still accept connections based on the acceptCount setting". Here again my confusion rises: Does this only apply/make sense if the OS overrides the acceptCount? If so, would Tomcat still be able to monitor those connections in the server socket created by the Connector or not? If the OS doesn't override the acceptCount, is Tomcat then able to monitor the connections? Furthermore: The documentation implies that if acceptCount < maxConnections, than it will only allow maxConnections, but as you pointed out yourself earlier, this doesn't seem to be the case, but instead it's "maxConnections (3) + acceptCount (2) + 1". What suggests to me that the OS doesn't override the acceptCount in my case is, that the ESTABLISHED connections always correlate to my acceptCount setting, but still connections don't get refused. This seems like a bug to me.
- "When you say "accept queue", I think "TCP stack backlog" which is 100% in the OS and not visible to Tomcat. Tomcat itself has its own queue of incoming connections (ones that have been accept()ed but no thread has yet been assigned). I suppose you mean this Tomcat-queue when you say "requests outside the queue, but still within the maxConnections limit"?"" - with accept queue I mean the internal one. How do you know what goes into the TCP stack backlog and what goes into the internal queue? What I mean by "requests outside the queue, but still within the maxConnections limit" is the following: Imagine your acceptCount is 5, but you accept more, because maxConnection 13. So maxConnection - acceptCount = 8. What happens with those requests? Do they land in the TCP stack backlog? As discussed before, I actually see 19 ESTABLISHED connections with a configuration like that. So do actually 13 land in the backlog? To me this seems like a (documentation) bug.
I hope this makes it clearer.
>
> > Any insight would be appreciated. If there is anything in here, that I
> > should report as bug, please let me know. I am happy to file one.
>
> We will need more information. In your next post, please provide either your
> <Connector> if you are using XML-based configuration or all your code that is
> creating/configuring the Tomcat Connector. Also, please give some default
> about your JVM, OS, and kernel version, etc.
I don't use XML config and use the default settings of embedded tomcat 9.0.45 as shipped with spring-boot 2.4.5
The JVM is: openjdk 11.0.9.1
The kernel version is: 5.4.0-73-generic
If any additional info is needed, let me know.
________________________________
Pflichtangaben anzeigen<http://www.deutschebahn.com/pflichtangaben/20210430>
Nähere Informationen zur Datenverarbeitung im DB-Konzern finden Sie hier: http://www.deutschebahn.com/de/konzern/datenschutz
Re: maxConnections behaving unexpected - no connection gets ever
refused
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Paul,
On 5/18/21 07:44, Paul P Wolf wrote:
> Hi,
>
> I am trying to run a spring boot application with an embedded tomcat. In a scenario, where there is a lot of load on the service, I would like tomcat to refuse connections or return something like a 503 Service Unavailable. From what I understand, you could have this behaviour by setting maxConnections and and any additional connections get refused. At least this was how the old blocking io acceptor worked, from what I understand.
>
> The documentation says "The maximum number of connections that the server will accept and process at any given time. When this number has been reached, the server will accept, but not process, one further connection. This additional connection be blocked until [...]".
> However the documentation doesn't really state what happens if maxConnection+2 connections are reached.
>
> I tried to run my application with following settings (embedded-tomcat 9.0.45):
> maxConnections=3
> acceptCount=2
> maxThreads=1
> processorCache=1
>
> I created an endpoint, which just sleeps for a few seconds and
> returns a string. When I create 50 separate connection via curl
> instances to call that service I see the following behaviour with the
> NIO Acceptor:
>
> * 6 http connections are accepted immediately (maxThreads +
> acceptCount + maxConnections)
The maxThreads setting should not be relevant, here. maxConnections
counts the total connections without regard to how many of them are
actually having work done (which requires a thread from maxThreads).
So accepting 6 connections really means:
maxConnections (3) + acceptCount (2) + 1 (because maxConnections says it
will accept 1+maxConnections, which is a little confusing IMO).
> * 44 http connections aren't established just, but neither are they
> refused. I will call them "blocked", but different from the
> specification those are 44 blocked connections and not just 1
What is the TCP state of the first 6 connections? What about the other
44? What is the difference between "accepted" and "blocked", and how are
you telling them apart?
> * once the first request finishes, the latest (blocked) requests
> gets a connection and is being processed (not a request from the
> accept-queue or one of the other already established connections)
So the queue is behaving "unfairly"?
> * when there are no further blocked requests, the requests still
> get processed in last in first out order
More unfairness.
You didn't post your <Connector> configuration (which is pretty critical
for trying to think about these things), but I suspect you aren't using
an <Executor>, which may ensure fairness. (Older Tomcats used an
internal thread pool which was NOT an "Executor" but later Tomcat should
always be using an Executor, which was intended to enforce fairness. Hmm.)
> * I see some timeouts after a while with this setup, depending on
> what timeouts I set on curl. The requests without an established
> connection timeout with "connect-timeout" parameter and the ones with
> established connections depending on the "max-time" parameter.
When you get a timeout, what kind of timeout are you encountering (on
the client side)? Is this a "connect timeout" or a "read timeout"?
> Now I have a lot of questions and I suspect some bugs, but I wanted to ask on this list first:
>
> 1. Is there a way to refuse connections, instead of blocking them?
maxConnections + acceptCount *should* be doing this. Remember that
acceptCount is just a suggestion to the OS. The OS is free to always
accept 65535 connections into its TCP/IP stack whether you say so or
not. Likewise, if you are connecting through other network devices (e.g.
load-balancer, whatever), there may be different rules in different
places. I'm assuming you are connecting directly to Tomcat on localhost,
but you didn't specify.
> 2. I can't control the connect-timeout of the clients - is there a
> way to control it on the server side? (I tried to change
> keepAliveTimeout and connectionTimeout without success)
It's impossible for the server to instruct the client to time-out. The
situation here is that the client can't reach the server therefore the
server can't communicate anything to the client.
If you can't change the connect-timeout of the clients, do you know what
the effective timeout actually is?
> 3. Why doesn't the specification state what happens if many more
> requests are being made than maxConnections allows for? (esp. with
> the behaviour being different among acceptors)
Which specification? The Tomcat documentation explains what happens with
0-maxConnections, then maxConnections + 1, then maxConnection + N where
N > 1, so ... what are you actually asking?
> 4. The processing order is troublesome - early requests get starved
> by following ones. Why is this lifo instead of fifo?
Please try explicitly configuring <Executor> and see if that changes
anything.
> 5. The number of accepted connections is unintuitively. Why are
> active connections with requests either being processed or being in
> the accept queue not counted as active connections (actual
> connections = maxConnections + acceptCount + maxThreads)?
Tomcat cannot see the OS's TCP backlog (the "accept queue") so nothing
can be counted there. The only thing Tomcat can see is up to
maxConnections + 1 at which point the acceptor thread will stop
accepting incoming connections (and those client connections will
queue-up in the TCP stack's backlog).
> 6. What is the difference between requests in the accept queue and
> requests outside the queue, but still within the maxConnections
> limit?
Let's be clear about nomenclature. When you say "accept queue", I think
"TCP stack backlog" which is 100% in the OS and not visible to Tomcat.
Tomcat itself has its own queue of incoming connections (ones that have
been accept()ed but no thread has yet been assigned). I suppose you mean
this Tomcat-queue when you say "requests outside the queue, but still
within the maxConnections limit"?
The difference is that the former (backlog) is not visible to Tomcat at
all (or any server that you might be using, this is not Tomcat- or
Java-specific) and the latter (having been accept()ed by Tomcat) are
being managed by Tomcat.
> Any insight would be appreciated. If there is anything in here, that
> I should report as bug, please let me know. I am happy to file one.
We will need more information. In your next post, please provide either
your <Connector> if you are using XML-based configuration or all your
code that is creating/configuring the Tomcat Connector. Also, please
give some default about your JVM, OS, and kernel version, etc.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org