You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Jo...@wellsfargo.com.INVALID on 2017/02/03 16:17:06 UTC
Increasing maxThreads results in more "Connection refused" errors?
All,
I'm doing some scalability testing of an app in Tomcat 7.0.73.
Some relevant connector config details:
maxThreads=80
maxKeepAliveRequests=100
keepAliveTimeout=10000
maxConnections unspecified (defaults to maxThreads according to the docs)
acceptCount unspecified (100 according to the docs)
clientAuth=true
FWIW, I'm testing two Tomcat instances on the same server. They are behind a load balancer.
It appears that when the load generator tries to exceed maxThreads, the new connection rate goes up quickly and CPU usage shoots up with it. I assume this is because Tomcat is proactively closing idle keep alive connections to service new connections. In an effort to keep the CPU in check, I tried increasing maxThreads from 80 to 120. This seemed to work well in a lot of ways. New connection rate didn't increase as much, CPU didn't increase as much, there was more connection reuse (more requests per connection,) and response times didn't deteriorate as much.
Great, right?
Then I noticed a large increase in "Connection refused" errors on the load generator. In other words, a higher maxThreads also results in a high error rate. The total hits per second from the client's perspective is about 60 in both cases. With maxThreads=80, there are about 3 connection refused errors per second at that volume. With maxThreads=120, there are about 10 connection refused errors per second.
I have no idea why this is. Can someone explain this or what I can do about it?
Thanks
John
RE: Increasing maxThreads results in more "Connection refused"
errors?
Posted by Jo...@wellsfargo.com.INVALID.
Chris
> -----Original Message-----
> From: Christopher Schultz [mailto:chris@christopherschultz.net]
> Sent: Friday, February 03, 2017 11:08 AM
> To: Tomcat Users List
> Subject: Re: Increasing maxThreads results in more "Connection refused" errors?
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> John,
>
> On 2/3/17 11:17 AM, John.E.Gregg@wellsfargo.com.INVALID wrote:
> > I'm doing some scalability testing of an app in Tomcat 7.0.73.
> >
> > Some relevant connector config details: maxThreads=80
> > maxKeepAliveRequests=100 keepAliveTimeout=10000 maxConnections
> > unspecified (defaults to maxThreads according to the docs) acceptCount
> > unspecified (100 according to the docs) clientAuth=true
>
> Which connector?
>
> > FWIW, I'm testing two Tomcat instances on the same server. They are
> > behind a load balancer.
>
> Which load-balancer using which protocol?
>
> > It appears that when the load generator tries to exceed maxThreads,
> > the new connection rate goes up quickly and CPU usage shoots up with
> > it.
>
> When you say the "new connection rate" goes up, what exactly do you mean?
> When using a load-generator, I would certainly expect the "new connection
> rate" to go up.
>
> > I assume this is because Tomcat is proactively closing idle keep alive
> > connections to service new connections.
>
> No. Tomcat behaves the same under load as when not under load. If you are
> using the BIO connector (it sounds like you are), then each connection will have
> to wait for the KeepAlive timeout to expire before servicing another request
> from another client.
>
> This is complicated by the reverse-proxy and so those details are important to
> provide.
>
> > In an effort to keep the CPU in check, I tried increasing maxThreads
> > from 80 to 120.
>
> If the CPU is very busy, allowing more threads to run is counter-productive, isn't
> it?
>
> > This seemed to work well in a lot of ways. New connection rate didn't
> > increase as much, CPU didn't increase as much, there was more
> > connection reuse (more requests per connection,) and response times
> > didn't deteriorate as much.
> >
> > Great, right?
> >
> > Then I noticed a large increase in "Connection refused" errors on the
> > load generator. In other words, a higher maxThreads also results in a
> > high error rate. The total hits per second from the client's
> > perspective is about 60 in both cases. With maxThreads=80, there are
> > about 3 connection refused errors per second at that volume. With
> > maxThreads=120, there are about 10 connection refused errors per
> > second.
>
> Are you hitting the load-balancer or Tomcat directly during your load-test?
>
> If you are hitting the load-balancer, then the details of that configuration are
> more important for the client: if the lb will only accept 10 connections, it doesn't
> matter what Tomcat is willing to accep t.
>
> > I have no idea why this is. Can someone explain this or what I can do
> > about it?
>
> Post more data. We'll get to the bottom of it.
>
> - -chris
Thanks for asking those good questions. When I checked with the load balancer people, they told me there was a 70 connection limit to each Tomcat server enforced by the load balancer. It took a while to get that limit removed, but now that it's gone, the connection errors completely went away. I think the limit is a good idea and we will probably re-establish a higher one after additional testing.
Thanks
John
Re: Increasing maxThreads results in more "Connection refused"
errors?
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
John,
On 2/3/17 11:17 AM, John.E.Gregg@wellsfargo.com.INVALID wrote:
> I'm doing some scalability testing of an app in Tomcat 7.0.73.
>
> Some relevant connector config details: maxThreads=80
> maxKeepAliveRequests=100 keepAliveTimeout=10000 maxConnections
> unspecified (defaults to maxThreads according to the docs)
> acceptCount unspecified (100 according to the docs)
> clientAuth=true
Which connector?
> FWIW, I'm testing two Tomcat instances on the same server. They
> are behind a load balancer.
Which load-balancer using which protocol?
> It appears that when the load generator tries to exceed
> maxThreads, the new connection rate goes up quickly and CPU usage
> shoots up with it.
When you say the "new connection rate" goes up, what exactly do you
mean? When using a load-generator, I would certainly expect the "new
connection rate" to go up.
> I assume this is because Tomcat is proactively closing idle keep
> alive connections to service new connections.
No. Tomcat behaves the same under load as when not under load. If you
are using the BIO connector (it sounds like you are), then each
connection will have to wait for the KeepAlive timeout to expire
before servicing another request from another client.
This is complicated by the reverse-proxy and so those details are
important to provide.
> In an effort to keep the CPU in check, I tried increasing
> maxThreads from 80 to 120.
If the CPU is very busy, allowing more threads to run is
counter-productive, isn't it?
> This seemed to work well in a lot of ways. New connection rate
> didn't increase as much, CPU didn't increase as much, there was
> more connection reuse (more requests per connection,) and response
> times didn't deteriorate as much.
>
> Great, right?
>
> Then I noticed a large increase in "Connection refused" errors on
> the load generator. In other words, a higher maxThreads also
> results in a high error rate. The total hits per second from the
> client's perspective is about 60 in both cases. With maxThreads=80,
> there are about 3 connection refused errors per second at that
> volume. With maxThreads=120, there are about 10 connection refused
> errors per second.
Are you hitting the load-balancer or Tomcat directly during your
load-test?
If you are hitting the load-balancer, then the details of that
configuration are more important for the client: if the lb will only
accept 10 connections, it doesn't matter what Tomcat is willing to accep
t.
> I have no idea why this is. Can someone explain this or what I can
> do about it?
Post more data. We'll get to the bottom of it.
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJYlLjdAAoJEBzwKT+lPKRYRyYQAKs8F2HQ60cmlInZuubIQaq5
MuJxMZV/rCSgFMwoBzR+KDMS2PlTUvpoUMlp3/tTz8oON4x6j3jGzQdA9EGcjje4
W7D/G6TmoMptjeI8Zz0UbYnIRM5JXEvxOzuuGURbER7iCaa9VVsoWnhcv+Ja02Ut
Xl9dzEwFpfXasr8ioHL5Yn4W7F0QUDz9vboBQ/9TwYAib9UAKEP8uzWWjh6ylLaw
Z9ShyJtCU5rQ47ZVYhCcd+ZgCnP0sbcjD9kloD29dDFHgALTWPuAQA50ZDPiAUBR
I0WuDMGITpsl0ni4jbwMjrnv3voaVD0HtSCZon6fynux2BciaHVGx8s0lGtRMz6v
isG5L6+io8JR3asNdb8Ug3G+5zX8FMZx5VwlsQNVqo10xsEXYosACtqTmASyLPx6
EwgLDnCeMwKK3A6hmKqw0aEl1R9TqlpLKK5h9g7+RPpm6imUdsZvHAjSL3OWJa8e
Laqujf8MxCN/DsZhKOocwv5F98zcsn/mD9VZ/2IG+78JqfB3qU4wa5brD11n71xe
uYMgGIH0zIGN9ZGRtWdAbPbY2JIu2qIkFzz06hgYr/W5QLrZzy2dGYkyGBPMHrDV
TPR1+EbNNOtGVq5IF6t8ZChOE6toNnIdWdy5bVNiXEhIpVXquWsfoysNVlBmyA4V
VICVGii1G2WK9kfP7Q82
=UyRB
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org