You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Norbert Elbanbuena <no...@teligentip.com> on 2020/08/16 16:35:11 UTC
Tomcat Handling close_waits
Hi,
We are experiencing a flood of close_waits in our server. Interestingly, all of the sessions stuck in close_waits are sourced from Amazon IP addresses. Our second server running the same setup (hardware spec, OS version, Tomcat version, etc.) and had almost the same session count and our application and Tomcat didn't go unresponsive.
Is there any tuneable parameter for the APR connector for Tomcat to close the stuck sessions, rather than waiting for the application?
Warm regards,
Norbert Elbanbuena
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat Handling close_waits
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Norbert,
On 8/16/20 13:16, Norbert Elbanbuena wrote:
> I also noticed that while server receives the connection requests,
> we are seeing multiple requests from the same sources. Some same
> source requests (FIN-WAIT) are all in state while other same
> sources requests are in other state (some in FIN-WAIT or close_wait
> and some Established).
>
> Why are we seeing multiple requests from the same source at the
> same time? Doesn't each socket request exhaust a thread on the
> application?
Not always.
See the table at the bottom of this page:
http://tomcat.apache.org/tomcat-9.0-doc/config/http.html#Connector_Compa
rison
The NIO and APR connectors are non-blocking while waiting for the
"next request", so if a client connects and never explicitly
disconnects, you can have a rising connection count without running
out of request-processing threads.
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl88Xo4ACgkQHPApP6U8
pFjvVBAAneeiMR8ZeEwa2kzymC4PKAAu6ZhiBUjMpGeyfBjgPrOxegQV+9ucyOo/
Ydk++ubxpYu5NRVruvCyhGbKL6x0x8Ds9jUTKvctrMLrmXK2Fxm/+O50Y0Q2ddfr
n1QYT9jmlv4s9rybSoBinm6tOCTjZLxExp+f4eAgC6VM5Zfl9Z3QkvUHU+YoV1l6
S4AJteXjZZEytmURxtwYpmtHPVknWvg6vlGAP8FFbH2KAKoaqv0DUB+zWe3vtSoD
0+55j2mJJphQkY00nfqzxEZDgfy68+bcqA+eLPM7ya+DsDjHiaUWOXOFwA8hHAJ3
L7+USQtP28EgEvHU6LsTBwwZxIYo4bCIAiC/Ncs8AdxBUCDiq2dNaS+pESUIkef5
IgFu+mrcZycNxaabmebfiCexIt4Oc+JQzUo1VhGYEuulPic67WyYFeYAgoasSIqc
MBAwV6/ahYNfoZwiJWjBPkkGBTNDuVQbk8PNbJ9JmvP0yWVolZKZDr9OLJKVvfSF
r576LhEIa0fwdA1x115ipnjYGWzPaobnX7xzVjpNNUXtbISG9gGXut57z7u7Ttdw
AEBedOx3vFdmBIn4Rw4y3o8it7hOVlMNQqW6f4LW7+KCDKf3yK/4+dipF7exjh2e
+7dBdb0UU2xcLOre6ZRJD3XFp4V3bTYFu3QiZl2LStpJjMsfqmM=
=r7Rk
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat Handling close_waits
Posted by "André Warnier (tomcat/perl)" <aw...@ice-sa.com>.
Hi.
On 17.08.2020 02:51, Norbert Elbanbuena wrote:
> Hi Paul,
>
> Yes these are some samples, I have about 300 of them getting stuck hourly
>
> tcp 761 0 192.168.1.50:58870 74.112.28.109:8011 CLOSE_WAIT
> tcp 0 0 192.168.1.50:56938 192.168.1.50:61616 CLOSE_WAIT
> tcp 0 0 192.168.1.50:56924 192.168.1.50:61616 CLOSE_WAIT
> tcp 0 0 192.168.1.50:56910 192.168.1.50:61616 CLOSE_WAIT
> tcp 0 0 192.168.1.50:56912 192.168.1.50:61616 CLOSE_WAIT
> tcp6 0 6240 192.168.1.50:443 52.11.72.45:47123 CLOSE_WAIT
> tcp6 32 0 192.168.1.50:443 34.209.104.242:13402 CLOSE_WAIT
> tcp6 268 0 192.168.1.50:443 108.162.244.28:40864 CLOSE_WAIT
> tcp6 32 0 192.168.1.50:443 35.167.185.49:10019 CLOSE_WAIT
> tcp6 32 0 192.168.1.50:443 52.24.48.141:60660 CLOSE_WAIT
> tcp6 202 0 192.168.1.50:443 199.189.191.86:51716 CLOSE_WAIT
> tcp6 202 0 192.168.1.50:443 199.189.191.86:1386 CLOSE_WAIT
>
>
Try forcing a Java Garbage Collection in Tomcat, and check if these CLOSE_WAIT sockets are
still there after that. If they are gone, you probably have a server application which
leaves these sockets "dangling" and does not properly close them.
Other people on this list may be able to provide a simpler way to trigger a GC, but I used
a script with "jmxsh", like this :
# gc_tomcat.jmxsh
# force the target JVM to do a GC, from the jmxsh shell
# invoke as :
# java -jar jmxsh.jar gc_tomcat.jmxsh
#
# In the following command, replace the port number (-p)
# by the port which has been specified in the parameter :
# -Dcom.sun.management.jmxremote.port=xxxxx
# of the target Java JVM startup line
jmx_connect -h 127.0.0.1 -p (port) -U (userid) -P (password)
jmx_invoke -m java.lang:type=Memory gc
jmx_close
for jmxsh, see :
- https://github.com/davr/jmxsh
- http://isbyr.com/use-jmxsh-jmx-cli-tool-to-troubleshoot-remote-jmx-connection/
>
> Warm regards,
>
> Norbert
>
> -----Original Message-----
> From: Paul Carter-Brown <pa...@jini.guru>
> Sent: Sunday, August 16, 2020 2:43 PM
> To: Tomcat Users List <us...@tomcat.apache.org>
> Subject: Re: Tomcat Handling close_waits
>
> Hi Norbert,
>
> The TCP socket states and timers are managed by the underlying OS and not by Tomcat. Can you paste a netstat -an result so I can see what you mean.
> Also, is the client using HTTP 1.1 with keep-alive or not? What kind of traffic is this?
>
> Paul
>
>
> On Sun, Aug 16, 2020 at 7:16 PM Norbert Elbanbuena <no...@teligentip.com>
> wrote:
>
>> I also noticed that while server receives the connection requests, we
>> are seeing multiple requests from the same sources. Some same source
>> requests
>> (FIN-WAIT) are all in state while other same sources requests are in
>> other state (some in FIN-WAIT or close_wait and some Established).
>>
>> Why are we seeing multiple requests from the same source at the same time?
>> Doesn't each socket request exhaust a thread on the application?
>>
>>
>> Warm regards,
>>
>> Norbert Elbanbuena
>>
>> -----Original Message-----
>> From: Norbert Elbanbuena <no...@teligentip.com>
>> Sent: Sunday, August 16, 2020 11:35 AM
>> To: Tomcat Users List <us...@tomcat.apache.org>
>> Subject: Tomcat Handling close_waits
>>
>> Hi,
>>
>> We are experiencing a flood of close_waits in our server.
>> Interestingly, all of the sessions stuck in close_waits are sourced
>> from Amazon IP addresses. Our second server running the same setup
>> (hardware spec, OS version, Tomcat version, etc.) and had almost the
>> same session count and our application and Tomcat didn't go unresponsive.
>>
>> Is there any tuneable parameter for the APR connector for Tomcat to
>> close the stuck sessions, rather than waiting for the application?
>>
>> Warm regards,
>>
>> Norbert Elbanbuena
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat Handling close_waits
Posted by Norbert Elbanbuena <no...@teligentip.com>.
Hi Paul,
Yes these are some samples, I have about 300 of them getting stuck hourly
tcp 761 0 192.168.1.50:58870 74.112.28.109:8011 CLOSE_WAIT
tcp 0 0 192.168.1.50:56938 192.168.1.50:61616 CLOSE_WAIT
tcp 0 0 192.168.1.50:56924 192.168.1.50:61616 CLOSE_WAIT
tcp 0 0 192.168.1.50:56910 192.168.1.50:61616 CLOSE_WAIT
tcp 0 0 192.168.1.50:56912 192.168.1.50:61616 CLOSE_WAIT
tcp6 0 6240 192.168.1.50:443 52.11.72.45:47123 CLOSE_WAIT
tcp6 32 0 192.168.1.50:443 34.209.104.242:13402 CLOSE_WAIT
tcp6 268 0 192.168.1.50:443 108.162.244.28:40864 CLOSE_WAIT
tcp6 32 0 192.168.1.50:443 35.167.185.49:10019 CLOSE_WAIT
tcp6 32 0 192.168.1.50:443 52.24.48.141:60660 CLOSE_WAIT
tcp6 202 0 192.168.1.50:443 199.189.191.86:51716 CLOSE_WAIT
tcp6 202 0 192.168.1.50:443 199.189.191.86:1386 CLOSE_WAIT
Warm regards,
Norbert
-----Original Message-----
From: Paul Carter-Brown <pa...@jini.guru>
Sent: Sunday, August 16, 2020 2:43 PM
To: Tomcat Users List <us...@tomcat.apache.org>
Subject: Re: Tomcat Handling close_waits
Hi Norbert,
The TCP socket states and timers are managed by the underlying OS and not by Tomcat. Can you paste a netstat -an result so I can see what you mean.
Also, is the client using HTTP 1.1 with keep-alive or not? What kind of traffic is this?
Paul
On Sun, Aug 16, 2020 at 7:16 PM Norbert Elbanbuena <no...@teligentip.com>
wrote:
> I also noticed that while server receives the connection requests, we
> are seeing multiple requests from the same sources. Some same source
> requests
> (FIN-WAIT) are all in state while other same sources requests are in
> other state (some in FIN-WAIT or close_wait and some Established).
>
> Why are we seeing multiple requests from the same source at the same time?
> Doesn't each socket request exhaust a thread on the application?
>
>
> Warm regards,
>
> Norbert Elbanbuena
>
> -----Original Message-----
> From: Norbert Elbanbuena <no...@teligentip.com>
> Sent: Sunday, August 16, 2020 11:35 AM
> To: Tomcat Users List <us...@tomcat.apache.org>
> Subject: Tomcat Handling close_waits
>
> Hi,
>
> We are experiencing a flood of close_waits in our server.
> Interestingly, all of the sessions stuck in close_waits are sourced
> from Amazon IP addresses. Our second server running the same setup
> (hardware spec, OS version, Tomcat version, etc.) and had almost the
> same session count and our application and Tomcat didn't go unresponsive.
>
> Is there any tuneable parameter for the APR connector for Tomcat to
> close the stuck sessions, rather than waiting for the application?
>
> Warm regards,
>
> Norbert Elbanbuena
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat Handling close_waits
Posted by Paul Carter-Brown <pa...@jini.guru>.
Hi Norbert,
The TCP socket states and timers are managed by the underlying OS and not
by Tomcat. Can you paste a netstat -an result so I can see what you mean.
Also, is the client using HTTP 1.1 with keep-alive or not? What kind of
traffic is this?
Paul
On Sun, Aug 16, 2020 at 7:16 PM Norbert Elbanbuena <no...@teligentip.com>
wrote:
> I also noticed that while server receives the connection requests, we are
> seeing multiple requests from the same sources. Some same source requests
> (FIN-WAIT) are all in state while other same sources requests are in other
> state (some in FIN-WAIT or close_wait and some Established).
>
> Why are we seeing multiple requests from the same source at the same time?
> Doesn't each socket request exhaust a thread on the application?
>
>
> Warm regards,
>
> Norbert Elbanbuena
>
> -----Original Message-----
> From: Norbert Elbanbuena <no...@teligentip.com>
> Sent: Sunday, August 16, 2020 11:35 AM
> To: Tomcat Users List <us...@tomcat.apache.org>
> Subject: Tomcat Handling close_waits
>
> Hi,
>
> We are experiencing a flood of close_waits in our server. Interestingly,
> all of the sessions stuck in close_waits are sourced from Amazon IP
> addresses. Our second server running the same setup (hardware spec, OS
> version, Tomcat version, etc.) and had almost the same session count and
> our application and Tomcat didn't go unresponsive.
>
> Is there any tuneable parameter for the APR connector for Tomcat to close
> the stuck sessions, rather than waiting for the application?
>
> Warm regards,
>
> Norbert Elbanbuena
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
RE: Tomcat Handling close_waits
Posted by Norbert Elbanbuena <no...@teligentip.com>.
I also noticed that while server receives the connection requests, we are seeing multiple requests from the same sources. Some same source requests (FIN-WAIT) are all in state while other same sources requests are in other state (some in FIN-WAIT or close_wait and some Established).
Why are we seeing multiple requests from the same source at the same time? Doesn't each socket request exhaust a thread on the application?
Warm regards,
Norbert Elbanbuena
-----Original Message-----
From: Norbert Elbanbuena <no...@teligentip.com>
Sent: Sunday, August 16, 2020 11:35 AM
To: Tomcat Users List <us...@tomcat.apache.org>
Subject: Tomcat Handling close_waits
Hi,
We are experiencing a flood of close_waits in our server. Interestingly, all of the sessions stuck in close_waits are sourced from Amazon IP addresses. Our second server running the same setup (hardware spec, OS version, Tomcat version, etc.) and had almost the same session count and our application and Tomcat didn't go unresponsive.
Is there any tuneable parameter for the APR connector for Tomcat to close the stuck sessions, rather than waiting for the application?
Warm regards,
Norbert Elbanbuena
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org