Close_wait state

["Elias, Michael" <El...@dnb.com.INVALID>]

Hi -
We are running tomcat version 7.0.50.
Starting 2 days ago are application stopped responding to requests. Our investigation showed us that we are not closing connections. We see after 300 tcp sessions, for the tomcat PID, in CLOSE_WAIT state out app stops responding. Restarting the app clears the state.

We took tcpdumps between our web layer and our tomcat layer. What we see in a successful connection is, after the response, tomcat sends a FIN, web ACK's, then web sends a FIN and Tomcat ACK's.. connection closes


In a bad connection, tomcat does not send its FIN after the response, after 3 minutes, the WEB sends a FIN and tomcat ACK's. the connection goes into CLOSE_WAIT and stays in that state until restart of tomcat.


Any help would be greatly appreciated.

Thanks
Michael

Re: Close_wait state

["André Warnier (tomcat)" <aw...@ice-sa.com>]

On 18.02.2016 16:50, Elias, Michael wrote:
> Hi -
> We are running tomcat version 7.0.50.
> Starting 2 days ago are application stopped responding to requests. Our investigation showed us that we are not closing connections. We see after 300 tcp sessions, for the tomcat PID, in CLOSE_WAIT state out app stops responding. Restarting the app clears the state.
>
> We took tcpdumps between our web layer and our tomcat layer. What we see in a successful connection is, after the response, tomcat sends a FIN, web ACK's, then web sends a FIN and Tomcat ACK's.. connection closes
>
>
> In a bad connection, tomcat does not send its FIN after the response, after 3 minutes, the WEB sends a FIN and tomcat ACK's. the connection goes into CLOSE_WAIT and stays in that state until restart of tomcat.
>
>
> Any help would be greatly appreciated.
>

I have a question, and a story to this :

Question : what happens to your connections in CLOSE_WAIT, if you force Tomcat (or rather, 
its JVM) to do a GC (garbage collection) ?
(There are probably different ways to do that, but I know only one and it is lengthy to 
set up.  Maybe someone has a quick suggestion ?)

Story :
One case in the past in which I had a similar issue, was with a webapp which :
- created an object which itself created a TCP connection to some external process
- used that object (its methods) to access that connection
- and when the time came to close this connection, it just "forgot" the object, and left 
it to the JVM to cleanup when it destroyed the object

And the JVM ended up with hundreds of connections in the CLOSE_WAIT state, up to a point 
(under Linux) where the entire TCP stack became unresponsive.

My interpretation of what happened then is :

Because in Java the garbage collection is asynchronous with the rest and only happens when 
needed, this unreferenced object could stay on the heap for quite a while.
(As a matter of fact, the more comfortable the heap, the longer it stays).
And because the JVM, to create a socket, uses ultimately some native code and some 
underlying native socket structure, this underlying OS-level socket remained also, in its 
CLOSE_WAIT state, for a long time after the original java object and wrapped connection 
had long ceased to be used by the webapp.
A GC cleared that, because it finally eliminates and destroys unreferenced objects, and 
their linked native structures at the same time, which has the effect of finally closing 
the connection properly. So a GC magically deleted these hundreds of CLOSE_WAIT connections.

Maybe your case is similar ?

The proper solution of course is to make sure that the webapp properly closes the 
underlying connection before it drops the object that encapsulates it.

An unproper and temporary (but in the meantime working) solution for me - because we had 
no access to the bad code - was to write a script which ran regularly, and forced the 
Tomcat JVM to do a GC.



---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org