Large numbers of sockets in CLOSE_WAIT on Linux

[Michael Ryan <Mi...@stellent.com>]

We are seeing large numbers of sockets lingering in the CLOSE_WAIT state
(waiting for the client to issue a close() call) while running our
client application on Linux.  The problem does not seem to occur on
Windows or Solaris, or at least is not nearly as obvious on Solaris.
When the process ends, the sockets are all closed, so it does not appear
to be an operating system bug.  It does appear that the sockets are
eventually closed, but that leads me to believe that they are being left
around for the garbage collector to deal with.
 
Does anyone have any insight into this?  I read several other peoples
messages regarding this, so I am not the only one experiencing this.  Is
there a way to force the Axis client to call close()?  Is this a JVM
issue?
 
Thanks for any help.
Mike Ryan

Re: Large numbers of sockets in CLOSE_WAIT on Linux

[Jason Cwik <bi...@gmail.com>]

I've seen sockets lingering in CLOSE_WAIT on Linux, but only when the
*server* is on Linux.  Your email indicates that your client is on linux?
Where is the server?  Also, what web container are you running your Axis
under?  Tomcat, Websphere?

I ran 30,000 test calls in 5 minutes against my Linux Axis server, and there
was a bunch of CLOSE_WAIT sockets, but they eventually disappeared, probably
when the finalizer ran.  It's possible that the web container or JVM doesn't
actually close() the native socket until it's finalized.

On 12/12/05, Michael Ryan <Mi...@stellent.com> wrote:
>
> We are seeing large numbers of sockets lingering in the CLOSE_WAIT state
> (waiting for the client to issue a close() call) while running our client
> application on Linux.  The problem does not seem to occur on Windows or
> Solaris, or at least is not nearly as obvious on Solaris.  When the process
> ends, the sockets are all closed, so it does not appear to be an operating
> system bug.  It does appear that the sockets are eventually closed, but that
> leads me to believe that they are being left around for the garbage
> collector to deal with.
>
> Does anyone have any insight into this?  I read several other peoples
> messages regarding this, so I am not the only one experiencing this.  Is
> there a way to force the Axis client to call close()?  Is this a JVM issue?
>
> Thanks for any help.
> Mike Ryan
>

Re: Large numbers of sockets in CLOSE_WAIT on Linux

[iksrazal <ik...@gmail.com>]

I've battled this problem but I've never seen it with axis. Anyways,  
CLOSE_WAIT state can be caused by at least two methods; The first being far 
more common and is what I have seen: 

a) The remote side has closed the
connection, but the local socket is still open.
b)  remote side has called shutdown(SHUT_WR). This would make the TCP
connection unidirectional, from the local to the remote side.

This has nothing specific to linux - I use linux exclusively. Java has some 
issues with sockets:

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4726957

The one thing you mention is that the problem fixes itself after a certain 
period of time. The way to check if that is really true is to make an axis 
call and watch on the client via  'netstat -an' and watch for close_wait 
going to time_wait - the normal sequence. If it does work as expected, 
somewhere on the client side axis performs a final read() to 
the filedescriptor of the socket port and clears CLOSE_WAIT. It'd help if you 
posted exactly how long it takes if you think its taking too long. 

If 'netstat -an' shows the unexpected you'll need to dig deeper. You don't 
mention which version of Axis you are using - I'm using axis2 .93 at the 
moment. Nevertheless, there are test cases in the above JDK bug report and 
it'd be worth checking the axis source to see if shutdownOutput() or  
selectNow() is properly called when setSoTimeout() is used. 

HTH,
iksrazal


Em Segunda 12 Dezembro 2005 14:49, o Michael Ryan escreveu:
> We are seeing large numbers of sockets lingering in the CLOSE_WAIT state
> (waiting for the client to issue a close() call) while running our
> client application on Linux.  The problem does not seem to occur on
> Windows or Solaris, or at least is not nearly as obvious on Solaris.
> When the process ends, the sockets are all closed, so it does not appear
> to be an operating system bug.  It does appear that the sockets are
> eventually closed, but that leads me to believe that they are being left
> around for the garbage collector to deal with.
>
> Does anyone have any insight into this?  I read several other peoples
> messages regarding this, so I am not the only one experiencing this.  Is
> there a way to force the Axis client to call close()?  Is this a JVM
> issue?
>
> Thanks for any help.
> Mike Ryan