You are viewing a plain text version of this content. The canonical link for it is here.
Posted to httpclient-users@hc.apache.org by David Cotter <da...@gmail.com> on 2006/11/27 13:26:45 UTC
Many sockets open in CLOSE_WAIT state
Hi there,
I am having a problem with too many open files on my server. I am using
HttpClient v.3.0 and the server application is a proxy web app that
retrieves the content using HttpClient and passes it back to the user using
a Tomcat servlet. I am serving 1000s of different users with several sever
requests per second. As time goes on the JVM memory usage steadily increases
as the number of open files/sockets in CLOSE_WAIT state on the Linux machine
increases. I can see from inspection that many of the open files are socket
connection to third party sites opened with HttpClient.
For every method executed I use a shared MultiThreadedHttpConnectionManager
and always call method.releaseConnection().
I have instantiated IdleConnectionTimeoutThread and also have another
thread that occasionally calls:
multiThreadedHttpConnectionManager.closeIdleConnections(idleTimeoutMillis);
multiThreadedHttpConnectionManager.deleteClosedConnections();
I have overridden MultiThreadedHttpConnectionManager.releaseConnection ()
and included a connection.close() in this function but have not seen any
results - always steadily increasing open files.
I have seen reference to overriding
MultiThreadedHttpConnectionManager.releaseConnection () to force sockets to
close on this list but nothing concrete as to what to put in the overrided
function.
I would appreciate any help on this.
Regards,
David
RE: Many sockets open in CLOSE_WAIT state
Posted by Mark Claassen <mc...@ocie.net>.
I had this error as well, and for me it came from me using the API
incorrectly. My lesson as a newbie was: follow the examples as closely as
possible. I found the api very easy to use incorrectly. However, when I
changed my usage to follow the examples more closely, all my problems
magically disappeared.
-----Original Message-----
From: David Cotter [mailto:davidcotter@gmail.com]
Sent: Tuesday, November 28, 2006 11:29 AM
To: httpclient-user@jakarta.apache.org
Subject: Re: Many sockets open in CLOSE_WAIT state
On closer inspection there are even more sockets open: For example I have
20,000 open sockets: 12000 are in ESTABLISHED and 8000 are in CLOSE_WAIT.
Regards,
David
On Mon, 2006-11-27 at 12:26 +0000, David Cotter wrote:
> > Hi there,
> >
> > I am having a problem with too many open files on my server. I am
> > using HttpClient v.3.0 and the server application is a proxy web app
> > that retrieves the content using HttpClient and passes it back to
> > the user
> using
> > a Tomcat servlet. I am serving 1000s of different users with
> > several
> sever
> > requests per second. As time goes on the JVM memory usage steadily
> increases
> > as the number of open files/sockets in CLOSE_WAIT state on the Linux
> machine
> > increases. I can see from inspection that many of the open files are
> socket
> > connection to third party sites opened with HttpClient.
> >
> > For every method executed I use a shared
> MultiThreadedHttpConnectionManager
> > and always call method.releaseConnection().
> >
> > I have instantiated IdleConnectionTimeoutThread and also have
> > another thread that occasionally calls:
> > multiThreadedHttpConnectionManager.closeIdleConnections
> (idleTimeoutMillis);
> > multiThreadedHttpConnectionManager.deleteClosedConnections();
> >
> > I have overridden
> > MultiThreadedHttpConnectionManager.releaseConnection()
> > and included a connection.close() in this function but have not seen
> > any results - always steadily increasing open files.
> >
> > I have seen reference to overriding
> > MultiThreadedHttpConnectionManager.releaseConnection () to force
> > sockets
> to
> > close on this list but nothing concrete as to what to put in the
> overrided
> > function.
> >
> > I would appreciate any help on this.
> >
>
> > Regards,
> > David
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
Re: Many sockets open in CLOSE_WAIT state
Posted by David Cotter <da...@gmail.com>.
On closer inspection there are even more sockets open: For example I have
20,000 open sockets: 12000 are in ESTABLISHED and 8000 are in CLOSE_WAIT.
Regards,
David
On Mon, 2006-11-27 at 12:26 +0000, David Cotter wrote:
> > Hi there,
> >
> > I am having a problem with too many open files on my server. I am using
> > HttpClient v.3.0 and the server application is a proxy web app that
> > retrieves the content using HttpClient and passes it back to the user
> using
> > a Tomcat servlet. I am serving 1000s of different users with several
> sever
> > requests per second. As time goes on the JVM memory usage steadily
> increases
> > as the number of open files/sockets in CLOSE_WAIT state on the Linux
> machine
> > increases. I can see from inspection that many of the open files are
> socket
> > connection to third party sites opened with HttpClient.
> >
> > For every method executed I use a shared
> MultiThreadedHttpConnectionManager
> > and always call method.releaseConnection().
> >
> > I have instantiated IdleConnectionTimeoutThread and also have another
> > thread that occasionally calls:
> > multiThreadedHttpConnectionManager.closeIdleConnections
> (idleTimeoutMillis);
> > multiThreadedHttpConnectionManager.deleteClosedConnections();
> >
> > I have overridden MultiThreadedHttpConnectionManager.releaseConnection()
> > and included a connection.close() in this function but have not seen any
> > results - always steadily increasing open files.
> >
> > I have seen reference to overriding
> > MultiThreadedHttpConnectionManager.releaseConnection () to force sockets
> to
> > close on this list but nothing concrete as to what to put in the
> overrided
> > function.
> >
> > I would appreciate any help on this.
> >
>
> > Regards,
> > David
>
>
Re: Many sockets open in CLOSE_WAIT state
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Mon, 2006-11-27 at 12:26 +0000, David Cotter wrote:
> Hi there,
>
> I am having a problem with too many open files on my server. I am using
> HttpClient v.3.0 and the server application is a proxy web app that
> retrieves the content using HttpClient and passes it back to the user using
> a Tomcat servlet. I am serving 1000s of different users with several sever
> requests per second. As time goes on the JVM memory usage steadily increases
> as the number of open files/sockets in CLOSE_WAIT state on the Linux machine
> increases. I can see from inspection that many of the open files are socket
> connection to third party sites opened with HttpClient.
>
> For every method executed I use a shared MultiThreadedHttpConnectionManager
> and always call method.releaseConnection().
>
> I have instantiated IdleConnectionTimeoutThread and also have another
> thread that occasionally calls:
> multiThreadedHttpConnectionManager.closeIdleConnections(idleTimeoutMillis);
> multiThreadedHttpConnectionManager.deleteClosedConnections();
>
> I have overridden MultiThreadedHttpConnectionManager.releaseConnection ()
> and included a connection.close() in this function but have not seen any
> results - always steadily increasing open files.
>
> I have seen reference to overriding
> MultiThreadedHttpConnectionManager.releaseConnection () to force sockets to
> close on this list but nothing concrete as to what to put in the overrided
> function.
>
> I would appreciate any help on this.
>
David,
The only rational explanation I can think of is that sockets remain in
the CLOSE_WAIT state due to some OS level settings. Calling #close() on
all connections released back to the connection manager should
definitely leave no sockets in the CLOSE_WAIT state.
Oleg
> Regards,
> David
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org