You are viewing a plain text version of this content. The canonical link for it is here.
Posted to httpclient-users@hc.apache.org by Jaya Christina B <bj...@TechMahindra.com> on 2006/12/01 08:19:09 UTC
RE: Many sockets open in CLOSE_WAIT state
I got this problem when I was creating one HttpClient object for every
request.
The problem was resolved when I created one singleton HttpClient for all
my requests with a MultiThreadedConnnectionManager
HTH
Cheers,
Jaya.
-----Original Message-----
From: Mark Claassen [mailto:mclaassen@ocie.net]
Sent: Tuesday, November 28, 2006 10:41 PM
To: 'HttpClient User Discussion'; davidcotter@gmail.com
Subject: RE: Many sockets open in CLOSE_WAIT state
I had this error as well, and for me it came from me using the API
incorrectly. My lesson as a newbie was: follow the examples as closely
as possible. I found the api very easy to use incorrectly. However,
when I changed my usage to follow the examples more closely, all my
problems magically disappeared.
-----Original Message-----
From: David Cotter [mailto:davidcotter@gmail.com]
Sent: Tuesday, November 28, 2006 11:29 AM
To: httpclient-user@jakarta.apache.org
Subject: Re: Many sockets open in CLOSE_WAIT state
On closer inspection there are even more sockets open: For example I
have 20,000 open sockets: 12000 are in ESTABLISHED and 8000 are in
CLOSE_WAIT.
Regards,
David
On Mon, 2006-11-27 at 12:26 +0000, David Cotter wrote:
> > Hi there,
> >
> > I am having a problem with too many open files on my server. I am
> > using HttpClient v.3.0 and the server application is a proxy web app
> > that retrieves the content using HttpClient and passes it back to
> > the user
> using
> > a Tomcat servlet. I am serving 1000s of different users with
> > several
> sever
> > requests per second. As time goes on the JVM memory usage steadily
> increases
> > as the number of open files/sockets in CLOSE_WAIT state on the Linux
> machine
> > increases. I can see from inspection that many of the open files are
> socket
> > connection to third party sites opened with HttpClient.
> >
> > For every method executed I use a shared
> MultiThreadedHttpConnectionManager
> > and always call method.releaseConnection().
> >
> > I have instantiated IdleConnectionTimeoutThread and also have
> > another thread that occasionally calls:
> > multiThreadedHttpConnectionManager.closeIdleConnections
> (idleTimeoutMillis);
> > multiThreadedHttpConnectionManager.deleteClosedConnections();
> >
> > I have overridden
> > MultiThreadedHttpConnectionManager.releaseConnection()
> > and included a connection.close() in this function but have not seen
> > any results - always steadily increasing open files.
> >
> > I have seen reference to overriding
> > MultiThreadedHttpConnectionManager.releaseConnection () to force
> > sockets
> to
> > close on this list but nothing concrete as to what to put in the
> overrided
> > function.
> >
> > I would appreciate any help on this.
> >
>
> > Regards,
> > David
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
============================================================================================================================
Tech Mahindra, formerly Mahindra-British Telecom.
Disclaimer:
This message and the information contained herein is proprietary and confidential and subject to the Tech Mahindra policy statement, you may review at <a href="http://www.techmahindra.com/Disclaimer.html">http://www.techmahindra.com/Disclaimer.html</a> externally and <a href="http://tim.techmahindra.com/Disclaimer.html">http://tim.techmahindra.com/Disclaimer.html</a> internally within Tech Mahindra.
============================================================================================================================
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
Re: Many sockets open in CLOSE_WAIT state
Posted by David Cotter <da...@gmail.com>.
Cool thanks - I will check out these other methods.
David.
On 12/4/06, Roland Weber <RO...@de.ibm.com> wrote:
>
> Hello David,
>
> > In this case I would have to synchrozize this block of code
> >
> > synchronized(foo) {
> > httpClient.setState(getStateForThisUser());
> > HttpMethod getMethod = new HttpMethod();
> > client.executeMethod(getMethod);
> > saveStateForThisUser(httpClient.getState())
> > }
>
> No! I suggested to pass a user-specific state object *instead*
> of using the default state. Your code would replace the default
> state instead of not using it.
>
> > It seems to me I will have to synchronize until the method is complete
> and
> > the state in HttpClient updated which is quite a bottleneck. Or am I
> missing
> > something?
>
> Please have a look at the JavaDocs of the HttpClient class.
> There are several execute methods, and almost all of them
> accept an HttpState object as an argument (iirc).
>
> hope that helps,
> Roland
>
>
>
>
--
-----------------------------
David Cotter CTO
Alatto Technologies Ltd
-----------------------------
david.cotter@alatto.com
m. +353 87 6293698
t. +353 1 2090700 (main)
t. +353 1 2090779 (direct)
f. +353 1 2090707
www.alatto.com
-----------------------------
The new way to mobile browse: http://tribes.cc/demo
Re: Many sockets open in CLOSE_WAIT state
Posted by Roland Weber <RO...@de.ibm.com>.
Hello David,
> In this case I would have to synchrozize this block of code
>
> synchronized(foo) {
> httpClient.setState(getStateForThisUser());
> HttpMethod getMethod = new HttpMethod();
> client.executeMethod(getMethod);
> saveStateForThisUser(httpClient.getState())
> }
No! I suggested to pass a user-specific state object *instead*
of using the default state. Your code would replace the default
state instead of not using it.
> It seems to me I will have to synchronize until the method is complete
and
> the state in HttpClient updated which is quite a bottleneck. Or am I
missing
> something?
Please have a look at the JavaDocs of the HttpClient class.
There are several execute methods, and almost all of them
accept an HttpState object as an argument (iirc).
hope that helps,
Roland
Re: Many sockets open in CLOSE_WAIT state
Posted by David Cotter <da...@gmail.com>.
Thanks Roland
>
> Using different HttpClient objects for each user is a waste of resources.
> Cookies are not stored in the HttpClient object, but in an HttpState
> object. Keep different HttpState objects for each user and pass them
> to a single HttpClient object when executing methods, instead of using
> the default state of different HttpClient objects.
>
In this case I would have to synchrozize this block of code
synchronized(foo) {
httpClient.setState(getStateForThisUser());
HttpMethod getMethod = new HttpMethod();
client.executeMethod(getMethod);
saveStateForThisUser(httpClient.getState())
}
It seems to me I will have to synchronize until the method is complete and
the state in HttpClient updated which is quite a bottleneck. Or am I missing
something?
Many thanks,
David
Re: Many sockets open in CLOSE_WAIT state
Posted by Roland Weber <RO...@de.ibm.com>.
Hello David,
> I am using a HttpClient object for every user coming to the system.
These
> are used to store state like cookies etc. Is there a reason why having
many
> HttpClient objects in memory would keep many sockets open.
by default, each HttpClient object will have it's own connection pool.
If you make sure that all your HttpClient objects share a single
MultiThreadedHttpConnectionManager, there shouldn't be a problem.
Using different HttpClient objects for each user is a waste of resources.
Cookies are not stored in the HttpClient object, but in an HttpState
object. Keep different HttpState objects for each user and pass them
to a single HttpClient object when executing methods, instead of using
the default state of different HttpClient objects.
hope that helps,
Roland
Re: Many sockets open in CLOSE_WAIT state
Posted by David Cotter <da...@gmail.com>.
I am using a HttpClient object for every user coming to the system. These
are used to store state like cookies etc. Is there a reason why having many
HttpClient objects in memory would keep many sockets open.
Thanks,
David
On 12/1/06, Jaya Christina B <bj...@techmahindra.com> wrote:
>
>
> I got this problem when I was creating one HttpClient object for every
> request.
> The problem was resolved when I created one singleton HttpClient for all
> my requests with a MultiThreadedConnnectionManager
>
> HTH
>
> Cheers,
> Jaya.
>
>
> -----Original Message-----
> From: Mark Claassen [mailto:mclaassen@ocie.net]
> Sent: Tuesday, November 28, 2006 10:41 PM
> To: 'HttpClient User Discussion'; davidcotter@gmail.com
> Subject: RE: Many sockets open in CLOSE_WAIT state
>
> I had this error as well, and for me it came from me using the API
> incorrectly. My lesson as a newbie was: follow the examples as closely
> as possible. I found the api very easy to use incorrectly. However,
> when I changed my usage to follow the examples more closely, all my
> problems magically disappeared.
>
> -----Original Message-----
> From: David Cotter [mailto:davidcotter@gmail.com]
> Sent: Tuesday, November 28, 2006 11:29 AM
> To: httpclient-user@jakarta.apache.org
> Subject: Re: Many sockets open in CLOSE_WAIT state
>
> On closer inspection there are even more sockets open: For example I
> have 20,000 open sockets: 12000 are in ESTABLISHED and 8000 are in
> CLOSE_WAIT.
>
> Regards,
> David
>
>
> On Mon, 2006-11-27 at 12:26 +0000, David Cotter wrote:
> > > Hi there,
> > >
> > > I am having a problem with too many open files on my server. I am
> > > using HttpClient v.3.0 and the server application is a proxy web app
>
> > > that retrieves the content using HttpClient and passes it back to
> > > the user
> > using
> > > a Tomcat servlet. I am serving 1000s of different users with
> > > several
> > sever
> > > requests per second. As time goes on the JVM memory usage steadily
> > increases
> > > as the number of open files/sockets in CLOSE_WAIT state on the Linux
> > machine
> > > increases. I can see from inspection that many of the open files are
> > socket
> > > connection to third party sites opened with HttpClient.
> > >
> > > For every method executed I use a shared
> > MultiThreadedHttpConnectionManager
> > > and always call method.releaseConnection().
> > >
> > > I have instantiated IdleConnectionTimeoutThread and also have
> > > another thread that occasionally calls:
> > > multiThreadedHttpConnectionManager.closeIdleConnections
> > (idleTimeoutMillis);
> > > multiThreadedHttpConnectionManager.deleteClosedConnections();
> > >
> > > I have overridden
> > > MultiThreadedHttpConnectionManager.releaseConnection()
> > > and included a connection.close() in this function but have not seen
>
> > > any results - always steadily increasing open files.
> > >
> > > I have seen reference to overriding
> > > MultiThreadedHttpConnectionManager.releaseConnection () to force
> > > sockets
> > to
> > > close on this list but nothing concrete as to what to put in the
> > overrided
> > > function.
> > >
> > > I would appreciate any help on this.
> > >
> >
> > > Regards,
> > > David
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
>
>
>
> ============================================================================================================================
>
> Tech Mahindra, formerly Mahindra-British Telecom.
>
> Disclaimer:
>
> This message and the information contained herein is proprietary and
> confidential and subject to the Tech Mahindra policy statement, you may
> review at <a href="http://www.techmahindra.com/Disclaimer.html">
> http://www.techmahindra.com/Disclaimer.html</a> externally and <a href="
> http://tim.techmahindra.com/Disclaimer.html">
> http://tim.techmahindra.com/Disclaimer.html</a> internally within Tech
> Mahindra.
>
>
> ============================================================================================================================
>