You are viewing a plain text version of this content. The canonical link for it is here.
Posted to httpclient-users@hc.apache.org by Prasad Sethumadhavan <PS...@sandcherry.com> on 2005/02/02 05:14:51 UTC
HttpClient under load
Hi,
I need some help or pointers regarding using the HttpClient under heavy
load. I am currently using the HttpClient under moderate load (50
concurrent threads). Each thread instatiates a new HttpClient client,
uses it and does the method.releaseConnection() before exiting.
SimpleHttpConnectionManager manager = new
SimpleHttpConnectionManager();
manager.getParams().setStaleCheckingEnabled(true);
manager.closeIdleConnections(1000L);
manager.getParams().setConnectionTimeout(
(int) DEFAULT_CONNECTION_TIMEOUT);
httpclient = new HttpClient(manager);
httpclient.getParams().setCookiePolicy(
CookiePolicy.BROWSER_COMPATIBILITY);
However when I run this setup under load, I keep getting socket
exceptions on the client side. I am using tomcat web server and I have
increased the max-processor & access-count. I don't see any errors or
exception in the tomcat log.
However I keep getting a lot of these client socket bind exceptions
after sometime. I guess it is happening due to the unavailability of
ports. I would like to know some best practices in using the HttpClient
under load. I am sure tha it is my bad, but wanted to get some input
from the user/dev community. Should I have a pool of HttpClient objects
and reuse them? Should I use the multi-threaded connection manager? I
need to run this under heavier load and need to fix this soon. Any help,
code snippets, or guidance would be greatly appreciated. Please let me
know if you need any other information.
Thanks in advance
Prasad
Client StackTrace:
[ERROR] 01 Feb 2005 20:43:56,177 NDC[] T[Thread-20] HTTP HTTP
I/O transport exception occurred
java.net.BindException: Address already in use: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at
java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:364)
at java.net.Socket.connect(Socket.java:507)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at
org.apache.commons.httpclient.protocol.ReflectionSocketFactory.createSoc
ket(ReflectionSocketFactory.java:128)
at
org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.crea
teSocket(DefaultProtocolSocketFactory.java:124)
at
org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:70
5)
at
org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMe
thodDirector.java:359)
at
org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMetho
dDirector.java:168)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:3
93)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:3
24)
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
Re: HttpClient under load
Posted by Michael Becke <be...@u.washington.edu>.
Hi Prasad,
Given the scenario you describe below you are more likely leaking one
connection/socket per request. You definitely want to reuse instances
of HttpClient and especially instances of a ConnectionManager. You
can either use the multi-threaded connection manager from all threads
or have a dedicated SimpleHttpConnectionManager per thread. Please
have a look at the performance guide for some more ideas
<http://jakarta.apache.org/commons/httpclient/3.0/performance.html>.
Also please note that calling manager.closeIdleConnections(1000L) only
closes connections that have already been idle for 1000ms. It does not
continuously poll for idle connections, closing them when they've been
idle for 1000ms.
Mike
On Feb 1, 2005, at 11:14 PM, Prasad Sethumadhavan wrote:
> Hi,
> I need some help or pointers regarding using the HttpClient under heavy
> load. I am currently using the HttpClient under moderate load (50
> concurrent threads). Each thread instatiates a new HttpClient client,
> uses it and does the method.releaseConnection() before exiting.
>
> SimpleHttpConnectionManager manager = new
> SimpleHttpConnectionManager();
> manager.getParams().setStaleCheckingEnabled(true);
> manager.closeIdleConnections(1000L);
> manager.getParams().setConnectionTimeout(
> (int) DEFAULT_CONNECTION_TIMEOUT);
>
> httpclient = new HttpClient(manager);
> httpclient.getParams().setCookiePolicy(
> CookiePolicy.BROWSER_COMPATIBILITY);
>
> However when I run this setup under load, I keep getting socket
> exceptions on the client side. I am using tomcat web server and I have
> increased the max-processor & access-count. I don't see any errors or
> exception in the tomcat log.
>
> However I keep getting a lot of these client socket bind exceptions
> after sometime. I guess it is happening due to the unavailability of
> ports. I would like to know some best practices in using the HttpClient
> under load. I am sure tha it is my bad, but wanted to get some input
> from the user/dev community. Should I have a pool of HttpClient objects
> and reuse them? Should I use the multi-threaded connection manager? I
> need to run this under heavier load and need to fix this soon. Any
> help,
> code snippets, or guidance would be greatly appreciated. Please let me
> know if you need any other information.
>
> Thanks in advance
> Prasad
>
>
> Client StackTrace:
>
> [ERROR] 01 Feb 2005 20:43:56,177 NDC[] T[Thread-20] HTTP HTTP
> I/O transport exception occurred
> java.net.BindException: Address already in use: connect
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
> at
> java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:364)
> at java.net.Socket.connect(Socket.java:507)
> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso
> r
> Impl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at
> org.apache.commons.httpclient.protocol.ReflectionSocketFactory.createSo
> c
> ket(ReflectionSocketFactory.java:128)
> at
> org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.cre
> a
> teSocket(DefaultProtocolSocketFactory.java:124)
> at
> org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:
> 70
> 5)
> at
> org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpM
> e
> thodDirector.java:359)
> at
> org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMeth
> o
> dDirector.java:168)
> at
> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:
> 3
> 93)
> at
> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:
> 3
> 24)
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail:
> httpclient-user-help@jakarta.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org