You are viewing a plain text version of this content. The canonical link for it is here.
Posted to httpclient-users@hc.apache.org by "Chen, Drew T" <DT...@DIRECTV.com> on 2012/01/24 16:03:37 UTC
HttpAsyncClient max connections
I'm using DefaultHttpAsyncClient to connect to approximately 100 remote hosts.
What I'm seeing through Wireshark is that only 19 or 20 requests are made immediately. As those 20 finish, then the rest of the requests in the queue are processed gradually.
I want DefaultHttyAsyncClient to be able to initiate all 100 immediately because these requests remain idle for a very long time, like 20 seconds each)
Looking into the source, I can see PoolingAsyncClientConnectionManager uses HttpNIOConnPool which the underlying base class shows a max of 20. The pool.setMaxTotal() I would expect to fix that does not seem to have any effect when I run it.
Here is a snippet of the code:
DefaultConnectingIOReactor defaultioreactor = new DefaultConnectingIOReactor();
PoolingAsyncClientConnectionManager pool = new PoolingAsyncClientConnectionManager(defaultioreactor);
pool.setMaxTotal(200); // This I would expect to enable the DefaultHttpAsyncClient to immediately connect up to 200 socket channels.
DefaultHttpAsyncClient defaultHttpClient = new DefaultHttpAsyncClient(pool);
httpClient = defaultHttpClient;
httpClient.getParams()
.setIntParameter(CoreConnectionPNames.SO_TIMEOUT, 60000)
.setIntParameter(CoreConnectionPNames.CONNECTION_TIMEOUT, 60000)
.setBooleanParameter(CoreProtocolPNames.USE_EXPECT_CONTINUE, true)
.setIntParameter(CoreProtocolPNames.WAIT_FOR_CONTINUE, 500);
defaultHttpClient.getCredentialsProvider().setCredentials(
AuthScope.ANY,
new UsernamePasswordCredentials("user", "password"));
httpClient.start();
targets = ...; //targets is a Map<String, HttpHost>
CountDownLatch latch = new CountDownLatch(targets.size()); // size will be approx. 100.
for (Map.Entry<String,HttpHost> SBB : targets.entrySet()) {
String name = SBB.getKey();
HttpHost target = SBB.getValue();
HttpRequestBase targetRequest = // function to create HttpGet();
SHEFFutureCallback futresponse = // new future callback object.
httpClient.execute(target, targetRequest, futresponse);
}
And for context, I'm running it on a Windows XP Intel Core2 Duo machine with a Gigabit NIC.
----------------------------------------
Drew Chen
22516 Gateway Center Dr.
Suite 100
Clarksburg, MD 20871
dtchen@directv.com<ma...@directv.com>
Office: 301-444-2993
Fax: 301-444-2998
RE: HttpAsyncClient max connections
Posted by "Chen, Drew T" <DT...@DIRECTV.com>.
After attempting to create a log output of the problem, the problem seems to have gone away. I did attempt resizing the Max Total with various numbers from 100-2000. But without any other changes, the AsyncClient seems to initiate all 100 connections immediately now.
-Drew
-----Original Message-----
From: Oleg Kalnichevski [mailto:olegk@apache.org]
Sent: Tuesday, January 24, 2012 10:35 AM
To: HttpClient User Discussion
Subject: Re: HttpAsyncClient max connections
On Tue, 2012-01-24 at 08:03 -0700, Chen, Drew T wrote:
> I'm using DefaultHttpAsyncClient to connect to approximately 100 remote hosts.
> What I'm seeing through Wireshark is that only 19 or 20 requests are made immediately. As those 20 finish, then the rest of the requests in the queue are processed gradually.
>
> I want DefaultHttyAsyncClient to be able to initiate all 100 immediately because these requests remain idle for a very long time, like 20 seconds each)
>
> Looking into the source, I can see PoolingAsyncClientConnectionManager uses HttpNIOConnPool which the underlying base class shows a max of 20. The pool.setMaxTotal() I would expect to fix that does not seem to have any effect when I run it.
>
> Here is a snippet of the code:
>
> DefaultConnectingIOReactor defaultioreactor = new DefaultConnectingIOReactor();
> PoolingAsyncClientConnectionManager pool = new PoolingAsyncClientConnectionManager(defaultioreactor);
> pool.setMaxTotal(200); // This I would expect to enable the DefaultHttpAsyncClient to immediately connect up to 200 socket channels.
>
> DefaultHttpAsyncClient defaultHttpClient = new DefaultHttpAsyncClient(pool);
> httpClient = defaultHttpClient;
> httpClient.getParams()
> .setIntParameter(CoreConnectionPNames.SO_TIMEOUT, 60000)
> .setIntParameter(CoreConnectionPNames.CONNECTION_TIMEOUT, 60000)
> .setBooleanParameter(CoreProtocolPNames.USE_EXPECT_CONTINUE, true)
> .setIntParameter(CoreProtocolPNames.WAIT_FOR_CONTINUE, 500);
>
> defaultHttpClient.getCredentialsProvider().setCredentials(
> AuthScope.ANY,
> new UsernamePasswordCredentials("user", "password"));
>
> httpClient.start();
>
> targets = ...; //targets is a Map<String, HttpHost>
>
> CountDownLatch latch = new CountDownLatch(targets.size()); // size will be approx. 100.
>
> for (Map.Entry<String,HttpHost> SBB : targets.entrySet()) {
> String name = SBB.getKey();
> HttpHost target = SBB.getValue();
>
> HttpRequestBase targetRequest = // function to create HttpGet();
> SHEFFutureCallback futresponse = // new future callback object.
>
> httpClient.execute(target, targetRequest, futresponse);
> }
>
> And for context, I'm running it on a Windows XP Intel Core2 Duo machine with a Gigabit NIC.
Please produce connection management context log as described here [1]
and post it to this list
Oleg
[1] http://hc.apache.org/httpcomponents-client-ga/logging.html
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: HttpAsyncClient max connections
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Tue, 2012-01-24 at 08:03 -0700, Chen, Drew T wrote:
> I'm using DefaultHttpAsyncClient to connect to approximately 100 remote hosts.
> What I'm seeing through Wireshark is that only 19 or 20 requests are made immediately. As those 20 finish, then the rest of the requests in the queue are processed gradually.
>
> I want DefaultHttyAsyncClient to be able to initiate all 100 immediately because these requests remain idle for a very long time, like 20 seconds each)
>
> Looking into the source, I can see PoolingAsyncClientConnectionManager uses HttpNIOConnPool which the underlying base class shows a max of 20. The pool.setMaxTotal() I would expect to fix that does not seem to have any effect when I run it.
>
> Here is a snippet of the code:
>
> DefaultConnectingIOReactor defaultioreactor = new DefaultConnectingIOReactor();
> PoolingAsyncClientConnectionManager pool = new PoolingAsyncClientConnectionManager(defaultioreactor);
> pool.setMaxTotal(200); // This I would expect to enable the DefaultHttpAsyncClient to immediately connect up to 200 socket channels.
>
> DefaultHttpAsyncClient defaultHttpClient = new DefaultHttpAsyncClient(pool);
> httpClient = defaultHttpClient;
> httpClient.getParams()
> .setIntParameter(CoreConnectionPNames.SO_TIMEOUT, 60000)
> .setIntParameter(CoreConnectionPNames.CONNECTION_TIMEOUT, 60000)
> .setBooleanParameter(CoreProtocolPNames.USE_EXPECT_CONTINUE, true)
> .setIntParameter(CoreProtocolPNames.WAIT_FOR_CONTINUE, 500);
>
> defaultHttpClient.getCredentialsProvider().setCredentials(
> AuthScope.ANY,
> new UsernamePasswordCredentials("user", "password"));
>
> httpClient.start();
>
> targets = ...; //targets is a Map<String, HttpHost>
>
> CountDownLatch latch = new CountDownLatch(targets.size()); // size will be approx. 100.
>
> for (Map.Entry<String,HttpHost> SBB : targets.entrySet()) {
> String name = SBB.getKey();
> HttpHost target = SBB.getValue();
>
> HttpRequestBase targetRequest = // function to create HttpGet();
> SHEFFutureCallback futresponse = // new future callback object.
>
> httpClient.execute(target, targetRequest, futresponse);
> }
>
> And for context, I'm running it on a Windows XP Intel Core2 Duo machine with a Gigabit NIC.
Please produce connection management context log as described here [1]
and post it to this list
Oleg
[1] http://hc.apache.org/httpcomponents-client-ga/logging.html
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org