too long dns lookup time

[Li Li <fa...@gmail.com>]

I am using http client 4.3.1 to crawl webpages. but I found most of
the time is used by dns look up
I use jstack many times and found all of the threads are "lookupAllHostAddr"

"Thread-9" prio=10 tid=0x00007f5174314800 nid=0xf50 runnable
[0x00007f5169ed8000]
   java.lang.Thread.State: RUNNABLE
        at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
        at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:901)
        at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1293)
        at java.net.InetAddress.getAllByName0(InetAddress.java:1246)
        at java.net.InetAddress.getAllByName(InetAddress.java:1162)
        at java.net.InetAddress.getAllByName(InetAddress.java:1098)
        at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:44)
        at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:102)
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:314)
        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:357)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:218)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:194)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:85)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:214)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:160)


my codes:
PoolingHttpClientConnectionManager connManager = new
PoolingHttpClientConnectionManager();
connManager.setMaxTotal(this.getMaxTotalConnection());
connManager.setDefaultMaxPerRoute(this.getMaxConnectionPerRoute());
defaultRequestConfig = RequestConfig.custom()
           .setCookieSpec(CookieSpecs.BEST_MATCH)
           .setExpectContinueEnabled(true)
           //.setStaleConnectionCheckEnabled(true)
           .setConnectTimeout(this.getConnectTimeout())
           .setConnectionRequestTimeout(this.getConnectionRequestTimeout())
           .setSocketTimeout(this.getReadTimeout())
           .build();
HttpClientBuilder builder=HttpClients.custom();

client = builder.setConnectionManager(connManager)
.setProxy(getProxy())
.setRetryHandler(new MyHttpRequestRetryHander(retryCount))
.setDefaultRequestConfig(defaultRequestConfig)
.setUserAgent(this.getUserAgent())
.build();

10 threads share this client in a process

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org