You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hc.apache.org by "Oleg Kalnichevski (JIRA)" <ji...@apache.org> on 2013/06/15 18:35:20 UTC
[jira] [Resolved] (HTTPCORE-343) ConcurrentModificationException
into ProcessPendingRequest when queries are reinjected into the
ResponseConsumer#failed event (after a Timeout failure of the original
query)
[ https://issues.apache.org/jira/browse/HTTPCORE-343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Oleg Kalnichevski resolved HTTPCORE-343.
----------------------------------------
Resolution: Fixed
Fix Version/s: 4.3-beta3
I changed AbstractNIOConnPool to fire request callbacks outside the pool lock which makes it possible to re-enter pool methods from a callback event. This should also resolve the original issue with concurrent (recursive) modification of the request queue.
This is a fairly major change made relatively late in the development process.
Frédéric, could you please test your application with the latest snapshot off SVN trunk?
Oleg
> ConcurrentModificationException into ProcessPendingRequest when queries are reinjected into the ResponseConsumer#failed event (after a Timeout failure of the original query)
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HTTPCORE-343
> URL: https://issues.apache.org/jira/browse/HTTPCORE-343
> Project: HttpComponents HttpCore
> Issue Type: Bug
> Components: HttpCore NIO
> Affects Versions: 4.2.3
> Environment: a lot of client thread (32 in my test), a lot of queries per second per thread (64 QPS in my test), some servers (8 in my test) and very big queries (servers respond between 10 to 25 seconds in my test)
> Reporter: Frédéric Passaniti
> Fix For: 4.3-beta3
>
>
> initial discussion :
> https://issues.apache.org/jira/browse/HTTPCORE-164?focusedCommentId=13671852&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13671852
> Here are the errors :
> SEVERE: shard[shard_v5] target[ng51.exa.com:9810] take[1] resp failed : java.util.concurrent.TimeoutException
> java.util.concurrent.TimeoutException
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exalead.elastic_ha.searng51.exa.com : 9810
> ch_client_federate.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.release(AbstractNIOConnPool.java:225)
> at org.apache.http.nio.pool.AbstractNIOConnPool.release(AbstractNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester$RequestExecutionCallback.completed(HttpAsyncRequester.java:324)
> at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:109)
> at org.apache.http.nio.protocol.BasicAsyncRequestExecutionHandler.responseCompleted(BasicAsyncRequestExecutionHandler.java:205)
> at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:368)
> at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:252)
> at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:190)
> at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:126)
> at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:1)
> at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:112)
> at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160)
> at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:348)
> at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:326)
> at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
> at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
> at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
> at java.lang.Thread.run(Unknown Source)
> java.util.ConcurrentModificationException
> May 31, 2013 5:55:25 PM com.exa.search_client.EHASearchClient getResults
> SEVERE: problem with query: [q=expects&b=0&hf=150000] affinity: [null] : java.util.ConcurrentModificationException
> java.util.ConcurrentModificationException
> at java.util.LinkedList$ListItr.checkForComodification(Unknown Source)
> at java.util.LinkedList$ListItr.next(Unknown Source)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:237)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.release(AbstractNIOConnPool.java:225)
> at org.apache.http.nio.pool.AbstractNIOConnPool.release(AbstractNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester$RequestExecutionCallback.completed(HttpAsyncRequester.java:324)
> at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:109)
> at org.apache.http.nio.protocol.BasicAsyncRequestExecutionHandler.responseCompleted(BasicAsyncRequestExecutionHandler.java:205)
> at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:368)
> at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:252)
> at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:190)
> at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:126)
> at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:1)
> at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:112)
> at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160)
> at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:348)
> at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:326)
> at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
> at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
> at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
> at java.lang.Thread.run(Unknown Source)
> 187781 [Thread-14] WARN search.qps 11 - Search failed
> May 31, 2013 5:55:25 PM com.exa.search_client.ShardQuery logError
> SEVERE: shard[shard_v5] target[ng51.exa.com:9210] take[0] Could not reinject failover query
> java.util.ConcurrentModificationException
> at java.util.LinkedList$ListItr.checkForComodification(Unknown Source)
> at java.util.LinkedList$ListItr.next(Unknown Source)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:237)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.release(AbstractNIOConnPool.java:225)
> at org.apache.http.nio.pool.AbstractNIOConnPool.release(AbstractNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester$RequestExecutionCallback.completed(HttpAsyncRequester.java:324)
> at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:109)
> at org.apache.http.nio.protocol.BasicAsyncRequestExecutionHandler.responseCompleted(BasicAsyncRequestExecutionHandler.java:205)
> at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:368)
> at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:252)
> at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:190)
> at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:126)
> at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:1)
> at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:112)
> at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160)
> at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:348)
> at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:326)
> at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
> at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
> at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
> at java.lang.Thread.run(Unknown Source)
> java.util.ConcurrentModificationException
> May 31, 2013 5:55:25 PM com.exa.search_client.ShardQuery logError
> SEVERE: shard[shard_v5] target[ng51.exa.com:9110] take[0] Could not reinject failover query
> java.util.ConcurrentModificationException
> at java.util.LinkedList$ListItr.checkForComodification(Unknown Source)
> at java.util.LinkedList$ListItr.next(Unknown Source)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:237)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:190)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:211)
> at org.apache.http.nio.protocol.HttpAsyncRequester.execute(HttpAsyncRequester.java:228)
> at com.exa.search_client.NioHttpClient.sendRequests(NioHttpClient.java:102)
> at com.exa.search_client.NioHttpResponseConsumer.failed(NioHttpResponseConsumer.java:95)
> at org.apache.http.nio.protocol.HttpAsyncRequester$ConnRequestCallback.failed(HttpAsyncRequester.java:269)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:122)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequests(AbstractNIOConnPool.java:251)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:193)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:111)
> at org.apache.http.impl.nio.pool.BasicNIOConnPool.lease(BasicNIOConnPool.java:1)
> at org.apache.http.nio.protocol.HttpAs ..........
> So :
> a concurrent exception is raised while the AbstractNIOConnectionPool is iterating over the leasingRequest AND at the same time (I think) a thread is trying to remove an element of this list.
> This occurs after a timeout exception.
> To reproduce it I used this setup :
> 32 clients thread, that try to execute a total of 64 QPS (query per second) to 8 servers (but I think this part is not a problem)
> and the MASTER KEY is to have very very long response size !!
> If my clients are able to read the answer within less than 1 or 2 seconds the issue doesn't occurs (or almost never).
> If I ask for a very big result (response time between 10 and 25 seconds) it take 1 or 2 minuts and then explode.
> The error occurs when it's spending some times in this call : conn.consumeInput(this.handler); (DefaultHttpClientIODispatch)
> Here is my code and setup :
> init :
> params_ = new SyncBasicHttpParams();
> params_
> .setIntParameter(CoreConnectionPNames.SO_TIMEOUT, http_params.so_timeout_)
> .setIntParameter(CoreConnectionPNames.CONNECTION_TIMEOUT, http_params.conn_timeout_)
> .setIntParameter(CoreConnectionPNames.SOCKET_BUFFER_SIZE, http_params.so_buffersize_)
> .setParameter(CoreProtocolPNames.USER_AGENT, "search-client");
> // <SocketTimeout>120000</SocketTimeout>
> // <ConnectionTimeout>80000</ConnectionTimeout>
> // <SocketBufferSize>4096</SocketBufferSize>
> final HttpAsyncRequestExecutor protocolHandler = new HttpAsyncRequestExecutor();
> final IOEventDispatch ioEventDispatch = new DefaultHttpClientIODispatch(protocolHandler, params_);
> ioReactor_ = new DefaultConnectingIOReactor();
> pool_ = new BasicNIOConnPool(ioReactor_, params_);
> pool_.setDefaultMaxPerRoute(2);
> pool_.setMaxTotal(4);
> // Run the I/O reactor in a separate thread
> reactor_daemon_ = new Thread(new Runnable(){
> public void run(){
> LOGGER.debug("IOReactor thread started");
> try {
> ioReactor_.execute(ioEventDispatch);
> } catch (final Exception ex) {
> LOGGER.error("error in the IOReactor thread: ", ex);
> }
> LOGGER.debug("IOReactor thread shutdown");
> }
> });
> reactor_daemon_.setDaemon(true);
> reactor_daemon_.start();
> httpproc_ = new ImmutableHttpProcessor(new HttpRequestInterceptor[] {
> new RequestContent(),
> new RequestTargetHost(),
> new RequestConnControl(),
> new RequestUserAgent(),
> new RequestExpectContinue()});
> requester_ = new HttpAsyncRequester(httpproc_, new DefaultConnectionReuseStrategy(), params_);
> query call :
> final CountDownLatch latch = new CountDownLatch(queries.length);
> for (final ShardQuery query : queries) {
> requester_.execute(
> new NioHttpRequestProducer(query),
> new NioHttpResponseConsumer(query, latch, this),
> pool_);
> }
> latch.await();
> Just a particularity (AND IT SEEMS TO BE THE PROBLEM) :
> into my NioHttpResponseConsumer, if my query failed (timeout), I invalidate the server, get the next available one, and try to re-execute the query (by calling again the HttpAsyncRequester.execute
> here is the code of the NioHttpResponseConsumer :
> @Override
> public void failed (final Exception ex)
> {
> if (! close_1()) {
> return;
> }
> query_.logError("resp failed : " + ex, ex);
> ex_ = ex;
> query_.flagCurrentTargetFailure();
> try {
> query_.nextOKTarget(); // get the next server of my list
> client_.sendRequests(latch_, query_); // call the HTTP requester
> } catch (final Throwable inner_th) {
> query_.error_ = inner_th;
> latch_.countDown();
> query_.logError("Could not reinject failover query", inner_th);
> }
> }
> I'm using this version :
> <modelVersion>4.0.0</modelVersion>
> <parent>
> <groupId>org.apache.httpcomponents</groupId>
> <artifactId>httpcomponents-core</artifactId>
> <version>4.2.3</version>
> </parent>
> <artifactId>httpcore-nio</artifactId>
> <name>HttpCore NIO</name>
> <inceptionYear>2005</inceptionYear>
>
> thanks for your help
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org