You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hc.apache.org by "Anurag Agarwal (Jira)" <ji...@apache.org> on 2019/11/02 15:45:00 UTC
[jira] [Commented] (HTTPASYNC-152) "Connection must be created by
connection manager" raised when requests are reused after being aborted
[ https://issues.apache.org/jira/browse/HTTPASYNC-152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16965406#comment-16965406 ]
Anurag Agarwal commented on HTTPASYNC-152:
------------------------------------------
Hey [~olegk], I think the bug is not 100% resolved even in 4.12, but by far the frequency has reduced to like 1 box in a day under heavy traffic out of 300-400 boxes we run.
I was trying to find out from where can this still occur. Yet I have only been able to suspect a single place from where this error can still occur. But I am not 100% sure if the source of the error we are getting in production environment is because of this. I will try to explain you the situation which may lead to this bug, but again it is difficult to reproduce it.
# Let there be a pending request for the connection to be established.
# Let everything run perfectly till [https://github.com/apache/httpcomponents-core/blob/023823b9f2bba59cf7338d7398cc4a0db1402bfc/httpcore-nio/src/main/java/org/apache/http/impl/nio/reactor/AbstractIOReactor.java#L426.] but not executing this line yet.
# And then shutdown the ConnectionPool (only till the point when the shutdown variable is just set to true, but session request is not yet cancelled).
# When following the `sessionRequest.completed` callbacks it will reach [https://github.com/apache/httpcomponents-core/blob/023823b9f2bba59cf7338d7398cc4a0db1402bfc/httpcore-nio/src/main/java/org/apache/http/nio/pool/AbstractNIOConnPool.java#L545-L547] where it will be returned and for this session, connection attribute won't be set.
# Since the session is still not terminated it will follow sessionCreated(key, session); line and proceed to give the above error, since the connection attribute is not attached and will return null.
Please also do look from your end as if there are any edge cases left, I am also looking from my end. Also confirm if my speculations are correct while I will confirm if we are getting the issue because of the above explained case.
> "Connection must be created by connection manager" raised when requests are reused after being aborted
> ------------------------------------------------------------------------------------------------------
>
> Key: HTTPASYNC-152
> URL: https://issues.apache.org/jira/browse/HTTPASYNC-152
> Project: HttpComponents HttpAsyncClient
> Issue Type: Bug
> Affects Versions: 4.1.4
> Reporter: Luca Cavanna
> Assignee: Oleg Kalnichevski
> Priority: Major
> Fix For: 4.1.5
>
>
> We are using http async client in the official Java REST client for Elasticsearch. We recently introduced the ability to cancel requests. While testing such new feature, I encountered a test failure that can be reproduced when the same request instance is reused (although reset is called after each run) after being cancelled. The following exception is raised, which calls the reactor to shut down unexpectedly, which fails all subsequent requests:
> {noformat}
> Aug 15, 2019 2:04:52 PM org.apache.http.impl.nio.client.InternalHttpAsyncClient run
> SEVERE: I/O reactor terminated abnormally
> org.apache.http.nio.reactor.IOReactorException: I/O dispatch worker terminated abnormally
> at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:359)
> at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
> at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
> at java.base/java.lang.Thread.run(Thread.java:844)
> Caused by: java.lang.IllegalStateException: Connection must be created by connection manager
> at org.apache.http.impl.nio.client.InternalIODispatch.createConnection(InternalIODispatch.java:56)
> at org.apache.http.impl.nio.client.InternalIODispatch.createConnection(InternalIODispatch.java:39)
> at org.apache.http.impl.nio.reactor.AbstractIODispatch.connected(AbstractIODispatch.java:70)
> at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionCreated(BaseIOReactor.java:248)
> at org.apache.http.impl.nio.reactor.AbstractIOReactor.processNewChannels(AbstractIOReactor.java:427)
> at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:287)
> at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
> at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
> ... 1 more
> Exception in thread "main" java.util.concurrent.ExecutionException: java.lang.IllegalStateException: I/O reactor has been shut down
> at org.apache.http.concurrent.BasicFuture.getResult(BasicFuture.java:71)
> at org.apache.http.concurrent.BasicFuture.get(BasicFuture.java:84)
> at org.apache.http.impl.nio.client.FutureWrapper.get(FutureWrapper.java:70)
> at ClientTest.main(ClientTest.java:31)
> Caused by: java.lang.IllegalStateException: I/O reactor has been shut down
> at org.apache.http.util.Asserts.check(Asserts.java:34)
> at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.connect(DefaultConnectingIOReactor.java:228)
> at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequest(AbstractNIOConnPool.java:481)
> at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:280)
> at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requestConnection(PoolingNHttpClientConnectionManager.java:295)
> at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestConnection(AbstractClientExchangeHandler.java:377)
> at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:129)
> at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141)
> at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:75)
> at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:85)
> at ClientTest.main(ClientTest.java:28)
> {noformat}
> The following snippet reproduces it most of the times:
> {code:java}
> import com.sun.net.httpserver.HttpServer;
> import org.apache.http.HttpHost;
> import org.apache.http.HttpResponse;
> import org.apache.http.client.methods.HttpGet;
> import org.apache.http.impl.nio.client.CloseableHttpAsyncClient;
> import org.apache.http.impl.nio.client.HttpAsyncClientBuilder;
> import java.net.InetAddress;
> import java.net.InetSocketAddress;
> import java.util.concurrent.CancellationException;
> import java.util.concurrent.Future;
> public class ClientTest {
> public static void main(String[] args) throws Exception {
> HttpServer httpServer = HttpServer.create(new InetSocketAddress(InetAddress.getLoopbackAddress(), 0), 0);
> httpServer.start();
> httpServer.createContext("/test", exchange -> {
> exchange.sendResponseHeaders(200, 0);
> exchange.close();
> });
> HttpHost httpHost = new HttpHost(httpServer.getAddress().getHostString(), httpServer.getAddress().getPort());
> try (CloseableHttpAsyncClient client = HttpAsyncClientBuilder.create().build()) {
> client.start();
> HttpGet httpGet = new HttpGet("/test");
> for (int i = 0; i < 10000; i++) {
> httpGet.reset();
> Future<HttpResponse> future = client.execute(httpHost, httpGet, null);
> httpGet.abort();
> try {
> future.get();
> assert false;
> } catch(CancellationException e) {
> //expected
> }
> }
> } finally {
> httpServer.stop(0);
> }
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org