You are viewing a plain text version of this content. The canonical link for it is here.
Posted to httpclient-users@hc.apache.org by Poonam Chawla <Po...@mathworks.com> on 2016/06/10 14:15:09 UTC
ManagedNHttpClientConnectionImpl Shutdown Vs. Close
Hello Oleg,
I am using Apache HttpAsycnClient 4.1 and I noticed that the connections are not released on calling close.
Here is what I am trying to do: (Please note that it is not a compileable code as I had to remove lines that are internal to my company).
- Create CloseableHttpAsyncClient
CloseableHttpAsyncClient httpClientAsync;
SSLIOSessionStrategy sslsf = new SSLIOSessionStrategy(sslContext,
hostnameVerifier) {
@Override
protected void verifySession(HttpHost host,
org.apache.http.nio.reactor.IOSession iosession,
SSLSession sslsession) throws SSLException {
verifySSLSession(iosession, sslsession, sslConfig);
}
};
Registry<SchemeIOSessionStrategy> sessionStrategyRegistry = RegistryBuilder
.<SchemeIOSessionStrategy> create()
.register("http", NoopIOSessionStrategy.INSTANCE)
.register("https", sslsf).build();
IOReactorConfig ioReactorConfig = IOReactorConfig.custom()
.setConnectTimeout((int) clientConfig.getTimeOutMs())
.setSoTimeout((int) clientConfig.getTimeOutMs())
.setTcpNoDelay(false).build();
ConnectingIOReactor ioReactor = new DefaultConnectingIOReactor(
ioReactorConfig);
NHttpConnectionFactory<ManagedNHttpClientConnection> connFactory = new ManagedNHttpClientConnectionFactory();
PoolingNHttpClientConnectionManager connManager = new PoolingNHttpClientConnectionManager(
ioReactor, connFactory, sessionStrategyRegistry);
connManager.setDefaultMaxPerRoute(clientConfig
.getMaxConnectionsPerAddress());
httpClientAsync = HttpAsyncClients.custom()
.setUserAgent(ApacheHttpClient.class.getName())
.setConnectionManager(connManager)
.setSSLContext(sslContext)
.setSSLHostnameVerifier(hostnameVerifier)
.setSSLStrategy(sslsf).build();
httpClientAsync.start();
- Execute HTTPPosts using the httpClientAsync
try{
// underneath creates the CloseableHttpAsyncClient
MWHttpClient client = new MWHttpClient();
//underneath calls httpClientAsync.execute(….)
Future<> f = client.invoke(….)
f.get()
//underneath calls httpClientAsync.execute(….)
f1 = httpClientAsync.execute(…)
catch{
}finally{
// spawn a new thread to close the httpClientAsync
// If there are pending requests waiting for a response and the user calls close,
// abort the pending requests using request.abort();
// underneath calls httpClientAsync.close();
client.close();
}
Here is a part of the entire log I am confused about:
2016/06/09 16:30:34:753 EDT [DEBUG] headers - http-outgoing-0 >> GET /~81b806db-4546-4a7d-b623-6f0848ff6b05/requests?since=2448&format=protobuf&clients=97628818-0daf-405c-a14c-4891738587b7 HTTP/1.1
2016/06/09 16:30:34:753 EDT [DEBUG] headers - http-outgoing-0 >> Host: localhost:53097
2016/06/09 16:30:34:753 EDT [DEBUG] headers - http-outgoing-0 >> Connection: Keep-Alive
2016/06/09 16:30:34:753 EDT [DEBUG] headers - http-outgoing-0 >> User-Agent: com.mathworks.mps.client.internal.ApacheHttpClient
2016/06/09 16:30:34:753 EDT [DEBUG] ManagedNHttpClientConnectionImpl - http-outgoing-0 127.0.0.1:54726<->127.0.0.1:53097[ACTIVE][rw:w]: Event set [w]
2016/06/09 16:30:34:753 EDT [DEBUG] MainClientExec - [exchange: 12] Request completed
2016/06/09 16:30:34:753 EDT [DEBUG] ManagedNHttpClientConnectionImpl - http-outgoing-0 127.0.0.1:54726<->127.0.0.1:53097[ACTIVE][rw:w]: 247 bytes written
2016/06/09 16:30:34:753 EDT [DEBUG] ManagedNHttpClientConnectionImpl - http-outgoing-0 127.0.0.1:54726<->127.0.0.1:53097[ACTIVE][r:w]: Event cleared [w]
2016/06/09 16:30:34:852 EDT [DEBUG] InternalHttpAsyncClient - [exchange: 12] Cancelled
2016/06/09 16:30:34:853 EDT [DEBUG] ManagedNHttpClientConnectionImpl - http-outgoing-0 127.0.0.1:54726<->127.0.0.1:53097[ACTIVE][r:w]: Shutdown
2016/06/09 16:30:34:853 EDT [DEBUG] PoolingNHttpClientConnectionManager - Connection manager is shutting down
2016/06/09 16:30:34:853 EDT [DEBUG] ManagedNHttpClientConnectionImpl - http-outgoing-1 127.0.0.1:54727<->127.0.0.1:53097[ACTIVE][r:r]: Close
2016/06/09 16:30:34:853 EDT [DEBUG] InternalHttpAsyncClient - [exchange: 12] connection aborted
2016/06/09 16:30:34:854 EDT [DEBUG] InternalIODispatch - http-outgoing-0 [CLOSED]: Disconnected
2016/06/09 16:30:34:855 EDT [DEBUG] InternalIODispatch - http-outgoing-1 [CLOSED]: Disconnected
2016/06/09 16:30:34:855 EDT [DEBUG] PoolingNHttpClientConnectionManager - Connection manager shut down
2016/06/09 16:30:34:855 EDT [DEBUG] PoolingNHttpClientConnectionManager - Releasing connection: [id: http-outgoing-0][route: {}->http://localhost:53097][total kept alive: 0; route allocated: 0 of 10; total allocated: 0 of 20]
2016/06/09 16:30:34:855 EDT [DEBUG] PoolingNHttpClientConnectionManager - Connection released: [id: http-outgoing-0][route: {}->http://localhost:53097][total kept alive: 0; route allocated: 0 of 10; total allocated: 0 of 20]
Even though the log at the end states that there are 0 of 20 connections allocated, this is what I see in the “Open files and ports” on Mac:
./.mps_socket/mathworks-mds-local-socket.83924.1
->0x60dc6d646a6b4acd
localhost:53097->localhost:54726
I tried to run this code in a loop and what is interesting is any connection that is “SHUTDOWN” is still open when you look at the activity monitor, whereas any connection that is “CLOSED” is no longer open.
This leak is eventually causing failures on our side with “Too many files open” error.
I would really appreciate if you could help me find a workaround to this issue.
Thanks,
Poonam
Re: ManagedNHttpClientConnectionImpl Shutdown Vs. Close
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Fri, 2016-06-10 at 17:35 +0000, Poonam Chawla wrote:
> Hello Oleg,
>
> About the shutdown vs close, I highlighted two lines from the log:
>
> 2016/06/09 16:30:34:853 EDT [DEBUG] ManagedNHttpClientConnectionImpl - http-outgoing-0 127.0.0.1:54726<->127.0.0.1:53097[ACTIVE][r:w]: Shutdown
> 2016/06/09 16:30:34:853 EDT [DEBUG] PoolingNHttpClientConnectionManager - Connection manager is shutting down
> 2016/06/09 16:30:34:853 EDT [DEBUG] ManagedNHttpClientConnectionImpl - http-outgoing-1 127.0.0.1:54727<->127.0.0.1:53097[ACTIVE][r:r]: Close
> 2016/06/09 16:30:34:853 EDT [DEBUG] InternalHttpAsyncClient - [exchange: 12] connection aborted
>
> http-outgoing-0 127.0.0.1:54726<->127.0.0.1:53097[ACTIVE][r:w]: Shutdown : This connection (where the log says \u201cSHUTDOWN\u201d) stays in CLOSED_WAIT state forever.
>
> mps_maste 83924 pchawla 29u IPv4 0x60dc6d64510bb295 0t0 TCP localhost:53097->localhost:54726 (CLOSE_WAIT)
>
> However, the connection that was closed has been released. http-outgoing-1 127.0.0.1:54727<->127.0.0.1:53097[ACTIVE][r:r]: Close
>
I cannot think of any practical difference between closing connection
and shutting down plain (non-SSL) connection other than flushing and not
flushing session buffers.
I do not think I will be able to help you unless you put together a test
app to reproduce the issue.
> I am not sure if it is related to this bug report : https://issues.apache.org/jira/browse/HTTPASYNC-69
>
I do not think so.
Oleg
> Thanks,
> Poonam
>
>
> On Jun 10, 2016, at 11:48 AM, Oleg Kalnichevski <ol...@apache.org>> wrote:
>
> On Fri, 2016-06-10 at 14:15 +0000, Poonam Chawla wrote:
> Hello Oleg,
>
>
> I am using Apache HttpAsycnClient 4.1 and I noticed that the
> connections are not released on calling close.
> Here is what I am trying to do: (Please note that it is not a
> compileable code as I had to remove lines that are internal to my
> company).
>
>
> - Create CloseableHttpAsyncClient
>
>
> CloseableHttpAsyncClient httpClientAsync;
>
>
> SSLIOSessionStrategy sslsf = new SSLIOSessionStrategy(sslContext,
> hostnameVerifier) {
> @Override
> protected void verifySession(HttpHost host,
> org.apache.http.nio.reactor.IOSession
> iosession,
> SSLSession sslsession) throws SSLException {
> verifySSLSession(iosession, sslsession,
> sslConfig);
> }
> };
>
>
> Registry<SchemeIOSessionStrategy> sessionStrategyRegistry
> = RegistryBuilder
> .<SchemeIOSessionStrategy> create()
> .register("http", NoopIOSessionStrategy.INSTANCE)
> .register("https", sslsf).build();
>
>
> IOReactorConfig ioReactorConfig = IOReactorConfig.custom()
> .setConnectTimeout((int)
> clientConfig.getTimeOutMs())
> .setSoTimeout((int) clientConfig.getTimeOutMs())
> .setTcpNoDelay(false).build();
>
>
> ConnectingIOReactor ioReactor = new
> DefaultConnectingIOReactor(
> ioReactorConfig);
>
>
> NHttpConnectionFactory<ManagedNHttpClientConnection>
> connFactory = new ManagedNHttpClientConnectionFactory();
> PoolingNHttpClientConnectionManager connManager = new
> PoolingNHttpClientConnectionManager(
> ioReactor,connFactory, sessionStrategyRegistry);
> connManager.setDefaultMaxPerRoute(clientConfig
> .getMaxConnectionsPerAddress());
>
>
> httpClientAsync = HttpAsyncClients.custom()
> .setUserAgent(ApacheHttpClient.class.getName())
> .setConnectionManager(connManager)
> .setSSLContext(sslContext)
> .setSSLHostnameVerifier(hostnameVerifier)
> .setSSLStrategy(sslsf).build();
> httpClientAsync.start();
>
>
>
>
> - Execute HTTPPosts using the httpClientAsync
>
>
> try{
> // underneath creates the CloseableHttpAsyncClient
> MWHttpClient client = new MWHttpClient();
>
> //underneath calls httpClientAsync.execute(\u2026.)
> Future<> f = client.invoke(\u2026.)
> f.get()
>
>
> //underneath calls httpClientAsync.execute(\u2026.)
> f1 = httpClientAsync.execute(\u2026)
>
>
> catch{
> }finally{
> // spawn a new thread to close the httpClientAsync
> // If there are pending requests waiting for a response and the user
> calls close,
> // abort the pending requests using request.abort();
> // underneath calls httpClientAsync.close();
>
>
> client.close();
> }
>
>
>
>
> Here is a part of the entire log I am confused about:
>
>
> 2016/06/09 16:30:34:753 EDT [DEBUG] headers - http-outgoing-0 >>
> GET /~81b806db-4546-4a7d-b623-6f0848ff6b05/requests?since=2448&format=protobuf&clients=97628818-0daf-405c-a14c-4891738587b7 HTTP/1.1
> 2016/06/09 16:30:34:753 EDT [DEBUG] headers - http-outgoing-0 >> Host:
> localhost:53097
> 2016/06/09 16:30:34:753 EDT [DEBUG] headers - http-outgoing-0 >>
> Connection: Keep-Alive
> 2016/06/09 16:30:34:753 EDT [DEBUG] headers - http-outgoing-0 >>
> User-Agent: com.mathworks.mps.client.internal.ApacheHttpClient
> 2016/06/09 16:30:34:753 EDT [DEBUG] ManagedNHttpClientConnectionImpl -
> http-outgoing-0 127.0.0.1:54726<->127.0.0.1:53097[ACTIVE][rw:w]: Event
> set [w]
> 2016/06/09 16:30:34:753 EDT [DEBUG] MainClientExec - [exchange: 12]
> Request completed
> 2016/06/09 16:30:34:753 EDT [DEBUG] ManagedNHttpClientConnectionImpl -
> http-outgoing-0 127.0.0.1:54726<->127.0.0.1:53097[ACTIVE][rw:w]: 247
> bytes written
> 2016/06/09 16:30:34:753 EDT [DEBUG] ManagedNHttpClientConnectionImpl -
> http-outgoing-0 127.0.0.1:54726<->127.0.0.1:53097[ACTIVE][r:w]: Event
> cleared [w]
> 2016/06/09 16:30:34:852 EDT [DEBUG] InternalHttpAsyncClient -
> [exchange: 12] Cancelled
> 2016/06/09 16:30:34:853 EDT [DEBUG] ManagedNHttpClientConnectionImpl -
> http-outgoing-0 127.0.0.1:54726<->127.0.0.1:53097[ACTIVE][r:w]:
> Shutdown
> 2016/06/09 16:30:34:853 EDT [DEBUG]
> PoolingNHttpClientConnectionManager - Connection manager is shutting
> down
> 2016/06/09 16:30:34:853 EDT [DEBUG] ManagedNHttpClientConnectionImpl -
> http-outgoing-1 127.0.0.1:54727<->127.0.0.1:53097[ACTIVE][r:r]: Close
> 2016/06/09 16:30:34:853 EDT [DEBUG] InternalHttpAsyncClient -
> [exchange: 12] connection aborted
> 2016/06/09 16:30:34:854 EDT [DEBUG] InternalIODispatch -
> http-outgoing-0 [CLOSED]: Disconnected
> 2016/06/09 16:30:34:855 EDT [DEBUG] InternalIODispatch -
> http-outgoing-1 [CLOSED]: Disconnected
> 2016/06/09 16:30:34:855 EDT [DEBUG]
> PoolingNHttpClientConnectionManager - Connection manager shut down
> 2016/06/09 16:30:34:855 EDT [DEBUG]
> PoolingNHttpClientConnectionManager - Releasing connection: [id:
> http-outgoing-0][route: {}->http://localhost:53097][total kept alive:
> 0; route allocated: 0 of 10; total allocated: 0 of 20]
> 2016/06/09 16:30:34:855 EDT [DEBUG]
> PoolingNHttpClientConnectionManager - Connection released: [id:
> http-outgoing-0][route: {}->http://localhost:53097][total kept alive:
> 0; route allocated: 0 of 10; total allocated: 0 of 20]
>
>
> Even though the log at the end states that there are 0 of 20
> connections allocated, this is what I see in the \u201cOpen files and
> ports\u201d on Mac:
> ./.mps_socket/mathworks-mds-local-socket.83924.1
> ->0x60dc6d646a6b4acd
> localhost:53097->localhost:54726
>
>
> Please note that the OS may still keep the underlying TCP connection
> open in TIME-WAIT even if the JVM considers the socket of that
> connection fully closed.
>
>
> I tried to run this code in a loop and what is interesting is any
> connection that is \u201cSHUTDOWN\u201d is still open when you look at the
> activity monitor, whereas any connection that is \u201cCLOSED\u201d is no longer
> open.
>
>
> I am not sure I understand this bit. Could you please provide a test app
> demonstrating the issue?
>
> Oleg
>
>
> This leak is eventually causing failures on our side with \u201cToo many
> files open\u201d error.
> I would really appreciate if you could help me find a workaround to
> this issue.
>
>
> Thanks,
> Poonam
>
>
>
>
>
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org<ma...@hc.apache.org>
> For additional commands, e-mail: httpclient-users-help@hc.apache.org<ma...@hc.apache.org>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: ManagedNHttpClientConnectionImpl Shutdown Vs. Close
Posted by Poonam Chawla <Po...@mathworks.com>.
Hello Oleg,
About the shutdown vs close, I highlighted two lines from the log:
2016/06/09 16:30:34:853 EDT [DEBUG] ManagedNHttpClientConnectionImpl - http-outgoing-0 127.0.0.1:54726<->127.0.0.1:53097[ACTIVE][r:w]: Shutdown
2016/06/09 16:30:34:853 EDT [DEBUG] PoolingNHttpClientConnectionManager - Connection manager is shutting down
2016/06/09 16:30:34:853 EDT [DEBUG] ManagedNHttpClientConnectionImpl - http-outgoing-1 127.0.0.1:54727<->127.0.0.1:53097[ACTIVE][r:r]: Close
2016/06/09 16:30:34:853 EDT [DEBUG] InternalHttpAsyncClient - [exchange: 12] connection aborted
http-outgoing-0 127.0.0.1:54726<->127.0.0.1:53097[ACTIVE][r:w]: Shutdown : This connection (where the log says “SHUTDOWN”) stays in CLOSED_WAIT state forever.
mps_maste 83924 pchawla 29u IPv4 0x60dc6d64510bb295 0t0 TCP localhost:53097->localhost:54726 (CLOSE_WAIT)
However, the connection that was closed has been released. http-outgoing-1 127.0.0.1:54727<->127.0.0.1:53097[ACTIVE][r:r]: Close
I am not sure if it is related to this bug report : https://issues.apache.org/jira/browse/HTTPASYNC-69
Thanks,
Poonam
On Jun 10, 2016, at 11:48 AM, Oleg Kalnichevski <ol...@apache.org>> wrote:
On Fri, 2016-06-10 at 14:15 +0000, Poonam Chawla wrote:
Hello Oleg,
I am using Apache HttpAsycnClient 4.1 and I noticed that the
connections are not released on calling close.
Here is what I am trying to do: (Please note that it is not a
compileable code as I had to remove lines that are internal to my
company).
- Create CloseableHttpAsyncClient
CloseableHttpAsyncClient httpClientAsync;
SSLIOSessionStrategy sslsf = new SSLIOSessionStrategy(sslContext,
hostnameVerifier) {
@Override
protected void verifySession(HttpHost host,
org.apache.http.nio.reactor.IOSession
iosession,
SSLSession sslsession) throws SSLException {
verifySSLSession(iosession, sslsession,
sslConfig);
}
};
Registry<SchemeIOSessionStrategy> sessionStrategyRegistry
= RegistryBuilder
.<SchemeIOSessionStrategy> create()
.register("http", NoopIOSessionStrategy.INSTANCE)
.register("https", sslsf).build();
IOReactorConfig ioReactorConfig = IOReactorConfig.custom()
.setConnectTimeout((int)
clientConfig.getTimeOutMs())
.setSoTimeout((int) clientConfig.getTimeOutMs())
.setTcpNoDelay(false).build();
ConnectingIOReactor ioReactor = new
DefaultConnectingIOReactor(
ioReactorConfig);
NHttpConnectionFactory<ManagedNHttpClientConnection>
connFactory = new ManagedNHttpClientConnectionFactory();
PoolingNHttpClientConnectionManager connManager = new
PoolingNHttpClientConnectionManager(
ioReactor,connFactory, sessionStrategyRegistry);
connManager.setDefaultMaxPerRoute(clientConfig
.getMaxConnectionsPerAddress());
httpClientAsync = HttpAsyncClients.custom()
.setUserAgent(ApacheHttpClient.class.getName())
.setConnectionManager(connManager)
.setSSLContext(sslContext)
.setSSLHostnameVerifier(hostnameVerifier)
.setSSLStrategy(sslsf).build();
httpClientAsync.start();
- Execute HTTPPosts using the httpClientAsync
try{
// underneath creates the CloseableHttpAsyncClient
MWHttpClient client = new MWHttpClient();
//underneath calls httpClientAsync.execute(….)
Future<> f = client.invoke(….)
f.get()
//underneath calls httpClientAsync.execute(….)
f1 = httpClientAsync.execute(…)
catch{
}finally{
// spawn a new thread to close the httpClientAsync
// If there are pending requests waiting for a response and the user
calls close,
// abort the pending requests using request.abort();
// underneath calls httpClientAsync.close();
client.close();
}
Here is a part of the entire log I am confused about:
2016/06/09 16:30:34:753 EDT [DEBUG] headers - http-outgoing-0 >>
GET /~81b806db-4546-4a7d-b623-6f0848ff6b05/requests?since=2448&format=protobuf&clients=97628818-0daf-405c-a14c-4891738587b7 HTTP/1.1
2016/06/09 16:30:34:753 EDT [DEBUG] headers - http-outgoing-0 >> Host:
localhost:53097
2016/06/09 16:30:34:753 EDT [DEBUG] headers - http-outgoing-0 >>
Connection: Keep-Alive
2016/06/09 16:30:34:753 EDT [DEBUG] headers - http-outgoing-0 >>
User-Agent: com.mathworks.mps.client.internal.ApacheHttpClient
2016/06/09 16:30:34:753 EDT [DEBUG] ManagedNHttpClientConnectionImpl -
http-outgoing-0 127.0.0.1:54726<->127.0.0.1:53097[ACTIVE][rw:w]: Event
set [w]
2016/06/09 16:30:34:753 EDT [DEBUG] MainClientExec - [exchange: 12]
Request completed
2016/06/09 16:30:34:753 EDT [DEBUG] ManagedNHttpClientConnectionImpl -
http-outgoing-0 127.0.0.1:54726<->127.0.0.1:53097[ACTIVE][rw:w]: 247
bytes written
2016/06/09 16:30:34:753 EDT [DEBUG] ManagedNHttpClientConnectionImpl -
http-outgoing-0 127.0.0.1:54726<->127.0.0.1:53097[ACTIVE][r:w]: Event
cleared [w]
2016/06/09 16:30:34:852 EDT [DEBUG] InternalHttpAsyncClient -
[exchange: 12] Cancelled
2016/06/09 16:30:34:853 EDT [DEBUG] ManagedNHttpClientConnectionImpl -
http-outgoing-0 127.0.0.1:54726<->127.0.0.1:53097[ACTIVE][r:w]:
Shutdown
2016/06/09 16:30:34:853 EDT [DEBUG]
PoolingNHttpClientConnectionManager - Connection manager is shutting
down
2016/06/09 16:30:34:853 EDT [DEBUG] ManagedNHttpClientConnectionImpl -
http-outgoing-1 127.0.0.1:54727<->127.0.0.1:53097[ACTIVE][r:r]: Close
2016/06/09 16:30:34:853 EDT [DEBUG] InternalHttpAsyncClient -
[exchange: 12] connection aborted
2016/06/09 16:30:34:854 EDT [DEBUG] InternalIODispatch -
http-outgoing-0 [CLOSED]: Disconnected
2016/06/09 16:30:34:855 EDT [DEBUG] InternalIODispatch -
http-outgoing-1 [CLOSED]: Disconnected
2016/06/09 16:30:34:855 EDT [DEBUG]
PoolingNHttpClientConnectionManager - Connection manager shut down
2016/06/09 16:30:34:855 EDT [DEBUG]
PoolingNHttpClientConnectionManager - Releasing connection: [id:
http-outgoing-0][route: {}->http://localhost:53097][total kept alive:
0; route allocated: 0 of 10; total allocated: 0 of 20]
2016/06/09 16:30:34:855 EDT [DEBUG]
PoolingNHttpClientConnectionManager - Connection released: [id:
http-outgoing-0][route: {}->http://localhost:53097][total kept alive:
0; route allocated: 0 of 10; total allocated: 0 of 20]
Even though the log at the end states that there are 0 of 20
connections allocated, this is what I see in the “Open files and
ports” on Mac:
./.mps_socket/mathworks-mds-local-socket.83924.1
->0x60dc6d646a6b4acd
localhost:53097->localhost:54726
Please note that the OS may still keep the underlying TCP connection
open in TIME-WAIT even if the JVM considers the socket of that
connection fully closed.
I tried to run this code in a loop and what is interesting is any
connection that is “SHUTDOWN” is still open when you look at the
activity monitor, whereas any connection that is “CLOSED” is no longer
open.
I am not sure I understand this bit. Could you please provide a test app
demonstrating the issue?
Oleg
This leak is eventually causing failures on our side with “Too many
files open” error.
I would really appreciate if you could help me find a workaround to
this issue.
Thanks,
Poonam
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org<ma...@hc.apache.org>
For additional commands, e-mail: httpclient-users-help@hc.apache.org<ma...@hc.apache.org>
Re: ManagedNHttpClientConnectionImpl Shutdown Vs. Close
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Fri, 2016-06-10 at 14:15 +0000, Poonam Chawla wrote:
> Hello Oleg,
>
>
> I am using Apache HttpAsycnClient 4.1 and I noticed that the
> connections are not released on calling close.
> Here is what I am trying to do: (Please note that it is not a
> compileable code as I had to remove lines that are internal to my
> company).
>
>
> - Create CloseableHttpAsyncClient
>
>
> CloseableHttpAsyncClient httpClientAsync;
>
>
> SSLIOSessionStrategy sslsf = new SSLIOSessionStrategy(sslContext,
> hostnameVerifier) {
> @Override
> protected void verifySession(HttpHost host,
> org.apache.http.nio.reactor.IOSession
> iosession,
> SSLSession sslsession) throws SSLException {
> verifySSLSession(iosession, sslsession,
> sslConfig);
> }
> };
>
>
> Registry<SchemeIOSessionStrategy> sessionStrategyRegistry
> = RegistryBuilder
> .<SchemeIOSessionStrategy> create()
> .register("http", NoopIOSessionStrategy.INSTANCE)
> .register("https", sslsf).build();
>
>
> IOReactorConfig ioReactorConfig = IOReactorConfig.custom()
> .setConnectTimeout((int)
> clientConfig.getTimeOutMs())
> .setSoTimeout((int) clientConfig.getTimeOutMs())
> .setTcpNoDelay(false).build();
>
>
> ConnectingIOReactor ioReactor = new
> DefaultConnectingIOReactor(
> ioReactorConfig);
>
>
> NHttpConnectionFactory<ManagedNHttpClientConnection>
> connFactory = new ManagedNHttpClientConnectionFactory();
> PoolingNHttpClientConnectionManager connManager = new
> PoolingNHttpClientConnectionManager(
> ioReactor,connFactory, sessionStrategyRegistry);
> connManager.setDefaultMaxPerRoute(clientConfig
> .getMaxConnectionsPerAddress());
>
>
> httpClientAsync = HttpAsyncClients.custom()
> .setUserAgent(ApacheHttpClient.class.getName())
> .setConnectionManager(connManager)
> .setSSLContext(sslContext)
> .setSSLHostnameVerifier(hostnameVerifier)
> .setSSLStrategy(sslsf).build();
> httpClientAsync.start();
>
>
>
>
> - Execute HTTPPosts using the httpClientAsync
>
>
> try{
> // underneath creates the CloseableHttpAsyncClient
> MWHttpClient client = new MWHttpClient();
>
> //underneath calls httpClientAsync.execute(\u2026.)
> Future<> f = client.invoke(\u2026.)
> f.get()
>
>
> //underneath calls httpClientAsync.execute(\u2026.)
> f1 = httpClientAsync.execute(\u2026)
>
>
> catch{
> }finally{
> // spawn a new thread to close the httpClientAsync
> // If there are pending requests waiting for a response and the user
> calls close,
> // abort the pending requests using request.abort();
> // underneath calls httpClientAsync.close();
>
>
> client.close();
> }
>
>
>
>
> Here is a part of the entire log I am confused about:
>
>
> 2016/06/09 16:30:34:753 EDT [DEBUG] headers - http-outgoing-0 >>
> GET /~81b806db-4546-4a7d-b623-6f0848ff6b05/requests?since=2448&format=protobuf&clients=97628818-0daf-405c-a14c-4891738587b7 HTTP/1.1
> 2016/06/09 16:30:34:753 EDT [DEBUG] headers - http-outgoing-0 >> Host:
> localhost:53097
> 2016/06/09 16:30:34:753 EDT [DEBUG] headers - http-outgoing-0 >>
> Connection: Keep-Alive
> 2016/06/09 16:30:34:753 EDT [DEBUG] headers - http-outgoing-0 >>
> User-Agent: com.mathworks.mps.client.internal.ApacheHttpClient
> 2016/06/09 16:30:34:753 EDT [DEBUG] ManagedNHttpClientConnectionImpl -
> http-outgoing-0 127.0.0.1:54726<->127.0.0.1:53097[ACTIVE][rw:w]: Event
> set [w]
> 2016/06/09 16:30:34:753 EDT [DEBUG] MainClientExec - [exchange: 12]
> Request completed
> 2016/06/09 16:30:34:753 EDT [DEBUG] ManagedNHttpClientConnectionImpl -
> http-outgoing-0 127.0.0.1:54726<->127.0.0.1:53097[ACTIVE][rw:w]: 247
> bytes written
> 2016/06/09 16:30:34:753 EDT [DEBUG] ManagedNHttpClientConnectionImpl -
> http-outgoing-0 127.0.0.1:54726<->127.0.0.1:53097[ACTIVE][r:w]: Event
> cleared [w]
> 2016/06/09 16:30:34:852 EDT [DEBUG] InternalHttpAsyncClient -
> [exchange: 12] Cancelled
> 2016/06/09 16:30:34:853 EDT [DEBUG] ManagedNHttpClientConnectionImpl -
> http-outgoing-0 127.0.0.1:54726<->127.0.0.1:53097[ACTIVE][r:w]:
> Shutdown
> 2016/06/09 16:30:34:853 EDT [DEBUG]
> PoolingNHttpClientConnectionManager - Connection manager is shutting
> down
> 2016/06/09 16:30:34:853 EDT [DEBUG] ManagedNHttpClientConnectionImpl -
> http-outgoing-1 127.0.0.1:54727<->127.0.0.1:53097[ACTIVE][r:r]: Close
> 2016/06/09 16:30:34:853 EDT [DEBUG] InternalHttpAsyncClient -
> [exchange: 12] connection aborted
> 2016/06/09 16:30:34:854 EDT [DEBUG] InternalIODispatch -
> http-outgoing-0 [CLOSED]: Disconnected
> 2016/06/09 16:30:34:855 EDT [DEBUG] InternalIODispatch -
> http-outgoing-1 [CLOSED]: Disconnected
> 2016/06/09 16:30:34:855 EDT [DEBUG]
> PoolingNHttpClientConnectionManager - Connection manager shut down
> 2016/06/09 16:30:34:855 EDT [DEBUG]
> PoolingNHttpClientConnectionManager - Releasing connection: [id:
> http-outgoing-0][route: {}->http://localhost:53097][total kept alive:
> 0; route allocated: 0 of 10; total allocated: 0 of 20]
> 2016/06/09 16:30:34:855 EDT [DEBUG]
> PoolingNHttpClientConnectionManager - Connection released: [id:
> http-outgoing-0][route: {}->http://localhost:53097][total kept alive:
> 0; route allocated: 0 of 10; total allocated: 0 of 20]
>
>
> Even though the log at the end states that there are 0 of 20
> connections allocated, this is what I see in the \u201cOpen files and
> ports\u201d on Mac:
> ./.mps_socket/mathworks-mds-local-socket.83924.1
> ->0x60dc6d646a6b4acd
> localhost:53097->localhost:54726
>
Please note that the OS may still keep the underlying TCP connection
open in TIME-WAIT even if the JVM considers the socket of that
connection fully closed.
>
> I tried to run this code in a loop and what is interesting is any
> connection that is \u201cSHUTDOWN\u201d is still open when you look at the
> activity monitor, whereas any connection that is \u201cCLOSED\u201d is no longer
> open.
>
I am not sure I understand this bit. Could you please provide a test app
demonstrating the issue?
Oleg
>
> This leak is eventually causing failures on our side with \u201cToo many
> files open\u201d error.
> I would really appreciate if you could help me find a workaround to
> this issue.
>
>
> Thanks,
> Poonam
>
>
>
>
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org