You are viewing a plain text version of this content. The canonical link for it is here.
Posted to httpclient-users@hc.apache.org by Harold Rosenberg <hr...@vmware.com> on 2016/03/23 19:54:29 UTC
Heap usage issue in Async Client
I am seeing unexpected behavior in the memory usage of a program that uses the Async Http Client. The issue arises when I switch from using exclusively HTTP connections to exclusively HTTPS connections. In the latter case, there is a large amount of memory used for byte arrays that seem to be related to SSL processing, and that aren’t ever cleaned up by garbage collection.
My question is whether this is expected due to differences between SSL and non-SSL processing, or whether I am failing to properly clean up my SSL connections.
The program is a workload driver for a performance benchmark. It simulates user activity to a web service. Each simulated user has about four active connections to the service. Normally the connections are all over SSL (really TLS), but for testing purposes there is a switch to use only non-SSL connections. The configuration of the HTTP client is identical for in both cases.
I gathered garbage-collection data with and without TLS for runs with 7,500 simulated users, which would average about 30,000 active connections. A user would be active for about five minutes, at which point the they log out and a new user logs in. While the user is logged in the connections will occasionally be idle and thus closed due to keepalive timeout. The runs were two hours long and the data was collected during a steady-state. Data is generated in the heap at about the same rate in both cases, but when running with HTTPS there is about 3.2GB of live data left in the heap after a full garbage-collection, while with just HTTP there is only about 890MB. The amount of live data in the heap doesn’t grow once it reaches the 3.2GB mark, so it isn’t a memory leak, but it is definitely proportional to the number of active connections. When I do the same run with 15,000 users, at steady-state the live data in the heap after a full GC is about 6.4GB.
Of the 3.2GB of live data, about 2.8GB is in 876,041 instances of byte[]. Almost all of the instances seem to be related to SSL processing. Here is the path to the nearest GC root for a few random instances:
this - value: byte[] #483799
<- hb - class: java.nio.HeapByteBuffer, value: byte[] #483799
<- buffer - class: org.apache.http.nio.reactor.ssl.PermanentSSLBufferManagementStrategy$InternalBuffer, value: java.nio.HeapByteBuffer #89456
<- outPlain - class: org.apache.http.nio.reactor.ssl.SSLIOSession, value: org.apache.http.nio.reactor.ssl.PermanentSSLBufferManagementStrategy$InternalBuffer #59632
<- bufferStatus - class: org.apache.http.impl.nio.reactor.IOSessionImpl, value: org.apache.http.nio.reactor.ssl.SSLIOSession #14910
<- attachment - class: sun.nio.ch.SelectionKeyImpl, value: org.apache.http.impl.nio.reactor.IOSessionImpl #14910
<- value - class: java.util.HashMap$Entry, value: sun.nio.ch.SelectionKeyImpl #14910
<- [355] - class: java.util.HashMap$Entry[], value: java.util.HashMap$Entry #197065
<- table - class: java.util.HashMap, value: java.util.HashMap$Entry[] #4612
<- fdToKey (Java frame) - class: sun.nio.ch.EPollSelectorImpl, value: java.util.HashMap #10016
this - value: byte[] #60139
<- key - class: com.sun.crypto.provider.TlsMasterSecretGenerator$TlsMasterSecretKey, value: byte[] #60139
<- masterSecret - class: sun.security.ssl.SSLSessionImpl, value: com.sun.crypto.provider.TlsMasterSecretGenerator$TlsMasterSecretKey #4827
<- sess - class: sun.security.ssl.SSLEngineImpl, value: sun.security.ssl.SSLSessionImpl #4829
<- sslEngine - class: org.apache.http.nio.reactor.ssl.SSLIOSession, value: sun.security.ssl.SSLEngineImpl #20572
<- bufferStatus - class: org.apache.http.impl.nio.reactor.IOSessionImpl, value: org.apache.http.nio.reactor.ssl.SSLIOSession #20572
<- attachment - class: sun.nio.ch.SelectionKeyImpl, value: org.apache.http.impl.nio.reactor.IOSessionImpl #20572
<- value - class: java.util.HashMap$Entry, value: sun.nio.ch.SelectionKeyImpl #20572
<- [1182] - class: java.util.HashMap$Entry[], value: java.util.HashMap$Entry #254234
<- table - class: java.util.HashMap, value: java.util.HashMap$Entry[] #6977
<- fdToKey (Java frame) - class: sun.nio.ch.EPollSelectorImpl, value: java.util.HashMap #9595
this - value: byte[] #71913
<- iv - class: com.sun.crypto.provider.CipherBlockChaining, value: byte[] #71913
<- cipher - class: com.sun.crypto.provider.CipherCore, value: com.sun.crypto.provider.CipherBlockChaining #3709
<- core - class: com.sun.crypto.provider.AESCipher$General, value: com.sun.crypto.provider.CipherCore #3709
<- spi - class: javax.crypto.Cipher, value: com.sun.crypto.provider.AESCipher$General #3709
<- cipher - class: sun.security.ssl.CipherBox, value: javax.crypto.Cipher #3709
<- readCipher - class: sun.security.ssl.SSLEngineImpl, value: sun.security.ssl.CipherBox #3710
<- sslEngine - class: org.apache.http.nio.reactor.ssl.SSLIOSession, value: sun.security.ssl.SSLEngineImpl #1855
<- bufferStatus - class: org.apache.http.impl.nio.reactor.IOSessionImpl, value: org.apache.http.nio.reactor.ssl.SSLIOSession #1855
<- key - class: java.util.HashMap$Entry, value: org.apache.http.impl.nio.reactor.IOSessionImpl #1855
<- [4000] - class: java.util.HashMap$Entry[], value: java.util.HashMap$Entry #71715
<- table - class: java.util.HashMap, value: java.util.HashMap$Entry[] #24138
<- map - class: java.util.HashSet, value: java.util.HashMap #8449
<- c - class: java.util.Collections$SynchronizedSet, value: java.util.HashSet #3312
<- sessions (Java frame) - class: org.apache.http.impl.nio.reactor.BaseIOReactor, value: java.util.Collections$SynchronizedSet #6
I can give more details about the configuration of the HTTP Client and the SSL sessions if it would be helpful, but I don’t want to give too much info if this is expected behavior.
Any advice would be appreciated.
Thanks,
Hal
Re: Heap usage issue in Async Client
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Thu, 2016-03-24 at 12:26 +0000, Harold Rosenberg wrote:
> Using the ReleasableSSLBufferManagementStrategy gave a nice improvement. For the same run with 7500 simulated users, the change reduced the average live data in the heap after a full GC from 3.2GB to about 1.5GB. The amount of live data is still proportional to the number of connections (about 3GB of live data for a run with 15,000 simulated users), but that is to be expected.
>
> Thanks for the help.
>
> Hal
>
I am glad it has helped.
Feel free to raise JIRA with a feature request to make use of custom
SSLBufferManagementStrategy instances less cumbersome.
Oleg
>
>
>
> On 3/23/16, 3:59 PM, "Oleg Kalnichevski" <ol...@apache.org> wrote:
>
> >On Wed, 2016-03-23 at 18:54 +0000, Harold Rosenberg wrote:
> >> I am seeing unexpected behavior in the memory usage of a program that uses the Async Http Client. The issue arises when I switch from using exclusively HTTP connections to exclusively HTTPS connections. In the latter case, there is a large amount of memory used for byte arrays that seem to be related to SSL processing, and that aren’t ever cleaned up by garbage collection.
> >>
> >> My question is whether this is expected due to differences between SSL and non-SSL processing, or whether I am failing to properly clean up my SSL connections.
> >>
> >> The program is a workload driver for a performance benchmark. It simulates user activity to a web service. Each simulated user has about four active connections to the service. Normally the connections are all over SSL (really TLS), but for testing purposes there is a switch to use only non-SSL connections. The configuration of the HTTP client is identical for in both cases.
> >>
> >> I gathered garbage-collection data with and without TLS for runs with 7,500 simulated users, which would average about 30,000 active connections. A user would be active for about five minutes, at which point the they log out and a new user logs in. While the user is logged in the connections will occasionally be idle and thus closed due to keepalive timeout. The runs were two hours long and the data was collected during a steady-state. Data is generated in the heap at about the same rate in both cases, but when running with HTTPS there is about 3.2GB of live data left in the heap after a full garbage-collection, while with just HTTP there is only about 890MB. The amount of live data in the heap doesn’t grow once it reaches the 3.2GB mark, so it isn’t a memory leak, but it is definitely proportional to the number of active connections. When I do the same run with 15,000 users, at steady-state the live data in the heap after a full GC is about 6.4GB.
> >>
> >> Of the 3.2GB of live data, about 2.8GB is in 876,041 instances of byte[]. Almost all of the instances seem to be related to SSL processing. Here is the path to the nearest GC root for a few random instances:
> >>
> >
> >Hi Harold,
> >
> >Could you try using ReleasableSSLBufferManagementStrategy instead of
> >default PermanentSSLBufferManagementStrategy?
> >
> >---
> >DefaultHostnameVerifier hostnameVerifier = new DefaultHostnameVerifier(PublicSuffixMatcherLoader.getDefault());
> >Registry<SchemeIOSessionStrategy> registry = RegistryBuilder.<SchemeIOSessionStrategy>create()
> > .register("http", NoopIOSessionStrategy.INSTANCE)
> > .register("https", new SchemeIOSessionStrategy() {
> > @Override
> > public boolean isLayeringRequired() {
> > return true;
> > }
> >
> > @Override
> > public IOSession upgrade(final HttpHost host, final IOSession iosession) throws IOException {
> > final SSLIOSession ssliosession = new SSLIOSession(
> > iosession,
> > SSLMode.CLIENT,
> > host,
> > SSLContexts.createSystemDefault(),
> > new SSLSetupHandler() {
> >
> > @Override
> > public void initalize(final SSLEngine sslengine) throws SSLException {
> > }
> >
> > @Override
> > public void verify(
> > final IOSession iosession,
> > final SSLSession sslsession) throws SSLException {
> > if (!hostnameVerifier.verify(host.getHostName(), sslsession)) {
> > final Certificate[] certs = sslsession.getPeerCertificates();
> > final X509Certificate x509 = (X509Certificate) certs[0];
> > final X500Principal x500Principal = x509.getSubjectX500Principal();
> > throw new SSLPeerUnverifiedException("Host name '" + host.getHostName() + "' does not match " +
> > "the certificate subject provided by the peer (" + x500Principal.toString() + ")");
> > }
> > }
> >
> > },
> > new ReleasableSSLBufferManagementStrategy());
> > iosession.setAttribute(SSLIOSession.SESSION_KEY, ssliosession);
> > ssliosession.initialize();
> > return ssliosession;
> > }
> > })
> > .build();
> >
> >ConnectingIOReactor ioReactor = new DefaultConnectingIOReactor();
> >PoolingNHttpClientConnectionManager cm = new PoolingNHttpClientConnectionManager(ioReactor, registry);
> >CloseableHttpAsyncClient httpclient = HttpAsyncClients.createMinimal(cm);
> >try {
> > httpclient.start();
> > HttpGet request = new HttpGet("https://urldefense.proofpoint.com/v2/url?u=https-3A__verisign.com_&d=BQIDaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=9uYCHvq1QDKcMAIXCt6iT2etdTpbRn0toGoRKE6QxRo&m=HkvNV6bDDfPghsn_MF5SgU7KlMhzmAGjcfgAmviJCSs&s=w1-QtI1DaVx22Oo8UumpY0uvEblsK5t55MihZ_UwmPM&e= ");
> > Future<HttpResponse> future = httpclient.execute(request, null);
> > HttpResponse response = future.get();
> > System.out.println("Response: " + response.getStatusLine());
> > System.out.println("Shutting down");
> >} finally {
> > httpclient.close();
> >}
> >System.out.println("Done");
> >---
> >
> >Oleg
> >
> >> this - value: byte[] #483799
> >> <- hb - class: java.nio.HeapByteBuffer, value: byte[] #483799
> >> <- buffer - class: org.apache.http.nio.reactor.ssl.PermanentSSLBufferManagementStrategy$InternalBuffer, value: java.nio.HeapByteBuffer #89456
> >> <- outPlain - class: org.apache.http.nio.reactor.ssl.SSLIOSession, value: org.apache.http.nio.reactor.ssl.PermanentSSLBufferManagementStrategy$InternalBuffer #59632
> >> <- bufferStatus - class: org.apache.http.impl.nio.reactor.IOSessionImpl, value: org.apache.http.nio.reactor.ssl.SSLIOSession #14910
> >> <- attachment - class: sun.nio.ch.SelectionKeyImpl, value: org.apache.http.impl.nio.reactor.IOSessionImpl #14910
> >> <- value - class: java.util.HashMap$Entry, value: sun.nio.ch.SelectionKeyImpl #14910
> >> <- [355] - class: java.util.HashMap$Entry[], value: java.util.HashMap$Entry #197065
> >> <- table - class: java.util.HashMap, value: java.util.HashMap$Entry[] #4612
> >> <- fdToKey (Java frame) - class: sun.nio.ch.EPollSelectorImpl, value: java.util.HashMap #10016
> >>
> >> this - value: byte[] #60139
> >> <- key - class: com.sun.crypto.provider.TlsMasterSecretGenerator$TlsMasterSecretKey, value: byte[] #60139
> >> <- masterSecret - class: sun.security.ssl.SSLSessionImpl, value: com.sun.crypto.provider.TlsMasterSecretGenerator$TlsMasterSecretKey #4827
> >> <- sess - class: sun.security.ssl.SSLEngineImpl, value: sun.security.ssl.SSLSessionImpl #4829
> >> <- sslEngine - class: org.apache.http.nio.reactor.ssl.SSLIOSession, value: sun.security.ssl.SSLEngineImpl #20572
> >> <- bufferStatus - class: org.apache.http.impl.nio.reactor.IOSessionImpl, value: org.apache.http.nio.reactor.ssl.SSLIOSession #20572
> >> <- attachment - class: sun.nio.ch.SelectionKeyImpl, value: org.apache.http.impl.nio.reactor.IOSessionImpl #20572
> >> <- value - class: java.util.HashMap$Entry, value: sun.nio.ch.SelectionKeyImpl #20572
> >> <- [1182] - class: java.util.HashMap$Entry[], value: java.util.HashMap$Entry #254234
> >> <- table - class: java.util.HashMap, value: java.util.HashMap$Entry[] #6977
> >> <- fdToKey (Java frame) - class: sun.nio.ch.EPollSelectorImpl, value: java.util.HashMap #9595
> >>
> >> this - value: byte[] #71913
> >> <- iv - class: com.sun.crypto.provider.CipherBlockChaining, value: byte[] #71913
> >> <- cipher - class: com.sun.crypto.provider.CipherCore, value: com.sun.crypto.provider.CipherBlockChaining #3709
> >> <- core - class: com.sun.crypto.provider.AESCipher$General, value: com.sun.crypto.provider.CipherCore #3709
> >> <- spi - class: javax.crypto.Cipher, value: com.sun.crypto.provider.AESCipher$General #3709
> >> <- cipher - class: sun.security.ssl.CipherBox, value: javax.crypto.Cipher #3709
> >> <- readCipher - class: sun.security.ssl.SSLEngineImpl, value: sun.security.ssl.CipherBox #3710
> >> <- sslEngine - class: org.apache.http.nio.reactor.ssl.SSLIOSession, value: sun.security.ssl.SSLEngineImpl #1855
> >> <- bufferStatus - class: org.apache.http.impl.nio.reactor.IOSessionImpl, value: org.apache.http.nio.reactor.ssl.SSLIOSession #1855
> >> <- key - class: java.util.HashMap$Entry, value: org.apache.http.impl.nio.reactor.IOSessionImpl #1855
> >> <- [4000] - class: java.util.HashMap$Entry[], value: java.util.HashMap$Entry #71715
> >> <- table - class: java.util.HashMap, value: java.util.HashMap$Entry[] #24138
> >> <- map - class: java.util.HashSet, value: java.util.HashMap #8449
> >> <- c - class: java.util.Collections$SynchronizedSet, value: java.util.HashSet #3312
> >> <- sessions (Java frame) - class: org.apache.http.impl.nio.reactor.BaseIOReactor, value: java.util.Collections$SynchronizedSet #6
> >>
> >> I can give more details about the configuration of the HTTP Client and the SSL sessions if it would be helpful, but I don’t want to give too much info if this is expected behavior.
> >>
> >> Any advice would be appreciated.
> >>
> >> Thanks,
> >>
> >> Hal
> >>
> >
> >
> >
> >---------------------------------------------------------------------
> >To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> >For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Heap usage issue in Async Client
Posted by Harold Rosenberg <hr...@vmware.com>.
Using the ReleasableSSLBufferManagementStrategy gave a nice improvement. For the same run with 7500 simulated users, the change reduced the average live data in the heap after a full GC from 3.2GB to about 1.5GB. The amount of live data is still proportional to the number of connections (about 3GB of live data for a run with 15,000 simulated users), but that is to be expected.
Thanks for the help.
Hal
On 3/23/16, 3:59 PM, "Oleg Kalnichevski" <ol...@apache.org> wrote:
>On Wed, 2016-03-23 at 18:54 +0000, Harold Rosenberg wrote:
>> I am seeing unexpected behavior in the memory usage of a program that uses the Async Http Client. The issue arises when I switch from using exclusively HTTP connections to exclusively HTTPS connections. In the latter case, there is a large amount of memory used for byte arrays that seem to be related to SSL processing, and that aren’t ever cleaned up by garbage collection.
>>
>> My question is whether this is expected due to differences between SSL and non-SSL processing, or whether I am failing to properly clean up my SSL connections.
>>
>> The program is a workload driver for a performance benchmark. It simulates user activity to a web service. Each simulated user has about four active connections to the service. Normally the connections are all over SSL (really TLS), but for testing purposes there is a switch to use only non-SSL connections. The configuration of the HTTP client is identical for in both cases.
>>
>> I gathered garbage-collection data with and without TLS for runs with 7,500 simulated users, which would average about 30,000 active connections. A user would be active for about five minutes, at which point the they log out and a new user logs in. While the user is logged in the connections will occasionally be idle and thus closed due to keepalive timeout. The runs were two hours long and the data was collected during a steady-state. Data is generated in the heap at about the same rate in both cases, but when running with HTTPS there is about 3.2GB of live data left in the heap after a full garbage-collection, while with just HTTP there is only about 890MB. The amount of live data in the heap doesn’t grow once it reaches the 3.2GB mark, so it isn’t a memory leak, but it is definitely proportional to the number of active connections. When I do the same run with 15,000 users, at steady-state the live data in the heap after a full GC is about 6.4GB.
>>
>> Of the 3.2GB of live data, about 2.8GB is in 876,041 instances of byte[]. Almost all of the instances seem to be related to SSL processing. Here is the path to the nearest GC root for a few random instances:
>>
>
>Hi Harold,
>
>Could you try using ReleasableSSLBufferManagementStrategy instead of
>default PermanentSSLBufferManagementStrategy?
>
>---
>DefaultHostnameVerifier hostnameVerifier = new DefaultHostnameVerifier(PublicSuffixMatcherLoader.getDefault());
>Registry<SchemeIOSessionStrategy> registry = RegistryBuilder.<SchemeIOSessionStrategy>create()
> .register("http", NoopIOSessionStrategy.INSTANCE)
> .register("https", new SchemeIOSessionStrategy() {
> @Override
> public boolean isLayeringRequired() {
> return true;
> }
>
> @Override
> public IOSession upgrade(final HttpHost host, final IOSession iosession) throws IOException {
> final SSLIOSession ssliosession = new SSLIOSession(
> iosession,
> SSLMode.CLIENT,
> host,
> SSLContexts.createSystemDefault(),
> new SSLSetupHandler() {
>
> @Override
> public void initalize(final SSLEngine sslengine) throws SSLException {
> }
>
> @Override
> public void verify(
> final IOSession iosession,
> final SSLSession sslsession) throws SSLException {
> if (!hostnameVerifier.verify(host.getHostName(), sslsession)) {
> final Certificate[] certs = sslsession.getPeerCertificates();
> final X509Certificate x509 = (X509Certificate) certs[0];
> final X500Principal x500Principal = x509.getSubjectX500Principal();
> throw new SSLPeerUnverifiedException("Host name '" + host.getHostName() + "' does not match " +
> "the certificate subject provided by the peer (" + x500Principal.toString() + ")");
> }
> }
>
> },
> new ReleasableSSLBufferManagementStrategy());
> iosession.setAttribute(SSLIOSession.SESSION_KEY, ssliosession);
> ssliosession.initialize();
> return ssliosession;
> }
> })
> .build();
>
>ConnectingIOReactor ioReactor = new DefaultConnectingIOReactor();
>PoolingNHttpClientConnectionManager cm = new PoolingNHttpClientConnectionManager(ioReactor, registry);
>CloseableHttpAsyncClient httpclient = HttpAsyncClients.createMinimal(cm);
>try {
> httpclient.start();
> HttpGet request = new HttpGet("https://urldefense.proofpoint.com/v2/url?u=https-3A__verisign.com_&d=BQIDaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=9uYCHvq1QDKcMAIXCt6iT2etdTpbRn0toGoRKE6QxRo&m=HkvNV6bDDfPghsn_MF5SgU7KlMhzmAGjcfgAmviJCSs&s=w1-QtI1DaVx22Oo8UumpY0uvEblsK5t55MihZ_UwmPM&e= ");
> Future<HttpResponse> future = httpclient.execute(request, null);
> HttpResponse response = future.get();
> System.out.println("Response: " + response.getStatusLine());
> System.out.println("Shutting down");
>} finally {
> httpclient.close();
>}
>System.out.println("Done");
>---
>
>Oleg
>
>> this - value: byte[] #483799
>> <- hb - class: java.nio.HeapByteBuffer, value: byte[] #483799
>> <- buffer - class: org.apache.http.nio.reactor.ssl.PermanentSSLBufferManagementStrategy$InternalBuffer, value: java.nio.HeapByteBuffer #89456
>> <- outPlain - class: org.apache.http.nio.reactor.ssl.SSLIOSession, value: org.apache.http.nio.reactor.ssl.PermanentSSLBufferManagementStrategy$InternalBuffer #59632
>> <- bufferStatus - class: org.apache.http.impl.nio.reactor.IOSessionImpl, value: org.apache.http.nio.reactor.ssl.SSLIOSession #14910
>> <- attachment - class: sun.nio.ch.SelectionKeyImpl, value: org.apache.http.impl.nio.reactor.IOSessionImpl #14910
>> <- value - class: java.util.HashMap$Entry, value: sun.nio.ch.SelectionKeyImpl #14910
>> <- [355] - class: java.util.HashMap$Entry[], value: java.util.HashMap$Entry #197065
>> <- table - class: java.util.HashMap, value: java.util.HashMap$Entry[] #4612
>> <- fdToKey (Java frame) - class: sun.nio.ch.EPollSelectorImpl, value: java.util.HashMap #10016
>>
>> this - value: byte[] #60139
>> <- key - class: com.sun.crypto.provider.TlsMasterSecretGenerator$TlsMasterSecretKey, value: byte[] #60139
>> <- masterSecret - class: sun.security.ssl.SSLSessionImpl, value: com.sun.crypto.provider.TlsMasterSecretGenerator$TlsMasterSecretKey #4827
>> <- sess - class: sun.security.ssl.SSLEngineImpl, value: sun.security.ssl.SSLSessionImpl #4829
>> <- sslEngine - class: org.apache.http.nio.reactor.ssl.SSLIOSession, value: sun.security.ssl.SSLEngineImpl #20572
>> <- bufferStatus - class: org.apache.http.impl.nio.reactor.IOSessionImpl, value: org.apache.http.nio.reactor.ssl.SSLIOSession #20572
>> <- attachment - class: sun.nio.ch.SelectionKeyImpl, value: org.apache.http.impl.nio.reactor.IOSessionImpl #20572
>> <- value - class: java.util.HashMap$Entry, value: sun.nio.ch.SelectionKeyImpl #20572
>> <- [1182] - class: java.util.HashMap$Entry[], value: java.util.HashMap$Entry #254234
>> <- table - class: java.util.HashMap, value: java.util.HashMap$Entry[] #6977
>> <- fdToKey (Java frame) - class: sun.nio.ch.EPollSelectorImpl, value: java.util.HashMap #9595
>>
>> this - value: byte[] #71913
>> <- iv - class: com.sun.crypto.provider.CipherBlockChaining, value: byte[] #71913
>> <- cipher - class: com.sun.crypto.provider.CipherCore, value: com.sun.crypto.provider.CipherBlockChaining #3709
>> <- core - class: com.sun.crypto.provider.AESCipher$General, value: com.sun.crypto.provider.CipherCore #3709
>> <- spi - class: javax.crypto.Cipher, value: com.sun.crypto.provider.AESCipher$General #3709
>> <- cipher - class: sun.security.ssl.CipherBox, value: javax.crypto.Cipher #3709
>> <- readCipher - class: sun.security.ssl.SSLEngineImpl, value: sun.security.ssl.CipherBox #3710
>> <- sslEngine - class: org.apache.http.nio.reactor.ssl.SSLIOSession, value: sun.security.ssl.SSLEngineImpl #1855
>> <- bufferStatus - class: org.apache.http.impl.nio.reactor.IOSessionImpl, value: org.apache.http.nio.reactor.ssl.SSLIOSession #1855
>> <- key - class: java.util.HashMap$Entry, value: org.apache.http.impl.nio.reactor.IOSessionImpl #1855
>> <- [4000] - class: java.util.HashMap$Entry[], value: java.util.HashMap$Entry #71715
>> <- table - class: java.util.HashMap, value: java.util.HashMap$Entry[] #24138
>> <- map - class: java.util.HashSet, value: java.util.HashMap #8449
>> <- c - class: java.util.Collections$SynchronizedSet, value: java.util.HashSet #3312
>> <- sessions (Java frame) - class: org.apache.http.impl.nio.reactor.BaseIOReactor, value: java.util.Collections$SynchronizedSet #6
>>
>> I can give more details about the configuration of the HTTP Client and the SSL sessions if it would be helpful, but I don’t want to give too much info if this is expected behavior.
>>
>> Any advice would be appreciated.
>>
>> Thanks,
>>
>> Hal
>>
>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
Re: Heap usage issue in Async Client
Posted by Harold Rosenberg <hr...@vmware.com>.
Thanks for the suggestion. I’ll give this a try and let you know what happens.
Hal
On 3/23/16, 3:59 PM, "Oleg Kalnichevski" <ol...@apache.org> wrote:
>On Wed, 2016-03-23 at 18:54 +0000, Harold Rosenberg wrote:
>> I am seeing unexpected behavior in the memory usage of a program that uses the Async Http Client. The issue arises when I switch from using exclusively HTTP connections to exclusively HTTPS connections. In the latter case, there is a large amount of memory used for byte arrays that seem to be related to SSL processing, and that aren’t ever cleaned up by garbage collection.
>>
>> My question is whether this is expected due to differences between SSL and non-SSL processing, or whether I am failing to properly clean up my SSL connections.
>>
>> The program is a workload driver for a performance benchmark. It simulates user activity to a web service. Each simulated user has about four active connections to the service. Normally the connections are all over SSL (really TLS), but for testing purposes there is a switch to use only non-SSL connections. The configuration of the HTTP client is identical for in both cases.
>>
>> I gathered garbage-collection data with and without TLS for runs with 7,500 simulated users, which would average about 30,000 active connections. A user would be active for about five minutes, at which point the they log out and a new user logs in. While the user is logged in the connections will occasionally be idle and thus closed due to keepalive timeout. The runs were two hours long and the data was collected during a steady-state. Data is generated in the heap at about the same rate in both cases, but when running with HTTPS there is about 3.2GB of live data left in the heap after a full garbage-collection, while with just HTTP there is only about 890MB. The amount of live data in the heap doesn’t grow once it reaches the 3.2GB mark, so it isn’t a memory leak, but it is definitely proportional to the number of active connections. When I do the same run with 15,000 users, at steady-state the live data in the heap after a full GC is about 6.4GB.
>>
>> Of the 3.2GB of live data, about 2.8GB is in 876,041 instances of byte[]. Almost all of the instances seem to be related to SSL processing. Here is the path to the nearest GC root for a few random instances:
>>
>
>Hi Harold,
>
>Could you try using ReleasableSSLBufferManagementStrategy instead of
>default PermanentSSLBufferManagementStrategy?
>
>---
>DefaultHostnameVerifier hostnameVerifier = new DefaultHostnameVerifier(PublicSuffixMatcherLoader.getDefault());
>Registry<SchemeIOSessionStrategy> registry = RegistryBuilder.<SchemeIOSessionStrategy>create()
> .register("http", NoopIOSessionStrategy.INSTANCE)
> .register("https", new SchemeIOSessionStrategy() {
> @Override
> public boolean isLayeringRequired() {
> return true;
> }
>
> @Override
> public IOSession upgrade(final HttpHost host, final IOSession iosession) throws IOException {
> final SSLIOSession ssliosession = new SSLIOSession(
> iosession,
> SSLMode.CLIENT,
> host,
> SSLContexts.createSystemDefault(),
> new SSLSetupHandler() {
>
> @Override
> public void initalize(final SSLEngine sslengine) throws SSLException {
> }
>
> @Override
> public void verify(
> final IOSession iosession,
> final SSLSession sslsession) throws SSLException {
> if (!hostnameVerifier.verify(host.getHostName(), sslsession)) {
> final Certificate[] certs = sslsession.getPeerCertificates();
> final X509Certificate x509 = (X509Certificate) certs[0];
> final X500Principal x500Principal = x509.getSubjectX500Principal();
> throw new SSLPeerUnverifiedException("Host name '" + host.getHostName() + "' does not match " +
> "the certificate subject provided by the peer (" + x500Principal.toString() + ")");
> }
> }
>
> },
> new ReleasableSSLBufferManagementStrategy());
> iosession.setAttribute(SSLIOSession.SESSION_KEY, ssliosession);
> ssliosession.initialize();
> return ssliosession;
> }
> })
> .build();
>
>ConnectingIOReactor ioReactor = new DefaultConnectingIOReactor();
>PoolingNHttpClientConnectionManager cm = new PoolingNHttpClientConnectionManager(ioReactor, registry);
>CloseableHttpAsyncClient httpclient = HttpAsyncClients.createMinimal(cm);
>try {
> httpclient.start();
> HttpGet request = new HttpGet("https://urldefense.proofpoint.com/v2/url?u=https-3A__verisign.com_&d=BQIDaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=9uYCHvq1QDKcMAIXCt6iT2etdTpbRn0toGoRKE6QxRo&m=HkvNV6bDDfPghsn_MF5SgU7KlMhzmAGjcfgAmviJCSs&s=w1-QtI1DaVx22Oo8UumpY0uvEblsK5t55MihZ_UwmPM&e= ");
> Future<HttpResponse> future = httpclient.execute(request, null);
> HttpResponse response = future.get();
> System.out.println("Response: " + response.getStatusLine());
> System.out.println("Shutting down");
>} finally {
> httpclient.close();
>}
>System.out.println("Done");
>---
>
>Oleg
>
>> this - value: byte[] #483799
>> <- hb - class: java.nio.HeapByteBuffer, value: byte[] #483799
>> <- buffer - class: org.apache.http.nio.reactor.ssl.PermanentSSLBufferManagementStrategy$InternalBuffer, value: java.nio.HeapByteBuffer #89456
>> <- outPlain - class: org.apache.http.nio.reactor.ssl.SSLIOSession, value: org.apache.http.nio.reactor.ssl.PermanentSSLBufferManagementStrategy$InternalBuffer #59632
>> <- bufferStatus - class: org.apache.http.impl.nio.reactor.IOSessionImpl, value: org.apache.http.nio.reactor.ssl.SSLIOSession #14910
>> <- attachment - class: sun.nio.ch.SelectionKeyImpl, value: org.apache.http.impl.nio.reactor.IOSessionImpl #14910
>> <- value - class: java.util.HashMap$Entry, value: sun.nio.ch.SelectionKeyImpl #14910
>> <- [355] - class: java.util.HashMap$Entry[], value: java.util.HashMap$Entry #197065
>> <- table - class: java.util.HashMap, value: java.util.HashMap$Entry[] #4612
>> <- fdToKey (Java frame) - class: sun.nio.ch.EPollSelectorImpl, value: java.util.HashMap #10016
>>
>> this - value: byte[] #60139
>> <- key - class: com.sun.crypto.provider.TlsMasterSecretGenerator$TlsMasterSecretKey, value: byte[] #60139
>> <- masterSecret - class: sun.security.ssl.SSLSessionImpl, value: com.sun.crypto.provider.TlsMasterSecretGenerator$TlsMasterSecretKey #4827
>> <- sess - class: sun.security.ssl.SSLEngineImpl, value: sun.security.ssl.SSLSessionImpl #4829
>> <- sslEngine - class: org.apache.http.nio.reactor.ssl.SSLIOSession, value: sun.security.ssl.SSLEngineImpl #20572
>> <- bufferStatus - class: org.apache.http.impl.nio.reactor.IOSessionImpl, value: org.apache.http.nio.reactor.ssl.SSLIOSession #20572
>> <- attachment - class: sun.nio.ch.SelectionKeyImpl, value: org.apache.http.impl.nio.reactor.IOSessionImpl #20572
>> <- value - class: java.util.HashMap$Entry, value: sun.nio.ch.SelectionKeyImpl #20572
>> <- [1182] - class: java.util.HashMap$Entry[], value: java.util.HashMap$Entry #254234
>> <- table - class: java.util.HashMap, value: java.util.HashMap$Entry[] #6977
>> <- fdToKey (Java frame) - class: sun.nio.ch.EPollSelectorImpl, value: java.util.HashMap #9595
>>
>> this - value: byte[] #71913
>> <- iv - class: com.sun.crypto.provider.CipherBlockChaining, value: byte[] #71913
>> <- cipher - class: com.sun.crypto.provider.CipherCore, value: com.sun.crypto.provider.CipherBlockChaining #3709
>> <- core - class: com.sun.crypto.provider.AESCipher$General, value: com.sun.crypto.provider.CipherCore #3709
>> <- spi - class: javax.crypto.Cipher, value: com.sun.crypto.provider.AESCipher$General #3709
>> <- cipher - class: sun.security.ssl.CipherBox, value: javax.crypto.Cipher #3709
>> <- readCipher - class: sun.security.ssl.SSLEngineImpl, value: sun.security.ssl.CipherBox #3710
>> <- sslEngine - class: org.apache.http.nio.reactor.ssl.SSLIOSession, value: sun.security.ssl.SSLEngineImpl #1855
>> <- bufferStatus - class: org.apache.http.impl.nio.reactor.IOSessionImpl, value: org.apache.http.nio.reactor.ssl.SSLIOSession #1855
>> <- key - class: java.util.HashMap$Entry, value: org.apache.http.impl.nio.reactor.IOSessionImpl #1855
>> <- [4000] - class: java.util.HashMap$Entry[], value: java.util.HashMap$Entry #71715
>> <- table - class: java.util.HashMap, value: java.util.HashMap$Entry[] #24138
>> <- map - class: java.util.HashSet, value: java.util.HashMap #8449
>> <- c - class: java.util.Collections$SynchronizedSet, value: java.util.HashSet #3312
>> <- sessions (Java frame) - class: org.apache.http.impl.nio.reactor.BaseIOReactor, value: java.util.Collections$SynchronizedSet #6
>>
>> I can give more details about the configuration of the HTTP Client and the SSL sessions if it would be helpful, but I don’t want to give too much info if this is expected behavior.
>>
>> Any advice would be appreciated.
>>
>> Thanks,
>>
>> Hal
>>
>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Heap usage issue in Async Client
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Wed, 2016-03-23 at 18:54 +0000, Harold Rosenberg wrote:
> I am seeing unexpected behavior in the memory usage of a program that uses the Async Http Client. The issue arises when I switch from using exclusively HTTP connections to exclusively HTTPS connections. In the latter case, there is a large amount of memory used for byte arrays that seem to be related to SSL processing, and that aren’t ever cleaned up by garbage collection.
>
> My question is whether this is expected due to differences between SSL and non-SSL processing, or whether I am failing to properly clean up my SSL connections.
>
> The program is a workload driver for a performance benchmark. It simulates user activity to a web service. Each simulated user has about four active connections to the service. Normally the connections are all over SSL (really TLS), but for testing purposes there is a switch to use only non-SSL connections. The configuration of the HTTP client is identical for in both cases.
>
> I gathered garbage-collection data with and without TLS for runs with 7,500 simulated users, which would average about 30,000 active connections. A user would be active for about five minutes, at which point the they log out and a new user logs in. While the user is logged in the connections will occasionally be idle and thus closed due to keepalive timeout. The runs were two hours long and the data was collected during a steady-state. Data is generated in the heap at about the same rate in both cases, but when running with HTTPS there is about 3.2GB of live data left in the heap after a full garbage-collection, while with just HTTP there is only about 890MB. The amount of live data in the heap doesn’t grow once it reaches the 3.2GB mark, so it isn’t a memory leak, but it is definitely proportional to the number of active connections. When I do the same run with 15,000 users, at steady-state the live data in the heap after a full GC is about 6.4GB.
>
> Of the 3.2GB of live data, about 2.8GB is in 876,041 instances of byte[]. Almost all of the instances seem to be related to SSL processing. Here is the path to the nearest GC root for a few random instances:
>
Hi Harold,
Could you try using ReleasableSSLBufferManagementStrategy instead of
default PermanentSSLBufferManagementStrategy?
---
DefaultHostnameVerifier hostnameVerifier = new DefaultHostnameVerifier(PublicSuffixMatcherLoader.getDefault());
Registry<SchemeIOSessionStrategy> registry = RegistryBuilder.<SchemeIOSessionStrategy>create()
.register("http", NoopIOSessionStrategy.INSTANCE)
.register("https", new SchemeIOSessionStrategy() {
@Override
public boolean isLayeringRequired() {
return true;
}
@Override
public IOSession upgrade(final HttpHost host, final IOSession iosession) throws IOException {
final SSLIOSession ssliosession = new SSLIOSession(
iosession,
SSLMode.CLIENT,
host,
SSLContexts.createSystemDefault(),
new SSLSetupHandler() {
@Override
public void initalize(final SSLEngine sslengine) throws SSLException {
}
@Override
public void verify(
final IOSession iosession,
final SSLSession sslsession) throws SSLException {
if (!hostnameVerifier.verify(host.getHostName(), sslsession)) {
final Certificate[] certs = sslsession.getPeerCertificates();
final X509Certificate x509 = (X509Certificate) certs[0];
final X500Principal x500Principal = x509.getSubjectX500Principal();
throw new SSLPeerUnverifiedException("Host name '" + host.getHostName() + "' does not match " +
"the certificate subject provided by the peer (" + x500Principal.toString() + ")");
}
}
},
new ReleasableSSLBufferManagementStrategy());
iosession.setAttribute(SSLIOSession.SESSION_KEY, ssliosession);
ssliosession.initialize();
return ssliosession;
}
})
.build();
ConnectingIOReactor ioReactor = new DefaultConnectingIOReactor();
PoolingNHttpClientConnectionManager cm = new PoolingNHttpClientConnectionManager(ioReactor, registry);
CloseableHttpAsyncClient httpclient = HttpAsyncClients.createMinimal(cm);
try {
httpclient.start();
HttpGet request = new HttpGet("https://verisign.com/");
Future<HttpResponse> future = httpclient.execute(request, null);
HttpResponse response = future.get();
System.out.println("Response: " + response.getStatusLine());
System.out.println("Shutting down");
} finally {
httpclient.close();
}
System.out.println("Done");
---
Oleg
> this - value: byte[] #483799
> <- hb - class: java.nio.HeapByteBuffer, value: byte[] #483799
> <- buffer - class: org.apache.http.nio.reactor.ssl.PermanentSSLBufferManagementStrategy$InternalBuffer, value: java.nio.HeapByteBuffer #89456
> <- outPlain - class: org.apache.http.nio.reactor.ssl.SSLIOSession, value: org.apache.http.nio.reactor.ssl.PermanentSSLBufferManagementStrategy$InternalBuffer #59632
> <- bufferStatus - class: org.apache.http.impl.nio.reactor.IOSessionImpl, value: org.apache.http.nio.reactor.ssl.SSLIOSession #14910
> <- attachment - class: sun.nio.ch.SelectionKeyImpl, value: org.apache.http.impl.nio.reactor.IOSessionImpl #14910
> <- value - class: java.util.HashMap$Entry, value: sun.nio.ch.SelectionKeyImpl #14910
> <- [355] - class: java.util.HashMap$Entry[], value: java.util.HashMap$Entry #197065
> <- table - class: java.util.HashMap, value: java.util.HashMap$Entry[] #4612
> <- fdToKey (Java frame) - class: sun.nio.ch.EPollSelectorImpl, value: java.util.HashMap #10016
>
> this - value: byte[] #60139
> <- key - class: com.sun.crypto.provider.TlsMasterSecretGenerator$TlsMasterSecretKey, value: byte[] #60139
> <- masterSecret - class: sun.security.ssl.SSLSessionImpl, value: com.sun.crypto.provider.TlsMasterSecretGenerator$TlsMasterSecretKey #4827
> <- sess - class: sun.security.ssl.SSLEngineImpl, value: sun.security.ssl.SSLSessionImpl #4829
> <- sslEngine - class: org.apache.http.nio.reactor.ssl.SSLIOSession, value: sun.security.ssl.SSLEngineImpl #20572
> <- bufferStatus - class: org.apache.http.impl.nio.reactor.IOSessionImpl, value: org.apache.http.nio.reactor.ssl.SSLIOSession #20572
> <- attachment - class: sun.nio.ch.SelectionKeyImpl, value: org.apache.http.impl.nio.reactor.IOSessionImpl #20572
> <- value - class: java.util.HashMap$Entry, value: sun.nio.ch.SelectionKeyImpl #20572
> <- [1182] - class: java.util.HashMap$Entry[], value: java.util.HashMap$Entry #254234
> <- table - class: java.util.HashMap, value: java.util.HashMap$Entry[] #6977
> <- fdToKey (Java frame) - class: sun.nio.ch.EPollSelectorImpl, value: java.util.HashMap #9595
>
> this - value: byte[] #71913
> <- iv - class: com.sun.crypto.provider.CipherBlockChaining, value: byte[] #71913
> <- cipher - class: com.sun.crypto.provider.CipherCore, value: com.sun.crypto.provider.CipherBlockChaining #3709
> <- core - class: com.sun.crypto.provider.AESCipher$General, value: com.sun.crypto.provider.CipherCore #3709
> <- spi - class: javax.crypto.Cipher, value: com.sun.crypto.provider.AESCipher$General #3709
> <- cipher - class: sun.security.ssl.CipherBox, value: javax.crypto.Cipher #3709
> <- readCipher - class: sun.security.ssl.SSLEngineImpl, value: sun.security.ssl.CipherBox #3710
> <- sslEngine - class: org.apache.http.nio.reactor.ssl.SSLIOSession, value: sun.security.ssl.SSLEngineImpl #1855
> <- bufferStatus - class: org.apache.http.impl.nio.reactor.IOSessionImpl, value: org.apache.http.nio.reactor.ssl.SSLIOSession #1855
> <- key - class: java.util.HashMap$Entry, value: org.apache.http.impl.nio.reactor.IOSessionImpl #1855
> <- [4000] - class: java.util.HashMap$Entry[], value: java.util.HashMap$Entry #71715
> <- table - class: java.util.HashMap, value: java.util.HashMap$Entry[] #24138
> <- map - class: java.util.HashSet, value: java.util.HashMap #8449
> <- c - class: java.util.Collections$SynchronizedSet, value: java.util.HashSet #3312
> <- sessions (Java frame) - class: org.apache.http.impl.nio.reactor.BaseIOReactor, value: java.util.Collections$SynchronizedSet #6
>
> I can give more details about the configuration of the HTTP Client and the SSL sessions if it would be helpful, but I don’t want to give too much info if this is expected behavior.
>
> Any advice would be appreciated.
>
> Thanks,
>
> Hal
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org