You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Michał Kapituła <mi...@comarch.pl> on 2011/05/04 11:03:35 UTC
Slow SSL connections after Tomcat 5 to 7 migration
Hi,
I've been experiencing trouble after migrating our company's web
application from Tomcat 5 to Tomcat 7. There's a significant slowdown
with https connections after the migration.
I'm aware that application has a flaw - there's a large number of http
request needed for generating a page (> 100 requests), but these are
mainly static content (css, js, images etc.) and only very limited (1-3)
ammount of requests are for dynamically generated content.
I've just tested the page generation time via local network (and
browser's cache off) and the page is being fully loaded in circa 2
seconds (with Firebug on) when using http, while loading the same page
using secure connection took as much as 10 times longer. It's extremely
weird for me, since I haven't noticed any significant differences
between Tomcat 5 and 7 approaches to SSL and the server.xml file is
pretty much the same.
Problems associated with server load, heavy user traffic or JVM params
are rather not an option here. The memory and CPU usage constantly stays
lower than with Tomcat 5.
Here are crucial parts of our server.xml file:
<Executor
name="tomcatThreadPool" namePrefix="catalina-exec-"
maxThreads="400" minSpareThreads="300" prestartminSpareThreads="true" />
<Connector port="80" protocol="HTTP/1.1" executor="tomcatThreadPool"
enableLookups="false" redirectPort="443" acceptCount="200"
tcpNoDelay="true" connectionTimeout="20000" maxKeepAliveRequests="200"
disableUploadTimeout="true" URIEncoding="UTF-8" compression="on"
compressableMimeType="application/javascript,text/html,text/xml,text/javascript,text/css,text/plain,application/json,text/json,application/xml"
/>
<Connector port="443" protocol="org.apache.coyote.http11.Http11Protocol"
executor="tomcatThreadPool"
address="192.168.100.100"
enableLookups="false" disableUploadTimeout="true"
keystorePass="XXX" keystoreFile="XXX"
acceptCount="200" scheme="https" secure="true" SSLEnabled="true"
clientAuth="true" sslProtocol="TLS"
truststoreFile="XXX" truststorePass="XXX"
URIEncoding="UTF-8" compression="on"
compressableMimeType="application/javascript,text/html,text/xml,text/javascript,text/css,text/plain,application/json,text/json,application/xml"
/>
<Connector port="443" protocol="org.apache.coyote.http11.Http11Protocol"
executor="tomcatThreadPool"
enableLookups="false" disableUploadTimeout="true"
acceptCount="200" scheme="https" secure="true" SSLEnabled="true"
address="192.168.100.101"
clientAuth="true" sslProtocol="TLS" keystorePass="XXX" keystoreFile="XXX"
truststoreFile="XXX" truststorePass="XXX"
URIEncoding="UTF-8" compression="on"
compressableMimeType="application/javascript,text/html,text/xml,text/javascript,text/css,text/plain,application/json,text/json,application/xml"
/>
Well, except for defining global thread pool nothing really changed
between our versions' configuration. I've experimented with Java Nio
Blocking Connector, but haven't noticed any significant improvement.
Also can't use APR Connector, because it's some kind of requirement that
we must use keystore files we already have.
Any ideas on how to improve SSL performance and what could possibly go
wrong?
Sorry for my English, for I'm not a native speaker :).
Regards,
MK
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Slow SSL connections after Tomcat 5 to 7 migration
Posted by Francis GALIEGUE <fg...@one2team.com>.
On Wed, May 4, 2011 at 15:49, Caldarale, Charles R
<Ch...@unisys.com> wrote:
>> From: Michał Kapituła [mailto:michal.kapitula@comarch.pl]
>> Subject: Slow SSL connections after Tomcat 5 to 7 migration
>
>> I've just tested the page generation time via local network (and
>> browser's cache off) and the page is being fully loaded in circa 2
>> seconds (with Firebug on) when using http, while loading the same
>> page using secure connection took as much as 10 times longer.
>
> If the CPU usage is not high during page generation over SSL, and the network isn't inducing delays, then the threads must be waiting for something. Can you take some thread dumps during the delay period and find out what they're stuck on? (My suspicion is reads from /dev/random, but I thought that was only during Tomcat initialization.)
>
> Also, is Tomcat front-ended by anything (e.g., httpd), or are you accessing it directly?
>
> Did anything else change when you upgraded Tomcat? For example, the JVM version?
>
Can it be the same problem that I encountered with the long deployment
time of the manager webapp, which is cured by changing the entropy
source file?
--
Francis Galiegue
ONE2TEAM
Ingénieur système
Mob : +33 (0) 683 877 875
Tel : +33 (0) 178 945 552
fge@one2team.com
40 avenue Raymond Poincaré
75116 Paris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Slow SSL connections after Tomcat 5 to 7 migration
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Michał Kapituła [mailto:michal.kapitula@comarch.pl]
> Subject: Re: Slow SSL connections after Tomcat 5 to 7 migration
> Really I don't know what to search for there (maybe you
> could give me some clues?).
Look to see what the threads are doing. Some will be waiting on monitors, some might actually be doing work, some might be stuck on I/O requests. You have to look at each call stack to see (a laborious process, unfortunately). It's often useful to take several thread dumps a few seconds apart and see what's changing and what isn't.
> Could you explain why reading from /dev/random possibly
> could be the case here?
Because /dev/random (unlike /dev/urandom) can wait an arbitrarily long time to gather enough randomness before returning data. This is especially problematic on virtual machines running as servers, which have very little going on to generate entropy.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
Re: Slow SSL connections after Tomcat 5 to 7 migration
Posted by Michał Kapituła <mi...@comarch.pl>.
W dniu 2011-05-05 18:50, Christopher Schultz pisze:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Michał,
>
> On 5/5/2011 10:21 AM, Michał Kapituła wrote:
>> I thought that seperate SSL Handshake could be made for each of the
>> requests, but I can't see such behaviour in the dumps. Only one or zero
>> threads commiting a handshake, others appear to autoaccept it (maybe not
>> a precise term).
> [snip]
>
>> No, nothing changed. Btw - the Tomcat version is 7.0.12 and JDK's is 6.0.14
> If you are using the BIO connector (which it looks like you are due to
> "org.apache.coyote.http11.Http11Protocol" in your<Connector>), then
> Tomcat doesn't have a great deal of control over the SSL handshake, etc.
> process. That's all up to the JRE itself since all of the SSL
> implementation comes from there. Tomcat can affect certain things like
> the ciphers used, etc. but basically everything is done by the runtime,
> not by Tomcat.
>
> Can you double-check to see if you are using the APR versions of the
> HTTP connectors? The best way to tell what's going on is to look at the
> startup messages when the connectors initialize themselves and emit a
> short log message to catalina.out. That will tell us what connector is
> actually being used.
Yes, I'm positive that I'm using Java BIO connector, not the APR one
(as I said before, I can't use one).
I've managed to run exact copy of production system's Tomcat on my own
machine, so I'll have an opportunity to run these apps in Eclipse,
experiment more with NIO connector and different server configurations,
look in the jdk and tomcat source and maybe I'll find what's taking so long.
Still I'm open for any new ideas from anyone.
Regards,
Michał
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Slow SSL connections after Tomcat 5 to 7 migration
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Michał,
On 5/5/2011 10:21 AM, Michał Kapituła wrote:
> I thought that seperate SSL Handshake could be made for each of the
> requests, but I can't see such behaviour in the dumps. Only one or zero
> threads commiting a handshake, others appear to autoaccept it (maybe not
> a precise term).
[snip]
> No, nothing changed. Btw - the Tomcat version is 7.0.12 and JDK's is 6.0.14
If you are using the BIO connector (which it looks like you are due to
"org.apache.coyote.http11.Http11Protocol" in your <Connector>), then
Tomcat doesn't have a great deal of control over the SSL handshake, etc.
process. That's all up to the JRE itself since all of the SSL
implementation comes from there. Tomcat can affect certain things like
the ciphers used, etc. but basically everything is done by the runtime,
not by Tomcat.
Can you double-check to see if you are using the APR versions of the
HTTP connectors? The best way to tell what's going on is to look at the
startup messages when the connectors initialize themselves and emit a
short log message to catalina.out. That will tell us what connector is
actually being used.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk3C1WgACgkQ9CaO5/Lv0PAUqQCfQ7hQtIx6VTlUTIm34BagDGMh
9gAAmwWcAqVhBhqt0CngBQT9fmwCX30I
=0pxy
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Slow SSL connections after Tomcat 5 to 7 migration
Posted by Michał Kapituła <mi...@comarch.pl>.
Hello,
I've made some thread dumps with https connection in progress.
W dniu 2011-05-04 15:49, Caldarale, Charles R pisze:
>> From: Michał Kapituła [mailto:michal.kapitula@comarch.pl]
>> Subject: Slow SSL connections after Tomcat 5 to 7 migration
>> I've just tested the page generation time via local network (and
>> browser's cache off) and the page is being fully loaded in circa 2
>> seconds (with Firebug on) when using http, while loading the same
>> page using secure connection took as much as 10 times longer.
> If the CPU usage is not high during page generation over SSL, and the network isn't inducing delays, then the threads must be waiting for something. Can you take some thread dumps during the delay period and find out what they're stuck on? (My suspicion is reads from /dev/random, but I thought that was only during Tomcat initialization.)
Really I don't know what to search for there (maybe you could give me
some clues?). Could you explain why reading from /dev/random possibly
could be the case here?
I thought that seperate SSL Handshake could be made for each of the
requests, but I can't see such behaviour in the dumps. Only one or zero
threads commiting a handshake, others appear to autoaccept it (maybe not
a precise term).
> Also, is Tomcat front-ended by anything (e.g., httpd), or are you accessing it directly?
No, it isn't.
> Did anything else change when you upgraded Tomcat? For example, the JVM version?
No, nothing changed. Btw - the Tomcat version is 7.0.12 and JDK's is 6.0.14
I'll update JDK tommorow, I'm not sure if it will change anything, but
won't hurt neither...
I'll appretiate any additional help from you guys. Maybe there's any
additional info I could provide to make things more clear? Please let me
know.
Regards,
Michał
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Slow SSL connections after Tomcat 5 to 7 migration
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Michał Kapituła [mailto:michal.kapitula@comarch.pl]
> Subject: Slow SSL connections after Tomcat 5 to 7 migration
> I've just tested the page generation time via local network (and
> browser's cache off) and the page is being fully loaded in circa 2
> seconds (with Firebug on) when using http, while loading the same
> page using secure connection took as much as 10 times longer.
If the CPU usage is not high during page generation over SSL, and the network isn't inducing delays, then the threads must be waiting for something. Can you take some thread dumps during the delay period and find out what they're stuck on? (My suspicion is reads from /dev/random, but I thought that was only during Tomcat initialization.)
Also, is Tomcat front-ended by anything (e.g., httpd), or are you accessing it directly?
Did anything else change when you upgraded Tomcat? For example, the JVM version?
> Sorry for my English, for I'm not a native speaker :).
It's better than that of many who are...
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.