You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by "Brewer, Edward L" <le...@Vanderbilt.Edu> on 2014/10/07 20:30:38 UTC
Help with Apache Tomcat/7.0.53 SSL issue
To all,
I am using Apache Tomcat 7.0.53 and I am having an intermittent issue with SSL. I am currently running three environments (Dev, UAT, and Prod. Prod comprises 4 VMs (uname states version as "2.6.32-431.11.2.el6.x86_x86_64 GNU/Linux" ) with each containing a local version of Java [ Java(TM) SE Runtime Environment (build 1.7.0_55-b13) Java HotSpot(TM) 64-Bit Server VM (build 24.55-b03, mixed mode) ] As well Tomcat and Java are owned by the user running the app. The VMs are load balanced over two pair of LTMs (LTM1 balances node 1 and node 2; LTM2 balances node 3 and node 4). The test environment is scaled down to just one LTM with two nodes and development is just a single VM.
Now, when I deployed dev and test I did not have any issues with SSL.... everything went as planned. When I deployed into production, I started to get complaints about timeouts to the service. After much troubleshooting... we were able to discern, using curl, that in production the LTM was not getting a response back from the application (using TCPDUMP) intermittently. Our LTMs are configured to server as a SSL proxy. On the VM, TCPDUMP shows that traffic is being presented to the socket but there is no response. As far as I can tell the three environments (TOMCAT and JAVA) are the same. I find nothing in the logs from both access and catalina.out. When I restart the servers the problem goes away for about one hour then it comes back rapidly. Using top and sar I do not see any issues with operating system performance. Also, by going done to one node the problem persists. As well here are the options that are in setenv.sh
export JAVA_OPTS="$JAVA_OPTS\
-verbosegc\
-Xms256m\
-XX:+DisableExplicitGC\
-Xmx2g"
Here is the error that I see from curl
curl: (52) SSL read: error:00000000:lib(0):func(0):reason(0), errno 104
Help,
Lee Brewer
Lee Brewer | Application Developer | Information Technology | Vanderbilt University
lee.brewer@vanderbilt.edu | phone 615.343.2802 | it.vanderbilt.edu<http://it.vanderbilt.edu/>
[Vanderbilt IT logo]
Re: Help with Apache Tomcat/7.0.53 SSL issue
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Edward,
On 10/7/14 2:35 PM, Brewer, Edward L wrote:
> Oh... Here is the entry in our server.xml (probably the most
> important part)
>
> <Connector port="<Omitted>" address="<Omitted>"
> protocol="HTTP/1.1" SSLEnabled="true" maxThreads="150"
> scheme="https" secure="true" clientAuth="false"
> ciphers="SSL_RSA_WITH_RC4_128_MD5,SSL_RSA_WITH_RC4_128_SHA,SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA"
>
>
keyAlias="<omitted>"
> keystoreFile="/app001/shibboleth/idp/epass/current/credentials/idp.jks"
>
>
keystorePass="<omitted>" />
So you are using JSSE and haven't specified an sslProtocol, so you are
getting the default which is TLS (which, for Java, really means SSLv3,
TLSv1, TLSv1.1, and TLSv1.2).
You are specifying a very small number of cipher suites (only 3) so
perhaps that's the problem. Note that all your cipher suites start
with SSL_* and none with TLS_*. That's not in itself a problem, but
you are restricting your server to using old cipher suites and not
allowing new ones. You can find code in the archives to pull the list
of supported and enabled-by-default cipher suites for your JVM.
What happens if you lift the restriction on the ciphers list so that
JSSE will use its default set?
> Here is the error that I see from curl
>
> curl: (52) SSL read: error:00000000:lib(0):func(0):reason(0), errno
> 104
Try using "openssl s_client" -- it gives much more information about
the connection.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
iQIcBAEBCAAGBQJUR7P4AAoJEBzwKT+lPKRY1SEP/1A+8i4Td8xD0xOcUe+P8oBK
wA6yjoo76MUqj4Nei0ZghXmzsrIUss/RsuazmLTJFTnJcEg3GThmjh1uKlHloUBR
2dFg6FhUDn4v+7P2sQiDuwtEd9oDx6aFA5j/DxSFCclnR7jq66vU0lxTjFdgd3jw
/G0dlF+iBnvBVEM2hojZAbv30qoIsxPAHXdsf7T13vcUQ/bVywmbqUPtoSR8hWzh
Mg+B+y7MEYJSUzeZf4JOqHuCe3nLHxOV7XNF7Mw5sZZ8DOvoay+tNU8mmeXmnHY0
zJe/4TICGz6BPYKaZNELwv8PiLZZ76mnu+c9I3Bcv3ZBC6D8p+yISA01apYOujgv
0Mfo9ilm/3E9dORHCX4497FyKLq6KjX3dPnlLD2G0YC7qRU6o1iA8pjFkbt38UgU
CeE8AMxu4sgQAyQVXkVlfs9T72JJmUdd3y+Jm5/WUreZoiTjS0gCEhwue9rUDOSo
B6wf7V971IlKQbbxMhpiqbf/2TsoS15REPviepsqCHXWVHxoOT/5etTN9V8vP2G6
fxeI4GaBIulGld+tNeVnR1Izi8sHz1GPYbGfD2zhwC1Br18MxiBdEtYQQI++LcTh
S2JdWtWmJBzgk/uHPB9Lm8oBwYplQYIHUPrF9XO3WJVBuThdeCDf9l5xfefSJktM
7aOx60/EkV878XIK/8Pm
=YDwk
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Help with Apache Tomcat/7.0.53 SSL issue
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Jeffrey Janner [mailto:Jeffrey.Janner@PolyDyne.com]
> Subject: RE: Help with Apache Tomcat/7.0.53 SSL issue
> Is 2g a valid value for -Xmx?
Yes, at least with the Sun/Oracle JVM. However, on 32-bit systems, that large a heap size will usually fail.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Help with Apache Tomcat/7.0.53 SSL issue
Posted by Jeffrey Janner <Je...@PolyDyne.com>.
> -----Original Message-----
> From: Brewer, Edward L [mailto:lee.brewer@Vanderbilt.Edu]
> Sent: Tuesday, October 07, 2014 1:36 PM
> To: Tomcat Users List
> Subject: RE: Help with Apache Tomcat/7.0.53 SSL issue
>
> To all,
>
>
> Oh... Here is the entry in our server.xml (probably the most important part)
>
> <Connector port="<Omitted>" address="<Omitted>" protocol="HTTP/1.1"
> SSLEnabled="true" maxThreads="150" scheme="https" secure="true"
> clientAuth="false"
> ciphers="SSL_RSA_WITH_RC4_128_MD5,SSL_RSA_WITH_RC4_128_SHA,SSL_
> DHE_RSA_WITH_3DES_EDE_CBC_SHA" keyAlias="<omitted>"
> keystoreFile="/app001/shibboleth/idp/epass/current/credentials/idp.jks"
> keystorePass="<omitted>" />
>
> <Connector port="<omitted>" address="<omitted>"
> protocol="org.apache.coyote.http11.Http11Protocol" maxthreads="150"
> scheme="https" SSLEnabled="true" secure="true" clientAuth="want"
> ciphers="SSL_RSA_WITH_RC4_128_MD5,SSL_RSA_WITH_RC4_128_SHA,SSL_
> DHE_RSA_WITH_3DES_EDE_CBC_SHA" keyAlias="<omitted>"
> keystoreFile="/app001/shibboleth/idp/epass/current/credentials/idp.jks"
> keystorePass="<omitted>" />
>
> Users connect directly to first listed connection.... The second SSL port is not
> currently used.
>
> Thanks,
> Lee
>
> From: Brewer, Edward L [mailto:lee.brewer@Vanderbilt.Edu]
> Sent: Tuesday, October 07, 2014 1:31 PM
> To: users@tomcat.apache.org
> Subject: Help with Apache Tomcat/7.0.53 SSL issue
>
> To all,
>
> I am using Apache Tomcat 7.0.53 and I am having an intermittent issue with
> SSL. I am currently running three environments (Dev, UAT, and Prod. Prod
> comprises 4 VMs (uname states version as "2.6.32-431.11.2.el6.x86_x86_64
> GNU/Linux" ) with each containing a local version of Java [ Java(TM) SE
> Runtime Environment (build 1.7.0_55-b13) Java HotSpot(TM) 64-Bit Server
> VM (build 24.55-b03, mixed mode) ] As well Tomcat and Java are owned by
> the user running the app. The VMs are load balanced over two pair of LTMs
> (LTM1 balances node 1 and node 2; LTM2 balances node 3 and node 4). The
> test environment is scaled down to just one LTM with two nodes and
> development is just a single VM.
>
> Now, when I deployed dev and test I did not have any issues with SSL....
> everything went as planned. When I deployed into production, I started to
> get complaints about timeouts to the service. After much troubleshooting...
> we were able to discern, using curl, that in production the LTM was not
> getting a response back from the application (using TCPDUMP)
> intermittently. Our LTMs are configured to server as a SSL proxy. On the
> VM, TCPDUMP shows that traffic is being presented to the socket but there
> is no response. As far as I can tell the three environments (TOMCAT and
> JAVA) are the same. I find nothing in the logs from both access and
> catalina.out. When I restart the servers the problem goes away for about
> one hour then it comes back rapidly. Using top and sar I do not see any
> issues with operating system performance. Also, by going done to one node
> the problem persists. As well here are the options that are in setenv.sh
>
> export JAVA_OPTS="$JAVA_OPTS\
> -verbosegc\
> -Xms256m\
> -XX:+DisableExplicitGC\
> -Xmx2g"
>
>
> Here is the error that I see from curl
>
> curl: (52) SSL read: error:00000000:lib(0):func(0):reason(0), errno 104
>
> Help,
> Lee Brewer
Lee, you say you checked the access & catalina logs, but did you check the stdout & stderr logs?
Since the problem goes away for about an hour after you restart, could you be having memory issues? Those are usually reported in the stderr log.
Is 2g a valid value for -Xmx? I've always specified it in terms of Megs, that is -Xmx2048m.
Jeff
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Help with Apache Tomcat/7.0.53 SSL issue
Posted by "Brewer, Edward L" <le...@Vanderbilt.Edu>.
To all,
Oh... Here is the entry in our server.xml (probably the most important part)
<Connector port="<Omitted>" address="<Omitted>" protocol="HTTP/1.1" SSLEnabled="true" maxThreads="150" scheme="https" secure="true" clientAuth="false" ciphers="SSL_RSA_WITH_RC4_128_MD5,SSL_RSA_WITH_RC4_128_SHA,SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA" keyAlias="<omitted>" keystoreFile="/app001/shibboleth/idp/epass/current/credentials/idp.jks" keystorePass="<omitted>" />
<Connector port="<omitted>" address="<omitted>" protocol="org.apache.coyote.http11.Http11Protocol" maxthreads="150" scheme="https" SSLEnabled="true" secure="true" clientAuth="want" ciphers="SSL_RSA_WITH_RC4_128_MD5,SSL_RSA_WITH_RC4_128_SHA,SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA" keyAlias="<omitted>" keystoreFile="/app001/shibboleth/idp/epass/current/credentials/idp.jks" keystorePass="<omitted>" />
Users connect directly to first listed connection.... The second SSL port is not currently used.
Thanks,
Lee
From: Brewer, Edward L [mailto:lee.brewer@Vanderbilt.Edu]
Sent: Tuesday, October 07, 2014 1:31 PM
To: users@tomcat.apache.org
Subject: Help with Apache Tomcat/7.0.53 SSL issue
To all,
I am using Apache Tomcat 7.0.53 and I am having an intermittent issue with SSL. I am currently running three environments (Dev, UAT, and Prod. Prod comprises 4 VMs (uname states version as "2.6.32-431.11.2.el6.x86_x86_64 GNU/Linux" ) with each containing a local version of Java [ Java(TM) SE Runtime Environment (build 1.7.0_55-b13) Java HotSpot(TM) 64-Bit Server VM (build 24.55-b03, mixed mode) ] As well Tomcat and Java are owned by the user running the app. The VMs are load balanced over two pair of LTMs (LTM1 balances node 1 and node 2; LTM2 balances node 3 and node 4). The test environment is scaled down to just one LTM with two nodes and development is just a single VM.
Now, when I deployed dev and test I did not have any issues with SSL.... everything went as planned. When I deployed into production, I started to get complaints about timeouts to the service. After much troubleshooting... we were able to discern, using curl, that in production the LTM was not getting a response back from the application (using TCPDUMP) intermittently. Our LTMs are configured to server as a SSL proxy. On the VM, TCPDUMP shows that traffic is being presented to the socket but there is no response. As far as I can tell the three environments (TOMCAT and JAVA) are the same. I find nothing in the logs from both access and catalina.out. When I restart the servers the problem goes away for about one hour then it comes back rapidly. Using top and sar I do not see any issues with operating system performance. Also, by going done to one node the problem persists. As well here are the options that are in setenv.sh
export JAVA_OPTS="$JAVA_OPTS\
-verbosegc\
-Xms256m\
-XX:+DisableExplicitGC\
-Xmx2g"
Here is the error that I see from curl
curl: (52) SSL read: error:00000000:lib(0):func(0):reason(0), errno 104
Help,
Lee Brewer
Lee Brewer | Application Developer | Information Technology | Vanderbilt University
lee.brewer@vanderbilt.edu<ma...@vanderbilt.edu> | phone 615.343.2802 | it.vanderbilt.edu<http://it.vanderbilt.edu/>
[Vanderbilt IT logo]