You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by "James H. H. Lampert" <ja...@touchtonecorp.com> on 2017/10/16 16:57:39 UTC
Weird problem: browsers refusing to connect; more "fun with ciphers
clauses on the connector tag"
Can somebody explain what just happened?
This morning, we got a call from a customer whose Tomcat server (on
their own hardware) we administer.
It seems that suddenly, and without any advance warning, all but the
oldest browsers were refusing to connect to the server, without offering
any override option. Dozens of their users were "left out in the cold,"
and unable to do their work.
I ran some tests of my own. Firefox 17 (Mac), for which I've blocked all
updates, connected just fine; Chrome 60 (Mac) was refusing to connect,
with extreme prejudice. I tried the same with our own Tomcat server, and
both browsers connected without complaint.
I then ran both sites through https://www.ssllabs.com/ssltest, and found
that our own Tomcat supported cipher suites
> TLS_RSA_WITH_AES_128_CBC_SHA (0x2f)
> TLS_RSA_WITH_AES_256_CBC_SHA (0x35)
while theirs supported
> TLS_RSA_WITH_AES_128_CBC_SHA (0x2f)
> TLS_DHE_RSA_WITH_AES_128_CBC_SHA (0x33) DH 1024 bits FS WEAK
> TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013) ECDH secp256r1 (eq. 3072 bits RSA) FS
> TLS_RSA_WITH_AES_256_CBC_SHA (0x35)
> TLS_DHE_RSA_WITH_AES_256_CBC_SHA (0x39) DH 1024 bits FS WEAK
> TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014) ECDH secp521r1 (eq. 15360 bits RSA) FS
Both were running Tomcat 7, both under Java 1.6. So I compared the
connector tags in the respective server.xml files, found that ours had a
"ciphers" clause restricting it to the two listed for it, while theirs
did not.
The obvious next step was to try removing the two DH ciphers, so I gave
them a "ciphers" clause accepting the other four protocols, restarted
Tomcat, cleared my Chrome cache, and tried connecting again. Still
refusing to connect, with extreme prejudice. Even tried shutting Chrome
down, relaunching it, clearing the cache, shutting it down again, and
relaunching it again, and still no change in behavior.
Finally, when I gave them the same "ciphers" clause we have on our
server, and restarted Tomcat again, everything started working properly.
I'll also note that the "Handshake Simulation" section of the SSLLabs
test showed 12 "Server sent fatal alert: internal_error" results for
both the "no ciphers clause" case and the "ECDHE" enabled case, but only
one (IE8/XP) for the "same ciphers clause as our own server" case.
But that leaves me with three big unanswered questions:
1. Why would a browser care about unsupported protocols if supported
ones were available?
2. My understanding is that DH is old, and considered weak, while ECDHE
is new, and considered extremely strong. So why do the ECDHE ciphers
still get rejected with extreme prejudice?
3. (and excuse me for screaming my head off; I'm not screaming AT
anybody) WHY DID THIS "REJECTION WITH EXTREME PREJUDICE" HAPPEN WITH
NEARLY ALL OF THE USERS OVER A WEEKEND, WITH NO WARNING AT ALL?
--
JHHL
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: More, Re: Weird problem: browsers refusing to connect; more "fun
with ciphers clauses on the connector tag"
Posted by "James H. H. Lampert" <ja...@touchtonecorp.com>.
On 10/19/17, 10:02 AM, Christopher Schultz wrote:
> The browser tells the server what cipher suites it supports during the
> initial handshake, and the server decides which algorithm to use. The
> client doesn't try multiple different connections to see which one
> sticks. The server either replies saying "okay, were using "suite X"
> or "go away, we can't talk to each other because we don't have any
> algorithms in common".
That actually makes a certain amount of sense, if the JVM somehow
*thinks* it can handle certain ciphers (e.g., the ECDHE ciphers), but
can't. And if the Midrange box JVMs are deferring cipher implementation
to the OS, rather than implementing it internally, I can see how that
could happen. I'll note that the ones that have ECDHE ciphers enabled,
and aren't blowing up not only Chrome 60 but a bunch of simulated
browser handshakes, are all on a much newer revision of the OS, one
that, I think, supports ECDHE at the OS level, whereas the ones on which
I had to disable ECDHE are on an OS revision that does NOT support it at
the OS level.
Some pretty big assumptions, but if they're right, it would explain much.
--
JHHL
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: More, Re: Weird problem: browsers refusing to connect; more "fun
with ciphers clauses on the connector tag"
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
James,
On 10/16/17 2:35 PM, James H. H. Lampert wrote:
> I just got finished going through 20 other customer Tomcat
> installations we administer.
>
> First, I found that most of them were accepting the DHE ciphers
> I'd disabled on the problem installation, and SSLLabs was giving
> them bad ratings for doing so.
Just for completeness, there's nothing wrong with DHE other than that
it's slow. Your previous post showed that it was using "weak primes"
which IS a concern, but unfortunately there are very old clients that
don't support larger primes (I'm looking at YOU, Java 1.6) and so
options are limited.
> Second, I found that two of the other installations were accepting
> the ECDHE ciphers I'd disabled, and yet they were working just fine
> in Chrome 60 (one of them still on a "test" certificate that was
> self-signed, wrong domain, and expired! -- we can encourage them to
> pick a CA, and let us help them install a signed cert, but we can't
> MAKE them do it).
You should enable ECDHE. It's the best per-CPU-cycle key-agreement
algorithm available today. EC keys (for authentication) are a little
less useful than RSA these days, but if you can support both, I would
recommend doing ahead to support both.
A thought just occurred to me: are some clients using libtcnative/APR
and others not? I believe Tomcat still prefers APR/tcnative to the
Java NIO connector is the libraries are available. For Tomcat 7, the
configuration is sensitive to the type of connector being used, so if
you "disabled a cipher suite" and it seems to be still enabled (e.g.
ECDHE) that could be an explanation.
> That one is running on an AS/400 at V7R3, with a JVM that IDs in
> Manager as "jvmap3270sr10fp1-20170215_012.6"; the other one that's
> working fine with the ECDHE certs active is an AS/400 at V7R1, with
> a JVM that IDs as "jvmap3260sr16fp15-20151029_01."
I think most of that is going to be meaningless to us, unfortunately.
The output of "java -version" is going to be more useful.
> In all cases, SSL is via JSSE, not OpenSSL (we've never even heard
> of anybody getting Tomcat running via OpenSSL on an AS/400).
Okay, that's good to know.
> Could it be that the browser is trying to use the ECDHE ciphers,
> and something in the OS or the JVM is blowing up when it tries to
> use them?
The browser tells the server what cipher suites it supports during the
initial handshake, and the server decides which algorithm to use. The
client doesn't try multiple different connections to see which one
sticks. The server either replies saying "okay, were using "suite X"
or "go away, we can't talk to each other because we don't have any
algorithms in common".
Is this just happening at a single client? Oh, maybe not, because you
said you tried to use Google Chrome yourself and it didn't work. I was
thinking "maybe some group policy disabled a whole swath of cipher
suites in the org" but that's unlikely to be the case unless your
Chrome instance was also affected.
Is it possible that the browsers within the org are contacting a
DIFFERENT server than the Tomcat itself? Perhaps a load-balancer or
reverse-proxy that was broken, and it only *appears* like Tomcat is
broken? You ran SSLLabs tool against the site and it says it supports
RSA-AES/CBC-SHA which is definitely a decent algorithm (though GCM
should be preferred over CBC mode in nearly all cases).
Honestly, your server should be supporting more than just those two
protocols, but those two should be sufficient to establish a
connection with a server that only supports those two.
Did you know that Qualys/SSLLabs also has a *client test* as well? Try
pointing your Google Chrome instance at this page and see what it says:
https://www.ssllabs.com/ssltest/viewMyClient.html
I just tried it with my Firefox 56 on Macos and these two cipher
suites are supported (as well as many other, of course):
TLS_RSA_WITH_AES_128_CBC_SHA (0x2f)
TLS_RSA_WITH_AES_256_CBC_SHA (0x35)
I did the same with Chrome 60 (latest) and is says it supports a
virtually identical list of cipher suites that Firefox does. I didn't
check any old versions.
Maybe have someone from inside the client's org share their screen
with you and have them run SSLLabs test against the hostname as they
see it from within.
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlno2pEACgkQHPApP6U8
pFhDGBAAk9FcFLmeNy/tScYF9S5SB557qHpGoCfcHMvMTGVy3n8eKsqv2OPT3mSq
JwYxDdil2opagLh061Pa1TiNBMfXqWOMdQi573bBeHdWgMrAllfd+hDVSUSqHpIL
3AqhTZI2NDC1Yvg8mAnoF7vbVFSDVnGVs5duyAjcbCJ+nrjBGK+oGteFAC2RiTN5
e4fZmte01Gie+9ihAWeTLc4YQlJT/cygIA1l4EJO0vwJJonwaR/1QGa8PhPxnIgj
XXL5zgg3KS6MEf9ioyhZKEJDjCACigal0OIwOGpDHHHkHjwKOHJQ2eJsUz24Ub6U
TNCKq1MJJ7yZ1PsBQJZ3QsHZNvkJ6vcXOY70M+t6mvdDGYalw8xTcnUFcdbT2RbZ
O+Q/rT0RhTQJaOkg3Jivd81KYdfxhE1+DX8Xz6E7e4Ygpblwa0hZokGk9fosttkK
yAMRJqD6zuRda5XLhiaFKN3qUiJH27RKKMTdHPmwYk0DljD8oEatUqxBJWT6+jeY
y/7BDRMnRoSLVoWs4IevuGsIbeW0//NBMq8/vTDo/GfJnMDe6iZe3Yj0+0mlBO7C
rQ/5OABGZBTouVDSN6gyZcGmU7R6vpYviEdb0Fwe0wPaMVyAdeuQNERtiY/QR8Ik
STslaBriDzHEDAK/TWj7ZHUg+HgyRrNUIKx3lT9mGJ99WBlwaRg=
=F0Q0
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
More, Re: Weird problem: browsers refusing to connect; more "fun
with ciphers clauses on the connector tag"
Posted by "James H. H. Lampert" <ja...@touchtonecorp.com>.
I just got finished going through 20 other customer Tomcat installations
we administer.
First, I found that most of them were accepting the DHE ciphers I'd
disabled on the problem installation, and SSLLabs was giving them bad
ratings for doing so.
Second, I found that two of the other installations were accepting the
ECDHE ciphers I'd disabled, and yet they were working just fine in
Chrome 60 (one of them still on a "test" certificate that was
self-signed, wrong domain, and expired! -- we can encourage them to pick
a CA, and let us help them install a signed cert, but we can't MAKE them
do it).
That one is running on an AS/400 at V7R3, with a JVM that IDs in Manager
as "jvmap3270sr10fp1-20170215_012.6"; the other one that's working fine
with the ECDHE certs active is an AS/400 at V7R1, with a JVM that IDs as
"jvmap3260sr16fp15-20151029_01."
In all cases, SSL is via JSSE, not OpenSSL (we've never even heard of
anybody getting Tomcat running via OpenSSL on an AS/400).
Could it be that the browser is trying to use the ECDHE ciphers, and
something in the OS or the JVM is blowing up when it tries to use them?
--
JHHL
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Weird problem: browsers refusing to connect; more "fun with
ciphers clauses on the connector tag"
Posted by Mark Thomas <ma...@apache.org>.
On 16/10/17 17:57, James H. H. Lampert wrote:
> Can somebody explain what just happened?
>
> This morning, we got a call from a customer whose Tomcat server (on
> their own hardware) we administer.
>
> It seems that suddenly, and without any advance warning, all but the
> oldest browsers were refusing to connect to the server, without offering
> any override option. Dozens of their users were "left out in the cold,"
> and unable to do their work.
What changed?
> I ran some tests of my own. Firefox 17 (Mac), for which I've blocked all
> updates, connected just fine; Chrome 60 (Mac) was refusing to connect,
> with extreme prejudice. I tried the same with our own Tomcat server, and
> both browsers connected without complaint.
>
> I then ran both sites through https://www.ssllabs.com/ssltest, and found
> that our own Tomcat supported cipher suites
>
>> TLS_RSA_WITH_AES_128_CBC_SHA (0x2f)
>> TLS_RSA_WITH_AES_256_CBC_SHA (0x35)
>
> while theirs supported
>
>> TLS_RSA_WITH_AES_128_CBC_SHA (0x2f)
>> TLS_DHE_RSA_WITH_AES_128_CBC_SHA (0x33) DH 1024 bits FS WEAK
>> TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013) ECDH secp256r1 (eq. 3072
>> bits RSA) FS
>> TLS_RSA_WITH_AES_256_CBC_SHA (0x35)
>> TLS_DHE_RSA_WITH_AES_256_CBC_SHA (0x39) DH 1024 bits FS WEAK
>> TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014) ECDH secp521r1 (eq.
>> 15360 bits RSA) FS
>
> Both were running Tomcat 7, both under Java 1.6. So I compared the
> connector tags in the respective server.xml files, found that ours had a
> "ciphers" clause restricting it to the two listed for it, while theirs
> did not.
>
> The obvious next step was to try removing the two DH ciphers, so I gave
> them a "ciphers" clause accepting the other four protocols, restarted
> Tomcat, cleared my Chrome cache, and tried connecting again. Still
> refusing to connect, with extreme prejudice. Even tried shutting Chrome
> down, relaunching it, clearing the cache, shutting it down again, and
> relaunching it again, and still no change in behavior.
>
> Finally, when I gave them the same "ciphers" clause we have on our
> server, and restarted Tomcat again, everything started working properly.
>
> I'll also note that the "Handshake Simulation" section of the SSLLabs
> test showed 12 "Server sent fatal alert: internal_error" results for
> both the "no ciphers clause" case and the "ECDHE" enabled case, but only
> one (IE8/XP) for the "same ciphers clause as our own server" case.
>
> But that leaves me with three big unanswered questions:
>
> 1. Why would a browser care about unsupported protocols if supported
> ones were available?
It is the server that appears to be rejecting the connection. Without
knowing what has changed it is hard to speculate what might be
triggering this.
> 2. My understanding is that DH is old, and considered weak, while ECDHE
> is new, and considered extremely strong. So why do the ECDHE ciphers
> still get rejected with extreme prejudice?
See previous response.
> 3. (and excuse me for screaming my head off; I'm not screaming AT
> anybody) WHY DID THIS "REJECTION WITH EXTREME PREJUDICE" HAPPEN WITH
> NEARLY ALL OF THE USERS OVER A WEEKEND, WITH NO WARNING AT ALL?
What changed? Something must have. Client update? OS update? Java
update? Tomcat update? Certificate expiry?
Getting to the bottom of what changed would be the first step in finding
what went on over the weekend.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org