You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tomcat.apache.org by bu...@apache.org on 2019/06/22 20:21:42 UTC
[Bug 63524] New: Private key must be accompanied by certificate
chain
https://bz.apache.org/bugzilla/show_bug.cgi?id=63524
Bug ID: 63524
Summary: Private key must be accompanied by certificate chain
Product: Tomcat 8
Version: 8.5.40
Hardware: PC
OS: Linux
Status: NEW
Severity: normal
Priority: P2
Component: Connectors
Assignee: dev@tomcat.apache.org
Reporter: arnaud.kleinveld@gmail.com
Target Milestone: ----
Upgrade to 8.5.40 broke the SSL connector. Connecting at port 8443 causes a
connection time out. Catalina.out is reporting that is fails to initialise the
connector at port 8443 as follows:
<snip>
Caused by: org.apache.catalina.LifecycleException: Protocol handler
initialization failed
at
org.apache.catalina.connector.Connector.initInternal(Connector.java:995)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:107)
... 12 more
Caused by: java.lang.IllegalArgumentException: Private key must be accompanied
by certificate chain
at
org.apache.tomcat.util.net.AprEndpoint.createSSLContext(AprEndpoint.java:404)
at org.apache.tomcat.util.net.AprEndpoint.bind(AprEndpoint.java:368)
at
org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1105)
at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:581)
at
org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:68)
at
org.apache.catalina.connector.Connector.initInternal(Connector.java:993)
... 13 more
Caused by: java.lang.IllegalArgumentException: Private key must be accompanied
by certificate chain
at java.security.KeyStore.setKeyEntry(KeyStore.java:1136)
at
org.apache.tomcat.util.net.SSLUtilBase.getKeyManagers(SSLUtilBase.java:313)
at
org.apache.tomcat.util.net.openssl.OpenSSLUtil.getKeyManagers(OpenSSLUtil.java:105)
at
org.apache.tomcat.util.net.SSLUtilBase.createSSLContext(SSLUtilBase.java:239)
at
org.apache.tomcat.util.net.AprEndpoint.createSSLContext(AprEndpoint.java:402)
... 18 more
Tomcat is configured to serve two domain names on two connectors each on a
different ip address with the following configuration:
<Connector port="8443" address="172.30.0.186"
protocol="org.apache.coyote.http11.Http11AprProtocol"
maxThreads="200" SSLEnabled="true"
scheme="https" secure="true" SSLVerifyClient="optional"
SSLProtocol="TLSv1+TLSv1.1+TLSv1.2"
SSLCertificateFile="/etc/pki/tls/certs/domain1.crt"
SSLCACertificateFile="/etc/pki/tls/certs/comodo-ca-bundle.crt"
SSLCertificateKeyFile="/etc/pki/tls/private/domain1.key" />
<Connector port="8443" address="172.30.0.94"
protocol="org.apache.coyote.http11.Http11AprProtocol"
maxThreads="200" SSLEnabled="true"
scheme="https" secure="true" SSLVerifyClient="optional"
SSLProtocol="TLSv1+TLSv1.1+TLSv1.2"
SSLCertificateFile="/etc/pki/tls/certs/domain2.crt"
SSLCACertificateFile="/etc/pki/tls/certs/lets-encrypt-x1-cross-signed.pem"
SSLCertificateKeyFile="/etc/pki/tls/private/domain2.key" />
The above configuration was working fine until Tomcat upgrade to the latest AWS
provided version which is 8.5.40 built May 2 2019 18:02:51 UTC
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 63524] Private key must be accompanied by certificate chain
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=63524
--- Comment #17 from Mark Thomas <ma...@apache.org> ---
There are two separate issues here.
The first is that the mechanism we are using to translate keys and certs to a
common format internally is stricter than OpenSSL and requires a valid
certificate chain. I have a patch that allows fall-back to direct OpenSSL
configuration in this case. Alternatively, the issue can be worked-around by
installed the cert chain. In this instance it is the "Sectigo RSA DV Bundle"
from https://support.sectigo.com/Com_KnowledgeDetailPage?Id=kA01N000000rfBO
The second issue is that the mechanism we are using to translate keys and
certificates to a common format doesn't support PKCS#1. Annoyingly, everything
we need is in the JRE but in the non-public sun.security.util package. I have a
patch to add PKCS#1 support as well.
I need to tidy the patches up, fill in the i18n that I skipped over and then
I'll be in a position to commit this.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 63524] Private key must be accompanied by certificate chain
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=63524
--- Comment #12 from Arnaud Kleinveld <ar...@gmail.com> ---
Hi, sorry for my late reply. I have gone through various options but I don't
see how I can reproduce this error. The Apache httpd server is using the same
certificates without any issues. Is there some kind of verbose logging I can
filter on Tomcat that will help you?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 63524] Private key must be accompanied by certificate chain
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=63524
--- Comment #2 from Mark Thomas <ma...@apache.org> ---
That sounds very much like bug 62526. That should be fixed in 8.5.40 though.
Are you using AWS Elastic Beanstalk? If not can you provide more environment
details please. We may need to re-create this on AWS so we'll need to know
exactly which service.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 63524] Private key must be accompanied by certificate chain
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=63524
Mark Thomas <ma...@apache.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEEDINFO |NEW
--- Comment #16 from Mark Thomas <ma...@apache.org> ---
Thanks. I have received the reproduction information and I can reproduce this
issue. I'm looking into what is going wrong now.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 63524] Private key must be accompanied by certificate chain
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=63524
--- Comment #20 from Mark Thomas <ma...@apache.org> ---
Yes. For the Sertigo key/cert use certificateChainFile="/path/to/dv/bundle".
For the letsencrypt key, if you convert it from PKCS#1 to PKCS#8 (opensssl can
do this) that should be OK as well.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 63524] Private key must be accompanied by certificate chain
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=63524
Mark Thomas <ma...@apache.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution|--- |FIXED
--- Comment #18 from Mark Thomas <ma...@apache.org> ---
Fixed in:
- master for 9.0.23 onwards
- 8.5.x for 8.5.44 onwards
The fix isn't needed in 7.0.x since the JSSE and OpenSSL configurations remain
separate in 7.0.x
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 63524] Private key must be accompanied by certificate chain
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=63524
--- Comment #3 from Mark Thomas <ma...@apache.org> ---
Looking at this a bit more I haven't been able to reproduce it yet. I suspect
it is related to the cert files being used. Is it possible for you to create a
set of test files that reproduces the issue?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 63524] Private key must be accompanied by certificate chain
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=63524
--- Comment #13 from Mark Thomas <ma...@apache.org> ---
There isn't much in the way of logging to enable that would help here. What we
really need is a set of keys/certs to reproduce the issue.
The simple solution (send me the keys/certs you are having the issue with)
isn't really an option unless you really, really trust me since it means
handing over your current private key(s). Not something I'd recommend.
Are you able to recreate the issue on a test server, revoke the cert(s) and
then send me the files?
Another option would be to create a set of problematic keys/certs from scratch
with OpenSSL and then add the steps to re-create those test files to this
issue.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 63524] Private key must be accompanied by certificate chain
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=63524
--- Comment #4 from Mark Thomas <ma...@apache.org> ---
Success! From a certain point of view. I have been able to recreate this. You
will see this error if you certs are in DER rather than PEM format.
OpenSSL can handle DER quite happily but the code we added to enable you to
switch seamlessly between OpenSSL and JSSE only works with PEM.
As a minimum, we should be able to get DER certs working again with OpenSSL.
Getting them working with JSSE as well might be more of a challenge. Meanwhile
conversion to PEM format should get everything working for you.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 63524] Private key must be accompanied by certificate chain
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=63524
--- Comment #19 from Arnaud Kleinveld <ar...@gmail.com> ---
Great and thank you for your excellent support. Looking forward to AWS update
announcement. This may take a while I guess. Do I understand correctly that in
the meantime I can solve the problem by installing the named bundle and using
certs in another format than pkcs#?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 63524] Private key must be accompanied by certificate chain
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=63524
--- Comment #8 from Arnaud Kleinveld <ar...@gmail.com> ---
(In reply to Mark Thomas from comment #7)
> There should be lines of headers / footers in each file starting "---". Can
> you list all the headers and footers present in each file please (this
> should help to ID the format being used).
domain2.crt
-----BEGIN CERTIFICATE-----
<random chars>
-----END CERTIFICATE-----
domain2.key
-----BEGIN RSA PRIVATE KEY-----
<random chars>
-----END RSA PRIVATE KEY-----
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 63524] Private key must be accompanied by certificate chain
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=63524
--- Comment #21 from Arnaud Kleinveld <ar...@gmail.com> ---
(In reply to Mark Thomas from comment #20)
> Yes. For the Sertigo key/cert use certificateChainFile="/path/to/dv/bundle".
>
> For the letsencrypt key, if you convert it from PKCS#1 to PKCS#8 (opensssl
> can do this) that should be OK as well.
Because the tomcat server is using APR I configured the server with
SSLCACertificateFile. Is this correct or should the configuration be changed,
to NIO2 I just read?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 63524] Private key must be accompanied by certificate chain
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=63524
--- Comment #5 from Arnaud Kleinveld <ar...@gmail.com> ---
Hi Mark, thank you for your quick and comprehensive replies. I am not using
Beanstalk and have instead manually configured Tomcat. As far as I know my
certificates are in PEM format because I can read them as text files. I Googled
for a method to understand which type it is. According the article found DER
certificates should be in binary format.
Another thing I forgot to mention is that I have another server running httpd
using the same certificates and keys and that server works fine. Maybe that
helps.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 63524] Private key must be accompanied by certificate chain
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=63524
--- Comment #15 from Mark Thomas <ma...@apache.org> ---
My public keys for markt@apache.org are listed here:
http://people.apache.org/keys/committer/
The first one (A9C5 ...) is my preferred one.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 63524] Private key must be accompanied by certificate chain
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=63524
--- Comment #6 from Arnaud Kleinveld <ar...@gmail.com> ---
(In reply to Mark Thomas from comment #4)
> Success! From a certain point of view. I have been able to recreate this.
> You will see this error if you certs are in DER rather than PEM format.
>
> OpenSSL can handle DER quite happily but the code we added to enable you to
> switch seamlessly between OpenSSL and JSSE only works with PEM.
>
> As a minimum, we should be able to get DER certs working again with OpenSSL.
> Getting them working with JSSE as well might be more of a challenge.
> Meanwhile conversion to PEM format should get everything working for you.
I noticed a message in Tomcat catalina.out log regarding this that may help:
"The certificate [/etc/pki/tls/certs/domain2.crt] or its private key
[/etc/pki/tls/private/domain2.key] could not be processed using a JSSE key
manager and will be given directly to OpenSSL
On another note I also use Tomcat APR which is configured to use OpenSSL.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 63524] Private key must be accompanied by certificate chain
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=63524
--- Comment #9 from Mark Thomas <ma...@apache.org> ---
Thanks. That looks like a PEM encoded PKCS#1 key and a PEM encoded X509 cert.
When I start 8.5.40 with those I don't see the error you see. I've tested with
the oldest and latest versions of OpenSSL.
I think we are back to me asking you to provide a set of test files (key and
cert(s)) that reproduce the issue.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 63524] Private key must be accompanied by certificate chain
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=63524
--- Comment #1 from Arnaud Kleinveld <ar...@gmail.com> ---
The domain1 configuration is also in use by a httpd server which is working
fine. Upgraded lets-encrypt-x1-cross-signed.pem to
lets-encrypt-x3-cross-signed.pem for domain2, unfortunately that didn't make a
difference.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 63524] Private key must be accompanied by certificate chain
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=63524
--- Comment #10 from Christopher Schultz <ch...@christopherschultz.net> ---
I realize that this conversation is headed in another direction, but...
(In reply to Mark Thomas from comment #4)
> OpenSSL can handle DER quite happily but the code we added to enable you to
> switch seamlessly between OpenSSL and JSSE only works with PEM.
The hard part is detecting the DER file, not reading it. JSSE will happily read
a DER file in the same way it reads a PEM-encoded file:
FileInputStream fis = new FileInputStream("certificate.der");
CertificateFactory cf = CertificateFactory.getInstance("X.509");
Certificate cert = cf.generateCertificate(fis);
This will work either raw DER or PEM-encoded DER files. There are no hard and
fast rules for reading multiple certificates in DER format so I think that DER
files would have to be either single-cert only or we'd need to do some
digging-around to see how other software handles multiple certs without PEM
encoding.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 63524] Private key must be accompanied by certificate chain
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=63524
--- Comment #22 from Mark Thomas <ma...@apache.org> ---
No need to switch from APR/Native to NIO2.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 63524] Private key must be accompanied by certificate chain
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=63524
Mark Thomas <ma...@apache.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |NEEDINFO
--- Comment #11 from Mark Thomas <ma...@apache.org> ---
I've found various ways to trigger this error but I am not confident I have
found the way the Arnaud is triggering the error.
Moving this to NEEDINFO until we get a set of test keys/certs that reproduce
this.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 63524] Private key must be accompanied by certificate chain
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=63524
--- Comment #7 from Mark Thomas <ma...@apache.org> ---
There should be lines of headers / footers in each file starting "---". Can you
list all the headers and footers present in each file please (this should help
to ID the format being used).
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 63524] Private key must be accompanied by certificate chain
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=63524
--- Comment #14 from Arnaud Kleinveld <ar...@gmail.com> ---
Perhaps I can send to your email if you have a personal public key.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org