You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by René Buffat <bu...@gmail.com> on 2022/04/06 20:54:45 UTC
TLS quorum host name verification issue with docker-compose
Hello *
In order to get familiar with Zookeeper / Kafka I'm in the process of
creating a docker-compose based setup. The aim is that all communications
are secured using mTLS. Thereby I run into the following issue with the
host name verification for the zookeeper quorum.
In short, I currently plan to have 3 zookeeper nodes. The nodes have the
hostnames zookeeper, zookeeper2 and zookeeper3. The core issue is, that
while it is possible to ping the container using these names, the reverse
DNS lookup differs:
E.g.:
$ ping zookeeper2
PING zookeeper2 (172.21.0.4) 56(84) bytes of data.
64 bytes from kafka_ssl-zookeeper2-1.kafka_ssl_default (172.21.0.4):
icmp_seq=1 ttl=64 time=0.069 ms
$ dig -x 172.21.0.4
; <<>> DiG 9.11.5-P4-5.1+deb10u7-Debian <<>> -x 172.21.0.4
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 63658
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;4.0.21.172.in-addr.arpa. IN PTR
;; ANSWER SECTION:
4.0.21.172.in-addr.arpa. 600 IN PTR
kafka_ssl-zookeeper2-1.kafka_ssl_default.
;; Query time: 0 msec
;; SERVER: 127.0.0.11#53(127.0.0.11)
;; WHEN: Wed Apr 06 18:53:41 UTC 2022
;; MSG SIZE rcvd: 118
kafka_ssl-zookeeper2-1.kafka_ssl_default seems to represent <docker-compose
project name>-<container-name>-<id>.<docker-network-name>
When I use zookeeper, zookeeper2 or zookeeper3 as the common name in the
certificates, the zookeeper reports the following errors:
javax.net.ssl.SSLPeerUnverifiedException: Certificate for
<kafka_ssl-zookeeper2-1.kafka_ssl_default> doesn't match common name of the
certificate subject: zookeeper2
java.security.cert.CertificateException: Failed to verify both host address
and host name
When disabling host name verification using ssl.quorum.hostnameVerification
the setup seems to work however.
When using kafka_ssl-zookeeper2-1.kafka_ssl_default as common names in the
certificates, these names can not be resolved: java.net.SocketException:
Unresolved address
A fully working example to reproducing this issue (including the full logs,
script to generate the certificates etc.) can be found here:
https://github.com/rbuffat/kafka_playground/tree/main/kafka_ssl for the
zookeeper* as common name case and
https://github.com/rbuffat/kafka_playground/tree/main/kafka_ssl_v2 for the
case when using kafka_ssl-zookeeper2-1.kafka_ssl_defaultas the common name
case.
Any insights into what to do to resolve this issue is much appreciated.
lg rene
Re: TLS quorum host name verification issue with docker-compose
Posted by René Buffat <bu...@gmail.com>.
Thanks Shawn for your reply.
I now managed to get a working example running:
https://github.com/rbuffat/kafka_playground/tree/main/kafkasslv3
What I did is to configure docker-compose so that the resulting FQDN is
resolvable. Thus having a directory name (and thus the docker-compose
project name) without underscores and defining a custom network with
explicitly setting a name for it in the docker-compose.yml. This results
in that "kafka_ssl_v2-zookeeper-1.kafka_ssl_default2" becomes to
"kafkasslv3-zookeeper-1.kafka", which is resolvable.
All configuration options are set in the docker-compose.yml[1]. As docker
automatically assigns ip addresses to the containers the configuration only
uses host-names.
It is indeed strange. Especially as it seems to work with kafka (given
there is not something misconfigured there at the moment). The reverse DNS
is my best guess at the moment.
The zookeeper documentations mentions the following[2]:
> Please note that the alias (-alias) and the distinguished name (-dname)
must match the hostname of the machine that is associated with, otherwise
hostname verification won't work.
> keytool -genkeypair -alias $(hostname -f) -keyalg RSA -keysize 2048
-dname "cn=$(hostname -f)" -keypass password -keystore keystore.jks
-storepass password
But this also does not really work for docker containers as to my knowledge
docker uses the container ids as hostname which is not known beforehand:
$ hostname -f
a5ee07583d8b
Do you think it would warrant creating an issue in the zookeeper issue
tracker?
lg rene
[1]
https://github.com/rbuffat/kafka_playground/blob/main/kafka_ssl_v2/docker-compose.yml
[2] https://zookeeper.apache.org/doc/r3.8.0/zookeeperAdmin.html#Quorum+TLS
On Thu, 7 Apr 2022 at 06:09, Shawn Heisey <ap...@elyograg.org> wrote:
> On 2022-04-06 14:54, René Buffat wrote:
> > javax.net.ssl.SSLPeerUnverifiedException: Certificate for
> > <kafka_ssl-zookeeper2-1.kafka_ssl_default> doesn't match common name of
> > the
> > certificate subject: zookeeper2
> > java.security.cert.CertificateException: Failed to verify both host
> > address
> > and host name
>
> Generally speaking, I have never heard of anything SSL-related that
> looks at or cares about reverse DNS.
>
> What should matter is what the CN (or SAN, for certificates that handle
> multiple names) is, and what hostname the client is using to connect to
> the server. Those have to match. When using SSL, you do not want to
> specify an IP address for the host, you want to give it a name, because
> it is very unlikely that you'll see an IP address in a certificate
> unless you create it with a private CA.
>
> If reverse DNS is the only place that the longer name appears, then
> either the SSL verification is entirely too picky, or you have given the
> client an IP address for the server instead of a name, and it is looking
> up the reverse DNS so that it has a name to compare to the cert. I have
> no idea whether the ZK client (or maybe Java itself) does this, but it
> wouldn't surprise me.
>
> Thanks,
> Shawn
>
Re: TLS quorum host name verification issue with docker-compose
Posted by Shawn Heisey <ap...@elyograg.org>.
On 2022-04-06 14:54, René Buffat wrote:
> javax.net.ssl.SSLPeerUnverifiedException: Certificate for
> <kafka_ssl-zookeeper2-1.kafka_ssl_default> doesn't match common name of
> the
> certificate subject: zookeeper2
> java.security.cert.CertificateException: Failed to verify both host
> address
> and host name
Generally speaking, I have never heard of anything SSL-related that
looks at or cares about reverse DNS.
What should matter is what the CN (or SAN, for certificates that handle
multiple names) is, and what hostname the client is using to connect to
the server. Those have to match. When using SSL, you do not want to
specify an IP address for the host, you want to give it a name, because
it is very unlikely that you'll see an IP address in a certificate
unless you create it with a private CA.
If reverse DNS is the only place that the longer name appears, then
either the SSL verification is entirely too picky, or you have given the
client an IP address for the server instead of a name, and it is looking
up the reverse DNS so that it has a name to compare to the cert. I have
no idea whether the ZK client (or maybe Java itself) does this, but it
wouldn't surprise me.
Thanks,
Shawn