You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Jose Manuel Vega Monroy <jo...@williamhill.com> on 2019/10/28 07:09:25 UTC
Re: [EXTERNAL] SSL setup failing
@Peter
It looks like communication problem between brokers. But ensure:
1) Crtificates are valid and properly signed by root CA or intermediate one in the chain
2) Clients and brokers having private key and certificate in their keystore and properly configured to point to its path
3) Clients and brokers having CA certificates in the truststore and properly configured to point to its path
4) Clients and brokersbroker having root CA certificate in their keystore and properly configured to.point to its path
5) Permissions are right ones fro trustore and keystore
Thanks
Get Outlook for Android<https://aka.ms/ghei36>
________________________________
From: Péter Nagykátai <st...@gmail.com>
Sent: Monday, 28 October 2019, 00:13
To: users@kafka.apache.org
Subject: [EXTERNAL] SSL setup failing
Hi!
I'm experimenting with setting up a log ingesting cluster and Kafka would
be part of it. Unfortunately, I can't get the first broker started. I need
to secure the communication between a dozen nodes and Kaquiafka would only be
one part of it. I have a secured node where I generate certificates for
every server in the cluster (with an intermediate CA). AFAIK, I need to use
'.jks' files for Kafka, so I've generated a '.p12' file from the openssl
certificate and key then used `keytool` to generate a keystore:
`keytool -importkeystore -srckeystore kafka-1.p12 -srcstoretype PKCS12
-alias kafka-1 -destkeystore kafka-1.jks`
I generated a truststore for the root and intermediate chain as well:
`keytool -importcert -alias ca-root -keystore truststore.jks -file
ca-chain.cert.pem
Relevant part of the 'server.properties' configuration:
####
listeners=EXTERNAL://kafka-1:9092,INTERNAL://kafka-1:9093
advertised.listeners=EXTERNAL://kafka-1:9092,INTERNAL://kafka-1:9093
inter.broker.listener.name=INTERNAL
listener.security.protocol.map=EXTERNAL:SSL,INTERNAL:SSL
security.protocol=SSL
ssl.client.auth=required
ssl.truststore.location=/*******/truststore.jks
ssl.truststore.password=*************
ssl.keystore.location=/*******/kafka-1.jks
ssl.keystore.password=*************
####
After starting Kafka (as a service) I get the the following in the
'server.log':
>>...
>> INFO [KafkaServer id=1001] started (kafka.server.KafkaServer)
>> INFO [SocketServer brokerId=1001] Failed authentication with
/XXX.XXX.XXX.XXX (SSL handshake failed)
(org.apache.kafka.common.network.Selector)
>> INFO [Controller id=1001, targetBrokerId=1001] Failed authentication
with kafka-1/XXX.XXX.XXX.XXX (SSL handshake failed)
(org.apache.kafka.common.network.Selector)
>> ERROR [Controller id=1001, targetBrokerId=1001] Connection to node 1001
(kafka-1/XXX.XXX.XXX.XXX:9093) failed authentication due to: SSL handshake
failed (org.apache.kafka.clients.NetworkClient)
>>...
>> WARN SSL handshake failed (kafka.utils.CoreUtils$)
>> org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake
failed
>> Caused by: javax.net.ssl.SSLProtocolException: Unexpected handshake
message: server_hello
>>...
I couldn't find any lead with that error message and got stuck. Any ideas
what that error message means and how to solve it?
Specs:
- Ubuntu 18.04.3 LTS
- OpenJDK Runtime Environment (build 11.0.4+11-post-Ubuntu-1ubuntu218.04.3)
- Kafka 2.2.1 (from kafka_2.12-2.2.1.tgz)
- OpenSSL 1.1.1
Thank you!
Peter
Confidentiality: The contents of this e-mail and any attachments transmitted with it are intended to be confidential to the intended recipient; and may be privileged or otherwise protected from disclosure. If you are not an intended recipient of this e-mail, do not duplicate or redistribute it by any means. Please delete it and any attachments and notify the sender that you have received it in error. This e-mail is sent by a William Hill PLC group company. The William Hill group companies include, among others, William Hill PLC (registered number 4212563), William Hill Organization Limited (registered number 278208), William Hill US HoldCo Inc, WHG (International) Limited (registered number 99191) and Mr Green Limited (registered number C43260). Each of William Hill PLC and William Hill Organization Limited is registered in England and Wales and has its registered office at 1 Bedford Avenue, London, WC1B 3AU, UK. William Hill U.S. HoldCo, Inc. is registered in Delaware and has its registered office at 1007 N. Orange Street, 9 Floor, Wilmington, New Castle County DE 19801 Delaware, United States of America. WHG (International) Limited is registered in Gibraltar and has its registered office at 6/1 Waterport Place, Gibraltar. Mr Green Limited is registered in Malta and has its registered office at Tagliaferro Business Centre, Level 7, 14 High Street, Sliema SLM 1549, Malta. Unless specifically indicated otherwise, the contents of this e-mail are subject to contract; and are not an official statement, and do not necessarily represent the views, of William Hill PLC, its subsidiaries or affiliated companies. Please note that neither William Hill PLC, nor its subsidiaries and affiliated companies can accept any responsibility for any viruses contained within this e-mail and it is your responsibility to scan any emails and their attachments. William Hill PLC, its subsidiaries and affiliated companies may monitor e-mail traffic data and also the content of e-mails for effective operation of the e-mail system, or for security, purposes.
Re: [EXTERNAL] SSL setup failing
Posted by Péter Nagykátai <st...@gmail.com>.
SOLVED
@Jose
>If so, the review SSL conf related to that.
It turned out that in the SSL configuration file, the `extendedKeyUsage`
attribute was set to "serverAuth". So I extended it to "serverAuth,
clientAuth" which solved the problem. At the moment it seems everything
works as intended. The next step is to verify that every communication (
producers-Kafka cluster, Kafka inter-cluster, consumer-Kafka cluster,
Zookeeper-Kafka, Zookeeper inter-cluster) is indeed secured.
Thank you for your help!
On Mon, Oct 28, 2019 at 6:36 PM Jose Manuel Vega Monroy <
jose.monroy@williamhill.com> wrote:
> @Peter
>
> I have the feeling is related to client.auth required, in the end each
> broker is a client for the rest in the cluster.
>
> Try with client.auth=none, and check if the connect.
>
> If so, the review SSL conf related to that.
>
> Cheers
>
> Get Outlook for Android <https://aka.ms/ghei36>
> ------------------------------
> *From:* Péter Nagykátai <st...@gmail.com>
> *Sent:* Monday, October 28, 2019 2:47:38 PM
> *To:* users@kafka.apache.org <us...@kafka.apache.org>
> *Subject:* Re: [EXTERNAL] SSL setup failing
>
> @Jose
> >9092 is as well SSL protocol?
> Yes, it is. As you see in the config snippet from my initial email.
>
> > Zookeeper is connecting over SSL?
> Yes, at least as far as I can tell. It's set up there too but neither of
> those are making verifying that easy...
>
> >So then I would review all certificates to check if valid.
> I did that after your first response.
>
> >As well there is a Kafka broker property 'advertised.host.name' you could
> set with same hostname in the certificate.
> I added this property but didn't change anything, I get the exact same
> error messages.
>
> Thanks
>
> On Mon, Oct 28, 2019 at 2:08 PM Jose Manuel Vega Monroy <
> jose.monroy@williamhill.com> wrote:
>
> > @Peter
> >
> > 9092 is as well SSL protocol? Zookeeper is connecting over SSL?
> >
> > So then I would review all certificates to check if valid.
> >
> > As well there is a Kafka broker property 'advertised.host.name' you
> could
> > set with same hostname in the certificate.
> >
> > Thanks
> >
> > <http://www.williamhill.com/>
> > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.whenthefunstops.co.uk_&d=DwIFaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=EpV0EqDiDgDPupfjZMqzuAv4qAvJraWxyVssyKeT39o&s=nltRImLIvceytcuboTTrAYca_JBmNpjbPGfTEcexHVw&e=
> >
> > Jose Manuel Vega Monroy
> > Java Developer / Software Developer Engineer in Test
> > Direct: +0035 0 2008038 (Ext. 8038)
> > Email: jose.monroy@williamhill.com
> > William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA
> >
> >
> >
> >
> > On 28/10/2019, 14:03, "Péter Nagykátai" <st...@gmail.com> wrote:
> >
> > Sorry, if I was unclear before. I'm absolutely new to Kafka and how
> it
> > works.
> >
> > @Jose
> > >That happening when clients trying to SSL connect?
> > There are no clients at the moment just one Kafka broker which spews
> > the
> > errors in the server.log. To be specific, there is a ZooKeeper client
> > which
> > has no issues:
> > >> INFO [ZooKeeperClient] Connected.
> (kafka.zookeeper.ZooKeeperClient)
> >
> > @Manna
> > >Are you talking about local network loopback?
> > No, at least I don't think so. I'm simply trying to have the broker
> in
> > a
> > stable running state but after it starts, it tries to connect to the
> > listed
> > internal broker, which is itself since at the moment there aren't
> other
> > brokers in the cluster.
> >
> > >Also, have you tried ssl debug using openssl? What did you observe?
> > Per Jose's advice I checked the certificates I generated last week
> and
> > everything checked out on the 'rootCA' node.
> >
> > >How have you setup your signed certificates?
> > I have a secured node where I generate certificates for every server
> > in the
> > cluster (with an intermediate CA). Here are the commands I used:
> >
> > `openssl genrsa -out kafka-1.key.pem 2048`
> > `openssl req -config openssl_intermediate.cnf -key kafka-1.key.pem
> -new
> > -sha256 -out kafka-1.csr.pem`
> > `openssl ca -config openssl_intermediate.cnf -extensions server_cert
> > -days
> > 375 -notext -md sha256 -in kafka-1.csr.pem -out kafka-1.cert.pem`
> > `openssl pkcs12 -export -in kafka-1.cert.pem -inkey kafka-1.key.pem
> > -out
> > kafka-1.p12 -name kafka-1`
> > `keytool -importkeystore -srckeystore kafka-1.p12 -srcstoretype
> PKCS12
> > -alias kafka-1 -destkeystore kafka-1.jks`
> >
> > Also, for the root+intermediate chain:
> > `keytool -importcert -alias ca-root -keystore truststore.jks -file
> > ca-chain.cert.pem`
> >
> > >Does your CN/SAN matches with your advertised.listeners setup?
> > Yes.
> >
> > >Have you setup hostname verification correctly?
> > My Kafka configuration file only have the settings I pasted before,
> the
> > rest aren't network specific.
> >
> >
> > My (beginner) opinion is that Kafka tries to authenticate itself as a
> > client and gets confused when getting 'server_hello' message.
> > ("Unexpected
> > handshake message: server_hello")
> >
> > Thanks!
> >
> > On Mon, Oct 28, 2019 at 12:25 PM M. Manna <ma...@gmail.com>
> wrote:
> >
> > > Hi,
> > >
> > > not sure what it means "Tries to communicate with itself". Are you
> > talking
> > > about local network loopback?
> > >
> > > Also, have you tried ssl debug using openssl? What did you observe?
> > >
> > > The exception is handshake exception. This is quite common when
> your
> > cert
> > > validation fails. How have you setup your signed certificates? Does
> > your
> > > CN/SAN matches with your advertised.listeners setup? Have you setup
> > > hostname verification correctly?
> > >
> > > Thanks,
> > >
> > > On Mon, 28 Oct 2019 at 11:11, Péter Nagykátai <
> st4r.f1sch@gmail.com>
> > > wrote:
> > >
> > > > @Jose
> > > >
> > > > >It looks like communication problem between brokers.
> > > > As I mentioned, "I can't get the first broker started". The
> > message above
> > > > is from when the broker tries to communicate with "itself":
> > [Controller
> > > > id=1001, targetBrokerId=1001]).
> > > >
> > > > Nevertheless, I went through the checklist and everything is in
> > order.
> > > For
> > > > the first couple of tries, I got different SSL errors but I could
> > work
> > > > those out (that time I messed up the certificates), but now the
> > problem
> > > is:
> > > > >> Caused by: javax.net.ssl.SSLProtocolException: *Unexpected
> > handshake
> > > > **message:
> > > > server_hello*
> > > >
> > > > Peter
> > > >
> > > > On Mon, Oct 28, 2019 at 8:09 AM Jose Manuel Vega Monroy <
> > > > jose.monroy@williamhill.com> wrote:
> > > >
> > > > > @Peter
> > > > >
> > > > > It looks like communication problem between brokers. But
> ensure:
> > > > >
> > > > > 1) Crtificates are valid and properly signed by root CA or
> > intermediate
> > > > > one in the chain
> > > > > 2) Clients and brokers having private key and certificate in
> > their
> > > > > keystore and properly configured to point to its path
> > > > > 3) Clients and brokers having CA certificates in the truststore
> > and
> > > > > properly configured to point to its path
> > > > > 4) Clients and brokersbroker having root CA certificate in
> their
> > > keystore
> > > > > and properly configured to.point to its path
> > > > > 5) Permissions are right ones fro trustore and keystore
> > > > >
> > > > > Thanks
> > > > >
> > > > > Get Outlook for Android <
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__aka.ms_ghei36&d=DwIFaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=LbcoJotfup6doDvIgqyHQISw77V3SVM4PTr_r6Tbn-Y&s=M2vFRWGBcJ0-qtIHW1BhAMr8yANCnvghejnBHBuTeJA&e=
> > >
> > > > >
> > > > > ------------------------------
> > > > > *From:* Péter Nagykátai <st...@gmail.com>
> > > > > *Sent:* Monday, 28 October 2019, 00:13
> > > > > *To:* users@kafka.apache.org
> > > > > *Subject:* [EXTERNAL] SSL setup failing
> > > > >
> > > > > Hi!
> > > > >
> > > > > I'm experimenting with setting up a log ingesting cluster and
> > Kafka
> > > would
> > > > > be part of it. Unfortunately, I can't get the first broker
> > started. I
> > > > need
> > > > > to secure the communication between a dozen nodes and Kaquiafka
> > would
> > > > only
> > > > > be
> > > > > one part of it. I have a secured node where I generate
> > certificates for
> > > > > every server in the cluster (with an intermediate CA). AFAIK, I
> > need to
> > > > use
> > > > > '.jks' files for Kafka, so I've generated a '.p12' file from
> the
> > > openssl
> > > > > certificate and key then used `keytool` to generate a keystore:
> > > > > `keytool -importkeystore -srckeystore kafka-1.p12 -srcstoretype
> > PKCS12
> > > > > -alias kafka-1 -destkeystore kafka-1.jks`
> > > > > I generated a truststore for the root and intermediate chain as
> > well:
> > > > > `keytool -importcert -alias ca-root -keystore truststore.jks
> > -file
> > > > > ca-chain.cert.pem
> > > > >
> > > > > Relevant part of the 'server.properties' configuration:
> > > > > ####
> > > > > listeners=EXTERNAL://kafka-1:9092,INTERNAL://kafka-1:9093
> > > > >
> > advertised.listeners=EXTERNAL://kafka-1:9092,INTERNAL://kafka-1:9093
> > > > > inter.broker.listener.name=INTERNAL
> > > > > listener.security.protocol.map=EXTERNAL:SSL,INTERNAL:SSL
> > > > > security.protocol=SSL
> > > > > ssl.client.auth=required
> > > > > ssl.truststore.location=/*******/truststore.jks
> > > > > ssl.truststore.password=*************
> > > > > ssl.keystore.location=/*******/kafka-1.jks
> > > > > ssl.keystore.password=*************
> > > > > ####
> > > > >
> > > > > After starting Kafka (as a service) I get the the following in
> > the
> > > > > 'server.log':
> > > > > >>...
> > > > > >> INFO [KafkaServer id=1001] started
> (kafka.server.KafkaServer)
> > > > > >> INFO [SocketServer brokerId=1001] Failed authentication with
> > > > > /XXX.XXX.XXX.XXX (SSL handshake failed)
> > > > > (org.apache.kafka.common.network.Selector)
> > > > > >> INFO [Controller id=1001, targetBrokerId=1001] Failed
> > authentication
> > > > > with kafka-1/XXX.XXX.XXX.XXX (SSL handshake failed)
> > > > > (org.apache.kafka.common.network.Selector)
> > > > > >> ERROR [Controller id=1001, targetBrokerId=1001] Connection
> to
> > node
> > > > 1001
> > > > > (kafka-1/XXX.XXX.XXX.XXX:9093) failed authentication due to:
> SSL
> > > > handshake
> > > > > failed (org.apache.kafka.clients.NetworkClient)
> > > > > >>...
> > > > > >> WARN SSL handshake failed (kafka.utils.CoreUtils$)
> > > > > >> org.apache.kafka.common.errors.SslAuthenticationException:
> SSL
> > > > handshake
> > > > > failed
> > > > > >> Caused by: javax.net.ssl.SSLProtocolException: Unexpected
> > handshake
> > > > > message: server_hello
> > > > > >>...
> > > > >
> > > > > I couldn't find any lead with that error message and got stuck.
> > Any
> > > ideas
> > > > > what that error message means and how to solve it?
> > > > >
> > > > > Specs:
> > > > > - Ubuntu 18.04.3 LTS
> > > > > - OpenJDK Runtime Environment (build
> > > > 11.0.4+11-post-Ubuntu-1ubuntu218.04.3)
> > > > > - Kafka 2.2.1 (from kafka_2.12-2.2.1.tgz)
> > > > > - OpenSSL 1.1.1
> > > > >
> > > > > Thank you!
> > > > > Peter
> > > > >
> > > > > Confidentiality: The contents of this e-mail and any
> attachments
> > > > > transmitted with it are intended to be confidential to the
> > intended
> > > > > recipient; and may be privileged or otherwise protected from
> > > disclosure.
> > > > If
> > > > > you are not an intended recipient of this e-mail, do not
> > duplicate or
> > > > > redistribute it by any means. Please delete it and any
> > attachments and
> > > > > notify the sender that you have received it in error. This
> > e-mail is
> > > sent
> > > > > by a William Hill PLC group company. The William Hill group
> > companies
> > > > > include, among others, William Hill PLC (registered number
> > 4212563),
> > > > > William Hill Organization Limited (registered number 278208),
> > William
> > > > Hill
> > > > > US HoldCo Inc, WHG (International) Limited (registered number
> > 99191)
> > > and
> > > > Mr
> > > > > Green Limited (registered number C43260). Each of William Hill
> > PLC and
> > > > > William Hill Organization Limited is registered in England and
> > Wales
> > > and
> > > > > has its registered office at 1 Bedford Avenue, London, WC1B
> 3AU,
> > UK.
> > > > > William Hill U.S. HoldCo, Inc. is registered in Delaware and
> has
> > its
> > > > > registered office at 1007 N. Orange Street, 9 Floor,
> Wilmington,
> > New
> > > > Castle
> > > > > County DE 19801 Delaware, United States of America. WHG
> > (International)
> > > > > Limited is registered in Gibraltar and has its registered
> office
> > at 6/1
> > > > > Waterport Place, Gibraltar. Mr Green Limited is registered in
> > Malta and
> > > > has
> > > > > its registered office at Tagliaferro Business Centre, Level 7,
> > 14 High
> > > > > Street, Sliema SLM 1549, Malta. Unless specifically indicated
> > > otherwise,
> > > > > the contents of this e-mail are subject to contract; and are
> not
> > an
> > > > > official statement, and do not necessarily represent the views,
> > of
> > > > William
> > > > > Hill PLC, its subsidiaries or affiliated companies. Please note
> > that
> > > > > neither William Hill PLC, nor its subsidiaries and affiliated
> > companies
> > > > can
> > > > > accept any responsibility for any viruses contained within this
> > e-mail
> > > > and
> > > > > it is your responsibility to scan any emails and their
> > attachments.
> > > > William
> > > > > Hill PLC, its subsidiaries and affiliated companies may monitor
> > e-mail
> > > > > traffic data and also the content of e-mails for effective
> > operation of
> > > > the
> > > > > e-mail system, or for security, purposes.
> > > > >
> > > >
> > >
> >
> >
> > Confidentiality: The contents of this e-mail and any attachments
> > transmitted with it are intended to be confidential to the intended
> > recipient; and may be privileged or otherwise protected from disclosure.
> If
> > you are not an intended recipient of this e-mail, do not duplicate or
> > redistribute it by any means. Please delete it and any attachments and
> > notify the sender that you have received it in error. This e-mail is sent
> > by a William Hill PLC group company. The William Hill group companies
> > include, among others, William Hill PLC (registered number 4212563),
> > William Hill Organization Limited (registered number 278208), William
> Hill
> > US HoldCo Inc, WHG (International) Limited (registered number 99191) and
> Mr
> > Green Limited (registered number C43260). Each of William Hill PLC and
> > William Hill Organization Limited is registered in England and Wales and
> > has its registered office at 1 Bedford Avenue, London, WC1B 3AU, UK.
> > William Hill U.S. HoldCo, Inc. is registered in Delaware and has its
> > registered office at 1007 N. Orange Street, 9 Floor, Wilmington, New
> Castle
> > County DE 19801 Delaware, United States of America. WHG (International)
> > Limited is registered in Gibraltar and has its registered office at 6/1
> > Waterport Place, Gibraltar. Mr Green Limited is registered in Malta and
> has
> > its registered office at Tagliaferro Business Centre, Level 7, 14 High
> > Street, Sliema SLM 1549, Malta. Unless specifically indicated otherwise,
> > the contents of this e-mail are subject to contract; and are not an
> > official statement, and do not necessarily represent the views, of
> William
> > Hill PLC, its subsidiaries or affiliated companies. Please note that
> > neither William Hill PLC, nor its subsidiaries and affiliated companies
> can
> > accept any responsibility for any viruses contained within this e-mail
> and
> > it is your responsibility to scan any emails and their attachments.
> William
> > Hill PLC, its subsidiaries and affiliated companies may monitor e-mail
> > traffic data and also the content of e-mails for effective operation of
> the
> > e-mail system, or for security, purposes.
> >
> Confidentiality: The contents of this e-mail and any attachments
> transmitted with it are intended to be confidential to the intended
> recipient; and may be privileged or otherwise protected from disclosure. If
> you are not an intended recipient of this e-mail, do not duplicate or
> redistribute it by any means. Please delete it and any attachments and
> notify the sender that you have received it in error. This e-mail is sent
> by a William Hill PLC group company. The William Hill group companies
> include, among others, William Hill PLC (registered number 4212563),
> William Hill Organization Limited (registered number 278208), William Hill
> US HoldCo Inc, WHG (International) Limited (registered number 99191) and Mr
> Green Limited (registered number C43260). Each of William Hill PLC and
> William Hill Organization Limited is registered in England and Wales and
> has its registered office at 1 Bedford Avenue, London, WC1B 3AU, UK.
> William Hill U.S. HoldCo, Inc. is registered in Delaware and has its
> registered office at 1007 N. Orange Street, 9 Floor, Wilmington, New Castle
> County DE 19801 Delaware, United States of America. WHG (International)
> Limited is registered in Gibraltar and has its registered office at 6/1
> Waterport Place, Gibraltar. Mr Green Limited is registered in Malta and has
> its registered office at Tagliaferro Business Centre, Level 7, 14 High
> Street, Sliema SLM 1549, Malta. Unless specifically indicated otherwise,
> the contents of this e-mail are subject to contract; and are not an
> official statement, and do not necessarily represent the views, of William
> Hill PLC, its subsidiaries or affiliated companies. Please note that
> neither William Hill PLC, nor its subsidiaries and affiliated companies can
> accept any responsibility for any viruses contained within this e-mail and
> it is your responsibility to scan any emails and their attachments. William
> Hill PLC, its subsidiaries and affiliated companies may monitor e-mail
> traffic data and also the content of e-mails for effective operation of the
> e-mail system, or for security, purposes.
>
Re: [EXTERNAL] SSL setup failing
Posted by Jose Manuel Vega Monroy <jo...@williamhill.com>.
@Peter
I have the feeling is related to client.auth required, in the end each broker is a client for the rest in the cluster.
Try with client.auth=none, and check if the connect.
If so, the review SSL conf related to that.
Cheers
Get Outlook for Android<https://aka.ms/ghei36>
________________________________
From: Péter Nagykátai <st...@gmail.com>
Sent: Monday, October 28, 2019 2:47:38 PM
To: users@kafka.apache.org <us...@kafka.apache.org>
Subject: Re: [EXTERNAL] SSL setup failing
@Jose
>9092 is as well SSL protocol?
Yes, it is. As you see in the config snippet from my initial email.
> Zookeeper is connecting over SSL?
Yes, at least as far as I can tell. It's set up there too but neither of
those are making verifying that easy...
>So then I would review all certificates to check if valid.
I did that after your first response.
>As well there is a Kafka broker property 'advertised.host.name' you could
set with same hostname in the certificate.
I added this property but didn't change anything, I get the exact same
error messages.
Thanks
On Mon, Oct 28, 2019 at 2:08 PM Jose Manuel Vega Monroy <
jose.monroy@williamhill.com> wrote:
> @Peter
>
> 9092 is as well SSL protocol? Zookeeper is connecting over SSL?
>
> So then I would review all certificates to check if valid.
>
> As well there is a Kafka broker property 'advertised.host.name' you could
> set with same hostname in the certificate.
>
> Thanks
>
> <http://www.williamhill.com/>
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.whenthefunstops.co.uk_&d=DwIFaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=EpV0EqDiDgDPupfjZMqzuAv4qAvJraWxyVssyKeT39o&s=nltRImLIvceytcuboTTrAYca_JBmNpjbPGfTEcexHVw&e= >
> Jose Manuel Vega Monroy
> Java Developer / Software Developer Engineer in Test
> Direct: +0035 0 2008038 (Ext. 8038)
> Email: jose.monroy@williamhill.com
> William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA
>
>
>
>
> On 28/10/2019, 14:03, "Péter Nagykátai" <st...@gmail.com> wrote:
>
> Sorry, if I was unclear before. I'm absolutely new to Kafka and how it
> works.
>
> @Jose
> >That happening when clients trying to SSL connect?
> There are no clients at the moment just one Kafka broker which spews
> the
> errors in the server.log. To be specific, there is a ZooKeeper client
> which
> has no issues:
> >> INFO [ZooKeeperClient] Connected. (kafka.zookeeper.ZooKeeperClient)
>
> @Manna
> >Are you talking about local network loopback?
> No, at least I don't think so. I'm simply trying to have the broker in
> a
> stable running state but after it starts, it tries to connect to the
> listed
> internal broker, which is itself since at the moment there aren't other
> brokers in the cluster.
>
> >Also, have you tried ssl debug using openssl? What did you observe?
> Per Jose's advice I checked the certificates I generated last week and
> everything checked out on the 'rootCA' node.
>
> >How have you setup your signed certificates?
> I have a secured node where I generate certificates for every server
> in the
> cluster (with an intermediate CA). Here are the commands I used:
>
> `openssl genrsa -out kafka-1.key.pem 2048`
> `openssl req -config openssl_intermediate.cnf -key kafka-1.key.pem -new
> -sha256 -out kafka-1.csr.pem`
> `openssl ca -config openssl_intermediate.cnf -extensions server_cert
> -days
> 375 -notext -md sha256 -in kafka-1.csr.pem -out kafka-1.cert.pem`
> `openssl pkcs12 -export -in kafka-1.cert.pem -inkey kafka-1.key.pem
> -out
> kafka-1.p12 -name kafka-1`
> `keytool -importkeystore -srckeystore kafka-1.p12 -srcstoretype PKCS12
> -alias kafka-1 -destkeystore kafka-1.jks`
>
> Also, for the root+intermediate chain:
> `keytool -importcert -alias ca-root -keystore truststore.jks -file
> ca-chain.cert.pem`
>
> >Does your CN/SAN matches with your advertised.listeners setup?
> Yes.
>
> >Have you setup hostname verification correctly?
> My Kafka configuration file only have the settings I pasted before, the
> rest aren't network specific.
>
>
> My (beginner) opinion is that Kafka tries to authenticate itself as a
> client and gets confused when getting 'server_hello' message.
> ("Unexpected
> handshake message: server_hello")
>
> Thanks!
>
> On Mon, Oct 28, 2019 at 12:25 PM M. Manna <ma...@gmail.com> wrote:
>
> > Hi,
> >
> > not sure what it means "Tries to communicate with itself". Are you
> talking
> > about local network loopback?
> >
> > Also, have you tried ssl debug using openssl? What did you observe?
> >
> > The exception is handshake exception. This is quite common when your
> cert
> > validation fails. How have you setup your signed certificates? Does
> your
> > CN/SAN matches with your advertised.listeners setup? Have you setup
> > hostname verification correctly?
> >
> > Thanks,
> >
> > On Mon, 28 Oct 2019 at 11:11, Péter Nagykátai <st...@gmail.com>
> > wrote:
> >
> > > @Jose
> > >
> > > >It looks like communication problem between brokers.
> > > As I mentioned, "I can't get the first broker started". The
> message above
> > > is from when the broker tries to communicate with "itself":
> [Controller
> > > id=1001, targetBrokerId=1001]).
> > >
> > > Nevertheless, I went through the checklist and everything is in
> order.
> > For
> > > the first couple of tries, I got different SSL errors but I could
> work
> > > those out (that time I messed up the certificates), but now the
> problem
> > is:
> > > >> Caused by: javax.net.ssl.SSLProtocolException: *Unexpected
> handshake
> > > **message:
> > > server_hello*
> > >
> > > Peter
> > >
> > > On Mon, Oct 28, 2019 at 8:09 AM Jose Manuel Vega Monroy <
> > > jose.monroy@williamhill.com> wrote:
> > >
> > > > @Peter
> > > >
> > > > It looks like communication problem between brokers. But ensure:
> > > >
> > > > 1) Crtificates are valid and properly signed by root CA or
> intermediate
> > > > one in the chain
> > > > 2) Clients and brokers having private key and certificate in
> their
> > > > keystore and properly configured to point to its path
> > > > 3) Clients and brokers having CA certificates in the truststore
> and
> > > > properly configured to point to its path
> > > > 4) Clients and brokersbroker having root CA certificate in their
> > keystore
> > > > and properly configured to.point to its path
> > > > 5) Permissions are right ones fro trustore and keystore
> > > >
> > > > Thanks
> > > >
> > > > Get Outlook for Android <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__aka.ms_ghei36&d=DwIFaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=LbcoJotfup6doDvIgqyHQISw77V3SVM4PTr_r6Tbn-Y&s=M2vFRWGBcJ0-qtIHW1BhAMr8yANCnvghejnBHBuTeJA&e=
> >
> > > >
> > > > ------------------------------
> > > > *From:* Péter Nagykátai <st...@gmail.com>
> > > > *Sent:* Monday, 28 October 2019, 00:13
> > > > *To:* users@kafka.apache.org
> > > > *Subject:* [EXTERNAL] SSL setup failing
> > > >
> > > > Hi!
> > > >
> > > > I'm experimenting with setting up a log ingesting cluster and
> Kafka
> > would
> > > > be part of it. Unfortunately, I can't get the first broker
> started. I
> > > need
> > > > to secure the communication between a dozen nodes and Kaquiafka
> would
> > > only
> > > > be
> > > > one part of it. I have a secured node where I generate
> certificates for
> > > > every server in the cluster (with an intermediate CA). AFAIK, I
> need to
> > > use
> > > > '.jks' files for Kafka, so I've generated a '.p12' file from the
> > openssl
> > > > certificate and key then used `keytool` to generate a keystore:
> > > > `keytool -importkeystore -srckeystore kafka-1.p12 -srcstoretype
> PKCS12
> > > > -alias kafka-1 -destkeystore kafka-1.jks`
> > > > I generated a truststore for the root and intermediate chain as
> well:
> > > > `keytool -importcert -alias ca-root -keystore truststore.jks
> -file
> > > > ca-chain.cert.pem
> > > >
> > > > Relevant part of the 'server.properties' configuration:
> > > > ####
> > > > listeners=EXTERNAL://kafka-1:9092,INTERNAL://kafka-1:9093
> > > >
> advertised.listeners=EXTERNAL://kafka-1:9092,INTERNAL://kafka-1:9093
> > > > inter.broker.listener.name=INTERNAL
> > > > listener.security.protocol.map=EXTERNAL:SSL,INTERNAL:SSL
> > > > security.protocol=SSL
> > > > ssl.client.auth=required
> > > > ssl.truststore.location=/*******/truststore.jks
> > > > ssl.truststore.password=*************
> > > > ssl.keystore.location=/*******/kafka-1.jks
> > > > ssl.keystore.password=*************
> > > > ####
> > > >
> > > > After starting Kafka (as a service) I get the the following in
> the
> > > > 'server.log':
> > > > >>...
> > > > >> INFO [KafkaServer id=1001] started (kafka.server.KafkaServer)
> > > > >> INFO [SocketServer brokerId=1001] Failed authentication with
> > > > /XXX.XXX.XXX.XXX (SSL handshake failed)
> > > > (org.apache.kafka.common.network.Selector)
> > > > >> INFO [Controller id=1001, targetBrokerId=1001] Failed
> authentication
> > > > with kafka-1/XXX.XXX.XXX.XXX (SSL handshake failed)
> > > > (org.apache.kafka.common.network.Selector)
> > > > >> ERROR [Controller id=1001, targetBrokerId=1001] Connection to
> node
> > > 1001
> > > > (kafka-1/XXX.XXX.XXX.XXX:9093) failed authentication due to: SSL
> > > handshake
> > > > failed (org.apache.kafka.clients.NetworkClient)
> > > > >>...
> > > > >> WARN SSL handshake failed (kafka.utils.CoreUtils$)
> > > > >> org.apache.kafka.common.errors.SslAuthenticationException: SSL
> > > handshake
> > > > failed
> > > > >> Caused by: javax.net.ssl.SSLProtocolException: Unexpected
> handshake
> > > > message: server_hello
> > > > >>...
> > > >
> > > > I couldn't find any lead with that error message and got stuck.
> Any
> > ideas
> > > > what that error message means and how to solve it?
> > > >
> > > > Specs:
> > > > - Ubuntu 18.04.3 LTS
> > > > - OpenJDK Runtime Environment (build
> > > 11.0.4+11-post-Ubuntu-1ubuntu218.04.3)
> > > > - Kafka 2.2.1 (from kafka_2.12-2.2.1.tgz)
> > > > - OpenSSL 1.1.1
> > > >
> > > > Thank you!
> > > > Peter
> > > >
> > > > Confidentiality: The contents of this e-mail and any attachments
> > > > transmitted with it are intended to be confidential to the
> intended
> > > > recipient; and may be privileged or otherwise protected from
> > disclosure.
> > > If
> > > > you are not an intended recipient of this e-mail, do not
> duplicate or
> > > > redistribute it by any means. Please delete it and any
> attachments and
> > > > notify the sender that you have received it in error. This
> e-mail is
> > sent
> > > > by a William Hill PLC group company. The William Hill group
> companies
> > > > include, among others, William Hill PLC (registered number
> 4212563),
> > > > William Hill Organization Limited (registered number 278208),
> William
> > > Hill
> > > > US HoldCo Inc, WHG (International) Limited (registered number
> 99191)
> > and
> > > Mr
> > > > Green Limited (registered number C43260). Each of William Hill
> PLC and
> > > > William Hill Organization Limited is registered in England and
> Wales
> > and
> > > > has its registered office at 1 Bedford Avenue, London, WC1B 3AU,
> UK.
> > > > William Hill U.S. HoldCo, Inc. is registered in Delaware and has
> its
> > > > registered office at 1007 N. Orange Street, 9 Floor, Wilmington,
> New
> > > Castle
> > > > County DE 19801 Delaware, United States of America. WHG
> (International)
> > > > Limited is registered in Gibraltar and has its registered office
> at 6/1
> > > > Waterport Place, Gibraltar. Mr Green Limited is registered in
> Malta and
> > > has
> > > > its registered office at Tagliaferro Business Centre, Level 7,
> 14 High
> > > > Street, Sliema SLM 1549, Malta. Unless specifically indicated
> > otherwise,
> > > > the contents of this e-mail are subject to contract; and are not
> an
> > > > official statement, and do not necessarily represent the views,
> of
> > > William
> > > > Hill PLC, its subsidiaries or affiliated companies. Please note
> that
> > > > neither William Hill PLC, nor its subsidiaries and affiliated
> companies
> > > can
> > > > accept any responsibility for any viruses contained within this
> e-mail
> > > and
> > > > it is your responsibility to scan any emails and their
> attachments.
> > > William
> > > > Hill PLC, its subsidiaries and affiliated companies may monitor
> e-mail
> > > > traffic data and also the content of e-mails for effective
> operation of
> > > the
> > > > e-mail system, or for security, purposes.
> > > >
> > >
> >
>
>
> Confidentiality: The contents of this e-mail and any attachments
> transmitted with it are intended to be confidential to the intended
> recipient; and may be privileged or otherwise protected from disclosure. If
> you are not an intended recipient of this e-mail, do not duplicate or
> redistribute it by any means. Please delete it and any attachments and
> notify the sender that you have received it in error. This e-mail is sent
> by a William Hill PLC group company. The William Hill group companies
> include, among others, William Hill PLC (registered number 4212563),
> William Hill Organization Limited (registered number 278208), William Hill
> US HoldCo Inc, WHG (International) Limited (registered number 99191) and Mr
> Green Limited (registered number C43260). Each of William Hill PLC and
> William Hill Organization Limited is registered in England and Wales and
> has its registered office at 1 Bedford Avenue, London, WC1B 3AU, UK.
> William Hill U.S. HoldCo, Inc. is registered in Delaware and has its
> registered office at 1007 N. Orange Street, 9 Floor, Wilmington, New Castle
> County DE 19801 Delaware, United States of America. WHG (International)
> Limited is registered in Gibraltar and has its registered office at 6/1
> Waterport Place, Gibraltar. Mr Green Limited is registered in Malta and has
> its registered office at Tagliaferro Business Centre, Level 7, 14 High
> Street, Sliema SLM 1549, Malta. Unless specifically indicated otherwise,
> the contents of this e-mail are subject to contract; and are not an
> official statement, and do not necessarily represent the views, of William
> Hill PLC, its subsidiaries or affiliated companies. Please note that
> neither William Hill PLC, nor its subsidiaries and affiliated companies can
> accept any responsibility for any viruses contained within this e-mail and
> it is your responsibility to scan any emails and their attachments. William
> Hill PLC, its subsidiaries and affiliated companies may monitor e-mail
> traffic data and also the content of e-mails for effective operation of the
> e-mail system, or for security, purposes.
>
Confidentiality: The contents of this e-mail and any attachments transmitted with it are intended to be confidential to the intended recipient; and may be privileged or otherwise protected from disclosure. If you are not an intended recipient of this e-mail, do not duplicate or redistribute it by any means. Please delete it and any attachments and notify the sender that you have received it in error. This e-mail is sent by a William Hill PLC group company. The William Hill group companies include, among others, William Hill PLC (registered number 4212563), William Hill Organization Limited (registered number 278208), William Hill US HoldCo Inc, WHG (International) Limited (registered number 99191) and Mr Green Limited (registered number C43260). Each of William Hill PLC and William Hill Organization Limited is registered in England and Wales and has its registered office at 1 Bedford Avenue, London, WC1B 3AU, UK. William Hill U.S. HoldCo, Inc. is registered in Delaware and has its registered office at 1007 N. Orange Street, 9 Floor, Wilmington, New Castle County DE 19801 Delaware, United States of America. WHG (International) Limited is registered in Gibraltar and has its registered office at 6/1 Waterport Place, Gibraltar. Mr Green Limited is registered in Malta and has its registered office at Tagliaferro Business Centre, Level 7, 14 High Street, Sliema SLM 1549, Malta. Unless specifically indicated otherwise, the contents of this e-mail are subject to contract; and are not an official statement, and do not necessarily represent the views, of William Hill PLC, its subsidiaries or affiliated companies. Please note that neither William Hill PLC, nor its subsidiaries and affiliated companies can accept any responsibility for any viruses contained within this e-mail and it is your responsibility to scan any emails and their attachments. William Hill PLC, its subsidiaries and affiliated companies may monitor e-mail traffic data and also the content of e-mails for effective operation of the e-mail system, or for security, purposes.
Re: [EXTERNAL] SSL setup failing
Posted by Péter Nagykátai <st...@gmail.com>.
@Jose
>9092 is as well SSL protocol?
Yes, it is. As you see in the config snippet from my initial email.
> Zookeeper is connecting over SSL?
Yes, at least as far as I can tell. It's set up there too but neither of
those are making verifying that easy...
>So then I would review all certificates to check if valid.
I did that after your first response.
>As well there is a Kafka broker property 'advertised.host.name' you could
set with same hostname in the certificate.
I added this property but didn't change anything, I get the exact same
error messages.
Thanks
On Mon, Oct 28, 2019 at 2:08 PM Jose Manuel Vega Monroy <
jose.monroy@williamhill.com> wrote:
> @Peter
>
> 9092 is as well SSL protocol? Zookeeper is connecting over SSL?
>
> So then I would review all certificates to check if valid.
>
> As well there is a Kafka broker property 'advertised.host.name' you could
> set with same hostname in the certificate.
>
> Thanks
>
> <http://www.williamhill.com/>
> <http://www.whenthefunstops.co.uk/>
> Jose Manuel Vega Monroy
> Java Developer / Software Developer Engineer in Test
> Direct: +0035 0 2008038 (Ext. 8038)
> Email: jose.monroy@williamhill.com
> William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA
>
>
>
>
> On 28/10/2019, 14:03, "Péter Nagykátai" <st...@gmail.com> wrote:
>
> Sorry, if I was unclear before. I'm absolutely new to Kafka and how it
> works.
>
> @Jose
> >That happening when clients trying to SSL connect?
> There are no clients at the moment just one Kafka broker which spews
> the
> errors in the server.log. To be specific, there is a ZooKeeper client
> which
> has no issues:
> >> INFO [ZooKeeperClient] Connected. (kafka.zookeeper.ZooKeeperClient)
>
> @Manna
> >Are you talking about local network loopback?
> No, at least I don't think so. I'm simply trying to have the broker in
> a
> stable running state but after it starts, it tries to connect to the
> listed
> internal broker, which is itself since at the moment there aren't other
> brokers in the cluster.
>
> >Also, have you tried ssl debug using openssl? What did you observe?
> Per Jose's advice I checked the certificates I generated last week and
> everything checked out on the 'rootCA' node.
>
> >How have you setup your signed certificates?
> I have a secured node where I generate certificates for every server
> in the
> cluster (with an intermediate CA). Here are the commands I used:
>
> `openssl genrsa -out kafka-1.key.pem 2048`
> `openssl req -config openssl_intermediate.cnf -key kafka-1.key.pem -new
> -sha256 -out kafka-1.csr.pem`
> `openssl ca -config openssl_intermediate.cnf -extensions server_cert
> -days
> 375 -notext -md sha256 -in kafka-1.csr.pem -out kafka-1.cert.pem`
> `openssl pkcs12 -export -in kafka-1.cert.pem -inkey kafka-1.key.pem
> -out
> kafka-1.p12 -name kafka-1`
> `keytool -importkeystore -srckeystore kafka-1.p12 -srcstoretype PKCS12
> -alias kafka-1 -destkeystore kafka-1.jks`
>
> Also, for the root+intermediate chain:
> `keytool -importcert -alias ca-root -keystore truststore.jks -file
> ca-chain.cert.pem`
>
> >Does your CN/SAN matches with your advertised.listeners setup?
> Yes.
>
> >Have you setup hostname verification correctly?
> My Kafka configuration file only have the settings I pasted before, the
> rest aren't network specific.
>
>
> My (beginner) opinion is that Kafka tries to authenticate itself as a
> client and gets confused when getting 'server_hello' message.
> ("Unexpected
> handshake message: server_hello")
>
> Thanks!
>
> On Mon, Oct 28, 2019 at 12:25 PM M. Manna <ma...@gmail.com> wrote:
>
> > Hi,
> >
> > not sure what it means "Tries to communicate with itself". Are you
> talking
> > about local network loopback?
> >
> > Also, have you tried ssl debug using openssl? What did you observe?
> >
> > The exception is handshake exception. This is quite common when your
> cert
> > validation fails. How have you setup your signed certificates? Does
> your
> > CN/SAN matches with your advertised.listeners setup? Have you setup
> > hostname verification correctly?
> >
> > Thanks,
> >
> > On Mon, 28 Oct 2019 at 11:11, Péter Nagykátai <st...@gmail.com>
> > wrote:
> >
> > > @Jose
> > >
> > > >It looks like communication problem between brokers.
> > > As I mentioned, "I can't get the first broker started". The
> message above
> > > is from when the broker tries to communicate with "itself":
> [Controller
> > > id=1001, targetBrokerId=1001]).
> > >
> > > Nevertheless, I went through the checklist and everything is in
> order.
> > For
> > > the first couple of tries, I got different SSL errors but I could
> work
> > > those out (that time I messed up the certificates), but now the
> problem
> > is:
> > > >> Caused by: javax.net.ssl.SSLProtocolException: *Unexpected
> handshake
> > > **message:
> > > server_hello*
> > >
> > > Peter
> > >
> > > On Mon, Oct 28, 2019 at 8:09 AM Jose Manuel Vega Monroy <
> > > jose.monroy@williamhill.com> wrote:
> > >
> > > > @Peter
> > > >
> > > > It looks like communication problem between brokers. But ensure:
> > > >
> > > > 1) Crtificates are valid and properly signed by root CA or
> intermediate
> > > > one in the chain
> > > > 2) Clients and brokers having private key and certificate in
> their
> > > > keystore and properly configured to point to its path
> > > > 3) Clients and brokers having CA certificates in the truststore
> and
> > > > properly configured to point to its path
> > > > 4) Clients and brokersbroker having root CA certificate in their
> > keystore
> > > > and properly configured to.point to its path
> > > > 5) Permissions are right ones fro trustore and keystore
> > > >
> > > > Thanks
> > > >
> > > > Get Outlook for Android <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__aka.ms_ghei36&d=DwIFaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=LbcoJotfup6doDvIgqyHQISw77V3SVM4PTr_r6Tbn-Y&s=M2vFRWGBcJ0-qtIHW1BhAMr8yANCnvghejnBHBuTeJA&e=
> >
> > > >
> > > > ------------------------------
> > > > *From:* Péter Nagykátai <st...@gmail.com>
> > > > *Sent:* Monday, 28 October 2019, 00:13
> > > > *To:* users@kafka.apache.org
> > > > *Subject:* [EXTERNAL] SSL setup failing
> > > >
> > > > Hi!
> > > >
> > > > I'm experimenting with setting up a log ingesting cluster and
> Kafka
> > would
> > > > be part of it. Unfortunately, I can't get the first broker
> started. I
> > > need
> > > > to secure the communication between a dozen nodes and Kaquiafka
> would
> > > only
> > > > be
> > > > one part of it. I have a secured node where I generate
> certificates for
> > > > every server in the cluster (with an intermediate CA). AFAIK, I
> need to
> > > use
> > > > '.jks' files for Kafka, so I've generated a '.p12' file from the
> > openssl
> > > > certificate and key then used `keytool` to generate a keystore:
> > > > `keytool -importkeystore -srckeystore kafka-1.p12 -srcstoretype
> PKCS12
> > > > -alias kafka-1 -destkeystore kafka-1.jks`
> > > > I generated a truststore for the root and intermediate chain as
> well:
> > > > `keytool -importcert -alias ca-root -keystore truststore.jks
> -file
> > > > ca-chain.cert.pem
> > > >
> > > > Relevant part of the 'server.properties' configuration:
> > > > ####
> > > > listeners=EXTERNAL://kafka-1:9092,INTERNAL://kafka-1:9093
> > > >
> advertised.listeners=EXTERNAL://kafka-1:9092,INTERNAL://kafka-1:9093
> > > > inter.broker.listener.name=INTERNAL
> > > > listener.security.protocol.map=EXTERNAL:SSL,INTERNAL:SSL
> > > > security.protocol=SSL
> > > > ssl.client.auth=required
> > > > ssl.truststore.location=/*******/truststore.jks
> > > > ssl.truststore.password=*************
> > > > ssl.keystore.location=/*******/kafka-1.jks
> > > > ssl.keystore.password=*************
> > > > ####
> > > >
> > > > After starting Kafka (as a service) I get the the following in
> the
> > > > 'server.log':
> > > > >>...
> > > > >> INFO [KafkaServer id=1001] started (kafka.server.KafkaServer)
> > > > >> INFO [SocketServer brokerId=1001] Failed authentication with
> > > > /XXX.XXX.XXX.XXX (SSL handshake failed)
> > > > (org.apache.kafka.common.network.Selector)
> > > > >> INFO [Controller id=1001, targetBrokerId=1001] Failed
> authentication
> > > > with kafka-1/XXX.XXX.XXX.XXX (SSL handshake failed)
> > > > (org.apache.kafka.common.network.Selector)
> > > > >> ERROR [Controller id=1001, targetBrokerId=1001] Connection to
> node
> > > 1001
> > > > (kafka-1/XXX.XXX.XXX.XXX:9093) failed authentication due to: SSL
> > > handshake
> > > > failed (org.apache.kafka.clients.NetworkClient)
> > > > >>...
> > > > >> WARN SSL handshake failed (kafka.utils.CoreUtils$)
> > > > >> org.apache.kafka.common.errors.SslAuthenticationException: SSL
> > > handshake
> > > > failed
> > > > >> Caused by: javax.net.ssl.SSLProtocolException: Unexpected
> handshake
> > > > message: server_hello
> > > > >>...
> > > >
> > > > I couldn't find any lead with that error message and got stuck.
> Any
> > ideas
> > > > what that error message means and how to solve it?
> > > >
> > > > Specs:
> > > > - Ubuntu 18.04.3 LTS
> > > > - OpenJDK Runtime Environment (build
> > > 11.0.4+11-post-Ubuntu-1ubuntu218.04.3)
> > > > - Kafka 2.2.1 (from kafka_2.12-2.2.1.tgz)
> > > > - OpenSSL 1.1.1
> > > >
> > > > Thank you!
> > > > Peter
> > > >
> > > > Confidentiality: The contents of this e-mail and any attachments
> > > > transmitted with it are intended to be confidential to the
> intended
> > > > recipient; and may be privileged or otherwise protected from
> > disclosure.
> > > If
> > > > you are not an intended recipient of this e-mail, do not
> duplicate or
> > > > redistribute it by any means. Please delete it and any
> attachments and
> > > > notify the sender that you have received it in error. This
> e-mail is
> > sent
> > > > by a William Hill PLC group company. The William Hill group
> companies
> > > > include, among others, William Hill PLC (registered number
> 4212563),
> > > > William Hill Organization Limited (registered number 278208),
> William
> > > Hill
> > > > US HoldCo Inc, WHG (International) Limited (registered number
> 99191)
> > and
> > > Mr
> > > > Green Limited (registered number C43260). Each of William Hill
> PLC and
> > > > William Hill Organization Limited is registered in England and
> Wales
> > and
> > > > has its registered office at 1 Bedford Avenue, London, WC1B 3AU,
> UK.
> > > > William Hill U.S. HoldCo, Inc. is registered in Delaware and has
> its
> > > > registered office at 1007 N. Orange Street, 9 Floor, Wilmington,
> New
> > > Castle
> > > > County DE 19801 Delaware, United States of America. WHG
> (International)
> > > > Limited is registered in Gibraltar and has its registered office
> at 6/1
> > > > Waterport Place, Gibraltar. Mr Green Limited is registered in
> Malta and
> > > has
> > > > its registered office at Tagliaferro Business Centre, Level 7,
> 14 High
> > > > Street, Sliema SLM 1549, Malta. Unless specifically indicated
> > otherwise,
> > > > the contents of this e-mail are subject to contract; and are not
> an
> > > > official statement, and do not necessarily represent the views,
> of
> > > William
> > > > Hill PLC, its subsidiaries or affiliated companies. Please note
> that
> > > > neither William Hill PLC, nor its subsidiaries and affiliated
> companies
> > > can
> > > > accept any responsibility for any viruses contained within this
> e-mail
> > > and
> > > > it is your responsibility to scan any emails and their
> attachments.
> > > William
> > > > Hill PLC, its subsidiaries and affiliated companies may monitor
> e-mail
> > > > traffic data and also the content of e-mails for effective
> operation of
> > > the
> > > > e-mail system, or for security, purposes.
> > > >
> > >
> >
>
>
> Confidentiality: The contents of this e-mail and any attachments
> transmitted with it are intended to be confidential to the intended
> recipient; and may be privileged or otherwise protected from disclosure. If
> you are not an intended recipient of this e-mail, do not duplicate or
> redistribute it by any means. Please delete it and any attachments and
> notify the sender that you have received it in error. This e-mail is sent
> by a William Hill PLC group company. The William Hill group companies
> include, among others, William Hill PLC (registered number 4212563),
> William Hill Organization Limited (registered number 278208), William Hill
> US HoldCo Inc, WHG (International) Limited (registered number 99191) and Mr
> Green Limited (registered number C43260). Each of William Hill PLC and
> William Hill Organization Limited is registered in England and Wales and
> has its registered office at 1 Bedford Avenue, London, WC1B 3AU, UK.
> William Hill U.S. HoldCo, Inc. is registered in Delaware and has its
> registered office at 1007 N. Orange Street, 9 Floor, Wilmington, New Castle
> County DE 19801 Delaware, United States of America. WHG (International)
> Limited is registered in Gibraltar and has its registered office at 6/1
> Waterport Place, Gibraltar. Mr Green Limited is registered in Malta and has
> its registered office at Tagliaferro Business Centre, Level 7, 14 High
> Street, Sliema SLM 1549, Malta. Unless specifically indicated otherwise,
> the contents of this e-mail are subject to contract; and are not an
> official statement, and do not necessarily represent the views, of William
> Hill PLC, its subsidiaries or affiliated companies. Please note that
> neither William Hill PLC, nor its subsidiaries and affiliated companies can
> accept any responsibility for any viruses contained within this e-mail and
> it is your responsibility to scan any emails and their attachments. William
> Hill PLC, its subsidiaries and affiliated companies may monitor e-mail
> traffic data and also the content of e-mails for effective operation of the
> e-mail system, or for security, purposes.
>
Re: [EXTERNAL] SSL setup failing
Posted by Jose Manuel Vega Monroy <jo...@williamhill.com>.
@Peter
9092 is as well SSL protocol? Zookeeper is connecting over SSL?
So then I would review all certificates to check if valid.
As well there is a Kafka broker property 'advertised.host.name' you could set with same hostname in the certificate.
Thanks
<http://www.williamhill.com/>
<http://www.whenthefunstops.co.uk/>
Jose Manuel Vega Monroy
Java Developer / Software Developer Engineer in Test
Direct: +0035 0 2008038 (Ext. 8038)
Email: jose.monroy@williamhill.com
William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA
On 28/10/2019, 14:03, "Péter Nagykátai" <st...@gmail.com> wrote:
Sorry, if I was unclear before. I'm absolutely new to Kafka and how it
works.
@Jose
>That happening when clients trying to SSL connect?
There are no clients at the moment just one Kafka broker which spews the
errors in the server.log. To be specific, there is a ZooKeeper client which
has no issues:
>> INFO [ZooKeeperClient] Connected. (kafka.zookeeper.ZooKeeperClient)
@Manna
>Are you talking about local network loopback?
No, at least I don't think so. I'm simply trying to have the broker in a
stable running state but after it starts, it tries to connect to the listed
internal broker, which is itself since at the moment there aren't other
brokers in the cluster.
>Also, have you tried ssl debug using openssl? What did you observe?
Per Jose's advice I checked the certificates I generated last week and
everything checked out on the 'rootCA' node.
>How have you setup your signed certificates?
I have a secured node where I generate certificates for every server in the
cluster (with an intermediate CA). Here are the commands I used:
`openssl genrsa -out kafka-1.key.pem 2048`
`openssl req -config openssl_intermediate.cnf -key kafka-1.key.pem -new
-sha256 -out kafka-1.csr.pem`
`openssl ca -config openssl_intermediate.cnf -extensions server_cert -days
375 -notext -md sha256 -in kafka-1.csr.pem -out kafka-1.cert.pem`
`openssl pkcs12 -export -in kafka-1.cert.pem -inkey kafka-1.key.pem -out
kafka-1.p12 -name kafka-1`
`keytool -importkeystore -srckeystore kafka-1.p12 -srcstoretype PKCS12
-alias kafka-1 -destkeystore kafka-1.jks`
Also, for the root+intermediate chain:
`keytool -importcert -alias ca-root -keystore truststore.jks -file
ca-chain.cert.pem`
>Does your CN/SAN matches with your advertised.listeners setup?
Yes.
>Have you setup hostname verification correctly?
My Kafka configuration file only have the settings I pasted before, the
rest aren't network specific.
My (beginner) opinion is that Kafka tries to authenticate itself as a
client and gets confused when getting 'server_hello' message. ("Unexpected
handshake message: server_hello")
Thanks!
On Mon, Oct 28, 2019 at 12:25 PM M. Manna <ma...@gmail.com> wrote:
> Hi,
>
> not sure what it means "Tries to communicate with itself". Are you talking
> about local network loopback?
>
> Also, have you tried ssl debug using openssl? What did you observe?
>
> The exception is handshake exception. This is quite common when your cert
> validation fails. How have you setup your signed certificates? Does your
> CN/SAN matches with your advertised.listeners setup? Have you setup
> hostname verification correctly?
>
> Thanks,
>
> On Mon, 28 Oct 2019 at 11:11, Péter Nagykátai <st...@gmail.com>
> wrote:
>
> > @Jose
> >
> > >It looks like communication problem between brokers.
> > As I mentioned, "I can't get the first broker started". The message above
> > is from when the broker tries to communicate with "itself": [Controller
> > id=1001, targetBrokerId=1001]).
> >
> > Nevertheless, I went through the checklist and everything is in order.
> For
> > the first couple of tries, I got different SSL errors but I could work
> > those out (that time I messed up the certificates), but now the problem
> is:
> > >> Caused by: javax.net.ssl.SSLProtocolException: *Unexpected handshake
> > **message:
> > server_hello*
> >
> > Peter
> >
> > On Mon, Oct 28, 2019 at 8:09 AM Jose Manuel Vega Monroy <
> > jose.monroy@williamhill.com> wrote:
> >
> > > @Peter
> > >
> > > It looks like communication problem between brokers. But ensure:
> > >
> > > 1) Crtificates are valid and properly signed by root CA or intermediate
> > > one in the chain
> > > 2) Clients and brokers having private key and certificate in their
> > > keystore and properly configured to point to its path
> > > 3) Clients and brokers having CA certificates in the truststore and
> > > properly configured to point to its path
> > > 4) Clients and brokersbroker having root CA certificate in their
> keystore
> > > and properly configured to.point to its path
> > > 5) Permissions are right ones fro trustore and keystore
> > >
> > > Thanks
> > >
> > > Get Outlook for Android <https://urldefense.proofpoint.com/v2/url?u=https-3A__aka.ms_ghei36&d=DwIFaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=LbcoJotfup6doDvIgqyHQISw77V3SVM4PTr_r6Tbn-Y&s=M2vFRWGBcJ0-qtIHW1BhAMr8yANCnvghejnBHBuTeJA&e= >
> > >
> > > ------------------------------
> > > *From:* Péter Nagykátai <st...@gmail.com>
> > > *Sent:* Monday, 28 October 2019, 00:13
> > > *To:* users@kafka.apache.org
> > > *Subject:* [EXTERNAL] SSL setup failing
> > >
> > > Hi!
> > >
> > > I'm experimenting with setting up a log ingesting cluster and Kafka
> would
> > > be part of it. Unfortunately, I can't get the first broker started. I
> > need
> > > to secure the communication between a dozen nodes and Kaquiafka would
> > only
> > > be
> > > one part of it. I have a secured node where I generate certificates for
> > > every server in the cluster (with an intermediate CA). AFAIK, I need to
> > use
> > > '.jks' files for Kafka, so I've generated a '.p12' file from the
> openssl
> > > certificate and key then used `keytool` to generate a keystore:
> > > `keytool -importkeystore -srckeystore kafka-1.p12 -srcstoretype PKCS12
> > > -alias kafka-1 -destkeystore kafka-1.jks`
> > > I generated a truststore for the root and intermediate chain as well:
> > > `keytool -importcert -alias ca-root -keystore truststore.jks -file
> > > ca-chain.cert.pem
> > >
> > > Relevant part of the 'server.properties' configuration:
> > > ####
> > > listeners=EXTERNAL://kafka-1:9092,INTERNAL://kafka-1:9093
> > > advertised.listeners=EXTERNAL://kafka-1:9092,INTERNAL://kafka-1:9093
> > > inter.broker.listener.name=INTERNAL
> > > listener.security.protocol.map=EXTERNAL:SSL,INTERNAL:SSL
> > > security.protocol=SSL
> > > ssl.client.auth=required
> > > ssl.truststore.location=/*******/truststore.jks
> > > ssl.truststore.password=*************
> > > ssl.keystore.location=/*******/kafka-1.jks
> > > ssl.keystore.password=*************
> > > ####
> > >
> > > After starting Kafka (as a service) I get the the following in the
> > > 'server.log':
> > > >>...
> > > >> INFO [KafkaServer id=1001] started (kafka.server.KafkaServer)
> > > >> INFO [SocketServer brokerId=1001] Failed authentication with
> > > /XXX.XXX.XXX.XXX (SSL handshake failed)
> > > (org.apache.kafka.common.network.Selector)
> > > >> INFO [Controller id=1001, targetBrokerId=1001] Failed authentication
> > > with kafka-1/XXX.XXX.XXX.XXX (SSL handshake failed)
> > > (org.apache.kafka.common.network.Selector)
> > > >> ERROR [Controller id=1001, targetBrokerId=1001] Connection to node
> > 1001
> > > (kafka-1/XXX.XXX.XXX.XXX:9093) failed authentication due to: SSL
> > handshake
> > > failed (org.apache.kafka.clients.NetworkClient)
> > > >>...
> > > >> WARN SSL handshake failed (kafka.utils.CoreUtils$)
> > > >> org.apache.kafka.common.errors.SslAuthenticationException: SSL
> > handshake
> > > failed
> > > >> Caused by: javax.net.ssl.SSLProtocolException: Unexpected handshake
> > > message: server_hello
> > > >>...
> > >
> > > I couldn't find any lead with that error message and got stuck. Any
> ideas
> > > what that error message means and how to solve it?
> > >
> > > Specs:
> > > - Ubuntu 18.04.3 LTS
> > > - OpenJDK Runtime Environment (build
> > 11.0.4+11-post-Ubuntu-1ubuntu218.04.3)
> > > - Kafka 2.2.1 (from kafka_2.12-2.2.1.tgz)
> > > - OpenSSL 1.1.1
> > >
> > > Thank you!
> > > Peter
> > >
> > > Confidentiality: The contents of this e-mail and any attachments
> > > transmitted with it are intended to be confidential to the intended
> > > recipient; and may be privileged or otherwise protected from
> disclosure.
> > If
> > > you are not an intended recipient of this e-mail, do not duplicate or
> > > redistribute it by any means. Please delete it and any attachments and
> > > notify the sender that you have received it in error. This e-mail is
> sent
> > > by a William Hill PLC group company. The William Hill group companies
> > > include, among others, William Hill PLC (registered number 4212563),
> > > William Hill Organization Limited (registered number 278208), William
> > Hill
> > > US HoldCo Inc, WHG (International) Limited (registered number 99191)
> and
> > Mr
> > > Green Limited (registered number C43260). Each of William Hill PLC and
> > > William Hill Organization Limited is registered in England and Wales
> and
> > > has its registered office at 1 Bedford Avenue, London, WC1B 3AU, UK.
> > > William Hill U.S. HoldCo, Inc. is registered in Delaware and has its
> > > registered office at 1007 N. Orange Street, 9 Floor, Wilmington, New
> > Castle
> > > County DE 19801 Delaware, United States of America. WHG (International)
> > > Limited is registered in Gibraltar and has its registered office at 6/1
> > > Waterport Place, Gibraltar. Mr Green Limited is registered in Malta and
> > has
> > > its registered office at Tagliaferro Business Centre, Level 7, 14 High
> > > Street, Sliema SLM 1549, Malta. Unless specifically indicated
> otherwise,
> > > the contents of this e-mail are subject to contract; and are not an
> > > official statement, and do not necessarily represent the views, of
> > William
> > > Hill PLC, its subsidiaries or affiliated companies. Please note that
> > > neither William Hill PLC, nor its subsidiaries and affiliated companies
> > can
> > > accept any responsibility for any viruses contained within this e-mail
> > and
> > > it is your responsibility to scan any emails and their attachments.
> > William
> > > Hill PLC, its subsidiaries and affiliated companies may monitor e-mail
> > > traffic data and also the content of e-mails for effective operation of
> > the
> > > e-mail system, or for security, purposes.
> > >
> >
>
Confidentiality: The contents of this e-mail and any attachments transmitted with it are intended to be confidential to the intended recipient; and may be privileged or otherwise protected from disclosure. If you are not an intended recipient of this e-mail, do not duplicate or redistribute it by any means. Please delete it and any attachments and notify the sender that you have received it in error. This e-mail is sent by a William Hill PLC group company. The William Hill group companies include, among others, William Hill PLC (registered number 4212563), William Hill Organization Limited (registered number 278208), William Hill US HoldCo Inc, WHG (International) Limited (registered number 99191) and Mr Green Limited (registered number C43260). Each of William Hill PLC and William Hill Organization Limited is registered in England and Wales and has its registered office at 1 Bedford Avenue, London, WC1B 3AU, UK. William Hill U.S. HoldCo, Inc. is registered in Delaware and has its registered office at 1007 N. Orange Street, 9 Floor, Wilmington, New Castle County DE 19801 Delaware, United States of America. WHG (International) Limited is registered in Gibraltar and has its registered office at 6/1 Waterport Place, Gibraltar. Mr Green Limited is registered in Malta and has its registered office at Tagliaferro Business Centre, Level 7, 14 High Street, Sliema SLM 1549, Malta. Unless specifically indicated otherwise, the contents of this e-mail are subject to contract; and are not an official statement, and do not necessarily represent the views, of William Hill PLC, its subsidiaries or affiliated companies. Please note that neither William Hill PLC, nor its subsidiaries and affiliated companies can accept any responsibility for any viruses contained within this e-mail and it is your responsibility to scan any emails and their attachments. William Hill PLC, its subsidiaries and affiliated companies may monitor e-mail traffic data and also the content of e-mails for effective operation of the e-mail system, or for security, purposes.
Re: [EXTERNAL] SSL setup failing
Posted by Péter Nagykátai <st...@gmail.com>.
Sorry, if I was unclear before. I'm absolutely new to Kafka and how it
works.
@Jose
>That happening when clients trying to SSL connect?
There are no clients at the moment just one Kafka broker which spews the
errors in the server.log. To be specific, there is a ZooKeeper client which
has no issues:
>> INFO [ZooKeeperClient] Connected. (kafka.zookeeper.ZooKeeperClient)
@Manna
>Are you talking about local network loopback?
No, at least I don't think so. I'm simply trying to have the broker in a
stable running state but after it starts, it tries to connect to the listed
internal broker, which is itself since at the moment there aren't other
brokers in the cluster.
>Also, have you tried ssl debug using openssl? What did you observe?
Per Jose's advice I checked the certificates I generated last week and
everything checked out on the 'rootCA' node.
>How have you setup your signed certificates?
I have a secured node where I generate certificates for every server in the
cluster (with an intermediate CA). Here are the commands I used:
`openssl genrsa -out kafka-1.key.pem 2048`
`openssl req -config openssl_intermediate.cnf -key kafka-1.key.pem -new
-sha256 -out kafka-1.csr.pem`
`openssl ca -config openssl_intermediate.cnf -extensions server_cert -days
375 -notext -md sha256 -in kafka-1.csr.pem -out kafka-1.cert.pem`
`openssl pkcs12 -export -in kafka-1.cert.pem -inkey kafka-1.key.pem -out
kafka-1.p12 -name kafka-1`
`keytool -importkeystore -srckeystore kafka-1.p12 -srcstoretype PKCS12
-alias kafka-1 -destkeystore kafka-1.jks`
Also, for the root+intermediate chain:
`keytool -importcert -alias ca-root -keystore truststore.jks -file
ca-chain.cert.pem`
>Does your CN/SAN matches with your advertised.listeners setup?
Yes.
>Have you setup hostname verification correctly?
My Kafka configuration file only have the settings I pasted before, the
rest aren't network specific.
My (beginner) opinion is that Kafka tries to authenticate itself as a
client and gets confused when getting 'server_hello' message. ("Unexpected
handshake message: server_hello")
Thanks!
On Mon, Oct 28, 2019 at 12:25 PM M. Manna <ma...@gmail.com> wrote:
> Hi,
>
> not sure what it means "Tries to communicate with itself". Are you talking
> about local network loopback?
>
> Also, have you tried ssl debug using openssl? What did you observe?
>
> The exception is handshake exception. This is quite common when your cert
> validation fails. How have you setup your signed certificates? Does your
> CN/SAN matches with your advertised.listeners setup? Have you setup
> hostname verification correctly?
>
> Thanks,
>
> On Mon, 28 Oct 2019 at 11:11, Péter Nagykátai <st...@gmail.com>
> wrote:
>
> > @Jose
> >
> > >It looks like communication problem between brokers.
> > As I mentioned, "I can't get the first broker started". The message above
> > is from when the broker tries to communicate with "itself": [Controller
> > id=1001, targetBrokerId=1001]).
> >
> > Nevertheless, I went through the checklist and everything is in order.
> For
> > the first couple of tries, I got different SSL errors but I could work
> > those out (that time I messed up the certificates), but now the problem
> is:
> > >> Caused by: javax.net.ssl.SSLProtocolException: *Unexpected handshake
> > **message:
> > server_hello*
> >
> > Peter
> >
> > On Mon, Oct 28, 2019 at 8:09 AM Jose Manuel Vega Monroy <
> > jose.monroy@williamhill.com> wrote:
> >
> > > @Peter
> > >
> > > It looks like communication problem between brokers. But ensure:
> > >
> > > 1) Crtificates are valid and properly signed by root CA or intermediate
> > > one in the chain
> > > 2) Clients and brokers having private key and certificate in their
> > > keystore and properly configured to point to its path
> > > 3) Clients and brokers having CA certificates in the truststore and
> > > properly configured to point to its path
> > > 4) Clients and brokersbroker having root CA certificate in their
> keystore
> > > and properly configured to.point to its path
> > > 5) Permissions are right ones fro trustore and keystore
> > >
> > > Thanks
> > >
> > > Get Outlook for Android <https://aka.ms/ghei36>
> > >
> > > ------------------------------
> > > *From:* Péter Nagykátai <st...@gmail.com>
> > > *Sent:* Monday, 28 October 2019, 00:13
> > > *To:* users@kafka.apache.org
> > > *Subject:* [EXTERNAL] SSL setup failing
> > >
> > > Hi!
> > >
> > > I'm experimenting with setting up a log ingesting cluster and Kafka
> would
> > > be part of it. Unfortunately, I can't get the first broker started. I
> > need
> > > to secure the communication between a dozen nodes and Kaquiafka would
> > only
> > > be
> > > one part of it. I have a secured node where I generate certificates for
> > > every server in the cluster (with an intermediate CA). AFAIK, I need to
> > use
> > > '.jks' files for Kafka, so I've generated a '.p12' file from the
> openssl
> > > certificate and key then used `keytool` to generate a keystore:
> > > `keytool -importkeystore -srckeystore kafka-1.p12 -srcstoretype PKCS12
> > > -alias kafka-1 -destkeystore kafka-1.jks`
> > > I generated a truststore for the root and intermediate chain as well:
> > > `keytool -importcert -alias ca-root -keystore truststore.jks -file
> > > ca-chain.cert.pem
> > >
> > > Relevant part of the 'server.properties' configuration:
> > > ####
> > > listeners=EXTERNAL://kafka-1:9092,INTERNAL://kafka-1:9093
> > > advertised.listeners=EXTERNAL://kafka-1:9092,INTERNAL://kafka-1:9093
> > > inter.broker.listener.name=INTERNAL
> > > listener.security.protocol.map=EXTERNAL:SSL,INTERNAL:SSL
> > > security.protocol=SSL
> > > ssl.client.auth=required
> > > ssl.truststore.location=/*******/truststore.jks
> > > ssl.truststore.password=*************
> > > ssl.keystore.location=/*******/kafka-1.jks
> > > ssl.keystore.password=*************
> > > ####
> > >
> > > After starting Kafka (as a service) I get the the following in the
> > > 'server.log':
> > > >>...
> > > >> INFO [KafkaServer id=1001] started (kafka.server.KafkaServer)
> > > >> INFO [SocketServer brokerId=1001] Failed authentication with
> > > /XXX.XXX.XXX.XXX (SSL handshake failed)
> > > (org.apache.kafka.common.network.Selector)
> > > >> INFO [Controller id=1001, targetBrokerId=1001] Failed authentication
> > > with kafka-1/XXX.XXX.XXX.XXX (SSL handshake failed)
> > > (org.apache.kafka.common.network.Selector)
> > > >> ERROR [Controller id=1001, targetBrokerId=1001] Connection to node
> > 1001
> > > (kafka-1/XXX.XXX.XXX.XXX:9093) failed authentication due to: SSL
> > handshake
> > > failed (org.apache.kafka.clients.NetworkClient)
> > > >>...
> > > >> WARN SSL handshake failed (kafka.utils.CoreUtils$)
> > > >> org.apache.kafka.common.errors.SslAuthenticationException: SSL
> > handshake
> > > failed
> > > >> Caused by: javax.net.ssl.SSLProtocolException: Unexpected handshake
> > > message: server_hello
> > > >>...
> > >
> > > I couldn't find any lead with that error message and got stuck. Any
> ideas
> > > what that error message means and how to solve it?
> > >
> > > Specs:
> > > - Ubuntu 18.04.3 LTS
> > > - OpenJDK Runtime Environment (build
> > 11.0.4+11-post-Ubuntu-1ubuntu218.04.3)
> > > - Kafka 2.2.1 (from kafka_2.12-2.2.1.tgz)
> > > - OpenSSL 1.1.1
> > >
> > > Thank you!
> > > Peter
> > >
> > > Confidentiality: The contents of this e-mail and any attachments
> > > transmitted with it are intended to be confidential to the intended
> > > recipient; and may be privileged or otherwise protected from
> disclosure.
> > If
> > > you are not an intended recipient of this e-mail, do not duplicate or
> > > redistribute it by any means. Please delete it and any attachments and
> > > notify the sender that you have received it in error. This e-mail is
> sent
> > > by a William Hill PLC group company. The William Hill group companies
> > > include, among others, William Hill PLC (registered number 4212563),
> > > William Hill Organization Limited (registered number 278208), William
> > Hill
> > > US HoldCo Inc, WHG (International) Limited (registered number 99191)
> and
> > Mr
> > > Green Limited (registered number C43260). Each of William Hill PLC and
> > > William Hill Organization Limited is registered in England and Wales
> and
> > > has its registered office at 1 Bedford Avenue, London, WC1B 3AU, UK.
> > > William Hill U.S. HoldCo, Inc. is registered in Delaware and has its
> > > registered office at 1007 N. Orange Street, 9 Floor, Wilmington, New
> > Castle
> > > County DE 19801 Delaware, United States of America. WHG (International)
> > > Limited is registered in Gibraltar and has its registered office at 6/1
> > > Waterport Place, Gibraltar. Mr Green Limited is registered in Malta and
> > has
> > > its registered office at Tagliaferro Business Centre, Level 7, 14 High
> > > Street, Sliema SLM 1549, Malta. Unless specifically indicated
> otherwise,
> > > the contents of this e-mail are subject to contract; and are not an
> > > official statement, and do not necessarily represent the views, of
> > William
> > > Hill PLC, its subsidiaries or affiliated companies. Please note that
> > > neither William Hill PLC, nor its subsidiaries and affiliated companies
> > can
> > > accept any responsibility for any viruses contained within this e-mail
> > and
> > > it is your responsibility to scan any emails and their attachments.
> > William
> > > Hill PLC, its subsidiaries and affiliated companies may monitor e-mail
> > > traffic data and also the content of e-mails for effective operation of
> > the
> > > e-mail system, or for security, purposes.
> > >
> >
>
Re: [EXTERNAL] SSL setup failing
Posted by "M. Manna" <ma...@gmail.com>.
Hi,
not sure what it means "Tries to communicate with itself". Are you talking
about local network loopback?
Also, have you tried ssl debug using openssl? What did you observe?
The exception is handshake exception. This is quite common when your cert
validation fails. How have you setup your signed certificates? Does your
CN/SAN matches with your advertised.listeners setup? Have you setup
hostname verification correctly?
Thanks,
On Mon, 28 Oct 2019 at 11:11, Péter Nagykátai <st...@gmail.com> wrote:
> @Jose
>
> >It looks like communication problem between brokers.
> As I mentioned, "I can't get the first broker started". The message above
> is from when the broker tries to communicate with "itself": [Controller
> id=1001, targetBrokerId=1001]).
>
> Nevertheless, I went through the checklist and everything is in order. For
> the first couple of tries, I got different SSL errors but I could work
> those out (that time I messed up the certificates), but now the problem is:
> >> Caused by: javax.net.ssl.SSLProtocolException: *Unexpected handshake
> **message:
> server_hello*
>
> Peter
>
> On Mon, Oct 28, 2019 at 8:09 AM Jose Manuel Vega Monroy <
> jose.monroy@williamhill.com> wrote:
>
> > @Peter
> >
> > It looks like communication problem between brokers. But ensure:
> >
> > 1) Crtificates are valid and properly signed by root CA or intermediate
> > one in the chain
> > 2) Clients and brokers having private key and certificate in their
> > keystore and properly configured to point to its path
> > 3) Clients and brokers having CA certificates in the truststore and
> > properly configured to point to its path
> > 4) Clients and brokersbroker having root CA certificate in their keystore
> > and properly configured to.point to its path
> > 5) Permissions are right ones fro trustore and keystore
> >
> > Thanks
> >
> > Get Outlook for Android <https://aka.ms/ghei36>
> >
> > ------------------------------
> > *From:* Péter Nagykátai <st...@gmail.com>
> > *Sent:* Monday, 28 October 2019, 00:13
> > *To:* users@kafka.apache.org
> > *Subject:* [EXTERNAL] SSL setup failing
> >
> > Hi!
> >
> > I'm experimenting with setting up a log ingesting cluster and Kafka would
> > be part of it. Unfortunately, I can't get the first broker started. I
> need
> > to secure the communication between a dozen nodes and Kaquiafka would
> only
> > be
> > one part of it. I have a secured node where I generate certificates for
> > every server in the cluster (with an intermediate CA). AFAIK, I need to
> use
> > '.jks' files for Kafka, so I've generated a '.p12' file from the openssl
> > certificate and key then used `keytool` to generate a keystore:
> > `keytool -importkeystore -srckeystore kafka-1.p12 -srcstoretype PKCS12
> > -alias kafka-1 -destkeystore kafka-1.jks`
> > I generated a truststore for the root and intermediate chain as well:
> > `keytool -importcert -alias ca-root -keystore truststore.jks -file
> > ca-chain.cert.pem
> >
> > Relevant part of the 'server.properties' configuration:
> > ####
> > listeners=EXTERNAL://kafka-1:9092,INTERNAL://kafka-1:9093
> > advertised.listeners=EXTERNAL://kafka-1:9092,INTERNAL://kafka-1:9093
> > inter.broker.listener.name=INTERNAL
> > listener.security.protocol.map=EXTERNAL:SSL,INTERNAL:SSL
> > security.protocol=SSL
> > ssl.client.auth=required
> > ssl.truststore.location=/*******/truststore.jks
> > ssl.truststore.password=*************
> > ssl.keystore.location=/*******/kafka-1.jks
> > ssl.keystore.password=*************
> > ####
> >
> > After starting Kafka (as a service) I get the the following in the
> > 'server.log':
> > >>...
> > >> INFO [KafkaServer id=1001] started (kafka.server.KafkaServer)
> > >> INFO [SocketServer brokerId=1001] Failed authentication with
> > /XXX.XXX.XXX.XXX (SSL handshake failed)
> > (org.apache.kafka.common.network.Selector)
> > >> INFO [Controller id=1001, targetBrokerId=1001] Failed authentication
> > with kafka-1/XXX.XXX.XXX.XXX (SSL handshake failed)
> > (org.apache.kafka.common.network.Selector)
> > >> ERROR [Controller id=1001, targetBrokerId=1001] Connection to node
> 1001
> > (kafka-1/XXX.XXX.XXX.XXX:9093) failed authentication due to: SSL
> handshake
> > failed (org.apache.kafka.clients.NetworkClient)
> > >>...
> > >> WARN SSL handshake failed (kafka.utils.CoreUtils$)
> > >> org.apache.kafka.common.errors.SslAuthenticationException: SSL
> handshake
> > failed
> > >> Caused by: javax.net.ssl.SSLProtocolException: Unexpected handshake
> > message: server_hello
> > >>...
> >
> > I couldn't find any lead with that error message and got stuck. Any ideas
> > what that error message means and how to solve it?
> >
> > Specs:
> > - Ubuntu 18.04.3 LTS
> > - OpenJDK Runtime Environment (build
> 11.0.4+11-post-Ubuntu-1ubuntu218.04.3)
> > - Kafka 2.2.1 (from kafka_2.12-2.2.1.tgz)
> > - OpenSSL 1.1.1
> >
> > Thank you!
> > Peter
> >
> > Confidentiality: The contents of this e-mail and any attachments
> > transmitted with it are intended to be confidential to the intended
> > recipient; and may be privileged or otherwise protected from disclosure.
> If
> > you are not an intended recipient of this e-mail, do not duplicate or
> > redistribute it by any means. Please delete it and any attachments and
> > notify the sender that you have received it in error. This e-mail is sent
> > by a William Hill PLC group company. The William Hill group companies
> > include, among others, William Hill PLC (registered number 4212563),
> > William Hill Organization Limited (registered number 278208), William
> Hill
> > US HoldCo Inc, WHG (International) Limited (registered number 99191) and
> Mr
> > Green Limited (registered number C43260). Each of William Hill PLC and
> > William Hill Organization Limited is registered in England and Wales and
> > has its registered office at 1 Bedford Avenue, London, WC1B 3AU, UK.
> > William Hill U.S. HoldCo, Inc. is registered in Delaware and has its
> > registered office at 1007 N. Orange Street, 9 Floor, Wilmington, New
> Castle
> > County DE 19801 Delaware, United States of America. WHG (International)
> > Limited is registered in Gibraltar and has its registered office at 6/1
> > Waterport Place, Gibraltar. Mr Green Limited is registered in Malta and
> has
> > its registered office at Tagliaferro Business Centre, Level 7, 14 High
> > Street, Sliema SLM 1549, Malta. Unless specifically indicated otherwise,
> > the contents of this e-mail are subject to contract; and are not an
> > official statement, and do not necessarily represent the views, of
> William
> > Hill PLC, its subsidiaries or affiliated companies. Please note that
> > neither William Hill PLC, nor its subsidiaries and affiliated companies
> can
> > accept any responsibility for any viruses contained within this e-mail
> and
> > it is your responsibility to scan any emails and their attachments.
> William
> > Hill PLC, its subsidiaries and affiliated companies may monitor e-mail
> > traffic data and also the content of e-mails for effective operation of
> the
> > e-mail system, or for security, purposes.
> >
>
Re: [EXTERNAL] SSL setup failing
Posted by Jose Manuel Vega Monroy <jo...@williamhill.com>.
@Peter
That happening when clients trying to SSL connect?
Review SSL configuration related ssl.client.auth=required is right in client side.
Here you have explained possible errors, and you could find what happening: https://docs.apigee.com/api-platform/troubleshoot/runtime/ssl-handshake-failures
For example, could be certificate CN name not matching hostname used to connect client-server, throwing something like this:
java.security.cert.CertificateException: No name matching localhost found
Thanks
<http://www.williamhill.com/>
<http://www.whenthefunstops.co.uk/>
Jose Manuel Vega Monroy
Java Developer / Software Developer Engineer in Test
Direct: +0035 0 2008038 (Ext. 8038)
Email: jose.monroy@williamhill.com
William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA
On 28/10/2019, 12:11, "Péter Nagykátai" <st...@gmail.com> wrote:
@Jose
>It looks like communication problem between brokers.
As I mentioned, "I can't get the first broker started". The message above
is from when the broker tries to communicate with "itself": [Controller
id=1001, targetBrokerId=1001]).
Nevertheless, I went through the checklist and everything is in order. For
the first couple of tries, I got different SSL errors but I could work
those out (that time I messed up the certificates), but now the problem is:
>> Caused by: javax.net.ssl.SSLProtocolException: *Unexpected handshake **message:
server_hello*
Peter
On Mon, Oct 28, 2019 at 8:09 AM Jose Manuel Vega Monroy <
jose.monroy@williamhill.com> wrote:
> @Peter
>
> It looks like communication problem between brokers. But ensure:
>
> 1) Crtificates are valid and properly signed by root CA or intermediate
> one in the chain
> 2) Clients and brokers having private key and certificate in their
> keystore and properly configured to point to its path
> 3) Clients and brokers having CA certificates in the truststore and
> properly configured to point to its path
> 4) Clients and brokersbroker having root CA certificate in their keystore
> and properly configured to.point to its path
> 5) Permissions are right ones fro trustore and keystore
>
> Thanks
>
> Get Outlook for Android <https://urldefense.proofpoint.com/v2/url?u=https-3A__aka.ms_ghei36&d=DwIFaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=uwX9KbgU8pPFAgG1vr70bVs7l0XYYV9Rn8yZR07Fgsg&s=ib4HzMkTN7pBJB_cZVJPatBggBNpQjzQOfaDArTFrkQ&e= >
>
> ------------------------------
> *From:* Péter Nagykátai <st...@gmail.com>
> *Sent:* Monday, 28 October 2019, 00:13
> *To:* users@kafka.apache.org
> *Subject:* [EXTERNAL] SSL setup failing
>
> Hi!
>
> I'm experimenting with setting up a log ingesting cluster and Kafka would
> be part of it. Unfortunately, I can't get the first broker started. I need
> to secure the communication between a dozen nodes and Kaquiafka would only
> be
> one part of it. I have a secured node where I generate certificates for
> every server in the cluster (with an intermediate CA). AFAIK, I need to use
> '.jks' files for Kafka, so I've generated a '.p12' file from the openssl
> certificate and key then used `keytool` to generate a keystore:
> `keytool -importkeystore -srckeystore kafka-1.p12 -srcstoretype PKCS12
> -alias kafka-1 -destkeystore kafka-1.jks`
> I generated a truststore for the root and intermediate chain as well:
> `keytool -importcert -alias ca-root -keystore truststore.jks -file
> ca-chain.cert.pem
>
> Relevant part of the 'server.properties' configuration:
> ####
> listeners=EXTERNAL://kafka-1:9092,INTERNAL://kafka-1:9093
> advertised.listeners=EXTERNAL://kafka-1:9092,INTERNAL://kafka-1:9093
> inter.broker.listener.name=INTERNAL
> listener.security.protocol.map=EXTERNAL:SSL,INTERNAL:SSL
> security.protocol=SSL
> ssl.client.auth=required
> ssl.truststore.location=/*******/truststore.jks
> ssl.truststore.password=*************
> ssl.keystore.location=/*******/kafka-1.jks
> ssl.keystore.password=*************
> ####
>
> After starting Kafka (as a service) I get the the following in the
> 'server.log':
> >>...
> >> INFO [KafkaServer id=1001] started (kafka.server.KafkaServer)
> >> INFO [SocketServer brokerId=1001] Failed authentication with
> /XXX.XXX.XXX.XXX (SSL handshake failed)
> (org.apache.kafka.common.network.Selector)
> >> INFO [Controller id=1001, targetBrokerId=1001] Failed authentication
> with kafka-1/XXX.XXX.XXX.XXX (SSL handshake failed)
> (org.apache.kafka.common.network.Selector)
> >> ERROR [Controller id=1001, targetBrokerId=1001] Connection to node 1001
> (kafka-1/XXX.XXX.XXX.XXX:9093) failed authentication due to: SSL handshake
> failed (org.apache.kafka.clients.NetworkClient)
> >>...
> >> WARN SSL handshake failed (kafka.utils.CoreUtils$)
> >> org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake
> failed
> >> Caused by: javax.net.ssl.SSLProtocolException: Unexpected handshake
> message: server_hello
> >>...
>
> I couldn't find any lead with that error message and got stuck. Any ideas
> what that error message means and how to solve it?
>
> Specs:
> - Ubuntu 18.04.3 LTS
> - OpenJDK Runtime Environment (build 11.0.4+11-post-Ubuntu-1ubuntu218.04.3)
> - Kafka 2.2.1 (from kafka_2.12-2.2.1.tgz)
> - OpenSSL 1.1.1
>
> Thank you!
> Peter
>
> Confidentiality: The contents of this e-mail and any attachments
> transmitted with it are intended to be confidential to the intended
> recipient; and may be privileged or otherwise protected from disclosure. If
> you are not an intended recipient of this e-mail, do not duplicate or
> redistribute it by any means. Please delete it and any attachments and
> notify the sender that you have received it in error. This e-mail is sent
> by a William Hill PLC group company. The William Hill group companies
> include, among others, William Hill PLC (registered number 4212563),
> William Hill Organization Limited (registered number 278208), William Hill
> US HoldCo Inc, WHG (International) Limited (registered number 99191) and Mr
> Green Limited (registered number C43260). Each of William Hill PLC and
> William Hill Organization Limited is registered in England and Wales and
> has its registered office at 1 Bedford Avenue, London, WC1B 3AU, UK.
> William Hill U.S. HoldCo, Inc. is registered in Delaware and has its
> registered office at 1007 N. Orange Street, 9 Floor, Wilmington, New Castle
> County DE 19801 Delaware, United States of America. WHG (International)
> Limited is registered in Gibraltar and has its registered office at 6/1
> Waterport Place, Gibraltar. Mr Green Limited is registered in Malta and has
> its registered office at Tagliaferro Business Centre, Level 7, 14 High
> Street, Sliema SLM 1549, Malta. Unless specifically indicated otherwise,
> the contents of this e-mail are subject to contract; and are not an
> official statement, and do not necessarily represent the views, of William
> Hill PLC, its subsidiaries or affiliated companies. Please note that
> neither William Hill PLC, nor its subsidiaries and affiliated companies can
> accept any responsibility for any viruses contained within this e-mail and
> it is your responsibility to scan any emails and their attachments. William
> Hill PLC, its subsidiaries and affiliated companies may monitor e-mail
> traffic data and also the content of e-mails for effective operation of the
> e-mail system, or for security, purposes.
>
Confidentiality: The contents of this e-mail and any attachments transmitted with it are intended to be confidential to the intended recipient; and may be privileged or otherwise protected from disclosure. If you are not an intended recipient of this e-mail, do not duplicate or redistribute it by any means. Please delete it and any attachments and notify the sender that you have received it in error. This e-mail is sent by a William Hill PLC group company. The William Hill group companies include, among others, William Hill PLC (registered number 4212563), William Hill Organization Limited (registered number 278208), William Hill US HoldCo Inc, WHG (International) Limited (registered number 99191) and Mr Green Limited (registered number C43260). Each of William Hill PLC and William Hill Organization Limited is registered in England and Wales and has its registered office at 1 Bedford Avenue, London, WC1B 3AU, UK. William Hill U.S. HoldCo, Inc. is registered in Delaware and has its registered office at 1007 N. Orange Street, 9 Floor, Wilmington, New Castle County DE 19801 Delaware, United States of America. WHG (International) Limited is registered in Gibraltar and has its registered office at 6/1 Waterport Place, Gibraltar. Mr Green Limited is registered in Malta and has its registered office at Tagliaferro Business Centre, Level 7, 14 High Street, Sliema SLM 1549, Malta. Unless specifically indicated otherwise, the contents of this e-mail are subject to contract; and are not an official statement, and do not necessarily represent the views, of William Hill PLC, its subsidiaries or affiliated companies. Please note that neither William Hill PLC, nor its subsidiaries and affiliated companies can accept any responsibility for any viruses contained within this e-mail and it is your responsibility to scan any emails and their attachments. William Hill PLC, its subsidiaries and affiliated companies may monitor e-mail traffic data and also the content of e-mails for effective operation of the e-mail system, or for security, purposes.
Re: [EXTERNAL] SSL setup failing
Posted by Péter Nagykátai <st...@gmail.com>.
@Jose
>It looks like communication problem between brokers.
As I mentioned, "I can't get the first broker started". The message above
is from when the broker tries to communicate with "itself": [Controller
id=1001, targetBrokerId=1001]).
Nevertheless, I went through the checklist and everything is in order. For
the first couple of tries, I got different SSL errors but I could work
those out (that time I messed up the certificates), but now the problem is:
>> Caused by: javax.net.ssl.SSLProtocolException: *Unexpected handshake **message:
server_hello*
Peter
On Mon, Oct 28, 2019 at 8:09 AM Jose Manuel Vega Monroy <
jose.monroy@williamhill.com> wrote:
> @Peter
>
> It looks like communication problem between brokers. But ensure:
>
> 1) Crtificates are valid and properly signed by root CA or intermediate
> one in the chain
> 2) Clients and brokers having private key and certificate in their
> keystore and properly configured to point to its path
> 3) Clients and brokers having CA certificates in the truststore and
> properly configured to point to its path
> 4) Clients and brokersbroker having root CA certificate in their keystore
> and properly configured to.point to its path
> 5) Permissions are right ones fro trustore and keystore
>
> Thanks
>
> Get Outlook for Android <https://aka.ms/ghei36>
>
> ------------------------------
> *From:* Péter Nagykátai <st...@gmail.com>
> *Sent:* Monday, 28 October 2019, 00:13
> *To:* users@kafka.apache.org
> *Subject:* [EXTERNAL] SSL setup failing
>
> Hi!
>
> I'm experimenting with setting up a log ingesting cluster and Kafka would
> be part of it. Unfortunately, I can't get the first broker started. I need
> to secure the communication between a dozen nodes and Kaquiafka would only
> be
> one part of it. I have a secured node where I generate certificates for
> every server in the cluster (with an intermediate CA). AFAIK, I need to use
> '.jks' files for Kafka, so I've generated a '.p12' file from the openssl
> certificate and key then used `keytool` to generate a keystore:
> `keytool -importkeystore -srckeystore kafka-1.p12 -srcstoretype PKCS12
> -alias kafka-1 -destkeystore kafka-1.jks`
> I generated a truststore for the root and intermediate chain as well:
> `keytool -importcert -alias ca-root -keystore truststore.jks -file
> ca-chain.cert.pem
>
> Relevant part of the 'server.properties' configuration:
> ####
> listeners=EXTERNAL://kafka-1:9092,INTERNAL://kafka-1:9093
> advertised.listeners=EXTERNAL://kafka-1:9092,INTERNAL://kafka-1:9093
> inter.broker.listener.name=INTERNAL
> listener.security.protocol.map=EXTERNAL:SSL,INTERNAL:SSL
> security.protocol=SSL
> ssl.client.auth=required
> ssl.truststore.location=/*******/truststore.jks
> ssl.truststore.password=*************
> ssl.keystore.location=/*******/kafka-1.jks
> ssl.keystore.password=*************
> ####
>
> After starting Kafka (as a service) I get the the following in the
> 'server.log':
> >>...
> >> INFO [KafkaServer id=1001] started (kafka.server.KafkaServer)
> >> INFO [SocketServer brokerId=1001] Failed authentication with
> /XXX.XXX.XXX.XXX (SSL handshake failed)
> (org.apache.kafka.common.network.Selector)
> >> INFO [Controller id=1001, targetBrokerId=1001] Failed authentication
> with kafka-1/XXX.XXX.XXX.XXX (SSL handshake failed)
> (org.apache.kafka.common.network.Selector)
> >> ERROR [Controller id=1001, targetBrokerId=1001] Connection to node 1001
> (kafka-1/XXX.XXX.XXX.XXX:9093) failed authentication due to: SSL handshake
> failed (org.apache.kafka.clients.NetworkClient)
> >>...
> >> WARN SSL handshake failed (kafka.utils.CoreUtils$)
> >> org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake
> failed
> >> Caused by: javax.net.ssl.SSLProtocolException: Unexpected handshake
> message: server_hello
> >>...
>
> I couldn't find any lead with that error message and got stuck. Any ideas
> what that error message means and how to solve it?
>
> Specs:
> - Ubuntu 18.04.3 LTS
> - OpenJDK Runtime Environment (build 11.0.4+11-post-Ubuntu-1ubuntu218.04.3)
> - Kafka 2.2.1 (from kafka_2.12-2.2.1.tgz)
> - OpenSSL 1.1.1
>
> Thank you!
> Peter
>
> Confidentiality: The contents of this e-mail and any attachments
> transmitted with it are intended to be confidential to the intended
> recipient; and may be privileged or otherwise protected from disclosure. If
> you are not an intended recipient of this e-mail, do not duplicate or
> redistribute it by any means. Please delete it and any attachments and
> notify the sender that you have received it in error. This e-mail is sent
> by a William Hill PLC group company. The William Hill group companies
> include, among others, William Hill PLC (registered number 4212563),
> William Hill Organization Limited (registered number 278208), William Hill
> US HoldCo Inc, WHG (International) Limited (registered number 99191) and Mr
> Green Limited (registered number C43260). Each of William Hill PLC and
> William Hill Organization Limited is registered in England and Wales and
> has its registered office at 1 Bedford Avenue, London, WC1B 3AU, UK.
> William Hill U.S. HoldCo, Inc. is registered in Delaware and has its
> registered office at 1007 N. Orange Street, 9 Floor, Wilmington, New Castle
> County DE 19801 Delaware, United States of America. WHG (International)
> Limited is registered in Gibraltar and has its registered office at 6/1
> Waterport Place, Gibraltar. Mr Green Limited is registered in Malta and has
> its registered office at Tagliaferro Business Centre, Level 7, 14 High
> Street, Sliema SLM 1549, Malta. Unless specifically indicated otherwise,
> the contents of this e-mail are subject to contract; and are not an
> official statement, and do not necessarily represent the views, of William
> Hill PLC, its subsidiaries or affiliated companies. Please note that
> neither William Hill PLC, nor its subsidiaries and affiliated companies can
> accept any responsibility for any viruses contained within this e-mail and
> it is your responsibility to scan any emails and their attachments. William
> Hill PLC, its subsidiaries and affiliated companies may monitor e-mail
> traffic data and also the content of e-mails for effective operation of the
> e-mail system, or for security, purposes.
>