You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by karan alang <ka...@gmail.com> on 2017/09/19 21:44:36 UTC
Kafka SSL error
Hello All -
I was able to set up SSL for the Kafka brokers, using OpenSSL.
however, I'm having issues with setting up SSL using the pem file (i.e. SSL
certificate - certified by CA, provided by the company)
Here is what i've done -
created the server/client keystore & truststore files and imported the
provided cert.pem file
keytool -keystore kafka.server.keystore.jks -alias localhost -validity 365
-genkey
keytool -keystore kafka.server.truststore.jks -alias CARoot -import -file
cert.pem
keytool -keystore kafka.client.truststore.jks -alias CARoot -import -file
cert.pem
keytool -keystore kafka.server.keystore.jks -alias CARoot -import -file
cert.pem
keytool -keystore kafka.client.keystore.jks -alias localhost -validity 365
-genkey
keytool -keystore kafka.client.keystore.jks -alias CARoot -import -file
cert.pem
I've a console producer pushing data in to the topic, and gives error as
shown below ->
Caused by: javax.net.ssl.SSLProtocolException: Handshake message sequence
> violation, state = 1, type = 1
> at
> sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:213)
> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026)
> at sun.security.ssl.Handshaker$1.run(Handshaker.java:966)
> at sun.security.ssl.Handshaker$1.run(Handshaker.java:963)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1416)
> at
> org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:336)
> at
> org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:417)
> ... 7 more
Any ideas on what the issue might be ?
thanks for help in advance!
Re: Kafka SSL error
Posted by karan alang <ka...@gmail.com>.
Thanks Jakub .. for your inputs & help in this !
I was able to get this to work last week..
On Thu, Sep 21, 2017 at 12:22 AM, Jakub Scholz <ja...@scholz.cz> wrote:
> Hi,
>
> If you want the Kafka broker to present the whole chain you have to use the
> chain when creating the PKCS12 file (use the chain instead of the host
> certificate). As you mentioned, the chain should be in the order 1) server
> cert, 2) intermediate cert and 3) root cert. It will be then automatically
> imported into the keystore. When you use -showcerts option in "openssl
> s_client it should show the whole chain. The intermediate and root cert
> alone can be imported into the keystore / truststore directly, it doesn't
> need to be through PKCS12. But as mentioned before, to have the broker
> present the whole chain you have to load the chain already into the PKCS12.
> Loading the chain parts separately into the keystore will not help.
>
> In the openssl s_client you can also use the -CAfile option to point it to
> the intermediate certificate - that would also wirk but using the whole
> chain is probably better.
>
> Regards
> Jakub
>
> On Thu, Sep 21, 2017 at 2:00 AM, karan alang <ka...@gmail.com>
> wrote:
>
> > Another point ..
> > on adding the chain.p12 (PKCS#12 format) .. order of the certs in the
> > keystore is as shown below
> >
> > Alias : 1
> > Cert[1] -> server cert
> > Cert[2] -> Intermediate cert
> > Cert[3] -> Root cert
> >
> > mentioning that, since one of the articles i read mentioned that the
> certs
> > imported should be in following order -> 1) Root 2) Intermediate 3)
> Server
> >
> >
> > On Wed, Sep 20, 2017 at 4:48 PM, karan alang <ka...@gmail.com>
> > wrote:
> >
> > > Hi Jakub,
> > >
> > > Thanks for the detailed note...
> > >
> > > here is the update ->
> > >
> > > I was able to convert the host.cert.pem to PKCS#12 & import the cert
> into
> > > the kafka.server.keystore.jks
> > > (also into kafka.server.truststore.jks)
> > > wrt the host.root.pem & host.intermed.pem certs - i'm assuming i need
> to
> > > convert them to PKCS#12 as well ?
> > > Or do i import them as PEM file ?
> > >
> > > I was not able to convert them to PKCS#12(since i understand the
> PKCS#12
> > > format requires a private key & the root.pem & intermed.pem donot have
> a
> > > private key associated)
> > > the error is shown below -
> > >
> > > openssl pkcs12 -export -out intermed.p12 -in hostname.issuer.pem -inkey
> > >> privatekey-kafka04-0920.key.pem -password pass:<password>
> > >> Enter pass phrase for privatekey-kafka04-0920.key.pem:
> > >> No certificate matches private key
> > >
> > >
> > >
> > > What i did next ->
> > >
> > > The host.chain.cert has all the 3 certs (server, intermediate, root)
> > > and i converted this to PKCS format & also was able to import into the
> > > keystore/truststore.
> > > I'm able to see the 3 certificates in the keystore/truststore under ONE
> > > alias.
> > >
> > >
> > > Now i get a different error shown below ->
> > >
> > > Command :
> > > openssl s_client -debug -connect <hostname>:9192 -tls1
> > >
> > > Error :
> > >
> > > depth=0 CN = <hostname>, OU = <OU>, O = <Organization>, ST = <State>,
> C =
> > >> US
> > >> verify error:num=20:unable to get local issuer certificate
> > >> verify return:1
> > >> depth=0 CN = <hostname>, OU = <OU>, O = <Organization>, ST = <State>,
> C
> > =
> > >> US
> > >> verify error:num=27:certificate not trusted
> > >> verify return:1
> > >> depth=0 CN = <hostname>, OU = <OU>, O = <Organization>, ST = <State>,
> C
> > =
> > >> US
> > >> verify error:num=21:unable to verify the first certificate
> > >> verify return:1
> > >> write to 0xd47830 [0xd98100] (12 bytes => 12 (0xC))
> > >> ---------
> > >> ---------
> > >> 0930 - 74 20 47 6c 6f 62 61 6c-20 43 41 0e t Global CA.
> > >> 093f - <SPACES/NULS>
> > >> write to 0xd47830 [0xd98100] (37 bytes => -1 (0xFFFFFFFFFFFFFFFF))
> > >> 140456850737056:error:1408E0F4:SSL routines:SSL3_GET_MESSAGE:
> unexpected
> > >> message:s3_both.c:491:
> > >> --------
> > >> --------
> > >> SSL-Session:
> > >> Protocol : TLSv1
> > >> ------
> > >> ------
> > >> Timeout : 7200 (sec)
> > >> Verify return code: 21 (unable to verify the first certificate)
> > >
> > >
> > >
> > > Any ideas on how to fix this ?
> > >
> > >
> > > On Wed, Sep 20, 2017 at 2:22 PM, Jakub Scholz <ja...@scholz.cz> wrote:
> > >
> > >> ad 1) The problem is that the signed certificate (host.cert.pem) which
> > the
> > >> CA provides is only the public key. You have to combine it with the
> > >> private
> > >> key which you created when requesting the signed certificate. The
> > private
> > >> key is never sent to the CA so they cannot provide it back. You or
> > whoever
> > >> created the signing request should have it.
> > >>
> > >> ad 2) To create the keystore with your signed key, you have to take
> the
> > >> signed public key and the private key and create P12 file:
> > >> $ openssl pkcs12 -export -out server.p12 -in host.cert.pem -inkey
> > host.key
> > >> -password pass:somepassword
> > >>
> > >> And afterwards convert it to keystore format:
> > >> $ keytool -importkeystore -srckeystore server.p12 -srcstoretype PKCS12
> > >> -srcstorepass somepassword -destkeystore server.keystore
> -deststoretype
> > >> JKS
> > >> -deststorepass somepassword -noprompt
> > >>
> > >> You can add the intermediate and root to the keystore as well. But
> when
> > it
> > >> tells you that you already have them in the system wide keystore say
> > "yes"
> > >> that you want to add them to the keystore again.
> > >>
> > >> ad 3) Do you want your clients to authenticate with a client
> > certificates?
> > >> If not, you should not need the truststore for the broker. You only
> > need a
> > >> truststore for the client which should contain the root and the
> > >> intermediate certificates. If you want clients to authenticate using
> > >> certificates you have to do basically the same as above for the client
> > >> (You
> > >> should use different certificate for cleints from the one in server -
> > for
> > >> security reasons ;-)).
> > >>
> > >> ad 4) As I said, I think what you did before was that the broker
> > basically
> > >> uses the self-signed cert you generated. This is probably the default
> > >> subject distinguished name of the self signed certificate. And since
> it
> > is
> > >> self-signed openssl cannot verify its identity (unless you pass it the
> > >> public key of this exact self signed certificate).
> > >>
> > >> Jakub
> > >>
> > >> On Wed, Sep 20, 2017 at 10:46 PM, karan alang <ka...@gmail.com>
> > >> wrote:
> > >>
> > >> > Hello
> > >> > - thanks for the response
> > >> > Here is the update on the issue.
> > >> >
> > >> > I'm using certs signed/provided by org-wide CA (geotrust, not a
> > >> self-signed
> > >> > cert)
> > >> > The Signed(by the CA - geotrust) cert provided has 3 certificates
> > >> > - host.chain.pem (certificate chain - contains the Root,
> Intermediate,
> > >> > Signed Server cert )
> > >> > - host.intermediate.pem (intermediate certificate)
> > >> > - host.cert.pem (signed server sert)
> > >> > Steps :
> > >> > 1) Added the signed cert to kafka.server.keystore.jks
> > >> >
> > >> >
> > >> > > keytool -import -alias servercert -trustcacerts -file
> host.cert.pem
> > >> > > -keystore kafka.server.keystore.jks
> > >> >
> > >> >
> > >> > *btw, Is the command above correct or needs to be changed ? *
> > >> >
> > >> > 2) Added the Intermediate & Signed cert to kafka.server.keystore.jks
> > >> >
> > >> > > keytool -import -alias intermediate -trustcacerts -file
> > >> host.issuer.pem
> > >> > > -keystore kafka.server.truststore.jks
> > >> > > keytool -import -alias servercert -trustcacerts -file
> host.cert.pem
> > >> > > -keystore kafka.server.truststore.jks
> > >> >
> > >> >
> > >> > When i try to add the Root Cert, it shows the following message.
> > >> >
> > >> > *Enter keystore password: *
> > >> > * Certificate already exists in system-wide CA keystore under alias
> > >> > <geotrustglobalca [jdk]>*
> > >> > * Do you still want to add it to your own keystore? [no]: *
> > >> >
> > >> > So, i assume the CA is already trusted .. do i still need to add
> this
> > to
> > >> > the kafka.server.truststore.jks ?
> > >> > (I think i did that, and it did not fix the issue)
> > >> > 3) In the Broker side (server.properties file), the setting is as
> > shown
> > >> > below
> > >> > (this is the same setting when i enabled SSL using OpenSSL, which is
> > >> > working fine btw)
> > >> >
> > >> > > ssl.keystore.location=/usr/hdp/2.5.3.0-37/confluent-3.2.
> > >> > > 2/kafkaSSL/kafka.server.keystore.jks
> > >> > > ssl.keystore.password=<passwd>
> > >> > > ssl.key.password=<passwd>
> > >> > > ssl.truststore.location=/usr/hdp/2.5.3.0-37/confluent-3.2.
> > >> > > 2/kafkaSSL/kafka.server.truststore.jks
> > >> > > ssl.truststore.password=<passwd>
> > >> > > ssl.client.auth=required
> > >> >
> > >> > 4) when i verify the SSL using command below
> > >> >
> > >> > Command :
> > >> >
> > >> > > openssl s_client -debug -connect <hostname>:9192 -tls1
> > >> >
> > >> >
> > >> > Error snippet :
> > >> >
> > >> >
> > >> > >
> > >> > >
> > >> > > depth=0 C = US, ST = <State Name>, L = Unknown, O = <org name>.,
> OU
> > =
> > >> > > <OU>, CN = <localhost>
> > >> > >
> > >> > verify error:num=18:self signed certificate
> > >> > >
> > >> > > verify return:1
> > >> > > ....................
> > >> > >
> > >> > -----------------
> > >> > ------------------
> > >> >
> > >> >
> > >> > > 063d - <SPACES/NULS>
> > >> > > write to 0x860830 [0x8b1100] (37 bytes => -1 (0xFFFFFFFFFFFFFFFF))
> > >> > > 140487373043616:error:1408E0F4:SSL routines:SSL3_GET_MESSAGE:unex
> > >> pected
> > >> > > message:s3_both.c:491:
> > >> >
> > >> >
> > >> > Any ideas on this ?
> > >> > What needs to be done to debug or fix this ?
> > >> >
> > >> > On Wed, Sep 20, 2017 at 12:33 AM, Jakub Scholz <ja...@scholz.cz>
> > wrote:
> > >> >
> > >> > > Hi,
> > >> > >
> > >> > > Looking at your commands it looks as if you generated a self
> signed
> > >> key
> > >> > for
> > >> > > server, self signed key for client and then imported the CA keys
> > >> public
> > >> > > keys into the truststores. I don’t think this will work because
> now
> > >> you
> > >> > > have two different self signed keys in the keystores and the
> > >> presumably
> > >> > the
> > >> > > CA public key in your truststores.
> > >> > >
> > >> > > Does the cert.pem actualy contain the private key? In case it
> > doesn’t
> > >> > > contain the private key you need to get the private key first. If
> > yes,
> > >> > you
> > >> > > have to convert it into the keystore format using this guide:
> > >> > > https://docs.oracle.com/cd/E35976_01/server.740/es_admin/
> > >> > > src/tadm_ssl_convert_pem_to_jks.html
> > >> > > You basically have to first use OpenSSL to create PKCS12 key and
> > that
> > >> can
> > >> > > be converted into keystore.
> > >> > >
> > >> > > BTW: If you run your application with system property
> > >> “javax.net.debug”
> > >> > set
> > >> > > to “ssl” it will generate a lot of useful debug information which
> > will
> > >> > help
> > >> > > to understand what is going on and fix this.
> > >> > >
> > >> > > Jakub
> > >> > >
> > >> > > On Tue, 19 Sep 2017 at 23:44, karan alang <ka...@gmail.com>
> > >> wrote:
> > >> > >
> > >> > > > Hello All -
> > >> > > > I was able to set up SSL for the Kafka brokers, using OpenSSL.
> > >> > > >
> > >> > > > however, I'm having issues with setting up SSL using the pem
> file
> > >> (i.e.
> > >> > > SSL
> > >> > > > certificate - certified by CA, provided by the company)
> > >> > > >
> > >> > > > Here is what i've done -
> > >> > > > created the server/client keystore & truststore files and
> imported
> > >> the
> > >> > > > provided cert.pem file
> > >> > > >
> > >> > > > keytool -keystore kafka.server.keystore.jks -alias localhost
> > >> -validity
> > >> > > 365
> > >> > > > -genkey
> > >> > > > keytool -keystore kafka.server.truststore.jks -alias CARoot
> > -import
> > >> > -file
> > >> > > > cert.pem
> > >> > > > keytool -keystore kafka.client.truststore.jks -alias CARoot
> > -import
> > >> > -file
> > >> > > > cert.pem
> > >> > > > keytool -keystore kafka.server.keystore.jks -alias CARoot
> -import
> > >> -file
> > >> > > > cert.pem
> > >> > > > keytool -keystore kafka.client.keystore.jks -alias localhost
> > >> -validity
> > >> > > 365
> > >> > > > -genkey
> > >> > > > keytool -keystore kafka.client.keystore.jks -alias CARoot
> -import
> > >> -file
> > >> > > > cert.pem
> > >> > > >
> > >> > > > I've a console producer pushing data in to the topic, and gives
> > >> error
> > >> > as
> > >> > > > shown below ->
> > >> > > >
> > >> > > >
> > >> > > > Caused by: javax.net.ssl.SSLProtocolException: Handshake
> message
> > >> > > sequence
> > >> > > > > violation, state = 1, type = 1
> > >> > > > > at
> > >> > > > >
> > >> > > > sun.security.ssl.ServerHandshaker.processMessage(
> > >> > > ServerHandshaker.java:213)
> > >> > > > > at sun.security.ssl.Handshaker.processLoop(Handshaker.java:
> > 1026)
> > >> > > > > at sun.security.ssl.Handshaker$1.run(Handshaker.java:966)
> > >> > > > > at sun.security.ssl.Handshaker$1.run(Handshaker.java:963)
> > >> > > > > at java.security.AccessController.doPrivileged(Native Method)
> > >> > > > > at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.
> > >> > java:1416)
> > >> > > > > at
> > >> > > > >
> > >> > > > org.apache.kafka.common.network.SslTransportLayer.runDelegat
> > >> edTasks(
> > >> > > SslTransportLayer.java:336)
> > >> > > > > at
> > >> > > > >
> > >> > > > org.apache.kafka.common.network.SslTransportLayer.
> > handshakeUnwrap(
> > >> > > SslTransportLayer.java:417)
> > >> > > > > ... 7 more
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > Any ideas on what the issue might be ?
> > >> > > > thanks for help in advance!
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>
Re: Kafka SSL error
Posted by Jakub Scholz <ja...@scholz.cz>.
Hi,
If you want the Kafka broker to present the whole chain you have to use the
chain when creating the PKCS12 file (use the chain instead of the host
certificate). As you mentioned, the chain should be in the order 1) server
cert, 2) intermediate cert and 3) root cert. It will be then automatically
imported into the keystore. When you use -showcerts option in "openssl
s_client it should show the whole chain. The intermediate and root cert
alone can be imported into the keystore / truststore directly, it doesn't
need to be through PKCS12. But as mentioned before, to have the broker
present the whole chain you have to load the chain already into the PKCS12.
Loading the chain parts separately into the keystore will not help.
In the openssl s_client you can also use the -CAfile option to point it to
the intermediate certificate - that would also wirk but using the whole
chain is probably better.
Regards
Jakub
On Thu, Sep 21, 2017 at 2:00 AM, karan alang <ka...@gmail.com> wrote:
> Another point ..
> on adding the chain.p12 (PKCS#12 format) .. order of the certs in the
> keystore is as shown below
>
> Alias : 1
> Cert[1] -> server cert
> Cert[2] -> Intermediate cert
> Cert[3] -> Root cert
>
> mentioning that, since one of the articles i read mentioned that the certs
> imported should be in following order -> 1) Root 2) Intermediate 3) Server
>
>
> On Wed, Sep 20, 2017 at 4:48 PM, karan alang <ka...@gmail.com>
> wrote:
>
> > Hi Jakub,
> >
> > Thanks for the detailed note...
> >
> > here is the update ->
> >
> > I was able to convert the host.cert.pem to PKCS#12 & import the cert into
> > the kafka.server.keystore.jks
> > (also into kafka.server.truststore.jks)
> > wrt the host.root.pem & host.intermed.pem certs - i'm assuming i need to
> > convert them to PKCS#12 as well ?
> > Or do i import them as PEM file ?
> >
> > I was not able to convert them to PKCS#12(since i understand the PKCS#12
> > format requires a private key & the root.pem & intermed.pem donot have a
> > private key associated)
> > the error is shown below -
> >
> > openssl pkcs12 -export -out intermed.p12 -in hostname.issuer.pem -inkey
> >> privatekey-kafka04-0920.key.pem -password pass:<password>
> >> Enter pass phrase for privatekey-kafka04-0920.key.pem:
> >> No certificate matches private key
> >
> >
> >
> > What i did next ->
> >
> > The host.chain.cert has all the 3 certs (server, intermediate, root)
> > and i converted this to PKCS format & also was able to import into the
> > keystore/truststore.
> > I'm able to see the 3 certificates in the keystore/truststore under ONE
> > alias.
> >
> >
> > Now i get a different error shown below ->
> >
> > Command :
> > openssl s_client -debug -connect <hostname>:9192 -tls1
> >
> > Error :
> >
> > depth=0 CN = <hostname>, OU = <OU>, O = <Organization>, ST = <State>, C =
> >> US
> >> verify error:num=20:unable to get local issuer certificate
> >> verify return:1
> >> depth=0 CN = <hostname>, OU = <OU>, O = <Organization>, ST = <State>, C
> =
> >> US
> >> verify error:num=27:certificate not trusted
> >> verify return:1
> >> depth=0 CN = <hostname>, OU = <OU>, O = <Organization>, ST = <State>, C
> =
> >> US
> >> verify error:num=21:unable to verify the first certificate
> >> verify return:1
> >> write to 0xd47830 [0xd98100] (12 bytes => 12 (0xC))
> >> ---------
> >> ---------
> >> 0930 - 74 20 47 6c 6f 62 61 6c-20 43 41 0e t Global CA.
> >> 093f - <SPACES/NULS>
> >> write to 0xd47830 [0xd98100] (37 bytes => -1 (0xFFFFFFFFFFFFFFFF))
> >> 140456850737056:error:1408E0F4:SSL routines:SSL3_GET_MESSAGE:unexpected
> >> message:s3_both.c:491:
> >> --------
> >> --------
> >> SSL-Session:
> >> Protocol : TLSv1
> >> ------
> >> ------
> >> Timeout : 7200 (sec)
> >> Verify return code: 21 (unable to verify the first certificate)
> >
> >
> >
> > Any ideas on how to fix this ?
> >
> >
> > On Wed, Sep 20, 2017 at 2:22 PM, Jakub Scholz <ja...@scholz.cz> wrote:
> >
> >> ad 1) The problem is that the signed certificate (host.cert.pem) which
> the
> >> CA provides is only the public key. You have to combine it with the
> >> private
> >> key which you created when requesting the signed certificate. The
> private
> >> key is never sent to the CA so they cannot provide it back. You or
> whoever
> >> created the signing request should have it.
> >>
> >> ad 2) To create the keystore with your signed key, you have to take the
> >> signed public key and the private key and create P12 file:
> >> $ openssl pkcs12 -export -out server.p12 -in host.cert.pem -inkey
> host.key
> >> -password pass:somepassword
> >>
> >> And afterwards convert it to keystore format:
> >> $ keytool -importkeystore -srckeystore server.p12 -srcstoretype PKCS12
> >> -srcstorepass somepassword -destkeystore server.keystore -deststoretype
> >> JKS
> >> -deststorepass somepassword -noprompt
> >>
> >> You can add the intermediate and root to the keystore as well. But when
> it
> >> tells you that you already have them in the system wide keystore say
> "yes"
> >> that you want to add them to the keystore again.
> >>
> >> ad 3) Do you want your clients to authenticate with a client
> certificates?
> >> If not, you should not need the truststore for the broker. You only
> need a
> >> truststore for the client which should contain the root and the
> >> intermediate certificates. If you want clients to authenticate using
> >> certificates you have to do basically the same as above for the client
> >> (You
> >> should use different certificate for cleints from the one in server -
> for
> >> security reasons ;-)).
> >>
> >> ad 4) As I said, I think what you did before was that the broker
> basically
> >> uses the self-signed cert you generated. This is probably the default
> >> subject distinguished name of the self signed certificate. And since it
> is
> >> self-signed openssl cannot verify its identity (unless you pass it the
> >> public key of this exact self signed certificate).
> >>
> >> Jakub
> >>
> >> On Wed, Sep 20, 2017 at 10:46 PM, karan alang <ka...@gmail.com>
> >> wrote:
> >>
> >> > Hello
> >> > - thanks for the response
> >> > Here is the update on the issue.
> >> >
> >> > I'm using certs signed/provided by org-wide CA (geotrust, not a
> >> self-signed
> >> > cert)
> >> > The Signed(by the CA - geotrust) cert provided has 3 certificates
> >> > - host.chain.pem (certificate chain - contains the Root, Intermediate,
> >> > Signed Server cert )
> >> > - host.intermediate.pem (intermediate certificate)
> >> > - host.cert.pem (signed server sert)
> >> > Steps :
> >> > 1) Added the signed cert to kafka.server.keystore.jks
> >> >
> >> >
> >> > > keytool -import -alias servercert -trustcacerts -file host.cert.pem
> >> > > -keystore kafka.server.keystore.jks
> >> >
> >> >
> >> > *btw, Is the command above correct or needs to be changed ? *
> >> >
> >> > 2) Added the Intermediate & Signed cert to kafka.server.keystore.jks
> >> >
> >> > > keytool -import -alias intermediate -trustcacerts -file
> >> host.issuer.pem
> >> > > -keystore kafka.server.truststore.jks
> >> > > keytool -import -alias servercert -trustcacerts -file host.cert.pem
> >> > > -keystore kafka.server.truststore.jks
> >> >
> >> >
> >> > When i try to add the Root Cert, it shows the following message.
> >> >
> >> > *Enter keystore password: *
> >> > * Certificate already exists in system-wide CA keystore under alias
> >> > <geotrustglobalca [jdk]>*
> >> > * Do you still want to add it to your own keystore? [no]: *
> >> >
> >> > So, i assume the CA is already trusted .. do i still need to add this
> to
> >> > the kafka.server.truststore.jks ?
> >> > (I think i did that, and it did not fix the issue)
> >> > 3) In the Broker side (server.properties file), the setting is as
> shown
> >> > below
> >> > (this is the same setting when i enabled SSL using OpenSSL, which is
> >> > working fine btw)
> >> >
> >> > > ssl.keystore.location=/usr/hdp/2.5.3.0-37/confluent-3.2.
> >> > > 2/kafkaSSL/kafka.server.keystore.jks
> >> > > ssl.keystore.password=<passwd>
> >> > > ssl.key.password=<passwd>
> >> > > ssl.truststore.location=/usr/hdp/2.5.3.0-37/confluent-3.2.
> >> > > 2/kafkaSSL/kafka.server.truststore.jks
> >> > > ssl.truststore.password=<passwd>
> >> > > ssl.client.auth=required
> >> >
> >> > 4) when i verify the SSL using command below
> >> >
> >> > Command :
> >> >
> >> > > openssl s_client -debug -connect <hostname>:9192 -tls1
> >> >
> >> >
> >> > Error snippet :
> >> >
> >> >
> >> > >
> >> > >
> >> > > depth=0 C = US, ST = <State Name>, L = Unknown, O = <org name>., OU
> =
> >> > > <OU>, CN = <localhost>
> >> > >
> >> > verify error:num=18:self signed certificate
> >> > >
> >> > > verify return:1
> >> > > ....................
> >> > >
> >> > -----------------
> >> > ------------------
> >> >
> >> >
> >> > > 063d - <SPACES/NULS>
> >> > > write to 0x860830 [0x8b1100] (37 bytes => -1 (0xFFFFFFFFFFFFFFFF))
> >> > > 140487373043616:error:1408E0F4:SSL routines:SSL3_GET_MESSAGE:unex
> >> pected
> >> > > message:s3_both.c:491:
> >> >
> >> >
> >> > Any ideas on this ?
> >> > What needs to be done to debug or fix this ?
> >> >
> >> > On Wed, Sep 20, 2017 at 12:33 AM, Jakub Scholz <ja...@scholz.cz>
> wrote:
> >> >
> >> > > Hi,
> >> > >
> >> > > Looking at your commands it looks as if you generated a self signed
> >> key
> >> > for
> >> > > server, self signed key for client and then imported the CA keys
> >> public
> >> > > keys into the truststores. I don’t think this will work because now
> >> you
> >> > > have two different self signed keys in the keystores and the
> >> presumably
> >> > the
> >> > > CA public key in your truststores.
> >> > >
> >> > > Does the cert.pem actualy contain the private key? In case it
> doesn’t
> >> > > contain the private key you need to get the private key first. If
> yes,
> >> > you
> >> > > have to convert it into the keystore format using this guide:
> >> > > https://docs.oracle.com/cd/E35976_01/server.740/es_admin/
> >> > > src/tadm_ssl_convert_pem_to_jks.html
> >> > > You basically have to first use OpenSSL to create PKCS12 key and
> that
> >> can
> >> > > be converted into keystore.
> >> > >
> >> > > BTW: If you run your application with system property
> >> “javax.net.debug”
> >> > set
> >> > > to “ssl” it will generate a lot of useful debug information which
> will
> >> > help
> >> > > to understand what is going on and fix this.
> >> > >
> >> > > Jakub
> >> > >
> >> > > On Tue, 19 Sep 2017 at 23:44, karan alang <ka...@gmail.com>
> >> wrote:
> >> > >
> >> > > > Hello All -
> >> > > > I was able to set up SSL for the Kafka brokers, using OpenSSL.
> >> > > >
> >> > > > however, I'm having issues with setting up SSL using the pem file
> >> (i.e.
> >> > > SSL
> >> > > > certificate - certified by CA, provided by the company)
> >> > > >
> >> > > > Here is what i've done -
> >> > > > created the server/client keystore & truststore files and imported
> >> the
> >> > > > provided cert.pem file
> >> > > >
> >> > > > keytool -keystore kafka.server.keystore.jks -alias localhost
> >> -validity
> >> > > 365
> >> > > > -genkey
> >> > > > keytool -keystore kafka.server.truststore.jks -alias CARoot
> -import
> >> > -file
> >> > > > cert.pem
> >> > > > keytool -keystore kafka.client.truststore.jks -alias CARoot
> -import
> >> > -file
> >> > > > cert.pem
> >> > > > keytool -keystore kafka.server.keystore.jks -alias CARoot -import
> >> -file
> >> > > > cert.pem
> >> > > > keytool -keystore kafka.client.keystore.jks -alias localhost
> >> -validity
> >> > > 365
> >> > > > -genkey
> >> > > > keytool -keystore kafka.client.keystore.jks -alias CARoot -import
> >> -file
> >> > > > cert.pem
> >> > > >
> >> > > > I've a console producer pushing data in to the topic, and gives
> >> error
> >> > as
> >> > > > shown below ->
> >> > > >
> >> > > >
> >> > > > Caused by: javax.net.ssl.SSLProtocolException: Handshake message
> >> > > sequence
> >> > > > > violation, state = 1, type = 1
> >> > > > > at
> >> > > > >
> >> > > > sun.security.ssl.ServerHandshaker.processMessage(
> >> > > ServerHandshaker.java:213)
> >> > > > > at sun.security.ssl.Handshaker.processLoop(Handshaker.java:
> 1026)
> >> > > > > at sun.security.ssl.Handshaker$1.run(Handshaker.java:966)
> >> > > > > at sun.security.ssl.Handshaker$1.run(Handshaker.java:963)
> >> > > > > at java.security.AccessController.doPrivileged(Native Method)
> >> > > > > at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.
> >> > java:1416)
> >> > > > > at
> >> > > > >
> >> > > > org.apache.kafka.common.network.SslTransportLayer.runDelegat
> >> edTasks(
> >> > > SslTransportLayer.java:336)
> >> > > > > at
> >> > > > >
> >> > > > org.apache.kafka.common.network.SslTransportLayer.
> handshakeUnwrap(
> >> > > SslTransportLayer.java:417)
> >> > > > > ... 7 more
> >> > > >
> >> > > >
> >> > > >
> >> > > > Any ideas on what the issue might be ?
> >> > > > thanks for help in advance!
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>
Re: Kafka SSL error
Posted by karan alang <ka...@gmail.com>.
Another point ..
on adding the chain.p12 (PKCS#12 format) .. order of the certs in the
keystore is as shown below
Alias : 1
Cert[1] -> server cert
Cert[2] -> Intermediate cert
Cert[3] -> Root cert
mentioning that, since one of the articles i read mentioned that the certs
imported should be in following order -> 1) Root 2) Intermediate 3) Server
On Wed, Sep 20, 2017 at 4:48 PM, karan alang <ka...@gmail.com> wrote:
> Hi Jakub,
>
> Thanks for the detailed note...
>
> here is the update ->
>
> I was able to convert the host.cert.pem to PKCS#12 & import the cert into
> the kafka.server.keystore.jks
> (also into kafka.server.truststore.jks)
> wrt the host.root.pem & host.intermed.pem certs - i'm assuming i need to
> convert them to PKCS#12 as well ?
> Or do i import them as PEM file ?
>
> I was not able to convert them to PKCS#12(since i understand the PKCS#12
> format requires a private key & the root.pem & intermed.pem donot have a
> private key associated)
> the error is shown below -
>
> openssl pkcs12 -export -out intermed.p12 -in hostname.issuer.pem -inkey
>> privatekey-kafka04-0920.key.pem -password pass:<password>
>> Enter pass phrase for privatekey-kafka04-0920.key.pem:
>> No certificate matches private key
>
>
>
> What i did next ->
>
> The host.chain.cert has all the 3 certs (server, intermediate, root)
> and i converted this to PKCS format & also was able to import into the
> keystore/truststore.
> I'm able to see the 3 certificates in the keystore/truststore under ONE
> alias.
>
>
> Now i get a different error shown below ->
>
> Command :
> openssl s_client -debug -connect <hostname>:9192 -tls1
>
> Error :
>
> depth=0 CN = <hostname>, OU = <OU>, O = <Organization>, ST = <State>, C =
>> US
>> verify error:num=20:unable to get local issuer certificate
>> verify return:1
>> depth=0 CN = <hostname>, OU = <OU>, O = <Organization>, ST = <State>, C =
>> US
>> verify error:num=27:certificate not trusted
>> verify return:1
>> depth=0 CN = <hostname>, OU = <OU>, O = <Organization>, ST = <State>, C =
>> US
>> verify error:num=21:unable to verify the first certificate
>> verify return:1
>> write to 0xd47830 [0xd98100] (12 bytes => 12 (0xC))
>> ---------
>> ---------
>> 0930 - 74 20 47 6c 6f 62 61 6c-20 43 41 0e t Global CA.
>> 093f - <SPACES/NULS>
>> write to 0xd47830 [0xd98100] (37 bytes => -1 (0xFFFFFFFFFFFFFFFF))
>> 140456850737056:error:1408E0F4:SSL routines:SSL3_GET_MESSAGE:unexpected
>> message:s3_both.c:491:
>> --------
>> --------
>> SSL-Session:
>> Protocol : TLSv1
>> ------
>> ------
>> Timeout : 7200 (sec)
>> Verify return code: 21 (unable to verify the first certificate)
>
>
>
> Any ideas on how to fix this ?
>
>
> On Wed, Sep 20, 2017 at 2:22 PM, Jakub Scholz <ja...@scholz.cz> wrote:
>
>> ad 1) The problem is that the signed certificate (host.cert.pem) which the
>> CA provides is only the public key. You have to combine it with the
>> private
>> key which you created when requesting the signed certificate. The private
>> key is never sent to the CA so they cannot provide it back. You or whoever
>> created the signing request should have it.
>>
>> ad 2) To create the keystore with your signed key, you have to take the
>> signed public key and the private key and create P12 file:
>> $ openssl pkcs12 -export -out server.p12 -in host.cert.pem -inkey host.key
>> -password pass:somepassword
>>
>> And afterwards convert it to keystore format:
>> $ keytool -importkeystore -srckeystore server.p12 -srcstoretype PKCS12
>> -srcstorepass somepassword -destkeystore server.keystore -deststoretype
>> JKS
>> -deststorepass somepassword -noprompt
>>
>> You can add the intermediate and root to the keystore as well. But when it
>> tells you that you already have them in the system wide keystore say "yes"
>> that you want to add them to the keystore again.
>>
>> ad 3) Do you want your clients to authenticate with a client certificates?
>> If not, you should not need the truststore for the broker. You only need a
>> truststore for the client which should contain the root and the
>> intermediate certificates. If you want clients to authenticate using
>> certificates you have to do basically the same as above for the client
>> (You
>> should use different certificate for cleints from the one in server - for
>> security reasons ;-)).
>>
>> ad 4) As I said, I think what you did before was that the broker basically
>> uses the self-signed cert you generated. This is probably the default
>> subject distinguished name of the self signed certificate. And since it is
>> self-signed openssl cannot verify its identity (unless you pass it the
>> public key of this exact self signed certificate).
>>
>> Jakub
>>
>> On Wed, Sep 20, 2017 at 10:46 PM, karan alang <ka...@gmail.com>
>> wrote:
>>
>> > Hello
>> > - thanks for the response
>> > Here is the update on the issue.
>> >
>> > I'm using certs signed/provided by org-wide CA (geotrust, not a
>> self-signed
>> > cert)
>> > The Signed(by the CA - geotrust) cert provided has 3 certificates
>> > - host.chain.pem (certificate chain - contains the Root, Intermediate,
>> > Signed Server cert )
>> > - host.intermediate.pem (intermediate certificate)
>> > - host.cert.pem (signed server sert)
>> > Steps :
>> > 1) Added the signed cert to kafka.server.keystore.jks
>> >
>> >
>> > > keytool -import -alias servercert -trustcacerts -file host.cert.pem
>> > > -keystore kafka.server.keystore.jks
>> >
>> >
>> > *btw, Is the command above correct or needs to be changed ? *
>> >
>> > 2) Added the Intermediate & Signed cert to kafka.server.keystore.jks
>> >
>> > > keytool -import -alias intermediate -trustcacerts -file
>> host.issuer.pem
>> > > -keystore kafka.server.truststore.jks
>> > > keytool -import -alias servercert -trustcacerts -file host.cert.pem
>> > > -keystore kafka.server.truststore.jks
>> >
>> >
>> > When i try to add the Root Cert, it shows the following message.
>> >
>> > *Enter keystore password: *
>> > * Certificate already exists in system-wide CA keystore under alias
>> > <geotrustglobalca [jdk]>*
>> > * Do you still want to add it to your own keystore? [no]: *
>> >
>> > So, i assume the CA is already trusted .. do i still need to add this to
>> > the kafka.server.truststore.jks ?
>> > (I think i did that, and it did not fix the issue)
>> > 3) In the Broker side (server.properties file), the setting is as shown
>> > below
>> > (this is the same setting when i enabled SSL using OpenSSL, which is
>> > working fine btw)
>> >
>> > > ssl.keystore.location=/usr/hdp/2.5.3.0-37/confluent-3.2.
>> > > 2/kafkaSSL/kafka.server.keystore.jks
>> > > ssl.keystore.password=<passwd>
>> > > ssl.key.password=<passwd>
>> > > ssl.truststore.location=/usr/hdp/2.5.3.0-37/confluent-3.2.
>> > > 2/kafkaSSL/kafka.server.truststore.jks
>> > > ssl.truststore.password=<passwd>
>> > > ssl.client.auth=required
>> >
>> > 4) when i verify the SSL using command below
>> >
>> > Command :
>> >
>> > > openssl s_client -debug -connect <hostname>:9192 -tls1
>> >
>> >
>> > Error snippet :
>> >
>> >
>> > >
>> > >
>> > > depth=0 C = US, ST = <State Name>, L = Unknown, O = <org name>., OU =
>> > > <OU>, CN = <localhost>
>> > >
>> > verify error:num=18:self signed certificate
>> > >
>> > > verify return:1
>> > > ....................
>> > >
>> > -----------------
>> > ------------------
>> >
>> >
>> > > 063d - <SPACES/NULS>
>> > > write to 0x860830 [0x8b1100] (37 bytes => -1 (0xFFFFFFFFFFFFFFFF))
>> > > 140487373043616:error:1408E0F4:SSL routines:SSL3_GET_MESSAGE:unex
>> pected
>> > > message:s3_both.c:491:
>> >
>> >
>> > Any ideas on this ?
>> > What needs to be done to debug or fix this ?
>> >
>> > On Wed, Sep 20, 2017 at 12:33 AM, Jakub Scholz <ja...@scholz.cz> wrote:
>> >
>> > > Hi,
>> > >
>> > > Looking at your commands it looks as if you generated a self signed
>> key
>> > for
>> > > server, self signed key for client and then imported the CA keys
>> public
>> > > keys into the truststores. I don’t think this will work because now
>> you
>> > > have two different self signed keys in the keystores and the
>> presumably
>> > the
>> > > CA public key in your truststores.
>> > >
>> > > Does the cert.pem actualy contain the private key? In case it doesn’t
>> > > contain the private key you need to get the private key first. If yes,
>> > you
>> > > have to convert it into the keystore format using this guide:
>> > > https://docs.oracle.com/cd/E35976_01/server.740/es_admin/
>> > > src/tadm_ssl_convert_pem_to_jks.html
>> > > You basically have to first use OpenSSL to create PKCS12 key and that
>> can
>> > > be converted into keystore.
>> > >
>> > > BTW: If you run your application with system property
>> “javax.net.debug”
>> > set
>> > > to “ssl” it will generate a lot of useful debug information which will
>> > help
>> > > to understand what is going on and fix this.
>> > >
>> > > Jakub
>> > >
>> > > On Tue, 19 Sep 2017 at 23:44, karan alang <ka...@gmail.com>
>> wrote:
>> > >
>> > > > Hello All -
>> > > > I was able to set up SSL for the Kafka brokers, using OpenSSL.
>> > > >
>> > > > however, I'm having issues with setting up SSL using the pem file
>> (i.e.
>> > > SSL
>> > > > certificate - certified by CA, provided by the company)
>> > > >
>> > > > Here is what i've done -
>> > > > created the server/client keystore & truststore files and imported
>> the
>> > > > provided cert.pem file
>> > > >
>> > > > keytool -keystore kafka.server.keystore.jks -alias localhost
>> -validity
>> > > 365
>> > > > -genkey
>> > > > keytool -keystore kafka.server.truststore.jks -alias CARoot -import
>> > -file
>> > > > cert.pem
>> > > > keytool -keystore kafka.client.truststore.jks -alias CARoot -import
>> > -file
>> > > > cert.pem
>> > > > keytool -keystore kafka.server.keystore.jks -alias CARoot -import
>> -file
>> > > > cert.pem
>> > > > keytool -keystore kafka.client.keystore.jks -alias localhost
>> -validity
>> > > 365
>> > > > -genkey
>> > > > keytool -keystore kafka.client.keystore.jks -alias CARoot -import
>> -file
>> > > > cert.pem
>> > > >
>> > > > I've a console producer pushing data in to the topic, and gives
>> error
>> > as
>> > > > shown below ->
>> > > >
>> > > >
>> > > > Caused by: javax.net.ssl.SSLProtocolException: Handshake message
>> > > sequence
>> > > > > violation, state = 1, type = 1
>> > > > > at
>> > > > >
>> > > > sun.security.ssl.ServerHandshaker.processMessage(
>> > > ServerHandshaker.java:213)
>> > > > > at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026)
>> > > > > at sun.security.ssl.Handshaker$1.run(Handshaker.java:966)
>> > > > > at sun.security.ssl.Handshaker$1.run(Handshaker.java:963)
>> > > > > at java.security.AccessController.doPrivileged(Native Method)
>> > > > > at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.
>> > java:1416)
>> > > > > at
>> > > > >
>> > > > org.apache.kafka.common.network.SslTransportLayer.runDelegat
>> edTasks(
>> > > SslTransportLayer.java:336)
>> > > > > at
>> > > > >
>> > > > org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(
>> > > SslTransportLayer.java:417)
>> > > > > ... 7 more
>> > > >
>> > > >
>> > > >
>> > > > Any ideas on what the issue might be ?
>> > > > thanks for help in advance!
>> > > >
>> > >
>> >
>>
>
>
Re: Kafka SSL error
Posted by karan alang <ka...@gmail.com>.
Hi Jakub,
Thanks for the detailed note...
here is the update ->
I was able to convert the host.cert.pem to PKCS#12 & import the cert into
the kafka.server.keystore.jks
(also into kafka.server.truststore.jks)
wrt the host.root.pem & host.intermed.pem certs - i'm assuming i need to
convert them to PKCS#12 as well ?
Or do i import them as PEM file ?
I was not able to convert them to PKCS#12(since i understand the PKCS#12
format requires a private key & the root.pem & intermed.pem donot have a
private key associated)
the error is shown below -
openssl pkcs12 -export -out intermed.p12 -in hostname.issuer.pem -inkey
> privatekey-kafka04-0920.key.pem -password pass:<password>
> Enter pass phrase for privatekey-kafka04-0920.key.pem:
> No certificate matches private key
What i did next ->
The host.chain.cert has all the 3 certs (server, intermediate, root)
and i converted this to PKCS format & also was able to import into the
keystore/truststore.
I'm able to see the 3 certificates in the keystore/truststore under ONE
alias.
Now i get a different error shown below ->
Command :
openssl s_client -debug -connect <hostname>:9192 -tls1
Error :
depth=0 CN = <hostname>, OU = <OU>, O = <Organization>, ST = <State>, C = US
> verify error:num=20:unable to get local issuer certificate
> verify return:1
> depth=0 CN = <hostname>, OU = <OU>, O = <Organization>, ST = <State>, C =
> US
> verify error:num=27:certificate not trusted
> verify return:1
> depth=0 CN = <hostname>, OU = <OU>, O = <Organization>, ST = <State>, C =
> US
> verify error:num=21:unable to verify the first certificate
> verify return:1
> write to 0xd47830 [0xd98100] (12 bytes => 12 (0xC))
> ---------
> ---------
> 0930 - 74 20 47 6c 6f 62 61 6c-20 43 41 0e t Global CA.
> 093f - <SPACES/NULS>
> write to 0xd47830 [0xd98100] (37 bytes => -1 (0xFFFFFFFFFFFFFFFF))
> 140456850737056:error:1408E0F4:SSL routines:SSL3_GET_MESSAGE:unexpected
> message:s3_both.c:491:
> --------
> --------
> SSL-Session:
> Protocol : TLSv1
> ------
> ------
> Timeout : 7200 (sec)
> Verify return code: 21 (unable to verify the first certificate)
Any ideas on how to fix this ?
On Wed, Sep 20, 2017 at 2:22 PM, Jakub Scholz <ja...@scholz.cz> wrote:
> ad 1) The problem is that the signed certificate (host.cert.pem) which the
> CA provides is only the public key. You have to combine it with the private
> key which you created when requesting the signed certificate. The private
> key is never sent to the CA so they cannot provide it back. You or whoever
> created the signing request should have it.
>
> ad 2) To create the keystore with your signed key, you have to take the
> signed public key and the private key and create P12 file:
> $ openssl pkcs12 -export -out server.p12 -in host.cert.pem -inkey host.key
> -password pass:somepassword
>
> And afterwards convert it to keystore format:
> $ keytool -importkeystore -srckeystore server.p12 -srcstoretype PKCS12
> -srcstorepass somepassword -destkeystore server.keystore -deststoretype JKS
> -deststorepass somepassword -noprompt
>
> You can add the intermediate and root to the keystore as well. But when it
> tells you that you already have them in the system wide keystore say "yes"
> that you want to add them to the keystore again.
>
> ad 3) Do you want your clients to authenticate with a client certificates?
> If not, you should not need the truststore for the broker. You only need a
> truststore for the client which should contain the root and the
> intermediate certificates. If you want clients to authenticate using
> certificates you have to do basically the same as above for the client (You
> should use different certificate for cleints from the one in server - for
> security reasons ;-)).
>
> ad 4) As I said, I think what you did before was that the broker basically
> uses the self-signed cert you generated. This is probably the default
> subject distinguished name of the self signed certificate. And since it is
> self-signed openssl cannot verify its identity (unless you pass it the
> public key of this exact self signed certificate).
>
> Jakub
>
> On Wed, Sep 20, 2017 at 10:46 PM, karan alang <ka...@gmail.com>
> wrote:
>
> > Hello
> > - thanks for the response
> > Here is the update on the issue.
> >
> > I'm using certs signed/provided by org-wide CA (geotrust, not a
> self-signed
> > cert)
> > The Signed(by the CA - geotrust) cert provided has 3 certificates
> > - host.chain.pem (certificate chain - contains the Root, Intermediate,
> > Signed Server cert )
> > - host.intermediate.pem (intermediate certificate)
> > - host.cert.pem (signed server sert)
> > Steps :
> > 1) Added the signed cert to kafka.server.keystore.jks
> >
> >
> > > keytool -import -alias servercert -trustcacerts -file host.cert.pem
> > > -keystore kafka.server.keystore.jks
> >
> >
> > *btw, Is the command above correct or needs to be changed ? *
> >
> > 2) Added the Intermediate & Signed cert to kafka.server.keystore.jks
> >
> > > keytool -import -alias intermediate -trustcacerts -file host.issuer.pem
> > > -keystore kafka.server.truststore.jks
> > > keytool -import -alias servercert -trustcacerts -file host.cert.pem
> > > -keystore kafka.server.truststore.jks
> >
> >
> > When i try to add the Root Cert, it shows the following message.
> >
> > *Enter keystore password: *
> > * Certificate already exists in system-wide CA keystore under alias
> > <geotrustglobalca [jdk]>*
> > * Do you still want to add it to your own keystore? [no]: *
> >
> > So, i assume the CA is already trusted .. do i still need to add this to
> > the kafka.server.truststore.jks ?
> > (I think i did that, and it did not fix the issue)
> > 3) In the Broker side (server.properties file), the setting is as shown
> > below
> > (this is the same setting when i enabled SSL using OpenSSL, which is
> > working fine btw)
> >
> > > ssl.keystore.location=/usr/hdp/2.5.3.0-37/confluent-3.2.
> > > 2/kafkaSSL/kafka.server.keystore.jks
> > > ssl.keystore.password=<passwd>
> > > ssl.key.password=<passwd>
> > > ssl.truststore.location=/usr/hdp/2.5.3.0-37/confluent-3.2.
> > > 2/kafkaSSL/kafka.server.truststore.jks
> > > ssl.truststore.password=<passwd>
> > > ssl.client.auth=required
> >
> > 4) when i verify the SSL using command below
> >
> > Command :
> >
> > > openssl s_client -debug -connect <hostname>:9192 -tls1
> >
> >
> > Error snippet :
> >
> >
> > >
> > >
> > > depth=0 C = US, ST = <State Name>, L = Unknown, O = <org name>., OU =
> > > <OU>, CN = <localhost>
> > >
> > verify error:num=18:self signed certificate
> > >
> > > verify return:1
> > > ....................
> > >
> > -----------------
> > ------------------
> >
> >
> > > 063d - <SPACES/NULS>
> > > write to 0x860830 [0x8b1100] (37 bytes => -1 (0xFFFFFFFFFFFFFFFF))
> > > 140487373043616:error:1408E0F4:SSL routines:SSL3_GET_MESSAGE:
> unexpected
> > > message:s3_both.c:491:
> >
> >
> > Any ideas on this ?
> > What needs to be done to debug or fix this ?
> >
> > On Wed, Sep 20, 2017 at 12:33 AM, Jakub Scholz <ja...@scholz.cz> wrote:
> >
> > > Hi,
> > >
> > > Looking at your commands it looks as if you generated a self signed key
> > for
> > > server, self signed key for client and then imported the CA keys public
> > > keys into the truststores. I don’t think this will work because now you
> > > have two different self signed keys in the keystores and the presumably
> > the
> > > CA public key in your truststores.
> > >
> > > Does the cert.pem actualy contain the private key? In case it doesn’t
> > > contain the private key you need to get the private key first. If yes,
> > you
> > > have to convert it into the keystore format using this guide:
> > > https://docs.oracle.com/cd/E35976_01/server.740/es_admin/
> > > src/tadm_ssl_convert_pem_to_jks.html
> > > You basically have to first use OpenSSL to create PKCS12 key and that
> can
> > > be converted into keystore.
> > >
> > > BTW: If you run your application with system property “javax.net.debug”
> > set
> > > to “ssl” it will generate a lot of useful debug information which will
> > help
> > > to understand what is going on and fix this.
> > >
> > > Jakub
> > >
> > > On Tue, 19 Sep 2017 at 23:44, karan alang <ka...@gmail.com>
> wrote:
> > >
> > > > Hello All -
> > > > I was able to set up SSL for the Kafka brokers, using OpenSSL.
> > > >
> > > > however, I'm having issues with setting up SSL using the pem file
> (i.e.
> > > SSL
> > > > certificate - certified by CA, provided by the company)
> > > >
> > > > Here is what i've done -
> > > > created the server/client keystore & truststore files and imported
> the
> > > > provided cert.pem file
> > > >
> > > > keytool -keystore kafka.server.keystore.jks -alias localhost
> -validity
> > > 365
> > > > -genkey
> > > > keytool -keystore kafka.server.truststore.jks -alias CARoot -import
> > -file
> > > > cert.pem
> > > > keytool -keystore kafka.client.truststore.jks -alias CARoot -import
> > -file
> > > > cert.pem
> > > > keytool -keystore kafka.server.keystore.jks -alias CARoot -import
> -file
> > > > cert.pem
> > > > keytool -keystore kafka.client.keystore.jks -alias localhost
> -validity
> > > 365
> > > > -genkey
> > > > keytool -keystore kafka.client.keystore.jks -alias CARoot -import
> -file
> > > > cert.pem
> > > >
> > > > I've a console producer pushing data in to the topic, and gives error
> > as
> > > > shown below ->
> > > >
> > > >
> > > > Caused by: javax.net.ssl.SSLProtocolException: Handshake message
> > > sequence
> > > > > violation, state = 1, type = 1
> > > > > at
> > > > >
> > > > sun.security.ssl.ServerHandshaker.processMessage(
> > > ServerHandshaker.java:213)
> > > > > at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026)
> > > > > at sun.security.ssl.Handshaker$1.run(Handshaker.java:966)
> > > > > at sun.security.ssl.Handshaker$1.run(Handshaker.java:963)
> > > > > at java.security.AccessController.doPrivileged(Native Method)
> > > > > at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.
> > java:1416)
> > > > > at
> > > > >
> > > > org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(
> > > SslTransportLayer.java:336)
> > > > > at
> > > > >
> > > > org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(
> > > SslTransportLayer.java:417)
> > > > > ... 7 more
> > > >
> > > >
> > > >
> > > > Any ideas on what the issue might be ?
> > > > thanks for help in advance!
> > > >
> > >
> >
>
Re: Kafka SSL error
Posted by Jakub Scholz <ja...@scholz.cz>.
ad 1) The problem is that the signed certificate (host.cert.pem) which the
CA provides is only the public key. You have to combine it with the private
key which you created when requesting the signed certificate. The private
key is never sent to the CA so they cannot provide it back. You or whoever
created the signing request should have it.
ad 2) To create the keystore with your signed key, you have to take the
signed public key and the private key and create P12 file:
$ openssl pkcs12 -export -out server.p12 -in host.cert.pem -inkey host.key
-password pass:somepassword
And afterwards convert it to keystore format:
$ keytool -importkeystore -srckeystore server.p12 -srcstoretype PKCS12
-srcstorepass somepassword -destkeystore server.keystore -deststoretype JKS
-deststorepass somepassword -noprompt
You can add the intermediate and root to the keystore as well. But when it
tells you that you already have them in the system wide keystore say "yes"
that you want to add them to the keystore again.
ad 3) Do you want your clients to authenticate with a client certificates?
If not, you should not need the truststore for the broker. You only need a
truststore for the client which should contain the root and the
intermediate certificates. If you want clients to authenticate using
certificates you have to do basically the same as above for the client (You
should use different certificate for cleints from the one in server - for
security reasons ;-)).
ad 4) As I said, I think what you did before was that the broker basically
uses the self-signed cert you generated. This is probably the default
subject distinguished name of the self signed certificate. And since it is
self-signed openssl cannot verify its identity (unless you pass it the
public key of this exact self signed certificate).
Jakub
On Wed, Sep 20, 2017 at 10:46 PM, karan alang <ka...@gmail.com> wrote:
> Hello
> - thanks for the response
> Here is the update on the issue.
>
> I'm using certs signed/provided by org-wide CA (geotrust, not a self-signed
> cert)
> The Signed(by the CA - geotrust) cert provided has 3 certificates
> - host.chain.pem (certificate chain - contains the Root, Intermediate,
> Signed Server cert )
> - host.intermediate.pem (intermediate certificate)
> - host.cert.pem (signed server sert)
> Steps :
> 1) Added the signed cert to kafka.server.keystore.jks
>
>
> > keytool -import -alias servercert -trustcacerts -file host.cert.pem
> > -keystore kafka.server.keystore.jks
>
>
> *btw, Is the command above correct or needs to be changed ? *
>
> 2) Added the Intermediate & Signed cert to kafka.server.keystore.jks
>
> > keytool -import -alias intermediate -trustcacerts -file host.issuer.pem
> > -keystore kafka.server.truststore.jks
> > keytool -import -alias servercert -trustcacerts -file host.cert.pem
> > -keystore kafka.server.truststore.jks
>
>
> When i try to add the Root Cert, it shows the following message.
>
> *Enter keystore password: *
> * Certificate already exists in system-wide CA keystore under alias
> <geotrustglobalca [jdk]>*
> * Do you still want to add it to your own keystore? [no]: *
>
> So, i assume the CA is already trusted .. do i still need to add this to
> the kafka.server.truststore.jks ?
> (I think i did that, and it did not fix the issue)
> 3) In the Broker side (server.properties file), the setting is as shown
> below
> (this is the same setting when i enabled SSL using OpenSSL, which is
> working fine btw)
>
> > ssl.keystore.location=/usr/hdp/2.5.3.0-37/confluent-3.2.
> > 2/kafkaSSL/kafka.server.keystore.jks
> > ssl.keystore.password=<passwd>
> > ssl.key.password=<passwd>
> > ssl.truststore.location=/usr/hdp/2.5.3.0-37/confluent-3.2.
> > 2/kafkaSSL/kafka.server.truststore.jks
> > ssl.truststore.password=<passwd>
> > ssl.client.auth=required
>
> 4) when i verify the SSL using command below
>
> Command :
>
> > openssl s_client -debug -connect <hostname>:9192 -tls1
>
>
> Error snippet :
>
>
> >
> >
> > depth=0 C = US, ST = <State Name>, L = Unknown, O = <org name>., OU =
> > <OU>, CN = <localhost>
> >
> verify error:num=18:self signed certificate
> >
> > verify return:1
> > ....................
> >
> -----------------
> ------------------
>
>
> > 063d - <SPACES/NULS>
> > write to 0x860830 [0x8b1100] (37 bytes => -1 (0xFFFFFFFFFFFFFFFF))
> > 140487373043616:error:1408E0F4:SSL routines:SSL3_GET_MESSAGE:unexpected
> > message:s3_both.c:491:
>
>
> Any ideas on this ?
> What needs to be done to debug or fix this ?
>
> On Wed, Sep 20, 2017 at 12:33 AM, Jakub Scholz <ja...@scholz.cz> wrote:
>
> > Hi,
> >
> > Looking at your commands it looks as if you generated a self signed key
> for
> > server, self signed key for client and then imported the CA keys public
> > keys into the truststores. I don’t think this will work because now you
> > have two different self signed keys in the keystores and the presumably
> the
> > CA public key in your truststores.
> >
> > Does the cert.pem actualy contain the private key? In case it doesn’t
> > contain the private key you need to get the private key first. If yes,
> you
> > have to convert it into the keystore format using this guide:
> > https://docs.oracle.com/cd/E35976_01/server.740/es_admin/
> > src/tadm_ssl_convert_pem_to_jks.html
> > You basically have to first use OpenSSL to create PKCS12 key and that can
> > be converted into keystore.
> >
> > BTW: If you run your application with system property “javax.net.debug”
> set
> > to “ssl” it will generate a lot of useful debug information which will
> help
> > to understand what is going on and fix this.
> >
> > Jakub
> >
> > On Tue, 19 Sep 2017 at 23:44, karan alang <ka...@gmail.com> wrote:
> >
> > > Hello All -
> > > I was able to set up SSL for the Kafka brokers, using OpenSSL.
> > >
> > > however, I'm having issues with setting up SSL using the pem file (i.e.
> > SSL
> > > certificate - certified by CA, provided by the company)
> > >
> > > Here is what i've done -
> > > created the server/client keystore & truststore files and imported the
> > > provided cert.pem file
> > >
> > > keytool -keystore kafka.server.keystore.jks -alias localhost -validity
> > 365
> > > -genkey
> > > keytool -keystore kafka.server.truststore.jks -alias CARoot -import
> -file
> > > cert.pem
> > > keytool -keystore kafka.client.truststore.jks -alias CARoot -import
> -file
> > > cert.pem
> > > keytool -keystore kafka.server.keystore.jks -alias CARoot -import -file
> > > cert.pem
> > > keytool -keystore kafka.client.keystore.jks -alias localhost -validity
> > 365
> > > -genkey
> > > keytool -keystore kafka.client.keystore.jks -alias CARoot -import -file
> > > cert.pem
> > >
> > > I've a console producer pushing data in to the topic, and gives error
> as
> > > shown below ->
> > >
> > >
> > > Caused by: javax.net.ssl.SSLProtocolException: Handshake message
> > sequence
> > > > violation, state = 1, type = 1
> > > > at
> > > >
> > > sun.security.ssl.ServerHandshaker.processMessage(
> > ServerHandshaker.java:213)
> > > > at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026)
> > > > at sun.security.ssl.Handshaker$1.run(Handshaker.java:966)
> > > > at sun.security.ssl.Handshaker$1.run(Handshaker.java:963)
> > > > at java.security.AccessController.doPrivileged(Native Method)
> > > > at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.
> java:1416)
> > > > at
> > > >
> > > org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(
> > SslTransportLayer.java:336)
> > > > at
> > > >
> > > org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(
> > SslTransportLayer.java:417)
> > > > ... 7 more
> > >
> > >
> > >
> > > Any ideas on what the issue might be ?
> > > thanks for help in advance!
> > >
> >
>
Re: Kafka SSL error
Posted by karan alang <ka...@gmail.com>.
Hello
- thanks for the response
Here is the update on the issue.
I'm using certs signed/provided by org-wide CA (geotrust, not a self-signed
cert)
The Signed(by the CA - geotrust) cert provided has 3 certificates
- host.chain.pem (certificate chain - contains the Root, Intermediate,
Signed Server cert )
- host.intermediate.pem (intermediate certificate)
- host.cert.pem (signed server sert)
Steps :
1) Added the signed cert to kafka.server.keystore.jks
> keytool -import -alias servercert -trustcacerts -file host.cert.pem
> -keystore kafka.server.keystore.jks
*btw, Is the command above correct or needs to be changed ? *
2) Added the Intermediate & Signed cert to kafka.server.keystore.jks
> keytool -import -alias intermediate -trustcacerts -file host.issuer.pem
> -keystore kafka.server.truststore.jks
> keytool -import -alias servercert -trustcacerts -file host.cert.pem
> -keystore kafka.server.truststore.jks
When i try to add the Root Cert, it shows the following message.
*Enter keystore password: *
* Certificate already exists in system-wide CA keystore under alias
<geotrustglobalca [jdk]>*
* Do you still want to add it to your own keystore? [no]: *
So, i assume the CA is already trusted .. do i still need to add this to
the kafka.server.truststore.jks ?
(I think i did that, and it did not fix the issue)
3) In the Broker side (server.properties file), the setting is as shown
below
(this is the same setting when i enabled SSL using OpenSSL, which is
working fine btw)
> ssl.keystore.location=/usr/hdp/2.5.3.0-37/confluent-3.2.
> 2/kafkaSSL/kafka.server.keystore.jks
> ssl.keystore.password=<passwd>
> ssl.key.password=<passwd>
> ssl.truststore.location=/usr/hdp/2.5.3.0-37/confluent-3.2.
> 2/kafkaSSL/kafka.server.truststore.jks
> ssl.truststore.password=<passwd>
> ssl.client.auth=required
4) when i verify the SSL using command below
Command :
> openssl s_client -debug -connect <hostname>:9192 -tls1
Error snippet :
>
>
> depth=0 C = US, ST = <State Name>, L = Unknown, O = <org name>., OU =
> <OU>, CN = <localhost>
>
verify error:num=18:self signed certificate
>
> verify return:1
> ....................
>
-----------------
------------------
> 063d - <SPACES/NULS>
> write to 0x860830 [0x8b1100] (37 bytes => -1 (0xFFFFFFFFFFFFFFFF))
> 140487373043616:error:1408E0F4:SSL routines:SSL3_GET_MESSAGE:unexpected
> message:s3_both.c:491:
Any ideas on this ?
What needs to be done to debug or fix this ?
On Wed, Sep 20, 2017 at 12:33 AM, Jakub Scholz <ja...@scholz.cz> wrote:
> Hi,
>
> Looking at your commands it looks as if you generated a self signed key for
> server, self signed key for client and then imported the CA keys public
> keys into the truststores. I don’t think this will work because now you
> have two different self signed keys in the keystores and the presumably the
> CA public key in your truststores.
>
> Does the cert.pem actualy contain the private key? In case it doesn’t
> contain the private key you need to get the private key first. If yes, you
> have to convert it into the keystore format using this guide:
> https://docs.oracle.com/cd/E35976_01/server.740/es_admin/
> src/tadm_ssl_convert_pem_to_jks.html
> You basically have to first use OpenSSL to create PKCS12 key and that can
> be converted into keystore.
>
> BTW: If you run your application with system property “javax.net.debug” set
> to “ssl” it will generate a lot of useful debug information which will help
> to understand what is going on and fix this.
>
> Jakub
>
> On Tue, 19 Sep 2017 at 23:44, karan alang <ka...@gmail.com> wrote:
>
> > Hello All -
> > I was able to set up SSL for the Kafka brokers, using OpenSSL.
> >
> > however, I'm having issues with setting up SSL using the pem file (i.e.
> SSL
> > certificate - certified by CA, provided by the company)
> >
> > Here is what i've done -
> > created the server/client keystore & truststore files and imported the
> > provided cert.pem file
> >
> > keytool -keystore kafka.server.keystore.jks -alias localhost -validity
> 365
> > -genkey
> > keytool -keystore kafka.server.truststore.jks -alias CARoot -import -file
> > cert.pem
> > keytool -keystore kafka.client.truststore.jks -alias CARoot -import -file
> > cert.pem
> > keytool -keystore kafka.server.keystore.jks -alias CARoot -import -file
> > cert.pem
> > keytool -keystore kafka.client.keystore.jks -alias localhost -validity
> 365
> > -genkey
> > keytool -keystore kafka.client.keystore.jks -alias CARoot -import -file
> > cert.pem
> >
> > I've a console producer pushing data in to the topic, and gives error as
> > shown below ->
> >
> >
> > Caused by: javax.net.ssl.SSLProtocolException: Handshake message
> sequence
> > > violation, state = 1, type = 1
> > > at
> > >
> > sun.security.ssl.ServerHandshaker.processMessage(
> ServerHandshaker.java:213)
> > > at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026)
> > > at sun.security.ssl.Handshaker$1.run(Handshaker.java:966)
> > > at sun.security.ssl.Handshaker$1.run(Handshaker.java:963)
> > > at java.security.AccessController.doPrivileged(Native Method)
> > > at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1416)
> > > at
> > >
> > org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(
> SslTransportLayer.java:336)
> > > at
> > >
> > org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(
> SslTransportLayer.java:417)
> > > ... 7 more
> >
> >
> >
> > Any ideas on what the issue might be ?
> > thanks for help in advance!
> >
>
Re: Kafka SSL error
Posted by Jakub Scholz <ja...@scholz.cz>.
Hi,
Looking at your commands it looks as if you generated a self signed key for
server, self signed key for client and then imported the CA keys public
keys into the truststores. I don’t think this will work because now you
have two different self signed keys in the keystores and the presumably the
CA public key in your truststores.
Does the cert.pem actualy contain the private key? In case it doesn’t
contain the private key you need to get the private key first. If yes, you
have to convert it into the keystore format using this guide:
https://docs.oracle.com/cd/E35976_01/server.740/es_admin/src/tadm_ssl_convert_pem_to_jks.html
You basically have to first use OpenSSL to create PKCS12 key and that can
be converted into keystore.
BTW: If you run your application with system property “javax.net.debug” set
to “ssl” it will generate a lot of useful debug information which will help
to understand what is going on and fix this.
Jakub
On Tue, 19 Sep 2017 at 23:44, karan alang <ka...@gmail.com> wrote:
> Hello All -
> I was able to set up SSL for the Kafka brokers, using OpenSSL.
>
> however, I'm having issues with setting up SSL using the pem file (i.e. SSL
> certificate - certified by CA, provided by the company)
>
> Here is what i've done -
> created the server/client keystore & truststore files and imported the
> provided cert.pem file
>
> keytool -keystore kafka.server.keystore.jks -alias localhost -validity 365
> -genkey
> keytool -keystore kafka.server.truststore.jks -alias CARoot -import -file
> cert.pem
> keytool -keystore kafka.client.truststore.jks -alias CARoot -import -file
> cert.pem
> keytool -keystore kafka.server.keystore.jks -alias CARoot -import -file
> cert.pem
> keytool -keystore kafka.client.keystore.jks -alias localhost -validity 365
> -genkey
> keytool -keystore kafka.client.keystore.jks -alias CARoot -import -file
> cert.pem
>
> I've a console producer pushing data in to the topic, and gives error as
> shown below ->
>
>
> Caused by: javax.net.ssl.SSLProtocolException: Handshake message sequence
> > violation, state = 1, type = 1
> > at
> >
> sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:213)
> > at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026)
> > at sun.security.ssl.Handshaker$1.run(Handshaker.java:966)
> > at sun.security.ssl.Handshaker$1.run(Handshaker.java:963)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1416)
> > at
> >
> org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:336)
> > at
> >
> org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:417)
> > ... 7 more
>
>
>
> Any ideas on what the issue might be ?
> thanks for help in advance!
>