You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Mayuresh Gharat <gh...@gmail.com> on 2016/05/12 21:35:58 UTC
Peer not authenticated when trying to send message to kafka over SSL
with Authentication
HI I am trying to establish an SSL connection from kafkaProducer and send
certificate to the Kafka Broker.
I deploy my kafka broker locally running 2 ports :
*listeners = PLAINTEXT://:9092,SSL://:16637 *
*My KafkaBroker SSL configs look like this :*
ssl.protocol = TLS
ssl.trustmanager.algorithm = SunX509
ssl.keymanager.algorithm = SunX509
ssl.keystore.type = VALUE1
ssl.keystore.location = /a/b/c
ssl.keystore.password = xyz
ssl.key.password = xyz
ssl.truststore.type = JKS
ssl.truststore.location = /u/v/w
ssl.truststore.password = 123
I run my producer locally on the same linux box as my KafkaBroker.
My produce command looks like this :
*bin/kafka-producer-perf-test.sh --num-records 10 --topic testToic_1
--record-size 10 --throughput 1 --producer-props *
bootstrap.servers = localhost://:16637
security.protocol = SSL
ssl.protocol = TLS
ssl.trustmanager.algorithm = SunX509
ssl.keymanager.algorithm = SunX509
ssl.keystore.type = VALUE1
ssl.keystore.location = /a/b/c
ssl.keystore.password = xyz
ssl.key.password = xyz
ssl.truststore.type = JKS
ssl.truststore.location = /u/v/w
ssl.truststore.password = 123
On kafka broker, when I do inside buildPrincipal() api of PricipalBuilder
SSLSession session = ((SslTransportLayer)transportLayer).sslSession();
session.getPeerCertificates()
I get:
*org.apache.kafka.common.KafkaException:
javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated*
I ran this command as listed here
http://kafka.apache.org/documentation.html#security_ssl :
*openssl s_client -debug -connect localhost:16637 -tls1*
and was able to see the certificate.
I am not able to understand the peer not authenticated exception here.
Am I missing any SSL config on producer request?
--
-Regards,
Mayuresh R. Gharat
(862) 250-7125
Re: Peer not authenticated when trying to send message to kafka over
SSL with Authentication
Posted by Mayuresh Gharat <gh...@gmail.com>.
I think I was specifying the wrong interBrokerProtocolVersion.
Thanks,
Mayuresh
On Thu, May 12, 2016 at 6:05 PM, Mayuresh Gharat <gharatmayuresh15@gmail.com
> wrote:
> Its pre 0.9 release i think.
> I have trunk and some cherry picks, so can't tell you the exact released
> version as such.
> But if I set those advertised.*="" then those should not be causing
> *"kafka.common.**BrokerEndPointNotAvailableExce*
>
> *ption: End point SSL not found for broker 0".*
> Thanks,
>
> Mayuresh
>
>
>
> On Thu, May 12, 2016 at 5:45 PM, Ismael Juma <is...@juma.me.uk> wrote:
>
>> `Stream` is a lazy collection and only the first element is printed
>> (that's
>> why you see the `?` in the output). What version of Kafka are you using? I
>> know I changed the code in that method not to use `Stream` at some point
>> during the 0.10 development cycle.
>>
>> Ismael
>>
>> On Fri, May 13, 2016 at 1:35 AM, Mayuresh Gharat <
>> gharatmayuresh15@gmail.com
>> > wrote:
>>
>> > All of them are set to "".
>> >
>> > BTW, I printed out the BrokerEndoints on the broker in
>> > KafkaApis.handleTopicMetadataRequest() and surprisingly it prints :
>> > WARN [KafkaApis] [kafka-request-handler-9] [kafka-server] []
>> [KafkaApi-0]
>> > ENDPOINTS: Stream(Map(PLAINTEXT -> EndPoint(hostname,9092,PLAINTEXT)),
>> ?) .
>> >
>> > It does not print the SSL protocol endpoint like
>> > EndPoint(hostname,16637,SSL).
>> >
>> > But my producer is able to talk to the broker on that SSL port and even
>> > send certs across and make requests.
>> >
>> > I checked the kafkaconfig values printed in the log :
>> > listeners = PLAINTEXT://:9092,SSL://:16637
>> >
>> > Thanks,
>> >
>> > Mayuresh
>> >
>> >
>> > On Thu, May 12, 2016 at 4:51 PM, Ismael Juma <is...@juma.me.uk> wrote:
>> >
>> > > Are you using any of the advertised.* configs by any chance? If so,
>> you
>> > > have to use advertised.listeners only, the other ones will only
>> > advertise a
>> > > PLAINTEXT listener.
>> > >
>> > > Ismael
>> > >
>> > > On Fri, May 13, 2016 at 12:24 AM, Mayuresh Gharat <
>> > > gharatmayuresh15@gmail.com> wrote:
>> > >
>> > > > Thanks a lot Ismael :)
>> > > >
>> > > > Was able to get pass it.
>> > > >
>> > > >
>> > > > Now trying to figure out from broker logs :
>> > > >
>> > > >
>> > > > INFO [KafkaApis] [kafka-request-handler-0] [kafka-server] []
>> > [KafkaApi-0]
>> > > > Auto creation of topic testToic_1 with 8 partitions and replication
>> > > factor
>> > > > 1 is successful!
>> > > > [KafkaApis] [kafka-request-handler-0] [kafka-server] [] [KafkaApi-0]
>> > > Error
>> > > > when handling request Name: TopicMetadataRequest; Version: 0;
>> > > > CorrelationId: 0; ClientId: producer-1; Topics: testToic_1
>> > > > kafka.common.BrokerEndPointNotAvailableException: End point SSL not
>> > found
>> > > > for broker 0
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > Thanks,
>> > > >
>> > > > Mayuresh
>> > > >
>> > > > On Thu, May 12, 2016 at 2:40 PM, Ismael Juma <is...@juma.me.uk>
>> > wrote:
>> > > >
>> > > > > Hi Mayuresh,
>> > > > >
>> > > > > You need to enable client authentication by setting
>> `ssl.client.auth`
>> > > to
>> > > > > `required` or `requested` (I suggest the former).
>> > > > >
>> > > > > Ismael
>> > > > >
>> > > > > On Thu, May 12, 2016 at 10:35 PM, Mayuresh Gharat <
>> > > > > gharatmayuresh15@gmail.com> wrote:
>> > > > >
>> > > > > > HI I am trying to establish an SSL connection from kafkaProducer
>> > and
>> > > > send
>> > > > > > certificate to the Kafka Broker.
>> > > > > >
>> > > > > >
>> > > > > > I deploy my kafka broker locally running 2 ports :
>> > > > > > *listeners = PLAINTEXT://:9092,SSL://:16637 *
>> > > > > >
>> > > > > > *My KafkaBroker SSL configs look like this :*
>> > > > > >
>> > > > > > ssl.protocol = TLS
>> > > > > > ssl.trustmanager.algorithm = SunX509
>> > > > > > ssl.keymanager.algorithm = SunX509
>> > > > > > ssl.keystore.type = VALUE1
>> > > > > > ssl.keystore.location = /a/b/c
>> > > > > > ssl.keystore.password = xyz
>> > > > > > ssl.key.password = xyz
>> > > > > > ssl.truststore.type = JKS
>> > > > > > ssl.truststore.location = /u/v/w
>> > > > > > ssl.truststore.password = 123
>> > > > > >
>> > > > > > I run my producer locally on the same linux box as my
>> KafkaBroker.
>> > > > > > My produce command looks like this :
>> > > > > >
>> > > > > > *bin/kafka-producer-perf-test.sh --num-records 10 --topic
>> > testToic_1
>> > > > > > --record-size 10 --throughput 1 --producer-props *
>> > > > > > bootstrap.servers = localhost://:16637
>> > > > > > security.protocol = SSL
>> > > > > > ssl.protocol = TLS
>> > > > > > ssl.trustmanager.algorithm = SunX509
>> > > > > > ssl.keymanager.algorithm = SunX509
>> > > > > > ssl.keystore.type = VALUE1
>> > > > > > ssl.keystore.location = /a/b/c
>> > > > > > ssl.keystore.password = xyz
>> > > > > > ssl.key.password = xyz
>> > > > > > ssl.truststore.type = JKS
>> > > > > > ssl.truststore.location = /u/v/w
>> > > > > > ssl.truststore.password = 123
>> > > > > >
>> > > > > >
>> > > > > > On kafka broker, when I do inside buildPrincipal() api of
>> > > > PricipalBuilder
>> > > > > >
>> > > > > > SSLSession session =
>> > > ((SslTransportLayer)transportLayer).sslSession();
>> > > > > > session.getPeerCertificates()
>> > > > > >
>> > > > > > I get:
>> > > > > > *org.apache.kafka.common.KafkaException:
>> > > > > > javax.net.ssl.SSLPeerUnverifiedException: peer not
>> authenticated*
>> > > > > >
>> > > > > >
>> > > > > > I ran this command as listed here
>> > > > > > http://kafka.apache.org/documentation.html#security_ssl :
>> > > > > >
>> > > > > > *openssl s_client -debug -connect localhost:16637 -tls1*
>> > > > > >
>> > > > > > and was able to see the certificate.
>> > > > > >
>> > > > > > I am not able to understand the peer not authenticated exception
>> > > here.
>> > > > > > Am I missing any SSL config on producer request?
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > --
>> > > > > > -Regards,
>> > > > > > Mayuresh R. Gharat
>> > > > > > (862) 250-7125
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > -Regards,
>> > > > Mayuresh R. Gharat
>> > > > (862) 250-7125
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > -Regards,
>> > Mayuresh R. Gharat
>> > (862) 250-7125
>> >
>>
>
>
>
> --
> -Regards,
> Mayuresh R. Gharat
> (862) 250-7125
>
--
-Regards,
Mayuresh R. Gharat
(862) 250-7125
Re: Peer not authenticated when trying to send message to kafka over
SSL with Authentication
Posted by Mayuresh Gharat <gh...@gmail.com>.
Its pre 0.9 release i think.
I have trunk and some cherry picks, so can't tell you the exact released
version as such.
But if I set those advertised.*="" then those should not be causing
*"kafka.common.**BrokerEndPointNotAvailableExce*
*ption: End point SSL not found for broker 0".*
Thanks,
Mayuresh
On Thu, May 12, 2016 at 5:45 PM, Ismael Juma <is...@juma.me.uk> wrote:
> `Stream` is a lazy collection and only the first element is printed (that's
> why you see the `?` in the output). What version of Kafka are you using? I
> know I changed the code in that method not to use `Stream` at some point
> during the 0.10 development cycle.
>
> Ismael
>
> On Fri, May 13, 2016 at 1:35 AM, Mayuresh Gharat <
> gharatmayuresh15@gmail.com
> > wrote:
>
> > All of them are set to "".
> >
> > BTW, I printed out the BrokerEndoints on the broker in
> > KafkaApis.handleTopicMetadataRequest() and surprisingly it prints :
> > WARN [KafkaApis] [kafka-request-handler-9] [kafka-server] []
> [KafkaApi-0]
> > ENDPOINTS: Stream(Map(PLAINTEXT -> EndPoint(hostname,9092,PLAINTEXT)),
> ?) .
> >
> > It does not print the SSL protocol endpoint like
> > EndPoint(hostname,16637,SSL).
> >
> > But my producer is able to talk to the broker on that SSL port and even
> > send certs across and make requests.
> >
> > I checked the kafkaconfig values printed in the log :
> > listeners = PLAINTEXT://:9092,SSL://:16637
> >
> > Thanks,
> >
> > Mayuresh
> >
> >
> > On Thu, May 12, 2016 at 4:51 PM, Ismael Juma <is...@juma.me.uk> wrote:
> >
> > > Are you using any of the advertised.* configs by any chance? If so, you
> > > have to use advertised.listeners only, the other ones will only
> > advertise a
> > > PLAINTEXT listener.
> > >
> > > Ismael
> > >
> > > On Fri, May 13, 2016 at 12:24 AM, Mayuresh Gharat <
> > > gharatmayuresh15@gmail.com> wrote:
> > >
> > > > Thanks a lot Ismael :)
> > > >
> > > > Was able to get pass it.
> > > >
> > > >
> > > > Now trying to figure out from broker logs :
> > > >
> > > >
> > > > INFO [KafkaApis] [kafka-request-handler-0] [kafka-server] []
> > [KafkaApi-0]
> > > > Auto creation of topic testToic_1 with 8 partitions and replication
> > > factor
> > > > 1 is successful!
> > > > [KafkaApis] [kafka-request-handler-0] [kafka-server] [] [KafkaApi-0]
> > > Error
> > > > when handling request Name: TopicMetadataRequest; Version: 0;
> > > > CorrelationId: 0; ClientId: producer-1; Topics: testToic_1
> > > > kafka.common.BrokerEndPointNotAvailableException: End point SSL not
> > found
> > > > for broker 0
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Thanks,
> > > >
> > > > Mayuresh
> > > >
> > > > On Thu, May 12, 2016 at 2:40 PM, Ismael Juma <is...@juma.me.uk>
> > wrote:
> > > >
> > > > > Hi Mayuresh,
> > > > >
> > > > > You need to enable client authentication by setting
> `ssl.client.auth`
> > > to
> > > > > `required` or `requested` (I suggest the former).
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Thu, May 12, 2016 at 10:35 PM, Mayuresh Gharat <
> > > > > gharatmayuresh15@gmail.com> wrote:
> > > > >
> > > > > > HI I am trying to establish an SSL connection from kafkaProducer
> > and
> > > > send
> > > > > > certificate to the Kafka Broker.
> > > > > >
> > > > > >
> > > > > > I deploy my kafka broker locally running 2 ports :
> > > > > > *listeners = PLAINTEXT://:9092,SSL://:16637 *
> > > > > >
> > > > > > *My KafkaBroker SSL configs look like this :*
> > > > > >
> > > > > > ssl.protocol = TLS
> > > > > > ssl.trustmanager.algorithm = SunX509
> > > > > > ssl.keymanager.algorithm = SunX509
> > > > > > ssl.keystore.type = VALUE1
> > > > > > ssl.keystore.location = /a/b/c
> > > > > > ssl.keystore.password = xyz
> > > > > > ssl.key.password = xyz
> > > > > > ssl.truststore.type = JKS
> > > > > > ssl.truststore.location = /u/v/w
> > > > > > ssl.truststore.password = 123
> > > > > >
> > > > > > I run my producer locally on the same linux box as my
> KafkaBroker.
> > > > > > My produce command looks like this :
> > > > > >
> > > > > > *bin/kafka-producer-perf-test.sh --num-records 10 --topic
> > testToic_1
> > > > > > --record-size 10 --throughput 1 --producer-props *
> > > > > > bootstrap.servers = localhost://:16637
> > > > > > security.protocol = SSL
> > > > > > ssl.protocol = TLS
> > > > > > ssl.trustmanager.algorithm = SunX509
> > > > > > ssl.keymanager.algorithm = SunX509
> > > > > > ssl.keystore.type = VALUE1
> > > > > > ssl.keystore.location = /a/b/c
> > > > > > ssl.keystore.password = xyz
> > > > > > ssl.key.password = xyz
> > > > > > ssl.truststore.type = JKS
> > > > > > ssl.truststore.location = /u/v/w
> > > > > > ssl.truststore.password = 123
> > > > > >
> > > > > >
> > > > > > On kafka broker, when I do inside buildPrincipal() api of
> > > > PricipalBuilder
> > > > > >
> > > > > > SSLSession session =
> > > ((SslTransportLayer)transportLayer).sslSession();
> > > > > > session.getPeerCertificates()
> > > > > >
> > > > > > I get:
> > > > > > *org.apache.kafka.common.KafkaException:
> > > > > > javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated*
> > > > > >
> > > > > >
> > > > > > I ran this command as listed here
> > > > > > http://kafka.apache.org/documentation.html#security_ssl :
> > > > > >
> > > > > > *openssl s_client -debug -connect localhost:16637 -tls1*
> > > > > >
> > > > > > and was able to see the certificate.
> > > > > >
> > > > > > I am not able to understand the peer not authenticated exception
> > > here.
> > > > > > Am I missing any SSL config on producer request?
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > -Regards,
> > > > > > Mayuresh R. Gharat
> > > > > > (862) 250-7125
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -Regards,
> > > > Mayuresh R. Gharat
> > > > (862) 250-7125
> > > >
> > >
> >
> >
> >
> > --
> > -Regards,
> > Mayuresh R. Gharat
> > (862) 250-7125
> >
>
--
-Regards,
Mayuresh R. Gharat
(862) 250-7125
Re: Peer not authenticated when trying to send message to kafka over
SSL with Authentication
Posted by Ismael Juma <is...@juma.me.uk>.
`Stream` is a lazy collection and only the first element is printed (that's
why you see the `?` in the output). What version of Kafka are you using? I
know I changed the code in that method not to use `Stream` at some point
during the 0.10 development cycle.
Ismael
On Fri, May 13, 2016 at 1:35 AM, Mayuresh Gharat <gharatmayuresh15@gmail.com
> wrote:
> All of them are set to "".
>
> BTW, I printed out the BrokerEndoints on the broker in
> KafkaApis.handleTopicMetadataRequest() and surprisingly it prints :
> WARN [KafkaApis] [kafka-request-handler-9] [kafka-server] [] [KafkaApi-0]
> ENDPOINTS: Stream(Map(PLAINTEXT -> EndPoint(hostname,9092,PLAINTEXT)), ?) .
>
> It does not print the SSL protocol endpoint like
> EndPoint(hostname,16637,SSL).
>
> But my producer is able to talk to the broker on that SSL port and even
> send certs across and make requests.
>
> I checked the kafkaconfig values printed in the log :
> listeners = PLAINTEXT://:9092,SSL://:16637
>
> Thanks,
>
> Mayuresh
>
>
> On Thu, May 12, 2016 at 4:51 PM, Ismael Juma <is...@juma.me.uk> wrote:
>
> > Are you using any of the advertised.* configs by any chance? If so, you
> > have to use advertised.listeners only, the other ones will only
> advertise a
> > PLAINTEXT listener.
> >
> > Ismael
> >
> > On Fri, May 13, 2016 at 12:24 AM, Mayuresh Gharat <
> > gharatmayuresh15@gmail.com> wrote:
> >
> > > Thanks a lot Ismael :)
> > >
> > > Was able to get pass it.
> > >
> > >
> > > Now trying to figure out from broker logs :
> > >
> > >
> > > INFO [KafkaApis] [kafka-request-handler-0] [kafka-server] []
> [KafkaApi-0]
> > > Auto creation of topic testToic_1 with 8 partitions and replication
> > factor
> > > 1 is successful!
> > > [KafkaApis] [kafka-request-handler-0] [kafka-server] [] [KafkaApi-0]
> > Error
> > > when handling request Name: TopicMetadataRequest; Version: 0;
> > > CorrelationId: 0; ClientId: producer-1; Topics: testToic_1
> > > kafka.common.BrokerEndPointNotAvailableException: End point SSL not
> found
> > > for broker 0
> > >
> > >
> > >
> > >
> > >
> > > Thanks,
> > >
> > > Mayuresh
> > >
> > > On Thu, May 12, 2016 at 2:40 PM, Ismael Juma <is...@juma.me.uk>
> wrote:
> > >
> > > > Hi Mayuresh,
> > > >
> > > > You need to enable client authentication by setting `ssl.client.auth`
> > to
> > > > `required` or `requested` (I suggest the former).
> > > >
> > > > Ismael
> > > >
> > > > On Thu, May 12, 2016 at 10:35 PM, Mayuresh Gharat <
> > > > gharatmayuresh15@gmail.com> wrote:
> > > >
> > > > > HI I am trying to establish an SSL connection from kafkaProducer
> and
> > > send
> > > > > certificate to the Kafka Broker.
> > > > >
> > > > >
> > > > > I deploy my kafka broker locally running 2 ports :
> > > > > *listeners = PLAINTEXT://:9092,SSL://:16637 *
> > > > >
> > > > > *My KafkaBroker SSL configs look like this :*
> > > > >
> > > > > ssl.protocol = TLS
> > > > > ssl.trustmanager.algorithm = SunX509
> > > > > ssl.keymanager.algorithm = SunX509
> > > > > ssl.keystore.type = VALUE1
> > > > > ssl.keystore.location = /a/b/c
> > > > > ssl.keystore.password = xyz
> > > > > ssl.key.password = xyz
> > > > > ssl.truststore.type = JKS
> > > > > ssl.truststore.location = /u/v/w
> > > > > ssl.truststore.password = 123
> > > > >
> > > > > I run my producer locally on the same linux box as my KafkaBroker.
> > > > > My produce command looks like this :
> > > > >
> > > > > *bin/kafka-producer-perf-test.sh --num-records 10 --topic
> testToic_1
> > > > > --record-size 10 --throughput 1 --producer-props *
> > > > > bootstrap.servers = localhost://:16637
> > > > > security.protocol = SSL
> > > > > ssl.protocol = TLS
> > > > > ssl.trustmanager.algorithm = SunX509
> > > > > ssl.keymanager.algorithm = SunX509
> > > > > ssl.keystore.type = VALUE1
> > > > > ssl.keystore.location = /a/b/c
> > > > > ssl.keystore.password = xyz
> > > > > ssl.key.password = xyz
> > > > > ssl.truststore.type = JKS
> > > > > ssl.truststore.location = /u/v/w
> > > > > ssl.truststore.password = 123
> > > > >
> > > > >
> > > > > On kafka broker, when I do inside buildPrincipal() api of
> > > PricipalBuilder
> > > > >
> > > > > SSLSession session =
> > ((SslTransportLayer)transportLayer).sslSession();
> > > > > session.getPeerCertificates()
> > > > >
> > > > > I get:
> > > > > *org.apache.kafka.common.KafkaException:
> > > > > javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated*
> > > > >
> > > > >
> > > > > I ran this command as listed here
> > > > > http://kafka.apache.org/documentation.html#security_ssl :
> > > > >
> > > > > *openssl s_client -debug -connect localhost:16637 -tls1*
> > > > >
> > > > > and was able to see the certificate.
> > > > >
> > > > > I am not able to understand the peer not authenticated exception
> > here.
> > > > > Am I missing any SSL config on producer request?
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -Regards,
> > > > > Mayuresh R. Gharat
> > > > > (862) 250-7125
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > -Regards,
> > > Mayuresh R. Gharat
> > > (862) 250-7125
> > >
> >
>
>
>
> --
> -Regards,
> Mayuresh R. Gharat
> (862) 250-7125
>
Re: Peer not authenticated when trying to send message to kafka over
SSL with Authentication
Posted by Mayuresh Gharat <gh...@gmail.com>.
All of them are set to "".
BTW, I printed out the BrokerEndoints on the broker in
KafkaApis.handleTopicMetadataRequest() and surprisingly it prints :
WARN [KafkaApis] [kafka-request-handler-9] [kafka-server] [] [KafkaApi-0]
ENDPOINTS: Stream(Map(PLAINTEXT -> EndPoint(hostname,9092,PLAINTEXT)), ?) .
It does not print the SSL protocol endpoint like
EndPoint(hostname,16637,SSL).
But my producer is able to talk to the broker on that SSL port and even
send certs across and make requests.
I checked the kafkaconfig values printed in the log :
listeners = PLAINTEXT://:9092,SSL://:16637
Thanks,
Mayuresh
On Thu, May 12, 2016 at 4:51 PM, Ismael Juma <is...@juma.me.uk> wrote:
> Are you using any of the advertised.* configs by any chance? If so, you
> have to use advertised.listeners only, the other ones will only advertise a
> PLAINTEXT listener.
>
> Ismael
>
> On Fri, May 13, 2016 at 12:24 AM, Mayuresh Gharat <
> gharatmayuresh15@gmail.com> wrote:
>
> > Thanks a lot Ismael :)
> >
> > Was able to get pass it.
> >
> >
> > Now trying to figure out from broker logs :
> >
> >
> > INFO [KafkaApis] [kafka-request-handler-0] [kafka-server] [] [KafkaApi-0]
> > Auto creation of topic testToic_1 with 8 partitions and replication
> factor
> > 1 is successful!
> > [KafkaApis] [kafka-request-handler-0] [kafka-server] [] [KafkaApi-0]
> Error
> > when handling request Name: TopicMetadataRequest; Version: 0;
> > CorrelationId: 0; ClientId: producer-1; Topics: testToic_1
> > kafka.common.BrokerEndPointNotAvailableException: End point SSL not found
> > for broker 0
> >
> >
> >
> >
> >
> > Thanks,
> >
> > Mayuresh
> >
> > On Thu, May 12, 2016 at 2:40 PM, Ismael Juma <is...@juma.me.uk> wrote:
> >
> > > Hi Mayuresh,
> > >
> > > You need to enable client authentication by setting `ssl.client.auth`
> to
> > > `required` or `requested` (I suggest the former).
> > >
> > > Ismael
> > >
> > > On Thu, May 12, 2016 at 10:35 PM, Mayuresh Gharat <
> > > gharatmayuresh15@gmail.com> wrote:
> > >
> > > > HI I am trying to establish an SSL connection from kafkaProducer and
> > send
> > > > certificate to the Kafka Broker.
> > > >
> > > >
> > > > I deploy my kafka broker locally running 2 ports :
> > > > *listeners = PLAINTEXT://:9092,SSL://:16637 *
> > > >
> > > > *My KafkaBroker SSL configs look like this :*
> > > >
> > > > ssl.protocol = TLS
> > > > ssl.trustmanager.algorithm = SunX509
> > > > ssl.keymanager.algorithm = SunX509
> > > > ssl.keystore.type = VALUE1
> > > > ssl.keystore.location = /a/b/c
> > > > ssl.keystore.password = xyz
> > > > ssl.key.password = xyz
> > > > ssl.truststore.type = JKS
> > > > ssl.truststore.location = /u/v/w
> > > > ssl.truststore.password = 123
> > > >
> > > > I run my producer locally on the same linux box as my KafkaBroker.
> > > > My produce command looks like this :
> > > >
> > > > *bin/kafka-producer-perf-test.sh --num-records 10 --topic testToic_1
> > > > --record-size 10 --throughput 1 --producer-props *
> > > > bootstrap.servers = localhost://:16637
> > > > security.protocol = SSL
> > > > ssl.protocol = TLS
> > > > ssl.trustmanager.algorithm = SunX509
> > > > ssl.keymanager.algorithm = SunX509
> > > > ssl.keystore.type = VALUE1
> > > > ssl.keystore.location = /a/b/c
> > > > ssl.keystore.password = xyz
> > > > ssl.key.password = xyz
> > > > ssl.truststore.type = JKS
> > > > ssl.truststore.location = /u/v/w
> > > > ssl.truststore.password = 123
> > > >
> > > >
> > > > On kafka broker, when I do inside buildPrincipal() api of
> > PricipalBuilder
> > > >
> > > > SSLSession session =
> ((SslTransportLayer)transportLayer).sslSession();
> > > > session.getPeerCertificates()
> > > >
> > > > I get:
> > > > *org.apache.kafka.common.KafkaException:
> > > > javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated*
> > > >
> > > >
> > > > I ran this command as listed here
> > > > http://kafka.apache.org/documentation.html#security_ssl :
> > > >
> > > > *openssl s_client -debug -connect localhost:16637 -tls1*
> > > >
> > > > and was able to see the certificate.
> > > >
> > > > I am not able to understand the peer not authenticated exception
> here.
> > > > Am I missing any SSL config on producer request?
> > > >
> > > >
> > > >
> > > > --
> > > > -Regards,
> > > > Mayuresh R. Gharat
> > > > (862) 250-7125
> > > >
> > >
> >
> >
> >
> > --
> > -Regards,
> > Mayuresh R. Gharat
> > (862) 250-7125
> >
>
--
-Regards,
Mayuresh R. Gharat
(862) 250-7125
Re: Peer not authenticated when trying to send message to kafka over
SSL with Authentication
Posted by Ismael Juma <is...@juma.me.uk>.
Are you using any of the advertised.* configs by any chance? If so, you
have to use advertised.listeners only, the other ones will only advertise a
PLAINTEXT listener.
Ismael
On Fri, May 13, 2016 at 12:24 AM, Mayuresh Gharat <
gharatmayuresh15@gmail.com> wrote:
> Thanks a lot Ismael :)
>
> Was able to get pass it.
>
>
> Now trying to figure out from broker logs :
>
>
> INFO [KafkaApis] [kafka-request-handler-0] [kafka-server] [] [KafkaApi-0]
> Auto creation of topic testToic_1 with 8 partitions and replication factor
> 1 is successful!
> [KafkaApis] [kafka-request-handler-0] [kafka-server] [] [KafkaApi-0] Error
> when handling request Name: TopicMetadataRequest; Version: 0;
> CorrelationId: 0; ClientId: producer-1; Topics: testToic_1
> kafka.common.BrokerEndPointNotAvailableException: End point SSL not found
> for broker 0
>
>
>
>
>
> Thanks,
>
> Mayuresh
>
> On Thu, May 12, 2016 at 2:40 PM, Ismael Juma <is...@juma.me.uk> wrote:
>
> > Hi Mayuresh,
> >
> > You need to enable client authentication by setting `ssl.client.auth` to
> > `required` or `requested` (I suggest the former).
> >
> > Ismael
> >
> > On Thu, May 12, 2016 at 10:35 PM, Mayuresh Gharat <
> > gharatmayuresh15@gmail.com> wrote:
> >
> > > HI I am trying to establish an SSL connection from kafkaProducer and
> send
> > > certificate to the Kafka Broker.
> > >
> > >
> > > I deploy my kafka broker locally running 2 ports :
> > > *listeners = PLAINTEXT://:9092,SSL://:16637 *
> > >
> > > *My KafkaBroker SSL configs look like this :*
> > >
> > > ssl.protocol = TLS
> > > ssl.trustmanager.algorithm = SunX509
> > > ssl.keymanager.algorithm = SunX509
> > > ssl.keystore.type = VALUE1
> > > ssl.keystore.location = /a/b/c
> > > ssl.keystore.password = xyz
> > > ssl.key.password = xyz
> > > ssl.truststore.type = JKS
> > > ssl.truststore.location = /u/v/w
> > > ssl.truststore.password = 123
> > >
> > > I run my producer locally on the same linux box as my KafkaBroker.
> > > My produce command looks like this :
> > >
> > > *bin/kafka-producer-perf-test.sh --num-records 10 --topic testToic_1
> > > --record-size 10 --throughput 1 --producer-props *
> > > bootstrap.servers = localhost://:16637
> > > security.protocol = SSL
> > > ssl.protocol = TLS
> > > ssl.trustmanager.algorithm = SunX509
> > > ssl.keymanager.algorithm = SunX509
> > > ssl.keystore.type = VALUE1
> > > ssl.keystore.location = /a/b/c
> > > ssl.keystore.password = xyz
> > > ssl.key.password = xyz
> > > ssl.truststore.type = JKS
> > > ssl.truststore.location = /u/v/w
> > > ssl.truststore.password = 123
> > >
> > >
> > > On kafka broker, when I do inside buildPrincipal() api of
> PricipalBuilder
> > >
> > > SSLSession session = ((SslTransportLayer)transportLayer).sslSession();
> > > session.getPeerCertificates()
> > >
> > > I get:
> > > *org.apache.kafka.common.KafkaException:
> > > javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated*
> > >
> > >
> > > I ran this command as listed here
> > > http://kafka.apache.org/documentation.html#security_ssl :
> > >
> > > *openssl s_client -debug -connect localhost:16637 -tls1*
> > >
> > > and was able to see the certificate.
> > >
> > > I am not able to understand the peer not authenticated exception here.
> > > Am I missing any SSL config on producer request?
> > >
> > >
> > >
> > > --
> > > -Regards,
> > > Mayuresh R. Gharat
> > > (862) 250-7125
> > >
> >
>
>
>
> --
> -Regards,
> Mayuresh R. Gharat
> (862) 250-7125
>
Re: Peer not authenticated when trying to send message to kafka over
SSL with Authentication
Posted by Mayuresh Gharat <gh...@gmail.com>.
Thanks a lot Ismael :)
Was able to get pass it.
Now trying to figure out from broker logs :
INFO [KafkaApis] [kafka-request-handler-0] [kafka-server] [] [KafkaApi-0]
Auto creation of topic testToic_1 with 8 partitions and replication factor
1 is successful!
[KafkaApis] [kafka-request-handler-0] [kafka-server] [] [KafkaApi-0] Error
when handling request Name: TopicMetadataRequest; Version: 0;
CorrelationId: 0; ClientId: producer-1; Topics: testToic_1
kafka.common.BrokerEndPointNotAvailableException: End point SSL not found
for broker 0
Thanks,
Mayuresh
On Thu, May 12, 2016 at 2:40 PM, Ismael Juma <is...@juma.me.uk> wrote:
> Hi Mayuresh,
>
> You need to enable client authentication by setting `ssl.client.auth` to
> `required` or `requested` (I suggest the former).
>
> Ismael
>
> On Thu, May 12, 2016 at 10:35 PM, Mayuresh Gharat <
> gharatmayuresh15@gmail.com> wrote:
>
> > HI I am trying to establish an SSL connection from kafkaProducer and send
> > certificate to the Kafka Broker.
> >
> >
> > I deploy my kafka broker locally running 2 ports :
> > *listeners = PLAINTEXT://:9092,SSL://:16637 *
> >
> > *My KafkaBroker SSL configs look like this :*
> >
> > ssl.protocol = TLS
> > ssl.trustmanager.algorithm = SunX509
> > ssl.keymanager.algorithm = SunX509
> > ssl.keystore.type = VALUE1
> > ssl.keystore.location = /a/b/c
> > ssl.keystore.password = xyz
> > ssl.key.password = xyz
> > ssl.truststore.type = JKS
> > ssl.truststore.location = /u/v/w
> > ssl.truststore.password = 123
> >
> > I run my producer locally on the same linux box as my KafkaBroker.
> > My produce command looks like this :
> >
> > *bin/kafka-producer-perf-test.sh --num-records 10 --topic testToic_1
> > --record-size 10 --throughput 1 --producer-props *
> > bootstrap.servers = localhost://:16637
> > security.protocol = SSL
> > ssl.protocol = TLS
> > ssl.trustmanager.algorithm = SunX509
> > ssl.keymanager.algorithm = SunX509
> > ssl.keystore.type = VALUE1
> > ssl.keystore.location = /a/b/c
> > ssl.keystore.password = xyz
> > ssl.key.password = xyz
> > ssl.truststore.type = JKS
> > ssl.truststore.location = /u/v/w
> > ssl.truststore.password = 123
> >
> >
> > On kafka broker, when I do inside buildPrincipal() api of PricipalBuilder
> >
> > SSLSession session = ((SslTransportLayer)transportLayer).sslSession();
> > session.getPeerCertificates()
> >
> > I get:
> > *org.apache.kafka.common.KafkaException:
> > javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated*
> >
> >
> > I ran this command as listed here
> > http://kafka.apache.org/documentation.html#security_ssl :
> >
> > *openssl s_client -debug -connect localhost:16637 -tls1*
> >
> > and was able to see the certificate.
> >
> > I am not able to understand the peer not authenticated exception here.
> > Am I missing any SSL config on producer request?
> >
> >
> >
> > --
> > -Regards,
> > Mayuresh R. Gharat
> > (862) 250-7125
> >
>
--
-Regards,
Mayuresh R. Gharat
(862) 250-7125
Re: Peer not authenticated when trying to send message to kafka over
SSL with Authentication
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Mayuresh,
You need to enable client authentication by setting `ssl.client.auth` to
`required` or `requested` (I suggest the former).
Ismael
On Thu, May 12, 2016 at 10:35 PM, Mayuresh Gharat <
gharatmayuresh15@gmail.com> wrote:
> HI I am trying to establish an SSL connection from kafkaProducer and send
> certificate to the Kafka Broker.
>
>
> I deploy my kafka broker locally running 2 ports :
> *listeners = PLAINTEXT://:9092,SSL://:16637 *
>
> *My KafkaBroker SSL configs look like this :*
>
> ssl.protocol = TLS
> ssl.trustmanager.algorithm = SunX509
> ssl.keymanager.algorithm = SunX509
> ssl.keystore.type = VALUE1
> ssl.keystore.location = /a/b/c
> ssl.keystore.password = xyz
> ssl.key.password = xyz
> ssl.truststore.type = JKS
> ssl.truststore.location = /u/v/w
> ssl.truststore.password = 123
>
> I run my producer locally on the same linux box as my KafkaBroker.
> My produce command looks like this :
>
> *bin/kafka-producer-perf-test.sh --num-records 10 --topic testToic_1
> --record-size 10 --throughput 1 --producer-props *
> bootstrap.servers = localhost://:16637
> security.protocol = SSL
> ssl.protocol = TLS
> ssl.trustmanager.algorithm = SunX509
> ssl.keymanager.algorithm = SunX509
> ssl.keystore.type = VALUE1
> ssl.keystore.location = /a/b/c
> ssl.keystore.password = xyz
> ssl.key.password = xyz
> ssl.truststore.type = JKS
> ssl.truststore.location = /u/v/w
> ssl.truststore.password = 123
>
>
> On kafka broker, when I do inside buildPrincipal() api of PricipalBuilder
>
> SSLSession session = ((SslTransportLayer)transportLayer).sslSession();
> session.getPeerCertificates()
>
> I get:
> *org.apache.kafka.common.KafkaException:
> javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated*
>
>
> I ran this command as listed here
> http://kafka.apache.org/documentation.html#security_ssl :
>
> *openssl s_client -debug -connect localhost:16637 -tls1*
>
> and was able to see the certificate.
>
> I am not able to understand the peer not authenticated exception here.
> Am I missing any SSL config on producer request?
>
>
>
> --
> -Regards,
> Mayuresh R. Gharat
> (862) 250-7125
>
Re: Peer not authenticated when trying to send message to kafka over
SSL with Authentication
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Mayuresh,
You need to enable client authentication by setting `ssl.client.auth` to
`required` or `requested` (I suggest the former).
Ismael
On Thu, May 12, 2016 at 10:35 PM, Mayuresh Gharat <
gharatmayuresh15@gmail.com> wrote:
> HI I am trying to establish an SSL connection from kafkaProducer and send
> certificate to the Kafka Broker.
>
>
> I deploy my kafka broker locally running 2 ports :
> *listeners = PLAINTEXT://:9092,SSL://:16637 *
>
> *My KafkaBroker SSL configs look like this :*
>
> ssl.protocol = TLS
> ssl.trustmanager.algorithm = SunX509
> ssl.keymanager.algorithm = SunX509
> ssl.keystore.type = VALUE1
> ssl.keystore.location = /a/b/c
> ssl.keystore.password = xyz
> ssl.key.password = xyz
> ssl.truststore.type = JKS
> ssl.truststore.location = /u/v/w
> ssl.truststore.password = 123
>
> I run my producer locally on the same linux box as my KafkaBroker.
> My produce command looks like this :
>
> *bin/kafka-producer-perf-test.sh --num-records 10 --topic testToic_1
> --record-size 10 --throughput 1 --producer-props *
> bootstrap.servers = localhost://:16637
> security.protocol = SSL
> ssl.protocol = TLS
> ssl.trustmanager.algorithm = SunX509
> ssl.keymanager.algorithm = SunX509
> ssl.keystore.type = VALUE1
> ssl.keystore.location = /a/b/c
> ssl.keystore.password = xyz
> ssl.key.password = xyz
> ssl.truststore.type = JKS
> ssl.truststore.location = /u/v/w
> ssl.truststore.password = 123
>
>
> On kafka broker, when I do inside buildPrincipal() api of PricipalBuilder
>
> SSLSession session = ((SslTransportLayer)transportLayer).sslSession();
> session.getPeerCertificates()
>
> I get:
> *org.apache.kafka.common.KafkaException:
> javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated*
>
>
> I ran this command as listed here
> http://kafka.apache.org/documentation.html#security_ssl :
>
> *openssl s_client -debug -connect localhost:16637 -tls1*
>
> and was able to see the certificate.
>
> I am not able to understand the peer not authenticated exception here.
> Am I missing any SSL config on producer request?
>
>
>
> --
> -Regards,
> Mayuresh R. Gharat
> (862) 250-7125
>