You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Raghav <ra...@gmail.com> on 2017/05/23 15:44:33 UTC
Kafka Authorization and ACLs Broken
Hello Kafka Users
I am a new Kafka user and trying to make Kafka SSL work with Authorization
and ACLs. I followed posts from Kafka and Confluent docs exactly to the
point but my producer cannot write to kafka broker. I get
"LEADER_NOT_FOUND" errors. And even Consumer throws the same errors.
Can someone please share their config which worked with ACLs.
Here is my config. Please help.
server.properties config
------------------------------------------------------------
------------------------------------------------
broker.id=0
auto.create.topics.enable=true
delete.topic.enable=true
listeners=PLAINTEXT://kafka1.example.com:9092
<http://kafka-dev1.example.com:9092/>,SSL://kafka1.example.com:9093
<http://kafka-dev1.example.com:9093/>
host.name=kafka1.example.com <http://kafka-dev1.example.com/>
ssl.keystore.location=/var/private/kafka1.keystore.jks
ssl.keystore.password=12345678
ssl.key.password=12345678
ssl.truststore.location=/var/private/kafka1.truststore.jks
ssl.truststore.password=12345678
ssl.client.auth=required
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
ssl.keystore.type=JKS
ssl.truststore.type=JKS
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
------------------------------------------------------------
------------------------------------------------
Here is producer Config(producer.properties)
------------------------------------------------------------
------------------------------------------------
security.protocol=SSL
ssl.truststore.location=/var/private/kafka2.truststore.jks
ssl.truststore.password=12345678
ssl.keystore.location=/var/private/kafka2.keystore.jks
ssl.keystore.password=12345678
ssl.key.password=12345678
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
ssl.truststore.type=JKS
ssl.keystore.type=JKS
------------------------------------------------------------
------------------------------------------------
Raqhav
Re: Kafka Authorization and ACLs Broken
Posted by Raghav <ra...@gmail.com>.
Rajini
I will try and report to you shortly. Many thanks.
Raghav
On Wed, May 24, 2017 at 7:04 AM, Rajini Sivaram <ra...@gmail.com>
wrote:
> Raghav/Darshan,
>
> Can you try these steps on a clean installation of Kafka? It works for me,
> so hopefully it will work for you. And then you can adapt to your scenario.
>
> *Create keystores and truststores:*
>
> keytool -genkey -alias kafka -keystore server.keystore.jks -dname
> "CN=KafkaBroker,O=Pivotal,C=UK" -storepass server-keystore-password
> -keypass server-key-password
>
> keytool -exportcert -file server-cert-file -keystore server.keystore.jks
> -alias kafka -storepass server-keystore-password
>
> keytool -importcert -file server-cert-file -keystore server.truststore.jks
> -alias kafka -storepass server-truststore-password -noprompt
>
> keytool -importcert -file server-cert-file -keystore client.truststore.jks
> -alias kafkaclient -storepass client-truststore-password -noprompt
>
>
> keytool -genkey -alias kafkaclient -keystore client.keystore.jks -dname
> "CN=KafkaClient,O=Pivotal,C=UK" -storepass client-keystore-password
> -keypass client-key-password
>
> keytool -exportcert -file client-cert-file -keystore client.keystore.jks
> -alias kafkaclient -storepass client-keystore-password
>
> keytool -importcert -file client-cert-file -keystore server.truststore.jks
> -alias kafkaclient -storepass server-truststore-password -noprompt
>
> *Configure broker: Add these lines at the end of your server.properties*
>
> listeners=SSL://:9093
>
> advertised.listeners=SSL://127.0.0.1:9093
>
> ssl.keystore.location=/tmp/acl/server.keystore.jks
>
> ssl.keystore.password=server-keystore-password
>
> ssl.key.password=server-key-password
>
> ssl.truststore.location=/tmp/acl/server.truststore.jks
>
> ssl.truststore.password=server-truststore-password
>
> security.inter.broker.protocol=SSL
>
> security.protocol=SSL
>
> ssl.client.auth=required
>
> allow.everyone.if.no.acl.found=false
>
> authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
>
> super.users=User:CN=KafkaBroker,O=Pivotal,C=UK
>
> *Configure producer: producer.properties*
>
> security.protocol=SSL
>
> ssl.truststore.location=/tmp/acl/client.truststore.jks
>
> ssl.truststore.password=client-truststore-password
>
> ssl.keystore.location=/tmp/acl/client.keystore.jks
>
> ssl.keystore.password=client-keystore-password
>
> ssl.key.password=client-key-password
>
>
> *Configure consumer: consumer.properties*
>
> security.protocol=SSL
>
> ssl.truststore.location=/tmp/acl/client.truststore.jks
>
> ssl.truststore.password=client-truststore-password
>
> ssl.keystore.location=/tmp/acl/client.keystore.jks
>
> ssl.keystore.password=client-keystore-password
>
> ssl.key.password=client-key-password
>
> group.id=testgroup
>
> *Create topic:*
>
> bin/kafka-topics.sh --zookeeper localhost --create --topic testtopic
> --replication-factor 1 --partitions 1
>
>
> *Configure ACLs:*
>
> bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181
> --add --allow-principal "User:CN=KafkaClient,O=Pivotal,C=UK" --producer
> --topic testtopic
>
> bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181
> --add --allow-principal "User:CN=KafkaClient,O=Pivotal,C=UK" --consumer
> --topic testtopic --group test group
>
>
> *Run console producer and type in some messages:*
>
> bin/kafka-console-producer.sh --producer.config
> /tmp/acl/producer.properties --topic testtopic --broker-list
> 127.0.0.1:9093
>
>
> *Run console consumer, you should see messages from above:*
>
> bin/kafka-console-consumer.sh --consumer.config
> /tmp/acl/consumer.properties --topic testtopic --bootstrap-server
> 127.0.0.1:9093 --from-beginning
>
>
>
> On Tue, May 23, 2017 at 12:57 PM, Raghav <ra...@gmail.com> wrote:
>
>> Darshan,
>>
>> I have not yet successfully gotten the ACLs to work in Kafka. I am still
>> looking for help. I will update this email thread if I do find. In case
>> you
>> get it working, please let me know.
>>
>> Thanks.
>>
>> R
>>
>> On Tue, May 23, 2017 at 8:49 AM, Darshan Purandare <
>> purandare.darshan@gmail.com> wrote:
>>
>> > Raghav
>> >
>> > I saw few posts of yours around Kafka ACLs and the problems. I have seen
>> > similar issues where Writer has not been able to write to any topic. I
>> have
>> > seen "leader not available" and sometimes "unknown topic or partition",
>> and
>> > "topic_authorization_failed" error.
>> >
>> > Let me know if you find a valid config that works.
>> >
>> > Thanks.
>> >
>> >
>> >
>> > On Tue, May 23, 2017 at 8:44 AM, Raghav <ra...@gmail.com> wrote:
>> >
>> >> Hello Kafka Users
>> >>
>> >> I am a new Kafka user and trying to make Kafka SSL work with
>> Authorization
>> >> and ACLs. I followed posts from Kafka and Confluent docs exactly to the
>> >> point but my producer cannot write to kafka broker. I get
>> >> "LEADER_NOT_FOUND" errors. And even Consumer throws the same errors.
>> >>
>> >> Can someone please share their config which worked with ACLs.
>> >>
>> >> Here is my config. Please help.
>> >>
>> >> server.properties config
>> >> ------------------------------------------------------------
>> >> ------------------------------------------------
>> >> broker.id=0
>> >> auto.create.topics.enable=true
>> >> delete.topic.enable=true
>> >>
>> >> listeners=PLAINTEXT://kafka1.example.com:9092
>> >> <http://kafka-dev1.example.com:9092/>,SSL://kafka1.example.com:9093
>> >> <http://kafka-dev1.example.com:9093/>
>> >> host.name=kafka1.example.com <http://kafka-dev1.example.com/>
>> >>
>> >>
>> >>
>> >> ssl.keystore.location=/var/private/kafka1.keystore.jks
>> >> ssl.keystore.password=12345678
>> >> ssl.key.password=12345678
>> >>
>> >> ssl.truststore.location=/var/private/kafka1.truststore.jks
>> >> ssl.truststore.password=12345678
>> >>
>> >> ssl.client.auth=required
>> >> ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
>> >> ssl.keystore.type=JKS
>> >> ssl.truststore.type=JKS
>> >>
>> >> authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
>> >> ------------------------------------------------------------
>> >> ------------------------------------------------
>> >>
>> >>
>> >>
>> >> Here is producer Config(producer.properties)
>> >> ------------------------------------------------------------
>> >> ------------------------------------------------
>> >> security.protocol=SSL
>> >> ssl.truststore.location=/var/private/kafka2.truststore.jks
>> >> ssl.truststore.password=12345678
>> >>
>> >> ssl.keystore.location=/var/private/kafka2.keystore.jks
>> >> ssl.keystore.password=12345678
>> >> ssl.key.password=12345678
>> >>
>> >> ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
>> >> ssl.truststore.type=JKS
>> >> ssl.keystore.type=JKS
>> >>
>> >> ------------------------------------------------------------
>> >> ------------------------------------------------
>> >>
>> >>
>> >> Raqhav
>> >>
>> >
>> >
>>
>>
>> --
>> Raghav
>>
>
>
--
Raghav
Re: Kafka Authorization and ACLs Broken
Posted by Raghav <ra...@gmail.com>.
Hi Alex
In fact I copied the same configuration that Rajini pasted above and it
worked for me. You can try the same. Let me know if it doesn't work.
Thanks.
On Fri, May 26, 2017 at 4:19 AM, Kamalov, Alex <Al...@bnymellon.com>
wrote:
> Hey Raghav,
>
>
>
> Yes, I would very much love to get your configs, so I can model against it.
>
>
>
> Thanks again,
>
>
>
> Alex
>
>
>
> *From: *Raghav <ra...@gmail.com>
> *Date: *Thursday, May 25, 2017 at 10:54 PM
> *To: *Mike Marzo <pr...@gmail.com>
> *Cc: *Darshan Purandare <pu...@gmail.com>, Rajini Sivaram <
> rajinisivaram@gmail.com>, Users <us...@kafka.apache.org>, Alex Kamalov <
> Alex.Kamalov@bnymellon.com>
> *Subject: *Re: Kafka Authorization and ACLs Broken
>
>
>
> In SSL cert, there is a field which has a CN (Common Name). So when ACLs
> are set, they are set for that CN. This is how the ACLs are configured and
> matched against. I am still pretty new to Kafka in general, but this is how
> I think it works. I can copy my config if you want.
>
>
>
> On Thu, May 25, 2017 at 12:51 PM, Mike Marzo <
> precisionarcheryllc@gmail.com> wrote:
>
> Stupid question....
>
> If u don't specify a jaas file how does the consumer and producer specify
> the Id that acl's are configured against.... boy I am getting more and
> more perplexed by this...
>
> mike marzo
> 908 209-4484 <(908)%20209-4484>
>
>
>
> On May 24, 2017 9:29 PM, "Raghav" <ra...@gmail.com> wrote:
>
> Mike
>
>
>
> I am not using jaas file. I literally took the config Rajini gave in the
> previous email and it worked for me. I am using ssl Kafka with ACLs. I am
> not suing kerberos.
>
>
>
> Thanks.
>
>
>
> On Wed, May 24, 2017 at 11:29 AM, Mike Marzo <
> precisionarcheryllc@gmail.com> wrote:
>
> I'm also having issues getting acls to work. Out of intereat, are you
> starting ur brokers with a jaas file, if so do u mind sharing the client
> and server side jaas entries so I can validate what I'm doing.
>
> mike marzo
> 908 209-4484
>
> On May 24, 2017 10:54 AM, "Raghav" <ra...@gmail.com> wrote:
>
> > Hi Rajini
> >
> > Thank you very much. It perfectly works.
> >
> > I think in my setup I was trying to use a CA (certificate authority) to
> > sign the certificates from client and server, and then adding it to trust
> > store and keystore. I think in that process, I may have messed
> something. I
> > will try above config with a CA to sign certificates. Hopefully that
> would
> > work too.
> >
> > Thanks a lot again.
> >
> > Raghav
> >
> >
> >
> >
> > On Wed, May 24, 2017 at 7:04 AM, Rajini Sivaram <rajinisivaram@gmail.com
> >
> > wrote:
> >
> > > Raghav/Darshan,
> > >
> > > Can you try these steps on a clean installation of Kafka? It works for
> > me,
> > > so hopefully it will work for you. And then you can adapt to your
> > scenario.
> > >
> > > *Create keystores and truststores:*
> > >
> > > keytool -genkey -alias kafka -keystore server.keystore.jks -dname
> > > "CN=KafkaBroker,O=Pivotal,C=UK" -storepass server-keystore-password
> > > -keypass server-key-password
> > >
> > > keytool -exportcert -file server-cert-file -keystore
> server.keystore.jks
> > > -alias kafka -storepass server-keystore-password
> > >
> > > keytool -importcert -file server-cert-file -keystore
> > server.truststore.jks
> > > -alias kafka -storepass server-truststore-password -noprompt
> > >
> > > keytool -importcert -file server-cert-file -keystore
> > client.truststore.jks
> > > -alias kafkaclient -storepass client-truststore-password -noprompt
> > >
> > >
> > > keytool -genkey -alias kafkaclient -keystore client.keystore.jks -dname
> > > "CN=KafkaClient,O=Pivotal,C=UK" -storepass client-keystore-password
> > > -keypass client-key-password
> > >
> > > keytool -exportcert -file client-cert-file -keystore
> client.keystore.jks
> > > -alias kafkaclient -storepass client-keystore-password
> > >
> > > keytool -importcert -file client-cert-file -keystore
> > server.truststore.jks
> > > -alias kafkaclient -storepass server-truststore-password -noprompt
> > >
> > > *Configure broker: Add these lines at the end of your
> server.properties*
> > >
> > > listeners=SSL://:9093
> > >
> > > advertised.listeners=SSL://127.0.0.1:9093
> > >
> > > ssl.keystore.location=/tmp/acl/server.keystore.jks
> > >
> > > ssl.keystore.password=server-keystore-password
> > >
> > > ssl.key.password=server-key-password
> > >
> > > ssl.truststore.location=/tmp/acl/server.truststore.jks
> > >
> > > ssl.truststore.password=server-truststore-password
> > >
> > > security.inter.broker.protocol=SSL
> > >
> > > security.protocol=SSL
> > >
> > > ssl.client.auth=required
> > >
> > > allow.everyone.if.no.acl.found=false
> > >
> > > authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
> > >
> > > super.users=User:CN=KafkaBroker,O=Pivotal,C=UK
> > >
> > > *Configure producer: producer.properties*
> > >
> > > security.protocol=SSL
> > >
> > > ssl.truststore.location=/tmp/acl/client.truststore.jks
> > >
> > > ssl.truststore.password=client-truststore-password
> > >
> > > ssl.keystore.location=/tmp/acl/client.keystore.jks
> > >
> > > ssl.keystore.password=client-keystore-password
> > >
> > > ssl.key.password=client-key-password
> > >
> > >
> > > *Configure consumer: consumer.properties*
> > >
> > > security.protocol=SSL
> > >
> > > ssl.truststore.location=/tmp/acl/client.truststore.jks
> > >
> > > ssl.truststore.password=client-truststore-password
> > >
> > > ssl.keystore.location=/tmp/acl/client.keystore.jks
> > >
> > > ssl.keystore.password=client-keystore-password
> > >
> > > ssl.key.password=client-key-password
> > >
> > > group.id=testgroup
> > >
> > > *Create topic:*
> > >
> > > bin/kafka-topics.sh --zookeeper localhost --create --topic testtopic
> > > --replication-factor 1 --partitions 1
> > >
> > >
> > > *Configure ACLs:*
> > >
> > > bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:
> > 2181
> > > --add --allow-principal "User:CN=KafkaClient,O=Pivotal,C=UK"
> --producer
> > > --topic testtopic
> > >
> > > bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:
> > 2181
> > > --add --allow-principal "User:CN=KafkaClient,O=Pivotal,C=UK"
> --consumer
> > > --topic testtopic --group test group
> > >
> > >
> > > *Run console producer and type in some messages:*
> > >
> > > bin/kafka-console-producer.sh --producer.config
> > > /tmp/acl/producer.properties --topic testtopic --broker-list
> > > 127.0.0.1:9093
> > >
> > >
> > > *Run console consumer, you should see messages from above:*
>
> > >
> > > bin/kafka-console-consumer.sh --consumer.config
> > > /tmp/acl/consumer.properties --topic testtopic --bootstrap-server
> > > 127.0.0.1:9093 --from-beginning
> > >
> > >
> > >
> > > On Tue, May 23, 2017 at 12:57 PM, Raghav <ra...@gmail.com>
> wrote:
> > >
> > >> Darshan,
> > >>
> > >> I have not yet successfully gotten the ACLs to work in Kafka. I am
> still
> > >> looking for help. I will update this email thread if I do find. In
> case
> > >> you
> > >> get it working, please let me know.
> > >>
> > >> Thanks.
> > >>
> > >> R
> > >>
> > >> On Tue, May 23, 2017 at 8:49 AM, Darshan Purandare <
> > >> purandare.darshan@gmail.com> wrote:
> > >>
> > >> > Raghav
> > >> >
> > >> > I saw few posts of yours around Kafka ACLs and the problems. I have
> > seen
> > >> > similar issues where Writer has not been able to write to any
> topic. I
> > >> have
> > >> > seen "leader not available" and sometimes "unknown topic or
> > partition",
> > >> and
> > >> > "topic_authorization_failed" error.
> > >> >
> > >> > Let me know if you find a valid config that works.
> > >> >
> > >> > Thanks.
> > >> >
> > >> >
> > >> >
> > >> > On Tue, May 23, 2017 at 8:44 AM, Raghav <ra...@gmail.com>
> > wrote:
> > >> >
> > >> >> Hello Kafka Users
> > >> >>
> > >> >> I am a new Kafka user and trying to make Kafka SSL work with
> > >> Authorization
> > >> >> and ACLs. I followed posts from Kafka and Confluent docs exactly to
> > the
> > >> >> point but my producer cannot write to kafka broker. I get
> > >> >> "LEADER_NOT_FOUND" errors. And even Consumer throws the same
> errors.
> > >> >>
> > >> >> Can someone please share their config which worked with ACLs.
> > >> >>
> > >> >> Here is my config. Please help.
> > >> >>
> > >> >> server.properties config
> > >> >> ------------------------------------------------------------
> > >> >> ------------------------------------------------
> > >> >> broker.id=0
> > >> >> auto.create.topics.enable=true
> > >> >> delete.topic.enable=true
> > >> >>
> > >> >> listeners=PLAINTEXT://kafka1.example.com:9092
> > >> >> <http://kafka-dev1.example.com:9092/>,SSL://kafka1.
> example.com:9093
> > >> >> <http://kafka-dev1.example.com:9093/>
> > >> >> host.name=kafka1.example.com <http://kafka-dev1.example.com/>
> > >> >>
> > >> >>
> > >> >>
> > >> >> ssl.keystore.location=/var/private/kafka1.keystore.jks
> > >> >> ssl.keystore.password=12345678
> > >> >> ssl.key.password=12345678
> > >> >>
> > >> >> ssl.truststore.location=/var/private/kafka1.truststore.jks
> > >> >> ssl.truststore.password=12345678
> > >> >>
> > >> >> ssl.client.auth=required
> > >> >> ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
> > >> >> ssl.keystore.type=JKS
> > >> >> ssl.truststore.type=JKS
> > >> >>
> > >> >> authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
> > >> >> ------------------------------------------------------------
> > >> >> ------------------------------------------------
> > >> >>
> > >> >>
> > >> >>
> > >> >> Here is producer Config(producer.properties)
> > >> >> ------------------------------------------------------------
> > >> >> ------------------------------------------------
> > >> >> security.protocol=SSL
> > >> >> ssl.truststore.location=/var/private/kafka2.truststore.jks
> > >> >> ssl.truststore.password=12345678
> > >> >>
> > >> >> ssl.keystore.location=/var/private/kafka2.keystore.jks
> > >> >> ssl.keystore.password=12345678
> > >> >> ssl.key.password=12345678
> > >> >>
> > >> >> ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
> > >> >> ssl.truststore.type=JKS
> > >> >> ssl.keystore.type=JKS
> > >> >>
> > >> >> ------------------------------------------------------------
> > >> >> ------------------------------------------------
> > >> >>
> > >> >>
> > >> >> Raqhav
> > >> >>
> > >> >
> > >> >
> > >>
> > >>
> > >> --
> > >> Raghav
> > >>
> > >
> > >
> >
> >
> > --
> > Raghav
> >
>
>
>
>
>
> --
>
> Raghav
>
>
>
>
>
> --
>
> Raghav
>
> The information contained in this e-mail, and any attachment, is
> confidential and is intended solely for the use of the intended recipient.
> Access, copying or re-use of the e-mail or any attachment, or any
> information contained therein, by any other person is not authorized. If
> you are not the intended recipient please return the e-mail to the sender
> and delete it from your computer. Although we attempt to sweep e-mail and
> attachments for viruses, we do not guarantee that either are virus-free and
> accept no liability for any damage sustained as a result of viruses.
>
> Please refer to http://disclaimer.bnymellon.com/eu.htm for certain
> disclosures relating to European legal entities.
>
--
Raghav
Re: Kafka Authorization and ACLs Broken
Posted by "Kamalov, Alex" <Al...@bnymellon.com>.
Hey Raghav,
Yes, I would very much love to get your configs, so I can model against it.
Thanks again,
Alex
From: Raghav <ra...@gmail.com>
Date: Thursday, May 25, 2017 at 10:54 PM
To: Mike Marzo <pr...@gmail.com>
Cc: Darshan Purandare <pu...@gmail.com>, Rajini Sivaram <ra...@gmail.com>, Users <us...@kafka.apache.org>, Alex Kamalov <Al...@bnymellon.com>
Subject: Re: Kafka Authorization and ACLs Broken
In SSL cert, there is a field which has a CN (Common Name). So when ACLs are set, they are set for that CN. This is how the ACLs are configured and matched against. I am still pretty new to Kafka in general, but this is how I think it works. I can copy my config if you want.
On Thu, May 25, 2017 at 12:51 PM, Mike Marzo <pr...@gmail.com>> wrote:
Stupid question....
If u don't specify a jaas file how does the consumer and producer specify the Id that acl's are configured against.... boy I am getting more and more perplexed by this...
mike marzo
908 209-4484<tel:(908)%20209-4484>
On May 24, 2017 9:29 PM, "Raghav" <ra...@gmail.com>> wrote:
Mike
I am not using jaas file. I literally took the config Rajini gave in the previous email and it worked for me. I am using ssl Kafka with ACLs. I am not suing kerberos.
Thanks.
On Wed, May 24, 2017 at 11:29 AM, Mike Marzo <pr...@gmail.com>> wrote:
I'm also having issues getting acls to work. Out of intereat, are you
starting ur brokers with a jaas file, if so do u mind sharing the client
and server side jaas entries so I can validate what I'm doing.
mike marzo
908 209-4484<tel:908%20209-4484>
On May 24, 2017 10:54 AM, "Raghav" <ra...@gmail.com>> wrote:
> Hi Rajini
>
> Thank you very much. It perfectly works.
>
> I think in my setup I was trying to use a CA (certificate authority) to
> sign the certificates from client and server, and then adding it to trust
> store and keystore. I think in that process, I may have messed something. I
> will try above config with a CA to sign certificates. Hopefully that would
> work too.
>
> Thanks a lot again.
>
> Raghav
>
>
>
>
> On Wed, May 24, 2017 at 7:04 AM, Rajini Sivaram <ra...@gmail.com>>
> wrote:
>
> > Raghav/Darshan,
> >
> > Can you try these steps on a clean installation of Kafka? It works for
> me,
> > so hopefully it will work for you. And then you can adapt to your
> scenario.
> >
> > *Create keystores and truststores:*
> >
> > keytool -genkey -alias kafka -keystore server.keystore.jks -dname
> > "CN=KafkaBroker,O=Pivotal,C=UK" -storepass server-keystore-password
> > -keypass server-key-password
> >
> > keytool -exportcert -file server-cert-file -keystore server.keystore.jks
> > -alias kafka -storepass server-keystore-password
> >
> > keytool -importcert -file server-cert-file -keystore
> server.truststore.jks
> > -alias kafka -storepass server-truststore-password -noprompt
> >
> > keytool -importcert -file server-cert-file -keystore
> client.truststore.jks
> > -alias kafkaclient -storepass client-truststore-password -noprompt
> >
> >
> > keytool -genkey -alias kafkaclient -keystore client.keystore.jks -dname
> > "CN=KafkaClient,O=Pivotal,C=UK" -storepass client-keystore-password
> > -keypass client-key-password
> >
> > keytool -exportcert -file client-cert-file -keystore client.keystore.jks
> > -alias kafkaclient -storepass client-keystore-password
> >
> > keytool -importcert -file client-cert-file -keystore
> server.truststore.jks
> > -alias kafkaclient -storepass server-truststore-password -noprompt
> >
> > *Configure broker: Add these lines at the end of your server.properties*
> >
> > listeners=SSL://:9093
> >
> > advertised.listeners=SSL://127.0.0.1:9093<http://127.0.0.1:9093>
> >
> > ssl.keystore.location=/tmp/acl/server.keystore.jks
> >
> > ssl.keystore.password=server-keystore-password
> >
> > ssl.key.password=server-key-password
> >
> > ssl.truststore.location=/tmp/acl/server.truststore.jks
> >
> > ssl.truststore.password=server-truststore-password
> >
> > security.inter.broker.protocol=SSL
> >
> > security.protocol=SSL
> >
> > ssl.client.auth=required
> >
> > allow.everyone.if.no.acl.found=false
> >
> > authorizer.class.name<http://authorizer.class.name>=kafka.se<http://kafka.se>curity.auth.SimpleAclAuthorizer
> >
> > super.users=User:CN=KafkaBroker,O=Pivotal,C=UK
> >
> > *Configure producer: producer.properties*
> >
> > security.protocol=SSL
> >
> > ssl.truststore.location=/tmp/acl/client.truststore.jks
> >
> > ssl.truststore.password=client-truststore-password
> >
> > ssl.keystore.location=/tmp/acl/client.keystore.jks
> >
> > ssl.keystore.password=client-keystore-password
> >
> > ssl.key.password=client-key-password
> >
> >
> > *Configure consumer: consumer.properties*
> >
> > security.protocol=SSL
> >
> > ssl.truststore.location=/tmp/acl/client.truststore.jks
> >
> > ssl.truststore.password=client-truststore-password
> >
> > ssl.keystore.location=/tmp/acl/client.keystore.jks
> >
> > ssl.keystore.password=client-keystore-password
> >
> > ssl.key.password=client-key-password
> >
> > group.id<http://group.id>=testgroup
> >
> > *Create topic:*
> >
> > bin/kafka-topics.sh --zookeeper localhost --create --topic testtopic
> > --replication-factor 1 --partitions 1
> >
> >
> > *Configure ACLs:*
> >
> > bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:
> 2181
> > --add --allow-principal "User:CN=KafkaClient,O=Pivotal,C=UK" --producer
> > --topic testtopic
> >
> > bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:
> 2181
> > --add --allow-principal "User:CN=KafkaClient,O=Pivotal,C=UK" --consumer
> > --topic testtopic --group test group
> >
> >
> > *Run console producer and type in some messages:*
> >
> > bin/kafka-console-producer.sh --producer.config
> > /tmp/acl/producer.properties --topic testtopic --broker-list
> > 127.0.0.1:9093<http://127.0.0.1:9093>
> >
> >
> > *Run console consumer, you should see messages from above:*
> >
> > bin/kafka-console-consumer.sh --consumer.config
> > /tmp/acl/consumer.properties --topic testtopic --bootstrap-server
> > 127.0.0.1:9093<http://127.0.0.1:9093> --from-beginning
> >
> >
> >
> > On Tue, May 23, 2017 at 12:57 PM, Raghav <ra...@gmail.com>> wrote:
> >
> >> Darshan,
> >>
> >> I have not yet successfully gotten the ACLs to work in Kafka. I am still
> >> looking for help. I will update this email thread if I do find. In case
> >> you
> >> get it working, please let me know.
> >>
> >> Thanks.
> >>
> >> R
> >>
> >> On Tue, May 23, 2017 at 8:49 AM, Darshan Purandare <
> >> purandare.darshan@gmail.com<ma...@gmail.com>> wrote:
> >>
> >> > Raghav
> >> >
> >> > I saw few posts of yours around Kafka ACLs and the problems. I have
> seen
> >> > similar issues where Writer has not been able to write to any topic. I
> >> have
> >> > seen "leader not available" and sometimes "unknown topic or
> partition",
> >> and
> >> > "topic_authorization_failed" error.
> >> >
> >> > Let me know if you find a valid config that works.
> >> >
> >> > Thanks.
> >> >
> >> >
> >> >
> >> > On Tue, May 23, 2017 at 8:44 AM, Raghav <ra...@gmail.com>>
> wrote:
> >> >
> >> >> Hello Kafka Users
> >> >>
> >> >> I am a new Kafka user and trying to make Kafka SSL work with
> >> Authorization
> >> >> and ACLs. I followed posts from Kafka and Confluent docs exactly to
> the
> >> >> point but my producer cannot write to kafka broker. I get
> >> >> "LEADER_NOT_FOUND" errors. And even Consumer throws the same errors.
> >> >>
> >> >> Can someone please share their config which worked with ACLs.
> >> >>
> >> >> Here is my config. Please help.
> >> >>
> >> >> server.properties config
> >> >> ------------------------------------------------------------
> >> >> ------------------------------------------------
> >> >> broker.id<http://broker.id>=0
> >> >> auto.create.topics.enable=true
> >> >> delete.topic.enable=true
> >> >>
> >> >> listeners=PLAINTEXT://kafka1.example.com:9092<http://kafka1.example.com:9092>
> >> >> <http://kafka-dev1.example.com:9092/>,SSL://kafka1.example.com:9093<http://kafka1.example.com:9093>
> >> >> <http://kafka-dev1.example.com:9093/>
> >> >> host.name<http://host.name>=kafka1.example.com<http://kafka1.example.com> <http://kafka-dev1.example.com/>
> >> >>
> >> >>
> >> >>
> >> >> ssl.keystore.location=/var/private/kafka1.keystore.jks
> >> >> ssl.keystore.password=12345678
> >> >> ssl.key.password=12345678
> >> >>
> >> >> ssl.truststore.location=/var/private/kafka1.truststore.jks
> >> >> ssl.truststore.password=12345678
> >> >>
> >> >> ssl.client.auth=required
> >> >> ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
> >> >> ssl.keystore.type=JKS
> >> >> ssl.truststore.type=JKS
> >> >>
> >> >> authorizer.class.name<http://authorizer.class.name>=kafka.se<http://kafka.se>curity.auth.SimpleAclAuthorizer
> >> >> ------------------------------------------------------------
> >> >> ------------------------------------------------
> >> >>
> >> >>
> >> >>
> >> >> Here is producer Config(producer.properties)
> >> >> ------------------------------------------------------------
> >> >> ------------------------------------------------
> >> >> security.protocol=SSL
> >> >> ssl.truststore.location=/var/private/kafka2.truststore.jks
> >> >> ssl.truststore.password=12345678
> >> >>
> >> >> ssl.keystore.location=/var/private/kafka2.keystore.jks
> >> >> ssl.keystore.password=12345678
> >> >> ssl.key.password=12345678
> >> >>
> >> >> ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
> >> >> ssl.truststore.type=JKS
> >> >> ssl.keystore.type=JKS
> >> >>
> >> >> ------------------------------------------------------------
> >> >> ------------------------------------------------
> >> >>
> >> >>
> >> >> Raqhav
> >> >>
> >> >
> >> >
> >>
> >>
> >> --
> >> Raghav
> >>
> >
> >
>
>
> --
> Raghav
>
--
Raghav
--
Raghav
The information contained in this e-mail, and any attachment, is confidential and is intended solely for the use of the intended recipient. Access, copying or re-use of the e-mail or any attachment, or any information contained therein, by any other person is not authorized. If you are not the intended recipient please return the e-mail to the sender and delete it from your computer. Although we attempt to sweep e-mail and attachments for viruses, we do not guarantee that either are virus-free and accept no liability for any damage sustained as a result of viruses.
Please refer to http://disclaimer.bnymellon.com/eu.htm for certain disclosures relating to European legal entities.
Re: Kafka Authorization and ACLs Broken
Posted by Raghav <ra...@gmail.com>.
In SSL cert, there is a field which has a CN (Common Name). So when ACLs
are set, they are set for that CN. This is how the ACLs are configured and
matched against. I am still pretty new to Kafka in general, but this is how
I think it works. I can copy my config if you want.
On Thu, May 25, 2017 at 12:51 PM, Mike Marzo <pr...@gmail.com>
wrote:
> Stupid question....
> If u don't specify a jaas file how does the consumer and producer specify
> the Id that acl's are configured against.... boy I am getting more and
> more perplexed by this...
>
> mike marzo
> 908 209-4484 <(908)%20209-4484>
>
> On May 24, 2017 9:29 PM, "Raghav" <ra...@gmail.com> wrote:
>
>> Mike
>>
>> I am not using jaas file. I literally took the config Rajini gave in the
>> previous email and it worked for me. I am using ssl Kafka with ACLs. I am
>> not suing kerberos.
>>
>> Thanks.
>>
>> On Wed, May 24, 2017 at 11:29 AM, Mike Marzo <
>> precisionarcheryllc@gmail.com> wrote:
>>
>>> I'm also having issues getting acls to work. Out of intereat, are you
>>> starting ur brokers with a jaas file, if so do u mind sharing the client
>>> and server side jaas entries so I can validate what I'm doing.
>>>
>>> mike marzo
>>> 908 209-4484
>>>
>>> On May 24, 2017 10:54 AM, "Raghav" <ra...@gmail.com> wrote:
>>>
>>> > Hi Rajini
>>> >
>>> > Thank you very much. It perfectly works.
>>> >
>>> > I think in my setup I was trying to use a CA (certificate authority) to
>>> > sign the certificates from client and server, and then adding it to
>>> trust
>>> > store and keystore. I think in that process, I may have messed
>>> something. I
>>> > will try above config with a CA to sign certificates. Hopefully that
>>> would
>>> > work too.
>>> >
>>> > Thanks a lot again.
>>> >
>>> > Raghav
>>> >
>>> >
>>> >
>>> >
>>> > On Wed, May 24, 2017 at 7:04 AM, Rajini Sivaram <
>>> rajinisivaram@gmail.com>
>>> > wrote:
>>> >
>>> > > Raghav/Darshan,
>>> > >
>>> > > Can you try these steps on a clean installation of Kafka? It works
>>> for
>>> > me,
>>> > > so hopefully it will work for you. And then you can adapt to your
>>> > scenario.
>>> > >
>>> > > *Create keystores and truststores:*
>>> > >
>>> > > keytool -genkey -alias kafka -keystore server.keystore.jks -dname
>>> > > "CN=KafkaBroker,O=Pivotal,C=UK" -storepass server-keystore-password
>>> > > -keypass server-key-password
>>> > >
>>> > > keytool -exportcert -file server-cert-file -keystore
>>> server.keystore.jks
>>> > > -alias kafka -storepass server-keystore-password
>>> > >
>>> > > keytool -importcert -file server-cert-file -keystore
>>> > server.truststore.jks
>>> > > -alias kafka -storepass server-truststore-password -noprompt
>>> > >
>>> > > keytool -importcert -file server-cert-file -keystore
>>> > client.truststore.jks
>>> > > -alias kafkaclient -storepass client-truststore-password -noprompt
>>> > >
>>> > >
>>> > > keytool -genkey -alias kafkaclient -keystore client.keystore.jks
>>> -dname
>>> > > "CN=KafkaClient,O=Pivotal,C=UK" -storepass client-keystore-password
>>> > > -keypass client-key-password
>>> > >
>>> > > keytool -exportcert -file client-cert-file -keystore
>>> client.keystore.jks
>>> > > -alias kafkaclient -storepass client-keystore-password
>>> > >
>>> > > keytool -importcert -file client-cert-file -keystore
>>> > server.truststore.jks
>>> > > -alias kafkaclient -storepass server-truststore-password -noprompt
>>> > >
>>> > > *Configure broker: Add these lines at the end of your
>>> server.properties*
>>> > >
>>> > > listeners=SSL://:9093
>>> > >
>>> > > advertised.listeners=SSL://127.0.0.1:9093
>>> > >
>>> > > ssl.keystore.location=/tmp/acl/server.keystore.jks
>>> > >
>>> > > ssl.keystore.password=server-keystore-password
>>> > >
>>> > > ssl.key.password=server-key-password
>>> > >
>>> > > ssl.truststore.location=/tmp/acl/server.truststore.jks
>>> > >
>>> > > ssl.truststore.password=server-truststore-password
>>> > >
>>> > > security.inter.broker.protocol=SSL
>>> > >
>>> > > security.protocol=SSL
>>> > >
>>> > > ssl.client.auth=required
>>> > >
>>> > > allow.everyone.if.no.acl.found=false
>>> > >
>>> > > authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
>>> > >
>>> > > super.users=User:CN=KafkaBroker,O=Pivotal,C=UK
>>> > >
>>> > > *Configure producer: producer.properties*
>>> > >
>>> > > security.protocol=SSL
>>> > >
>>> > > ssl.truststore.location=/tmp/acl/client.truststore.jks
>>> > >
>>> > > ssl.truststore.password=client-truststore-password
>>> > >
>>> > > ssl.keystore.location=/tmp/acl/client.keystore.jks
>>> > >
>>> > > ssl.keystore.password=client-keystore-password
>>> > >
>>> > > ssl.key.password=client-key-password
>>> > >
>>> > >
>>> > > *Configure consumer: consumer.properties*
>>> > >
>>> > > security.protocol=SSL
>>> > >
>>> > > ssl.truststore.location=/tmp/acl/client.truststore.jks
>>> > >
>>> > > ssl.truststore.password=client-truststore-password
>>> > >
>>> > > ssl.keystore.location=/tmp/acl/client.keystore.jks
>>> > >
>>> > > ssl.keystore.password=client-keystore-password
>>> > >
>>> > > ssl.key.password=client-key-password
>>> > >
>>> > > group.id=testgroup
>>> > >
>>> > > *Create topic:*
>>> > >
>>> > > bin/kafka-topics.sh --zookeeper localhost --create --topic testtopic
>>> > > --replication-factor 1 --partitions 1
>>> > >
>>> > >
>>> > > *Configure ACLs:*
>>> > >
>>> > > bin/kafka-acls.sh --authorizer-properties
>>> zookeeper.connect=localhost:
>>> > 2181
>>> > > --add --allow-principal "User:CN=KafkaClient,O=Pivotal,C=UK"
>>> --producer
>>> > > --topic testtopic
>>> > >
>>> > > bin/kafka-acls.sh --authorizer-properties
>>> zookeeper.connect=localhost:
>>> > 2181
>>> > > --add --allow-principal "User:CN=KafkaClient,O=Pivotal,C=UK"
>>> --consumer
>>> > > --topic testtopic --group test group
>>> > >
>>> > >
>>> > > *Run console producer and type in some messages:*
>>> > >
>>> > > bin/kafka-console-producer.sh --producer.config
>>> > > /tmp/acl/producer.properties --topic testtopic --broker-list
>>> > > 127.0.0.1:9093
>>> > >
>>> > >
>>> > > *Run console consumer, you should see messages from above:*
>>> > >
>>> > > bin/kafka-console-consumer.sh --consumer.config
>>> > > /tmp/acl/consumer.properties --topic testtopic --bootstrap-server
>>> > > 127.0.0.1:9093 --from-beginning
>>> > >
>>> > >
>>> > >
>>> > > On Tue, May 23, 2017 at 12:57 PM, Raghav <ra...@gmail.com>
>>> wrote:
>>> > >
>>> > >> Darshan,
>>> > >>
>>> > >> I have not yet successfully gotten the ACLs to work in Kafka. I am
>>> still
>>> > >> looking for help. I will update this email thread if I do find. In
>>> case
>>> > >> you
>>> > >> get it working, please let me know.
>>> > >>
>>> > >> Thanks.
>>> > >>
>>> > >> R
>>> > >>
>>> > >> On Tue, May 23, 2017 at 8:49 AM, Darshan Purandare <
>>> > >> purandare.darshan@gmail.com> wrote:
>>> > >>
>>> > >> > Raghav
>>> > >> >
>>> > >> > I saw few posts of yours around Kafka ACLs and the problems. I
>>> have
>>> > seen
>>> > >> > similar issues where Writer has not been able to write to any
>>> topic. I
>>> > >> have
>>> > >> > seen "leader not available" and sometimes "unknown topic or
>>> > partition",
>>> > >> and
>>> > >> > "topic_authorization_failed" error.
>>> > >> >
>>> > >> > Let me know if you find a valid config that works.
>>> > >> >
>>> > >> > Thanks.
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> > On Tue, May 23, 2017 at 8:44 AM, Raghav <ra...@gmail.com>
>>> > wrote:
>>> > >> >
>>> > >> >> Hello Kafka Users
>>> > >> >>
>>> > >> >> I am a new Kafka user and trying to make Kafka SSL work with
>>> > >> Authorization
>>> > >> >> and ACLs. I followed posts from Kafka and Confluent docs exactly
>>> to
>>> > the
>>> > >> >> point but my producer cannot write to kafka broker. I get
>>> > >> >> "LEADER_NOT_FOUND" errors. And even Consumer throws the same
>>> errors.
>>> > >> >>
>>> > >> >> Can someone please share their config which worked with ACLs.
>>> > >> >>
>>> > >> >> Here is my config. Please help.
>>> > >> >>
>>> > >> >> server.properties config
>>> > >> >> ------------------------------------------------------------
>>> > >> >> ------------------------------------------------
>>> > >> >> broker.id=0
>>> > >> >> auto.create.topics.enable=true
>>> > >> >> delete.topic.enable=true
>>> > >> >>
>>> > >> >> listeners=PLAINTEXT://kafka1.example.com:9092
>>> > >> >> <http://kafka-dev1.example.com:9092/>,SSL://kafka1.example.c
>>> om:9093
>>> > >> >> <http://kafka-dev1.example.com:9093/>
>>> > >> >> host.name=kafka1.example.com <http://kafka-dev1.example.com/>
>>> > >> >>
>>> > >> >>
>>> > >> >>
>>> > >> >> ssl.keystore.location=/var/private/kafka1.keystore.jks
>>> > >> >> ssl.keystore.password=12345678
>>> > >> >> ssl.key.password=12345678
>>> > >> >>
>>> > >> >> ssl.truststore.location=/var/private/kafka1.truststore.jks
>>> > >> >> ssl.truststore.password=12345678
>>> > >> >>
>>> > >> >> ssl.client.auth=required
>>> > >> >> ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
>>> > >> >> ssl.keystore.type=JKS
>>> > >> >> ssl.truststore.type=JKS
>>> > >> >>
>>> > >> >> authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
>>> > >> >> ------------------------------------------------------------
>>> > >> >> ------------------------------------------------
>>> > >> >>
>>> > >> >>
>>> > >> >>
>>> > >> >> Here is producer Config(producer.properties)
>>> > >> >> ------------------------------------------------------------
>>> > >> >> ------------------------------------------------
>>> > >> >> security.protocol=SSL
>>> > >> >> ssl.truststore.location=/var/private/kafka2.truststore.jks
>>> > >> >> ssl.truststore.password=12345678
>>> > >> >>
>>> > >> >> ssl.keystore.location=/var/private/kafka2.keystore.jks
>>> > >> >> ssl.keystore.password=12345678
>>> > >> >> ssl.key.password=12345678
>>> > >> >>
>>> > >> >> ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
>>> > >> >> ssl.truststore.type=JKS
>>> > >> >> ssl.keystore.type=JKS
>>> > >> >>
>>> > >> >> ------------------------------------------------------------
>>> > >> >> ------------------------------------------------
>>> > >> >>
>>> > >> >>
>>> > >> >> Raqhav
>>> > >> >>
>>> > >> >
>>> > >> >
>>> > >>
>>> > >>
>>> > >> --
>>> > >> Raghav
>>> > >>
>>> > >
>>> > >
>>> >
>>> >
>>> > --
>>> > Raghav
>>> >
>>>
>>
>>
>>
>> --
>> Raghav
>>
>
--
Raghav
Re: Kafka Authorization and ACLs Broken
Posted by Mike Marzo <pr...@gmail.com>.
Stupid question....
If u don't specify a jaas file how does the consumer and producer specify
the Id that acl's are configured against.... boy I am getting more and
more perplexed by this...
mike marzo
908 209-4484
On May 24, 2017 9:29 PM, "Raghav" <ra...@gmail.com> wrote:
> Mike
>
> I am not using jaas file. I literally took the config Rajini gave in the
> previous email and it worked for me. I am using ssl Kafka with ACLs. I am
> not suing kerberos.
>
> Thanks.
>
> On Wed, May 24, 2017 at 11:29 AM, Mike Marzo <
> precisionarcheryllc@gmail.com> wrote:
>
>> I'm also having issues getting acls to work. Out of intereat, are you
>> starting ur brokers with a jaas file, if so do u mind sharing the client
>> and server side jaas entries so I can validate what I'm doing.
>>
>> mike marzo
>> 908 209-4484
>>
>> On May 24, 2017 10:54 AM, "Raghav" <ra...@gmail.com> wrote:
>>
>> > Hi Rajini
>> >
>> > Thank you very much. It perfectly works.
>> >
>> > I think in my setup I was trying to use a CA (certificate authority) to
>> > sign the certificates from client and server, and then adding it to
>> trust
>> > store and keystore. I think in that process, I may have messed
>> something. I
>> > will try above config with a CA to sign certificates. Hopefully that
>> would
>> > work too.
>> >
>> > Thanks a lot again.
>> >
>> > Raghav
>> >
>> >
>> >
>> >
>> > On Wed, May 24, 2017 at 7:04 AM, Rajini Sivaram <
>> rajinisivaram@gmail.com>
>> > wrote:
>> >
>> > > Raghav/Darshan,
>> > >
>> > > Can you try these steps on a clean installation of Kafka? It works for
>> > me,
>> > > so hopefully it will work for you. And then you can adapt to your
>> > scenario.
>> > >
>> > > *Create keystores and truststores:*
>> > >
>> > > keytool -genkey -alias kafka -keystore server.keystore.jks -dname
>> > > "CN=KafkaBroker,O=Pivotal,C=UK" -storepass server-keystore-password
>> > > -keypass server-key-password
>> > >
>> > > keytool -exportcert -file server-cert-file -keystore
>> server.keystore.jks
>> > > -alias kafka -storepass server-keystore-password
>> > >
>> > > keytool -importcert -file server-cert-file -keystore
>> > server.truststore.jks
>> > > -alias kafka -storepass server-truststore-password -noprompt
>> > >
>> > > keytool -importcert -file server-cert-file -keystore
>> > client.truststore.jks
>> > > -alias kafkaclient -storepass client-truststore-password -noprompt
>> > >
>> > >
>> > > keytool -genkey -alias kafkaclient -keystore client.keystore.jks
>> -dname
>> > > "CN=KafkaClient,O=Pivotal,C=UK" -storepass client-keystore-password
>> > > -keypass client-key-password
>> > >
>> > > keytool -exportcert -file client-cert-file -keystore
>> client.keystore.jks
>> > > -alias kafkaclient -storepass client-keystore-password
>> > >
>> > > keytool -importcert -file client-cert-file -keystore
>> > server.truststore.jks
>> > > -alias kafkaclient -storepass server-truststore-password -noprompt
>> > >
>> > > *Configure broker: Add these lines at the end of your
>> server.properties*
>> > >
>> > > listeners=SSL://:9093
>> > >
>> > > advertised.listeners=SSL://127.0.0.1:9093
>> > >
>> > > ssl.keystore.location=/tmp/acl/server.keystore.jks
>> > >
>> > > ssl.keystore.password=server-keystore-password
>> > >
>> > > ssl.key.password=server-key-password
>> > >
>> > > ssl.truststore.location=/tmp/acl/server.truststore.jks
>> > >
>> > > ssl.truststore.password=server-truststore-password
>> > >
>> > > security.inter.broker.protocol=SSL
>> > >
>> > > security.protocol=SSL
>> > >
>> > > ssl.client.auth=required
>> > >
>> > > allow.everyone.if.no.acl.found=false
>> > >
>> > > authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
>> > >
>> > > super.users=User:CN=KafkaBroker,O=Pivotal,C=UK
>> > >
>> > > *Configure producer: producer.properties*
>> > >
>> > > security.protocol=SSL
>> > >
>> > > ssl.truststore.location=/tmp/acl/client.truststore.jks
>> > >
>> > > ssl.truststore.password=client-truststore-password
>> > >
>> > > ssl.keystore.location=/tmp/acl/client.keystore.jks
>> > >
>> > > ssl.keystore.password=client-keystore-password
>> > >
>> > > ssl.key.password=client-key-password
>> > >
>> > >
>> > > *Configure consumer: consumer.properties*
>> > >
>> > > security.protocol=SSL
>> > >
>> > > ssl.truststore.location=/tmp/acl/client.truststore.jks
>> > >
>> > > ssl.truststore.password=client-truststore-password
>> > >
>> > > ssl.keystore.location=/tmp/acl/client.keystore.jks
>> > >
>> > > ssl.keystore.password=client-keystore-password
>> > >
>> > > ssl.key.password=client-key-password
>> > >
>> > > group.id=testgroup
>> > >
>> > > *Create topic:*
>> > >
>> > > bin/kafka-topics.sh --zookeeper localhost --create --topic testtopic
>> > > --replication-factor 1 --partitions 1
>> > >
>> > >
>> > > *Configure ACLs:*
>> > >
>> > > bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:
>> > 2181
>> > > --add --allow-principal "User:CN=KafkaClient,O=Pivotal,C=UK"
>> --producer
>> > > --topic testtopic
>> > >
>> > > bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:
>> > 2181
>> > > --add --allow-principal "User:CN=KafkaClient,O=Pivotal,C=UK"
>> --consumer
>> > > --topic testtopic --group test group
>> > >
>> > >
>> > > *Run console producer and type in some messages:*
>> > >
>> > > bin/kafka-console-producer.sh --producer.config
>> > > /tmp/acl/producer.properties --topic testtopic --broker-list
>> > > 127.0.0.1:9093
>> > >
>> > >
>> > > *Run console consumer, you should see messages from above:*
>> > >
>> > > bin/kafka-console-consumer.sh --consumer.config
>> > > /tmp/acl/consumer.properties --topic testtopic --bootstrap-server
>> > > 127.0.0.1:9093 --from-beginning
>> > >
>> > >
>> > >
>> > > On Tue, May 23, 2017 at 12:57 PM, Raghav <ra...@gmail.com>
>> wrote:
>> > >
>> > >> Darshan,
>> > >>
>> > >> I have not yet successfully gotten the ACLs to work in Kafka. I am
>> still
>> > >> looking for help. I will update this email thread if I do find. In
>> case
>> > >> you
>> > >> get it working, please let me know.
>> > >>
>> > >> Thanks.
>> > >>
>> > >> R
>> > >>
>> > >> On Tue, May 23, 2017 at 8:49 AM, Darshan Purandare <
>> > >> purandare.darshan@gmail.com> wrote:
>> > >>
>> > >> > Raghav
>> > >> >
>> > >> > I saw few posts of yours around Kafka ACLs and the problems. I have
>> > seen
>> > >> > similar issues where Writer has not been able to write to any
>> topic. I
>> > >> have
>> > >> > seen "leader not available" and sometimes "unknown topic or
>> > partition",
>> > >> and
>> > >> > "topic_authorization_failed" error.
>> > >> >
>> > >> > Let me know if you find a valid config that works.
>> > >> >
>> > >> > Thanks.
>> > >> >
>> > >> >
>> > >> >
>> > >> > On Tue, May 23, 2017 at 8:44 AM, Raghav <ra...@gmail.com>
>> > wrote:
>> > >> >
>> > >> >> Hello Kafka Users
>> > >> >>
>> > >> >> I am a new Kafka user and trying to make Kafka SSL work with
>> > >> Authorization
>> > >> >> and ACLs. I followed posts from Kafka and Confluent docs exactly
>> to
>> > the
>> > >> >> point but my producer cannot write to kafka broker. I get
>> > >> >> "LEADER_NOT_FOUND" errors. And even Consumer throws the same
>> errors.
>> > >> >>
>> > >> >> Can someone please share their config which worked with ACLs.
>> > >> >>
>> > >> >> Here is my config. Please help.
>> > >> >>
>> > >> >> server.properties config
>> > >> >> ------------------------------------------------------------
>> > >> >> ------------------------------------------------
>> > >> >> broker.id=0
>> > >> >> auto.create.topics.enable=true
>> > >> >> delete.topic.enable=true
>> > >> >>
>> > >> >> listeners=PLAINTEXT://kafka1.example.com:9092
>> > >> >> <http://kafka-dev1.example.com:9092/>,SSL://kafka1.example.
>> com:9093
>> > >> >> <http://kafka-dev1.example.com:9093/>
>> > >> >> host.name=kafka1.example.com <http://kafka-dev1.example.com/>
>> > >> >>
>> > >> >>
>> > >> >>
>> > >> >> ssl.keystore.location=/var/private/kafka1.keystore.jks
>> > >> >> ssl.keystore.password=12345678
>> > >> >> ssl.key.password=12345678
>> > >> >>
>> > >> >> ssl.truststore.location=/var/private/kafka1.truststore.jks
>> > >> >> ssl.truststore.password=12345678
>> > >> >>
>> > >> >> ssl.client.auth=required
>> > >> >> ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
>> > >> >> ssl.keystore.type=JKS
>> > >> >> ssl.truststore.type=JKS
>> > >> >>
>> > >> >> authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
>> > >> >> ------------------------------------------------------------
>> > >> >> ------------------------------------------------
>> > >> >>
>> > >> >>
>> > >> >>
>> > >> >> Here is producer Config(producer.properties)
>> > >> >> ------------------------------------------------------------
>> > >> >> ------------------------------------------------
>> > >> >> security.protocol=SSL
>> > >> >> ssl.truststore.location=/var/private/kafka2.truststore.jks
>> > >> >> ssl.truststore.password=12345678
>> > >> >>
>> > >> >> ssl.keystore.location=/var/private/kafka2.keystore.jks
>> > >> >> ssl.keystore.password=12345678
>> > >> >> ssl.key.password=12345678
>> > >> >>
>> > >> >> ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
>> > >> >> ssl.truststore.type=JKS
>> > >> >> ssl.keystore.type=JKS
>> > >> >>
>> > >> >> ------------------------------------------------------------
>> > >> >> ------------------------------------------------
>> > >> >>
>> > >> >>
>> > >> >> Raqhav
>> > >> >>
>> > >> >
>> > >> >
>> > >>
>> > >>
>> > >> --
>> > >> Raghav
>> > >>
>> > >
>> > >
>> >
>> >
>> > --
>> > Raghav
>> >
>>
>
>
>
> --
> Raghav
>
Re: Kafka Authorization and ACLs Broken
Posted by Raghav <ra...@gmail.com>.
I initially tried kerberos, but it felt too complicated, so gave up and
only tried SSL.
On Wed, May 24, 2017 at 7:47 PM, Mike Marzo <pr...@gmail.com>
wrote:
> Thanks. We will try it. Struggling with krb5 and acls
>
> mike marzo
> 908 209-4484 <(908)%20209-4484>
>
> On May 24, 2017 9:29 PM, "Raghav" <ra...@gmail.com> wrote:
>
>> Mike
>>
>> I am not using jaas file. I literally took the config Rajini gave in the
>> previous email and it worked for me. I am using ssl Kafka with ACLs. I am
>> not suing kerberos.
>>
>> Thanks.
>>
>> On Wed, May 24, 2017 at 11:29 AM, Mike Marzo <
>> precisionarcheryllc@gmail.com> wrote:
>>
>>> I'm also having issues getting acls to work. Out of intereat, are you
>>> starting ur brokers with a jaas file, if so do u mind sharing the client
>>> and server side jaas entries so I can validate what I'm doing.
>>>
>>> mike marzo
>>> 908 209-4484
>>>
>>> On May 24, 2017 10:54 AM, "Raghav" <ra...@gmail.com> wrote:
>>>
>>> > Hi Rajini
>>> >
>>> > Thank you very much. It perfectly works.
>>> >
>>> > I think in my setup I was trying to use a CA (certificate authority) to
>>> > sign the certificates from client and server, and then adding it to
>>> trust
>>> > store and keystore. I think in that process, I may have messed
>>> something. I
>>> > will try above config with a CA to sign certificates. Hopefully that
>>> would
>>> > work too.
>>> >
>>> > Thanks a lot again.
>>> >
>>> > Raghav
>>> >
>>> >
>>> >
>>> >
>>> > On Wed, May 24, 2017 at 7:04 AM, Rajini Sivaram <
>>> rajinisivaram@gmail.com>
>>> > wrote:
>>> >
>>> > > Raghav/Darshan,
>>> > >
>>> > > Can you try these steps on a clean installation of Kafka? It works
>>> for
>>> > me,
>>> > > so hopefully it will work for you. And then you can adapt to your
>>> > scenario.
>>> > >
>>> > > *Create keystores and truststores:*
>>> > >
>>> > > keytool -genkey -alias kafka -keystore server.keystore.jks -dname
>>> > > "CN=KafkaBroker,O=Pivotal,C=UK" -storepass server-keystore-password
>>> > > -keypass server-key-password
>>> > >
>>> > > keytool -exportcert -file server-cert-file -keystore
>>> server.keystore.jks
>>> > > -alias kafka -storepass server-keystore-password
>>> > >
>>> > > keytool -importcert -file server-cert-file -keystore
>>> > server.truststore.jks
>>> > > -alias kafka -storepass server-truststore-password -noprompt
>>> > >
>>> > > keytool -importcert -file server-cert-file -keystore
>>> > client.truststore.jks
>>> > > -alias kafkaclient -storepass client-truststore-password -noprompt
>>> > >
>>> > >
>>> > > keytool -genkey -alias kafkaclient -keystore client.keystore.jks
>>> -dname
>>> > > "CN=KafkaClient,O=Pivotal,C=UK" -storepass client-keystore-password
>>> > > -keypass client-key-password
>>> > >
>>> > > keytool -exportcert -file client-cert-file -keystore
>>> client.keystore.jks
>>> > > -alias kafkaclient -storepass client-keystore-password
>>> > >
>>> > > keytool -importcert -file client-cert-file -keystore
>>> > server.truststore.jks
>>> > > -alias kafkaclient -storepass server-truststore-password -noprompt
>>> > >
>>> > > *Configure broker: Add these lines at the end of your
>>> server.properties*
>>> > >
>>> > > listeners=SSL://:9093
>>> > >
>>> > > advertised.listeners=SSL://127.0.0.1:9093
>>> > >
>>> > > ssl.keystore.location=/tmp/acl/server.keystore.jks
>>> > >
>>> > > ssl.keystore.password=server-keystore-password
>>> > >
>>> > > ssl.key.password=server-key-password
>>> > >
>>> > > ssl.truststore.location=/tmp/acl/server.truststore.jks
>>> > >
>>> > > ssl.truststore.password=server-truststore-password
>>> > >
>>> > > security.inter.broker.protocol=SSL
>>> > >
>>> > > security.protocol=SSL
>>> > >
>>> > > ssl.client.auth=required
>>> > >
>>> > > allow.everyone.if.no.acl.found=false
>>> > >
>>> > > authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
>>> > >
>>> > > super.users=User:CN=KafkaBroker,O=Pivotal,C=UK
>>> > >
>>> > > *Configure producer: producer.properties*
>>> > >
>>> > > security.protocol=SSL
>>> > >
>>> > > ssl.truststore.location=/tmp/acl/client.truststore.jks
>>> > >
>>> > > ssl.truststore.password=client-truststore-password
>>> > >
>>> > > ssl.keystore.location=/tmp/acl/client.keystore.jks
>>> > >
>>> > > ssl.keystore.password=client-keystore-password
>>> > >
>>> > > ssl.key.password=client-key-password
>>> > >
>>> > >
>>> > > *Configure consumer: consumer.properties*
>>> > >
>>> > > security.protocol=SSL
>>> > >
>>> > > ssl.truststore.location=/tmp/acl/client.truststore.jks
>>> > >
>>> > > ssl.truststore.password=client-truststore-password
>>> > >
>>> > > ssl.keystore.location=/tmp/acl/client.keystore.jks
>>> > >
>>> > > ssl.keystore.password=client-keystore-password
>>> > >
>>> > > ssl.key.password=client-key-password
>>> > >
>>> > > group.id=testgroup
>>> > >
>>> > > *Create topic:*
>>> > >
>>> > > bin/kafka-topics.sh --zookeeper localhost --create --topic testtopic
>>> > > --replication-factor 1 --partitions 1
>>> > >
>>> > >
>>> > > *Configure ACLs:*
>>> > >
>>> > > bin/kafka-acls.sh --authorizer-properties
>>> zookeeper.connect=localhost:
>>> > 2181
>>> > > --add --allow-principal "User:CN=KafkaClient,O=Pivotal,C=UK"
>>> --producer
>>> > > --topic testtopic
>>> > >
>>> > > bin/kafka-acls.sh --authorizer-properties
>>> zookeeper.connect=localhost:
>>> > 2181
>>> > > --add --allow-principal "User:CN=KafkaClient,O=Pivotal,C=UK"
>>> --consumer
>>> > > --topic testtopic --group test group
>>> > >
>>> > >
>>> > > *Run console producer and type in some messages:*
>>> > >
>>> > > bin/kafka-console-producer.sh --producer.config
>>> > > /tmp/acl/producer.properties --topic testtopic --broker-list
>>> > > 127.0.0.1:9093
>>> > >
>>> > >
>>> > > *Run console consumer, you should see messages from above:*
>>> > >
>>> > > bin/kafka-console-consumer.sh --consumer.config
>>> > > /tmp/acl/consumer.properties --topic testtopic --bootstrap-server
>>> > > 127.0.0.1:9093 --from-beginning
>>> > >
>>> > >
>>> > >
>>> > > On Tue, May 23, 2017 at 12:57 PM, Raghav <ra...@gmail.com>
>>> wrote:
>>> > >
>>> > >> Darshan,
>>> > >>
>>> > >> I have not yet successfully gotten the ACLs to work in Kafka. I am
>>> still
>>> > >> looking for help. I will update this email thread if I do find. In
>>> case
>>> > >> you
>>> > >> get it working, please let me know.
>>> > >>
>>> > >> Thanks.
>>> > >>
>>> > >> R
>>> > >>
>>> > >> On Tue, May 23, 2017 at 8:49 AM, Darshan Purandare <
>>> > >> purandare.darshan@gmail.com> wrote:
>>> > >>
>>> > >> > Raghav
>>> > >> >
>>> > >> > I saw few posts of yours around Kafka ACLs and the problems. I
>>> have
>>> > seen
>>> > >> > similar issues where Writer has not been able to write to any
>>> topic. I
>>> > >> have
>>> > >> > seen "leader not available" and sometimes "unknown topic or
>>> > partition",
>>> > >> and
>>> > >> > "topic_authorization_failed" error.
>>> > >> >
>>> > >> > Let me know if you find a valid config that works.
>>> > >> >
>>> > >> > Thanks.
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> > On Tue, May 23, 2017 at 8:44 AM, Raghav <ra...@gmail.com>
>>> > wrote:
>>> > >> >
>>> > >> >> Hello Kafka Users
>>> > >> >>
>>> > >> >> I am a new Kafka user and trying to make Kafka SSL work with
>>> > >> Authorization
>>> > >> >> and ACLs. I followed posts from Kafka and Confluent docs exactly
>>> to
>>> > the
>>> > >> >> point but my producer cannot write to kafka broker. I get
>>> > >> >> "LEADER_NOT_FOUND" errors. And even Consumer throws the same
>>> errors.
>>> > >> >>
>>> > >> >> Can someone please share their config which worked with ACLs.
>>> > >> >>
>>> > >> >> Here is my config. Please help.
>>> > >> >>
>>> > >> >> server.properties config
>>> > >> >> ------------------------------------------------------------
>>> > >> >> ------------------------------------------------
>>> > >> >> broker.id=0
>>> > >> >> auto.create.topics.enable=true
>>> > >> >> delete.topic.enable=true
>>> > >> >>
>>> > >> >> listeners=PLAINTEXT://kafka1.example.com:9092
>>> > >> >> <http://kafka-dev1.example.com:9092/>,SSL://kafka1.example.c
>>> om:9093
>>> > >> >> <http://kafka-dev1.example.com:9093/>
>>> > >> >> host.name=kafka1.example.com <http://kafka-dev1.example.com/>
>>> > >> >>
>>> > >> >>
>>> > >> >>
>>> > >> >> ssl.keystore.location=/var/private/kafka1.keystore.jks
>>> > >> >> ssl.keystore.password=12345678
>>> > >> >> ssl.key.password=12345678
>>> > >> >>
>>> > >> >> ssl.truststore.location=/var/private/kafka1.truststore.jks
>>> > >> >> ssl.truststore.password=12345678
>>> > >> >>
>>> > >> >> ssl.client.auth=required
>>> > >> >> ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
>>> > >> >> ssl.keystore.type=JKS
>>> > >> >> ssl.truststore.type=JKS
>>> > >> >>
>>> > >> >> authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
>>> > >> >> ------------------------------------------------------------
>>> > >> >> ------------------------------------------------
>>> > >> >>
>>> > >> >>
>>> > >> >>
>>> > >> >> Here is producer Config(producer.properties)
>>> > >> >> ------------------------------------------------------------
>>> > >> >> ------------------------------------------------
>>> > >> >> security.protocol=SSL
>>> > >> >> ssl.truststore.location=/var/private/kafka2.truststore.jks
>>> > >> >> ssl.truststore.password=12345678
>>> > >> >>
>>> > >> >> ssl.keystore.location=/var/private/kafka2.keystore.jks
>>> > >> >> ssl.keystore.password=12345678
>>> > >> >> ssl.key.password=12345678
>>> > >> >>
>>> > >> >> ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
>>> > >> >> ssl.truststore.type=JKS
>>> > >> >> ssl.keystore.type=JKS
>>> > >> >>
>>> > >> >> ------------------------------------------------------------
>>> > >> >> ------------------------------------------------
>>> > >> >>
>>> > >> >>
>>> > >> >> Raqhav
>>> > >> >>
>>> > >> >
>>> > >> >
>>> > >>
>>> > >>
>>> > >> --
>>> > >> Raghav
>>> > >>
>>> > >
>>> > >
>>> >
>>> >
>>> > --
>>> > Raghav
>>> >
>>>
>>
>>
>>
>> --
>> Raghav
>>
>
--
Raghav
Re: Kafka Authorization and ACLs Broken
Posted by Mike Marzo <pr...@gmail.com>.
Thanks. We will try it. Struggling with krb5 and acls
mike marzo
908 209-4484
On May 24, 2017 9:29 PM, "Raghav" <ra...@gmail.com> wrote:
> Mike
>
> I am not using jaas file. I literally took the config Rajini gave in the
> previous email and it worked for me. I am using ssl Kafka with ACLs. I am
> not suing kerberos.
>
> Thanks.
>
> On Wed, May 24, 2017 at 11:29 AM, Mike Marzo <
> precisionarcheryllc@gmail.com> wrote:
>
>> I'm also having issues getting acls to work. Out of intereat, are you
>> starting ur brokers with a jaas file, if so do u mind sharing the client
>> and server side jaas entries so I can validate what I'm doing.
>>
>> mike marzo
>> 908 209-4484
>>
>> On May 24, 2017 10:54 AM, "Raghav" <ra...@gmail.com> wrote:
>>
>> > Hi Rajini
>> >
>> > Thank you very much. It perfectly works.
>> >
>> > I think in my setup I was trying to use a CA (certificate authority) to
>> > sign the certificates from client and server, and then adding it to
>> trust
>> > store and keystore. I think in that process, I may have messed
>> something. I
>> > will try above config with a CA to sign certificates. Hopefully that
>> would
>> > work too.
>> >
>> > Thanks a lot again.
>> >
>> > Raghav
>> >
>> >
>> >
>> >
>> > On Wed, May 24, 2017 at 7:04 AM, Rajini Sivaram <
>> rajinisivaram@gmail.com>
>> > wrote:
>> >
>> > > Raghav/Darshan,
>> > >
>> > > Can you try these steps on a clean installation of Kafka? It works for
>> > me,
>> > > so hopefully it will work for you. And then you can adapt to your
>> > scenario.
>> > >
>> > > *Create keystores and truststores:*
>> > >
>> > > keytool -genkey -alias kafka -keystore server.keystore.jks -dname
>> > > "CN=KafkaBroker,O=Pivotal,C=UK" -storepass server-keystore-password
>> > > -keypass server-key-password
>> > >
>> > > keytool -exportcert -file server-cert-file -keystore
>> server.keystore.jks
>> > > -alias kafka -storepass server-keystore-password
>> > >
>> > > keytool -importcert -file server-cert-file -keystore
>> > server.truststore.jks
>> > > -alias kafka -storepass server-truststore-password -noprompt
>> > >
>> > > keytool -importcert -file server-cert-file -keystore
>> > client.truststore.jks
>> > > -alias kafkaclient -storepass client-truststore-password -noprompt
>> > >
>> > >
>> > > keytool -genkey -alias kafkaclient -keystore client.keystore.jks
>> -dname
>> > > "CN=KafkaClient,O=Pivotal,C=UK" -storepass client-keystore-password
>> > > -keypass client-key-password
>> > >
>> > > keytool -exportcert -file client-cert-file -keystore
>> client.keystore.jks
>> > > -alias kafkaclient -storepass client-keystore-password
>> > >
>> > > keytool -importcert -file client-cert-file -keystore
>> > server.truststore.jks
>> > > -alias kafkaclient -storepass server-truststore-password -noprompt
>> > >
>> > > *Configure broker: Add these lines at the end of your
>> server.properties*
>> > >
>> > > listeners=SSL://:9093
>> > >
>> > > advertised.listeners=SSL://127.0.0.1:9093
>> > >
>> > > ssl.keystore.location=/tmp/acl/server.keystore.jks
>> > >
>> > > ssl.keystore.password=server-keystore-password
>> > >
>> > > ssl.key.password=server-key-password
>> > >
>> > > ssl.truststore.location=/tmp/acl/server.truststore.jks
>> > >
>> > > ssl.truststore.password=server-truststore-password
>> > >
>> > > security.inter.broker.protocol=SSL
>> > >
>> > > security.protocol=SSL
>> > >
>> > > ssl.client.auth=required
>> > >
>> > > allow.everyone.if.no.acl.found=false
>> > >
>> > > authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
>> > >
>> > > super.users=User:CN=KafkaBroker,O=Pivotal,C=UK
>> > >
>> > > *Configure producer: producer.properties*
>> > >
>> > > security.protocol=SSL
>> > >
>> > > ssl.truststore.location=/tmp/acl/client.truststore.jks
>> > >
>> > > ssl.truststore.password=client-truststore-password
>> > >
>> > > ssl.keystore.location=/tmp/acl/client.keystore.jks
>> > >
>> > > ssl.keystore.password=client-keystore-password
>> > >
>> > > ssl.key.password=client-key-password
>> > >
>> > >
>> > > *Configure consumer: consumer.properties*
>> > >
>> > > security.protocol=SSL
>> > >
>> > > ssl.truststore.location=/tmp/acl/client.truststore.jks
>> > >
>> > > ssl.truststore.password=client-truststore-password
>> > >
>> > > ssl.keystore.location=/tmp/acl/client.keystore.jks
>> > >
>> > > ssl.keystore.password=client-keystore-password
>> > >
>> > > ssl.key.password=client-key-password
>> > >
>> > > group.id=testgroup
>> > >
>> > > *Create topic:*
>> > >
>> > > bin/kafka-topics.sh --zookeeper localhost --create --topic testtopic
>> > > --replication-factor 1 --partitions 1
>> > >
>> > >
>> > > *Configure ACLs:*
>> > >
>> > > bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:
>> > 2181
>> > > --add --allow-principal "User:CN=KafkaClient,O=Pivotal,C=UK"
>> --producer
>> > > --topic testtopic
>> > >
>> > > bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:
>> > 2181
>> > > --add --allow-principal "User:CN=KafkaClient,O=Pivotal,C=UK"
>> --consumer
>> > > --topic testtopic --group test group
>> > >
>> > >
>> > > *Run console producer and type in some messages:*
>> > >
>> > > bin/kafka-console-producer.sh --producer.config
>> > > /tmp/acl/producer.properties --topic testtopic --broker-list
>> > > 127.0.0.1:9093
>> > >
>> > >
>> > > *Run console consumer, you should see messages from above:*
>> > >
>> > > bin/kafka-console-consumer.sh --consumer.config
>> > > /tmp/acl/consumer.properties --topic testtopic --bootstrap-server
>> > > 127.0.0.1:9093 --from-beginning
>> > >
>> > >
>> > >
>> > > On Tue, May 23, 2017 at 12:57 PM, Raghav <ra...@gmail.com>
>> wrote:
>> > >
>> > >> Darshan,
>> > >>
>> > >> I have not yet successfully gotten the ACLs to work in Kafka. I am
>> still
>> > >> looking for help. I will update this email thread if I do find. In
>> case
>> > >> you
>> > >> get it working, please let me know.
>> > >>
>> > >> Thanks.
>> > >>
>> > >> R
>> > >>
>> > >> On Tue, May 23, 2017 at 8:49 AM, Darshan Purandare <
>> > >> purandare.darshan@gmail.com> wrote:
>> > >>
>> > >> > Raghav
>> > >> >
>> > >> > I saw few posts of yours around Kafka ACLs and the problems. I have
>> > seen
>> > >> > similar issues where Writer has not been able to write to any
>> topic. I
>> > >> have
>> > >> > seen "leader not available" and sometimes "unknown topic or
>> > partition",
>> > >> and
>> > >> > "topic_authorization_failed" error.
>> > >> >
>> > >> > Let me know if you find a valid config that works.
>> > >> >
>> > >> > Thanks.
>> > >> >
>> > >> >
>> > >> >
>> > >> > On Tue, May 23, 2017 at 8:44 AM, Raghav <ra...@gmail.com>
>> > wrote:
>> > >> >
>> > >> >> Hello Kafka Users
>> > >> >>
>> > >> >> I am a new Kafka user and trying to make Kafka SSL work with
>> > >> Authorization
>> > >> >> and ACLs. I followed posts from Kafka and Confluent docs exactly
>> to
>> > the
>> > >> >> point but my producer cannot write to kafka broker. I get
>> > >> >> "LEADER_NOT_FOUND" errors. And even Consumer throws the same
>> errors.
>> > >> >>
>> > >> >> Can someone please share their config which worked with ACLs.
>> > >> >>
>> > >> >> Here is my config. Please help.
>> > >> >>
>> > >> >> server.properties config
>> > >> >> ------------------------------------------------------------
>> > >> >> ------------------------------------------------
>> > >> >> broker.id=0
>> > >> >> auto.create.topics.enable=true
>> > >> >> delete.topic.enable=true
>> > >> >>
>> > >> >> listeners=PLAINTEXT://kafka1.example.com:9092
>> > >> >> <http://kafka-dev1.example.com:9092/>,SSL://kafka1.example.
>> com:9093
>> > >> >> <http://kafka-dev1.example.com:9093/>
>> > >> >> host.name=kafka1.example.com <http://kafka-dev1.example.com/>
>> > >> >>
>> > >> >>
>> > >> >>
>> > >> >> ssl.keystore.location=/var/private/kafka1.keystore.jks
>> > >> >> ssl.keystore.password=12345678
>> > >> >> ssl.key.password=12345678
>> > >> >>
>> > >> >> ssl.truststore.location=/var/private/kafka1.truststore.jks
>> > >> >> ssl.truststore.password=12345678
>> > >> >>
>> > >> >> ssl.client.auth=required
>> > >> >> ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
>> > >> >> ssl.keystore.type=JKS
>> > >> >> ssl.truststore.type=JKS
>> > >> >>
>> > >> >> authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
>> > >> >> ------------------------------------------------------------
>> > >> >> ------------------------------------------------
>> > >> >>
>> > >> >>
>> > >> >>
>> > >> >> Here is producer Config(producer.properties)
>> > >> >> ------------------------------------------------------------
>> > >> >> ------------------------------------------------
>> > >> >> security.protocol=SSL
>> > >> >> ssl.truststore.location=/var/private/kafka2.truststore.jks
>> > >> >> ssl.truststore.password=12345678
>> > >> >>
>> > >> >> ssl.keystore.location=/var/private/kafka2.keystore.jks
>> > >> >> ssl.keystore.password=12345678
>> > >> >> ssl.key.password=12345678
>> > >> >>
>> > >> >> ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
>> > >> >> ssl.truststore.type=JKS
>> > >> >> ssl.keystore.type=JKS
>> > >> >>
>> > >> >> ------------------------------------------------------------
>> > >> >> ------------------------------------------------
>> > >> >>
>> > >> >>
>> > >> >> Raqhav
>> > >> >>
>> > >> >
>> > >> >
>> > >>
>> > >>
>> > >> --
>> > >> Raghav
>> > >>
>> > >
>> > >
>> >
>> >
>> > --
>> > Raghav
>> >
>>
>
>
>
> --
> Raghav
>
Re: Kafka Authorization and ACLs Broken
Posted by Raghav <ra...@gmail.com>.
Mike
I am not using jaas file. I literally took the config Rajini gave in the
previous email and it worked for me. I am using ssl Kafka with ACLs. I am
not suing kerberos.
Thanks.
On Wed, May 24, 2017 at 11:29 AM, Mike Marzo <pr...@gmail.com>
wrote:
> I'm also having issues getting acls to work. Out of intereat, are you
> starting ur brokers with a jaas file, if so do u mind sharing the client
> and server side jaas entries so I can validate what I'm doing.
>
> mike marzo
> 908 209-4484
>
> On May 24, 2017 10:54 AM, "Raghav" <ra...@gmail.com> wrote:
>
> > Hi Rajini
> >
> > Thank you very much. It perfectly works.
> >
> > I think in my setup I was trying to use a CA (certificate authority) to
> > sign the certificates from client and server, and then adding it to trust
> > store and keystore. I think in that process, I may have messed
> something. I
> > will try above config with a CA to sign certificates. Hopefully that
> would
> > work too.
> >
> > Thanks a lot again.
> >
> > Raghav
> >
> >
> >
> >
> > On Wed, May 24, 2017 at 7:04 AM, Rajini Sivaram <rajinisivaram@gmail.com
> >
> > wrote:
> >
> > > Raghav/Darshan,
> > >
> > > Can you try these steps on a clean installation of Kafka? It works for
> > me,
> > > so hopefully it will work for you. And then you can adapt to your
> > scenario.
> > >
> > > *Create keystores and truststores:*
> > >
> > > keytool -genkey -alias kafka -keystore server.keystore.jks -dname
> > > "CN=KafkaBroker,O=Pivotal,C=UK" -storepass server-keystore-password
> > > -keypass server-key-password
> > >
> > > keytool -exportcert -file server-cert-file -keystore
> server.keystore.jks
> > > -alias kafka -storepass server-keystore-password
> > >
> > > keytool -importcert -file server-cert-file -keystore
> > server.truststore.jks
> > > -alias kafka -storepass server-truststore-password -noprompt
> > >
> > > keytool -importcert -file server-cert-file -keystore
> > client.truststore.jks
> > > -alias kafkaclient -storepass client-truststore-password -noprompt
> > >
> > >
> > > keytool -genkey -alias kafkaclient -keystore client.keystore.jks -dname
> > > "CN=KafkaClient,O=Pivotal,C=UK" -storepass client-keystore-password
> > > -keypass client-key-password
> > >
> > > keytool -exportcert -file client-cert-file -keystore
> client.keystore.jks
> > > -alias kafkaclient -storepass client-keystore-password
> > >
> > > keytool -importcert -file client-cert-file -keystore
> > server.truststore.jks
> > > -alias kafkaclient -storepass server-truststore-password -noprompt
> > >
> > > *Configure broker: Add these lines at the end of your
> server.properties*
> > >
> > > listeners=SSL://:9093
> > >
> > > advertised.listeners=SSL://127.0.0.1:9093
> > >
> > > ssl.keystore.location=/tmp/acl/server.keystore.jks
> > >
> > > ssl.keystore.password=server-keystore-password
> > >
> > > ssl.key.password=server-key-password
> > >
> > > ssl.truststore.location=/tmp/acl/server.truststore.jks
> > >
> > > ssl.truststore.password=server-truststore-password
> > >
> > > security.inter.broker.protocol=SSL
> > >
> > > security.protocol=SSL
> > >
> > > ssl.client.auth=required
> > >
> > > allow.everyone.if.no.acl.found=false
> > >
> > > authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
> > >
> > > super.users=User:CN=KafkaBroker,O=Pivotal,C=UK
> > >
> > > *Configure producer: producer.properties*
> > >
> > > security.protocol=SSL
> > >
> > > ssl.truststore.location=/tmp/acl/client.truststore.jks
> > >
> > > ssl.truststore.password=client-truststore-password
> > >
> > > ssl.keystore.location=/tmp/acl/client.keystore.jks
> > >
> > > ssl.keystore.password=client-keystore-password
> > >
> > > ssl.key.password=client-key-password
> > >
> > >
> > > *Configure consumer: consumer.properties*
> > >
> > > security.protocol=SSL
> > >
> > > ssl.truststore.location=/tmp/acl/client.truststore.jks
> > >
> > > ssl.truststore.password=client-truststore-password
> > >
> > > ssl.keystore.location=/tmp/acl/client.keystore.jks
> > >
> > > ssl.keystore.password=client-keystore-password
> > >
> > > ssl.key.password=client-key-password
> > >
> > > group.id=testgroup
> > >
> > > *Create topic:*
> > >
> > > bin/kafka-topics.sh --zookeeper localhost --create --topic testtopic
> > > --replication-factor 1 --partitions 1
> > >
> > >
> > > *Configure ACLs:*
> > >
> > > bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:
> > 2181
> > > --add --allow-principal "User:CN=KafkaClient,O=Pivotal,C=UK"
> --producer
> > > --topic testtopic
> > >
> > > bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:
> > 2181
> > > --add --allow-principal "User:CN=KafkaClient,O=Pivotal,C=UK"
> --consumer
> > > --topic testtopic --group test group
> > >
> > >
> > > *Run console producer and type in some messages:*
> > >
> > > bin/kafka-console-producer.sh --producer.config
> > > /tmp/acl/producer.properties --topic testtopic --broker-list
> > > 127.0.0.1:9093
> > >
> > >
> > > *Run console consumer, you should see messages from above:*
> > >
> > > bin/kafka-console-consumer.sh --consumer.config
> > > /tmp/acl/consumer.properties --topic testtopic --bootstrap-server
> > > 127.0.0.1:9093 --from-beginning
> > >
> > >
> > >
> > > On Tue, May 23, 2017 at 12:57 PM, Raghav <ra...@gmail.com>
> wrote:
> > >
> > >> Darshan,
> > >>
> > >> I have not yet successfully gotten the ACLs to work in Kafka. I am
> still
> > >> looking for help. I will update this email thread if I do find. In
> case
> > >> you
> > >> get it working, please let me know.
> > >>
> > >> Thanks.
> > >>
> > >> R
> > >>
> > >> On Tue, May 23, 2017 at 8:49 AM, Darshan Purandare <
> > >> purandare.darshan@gmail.com> wrote:
> > >>
> > >> > Raghav
> > >> >
> > >> > I saw few posts of yours around Kafka ACLs and the problems. I have
> > seen
> > >> > similar issues where Writer has not been able to write to any
> topic. I
> > >> have
> > >> > seen "leader not available" and sometimes "unknown topic or
> > partition",
> > >> and
> > >> > "topic_authorization_failed" error.
> > >> >
> > >> > Let me know if you find a valid config that works.
> > >> >
> > >> > Thanks.
> > >> >
> > >> >
> > >> >
> > >> > On Tue, May 23, 2017 at 8:44 AM, Raghav <ra...@gmail.com>
> > wrote:
> > >> >
> > >> >> Hello Kafka Users
> > >> >>
> > >> >> I am a new Kafka user and trying to make Kafka SSL work with
> > >> Authorization
> > >> >> and ACLs. I followed posts from Kafka and Confluent docs exactly to
> > the
> > >> >> point but my producer cannot write to kafka broker. I get
> > >> >> "LEADER_NOT_FOUND" errors. And even Consumer throws the same
> errors.
> > >> >>
> > >> >> Can someone please share their config which worked with ACLs.
> > >> >>
> > >> >> Here is my config. Please help.
> > >> >>
> > >> >> server.properties config
> > >> >> ------------------------------------------------------------
> > >> >> ------------------------------------------------
> > >> >> broker.id=0
> > >> >> auto.create.topics.enable=true
> > >> >> delete.topic.enable=true
> > >> >>
> > >> >> listeners=PLAINTEXT://kafka1.example.com:9092
> > >> >> <http://kafka-dev1.example.com:9092/>,SSL://kafka1.
> example.com:9093
> > >> >> <http://kafka-dev1.example.com:9093/>
> > >> >> host.name=kafka1.example.com <http://kafka-dev1.example.com/>
> > >> >>
> > >> >>
> > >> >>
> > >> >> ssl.keystore.location=/var/private/kafka1.keystore.jks
> > >> >> ssl.keystore.password=12345678
> > >> >> ssl.key.password=12345678
> > >> >>
> > >> >> ssl.truststore.location=/var/private/kafka1.truststore.jks
> > >> >> ssl.truststore.password=12345678
> > >> >>
> > >> >> ssl.client.auth=required
> > >> >> ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
> > >> >> ssl.keystore.type=JKS
> > >> >> ssl.truststore.type=JKS
> > >> >>
> > >> >> authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
> > >> >> ------------------------------------------------------------
> > >> >> ------------------------------------------------
> > >> >>
> > >> >>
> > >> >>
> > >> >> Here is producer Config(producer.properties)
> > >> >> ------------------------------------------------------------
> > >> >> ------------------------------------------------
> > >> >> security.protocol=SSL
> > >> >> ssl.truststore.location=/var/private/kafka2.truststore.jks
> > >> >> ssl.truststore.password=12345678
> > >> >>
> > >> >> ssl.keystore.location=/var/private/kafka2.keystore.jks
> > >> >> ssl.keystore.password=12345678
> > >> >> ssl.key.password=12345678
> > >> >>
> > >> >> ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
> > >> >> ssl.truststore.type=JKS
> > >> >> ssl.keystore.type=JKS
> > >> >>
> > >> >> ------------------------------------------------------------
> > >> >> ------------------------------------------------
> > >> >>
> > >> >>
> > >> >> Raqhav
> > >> >>
> > >> >
> > >> >
> > >>
> > >>
> > >> --
> > >> Raghav
> > >>
> > >
> > >
> >
> >
> > --
> > Raghav
> >
>
--
Raghav
Re: Kafka Authorization and ACLs Broken
Posted by Mike Marzo <pr...@gmail.com>.
I'm also having issues getting acls to work. Out of intereat, are you
starting ur brokers with a jaas file, if so do u mind sharing the client
and server side jaas entries so I can validate what I'm doing.
mike marzo
908 209-4484
On May 24, 2017 10:54 AM, "Raghav" <ra...@gmail.com> wrote:
> Hi Rajini
>
> Thank you very much. It perfectly works.
>
> I think in my setup I was trying to use a CA (certificate authority) to
> sign the certificates from client and server, and then adding it to trust
> store and keystore. I think in that process, I may have messed something. I
> will try above config with a CA to sign certificates. Hopefully that would
> work too.
>
> Thanks a lot again.
>
> Raghav
>
>
>
>
> On Wed, May 24, 2017 at 7:04 AM, Rajini Sivaram <ra...@gmail.com>
> wrote:
>
> > Raghav/Darshan,
> >
> > Can you try these steps on a clean installation of Kafka? It works for
> me,
> > so hopefully it will work for you. And then you can adapt to your
> scenario.
> >
> > *Create keystores and truststores:*
> >
> > keytool -genkey -alias kafka -keystore server.keystore.jks -dname
> > "CN=KafkaBroker,O=Pivotal,C=UK" -storepass server-keystore-password
> > -keypass server-key-password
> >
> > keytool -exportcert -file server-cert-file -keystore server.keystore.jks
> > -alias kafka -storepass server-keystore-password
> >
> > keytool -importcert -file server-cert-file -keystore
> server.truststore.jks
> > -alias kafka -storepass server-truststore-password -noprompt
> >
> > keytool -importcert -file server-cert-file -keystore
> client.truststore.jks
> > -alias kafkaclient -storepass client-truststore-password -noprompt
> >
> >
> > keytool -genkey -alias kafkaclient -keystore client.keystore.jks -dname
> > "CN=KafkaClient,O=Pivotal,C=UK" -storepass client-keystore-password
> > -keypass client-key-password
> >
> > keytool -exportcert -file client-cert-file -keystore client.keystore.jks
> > -alias kafkaclient -storepass client-keystore-password
> >
> > keytool -importcert -file client-cert-file -keystore
> server.truststore.jks
> > -alias kafkaclient -storepass server-truststore-password -noprompt
> >
> > *Configure broker: Add these lines at the end of your server.properties*
> >
> > listeners=SSL://:9093
> >
> > advertised.listeners=SSL://127.0.0.1:9093
> >
> > ssl.keystore.location=/tmp/acl/server.keystore.jks
> >
> > ssl.keystore.password=server-keystore-password
> >
> > ssl.key.password=server-key-password
> >
> > ssl.truststore.location=/tmp/acl/server.truststore.jks
> >
> > ssl.truststore.password=server-truststore-password
> >
> > security.inter.broker.protocol=SSL
> >
> > security.protocol=SSL
> >
> > ssl.client.auth=required
> >
> > allow.everyone.if.no.acl.found=false
> >
> > authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
> >
> > super.users=User:CN=KafkaBroker,O=Pivotal,C=UK
> >
> > *Configure producer: producer.properties*
> >
> > security.protocol=SSL
> >
> > ssl.truststore.location=/tmp/acl/client.truststore.jks
> >
> > ssl.truststore.password=client-truststore-password
> >
> > ssl.keystore.location=/tmp/acl/client.keystore.jks
> >
> > ssl.keystore.password=client-keystore-password
> >
> > ssl.key.password=client-key-password
> >
> >
> > *Configure consumer: consumer.properties*
> >
> > security.protocol=SSL
> >
> > ssl.truststore.location=/tmp/acl/client.truststore.jks
> >
> > ssl.truststore.password=client-truststore-password
> >
> > ssl.keystore.location=/tmp/acl/client.keystore.jks
> >
> > ssl.keystore.password=client-keystore-password
> >
> > ssl.key.password=client-key-password
> >
> > group.id=testgroup
> >
> > *Create topic:*
> >
> > bin/kafka-topics.sh --zookeeper localhost --create --topic testtopic
> > --replication-factor 1 --partitions 1
> >
> >
> > *Configure ACLs:*
> >
> > bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:
> 2181
> > --add --allow-principal "User:CN=KafkaClient,O=Pivotal,C=UK" --producer
> > --topic testtopic
> >
> > bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:
> 2181
> > --add --allow-principal "User:CN=KafkaClient,O=Pivotal,C=UK" --consumer
> > --topic testtopic --group test group
> >
> >
> > *Run console producer and type in some messages:*
> >
> > bin/kafka-console-producer.sh --producer.config
> > /tmp/acl/producer.properties --topic testtopic --broker-list
> > 127.0.0.1:9093
> >
> >
> > *Run console consumer, you should see messages from above:*
> >
> > bin/kafka-console-consumer.sh --consumer.config
> > /tmp/acl/consumer.properties --topic testtopic --bootstrap-server
> > 127.0.0.1:9093 --from-beginning
> >
> >
> >
> > On Tue, May 23, 2017 at 12:57 PM, Raghav <ra...@gmail.com> wrote:
> >
> >> Darshan,
> >>
> >> I have not yet successfully gotten the ACLs to work in Kafka. I am still
> >> looking for help. I will update this email thread if I do find. In case
> >> you
> >> get it working, please let me know.
> >>
> >> Thanks.
> >>
> >> R
> >>
> >> On Tue, May 23, 2017 at 8:49 AM, Darshan Purandare <
> >> purandare.darshan@gmail.com> wrote:
> >>
> >> > Raghav
> >> >
> >> > I saw few posts of yours around Kafka ACLs and the problems. I have
> seen
> >> > similar issues where Writer has not been able to write to any topic. I
> >> have
> >> > seen "leader not available" and sometimes "unknown topic or
> partition",
> >> and
> >> > "topic_authorization_failed" error.
> >> >
> >> > Let me know if you find a valid config that works.
> >> >
> >> > Thanks.
> >> >
> >> >
> >> >
> >> > On Tue, May 23, 2017 at 8:44 AM, Raghav <ra...@gmail.com>
> wrote:
> >> >
> >> >> Hello Kafka Users
> >> >>
> >> >> I am a new Kafka user and trying to make Kafka SSL work with
> >> Authorization
> >> >> and ACLs. I followed posts from Kafka and Confluent docs exactly to
> the
> >> >> point but my producer cannot write to kafka broker. I get
> >> >> "LEADER_NOT_FOUND" errors. And even Consumer throws the same errors.
> >> >>
> >> >> Can someone please share their config which worked with ACLs.
> >> >>
> >> >> Here is my config. Please help.
> >> >>
> >> >> server.properties config
> >> >> ------------------------------------------------------------
> >> >> ------------------------------------------------
> >> >> broker.id=0
> >> >> auto.create.topics.enable=true
> >> >> delete.topic.enable=true
> >> >>
> >> >> listeners=PLAINTEXT://kafka1.example.com:9092
> >> >> <http://kafka-dev1.example.com:9092/>,SSL://kafka1.example.com:9093
> >> >> <http://kafka-dev1.example.com:9093/>
> >> >> host.name=kafka1.example.com <http://kafka-dev1.example.com/>
> >> >>
> >> >>
> >> >>
> >> >> ssl.keystore.location=/var/private/kafka1.keystore.jks
> >> >> ssl.keystore.password=12345678
> >> >> ssl.key.password=12345678
> >> >>
> >> >> ssl.truststore.location=/var/private/kafka1.truststore.jks
> >> >> ssl.truststore.password=12345678
> >> >>
> >> >> ssl.client.auth=required
> >> >> ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
> >> >> ssl.keystore.type=JKS
> >> >> ssl.truststore.type=JKS
> >> >>
> >> >> authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
> >> >> ------------------------------------------------------------
> >> >> ------------------------------------------------
> >> >>
> >> >>
> >> >>
> >> >> Here is producer Config(producer.properties)
> >> >> ------------------------------------------------------------
> >> >> ------------------------------------------------
> >> >> security.protocol=SSL
> >> >> ssl.truststore.location=/var/private/kafka2.truststore.jks
> >> >> ssl.truststore.password=12345678
> >> >>
> >> >> ssl.keystore.location=/var/private/kafka2.keystore.jks
> >> >> ssl.keystore.password=12345678
> >> >> ssl.key.password=12345678
> >> >>
> >> >> ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
> >> >> ssl.truststore.type=JKS
> >> >> ssl.keystore.type=JKS
> >> >>
> >> >> ------------------------------------------------------------
> >> >> ------------------------------------------------
> >> >>
> >> >>
> >> >> Raqhav
> >> >>
> >> >
> >> >
> >>
> >>
> >> --
> >> Raghav
> >>
> >
> >
>
>
> --
> Raghav
>
Re: Kafka Authorization and ACLs Broken
Posted by Raghav <ra...@gmail.com>.
Hi Rajini
Thank you very much. It perfectly works.
I think in my setup I was trying to use a CA (certificate authority) to
sign the certificates from client and server, and then adding it to trust
store and keystore. I think in that process, I may have messed something. I
will try above config with a CA to sign certificates. Hopefully that would
work too.
Thanks a lot again.
Raghav
On Wed, May 24, 2017 at 7:04 AM, Rajini Sivaram <ra...@gmail.com>
wrote:
> Raghav/Darshan,
>
> Can you try these steps on a clean installation of Kafka? It works for me,
> so hopefully it will work for you. And then you can adapt to your scenario.
>
> *Create keystores and truststores:*
>
> keytool -genkey -alias kafka -keystore server.keystore.jks -dname
> "CN=KafkaBroker,O=Pivotal,C=UK" -storepass server-keystore-password
> -keypass server-key-password
>
> keytool -exportcert -file server-cert-file -keystore server.keystore.jks
> -alias kafka -storepass server-keystore-password
>
> keytool -importcert -file server-cert-file -keystore server.truststore.jks
> -alias kafka -storepass server-truststore-password -noprompt
>
> keytool -importcert -file server-cert-file -keystore client.truststore.jks
> -alias kafkaclient -storepass client-truststore-password -noprompt
>
>
> keytool -genkey -alias kafkaclient -keystore client.keystore.jks -dname
> "CN=KafkaClient,O=Pivotal,C=UK" -storepass client-keystore-password
> -keypass client-key-password
>
> keytool -exportcert -file client-cert-file -keystore client.keystore.jks
> -alias kafkaclient -storepass client-keystore-password
>
> keytool -importcert -file client-cert-file -keystore server.truststore.jks
> -alias kafkaclient -storepass server-truststore-password -noprompt
>
> *Configure broker: Add these lines at the end of your server.properties*
>
> listeners=SSL://:9093
>
> advertised.listeners=SSL://127.0.0.1:9093
>
> ssl.keystore.location=/tmp/acl/server.keystore.jks
>
> ssl.keystore.password=server-keystore-password
>
> ssl.key.password=server-key-password
>
> ssl.truststore.location=/tmp/acl/server.truststore.jks
>
> ssl.truststore.password=server-truststore-password
>
> security.inter.broker.protocol=SSL
>
> security.protocol=SSL
>
> ssl.client.auth=required
>
> allow.everyone.if.no.acl.found=false
>
> authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
>
> super.users=User:CN=KafkaBroker,O=Pivotal,C=UK
>
> *Configure producer: producer.properties*
>
> security.protocol=SSL
>
> ssl.truststore.location=/tmp/acl/client.truststore.jks
>
> ssl.truststore.password=client-truststore-password
>
> ssl.keystore.location=/tmp/acl/client.keystore.jks
>
> ssl.keystore.password=client-keystore-password
>
> ssl.key.password=client-key-password
>
>
> *Configure consumer: consumer.properties*
>
> security.protocol=SSL
>
> ssl.truststore.location=/tmp/acl/client.truststore.jks
>
> ssl.truststore.password=client-truststore-password
>
> ssl.keystore.location=/tmp/acl/client.keystore.jks
>
> ssl.keystore.password=client-keystore-password
>
> ssl.key.password=client-key-password
>
> group.id=testgroup
>
> *Create topic:*
>
> bin/kafka-topics.sh --zookeeper localhost --create --topic testtopic
> --replication-factor 1 --partitions 1
>
>
> *Configure ACLs:*
>
> bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181
> --add --allow-principal "User:CN=KafkaClient,O=Pivotal,C=UK" --producer
> --topic testtopic
>
> bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181
> --add --allow-principal "User:CN=KafkaClient,O=Pivotal,C=UK" --consumer
> --topic testtopic --group test group
>
>
> *Run console producer and type in some messages:*
>
> bin/kafka-console-producer.sh --producer.config
> /tmp/acl/producer.properties --topic testtopic --broker-list
> 127.0.0.1:9093
>
>
> *Run console consumer, you should see messages from above:*
>
> bin/kafka-console-consumer.sh --consumer.config
> /tmp/acl/consumer.properties --topic testtopic --bootstrap-server
> 127.0.0.1:9093 --from-beginning
>
>
>
> On Tue, May 23, 2017 at 12:57 PM, Raghav <ra...@gmail.com> wrote:
>
>> Darshan,
>>
>> I have not yet successfully gotten the ACLs to work in Kafka. I am still
>> looking for help. I will update this email thread if I do find. In case
>> you
>> get it working, please let me know.
>>
>> Thanks.
>>
>> R
>>
>> On Tue, May 23, 2017 at 8:49 AM, Darshan Purandare <
>> purandare.darshan@gmail.com> wrote:
>>
>> > Raghav
>> >
>> > I saw few posts of yours around Kafka ACLs and the problems. I have seen
>> > similar issues where Writer has not been able to write to any topic. I
>> have
>> > seen "leader not available" and sometimes "unknown topic or partition",
>> and
>> > "topic_authorization_failed" error.
>> >
>> > Let me know if you find a valid config that works.
>> >
>> > Thanks.
>> >
>> >
>> >
>> > On Tue, May 23, 2017 at 8:44 AM, Raghav <ra...@gmail.com> wrote:
>> >
>> >> Hello Kafka Users
>> >>
>> >> I am a new Kafka user and trying to make Kafka SSL work with
>> Authorization
>> >> and ACLs. I followed posts from Kafka and Confluent docs exactly to the
>> >> point but my producer cannot write to kafka broker. I get
>> >> "LEADER_NOT_FOUND" errors. And even Consumer throws the same errors.
>> >>
>> >> Can someone please share their config which worked with ACLs.
>> >>
>> >> Here is my config. Please help.
>> >>
>> >> server.properties config
>> >> ------------------------------------------------------------
>> >> ------------------------------------------------
>> >> broker.id=0
>> >> auto.create.topics.enable=true
>> >> delete.topic.enable=true
>> >>
>> >> listeners=PLAINTEXT://kafka1.example.com:9092
>> >> <http://kafka-dev1.example.com:9092/>,SSL://kafka1.example.com:9093
>> >> <http://kafka-dev1.example.com:9093/>
>> >> host.name=kafka1.example.com <http://kafka-dev1.example.com/>
>> >>
>> >>
>> >>
>> >> ssl.keystore.location=/var/private/kafka1.keystore.jks
>> >> ssl.keystore.password=12345678
>> >> ssl.key.password=12345678
>> >>
>> >> ssl.truststore.location=/var/private/kafka1.truststore.jks
>> >> ssl.truststore.password=12345678
>> >>
>> >> ssl.client.auth=required
>> >> ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
>> >> ssl.keystore.type=JKS
>> >> ssl.truststore.type=JKS
>> >>
>> >> authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
>> >> ------------------------------------------------------------
>> >> ------------------------------------------------
>> >>
>> >>
>> >>
>> >> Here is producer Config(producer.properties)
>> >> ------------------------------------------------------------
>> >> ------------------------------------------------
>> >> security.protocol=SSL
>> >> ssl.truststore.location=/var/private/kafka2.truststore.jks
>> >> ssl.truststore.password=12345678
>> >>
>> >> ssl.keystore.location=/var/private/kafka2.keystore.jks
>> >> ssl.keystore.password=12345678
>> >> ssl.key.password=12345678
>> >>
>> >> ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
>> >> ssl.truststore.type=JKS
>> >> ssl.keystore.type=JKS
>> >>
>> >> ------------------------------------------------------------
>> >> ------------------------------------------------
>> >>
>> >>
>> >> Raqhav
>> >>
>> >
>> >
>>
>>
>> --
>> Raghav
>>
>
>
--
Raghav
Re: Kafka Authorization and ACLs Broken
Posted by Rajini Sivaram <ra...@gmail.com>.
Hi Raghav,
Yes, you should be able to use AdminClient from 0.11.0. Take a look at the
Javadocs (
https://kafka.apache.org/0110/javadoc/org/apache/kafka/clients/admin/package-summary.html).
The integration tests may be useful too (
https://github.com/apache/kafka/blob/trunk/core/src/test/scala/integration/kafka/api/AdminClientIntegrationTest.scala
,
https://github.com/apache/kafka/blob/trunk/core/src/test/scala/integration/kafka/api/SaslSslAdminClientIntegrationTest.scala
).
Regards,
Rajini
On Wed, Jul 5, 2017 at 4:10 PM, Raghav <ra...@gmail.com> wrote:
> Hi Rajini
>
> Now that 0.11.0 is out, can we use the Admin client ? Are there some
> example code for these ?
>
> Thanks.
>
> On Wed, May 24, 2017 at 9:06 PM, Rajini Sivaram <ra...@gmail.com>
> wrote:
>
>> Hi Raghav,
>>
>> Yes, you can create ACLs programmatically. Take a look at the use of
>> AclCommand.main in https://github.com/apache/kafk
>> a/blob/trunk/core/src/test/scala/integration/kafka/api/
>> EndToEndAuthorizationTest.scala
>>
>> If you can wait for the next release 0.11.0 that will be out next month,
>> you can use the new Java AdminClient, which allows you to do this in a much
>> neater way. Take a look at the interface https://github.com/a
>> pache/kafka/blob/trunk/clients/src/main/java/org/apache/
>> kafka/clients/admin/AdminClient.java
>> <https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/admin/AdminClient.java>
>>
>> If your release is not imminent, then you could build Kafka from the
>> 0.11.0 branch and use the new AdminClient. When the release is out, you can
>> switch over to the binary release.
>>
>> Regards,
>>
>> Rajini
>>
>>
>>
>> On Wed, May 24, 2017 at 4:13 PM, Raghav <ra...@gmail.com> wrote:
>>
>>> Hi Rajini
>>>
>>> Quick question on Configuring ACLs: We used bin/kafka-acls.sh to
>>> configure ACL rules, which internally uses Kafka Admin APIs to configure
>>> the ACLs.
>>>
>>> Can I add, remove and list ACLs via zk client libraries ? I want to be
>>> able to add, remove, list ACLs via my code rather than using Kafka-acl.sh.
>>> Is there a guideline for recommended set of libraries to use to do such
>>> operations ?
>>>
>>> As always thanks so much.
>>>
>>>
>>>
>>> On Wed, May 24, 2017 at 7:04 AM, Rajini Sivaram <rajinisivaram@gmail.com
>>> > wrote:
>>>
>>>> Raghav/Darshan,
>>>>
>>>> Can you try these steps on a clean installation of Kafka? It works for
>>>> me, so hopefully it will work for you. And then you can adapt to your
>>>> scenario.
>>>>
>>>> *Create keystores and truststores:*
>>>>
>>>> keytool -genkey -alias kafka -keystore server.keystore.jks -dname
>>>> "CN=KafkaBroker,O=Pivotal,C=UK" -storepass server-keystore-password
>>>> -keypass server-key-password
>>>>
>>>> keytool -exportcert -file server-cert-file -keystore
>>>> server.keystore.jks -alias kafka -storepass server-keystore-password
>>>>
>>>> keytool -importcert -file server-cert-file -keystore
>>>> server.truststore.jks -alias kafka -storepass server-truststore-password
>>>> -noprompt
>>>>
>>>> keytool -importcert -file server-cert-file -keystore
>>>> client.truststore.jks -alias kafkaclient -storepass
>>>> client-truststore-password -noprompt
>>>>
>>>>
>>>> keytool -genkey -alias kafkaclient -keystore client.keystore.jks -dname
>>>> "CN=KafkaClient,O=Pivotal,C=UK" -storepass client-keystore-password
>>>> -keypass client-key-password
>>>>
>>>> keytool -exportcert -file client-cert-file -keystore
>>>> client.keystore.jks -alias kafkaclient -storepass client-keystore-password
>>>>
>>>> keytool -importcert -file client-cert-file -keystore
>>>> server.truststore.jks -alias kafkaclient -storepass
>>>> server-truststore-password -noprompt
>>>>
>>>> *Configure broker: Add these lines at the end of your server.properties*
>>>>
>>>> listeners=SSL://:9093
>>>>
>>>> advertised.listeners=SSL://127.0.0.1:9093
>>>>
>>>> ssl.keystore.location=/tmp/acl/server.keystore.jks
>>>>
>>>> ssl.keystore.password=server-keystore-password
>>>>
>>>> ssl.key.password=server-key-password
>>>>
>>>> ssl.truststore.location=/tmp/acl/server.truststore.jks
>>>>
>>>> ssl.truststore.password=server-truststore-password
>>>>
>>>> security.inter.broker.protocol=SSL
>>>>
>>>> security.protocol=SSL
>>>>
>>>> ssl.client.auth=required
>>>>
>>>> allow.everyone.if.no.acl.found=false
>>>>
>>>> authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
>>>>
>>>> super.users=User:CN=KafkaBroker,O=Pivotal,C=UK
>>>>
>>>> *Configure producer: producer.properties*
>>>>
>>>> security.protocol=SSL
>>>>
>>>> ssl.truststore.location=/tmp/acl/client.truststore.jks
>>>>
>>>> ssl.truststore.password=client-truststore-password
>>>>
>>>> ssl.keystore.location=/tmp/acl/client.keystore.jks
>>>>
>>>> ssl.keystore.password=client-keystore-password
>>>>
>>>> ssl.key.password=client-key-password
>>>>
>>>>
>>>> *Configure consumer: consumer.properties*
>>>>
>>>> security.protocol=SSL
>>>>
>>>> ssl.truststore.location=/tmp/acl/client.truststore.jks
>>>>
>>>> ssl.truststore.password=client-truststore-password
>>>>
>>>> ssl.keystore.location=/tmp/acl/client.keystore.jks
>>>>
>>>> ssl.keystore.password=client-keystore-password
>>>>
>>>> ssl.key.password=client-key-password
>>>>
>>>> group.id=testgroup
>>>>
>>>> *Create topic:*
>>>>
>>>> bin/kafka-topics.sh --zookeeper localhost --create --topic testtopic
>>>> --replication-factor 1 --partitions 1
>>>>
>>>>
>>>> *Configure ACLs:*
>>>>
>>>> bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181
>>>> --add --allow-principal "User:CN=KafkaClient,O=Pivotal,C=UK"
>>>> --producer --topic testtopic
>>>>
>>>> bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181
>>>> --add --allow-principal "User:CN=KafkaClient,O=Pivotal,C=UK"
>>>> --consumer --topic testtopic --group test group
>>>>
>>>>
>>>> *Run console producer and type in some messages:*
>>>>
>>>> bin/kafka-console-producer.sh --producer.config
>>>> /tmp/acl/producer.properties --topic testtopic --broker-list
>>>> 127.0.0.1:9093
>>>>
>>>>
>>>> *Run console consumer, you should see messages from above:*
>>>>
>>>> bin/kafka-console-consumer.sh --consumer.config
>>>> /tmp/acl/consumer.properties --topic testtopic --bootstrap-server
>>>> 127.0.0.1:9093 --from-beginning
>>>>
>>>>
>>>>
>>>> On Tue, May 23, 2017 at 12:57 PM, Raghav <ra...@gmail.com> wrote:
>>>>
>>>>> Darshan,
>>>>>
>>>>> I have not yet successfully gotten the ACLs to work in Kafka. I am
>>>>> still
>>>>> looking for help. I will update this email thread if I do find. In
>>>>> case you
>>>>> get it working, please let me know.
>>>>>
>>>>> Thanks.
>>>>>
>>>>> R
>>>>>
>>>>> On Tue, May 23, 2017 at 8:49 AM, Darshan Purandare <
>>>>> purandare.darshan@gmail.com> wrote:
>>>>>
>>>>> > Raghav
>>>>> >
>>>>> > I saw few posts of yours around Kafka ACLs and the problems. I have
>>>>> seen
>>>>> > similar issues where Writer has not been able to write to any topic.
>>>>> I have
>>>>> > seen "leader not available" and sometimes "unknown topic or
>>>>> partition", and
>>>>> > "topic_authorization_failed" error.
>>>>> >
>>>>> > Let me know if you find a valid config that works.
>>>>> >
>>>>> > Thanks.
>>>>> >
>>>>> >
>>>>> >
>>>>> > On Tue, May 23, 2017 at 8:44 AM, Raghav <ra...@gmail.com>
>>>>> wrote:
>>>>> >
>>>>> >> Hello Kafka Users
>>>>> >>
>>>>> >> I am a new Kafka user and trying to make Kafka SSL work with
>>>>> Authorization
>>>>> >> and ACLs. I followed posts from Kafka and Confluent docs exactly to
>>>>> the
>>>>> >> point but my producer cannot write to kafka broker. I get
>>>>> >> "LEADER_NOT_FOUND" errors. And even Consumer throws the same errors.
>>>>> >>
>>>>> >> Can someone please share their config which worked with ACLs.
>>>>> >>
>>>>> >> Here is my config. Please help.
>>>>> >>
>>>>> >> server.properties config
>>>>> >> ------------------------------------------------------------
>>>>> >> ------------------------------------------------
>>>>> >> broker.id=0
>>>>> >> auto.create.topics.enable=true
>>>>> >> delete.topic.enable=true
>>>>> >>
>>>>> >> listeners=PLAINTEXT://kafka1.example.com:9092
>>>>> >> <http://kafka-dev1.example.com:9092/>,SSL://kafka1.example.com:9093
>>>>> >> <http://kafka-dev1.example.com:9093/>
>>>>> >> host.name=kafka1.example.com <http://kafka-dev1.example.com/>
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> ssl.keystore.location=/var/private/kafka1.keystore.jks
>>>>> >> ssl.keystore.password=12345678
>>>>> >> ssl.key.password=12345678
>>>>> >>
>>>>> >> ssl.truststore.location=/var/private/kafka1.truststore.jks
>>>>> >> ssl.truststore.password=12345678
>>>>> >>
>>>>> >> ssl.client.auth=required
>>>>> >> ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
>>>>> >> ssl.keystore.type=JKS
>>>>> >> ssl.truststore.type=JKS
>>>>> >>
>>>>> >> authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
>>>>> >> ------------------------------------------------------------
>>>>> >> ------------------------------------------------
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> Here is producer Config(producer.properties)
>>>>> >> ------------------------------------------------------------
>>>>> >> ------------------------------------------------
>>>>> >> security.protocol=SSL
>>>>> >> ssl.truststore.location=/var/private/kafka2.truststore.jks
>>>>> >> ssl.truststore.password=12345678
>>>>> >>
>>>>> >> ssl.keystore.location=/var/private/kafka2.keystore.jks
>>>>> >> ssl.keystore.password=12345678
>>>>> >> ssl.key.password=12345678
>>>>> >>
>>>>> >> ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
>>>>> >> ssl.truststore.type=JKS
>>>>> >> ssl.keystore.type=JKS
>>>>> >>
>>>>> >> ------------------------------------------------------------
>>>>> >> ------------------------------------------------
>>>>> >>
>>>>> >>
>>>>> >> Raqhav
>>>>> >>
>>>>> >
>>>>> >
>>>>>
>>>>>
>>>>> --
>>>>> Raghav
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Raghav
>>>
>>
>>
>
>
> --
> Raghav
>
Re: Kafka Authorization and ACLs Broken
Posted by Raghav <ra...@gmail.com>.
Hi Rajini
Now that 0.11.0 is out, can we use the Admin client ? Are there some
example code for these ?
Thanks.
On Wed, May 24, 2017 at 9:06 PM, Rajini Sivaram <ra...@gmail.com>
wrote:
> Hi Raghav,
>
> Yes, you can create ACLs programmatically. Take a look at the use of
> AclCommand.main in https://github.com/apache/kafka/blob/trunk/core/src/
> test/scala/integration/kafka/api/EndToEndAuthorizationTest.scala
>
> If you can wait for the next release 0.11.0 that will be out next month,
> you can use the new Java AdminClient, which allows you to do this in a much
> neater way. Take a look at the interface https://github.com/
> apache/kafka/blob/trunk/clients/src/main/java/org/
> apache/kafka/clients/admin/AdminClient.java
> <https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/admin/AdminClient.java>
>
> If your release is not imminent, then you could build Kafka from the
> 0.11.0 branch and use the new AdminClient. When the release is out, you can
> switch over to the binary release.
>
> Regards,
>
> Rajini
>
>
>
> On Wed, May 24, 2017 at 4:13 PM, Raghav <ra...@gmail.com> wrote:
>
>> Hi Rajini
>>
>> Quick question on Configuring ACLs: We used bin/kafka-acls.sh to
>> configure ACL rules, which internally uses Kafka Admin APIs to configure
>> the ACLs.
>>
>> Can I add, remove and list ACLs via zk client libraries ? I want to be
>> able to add, remove, list ACLs via my code rather than using Kafka-acl.sh.
>> Is there a guideline for recommended set of libraries to use to do such
>> operations ?
>>
>> As always thanks so much.
>>
>>
>>
>> On Wed, May 24, 2017 at 7:04 AM, Rajini Sivaram <ra...@gmail.com>
>> wrote:
>>
>>> Raghav/Darshan,
>>>
>>> Can you try these steps on a clean installation of Kafka? It works for
>>> me, so hopefully it will work for you. And then you can adapt to your
>>> scenario.
>>>
>>> *Create keystores and truststores:*
>>>
>>> keytool -genkey -alias kafka -keystore server.keystore.jks -dname
>>> "CN=KafkaBroker,O=Pivotal,C=UK" -storepass server-keystore-password
>>> -keypass server-key-password
>>>
>>> keytool -exportcert -file server-cert-file -keystore server.keystore.jks
>>> -alias kafka -storepass server-keystore-password
>>>
>>> keytool -importcert -file server-cert-file -keystore
>>> server.truststore.jks -alias kafka -storepass server-truststore-password
>>> -noprompt
>>>
>>> keytool -importcert -file server-cert-file -keystore
>>> client.truststore.jks -alias kafkaclient -storepass
>>> client-truststore-password -noprompt
>>>
>>>
>>> keytool -genkey -alias kafkaclient -keystore client.keystore.jks -dname
>>> "CN=KafkaClient,O=Pivotal,C=UK" -storepass client-keystore-password
>>> -keypass client-key-password
>>>
>>> keytool -exportcert -file client-cert-file -keystore client.keystore.jks
>>> -alias kafkaclient -storepass client-keystore-password
>>>
>>> keytool -importcert -file client-cert-file -keystore
>>> server.truststore.jks -alias kafkaclient -storepass
>>> server-truststore-password -noprompt
>>>
>>> *Configure broker: Add these lines at the end of your server.properties*
>>>
>>> listeners=SSL://:9093
>>>
>>> advertised.listeners=SSL://127.0.0.1:9093
>>>
>>> ssl.keystore.location=/tmp/acl/server.keystore.jks
>>>
>>> ssl.keystore.password=server-keystore-password
>>>
>>> ssl.key.password=server-key-password
>>>
>>> ssl.truststore.location=/tmp/acl/server.truststore.jks
>>>
>>> ssl.truststore.password=server-truststore-password
>>>
>>> security.inter.broker.protocol=SSL
>>>
>>> security.protocol=SSL
>>>
>>> ssl.client.auth=required
>>>
>>> allow.everyone.if.no.acl.found=false
>>>
>>> authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
>>>
>>> super.users=User:CN=KafkaBroker,O=Pivotal,C=UK
>>>
>>> *Configure producer: producer.properties*
>>>
>>> security.protocol=SSL
>>>
>>> ssl.truststore.location=/tmp/acl/client.truststore.jks
>>>
>>> ssl.truststore.password=client-truststore-password
>>>
>>> ssl.keystore.location=/tmp/acl/client.keystore.jks
>>>
>>> ssl.keystore.password=client-keystore-password
>>>
>>> ssl.key.password=client-key-password
>>>
>>>
>>> *Configure consumer: consumer.properties*
>>>
>>> security.protocol=SSL
>>>
>>> ssl.truststore.location=/tmp/acl/client.truststore.jks
>>>
>>> ssl.truststore.password=client-truststore-password
>>>
>>> ssl.keystore.location=/tmp/acl/client.keystore.jks
>>>
>>> ssl.keystore.password=client-keystore-password
>>>
>>> ssl.key.password=client-key-password
>>>
>>> group.id=testgroup
>>>
>>> *Create topic:*
>>>
>>> bin/kafka-topics.sh --zookeeper localhost --create --topic testtopic
>>> --replication-factor 1 --partitions 1
>>>
>>>
>>> *Configure ACLs:*
>>>
>>> bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181
>>> --add --allow-principal "User:CN=KafkaClient,O=Pivotal,C=UK" --producer
>>> --topic testtopic
>>>
>>> bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181
>>> --add --allow-principal "User:CN=KafkaClient,O=Pivotal,C=UK" --consumer
>>> --topic testtopic --group test group
>>>
>>>
>>> *Run console producer and type in some messages:*
>>>
>>> bin/kafka-console-producer.sh --producer.config
>>> /tmp/acl/producer.properties --topic testtopic --broker-list
>>> 127.0.0.1:9093
>>>
>>>
>>> *Run console consumer, you should see messages from above:*
>>>
>>> bin/kafka-console-consumer.sh --consumer.config
>>> /tmp/acl/consumer.properties --topic testtopic --bootstrap-server
>>> 127.0.0.1:9093 --from-beginning
>>>
>>>
>>>
>>> On Tue, May 23, 2017 at 12:57 PM, Raghav <ra...@gmail.com> wrote:
>>>
>>>> Darshan,
>>>>
>>>> I have not yet successfully gotten the ACLs to work in Kafka. I am still
>>>> looking for help. I will update this email thread if I do find. In case
>>>> you
>>>> get it working, please let me know.
>>>>
>>>> Thanks.
>>>>
>>>> R
>>>>
>>>> On Tue, May 23, 2017 at 8:49 AM, Darshan Purandare <
>>>> purandare.darshan@gmail.com> wrote:
>>>>
>>>> > Raghav
>>>> >
>>>> > I saw few posts of yours around Kafka ACLs and the problems. I have
>>>> seen
>>>> > similar issues where Writer has not been able to write to any topic.
>>>> I have
>>>> > seen "leader not available" and sometimes "unknown topic or
>>>> partition", and
>>>> > "topic_authorization_failed" error.
>>>> >
>>>> > Let me know if you find a valid config that works.
>>>> >
>>>> > Thanks.
>>>> >
>>>> >
>>>> >
>>>> > On Tue, May 23, 2017 at 8:44 AM, Raghav <ra...@gmail.com>
>>>> wrote:
>>>> >
>>>> >> Hello Kafka Users
>>>> >>
>>>> >> I am a new Kafka user and trying to make Kafka SSL work with
>>>> Authorization
>>>> >> and ACLs. I followed posts from Kafka and Confluent docs exactly to
>>>> the
>>>> >> point but my producer cannot write to kafka broker. I get
>>>> >> "LEADER_NOT_FOUND" errors. And even Consumer throws the same errors.
>>>> >>
>>>> >> Can someone please share their config which worked with ACLs.
>>>> >>
>>>> >> Here is my config. Please help.
>>>> >>
>>>> >> server.properties config
>>>> >> ------------------------------------------------------------
>>>> >> ------------------------------------------------
>>>> >> broker.id=0
>>>> >> auto.create.topics.enable=true
>>>> >> delete.topic.enable=true
>>>> >>
>>>> >> listeners=PLAINTEXT://kafka1.example.com:9092
>>>> >> <http://kafka-dev1.example.com:9092/>,SSL://kafka1.example.com:9093
>>>> >> <http://kafka-dev1.example.com:9093/>
>>>> >> host.name=kafka1.example.com <http://kafka-dev1.example.com/>
>>>> >>
>>>> >>
>>>> >>
>>>> >> ssl.keystore.location=/var/private/kafka1.keystore.jks
>>>> >> ssl.keystore.password=12345678
>>>> >> ssl.key.password=12345678
>>>> >>
>>>> >> ssl.truststore.location=/var/private/kafka1.truststore.jks
>>>> >> ssl.truststore.password=12345678
>>>> >>
>>>> >> ssl.client.auth=required
>>>> >> ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
>>>> >> ssl.keystore.type=JKS
>>>> >> ssl.truststore.type=JKS
>>>> >>
>>>> >> authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
>>>> >> ------------------------------------------------------------
>>>> >> ------------------------------------------------
>>>> >>
>>>> >>
>>>> >>
>>>> >> Here is producer Config(producer.properties)
>>>> >> ------------------------------------------------------------
>>>> >> ------------------------------------------------
>>>> >> security.protocol=SSL
>>>> >> ssl.truststore.location=/var/private/kafka2.truststore.jks
>>>> >> ssl.truststore.password=12345678
>>>> >>
>>>> >> ssl.keystore.location=/var/private/kafka2.keystore.jks
>>>> >> ssl.keystore.password=12345678
>>>> >> ssl.key.password=12345678
>>>> >>
>>>> >> ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
>>>> >> ssl.truststore.type=JKS
>>>> >> ssl.keystore.type=JKS
>>>> >>
>>>> >> ------------------------------------------------------------
>>>> >> ------------------------------------------------
>>>> >>
>>>> >>
>>>> >> Raqhav
>>>> >>
>>>> >
>>>> >
>>>>
>>>>
>>>> --
>>>> Raghav
>>>>
>>>
>>>
>>
>>
>> --
>> Raghav
>>
>
>
--
Raghav
Re: Kafka Authorization and ACLs Broken
Posted by Rajini Sivaram <ra...@gmail.com>.
Raghav/Darshan,
Can you try these steps on a clean installation of Kafka? It works for me,
so hopefully it will work for you. And then you can adapt to your scenario.
*Create keystores and truststores:*
keytool -genkey -alias kafka -keystore server.keystore.jks -dname
"CN=KafkaBroker,O=Pivotal,C=UK" -storepass server-keystore-password
-keypass server-key-password
keytool -exportcert -file server-cert-file -keystore server.keystore.jks
-alias kafka -storepass server-keystore-password
keytool -importcert -file server-cert-file -keystore server.truststore.jks
-alias kafka -storepass server-truststore-password -noprompt
keytool -importcert -file server-cert-file -keystore client.truststore.jks
-alias kafkaclient -storepass client-truststore-password -noprompt
keytool -genkey -alias kafkaclient -keystore client.keystore.jks -dname
"CN=KafkaClient,O=Pivotal,C=UK" -storepass client-keystore-password
-keypass client-key-password
keytool -exportcert -file client-cert-file -keystore client.keystore.jks
-alias kafkaclient -storepass client-keystore-password
keytool -importcert -file client-cert-file -keystore server.truststore.jks
-alias kafkaclient -storepass server-truststore-password -noprompt
*Configure broker: Add these lines at the end of your server.properties*
listeners=SSL://:9093
advertised.listeners=SSL://127.0.0.1:9093
ssl.keystore.location=/tmp/acl/server.keystore.jks
ssl.keystore.password=server-keystore-password
ssl.key.password=server-key-password
ssl.truststore.location=/tmp/acl/server.truststore.jks
ssl.truststore.password=server-truststore-password
security.inter.broker.protocol=SSL
security.protocol=SSL
ssl.client.auth=required
allow.everyone.if.no.acl.found=false
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
super.users=User:CN=KafkaBroker,O=Pivotal,C=UK
*Configure producer: producer.properties*
security.protocol=SSL
ssl.truststore.location=/tmp/acl/client.truststore.jks
ssl.truststore.password=client-truststore-password
ssl.keystore.location=/tmp/acl/client.keystore.jks
ssl.keystore.password=client-keystore-password
ssl.key.password=client-key-password
*Configure consumer: consumer.properties*
security.protocol=SSL
ssl.truststore.location=/tmp/acl/client.truststore.jks
ssl.truststore.password=client-truststore-password
ssl.keystore.location=/tmp/acl/client.keystore.jks
ssl.keystore.password=client-keystore-password
ssl.key.password=client-key-password
group.id=testgroup
*Create topic:*
bin/kafka-topics.sh --zookeeper localhost --create --topic testtopic
--replication-factor 1 --partitions 1
*Configure ACLs:*
bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181
--add --allow-principal "User:CN=KafkaClient,O=Pivotal,C=UK" --producer
--topic testtopic
bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181
--add --allow-principal "User:CN=KafkaClient,O=Pivotal,C=UK" --consumer
--topic testtopic --group test group
*Run console producer and type in some messages:*
bin/kafka-console-producer.sh --producer.config
/tmp/acl/producer.properties --topic testtopic --broker-list 127.0.0.1:9093
*Run console consumer, you should see messages from above:*
bin/kafka-console-consumer.sh --consumer.config
/tmp/acl/consumer.properties --topic testtopic --bootstrap-server
127.0.0.1:9093 --from-beginning
On Tue, May 23, 2017 at 12:57 PM, Raghav <ra...@gmail.com> wrote:
> Darshan,
>
> I have not yet successfully gotten the ACLs to work in Kafka. I am still
> looking for help. I will update this email thread if I do find. In case you
> get it working, please let me know.
>
> Thanks.
>
> R
>
> On Tue, May 23, 2017 at 8:49 AM, Darshan Purandare <
> purandare.darshan@gmail.com> wrote:
>
> > Raghav
> >
> > I saw few posts of yours around Kafka ACLs and the problems. I have seen
> > similar issues where Writer has not been able to write to any topic. I
> have
> > seen "leader not available" and sometimes "unknown topic or partition",
> and
> > "topic_authorization_failed" error.
> >
> > Let me know if you find a valid config that works.
> >
> > Thanks.
> >
> >
> >
> > On Tue, May 23, 2017 at 8:44 AM, Raghav <ra...@gmail.com> wrote:
> >
> >> Hello Kafka Users
> >>
> >> I am a new Kafka user and trying to make Kafka SSL work with
> Authorization
> >> and ACLs. I followed posts from Kafka and Confluent docs exactly to the
> >> point but my producer cannot write to kafka broker. I get
> >> "LEADER_NOT_FOUND" errors. And even Consumer throws the same errors.
> >>
> >> Can someone please share their config which worked with ACLs.
> >>
> >> Here is my config. Please help.
> >>
> >> server.properties config
> >> ------------------------------------------------------------
> >> ------------------------------------------------
> >> broker.id=0
> >> auto.create.topics.enable=true
> >> delete.topic.enable=true
> >>
> >> listeners=PLAINTEXT://kafka1.example.com:9092
> >> <http://kafka-dev1.example.com:9092/>,SSL://kafka1.example.com:9093
> >> <http://kafka-dev1.example.com:9093/>
> >> host.name=kafka1.example.com <http://kafka-dev1.example.com/>
> >>
> >>
> >>
> >> ssl.keystore.location=/var/private/kafka1.keystore.jks
> >> ssl.keystore.password=12345678
> >> ssl.key.password=12345678
> >>
> >> ssl.truststore.location=/var/private/kafka1.truststore.jks
> >> ssl.truststore.password=12345678
> >>
> >> ssl.client.auth=required
> >> ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
> >> ssl.keystore.type=JKS
> >> ssl.truststore.type=JKS
> >>
> >> authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
> >> ------------------------------------------------------------
> >> ------------------------------------------------
> >>
> >>
> >>
> >> Here is producer Config(producer.properties)
> >> ------------------------------------------------------------
> >> ------------------------------------------------
> >> security.protocol=SSL
> >> ssl.truststore.location=/var/private/kafka2.truststore.jks
> >> ssl.truststore.password=12345678
> >>
> >> ssl.keystore.location=/var/private/kafka2.keystore.jks
> >> ssl.keystore.password=12345678
> >> ssl.key.password=12345678
> >>
> >> ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
> >> ssl.truststore.type=JKS
> >> ssl.keystore.type=JKS
> >>
> >> ------------------------------------------------------------
> >> ------------------------------------------------
> >>
> >>
> >> Raqhav
> >>
> >
> >
>
>
> --
> Raghav
>
Re: Kafka Authorization and ACLs Broken
Posted by Raghav <ra...@gmail.com>.
Darshan,
I have not yet successfully gotten the ACLs to work in Kafka. I am still
looking for help. I will update this email thread if I do find. In case you
get it working, please let me know.
Thanks.
R
On Tue, May 23, 2017 at 8:49 AM, Darshan Purandare <
purandare.darshan@gmail.com> wrote:
> Raghav
>
> I saw few posts of yours around Kafka ACLs and the problems. I have seen
> similar issues where Writer has not been able to write to any topic. I have
> seen "leader not available" and sometimes "unknown topic or partition", and
> "topic_authorization_failed" error.
>
> Let me know if you find a valid config that works.
>
> Thanks.
>
>
>
> On Tue, May 23, 2017 at 8:44 AM, Raghav <ra...@gmail.com> wrote:
>
>> Hello Kafka Users
>>
>> I am a new Kafka user and trying to make Kafka SSL work with Authorization
>> and ACLs. I followed posts from Kafka and Confluent docs exactly to the
>> point but my producer cannot write to kafka broker. I get
>> "LEADER_NOT_FOUND" errors. And even Consumer throws the same errors.
>>
>> Can someone please share their config which worked with ACLs.
>>
>> Here is my config. Please help.
>>
>> server.properties config
>> ------------------------------------------------------------
>> ------------------------------------------------
>> broker.id=0
>> auto.create.topics.enable=true
>> delete.topic.enable=true
>>
>> listeners=PLAINTEXT://kafka1.example.com:9092
>> <http://kafka-dev1.example.com:9092/>,SSL://kafka1.example.com:9093
>> <http://kafka-dev1.example.com:9093/>
>> host.name=kafka1.example.com <http://kafka-dev1.example.com/>
>>
>>
>>
>> ssl.keystore.location=/var/private/kafka1.keystore.jks
>> ssl.keystore.password=12345678
>> ssl.key.password=12345678
>>
>> ssl.truststore.location=/var/private/kafka1.truststore.jks
>> ssl.truststore.password=12345678
>>
>> ssl.client.auth=required
>> ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
>> ssl.keystore.type=JKS
>> ssl.truststore.type=JKS
>>
>> authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
>> ------------------------------------------------------------
>> ------------------------------------------------
>>
>>
>>
>> Here is producer Config(producer.properties)
>> ------------------------------------------------------------
>> ------------------------------------------------
>> security.protocol=SSL
>> ssl.truststore.location=/var/private/kafka2.truststore.jks
>> ssl.truststore.password=12345678
>>
>> ssl.keystore.location=/var/private/kafka2.keystore.jks
>> ssl.keystore.password=12345678
>> ssl.key.password=12345678
>>
>> ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
>> ssl.truststore.type=JKS
>> ssl.keystore.type=JKS
>>
>> ------------------------------------------------------------
>> ------------------------------------------------
>>
>>
>> Raqhav
>>
>
>
--
Raghav
Re: Kafka Authorization and ACLs Broken
Posted by Darshan Purandare <pu...@gmail.com>.
Raghav
I saw few posts of yours around Kafka ACLs and the problems. I have seen
similar issues where Writer has not been able to write to any topic. I have
seen "leader not available" and sometimes "unknown topic or partition", and
"topic_authorization_failed" error.
Let me know if you find a valid config that works.
Thanks.
On Tue, May 23, 2017 at 8:44 AM, Raghav <ra...@gmail.com> wrote:
> Hello Kafka Users
>
> I am a new Kafka user and trying to make Kafka SSL work with Authorization
> and ACLs. I followed posts from Kafka and Confluent docs exactly to the
> point but my producer cannot write to kafka broker. I get
> "LEADER_NOT_FOUND" errors. And even Consumer throws the same errors.
>
> Can someone please share their config which worked with ACLs.
>
> Here is my config. Please help.
>
> server.properties config
> ------------------------------------------------------------
> ------------------------------------------------
> broker.id=0
> auto.create.topics.enable=true
> delete.topic.enable=true
>
> listeners=PLAINTEXT://kafka1.example.com:9092
> <http://kafka-dev1.example.com:9092/>,SSL://kafka1.example.com:9093
> <http://kafka-dev1.example.com:9093/>
> host.name=kafka1.example.com <http://kafka-dev1.example.com/>
>
>
> ssl.keystore.location=/var/private/kafka1.keystore.jks
> ssl.keystore.password=12345678
> ssl.key.password=12345678
>
> ssl.truststore.location=/var/private/kafka1.truststore.jks
> ssl.truststore.password=12345678
>
> ssl.client.auth=required
> ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
> ssl.keystore.type=JKS
> ssl.truststore.type=JKS
>
> authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
> ------------------------------------------------------------
> ------------------------------------------------
>
>
>
> Here is producer Config(producer.properties)
> ------------------------------------------------------------
> ------------------------------------------------
> security.protocol=SSL
> ssl.truststore.location=/var/private/kafka2.truststore.jks
> ssl.truststore.password=12345678
>
> ssl.keystore.location=/var/private/kafka2.keystore.jks
> ssl.keystore.password=12345678
> ssl.key.password=12345678
>
> ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
> ssl.truststore.type=JKS
> ssl.keystore.type=JKS
>
> ------------------------------------------------------------
> ------------------------------------------------
>
>
> Raqhav
>