You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Raghav <ra...@gmail.com> on 2017/07/05 15:10:23 UTC
Re: Kafka Authorization and ACLs Broken
Hi Rajini
Now that 0.11.0 is out, can we use the Admin client ? Are there some
example code for these ?
Thanks.
On Wed, May 24, 2017 at 9:06 PM, Rajini Sivaram <ra...@gmail.com>
wrote:
> Hi Raghav,
>
> Yes, you can create ACLs programmatically. Take a look at the use of
> AclCommand.main in https://github.com/apache/kafka/blob/trunk/core/src/
> test/scala/integration/kafka/api/EndToEndAuthorizationTest.scala
>
> If you can wait for the next release 0.11.0 that will be out next month,
> you can use the new Java AdminClient, which allows you to do this in a much
> neater way. Take a look at the interface https://github.com/
> apache/kafka/blob/trunk/clients/src/main/java/org/
> apache/kafka/clients/admin/AdminClient.java
> <https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/admin/AdminClient.java>
>
> If your release is not imminent, then you could build Kafka from the
> 0.11.0 branch and use the new AdminClient. When the release is out, you can
> switch over to the binary release.
>
> Regards,
>
> Rajini
>
>
>
> On Wed, May 24, 2017 at 4:13 PM, Raghav <ra...@gmail.com> wrote:
>
>> Hi Rajini
>>
>> Quick question on Configuring ACLs: We used bin/kafka-acls.sh to
>> configure ACL rules, which internally uses Kafka Admin APIs to configure
>> the ACLs.
>>
>> Can I add, remove and list ACLs via zk client libraries ? I want to be
>> able to add, remove, list ACLs via my code rather than using Kafka-acl.sh.
>> Is there a guideline for recommended set of libraries to use to do such
>> operations ?
>>
>> As always thanks so much.
>>
>>
>>
>> On Wed, May 24, 2017 at 7:04 AM, Rajini Sivaram <ra...@gmail.com>
>> wrote:
>>
>>> Raghav/Darshan,
>>>
>>> Can you try these steps on a clean installation of Kafka? It works for
>>> me, so hopefully it will work for you. And then you can adapt to your
>>> scenario.
>>>
>>> *Create keystores and truststores:*
>>>
>>> keytool -genkey -alias kafka -keystore server.keystore.jks -dname
>>> "CN=KafkaBroker,O=Pivotal,C=UK" -storepass server-keystore-password
>>> -keypass server-key-password
>>>
>>> keytool -exportcert -file server-cert-file -keystore server.keystore.jks
>>> -alias kafka -storepass server-keystore-password
>>>
>>> keytool -importcert -file server-cert-file -keystore
>>> server.truststore.jks -alias kafka -storepass server-truststore-password
>>> -noprompt
>>>
>>> keytool -importcert -file server-cert-file -keystore
>>> client.truststore.jks -alias kafkaclient -storepass
>>> client-truststore-password -noprompt
>>>
>>>
>>> keytool -genkey -alias kafkaclient -keystore client.keystore.jks -dname
>>> "CN=KafkaClient,O=Pivotal,C=UK" -storepass client-keystore-password
>>> -keypass client-key-password
>>>
>>> keytool -exportcert -file client-cert-file -keystore client.keystore.jks
>>> -alias kafkaclient -storepass client-keystore-password
>>>
>>> keytool -importcert -file client-cert-file -keystore
>>> server.truststore.jks -alias kafkaclient -storepass
>>> server-truststore-password -noprompt
>>>
>>> *Configure broker: Add these lines at the end of your server.properties*
>>>
>>> listeners=SSL://:9093
>>>
>>> advertised.listeners=SSL://127.0.0.1:9093
>>>
>>> ssl.keystore.location=/tmp/acl/server.keystore.jks
>>>
>>> ssl.keystore.password=server-keystore-password
>>>
>>> ssl.key.password=server-key-password
>>>
>>> ssl.truststore.location=/tmp/acl/server.truststore.jks
>>>
>>> ssl.truststore.password=server-truststore-password
>>>
>>> security.inter.broker.protocol=SSL
>>>
>>> security.protocol=SSL
>>>
>>> ssl.client.auth=required
>>>
>>> allow.everyone.if.no.acl.found=false
>>>
>>> authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
>>>
>>> super.users=User:CN=KafkaBroker,O=Pivotal,C=UK
>>>
>>> *Configure producer: producer.properties*
>>>
>>> security.protocol=SSL
>>>
>>> ssl.truststore.location=/tmp/acl/client.truststore.jks
>>>
>>> ssl.truststore.password=client-truststore-password
>>>
>>> ssl.keystore.location=/tmp/acl/client.keystore.jks
>>>
>>> ssl.keystore.password=client-keystore-password
>>>
>>> ssl.key.password=client-key-password
>>>
>>>
>>> *Configure consumer: consumer.properties*
>>>
>>> security.protocol=SSL
>>>
>>> ssl.truststore.location=/tmp/acl/client.truststore.jks
>>>
>>> ssl.truststore.password=client-truststore-password
>>>
>>> ssl.keystore.location=/tmp/acl/client.keystore.jks
>>>
>>> ssl.keystore.password=client-keystore-password
>>>
>>> ssl.key.password=client-key-password
>>>
>>> group.id=testgroup
>>>
>>> *Create topic:*
>>>
>>> bin/kafka-topics.sh --zookeeper localhost --create --topic testtopic
>>> --replication-factor 1 --partitions 1
>>>
>>>
>>> *Configure ACLs:*
>>>
>>> bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181
>>> --add --allow-principal "User:CN=KafkaClient,O=Pivotal,C=UK" --producer
>>> --topic testtopic
>>>
>>> bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181
>>> --add --allow-principal "User:CN=KafkaClient,O=Pivotal,C=UK" --consumer
>>> --topic testtopic --group test group
>>>
>>>
>>> *Run console producer and type in some messages:*
>>>
>>> bin/kafka-console-producer.sh --producer.config
>>> /tmp/acl/producer.properties --topic testtopic --broker-list
>>> 127.0.0.1:9093
>>>
>>>
>>> *Run console consumer, you should see messages from above:*
>>>
>>> bin/kafka-console-consumer.sh --consumer.config
>>> /tmp/acl/consumer.properties --topic testtopic --bootstrap-server
>>> 127.0.0.1:9093 --from-beginning
>>>
>>>
>>>
>>> On Tue, May 23, 2017 at 12:57 PM, Raghav <ra...@gmail.com> wrote:
>>>
>>>> Darshan,
>>>>
>>>> I have not yet successfully gotten the ACLs to work in Kafka. I am still
>>>> looking for help. I will update this email thread if I do find. In case
>>>> you
>>>> get it working, please let me know.
>>>>
>>>> Thanks.
>>>>
>>>> R
>>>>
>>>> On Tue, May 23, 2017 at 8:49 AM, Darshan Purandare <
>>>> purandare.darshan@gmail.com> wrote:
>>>>
>>>> > Raghav
>>>> >
>>>> > I saw few posts of yours around Kafka ACLs and the problems. I have
>>>> seen
>>>> > similar issues where Writer has not been able to write to any topic.
>>>> I have
>>>> > seen "leader not available" and sometimes "unknown topic or
>>>> partition", and
>>>> > "topic_authorization_failed" error.
>>>> >
>>>> > Let me know if you find a valid config that works.
>>>> >
>>>> > Thanks.
>>>> >
>>>> >
>>>> >
>>>> > On Tue, May 23, 2017 at 8:44 AM, Raghav <ra...@gmail.com>
>>>> wrote:
>>>> >
>>>> >> Hello Kafka Users
>>>> >>
>>>> >> I am a new Kafka user and trying to make Kafka SSL work with
>>>> Authorization
>>>> >> and ACLs. I followed posts from Kafka and Confluent docs exactly to
>>>> the
>>>> >> point but my producer cannot write to kafka broker. I get
>>>> >> "LEADER_NOT_FOUND" errors. And even Consumer throws the same errors.
>>>> >>
>>>> >> Can someone please share their config which worked with ACLs.
>>>> >>
>>>> >> Here is my config. Please help.
>>>> >>
>>>> >> server.properties config
>>>> >> ------------------------------------------------------------
>>>> >> ------------------------------------------------
>>>> >> broker.id=0
>>>> >> auto.create.topics.enable=true
>>>> >> delete.topic.enable=true
>>>> >>
>>>> >> listeners=PLAINTEXT://kafka1.example.com:9092
>>>> >> <http://kafka-dev1.example.com:9092/>,SSL://kafka1.example.com:9093
>>>> >> <http://kafka-dev1.example.com:9093/>
>>>> >> host.name=kafka1.example.com <http://kafka-dev1.example.com/>
>>>> >>
>>>> >>
>>>> >>
>>>> >> ssl.keystore.location=/var/private/kafka1.keystore.jks
>>>> >> ssl.keystore.password=12345678
>>>> >> ssl.key.password=12345678
>>>> >>
>>>> >> ssl.truststore.location=/var/private/kafka1.truststore.jks
>>>> >> ssl.truststore.password=12345678
>>>> >>
>>>> >> ssl.client.auth=required
>>>> >> ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
>>>> >> ssl.keystore.type=JKS
>>>> >> ssl.truststore.type=JKS
>>>> >>
>>>> >> authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
>>>> >> ------------------------------------------------------------
>>>> >> ------------------------------------------------
>>>> >>
>>>> >>
>>>> >>
>>>> >> Here is producer Config(producer.properties)
>>>> >> ------------------------------------------------------------
>>>> >> ------------------------------------------------
>>>> >> security.protocol=SSL
>>>> >> ssl.truststore.location=/var/private/kafka2.truststore.jks
>>>> >> ssl.truststore.password=12345678
>>>> >>
>>>> >> ssl.keystore.location=/var/private/kafka2.keystore.jks
>>>> >> ssl.keystore.password=12345678
>>>> >> ssl.key.password=12345678
>>>> >>
>>>> >> ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
>>>> >> ssl.truststore.type=JKS
>>>> >> ssl.keystore.type=JKS
>>>> >>
>>>> >> ------------------------------------------------------------
>>>> >> ------------------------------------------------
>>>> >>
>>>> >>
>>>> >> Raqhav
>>>> >>
>>>> >
>>>> >
>>>>
>>>>
>>>> --
>>>> Raghav
>>>>
>>>
>>>
>>
>>
>> --
>> Raghav
>>
>
>
--
Raghav
Re: Kafka Authorization and ACLs Broken
Posted by Rajini Sivaram <ra...@gmail.com>.
Hi Raghav,
Yes, you should be able to use AdminClient from 0.11.0. Take a look at the
Javadocs (
https://kafka.apache.org/0110/javadoc/org/apache/kafka/clients/admin/package-summary.html).
The integration tests may be useful too (
https://github.com/apache/kafka/blob/trunk/core/src/test/scala/integration/kafka/api/AdminClientIntegrationTest.scala
,
https://github.com/apache/kafka/blob/trunk/core/src/test/scala/integration/kafka/api/SaslSslAdminClientIntegrationTest.scala
).
Regards,
Rajini
On Wed, Jul 5, 2017 at 4:10 PM, Raghav <ra...@gmail.com> wrote:
> Hi Rajini
>
> Now that 0.11.0 is out, can we use the Admin client ? Are there some
> example code for these ?
>
> Thanks.
>
> On Wed, May 24, 2017 at 9:06 PM, Rajini Sivaram <ra...@gmail.com>
> wrote:
>
>> Hi Raghav,
>>
>> Yes, you can create ACLs programmatically. Take a look at the use of
>> AclCommand.main in https://github.com/apache/kafk
>> a/blob/trunk/core/src/test/scala/integration/kafka/api/
>> EndToEndAuthorizationTest.scala
>>
>> If you can wait for the next release 0.11.0 that will be out next month,
>> you can use the new Java AdminClient, which allows you to do this in a much
>> neater way. Take a look at the interface https://github.com/a
>> pache/kafka/blob/trunk/clients/src/main/java/org/apache/
>> kafka/clients/admin/AdminClient.java
>> <https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/admin/AdminClient.java>
>>
>> If your release is not imminent, then you could build Kafka from the
>> 0.11.0 branch and use the new AdminClient. When the release is out, you can
>> switch over to the binary release.
>>
>> Regards,
>>
>> Rajini
>>
>>
>>
>> On Wed, May 24, 2017 at 4:13 PM, Raghav <ra...@gmail.com> wrote:
>>
>>> Hi Rajini
>>>
>>> Quick question on Configuring ACLs: We used bin/kafka-acls.sh to
>>> configure ACL rules, which internally uses Kafka Admin APIs to configure
>>> the ACLs.
>>>
>>> Can I add, remove and list ACLs via zk client libraries ? I want to be
>>> able to add, remove, list ACLs via my code rather than using Kafka-acl.sh.
>>> Is there a guideline for recommended set of libraries to use to do such
>>> operations ?
>>>
>>> As always thanks so much.
>>>
>>>
>>>
>>> On Wed, May 24, 2017 at 7:04 AM, Rajini Sivaram <rajinisivaram@gmail.com
>>> > wrote:
>>>
>>>> Raghav/Darshan,
>>>>
>>>> Can you try these steps on a clean installation of Kafka? It works for
>>>> me, so hopefully it will work for you. And then you can adapt to your
>>>> scenario.
>>>>
>>>> *Create keystores and truststores:*
>>>>
>>>> keytool -genkey -alias kafka -keystore server.keystore.jks -dname
>>>> "CN=KafkaBroker,O=Pivotal,C=UK" -storepass server-keystore-password
>>>> -keypass server-key-password
>>>>
>>>> keytool -exportcert -file server-cert-file -keystore
>>>> server.keystore.jks -alias kafka -storepass server-keystore-password
>>>>
>>>> keytool -importcert -file server-cert-file -keystore
>>>> server.truststore.jks -alias kafka -storepass server-truststore-password
>>>> -noprompt
>>>>
>>>> keytool -importcert -file server-cert-file -keystore
>>>> client.truststore.jks -alias kafkaclient -storepass
>>>> client-truststore-password -noprompt
>>>>
>>>>
>>>> keytool -genkey -alias kafkaclient -keystore client.keystore.jks -dname
>>>> "CN=KafkaClient,O=Pivotal,C=UK" -storepass client-keystore-password
>>>> -keypass client-key-password
>>>>
>>>> keytool -exportcert -file client-cert-file -keystore
>>>> client.keystore.jks -alias kafkaclient -storepass client-keystore-password
>>>>
>>>> keytool -importcert -file client-cert-file -keystore
>>>> server.truststore.jks -alias kafkaclient -storepass
>>>> server-truststore-password -noprompt
>>>>
>>>> *Configure broker: Add these lines at the end of your server.properties*
>>>>
>>>> listeners=SSL://:9093
>>>>
>>>> advertised.listeners=SSL://127.0.0.1:9093
>>>>
>>>> ssl.keystore.location=/tmp/acl/server.keystore.jks
>>>>
>>>> ssl.keystore.password=server-keystore-password
>>>>
>>>> ssl.key.password=server-key-password
>>>>
>>>> ssl.truststore.location=/tmp/acl/server.truststore.jks
>>>>
>>>> ssl.truststore.password=server-truststore-password
>>>>
>>>> security.inter.broker.protocol=SSL
>>>>
>>>> security.protocol=SSL
>>>>
>>>> ssl.client.auth=required
>>>>
>>>> allow.everyone.if.no.acl.found=false
>>>>
>>>> authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
>>>>
>>>> super.users=User:CN=KafkaBroker,O=Pivotal,C=UK
>>>>
>>>> *Configure producer: producer.properties*
>>>>
>>>> security.protocol=SSL
>>>>
>>>> ssl.truststore.location=/tmp/acl/client.truststore.jks
>>>>
>>>> ssl.truststore.password=client-truststore-password
>>>>
>>>> ssl.keystore.location=/tmp/acl/client.keystore.jks
>>>>
>>>> ssl.keystore.password=client-keystore-password
>>>>
>>>> ssl.key.password=client-key-password
>>>>
>>>>
>>>> *Configure consumer: consumer.properties*
>>>>
>>>> security.protocol=SSL
>>>>
>>>> ssl.truststore.location=/tmp/acl/client.truststore.jks
>>>>
>>>> ssl.truststore.password=client-truststore-password
>>>>
>>>> ssl.keystore.location=/tmp/acl/client.keystore.jks
>>>>
>>>> ssl.keystore.password=client-keystore-password
>>>>
>>>> ssl.key.password=client-key-password
>>>>
>>>> group.id=testgroup
>>>>
>>>> *Create topic:*
>>>>
>>>> bin/kafka-topics.sh --zookeeper localhost --create --topic testtopic
>>>> --replication-factor 1 --partitions 1
>>>>
>>>>
>>>> *Configure ACLs:*
>>>>
>>>> bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181
>>>> --add --allow-principal "User:CN=KafkaClient,O=Pivotal,C=UK"
>>>> --producer --topic testtopic
>>>>
>>>> bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181
>>>> --add --allow-principal "User:CN=KafkaClient,O=Pivotal,C=UK"
>>>> --consumer --topic testtopic --group test group
>>>>
>>>>
>>>> *Run console producer and type in some messages:*
>>>>
>>>> bin/kafka-console-producer.sh --producer.config
>>>> /tmp/acl/producer.properties --topic testtopic --broker-list
>>>> 127.0.0.1:9093
>>>>
>>>>
>>>> *Run console consumer, you should see messages from above:*
>>>>
>>>> bin/kafka-console-consumer.sh --consumer.config
>>>> /tmp/acl/consumer.properties --topic testtopic --bootstrap-server
>>>> 127.0.0.1:9093 --from-beginning
>>>>
>>>>
>>>>
>>>> On Tue, May 23, 2017 at 12:57 PM, Raghav <ra...@gmail.com> wrote:
>>>>
>>>>> Darshan,
>>>>>
>>>>> I have not yet successfully gotten the ACLs to work in Kafka. I am
>>>>> still
>>>>> looking for help. I will update this email thread if I do find. In
>>>>> case you
>>>>> get it working, please let me know.
>>>>>
>>>>> Thanks.
>>>>>
>>>>> R
>>>>>
>>>>> On Tue, May 23, 2017 at 8:49 AM, Darshan Purandare <
>>>>> purandare.darshan@gmail.com> wrote:
>>>>>
>>>>> > Raghav
>>>>> >
>>>>> > I saw few posts of yours around Kafka ACLs and the problems. I have
>>>>> seen
>>>>> > similar issues where Writer has not been able to write to any topic.
>>>>> I have
>>>>> > seen "leader not available" and sometimes "unknown topic or
>>>>> partition", and
>>>>> > "topic_authorization_failed" error.
>>>>> >
>>>>> > Let me know if you find a valid config that works.
>>>>> >
>>>>> > Thanks.
>>>>> >
>>>>> >
>>>>> >
>>>>> > On Tue, May 23, 2017 at 8:44 AM, Raghav <ra...@gmail.com>
>>>>> wrote:
>>>>> >
>>>>> >> Hello Kafka Users
>>>>> >>
>>>>> >> I am a new Kafka user and trying to make Kafka SSL work with
>>>>> Authorization
>>>>> >> and ACLs. I followed posts from Kafka and Confluent docs exactly to
>>>>> the
>>>>> >> point but my producer cannot write to kafka broker. I get
>>>>> >> "LEADER_NOT_FOUND" errors. And even Consumer throws the same errors.
>>>>> >>
>>>>> >> Can someone please share their config which worked with ACLs.
>>>>> >>
>>>>> >> Here is my config. Please help.
>>>>> >>
>>>>> >> server.properties config
>>>>> >> ------------------------------------------------------------
>>>>> >> ------------------------------------------------
>>>>> >> broker.id=0
>>>>> >> auto.create.topics.enable=true
>>>>> >> delete.topic.enable=true
>>>>> >>
>>>>> >> listeners=PLAINTEXT://kafka1.example.com:9092
>>>>> >> <http://kafka-dev1.example.com:9092/>,SSL://kafka1.example.com:9093
>>>>> >> <http://kafka-dev1.example.com:9093/>
>>>>> >> host.name=kafka1.example.com <http://kafka-dev1.example.com/>
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> ssl.keystore.location=/var/private/kafka1.keystore.jks
>>>>> >> ssl.keystore.password=12345678
>>>>> >> ssl.key.password=12345678
>>>>> >>
>>>>> >> ssl.truststore.location=/var/private/kafka1.truststore.jks
>>>>> >> ssl.truststore.password=12345678
>>>>> >>
>>>>> >> ssl.client.auth=required
>>>>> >> ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
>>>>> >> ssl.keystore.type=JKS
>>>>> >> ssl.truststore.type=JKS
>>>>> >>
>>>>> >> authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
>>>>> >> ------------------------------------------------------------
>>>>> >> ------------------------------------------------
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> Here is producer Config(producer.properties)
>>>>> >> ------------------------------------------------------------
>>>>> >> ------------------------------------------------
>>>>> >> security.protocol=SSL
>>>>> >> ssl.truststore.location=/var/private/kafka2.truststore.jks
>>>>> >> ssl.truststore.password=12345678
>>>>> >>
>>>>> >> ssl.keystore.location=/var/private/kafka2.keystore.jks
>>>>> >> ssl.keystore.password=12345678
>>>>> >> ssl.key.password=12345678
>>>>> >>
>>>>> >> ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
>>>>> >> ssl.truststore.type=JKS
>>>>> >> ssl.keystore.type=JKS
>>>>> >>
>>>>> >> ------------------------------------------------------------
>>>>> >> ------------------------------------------------
>>>>> >>
>>>>> >>
>>>>> >> Raqhav
>>>>> >>
>>>>> >
>>>>> >
>>>>>
>>>>>
>>>>> --
>>>>> Raghav
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Raghav
>>>
>>
>>
>
>
> --
> Raghav
>