You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Raghu B <ra...@gmail.com> on 2016/12/14 00:12:13 UTC

Kafka ACL's with SSL Protocol is not working

Hi All,

I am trying to enable ACL's in my Kafka cluster with along with SSL
Protocol.

I tried with each and every parameters but no luck, so I need help to
enable the SSL(without Kerberos) and I am attaching all the configuration
details in this.

Kindly Help me.


*I tested SSL without ACL, it worked fine
(listeners=SSL://10.247.195.122:9093 <http://10.247.195.122:9093>)*


*This is my Kafka server properties file:*

*############################# ACL SETTINGS #############################*

*auto.create.topics.enable=true*

*authorizer.class.name
<http://authorizer.class.name>=kafka.security.auth.SimpleAclAuthorizer*

*security.inter.broker.protocol=SSL*

*#allow.everyone.if.no.acl.found=true*

*#principal.builder.class=CustomizedPrincipalBuilderClass*

*#super.users=User:"CN=writeuser,OU=Unknown,O=Unknown,L=Unknown,ST=Unknown,C=Unknown"*

*#super.users=User:Raghu;User:Admin*

*#offsets.storage=kafka*

*#dual.commit.enabled=true*

*listeners=SSL://10.247.195.122:9093 <http://10.247.195.122:9093>*

*#listeners=PLAINTEXT://10.247.195.122:9092 <http://10.247.195.122:9092>*

*#listeners=PLAINTEXT://10.247.195.122:9092
<http://10.247.195.122:9092>,SSL://10.247.195.122:9093
<http://10.247.195.122:9093>*

*#advertised.listeners=PLAINTEXT://10.247.195.122:9092
<http://10.247.195.122:9092>*


*
ssl.keystore.location=/home/raghu/kafka/security/server.keystore.jks*

*        ssl.keystore.password=123456*

*        ssl.key.password=123456*

*
ssl.truststore.location=/home/raghu/kafka/security/server.truststore.jks*

*        ssl.truststore.password=123456*



*Set the ACL from Authorizer CLI:*

> *bin/kafka-acls.sh --authorizer-properties
zookeeper.connect=10.247.195.122:2181 <http://10.247.195.122:2181> --list
--topic ssltopic*

*Current ACLs for resource `Topic:ssltopic`: *

*  User:CN=writeuser, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown,
C=Unknown has Allow permission for operations: Write from hosts: * *


*XXXWMXXX-7:kafka_2.11-0.10.1.0 rbaddam$ bin/kafka-console-producer.sh
--broker-list 10.247.195.122:9093 <http://10.247.195.122:9093> --topic
ssltopic --producer.config client-ssl.properties*


*[2016-12-13 14:53:45,839] WARN Error while fetching metadata with
correlation id 0 : {ssltopic=UNKNOWN_TOPIC_OR_PARTITION}
(org.apache.kafka.clients.NetworkClient)*

*[2016-12-13 14:53:45,984] WARN Error while fetching metadata with
correlation id 1 : {ssltopic=UNKNOWN_TOPIC_OR_PARTITION}
(org.apache.kafka.clients.NetworkClient)*


*XXXWMXXX-7:kafka_2.11-0.10.1.0 rbaddam$ cat client-ssl.properties*

*#group.id <http://group.id>=sslgroup*

*security.protocol=SSL*

*ssl.truststore.location=/Users/rbaddam/Desktop/Dev/kafka_2.11-0.10.1.0/ssl/client.truststore.jks*

*ssl.truststore.password=123456*

* #Configure Below if you use Client Auth*


*ssl.keystore.location=/Users/rbaddam/Desktop/Dev/kafka_2.11-0.10.1.0/ssl/client.keystore.jks*

*ssl.keystore.password=123456*

*ssl.key.password=123456*


*XXXWMXXX-7:kafka_2.11-0.10.1.0 rbaddam$ bin/kafka-console-consumer.sh
--bootstrap-server 10.247.195.122:9093 <http://10.247.195.122:9093>
--new-consumer --consumer.config client-ssl.properties --topic ssltopic
--from-beginning*

*[2016-12-13 14:53:28,817] WARN Error while fetching metadata with
correlation id 1 : {ssltopic=UNKNOWN_TOPIC_OR_PARTITION}
(org.apache.kafka.clients.NetworkClient)*

*[2016-12-13 14:53:28,819] ERROR Unknown error when running consumer:
(kafka.tools.ConsoleConsumer$)*

*org.apache.kafka.common.errors.GroupAuthorizationException: Not authorized
to access group: console-consumer-52826*


Thanks in advance,

Raghu - raghu98499@gmail.com

Re: Kafka ACL's with SSL Protocol is not working

Posted by Rajini Sivaram <ra...@gmail.com>.
Raghu,

Only the principal used for inter broker communication needs to be a super
user. For other users, you can set ACLs based on their role. You will need
different keystores for broker and clients with different principals so
that you can configure different permissions. You can configure User:Broker
as superuser and User:User_1 with produce permissions and User:User_2 with
consume permissions.

On Mon, Dec 19, 2016 at 8:10 PM, Raghu B <ra...@gmail.com> wrote:

> Thanks Rajani for the above Info but I want to restrict a user from
> performing all the operations (I think that defines ACL), I just want
> User_1 to produce messages and User_2 to consume messages.
>
> How can we achieve that.
>
> Thanks in advance
>
> On Mon, Dec 19, 2016 at 3:13 AM, Rajini Sivaram <ra...@gmail.com>
> wrote:
>
>> Raghu,
>>
>> It could be because the principal used for inter broker communication
>> doesn't have all the necessary permissions. If you are using PLAINTEXT for
>> inter-broker, the principal is ANONYMOUS, if using SSL, it would be
>> similar
>> to the one you are setting for client. You can configure broker principal
>> as super.users to give full access.
>>
>> On Fri, Dec 16, 2016 at 10:16 PM, Raghu B <ra...@gmail.com> wrote:
>>
>> > Thank you Rajani, your suggestion is really helpful.
>> >
>> >
>> > [2016-12-16 21:55:36,720] DEBUG Principal =
>> > User:CN=writeuser,OU=Unknown,O=Unknown,L=Unknown,ST=Unknown,C=Unknown
>> is
>> > Allowed Operation = Create from host = 172.28.89.63 on resource =
>> > Cluster:kafka-cluster (kafka.authorizer.logger)
>> >
>> > Finally I am getting the user as exactly what I set in my SSL-Cert (Not
>> > Anonymous).
>> >
>> > But, I am getting another Error i.e
>> >
>> >
>> > [2016-12-16 13:55:36,449] WARN Error while fetching metadata with
>> > correlation id 45 : {my-ssl-topic=LEADER_NOT_AVAILABLE}
>> > (org.apache.kafka.clients.NetworkClient)
>> > [2016-12-16 13:55:36,609] WARN Error while fetching metadata with
>> > correlation id 46 : {my-ssl-topic=LEADER_NOT_AVAILABLE}
>> > (org.apache.kafka.clients.NetworkClient)
>> > [2016-12-16 13:55:36,766] WARN Error while fetching metadata with
>> > correlation id 47 : {my-ssl-topic=LEADER_NOT_AVAILABLE}
>> > (org.apache.kafka.clients.NetworkClient)
>> >
>> >
>> > I created the topic and my kafka node is working without any issues (I
>> > restarted several time)
>> >
>> > [raghu@Kafka-238343-1-33109167 kafka_2.11-0.10.1.0]$
>> *bin/kafka-topics.sh
>> > --describe --zookeeper localhost:2181 --topic my-ssl-topic*
>> >
>> > Topic:my-ssl-topic PartitionCount:1 ReplicationFactor:1 Configs:
>> > Topic: my-ssl-topic Partition: 0 Leader: 0 Replicas: 0 Isr: 0
>> >
>> > Thanks in advance,
>> > Raghu
>> >
>> >
>> > On Fri, Dec 16, 2016 at 1:30 AM, Rajini Sivaram <rs...@pivotal.io>
>> > wrote:
>> >
>> > > You need to set ssl.client.auth="required" in server.properties.
>> > >
>> > > Regards,
>> > >
>> > > Rajini
>> > >
>> > > On Wed, Dec 14, 2016 at 12:12 AM, Raghu B <ra...@gmail.com>
>> wrote:
>> > >
>> > > > Hi All,
>> > > >
>> > > > I am trying to enable ACL's in my Kafka cluster with along with SSL
>> > > > Protocol.
>> > > >
>> > > > I tried with each and every parameters but no luck, so I need help
>> to
>> > > > enable the SSL(without Kerberos) and I am attaching all the
>> > configuration
>> > > > details in this.
>> > > >
>> > > > Kindly Help me.
>> > > >
>> > > >
>> > > > *I tested SSL without ACL, it worked fine
>> > > > (listeners=SSL://10.247.195.122:9093 <http://10.247.195.122:9093>)*
>> > > >
>> > > >
>> > > > *This is my Kafka server properties file:*
>> > > >
>> > > > *############################# ACL SETTINGS
>> > > #############################*
>> > > >
>> > > > *auto.create.topics.enable=true*
>> > > >
>> > > > *authorizer.class.name
>> > > > <http://authorizer.class.name>=kafka.security.auth.
>> > SimpleAclAuthorizer*
>> > > >
>> > > > *security.inter.broker.protocol=SSL*
>> > > >
>> > > > *#allow.everyone.if.no.acl.found=true*
>> > > >
>> > > > *#principal.builder.class=CustomizedPrincipalBuilderClass*
>> > > >
>> > > > *#super.users=User:"CN=writeuser,OU=Unknown,O=
>> > > > Unknown,L=Unknown,ST=Unknown,C=Unknown"*
>> > > >
>> > > > *#super.users=User:Raghu;User:Admin*
>> > > >
>> > > > *#offsets.storage=kafka*
>> > > >
>> > > > *#dual.commit.enabled=true*
>> > > >
>> > > > *listeners=SSL://10.247.195.122:9093 <http://10.247.195.122:9093>*
>> > > >
>> > > > *#listeners=PLAINTEXT://10.247.195.122:9092 <
>> > http://10.247.195.122:9092
>> > > >*
>> > > >
>> > > > *#listeners=PLAINTEXT://10.247.195.122:9092
>> > > > <http://10.247.195.122:9092>,SSL://10.247.195.122:9093
>> > > > <http://10.247.195.122:9093>*
>> > > >
>> > > > *#advertised.listeners=PLAINTEXT://10.247.195.122:9092
>> > > > <http://10.247.195.122:9092>*
>> > > >
>> > > >
>> > > > *
>> > > > ssl.keystore.location=/home/raghu/kafka/security/server.keys
>> tore.jks*
>> > > >
>> > > > *        ssl.keystore.password=123456*
>> > > >
>> > > > *        ssl.key.password=123456*
>> > > >
>> > > > *
>> > > > ssl.truststore.location=/home/raghu/kafka/security/server.
>> > > truststore.jks*
>> > > >
>> > > > *        ssl.truststore.password=123456*
>> > > >
>> > > >
>> > > >
>> > > > *Set the ACL from Authorizer CLI:*
>> > > >
>> > > > > *bin/kafka-acls.sh --authorizer-properties
>> > > > zookeeper.connect=10.247.195.122:2181 <http://10.247.195.122:2181>
>> > > --list
>> > > > --topic ssltopic*
>> > > >
>> > > > *Current ACLs for resource `Topic:ssltopic`: *
>> > > >
>> > > > *  User:CN=writeuser, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown,
>> > > > C=Unknown has Allow permission for operations: Write from hosts: * *
>> > > >
>> > > >
>> > > > *XXXWMXXX-7:kafka_2.11-0.10.1.0 rbaddam$
>> bin/kafka-console-producer.sh
>> > > > --broker-list 10.247.195.122:9093 <http://10.247.195.122:9093>
>> --topic
>> > > > ssltopic --producer.config client-ssl.properties*
>> > > >
>> > > >
>> > > > *[2016-12-13 14:53:45,839] WARN Error while fetching metadata with
>> > > > correlation id 0 : {ssltopic=UNKNOWN_TOPIC_OR_PARTITION}
>> > > > (org.apache.kafka.clients.NetworkClient)*
>> > > >
>> > > > *[2016-12-13 14:53:45,984] WARN Error while fetching metadata with
>> > > > correlation id 1 : {ssltopic=UNKNOWN_TOPIC_OR_PARTITION}
>> > > > (org.apache.kafka.clients.NetworkClient)*
>> > > >
>> > > >
>> > > > *XXXWMXXX-7:kafka_2.11-0.10.1.0 rbaddam$ cat client-ssl.properties*
>> > > >
>> > > > *#group.id <http://group.id>=sslgroup*
>> > > >
>> > > > *security.protocol=SSL*
>> > > >
>> > > > *ssl.truststore.location=/Users/rbaddam/Desktop/Dev/
>> > > > kafka_2.11-0.10.1.0/ssl/client.truststore.jks*
>> > > >
>> > > > *ssl.truststore.password=123456*
>> > > >
>> > > > * #Configure Below if you use Client Auth*
>> > > >
>> > > >
>> > > > *ssl.keystore.location=/Users/rbaddam/Desktop/Dev/kafka_2.
>> > > > 11-0.10.1.0/ssl/client.keystore.jks*
>> > > >
>> > > > *ssl.keystore.password=123456*
>> > > >
>> > > > *ssl.key.password=123456*
>> > > >
>> > > >
>> > > > *XXXWMXXX-7:kafka_2.11-0.10.1.0 rbaddam$
>> bin/kafka-console-consumer.sh
>> > > > --bootstrap-server 10.247.195.122:9093 <http://10.247.195.122:9093>
>> > > > --new-consumer --consumer.config client-ssl.properties --topic
>> ssltopic
>> > > > --from-beginning*
>> > > >
>> > > > *[2016-12-13 14:53:28,817] WARN Error while fetching metadata with
>> > > > correlation id 1 : {ssltopic=UNKNOWN_TOPIC_OR_PARTITION}
>> > > > (org.apache.kafka.clients.NetworkClient)*
>> > > >
>> > > > *[2016-12-13 14:53:28,819] ERROR Unknown error when running
>> consumer:
>> > > > (kafka.tools.ConsoleConsumer$)*
>> > > >
>> > > > *org.apache.kafka.common.errors.GroupAuthorizationException: Not
>> > > > authorized
>> > > > to access group: console-consumer-52826*
>> > > >
>> > > >
>> > > > Thanks in advance,
>> > > >
>> > > > Raghu - raghu98499@gmail.com
>> > > >
>> > >
>> >
>>
>>
>>
>> --
>> Regards,
>>
>> Rajini
>>
>
>

Re: Kafka ACL's with SSL Protocol is not working

Posted by Raghu B <ra...@gmail.com>.
Thanks Rajani for the above Info but I want to restrict a user from
performing all the operations (I think that defines ACL), I just want
User_1 to produce messages and User_2 to consume messages.

How can we achieve that.

Thanks in advance

On Mon, Dec 19, 2016 at 3:13 AM, Rajini Sivaram <ra...@gmail.com>
wrote:

> Raghu,
>
> It could be because the principal used for inter broker communication
> doesn't have all the necessary permissions. If you are using PLAINTEXT for
> inter-broker, the principal is ANONYMOUS, if using SSL, it would be similar
> to the one you are setting for client. You can configure broker principal
> as super.users to give full access.
>
> On Fri, Dec 16, 2016 at 10:16 PM, Raghu B <ra...@gmail.com> wrote:
>
> > Thank you Rajani, your suggestion is really helpful.
> >
> >
> > [2016-12-16 21:55:36,720] DEBUG Principal =
> > User:CN=writeuser,OU=Unknown,O=Unknown,L=Unknown,ST=Unknown,C=Unknown is
> > Allowed Operation = Create from host = 172.28.89.63 on resource =
> > Cluster:kafka-cluster (kafka.authorizer.logger)
> >
> > Finally I am getting the user as exactly what I set in my SSL-Cert (Not
> > Anonymous).
> >
> > But, I am getting another Error i.e
> >
> >
> > [2016-12-16 13:55:36,449] WARN Error while fetching metadata with
> > correlation id 45 : {my-ssl-topic=LEADER_NOT_AVAILABLE}
> > (org.apache.kafka.clients.NetworkClient)
> > [2016-12-16 13:55:36,609] WARN Error while fetching metadata with
> > correlation id 46 : {my-ssl-topic=LEADER_NOT_AVAILABLE}
> > (org.apache.kafka.clients.NetworkClient)
> > [2016-12-16 13:55:36,766] WARN Error while fetching metadata with
> > correlation id 47 : {my-ssl-topic=LEADER_NOT_AVAILABLE}
> > (org.apache.kafka.clients.NetworkClient)
> >
> >
> > I created the topic and my kafka node is working without any issues (I
> > restarted several time)
> >
> > [raghu@Kafka-238343-1-33109167 kafka_2.11-0.10.1.0]$
> *bin/kafka-topics.sh
> > --describe --zookeeper localhost:2181 --topic my-ssl-topic*
> >
> > Topic:my-ssl-topic PartitionCount:1 ReplicationFactor:1 Configs:
> > Topic: my-ssl-topic Partition: 0 Leader: 0 Replicas: 0 Isr: 0
> >
> > Thanks in advance,
> > Raghu
> >
> >
> > On Fri, Dec 16, 2016 at 1:30 AM, Rajini Sivaram <rs...@pivotal.io>
> > wrote:
> >
> > > You need to set ssl.client.auth="required" in server.properties.
> > >
> > > Regards,
> > >
> > > Rajini
> > >
> > > On Wed, Dec 14, 2016 at 12:12 AM, Raghu B <ra...@gmail.com>
> wrote:
> > >
> > > > Hi All,
> > > >
> > > > I am trying to enable ACL's in my Kafka cluster with along with SSL
> > > > Protocol.
> > > >
> > > > I tried with each and every parameters but no luck, so I need help to
> > > > enable the SSL(without Kerberos) and I am attaching all the
> > configuration
> > > > details in this.
> > > >
> > > > Kindly Help me.
> > > >
> > > >
> > > > *I tested SSL without ACL, it worked fine
> > > > (listeners=SSL://10.247.195.122:9093 <http://10.247.195.122:9093>)*
> > > >
> > > >
> > > > *This is my Kafka server properties file:*
> > > >
> > > > *############################# ACL SETTINGS
> > > #############################*
> > > >
> > > > *auto.create.topics.enable=true*
> > > >
> > > > *authorizer.class.name
> > > > <http://authorizer.class.name>=kafka.security.auth.
> > SimpleAclAuthorizer*
> > > >
> > > > *security.inter.broker.protocol=SSL*
> > > >
> > > > *#allow.everyone.if.no.acl.found=true*
> > > >
> > > > *#principal.builder.class=CustomizedPrincipalBuilderClass*
> > > >
> > > > *#super.users=User:"CN=writeuser,OU=Unknown,O=
> > > > Unknown,L=Unknown,ST=Unknown,C=Unknown"*
> > > >
> > > > *#super.users=User:Raghu;User:Admin*
> > > >
> > > > *#offsets.storage=kafka*
> > > >
> > > > *#dual.commit.enabled=true*
> > > >
> > > > *listeners=SSL://10.247.195.122:9093 <http://10.247.195.122:9093>*
> > > >
> > > > *#listeners=PLAINTEXT://10.247.195.122:9092 <
> > http://10.247.195.122:9092
> > > >*
> > > >
> > > > *#listeners=PLAINTEXT://10.247.195.122:9092
> > > > <http://10.247.195.122:9092>,SSL://10.247.195.122:9093
> > > > <http://10.247.195.122:9093>*
> > > >
> > > > *#advertised.listeners=PLAINTEXT://10.247.195.122:9092
> > > > <http://10.247.195.122:9092>*
> > > >
> > > >
> > > > *
> > > > ssl.keystore.location=/home/raghu/kafka/security/server.
> keystore.jks*
> > > >
> > > > *        ssl.keystore.password=123456*
> > > >
> > > > *        ssl.key.password=123456*
> > > >
> > > > *
> > > > ssl.truststore.location=/home/raghu/kafka/security/server.
> > > truststore.jks*
> > > >
> > > > *        ssl.truststore.password=123456*
> > > >
> > > >
> > > >
> > > > *Set the ACL from Authorizer CLI:*
> > > >
> > > > > *bin/kafka-acls.sh --authorizer-properties
> > > > zookeeper.connect=10.247.195.122:2181 <http://10.247.195.122:2181>
> > > --list
> > > > --topic ssltopic*
> > > >
> > > > *Current ACLs for resource `Topic:ssltopic`: *
> > > >
> > > > *  User:CN=writeuser, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown,
> > > > C=Unknown has Allow permission for operations: Write from hosts: * *
> > > >
> > > >
> > > > *XXXWMXXX-7:kafka_2.11-0.10.1.0 rbaddam$
> bin/kafka-console-producer.sh
> > > > --broker-list 10.247.195.122:9093 <http://10.247.195.122:9093>
> --topic
> > > > ssltopic --producer.config client-ssl.properties*
> > > >
> > > >
> > > > *[2016-12-13 14:53:45,839] WARN Error while fetching metadata with
> > > > correlation id 0 : {ssltopic=UNKNOWN_TOPIC_OR_PARTITION}
> > > > (org.apache.kafka.clients.NetworkClient)*
> > > >
> > > > *[2016-12-13 14:53:45,984] WARN Error while fetching metadata with
> > > > correlation id 1 : {ssltopic=UNKNOWN_TOPIC_OR_PARTITION}
> > > > (org.apache.kafka.clients.NetworkClient)*
> > > >
> > > >
> > > > *XXXWMXXX-7:kafka_2.11-0.10.1.0 rbaddam$ cat client-ssl.properties*
> > > >
> > > > *#group.id <http://group.id>=sslgroup*
> > > >
> > > > *security.protocol=SSL*
> > > >
> > > > *ssl.truststore.location=/Users/rbaddam/Desktop/Dev/
> > > > kafka_2.11-0.10.1.0/ssl/client.truststore.jks*
> > > >
> > > > *ssl.truststore.password=123456*
> > > >
> > > > * #Configure Below if you use Client Auth*
> > > >
> > > >
> > > > *ssl.keystore.location=/Users/rbaddam/Desktop/Dev/kafka_2.
> > > > 11-0.10.1.0/ssl/client.keystore.jks*
> > > >
> > > > *ssl.keystore.password=123456*
> > > >
> > > > *ssl.key.password=123456*
> > > >
> > > >
> > > > *XXXWMXXX-7:kafka_2.11-0.10.1.0 rbaddam$
> bin/kafka-console-consumer.sh
> > > > --bootstrap-server 10.247.195.122:9093 <http://10.247.195.122:9093>
> > > > --new-consumer --consumer.config client-ssl.properties --topic
> ssltopic
> > > > --from-beginning*
> > > >
> > > > *[2016-12-13 14:53:28,817] WARN Error while fetching metadata with
> > > > correlation id 1 : {ssltopic=UNKNOWN_TOPIC_OR_PARTITION}
> > > > (org.apache.kafka.clients.NetworkClient)*
> > > >
> > > > *[2016-12-13 14:53:28,819] ERROR Unknown error when running consumer:
> > > > (kafka.tools.ConsoleConsumer$)*
> > > >
> > > > *org.apache.kafka.common.errors.GroupAuthorizationException: Not
> > > > authorized
> > > > to access group: console-consumer-52826*
> > > >
> > > >
> > > > Thanks in advance,
> > > >
> > > > Raghu - raghu98499@gmail.com
> > > >
> > >
> >
>
>
>
> --
> Regards,
>
> Rajini
>

Re: Kafka ACL's with SSL Protocol is not working

Posted by Rajini Sivaram <ra...@gmail.com>.
Raghu,

It could be because the principal used for inter broker communication
doesn't have all the necessary permissions. If you are using PLAINTEXT for
inter-broker, the principal is ANONYMOUS, if using SSL, it would be similar
to the one you are setting for client. You can configure broker principal
as super.users to give full access.

On Fri, Dec 16, 2016 at 10:16 PM, Raghu B <ra...@gmail.com> wrote:

> Thank you Rajani, your suggestion is really helpful.
>
>
> [2016-12-16 21:55:36,720] DEBUG Principal =
> User:CN=writeuser,OU=Unknown,O=Unknown,L=Unknown,ST=Unknown,C=Unknown is
> Allowed Operation = Create from host = 172.28.89.63 on resource =
> Cluster:kafka-cluster (kafka.authorizer.logger)
>
> Finally I am getting the user as exactly what I set in my SSL-Cert (Not
> Anonymous).
>
> But, I am getting another Error i.e
>
>
> [2016-12-16 13:55:36,449] WARN Error while fetching metadata with
> correlation id 45 : {my-ssl-topic=LEADER_NOT_AVAILABLE}
> (org.apache.kafka.clients.NetworkClient)
> [2016-12-16 13:55:36,609] WARN Error while fetching metadata with
> correlation id 46 : {my-ssl-topic=LEADER_NOT_AVAILABLE}
> (org.apache.kafka.clients.NetworkClient)
> [2016-12-16 13:55:36,766] WARN Error while fetching metadata with
> correlation id 47 : {my-ssl-topic=LEADER_NOT_AVAILABLE}
> (org.apache.kafka.clients.NetworkClient)
>
>
> I created the topic and my kafka node is working without any issues (I
> restarted several time)
>
> [raghu@Kafka-238343-1-33109167 kafka_2.11-0.10.1.0]$ *bin/kafka-topics.sh
> --describe --zookeeper localhost:2181 --topic my-ssl-topic*
>
> Topic:my-ssl-topic PartitionCount:1 ReplicationFactor:1 Configs:
> Topic: my-ssl-topic Partition: 0 Leader: 0 Replicas: 0 Isr: 0
>
> Thanks in advance,
> Raghu
>
>
> On Fri, Dec 16, 2016 at 1:30 AM, Rajini Sivaram <rs...@pivotal.io>
> wrote:
>
> > You need to set ssl.client.auth="required" in server.properties.
> >
> > Regards,
> >
> > Rajini
> >
> > On Wed, Dec 14, 2016 at 12:12 AM, Raghu B <ra...@gmail.com> wrote:
> >
> > > Hi All,
> > >
> > > I am trying to enable ACL's in my Kafka cluster with along with SSL
> > > Protocol.
> > >
> > > I tried with each and every parameters but no luck, so I need help to
> > > enable the SSL(without Kerberos) and I am attaching all the
> configuration
> > > details in this.
> > >
> > > Kindly Help me.
> > >
> > >
> > > *I tested SSL without ACL, it worked fine
> > > (listeners=SSL://10.247.195.122:9093 <http://10.247.195.122:9093>)*
> > >
> > >
> > > *This is my Kafka server properties file:*
> > >
> > > *############################# ACL SETTINGS
> > #############################*
> > >
> > > *auto.create.topics.enable=true*
> > >
> > > *authorizer.class.name
> > > <http://authorizer.class.name>=kafka.security.auth.
> SimpleAclAuthorizer*
> > >
> > > *security.inter.broker.protocol=SSL*
> > >
> > > *#allow.everyone.if.no.acl.found=true*
> > >
> > > *#principal.builder.class=CustomizedPrincipalBuilderClass*
> > >
> > > *#super.users=User:"CN=writeuser,OU=Unknown,O=
> > > Unknown,L=Unknown,ST=Unknown,C=Unknown"*
> > >
> > > *#super.users=User:Raghu;User:Admin*
> > >
> > > *#offsets.storage=kafka*
> > >
> > > *#dual.commit.enabled=true*
> > >
> > > *listeners=SSL://10.247.195.122:9093 <http://10.247.195.122:9093>*
> > >
> > > *#listeners=PLAINTEXT://10.247.195.122:9092 <
> http://10.247.195.122:9092
> > >*
> > >
> > > *#listeners=PLAINTEXT://10.247.195.122:9092
> > > <http://10.247.195.122:9092>,SSL://10.247.195.122:9093
> > > <http://10.247.195.122:9093>*
> > >
> > > *#advertised.listeners=PLAINTEXT://10.247.195.122:9092
> > > <http://10.247.195.122:9092>*
> > >
> > >
> > > *
> > > ssl.keystore.location=/home/raghu/kafka/security/server.keystore.jks*
> > >
> > > *        ssl.keystore.password=123456*
> > >
> > > *        ssl.key.password=123456*
> > >
> > > *
> > > ssl.truststore.location=/home/raghu/kafka/security/server.
> > truststore.jks*
> > >
> > > *        ssl.truststore.password=123456*
> > >
> > >
> > >
> > > *Set the ACL from Authorizer CLI:*
> > >
> > > > *bin/kafka-acls.sh --authorizer-properties
> > > zookeeper.connect=10.247.195.122:2181 <http://10.247.195.122:2181>
> > --list
> > > --topic ssltopic*
> > >
> > > *Current ACLs for resource `Topic:ssltopic`: *
> > >
> > > *  User:CN=writeuser, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown,
> > > C=Unknown has Allow permission for operations: Write from hosts: * *
> > >
> > >
> > > *XXXWMXXX-7:kafka_2.11-0.10.1.0 rbaddam$ bin/kafka-console-producer.sh
> > > --broker-list 10.247.195.122:9093 <http://10.247.195.122:9093> --topic
> > > ssltopic --producer.config client-ssl.properties*
> > >
> > >
> > > *[2016-12-13 14:53:45,839] WARN Error while fetching metadata with
> > > correlation id 0 : {ssltopic=UNKNOWN_TOPIC_OR_PARTITION}
> > > (org.apache.kafka.clients.NetworkClient)*
> > >
> > > *[2016-12-13 14:53:45,984] WARN Error while fetching metadata with
> > > correlation id 1 : {ssltopic=UNKNOWN_TOPIC_OR_PARTITION}
> > > (org.apache.kafka.clients.NetworkClient)*
> > >
> > >
> > > *XXXWMXXX-7:kafka_2.11-0.10.1.0 rbaddam$ cat client-ssl.properties*
> > >
> > > *#group.id <http://group.id>=sslgroup*
> > >
> > > *security.protocol=SSL*
> > >
> > > *ssl.truststore.location=/Users/rbaddam/Desktop/Dev/
> > > kafka_2.11-0.10.1.0/ssl/client.truststore.jks*
> > >
> > > *ssl.truststore.password=123456*
> > >
> > > * #Configure Below if you use Client Auth*
> > >
> > >
> > > *ssl.keystore.location=/Users/rbaddam/Desktop/Dev/kafka_2.
> > > 11-0.10.1.0/ssl/client.keystore.jks*
> > >
> > > *ssl.keystore.password=123456*
> > >
> > > *ssl.key.password=123456*
> > >
> > >
> > > *XXXWMXXX-7:kafka_2.11-0.10.1.0 rbaddam$ bin/kafka-console-consumer.sh
> > > --bootstrap-server 10.247.195.122:9093 <http://10.247.195.122:9093>
> > > --new-consumer --consumer.config client-ssl.properties --topic ssltopic
> > > --from-beginning*
> > >
> > > *[2016-12-13 14:53:28,817] WARN Error while fetching metadata with
> > > correlation id 1 : {ssltopic=UNKNOWN_TOPIC_OR_PARTITION}
> > > (org.apache.kafka.clients.NetworkClient)*
> > >
> > > *[2016-12-13 14:53:28,819] ERROR Unknown error when running consumer:
> > > (kafka.tools.ConsoleConsumer$)*
> > >
> > > *org.apache.kafka.common.errors.GroupAuthorizationException: Not
> > > authorized
> > > to access group: console-consumer-52826*
> > >
> > >
> > > Thanks in advance,
> > >
> > > Raghu - raghu98499@gmail.com
> > >
> >
>



-- 
Regards,

Rajini

Re: Kafka ACL's with SSL Protocol is not working

Posted by Raghu B <ra...@gmail.com>.
Thank you Rajani, your suggestion is really helpful.


[2016-12-16 21:55:36,720] DEBUG Principal =
User:CN=writeuser,OU=Unknown,O=Unknown,L=Unknown,ST=Unknown,C=Unknown is
Allowed Operation = Create from host = 172.28.89.63 on resource =
Cluster:kafka-cluster (kafka.authorizer.logger)

Finally I am getting the user as exactly what I set in my SSL-Cert (Not
Anonymous).

But, I am getting another Error i.e


[2016-12-16 13:55:36,449] WARN Error while fetching metadata with
correlation id 45 : {my-ssl-topic=LEADER_NOT_AVAILABLE}
(org.apache.kafka.clients.NetworkClient)
[2016-12-16 13:55:36,609] WARN Error while fetching metadata with
correlation id 46 : {my-ssl-topic=LEADER_NOT_AVAILABLE}
(org.apache.kafka.clients.NetworkClient)
[2016-12-16 13:55:36,766] WARN Error while fetching metadata with
correlation id 47 : {my-ssl-topic=LEADER_NOT_AVAILABLE}
(org.apache.kafka.clients.NetworkClient)


I created the topic and my kafka node is working without any issues (I
restarted several time)

[raghu@Kafka-238343-1-33109167 kafka_2.11-0.10.1.0]$ *bin/kafka-topics.sh
--describe --zookeeper localhost:2181 --topic my-ssl-topic*

Topic:my-ssl-topic PartitionCount:1 ReplicationFactor:1 Configs:
Topic: my-ssl-topic Partition: 0 Leader: 0 Replicas: 0 Isr: 0

Thanks in advance,
Raghu


On Fri, Dec 16, 2016 at 1:30 AM, Rajini Sivaram <rs...@pivotal.io> wrote:

> You need to set ssl.client.auth="required" in server.properties.
>
> Regards,
>
> Rajini
>
> On Wed, Dec 14, 2016 at 12:12 AM, Raghu B <ra...@gmail.com> wrote:
>
> > Hi All,
> >
> > I am trying to enable ACL's in my Kafka cluster with along with SSL
> > Protocol.
> >
> > I tried with each and every parameters but no luck, so I need help to
> > enable the SSL(without Kerberos) and I am attaching all the configuration
> > details in this.
> >
> > Kindly Help me.
> >
> >
> > *I tested SSL without ACL, it worked fine
> > (listeners=SSL://10.247.195.122:9093 <http://10.247.195.122:9093>)*
> >
> >
> > *This is my Kafka server properties file:*
> >
> > *############################# ACL SETTINGS
> #############################*
> >
> > *auto.create.topics.enable=true*
> >
> > *authorizer.class.name
> > <http://authorizer.class.name>=kafka.security.auth.SimpleAclAuthorizer*
> >
> > *security.inter.broker.protocol=SSL*
> >
> > *#allow.everyone.if.no.acl.found=true*
> >
> > *#principal.builder.class=CustomizedPrincipalBuilderClass*
> >
> > *#super.users=User:"CN=writeuser,OU=Unknown,O=
> > Unknown,L=Unknown,ST=Unknown,C=Unknown"*
> >
> > *#super.users=User:Raghu;User:Admin*
> >
> > *#offsets.storage=kafka*
> >
> > *#dual.commit.enabled=true*
> >
> > *listeners=SSL://10.247.195.122:9093 <http://10.247.195.122:9093>*
> >
> > *#listeners=PLAINTEXT://10.247.195.122:9092 <http://10.247.195.122:9092
> >*
> >
> > *#listeners=PLAINTEXT://10.247.195.122:9092
> > <http://10.247.195.122:9092>,SSL://10.247.195.122:9093
> > <http://10.247.195.122:9093>*
> >
> > *#advertised.listeners=PLAINTEXT://10.247.195.122:9092
> > <http://10.247.195.122:9092>*
> >
> >
> > *
> > ssl.keystore.location=/home/raghu/kafka/security/server.keystore.jks*
> >
> > *        ssl.keystore.password=123456*
> >
> > *        ssl.key.password=123456*
> >
> > *
> > ssl.truststore.location=/home/raghu/kafka/security/server.
> truststore.jks*
> >
> > *        ssl.truststore.password=123456*
> >
> >
> >
> > *Set the ACL from Authorizer CLI:*
> >
> > > *bin/kafka-acls.sh --authorizer-properties
> > zookeeper.connect=10.247.195.122:2181 <http://10.247.195.122:2181>
> --list
> > --topic ssltopic*
> >
> > *Current ACLs for resource `Topic:ssltopic`: *
> >
> > *  User:CN=writeuser, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown,
> > C=Unknown has Allow permission for operations: Write from hosts: * *
> >
> >
> > *XXXWMXXX-7:kafka_2.11-0.10.1.0 rbaddam$ bin/kafka-console-producer.sh
> > --broker-list 10.247.195.122:9093 <http://10.247.195.122:9093> --topic
> > ssltopic --producer.config client-ssl.properties*
> >
> >
> > *[2016-12-13 14:53:45,839] WARN Error while fetching metadata with
> > correlation id 0 : {ssltopic=UNKNOWN_TOPIC_OR_PARTITION}
> > (org.apache.kafka.clients.NetworkClient)*
> >
> > *[2016-12-13 14:53:45,984] WARN Error while fetching metadata with
> > correlation id 1 : {ssltopic=UNKNOWN_TOPIC_OR_PARTITION}
> > (org.apache.kafka.clients.NetworkClient)*
> >
> >
> > *XXXWMXXX-7:kafka_2.11-0.10.1.0 rbaddam$ cat client-ssl.properties*
> >
> > *#group.id <http://group.id>=sslgroup*
> >
> > *security.protocol=SSL*
> >
> > *ssl.truststore.location=/Users/rbaddam/Desktop/Dev/
> > kafka_2.11-0.10.1.0/ssl/client.truststore.jks*
> >
> > *ssl.truststore.password=123456*
> >
> > * #Configure Below if you use Client Auth*
> >
> >
> > *ssl.keystore.location=/Users/rbaddam/Desktop/Dev/kafka_2.
> > 11-0.10.1.0/ssl/client.keystore.jks*
> >
> > *ssl.keystore.password=123456*
> >
> > *ssl.key.password=123456*
> >
> >
> > *XXXWMXXX-7:kafka_2.11-0.10.1.0 rbaddam$ bin/kafka-console-consumer.sh
> > --bootstrap-server 10.247.195.122:9093 <http://10.247.195.122:9093>
> > --new-consumer --consumer.config client-ssl.properties --topic ssltopic
> > --from-beginning*
> >
> > *[2016-12-13 14:53:28,817] WARN Error while fetching metadata with
> > correlation id 1 : {ssltopic=UNKNOWN_TOPIC_OR_PARTITION}
> > (org.apache.kafka.clients.NetworkClient)*
> >
> > *[2016-12-13 14:53:28,819] ERROR Unknown error when running consumer:
> > (kafka.tools.ConsoleConsumer$)*
> >
> > *org.apache.kafka.common.errors.GroupAuthorizationException: Not
> > authorized
> > to access group: console-consumer-52826*
> >
> >
> > Thanks in advance,
> >
> > Raghu - raghu98499@gmail.com
> >
>

Re: Kafka ACL's with SSL Protocol is not working

Posted by Rajini Sivaram <rs...@pivotal.io>.
You need to set ssl.client.auth="required" in server.properties.

Regards,

Rajini

On Wed, Dec 14, 2016 at 12:12 AM, Raghu B <ra...@gmail.com> wrote:

> Hi All,
>
> I am trying to enable ACL's in my Kafka cluster with along with SSL
> Protocol.
>
> I tried with each and every parameters but no luck, so I need help to
> enable the SSL(without Kerberos) and I am attaching all the configuration
> details in this.
>
> Kindly Help me.
>
>
> *I tested SSL without ACL, it worked fine
> (listeners=SSL://10.247.195.122:9093 <http://10.247.195.122:9093>)*
>
>
> *This is my Kafka server properties file:*
>
> *############################# ACL SETTINGS #############################*
>
> *auto.create.topics.enable=true*
>
> *authorizer.class.name
> <http://authorizer.class.name>=kafka.security.auth.SimpleAclAuthorizer*
>
> *security.inter.broker.protocol=SSL*
>
> *#allow.everyone.if.no.acl.found=true*
>
> *#principal.builder.class=CustomizedPrincipalBuilderClass*
>
> *#super.users=User:"CN=writeuser,OU=Unknown,O=
> Unknown,L=Unknown,ST=Unknown,C=Unknown"*
>
> *#super.users=User:Raghu;User:Admin*
>
> *#offsets.storage=kafka*
>
> *#dual.commit.enabled=true*
>
> *listeners=SSL://10.247.195.122:9093 <http://10.247.195.122:9093>*
>
> *#listeners=PLAINTEXT://10.247.195.122:9092 <http://10.247.195.122:9092>*
>
> *#listeners=PLAINTEXT://10.247.195.122:9092
> <http://10.247.195.122:9092>,SSL://10.247.195.122:9093
> <http://10.247.195.122:9093>*
>
> *#advertised.listeners=PLAINTEXT://10.247.195.122:9092
> <http://10.247.195.122:9092>*
>
>
> *
> ssl.keystore.location=/home/raghu/kafka/security/server.keystore.jks*
>
> *        ssl.keystore.password=123456*
>
> *        ssl.key.password=123456*
>
> *
> ssl.truststore.location=/home/raghu/kafka/security/server.truststore.jks*
>
> *        ssl.truststore.password=123456*
>
>
>
> *Set the ACL from Authorizer CLI:*
>
> > *bin/kafka-acls.sh --authorizer-properties
> zookeeper.connect=10.247.195.122:2181 <http://10.247.195.122:2181> --list
> --topic ssltopic*
>
> *Current ACLs for resource `Topic:ssltopic`: *
>
> *  User:CN=writeuser, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown,
> C=Unknown has Allow permission for operations: Write from hosts: * *
>
>
> *XXXWMXXX-7:kafka_2.11-0.10.1.0 rbaddam$ bin/kafka-console-producer.sh
> --broker-list 10.247.195.122:9093 <http://10.247.195.122:9093> --topic
> ssltopic --producer.config client-ssl.properties*
>
>
> *[2016-12-13 14:53:45,839] WARN Error while fetching metadata with
> correlation id 0 : {ssltopic=UNKNOWN_TOPIC_OR_PARTITION}
> (org.apache.kafka.clients.NetworkClient)*
>
> *[2016-12-13 14:53:45,984] WARN Error while fetching metadata with
> correlation id 1 : {ssltopic=UNKNOWN_TOPIC_OR_PARTITION}
> (org.apache.kafka.clients.NetworkClient)*
>
>
> *XXXWMXXX-7:kafka_2.11-0.10.1.0 rbaddam$ cat client-ssl.properties*
>
> *#group.id <http://group.id>=sslgroup*
>
> *security.protocol=SSL*
>
> *ssl.truststore.location=/Users/rbaddam/Desktop/Dev/
> kafka_2.11-0.10.1.0/ssl/client.truststore.jks*
>
> *ssl.truststore.password=123456*
>
> * #Configure Below if you use Client Auth*
>
>
> *ssl.keystore.location=/Users/rbaddam/Desktop/Dev/kafka_2.
> 11-0.10.1.0/ssl/client.keystore.jks*
>
> *ssl.keystore.password=123456*
>
> *ssl.key.password=123456*
>
>
> *XXXWMXXX-7:kafka_2.11-0.10.1.0 rbaddam$ bin/kafka-console-consumer.sh
> --bootstrap-server 10.247.195.122:9093 <http://10.247.195.122:9093>
> --new-consumer --consumer.config client-ssl.properties --topic ssltopic
> --from-beginning*
>
> *[2016-12-13 14:53:28,817] WARN Error while fetching metadata with
> correlation id 1 : {ssltopic=UNKNOWN_TOPIC_OR_PARTITION}
> (org.apache.kafka.clients.NetworkClient)*
>
> *[2016-12-13 14:53:28,819] ERROR Unknown error when running consumer:
> (kafka.tools.ConsoleConsumer$)*
>
> *org.apache.kafka.common.errors.GroupAuthorizationException: Not
> authorized
> to access group: console-consumer-52826*
>
>
> Thanks in advance,
>
> Raghu - raghu98499@gmail.com
>

Re: Kafka ACL's with SSL Protocol is not working

Posted by Rajini Sivaram <rs...@pivotal.io>.
You need to set ssl.client.auth="required" in server.properties.

Regards,

Rajini

On Wed, Dec 14, 2016 at 12:12 AM, Raghu B <ra...@gmail.com> wrote:

> Hi All,
>
> I am trying to enable ACL's in my Kafka cluster with along with SSL
> Protocol.
>
> I tried with each and every parameters but no luck, so I need help to
> enable the SSL(without Kerberos) and I am attaching all the configuration
> details in this.
>
> Kindly Help me.
>
>
> *I tested SSL without ACL, it worked fine
> (listeners=SSL://10.247.195.122:9093 <http://10.247.195.122:9093>)*
>
>
> *This is my Kafka server properties file:*
>
> *############################# ACL SETTINGS #############################*
>
> *auto.create.topics.enable=true*
>
> *authorizer.class.name
> <http://authorizer.class.name>=kafka.security.auth.SimpleAclAuthorizer*
>
> *security.inter.broker.protocol=SSL*
>
> *#allow.everyone.if.no.acl.found=true*
>
> *#principal.builder.class=CustomizedPrincipalBuilderClass*
>
> *#super.users=User:"CN=writeuser,OU=Unknown,O=
> Unknown,L=Unknown,ST=Unknown,C=Unknown"*
>
> *#super.users=User:Raghu;User:Admin*
>
> *#offsets.storage=kafka*
>
> *#dual.commit.enabled=true*
>
> *listeners=SSL://10.247.195.122:9093 <http://10.247.195.122:9093>*
>
> *#listeners=PLAINTEXT://10.247.195.122:9092 <http://10.247.195.122:9092>*
>
> *#listeners=PLAINTEXT://10.247.195.122:9092
> <http://10.247.195.122:9092>,SSL://10.247.195.122:9093
> <http://10.247.195.122:9093>*
>
> *#advertised.listeners=PLAINTEXT://10.247.195.122:9092
> <http://10.247.195.122:9092>*
>
>
> *
> ssl.keystore.location=/home/raghu/kafka/security/server.keystore.jks*
>
> *        ssl.keystore.password=123456*
>
> *        ssl.key.password=123456*
>
> *
> ssl.truststore.location=/home/raghu/kafka/security/server.truststore.jks*
>
> *        ssl.truststore.password=123456*
>
>
>
> *Set the ACL from Authorizer CLI:*
>
> > *bin/kafka-acls.sh --authorizer-properties
> zookeeper.connect=10.247.195.122:2181 <http://10.247.195.122:2181> --list
> --topic ssltopic*
>
> *Current ACLs for resource `Topic:ssltopic`: *
>
> *  User:CN=writeuser, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown,
> C=Unknown has Allow permission for operations: Write from hosts: * *
>
>
> *XXXWMXXX-7:kafka_2.11-0.10.1.0 rbaddam$ bin/kafka-console-producer.sh
> --broker-list 10.247.195.122:9093 <http://10.247.195.122:9093> --topic
> ssltopic --producer.config client-ssl.properties*
>
>
> *[2016-12-13 14:53:45,839] WARN Error while fetching metadata with
> correlation id 0 : {ssltopic=UNKNOWN_TOPIC_OR_PARTITION}
> (org.apache.kafka.clients.NetworkClient)*
>
> *[2016-12-13 14:53:45,984] WARN Error while fetching metadata with
> correlation id 1 : {ssltopic=UNKNOWN_TOPIC_OR_PARTITION}
> (org.apache.kafka.clients.NetworkClient)*
>
>
> *XXXWMXXX-7:kafka_2.11-0.10.1.0 rbaddam$ cat client-ssl.properties*
>
> *#group.id <http://group.id>=sslgroup*
>
> *security.protocol=SSL*
>
> *ssl.truststore.location=/Users/rbaddam/Desktop/Dev/
> kafka_2.11-0.10.1.0/ssl/client.truststore.jks*
>
> *ssl.truststore.password=123456*
>
> * #Configure Below if you use Client Auth*
>
>
> *ssl.keystore.location=/Users/rbaddam/Desktop/Dev/kafka_2.
> 11-0.10.1.0/ssl/client.keystore.jks*
>
> *ssl.keystore.password=123456*
>
> *ssl.key.password=123456*
>
>
> *XXXWMXXX-7:kafka_2.11-0.10.1.0 rbaddam$ bin/kafka-console-consumer.sh
> --bootstrap-server 10.247.195.122:9093 <http://10.247.195.122:9093>
> --new-consumer --consumer.config client-ssl.properties --topic ssltopic
> --from-beginning*
>
> *[2016-12-13 14:53:28,817] WARN Error while fetching metadata with
> correlation id 1 : {ssltopic=UNKNOWN_TOPIC_OR_PARTITION}
> (org.apache.kafka.clients.NetworkClient)*
>
> *[2016-12-13 14:53:28,819] ERROR Unknown error when running consumer:
> (kafka.tools.ConsoleConsumer$)*
>
> *org.apache.kafka.common.errors.GroupAuthorizationException: Not
> authorized
> to access group: console-consumer-52826*
>
>
> Thanks in advance,
>
> Raghu - raghu98499@gmail.com
>