You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Chapin, Ryan" <Ry...@hughes.com> on 2019/01/07 19:11:47 UTC
Unable to Restrict Access to Consumer Group ACLs
I do not seem to be able to restrict access to specific users for specific consumer groups.
It seems as though I am doing something wrong, or that there is a bug as I imagine what I am trying to do is very straight forward.
We have an 11 node cluster running the 2.11-1.0.0 open source version of Kafka in conjunction with an open source Zookeeper distro (zookeeper-3.4.6-1.el7) from the bigtop repo. The cluster is secured via Kerberos (AD) authentication, over TLS. All nodes communicate with each other over authenticated TLS connections as well.
We are using the Zookeeper based authorization mechanism with the following configs from Kafka's server.properties (the comments also indicate the only Zookeeper config items specific to the ACL mechanism):
######################################################################
# This setting ensures that the acl settings for znodes that are
# created by kafka are not modifiable by other users that can connect
# to ZooKeeper. This has no effect on the data that can be accessed
# in kafka, but ensures that outside actors cannot modify the znodes
# that enable kafka to maintain state on the topics.
#
zookeeper.set.acl=true
######################################################################
# Given that we have figured out that the two following ZooKeeper
# configs:
#
# kerberos.removeHostFromPrincipal=true
# kerberos.removeRealmFromPrincipal=true
#
# will take a principal in the form of
#
# kafka/kafka01.prod.quasar.nadops.net@GTN.NADOPS.NET
#
# and 'convert' it to:
#
# kafka
#
# This means that all of the kafka brokers now 'share' a username and
# we can easily configure the super.users to grant all access to all
# znodes to all of the brokers without having to continually update
# this config and/or set it conditionally based on the environment
# and list of hosts.
#
super.users=User:kafka
######################################################################
# Authorization Configs:
#
# The following is a stepping stone using the Zookeeper based
# authorization mechanism
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
# Since we were able to sort out the ability to define the kafka
# brokers as super.users, we can, by default ensure that only those
# users that have specifically configured acls can get access to
# kafka resources.
#
allow.everyone.if.no.acl.found=false
I would like to be able to provide self-service access to certain topics such that certain topics allow read access to any authenticated user from any host.
Further, I want to ensure that only certain users can use specifically defined consumer groups when reading from those topics.
For example:
There is a topic "rchapin_topic" configured with three partitions. There is a production application that is reading from this topic and driving a customer facing dashboard. The production application is authenticating as the "rchapin" user with the "rchapin_group" consumer id.
Any other authenticated user should be able to read from that topic, but SHOULD NOT be able to do so with the "rchapin_group" consumer group. If any other consumer starts reading from the topic with that consumer group, it will start siphoning off data from the production application.
Following are the ACLs that we have attempted to put in place that we figure would allow us to configure such a use case.
Current ACLs for resource `Topic:rchapin_topic`:
User:rchapin has Allow permission for operations: Read from hosts: *
User:rchapin has Allow permission for operations: Write from hosts: *
User:rchapin has Allow permission for operations: Describe from hosts: *
User:* has Allow permission for operations: Describe from hosts: *
User:* has Allow permission for operations: Read from hosts: *
Current ACLs for resource `Group:rchapin_group`:
User:rchapin has Allow permission for operations: Read from hosts: *
With the aforementioned configurations I can connect as any other authenticated use to read data from the rchapin topic using the rchapin_group consumer group and begin siphoning off data from the rchapin production application.
If I add an ACL that specifically adds "deny-principal" for all users for the group as follows:
# kafka-acls.sh --authorizer-properties zookeeper.connect=zk2-01 --add --deny-principal User:* --group rchapin_group
Adding ACLs for resource `Group:rchapin_group`:
User:* has Deny permission for operations: All from hosts: *
Current ACLs for resource `Group:rchapin_group`:
User:rchapin has Allow permission for operations: Read from hosts: *
User:* has Deny permission for operations: All from hosts: *
Then no one is able to read from that topic with the rchapin_group consumer id, but any authenticated user can connect with another group id to consume from the topic.
Is it possible to restrict access to consumer group to a specific user? If so, what am I doing wrong? Or is there a bug in this version of Kafka?
Any help is greatly appreciated.
-- Ryan Chapin
RE: Unable to Restrict Access to Consumer Group ACLs
Posted by "Chapin, Ryan" <Ry...@hughes.com>.
Well, I feel a bit stupid on this one . . . .
Guess what happens when I discover the following ACL rule that was in place?
Current ACLs for resource `Group:*`:
User:* has Allow permission for operations: Read from hosts: *
And then guess what happens when I delete that rule?
Yes, you guessed it . . . it works as you would expect and limits access to the specifically defined group.
-----Original Message-----
From: Chapin, Ryan <Ry...@hughes.com>
Sent: Monday, January 07, 2019 2:12 PM
To: users@kafka.apache.org
Subject: Unable to Restrict Access to Consumer Group ACLs
WARNING: The sender of this email could not be validated and may not match the person in the "From" field.
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
I do not seem to be able to restrict access to specific users for specific consumer groups.
It seems as though I am doing something wrong, or that there is a bug as I imagine what I am trying to do is very straight forward.
We have an 11 node cluster running the 2.11-1.0.0 open source version of Kafka in conjunction with an open source Zookeeper distro (zookeeper-3.4.6-1.el7) from the bigtop repo. The cluster is secured via Kerberos (AD) authentication, over TLS. All nodes communicate with each other over authenticated TLS connections as well.
We are using the Zookeeper based authorization mechanism with the following configs from Kafka's server.properties (the comments also indicate the only Zookeeper config items specific to the ACL mechanism):
######################################################################
# This setting ensures that the acl settings for znodes that are # created by kafka are not modifiable by other users that can connect # to ZooKeeper. This has no effect on the data that can be accessed # in kafka, but ensures that outside actors cannot modify the znodes # that enable kafka to maintain state on the topics.
#
zookeeper.set.acl=true
######################################################################
# Given that we have figured out that the two following ZooKeeper # configs:
#
# kerberos.removeHostFromPrincipal=true
# kerberos.removeRealmFromPrincipal=true
#
# will take a principal in the form of
#
# kafka/kafka01.prod.quasar.nadops.net@GTN.NADOPS.NET
#
# and 'convert' it to:
#
# kafka
#
# This means that all of the kafka brokers now 'share' a username and # we can easily configure the super.users to grant all access to all # znodes to all of the brokers without having to continually update # this config and/or set it conditionally based on the environment # and list of hosts.
#
super.users=User:kafka
######################################################################
# Authorization Configs:
#
# The following is a stepping stone using the Zookeeper based # authorization mechanism
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
# Since we were able to sort out the ability to define the kafka # brokers as super.users, we can, by default ensure that only those # users that have specifically configured acls can get access to # kafka resources.
#
allow.everyone.if.no.acl.found=false
I would like to be able to provide self-service access to certain topics such that certain topics allow read access to any authenticated user from any host.
Further, I want to ensure that only certain users can use specifically defined consumer groups when reading from those topics.
For example:
There is a topic "rchapin_topic" configured with three partitions. There is a production application that is reading from this topic and driving a customer facing dashboard. The production application is authenticating as the "rchapin" user with the "rchapin_group" consumer id.
Any other authenticated user should be able to read from that topic, but SHOULD NOT be able to do so with the "rchapin_group" consumer group. If any other consumer starts reading from the topic with that consumer group, it will start siphoning off data from the production application.
Following are the ACLs that we have attempted to put in place that we figure would allow us to configure such a use case.
Current ACLs for resource `Topic:rchapin_topic`:
User:rchapin has Allow permission for operations: Read from hosts: *
User:rchapin has Allow permission for operations: Write from hosts: *
User:rchapin has Allow permission for operations: Describe from hosts: *
User:* has Allow permission for operations: Describe from hosts: *
User:* has Allow permission for operations: Read from hosts: *
Current ACLs for resource `Group:rchapin_group`:
User:rchapin has Allow permission for operations: Read from hosts: *
With the aforementioned configurations I can connect as any other authenticated use to read data from the rchapin topic using the rchapin_group consumer group and begin siphoning off data from the rchapin production application.
If I add an ACL that specifically adds "deny-principal" for all users for the group as follows:
# kafka-acls.sh --authorizer-properties zookeeper.connect=zk2-01 --add --deny-principal User:* --group rchapin_group Adding ACLs for resource `Group:rchapin_group`:
User:* has Deny permission for operations: All from hosts: *
Current ACLs for resource `Group:rchapin_group`:
User:rchapin has Allow permission for operations: Read from hosts: *
User:* has Deny permission for operations: All from hosts: *
Then no one is able to read from that topic with the rchapin_group consumer id, but any authenticated user can connect with another group id to consume from the topic.
Is it possible to restrict access to consumer group to a specific user? If so, what am I doing wrong? Or is there a bug in this version of Kafka?
Any help is greatly appreciated.
-- Ryan Chapin