Deploying Kafka topics in a kerberized Zookeeper without superuser (in a CI flow)

[Kristjan Peil <kr...@gmail.com>]

I'm running Kafka 1.1.1 and Zookeeper 3.4.6 in a cluster, both guarded by
Kerberos. My app stack includes a module containing topic configurations,
and my continuous integration build autodeploys changes to topics with
kafka-topics.sh and kafka-configs.sh.

When I try to use a non-superuser principal to authenticate in the scripts,
the topic metadata is created by kafka-topics.sh in Zookeeper in such a way
that Kafka cannot process it to create the actual topics in Kafka brokers -
partitions are not created in the broker. Also, running kafka-configs.sh to
alter configs of existing topics gets "NoAuth for /configs/<topicname>".

When I authenticate with the superuser principal "kafka" then everything
works fine. But making the "kafka" superuser credentials available in CI
context seems unsecure.

Is it possible to use kafka-topics.sh and kafka-configs.sh in a kerberized
environment with a non-superuser Kerberos principal and how can this be
made to happen?
Can you suggest an alternate solution to achieve CI for Kafka topics?

Best regards,
Kristjan Peil

Re: Deploying Kafka topics in a kerberized Zookeeper without superuser (in a CI flow)

[Harsha Chintalapani <ka...@harsha.io>]

Hi,
      When you kerberoize Kafka and enable zookeeper.set.acl to true, all the zookeeper nodes created under zookeeper root will have ACLs to allow only Kafka Broker’s principal. Since all topic creation will go directly to zookeeper, i.e Kafka-topic.sh script creates a zookeeper node under /broker/topics and it needs to have Kafka Broker’s principal set as ACL.  If you use any other principal it will create issues like you are seeing.
      One option is to disable zookeeper.set.acl. This means anyone who has access to zookeeper can create a topic. Better option would be to use KafkaAdminClient to createTopics which will send a createTopicRequest through brokers which can be authorized. Your CI can have its own principal and you can create authorization policy which will allow this principal to create topics.

-Harsha

On Jan 21, 2019, 3:00 AM -0800, Kristjan Peil <kr...@gmail.com>, wrote:
> I'm running Kafka 1.1.1 and Zookeeper 3.4.6 in a cluster, both guarded by
> Kerberos. My app stack includes a module containing topic configurations,
> and my continuous integration build autodeploys changes to topics with
> kafka-topics.sh and kafka-configs.sh.
>
> When I try to use a non-superuser principal to authenticate in the scripts,
> the topic metadata is created by kafka-topics.sh in Zookeeper in such a way
> that Kafka cannot process it to create the actual topics in Kafka brokers -
> partitions are not created in the broker. Also, running kafka-configs.sh to
> alter configs of existing topics gets "NoAuth for /configs/<topicname>".
>
> When I authenticate with the superuser principal "kafka" then everything
> works fine. But making the "kafka" superuser credentials available in CI
> context seems unsecure.
>
> Is it possible to use kafka-topics.sh and kafka-configs.sh in a kerberized
> environment with a non-superuser Kerberos principal and how can this be
> made to happen?
> Can you suggest an alternate solution to achieve CI for Kafka topics?
>
> Best regards,
> Kristjan Peil