[jira] [Created] (KAFKA-8821) Avoid pattern subscription to allow for stricter ACL settings

["Matthias J. Sax (Jira)" <ji...@apache.org>]

Matthias J. Sax created KAFKA-8821:
--------------------------------------

             Summary: Avoid pattern subscription to allow for stricter ACL settings
                 Key: KAFKA-8821
                 URL: https://issues.apache.org/jira/browse/KAFKA-8821
             Project: Kafka
          Issue Type: Improvement
          Components: streams
            Reporter: Matthias J. Sax


To avoid triggering auto topic creation (if `auto.create.topic.enable=true` on the brokers), Kafka Streams uses consumer pattern subscription. For this case, the consumer requests all metadata from the brokers and does client side filtering.

However, if users want to set ACL to restrict a Kafka Streams application, this may results in broker side ERROR logs that some metadata cannot be provided. The only way to avoid those broker side ERROR logs is to grant corresponding permissions.

As of 2.3 release it's possible to disable auto topic creation client side (via https://issues.apache.org/jira/browse/KAFKA-7320). Kafka Streams should use this new feature (note, that broker version 0.11 is required) to allow users to set strict ACLs without getting flooded with ERROR logs on the broker.

The proposal is that by default Kafka Streams disables auto-topic create client side (optimistically) and uses regular subscription (not pattern subscription). If an older broker is used, users need to explicitly enable `allow.auto.create.topic` client side. If we detect this setting, we switch back to pattern based subscription.

If users don't enable auto topic create client side and run with an older broker, we would just rethrow the exception to the user, adding some context information on how to fix the issue. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)