[jira] [Commented] (KAFKA-9806) Unable to consume when __consumer_offsets is created under insufficient cluster ACLs

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-9806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17074577#comment-17074577 ] 

ASF GitHub Bot commented on KAFKA-9806:
---------------------------------------

paolo-moriello commented on pull request #8415: KAFKA-9806: __consumer_offsets is not created under insufficient ACLs
URL: https://github.com/apache/kafka/pull/8415
 
 
   Fixing KAFKA-9806 bug: the __consumer_offsets topic gets created when a consumer try to consume on a new cluster for the first time. If there aren't sufficient cluster-level ACLs, FindCoordinator request succeed, anche __consumer_offsets gets created in zk, but subsequent UpdateMetadata and LeaderAndIsr requests fail, putting the offsets topic in a bad state. When this happens, it is not possible to consume even after updating ACLs (5a).
   
   This PR aims to introduce a new check before creating the __consumer_offsets topic, so that it does not get created if there aren't sufficient ACLs.
   
   This change was tested by following the steps reported in KAFKA-9806, and verifying that it is possible to consume after the code change (5b). Eg:
   1. Create a topic
   2. Set insufficient cluster level ACLs. Which precise ACL does not matter as long as ClusterAction initiated by administrative messages between brokers is denied. 
   3. Consume on the topic -> this request will fail with unauthorized errors on the client side as well
   4. Remove ACLs or set ACLs to allow inter-broker communication
   5a. (Before code change) Produce/consume on topic. Consumer won't be able to read any data.
   5b. (After code change) Produce/consume on topic. Consumer are able to read data.
   
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Unable to consume when __consumer_offsets is created under insufficient cluster ACLs
> ------------------------------------------------------------------------------------
>
>                 Key: KAFKA-9806
>                 URL: https://issues.apache.org/jira/browse/KAFKA-9806
>             Project: Kafka
>          Issue Type: Bug
>          Components: consumer, controller, core
>            Reporter: Paolo Moriello
>            Assignee: Paolo Moriello
>            Priority: Major
>
> h2. Description
> The __consumer_offsets topic is automatically created the first time consumption happens on a cluster. When a consumer start to consume from a topic, it first submits a FindCoordinator request, which tries to get metadata from the __consumer_offsets topic. If this does not exist, it gets created. On topic creation, the controller assigning partitions to the other brokers through UpdateMetadata and LeaderAndIsr requests.
> If there aren't sufficient cluster-level ACLs, UpdateMetadata and LeaderAndIsr are not authorized by peer brokers; so, the requests fail, but the auto-creation of the topic succeeds and zk-nodes are filled with the related metadata. No-operation is performed by the controller on handling of failed UpdateMetadata and LeaderAndIsr, so __consumer_offset topic does not get deleted.
> When ACLs are removed/fixed, __consumer_offset is in a bad state for the topics we tried to consume on under insufficient ACLs, and consumption is not possible: consumer does not return any data. No errors are thrown by the consumer, nor by the brokers. By enabling DEBUG logs on the consumer we only see that: 'The coordinator is not available.'
> Note: this is fixed with a broker restart or __consumer_offsets partition reassignment, since both operations trigger UpdateMetadata and LeaderAndIsr requests on the topic.
> h2. Steps to reproduce
> Clean Kafka cluster (no __consumer_offset) with more than 1 broker, and SSL enabled for inter-broker communication.
> + I had allow.everyone.if.no.acl.found set to true on my test cluster.
>  # Create a topic
>  # Set insufficient cluster level ACLs. Which precise ACL does not matter as long as ClusterAction initiated by administrative messages between brokers is denied. 
>  # Consume on the topic -> this request will fail with unauthorized errors on the client side as well
>  # Remove ACLs or set ACLs to allow inter-broker communication
>  # Produce/consume on topic. Consumer won't be able to read any data.
> Step 3 triggers creation of __consumer_offset topic. In addition to the client side errors, it is possible to see related UpdateMetadata and LeaderAndIsr exceptions in the broker logs. However at this point you can verify that __consumer_offset topic is created.
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)