[jira] [Comment Edited] (KAFKA-13612) internal topics won't be created in metadataRequest when auto.create.topics.enable=false

["Jason Gustafson (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-13612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17482006#comment-17482006 ] 

Jason Gustafson edited comment on KAFKA-13612 at 1/25/22, 5:59 PM:
-------------------------------------------------------------------

I think there is a higher level question here which the community probably has never fully agreed upon. My take (and I think Colin agrees with me) is that __consumer_offsets __ should not be accessed by clients directly. For example, we don't make any guarantees about the compatibility of the message format used to write group metadata and offset commits to this topic, so any tool reading data from it might break after an upgrade. And direct writes to the topic do not work in general because the coordinator does not materialize the data in its cache. What is supported is indirect access through the respective request APIs (e.g. `OffsetCommit` and `OffsetDelete`). Is there a gap in these APIs that requires some use cases to still access __ __consumer_offsets directly?

As for the behavior change, if we were still in the context of release 2.8, I'd probably take the old behavior without question. But now we've done 3 releases (3.1 is out now) with this modified behavior, so I'm inclined to step back a little to consider how we actually want this to work.


was (Author: hachikuji):
I think there is a higher level question here which the community probably has never fully agreed upon. My take (and I think Colin agrees with me) is that `__consumer_offsets` should not be accessed by clients directly. For example, we don't make any guarantees about the compatibility of the message format used to write group metadata and offset commits to this topic, so any tool reading data from it might break after an upgrade. And direct writes to the topic do not work in general because the coordinator does not materialize the data in its cache. What is supported is indirect access through the respective request APIs (e.g. `OffsetCommit` and `OffsetDelete`). Is there a gap in these APIs that requires some use cases to still access `__consumer_offsets` directly?

As for the behavior change, if we were still in the context of release 2.8, I'd probably take the old behavior without question. But now we've done 3 releases (3.1 is out now) with this modified behavior, so I'm inclined to step back a little to consider how we actually want this to work.



> internal topics won't be created in metadataRequest when auto.create.topics.enable=false
> ----------------------------------------------------------------------------------------
>
>                 Key: KAFKA-13612
>                 URL: https://issues.apache.org/jira/browse/KAFKA-13612
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 2.8.0, 3.0.0
>            Reporter: Luke Chen
>            Assignee: Luke Chen
>            Priority: Major
>
> In KAFKA-9751, when create internal topics through FindCoordinator or Metadata request, we route the topic creation request to the controller instead of handling by itself. We change logic in `KafkaApis#getTopicMetadata`, and make the internal topic won't get created when "auto.create.topics.enable=false`. 
> h4.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)