You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Sophie Blee-Goldman (Jira)" <ji...@apache.org> on 2020/01/08 02:00:11 UTC
[jira] [Commented] (KAFKA-7317) Use collections subscription for
main consumer to reduce metadata
[ https://issues.apache.org/jira/browse/KAFKA-7317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17010255#comment-17010255 ]
Sophie Blee-Goldman commented on KAFKA-7317:
--------------------------------------------
Disabling auto-topic creation is possible for brokers version 0.11.0.0+ as of KAFKA-5291. As of 2.2.1/2.3.x the compatibility support for Streams client changed, where the minimum broker version requirement is now also 0.11.x. This means we can now safely go back to using regular subscription.
Note, that users can still use pattern subscription themselves to subscribe to input topics. We should consider adding broker-side regex support and in general avoid fetching/sending metadata on every single topic in a cluster.
> Use collections subscription for main consumer to reduce metadata
> -----------------------------------------------------------------
>
> Key: KAFKA-7317
> URL: https://issues.apache.org/jira/browse/KAFKA-7317
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Matthias J. Sax
> Assignee: Sophie Blee-Goldman
> Priority: Major
>
> In KAFKA-4633 we switched from "collection subscription" to "pattern subscription" for `Consumer#subscribe()` to avoid triggering auto topic creating on the broker. In KAFKA-5291, the metadata request was extended to overwrite the broker config within the request itself. However, this feature is only used in `KafkaAdminClient`. KAFKA-7320 adds this feature for the consumer client, too.
> This ticket proposes to use the new feature within Kafka Streams to allow the usage of collection based subscription in consumer and admit clients to reduce the metadata response size than can be very large for large number of partitions in the cluster.
> Note, that Streams need to be able to distinguish if it connects to older brokers that do not support the new metadata request and still use pattern subscription for this case.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)