[jira] [Commented] (KAFKA-13332) New pattern-matched topic with more partitions than existing matched topics can crash Kafka Streams

["Matthias J. Sax (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-13332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17421883#comment-17421883 ] 

Matthias J. Sax commented on KAFKA-13332:
-----------------------------------------

The original idea to fail if the number of partitions for an existing repartition topic changes was meant as a "safe guard". Do we believe that this assumption was incorrect and that we don't need such a save guard (and just remove the check if the repartition topic exists?

Or could this be a case of "won't fix" and users would need to work around by expliclity calling `repartition()` and set the number of partitions manually?

This issue is kinda known since we introduce auto-repartitioning (I guess 0.10.1 release?), but it never came up in practice as a problem.

> New pattern-matched topic with more partitions than existing matched topics can crash Kafka Streams
> ---------------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-13332
>                 URL: https://issues.apache.org/jira/browse/KAFKA-13332
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>            Reporter: A. Sophie Blee-Goldman
>            Priority: Major
>
> The partition count resolution logic in Streams is used to determine the number of partitions for any repartition topics that don't already exist. This is done by parsing the topology to find the number of partitions of all upstream topics, and taking the max. For Pattern-subscribed subtopologies, this means you need to ensure that at least one topic matching this pattern is created prior to starting up the app. That topic, or topics, will determine the number of partitions for any downstream repartitions.
> The problem is that repartition topics are created once, the first time the app is started up. After that, during each rebalance Streams will validate all repartition topics including checking for their existence, and verifying they have the correct number of partitions. This check will fail if a new topic is created after the first initialization, which matches the pattern but has more partitions than any of the existing topics.
> This means that unfortunately, you can't create a new input topic that matches the pattern your app is subscribed to unless it has equal or fewer partitions than the existing matching topics. If you do, you would need to stop all instances and delete the existing repartition topics before creating this new topic



--
This message was sent by Atlassian Jira
(v8.3.4#803005)