[jira] [Created] (KAFKA-9282) Consider more flexible node grouping for Pattern subscription

["Sophie Blee-Goldman (Jira)" <ji...@apache.org>]

Sophie Blee-Goldman created KAFKA-9282:
------------------------------------------

             Summary: Consider more flexible node grouping for Pattern subscription
                 Key: KAFKA-9282
                 URL: https://issues.apache.org/jira/browse/KAFKA-9282
             Project: Kafka
          Issue Type: Improvement
          Components: streams
            Reporter: Sophie Blee-Goldman


The current grouping for input topics using pattern subscription creates a single node group for all matching topics, meaning the number of tasks scales with the maximum partition count across all topics. This reduces overhead and is suitable for some scenarios, but limits the ability to scale out and prevents easily parallelized processing of completely independent partitions. We should consider making it possible for the number of tasks to instead scale with the total number of partitions summed across all matching input topics.

Ideally Streams could just autoscale based on some heuristic and the currently available resources, but we would have to be careful if those things change. Alternatively we could just leave this up to the user to decide, potentially by augmenting the Pattern-based source KStream method with a new overload allowing this grouping to be specified. For example
{code:java}
StreamsBuilder {
  public KStream stream(topicPattern, numTasks);
}
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)