[jira] [Created] (KAFKA-20934) Considering Share Groups for decoupling stateless Kafka Streams processing parallelism from source partitions

["sanghyeok An (Jira)" <ji...@apache.org>]

sanghyeok An created KAFKA-20934:
------------------------------------

             Summary: Considering Share Groups for decoupling stateless Kafka Streams processing parallelism from source partitions
                 Key: KAFKA-20934
                 URL: https://issues.apache.org/jira/browse/KAFKA-20934
             Project: Kafka
          Issue Type: Improvement
            Reporter: sanghyeok An
            Assignee: sanghyeok An


I am creating this Jira ticket for ideation. If Kafka maintainers or the community think this direction is worth discussing, I would be happy to write a KIP and develop the discussion further.

Currently, Kafka Streams processing parallelism is closely tied to the number of partitions in the source topic. In contrast, Share Groups allow multiple consumers to share the same partition, and the number of consumers can exceed the number of partitions. Therefore, using Share Groups may provide a way to decouple processing parallelism from the number of source partitions.

Applying this model to existing stateful Kafka Streams topologies does not appear to be straightforward. Stateful processing in Kafka Streams is based on a model in which a task owns specific input partitions and local state stores. In addition, with Share Groups, records from the same partition may be processed by different consumers, and partition-level ordering is not guaranteed overall.

However, there may be room to use Share Groups for stateless Kafka Streams topologies where record processing is order-independent. In such topologies, processing records from the same source partition across multiple Streams instances would not introduce conflicts in terms of state ownership, potentially allowing the number of processing instances to exceed the number of source partitions.

Although proper performance evaluation would be necessary, this could potentially improve throughput for stateless topologies where application-side processing is the bottleneck. It could also reduce the need to over-partition topics solely to achieve higher processing parallelism, which may in turn reduce the operational overhead associated with maintaining a large number of partitions. KIP-932 also describes over-partitioning for parallel consumption as one of the problems that Share Groups are intended to address.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)