[jira] [Created] (KAFKA-13842) Add per-aggregation step before repartitioning

["Matthias J. Sax (Jira)" <ji...@apache.org>]

Matthias J. Sax created KAFKA-13842:
---------------------------------------

             Summary: Add per-aggregation step before repartitioning
                 Key: KAFKA-13842
                 URL: https://issues.apache.org/jira/browse/KAFKA-13842
             Project: Kafka
          Issue Type: Improvement
          Components: streams
            Reporter: Matthias J. Sax


Kafka Streams follows a continuous refinement model for aggregation. For this reason, we never implement a pre-aggregation step before data repartitioning, because it won't help much to reduce repartition cost (there is no natural boundary when a pre-aggregation is finished and when to emit it downstream for the actual aggregation roll-up).

With https://issues.apache.org/jira/browse/KAFKA-13785 we introduce a per-aggregation "emit final" feature (different to suppress()) that changes the continuous refinement model and thus it seems to be a good optimization to add a pre-aggregation step if this new feature is used.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)