[jira] [Created] (KAFKA-7608) A Kafka Streams DSL transform or processor call should trigger a repartition like a join

["Andy Bryant (JIRA)" <ji...@apache.org>]

Andy Bryant created KAFKA-7608:
----------------------------------

             Summary: A Kafka Streams DSL transform or processor call should trigger a repartition like a join
                 Key: KAFKA-7608
                 URL: https://issues.apache.org/jira/browse/KAFKA-7608
             Project: Kafka
          Issue Type: Bug
          Components: streams
    Affects Versions: 2.0.0
            Reporter: Andy Bryant


Currently in Kafka Streams, if any DSL operation occurs that may modify the keys of the record stream, the stream is flagged for repartitioning. Currently this flag is checked prior to a stream join or an aggregation and if set the stream is piped through a transient repartition topic. This ensures messages with the same key are always co-located in the same partition and hence same stream task and state store.

The same mechanism should be used to trigger repartitioning prior to stream {{transform}}, {{transformValues}} and {{process}} calls that specify one or more state stores.

Currently without the forced repartitioning, for streams where the key has been modified, there is no guarantee the same keys will be processed by the same task which would be what you expect when using a state store. Given that aggregations and joins already automatically make this guarantee it seems inconsistent that {{transform}} and {{process}} do not provide the same guarantees.

To achieve the same guarantees currently, developers must manually pipe the stream through a topic to force the repartitioning. This works, but is sub-optimal since you don't get the handy optimisation where the repartition topic contents is purged after use.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)