[jira] [Created] (KAFKA-10369) Introduce Distinct operation in KStream

["Ivan Ponomarev (Jira)" <ji...@apache.org>]

Ivan Ponomarev created KAFKA-10369:
--------------------------------------

             Summary: Introduce Distinct operation in KStream
                 Key: KAFKA-10369
                 URL: https://issues.apache.org/jira/browse/KAFKA-10369
             Project: Kafka
          Issue Type: Improvement
            Reporter: Ivan Ponomarev
            Assignee: Ivan Ponomarev


Message deduplication is a common task.

One example: we might have multiple data sources each reporting its state periodically with a relatively high frequency, their current states should be stored in a database. In case the actual change of the state occurs with a lower frequency than it is reported, in order to reduce the number of writes to the database we might want to filter out duplicated messages using Kafka Streams.

'Distinct' operation is common in data processing, e. g.
 * Java Stream has [distinct() |https://docs.oracle.com/javase/8/docs/api/java/util/stream/Stream.html#distinct--] operation,
 * SQL has DISTINCT keyword.

 

Hence it is natural to expect the similar functionality from Kafka Streams.

Although Kafka Streams Tutorials contains an [example|https://kafka-tutorials.confluent.io/finding-distinct-events/kstreams.html] of how distinct can be emulated , but this example is complicated: it involves low-level coding with local state store and a custom transformer. It might be much more convenient to have distinct as a first-class DSL operation.

Due to 'infinite' nature of KStream, distinct operation should be windowed, similar to windowed joins and aggregations for KStreams.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)