[jira] [Created] (KAFKA-3596) Kafka Streams: Window expiration needs to consider more than event time

["Henry Cai (JIRA)" <ji...@apache.org>]

Henry Cai created KAFKA-3596:
--------------------------------

             Summary: Kafka Streams: Window expiration needs to consider more than event time
                 Key: KAFKA-3596
                 URL: https://issues.apache.org/jira/browse/KAFKA-3596
             Project: Kafka
          Issue Type: Improvement
          Components: streams
    Affects Versions: 0.10.1.0
            Reporter: Henry Cai
            Assignee: Guozhang Wang
            Priority: Minor


Currently state store replication always go through a compact kafka topic.  For some state stores, e.g. JoinWindow, there are no duplicates in the store, there is not much benefit using a compacted topic.

The problem of using compacted topic is the records can stay in kafka broker forever.  In my use case, my key is ad_id, it's incrementing all the time, not bounded, I am worried the disk space on broker for that topic will go forever.

I think we either need the capability to purge the compacted records on broker, or allow us to specify different compact option for state store replication.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)