You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Dong Lin (JIRA)" <ji...@apache.org> on 2018/10/03 06:58:00 UTC
[jira] [Updated] (KAFKA-3596) Kafka Streams: Window expiration
needs to consider more than event time
[ https://issues.apache.org/jira/browse/KAFKA-3596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dong Lin updated KAFKA-3596:
----------------------------
Fix Version/s: (was: 2.1.0)
2.2.0
> Kafka Streams: Window expiration needs to consider more than event time
> -----------------------------------------------------------------------
>
> Key: KAFKA-3596
> URL: https://issues.apache.org/jira/browse/KAFKA-3596
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Affects Versions: 0.10.0.0
> Reporter: Henry Cai
> Priority: Minor
> Labels: architecture
> Fix For: 2.2.0
>
>
> Currently in Kafka Streams, the way the windows are expired in RocksDB is triggered by new event insertion. When a window is created at T0 with 10 minutes retention, when we saw a new record coming with event timestamp T0 + 10 +1, we will expire that window (remove it) out of RocksDB.
> In the real world, it's very easy to see event coming with future timestamp (or out-of-order events coming with big time gaps between events), this way of retiring a window based on one event's event timestamp is dangerous. I think at least we need to consider both the event's event time and server/stream time elapse.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)