[jira] [Commented] (KAFKA-20438) Allow dropping "future" records

["Arpit Goyal (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-20438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18073321#comment-18073321 ] 

Arpit Goyal commented on KAFKA-20438:
-------------------------------------

[~mjsax]  I am picking this up

> Allow dropping "future" records
> -------------------------------
>
>                 Key: KAFKA-20438
>                 URL: https://issues.apache.org/jira/browse/KAFKA-20438
>             Project: Kafka
>          Issue Type: New Feature
>          Components: streams
>            Reporter: Matthias J. Sax
>            Assignee: Arpit Goyal
>            Priority: Major
>              Labels: needs-kip
>
> In the Kafka Streams DSL, certain stateful operators advance their state based on stream-time, ie, the highest observes record timestamp over all input records.
> In particular, window- and session-stores (and other segmented state stores) roll segments base on stream-time, and also apply window-close and grace-periods base on it.
> The issue is, that a single malformed record with a timestamp far in the future, might advance stream-time incorrectly, thus triggering closing windows and rolling segments too early, and there is no good way to recover from it.
> Currently, users could build manual filtering and compare a record timestamps to current stream-time (which is tracked by the KS runtime) to avoid such issues.
> We propose to add a built-in mechanism (either via a global config, or a per-window parameter like "grace period"), to let Kafka Streams drop such malformed future records automatically.
> Note: while it could also make sense to write such records into a DLQ, we should tackle DLQ support separately.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)