[jira] [Created] (KAFKA-4325) Improve processing of late records for window operations

["Matthias J. Sax (JIRA)" <ji...@apache.org>]

Matthias J. Sax created KAFKA-4325:
--------------------------------------

             Summary: Improve processing of late records for window operations
                 Key: KAFKA-4325
                 URL: https://issues.apache.org/jira/browse/KAFKA-4325
             Project: Kafka
          Issue Type: Improvement
          Components: streams
            Reporter: Matthias J. Sax
            Assignee: Guozhang Wang
            Priority: Minor


Windows are kept until their retention time passed. If a late arriving record is processed that is older than any window kept, a new window is created containing this single late arriving record, the aggregation is computed and the window is immediately discarded afterward (as it is older than retention time).

This behavior might case problems for downstream application as the original window aggregate might we overwritten with the late single-record- aggregate value. Thus, we should rather not process the late arriving record for this case.

However, data loss might not be acceptable for all use cases. In order to enable the use to not lose any data, window operators should allow to register a handler function that is called instead of just dropping the late arriving record.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)