[jira] [Created] (SPARK-18124) Implement watermarking for handling late data

["Tathagata Das (JIRA)" <ji...@apache.org>]

Tathagata Das created SPARK-18124:
-------------------------------------

             Summary: Implement watermarking for handling late data
                 Key: SPARK-18124
                 URL: https://issues.apache.org/jira/browse/SPARK-18124
             Project: Spark
          Issue Type: Sub-task
            Reporter: Tathagata Das


Whenever we aggregate data by event time, we want to consider data is late and out-of-order in terms of its event time. Since we keep aggregate keyed by the time as state, the state will grow unbounded if we keep around all old aggregates in an attempt consider arbitrarily late data. Since the state is a store in-memory, we have to prevent building up of this unbounded state. Hence, we need a watermarking mechanism by which we will mark data that is older beyond a threshold as “too late”, and stop updating the aggregates with them. This would allow us to remove old aggregates that are never going to be updated, thus bounding the size of the state.

Here is the design doc - https://docs.google.com/document/d/1z-Pazs5v4rA31azvmYhu4I5xwqaNQl6ZLIS03xhkfCQ/edit?usp=sharing



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org