[jira] [Updated] (SPARK-19932) Disallow a case that might cause OOM for steaming deduplication

["Liwei Lin (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-19932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Liwei Lin updated SPARK-19932:
------------------------------
    Summary: Disallow a case that might cause OOM for steaming deduplication  (was: Disallow a case that might case OOM for steaming deduplication)

> Disallow a case that might cause OOM for steaming deduplication
> ---------------------------------------------------------------
>
>                 Key: SPARK-19932
>                 URL: https://issues.apache.org/jira/browse/SPARK-19932
>             Project: Spark
>          Issue Type: Improvement
>          Components: Structured Streaming
>    Affects Versions: 2.1.0
>            Reporter: Liwei Lin
>
> {code}
> spark
>    .readStream                 // schema: (word, eventTime), like ("a", 10), ("a", 11), ("b", 12) ...
>    ...
>    .withWatermark("eventTime", "10 seconds")
>    .dropDuplicates("word")     // note: "eventTime" is not part of the key columns
>    ...
> {code}
> As shown above, right now if watermark is specified for a streaming dropDuplicates query, but not specified as the key columns, then we'll still get the correct answer, but the state just keeps growing and will never get cleaned up.
> The reason is, the watermark attribute is not part of the key of the state store in this case. We're not saving event time information in the state store.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org