You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Tathagata Das (JIRA)" <ji...@apache.org> on 2016/11/15 21:42:59 UTC

[jira] [Resolved] (SPARK-18337) Memory Sink should be able to recover from checkpoints in Complete OutputMode

     [ https://issues.apache.org/jira/browse/SPARK-18337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tathagata Das resolved SPARK-18337.
-----------------------------------
       Resolution: Fixed
         Assignee: Burak Yavuz
    Fix Version/s: 2.1.0
                   2.0.3

> Memory Sink should be able to recover from checkpoints in Complete OutputMode
> -----------------------------------------------------------------------------
>
>                 Key: SPARK-18337
>                 URL: https://issues.apache.org/jira/browse/SPARK-18337
>             Project: Spark
>          Issue Type: Improvement
>          Components: Structured Streaming
>    Affects Versions: 2.0.1
>            Reporter: Burak Yavuz
>            Assignee: Burak Yavuz
>             Fix For: 2.0.3, 2.1.0
>
>
> Memory sinks are not meant to be fault tolerant, but there are certain cases, where it would be nice that it can recover from checkpoints. In cases where you may use a scalable StateStore in StructuredStreaming (when you have an aggregation), and you add a filter based on a key or value in your state, it's nice to be able to continue from where you left off after failures.
> For correctness reasons, the output will ONLY be correct in Complete mode, so we could support that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org