You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Burak Yavuz (JIRA)" <ji...@apache.org> on 2017/07/11 00:42:00 UTC

[jira] [Created] (SPARK-21370) Clarify In-Memory State Store purpose (read-only, read-write) with an additional state

Burak Yavuz created SPARK-21370:
-----------------------------------

             Summary: Clarify In-Memory State Store purpose (read-only, read-write) with an additional state
                 Key: SPARK-21370
                 URL: https://issues.apache.org/jira/browse/SPARK-21370
             Project: Spark
          Issue Type: Improvement
          Components: Structured Streaming
    Affects Versions: 2.1.1
            Reporter: Burak Yavuz
            Assignee: Burak Yavuz


Currently the HDFSBackedStateStore sets it's state as UPDATING as it is initialized.

For every trigger, we create two state stores, one used during "Restore" and one during "Save". The "Restore" StateStore is read-only. This state store gets "aborted" after a task is completed, which results in a file being created and immediately deleted.

This can be avoided if there is an INITIALIZED state and abort deletes files only when there is an update to the state store using "put" or "remove".



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org