You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Burak Yavuz (JIRA)" <ji...@apache.org> on 2017/07/11 00:42:00 UTC
[jira] [Created] (SPARK-21370) Clarify In-Memory State Store
purpose (read-only, read-write) with an additional state
Burak Yavuz created SPARK-21370:
-----------------------------------
Summary: Clarify In-Memory State Store purpose (read-only, read-write) with an additional state
Key: SPARK-21370
URL: https://issues.apache.org/jira/browse/SPARK-21370
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 2.1.1
Reporter: Burak Yavuz
Assignee: Burak Yavuz
Currently the HDFSBackedStateStore sets it's state as UPDATING as it is initialized.
For every trigger, we create two state stores, one used during "Restore" and one during "Save". The "Restore" StateStore is read-only. This state store gets "aborted" after a task is completed, which results in a file being created and immediately deleted.
This can be avoided if there is an INITIALIZED state and abort deletes files only when there is an update to the state store using "put" or "remove".
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org