[jira] [Created] (SPARK-45511) SPIP: State Data Source - Reader

["Jungtaek Lim (Jira)" <ji...@apache.org>]

Jungtaek Lim created SPARK-45511:
------------------------------------

             Summary: SPIP: State Data Source - Reader
                 Key: SPARK-45511
                 URL: https://issues.apache.org/jira/browse/SPARK-45511
             Project: Spark
          Issue Type: New Feature
          Components: Structured Streaming
    Affects Versions: 4.0.0
            Reporter: Jungtaek Lim


State Store has been a black box from the introduction of the stateful operator. It has been the “internal” data to the streaming query, and Spark does not expose the data outside of the streaming query. There is no feature/tool for users to read and modify the content of state stores.

Specific to the ability to read the state, the lack of feature brings up various limitations like following:
 * Users are unable to see the content in the state store, leading to inability to debug.
 * Users have to perform some indirect approach on verifying the content of the state store in unit tests. The only option they can take is relying on the output of the query.

Given that, we propose to introduce a feature which enables users to read the state from the outside of the streaming query.

SPIP: [https://docs.google.com/document/d/1HjEupRv8TRFeULtJuxRq_tEG1Wq-9UNu-ctGgCYRke0/edit?usp=sharing]

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org