You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Praveen Gattu (Jira)" <ji...@apache.org> on 2023/10/16 18:53:00 UTC
[jira] [Commented] (SPARK-45511) SPIP: State Data Source - Reader
[ https://issues.apache.org/jira/browse/SPARK-45511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17775886#comment-17775886 ]
Praveen Gattu commented on SPARK-45511:
---------------------------------------
Thank you Jungtaek for starting this discussion. Made a few comments in the doc. Making State less of a blackbox is very important for our customers and is the number 1 ask for customers running stateful applications.
> SPIP: State Data Source - Reader
> --------------------------------
>
> Key: SPARK-45511
> URL: https://issues.apache.org/jira/browse/SPARK-45511
> Project: Spark
> Issue Type: New Feature
> Components: Structured Streaming
> Affects Versions: 4.0.0
> Reporter: Jungtaek Lim
> Priority: Major
> Labels: SPIP
>
> State Store has been a black box from the introduction of the stateful operator. It has been the “internal” data to the streaming query, and Spark does not expose the data outside of the streaming query. There is no feature/tool for users to read and modify the content of state stores.
> Specific to the ability to read the state, the lack of feature brings up various limitations like following:
> * Users are unable to see the content in the state store, leading to inability to debug.
> * Users have to perform some indirect approach on verifying the content of the state store in unit tests. The only option they can take is relying on the output of the query.
> Given that, we propose to introduce a feature which enables users to read the state from the outside of the streaming query.
> SPIP: [https://docs.google.com/document/d/1_iVf_CIu2RZd3yWWF6KoRNlBiz5NbSIK0yThqG0EvPY/edit?usp=sharing]
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org