[jira] [Created] (SPARK-48772) State Data Source Read Change Feed

["Yuchen Liu (Jira)" <ji...@apache.org>]

Yuchen Liu created SPARK-48772:
----------------------------------

             Summary: State Data Source Read Change Feed
                 Key: SPARK-48772
                 URL: https://issues.apache.org/jira/browse/SPARK-48772
             Project: Spark
          Issue Type: New Feature
          Components: Structured Streaming
    Affects Versions: 4.0.0
            Reporter: Yuchen Liu


The current state reader can only return the entire state at a specific version. If an error occurs related to state, knowing the change of state across versions to find out at which version state starts to go wrong is important for debugging purposes. This adds ability of showing the evolution of state as Change Data Capture (CDC) format to state data source.

An example usage:

 
{code:java}
.format("statestore")
.option("readChangeFeed", true)
.option("changeStartBatchId", 5) #required 
.option("changeEndBatchId", 10)  #not required, default: latest batch Id available
{code}
 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org