You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Yuchen Liu (Jira)" <ji...@apache.org> on 2024/07/01 23:16:00 UTC
[jira] [Updated] (SPARK-48772) State Data Source Read Change Feed
[ https://issues.apache.org/jira/browse/SPARK-48772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yuchen Liu updated SPARK-48772:
-------------------------------
Description:
The current state reader can only return the entire state at a specific version. If an error occurs related to state, knowing the change of state across versions to find out at which version state starts to go wrong is important for debugging purposes. This adds ability of showing the evolution of state as Change Data Capture (CDC) format to state data source.
An example usage:
{code:java}
.format("statestore")
.option("readChangeFeed", true)
.option("changeStartBatchId", 5) #required
.option("changeEndBatchId", 10) #not required, default: latest batch Id available
{code}
was:
The current state reader can only return the entire state at a specific version. If an error occurs related to state, knowing the change of state across versions to find out at which version state starts to go wrong is important for debugging purposes. This adds ability of showing the evolution of state as Change Data Capture (CDC) format to state data source.
An example usage:
{code:java}
.format("statestore")
.option("readChangeFeed", true)
.option("changeStartBatchId", 5) #required
.option("changeEndBatchId", 10) #not required, default: latest batch Id available
{code}
> State Data Source Read Change Feed
> ----------------------------------
>
> Key: SPARK-48772
> URL: https://issues.apache.org/jira/browse/SPARK-48772
> Project: Spark
> Issue Type: New Feature
> Components: Structured Streaming
> Affects Versions: 4.0.0
> Reporter: Yuchen Liu
> Priority: Major
>
> The current state reader can only return the entire state at a specific version. If an error occurs related to state, knowing the change of state across versions to find out at which version state starts to go wrong is important for debugging purposes. This adds ability of showing the evolution of state as Change Data Capture (CDC) format to state data source.
> An example usage:
> {code:java}
> .format("statestore")
> .option("readChangeFeed", true)
> .option("changeStartBatchId", 5) #required
> .option("changeEndBatchId", 10) #not required, default: latest batch Id available
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org