You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "DB Tsai (Jira)" <ji...@apache.org> on 2026/05/22 23:56:00 UTC
[jira] [Assigned] (SPARK-56249) Auto CDC support
[ https://issues.apache.org/jira/browse/SPARK-56249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
DB Tsai reassigned SPARK-56249:
-------------------------------
Assignee: Andreas Neumann
> Auto CDC support
> ----------------
>
> Key: SPARK-56249
> URL: https://issues.apache.org/jira/browse/SPARK-56249
> Project: Spark
> Issue Type: Umbrella
> Components: Declarative Pipelines, PySpark, SQL
> Affects Versions: 4.2.0
> Reporter: Andreas Neumann
> Assignee: Andreas Neumann
> Priority: Major
> Labels: SPIP, pull-request-available
>
> *SPIP Document:* [SPIP: Auto CDC|https://docs.google.com/document/d/1Hp5BGEYJRHbk6J7XUph3bAPZKRQXKOuV1PEaqZMMRoQ/]
> *Motivation*
> With the upcoming introduction of standardized CDC support, Spark will soon have a unified way to _produce_ change data feeds. However, _consuming_ these feeds and applying them to a target table remains a significant challenge.
> Common patterns like *SCD Type 1* (maintaining a 1:1 replica) and *SCD Type 2* (tracking full change history) often require hand-crafted, complex MERGE logic. In distributed systems, these implementations are frequently error-prone when handling deletions or out-of-order data.
> *Proposal*
> This SPIP proposes a new *"Auto CDC"* flow type for Spark. It encapsulates the complex logic for SCD types and out-of-order data, allowing data engineers to configure a declarative flow instead of writing manual MERGE statements. This feature will be available in both {*}Python and SQL{*}.
> Example SQL:
> {code:java}
> -- Produce a change feed
> CREATE STREAMING TABLE cdc.users AS
> SELECT * FROM STREAM my_table CHANGES FROM VERSION 10;
> -- Consume the change feed
> CREATE FLOW flow
> AS AUTO CDC INTO
> target
> FROM stream(cdc_data.users)
> KEYS (userId)
> APPLY AS DELETE WHEN operation = "DELETE"
> SEQUENCE BY sequenceNum
> COLUMNS * EXCEPT (operation, sequenceNum)
> STORED AS SCD TYPE 2
> TRACK HISTORY ON * EXCEPT (city);
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org