You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Andreas Neumann <an...@apache.org> on 2026/04/03 21:59:48 UTC
[VOTE] SPIP: Auto CDC Support for Apache Spark
Hi Spark devs,
I'd like to call a vote on the SPIP*: Auto CDC Support for Apache Spark*
Motivation
With the upcoming introduction of standardized CDC support
<https://issues.apache.org/jira/browse/SPARK-55668>, Spark will soon have a
unified way to produce change data feeds. However, consuming these feeds
and applying them to a target table remains a significant challenge.
Common patterns like SCD Type 1 (maintaining a 1:1 replica) and SCD
Type 2 (tracking
full change history) often require hand-crafted, complex MERGE logic. In
distributed systems, these implementations are frequently error-prone when
handling deletions or out-of-order data.
Proposal
This SPIP proposes a new "Auto CDC" flow type for Spark. It encapsulates
the complex logic for SCD types and out-of-order data, allowing data
engineers to configure a declarative flow instead of writing manual
MERGE statements.
This feature will be available in both Python and SQL.
Example SQL:
-- Produce a change feed
CREATE STREAMING TABLE cdc.users AS
SELECT * FROM STREAM my_table CHANGES FROM VERSION 10;
-- Consume the change feed
CREATE FLOW flow
AS AUTO CDC INTO
target
FROM stream(cdc_data.users)
KEYS (userId)
APPLY AS DELETE WHEN operation = "DELETE"
SEQUENCE BY sequenceNum
COLUMNS * EXCEPT (operation, sequenceNum)
STORED AS SCD TYPE 2
TRACK HISTORY ON * EXCEPT (city);
*Relevant Links:*
- SPIP Document:
https://docs.google.com/document/d/1Hp5BGEYJRHbk6J7XUph3bAPZKRQXKOuV1PEaqZMMRoQ/
-
*Discussion Thread: *
https://lists.apache.org/thread/j6sj9wo9odgdpgzlxtvhoy7szs0jplf7
-
JIRA: <https://issues.apache.org/jira/browse/SPARK-55668>
https://issues.apache.org/jira/browse/SPARK-56249
*The vote will be open for at least 72 hours. *Please vote:
[ ] +1: Accept the proposal as an official SPIP
[ ] +0
[ ] -1: I don't think this is a good idea because ...
Cheers -Andreas
Re: [VOTE] SPIP: Auto CDC Support for Apache Spark
Posted by Andreas Neumann <an...@apache.org>.
Thank you all. I will close the vote now.
On Tue, Apr 7, 2026 at 8:50 AM Dongjoon Hyun <do...@apache.org> wrote:
> +1
>
> Dongjoon
>
> On 2026/04/07 07:18:19 Kousuke Saruta wrote:
> > +1
> >
> >
> > 2026年4月7日(火) 16:17 Peter Toth <pe...@gmail.com>:
> >
> > > +1 (non-binding)
> > >
> > > On Mon, Apr 6, 2026 at 10:13 PM Jungtaek Lim <
> kabhwan.opensource@gmail.com>
> > > wrote:
> > >
> > >> +1 (non-binding)
> > >>
> > >> On Tue, Apr 7, 2026 at 3:11 AM Liu Cao <tw...@gmail.com> wrote:
> > >>
> > >>> +1 (non-binding)
> > >>>
> > >>> On Mon, Apr 6, 2026 at 11:10 AM John Zhuge <jz...@apache.org>
> wrote:
> > >>>
> > >>>> +1 (non-binding)
> > >>>>
> > >>>> On Mon, Apr 6, 2026 at 11:07 AM Holden Karau <
> holden.karau@gmail.com>
> > >>>> wrote:
> > >>>>
> > >>>>> +1 (binding)
> > >>>>>
> > >>>>> On Mon, Apr 6, 2026 at 10:58 AM Anish Shrigondekar via dev <
> > >>>>> dev@spark.apache.org> wrote:
> > >>>>>
> > >>>>>> +1
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>> Anish
> > >>>>>>
> > >>>>>> On Mon, Apr 6, 2026 at 10:57 AM DB Tsai <db...@dbtsai.com>
> wrote:
> > >>>>>>
> > >>>>>>> +1 (binding)
> > >>>>>>>
> > >>>>>>> DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1
> > >>>>>>>
> > >>>>>>> On Apr 3, 2026, at 2:59 PM, Andreas Neumann <an...@apache.org>
> wrote:
> > >>>>>>>
> > >>>>>>> Hi Spark devs,
> > >>>>>>>
> > >>>>>>> I'd like to call a vote on the SPIP*: Auto CDC Support for Apache
> > >>>>>>> Spark*
> > >>>>>>> Motivation
> > >>>>>>>
> > >>>>>>> With the upcoming introduction of standardized CDC support
> > >>>>>>> <https://issues.apache.org/jira/browse/SPARK-55668>, Spark will
> > >>>>>>> soon have a unified way to produce change data feeds. However,
> > >>>>>>> consuming these feeds and applying them to a target table
> remains a
> > >>>>>>> significant challenge.
> > >>>>>>>
> > >>>>>>> Common patterns like SCD Type 1 (maintaining a 1:1 replica) and
> SCD
> > >>>>>>> Type 2 (tracking full change history) often require hand-crafted,
> > >>>>>>> complex MERGE logic. In distributed systems, these
> implementations
> > >>>>>>> are frequently error-prone when handling deletions or
> out-of-order data.
> > >>>>>>> Proposal
> > >>>>>>>
> > >>>>>>> This SPIP proposes a new "Auto CDC" flow type for Spark. It
> > >>>>>>> encapsulates the complex logic for SCD types and out-of-order
> data,
> > >>>>>>> allowing data engineers to configure a declarative flow instead
> of writing
> > >>>>>>> manual MERGE statements. This feature will be available in both
> Python
> > >>>>>>> and SQL.
> > >>>>>>> Example SQL:
> > >>>>>>> -- Produce a change feed
> > >>>>>>> CREATE STREAMING TABLE cdc.users AS
> > >>>>>>> SELECT * FROM STREAM my_table CHANGES FROM VERSION 10;
> > >>>>>>>
> > >>>>>>> -- Consume the change feed
> > >>>>>>> CREATE FLOW flow
> > >>>>>>> AS AUTO CDC INTO
> > >>>>>>> target
> > >>>>>>> FROM stream(cdc_data.users)
> > >>>>>>> KEYS (userId)
> > >>>>>>> APPLY AS DELETE WHEN operation = "DELETE"
> > >>>>>>> SEQUENCE BY sequenceNum
> > >>>>>>> COLUMNS * EXCEPT (operation, sequenceNum)
> > >>>>>>> STORED AS SCD TYPE 2
> > >>>>>>> TRACK HISTORY ON * EXCEPT (city);
> > >>>>>>>
> > >>>>>>> *Relevant Links:*
> > >>>>>>>
> > >>>>>>> - SPIP Document:
> > >>>>>>>
> https://docs.google.com/document/d/1Hp5BGEYJRHbk6J7XUph3bAPZKRQXKOuV1PEaqZMMRoQ/
> > >>>>>>> - *Discussion Thread: *
> > >>>>>>>
> https://lists.apache.org/thread/j6sj9wo9odgdpgzlxtvhoy7szs0jplf7
> > >>>>>>> -
> > >>>>>>>
> > >>>>>>> JIRA: <https://issues.apache.org/jira/browse/SPARK-55668>
> > >>>>>>> https://issues.apache.org/jira/browse/SPARK-56249
> > >>>>>>>
> > >>>>>>> *The vote will be open for at least 72 hours. *Please vote:
> > >>>>>>>
> > >>>>>>> [ ] +1: Accept the proposal as an official SPIP
> > >>>>>>> [ ] +0
> > >>>>>>> [ ] -1: I don't think this is a good idea because ...
> > >>>>>>> Cheers -Andreas
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>
> > >>>>> --
> > >>>>> Twitter: https://twitter.com/holdenkarau
> > >>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/
> > >>>>> <https://www.fighthealthinsurance.com/?q=hk_email>
> > >>>>> Books (Learning Spark, High Performance Spark, etc.):
> > >>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
> > >>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
> > >>>>> Pronouns: she/her
> > >>>>>
> > >>>>
> > >>>>
> > >>>> --
> > >>>> John Zhuge
> > >>>>
> > >>>
> > >>>
> > >>> --
> > >>>
> > >>> Liu Cao
> > >>>
> > >>>
> > >>>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
Re: [VOTE] SPIP: Auto CDC Support for Apache Spark
Posted by Dongjoon Hyun <do...@apache.org>.
+1
Dongjoon
On 2026/04/07 07:18:19 Kousuke Saruta wrote:
> +1
>
>
> 2026年4月7日(火) 16:17 Peter Toth <pe...@gmail.com>:
>
> > +1 (non-binding)
> >
> > On Mon, Apr 6, 2026 at 10:13 PM Jungtaek Lim <ka...@gmail.com>
> > wrote:
> >
> >> +1 (non-binding)
> >>
> >> On Tue, Apr 7, 2026 at 3:11 AM Liu Cao <tw...@gmail.com> wrote:
> >>
> >>> +1 (non-binding)
> >>>
> >>> On Mon, Apr 6, 2026 at 11:10 AM John Zhuge <jz...@apache.org> wrote:
> >>>
> >>>> +1 (non-binding)
> >>>>
> >>>> On Mon, Apr 6, 2026 at 11:07 AM Holden Karau <ho...@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> +1 (binding)
> >>>>>
> >>>>> On Mon, Apr 6, 2026 at 10:58 AM Anish Shrigondekar via dev <
> >>>>> dev@spark.apache.org> wrote:
> >>>>>
> >>>>>> +1
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Anish
> >>>>>>
> >>>>>> On Mon, Apr 6, 2026 at 10:57 AM DB Tsai <db...@dbtsai.com> wrote:
> >>>>>>
> >>>>>>> +1 (binding)
> >>>>>>>
> >>>>>>> DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1
> >>>>>>>
> >>>>>>> On Apr 3, 2026, at 2:59 PM, Andreas Neumann <an...@apache.org> wrote:
> >>>>>>>
> >>>>>>> Hi Spark devs,
> >>>>>>>
> >>>>>>> I'd like to call a vote on the SPIP*: Auto CDC Support for Apache
> >>>>>>> Spark*
> >>>>>>> Motivation
> >>>>>>>
> >>>>>>> With the upcoming introduction of standardized CDC support
> >>>>>>> <https://issues.apache.org/jira/browse/SPARK-55668>, Spark will
> >>>>>>> soon have a unified way to produce change data feeds. However,
> >>>>>>> consuming these feeds and applying them to a target table remains a
> >>>>>>> significant challenge.
> >>>>>>>
> >>>>>>> Common patterns like SCD Type 1 (maintaining a 1:1 replica) and SCD
> >>>>>>> Type 2 (tracking full change history) often require hand-crafted,
> >>>>>>> complex MERGE logic. In distributed systems, these implementations
> >>>>>>> are frequently error-prone when handling deletions or out-of-order data.
> >>>>>>> Proposal
> >>>>>>>
> >>>>>>> This SPIP proposes a new "Auto CDC" flow type for Spark. It
> >>>>>>> encapsulates the complex logic for SCD types and out-of-order data,
> >>>>>>> allowing data engineers to configure a declarative flow instead of writing
> >>>>>>> manual MERGE statements. This feature will be available in both Python
> >>>>>>> and SQL.
> >>>>>>> Example SQL:
> >>>>>>> -- Produce a change feed
> >>>>>>> CREATE STREAMING TABLE cdc.users AS
> >>>>>>> SELECT * FROM STREAM my_table CHANGES FROM VERSION 10;
> >>>>>>>
> >>>>>>> -- Consume the change feed
> >>>>>>> CREATE FLOW flow
> >>>>>>> AS AUTO CDC INTO
> >>>>>>> target
> >>>>>>> FROM stream(cdc_data.users)
> >>>>>>> KEYS (userId)
> >>>>>>> APPLY AS DELETE WHEN operation = "DELETE"
> >>>>>>> SEQUENCE BY sequenceNum
> >>>>>>> COLUMNS * EXCEPT (operation, sequenceNum)
> >>>>>>> STORED AS SCD TYPE 2
> >>>>>>> TRACK HISTORY ON * EXCEPT (city);
> >>>>>>>
> >>>>>>> *Relevant Links:*
> >>>>>>>
> >>>>>>> - SPIP Document:
> >>>>>>> https://docs.google.com/document/d/1Hp5BGEYJRHbk6J7XUph3bAPZKRQXKOuV1PEaqZMMRoQ/
> >>>>>>> - *Discussion Thread: *
> >>>>>>> https://lists.apache.org/thread/j6sj9wo9odgdpgzlxtvhoy7szs0jplf7
> >>>>>>> -
> >>>>>>>
> >>>>>>> JIRA: <https://issues.apache.org/jira/browse/SPARK-55668>
> >>>>>>> https://issues.apache.org/jira/browse/SPARK-56249
> >>>>>>>
> >>>>>>> *The vote will be open for at least 72 hours. *Please vote:
> >>>>>>>
> >>>>>>> [ ] +1: Accept the proposal as an official SPIP
> >>>>>>> [ ] +0
> >>>>>>> [ ] -1: I don't think this is a good idea because ...
> >>>>>>> Cheers -Andreas
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>>> --
> >>>>> Twitter: https://twitter.com/holdenkarau
> >>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/
> >>>>> <https://www.fighthealthinsurance.com/?q=hk_email>
> >>>>> Books (Learning Spark, High Performance Spark, etc.):
> >>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
> >>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
> >>>>> Pronouns: she/her
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> John Zhuge
> >>>>
> >>>
> >>>
> >>> --
> >>>
> >>> Liu Cao
> >>>
> >>>
> >>>
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: [VOTE] SPIP: Auto CDC Support for Apache Spark
Posted by Kousuke Saruta <sa...@apache.org>.
+1
2026年4月7日(火) 16:17 Peter Toth <pe...@gmail.com>:
> +1 (non-binding)
>
> On Mon, Apr 6, 2026 at 10:13 PM Jungtaek Lim <ka...@gmail.com>
> wrote:
>
>> +1 (non-binding)
>>
>> On Tue, Apr 7, 2026 at 3:11 AM Liu Cao <tw...@gmail.com> wrote:
>>
>>> +1 (non-binding)
>>>
>>> On Mon, Apr 6, 2026 at 11:10 AM John Zhuge <jz...@apache.org> wrote:
>>>
>>>> +1 (non-binding)
>>>>
>>>> On Mon, Apr 6, 2026 at 11:07 AM Holden Karau <ho...@gmail.com>
>>>> wrote:
>>>>
>>>>> +1 (binding)
>>>>>
>>>>> On Mon, Apr 6, 2026 at 10:58 AM Anish Shrigondekar via dev <
>>>>> dev@spark.apache.org> wrote:
>>>>>
>>>>>> +1
>>>>>>
>>>>>> Thanks,
>>>>>> Anish
>>>>>>
>>>>>> On Mon, Apr 6, 2026 at 10:57 AM DB Tsai <db...@dbtsai.com> wrote:
>>>>>>
>>>>>>> +1 (binding)
>>>>>>>
>>>>>>> DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1
>>>>>>>
>>>>>>> On Apr 3, 2026, at 2:59 PM, Andreas Neumann <an...@apache.org> wrote:
>>>>>>>
>>>>>>> Hi Spark devs,
>>>>>>>
>>>>>>> I'd like to call a vote on the SPIP*: Auto CDC Support for Apache
>>>>>>> Spark*
>>>>>>> Motivation
>>>>>>>
>>>>>>> With the upcoming introduction of standardized CDC support
>>>>>>> <https://issues.apache.org/jira/browse/SPARK-55668>, Spark will
>>>>>>> soon have a unified way to produce change data feeds. However,
>>>>>>> consuming these feeds and applying them to a target table remains a
>>>>>>> significant challenge.
>>>>>>>
>>>>>>> Common patterns like SCD Type 1 (maintaining a 1:1 replica) and SCD
>>>>>>> Type 2 (tracking full change history) often require hand-crafted,
>>>>>>> complex MERGE logic. In distributed systems, these implementations
>>>>>>> are frequently error-prone when handling deletions or out-of-order data.
>>>>>>> Proposal
>>>>>>>
>>>>>>> This SPIP proposes a new "Auto CDC" flow type for Spark. It
>>>>>>> encapsulates the complex logic for SCD types and out-of-order data,
>>>>>>> allowing data engineers to configure a declarative flow instead of writing
>>>>>>> manual MERGE statements. This feature will be available in both Python
>>>>>>> and SQL.
>>>>>>> Example SQL:
>>>>>>> -- Produce a change feed
>>>>>>> CREATE STREAMING TABLE cdc.users AS
>>>>>>> SELECT * FROM STREAM my_table CHANGES FROM VERSION 10;
>>>>>>>
>>>>>>> -- Consume the change feed
>>>>>>> CREATE FLOW flow
>>>>>>> AS AUTO CDC INTO
>>>>>>> target
>>>>>>> FROM stream(cdc_data.users)
>>>>>>> KEYS (userId)
>>>>>>> APPLY AS DELETE WHEN operation = "DELETE"
>>>>>>> SEQUENCE BY sequenceNum
>>>>>>> COLUMNS * EXCEPT (operation, sequenceNum)
>>>>>>> STORED AS SCD TYPE 2
>>>>>>> TRACK HISTORY ON * EXCEPT (city);
>>>>>>>
>>>>>>> *Relevant Links:*
>>>>>>>
>>>>>>> - SPIP Document:
>>>>>>> https://docs.google.com/document/d/1Hp5BGEYJRHbk6J7XUph3bAPZKRQXKOuV1PEaqZMMRoQ/
>>>>>>> - *Discussion Thread: *
>>>>>>> https://lists.apache.org/thread/j6sj9wo9odgdpgzlxtvhoy7szs0jplf7
>>>>>>> -
>>>>>>>
>>>>>>> JIRA: <https://issues.apache.org/jira/browse/SPARK-55668>
>>>>>>> https://issues.apache.org/jira/browse/SPARK-56249
>>>>>>>
>>>>>>> *The vote will be open for at least 72 hours. *Please vote:
>>>>>>>
>>>>>>> [ ] +1: Accept the proposal as an official SPIP
>>>>>>> [ ] +0
>>>>>>> [ ] -1: I don't think this is a good idea because ...
>>>>>>> Cheers -Andreas
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>> --
>>>>> Twitter: https://twitter.com/holdenkarau
>>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>>>>> <https://www.fighthealthinsurance.com/?q=hk_email>
>>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>>> Pronouns: she/her
>>>>>
>>>>
>>>>
>>>> --
>>>> John Zhuge
>>>>
>>>
>>>
>>> --
>>>
>>> Liu Cao
>>>
>>>
>>>
Re: [VOTE] SPIP: Auto CDC Support for Apache Spark
Posted by Peter Toth <pe...@gmail.com>.
+1 (non-binding)
On Mon, Apr 6, 2026 at 10:13 PM Jungtaek Lim <ka...@gmail.com>
wrote:
> +1 (non-binding)
>
> On Tue, Apr 7, 2026 at 3:11 AM Liu Cao <tw...@gmail.com> wrote:
>
>> +1 (non-binding)
>>
>> On Mon, Apr 6, 2026 at 11:10 AM John Zhuge <jz...@apache.org> wrote:
>>
>>> +1 (non-binding)
>>>
>>> On Mon, Apr 6, 2026 at 11:07 AM Holden Karau <ho...@gmail.com>
>>> wrote:
>>>
>>>> +1 (binding)
>>>>
>>>> On Mon, Apr 6, 2026 at 10:58 AM Anish Shrigondekar via dev <
>>>> dev@spark.apache.org> wrote:
>>>>
>>>>> +1
>>>>>
>>>>> Thanks,
>>>>> Anish
>>>>>
>>>>> On Mon, Apr 6, 2026 at 10:57 AM DB Tsai <db...@dbtsai.com> wrote:
>>>>>
>>>>>> +1 (binding)
>>>>>>
>>>>>> DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1
>>>>>>
>>>>>> On Apr 3, 2026, at 2:59 PM, Andreas Neumann <an...@apache.org> wrote:
>>>>>>
>>>>>> Hi Spark devs,
>>>>>>
>>>>>> I'd like to call a vote on the SPIP*: Auto CDC Support for Apache
>>>>>> Spark*
>>>>>> Motivation
>>>>>>
>>>>>> With the upcoming introduction of standardized CDC support
>>>>>> <https://issues.apache.org/jira/browse/SPARK-55668>, Spark will soon
>>>>>> have a unified way to produce change data feeds. However, consuming these
>>>>>> feeds and applying them to a target table remains a significant challenge.
>>>>>>
>>>>>> Common patterns like SCD Type 1 (maintaining a 1:1 replica) and SCD
>>>>>> Type 2 (tracking full change history) often require hand-crafted,
>>>>>> complex MERGE logic. In distributed systems, these implementations
>>>>>> are frequently error-prone when handling deletions or out-of-order data.
>>>>>> Proposal
>>>>>>
>>>>>> This SPIP proposes a new "Auto CDC" flow type for Spark. It
>>>>>> encapsulates the complex logic for SCD types and out-of-order data,
>>>>>> allowing data engineers to configure a declarative flow instead of writing
>>>>>> manual MERGE statements. This feature will be available in both Python
>>>>>> and SQL.
>>>>>> Example SQL:
>>>>>> -- Produce a change feed
>>>>>> CREATE STREAMING TABLE cdc.users AS
>>>>>> SELECT * FROM STREAM my_table CHANGES FROM VERSION 10;
>>>>>>
>>>>>> -- Consume the change feed
>>>>>> CREATE FLOW flow
>>>>>> AS AUTO CDC INTO
>>>>>> target
>>>>>> FROM stream(cdc_data.users)
>>>>>> KEYS (userId)
>>>>>> APPLY AS DELETE WHEN operation = "DELETE"
>>>>>> SEQUENCE BY sequenceNum
>>>>>> COLUMNS * EXCEPT (operation, sequenceNum)
>>>>>> STORED AS SCD TYPE 2
>>>>>> TRACK HISTORY ON * EXCEPT (city);
>>>>>>
>>>>>> *Relevant Links:*
>>>>>>
>>>>>> - SPIP Document:
>>>>>> https://docs.google.com/document/d/1Hp5BGEYJRHbk6J7XUph3bAPZKRQXKOuV1PEaqZMMRoQ/
>>>>>> - *Discussion Thread: *
>>>>>> https://lists.apache.org/thread/j6sj9wo9odgdpgzlxtvhoy7szs0jplf7
>>>>>> -
>>>>>>
>>>>>> JIRA: <https://issues.apache.org/jira/browse/SPARK-55668>
>>>>>> https://issues.apache.org/jira/browse/SPARK-56249
>>>>>>
>>>>>> *The vote will be open for at least 72 hours. *Please vote:
>>>>>>
>>>>>> [ ] +1: Accept the proposal as an official SPIP
>>>>>> [ ] +0
>>>>>> [ ] -1: I don't think this is a good idea because ...
>>>>>> Cheers -Andreas
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>> --
>>>> Twitter: https://twitter.com/holdenkarau
>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>>>> <https://www.fighthealthinsurance.com/?q=hk_email>
>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>> Pronouns: she/her
>>>>
>>>
>>>
>>> --
>>> John Zhuge
>>>
>>
>>
>> --
>>
>> Liu Cao
>>
>>
>>
Re: [VOTE] SPIP: Auto CDC Support for Apache Spark
Posted by Jungtaek Lim <ka...@gmail.com>.
+1 (non-binding)
On Tue, Apr 7, 2026 at 3:11 AM Liu Cao <tw...@gmail.com> wrote:
> +1 (non-binding)
>
> On Mon, Apr 6, 2026 at 11:10 AM John Zhuge <jz...@apache.org> wrote:
>
>> +1 (non-binding)
>>
>> On Mon, Apr 6, 2026 at 11:07 AM Holden Karau <ho...@gmail.com>
>> wrote:
>>
>>> +1 (binding)
>>>
>>> On Mon, Apr 6, 2026 at 10:58 AM Anish Shrigondekar via dev <
>>> dev@spark.apache.org> wrote:
>>>
>>>> +1
>>>>
>>>> Thanks,
>>>> Anish
>>>>
>>>> On Mon, Apr 6, 2026 at 10:57 AM DB Tsai <db...@dbtsai.com> wrote:
>>>>
>>>>> +1 (binding)
>>>>>
>>>>> DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1
>>>>>
>>>>> On Apr 3, 2026, at 2:59 PM, Andreas Neumann <an...@apache.org> wrote:
>>>>>
>>>>> Hi Spark devs,
>>>>>
>>>>> I'd like to call a vote on the SPIP*: Auto CDC Support for Apache
>>>>> Spark*
>>>>> Motivation
>>>>>
>>>>> With the upcoming introduction of standardized CDC support
>>>>> <https://issues.apache.org/jira/browse/SPARK-55668>, Spark will soon
>>>>> have a unified way to produce change data feeds. However, consuming these
>>>>> feeds and applying them to a target table remains a significant challenge.
>>>>>
>>>>> Common patterns like SCD Type 1 (maintaining a 1:1 replica) and SCD
>>>>> Type 2 (tracking full change history) often require hand-crafted,
>>>>> complex MERGE logic. In distributed systems, these implementations
>>>>> are frequently error-prone when handling deletions or out-of-order data.
>>>>> Proposal
>>>>>
>>>>> This SPIP proposes a new "Auto CDC" flow type for Spark. It
>>>>> encapsulates the complex logic for SCD types and out-of-order data,
>>>>> allowing data engineers to configure a declarative flow instead of writing
>>>>> manual MERGE statements. This feature will be available in both Python
>>>>> and SQL.
>>>>> Example SQL:
>>>>> -- Produce a change feed
>>>>> CREATE STREAMING TABLE cdc.users AS
>>>>> SELECT * FROM STREAM my_table CHANGES FROM VERSION 10;
>>>>>
>>>>> -- Consume the change feed
>>>>> CREATE FLOW flow
>>>>> AS AUTO CDC INTO
>>>>> target
>>>>> FROM stream(cdc_data.users)
>>>>> KEYS (userId)
>>>>> APPLY AS DELETE WHEN operation = "DELETE"
>>>>> SEQUENCE BY sequenceNum
>>>>> COLUMNS * EXCEPT (operation, sequenceNum)
>>>>> STORED AS SCD TYPE 2
>>>>> TRACK HISTORY ON * EXCEPT (city);
>>>>>
>>>>> *Relevant Links:*
>>>>>
>>>>> - SPIP Document:
>>>>> https://docs.google.com/document/d/1Hp5BGEYJRHbk6J7XUph3bAPZKRQXKOuV1PEaqZMMRoQ/
>>>>> - *Discussion Thread: *
>>>>> https://lists.apache.org/thread/j6sj9wo9odgdpgzlxtvhoy7szs0jplf7
>>>>> -
>>>>>
>>>>> JIRA: <https://issues.apache.org/jira/browse/SPARK-55668>
>>>>> https://issues.apache.org/jira/browse/SPARK-56249
>>>>>
>>>>> *The vote will be open for at least 72 hours. *Please vote:
>>>>>
>>>>> [ ] +1: Accept the proposal as an official SPIP
>>>>> [ ] +0
>>>>> [ ] -1: I don't think this is a good idea because ...
>>>>> Cheers -Andreas
>>>>>
>>>>>
>>>>>
>>>>>
>>>
>>> --
>>> Twitter: https://twitter.com/holdenkarau
>>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>>> <https://www.fighthealthinsurance.com/?q=hk_email>
>>> Books (Learning Spark, High Performance Spark, etc.):
>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>> Pronouns: she/her
>>>
>>
>>
>> --
>> John Zhuge
>>
>
>
> --
>
> Liu Cao
>
>
>
Re: [VOTE] SPIP: Auto CDC Support for Apache Spark
Posted by Liu Cao <tw...@gmail.com>.
+1 (non-binding)
On Mon, Apr 6, 2026 at 11:10 AM John Zhuge <jz...@apache.org> wrote:
> +1 (non-binding)
>
> On Mon, Apr 6, 2026 at 11:07 AM Holden Karau <ho...@gmail.com>
> wrote:
>
>> +1 (binding)
>>
>> On Mon, Apr 6, 2026 at 10:58 AM Anish Shrigondekar via dev <
>> dev@spark.apache.org> wrote:
>>
>>> +1
>>>
>>> Thanks,
>>> Anish
>>>
>>> On Mon, Apr 6, 2026 at 10:57 AM DB Tsai <db...@dbtsai.com> wrote:
>>>
>>>> +1 (binding)
>>>>
>>>> DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1
>>>>
>>>> On Apr 3, 2026, at 2:59 PM, Andreas Neumann <an...@apache.org> wrote:
>>>>
>>>> Hi Spark devs,
>>>>
>>>> I'd like to call a vote on the SPIP*: Auto CDC Support for Apache
>>>> Spark*
>>>> Motivation
>>>>
>>>> With the upcoming introduction of standardized CDC support
>>>> <https://issues.apache.org/jira/browse/SPARK-55668>, Spark will soon
>>>> have a unified way to produce change data feeds. However, consuming these
>>>> feeds and applying them to a target table remains a significant challenge.
>>>>
>>>> Common patterns like SCD Type 1 (maintaining a 1:1 replica) and SCD
>>>> Type 2 (tracking full change history) often require hand-crafted,
>>>> complex MERGE logic. In distributed systems, these implementations are
>>>> frequently error-prone when handling deletions or out-of-order data.
>>>> Proposal
>>>>
>>>> This SPIP proposes a new "Auto CDC" flow type for Spark. It
>>>> encapsulates the complex logic for SCD types and out-of-order data,
>>>> allowing data engineers to configure a declarative flow instead of writing
>>>> manual MERGE statements. This feature will be available in both Python
>>>> and SQL.
>>>> Example SQL:
>>>> -- Produce a change feed
>>>> CREATE STREAMING TABLE cdc.users AS
>>>> SELECT * FROM STREAM my_table CHANGES FROM VERSION 10;
>>>>
>>>> -- Consume the change feed
>>>> CREATE FLOW flow
>>>> AS AUTO CDC INTO
>>>> target
>>>> FROM stream(cdc_data.users)
>>>> KEYS (userId)
>>>> APPLY AS DELETE WHEN operation = "DELETE"
>>>> SEQUENCE BY sequenceNum
>>>> COLUMNS * EXCEPT (operation, sequenceNum)
>>>> STORED AS SCD TYPE 2
>>>> TRACK HISTORY ON * EXCEPT (city);
>>>>
>>>> *Relevant Links:*
>>>>
>>>> - SPIP Document:
>>>> https://docs.google.com/document/d/1Hp5BGEYJRHbk6J7XUph3bAPZKRQXKOuV1PEaqZMMRoQ/
>>>> - *Discussion Thread: *
>>>> https://lists.apache.org/thread/j6sj9wo9odgdpgzlxtvhoy7szs0jplf7
>>>> -
>>>>
>>>> JIRA: <https://issues.apache.org/jira/browse/SPARK-55668>
>>>> https://issues.apache.org/jira/browse/SPARK-56249
>>>>
>>>> *The vote will be open for at least 72 hours. *Please vote:
>>>>
>>>> [ ] +1: Accept the proposal as an official SPIP
>>>> [ ] +0
>>>> [ ] -1: I don't think this is a good idea because ...
>>>> Cheers -Andreas
>>>>
>>>>
>>>>
>>>>
>>
>> --
>> Twitter: https://twitter.com/holdenkarau
>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>> <https://www.fighthealthinsurance.com/?q=hk_email>
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>> Pronouns: she/her
>>
>
>
> --
> John Zhuge
>
--
Liu Cao
Re: [VOTE] SPIP: Auto CDC Support for Apache Spark
Posted by John Zhuge <jz...@apache.org>.
+1 (non-binding)
On Mon, Apr 6, 2026 at 11:07 AM Holden Karau <ho...@gmail.com> wrote:
> +1 (binding)
>
> On Mon, Apr 6, 2026 at 10:58 AM Anish Shrigondekar via dev <
> dev@spark.apache.org> wrote:
>
>> +1
>>
>> Thanks,
>> Anish
>>
>> On Mon, Apr 6, 2026 at 10:57 AM DB Tsai <db...@dbtsai.com> wrote:
>>
>>> +1 (binding)
>>>
>>> DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1
>>>
>>> On Apr 3, 2026, at 2:59 PM, Andreas Neumann <an...@apache.org> wrote:
>>>
>>> Hi Spark devs,
>>>
>>> I'd like to call a vote on the SPIP*: Auto CDC Support for Apache Spark*
>>> Motivation
>>>
>>> With the upcoming introduction of standardized CDC support
>>> <https://issues.apache.org/jira/browse/SPARK-55668>, Spark will soon
>>> have a unified way to produce change data feeds. However, consuming these
>>> feeds and applying them to a target table remains a significant challenge.
>>>
>>> Common patterns like SCD Type 1 (maintaining a 1:1 replica) and SCD
>>> Type 2 (tracking full change history) often require hand-crafted,
>>> complex MERGE logic. In distributed systems, these implementations are
>>> frequently error-prone when handling deletions or out-of-order data.
>>> Proposal
>>>
>>> This SPIP proposes a new "Auto CDC" flow type for Spark. It
>>> encapsulates the complex logic for SCD types and out-of-order data,
>>> allowing data engineers to configure a declarative flow instead of writing
>>> manual MERGE statements. This feature will be available in both Python
>>> and SQL.
>>> Example SQL:
>>> -- Produce a change feed
>>> CREATE STREAMING TABLE cdc.users AS
>>> SELECT * FROM STREAM my_table CHANGES FROM VERSION 10;
>>>
>>> -- Consume the change feed
>>> CREATE FLOW flow
>>> AS AUTO CDC INTO
>>> target
>>> FROM stream(cdc_data.users)
>>> KEYS (userId)
>>> APPLY AS DELETE WHEN operation = "DELETE"
>>> SEQUENCE BY sequenceNum
>>> COLUMNS * EXCEPT (operation, sequenceNum)
>>> STORED AS SCD TYPE 2
>>> TRACK HISTORY ON * EXCEPT (city);
>>>
>>> *Relevant Links:*
>>>
>>> - SPIP Document:
>>> https://docs.google.com/document/d/1Hp5BGEYJRHbk6J7XUph3bAPZKRQXKOuV1PEaqZMMRoQ/
>>> - *Discussion Thread: *
>>> https://lists.apache.org/thread/j6sj9wo9odgdpgzlxtvhoy7szs0jplf7
>>> -
>>>
>>> JIRA: <https://issues.apache.org/jira/browse/SPARK-55668>
>>> https://issues.apache.org/jira/browse/SPARK-56249
>>>
>>> *The vote will be open for at least 72 hours. *Please vote:
>>>
>>> [ ] +1: Accept the proposal as an official SPIP
>>> [ ] +0
>>> [ ] -1: I don't think this is a good idea because ...
>>> Cheers -Andreas
>>>
>>>
>>>
>>>
>
> --
> Twitter: https://twitter.com/holdenkarau
> Fight Health Insurance: https://www.fighthealthinsurance.com/
> <https://www.fighthealthinsurance.com/?q=hk_email>
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
> Pronouns: she/her
>
--
John Zhuge
Re: [VOTE] SPIP: Auto CDC Support for Apache Spark
Posted by Holden Karau <ho...@gmail.com>.
+1 (binding)
On Mon, Apr 6, 2026 at 10:58 AM Anish Shrigondekar via dev <
dev@spark.apache.org> wrote:
> +1
>
> Thanks,
> Anish
>
> On Mon, Apr 6, 2026 at 10:57 AM DB Tsai <db...@dbtsai.com> wrote:
>
>> +1 (binding)
>>
>> DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1
>>
>> On Apr 3, 2026, at 2:59 PM, Andreas Neumann <an...@apache.org> wrote:
>>
>> Hi Spark devs,
>>
>> I'd like to call a vote on the SPIP*: Auto CDC Support for Apache Spark*
>> Motivation
>>
>> With the upcoming introduction of standardized CDC support
>> <https://issues.apache.org/jira/browse/SPARK-55668>, Spark will soon
>> have a unified way to produce change data feeds. However, consuming these
>> feeds and applying them to a target table remains a significant challenge.
>>
>> Common patterns like SCD Type 1 (maintaining a 1:1 replica) and SCD Type
>> 2 (tracking full change history) often require hand-crafted, complex
>> MERGE logic. In distributed systems, these implementations are
>> frequently error-prone when handling deletions or out-of-order data.
>> Proposal
>>
>> This SPIP proposes a new "Auto CDC" flow type for Spark. It encapsulates
>> the complex logic for SCD types and out-of-order data, allowing data
>> engineers to configure a declarative flow instead of writing manual MERGE statements.
>> This feature will be available in both Python and SQL.
>> Example SQL:
>> -- Produce a change feed
>> CREATE STREAMING TABLE cdc.users AS
>> SELECT * FROM STREAM my_table CHANGES FROM VERSION 10;
>>
>> -- Consume the change feed
>> CREATE FLOW flow
>> AS AUTO CDC INTO
>> target
>> FROM stream(cdc_data.users)
>> KEYS (userId)
>> APPLY AS DELETE WHEN operation = "DELETE"
>> SEQUENCE BY sequenceNum
>> COLUMNS * EXCEPT (operation, sequenceNum)
>> STORED AS SCD TYPE 2
>> TRACK HISTORY ON * EXCEPT (city);
>>
>> *Relevant Links:*
>>
>> - SPIP Document:
>> https://docs.google.com/document/d/1Hp5BGEYJRHbk6J7XUph3bAPZKRQXKOuV1PEaqZMMRoQ/
>> - *Discussion Thread: *
>> https://lists.apache.org/thread/j6sj9wo9odgdpgzlxtvhoy7szs0jplf7
>> -
>>
>> JIRA: <https://issues.apache.org/jira/browse/SPARK-55668>
>> https://issues.apache.org/jira/browse/SPARK-56249
>>
>> *The vote will be open for at least 72 hours. *Please vote:
>>
>> [ ] +1: Accept the proposal as an official SPIP
>> [ ] +0
>> [ ] -1: I don't think this is a good idea because ...
>> Cheers -Andreas
>>
>>
>>
>>
--
Twitter: https://twitter.com/holdenkarau
Fight Health Insurance: https://www.fighthealthinsurance.com/
<https://www.fighthealthinsurance.com/?q=hk_email>
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
YouTube Live Streams: https://www.youtube.com/user/holdenkarau
Pronouns: she/her
Re: [VOTE] SPIP: Auto CDC Support for Apache Spark
Posted by Anish Shrigondekar via dev <de...@spark.apache.org>.
+1
Thanks,
Anish
On Mon, Apr 6, 2026 at 10:57 AM DB Tsai <db...@dbtsai.com> wrote:
> +1 (binding)
>
> DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1
>
> On Apr 3, 2026, at 2:59 PM, Andreas Neumann <an...@apache.org> wrote:
>
> Hi Spark devs,
>
> I'd like to call a vote on the SPIP*: Auto CDC Support for Apache Spark*
> Motivation
>
> With the upcoming introduction of standardized CDC support
> <https://issues.apache.org/jira/browse/SPARK-55668>, Spark will soon have
> a unified way to produce change data feeds. However, consuming these
> feeds and applying them to a target table remains a significant challenge.
>
> Common patterns like SCD Type 1 (maintaining a 1:1 replica) and SCD Type 2 (tracking
> full change history) often require hand-crafted, complex MERGE logic. In
> distributed systems, these implementations are frequently error-prone when
> handling deletions or out-of-order data.
> Proposal
>
> This SPIP proposes a new "Auto CDC" flow type for Spark. It encapsulates
> the complex logic for SCD types and out-of-order data, allowing data
> engineers to configure a declarative flow instead of writing manual MERGE statements.
> This feature will be available in both Python and SQL.
> Example SQL:
> -- Produce a change feed
> CREATE STREAMING TABLE cdc.users AS
> SELECT * FROM STREAM my_table CHANGES FROM VERSION 10;
>
> -- Consume the change feed
> CREATE FLOW flow
> AS AUTO CDC INTO
> target
> FROM stream(cdc_data.users)
> KEYS (userId)
> APPLY AS DELETE WHEN operation = "DELETE"
> SEQUENCE BY sequenceNum
> COLUMNS * EXCEPT (operation, sequenceNum)
> STORED AS SCD TYPE 2
> TRACK HISTORY ON * EXCEPT (city);
>
> *Relevant Links:*
>
> - SPIP Document:
> https://docs.google.com/document/d/1Hp5BGEYJRHbk6J7XUph3bAPZKRQXKOuV1PEaqZMMRoQ/
> - *Discussion Thread: *
> https://lists.apache.org/thread/j6sj9wo9odgdpgzlxtvhoy7szs0jplf7
> -
>
> JIRA: <https://issues.apache.org/jira/browse/SPARK-55668>
> https://issues.apache.org/jira/browse/SPARK-56249
>
> *The vote will be open for at least 72 hours. *Please vote:
>
> [ ] +1: Accept the proposal as an official SPIP
> [ ] +0
> [ ] -1: I don't think this is a good idea because ...
> Cheers -Andreas
>
>
>
>
Re: [VOTE] SPIP: Auto CDC Support for Apache Spark
Posted by DB Tsai <db...@dbtsai.com>.
+1 (binding)
DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1
> On Apr 3, 2026, at 2:59 PM, Andreas Neumann <an...@apache.org> wrote:
>
> Hi Spark devs,
>
> I'd like to call a vote on the SPIP: Auto CDC Support for Apache Spark
>
> Motivation
> With the upcoming introduction of standardized CDC support <https://issues.apache.org/jira/browse/SPARK-55668>, Spark will soon have a unified way to produce change data feeds. However, consuming these feeds and applying them to a target table remains a significant challenge.
>
> Common patterns like SCD Type 1 (maintaining a 1:1 replica) and SCD Type 2 (tracking full change history) often require hand-crafted, complex MERGE logic. In distributed systems, these implementations are frequently error-prone when handling deletions or out-of-order data.
>
> Proposal
> This SPIP proposes a new "Auto CDC" flow type for Spark. It encapsulates the complex logic for SCD types and out-of-order data, allowing data engineers to configure a declarative flow instead of writing manual MERGE statements. This feature will be available in both Python and SQL.
>
> Example SQL:
> -- Produce a change feed
> CREATE STREAMING TABLE cdc.users AS
> SELECT * FROM STREAM my_table CHANGES FROM VERSION 10;
>
> -- Consume the change feed
> CREATE FLOW flow
> AS AUTO CDC INTO
> target
> FROM stream(cdc_data.users)
> KEYS (userId)
> APPLY AS DELETE WHEN operation = "DELETE"
> SEQUENCE BY sequenceNum
> COLUMNS * EXCEPT (operation, sequenceNum)
> STORED AS SCD TYPE 2
> TRACK HISTORY ON * EXCEPT (city);
>
> Relevant Links:
> SPIP Document: https://docs.google.com/document/d/1Hp5BGEYJRHbk6J7XUph3bAPZKRQXKOuV1PEaqZMMRoQ/
> Discussion Thread: https://lists.apache.org/thread/j6sj9wo9odgdpgzlxtvhoy7szs0jplf7
> JIRA: <https://issues.apache.org/jira/browse/SPARK-55668>https://issues.apache.org/jira/browse/SPARK-56249
>
> The vote will be open for at least 72 hours. Please vote:
>
> [ ] +1: Accept the proposal as an official SPIP
> [ ] +0
> [ ] -1: I don't think this is a good idea because ...
>
> Cheers -Andreas
>
>
Re: [VOTE] SPIP: Auto CDC Support for Apache Spark
Posted by huaxin gao <hu...@gmail.com>.
+1
On Mon, Apr 6, 2026 at 10:10 AM Anton Okolnychyi <ao...@gmail.com>
wrote:
> +1 (non-binding)
>
> сб, 4 квіт. 2026 р. о 11:55 Gengliang Wang <lt...@gmail.com> пише:
>
>> +1
>>
>> On Sat, Apr 4, 2026 at 10:17 AM Xiao Li <ga...@gmail.com> wrote:
>>
>>> +1
>>>
>>> vaquar khan <va...@gmail.com> 于2026年4月4日周六 09:45写道：
>>>
>>>> +1
>>>>
>>>> Regards,
>>>> Viquar Khan
>>>>
>>>> On Sat, 4 Apr 2026 at 11:14, Lisa N. Cao <li...@gmail.com>
>>>> wrote:
>>>>
>>>>> +1 (non-binding)
>>>>>
>>>>> --
>>>>> LNC
>>>>>
>>>>> On Fri, Apr 3, 2026, 5:15 PM Shixiong Zhu <zs...@gmail.com> wrote:
>>>>>
>>>>>> +1
>>>>>>
>>>>>>
>>>>>> On Fri, Apr 3, 2026 at 5:03 PM Mich Talebzadeh <
>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>
>>>>>>> +1
>>>>>>>
>>>>>>> Dr Mich Talebzadeh,
>>>>>>> Data Scientist | Distributed Systems (Spark) | Financial Forensics &
>>>>>>> Metadata Analytics | Transaction Reconstruction | Audit & Evidence-Based
>>>>>>> Analytics
>>>>>>>
>>>>>>> view my Linkedin profile
>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, 3 Apr 2026 at 23:00, Andreas Neumann <an...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Spark devs,
>>>>>>>>
>>>>>>>> I'd like to call a vote on the SPIP*: Auto CDC Support for Apache
>>>>>>>> Spark*
>>>>>>>> Motivation
>>>>>>>>
>>>>>>>> With the upcoming introduction of standardized CDC support
>>>>>>>> <https://issues.apache.org/jira/browse/SPARK-55668>, Spark will
>>>>>>>> soon have a unified way to produce change data feeds. However,
>>>>>>>> consuming these feeds and applying them to a target table remains
>>>>>>>> a significant challenge.
>>>>>>>>
>>>>>>>> Common patterns like SCD Type 1 (maintaining a 1:1 replica) and SCD
>>>>>>>> Type 2 (tracking full change history) often require hand-crafted,
>>>>>>>> complex MERGE logic. In distributed systems, these implementations
>>>>>>>> are frequently error-prone when handling deletions or out-of-order data.
>>>>>>>> Proposal
>>>>>>>>
>>>>>>>> This SPIP proposes a new "Auto CDC" flow type for Spark. It
>>>>>>>> encapsulates the complex logic for SCD types and out-of-order data,
>>>>>>>> allowing data engineers to configure a declarative flow instead of writing
>>>>>>>> manual MERGE statements. This feature will be available in both Python
>>>>>>>> and SQL.
>>>>>>>>
>>>>>>>> Example SQL:
>>>>>>>>
>>>>>>>> -- Produce a change feed
>>>>>>>>
>>>>>>>> CREATE STREAMING TABLE cdc.users AS
>>>>>>>>
>>>>>>>> SELECT * FROM STREAM my_table CHANGES FROM VERSION 10;
>>>>>>>>
>>>>>>>>
>>>>>>>> -- Consume the change feed
>>>>>>>>
>>>>>>>> CREATE FLOW flow
>>>>>>>>
>>>>>>>> AS AUTO CDC INTO
>>>>>>>>
>>>>>>>> target
>>>>>>>>
>>>>>>>> FROM stream(cdc_data.users)
>>>>>>>>
>>>>>>>> KEYS (userId)
>>>>>>>>
>>>>>>>> APPLY AS DELETE WHEN operation = "DELETE"
>>>>>>>>
>>>>>>>> SEQUENCE BY sequenceNum
>>>>>>>>
>>>>>>>> COLUMNS * EXCEPT (operation, sequenceNum)
>>>>>>>>
>>>>>>>> STORED AS SCD TYPE 2
>>>>>>>>
>>>>>>>> TRACK HISTORY ON * EXCEPT (city);
>>>>>>>>
>>>>>>>>
>>>>>>>> *Relevant Links:*
>>>>>>>>
>>>>>>>> - SPIP Document:
>>>>>>>> https://docs.google.com/document/d/1Hp5BGEYJRHbk6J7XUph3bAPZKRQXKOuV1PEaqZMMRoQ/
>>>>>>>> -
>>>>>>>>
>>>>>>>> *Discussion Thread: *
>>>>>>>> https://lists.apache.org/thread/j6sj9wo9odgdpgzlxtvhoy7szs0jplf7
>>>>>>>> -
>>>>>>>>
>>>>>>>> JIRA: <https://issues.apache.org/jira/browse/SPARK-55668>
>>>>>>>> https://issues.apache.org/jira/browse/SPARK-56249
>>>>>>>>
>>>>>>>> *The vote will be open for at least 72 hours. *Please vote:
>>>>>>>>
>>>>>>>> [ ] +1: Accept the proposal as an official SPIP
>>>>>>>> [ ] +0
>>>>>>>> [ ] -1: I don't think this is a good idea because ...
>>>>>>>> Cheers -Andreas
>>>>>>>>
>>>>>>>>
>>>>>>>>
Re: [VOTE] SPIP: Auto CDC Support for Apache Spark
Posted by Anton Okolnychyi <ao...@gmail.com>.
+1 (non-binding)
сб, 4 квіт. 2026 р. о 11:55 Gengliang Wang <lt...@gmail.com> пише:
> +1
>
> On Sat, Apr 4, 2026 at 10:17 AM Xiao Li <ga...@gmail.com> wrote:
>
>> +1
>>
>> vaquar khan <va...@gmail.com> 于2026年4月4日周六 09:45写道：
>>
>>> +1
>>>
>>> Regards,
>>> Viquar Khan
>>>
>>> On Sat, 4 Apr 2026 at 11:14, Lisa N. Cao <li...@gmail.com>
>>> wrote:
>>>
>>>> +1 (non-binding)
>>>>
>>>> --
>>>> LNC
>>>>
>>>> On Fri, Apr 3, 2026, 5:15 PM Shixiong Zhu <zs...@gmail.com> wrote:
>>>>
>>>>> +1
>>>>>
>>>>>
>>>>> On Fri, Apr 3, 2026 at 5:03 PM Mich Talebzadeh <
>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>
>>>>>> +1
>>>>>>
>>>>>> Dr Mich Talebzadeh,
>>>>>> Data Scientist | Distributed Systems (Spark) | Financial Forensics &
>>>>>> Metadata Analytics | Transaction Reconstruction | Audit & Evidence-Based
>>>>>> Analytics
>>>>>>
>>>>>> view my Linkedin profile
>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, 3 Apr 2026 at 23:00, Andreas Neumann <an...@apache.org> wrote:
>>>>>>
>>>>>>> Hi Spark devs,
>>>>>>>
>>>>>>> I'd like to call a vote on the SPIP*: Auto CDC Support for Apache
>>>>>>> Spark*
>>>>>>> Motivation
>>>>>>>
>>>>>>> With the upcoming introduction of standardized CDC support
>>>>>>> <https://issues.apache.org/jira/browse/SPARK-55668>, Spark will
>>>>>>> soon have a unified way to produce change data feeds. However,
>>>>>>> consuming these feeds and applying them to a target table remains a
>>>>>>> significant challenge.
>>>>>>>
>>>>>>> Common patterns like SCD Type 1 (maintaining a 1:1 replica) and SCD
>>>>>>> Type 2 (tracking full change history) often require hand-crafted,
>>>>>>> complex MERGE logic. In distributed systems, these implementations
>>>>>>> are frequently error-prone when handling deletions or out-of-order data.
>>>>>>> Proposal
>>>>>>>
>>>>>>> This SPIP proposes a new "Auto CDC" flow type for Spark. It
>>>>>>> encapsulates the complex logic for SCD types and out-of-order data,
>>>>>>> allowing data engineers to configure a declarative flow instead of writing
>>>>>>> manual MERGE statements. This feature will be available in both Python
>>>>>>> and SQL.
>>>>>>>
>>>>>>> Example SQL:
>>>>>>>
>>>>>>> -- Produce a change feed
>>>>>>>
>>>>>>> CREATE STREAMING TABLE cdc.users AS
>>>>>>>
>>>>>>> SELECT * FROM STREAM my_table CHANGES FROM VERSION 10;
>>>>>>>
>>>>>>>
>>>>>>> -- Consume the change feed
>>>>>>>
>>>>>>> CREATE FLOW flow
>>>>>>>
>>>>>>> AS AUTO CDC INTO
>>>>>>>
>>>>>>> target
>>>>>>>
>>>>>>> FROM stream(cdc_data.users)
>>>>>>>
>>>>>>> KEYS (userId)
>>>>>>>
>>>>>>> APPLY AS DELETE WHEN operation = "DELETE"
>>>>>>>
>>>>>>> SEQUENCE BY sequenceNum
>>>>>>>
>>>>>>> COLUMNS * EXCEPT (operation, sequenceNum)
>>>>>>>
>>>>>>> STORED AS SCD TYPE 2
>>>>>>>
>>>>>>> TRACK HISTORY ON * EXCEPT (city);
>>>>>>>
>>>>>>>
>>>>>>> *Relevant Links:*
>>>>>>>
>>>>>>> - SPIP Document:
>>>>>>> https://docs.google.com/document/d/1Hp5BGEYJRHbk6J7XUph3bAPZKRQXKOuV1PEaqZMMRoQ/
>>>>>>> -
>>>>>>>
>>>>>>> *Discussion Thread: *
>>>>>>> https://lists.apache.org/thread/j6sj9wo9odgdpgzlxtvhoy7szs0jplf7
>>>>>>> -
>>>>>>>
>>>>>>> JIRA: <https://issues.apache.org/jira/browse/SPARK-55668>
>>>>>>> https://issues.apache.org/jira/browse/SPARK-56249
>>>>>>>
>>>>>>> *The vote will be open for at least 72 hours. *Please vote:
>>>>>>>
>>>>>>> [ ] +1: Accept the proposal as an official SPIP
>>>>>>> [ ] +0
>>>>>>> [ ] -1: I don't think this is a good idea because ...
>>>>>>> Cheers -Andreas
>>>>>>>
>>>>>>>
>>>>>>>
Re: [VOTE] SPIP: Auto CDC Support for Apache Spark
Posted by Gengliang Wang <lt...@gmail.com>.
+1
On Sat, Apr 4, 2026 at 10:17 AM Xiao Li <ga...@gmail.com> wrote:
> +1
>
> vaquar khan <va...@gmail.com> 于2026年4月4日周六 09:45写道：
>
>> +1
>>
>> Regards,
>> Viquar Khan
>>
>> On Sat, 4 Apr 2026 at 11:14, Lisa N. Cao <li...@gmail.com>
>> wrote:
>>
>>> +1 (non-binding)
>>>
>>> --
>>> LNC
>>>
>>> On Fri, Apr 3, 2026, 5:15 PM Shixiong Zhu <zs...@gmail.com> wrote:
>>>
>>>> +1
>>>>
>>>>
>>>> On Fri, Apr 3, 2026 at 5:03 PM Mich Talebzadeh <
>>>> mich.talebzadeh@gmail.com> wrote:
>>>>
>>>>> +1
>>>>>
>>>>> Dr Mich Talebzadeh,
>>>>> Data Scientist | Distributed Systems (Spark) | Financial Forensics &
>>>>> Metadata Analytics | Transaction Reconstruction | Audit & Evidence-Based
>>>>> Analytics
>>>>>
>>>>> view my Linkedin profile
>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, 3 Apr 2026 at 23:00, Andreas Neumann <an...@apache.org> wrote:
>>>>>
>>>>>> Hi Spark devs,
>>>>>>
>>>>>> I'd like to call a vote on the SPIP*: Auto CDC Support for Apache
>>>>>> Spark*
>>>>>> Motivation
>>>>>>
>>>>>> With the upcoming introduction of standardized CDC support
>>>>>> <https://issues.apache.org/jira/browse/SPARK-55668>, Spark will soon
>>>>>> have a unified way to produce change data feeds. However, consuming these
>>>>>> feeds and applying them to a target table remains a significant challenge.
>>>>>>
>>>>>> Common patterns like SCD Type 1 (maintaining a 1:1 replica) and SCD
>>>>>> Type 2 (tracking full change history) often require hand-crafted,
>>>>>> complex MERGE logic. In distributed systems, these implementations
>>>>>> are frequently error-prone when handling deletions or out-of-order data.
>>>>>> Proposal
>>>>>>
>>>>>> This SPIP proposes a new "Auto CDC" flow type for Spark. It
>>>>>> encapsulates the complex logic for SCD types and out-of-order data,
>>>>>> allowing data engineers to configure a declarative flow instead of writing
>>>>>> manual MERGE statements. This feature will be available in both Python
>>>>>> and SQL.
>>>>>>
>>>>>> Example SQL:
>>>>>>
>>>>>> -- Produce a change feed
>>>>>>
>>>>>> CREATE STREAMING TABLE cdc.users AS
>>>>>>
>>>>>> SELECT * FROM STREAM my_table CHANGES FROM VERSION 10;
>>>>>>
>>>>>>
>>>>>> -- Consume the change feed
>>>>>>
>>>>>> CREATE FLOW flow
>>>>>>
>>>>>> AS AUTO CDC INTO
>>>>>>
>>>>>> target
>>>>>>
>>>>>> FROM stream(cdc_data.users)
>>>>>>
>>>>>> KEYS (userId)
>>>>>>
>>>>>> APPLY AS DELETE WHEN operation = "DELETE"
>>>>>>
>>>>>> SEQUENCE BY sequenceNum
>>>>>>
>>>>>> COLUMNS * EXCEPT (operation, sequenceNum)
>>>>>>
>>>>>> STORED AS SCD TYPE 2
>>>>>>
>>>>>> TRACK HISTORY ON * EXCEPT (city);
>>>>>>
>>>>>>
>>>>>> *Relevant Links:*
>>>>>>
>>>>>> - SPIP Document:
>>>>>> https://docs.google.com/document/d/1Hp5BGEYJRHbk6J7XUph3bAPZKRQXKOuV1PEaqZMMRoQ/
>>>>>> -
>>>>>>
>>>>>> *Discussion Thread: *
>>>>>> https://lists.apache.org/thread/j6sj9wo9odgdpgzlxtvhoy7szs0jplf7
>>>>>> -
>>>>>>
>>>>>> JIRA: <https://issues.apache.org/jira/browse/SPARK-55668>
>>>>>> https://issues.apache.org/jira/browse/SPARK-56249
>>>>>>
>>>>>> *The vote will be open for at least 72 hours. *Please vote:
>>>>>>
>>>>>> [ ] +1: Accept the proposal as an official SPIP
>>>>>> [ ] +0
>>>>>> [ ] -1: I don't think this is a good idea because ...
>>>>>> Cheers -Andreas
>>>>>>
>>>>>>
>>>>>>
Re: [VOTE] SPIP: Auto CDC Support for Apache Spark
Posted by Xiao Li <ga...@gmail.com>.
+1
vaquar khan <va...@gmail.com> 于2026年4月4日周六 09:45写道：
> +1
>
> Regards,
> Viquar Khan
>
> On Sat, 4 Apr 2026 at 11:14, Lisa N. Cao <li...@gmail.com> wrote:
>
>> +1 (non-binding)
>>
>> --
>> LNC
>>
>> On Fri, Apr 3, 2026, 5:15 PM Shixiong Zhu <zs...@gmail.com> wrote:
>>
>>> +1
>>>
>>>
>>> On Fri, Apr 3, 2026 at 5:03 PM Mich Talebzadeh <
>>> mich.talebzadeh@gmail.com> wrote:
>>>
>>>> +1
>>>>
>>>> Dr Mich Talebzadeh,
>>>> Data Scientist | Distributed Systems (Spark) | Financial Forensics &
>>>> Metadata Analytics | Transaction Reconstruction | Audit & Evidence-Based
>>>> Analytics
>>>>
>>>> view my Linkedin profile
>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, 3 Apr 2026 at 23:00, Andreas Neumann <an...@apache.org> wrote:
>>>>
>>>>> Hi Spark devs,
>>>>>
>>>>> I'd like to call a vote on the SPIP*: Auto CDC Support for Apache
>>>>> Spark*
>>>>> Motivation
>>>>>
>>>>> With the upcoming introduction of standardized CDC support
>>>>> <https://issues.apache.org/jira/browse/SPARK-55668>, Spark will soon
>>>>> have a unified way to produce change data feeds. However, consuming these
>>>>> feeds and applying them to a target table remains a significant challenge.
>>>>>
>>>>> Common patterns like SCD Type 1 (maintaining a 1:1 replica) and SCD
>>>>> Type 2 (tracking full change history) often require hand-crafted,
>>>>> complex MERGE logic. In distributed systems, these implementations
>>>>> are frequently error-prone when handling deletions or out-of-order data.
>>>>> Proposal
>>>>>
>>>>> This SPIP proposes a new "Auto CDC" flow type for Spark. It
>>>>> encapsulates the complex logic for SCD types and out-of-order data,
>>>>> allowing data engineers to configure a declarative flow instead of writing
>>>>> manual MERGE statements. This feature will be available in both Python
>>>>> and SQL.
>>>>>
>>>>> Example SQL:
>>>>>
>>>>> -- Produce a change feed
>>>>>
>>>>> CREATE STREAMING TABLE cdc.users AS
>>>>>
>>>>> SELECT * FROM STREAM my_table CHANGES FROM VERSION 10;
>>>>>
>>>>>
>>>>> -- Consume the change feed
>>>>>
>>>>> CREATE FLOW flow
>>>>>
>>>>> AS AUTO CDC INTO
>>>>>
>>>>> target
>>>>>
>>>>> FROM stream(cdc_data.users)
>>>>>
>>>>> KEYS (userId)
>>>>>
>>>>> APPLY AS DELETE WHEN operation = "DELETE"
>>>>>
>>>>> SEQUENCE BY sequenceNum
>>>>>
>>>>> COLUMNS * EXCEPT (operation, sequenceNum)
>>>>>
>>>>> STORED AS SCD TYPE 2
>>>>>
>>>>> TRACK HISTORY ON * EXCEPT (city);
>>>>>
>>>>>
>>>>> *Relevant Links:*
>>>>>
>>>>> - SPIP Document:
>>>>> https://docs.google.com/document/d/1Hp5BGEYJRHbk6J7XUph3bAPZKRQXKOuV1PEaqZMMRoQ/
>>>>> -
>>>>>
>>>>> *Discussion Thread: *
>>>>> https://lists.apache.org/thread/j6sj9wo9odgdpgzlxtvhoy7szs0jplf7
>>>>> -
>>>>>
>>>>> JIRA: <https://issues.apache.org/jira/browse/SPARK-55668>
>>>>> https://issues.apache.org/jira/browse/SPARK-56249
>>>>>
>>>>> *The vote will be open for at least 72 hours. *Please vote:
>>>>>
>>>>> [ ] +1: Accept the proposal as an official SPIP
>>>>> [ ] +0
>>>>> [ ] -1: I don't think this is a good idea because ...
>>>>> Cheers -Andreas
>>>>>
>>>>>
>>>>>
Re: [VOTE] SPIP: Auto CDC Support for Apache Spark
Posted by vaquar khan <va...@gmail.com>.
+1
Regards,
Viquar Khan
On Sat, 4 Apr 2026 at 11:14, Lisa N. Cao <li...@gmail.com> wrote:
> +1 (non-binding)
>
> --
> LNC
>
> On Fri, Apr 3, 2026, 5:15 PM Shixiong Zhu <zs...@gmail.com> wrote:
>
>> +1
>>
>>
>> On Fri, Apr 3, 2026 at 5:03 PM Mich Talebzadeh <mi...@gmail.com>
>> wrote:
>>
>>> +1
>>>
>>> Dr Mich Talebzadeh,
>>> Data Scientist | Distributed Systems (Spark) | Financial Forensics &
>>> Metadata Analytics | Transaction Reconstruction | Audit & Evidence-Based
>>> Analytics
>>>
>>> view my Linkedin profile
>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>
>>>
>>>
>>>
>>>
>>> On Fri, 3 Apr 2026 at 23:00, Andreas Neumann <an...@apache.org> wrote:
>>>
>>>> Hi Spark devs,
>>>>
>>>> I'd like to call a vote on the SPIP*: Auto CDC Support for Apache
>>>> Spark*
>>>> Motivation
>>>>
>>>> With the upcoming introduction of standardized CDC support
>>>> <https://issues.apache.org/jira/browse/SPARK-55668>, Spark will soon
>>>> have a unified way to produce change data feeds. However, consuming these
>>>> feeds and applying them to a target table remains a significant challenge.
>>>>
>>>> Common patterns like SCD Type 1 (maintaining a 1:1 replica) and SCD
>>>> Type 2 (tracking full change history) often require hand-crafted,
>>>> complex MERGE logic. In distributed systems, these implementations are
>>>> frequently error-prone when handling deletions or out-of-order data.
>>>> Proposal
>>>>
>>>> This SPIP proposes a new "Auto CDC" flow type for Spark. It
>>>> encapsulates the complex logic for SCD types and out-of-order data,
>>>> allowing data engineers to configure a declarative flow instead of writing
>>>> manual MERGE statements. This feature will be available in both Python
>>>> and SQL.
>>>>
>>>> Example SQL:
>>>>
>>>> -- Produce a change feed
>>>>
>>>> CREATE STREAMING TABLE cdc.users AS
>>>>
>>>> SELECT * FROM STREAM my_table CHANGES FROM VERSION 10;
>>>>
>>>>
>>>> -- Consume the change feed
>>>>
>>>> CREATE FLOW flow
>>>>
>>>> AS AUTO CDC INTO
>>>>
>>>> target
>>>>
>>>> FROM stream(cdc_data.users)
>>>>
>>>> KEYS (userId)
>>>>
>>>> APPLY AS DELETE WHEN operation = "DELETE"
>>>>
>>>> SEQUENCE BY sequenceNum
>>>>
>>>> COLUMNS * EXCEPT (operation, sequenceNum)
>>>>
>>>> STORED AS SCD TYPE 2
>>>>
>>>> TRACK HISTORY ON * EXCEPT (city);
>>>>
>>>>
>>>> *Relevant Links:*
>>>>
>>>> - SPIP Document:
>>>> https://docs.google.com/document/d/1Hp5BGEYJRHbk6J7XUph3bAPZKRQXKOuV1PEaqZMMRoQ/
>>>> -
>>>>
>>>> *Discussion Thread: *
>>>> https://lists.apache.org/thread/j6sj9wo9odgdpgzlxtvhoy7szs0jplf7
>>>> -
>>>>
>>>> JIRA: <https://issues.apache.org/jira/browse/SPARK-55668>
>>>> https://issues.apache.org/jira/browse/SPARK-56249
>>>>
>>>> *The vote will be open for at least 72 hours. *Please vote:
>>>>
>>>> [ ] +1: Accept the proposal as an official SPIP
>>>> [ ] +0
>>>> [ ] -1: I don't think this is a good idea because ...
>>>> Cheers -Andreas
>>>>
>>>>
>>>>
Re: [VOTE] SPIP: Auto CDC Support for Apache Spark
Posted by "Lisa N. Cao" <li...@gmail.com>.
+1 (non-binding)
--
LNC
On Fri, Apr 3, 2026, 5:15 PM Shixiong Zhu <zs...@gmail.com> wrote:
> +1
>
>
> On Fri, Apr 3, 2026 at 5:03 PM Mich Talebzadeh <mi...@gmail.com>
> wrote:
>
>> +1
>>
>> Dr Mich Talebzadeh,
>> Data Scientist | Distributed Systems (Spark) | Financial Forensics &
>> Metadata Analytics | Transaction Reconstruction | Audit & Evidence-Based
>> Analytics
>>
>> view my Linkedin profile
>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>
>>
>>
>>
>>
>> On Fri, 3 Apr 2026 at 23:00, Andreas Neumann <an...@apache.org> wrote:
>>
>>> Hi Spark devs,
>>>
>>> I'd like to call a vote on the SPIP*: Auto CDC Support for Apache Spark*
>>> Motivation
>>>
>>> With the upcoming introduction of standardized CDC support
>>> <https://issues.apache.org/jira/browse/SPARK-55668>, Spark will soon
>>> have a unified way to produce change data feeds. However, consuming these
>>> feeds and applying them to a target table remains a significant challenge.
>>>
>>> Common patterns like SCD Type 1 (maintaining a 1:1 replica) and SCD
>>> Type 2 (tracking full change history) often require hand-crafted,
>>> complex MERGE logic. In distributed systems, these implementations are
>>> frequently error-prone when handling deletions or out-of-order data.
>>> Proposal
>>>
>>> This SPIP proposes a new "Auto CDC" flow type for Spark. It
>>> encapsulates the complex logic for SCD types and out-of-order data,
>>> allowing data engineers to configure a declarative flow instead of writing
>>> manual MERGE statements. This feature will be available in both Python
>>> and SQL.
>>>
>>> Example SQL:
>>>
>>> -- Produce a change feed
>>>
>>> CREATE STREAMING TABLE cdc.users AS
>>>
>>> SELECT * FROM STREAM my_table CHANGES FROM VERSION 10;
>>>
>>>
>>> -- Consume the change feed
>>>
>>> CREATE FLOW flow
>>>
>>> AS AUTO CDC INTO
>>>
>>> target
>>>
>>> FROM stream(cdc_data.users)
>>>
>>> KEYS (userId)
>>>
>>> APPLY AS DELETE WHEN operation = "DELETE"
>>>
>>> SEQUENCE BY sequenceNum
>>>
>>> COLUMNS * EXCEPT (operation, sequenceNum)
>>>
>>> STORED AS SCD TYPE 2
>>>
>>> TRACK HISTORY ON * EXCEPT (city);
>>>
>>>
>>> *Relevant Links:*
>>>
>>> - SPIP Document:
>>> https://docs.google.com/document/d/1Hp5BGEYJRHbk6J7XUph3bAPZKRQXKOuV1PEaqZMMRoQ/
>>> -
>>>
>>> *Discussion Thread: *
>>> https://lists.apache.org/thread/j6sj9wo9odgdpgzlxtvhoy7szs0jplf7
>>> -
>>>
>>> JIRA: <https://issues.apache.org/jira/browse/SPARK-55668>
>>> https://issues.apache.org/jira/browse/SPARK-56249
>>>
>>> *The vote will be open for at least 72 hours. *Please vote:
>>>
>>> [ ] +1: Accept the proposal as an official SPIP
>>> [ ] +0
>>> [ ] -1: I don't think this is a good idea because ...
>>> Cheers -Andreas
>>>
>>>
>>>
Re: [VOTE] SPIP: Auto CDC Support for Apache Spark
Posted by Shixiong Zhu <zs...@gmail.com>.
+1
On Fri, Apr 3, 2026 at 5:03 PM Mich Talebzadeh <mi...@gmail.com>
wrote:
> +1
>
> Dr Mich Talebzadeh,
> Data Scientist | Distributed Systems (Spark) | Financial Forensics &
> Metadata Analytics | Transaction Reconstruction | Audit & Evidence-Based
> Analytics
>
> view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
>
>
>
> On Fri, 3 Apr 2026 at 23:00, Andreas Neumann <an...@apache.org> wrote:
>
>> Hi Spark devs,
>>
>> I'd like to call a vote on the SPIP*: Auto CDC Support for Apache Spark*
>> Motivation
>>
>> With the upcoming introduction of standardized CDC support
>> <https://issues.apache.org/jira/browse/SPARK-55668>, Spark will soon
>> have a unified way to produce change data feeds. However, consuming these
>> feeds and applying them to a target table remains a significant challenge.
>>
>> Common patterns like SCD Type 1 (maintaining a 1:1 replica) and SCD Type
>> 2 (tracking full change history) often require hand-crafted, complex
>> MERGE logic. In distributed systems, these implementations are
>> frequently error-prone when handling deletions or out-of-order data.
>> Proposal
>>
>> This SPIP proposes a new "Auto CDC" flow type for Spark. It encapsulates
>> the complex logic for SCD types and out-of-order data, allowing data
>> engineers to configure a declarative flow instead of writing manual MERGE statements.
>> This feature will be available in both Python and SQL.
>>
>> Example SQL:
>>
>> -- Produce a change feed
>>
>> CREATE STREAMING TABLE cdc.users AS
>>
>> SELECT * FROM STREAM my_table CHANGES FROM VERSION 10;
>>
>>
>> -- Consume the change feed
>>
>> CREATE FLOW flow
>>
>> AS AUTO CDC INTO
>>
>> target
>>
>> FROM stream(cdc_data.users)
>>
>> KEYS (userId)
>>
>> APPLY AS DELETE WHEN operation = "DELETE"
>>
>> SEQUENCE BY sequenceNum
>>
>> COLUMNS * EXCEPT (operation, sequenceNum)
>>
>> STORED AS SCD TYPE 2
>>
>> TRACK HISTORY ON * EXCEPT (city);
>>
>>
>> *Relevant Links:*
>>
>> - SPIP Document:
>> https://docs.google.com/document/d/1Hp5BGEYJRHbk6J7XUph3bAPZKRQXKOuV1PEaqZMMRoQ/
>> -
>>
>> *Discussion Thread: *
>> https://lists.apache.org/thread/j6sj9wo9odgdpgzlxtvhoy7szs0jplf7
>> -
>>
>> JIRA: <https://issues.apache.org/jira/browse/SPARK-55668>
>> https://issues.apache.org/jira/browse/SPARK-56249
>>
>> *The vote will be open for at least 72 hours. *Please vote:
>>
>> [ ] +1: Accept the proposal as an official SPIP
>> [ ] +0
>> [ ] -1: I don't think this is a good idea because ...
>> Cheers -Andreas
>>
>>
>>
Re: [VOTE] SPIP: Auto CDC Support for Apache Spark
Posted by Mich Talebzadeh <mi...@gmail.com>.
+1
Dr Mich Talebzadeh,
Data Scientist | Distributed Systems (Spark) | Financial Forensics &
Metadata Analytics | Transaction Reconstruction | Audit & Evidence-Based
Analytics
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
On Fri, 3 Apr 2026 at 23:00, Andreas Neumann <an...@apache.org> wrote:
> Hi Spark devs,
>
> I'd like to call a vote on the SPIP*: Auto CDC Support for Apache Spark*
> Motivation
>
> With the upcoming introduction of standardized CDC support
> <https://issues.apache.org/jira/browse/SPARK-55668>, Spark will soon have
> a unified way to produce change data feeds. However, consuming these
> feeds and applying them to a target table remains a significant challenge.
>
> Common patterns like SCD Type 1 (maintaining a 1:1 replica) and SCD Type 2 (tracking
> full change history) often require hand-crafted, complex MERGE logic. In
> distributed systems, these implementations are frequently error-prone when
> handling deletions or out-of-order data.
> Proposal
>
> This SPIP proposes a new "Auto CDC" flow type for Spark. It encapsulates
> the complex logic for SCD types and out-of-order data, allowing data
> engineers to configure a declarative flow instead of writing manual MERGE statements.
> This feature will be available in both Python and SQL.
>
> Example SQL:
>
> -- Produce a change feed
>
> CREATE STREAMING TABLE cdc.users AS
>
> SELECT * FROM STREAM my_table CHANGES FROM VERSION 10;
>
>
> -- Consume the change feed
>
> CREATE FLOW flow
>
> AS AUTO CDC INTO
>
> target
>
> FROM stream(cdc_data.users)
>
> KEYS (userId)
>
> APPLY AS DELETE WHEN operation = "DELETE"
>
> SEQUENCE BY sequenceNum
>
> COLUMNS * EXCEPT (operation, sequenceNum)
>
> STORED AS SCD TYPE 2
>
> TRACK HISTORY ON * EXCEPT (city);
>
>
> *Relevant Links:*
>
> - SPIP Document:
> https://docs.google.com/document/d/1Hp5BGEYJRHbk6J7XUph3bAPZKRQXKOuV1PEaqZMMRoQ/
> -
>
> *Discussion Thread: *
> https://lists.apache.org/thread/j6sj9wo9odgdpgzlxtvhoy7szs0jplf7
> -
>
> JIRA: <https://issues.apache.org/jira/browse/SPARK-55668>
> https://issues.apache.org/jira/browse/SPARK-56249
>
> *The vote will be open for at least 72 hours. *Please vote:
>
> [ ] +1: Accept the proposal as an official SPIP
> [ ] +0
> [ ] -1: I don't think this is a good idea because ...
> Cheers -Andreas
>
>
>