You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Szehon Ho (Jira)" <ji...@apache.org> on 2026/08/27 22:30:00 UTC

[jira] [Assigned] (SPARK-58572) Add Cross-SCD Fuzz Testing

     [ https://issues.apache.org/jira/browse/SPARK-58572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Szehon Ho reassigned SPARK-58572:
---------------------------------

    Assignee: Anish Mahto

> Add Cross-SCD Fuzz Testing
> --------------------------
>
>                 Key: SPARK-58572
>                 URL: https://issues.apache.org/jira/browse/SPARK-58572
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Declarative Pipelines
>    Affects Versions: 4.4.0
>            Reporter: Anish Mahto
>            Assignee: Anish Mahto
>            Priority: Major
>              Labels: pull-request-available
>
> Given the same CDF (but can be shuffled in arbitrary order), every key's output row in SCD1 should be exactly equal (in data columns, not necessarily CDC metadata) to the last row per key in SCD2.
> Using this principle, and the existing AutoCDC fuzz test harnesses, we should setup a cross-SCD fuzz test so that one implementation can keep the other in check for correctness, and detect regressions made to one SCD's implementation but not the other.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org