[jira] [Created] (SPARK-59787) SCD1 Ignore-null support; Coalesce incoming microbatch rows per key

["Anish Mahto (Jira)" <ji...@apache.org>]

Anish Mahto created SPARK-59787:
-----------------------------------

             Summary: SCD1 Ignore-null support; Coalesce incoming microbatch rows per key
                 Key: SPARK-59787
                 URL: https://issues.apache.org/jira/browse/SPARK-59787
             Project: Spark
          Issue Type: Sub-task
          Components: Declarative Pipelines
    Affects Versions: 4.4.0
            Reporter: Anish Mahto


A microbatch can contain multiple CDC events targeting the same key. With SCD1 ignore-null handling, different events may author different leaves, so selecting a single winning row is insufficient.

Instead, collapse all microbatch rows for each key into one synthetic row. For every user-data leaf, select the value from the latest event that authors that leaf. Upserts determine authorship from their version maps, while deletes author null for every leaf.

The resulting row contains the winning value and authorship sequence for every leaf, a reconstructed version map, and the greatest row-wide upsert and delete sequences from the microbatch.

This ticket only reconciles events within one microbatch. It does not reconcile against persisted auxiliary-table tombstones or existing target-table state.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org