[jira] [Created] (SPARK-60350) SCD1 Ignore-null support; Target table merge

["Anish Mahto (Jira)" <ji...@apache.org>]

Anish Mahto created SPARK-60350:
-----------------------------------

             Summary: SCD1 Ignore-null support; Target table merge
                 Key: SPARK-60350
                 URL: https://issues.apache.org/jira/browse/SPARK-60350
             Project: Spark
          Issue Type: Sub-task
          Components: Declarative Pipelines
    Affects Versions: 4.4.0
            Reporter: Anish Mahto


Implement SCD1 target merge for ignore-null flows.

Once per microbatch, the target merge applies the reconciled changes, at most one per key, to the target table. Today an upsert at least as new replaces the whole row. Under ignore-null, the incoming change and the target row can each hold the newest value of different fields, tracked by this stack's per-row version map of when each leaf field was last written.

Introduce a new leaf-level merge strategy beside the existing whole-row one. It splits selected columns into leaves, takes each field from whichever side (microbatch or target) wrote it later, and rebuilds the columns and version map. A delete removes the row only if newer than its latest upsert, so later-written fields survive; ties still favor upserts.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org