[jira] [Created] (SPARK-60252) SCD1 Ignore-null support; Field Reconciliation Data Classes

["Anish Mahto (Jira)" <ji...@apache.org>]

Anish Mahto created SPARK-60252:
-----------------------------------

             Summary: SCD1 Ignore-null support; Field Reconciliation Data Classes
                 Key: SPARK-60252
                 URL: https://issues.apache.org/jira/browse/SPARK-60252
             Project: Spark
          Issue Type: Sub-task
          Components: Declarative Pipelines
    Affects Versions: 4.4.0
            Reporter: Anish Mahto


Introduce case classes and helpers that represent user-data fields during SCD1 ignore-null reconciliation.

SCD1 reconciliation combines a key's CDC events into one row holding the key's latest state. It first collapses the incoming microbatch to at most one row per key, then reconciles that row against the key's persisted tombstone and, on merge, against its target row. Without ignore-null, reconciliation is row-wide: one row wins every field. With ignore-null, an upsert authors a selected leaf only if it provides a non-null value, so different rows can win different fields.

`Scd1FieldToReconcile` describes one field that is reconciled as a unit. That is either a leaf of a column selected for ignore-null, or a whole column outside the selection, since every upsert authors all of that column's leaves. Reconciling it produces an `Scd1ReconciledField`, which holds the reconciled value and the sequence that authored it; the version map is built from those sequences. 

Both reconciliations, within the microbatch and against the target row, use these same two classes, so they define fields and build the version map the same way. The transition is one-way, and neither class changes behavior until follow-up tickets wire them in.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org