[jira] [Created] (SPARK-59183) SCD2 Ignore-null support; Populate version map per event row at ingest

["Anish Mahto (Jira)" <ji...@apache.org>]

Anish Mahto created SPARK-59183:
-----------------------------------

             Summary: SCD2 Ignore-null support; Populate version map per event row at ingest
                 Key: SPARK-59183
                 URL: https://issues.apache.org/jira/browse/SPARK-59183
             Project: Spark
          Issue Type: Sub-task
          Components: Declarative Pipelines
    Affects Versions: 4.4.0
            Reporter: Anish Mahto


When processing an incoming microbatch, if ignore-null is enabled for the AutoCDC flow, a non-null version map needs to be populated for every row in the microbatch.

As per the version map contract (see `Scd2VersionMap`'s scaladoc), the version map is populated with null-authorship information for every event, respecting the currently active ignore-null selection.

This ticket is not concerned with reconciling the microbatch's version maps with existing rows in the aux/target table.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org