You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Jose Torres (Jira)" <ji...@apache.org> on 2026/10/15 22:04:00 UTC

[jira] [Assigned] (SPARK-59358) SCD2 Ignore-null support; Coalesce ignored-null values

     [ https://issues.apache.org/jira/browse/SPARK-59358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jose Torres reassigned SPARK-59358:
-----------------------------------

    Assignee: Anish Mahto

> SCD2 Ignore-null support; Coalesce ignored-null values
> ------------------------------------------------------
>
>                 Key: SPARK-59358
>                 URL: https://issues.apache.org/jira/browse/SPARK-59358
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Declarative Pipelines
>    Affects Versions: 4.4.0
>            Reporter: Anish Mahto
>            Assignee: Anish Mahto
>            Priority: Major
>              Labels: pull-request-available
>
> Once the version map is constructed per incoming row in the microbatch, we need to start actually reconciling columns in the incoming microbatch that have been deduced as unauthored - they should instead inherit the last known authored value.
> While reconciling the affected existing rows from the aux/target table, it's also possible out of order received events in the current microbatch affect what authored value existing ignored-null columns inherit; these need to be updated too.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org