[jira] [Assigned] (SPARK-57356) Implement SCD2 Batch Processor; Cleanup Delete Encoding Rows Post-Reconciliation

["Szehon Ho (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-57356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Szehon Ho reassigned SPARK-57356:
---------------------------------

    Assignee: Anish Mahto

> Implement SCD2 Batch Processor; Cleanup Delete Encoding Rows Post-Reconciliation
> --------------------------------------------------------------------------------
>
>                 Key: SPARK-57356
>                 URL: https://issues.apache.org/jira/browse/SPARK-57356
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Declarative Pipelines
>    Affects Versions: 4.3.0
>            Reporter: Anish Mahto
>            Assignee: Anish Mahto
>            Priority: Major
>              Labels: pull-request-available
>
> {*}Preamble{*}:
> The SCD type 2 flow is a foreachBatch streaming query on an input change-data-feed, and is responsible for reconciling the incoming change data onto some target table that follows SCD2 replication semantics.
> SCD2 flows also maintain an "auxiliary" table to keep track of early-arriving out-of-order received events state. Each microbatch will need to reconcile against this auxiliary table as well, and update the auxiliary table's state appropriately for future microbatches.
> *Cleanup Delete Encoding Rows Post-Reconciliation*
> Reconciliation itself (re)assigns the start/end-ats of upsert representing columns in the microbatch-affected row set, using the next neighbor event (could be either another upsert that ends this upsert, or a delete event that ends this upsert).
> Once a delete event matches with an preceding row and reconciliation updates that row's end-at accordingly, that upsert row (now closed) fully encodes the delete event along with its upsert event. This means the delete event does not need to have its own row, and the tombstone/decomposition tail can be dropped. If a delete event does not match against a preceding upsert however, then it is necessary to bookkeep as a tombstone in the aux table.
> It is also possible that post-reconciliation, a decomposition tail survives as a non-redundant delete encoding event. This decomposition tail should be promoted to a tombstone and persisted in the aux to reconcile against future microbatches.
> As a trivial example image an existing closed upsert valid for start/end at [T=10, T=20) gets bisected by a delete event at T=15. Then the closed upsert gets decomposed into head and tail at [T=10, null) and (null, T=20], reconciliation creates a new closed upsert at [T=10, T=15), and the decomposition tail (null, T=20] should be promoted to a tombstone at instant T=20.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org