You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "AnishMahto (via GitHub)" <gi...@apache.org> on 2026/05/19 17:52:34 UTC
[PR] [SPARK-56927] Implement SCD1 Batch Processor; Merge Microbatch onto Target [spark]
AnishMahto opened a new pull request, #55996:
URL: https://github.com/apache/spark/pull/55996
Approved AutoCDC SPIP: https://lists.apache.org/thread/j6sj9wo9odgdpgzlxtvhoy7szs0jplf7
--------
This is a stacked PR. Review incremental diff here: https://github.com/AnishMahto/spark/compare/SPARK-56923-SCD1-merge-microbatch-onto-auxiliary...SPARK-56927-SCD1-merge-microbatch-onto-target
--------
Preamble:
The SCD type 1 flow is a foreachBatch streaming query on an input change-data-feed, and is responsible for reconciling the incoming change data onto some target table that follows SCD1 replication semantics.
SCD1 flows also maintain an "auxiliary" table to keep track of early-arriving out-of-order received events state. Each microbatch will need to reconcile against this auxiliary table as well, and update the auxiliary table's state appropriately for future microbatches.
Merge Microbatch onto Target:
The final part of an SCD1 reconciliation - merge the reconciled dataframe (deduplicated, target columns and operational metadata projected, and tombstone applied) onto the target table.
The target table's contents represent the SCD1 transformation of the change data feed, for the specified target column selection in the `ChangeArgs`.
Merging onto the target under SCD1 semantics means deleting existing rows that match against the reconciled microbatch's delete events, and inserting or updating rows as appropriate for the reconciled microbatch's upsert events.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56927][SDP] Implement SCD1 Batch Processor; Merge Microbatch onto Target [spark]
Posted by "AnishMahto (via GitHub)" <gi...@apache.org>.
AnishMahto closed pull request #55996: [SPARK-56927][SDP] Implement SCD1 Batch Processor; Merge Microbatch onto Target
URL: https://github.com/apache/spark/pull/55996
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56927][SDP] Implement SCD1 Batch Processor; Merge Microbatch onto Target [spark]
Posted by "AnishMahto (via GitHub)" <gi...@apache.org>.
AnishMahto commented on PR #55996:
URL: https://github.com/apache/spark/pull/55996#issuecomment-4531660435
Closing as these changes were merged with https://github.com/apache/spark/pull/56016
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org