You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Andreas Neumann (Jira)" <ji...@apache.org> on 2026/07/28 03:39:00 UTC

[jira] [Commented] (SPARK-58325) Validate AutoCDC reserved columns post-column-selection to allow chained replicas

    [ https://issues.apache.org/jira/browse/SPARK-58325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18099607#comment-18099607 ] 

Andreas Neumann commented on SPARK-58325:
-----------------------------------------

This is an interesting one. It has an element of complexity that we should account for:
 - we currently don't require that all columns that contribute to the sequencing expression are included in the column selection
 - during batch reconciliation, we first compute the sequencing and project it onto the batch, encoded into the `_cdc_{_}c{_}metadata` column
 - at that time, if the table already contains the `_cdc_{_}c{_}metadata` column, we would have a conflict.
 - so we would have to apply the column selection first, to remove that column. But if that column contributes to the sequencing expression, then that expression is broken.
 - we could, in theory, add the metadata column with a different name first, then apply the column selection, and then rename the metadata column to its actual name
 - but that seems overkill for a use case that is probably not common. 

This use case (AUTO CDC on a change feed produced by a table that was itself an AUTO CDC table) is valid. For example, a user might replicate an RDB table to the cloud, and then replicate that again to another availability zone for DR reasons. And in that case, it would even make sense to reuse the exact same sequencing, extracting it from the `_cdc_{_}c{_}metadata` column. And that is exactly the use case that this improvement would enable. 

But there is another way to do that: As part of the streaming query that provides the `source`, the user can extract the original sequence from the `_cdc_{_}c{_}metadata` column into a new column, and drop the`_cdc_{_}c{_}metadata` column itself from the input. Because this work-around exists, I suggest that we do not implement this feature, at lest for now. If there is overwhelming demand from users, we can always enable it later. 

 

 

 

> Validate AutoCDC reserved columns post-column-selection to allow chained replicas
> ---------------------------------------------------------------------------------
>
>                 Key: SPARK-58325
>                 URL: https://issues.apache.org/jira/browse/SPARK-58325
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Declarative Pipelines
>    Affects Versions: 5.0.0
>            Reporter: Andreas Neumann
>            Priority: Major
>
> AutoCDC currently rejects a source change feed that contains a reserved column BEFORE the user's column selection is applied. Two problems follow:
> 1. It blocks a legitimate use case. An SCD1 AutoCDC target physically carries the reserved _cdc_metadata column. Feeding that target into another AutoCDC flow (e.g. re-replicating an SCD1 replica as SCD2, or vice versa) means the source contains _cdc_metadata by construction, not by user error. The user's natural recourse -- COLUMNS * EXCEPT '__spark_autocdc_metadata' -- cannot work today because requireReservedPrefixAbsentInSourceColumns() runs against df.schema (the raw source) before column selection.
> 2. It is inconsistent across the framework columns. The prefixed _cdc_metadata is guarded pre-selection (in the shared prefix guard, Flow.requireReservedPrefixAbsentInSourceColumns), while the non-prefixed __START_AT / __END_AT are (per SPARK-57251) guarded by a separate framework-column guard and can already be excluded via the column selection. From the user's perspective all three are engine-owned columns.
> Proposal: validate that reserved columns are absent AFTER the column selection is applied (i.e. against the user-selected schema), not against the raw source. This:
> - Lets a user opt out of any reserved column via EXCEPT, enabling chained replicas in both directions.
> - Unifies the rule for all three framework columns (__START_AT, __END_AT, _cdc_metadata).
> - Remains safe: you can only fail to opt OUT; you can never opt IN (an INCLUDE naming a reserved column, or no selection over a colliding source, still fails because the column is still present post-selection). Backward compatible with the existing 'reject a source _cdc_metadata column with no selection' behavior.
> Coordination: this must move BOTH guards post-selection in lockstep to avoid flipping the inconsistency:
> - Flow.requireReservedPrefixAbsentInSourceColumns (prefix guard; shared with SCD1) -- validate against userSelectedSchema.
> - The SPARK-57251 framework-column guard for __START_AT / __END_AT -- same treatment. SPARK-57251 currently validates pre-selection (df.schema.fieldNames); align it here or sequence the two changes together.
> Note this touches shared SCD1 behavior, so it is intentionally NOT folded into the SCD2 schema-derivation sub-task (SPARK-58319). Add tests: EXCEPT of each reserved column from a colliding source is accepted and the engine column is re-appended; INCLUDE naming a reserved column still fails; a colliding source with no selection still fails.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org