[jira] [Resolved] (SPARK-58325) Validate AutoCDC reserved columns post-column-selection to allow chained replicas

["Andreas Neumann (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-58325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andreas Neumann resolved SPARK-58325.
-------------------------------------
    Resolution: Won't Do

> Validate AutoCDC reserved columns post-column-selection to allow chained replicas
> ---------------------------------------------------------------------------------
>
>                 Key: SPARK-58325
>                 URL: https://issues.apache.org/jira/browse/SPARK-58325
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Declarative Pipelines
>    Affects Versions: 5.0.0
>            Reporter: Andreas Neumann
>            Priority: Major
>
> AutoCDC currently rejects a source change feed that contains a reserved column BEFORE the user's column selection is applied. Two problems follow:
> 1. It blocks a legitimate use case. An SCD1 AutoCDC target physically carries the reserved _cdc_metadata column. Feeding that target into another AutoCDC flow (e.g. re-replicating an SCD1 replica as SCD2, or vice versa) means the source contains _cdc_metadata by construction, not by user error. The user's natural recourse -- COLUMNS * EXCEPT '__spark_autocdc_metadata' -- cannot work today because requireReservedPrefixAbsentInSourceColumns() runs against df.schema (the raw source) before column selection.
> 2. It is inconsistent across the framework columns. The prefixed _cdc_metadata is guarded pre-selection (in the shared prefix guard, Flow.requireReservedPrefixAbsentInSourceColumns), while the non-prefixed __START_AT / __END_AT are (per SPARK-57251) guarded by a separate framework-column guard and can already be excluded via the column selection. From the user's perspective all three are engine-owned columns.
> Proposal: validate that reserved columns are absent AFTER the column selection is applied (i.e. against the user-selected schema), not against the raw source. This:
> - Lets a user opt out of any reserved column via EXCEPT, enabling chained replicas in both directions.
> - Unifies the rule for all three framework columns (__START_AT, __END_AT, _cdc_metadata).
> - Remains safe: you can only fail to opt OUT; you can never opt IN (an INCLUDE naming a reserved column, or no selection over a colliding source, still fails because the column is still present post-selection). Backward compatible with the existing 'reject a source _cdc_metadata column with no selection' behavior.
> Coordination: this must move BOTH guards post-selection in lockstep to avoid flipping the inconsistency:
> - Flow.requireReservedPrefixAbsentInSourceColumns (prefix guard; shared with SCD1) -- validate against userSelectedSchema.
> - The SPARK-57251 framework-column guard for __START_AT / __END_AT -- same treatment. SPARK-57251 currently validates pre-selection (df.schema.fieldNames); align it here or sequence the two changes together.
> Note this touches shared SCD1 behavior, so it is intentionally NOT folded into the SCD2 schema-derivation sub-task (SPARK-58319). Add tests: EXCEPT of each reserved column from a colliding source is accepted and the engine column is re-appended; INCLUDE naming a reserved column still fails; a colliding source with no selection still fails.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org