[jira] [Updated] (SPARK-58478) SCD1 AutoCDC rejects dropping a nested struct/array field but accepts dropping a top-level column

["Andreas Neumann (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-58478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andreas Neumann updated SPARK-58478:
------------------------------------
    Description: 
h2. Summary

SCD Type 1 AutoCDC is internally inconsistent about subtractive schema evolution across runs:

* Dropping a *top-level* column from the flow's source (or narrowing the {{COLUMNS}} selection) is *accepted*: the column is preserved on already-written rows and set to {{NULL}} on newly-inserted rows. Covered by {{AutoCdcScd1SchemaEvolutionSuite}} ("a top-level column dropped from the source DF between runs is preserved on existing rows and left NULL on new rows").
* Dropping a *nested* struct field, or a field inside an {{array<struct>}} element, is *rejected* with {{INCOMPATIBLE_DATA_FOR_TABLE.CANNOT_FIND_DATA}} (same suite, "dropping a nested struct field between runs fails with INCOMPATIBLE_DATA_FOR_TABLE" and the {{array<struct>}} analog).

So the same logical operation -- the flow stops emitting a column that still exists on the target -- succeeds at the top level but fails one level down.

h2. Why

This is not a deliberate "reject subtractive evolution" policy; it is an artifact of where the failure surfaces in the MERGE write path:

* For a dropped top-level column, the SCD1 MERGE simply omits that column from its INSERT/UPDATE assignment maps, so nothing references the missing data and the write succeeds.
* For a dropped nested field, the v2 writer's {{TableOutputResolver}} walks into the target's struct and cannot find data for the nested path (e.g. {{value.b.c}}), so it throws {{CANNOT_FIND_DATA}}.

The nested case is stricter purely because the resolver descends into nested paths while the top-level assignment map does not.

h2. Contrast with SCD2

SPARK-58418 makes SCD2 handle both cases consistently: {{Scd2ForeachBatchHandler}} uses {{unionByName(..., allowMissingColumns = true)}}, which pads a missing column -- top-level or nested (recursing into structs/arrays) -- with {{NULL}} before the union/MERGE. As a result SCD2 preserves the field on existing records and writes {{NULL}} on new ones for a nested drop too, matching its own top-level behavior. SCD1 does not have this padding on its MERGE source, so it retains the nested-drop failure. SPARK-58418's tests document this as a deliberate SCD2-vs-SCD1 divergence
({{AutoCdcScd2ColumnEvolutionSuite}}).

h2. Proposal

Make SCD1 accept a dropped nested struct / array-element field the same additive-tolerant way it already accepts a dropped top-level column (preserve on existing rows, {{NULL}} on new rows), so its subtractive-evolution behavior is consistent across nesting levels and matches SCD2. Likely approach: pad the MERGE source with the missing nested fields (analogous to SCD2's {{allowMissingColumns}}) before the write, rather than letting the resolver reject it.

When this is fixed, the two "fails with INCOMPATIBLE_DATA_FOR_TABLE" tests in {{AutoCdcScd1SchemaEvolutionSuite}} should be converted to assert the preserve/NULL behavior, and the SCD2-vs-SCD1 divergence note in {{AutoCdcScd2ColumnEvolutionSuite}} can be removed.

Not a regression and not blocking; captured for consistency while implementing SPARK-58418.


> SCD1 AutoCDC rejects dropping a nested struct/array field but accepts dropping a top-level column
> -------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-58478
>                 URL: https://issues.apache.org/jira/browse/SPARK-58478
>             Project: Spark
>          Issue Type: Sub-task
>          Components: SQL
>    Affects Versions: 5.0.0
>            Reporter: Andreas Neumann
>            Priority: Major
>
> h2. Summary
> SCD Type 1 AutoCDC is internally inconsistent about subtractive schema evolution across runs:
> * Dropping a *top-level* column from the flow's source (or narrowing the {{COLUMNS}} selection) is *accepted*: the column is preserved on already-written rows and set to {{NULL}} on newly-inserted rows. Covered by {{AutoCdcScd1SchemaEvolutionSuite}} ("a top-level column dropped from the source DF between runs is preserved on existing rows and left NULL on new rows").
> * Dropping a *nested* struct field, or a field inside an {{array<struct>}} element, is *rejected* with {{INCOMPATIBLE_DATA_FOR_TABLE.CANNOT_FIND_DATA}} (same suite, "dropping a nested struct field between runs fails with INCOMPATIBLE_DATA_FOR_TABLE" and the {{array<struct>}} analog).
> So the same logical operation -- the flow stops emitting a column that still exists on the target -- succeeds at the top level but fails one level down.
> h2. Why
> This is not a deliberate "reject subtractive evolution" policy; it is an artifact of where the failure surfaces in the MERGE write path:
> * For a dropped top-level column, the SCD1 MERGE simply omits that column from its INSERT/UPDATE assignment maps, so nothing references the missing data and the write succeeds.
> * For a dropped nested field, the v2 writer's {{TableOutputResolver}} walks into the target's struct and cannot find data for the nested path (e.g. {{value.b.c}}), so it throws {{CANNOT_FIND_DATA}}.
> The nested case is stricter purely because the resolver descends into nested paths while the top-level assignment map does not.
> h2. Contrast with SCD2
> SPARK-58418 makes SCD2 handle both cases consistently: {{Scd2ForeachBatchHandler}} uses {{unionByName(..., allowMissingColumns = true)}}, which pads a missing column -- top-level or nested (recursing into structs/arrays) -- with {{NULL}} before the union/MERGE. As a result SCD2 preserves the field on existing records and writes {{NULL}} on new ones for a nested drop too, matching its own top-level behavior. SCD1 does not have this padding on its MERGE source, so it retains the nested-drop failure. SPARK-58418's tests document this as a deliberate SCD2-vs-SCD1 divergence
> ({{AutoCdcScd2ColumnEvolutionSuite}}).
> h2. Proposal
> Make SCD1 accept a dropped nested struct / array-element field the same additive-tolerant way it already accepts a dropped top-level column (preserve on existing rows, {{NULL}} on new rows), so its subtractive-evolution behavior is consistent across nesting levels and matches SCD2. Likely approach: pad the MERGE source with the missing nested fields (analogous to SCD2's {{allowMissingColumns}}) before the write, rather than letting the resolver reject it.
> When this is fixed, the two "fails with INCOMPATIBLE_DATA_FOR_TABLE" tests in {{AutoCdcScd1SchemaEvolutionSuite}} should be converted to assert the preserve/NULL behavior, and the SCD2-vs-SCD1 divergence note in {{AutoCdcScd2ColumnEvolutionSuite}} can be removed.
> Not a regression and not blocking; captured for consistency while implementing SPARK-58418.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org