[jira] [Assigned] (SPARK-59164) Support nested column schema evolution

["Szehon Ho (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-59164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Szehon Ho reassigned SPARK-59164:
---------------------------------

    Assignee: Anish Mahto

> Support nested column schema evolution
> --------------------------------------
>
>                 Key: SPARK-59164
>                 URL: https://issues.apache.org/jira/browse/SPARK-59164
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Declarative Pipelines
>    Affects Versions: 4.4.0
>            Reporter: Anish Mahto
>            Assignee: Anish Mahto
>            Priority: Major
>              Labels: pull-request-available
>
> `SchemaInferenceUtils.diffSchemas` is used to deduce the `TableChanges` that need to be sent to the catalog in order to reflect the latest schema evolved state of datasets declared by a pipeline, during dataset materialization.
> Today however, the diff does not support nested column level evolution. If some nested column (ex. struct type) has one of its sub-fields changed between runs, but the column's top level name stays the same, it does not emit the correct (or any) column add or delete changes to the catalog.
> The fix is to recursively diff the schema, traversing every nested column as necessary. This is similar to the existing `ResolveSchemaEvolution.computeSchemaChanges`, except it is not just additive schema evolution - it supports dropping columns in the latest dataset's schema declaration.
> Two things intentionally not handled by this change:
> 1. Diff'ing changes to a nested columns `metadata`. That was not respected before, and continues to not be respected after these changes, because DSv2 does not yet support `TableChanges` types for changes to column metadata
> 2. Respecting order that columns are defined in between the existing and new schemas. Ex. if the exact same column was previously declared as the first column in the schema, but now is declared as the second column in the schema. Instead, existing behavior is preserved; column equality is position invariant.
> This is a general improvement to SDP's schema evolution infrastructure, but is being tracked as part of the AutoCDC ignore-null implementation, which will start requiring AutoCDC's `_cdc_metadata` column to support schema evolution for a nested version map field. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org