[jira] [Updated] (SPARK-59014) DataSourceV2 refresh validation misses a data column that hides a captured metadata column

["Yan Yan (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-59014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yan Yan updated SPARK-59014:
----------------------------
    Description: 
When a DataSourceV2 relation is refreshed / re-resolved, {{V2TableUtil.validateCapturedMetadataColumns}} validates its captured metadata-column attributes against the current table. Two gaps could let incompatible captured metadata references pass validation.

h3. 1. A data column hides a captured metadata column

The existing check compared captured metadata columns only against the metadata columns the connector still reports, so a name conflict arriving on the *data* side was invisible to it.

If a data column takes the name of a captured metadata column and the connector does not rename conflicts ({{SupportsMetadataColumns.canRenameConflictingMetadataColumns()}} returns {{false}}, the default), {{LogicalPlan.metadataOutputWithOutConflicts}} suppresses the metadata column. A captured reference to it can then no longer be resolved. On a partially-pruned scan, {{PushDownUtils.toOutputAttrs}} maps the read schema onto the relation output by name and collapses the two same-named fields onto a single attribute, so a query that asked for the metadata column silently returns the *data* column's values.

*Expected:* refresh / re-resolution fails with a clear error.
*Actual:* on a suppressing connector with partial column pruning, the query silently returns the data column's values where the metadata column was expected (a wrong-results bug); with full pruning it happens to return the correct value.

The {{SupportsMetadataColumns}} contract already recommends that a non-renaming source reject such a data-column name, but that is only a recommendation and is not enforced, so Spark should validate it.

h3. 2. A renamed captured metadata column is incorrectly matched by its physical name

A connector that can rename conflicts keeps a conflicting metadata column reachable under a different physical attribute name. For example, a data column and metadata column both logically named {{index}} produce this relation shape:

{code}
AttributeReference.name = "_index"
AttributeReference.metadata.__metadata_col = "index"
relation.table.metadataColumns() = [index]
{code}

The internal {{__metadata_col}} entry stores the metadata column's logical connector name; it is not a Boolean marker. Previously, {{extractMetadataColumns}} collected {{AttributeReference.name}}, so it looked for physical name {{_index}} in {{relation.table.metadataColumns()}}. The connector reports logical name {{index}}, the lookup returned no match, and the captured metadata column was omitted from schema compatibility validation. A later type or nullability change to the logical {{index}} metadata column could therefore pass silently.

*Expected:* the physical rename remains valid, while the captured logical metadata column continues to be checked for type and nullability changes.
*Actual:* the renamed captured metadata column is dropped from validation because its physical name does not match the connector's logical name.

Renaming connectors are unaffected by the first issue's shadowing rejection because their metadata columns remain reachable. The second issue specifically restores schema validation for those renamed metadata attributes; an unchanged renamed metadata column remains valid.

h3. Fix

* Detect a still-reported metadata column hidden by a same-named data column in the shared {{validateCapturedMetadataColumns}} path and raise {{INCOMPATIBLE_TABLE_CHANGE_AFTER_ANALYSIS.METADATA_COLUMNS_MISMATCH}}. Only the suppressed case is rejected; the renaming case keeps working.
* Extract captured metadata attributes with {{MetadataAttributeWithLogicalName}} and match {{relation.table.metadataColumns()}} using the logical name stored in {{__metadata_col}}, so type and nullability changes to renamed metadata columns are validated.
* Correct the outdated internal comments: {{__metadata_col}} stores the logical column name, while {{__file_source_metadata_col}} is the Boolean marker.

PR: https://github.com/apache/spark/pull/58295

  was:
When a DataSourceV2 relation is refreshed / re-resolved, {{V2TableUtil.validateCapturedMetadataColumns}} validates the captured metadata columns against the current table. It compares the captured metadata columns only against the metadata columns the connector still reports, so a name conflict that arrives on the *data* side is invisible to it.

If a data column takes the name of a captured metadata column and the connector does not rename conflicts ({{SupportsMetadataColumns.canRenameConflictingMetadataColumns()}} returns {{false}}, the default), {{LogicalPlan.metadataOutputWithOutConflicts}} suppresses the metadata column. A captured reference to it can then no longer be resolved. On a partially-pruned scan, {{PushDownUtils.toOutputAttrs}} maps the read schema onto the relation output by name and collapses the two same-named fields onto a single attribute, so a query that asked for the metadata column silently returns the *data* column's values.

*Expected:* refresh / re-resolution fails with a clear error.
*Actual:* on a suppressing connector with partial column pruning, the query silently returns the data column's values where the metadata column was expected (a wrong-results bug); with full pruning it happens to return the correct value.

Connectors that rename conflicting metadata columns are unaffected. The {{SupportsMetadataColumns}} contract already recommends that a non-renaming source reject such a data-column name, but that is only a recommendation and is not enforced, so Spark should validate it.

*Fix:* detect the conflict in the shared {{validateCapturedMetadataColumns}} and raise {{INCOMPATIBLE_TABLE_CHANGE_AFTER_ANALYSIS.METADATA_COLUMNS_MISMATCH}}. Only the suppressed case is rejected; the renaming case keeps working.

PR: https://github.com/apache/spark/pull/58295


> DataSourceV2 refresh validation misses a data column that hides a captured metadata column
> ------------------------------------------------------------------------------------------
>
>                 Key: SPARK-59014
>                 URL: https://issues.apache.org/jira/browse/SPARK-59014
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 4.2.0
>            Reporter: Yan Yan
>            Priority: Minor
>
> When a DataSourceV2 relation is refreshed / re-resolved, {{V2TableUtil.validateCapturedMetadataColumns}} validates its captured metadata-column attributes against the current table. Two gaps could let incompatible captured metadata references pass validation.
> h3. 1. A data column hides a captured metadata column
> The existing check compared captured metadata columns only against the metadata columns the connector still reports, so a name conflict arriving on the *data* side was invisible to it.
> If a data column takes the name of a captured metadata column and the connector does not rename conflicts ({{SupportsMetadataColumns.canRenameConflictingMetadataColumns()}} returns {{false}}, the default), {{LogicalPlan.metadataOutputWithOutConflicts}} suppresses the metadata column. A captured reference to it can then no longer be resolved. On a partially-pruned scan, {{PushDownUtils.toOutputAttrs}} maps the read schema onto the relation output by name and collapses the two same-named fields onto a single attribute, so a query that asked for the metadata column silently returns the *data* column's values.
> *Expected:* refresh / re-resolution fails with a clear error.
> *Actual:* on a suppressing connector with partial column pruning, the query silently returns the data column's values where the metadata column was expected (a wrong-results bug); with full pruning it happens to return the correct value.
> The {{SupportsMetadataColumns}} contract already recommends that a non-renaming source reject such a data-column name, but that is only a recommendation and is not enforced, so Spark should validate it.
> h3. 2. A renamed captured metadata column is incorrectly matched by its physical name
> A connector that can rename conflicts keeps a conflicting metadata column reachable under a different physical attribute name. For example, a data column and metadata column both logically named {{index}} produce this relation shape:
> {code}
> AttributeReference.name = "_index"
> AttributeReference.metadata.__metadata_col = "index"
> relation.table.metadataColumns() = [index]
> {code}
> The internal {{__metadata_col}} entry stores the metadata column's logical connector name; it is not a Boolean marker. Previously, {{extractMetadataColumns}} collected {{AttributeReference.name}}, so it looked for physical name {{_index}} in {{relation.table.metadataColumns()}}. The connector reports logical name {{index}}, the lookup returned no match, and the captured metadata column was omitted from schema compatibility validation. A later type or nullability change to the logical {{index}} metadata column could therefore pass silently.
> *Expected:* the physical rename remains valid, while the captured logical metadata column continues to be checked for type and nullability changes.
> *Actual:* the renamed captured metadata column is dropped from validation because its physical name does not match the connector's logical name.
> Renaming connectors are unaffected by the first issue's shadowing rejection because their metadata columns remain reachable. The second issue specifically restores schema validation for those renamed metadata attributes; an unchanged renamed metadata column remains valid.
> h3. Fix
> * Detect a still-reported metadata column hidden by a same-named data column in the shared {{validateCapturedMetadataColumns}} path and raise {{INCOMPATIBLE_TABLE_CHANGE_AFTER_ANALYSIS.METADATA_COLUMNS_MISMATCH}}. Only the suppressed case is rejected; the renaming case keeps working.
> * Extract captured metadata attributes with {{MetadataAttributeWithLogicalName}} and match {{relation.table.metadataColumns()}} using the logical name stored in {{__metadata_col}}, so type and nullability changes to renamed metadata columns are validated.
> * Correct the outdated internal comments: {{__metadata_col}} stores the logical column name, while {{__file_source_metadata_col}} is the Boolean marker.
> PR: https://github.com/apache/spark/pull/58295



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org