You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Wenchen Fan (Jira)" <ji...@apache.org> on 2026/09/03 03:40:00 UTC
[jira] [Assigned] (SPARK-59014) DataSourceV2 refresh validation misses a data column that hides a captured metadata column
[ https://issues.apache.org/jira/browse/SPARK-59014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wenchen Fan reassigned SPARK-59014:
-----------------------------------
Assignee: Yan Yan
> DataSourceV2 refresh validation misses a data column that hides a captured metadata column
> ------------------------------------------------------------------------------------------
>
> Key: SPARK-59014
> URL: https://issues.apache.org/jira/browse/SPARK-59014
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Yan Yan
> Assignee: Yan Yan
> Priority: Minor
> Labels: pull-request-available
>
> When a DataSourceV2 relation is refreshed / re-resolved, {{V2TableUtil.validateCapturedMetadataColumns}} validates its captured metadata-column attributes against the current table. Two gaps could let incompatible captured metadata references pass validation.
> h3. 1. A data column hides a captured metadata column
> The existing check compared captured metadata columns only against the metadata columns the connector still reports, so a name conflict arriving on the *data* side was invisible to it.
> If a data column takes the name of a captured metadata column and the connector does not rename conflicts ({{SupportsMetadataColumns.canRenameConflictingMetadataColumns()}} returns {{false}}, the default), {{LogicalPlan.metadataOutputWithOutConflicts}} suppresses the metadata column. A captured reference to it can then no longer be resolved. On a partially-pruned scan, {{PushDownUtils.toOutputAttrs}} maps the read schema onto the relation output by name and collapses the two same-named fields onto a single attribute, so a query that asked for the metadata column silently returns the *data* column's values.
> *Expected:* refresh / re-resolution fails with a clear error.
> *Actual:* on a suppressing connector with partial column pruning, the query silently returns the data column's values where the metadata column was expected (a wrong-results bug); with full pruning it happens to return the correct value.
> The {{SupportsMetadataColumns}} contract already recommends that a non-renaming source reject such a data-column name, but that is only a recommendation and is not enforced, so Spark should validate it.
> h3. 2. A renamed captured metadata column is incorrectly matched by its physical name
> A connector that can rename conflicts keeps a conflicting metadata column reachable under a different physical attribute name. For example, a data column and metadata column both logically named {{index}} produce this relation shape:
> {code}
> AttributeReference.name = "_index"
> AttributeReference.metadata.__metadata_col = "index"
> relation.table.metadataColumns() = [index]
> {code}
> The internal {{__metadata_col}} entry stores the metadata column's logical connector name; it is not a Boolean marker. Previously, {{extractMetadataColumns}} collected {{AttributeReference.name}}, so it looked for physical name {{_index}} in {{relation.table.metadataColumns()}}. The connector reports logical name {{index}}, the lookup returned no match, and the captured metadata column was omitted from schema compatibility validation. A later type or nullability change to the logical {{index}} metadata column could therefore pass silently.
> *Expected:* the physical rename remains valid, while the captured logical metadata column continues to be checked for type and nullability changes.
> *Actual:* the renamed captured metadata column is dropped from validation because its physical name does not match the connector's logical name.
> Renaming connectors are unaffected by the first issue's shadowing rejection because their metadata columns remain reachable. The second issue specifically restores schema validation for those renamed metadata attributes; an unchanged renamed metadata column remains valid.
> h3. Fix
> * Detect a still-reported metadata column hidden by a same-named data column in the shared {{validateCapturedMetadataColumns}} path and raise {{INCOMPATIBLE_TABLE_CHANGE_AFTER_ANALYSIS.METADATA_COLUMNS_MISMATCH}}. Only the suppressed case is rejected; the renaming case keeps working.
> * Extract captured metadata attributes with {{MetadataAttributeWithLogicalName}} and match {{relation.table.metadataColumns()}} using the logical name stored in {{__metadata_col}}, so type and nullability changes to renamed metadata columns are validated.
> * Correct the outdated internal comments: {{__metadata_col}} stores the logical column name, while {{__file_source_metadata_col}} is the Boolean marker.
> PR: https://github.com/apache/spark/pull/58295
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org