[jira] [Created] (SPARK-59014) DataSourceV2 refresh validation misses a data column that hides a captured metadata column

["Yan Yan (Jira)" <ji...@apache.org>]

Yan Yan created SPARK-59014:
-------------------------------

             Summary: DataSourceV2 refresh validation misses a data column that hides a captured metadata column
                 Key: SPARK-59014
                 URL: https://issues.apache.org/jira/browse/SPARK-59014
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 4.2.0
            Reporter: Yan Yan


When a DataSourceV2 relation is refreshed / re-resolved, {{V2TableUtil.validateCapturedMetadataColumns}} validates the captured metadata columns against the current table. It compares the captured metadata columns only against the metadata columns the connector still reports, so a name conflict that arrives on the *data* side is invisible to it.

If a data column takes the name of a captured metadata column and the connector does not rename conflicts ({{SupportsMetadataColumns.canRenameConflictingMetadataColumns()}} returns {{false}}, the default), {{LogicalPlan.metadataOutputWithOutConflicts}} suppresses the metadata column. A captured reference to it can then no longer be resolved. On a partially-pruned scan, {{PushDownUtils.toOutputAttrs}} maps the read schema onto the relation output by name and collapses the two same-named fields onto a single attribute, so a query that asked for the metadata column silently returns the *data* column's values.

*Expected:* refresh / re-resolution fails with a clear error.
*Actual:* on a suppressing connector with partial column pruning, the query silently returns the data column's values where the metadata column was expected (a wrong-results bug); with full pruning it happens to return the correct value.

Connectors that rename conflicting metadata columns are unaffected. The {{SupportsMetadataColumns}} contract already recommends that a non-renaming source reject such a data-column name, but that is only a recommendation and is not enforced, so Spark should validate it.

*Fix:* detect the conflict in the shared {{validateCapturedMetadataColumns}} and raise {{INCOMPATIBLE_TABLE_CHANGE_AFTER_ANALYSIS.METADATA_COLUMNS_MISMATCH}}. Only the suppressed case is rejected; the renaming case keeps working.

PR: https://github.com/apache/spark/pull/58295



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org