[jira] [Created] (SPARK-60011) Keep a DSv2 nested-field ordering after schema pruning

["Yahya Kisana (Jira)" <ji...@apache.org>]

Yahya Kisana created SPARK-60011:
------------------------------------

             Summary: Keep a DSv2 nested-field ordering after schema pruning
                 Key: SPARK-60011
                 URL: https://issues.apache.org/jira/browse/SPARK-60011
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Yahya Kisana


[SPARK-59948|https://issues.apache.org/jira/projects/SPARK/issues/SPARK-59948] strips the Alias around a nested sort key, so a DSv2 scan's reported ordering on s.x can satisfy a required ordering on s.x. This only works while the scan keeps the whole struct. When nested schema pruning narrows the struct, Spark still adds a redundant
SortExec.

With s struct<x: bigint, y: bigint> and a reported ordering on s.x:
|query|scan output s|SortExec|
|SELECT * FROM t SORT BY s.x|struct<x, y>|no|
|SELECT id, s FROM t SORT BY s.x|struct<x, y>|no|
|SELECT id, s.x FROM t SORT BY s.x|struct<x>|yes|

 

*Cause:* V2ScanPartitioningAndOrdering resolves the ordering against the full DataSourceV2Relation, so its s attribute keeps the unpruned type. The required ordering uses s from the pruned scan output. AttributeReference equality includes the data type, so the two GetStructFields never match.

Rebinding s to the scan output by exprId is not enough on its own: the GetStructField ordinal also comes from the full struct and must be recomputed against the pruned type.

Query results are correct. The cost is an unnecessary sort.

Found by [~petertoth] while reviewing https://github.com/apache/spark/pull/59199.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org