[jira] [Created] (SPARK-59948) Strip aliases from a DSv2 reported ordering on a nested field

["Anurag Mantripragada (Jira)" <ji...@apache.org>]

Anurag Mantripragada created SPARK-59948:
--------------------------------------------

             Summary: Strip aliases from a DSv2 reported ordering on a nested field
                 Key: SPARK-59948
                 URL: https://issues.apache.org/jira/browse/SPARK-59948
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Anurag Mantripragada


{{V2ScanPartitioningAndOrdering}} converts a scan's {{SupportsReportOrdering}} ordering on every application of the rule, and unlike the partitioning pass, the ordering pass has no guard. For a nested sort key such as {{s.x}}, {{LogicalPlan.resolve}} returns a new {{Alias(GetStructField(...))}} with a fresh {{ExprId}} each time.

As a result:
* In testing, the "Early Filter and Projection Push-Down" Once batch fails its idempotence check:
{noformat}
Once strategy's idempotence is broken for batch Early Filter and Projection Push-Down
{noformat}
* In production, the aliased {{SortOrder}} never semantically equals a required {{SortOrder}} on {{s.x}}, so Spark adds a redundant {{SortExec}} even though the scan reports that ordering.
* {{PlanMerger}} treats two derivations of the same nested ordering as different, so scan merging can decline.

Repro: a DSv2 scan that implements {{SupportsReportOrdering}} and reports {{sort(FieldReference(Seq("s", "x")), ASC, NULLS_FIRST)}} on a table with a struct column {{s}}, then a query such as {{SELECT s.x FROM t}} or {{sortWithinPartitions($"s.x")}}.

Proposed fix: strip aliases from the converted sort orders in the ordering pass (e.g. {{trimAliases}} from {{AliasHelper}}), so that repeated applications produce equal plans and the reported ordering can satisfy a required ordering.

This is pre-existing on master. It was found while reviewing SPARK-59721 (https://github.com/apache/spark/pull/58979). A related but separate issue: a nested partition key gets the same {{Alias}} and is rejected by {{KeyedPartitioning.supportsExpressions}}, so storage-partitioned join never applies to it.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org