You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Peter Toth (Jira)" <ji...@apache.org> on 2026/10/16 11:09:00 UTC
[jira] [Commented] (SPARK-59948) Strip aliases from a DSv2 reported ordering on a nested field
[ https://issues.apache.org/jira/browse/SPARK-59948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18127791#comment-18127791 ]
Peter Toth commented on SPARK-59948:
------------------------------------
Changed the type to Improvement and the affected version to 4.4.0. Query results were never wrong. Without this fix, Spark added a redundant sort above a scan that reports an ordering on a nested field. {{PlanMerger}} also did not merge two scans of such a table. The {{Once}} batch idempotence check that failed runs only in tests.
> Strip aliases from a DSv2 reported ordering on a nested field
> -------------------------------------------------------------
>
> Key: SPARK-59948
> URL: https://issues.apache.org/jira/browse/SPARK-59948
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: Anurag Mantripragada
> Assignee: Yahya Kisana
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.4.0
>
>
> {{V2ScanPartitioningAndOrdering}} converts a scan's {{SupportsReportOrdering}} ordering on every application of the rule, and unlike the partitioning pass, the ordering pass has no guard. For a nested sort key such as {{s.x}}, {{LogicalPlan.resolve}} returns a new {{Alias(GetStructField(...))}} with a fresh {{ExprId}} each time.
> As a result:
> * In testing, the "Early Filter and Projection Push-Down" Once batch fails its idempotence check:
> {noformat}
> Once strategy's idempotence is broken for batch Early Filter and Projection Push-Down
> {noformat}
> * In production, the aliased {{SortOrder}} never semantically equals a required {{SortOrder}} on {{s.x}}, so Spark adds a redundant {{SortExec}} even though the scan reports that ordering.
> * {{PlanMerger}} treats two derivations of the same nested ordering as different, so scan merging can decline.
> Repro: a DSv2 scan that implements {{SupportsReportOrdering}} and reports {{sort(FieldReference(Seq("s", "x")), ASC, NULLS_FIRST)}} on a table with a struct column {{s}}, then a query such as {{SELECT s.x FROM t}} or {{sortWithinPartitions($"s.x")}}.
> Proposed fix: strip aliases from the converted sort orders in the ordering pass (e.g. {{trimAliases}} from {{AliasHelper}}), so that repeated applications produce equal plans and the reported ordering can satisfy a required ordering.
> This is pre-existing on master. It was found while reviewing SPARK-59721 (https://github.com/apache/spark/pull/58979). A related but separate issue: a nested partition key gets the same {{Alias}} and is rejected by {{KeyedPartitioning.supportsExpressions}}, so storage-partitioned join never applies to it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org