[jira] [Resolved] (SPARK-56339) Fix BatchScanExec canonicalization to enable ReuseSubquery for DSv2 scans

["Vitalii Li (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-56339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vitalii Li resolved SPARK-56339.
--------------------------------
    Resolution: Not A Bug

This is an issue on Delta side, fixed by https://github.com/delta-io/delta/pull/6477

> Fix BatchScanExec canonicalization to enable ReuseSubquery for DSv2 scans
> -------------------------------------------------------------------------
>
>                 Key: SPARK-56339
>                 URL: https://issues.apache.org/jira/browse/SPARK-56339
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 4.2.0
>            Reporter: Vitalii Li
>            Priority: Major
>              Labels: pull-request-available
>
>  `ReuseSubquery` fails to detect identical DSv2 subquery plans because `BatchScanExec` does not properly canonicalize or compare its scan/batch/table fields.
>   Problem
>   When a query contains multiple identical scalar subqueries scanning the same DSv2 table, `ReuseSubquery` should replace duplicates with `ReusedSubqueryExec`. This works for DSv1 (`FileSourceScanExec`) but fails for DSv2 (`BatchScanExec`).
>   Impact
>   Any query with repeated scalar subqueries on the same DSv2 table executes each subquery independently instead of reusing results. The number of redundant table scans scales linearly with the number of identical subqueries.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org