You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Peter Toth (Jira)" <ji...@apache.org> on 2026/08/05 12:51:00 UTC

[jira] [Resolved] (SPARK-58562) Document subplan merging in the SQL performance tuning guide

     [ https://issues.apache.org/jira/browse/SPARK-58562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Peter Toth resolved SPARK-58562.
--------------------------------
    Fix Version/s: 4.3.0
       Resolution: Fixed

Issue resolved by pull request 57765
[https://github.com/apache/spark/pull/57765]

> Document subplan merging in the SQL performance tuning guide
> ------------------------------------------------------------
>
>                 Key: SPARK-58562
>                 URL: https://issues.apache.org/jira/browse/SPARK-58562
>             Project: Spark
>          Issue Type: Documentation
>          Components: SQL
>    Affects Versions: 4.3.0
>            Reporter: Peter Toth
>            Assignee: Peter Toth
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.3.0
>
>
> The MergeSubplans optimizer rule (org.apache.spark.sql.execution.planmerging.MergeSubplans) merges one-row-result subplans - scalar subqueries and non-grouping aggregates - that read the same input, so the input is scanned once instead of once per subplan. The rule is on by default and has 4 public configs under spark.sql.optimizer.mergeSubplans.filterPropagation.*, 3 of which are disabled by default and need a user decision.
> Today none of this is described in the docs. The configs only show up in the generated SQL config table in configuration.md, and the single mention anywhere else is the 4.3 migration-guide note about the rule's package move. In particular users have no way to discover that symmetric filter propagation, while off by default because OR-widening the merged filter can cost IO pruning, gave multi-fold speedups on TPC-DS query shapes that compute several differently-filtered aggregates over the same table.
> Add a section to docs/sql-performance-tuning.md that covers what the rule does, the query shapes it applies to, the configs and their tradeoffs, and how to disable the rule.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org