You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2026/04/30 20:54:26 UTC
(spark) branch master updated: [SPARK-40193][SQL][FOLLOWUP] Rename `spark.sql.optimizer.mergeSubplans.( -> filterPropagation.)symmetricFilterPropagation.enabled`
This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 6593e21a2189 [SPARK-40193][SQL][FOLLOWUP] Rename `spark.sql.optimizer.mergeSubplans.( -> filterPropagation.)symmetricFilterPropagation.enabled`
6593e21a2189 is described below
commit 6593e21a21895d3b6cfc9ec73a8c42d48786c538
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Thu Apr 30 13:54:08 2026 -0700
[SPARK-40193][SQL][FOLLOWUP] Rename `spark.sql.optimizer.mergeSubplans.( -> filterPropagation.)symmetricFilterPropagation.enabled`
### What changes were proposed in this pull request?
This PR renames the SQL config key
`spark.sql.optimizer.mergeSubplans.symmetricFilterPropagation.enabled` to `spark.sql.optimizer.mergeSubplans.filterPropagation.symmetricFilterPropagation.enabled`.
### Why are the changes needed?
To follow the Apache Spark configuration namespace rule. The new key better reflects the configuration hierarchy. `spark.sql.optimizer.mergeSubplans.filterPropagation.enabled` is the parent gate for filter propagation in `MergeSubplans`, and the symmetric variant is a sub-feature of it. Nesting the symmetric flag under `filterPropagation.*` makes that relationship explicit, and groups related options together when listed alphabetically.
The previous key `mergeSubplans.symmetricFilterPropagation.enabled` placed the two sibling options at different levels of the namespace, which obscured the dependency (the symmetric flag has no effect when filter propagation is disabled, as documented in the config `doc`).
This config was added in 4.2.0 and has not been released yet, so no backwards-compat alias is required.
### Does this PR introduce _any_ user-facing change?
No. The config has not been released yet (introduced for 4.2.0).
### How was this patch tested?
Pass the CIs.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.7
Closes #55633 from dongjoon-hyun/dongjoon/suspicious-hofstadter-e56b11.
Authored-by: Dongjoon Hyun <do...@apache.org>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
.../org/apache/spark/sql/internal/SQLConf.scala | 28 +++++++++++-----------
1 file changed, 14 insertions(+), 14 deletions(-)
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index 99a5339950a5..a56296833d1e 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -6593,20 +6593,20 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
- val MERGE_SUBPLANS_SYMMETRIC_FILTER_PROPAGATION_ENABLED =
- buildConf("spark.sql.optimizer.mergeSubplans.symmetricFilterPropagation.enabled")
- .doc("When set to true, two non-grouping aggregate subplans that both have filter " +
- "conditions (but with different predicates) can be merged into a single scan using " +
- "FILTER (WHERE ...) clauses on each aggregate expression. " +
- "Merging two filtered scans broadens the combined filter to OR(f1, f2), which may " +
- "reduce IO pruning (e.g. partition or file skipping) compared to the individual " +
- "filters. Disabled by default; enable once the behaviour has been validated in your " +
- "workload, particularly on heavily partitioned or file-pruned tables. " +
- s"Has no effect when ${MERGE_SUBPLANS_FILTER_PROPAGATION_ENABLED.key} is false.")
- .version("4.2.0")
- .withBindingPolicy(ConfigBindingPolicy.SESSION)
- .booleanConf
- .createWithDefault(false)
+ val MERGE_SUBPLANS_SYMMETRIC_FILTER_PROPAGATION_ENABLED = buildConf(
+ "spark.sql.optimizer.mergeSubplans.filterPropagation.symmetricFilterPropagation.enabled")
+ .doc("When set to true, two non-grouping aggregate subplans that both have filter " +
+ "conditions (but with different predicates) can be merged into a single scan using " +
+ "FILTER (WHERE ...) clauses on each aggregate expression. " +
+ "Merging two filtered scans broadens the combined filter to OR(f1, f2), which may " +
+ "reduce IO pruning (e.g. partition or file skipping) compared to the individual " +
+ "filters. Disabled by default; enable once the behaviour has been validated in your " +
+ "workload, particularly on heavily partitioned or file-pruned tables. " +
+ s"Has no effect when ${MERGE_SUBPLANS_FILTER_PROPAGATION_ENABLED.key} is false.")
+ .version("4.2.0")
+ .withBindingPolicy(ConfigBindingPolicy.SESSION)
+ .booleanConf
+ .createWithDefault(false)
val ERROR_MESSAGE_FORMAT = buildConf("spark.sql.error.messageFormat")
.doc("When PRETTY, the error message consists of textual representation of error class, " +
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org