You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ch...@apache.org on 2026/08/28 02:28:23 UTC
(spark) branch branch-4.x updated: [SPARK-59023][SQL] Support none mode in spark.sql.ui.explainMode to skip plan description in SQL events
This is an automated email from the ASF dual-hosted git repository.
pan3793 pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 1dcefc6709c3 [SPARK-59023][SQL] Support none mode in spark.sql.ui.explainMode to skip plan description in SQL events
1dcefc6709c3 is described below
commit 1dcefc6709c3f220988f268399c47c30b044de11
Author: Cheng Pan <pa...@gmail.com>
AuthorDate: Fri Aug 28 10:26:03 2026 +0800
[SPARK-59023][SQL] Support none mode in spark.sql.ui.explainMode to skip plan description in SQL events
### What changes were proposed in this pull request?
Add a `none` option to `spark.sql.ui.explainMode`. When it is set, `SparkListenerSQLExecutionStart` and `SparkListenerSQLAdaptiveExecutionUpdate` carry a placeholder string instead of the rendered plan description:
```
No plan description because spark.sql.ui.explainMode=none
```
This skips the explain string generation for the SQL UI entirely. The change touches:
- `SQLConf.UI_EXPLAIN_MODE`: accepts `none` in addition to `simple`, `extended`, `codegen`, `cost`, `formatted` (doc and validation updated)
- `SQLExecution`: hosts `planDescription`, which returns the placeholder instead of calling `QueryExecution.explainString` when the mode is `none`
- `AdaptiveSparkPlanExec`: uses the same helper for the plan description posted with adaptive execution updates
### Why are the changes needed?
Rendering the plan description is done solely for the SQL UI. For workloads with large plans, or with adaptive execution enabled (where the plan description is re-rendered on every adaptive update), this generation is non-trivial overhead that users who do not use the SQL UI cannot avoid today, since every valid mode renders something. `none` lets them opt out of the cost.
For example, we have a customer job that constructs a huge plan whose `treeString` exceeds 280,000 lines. Rendering the plan description takes more than 3 minutes per iteration, and with AQE enabled, assembling the plan tree strings takes more than 40 minutes across the query execution. A string this large also pressures driver memory (we have observed driver OOM), and the browser becomes unresponsive when the SQL UI tries to render it.
`spark.sql.maxPlanStringLength` does not address this. `StringConcat.append` only skips storing text past the limit, and `TreeNode.generateTreeString` has no early exit, so every node's string is still materialized. It bounds the final string's memory, not the CPU spent building it.
### Does this PR introduce _any_ user-facing change?
Yes. `spark.sql.ui.explainMode` now accepts `none`. When set, the SQL UI shows the placeholder text above instead of the query plan description for SQL executions. The default value (`formatted`) is unchanged.
### How was this patch tested?
The existing `control a plan explain mode in listeners via SQLConf` tests are extended with a `none` case, asserting the placeholder is carried by the events:
- `AdaptiveQueryExecSuite` (covers `SparkListenerSQLAdaptiveExecutionUpdate`)
- `SQLAppStatusListenerSuite` (covers `SparkListenerSQLExecutionStart`)
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Qwen3.8 Max
Closes #58313 from pan3793/SPARK-59023.
Authored-by: Cheng Pan <pa...@gmail.com>
Signed-off-by: Cheng Pan <ch...@apache.org>
(cherry picked from commit d8429a174ec699a7e9ddff1ee9b7c82b60900ba7)
Signed-off-by: Cheng Pan <ch...@apache.org>
---
.../org/apache/spark/sql/internal/SQLConf.scala | 11 ++++-----
.../apache/spark/sql/execution/SQLExecution.scala | 26 +++++++++++++++++++---
.../execution/adaptive/AdaptiveSparkPlanExec.scala | 3 +--
.../adaptive/AdaptiveQueryExecSuite.scala | 3 ++-
.../execution/ui/SQLAppStatusListenerSuite.scala | 3 ++-
5 files changed, 34 insertions(+), 12 deletions(-)
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index fbbad38a8a5d..a9d79df84497 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -6664,14 +6664,15 @@ object SQLConf {
.createWithDefault(false)
val UI_EXPLAIN_MODE = buildConf("spark.sql.ui.explainMode")
- .doc("Configures the query explain mode used in the Spark SQL UI. The value can be 'simple', " +
- "'extended', 'codegen', 'cost', or 'formatted'. The default value is 'formatted'.")
+ .doc("Configures the query explain mode used in the Spark SQL UI. The value can be 'none', " +
+ "'simple', 'extended', 'codegen', 'cost', or 'formatted'. The default value is 'formatted'.")
.version("3.1.0")
.stringConf
.transform(_.toUpperCase(Locale.ROOT))
- .checkValue(mode => Set("SIMPLE", "EXTENDED", "CODEGEN", "COST", "FORMATTED").contains(mode),
- "Invalid value for 'spark.sql.ui.explainMode'. Valid values are 'simple', 'extended', " +
- "'codegen', 'cost' and 'formatted'.")
+ .checkValue(mode =>
+ Set("NONE", "SIMPLE", "EXTENDED", "CODEGEN", "COST", "FORMATTED").contains(mode),
+ "Invalid value for 'spark.sql.ui.explainMode'. Valid values are 'none', 'simple', " +
+ "'extended', 'codegen', 'cost' and 'formatted'.")
.createWithDefault("formatted")
val SOURCES_BINARY_FILE_MAX_LENGTH = buildConf("spark.sql.sources.binaryFile.maxLength")
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala
index f25e908a9cdb..e296e00ba48d 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala
@@ -120,6 +120,27 @@ object SQLExecution extends Logging {
}
}
+ private[sql] val NONE_EXPLAIN_MODE = "none"
+
+ private[sql] val NO_PLAN_DESCRIPTION =
+ s"No plan description because ${SQLConf.UI_EXPLAIN_MODE.key}=$NONE_EXPLAIN_MODE"
+
+ /**
+ * Returns the plan description carried by the SQL UI events, or a placeholder when the UI
+ * explain mode is `none`, in which case the explain string, which is expensive to build for
+ * large plans, is not generated at all.
+ *
+ * `none` is intentionally not an [[ExplainMode]]: it only makes sense for the UI events, and
+ * keeping it here avoids making `df.explain("none")` a valid public API.
+ */
+ private[sql] def planDescription(qe: QueryExecution, uiExplainMode: String): String = {
+ if (uiExplainMode.equalsIgnoreCase(NONE_EXPLAIN_MODE)) {
+ NO_PLAN_DESCRIPTION
+ } else {
+ qe.explainString(ExplainMode.fromString(uiExplainMode))
+ }
+ }
+
/**
* Wrap an action that will execute "queryExecution" to track all Spark jobs in the body so that
* we can connect them with an execution.
@@ -209,9 +230,8 @@ object SQLExecution extends Logging {
sc.listenerBus.post(startEvent)
throw e
case Right(f) =>
- val planDescriptionMode =
- ExplainMode.fromString(sparkSession.sessionState.conf.uiExplainMode)
- val planDesc = queryExecution.explainString(planDescriptionMode)
+ val planDesc = planDescription(
+ queryExecution, sparkSession.sessionState.conf.uiExplainMode)
val planInfo = try {
SparkPlanInfo.fromSparkPlan(queryExecution.executedPlan)
} catch {
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala
index a12b202b5b00..0ef9b02d32d0 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala
@@ -1145,10 +1145,9 @@ case class AdaptiveSparkPlanExec(
context.session.sparkContext.listenerBus.post(SparkListenerSQLAdaptiveSQLMetricUpdates(
executionId, newMetrics))
} else {
- val planDescriptionMode = ExplainMode.fromString(conf.uiExplainMode)
context.session.sparkContext.listenerBus.post(SparkListenerSQLAdaptiveExecutionUpdate(
executionId,
- context.qe.explainString(planDescriptionMode),
+ SQLExecution.planDescription(context.qe, conf.uiExplainMode),
SparkPlanInfo.fromSparkPlan(context.qe.executedPlan)))
}
}
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
index 8e98d7785f6c..3ac16235333c 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
@@ -2541,7 +2541,8 @@ class AdaptiveQueryExecSuite
"== Optimized Logical Plan ==", "== Physical Plan ==")),
("codegen", Seq("WholeStageCodegen subtrees")),
("cost", Seq("== Optimized Logical Plan ==", "Statistics(sizeInBytes")),
- ("formatted", Seq("== Physical Plan ==", "Output", "Arguments"))).foreach {
+ ("formatted", Seq("== Physical Plan ==", "Output", "Arguments")),
+ ("none", Seq(SQLExecution.NO_PLAN_DESCRIPTION))).foreach {
case (mode, expected) =>
checkPlanDescription(mode, expected)
}
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListenerSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListenerSuite.scala
index 9230eb48dcca..7d9c79ab9236 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListenerSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListenerSuite.scala
@@ -369,7 +369,8 @@ abstract class SQLAppStatusListenerSuite extends SharedSparkSession with JsonTes
"== Optimized Logical Plan ==", "== Physical Plan ==")),
("codegen", Seq("WholeStageCodegen subtrees")),
("cost", Seq("== Optimized Logical Plan ==", "Statistics(sizeInBytes")),
- ("formatted", Seq("== Physical Plan ==", "Output", "Arguments"))).foreach {
+ ("formatted", Seq("== Physical Plan ==", "Output", "Arguments")),
+ ("none", Seq(SQLExecution.NO_PLAN_DESCRIPTION))).foreach {
case (mode, expected) =>
checkPlanDescription(mode, expected)
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org