You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by we...@apache.org on 2025/03/05 05:17:18 UTC
(spark) branch branch-4.0 updated: [SPARK-51357][SQL] Preserve plan change logging level for views
This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.0 by this push:
new 1cfbe4608228 [SPARK-51357][SQL] Preserve plan change logging level for views
1cfbe4608228 is described below
commit 1cfbe46082289e47c48e4842c4cf45bc21e379fe
Author: Vladimir Golubev <vl...@databricks.com>
AuthorDate: Wed Mar 5 13:16:48 2025 +0800
[SPARK-51357][SQL] Preserve plan change logging level for views
### What changes were proposed in this pull request?
Preserve plan change logging level for views:
- `spark.sql.planChangeLog.level`
- `spark.sql.expressionTreeChangeLog.level`
Before:
After:
### Why are the changes needed?
Plan change logging level is not preserved for views. This hides part of the resolution process for queries with views and makes debugging harder.
### Does this PR introduce _any_ user-facing change?
No, just debugging becomes easier.
### How was this patch tested?
Existing tests.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #50118 from vladimirg-db/vladimir-golubev_data/preserve-plan-change-logging-setting-for-views.
Authored-by: Vladimir Golubev <vl...@databricks.com>
Signed-off-by: Wenchen Fan <we...@databricks.com>
(cherry picked from commit a6600afbcba1dec251e7c751c7bd463b9e9ee35d)
Signed-off-by: Wenchen Fan <we...@databricks.com>
---
.../plans/logical/basicLogicalOperators.scala | 28 +++++++++++++++-------
1 file changed, 19 insertions(+), 9 deletions(-)
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala
index fb1999148d60..7d279092668f 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala
@@ -840,20 +840,30 @@ object View {
// For temporary view, we always use captured sql configs
if (activeConf.useCurrentSQLConfigsForView && !isTempView) return activeConf
- val sqlConf = new SQLConf()
// We retain below configs from current session because they are not captured by view
// as optimization configs but they are still needed during the view resolution.
- // TODO: remove this `retainedConfigs` after the `RelationConversions` is moved to
+ // TODO: remove this `retainedHiveConfigs` after the `RelationConversions` is moved to
// optimization phase.
+ val retainedHiveConfigs = Seq(
+ "spark.sql.hive.convertMetastoreParquet",
+ "spark.sql.hive.convertMetastoreOrc",
+ "spark.sql.hive.convertInsertingPartitionedTable",
+ "spark.sql.hive.convertInsertingUnpartitionedTable",
+ "spark.sql.hive.convertMetastoreCtas"
+ )
+
+ val retainedLoggingConfigs = Seq(
+ "spark.sql.planChangeLog.level",
+ "spark.sql.expressionTreeChangeLog.level"
+ )
+
val retainedConfigs = activeConf.getAllConfs.filter { case (key, _) =>
- Seq(
- "spark.sql.hive.convertMetastoreParquet",
- "spark.sql.hive.convertMetastoreOrc",
- "spark.sql.hive.convertInsertingPartitionedTable",
- "spark.sql.hive.convertInsertingUnpartitionedTable",
- "spark.sql.hive.convertMetastoreCtas"
- ).contains(key) || key.startsWith("spark.sql.catalog.")
+ retainedHiveConfigs.contains(key) || retainedLoggingConfigs.contains(key) || key.startsWith(
+ "spark.sql.catalog."
+ )
}
+
+ val sqlConf = new SQLConf()
for ((k, v) <- configs ++ retainedConfigs) {
sqlConf.settings.put(k, v)
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org