You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "sebastianhillig-db (via GitHub)" <gi...@apache.org> on 2026/10/09 07:19:25 UTC
[PR] [SPARK-60105][SQL] Memoize the retained session-config overlay for view and SQL function body confs [spark]
sebastianhillig-db opened a new pull request, #59320:
URL: https://github.com/apache/spark/pull/59320
### What changes were proposed in this pull request?
This is a follow-up to SPARK-59997 (#59254). That change stopped `Analyzer.retainResolutionConfigsForAnalysis` from scanning every registered config entry. The method still runs once per view and SQL function body conf build, though, and each run copies every explicitly set session config and looks up the binding policy of each key.
This PR computes the retained configs once per existing conf and memoizes them on that conf:
- `SQLConf.settings` is now a `ModificationCountingMap` instead of a `Collections.synchronizedMap`. As before, every map method synchronizes on the map itself. In addition:
- it counts every mutating call;
- `snapshotWithModificationCount` returns a copy of the entries together with the count they correspond to;
- the key, value, and entry views are read-only, so no mutation can bypass the count.
- `object SQLConf` keeps a version of the registered SQL config entries, which every `register` and `unregister` advances. This covers configs from lazily loaded modules (e.g. sql/hive) that register after a memo was computed.
- `SQLConf.retainedResolutionConfigs` returns the retained subset of the settings: `spark.sql.catalog.*` keys and configs bound `SESSION` or `NOT_APPLICABLE`, the same rule as SPARK-59997.
- It reuses the memo while both the settings count and the entries version are unchanged.
- Nested view and function bodies install their own conf, so they get their own memo.
- `ReadOnlySQLConf`, whose settings are task local properties, keeps the unmemoized path.
- `retainResolutionConfigsForAnalysis` now overlays the memoized map with `putAll` and then calls `trySetAnsiValue`, in the same order and with the same precedence as before.
- A new internal, SESSION-bound conf, `spark.sql.analyzer.memoizeRetainedResolutionConfigs.enabled` (default `true`), turns the memo off and restores the per-build recomputation.
### Why are the changes needed?
One query can rebuild many body confs from the same session conf. Every reference to a view or SQL function, including nested ones, builds a new conf and overlays the session's retained configs onto it. Before this PR, each build iterates over all explicitly set session configs and looks up the binding policy of each. With the memo, a build copies the already filtered map, and the filter runs again only after the session conf or the set of registered configs changes. Queries that reference many views or SQL functions benefit the most.
### Does this PR introduce _any_ user-facing change?
No. The resulting view and SQL function body confs are unchanged.
Two internal details of `SQLConf.settings`, which is `protected[spark]`, do change:
- Its key, value, and entry views are now read-only, so mutating the settings through them throws `UnsupportedOperationException`. No code in this repository does that.
- The views no longer lock on each call. As with iteration, callers must hold the map's monitor while using them.
The new conf turns off only the memo, not these changes.
### How was this patch tested?
New tests:
- `ModificationCountingMapSuite`:
- the count advances on every mutating call, including the `java.util.Map` default methods and the mutators of the `asScala` wrapper;
- the snapshot pairs a copy of the entries with their count;
- the views are read-only.
- `RetainedResolutionConfigsSuite`:
- view and SQL function body confs are identical with and without the memo, across captured ANSI values, creation versions, and `alwaysSetAnsiValue`;
- the memo is recomputed after each kind of settings change, and after a config is registered or unregistered;
- a nested conf gets its own memo and leaves the outer one untouched;
- turning the conf off recomputes on every call;
- `ReadOnlySQLConf` filters without memoizing.
- `AnalysisConfOverrideSuite`: a temporary view, a persistent view, and a SQL function each follow `spark.sql.analyzer.lowerEmptyGroupingSetToGlobalAggregate.enabled`, a SESSION-bound config, when it changes with `SET` between queries.
I also checked that the tests catch a stale memo: with the memo's settings check forced to always match, this test and the recompute test in `RetainedResolutionConfigsSuite` both fail.
Existing suites, run locally:
- catalyst `*AnalysisSuite`;
- `AnalysisConfOverrideSuite`, `SimpleSQLViewSuite`, `LocalTempViewTestSuite`, `GlobalTempViewTestSuite`, `PersistedViewTestSuite`, `SQLFunctionSuite`, `SQLConfSuite`, `ExecutorSideSQLConfSuite`;
- `SQLQueryTestSuite -z grouping_set`;
- `SparkConfigBindingPolicySuite`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60105][SQL] Memoize the retained session-config overlay for view and SQL function body confs [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #59320:
URL: https://github.com/apache/spark/pull/59320#discussion_r4258683487
##########
sql/catalyst/benchmarks/RetainedResolutionConfigsBenchmark-results.txt:
##########
@@ -0,0 +1,382 @@
+================================================================================================
+view: 32 settings, 25% retained, warm
+================================================================================================
+
+OpenJDK 64-Bit Server VM 17.0.15+6-Ubuntu-0ubuntu120.04 on Linux 5.4.0-1163-aws-fips
Review Comment:
**Nit (P3):** These numbers come from a local machine (Ubuntu OpenJDK 17.0.15 on Linux 5.4.0-aws-fips, Xeon 8375C). The other result sets in `sql/catalyst/benchmarks/` are generated by the "Run benchmarks" GitHub Actions workflow and include `-jdk21-results.txt` and `-jdk25-results.txt`, so this file can't be compared with them or regenerated the same way. Could you regenerate it with that workflow for JDK 17, 21, and 25? The `create-commit` option can push the results to this branch.
**Verification:**
- **Inspection:** All three JDK result files exist for the benchmark, and their environment headers match the workflow runner environment used by the other committed catalyst benchmark results.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:
##########
@@ -326,15 +326,30 @@ object Analyzer {
newConf: SQLConf,
existingConf: SQLConf,
createSparkVersion: String = ""): Unit = {
- existingConf.getAllConfs.foreach { case (key, value) =>
- if (key.startsWith("spark.sql.catalog.") || SQLConf.isSessionBindingPolicy(key)) {
- newConf.settings.put(key, value)
+ // The retained configs are memoized on `existingConf` unless
+ // `ANALYZER_MEMOIZE_RETAINED_RESOLUTION_CONFIGS` is off, which recomputes them on every build.
+ if (existingConf.getConf(SQLConf.ANALYZER_MEMOIZE_RETAINED_RESOLUTION_CONFIGS)) {
+ newConf.settings.putAll(existingConf.retainedResolutionConfigs)
+ } else {
+ existingConf.getAllConfs.foreach { case (key, value) =>
+ if (bodyFollowsCaller(key)) {
+ newConf.settings.put(key, value)
+ }
}
}
trySetAnsiValue(newConf, createSparkVersion)
}
+ /**
+ * True if a view or SQL function body takes the caller's value for `key` rather than the
Review Comment:
**Nit (P3):** This reads as if the body always takes the caller's value for these keys, but the overlay copies only keys the caller has explicitly set, and only where session overrides are applied. For example, if a view is created while a SESSION-bound key such as `spark.sql.analyzer.lowerEmptyGroupingSetToGlobalAggregate.enabled` is explicitly `false` and the caller later `RESET`s it, the body keeps the saved `false`. Function confs built by `SessionCatalog` (`applySessionOverrides = false`) also keep saved values. Could the wording be narrowed, e.g. "True if the caller's explicitly set value for `key` is overlaid onto view and SQL function body confs (when session overrides apply), replacing the value saved with the body: catalog configs and SQL configs bound SESSION or NOT_APPLICABLE."?
**Verification:**
- **Inspection:** The revised doc matches the overlay behavior: only explicitly set caller keys selected by the predicate override saved values, and only on paths that call retainResolutionConfigsForAnalysis.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -2893,6 +2935,19 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val ANALYZER_MEMOIZE_RETAINED_RESOLUTION_CONFIGS =
+ buildConf("spark.sql.analyzer.memoizeRetainedResolutionConfigs.enabled")
+ .internal()
+ .version("4.4.0")
+ .withBindingPolicy(ConfigBindingPolicy.SESSION)
Review Comment:
**Nit (P3):** The `ConfigBindingPolicy` guidance for new configs says: "Can the config change the result of resolving the body of a view/UDF/procedure, i.e. the resolved plan? If not, use NOT_APPLICABLE." Since both values of this flag produce the same body conf, `NOT_APPLICABLE` seems like the right label. `isSessionBindingPolicy` retains `NOT_APPLICABLE` keys too, so nested bodies would still inherit the caller's explicitly set value.
**Verification:**
- **Inspection:** The config's declared binding policy is NOT_APPLICABLE, as the ConfigBindingPolicy selection rule requires for a config that cannot change the resolved body plan.
- **Compatibility:** Memo-on and memo-off view and SQL function body confs remain identical, and an explicitly set flag value still propagates into nested body confs.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/ModificationCountingMap.scala:
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.internal
+
+import java.util
+import java.util.function.{BiConsumer, BiFunction, Function => JFunction}
+
+/**
+ * A thread-safe map of [[SQLConf]] settings that counts the mutating calls on it, so values
+ * derived from its entries can be reused while [[modificationCount]] is unchanged.
+ *
+ * Like the `java.util.Collections.synchronizedMap` it replaces, every method of the map
+ * synchronizes on the map itself, so a caller iterating the map must hold its monitor. Unlike it,
+ * the key, value, and entry views are read-only, so every mutation goes through a method that
+ * advances the count, and unsynchronized, so a caller must hold the monitor while using them.
+ */
+private[sql] final class ModificationCountingMap extends util.Map[String, String] {
+
+ private val underlying = new util.HashMap[String, String]
+
+ private val readOnlyUnderlying = util.Collections.unmodifiableMap[String, String](underlying)
+
+ @volatile private var mutations = 0L
+
+ /**
+ * Number of mutating calls so far; a call may advance it without changing the entries. Values
+ * derived from the entries stay valid while it is unchanged; use
+ * [[snapshotWithModificationCount]] to pair a count with the entries it describes.
+ */
+ def modificationCount: Long = mutations
+
+ /** A copy of the entries and the [[modificationCount]] they correspond to. */
+ def snapshotWithModificationCount: (Long, util.Map[String, String]) = synchronized {
+ (mutations, new util.HashMap[String, String](underlying))
+ }
+
+ private def mutate[T](f: => T): T = synchronized {
Review Comment:
**Nit (P3):** `getAllConfs` (`settings.synchronized { ... }`), `setConf(Properties)`, and the `RuntimeConfig` PythonWorkerEnvironment check-then-write all depend on these methods locking the map itself. `Collections.synchronizedMap` guaranteed that before; now it's implemented here, and none of the new tests would fail if `mutate` (or the reads) switched to a private lock. A small test would pin it down. For example, assert `Thread.holdsLock(map)` inside a `compute`/`merge` callback, or check that a writer thread blocks while the test holds `map.synchronized`.
**Verification:**
- **Regression:** The new test passes against the current implementation and fails when mutate is changed to use a different lock or no lock.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60105][SQL] Memoize the retained session-config overlay for view and SQL function body confs [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #59320:
URL: https://github.com/apache/spark/pull/59320#discussion_r4271696331
##########
sql/core/src/main/scala/org/apache/spark/sql/classic/RuntimeConfig.scala:
##########
@@ -46,8 +46,8 @@ class RuntimeConfig private[sql](val sqlConf: SQLConf = new SQLConf) extends sql
// Connect config RPC writes through this method too, so both front ends behave the same way.
//
// Scoped to the prefix so an ordinary configuration write does not take the monitor. Check
- // and write under the monitor that guards the session configurations -- `settings` is a
- // `Collections.synchronizedMap`, so every `put` locks the wrapper this block locks -- because
+ // and write under the monitor that guards the session configurations -- every method of
Review Comment:
**Nit (P3):** This now says every method of `settings` locks the map itself, but `ModificationCountingMap`'s `keySet`, `values`, and `entrySet` return unsynchronized views. Its class doc and `ModificationCountingMapSuite` ("every method except the views locks the map itself") both carve them out. The block itself is still correct, because `put` goes through `mutate`, which synchronizes on the map. The old wording was accurate because `Collections.synchronizedMap` views lock the same mutex. The new wording could lead someone to call, say, `sqlConf.settings.keySet.contains(k)` without holding the monitor and race with a concurrent `SET`. Could it be narrowed to something like "every method of `settings` except its views locks the map itself, so every `put` locks the map this block locks"?
**Verification:**
- **Inspection:** The revised comment's locking claim matches ModificationCountingMap: non-view methods (including put) synchronize on the map itself, and keySet/values/entrySet are excluded from that claim.
##########
sql/catalyst/benchmarks/RetainedResolutionConfigsBenchmark-results.txt:
##########
@@ -0,0 +1,382 @@
+================================================================================================
+view: 32 settings, 25% retained, warm
+================================================================================================
+
+OpenJDK 64-Bit Server VM 17.0.15+6-Ubuntu-0ubuntu120.04 on Linux 5.4.0-1163-aws-fips
Review Comment:
Thanks, verified: all three result files now come from the "Run benchmarks" workflow runner (AMD EPYC 7763, Linux 6.17.0-1022-azure) for JDK 17, 21, and 25, consistent with the other catalyst benchmark results.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4258683487","thread_id":"inline:4258683487","verdict_sha256":"ae877c56c9e26aa21afebfcdd9e33ffa742e923b49b144b08d69c09e8df9705d"} -->
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:
##########
@@ -326,15 +326,30 @@ object Analyzer {
newConf: SQLConf,
existingConf: SQLConf,
createSparkVersion: String = ""): Unit = {
- existingConf.getAllConfs.foreach { case (key, value) =>
- if (key.startsWith("spark.sql.catalog.") || SQLConf.isSessionBindingPolicy(key)) {
- newConf.settings.put(key, value)
+ // The retained configs are memoized on `existingConf` unless
+ // `ANALYZER_MEMOIZE_RETAINED_RESOLUTION_CONFIGS` is off, which recomputes them on every build.
+ if (existingConf.getConf(SQLConf.ANALYZER_MEMOIZE_RETAINED_RESOLUTION_CONFIGS)) {
+ newConf.settings.putAll(existingConf.retainedResolutionConfigs)
+ } else {
+ existingConf.getAllConfs.foreach { case (key, value) =>
+ if (bodyFollowsCaller(key)) {
+ newConf.settings.put(key, value)
+ }
}
}
trySetAnsiValue(newConf, createSparkVersion)
}
+ /**
+ * True if a view or SQL function body takes the caller's value for `key` rather than the
Review Comment:
Thanks, verified: the narrowed doc now says only the caller's explicitly set keys replace saved values, and only on paths that go through `retainResolutionConfigsForAnalysis`.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4258683496","thread_id":"inline:4258683496","verdict_sha256":"ae877c56c9e26aa21afebfcdd9e33ffa742e923b49b144b08d69c09e8df9705d"} -->
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -2893,6 +2935,19 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val ANALYZER_MEMOIZE_RETAINED_RESOLUTION_CONFIGS =
+ buildConf("spark.sql.analyzer.memoizeRetainedResolutionConfigs.enabled")
+ .internal()
+ .version("4.4.0")
+ .withBindingPolicy(ConfigBindingPolicy.SESSION)
Review Comment:
Thanks, verified: the flag is now `NOT_APPLICABLE`, and the on/off equivalence tests still expect an explicitly set value to follow into body confs.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4258683499","thread_id":"inline:4258683499","verdict_sha256":"ae877c56c9e26aa21afebfcdd9e33ffa742e923b49b144b08d69c09e8df9705d"} -->
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/ModificationCountingMap.scala:
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.internal
+
+import java.util
+import java.util.function.{BiConsumer, BiFunction, Function => JFunction}
+
+/**
+ * A thread-safe map of [[SQLConf]] settings that counts the mutating calls on it, so values
+ * derived from its entries can be reused while [[modificationCount]] is unchanged.
+ *
+ * Like the `java.util.Collections.synchronizedMap` it replaces, every method of the map
+ * synchronizes on the map itself, so a caller iterating the map must hold its monitor. Unlike it,
+ * the key, value, and entry views are read-only, so every mutation goes through a method that
+ * advances the count, and unsynchronized, so a caller must hold the monitor while using them.
+ */
+private[sql] final class ModificationCountingMap extends util.Map[String, String] {
+
+ private val underlying = new util.HashMap[String, String]
+
+ private val readOnlyUnderlying = util.Collections.unmodifiableMap[String, String](underlying)
+
+ @volatile private var mutations = 0L
+
+ /**
+ * Number of mutating calls so far; a call may advance it without changing the entries. Values
+ * derived from the entries stay valid while it is unchanged; use
+ * [[snapshotWithModificationCount]] to pair a count with the entries it describes.
+ */
+ def modificationCount: Long = mutations
+
+ /** A copy of the entries and the [[modificationCount]] they correspond to. */
+ def snapshotWithModificationCount: (Long, util.Map[String, String]) = synchronized {
+ (mutations, new util.HashMap[String, String](underlying))
+ }
+
+ private def mutate[T](f: => T): T = synchronized {
Review Comment:
Thanks, verified: the new test blocks every mutator, `snapshotWithModificationCount`, and each read on the map's own monitor, so moving any of them to a private lock or removing the lock would fail it.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4258683512","thread_id":"inline:4258683512","verdict_sha256":"ae877c56c9e26aa21afebfcdd9e33ffa742e923b49b144b08d69c09e8df9705d"} -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60105][SQL] Memoize the retained session-config overlay for view and SQL function body confs [spark]
Posted by "sebastianhillig-db (via GitHub)" <gi...@apache.org>.
sebastianhillig-db commented on code in PR #59320:
URL: https://github.com/apache/spark/pull/59320#discussion_r4270837826
##########
sql/catalyst/benchmarks/RetainedResolutionConfigsBenchmark-results.txt:
##########
@@ -0,0 +1,382 @@
+================================================================================================
+view: 32 settings, 25% retained, warm
+================================================================================================
+
+OpenJDK 64-Bit Server VM 17.0.15+6-Ubuntu-0ubuntu120.04 on Linux 5.4.0-1163-aws-fips
Review Comment:
Regenerated with the "Run benchmarks" workflow for JDK 17, 21, and 25 ([17](https://github.com/sebastianhillig-db/spark/actions/runs/38560201246), [21](https://github.com/sebastianhillig-db/spark/actions/runs/38560212037), [25](https://github.com/sebastianhillig-db/spark/actions/runs/38560222559)); the results are in 8a305bd698d (JDK 21), 6bc31ddc15e (JDK 17), and b6c4ad9d489 (JDK 25).
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:
##########
@@ -326,15 +326,30 @@ object Analyzer {
newConf: SQLConf,
existingConf: SQLConf,
createSparkVersion: String = ""): Unit = {
- existingConf.getAllConfs.foreach { case (key, value) =>
- if (key.startsWith("spark.sql.catalog.") || SQLConf.isSessionBindingPolicy(key)) {
- newConf.settings.put(key, value)
+ // The retained configs are memoized on `existingConf` unless
+ // `ANALYZER_MEMOIZE_RETAINED_RESOLUTION_CONFIGS` is off, which recomputes them on every build.
+ if (existingConf.getConf(SQLConf.ANALYZER_MEMOIZE_RETAINED_RESOLUTION_CONFIGS)) {
+ newConf.settings.putAll(existingConf.retainedResolutionConfigs)
+ } else {
+ existingConf.getAllConfs.foreach { case (key, value) =>
+ if (bodyFollowsCaller(key)) {
+ newConf.settings.put(key, value)
+ }
}
}
trySetAnsiValue(newConf, createSparkVersion)
}
+ /**
+ * True if a view or SQL function body takes the caller's value for `key` rather than the
Review Comment:
Narrowed in 7f92399c393 and 062accf4ee6. The doc now says `retainResolutionConfigsForAnalysis` overlays the caller's explicitly set value, that keys the caller hasn't set keep their saved values, and that confs built without that method (such as the function confs from `SessionCatalog`) don't consult the predicate.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -2893,6 +2935,19 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val ANALYZER_MEMOIZE_RETAINED_RESOLUTION_CONFIGS =
+ buildConf("spark.sql.analyzer.memoizeRetainedResolutionConfigs.enabled")
+ .internal()
+ .version("4.4.0")
+ .withBindingPolicy(ConfigBindingPolicy.SESSION)
Review Comment:
Changed to `NOT_APPLICABLE` in 3876ddb700e. `isSessionBindingPolicy` retains `NOT_APPLICABLE` keys, so nested bodies still inherit an explicitly set value; the on/off equivalence tests are unchanged and pass.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/ModificationCountingMap.scala:
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.internal
+
+import java.util
+import java.util.function.{BiConsumer, BiFunction, Function => JFunction}
+
+/**
+ * A thread-safe map of [[SQLConf]] settings that counts the mutating calls on it, so values
+ * derived from its entries can be reused while [[modificationCount]] is unchanged.
+ *
+ * Like the `java.util.Collections.synchronizedMap` it replaces, every method of the map
+ * synchronizes on the map itself, so a caller iterating the map must hold its monitor. Unlike it,
+ * the key, value, and entry views are read-only, so every mutation goes through a method that
+ * advances the count, and unsynchronized, so a caller must hold the monitor while using them.
+ */
+private[sql] final class ModificationCountingMap extends util.Map[String, String] {
+
+ private val underlying = new util.HashMap[String, String]
+
+ private val readOnlyUnderlying = util.Collections.unmodifiableMap[String, String](underlying)
+
+ @volatile private var mutations = 0L
+
+ /**
+ * Number of mutating calls so far; a call may advance it without changing the entries. Values
+ * derived from the entries stay valid while it is unchanged; use
+ * [[snapshotWithModificationCount]] to pair a count with the entries it describes.
+ */
+ def modificationCount: Long = mutations
+
+ /** A copy of the entries and the [[modificationCount]] they correspond to. */
+ def snapshotWithModificationCount: (Long, util.Map[String, String]) = synchronized {
+ (mutations, new util.HashMap[String, String](underlying))
+ }
+
+ private def mutate[T](f: => T): T = synchronized {
Review Comment:
Added in 088f0fa6c67. The test holds the map's monitor and checks that each mutator, `snapshotWithModificationCount`, and each read blocks until it's released. It fails when `mutate` uses a private lock or no lock, and when `get` is unsynchronized.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60105][SQL] Memoize the retained session-config overlay for view and SQL function body confs [spark]
Posted by "vladimirg-db (via GitHub)" <gi...@apache.org>.
vladimirg-db commented on code in PR #59320:
URL: https://github.com/apache/spark/pull/59320#discussion_r4227804063
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:
##########
@@ -319,18 +319,15 @@ object AnalysisContext {
}
object Analyzer {
- // Configs with bindingPolicy SESSION or NOT_APPLICABLE are retained when resolving views and
- // SQL UDFs, so that their values propagate from the active session rather than falling back to
- // Spark defaults. Entries from lazily-loaded modules are recognized once registered.
+ // Overlays the session-following configs of `existingConf` onto `newConf`, a conf built to
+ // resolve a view or SQL function body, so their values come from the active session rather
+ // than Spark defaults. See `SQLConf.filterRetainedResolutionConfigs` for which configs are
+ // retained; they are memoized on `existingConf`.
def retainResolutionConfigsForAnalysis(
newConf: SQLConf,
existingConf: SQLConf,
createSparkVersion: String = ""): Unit = {
- existingConf.getAllConfs.foreach { case (key, value) =>
- if (key.startsWith("spark.sql.catalog.") || SQLConf.isSessionBindingPolicy(key)) {
- newConf.settings.put(key, value)
- }
- }
+ newConf.settings.putAll(existingConf.retainedResolutionConfigs)
Review Comment:
This is a very core change, can we make sure that we have a flag to flip back to the previous behavior cleanly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60105][SQL] Memoize the retained session-config overlay for view and SQL function body confs [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #59320: [SPARK-60105][SQL] Memoize the retained session-config overlay for view and SQL function body confs
URL: https://github.com/apache/spark/pull/59320
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60105][SQL] Memoize the retained session-config overlay for view and SQL function body confs [spark]
Posted by "sebastianhillig-db (via GitHub)" <gi...@apache.org>.
sebastianhillig-db commented on code in PR #59320:
URL: https://github.com/apache/spark/pull/59320#discussion_r4227811488
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:
##########
@@ -319,18 +319,15 @@ object AnalysisContext {
}
object Analyzer {
- // Configs with bindingPolicy SESSION or NOT_APPLICABLE are retained when resolving views and
- // SQL UDFs, so that their values propagate from the active session rather than falling back to
- // Spark defaults. Entries from lazily-loaded modules are recognized once registered.
+ // Overlays the session-following configs of `existingConf` onto `newConf`, a conf built to
+ // resolve a view or SQL function body, so their values come from the active session rather
+ // than Spark defaults. See `SQLConf.filterRetainedResolutionConfigs` for which configs are
+ // retained; they are memoized on `existingConf`.
def retainResolutionConfigsForAnalysis(
newConf: SQLConf,
existingConf: SQLConf,
createSparkVersion: String = ""): Unit = {
- existingConf.getAllConfs.foreach { case (key, value) =>
- if (key.startsWith("spark.sql.catalog.") || SQLConf.isSessionBindingPolicy(key)) {
- newConf.settings.put(key, value)
- }
- }
+ newConf.settings.putAll(existingConf.retainedResolutionConfigs)
Review Comment:
Yes, fair!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60105][SQL] Memoize the retained session-config overlay for view and SQL function body confs [spark]
Posted by "sebastianhillig-db (via GitHub)" <gi...@apache.org>.
sebastianhillig-db commented on PR #59320:
URL: https://github.com/apache/spark/pull/59320#issuecomment-6171566412
@HyukjinKwon all four comments are addressed; see the reply in each thread. Could you take another look when you have a chance?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60105][SQL] Memoize the retained session-config overlay for view and SQL function body confs [spark]
Posted by "sebastianhillig-db (via GitHub)" <gi...@apache.org>.
sebastianhillig-db commented on code in PR #59320:
URL: https://github.com/apache/spark/pull/59320#discussion_r4228144715
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:
##########
@@ -319,18 +319,15 @@ object AnalysisContext {
}
object Analyzer {
- // Configs with bindingPolicy SESSION or NOT_APPLICABLE are retained when resolving views and
- // SQL UDFs, so that their values propagate from the active session rather than falling back to
- // Spark defaults. Entries from lazily-loaded modules are recognized once registered.
+ // Overlays the session-following configs of `existingConf` onto `newConf`, a conf built to
+ // resolve a view or SQL function body, so their values come from the active session rather
+ // than Spark defaults. See `SQLConf.filterRetainedResolutionConfigs` for which configs are
+ // retained; they are memoized on `existingConf`.
def retainResolutionConfigsForAnalysis(
newConf: SQLConf,
existingConf: SQLConf,
createSparkVersion: String = ""): Unit = {
- existingConf.getAllConfs.foreach { case (key, value) =>
- if (key.startsWith("spark.sql.catalog.") || SQLConf.isSessionBindingPolicy(key)) {
- newConf.settings.put(key, value)
- }
- }
+ newConf.settings.putAll(existingConf.retainedResolutionConfigs)
Review Comment:
Gated this switch with the conf.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60105][SQL] Memoize the retained session-config overlay for view and SQL function body confs [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #59320:
URL: https://github.com/apache/spark/pull/59320#issuecomment-6186408684
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/de2035fcfbf2358eed0ef05feb8e8bd38509922f
- merged into branch-4.x https://github.com/apache/spark/commit/7845ccdf715303d228d6a9f713e5593c8932e1c1
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org