You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by we...@apache.org on 2026/06/11 14:06:20 UTC

(spark) branch master updated: [SPARK-57377][INFRA] Add CI check to prevent new entries in the config binding policy exceptions file

This is an automated email from the ASF dual-hosted git repository.

cloud-fan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 6693d43d11d7 [SPARK-57377][INFRA] Add CI check to prevent new entries in the config binding policy exceptions file
6693d43d11d7 is described below

commit 6693d43d11d7d6fb3fc3aa90bc0601c569e9c6ab
Author: Wenchen Fan <we...@databricks.com>
AuthorDate: Thu Jun 11 07:06:01 2026 -0700

    [SPARK-57377][INFRA] Add CI check to prevent new entries in the config binding policy exceptions file
    
    ### What changes were proposed in this pull request?
    
    This PR adds a new lightweight CI job `binding-policy` ("Config binding policy exceptions check") to `build_and_test.yml`. The job diffs `sql/hive/src/test/resources/conf/binding-policy-exceptions/configs-without-binding-policy-exceptions` against the latest apache/spark base (using the same fork-sync mechanism as the existing `buf` job) and fails if the PR adds any entries to the file, with an error message pointing the author to `.withBindingPolicy()` and the `ConfigBindingPolicy` s [...]
    
    The job runs on a bare runner in well under a minute, is gated on the existing `lint` flag from the `precondition` job, only runs on forks (where PR builds run, so the diff against apache/spark master is exactly the PR's change; on apache/spark itself the job is skipped at the scheduler level and consumes no runner), and skips cleanly on branches where the exceptions file does not exist.
    
    On top of the CI check, this PR also:
    
    - Documents how to choose a binding policy: the `ConfigBindingPolicy` scaladoc now has a 3-step decision flow (does the config change the result of resolving a view/UDF/procedure body? should persisted objects freeze the create-time value, like ANSI mode or session timezone? otherwise SESSION). `SparkConfigBindingPolicySuite` and the CI job's error message point to that scaladoc as the single source of truth, and the suite's failure message now explicitly says not to grow the exceptio [...]
    - Fixes the 12 recently-added configs that were put in the exceptions file instead of declaring a policy, all with `NOT_APPLICABLE` (none of them can change the result of resolving a view/UDF/procedure body): `spark.sql.execution.useHashAggregateExec`, `spark.sql.hive.thriftServer.http.sniHostCheckEnabled`, `spark.sql.streaming.queryEvolution.enableSinkEvolution`, `spark.sql.insertNestedTypeCoercion.enabled`, `spark.testing.injectShuffleFetchFailures`, `spark.ui.jetty.sniHostCheckEnab [...]
    - Relabels the four `spark.sql.window.segmentTree.*` configs from `SESSION` to `NOT_APPLICABLE` per the review discussion (runtime-identical, but `NOT_APPLICABLE` is the accurate label for physical-execution toggles).
    - Fixes the `version()` of `spark.sql.streaming.queryEvolution.enableSinkEvolution` from 4.1.0 to 4.3.0 (it was added after the branch-4.2 cut).
    
    ### Why are the changes needed?
    
    The exceptions file is a frozen list of pre-existing configs that were created before the binding policy was introduced, and it should only ever shrink. `SparkConfigBindingPolicySuite` forces every config to either declare a binding policy or be listed in this file, so PR authors who add a new config without a binding policy see a test failure and "fix" it by adding the config to the exceptions file, which defeats the purpose of the enforcement (see https://github.com/apache/spark/pul [...]
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. `SESSION` and `NOT_APPLICABLE` behave identically at runtime, and declaring either for a config that previously had no policy only affects view/UDF resolution for configs that are actually consulted there, which none of the touched configs are.
    
    ### How was this patch tested?
    
    The CI job was verified end to end on this PR itself: a temporary commit adding an entry to the exceptions file made the job fail with the guidance message in ~10 seconds (https://github.com/cloud-fan/spark/actions/runs/27302316185), and the commits that shrink the file pass the check. The config changes are covered by the existing `SparkConfigBindingPolicySuite`, which verifies that every config either declares a policy or is listed in the exceptions file, and that no config does both.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code
    
    This pull request and its description were written by Isaac.
    
    Closes #56437 from cloud-fan/binding-policy-ci-check.
    
    Authored-by: Wenchen Fan <we...@databricks.com>
    Signed-off-by: Wenchen Fan <we...@databricks.com>
---
 .github/workflows/build_and_test.yml               | 49 ++++++++++++++++++++++
 .../internal/config/ConfigBindingPolicy.scala      | 16 +++++++
 .../org/apache/spark/internal/config/History.scala |  1 +
 .../org/apache/spark/internal/config/Tests.scala   |  1 +
 .../org/apache/spark/internal/config/UI.scala      |  1 +
 .../org/apache/spark/internal/config/package.scala |  5 +++
 .../org/apache/spark/sql/internal/SQLConf.scala    | 13 +++---
 .../apache/spark/sql/internal/StaticSQLConf.scala  |  2 +
 .../configs-without-binding-policy-exceptions      | 12 ------
 .../SparkConfigBindingPolicySuite.scala            | 15 ++++++-
 10 files changed, 96 insertions(+), 19 deletions(-)

diff --git a/.github/workflows/build_and_test.yml b/.github/workflows/build_and_test.yml
index dcd91ac093dd..b7e3f23cd52a 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -1038,6 +1038,55 @@ jobs:
       if: inputs.branch != 'branch-3.5'
       run: ./dev/check-protos.py
 
+  # The config binding policy exceptions file lists pre-existing configs that were created before
+  # the binding policy was introduced. It must never grow: new configs must declare a binding
+  # policy via .withBindingPolicy() instead of being added to the exceptions file. This job fails
+  # if a PR adds entries to the file. It only runs on forks because PR builds run there, and the
+  # diff against apache/spark master is exactly the PR's change.
+  binding-policy:
+    needs: [precondition]
+    if: >-
+      (!cancelled()) &&
+      github.repository != 'apache/spark' &&
+      fromJson(needs.precondition.outputs.required).lint == 'true'
+    name: Config binding policy exceptions check
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    steps:
+    - name: Checkout Spark repository
+      uses: actions/checkout@v6
+      with:
+        fetch-depth: 0
+        repository: apache/spark
+        ref: ${{ needs.precondition.outputs.head_sha }}
+    - name: Sync the current branch with the latest in Apache Spark
+      run: |
+        echo "APACHE_SPARK_REF=$(git rev-parse HEAD)" >> $GITHUB_ENV
+        git fetch https://github.com/$GITHUB_REPOSITORY.git ${GITHUB_REF#refs/heads/}
+        git -c user.name='Apache Spark Test Account' -c user.email='sparktestacc@gmail.com' merge --no-commit --progress --squash FETCH_HEAD
+        git -c user.name='Apache Spark Test Account' -c user.email='sparktestacc@gmail.com' commit -m "Merged commit" --allow-empty
+    - name: Check no new entries in the binding policy exceptions file
+      run: |
+        EXCEPTIONS_FILE="sql/hive/src/test/resources/conf/binding-policy-exceptions/configs-without-binding-policy-exceptions"
+        if [ ! -f "$EXCEPTIONS_FILE" ]; then
+          echo "$EXCEPTIONS_FILE does not exist on this branch, skipping."
+          exit 0
+        fi
+        ADDED_ENTRIES=$(git diff "$APACHE_SPARK_REF" HEAD -- "$EXCEPTIONS_FILE" | grep '^+' | grep -v '^+++' || true)
+        if [ -n "$ADDED_ENTRIES" ]; then
+          echo "ERROR: This PR adds the following entries to $EXCEPTIONS_FILE:"
+          echo "$ADDED_ENTRIES"
+          echo ""
+          echo "This file is a frozen list of configs that predate the binding policy, and new"
+          echo "entries must not be added to it. Instead, declare a binding policy when building"
+          echo "the config entry, e.g."
+          echo "  .withBindingPolicy(ConfigBindingPolicy.SESSION/PERSISTED/NOT_APPLICABLE)"
+          echo "See the scaladoc of org.apache.spark.internal.config.ConfigBindingPolicy for the"
+          echo "semantics of each policy and how to choose one."
+          exit 1
+        fi
+        echo "No new entries added to the binding policy exceptions file."
+
   # Static analysis
   lint:
     needs: [precondition, infra-image]
diff --git a/common/utils/src/main/scala/org/apache/spark/internal/config/ConfigBindingPolicy.scala b/common/utils/src/main/scala/org/apache/spark/internal/config/ConfigBindingPolicy.scala
index 65009159f609..4369aa71b03a 100644
--- a/common/utils/src/main/scala/org/apache/spark/internal/config/ConfigBindingPolicy.scala
+++ b/common/utils/src/main/scala/org/apache/spark/internal/config/ConfigBindingPolicy.scala
@@ -29,6 +29,22 @@ package org.apache.spark.internal.config
  * is PERSISTED, session-level config changes will not apply to views/UDFs/procedures but only
  * to outer queries. In order for session-level changes to propagate correctly, this value must
  * be explicitly set to SESSION.
+ *
+ * How to choose a policy for a new config:
+ *
+ *  1. Can the config change the result of resolving the body of a view/UDF/procedure, i.e. the
+ *     resolved plan? If not, use NOT_APPLICABLE. This covers most configs. Note that the test
+ *     is whether the config changes the resolution result, not whether it is read during
+ *     resolution: view analysis can itself trigger query execution (e.g. a schema inference
+ *     job), so even physical planning or runtime configs may be read while resolving a view,
+ *     but they do not change what the body resolves to.
+ *  2. If it can, should the persisted object use the create-time value of the config,
+ *     so that it keeps computing the same result as it did at creation no matter who calls it
+ *     later (e.g. ANSI mode, session timezone)? If so, use PERSISTED. Note that changing query
+ *     behavior alone does not justify PERSISTED: a bug-fix flag also changes query behavior,
+ *     but views should not freeze the buggy behavior.
+ *  3. Otherwise, use SESSION: the caller's session value applies, keeping behavior uniform
+ *     across the entire query.
  */
 object ConfigBindingPolicy extends Enumeration {
   type ConfigBindingPolicy = Value
diff --git a/core/src/main/scala/org/apache/spark/internal/config/History.scala b/core/src/main/scala/org/apache/spark/internal/config/History.scala
index 90abc9d038db..1936ecf68103 100644
--- a/core/src/main/scala/org/apache/spark/internal/config/History.scala
+++ b/core/src/main/scala/org/apache/spark/internal/config/History.scala
@@ -76,6 +76,7 @@ private[spark] object History {
         "(e.g., S3 Lifecycle Policies) for those. " +
         "Example: \"s3a://.*,gs://.*\" disables scanning for all S3 and GCS directories.")
       .version("4.2.0")
+      .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
       .stringConf
       .toSequence
       .createWithDefault(Nil)
diff --git a/core/src/main/scala/org/apache/spark/internal/config/Tests.scala b/core/src/main/scala/org/apache/spark/internal/config/Tests.scala
index 8ecb14be1dfb..ed88fbe71ffc 100644
--- a/core/src/main/scala/org/apache/spark/internal/config/Tests.scala
+++ b/core/src/main/scala/org/apache/spark/internal/config/Tests.scala
@@ -43,6 +43,7 @@ private[spark] object Tests {
     ConfigBuilder("spark.testing.injectShuffleFetchFailures")
       .doc("Injecting fetch failures for shuffle stages by providing an invalid BlockManager " +
         "location for the first stage attempt. Testing only flag!")
+      .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
       .booleanConf
       .createWithDefault(false)
 
diff --git a/core/src/main/scala/org/apache/spark/internal/config/UI.scala b/core/src/main/scala/org/apache/spark/internal/config/UI.scala
index 20c99794cd21..3fae4c6d2f78 100644
--- a/core/src/main/scala/org/apache/spark/internal/config/UI.scala
+++ b/core/src/main/scala/org/apache/spark/internal/config/UI.scala
@@ -149,6 +149,7 @@ private[spark] object UI {
       "backward compatibility with standalone deployments. Set to true to enforce " +
       "SNI host checking for stricter security.")
     .version("4.2.0")
+    .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
     .booleanConf
     .createWithDefault(false)
 
diff --git a/core/src/main/scala/org/apache/spark/internal/config/package.scala b/core/src/main/scala/org/apache/spark/internal/config/package.scala
index 86e5422a8551..141e033ea992 100644
--- a/core/src/main/scala/org/apache/spark/internal/config/package.scala
+++ b/core/src/main/scala/org/apache/spark/internal/config/package.scala
@@ -2393,6 +2393,7 @@ package object config {
       .doc("When true, scheduler log messages for streaming tasks include " +
         "the structured streaming query ID and batch ID.")
       .version("4.2.0")
+      .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
       .booleanConf
       .createWithDefault(true)
 
@@ -2401,6 +2402,7 @@ package object config {
       .doc("Maximum number of characters of the streaming query ID to include " +
         "in scheduler log messages. Set to -1 to include the full query ID.")
       .version("4.2.0")
+      .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
       .intConf
       .createWithDefault(5)
 
@@ -2971,6 +2973,7 @@ package object config {
         "Application Master JVM options in client mode. In cluster mode, use " +
         "`spark.driver.limitActiveProcessorCount.enabled` instead.")
       .version("4.2.0")
+      .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
       .booleanConf
       .createWithDefault(false)
 
@@ -2979,6 +2982,7 @@ package object config {
       .doc("Whether to add -XX:ActiveProcessorCount=<spark.driver.cores> to the driver JVM " +
         "options. Currently, this only takes effect in YARN cluster mode.")
       .version("4.2.0")
+      .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
       .booleanConf
       .createWithDefault(false)
 
@@ -2987,6 +2991,7 @@ package object config {
       .doc("Whether to add -XX:ActiveProcessorCount=<spark.executor.cores> to executor JVM " +
         "options. Currently, this only takes effect in YARN mode.")
       .version("4.2.0")
+      .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
       .booleanConf
       .createWithDefault(false)
 }
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index 99b21f3b79b4..80a55cf5802d 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -3260,7 +3260,8 @@ object SQLConf {
       .doc("When true, streaming sinks can be named using the name() API on DataStreamWriter. " +
         "This enables sink evolution capability where sinks can be changed while maintaining " +
         "a historical record of all sinks used in the checkpoint.")
-      .version("4.1.0")
+      .version("4.3.0")
+      .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
       .booleanConf
       .createWithDefault(false)
 
@@ -3823,6 +3824,7 @@ object SQLConf {
     .doc("Decides if we use HashAggregateExec when possible, the rule takes precedence " +
       s"over ${USE_OBJECT_HASH_AGG.key}.")
     .version("4.3.0")
+    .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
     .booleanConf
     .createWithDefault(true)
 
@@ -4239,7 +4241,7 @@ object SQLConf {
 
   val WINDOW_SEGMENT_TREE_ENABLED =
     buildConf("spark.sql.window.segmentTree.enabled")
-      .withBindingPolicy(ConfigBindingPolicy.SESSION)
+      .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
       .doc("Use block-chunked segment tree for moving aggregate window frames " +
         "whose functions are all DeclarativeAggregate without FILTER/DISTINCT.")
       .version("4.2.0")
@@ -4248,7 +4250,7 @@ object SQLConf {
 
   val WINDOW_SEGMENT_TREE_MIN_PARTITION_ROWS =
     buildConf("spark.sql.window.segmentTree.minPartitionRows")
-      .withBindingPolicy(ConfigBindingPolicy.SESSION)
+      .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
       .doc("Minimum partition row count to activate the segment-tree moving frame. " +
         "Partitions smaller than this fall back to the default sliding implementation.")
       .version("4.2.0")
@@ -4259,7 +4261,7 @@ object SQLConf {
 
   val WINDOW_SEGMENT_TREE_BLOCK_SIZE =
     buildConf("spark.sql.window.segmentTree.blockSize")
-      .withBindingPolicy(ConfigBindingPolicy.SESSION)
+      .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
       .doc("Block size, in rows, for the block-chunked segment tree used by moving " +
         "window frames. Each leaf of the tree aggregates this many consecutive rows. " +
         "Smaller values reduce per-partition memory and speed up tree build for small " +
@@ -4274,7 +4276,7 @@ object SQLConf {
 
   val WINDOW_SEGMENT_TREE_FANOUT =
     buildConf("spark.sql.window.segmentTree.fanout")
-      .withBindingPolicy(ConfigBindingPolicy.SESSION)
+      .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
       .doc("Fanout of internal nodes for the block-chunked segment tree.")
       .version("4.2.0")
       .internal()
@@ -7446,6 +7448,7 @@ object SQLConf {
         "top-level columns are filled with their default value (or null). This is " +
         "experimental and the semantics may change.")
       .version("4.2.0")
+      .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
       .booleanConf
       .createWithDefault(false)
 
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala
index 25114a93f04c..ad9106ee3e06 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala
@@ -20,6 +20,7 @@ package org.apache.spark.sql.internal
 import java.util.Locale
 import java.util.concurrent.TimeUnit
 
+import org.apache.spark.internal.config.ConfigBindingPolicy
 import org.apache.spark.sql.connector.catalog.CatalogManager.SESSION_CATALOG_NAME
 import org.apache.spark.util.Utils
 
@@ -128,6 +129,7 @@ object StaticSQLConf {
       .doc("Whether to enable Jetty's SNI host check on the ThriftHttpCLIService HTTPS " +
         "connector. Set to false to restore the behavior prior to SPARK-45522 (Jetty 10+).")
       .version("4.3.0")
+      .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
       .booleanConf
       .createWithDefault(true)
 
diff --git a/sql/hive/src/test/resources/conf/binding-policy-exceptions/configs-without-binding-policy-exceptions b/sql/hive/src/test/resources/conf/binding-policy-exceptions/configs-without-binding-policy-exceptions
index 26f29ba24d10..a9d53bdb1036 100644
--- a/sql/hive/src/test/resources/conf/binding-policy-exceptions/configs-without-binding-policy-exceptions
+++ b/sql/hive/src/test/resources/conf/binding-policy-exceptions/configs-without-binding-policy-exceptions
@@ -43,7 +43,6 @@ spark.driver.extraClassPath
 spark.driver.extraJavaOptions
 spark.driver.extraLibraryPath
 spark.driver.host
-spark.driver.limitActiveProcessorCount.enabled
 spark.driver.log.allowErasureCoding
 spark.driver.log.dfsDir
 spark.driver.log.layout
@@ -120,7 +119,6 @@ spark.executor.id
 spark.executor.instances
 spark.executor.isolatedSessionCache.size
 spark.executor.killOnFatalError.depth
-spark.executor.limitActiveProcessorCount.enabled
 spark.executor.logs.redirectConsoleOutputs
 spark.executor.logs.rolling.enableCompression
 spark.executor.logs.rolling.maxRetainedFiles
@@ -169,7 +167,6 @@ spark.history.fs.numReplayThreads
 spark.history.fs.safemodeCheck.interval
 spark.history.fs.update.batchSize
 spark.history.fs.update.interval
-spark.history.fs.update.scanDisabledPathPatterns
 spark.history.kerberos.enabled
 spark.history.kerberos.keytab
 spark.history.kerberos.principal
@@ -319,8 +316,6 @@ spark.scheduler.numCancelledJobGroupsToTrack
 spark.scheduler.resource.profileMergeConflicts
 spark.scheduler.revive.interval
 spark.scheduler.stage.legacyAbortAfterKillTasks
-spark.scheduler.streaming.idAwareLogging.enabled
-spark.scheduler.streaming.idAwareLogging.queryIdLength
 spark.security.credentials.renewalRatio
 spark.security.credentials.retryWait
 spark.serializer
@@ -603,7 +598,6 @@ spark.sql.execution.replaceHashWithSortAgg
 spark.sql.execution.reuseSubquery
 spark.sql.execution.sortBeforeRepartition
 spark.sql.execution.topKSortFallbackThreshold
-spark.sql.execution.useHashAggregateExec
 spark.sql.execution.useObjectHashAggregateExec
 spark.sql.execution.usePartitionEvaluator
 spark.sql.extendedExplainProviders
@@ -649,7 +643,6 @@ spark.sql.hive.metastorePartitionPruningFastFallback
 spark.sql.hive.metastorePartitionPruningInSetThreshold
 spark.sql.hive.tablePropertyLengthThreshold
 spark.sql.hive.thriftServer.async
-spark.sql.hive.thriftServer.http.sniHostCheckEnabled
 spark.sql.hive.thriftServer.singleSession
 spark.sql.hive.useDelegateForSymlinkTextInputFormat
 spark.sql.hive.version
@@ -661,7 +654,6 @@ spark.sql.inMemoryColumnarStorage.hugeVectorReserveRatio
 spark.sql.inMemoryColumnarStorage.hugeVectorThreshold
 spark.sql.inMemoryColumnarStorage.partitionPruning
 spark.sql.inMemoryTableScanStatistics.enable
-spark.sql.insertNestedTypeCoercion.enabled
 spark.sql.join.preferSortMergeJoin
 spark.sql.json.enableExactStringParsing
 spark.sql.json.enablePartialResults
@@ -1038,7 +1030,6 @@ spark.sql.streaming.numRecentProgressUpdates
 spark.sql.streaming.offsetLog.formatVersion
 spark.sql.streaming.optimizeOneRowPlan.enabled
 spark.sql.streaming.pollingDelay
-spark.sql.streaming.queryEvolution.enableSinkEvolution
 spark.sql.streaming.queryEvolution.enableSourceEvolution
 spark.sql.streaming.ratioExtraSpaceAllowedInCheckpoint
 spark.sql.streaming.realTimeMode.allowlistCheck
@@ -1177,7 +1168,6 @@ spark.taskMetrics.trackUpdatedBlockStatuses
 spark.test.noStageRetry
 spark.testing
 spark.testing.dynamicAllocation.schedule.enabled
-spark.testing.injectShuffleFetchFailures
 spark.testing.memory
 spark.testing.nCoresPerExecutor
 spark.testing.nExecutorsPerHost
@@ -1194,7 +1184,6 @@ spark.ui.enabled
 spark.ui.filters
 spark.ui.groupSQLSubExecutionEnabled
 spark.ui.heapHistogramEnabled
-spark.ui.jetty.sniHostCheckEnabled
 spark.ui.jetty.stopTimeout
 spark.ui.killEnabled
 spark.ui.liveUpdate.minFlushPeriod
@@ -1229,7 +1218,6 @@ spark.unsafe.sorter.spill.read.ahead.enabled
 spark.unsafe.sorter.spill.reader.buffer.size
 spark.user.groups.mapping
 spark.worker.executorStateSync.maxAttempts
-spark.yarn.am.limitActiveProcessorCount.enabled
 spark.yarn.dist.forceDownloadSchemes
 spark.yarn.dist.pyFiles
 spark.yarn.isPython
diff --git a/sql/hive/src/test/scala/org/apache/spark/sql/configaudit/SparkConfigBindingPolicySuite.scala b/sql/hive/src/test/scala/org/apache/spark/sql/configaudit/SparkConfigBindingPolicySuite.scala
index e7b4669e2b96..78c3184d351e 100644
--- a/sql/hive/src/test/scala/org/apache/spark/sql/configaudit/SparkConfigBindingPolicySuite.scala
+++ b/sql/hive/src/test/scala/org/apache/spark/sql/configaudit/SparkConfigBindingPolicySuite.scala
@@ -25,6 +25,14 @@ import org.apache.spark.internal.config.{ConfigBindingPolicy, ConfigEntry}
 import org.apache.spark.sql.hive.HiveUtils
 import org.apache.spark.sql.internal.SQLConf
 
+/**
+ * Enforces that every Spark config declares a `ConfigBindingPolicy`. The exceptions file
+ * `conf/binding-policy-exceptions/configs-without-binding-policy-exceptions` is a frozen list
+ * of configs that predate the binding policy and must only ever shrink: new configs must
+ * declare a policy via `.withBindingPolicy()` when building the config entry (see the
+ * [[ConfigBindingPolicy]] scaladoc for how to choose a policy). The `binding-policy` CI job
+ * rejects any PR that adds entries to the exceptions file.
+ */
 class SparkConfigBindingPolicySuite extends SparkFunSuite {
 
   override def beforeAll(): Unit = {
@@ -57,8 +65,11 @@ class SparkConfigBindingPolicySuite extends SparkFunSuite {
     if (missingBindingPolicyConfigs.nonEmpty) {
       fail(
         s"The following configs do not have bindingPolicy field set. You need to define it " +
-        "by using .withBindingPolicy(ConfigBindingPolicy.SESSION/PERSISTED) when you build " +
-        "the config entry.\n" +
+        "by using .withBindingPolicy(ConfigBindingPolicy.SESSION/PERSISTED/NOT_APPLICABLE) " +
+        "when you build the config entry. See the ConfigBindingPolicy scaladoc for how to " +
+        "choose a policy. DO NOT add new entries to the exceptions file: it is a frozen " +
+        "list of configs that predate the binding policy and must only ever shrink (the " +
+        "binding-policy CI job rejects any addition).\n" +
         missingBindingPolicyConfigs.mkString("\n")
       )
     }


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org