You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2026/07/08 00:03:51 UTC
(spark-kubernetes-operator) branch main updated: [SPARK-57543] Make ConfigOption dynamic override opt-in and filter dynamic config refresh
This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new 1127bf9 [SPARK-57543] Make ConfigOption dynamic override opt-in and filter dynamic config refresh
1127bf9 is described below
commit 1127bf9fee08b55251401b86c847e6646b520c98
Author: Zhou JIANG <ji...@umich.edu>
AuthorDate: Tue Jul 7 17:03:44 2026 -0700
[SPARK-57543] Make ConfigOption dynamic override opt-in and filter dynamic config refresh
### What changes were proposed in this pull request?
This PR makes runtime dynamic config override opt-in and enforces it centrally when dynamic config is refreshed:
1. `ConfigOption`
- `enableDynamicOverride` now defaults to `false` — an option must explicitly opt in via `.enableDynamicOverride(true)` to be overridable at runtime.
- Each option registers itself (keyed by its config key) into a static registry on construction, exposing `dynamicOverrideEnabledKeys()` as the allow-list of overridable keys. `enableDynamicOverride` on each option remains the single source of truth.
2. `SparkOperatorConfManager.refresh(Map)` now filters the incoming ConfigMap data against `ConfigOption.dynamicOverrideEnabledKeys()`, applying only overridable keys and logging any dropped (unknown or non-overridable) keys at WARN. `getValue(String)` and `getAll()` therefore no longer surface override values for keys that are never honored.
3. `SparkOperatorConf` options now set `enableDynamicOverride`explicitly; this also fixes docs for `LOG_CONF` and `PERIODIC_GC_INTERVAL_SECONDS`, which are only read at startup and are non-overridable.
The existing per-option check in `ConfigOption.resolveValue()` is kept as defense-in-depth.
### Why are the changes needed?
Previously `refresh()` stored the entire ConfigMap into the override map, and `enableDynamicOverride` defaulted to `true`. As a result, an override value could be stored (and surfaced via `getValue`/`getAll`/the `LOG_CONF` dump) for keys that are not meant to be hot-reloaded, and any new option was overridable by default. Defaulting to opt-in and dropping non-allow-listed keys at refresh time makes the override surface explicit, safer, and self-documenting, and avoids silently ignored [...]
### Does this PR introduce _any_ user-facing change?
Yes. Dynamic config updates for keys that are not explicitly marked `enableDynamicOverride(true)` are now ignored (and logged at WARN) instead of being stored. In particular `spark.logConf` and `spark.kubernetes.operator.periodicGC.intervalSeconds` are no longer dynamically overridable (both are only read at startup). The generated config documentation's "Allow Hot Reloading" column reflects each option's flag.
### How was this patch tested?
Updated `ConfigOptionTest` and `SparkOperatorConfManagerTest`, and added `SparkOperatorConfManagerTest#testRefreshDropsUnknownAndNonOverridableKeys` covering the allow-list filtering. Existing `SparkOperatorConfigMapReconcilerTest` continues to exercise the dynamic-config refresh path with an overridable key.
### Was this patch authored or co-authored using generative AI tooling?
Co-Authored-By: Claude Opus 4.8
Closes #713 from jiangzho/conf_wl.
Authored-by: Zhou JIANG <ji...@umich.edu>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
docs/config_properties.md | 6 +-
.../spark/k8s/operator/config/ConfigOption.java | 70 ++++++++++++++++++----
.../k8s/operator/config/SparkOperatorConf.java | 20 +++++++
.../operator/config/SparkOperatorConfManager.java | 31 +++++++++-
.../spark/k8s/operator/SparkOperatorTest.java | 8 +++
.../k8s/operator/config/ConfigOptionTest.java | 12 +++-
.../config/SparkOperatorConfManagerTest.java | 39 +++++++++++-
.../apache/spark/k8s/operator/utils/UtilsTest.java | 8 ++-
8 files changed, 174 insertions(+), 20 deletions(-)
diff --git a/docs/config_properties.md b/docs/config_properties.md
index 9c14468..1a9ff64 100644
--- a/docs/config_properties.md
+++ b/docs/config_properties.md
@@ -35,7 +35,7 @@
| spark.kubernetes.operator.metrics.sanitizePrometheusMetricsNameEnabled | Boolean | true | false | Whether or not to enable automatic name sanitizing for all metrics based on best-practice guide from Prometheus https://prometheus.io/docs/practices/naming/ |
| spark.kubernetes.operator.name | String | spark-kubernetes-operator | false | Name of the operator. |
| spark.kubernetes.operator.namespace | String | default | false | Namespace that operator is deployed within. |
- | spark.kubernetes.operator.periodicGC.intervalSeconds | Long | 1800 | true | Interval (in seconds) between periodic System.gc() invocations. Set to 0 or a negative value to disable. Note that System.gc() is a no-op if JVM is started with -XX:+DisableExplicitGC. |
+ | spark.kubernetes.operator.periodicGC.intervalSeconds | Long | 1800 | false | Interval (in seconds) between periodic System.gc() invocations. Set to 0 or a negative value to disable. Note that System.gc() is a no-op if JVM is started with -XX:+DisableExplicitGC. |
| spark.kubernetes.operator.reconciler.appStatusListenerClassNames | String | | false | Comma-separated names of SparkAppStatusListener class implementations |
| spark.kubernetes.operator.reconciler.clusterStatusListenerClassNames | String | | false | Comma-separated names of SparkClusterStatusListener class implementations |
| spark.kubernetes.operator.reconciler.foregroundRequestTimeoutSeconds | Long | 60 | true | Timeout (in seconds) for requests made to API server. This applies only to foreground requests. |
@@ -50,6 +50,6 @@
| spark.kubernetes.operator.reconciler.terminationTimeoutSeconds | Integer | 30 | false | Grace period for operator shutdown before reconciliation threads are killed. |
| spark.kubernetes.operator.reconciler.trimStateTransitionHistoryEnabled | Boolean | true | true | When enabled, operator would trim state transition history when a new attempt starts, keeping previous attempt summary only. |
| spark.kubernetes.operator.watchedNamespaces | String | default | true | Comma-separated list of namespaces that the operator would be watching for Spark resources. If set to '*', operator would watch all namespaces. |
- | spark.logConf | Boolean | false | true | When enabled, operator will print configurations |
- | spark.redaction.regex | String | (?i)secret\|password\|token\|access[.]?key | true | Regex to decide which parts of configuration properties contain sensitive information, whose values would be redacted in operator logs. This reuses Spark's redaction configuration key and default pattern. |
+ | spark.logConf | Boolean | false | false | When enabled, operator will print configurations |
+ | spark.redaction.regex | String | (?i)secret\|password\|token\|access[.]?key | false | Regex to decide which parts of configuration properties contain sensitive information, whose values would be redacted in operator logs. This reuses Spark's redaction configuration key and default pattern. |
diff --git a/spark-operator/src/main/java/org/apache/spark/k8s/operator/config/ConfigOption.java b/spark-operator/src/main/java/org/apache/spark/k8s/operator/config/ConfigOption.java
index 1130866..d9c4aae 100644
--- a/spark-operator/src/main/java/org/apache/spark/k8s/operator/config/ConfigOption.java
+++ b/spark-operator/src/main/java/org/apache/spark/k8s/operator/config/ConfigOption.java
@@ -19,12 +19,15 @@
package org.apache.spark.k8s.operator.config;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+
import com.fasterxml.jackson.core.JsonProcessingException;
-import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.EqualsAndHashCode;
import lombok.Getter;
-import lombok.RequiredArgsConstructor;
import lombok.ToString;
import lombok.extern.slf4j.Slf4j;
@@ -34,20 +37,67 @@ import org.apache.spark.k8s.operator.utils.StringUtils;
/**
* Config options for Spark Operator. Supports primitive and serialized JSON.
*
+ * <p>Every option registers itself, keyed by {@link #key}, into a static registry on construction.
+ * {@link SparkOperatorConfManager#refresh(Map)} consults {@link #dynamicOverrideEnabledKeys()} to
+ * drop any dynamic-config update that targets an unknown key or one whose {@link
+ * #enableDynamicOverride} is {@code false}.
+ *
* @param <T> The type of the config option's value.
*/
-@RequiredArgsConstructor
-@AllArgsConstructor
@EqualsAndHashCode
@ToString
-@Builder
@Slf4j
public class ConfigOption<T> {
- @Getter @Builder.Default private final boolean enableDynamicOverride = true;
- @Getter private String key;
- @Getter private String description;
- @Getter private T defaultValue;
- @Getter private Class<T> typeParameterClass;
+ /** Indexes every declared option so dynamic-config refresh can check overridability. */
+ private static final Map<String, ConfigOption<?>> REGISTRY = new ConcurrentHashMap<>();
+
+ /**
+ * Whether this option may be overridden at runtime via dynamic config. Defaults to {@code false}
+ * (opt-in): dynamic override must be explicitly enabled through the builder.
+ */
+ @Getter private final boolean enableDynamicOverride;
+ @Getter private final String key;
+ @Getter private final String description;
+ @Getter private final T defaultValue;
+ @Getter private final Class<T> typeParameterClass;
+
+ @Builder
+ ConfigOption(
+ boolean enableDynamicOverride,
+ String key,
+ String description,
+ T defaultValue,
+ Class<T> typeParameterClass) {
+ this.enableDynamicOverride = enableDynamicOverride;
+ this.key = key;
+ this.description = description;
+ this.defaultValue = defaultValue;
+ this.typeParameterClass = typeParameterClass;
+ if (StringUtils.isNotEmpty(key)) {
+ REGISTRY.put(key, this);
+ }
+ }
+
+ /**
+ * Returns the keys of all declared options that permit runtime dynamic override. Used as the
+ * allow-list when refreshing dynamic config.
+ *
+ * <p><b>Note:</b> the returned set only includes options that have already been constructed.
+ * Options are {@code static final} fields (e.g. on {@link SparkOperatorConf}) that register
+ * themselves lazily on class initialization, so a caller that reads this before those classes
+ * are initialized would see an empty or partial set. {@link
+ * SparkOperatorConfManager#refresh(Map)} forces initialization via {@link
+ * SparkOperatorConf#ensureOptionsRegistered()} first to avoid that; other callers must ensure
+ * the relevant option classes are initialized.
+ *
+ * @return An immutable set of dynamically overridable config keys.
+ */
+ public static Set<String> dynamicOverrideEnabledKeys() {
+ return REGISTRY.values().stream()
+ .filter(ConfigOption::isEnableDynamicOverride)
+ .map(ConfigOption::getKey)
+ .collect(Collectors.toUnmodifiableSet());
+ }
/**
* Returns the resolved value of the config option.
diff --git a/spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConf.java b/spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConf.java
index 03ab285..a1a4c10 100644
--- a/spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConf.java
+++ b/spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConf.java
@@ -35,6 +35,7 @@ public final class SparkOperatorConf {
public static final ConfigOption<Boolean> LOG_CONF =
ConfigOption.<Boolean>builder()
.key("spark.logConf")
+ .enableDynamicOverride(false)
.description("When enabled, operator will print configurations")
.typeParameterClass(Boolean.class)
.defaultValue(false)
@@ -48,6 +49,9 @@ public final class SparkOperatorConf {
public static final ConfigOption<String> REDACTION_REGEX =
ConfigOption.<String>builder()
.key("spark.redaction.regex")
+ // Intentionally not dynamically overridable: a runtime override of the redaction
+ // pattern could weaken or disable log redaction, so this is only read at startup.
+ .enableDynamicOverride(false)
.description(
"Regex to decide which parts of configuration properties contain sensitive "
+ "information, whose values would be redacted in operator logs. This reuses "
@@ -63,6 +67,7 @@ public final class SparkOperatorConf {
public static final ConfigOption<Long> PERIODIC_GC_INTERVAL_SECONDS =
ConfigOption.<Long>builder()
.key("spark.kubernetes.operator.periodicGC.intervalSeconds")
+ .enableDynamicOverride(false)
.description(
"Interval (in seconds) between periodic System.gc() invocations. "
+ "Set to 0 or a negative value to disable. Note that System.gc() is a no-op "
@@ -868,4 +873,19 @@ public final class SparkOperatorConf {
}
return value;
}
+
+ /**
+ * Forces static initialization of this class so that every {@link ConfigOption} declared here
+ * has registered itself with {@link ConfigOption}'s registry.
+ *
+ * <p>{@link ConfigOption#dynamicOverrideEnabledKeys()} only reflects options that have already
+ * been constructed, and the options are {@code static final} fields that register lazily on
+ * class init. Invoking this method (an active use of the class) guarantees the allow-list is
+ * complete before dynamic config is refreshed, independent of operator boot ordering — see
+ * {@link SparkOperatorConfManager#refresh(java.util.Map)}. This method is intentionally a no-op;
+ * the side effect is triggering class initialization.
+ */
+ public static void ensureOptionsRegistered() {
+ // No-op: the mere invocation triggers this class's static initializer.
+ }
}
diff --git a/spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConfManager.java b/spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConfManager.java
index a0f4bfe..15f8118 100644
--- a/spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConfManager.java
+++ b/spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConfManager.java
@@ -23,8 +23,11 @@ import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.List;
import java.util.Map;
import java.util.Properties;
+import java.util.Set;
import lombok.extern.slf4j.Slf4j;
@@ -99,14 +102,36 @@ public class SparkOperatorConfManager {
}
/**
- * Refreshes the configuration overrides with new values from a map.
+ * Refreshes the configuration overrides with new values from a map. Only keys that are declared
+ * as dynamically overridable options (see {@link ConfigOption#dynamicOverrideEnabledKeys()}) are
+ * applied; unknown keys and keys whose option disables dynamic override are dropped and logged so
+ * an ineffective update is not silently ignored.
*
* @param updatedConfig A map containing the updated configuration properties.
*/
public void refresh(Map<String, String> updatedConfig) {
+ // Ensure every ConfigOption has registered before we read the allow-list. The options are
+ // static final fields on SparkOperatorConf and register lazily on class init, so without this
+ // the allow-list could be empty or partial if refresh() runs before SparkOperatorConf is
+ // touched, silently dropping every override. Forcing init here makes correctness independent
+ // of boot ordering rather than relying on an earlier incidental read of SparkOperatorConf.
+ SparkOperatorConf.ensureOptionsRegistered();
+ Set<String> allowedKeys = ConfigOption.dynamicOverrideEnabledKeys();
+ Properties filtered = new Properties();
+ List<String> droppedKeys = new ArrayList<>();
+ for (Map.Entry<String, String> entry : updatedConfig.entrySet()) {
+ if (allowedKeys.contains(entry.getKey())) {
+ filtered.put(entry.getKey(), entry.getValue());
+ } else {
+ droppedKeys.add(entry.getKey());
+ }
+ }
+ if (!droppedKeys.isEmpty()) {
+ log.warn(
+ "Ignoring dynamic config override for unknown or non-overridable keys: {}", droppedKeys);
+ }
synchronized (this) {
- this.configOverrides = new Properties();
- configOverrides.putAll(updatedConfig);
+ this.configOverrides = filtered;
}
}
diff --git a/spark-operator/src/test/java/org/apache/spark/k8s/operator/SparkOperatorTest.java b/spark-operator/src/test/java/org/apache/spark/k8s/operator/SparkOperatorTest.java
index 7c55c33..7e97d92 100644
--- a/spark-operator/src/test/java/org/apache/spark/k8s/operator/SparkOperatorTest.java
+++ b/spark-operator/src/test/java/org/apache/spark/k8s/operator/SparkOperatorTest.java
@@ -53,6 +53,7 @@ import org.mockito.MockedConstruction;
import org.mockito.MockedStatic;
import org.apache.spark.k8s.operator.client.KubernetesClientFactory;
+import org.apache.spark.k8s.operator.config.ConfigOption;
import org.apache.spark.k8s.operator.config.DynamicConfigMonitor;
import org.apache.spark.k8s.operator.config.SparkOperatorConf;
import org.apache.spark.k8s.operator.config.SparkOperatorConfManager;
@@ -285,6 +286,13 @@ class SparkOperatorTest {
MockedConstruction<KubernetesMetricsInterceptor> interceptorMockedConstruction =
mockConstruction(KubernetesMetricsInterceptor.class)) {
setConfigKey(SparkOperatorConf.LOG_CONF, true);
+ // Register the sensitive key as dynamically overridable so refresh() keeps it; refresh now
+ // drops any key that is not in the dynamic-override allow-list.
+ ConfigOption.<String>builder()
+ .key("spark.dummy.db.password")
+ .enableDynamicOverride(true)
+ .typeParameterClass(String.class)
+ .build();
SparkOperatorConfManager.INSTANCE.refresh(
Map.of("spark.dummy.db.password", "super-sensitive-value"));
mockMetricsSystemFactory
diff --git a/spark-operator/src/test/java/org/apache/spark/k8s/operator/config/ConfigOptionTest.java b/spark-operator/src/test/java/org/apache/spark/k8s/operator/config/ConfigOptionTest.java
index d1a9827..c380cc3 100644
--- a/spark-operator/src/test/java/org/apache/spark/k8s/operator/config/ConfigOptionTest.java
+++ b/spark-operator/src/test/java/org/apache/spark/k8s/operator/config/ConfigOptionTest.java
@@ -129,10 +129,10 @@ class ConfigOptionTest {
configOverride.put("foodouble", "10.0");
configOverride.put("foobool", "true");
configOverride.put("foo", "barbar");
- SparkOperatorConfManager.INSTANCE.refresh(configOverride);
ConfigOption<String> testStrConf =
ConfigOption.<String>builder()
.key("foo")
+ .enableDynamicOverride(true)
.typeParameterClass(String.class)
.description("foo foo.")
.defaultValue(defaultStringValue)
@@ -140,6 +140,7 @@ class ConfigOptionTest {
ConfigOption<Integer> testIntConf =
ConfigOption.<Integer>builder()
.key("fooint")
+ .enableDynamicOverride(true)
.typeParameterClass(Integer.class)
.description("foo foo.")
.defaultValue(defaultIntValue)
@@ -147,6 +148,7 @@ class ConfigOptionTest {
ConfigOption<Short> testShortConf =
ConfigOption.<Short>builder()
.key("fooshort")
+ .enableDynamicOverride(true)
.typeParameterClass(Short.class)
.description("foo foo.")
.defaultValue(defaultShortValue)
@@ -154,6 +156,7 @@ class ConfigOptionTest {
ConfigOption<Long> testLongConf =
ConfigOption.<Long>builder()
.key("foolong")
+ .enableDynamicOverride(true)
.typeParameterClass(Long.class)
.description("foo foo.")
.defaultValue(defaultLongValue)
@@ -161,6 +164,7 @@ class ConfigOptionTest {
ConfigOption<Boolean> testBooleanConf =
ConfigOption.<Boolean>builder()
.key("foobool")
+ .enableDynamicOverride(true)
.typeParameterClass(Boolean.class)
.description("foo foo.")
.defaultValue(defaultBooleanValue)
@@ -168,6 +172,7 @@ class ConfigOptionTest {
ConfigOption<Float> testFloatConf =
ConfigOption.<Float>builder()
.key("foofloat")
+ .enableDynamicOverride(true)
.typeParameterClass(Float.class)
.description("foo foo.")
.defaultValue(defaultFloatValue)
@@ -175,6 +180,7 @@ class ConfigOptionTest {
ConfigOption<Double> testDoubleConf =
ConfigOption.<Double>builder()
.key("foodouble")
+ .enableDynamicOverride(true)
.typeParameterClass(Double.class)
.description("foo foo.")
.defaultValue(defaultDoubleValue)
@@ -182,10 +188,14 @@ class ConfigOptionTest {
ConfigOption<Byte> testByteConf =
ConfigOption.<Byte>builder()
.key("foobyte")
+ .enableDynamicOverride(true)
.typeParameterClass(Byte.class)
.description("foo foo.")
.defaultValue(defaultByteValue)
.build();
+ // The options above opt in to dynamic override, so refreshing after they are built lets the
+ // allow-list filter retain these keys.
+ SparkOperatorConfManager.INSTANCE.refresh(configOverride);
Assertions.assertEquals(overrideStringValue, testStrConf.getValue());
Assertions.assertEquals(overrideIntValue, testIntConf.getValue());
Assertions.assertEquals(overrideLongValue, testLongConf.getValue());
diff --git a/spark-operator/src/test/java/org/apache/spark/k8s/operator/config/SparkOperatorConfManagerTest.java b/spark-operator/src/test/java/org/apache/spark/k8s/operator/config/SparkOperatorConfManagerTest.java
index 7d0538f..924b6e0 100644
--- a/spark-operator/src/test/java/org/apache/spark/k8s/operator/config/SparkOperatorConfManagerTest.java
+++ b/spark-operator/src/test/java/org/apache/spark/k8s/operator/config/SparkOperatorConfManagerTest.java
@@ -54,6 +54,7 @@ class SparkOperatorConfManagerTest {
String propBackUp = System.getProperty("spark.kubernetes.operator.foo");
System.setProperty("spark.kubernetes.operator.foo", "bar");
try {
+ registerDynamicOption("spark.kubernetes.operator.foo");
SparkOperatorConfManager confManager = new SparkOperatorConfManager();
Assertions.assertEquals("bar", confManager.getInitialValue("spark.kubernetes.operator.foo"));
Assertions.assertEquals("bar", confManager.getValue("spark.kubernetes.operator.foo"));
@@ -80,8 +81,10 @@ class SparkOperatorConfManagerTest {
String propBackUp = System.getProperty("spark.kubernetes.operator.foo");
System.setProperty("spark.kubernetes.operator.foo", "bar");
try {
+ registerDynamicOption("spark.kubernetes.operator.foo");
+ registerDynamicOption("k1");
+ registerDynamicOption("k2");
SparkOperatorConfManager confManager = new SparkOperatorConfManager();
-
// Check initial configurations.
int initialSize = confManager.getAll().size();
Assertions.assertEquals(initialSize, confManager.initialConfig.size());
@@ -111,4 +114,38 @@ class SparkOperatorConfManagerTest {
}
}
}
+
+ @Test
+ void testRefreshDropsUnknownAndNonOverridableKeys() {
+ String overridableKey = "spark.kubernetes.operator.test.overridable";
+ String nonOverridableKey = "spark.kubernetes.operator.test.nonOverridable";
+ String unknownKey = "spark.kubernetes.operator.test.unknown";
+ registerDynamicOption(overridableKey);
+ ConfigOption.<String>builder()
+ .key(nonOverridableKey)
+ .enableDynamicOverride(false)
+ .typeParameterClass(String.class)
+ .build();
+
+ SparkOperatorConfManager confManager = new SparkOperatorConfManager();
+ confManager.refresh(
+ Map.of(
+ overridableKey, "kept",
+ nonOverridableKey, "dropped",
+ unknownKey, "dropped"));
+
+ Assertions.assertEquals(1, confManager.configOverrides.size());
+ Assertions.assertEquals("kept", confManager.getValue(overridableKey));
+ Assertions.assertNull(confManager.getValue(nonOverridableKey));
+ Assertions.assertNull(confManager.getValue(unknownKey));
+ }
+
+ /** Builds a throwaway {@link ConfigOption} so {@code key} is registered as overridable. */
+ private static void registerDynamicOption(String key) {
+ ConfigOption.<String>builder()
+ .key(key)
+ .enableDynamicOverride(true)
+ .typeParameterClass(String.class)
+ .build();
+ }
}
diff --git a/spark-operator/src/test/java/org/apache/spark/k8s/operator/utils/UtilsTest.java b/spark-operator/src/test/java/org/apache/spark/k8s/operator/utils/UtilsTest.java
index 40e7e50..42c591d 100644
--- a/spark-operator/src/test/java/org/apache/spark/k8s/operator/utils/UtilsTest.java
+++ b/spark-operator/src/test/java/org/apache/spark/k8s/operator/utils/UtilsTest.java
@@ -94,14 +94,18 @@ class UtilsTest {
}
@Test
- void redactSensitiveInfoHonorsRedactionRegexFromOperatorConfig() {
+ void redactSensitiveInfoIgnoresDynamicRedactionRegexOverride() {
+ // spark.redaction.regex is intentionally not dynamically overridable: a runtime override
+ // could weaken or disable redaction, so refresh() drops it and the startup default pattern
+ // continues to apply.
SparkOperatorConfManager.INSTANCE.refresh(Map.of(REDACTION_REGEX_KEY, "(?i)confidential"));
Map<String, String> props =
Map.of(
"custom.confidential.conf", "custom-value",
"spark.kubernetes.operator.namespace", "default");
Map<String, String> redacted = Utils.redactSensitiveInfo(props);
- assertEquals(REDACTED_TEXT, redacted.get("custom.confidential.conf"));
+ // The override is ignored, so "confidential" is not treated as sensitive.
+ assertEquals("custom-value", redacted.get("custom.confidential.conf"));
assertEquals("default", redacted.get("spark.kubernetes.operator.namespace"));
assertEquals(props.keySet(), redacted.keySet());
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org