You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "sarutak (via GitHub)" <gi...@apache.org> on 2026/07/21 00:14:00 UTC
[PR] [SPARK-57893][CORE] Implement UserCredentialManager [spark]
sarutak opened a new pull request, #57387:
URL: https://github.com/apache/spark/pull/57387
### What changes were proposed in this pull request?
This is a part of the SPIP: OIDC Credential Propagation (SPARK-57703).
This PR adds `UserCredentialManager`, the driver-side credential renewal loop.
`UserCredentialManager` is a sibling of `HadoopDelegationTokenManager` and handles the OIDC credential propagation path independently. Both managers run on independent threads and can both be active simultaneously.
**Responsibilities:**
1. Reads the current identity token via `TokenIngestor.load()`
2. Calls `CredentialProvider.resolve()` for each configured scheme
3. Serializes the resulting `UserCredentials` and invokes the propagation callback
4. Schedules renewal based on `min(identity token expiry, service credential expiry) - safetyMargin`
5. Retries with exponential backoff on failure
**Key design decisions:**
- **Fail-fast on startup:** If the initial credential acquisition fails, the application fails to start rather than running with null credentials.
- **Per-provider error isolation:** One provider failure does not abort the entire resolution loop. Other providers continue to resolve.
- **Separation of concerns:** Credential *resolution* failures trigger backoff; *propagation* failures are logged and retried on the next scheduled renewal.
- **ObjectInputFilter:** Deserialization is restricted to only the classes needed for `UserCredentials`, preventing deserialization attacks.
**New files:**
- `core/.../deploy/security/UserCredentialManager.scala` — Main implementation
- `core/.../deploy/security/UserCredentialManagerSuite.scala` — Unit tests
**Modified files:**
- `CredentialProviderLoader.java` — Added `discoverAllSchemes()` for auto-discovery of registered schemes
- `config/package.scala` — Added `spark.security.credentials.*` configuration keys
- `configs-without-binding-policy-exceptions` — Registered new config keys
### Why are the changes needed?
There is no component to orchestrate credential acquisition, renewal scheduling, and propagation triggering on the driver for OIDC-based environments. This is the core driver-side engine that coordinates `TokenIngestor` and `CredentialProvider` to maintain fresh credentials.
### Does this PR introduce _any_ user-facing change?
No. All new behavior is gated by `spark.security.credentials.enabled=false` (default). No existing APIs are changed.
### How was this patch tested?
Unit tests in `UserCredentialManagerSuite` covering:
- Initial credential acquisition and callback invocation
- Fail-fast when TokenIngestor returns empty
- Serialization/deserialization round-trip with ObjectInputFilter security
- `computeRenewalDelay` edge cases (safety margin, min interval, no expiry)
- Exponential backoff (growth, cap, zero-failure edge case)
- Per-provider error isolation (partial failure, total failure)
- Token rotation triggers new credential acquisition and propagation
- Factory method (`create`) with various configurations
- Lifecycle safety (double-start guard, stop after start)
### Was this patch authored or co-authored using generative AI tooling?
Kiro CLI / Claude
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement UserCredentialManager [spark]
Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on PR #57387:
URL: https://github.com/apache/spark/pull/57387#issuecomment-5084689771
> It seems that we missed the configuration namespace conflict issue during SPIP discussion.
>
> Technically, `spark.security.credentials.*` is the namespace for `Hadoop delegation token` in Apache Spark.
>
> https://github.com/apache/spark/blob/b910d6f4101c53dd5f9b866122d92e4c41c768f3/core/src/main/scala/org/apache/spark/deploy/security/HadoopDelegationTokenManager.scala#L296
>
> So, the proposed configuration namespace is very misleading to the Apache Spark users. If you don't mind, I'd like to propose to use `spark.security.oidc.*` namespace for this **SPIP: OIDC Credential Propagation**.
>
> WDYT, @sarutak ?
Thanks for catching this, @dongjoon-hyun. I agree that reusing the `spark.security.credentials.*` for OIDC creates confusion. I'll rename to `spark.security.oidc.*` as you suggest.
Since the earlier sub-tasks (already merged) also reference the old namespace in `CredentialProviderLoader`, `CredentialProvider`, and `FileTokenIngestor`, I'll submit a preparatory PR to rename those first. Once that's merged, I'll rebase this PR with the full namespace update and address your other comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement `UserCredentialManager` [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #57387:
URL: https://github.com/apache/spark/pull/57387#discussion_r3667035666
##########
core/src/main/java/org/apache/spark/security/CredentialProviderLoader.java:
##########
@@ -221,10 +223,38 @@ private static List<CredentialProvider> loadProviders() {
return result;
}
+ /**
+ * Discovers all URI schemes supported by providers on the classpath.
+ * <p>
+ * This method queries all discovered {@link CredentialProvider} instances and collects
+ * their {@link CredentialProvider#supportedSchemes()} into a single set. Unlike
+ * {@link #providerFor(String, Map)}, this does not initialize providers or apply
+ * explicit selection rules -- it only reports what schemes are potentially available.
+ * <p>
+ * Intended for use by {@code UserCredentialManager} when no explicit scheme configuration
+ * (e.g., {@code spark.security.credentials.provider.<scheme>}) is provided.
+ *
+ * @return a set of all supported scheme names (lowercased), possibly empty
+ */
+ public static Set<String> discoverAllSchemes() {
+ List<CredentialProvider> providers = getProviders();
+ Set<String> schemes = new java.util.HashSet<>();
Review Comment:
> Added `import java.util.HashSet;` and changed to `new HashSet<>()`.
To @sarutak , the suggestion is not accept actually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement `UserCredentialManager` [spark]
Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on PR #57387:
URL: https://github.com/apache/spark/pull/57387#issuecomment-5110419320
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/86de29d22894b4ad131c6fbdedf953945e2dae24
- merged into branch-4.x https://github.com/apache/spark/commit/057b60796898749a641c160c706dad4c6885da70
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement `UserCredentialManager` [spark]
Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak closed pull request #57387: [SPARK-57893][CORE] Implement `UserCredentialManager`
URL: https://github.com/apache/spark/pull/57387
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement UserCredentialManager [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #57387:
URL: https://github.com/apache/spark/pull/57387#discussion_r3627521286
##########
core/src/main/scala/org/apache/spark/deploy/security/UserCredentialManager.scala:
##########
@@ -0,0 +1,433 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.security
+
+import java.io.{ByteArrayOutputStream, ObjectInputFilter, ObjectInputStream, ObjectOutputStream}
+import java.net.URI
+import java.nio.file.Paths
+import java.time.Instant
+import java.util.concurrent.{RejectedExecutionException, ScheduledExecutorService, TimeUnit}
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.LogKeys
+import org.apache.spark.internal.config._
+import org.apache.spark.security._
+import org.apache.spark.ui.UIUtils
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Manager for OIDC-based credential propagation on the driver.
+ *
+ * This class is a sibling of [[HadoopDelegationTokenManager]] and handles the OIDC credential
+ * propagation path independently. Both managers run on independent threads and can both be
+ * active simultaneously (e.g., a cluster accessing HDFS via Kerberos and S3 via OIDC).
+ *
+ * Responsibilities:
+ * 1. Reads the current identity token via a [[TokenIngestor]]
+ * 2. Calls [[CredentialProvider#resolve]] for each configured scheme
+ * 3. Serializes the resulting [[UserCredentials]] and invokes the propagation callback
+ * 4. Schedules renewal based on `min(identity token expiry, service credential expiry) -
+ * safetyMargin`
+ * 5. Retries with exponential backoff on failure
+ *
+ * Started from `CoarseGrainedSchedulerBackend.start()` when
+ * `spark.security.credentials.enabled=true`, independently of
+ * `UserGroupInformation.isSecurityEnabled()`.
+ *
+ * Lifecycle: call `start()` exactly once, then `stop()` to shut down.
+ * Calling `start()` after `stop()` is not supported.
+ */
+private[spark] class UserCredentialManager(
+ sparkConf: SparkConf,
+ tokenIngestor: TokenIngestor,
+ onCredentialsUpdate: Array[Byte] => Unit) extends Logging {
+
+ private val safetyMargin = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_SAFETY_MARGIN)
+ private val minInterval = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_MIN_INTERVAL)
+
+ // Thread-safe counter for exponential backoff calculation.
+ // Accessed from both the caller of start() and the scheduled renewal thread.
+ private val consecutiveFailures = new AtomicInteger(0)
+ private val maxBackoffMs: Long = TimeUnit.MINUTES.toMillis(10)
Review Comment:
`maxBackoffMs` is a hardcoded 10-minute magic constant, while its scheduling peers `safetyMargin` and `minInterval` are both configs (`SECURITY_CREDENTIALS_RENEWAL_*`). For consistency, consider making this a config too, or at least a named constant on the companion `object` beside `DEFAULT_RENEWAL_INTERVAL_NO_EXPIRY_MS`. Non-blocking.
##########
core/src/main/scala/org/apache/spark/deploy/security/UserCredentialManager.scala:
##########
@@ -0,0 +1,433 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.security
+
+import java.io.{ByteArrayOutputStream, ObjectInputFilter, ObjectInputStream, ObjectOutputStream}
+import java.net.URI
+import java.nio.file.Paths
+import java.time.Instant
+import java.util.concurrent.{RejectedExecutionException, ScheduledExecutorService, TimeUnit}
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.LogKeys
+import org.apache.spark.internal.config._
+import org.apache.spark.security._
+import org.apache.spark.ui.UIUtils
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Manager for OIDC-based credential propagation on the driver.
+ *
+ * This class is a sibling of [[HadoopDelegationTokenManager]] and handles the OIDC credential
+ * propagation path independently. Both managers run on independent threads and can both be
+ * active simultaneously (e.g., a cluster accessing HDFS via Kerberos and S3 via OIDC).
+ *
+ * Responsibilities:
+ * 1. Reads the current identity token via a [[TokenIngestor]]
+ * 2. Calls [[CredentialProvider#resolve]] for each configured scheme
+ * 3. Serializes the resulting [[UserCredentials]] and invokes the propagation callback
+ * 4. Schedules renewal based on `min(identity token expiry, service credential expiry) -
+ * safetyMargin`
+ * 5. Retries with exponential backoff on failure
+ *
+ * Started from `CoarseGrainedSchedulerBackend.start()` when
+ * `spark.security.credentials.enabled=true`, independently of
+ * `UserGroupInformation.isSecurityEnabled()`.
+ *
+ * Lifecycle: call `start()` exactly once, then `stop()` to shut down.
+ * Calling `start()` after `stop()` is not supported.
+ */
+private[spark] class UserCredentialManager(
+ sparkConf: SparkConf,
+ tokenIngestor: TokenIngestor,
+ onCredentialsUpdate: Array[Byte] => Unit) extends Logging {
+
+ private val safetyMargin = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_SAFETY_MARGIN)
+ private val minInterval = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_MIN_INTERVAL)
+
+ // Thread-safe counter for exponential backoff calculation.
+ // Accessed from both the caller of start() and the scheduled renewal thread.
+ private val consecutiveFailures = new AtomicInteger(0)
+ private val maxBackoffMs: Long = TimeUnit.MINUTES.toMillis(10)
+
+ @volatile private var renewalExecutor: ScheduledExecutorService = _
+
+ // Pre-compute the filtered credential configuration map (SparkConf is immutable).
+ private val credentialConfMap: java.util.Map[String, String] = sparkConf.getAll
+ .filter(_._1.startsWith("spark.security.credentials."))
+ .toMap.asJava
+
+ /**
+ * Start the credential manager. Acquires initial credentials and schedules renewal.
+ *
+ * The initial credential acquisition is fail-fast: if the token cannot be loaded or
+ * no credentials can be resolved, an exception is thrown. Subsequent renewal failures
+ * are handled with exponential backoff.
+ *
+ * @return The serialized initial [[UserCredentials]].
+ * @throws IllegalStateException if the initial credential acquisition fails.
+ */
+ def start(): Array[Byte] = {
+ require(renewalExecutor == null, "start() must not be called more than once")
Review Comment:
`start()` assigns `renewalExecutor` (next line) *before* the fail-fast acquisition, which can throw at line 101 (token missing) or via `resolveCredentials` at line 232 (no credentials resolved). On that throw, the daemon-thread executor has been created but is never returned or stopped, and because `start()` opens with this `require(renewalExecutor == null, ...)`, a caller that catches the exception can never retry — the guard now trips.
The analogue `HadoopDelegationTokenManager` avoids this: its `updateTokensTask()` catches and reschedules rather than throwing out of `start()`, so it never leaves `start()` with a live unmanaged executor.
Since the executor isn't used for the synchronous initial acquisition, the cleanest fix is to create it *after* the acquisition succeeds, just before `scheduleRenewal(renewalDelay)`. Alternatively, wrap the acquisition in a `try` that calls `renewalExecutor.shutdownNow()` and nulls the field before rethrowing.
##########
core/src/main/java/org/apache/spark/security/CredentialProviderLoader.java:
##########
@@ -221,10 +221,38 @@ private static List<CredentialProvider> loadProviders() {
return result;
}
+ /**
+ * Discovers all URI schemes supported by providers on the classpath.
+ * <p>
+ * This method queries all discovered {@link CredentialProvider} instances and collects
+ * their {@link CredentialProvider#supportedSchemes()} into a single set. Unlike
+ * {@link #providerFor(String, Map)}, this does not initialize providers or apply
+ * explicit selection rules -- it only reports what schemes are potentially available.
+ * <p>
+ * Intended for use by {@code UserCredentialManager} when no explicit scheme configuration
+ * (e.g., {@code spark.security.credentials.provider.<scheme>}) is provided.
+ *
+ * @param conf Spark configuration properties (used for potential future filtering)
+ * @return a set of all supported scheme names (lowercased), possibly empty
+ */
+ public static Set<String> discoverAllSchemes(Map<String, String> conf) {
Review Comment:
The `conf` parameter is never used in this method body — the Javadoc notes it's "used for potential future filtering." An unused parameter is YAGNI: consider dropping it until the filtering actually lands (the caller `discoverSchemes()` can add it back with the feature). Non-blocking.
##########
core/src/main/scala/org/apache/spark/deploy/security/UserCredentialManager.scala:
##########
@@ -0,0 +1,433 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.security
+
+import java.io.{ByteArrayOutputStream, ObjectInputFilter, ObjectInputStream, ObjectOutputStream}
+import java.net.URI
+import java.nio.file.Paths
+import java.time.Instant
+import java.util.concurrent.{RejectedExecutionException, ScheduledExecutorService, TimeUnit}
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.LogKeys
+import org.apache.spark.internal.config._
+import org.apache.spark.security._
+import org.apache.spark.ui.UIUtils
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Manager for OIDC-based credential propagation on the driver.
+ *
+ * This class is a sibling of [[HadoopDelegationTokenManager]] and handles the OIDC credential
+ * propagation path independently. Both managers run on independent threads and can both be
+ * active simultaneously (e.g., a cluster accessing HDFS via Kerberos and S3 via OIDC).
+ *
+ * Responsibilities:
+ * 1. Reads the current identity token via a [[TokenIngestor]]
+ * 2. Calls [[CredentialProvider#resolve]] for each configured scheme
+ * 3. Serializes the resulting [[UserCredentials]] and invokes the propagation callback
+ * 4. Schedules renewal based on `min(identity token expiry, service credential expiry) -
+ * safetyMargin`
+ * 5. Retries with exponential backoff on failure
+ *
+ * Started from `CoarseGrainedSchedulerBackend.start()` when
Review Comment:
Nit: this cites `CoarseGrainedSchedulerBackend.start()` as the caller, but no such call site exists yet on this branch (this is a WIP subtask). It reads as accurate to someone holding the full SPIP context, but is misleading on `master` today. Consider softening to "intended to be started from ..." or deferring the claim to the subtask that adds the wiring.
##########
core/src/main/java/org/apache/spark/security/CredentialProviderLoader.java:
##########
@@ -221,10 +221,38 @@ private static List<CredentialProvider> loadProviders() {
return result;
}
+ /**
+ * Discovers all URI schemes supported by providers on the classpath.
+ * <p>
+ * This method queries all discovered {@link CredentialProvider} instances and collects
+ * their {@link CredentialProvider#supportedSchemes()} into a single set. Unlike
+ * {@link #providerFor(String, Map)}, this does not initialize providers or apply
+ * explicit selection rules -- it only reports what schemes are potentially available.
+ * <p>
+ * Intended for use by {@code UserCredentialManager} when no explicit scheme configuration
+ * (e.g., {@code spark.security.credentials.provider.<scheme>}) is provided.
+ *
+ * @param conf Spark configuration properties (used for potential future filtering)
+ * @return a set of all supported scheme names (lowercased), possibly empty
+ */
+ public static Set<String> discoverAllSchemes(Map<String, String> conf) {
+ List<CredentialProvider> providers = getProviders();
+ Set<String> schemes = new java.util.HashSet<>();
+ for (CredentialProvider provider : providers) {
+ Set<String> providerSchemes = provider.supportedSchemes();
+ if (providerSchemes != null) {
+ for (String s : providerSchemes) {
+ schemes.add(s.toLowerCase(Locale.ROOT));
+ }
+ }
+ }
+ return schemes;
+ }
+
/**
* Resets the cached provider list and initialization tracking. Intended for testing only.
*/
- static void resetForTesting() {
+ public static void resetForTesting() {
Review Comment:
Nit: `resetForTesting()` is widened from package-private to `public` here. This is needed because the suite lives in a different package (`o.a.s.deploy.security` vs this class's `o.a.s.security`), and it's defensible on a `@Private` class — but annotating it `@VisibleForTesting` (used elsewhere in core) would document that the widened visibility is for tests, not general use.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement UserCredentialManager [spark]
Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on code in PR #57387:
URL: https://github.com/apache/spark/pull/57387#discussion_r3629885652
##########
core/src/main/scala/org/apache/spark/deploy/security/UserCredentialManager.scala:
##########
@@ -0,0 +1,433 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.security
+
+import java.io.{ByteArrayOutputStream, ObjectInputFilter, ObjectInputStream, ObjectOutputStream}
+import java.net.URI
+import java.nio.file.Paths
+import java.time.Instant
+import java.util.concurrent.{RejectedExecutionException, ScheduledExecutorService, TimeUnit}
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.LogKeys
+import org.apache.spark.internal.config._
+import org.apache.spark.security._
+import org.apache.spark.ui.UIUtils
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Manager for OIDC-based credential propagation on the driver.
+ *
+ * This class is a sibling of [[HadoopDelegationTokenManager]] and handles the OIDC credential
+ * propagation path independently. Both managers run on independent threads and can both be
+ * active simultaneously (e.g., a cluster accessing HDFS via Kerberos and S3 via OIDC).
+ *
+ * Responsibilities:
+ * 1. Reads the current identity token via a [[TokenIngestor]]
+ * 2. Calls [[CredentialProvider#resolve]] for each configured scheme
+ * 3. Serializes the resulting [[UserCredentials]] and invokes the propagation callback
+ * 4. Schedules renewal based on `min(identity token expiry, service credential expiry) -
+ * safetyMargin`
+ * 5. Retries with exponential backoff on failure
+ *
+ * Started from `CoarseGrainedSchedulerBackend.start()` when
+ * `spark.security.credentials.enabled=true`, independently of
+ * `UserGroupInformation.isSecurityEnabled()`.
+ *
+ * Lifecycle: call `start()` exactly once, then `stop()` to shut down.
+ * Calling `start()` after `stop()` is not supported.
+ */
+private[spark] class UserCredentialManager(
+ sparkConf: SparkConf,
+ tokenIngestor: TokenIngestor,
+ onCredentialsUpdate: Array[Byte] => Unit) extends Logging {
+
+ private val safetyMargin = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_SAFETY_MARGIN)
+ private val minInterval = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_MIN_INTERVAL)
+
+ // Thread-safe counter for exponential backoff calculation.
+ // Accessed from both the caller of start() and the scheduled renewal thread.
+ private val consecutiveFailures = new AtomicInteger(0)
+ private val maxBackoffMs: Long = TimeUnit.MINUTES.toMillis(10)
+
+ @volatile private var renewalExecutor: ScheduledExecutorService = _
+
+ // Pre-compute the filtered credential configuration map (SparkConf is immutable).
+ private val credentialConfMap: java.util.Map[String, String] = sparkConf.getAll
+ .filter(_._1.startsWith("spark.security.credentials."))
+ .toMap.asJava
+
+ /**
+ * Start the credential manager. Acquires initial credentials and schedules renewal.
+ *
+ * The initial credential acquisition is fail-fast: if the token cannot be loaded or
+ * no credentials can be resolved, an exception is thrown. Subsequent renewal failures
+ * are handled with exponential backoff.
+ *
+ * @return The serialized initial [[UserCredentials]].
+ * @throws IllegalStateException if the initial credential acquisition fails.
+ */
+ def start(): Array[Byte] = {
+ require(renewalExecutor == null, "start() must not be called more than once")
+ renewalExecutor =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("user-credential-renewal")
+
+ // Initial acquisition is fail-fast (no retry/backoff).
+ // This ensures the application fails to start if credentials cannot be obtained,
+ // rather than running with null credentials.
+ val userContext = tokenIngestor.load()
+ if (!userContext.isPresent) {
+ throw new IllegalStateException(
+ "Failed to start UserCredentialManager: identity token file is missing or malformed. " +
+ s"Check ${SECURITY_CREDENTIALS_IDENTITY_TOKEN_FILE.key} configuration.")
+ }
+
+ val ctx = userContext.get()
+ logInfo(log"Initial identity token loaded for principal " +
+ log"${MDC(LogKeys.PRINCIPAL, ctx.getPrincipal)} " +
+ log"(issuer: ${MDC(LogKeys.URI, ctx.getIssuer)})")
+
+ val (credentials, earliestExpiry) = resolveCredentials(ctx)
+ val serialized = serializeUserCredentials(credentials)
+
+ // Propagate initial credentials
+ onCredentialsUpdate(serialized)
+
+ // Schedule first renewal
+ val renewalDelay = computeRenewalDelay(ctx, earliestExpiry)
+ scheduleRenewal(renewalDelay)
+
+ logInfo(log"Credential acquisition successful. Next renewal in " +
+ log"${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(renewalDelay))}.")
+ serialized
+ }
+
+ def stop(): Unit = {
+ if (renewalExecutor != null) {
+ renewalExecutor.shutdownNow()
+ }
+ }
+
+ /**
+ * Scheduled renewal task: load identity token, resolve credentials, propagate,
+ * and schedule the next renewal. Failures are retried with exponential backoff.
+ */
+ private def renewCredentialsTask(): Unit = {
+ try {
+ val userContext = tokenIngestor.load()
Review Comment:
The timeout concern is specific to `FileTokenIngestor`'s `Files.readAllBytes` call rather than `UserCredentialManager`'s orchestration. The SPI contract (`TokenIngestor.load()`) doesn't prescribe blocking behavior, so a `Future`-based timeout or non-blocking read could be added inside `FileTokenIngestor` without changing the manager.
Since `FileTokenIngestor` was merged in Sub-task 3, I think it's better to track this as a follow-up issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement UserCredentialManager [spark]
Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on code in PR #57387:
URL: https://github.com/apache/spark/pull/57387#discussion_r3665747919
##########
core/src/main/scala/org/apache/spark/deploy/security/UserCredentialManager.scala:
##########
@@ -0,0 +1,443 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.security
+
+import java.io.{ByteArrayOutputStream, ObjectInputFilter, ObjectInputStream, ObjectOutputStream}
+import java.net.URI
+import java.nio.file.Paths
+import java.time.Instant
+import java.util.concurrent.{RejectedExecutionException, ScheduledExecutorService, TimeUnit}
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.LogKeys
+import org.apache.spark.internal.config._
+import org.apache.spark.security._
+import org.apache.spark.ui.UIUtils
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Manager for OIDC-based credential propagation on the driver.
+ *
+ * This class is a sibling of [[HadoopDelegationTokenManager]] and handles the OIDC credential
+ * propagation path independently. Both managers run on independent threads and can both be
+ * active simultaneously (e.g., a cluster accessing HDFS via Kerberos and S3 via OIDC).
+ *
+ * Responsibilities:
+ * 1. Reads the current identity token via a [[TokenIngestor]]
+ * 2. Calls [[CredentialProvider#resolve]] for each configured scheme
+ * 3. Serializes the resulting [[UserCredentials]] and invokes the propagation callback
+ * 4. Schedules renewal based on `min(identity token expiry, service credential expiry) -
+ * safetyMargin`
+ * 5. Retries with exponential backoff on failure
+ *
+ * Intended to be started from `CoarseGrainedSchedulerBackend.start()` when
+ * `spark.security.credentials.enabled=true`, independently of
+ * `UserGroupInformation.isSecurityEnabled()`.
+ *
+ * Lifecycle: call `start()` exactly once, then `stop()` to shut down.
+ * Calling `start()` after `stop()` is not supported.
+ */
+private[spark] class UserCredentialManager(
+ sparkConf: SparkConf,
+ tokenIngestor: TokenIngestor,
+ onCredentialsUpdate: Array[Byte] => Unit) extends Logging {
+
+ private val safetyMargin = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_SAFETY_MARGIN)
+ private val minInterval = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_MIN_INTERVAL)
+
+ // Thread-safe counter for exponential backoff calculation.
+ // Accessed from both the caller of start() and the scheduled renewal thread.
+ private val consecutiveFailures = new AtomicInteger(0)
+ private val maxBackoffMs: Long = UserCredentialManager.MAX_BACKOFF_MS
+
+ @volatile private var renewalExecutor: ScheduledExecutorService = _
+
+ // Pre-compute the filtered credential configuration map (SparkConf is immutable).
Review Comment:
Changed to "Snapshot of credential-related configuration at construction time".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement UserCredentialManager [spark]
Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on code in PR #57387:
URL: https://github.com/apache/spark/pull/57387#discussion_r3665736559
##########
core/src/main/scala/org/apache/spark/internal/config/package.scala:
##########
@@ -1679,6 +1679,44 @@ package object config {
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("1h")
+ private[spark] val SECURITY_CREDENTIALS_ENABLED =
+ ConfigBuilder("spark.security.credentials.enabled")
+ .doc("Whether to enable OIDC credential propagation. When enabled, the driver reads an " +
+ "identity token from a file, exchanges it for short-lived service credentials via " +
+ "CredentialProvider implementations, and propagates those credentials to executors.")
+ .version("4.3.0")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+ .booleanConf
+ .createWithDefault(false)
+
+ private[spark] val SECURITY_CREDENTIALS_IDENTITY_TOKEN_FILE =
+ ConfigBuilder("spark.security.credentials.identityToken.file")
+ .doc("Path to the OIDC identity token file on the driver. Required when " +
+ "spark.security.credentials.enabled is true. The file should contain a JWT token " +
+ "(e.g., a Kubernetes projected service account token).")
+ .version("4.3.0")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+ .stringConf
+ .createOptional
+
+ private[spark] val SECURITY_CREDENTIALS_RENEWAL_SAFETY_MARGIN =
+ ConfigBuilder("spark.security.credentials.renewal.safetyMargin")
+ .doc("How long before credential expiry to trigger renewal. Credentials are refreshed " +
+ "at min(identity token expiry, service credential expiry) minus this margin.")
+ .version("4.3.0")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+ .timeConf(TimeUnit.MILLISECONDS)
+ .createWithDefaultString("60s")
+
+ private[spark] val SECURITY_CREDENTIALS_RENEWAL_MIN_INTERVAL =
+ ConfigBuilder("spark.security.credentials.renewal.minInterval")
+ .doc("Minimum interval between credential renewal attempts. This prevents tight renewal " +
+ "loops when credentials have very short TTLs or when failures cause rapid retries.")
+ .version("4.3.0")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+ .timeConf(TimeUnit.MILLISECONDS)
+ .createWithDefaultString("30s")
Review Comment:
Added `.checkValue(_ > 0, ...)` to both `minInterval` and `safetyMargin` for consistency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement UserCredentialManager [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #57387:
URL: https://github.com/apache/spark/pull/57387#discussion_r3666891783
##########
core/src/main/scala/org/apache/spark/internal/config/package.scala:
##########
@@ -1690,6 +1690,46 @@ package object config {
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("1h")
+ private[spark] val SECURITY_CREDENTIALS_ENABLED =
Review Comment:
```suggestion
private[spark] val SECURITY_OIDC_ENABLED =
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement UserCredentialManager [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #57387:
URL: https://github.com/apache/spark/pull/57387#discussion_r3650937755
##########
core/src/main/scala/org/apache/spark/deploy/security/UserCredentialManager.scala:
##########
@@ -0,0 +1,443 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.security
+
+import java.io.{ByteArrayOutputStream, ObjectInputFilter, ObjectInputStream, ObjectOutputStream}
+import java.net.URI
+import java.nio.file.Paths
+import java.time.Instant
+import java.util.concurrent.{RejectedExecutionException, ScheduledExecutorService, TimeUnit}
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.LogKeys
+import org.apache.spark.internal.config._
+import org.apache.spark.security._
+import org.apache.spark.ui.UIUtils
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Manager for OIDC-based credential propagation on the driver.
+ *
+ * This class is a sibling of [[HadoopDelegationTokenManager]] and handles the OIDC credential
+ * propagation path independently. Both managers run on independent threads and can both be
+ * active simultaneously (e.g., a cluster accessing HDFS via Kerberos and S3 via OIDC).
+ *
+ * Responsibilities:
+ * 1. Reads the current identity token via a [[TokenIngestor]]
+ * 2. Calls [[CredentialProvider#resolve]] for each configured scheme
+ * 3. Serializes the resulting [[UserCredentials]] and invokes the propagation callback
+ * 4. Schedules renewal based on `min(identity token expiry, service credential expiry) -
+ * safetyMargin`
+ * 5. Retries with exponential backoff on failure
+ *
+ * Intended to be started from `CoarseGrainedSchedulerBackend.start()` when
+ * `spark.security.credentials.enabled=true`, independently of
+ * `UserGroupInformation.isSecurityEnabled()`.
+ *
+ * Lifecycle: call `start()` exactly once, then `stop()` to shut down.
+ * Calling `start()` after `stop()` is not supported.
+ */
+private[spark] class UserCredentialManager(
+ sparkConf: SparkConf,
+ tokenIngestor: TokenIngestor,
+ onCredentialsUpdate: Array[Byte] => Unit) extends Logging {
+
+ private val safetyMargin = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_SAFETY_MARGIN)
+ private val minInterval = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_MIN_INTERVAL)
+
+ // Thread-safe counter for exponential backoff calculation.
+ // Accessed from both the caller of start() and the scheduled renewal thread.
+ private val consecutiveFailures = new AtomicInteger(0)
+ private val maxBackoffMs: Long = UserCredentialManager.MAX_BACKOFF_MS
+
+ @volatile private var renewalExecutor: ScheduledExecutorService = _
+
+ // Pre-compute the filtered credential configuration map (SparkConf is immutable).
+ private val credentialConfMap: java.util.Map[String, String] = sparkConf.getAll
+ .filter(_._1.startsWith("spark.security.credentials."))
+ .toMap.asJava
+
+ /**
+ * Start the credential manager. Acquires initial credentials and schedules renewal.
+ *
+ * The initial credential acquisition is fail-fast: if the token cannot be loaded or
+ * no credentials can be resolved, an exception is thrown. Subsequent renewal failures
+ * are handled with exponential backoff.
+ *
+ * @return The serialized initial [[UserCredentials]].
+ * @throws IllegalStateException if the initial credential acquisition fails.
+ */
+ def start(): Array[Byte] = {
+ require(renewalExecutor == null, "start() must not be called more than once")
+
+ // Initial acquisition is fail-fast (no retry/backoff).
+ // This ensures the application fails to start if credentials cannot be obtained,
+ // rather than running with null credentials.
+ val userContext = tokenIngestor.load()
+ if (!userContext.isPresent) {
+ throw new IllegalStateException(
+ "Failed to start UserCredentialManager: identity token file is missing or malformed. " +
+ s"Check ${SECURITY_CREDENTIALS_IDENTITY_TOKEN_FILE.key} configuration.")
+ }
+
+ val ctx = userContext.get()
+ logInfo(log"Initial identity token loaded for principal " +
+ log"${MDC(LogKeys.PRINCIPAL, ctx.getPrincipal)} " +
+ log"(issuer: ${MDC(LogKeys.URI, ctx.getIssuer)})")
+
+ val (credentials, earliestExpiry) = resolveCredentials(ctx)
+ val serialized = serializeUserCredentials(credentials)
+
+ // Propagate initial credentials
+ onCredentialsUpdate(serialized)
+
+ // Create the renewal executor only after successful initial acquisition.
+ // This avoids leaking a daemon thread if the fail-fast path throws, and
+ // keeps the require() guard valid for retry scenarios.
+ renewalExecutor =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("user-credential-renewal")
+
+ // Schedule first renewal
+ val renewalDelay = computeRenewalDelay(ctx, earliestExpiry)
+ scheduleRenewal(renewalDelay)
+
+ logInfo(log"Credential acquisition successful. Next renewal in " +
+ log"${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(renewalDelay))}.")
+ serialized
+ }
+
+ def stop(): Unit = {
+ if (renewalExecutor != null) {
+ renewalExecutor.shutdownNow()
+ }
+ }
+
+ /**
+ * Scheduled renewal task: load identity token, resolve credentials, propagate,
+ * and schedule the next renewal. Failures are retried with exponential backoff.
+ */
+ private def renewCredentialsTask(): Unit = {
+ try {
+ val userContext = tokenIngestor.load()
+ if (!userContext.isPresent) {
+ throw new IllegalStateException(
+ "TokenIngestor returned empty - identity token file may be missing or malformed")
+ }
+
+ val ctx = userContext.get()
+ logInfo(log"Loaded identity token for principal " +
+ log"${MDC(LogKeys.PRINCIPAL, ctx.getPrincipal)} " +
+ log"(issuer: ${MDC(LogKeys.URI, ctx.getIssuer)})")
+
+ val (credentials, earliestExpiry) = resolveCredentials(ctx)
+ val serialized = serializeUserCredentials(credentials)
+
+ // Propagate credentials to executors. Errors here are logged separately
+ // so that credential-fetch success is not conflated with distribution failure.
+ try {
+ onCredentialsUpdate(serialized)
+ } catch {
+ case e: Exception =>
+ logWarning(log"Credentials were resolved successfully but failed to propagate " +
+ log"to executors. Executors will receive updated credentials on next renewal.", e)
+ }
+
+ // Reset backoff on successful credential resolution. This is intentionally done
+ // even if onCredentialsUpdate failed above: the backoff counter tracks credential
+ // *resolution* failures (STS/provider issues), not propagation failures.
+ // Propagation failures are transient and will be retried on next scheduled renewal.
+ consecutiveFailures.set(0)
+
+ // Schedule next renewal
+ val renewalDelay = computeRenewalDelay(ctx, earliestExpiry)
+ scheduleRenewal(renewalDelay)
+
+ logInfo(log"Credential renewal successful. Next renewal in " +
+ log"${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(renewalDelay))}.")
+ } catch {
+ case _: InterruptedException =>
+ // Shutting down, ignore
+ case e: Exception =>
+ val failures = consecutiveFailures.incrementAndGet()
+ val delay = computeBackoffDelay()
+ logWarning(log"Failed to renew user credentials (attempt " +
+ log"${MDC(LogKeys.NUM_RETRY, failures)}), " +
+ log"will retry in ${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(delay))}.", e)
+ scheduleRenewal(delay)
+ }
+ }
+
+ /**
+ * Resolve credentials for all schemes that have registered providers.
+ *
+ * @return Tuple of (UserCredentials, earliest expiry across all service credentials)
+ */
+ private def resolveCredentials(
+ ctx: UserContext): (UserCredentials, Option[Instant]) = {
+ val schemes = discoverSchemes()
+
+ val credentialMap = new mutable.HashMap[String, ServiceCredential]()
+ var earliestExpiry: Option[Instant] = None
+
+ for (scheme <- schemes) {
+ try {
+ val providerOpt = CredentialProviderLoader.providerFor(scheme, credentialConfMap)
+ if (providerOpt.isPresent) {
+ val provider = providerOpt.get()
+ // Use a synthetic target URI with just the scheme for initial resolution.
+ // The full URI is passed in future versions when path-specific resolution is needed.
+ // The "synthetic" authority signals this is not a real endpoint but a placeholder
+ // for scheme-based provider selection.
+ val target = new URI(scheme, UserCredentialManager.SYNTHETIC_TARGET_AUTHORITY,
+ "/", null, null)
+ val credential = provider.resolve(ctx, target)
+ credentialMap.put(scheme, credential)
+
+ val expiry = credential.getExpiresAt
+ if (expiry != null) {
+ earliestExpiry = earliestExpiry match {
+ case Some(existing) if existing.isBefore(expiry) => Some(existing)
+ case _ => Some(expiry)
+ }
+ }
+ } else {
+ logWarning(log"No credential provider found for scheme " +
+ log"${MDC(LogKeys.URI, scheme)}. Skipping.")
+ }
+ } catch {
+ case e: Exception =>
+ logWarning(log"Failed to resolve credentials for scheme " +
+ log"${MDC(LogKeys.URI, scheme)}. Skipping this provider.", e)
+ }
+ }
+
+ if (credentialMap.isEmpty) {
+ throw new IllegalStateException(
+ "No credential providers resolved any credentials. " +
+ "Check that providers are on the classpath and configured correctly.")
+ }
+
+ (new UserCredentials(credentialMap.asJava), earliestExpiry)
+ }
+
+ /**
+ * Discover all schemes that have at least one registered provider.
+ *
+ * Schemes are determined by explicit configuration keys of the form
+ * `spark.security.credentials.provider.<scheme>`. If no explicit configuration
+ * exists, the method queries `CredentialProviderLoader` to discover all providers
+ * registered via ServiceLoader and collects their supported schemes.
+ *
+ * Note: When using auto-discovery (no explicit config), multiple providers may
+ * support the same scheme. In that case, `CredentialProviderLoader.providerFor`
+ * will throw an `IllegalArgumentException` for that scheme. The caller
+ * (`resolveCredentials`) handles this gracefully via per-provider exception catching,
+ * logging a warning and continuing with remaining schemes.
+ */
+ private def discoverSchemes(): Set[String] = {
+ // Extract explicitly configured scheme -> provider mappings
+ val explicitSchemes = credentialConfMap.asScala
+ .filter { case (k, _) => k.startsWith("spark.security.credentials.provider.") }
+ .map { case (k, _) => k.stripPrefix("spark.security.credentials.provider.") }
+ .toSet
+
+ if (explicitSchemes.nonEmpty) {
+ explicitSchemes
+ } else {
+ // No explicit scheme configuration. Discover all schemes that have a provider
+ // available on the classpath by probing CredentialProviderLoader.
+ // This covers both built-in providers (e.g., connector/credential-aws for "s3a")
+ // and third-party providers registered via ServiceLoader.
+ CredentialProviderLoader.discoverAllSchemes().asScala.toSet
+ }
+ }
+
+ /**
+ * Compute the delay until next renewal.
+ * Uses min(identity token expiry, service credential expiry) - safetyMargin,
+ * bounded below by minInterval.
+ */
+ private[security] def computeRenewalDelay(
+ ctx: UserContext,
+ earliestCredentialExpiry: Option[Instant]): Long = {
+ val now = System.currentTimeMillis()
+
+ // Consider identity token expiry
+ val tokenExpiry: Option[Long] = Option(ctx.getExpiresAt).map(_.toEpochMilli)
+
+ // Consider earliest service credential expiry
+ val credExpiry: Option[Long] = earliestCredentialExpiry.map(_.toEpochMilli)
+
+ // Take the minimum of both
+ val effectiveExpiry: Option[Long] = (tokenExpiry, credExpiry) match {
+ case (Some(t), Some(c)) => Some(math.min(t, c))
+ case (Some(t), None) => Some(t)
+ case (None, Some(c)) => Some(c)
+ case (None, None) => None
+ }
+
+ effectiveExpiry match {
+ case Some(expiry) =>
+ math.max(expiry - now - safetyMargin, minInterval)
+ case None =>
+ // No expiry information available from either the identity token or service
+ // credentials. Use half of the default suggested TTL (15 min / 2 = 7.5 min,
+ // rounded down) as a conservative polling interval. This ensures credentials
+ // are refreshed even when providers don't report expiration times.
+ UserCredentialManager.DEFAULT_RENEWAL_INTERVAL_NO_EXPIRY_MS
+ }
+ }
+
+ /**
+ * Compute backoff delay using exponential backoff with jitter.
+ * Bounded by minInterval (floor) and maxBackoffMs (ceiling).
+ */
+ private[security] def computeBackoffDelay(): Long = {
+ // Guard against negative or zero shift amounts. consecutiveFailures should always
+ // be >= 1 when this is called (incremented before calling), but we protect against
+ // edge cases defensively.
+ val shiftAmount = math.max(0, math.min(consecutiveFailures.get() - 1, 6))
+ val baseDelay = minInterval * (1L << shiftAmount)
+ val cappedDelay = math.min(baseDelay, maxBackoffMs)
+ // Add 10% jitter to avoid thundering herd on recovery
+ val jitter = (cappedDelay * 0.1 * math.random()).toLong
+ math.max(cappedDelay + jitter, minInterval)
+ }
+
+ private def scheduleRenewal(delay: Long): Unit = {
+ try {
+ val renewalTask = new Runnable {
+ override def run(): Unit = {
+ renewCredentialsTask()
+ }
+ }
+ renewalExecutor.schedule(renewalTask, delay, TimeUnit.MILLISECONDS)
+ } catch {
+ case _: RejectedExecutionException =>
+ // Executor has been shut down (e.g., stop() called concurrently). This is expected
+ // during application shutdown -- no further renewals will be scheduled.
+ logDebug(log"Renewal scheduling rejected - executor is shut down.")
+ }
+ }
+
+ /**
+ * Serialize [[UserCredentials]] to a byte array using Java serialization.
+ */
+ private[security] def serializeUserCredentials(credentials: UserCredentials): Array[Byte] = {
+ val bos = new ByteArrayOutputStream()
+ try {
+ val oos = new ObjectOutputStream(bos)
+ try {
+ oos.writeObject(credentials)
+ oos.flush()
+ } finally {
+ oos.close()
+ }
+ bos.toByteArray
+ } finally {
+ bos.close()
+ }
+ }
+}
+
+private[spark] object UserCredentialManager {
+
+ /**
+ * Synthetic authority used in target URIs for scheme-based provider resolution.
+ * Providers should not rely on this value; it signals that no specific endpoint
+ * is targeted and only the URI scheme is meaningful.
+ */
+ private val SYNTHETIC_TARGET_AUTHORITY = "synthetic"
+
+ /**
+ * Maximum backoff delay between credential renewal retry attempts.
+ * Caps the exponential backoff to prevent excessively long gaps between retries.
+ */
+ private val MAX_BACKOFF_MS: Long = TimeUnit.MINUTES.toMillis(10)
+
+ /**
+ * Default renewal interval when neither the identity token nor service credentials
+ * report an expiration time. Set to 7 minutes (half of the default 15-minute
+ * suggested TTL from CredentialProvider.suggestedTtl()).
+ */
+ private val DEFAULT_RENEWAL_INTERVAL_NO_EXPIRY_MS: Long = TimeUnit.MINUTES.toMillis(7)
+
+ /**
+ * ObjectInputFilter pattern restricting deserialization to only the classes needed
+ * for UserCredentials. This prevents deserialization gadget chain attacks while
+ * allowing Java's built-in collection internal classes and arrays that HashMap uses.
+ */
+ private val DESERIALIZATION_FILTER: String =
+ "org.apache.spark.security.**;" +
+ "java.util.**;" +
+ "java.time.**;" +
+ "java.lang.**;" +
+ "maxdepth=10;" +
+ "!*"
+
+ /**
+ * Create a UserCredentialManager if OIDC credential propagation is enabled.
+ *
+ * @param sparkConf The Spark configuration
+ * @param onCredentialsUpdate Callback to propagate credentials to executors
+ * @return Some(manager) if enabled, None otherwise
+ */
+ def create(
+ sparkConf: SparkConf,
+ onCredentialsUpdate: Array[Byte] => Unit): Option[UserCredentialManager] = {
+ if (!sparkConf.get(SECURITY_CREDENTIALS_ENABLED)) {
+ None
+ } else {
+ val tokenFile = sparkConf.get(SECURITY_CREDENTIALS_IDENTITY_TOKEN_FILE).getOrElse {
+ throw new IllegalArgumentException(
+ s"${SECURITY_CREDENTIALS_IDENTITY_TOKEN_FILE.key} must be set when " +
+ s"${SECURITY_CREDENTIALS_ENABLED.key} is true")
+ }
+
+ val tokenIngestor = new FileTokenIngestor(Paths.get(tokenFile))
+ Some(new UserCredentialManager(sparkConf, tokenIngestor, onCredentialsUpdate))
+ }
+ }
+
+ /**
+ * Deserialize [[UserCredentials]] from a byte array.
+ *
+ * Uses an [[ObjectInputFilter]] to restrict deserialized classes to only those
+ * required for [[UserCredentials]], preventing deserialization attacks.
+ */
+ def deserializeUserCredentials(bytes: Array[Byte]): UserCredentials = {
+ val bis = new java.io.ByteArrayInputStream(bytes)
+ try {
+ val ois = new ObjectInputStream(bis)
+ try {
+ ois.setObjectInputFilter(
+ ObjectInputFilter.Config.createFilter(DESERIALIZATION_FILTER))
+ ois.readObject().asInstanceOf[UserCredentials]
+ } finally {
+ ois.close()
+ }
+ } finally {
+ bis.close()
+ }
Review Comment:
BTW, `serializeUserCredentials` has the same patter. Please simplify `serializeUserCredentials` in the same way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement UserCredentialManager [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #57387:
URL: https://github.com/apache/spark/pull/57387#issuecomment-5080301318
Sure, @sarutak .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement UserCredentialManager [spark]
Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on code in PR #57387:
URL: https://github.com/apache/spark/pull/57387#discussion_r3665812339
##########
core/src/main/scala/org/apache/spark/deploy/security/UserCredentialManager.scala:
##########
@@ -0,0 +1,443 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.security
+
+import java.io.{ByteArrayOutputStream, ObjectInputFilter, ObjectInputStream, ObjectOutputStream}
+import java.net.URI
+import java.nio.file.Paths
+import java.time.Instant
+import java.util.concurrent.{RejectedExecutionException, ScheduledExecutorService, TimeUnit}
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.LogKeys
+import org.apache.spark.internal.config._
+import org.apache.spark.security._
+import org.apache.spark.ui.UIUtils
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Manager for OIDC-based credential propagation on the driver.
+ *
+ * This class is a sibling of [[HadoopDelegationTokenManager]] and handles the OIDC credential
+ * propagation path independently. Both managers run on independent threads and can both be
+ * active simultaneously (e.g., a cluster accessing HDFS via Kerberos and S3 via OIDC).
+ *
+ * Responsibilities:
+ * 1. Reads the current identity token via a [[TokenIngestor]]
+ * 2. Calls [[CredentialProvider#resolve]] for each configured scheme
+ * 3. Serializes the resulting [[UserCredentials]] and invokes the propagation callback
+ * 4. Schedules renewal based on `min(identity token expiry, service credential expiry) -
+ * safetyMargin`
+ * 5. Retries with exponential backoff on failure
+ *
+ * Intended to be started from `CoarseGrainedSchedulerBackend.start()` when
+ * `spark.security.credentials.enabled=true`, independently of
+ * `UserGroupInformation.isSecurityEnabled()`.
+ *
+ * Lifecycle: call `start()` exactly once, then `stop()` to shut down.
+ * Calling `start()` after `stop()` is not supported.
+ */
+private[spark] class UserCredentialManager(
+ sparkConf: SparkConf,
+ tokenIngestor: TokenIngestor,
+ onCredentialsUpdate: Array[Byte] => Unit) extends Logging {
+
+ private val safetyMargin = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_SAFETY_MARGIN)
+ private val minInterval = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_MIN_INTERVAL)
+
+ // Thread-safe counter for exponential backoff calculation.
+ // Accessed from both the caller of start() and the scheduled renewal thread.
+ private val consecutiveFailures = new AtomicInteger(0)
+ private val maxBackoffMs: Long = UserCredentialManager.MAX_BACKOFF_MS
+
+ @volatile private var renewalExecutor: ScheduledExecutorService = _
+
+ // Pre-compute the filtered credential configuration map (SparkConf is immutable).
+ private val credentialConfMap: java.util.Map[String, String] = sparkConf.getAll
+ .filter(_._1.startsWith("spark.security.credentials."))
+ .toMap.asJava
+
+ /**
+ * Start the credential manager. Acquires initial credentials and schedules renewal.
+ *
+ * The initial credential acquisition is fail-fast: if the token cannot be loaded or
+ * no credentials can be resolved, an exception is thrown. Subsequent renewal failures
+ * are handled with exponential backoff.
+ *
+ * @return The serialized initial [[UserCredentials]].
+ * @throws IllegalStateException if the initial credential acquisition fails.
+ */
+ def start(): Array[Byte] = {
+ require(renewalExecutor == null, "start() must not be called more than once")
+
+ // Initial acquisition is fail-fast (no retry/backoff).
+ // This ensures the application fails to start if credentials cannot be obtained,
+ // rather than running with null credentials.
+ val userContext = tokenIngestor.load()
+ if (!userContext.isPresent) {
+ throw new IllegalStateException(
+ "Failed to start UserCredentialManager: identity token file is missing or malformed. " +
+ s"Check ${SECURITY_CREDENTIALS_IDENTITY_TOKEN_FILE.key} configuration.")
+ }
+
+ val ctx = userContext.get()
+ logInfo(log"Initial identity token loaded for principal " +
+ log"${MDC(LogKeys.PRINCIPAL, ctx.getPrincipal)} " +
+ log"(issuer: ${MDC(LogKeys.URI, ctx.getIssuer)})")
+
+ val (credentials, earliestExpiry) = resolveCredentials(ctx)
+ val serialized = serializeUserCredentials(credentials)
+
+ // Propagate initial credentials
+ onCredentialsUpdate(serialized)
+
+ // Create the renewal executor only after successful initial acquisition.
+ // This avoids leaking a daemon thread if the fail-fast path throws, and
+ // keeps the require() guard valid for retry scenarios.
+ renewalExecutor =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("user-credential-renewal")
+
+ // Schedule first renewal
+ val renewalDelay = computeRenewalDelay(ctx, earliestExpiry)
+ scheduleRenewal(renewalDelay)
+
+ logInfo(log"Credential acquisition successful. Next renewal in " +
+ log"${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(renewalDelay))}.")
+ serialized
+ }
+
+ def stop(): Unit = {
Review Comment:
`UserCredentialManager` follows the same lifecycle as `HadoopDelegationTokenManager`. Both are started from `CoarseGrainedSchedulerBackend.start()` and stopped from `CoarseGrainedSchedulerBackend.stop()`. Even when `stop()` is invoked from a different thread (via `SparkContext.stopInNewThread()`), it is always called after `start()` has completed. The `Thread.start()` in `stopInNewThread()` establishes a happens-before relationship that guarantees visibility of the `renewalExecutor` field without `@volatile`.
So, I left it as is without adding the `stopped` flag and removed `@volatile` from `renewalExecutor`.
This matches `HadoopDelegationTokenManager`, which also uses a plain `private var` for its executor field with no `@volatile` or `stopped` flag. I've kept the same pattern for consistency.
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement `UserCredentialManager` [spark]
Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on code in PR #57387:
URL: https://github.com/apache/spark/pull/57387#discussion_r3667140781
##########
core/src/main/java/org/apache/spark/security/CredentialProviderLoader.java:
##########
@@ -221,10 +223,38 @@ private static List<CredentialProvider> loadProviders() {
return result;
}
+ /**
+ * Discovers all URI schemes supported by providers on the classpath.
+ * <p>
+ * This method queries all discovered {@link CredentialProvider} instances and collects
+ * their {@link CredentialProvider#supportedSchemes()} into a single set. Unlike
+ * {@link #providerFor(String, Map)}, this does not initialize providers or apply
+ * explicit selection rules -- it only reports what schemes are potentially available.
+ * <p>
+ * Intended for use by {@code UserCredentialManager} when no explicit scheme configuration
+ * (e.g., {@code spark.security.credentials.provider.<scheme>}) is provided.
+ *
+ * @return a set of all supported scheme names (lowercased), possibly empty
+ */
+ public static Set<String> discoverAllSchemes() {
+ List<CredentialProvider> providers = getProviders();
+ Set<String> schemes = new java.util.HashSet<>();
Review Comment:
Oh... mistakenly thought I had already fixed it. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement UserCredentialManager [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #57387:
URL: https://github.com/apache/spark/pull/57387#discussion_r3666897856
##########
core/src/main/scala/org/apache/spark/internal/config/package.scala:
##########
@@ -1690,6 +1690,46 @@ package object config {
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("1h")
+ private[spark] val SECURITY_CREDENTIALS_ENABLED =
+ ConfigBuilder("spark.security.oidc.enabled")
+ .doc("Whether to enable OIDC credential propagation. When enabled, the driver reads an " +
+ "identity token from a file, exchanges it for short-lived service credentials via " +
+ "CredentialProvider implementations, and propagates those credentials to executors.")
+ .version("4.3.0")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+ .booleanConf
+ .createWithDefault(false)
+
+ private[spark] val SECURITY_CREDENTIALS_IDENTITY_TOKEN_FILE =
Review Comment:
```suggestion
private[spark] val SECURITY_OIDC_IDENTITY_TOKEN_FILE =
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement `UserCredentialManager` [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #57387:
URL: https://github.com/apache/spark/pull/57387#discussion_r3667025658
##########
core/src/test/scala/org/apache/spark/deploy/security/UserCredentialManagerSuite.scala:
##########
@@ -0,0 +1,474 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.security
+
+import java.time.Instant
+import java.util
+import java.util.Optional
+import java.util.concurrent.atomic.{AtomicInteger, AtomicReference}
+
+import org.apache.spark.{SparkConf, SparkFunSuite}
+import org.apache.spark.internal.config._
+import org.apache.spark.security._
+
+class UserCredentialManagerSuite extends SparkFunSuite {
+
+ override def beforeEach(): Unit = {
+ super.beforeEach()
+ CredentialProviderLoader.resetForTesting()
+ }
+
+ private def createSparkConf(): SparkConf = {
+ new SparkConf(loadDefaults = false)
+ .set(SECURITY_CREDENTIALS_ENABLED, true)
+ .set(SECURITY_CREDENTIALS_IDENTITY_TOKEN_FILE, "/tmp/fake-token")
+ .set(SECURITY_CREDENTIALS_RENEWAL_SAFETY_MARGIN, 5000L) // 5s for tests
+ .set(SECURITY_CREDENTIALS_RENEWAL_MIN_INTERVAL, 1000L) // 1s for tests
+ }
+
+ private def createUserContext(
+ expiresInSeconds: Long = 300): UserContext = {
+ new UserContext(
+ "test-user",
+ "https://issuer.example.com",
+ "fake.jwt.token",
+ Instant.now(),
+ Instant.now().plusSeconds(expiresInSeconds))
+ }
+
+ private def createIngestor(ctx: UserContext): TokenIngestor = {
+ new TokenIngestor {
+ override def load(): Optional[UserContext] = Optional.of(ctx)
+ }
+ }
+
+ private def createFailingIngestor(): TokenIngestor = {
+ new TokenIngestor {
+ override def load(): Optional[UserContext] = Optional.empty()
+ }
+ }
+
+ test("start() acquires initial credentials and invokes callback") {
+ val conf = createSparkConf()
+ val ctx = createUserContext()
+ val callbackRef = new AtomicReference[Array[Byte]]()
+
+ // Use CredentialProviderLoader with the FakeCredentialProvider from ServiceLoader
+ // FakeCredentialProvider supports scheme "fake"
+ conf.set("spark.security.oidc.provider.fake",
+ "org.apache.spark.security.FakeCredentialProvider")
+
+ val manager = new UserCredentialManager(
+ conf,
+ createIngestor(ctx),
+ bytes => callbackRef.set(bytes))
+
+ try {
+ val result = manager.start()
+ assert(result != null, "start() should return serialized credentials")
+ assert(callbackRef.get() != null, "callback should have been invoked")
+
+ // Verify deserialization
+ val creds = UserCredentialManager.deserializeUserCredentials(result)
+ assert(creds.forScheme("fake").isPresent,
+ "Should have credentials for 'fake' scheme")
+ assert(creds.forScheme("fake").get().getProperties.get("provider") === "fake")
+ } finally {
+ manager.stop()
+ }
+ }
+
+ test("start() throws when TokenIngestor returns empty (fail-fast)") {
+ val conf = createSparkConf()
+ conf.set("spark.security.oidc.provider.fake",
+ "org.apache.spark.security.FakeCredentialProvider")
+
+ val manager = new UserCredentialManager(
+ conf,
+ createFailingIngestor(),
+ _ => ())
+
+ try {
+ val ex = intercept[IllegalStateException] {
+ manager.start()
+ }
+ assert(ex.getMessage.contains("identity token file is missing or malformed"))
+ } finally {
+ manager.stop()
+ }
+ }
+
+ test("serialization round-trip of UserCredentials") {
+ val props = new util.HashMap[String, String]()
+ props.put("fs.s3a.access.key", "AKIAEXAMPLE")
+ props.put("fs.s3a.secret.key", "secret123")
+ props.put("fs.s3a.session.token", "token456")
+ val cred = new ServiceCredential(props, Instant.now().plusSeconds(3600))
+
+ val credsMap = new util.HashMap[String, ServiceCredential]()
+ credsMap.put("s3a", cred)
+ val original = new UserCredentials(credsMap)
+
+ val conf = createSparkConf()
+ val manager = new UserCredentialManager(conf, createFailingIngestor(), _ => ())
+
+ try {
+ val serialized = UserCredentialManager.serializeUserCredentials(original)
+ val deserialized = UserCredentialManager.deserializeUserCredentials(serialized)
+
+ assert(deserialized.forScheme("s3a").isPresent)
+ val restored = deserialized.forScheme("s3a").get()
+ assert(restored.getProperties.get("fs.s3a.access.key") === "AKIAEXAMPLE")
+ assert(restored.getProperties.get("fs.s3a.secret.key") === "secret123")
+ assert(restored.getProperties.get("fs.s3a.session.token") === "token456")
+ assert(restored.getExpiresAt === cred.getExpiresAt)
+ } finally {
+ manager.stop()
+ }
+ }
+
+ test("deserialization rejects unauthorized classes (ObjectInputFilter)") {
+ // Serialize a class that is NOT in the allowed filter pattern.
+ // The filter allows org.apache.spark.security.**, java.util.**, java.time.**,
+ // java.lang.** but blocks everything else (including java.io.File).
+ val bos = new java.io.ByteArrayOutputStream()
+ val oos = new java.io.ObjectOutputStream(bos)
+ oos.writeObject(new java.io.File("/tmp/malicious"))
+ oos.close()
+
+ val ex = intercept[java.io.InvalidClassException] {
+ UserCredentialManager.deserializeUserCredentials(bos.toByteArray)
+ }
+ // ObjectInputFilter rejects classes not in the allowlist
+ assert(ex.getMessage.contains("REJECTED") || ex.getMessage.contains("filter"),
+ s"Expected filter rejection, got: ${ex.getMessage}")
+ }
+
+ test("computeRenewalDelay respects safetyMargin and minInterval") {
+ val conf = createSparkConf()
+ .set(SECURITY_CREDENTIALS_RENEWAL_SAFETY_MARGIN, 10000L) // 10s
+ .set(SECURITY_CREDENTIALS_RENEWAL_MIN_INTERVAL, 5000L) // 5s
+
+ val manager = new UserCredentialManager(conf, createFailingIngestor(), _ => ())
+ try {
+ // Token expires in 60s, credential expires in 30s
+ // Expected: min(60s, 30s) - 10s = 20s
+ val ctx = createUserContext(expiresInSeconds = 60)
+ val credExpiry = Some(Instant.now().plusSeconds(30))
+ val delay = manager.computeRenewalDelay(ctx, credExpiry)
+
+ // Allow 1s tolerance for timing
+ assert(delay >= 19000 && delay <= 21000,
+ s"Expected ~20000ms, got ${delay}ms")
+ } finally {
+ manager.stop()
+ }
+ }
+
+ test("computeRenewalDelay uses minInterval when expiry is very close") {
+ val conf = createSparkConf()
+ .set(SECURITY_CREDENTIALS_RENEWAL_SAFETY_MARGIN, 10000L)
+ .set(SECURITY_CREDENTIALS_RENEWAL_MIN_INTERVAL, 5000L)
+
+ val manager = new UserCredentialManager(conf, createFailingIngestor(), _ => ())
+ try {
+ // Token expires in 5s, safetyMargin is 10s -> computed delay would be negative
+ // Should be bounded by minInterval (5s)
+ val ctx = createUserContext(expiresInSeconds = 5)
+ val credExpiry = Some(Instant.now().plusSeconds(5))
+ val delay = manager.computeRenewalDelay(ctx, credExpiry)
+
+ assert(delay === 5000L, s"Expected minInterval (5000ms), got ${delay}ms")
+ } finally {
+ manager.stop()
+ }
+ }
+
+ test("computeRenewalDelay uses identity token expiry when credential has no expiry") {
+ val conf = createSparkConf()
+ .set(SECURITY_CREDENTIALS_RENEWAL_SAFETY_MARGIN, 10000L) // 10s
+ .set(SECURITY_CREDENTIALS_RENEWAL_MIN_INTERVAL, 5000L) // 5s
+
+ val manager = new UserCredentialManager(conf, createFailingIngestor(), _ => ())
+ try {
+ // Token expires in 60s, no credential expiry
+ // Expected: 60s - 10s = 50s
+ val ctx = createUserContext(expiresInSeconds = 60)
+ val delay = manager.computeRenewalDelay(ctx, None)
+
+ assert(delay >= 49000 && delay <= 51000,
+ s"Expected ~50000ms, got ${delay}ms")
+ } finally {
+ manager.stop()
+ }
+ }
+
+ test("computeRenewalDelay returns default when no expiry information") {
+ val conf = createSparkConf()
+ val manager = new UserCredentialManager(conf, createFailingIngestor(), _ => ())
+ try {
+ // UserContext with null expiresAt
+ val ctx = new UserContext(
+ "test-user", "https://issuer.example.com", "fake.jwt.token",
+ Instant.now(), null)
+ val delay = manager.computeRenewalDelay(ctx, None)
+
+ // Should be DEFAULT_RENEWAL_INTERVAL_NO_EXPIRY_MS = 7 minutes = 420000ms
+ assert(delay === 420000L, s"Expected 420000ms (7 min), got ${delay}ms")
+ } finally {
+ manager.stop()
+ }
+ }
+
+ test("computeBackoffDelay increases exponentially") {
+ val conf = createSparkConf()
+ .set(SECURITY_CREDENTIALS_RENEWAL_MIN_INTERVAL, 1000L)
+
+ val manager = new UserCredentialManager(conf, createFailingIngestor(), _ => ())
+ try {
+ val failuresField = classOf[UserCredentialManager].getDeclaredField("consecutiveFailures")
+ failuresField.setAccessible(true)
+
+ // First failure: base = minInterval * 2^0 = 1000ms
+ failuresField.setInt(manager, 1)
+ val delay1 = manager.computeBackoffDelay()
+ assert(delay1 >= 1000 && delay1 <= 1200,
+ s"First backoff should be ~1000-1100ms, got ${delay1}ms")
+
+ // Second failure: base = minInterval * 2^1 = 2000ms
+ failuresField.setInt(manager, 2)
+ val delay2 = manager.computeBackoffDelay()
+ assert(delay2 >= 2000 && delay2 <= 2300,
+ s"Second backoff should be ~2000-2200ms, got ${delay2}ms")
+
+ // Third failure: base = minInterval * 2^2 = 4000ms
+ failuresField.setInt(manager, 3)
+ val delay3 = manager.computeBackoffDelay()
+ assert(delay3 >= 4000 && delay3 <= 4500,
+ s"Third backoff should be ~4000-4400ms, got ${delay3}ms")
+ } finally {
+ manager.stop()
+ }
+ }
+
+ test("computeBackoffDelay is capped at maxBackoffMs") {
+ val conf = createSparkConf()
+ .set(SECURITY_CREDENTIALS_RENEWAL_MIN_INTERVAL, 1000L)
+
+ val manager = new UserCredentialManager(conf, createFailingIngestor(), _ => ())
+ try {
+ val failuresField = classOf[UserCredentialManager].getDeclaredField("consecutiveFailures")
+ failuresField.setAccessible(true)
+
+ // Many failures: should be capped at 10 minutes (600000ms)
+ failuresField.setInt(manager, 20)
+ val delay = manager.computeBackoffDelay()
+ assert(delay <= 660000L, // 600000 + 10% jitter
+ s"Backoff should be capped at ~600000ms + jitter, got ${delay}ms")
+ } finally {
+ manager.stop()
+ }
+ }
+
+ test("computeBackoffDelay handles zero consecutiveFailures defensively") {
+ val conf = createSparkConf()
+ .set(SECURITY_CREDENTIALS_RENEWAL_MIN_INTERVAL, 1000L)
+
+ val manager = new UserCredentialManager(conf, createFailingIngestor(), _ => ())
+ try {
+ val failuresField = classOf[UserCredentialManager].getDeclaredField("consecutiveFailures")
+ failuresField.setAccessible(true)
+
+ // Edge case: 0 failures (should not happen in practice, but defensive)
+ failuresField.setInt(manager, 0)
+ val delay = manager.computeBackoffDelay()
+ // shiftAmount = max(0, min(0-1, 6)) = max(0, -1) = 0
+ // baseDelay = 1000 * 2^0 = 1000
+ assert(delay >= 1000 && delay <= 1200,
+ s"With 0 failures, backoff should be ~1000ms, got ${delay}ms")
+ } finally {
+ manager.stop()
+ }
+ }
+
+ test("UserCredentialManager.create returns None when disabled") {
+ val conf = new SparkConf(loadDefaults = false)
+ .set(SECURITY_CREDENTIALS_ENABLED, false)
+
+ val result = UserCredentialManager.create(conf, _ => ())
+ assert(result.isEmpty)
+ }
+
+ test("UserCredentialManager.create returns Some when enabled with valid config") {
+ val conf = createSparkConf()
+ val result = UserCredentialManager.create(conf, _ => ())
+ assert(result.isDefined)
+ }
+
+ test("UserCredentialManager.create throws when enabled without token file") {
+ val conf = new SparkConf(loadDefaults = false)
+ .set(SECURITY_CREDENTIALS_ENABLED, true)
+ // Deliberately not setting SECURITY_CREDENTIALS_IDENTITY_TOKEN_FILE
+
+ val ex = intercept[IllegalArgumentException] {
+ UserCredentialManager.create(conf, _ => ())
+ }
+ assert(ex.getMessage.contains("spark.security.oidc.identityToken.file"))
+ }
+
+ test("renewal is scheduled after successful credential acquisition") {
+ val conf = createSparkConf()
+ conf.set("spark.security.oidc.provider.fake",
+ "org.apache.spark.security.FakeCredentialProvider")
+ val ctx = createUserContext(expiresInSeconds = 60)
+ var callbackCount = 0
+
+ val manager = new UserCredentialManager(
+ conf,
+ createIngestor(ctx),
+ _ => { callbackCount += 1 })
+
+ try {
+ val result = manager.start()
+ assert(result != null)
+ assert(callbackCount === 1, "callback should be invoked once on start")
+ } finally {
+ manager.stop()
+ }
+ }
+
+ test("stop() after start() does not throw") {
+ val conf = createSparkConf()
+ conf.set("spark.security.oidc.provider.fake",
+ "org.apache.spark.security.FakeCredentialProvider")
+ val ctx = createUserContext()
+
+ val manager = new UserCredentialManager(conf, createIngestor(ctx), _ => ())
+ manager.start()
+ // Should not throw
+ manager.stop()
+ }
+
+ test("per-provider error isolation: one provider failure does not block others") {
+ // Configure two schemes: "fake" (will succeed) and "shared" (will trigger ambiguity
+ // because no explicit provider is set for it and both FakeCredentialProvider and
+ // AnotherFakeCredentialProvider support "shared").
+ // The per-provider catch should absorb the IllegalArgumentException for "shared"
+ // and still return credentials from "fake".
+ val conf = createSparkConf()
+ // Only set explicit provider for "fake", leave "shared" ambiguous
+ conf.set("spark.security.oidc.provider.fake",
+ "org.apache.spark.security.FakeCredentialProvider")
+ conf.set("spark.security.oidc.provider.shared", "nonexistent.Provider")
+
+ val ctx = createUserContext()
+ val callbackRef = new AtomicReference[Array[Byte]]()
+
+ val manager = new UserCredentialManager(
+ conf,
+ createIngestor(ctx),
+ bytes => callbackRef.set(bytes))
+
+ try {
+ val result = manager.start()
+ assert(result != null, "start() should return serialized credentials")
+
+ // Verify that "fake" credentials were resolved despite "shared" failing
+ val creds = UserCredentialManager.deserializeUserCredentials(result)
+ assert(creds.forScheme("fake").isPresent,
+ "Should have credentials for 'fake' scheme even though 'shared' failed")
+ assert(creds.forScheme("fake").get().getProperties.get("provider") === "fake")
+ // "shared" should NOT be present (its provider was not found)
+ assert(!creds.forScheme("shared").isPresent,
+ "'shared' scheme should not have credentials due to provider resolution failure")
+ } finally {
+ manager.stop()
+ }
+ }
+
+ test("per-provider error isolation: all providers fail throws IllegalStateException") {
+ // Configure only "shared" with a non-existent provider class
+ val conf = createSparkConf()
+ conf.set("spark.security.oidc.provider.shared", "nonexistent.Provider")
+
+ val ctx = createUserContext()
+
+ val manager = new UserCredentialManager(conf, createIngestor(ctx), _ => ())
+
+ try {
+ val ex = intercept[IllegalStateException] {
+ manager.start()
+ }
+ assert(ex.getMessage.contains("No credential providers resolved any credentials"))
+ } finally {
+ manager.stop()
+ }
+ }
+
+ test("token rotation triggers new credential acquisition and propagation") {
+ val conf = createSparkConf()
+ .set(SECURITY_CREDENTIALS_RENEWAL_SAFETY_MARGIN, 1000L) // 1s
+ .set(SECURITY_CREDENTIALS_RENEWAL_MIN_INTERVAL, 500L) // 0.5s
+ conf.set("spark.security.oidc.provider.fake",
+ "org.apache.spark.security.FakeCredentialProvider")
+
+ // First token: user-A
+ val ctx1 = new UserContext(
+ "user-A", "https://issuer.example.com", "token-A",
+ Instant.now(), Instant.now().plusSeconds(2)) // expires in 2s
+
+ // Second token: user-B (simulating rotation)
+ val ctx2 = new UserContext(
+ "user-B", "https://issuer.example.com", "token-B",
+ Instant.now(), Instant.now().plusSeconds(300))
+
+ // TokenIngestor that returns ctx1 initially, then ctx2 after first call
+ val callCount = new AtomicInteger(0)
+ val rotatingIngestor = new TokenIngestor {
+ override def load(): Optional[UserContext] = {
+ if (callCount.getAndIncrement() == 0) Optional.of(ctx1)
+ else Optional.of(ctx2)
+ }
+ }
+
+ val callbacks = new java.util.concurrent.CopyOnWriteArrayList[Array[Byte]]()
+ val manager = new UserCredentialManager(
+ conf,
+ rotatingIngestor,
+ bytes => callbacks.add(bytes))
+
+ try {
+ val initial = manager.start()
+ assert(initial != null)
+ assert(callbacks.size() === 1, "Should have one callback from start()")
+
+ // Wait for renewal to fire (token expires in 2s, safety margin 1s -> renews after ~1s)
+ Thread.sleep(2000)
Review Comment:
Please use `eventually` because our GitHub Actions is very slow sometimes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement `UserCredentialManager` [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #57387:
URL: https://github.com/apache/spark/pull/57387#issuecomment-5120033051
Thank you, @sarutak and all!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement `UserCredentialManager` [spark]
Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on code in PR #57387:
URL: https://github.com/apache/spark/pull/57387#discussion_r3667140781
##########
core/src/main/java/org/apache/spark/security/CredentialProviderLoader.java:
##########
@@ -221,10 +223,38 @@ private static List<CredentialProvider> loadProviders() {
return result;
}
+ /**
+ * Discovers all URI schemes supported by providers on the classpath.
+ * <p>
+ * This method queries all discovered {@link CredentialProvider} instances and collects
+ * their {@link CredentialProvider#supportedSchemes()} into a single set. Unlike
+ * {@link #providerFor(String, Map)}, this does not initialize providers or apply
+ * explicit selection rules -- it only reports what schemes are potentially available.
+ * <p>
+ * Intended for use by {@code UserCredentialManager} when no explicit scheme configuration
+ * (e.g., {@code spark.security.credentials.provider.<scheme>}) is provided.
+ *
+ * @return a set of all supported scheme names (lowercased), possibly empty
+ */
+ public static Set<String> discoverAllSchemes() {
+ List<CredentialProvider> providers = getProviders();
+ Set<String> schemes = new java.util.HashSet<>();
Review Comment:
Oh... mistakenly thought I had already it. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement `UserCredentialManager` [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #57387:
URL: https://github.com/apache/spark/pull/57387#issuecomment-5106372820
To @sarutak , I finished another review round. Please address newly added comments and the existing one, https://github.com/apache/spark/pull/57387#discussion_r3667035666 .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement UserCredentialManager [spark]
Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on code in PR #57387:
URL: https://github.com/apache/spark/pull/57387#discussion_r3665825665
##########
core/src/main/scala/org/apache/spark/deploy/security/UserCredentialManager.scala:
##########
@@ -0,0 +1,443 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.security
+
+import java.io.{ByteArrayOutputStream, ObjectInputFilter, ObjectInputStream, ObjectOutputStream}
+import java.net.URI
+import java.nio.file.Paths
+import java.time.Instant
+import java.util.concurrent.{RejectedExecutionException, ScheduledExecutorService, TimeUnit}
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.LogKeys
+import org.apache.spark.internal.config._
+import org.apache.spark.security._
+import org.apache.spark.ui.UIUtils
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Manager for OIDC-based credential propagation on the driver.
+ *
+ * This class is a sibling of [[HadoopDelegationTokenManager]] and handles the OIDC credential
+ * propagation path independently. Both managers run on independent threads and can both be
+ * active simultaneously (e.g., a cluster accessing HDFS via Kerberos and S3 via OIDC).
+ *
+ * Responsibilities:
+ * 1. Reads the current identity token via a [[TokenIngestor]]
+ * 2. Calls [[CredentialProvider#resolve]] for each configured scheme
+ * 3. Serializes the resulting [[UserCredentials]] and invokes the propagation callback
+ * 4. Schedules renewal based on `min(identity token expiry, service credential expiry) -
+ * safetyMargin`
+ * 5. Retries with exponential backoff on failure
+ *
+ * Intended to be started from `CoarseGrainedSchedulerBackend.start()` when
+ * `spark.security.credentials.enabled=true`, independently of
+ * `UserGroupInformation.isSecurityEnabled()`.
+ *
+ * Lifecycle: call `start()` exactly once, then `stop()` to shut down.
+ * Calling `start()` after `stop()` is not supported.
+ */
+private[spark] class UserCredentialManager(
+ sparkConf: SparkConf,
+ tokenIngestor: TokenIngestor,
+ onCredentialsUpdate: Array[Byte] => Unit) extends Logging {
+
+ private val safetyMargin = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_SAFETY_MARGIN)
+ private val minInterval = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_MIN_INTERVAL)
+
+ // Thread-safe counter for exponential backoff calculation.
+ // Accessed from both the caller of start() and the scheduled renewal thread.
+ private val consecutiveFailures = new AtomicInteger(0)
+ private val maxBackoffMs: Long = UserCredentialManager.MAX_BACKOFF_MS
+
+ @volatile private var renewalExecutor: ScheduledExecutorService = _
+
+ // Pre-compute the filtered credential configuration map (SparkConf is immutable).
+ private val credentialConfMap: java.util.Map[String, String] = sparkConf.getAll
+ .filter(_._1.startsWith("spark.security.credentials."))
+ .toMap.asJava
+
+ /**
+ * Start the credential manager. Acquires initial credentials and schedules renewal.
+ *
+ * The initial credential acquisition is fail-fast: if the token cannot be loaded or
+ * no credentials can be resolved, an exception is thrown. Subsequent renewal failures
+ * are handled with exponential backoff.
+ *
+ * @return The serialized initial [[UserCredentials]].
+ * @throws IllegalStateException if the initial credential acquisition fails.
+ */
+ def start(): Array[Byte] = {
+ require(renewalExecutor == null, "start() must not be called more than once")
+
+ // Initial acquisition is fail-fast (no retry/backoff).
+ // This ensures the application fails to start if credentials cannot be obtained,
+ // rather than running with null credentials.
+ val userContext = tokenIngestor.load()
+ if (!userContext.isPresent) {
+ throw new IllegalStateException(
+ "Failed to start UserCredentialManager: identity token file is missing or malformed. " +
+ s"Check ${SECURITY_CREDENTIALS_IDENTITY_TOKEN_FILE.key} configuration.")
+ }
+
+ val ctx = userContext.get()
+ logInfo(log"Initial identity token loaded for principal " +
+ log"${MDC(LogKeys.PRINCIPAL, ctx.getPrincipal)} " +
+ log"(issuer: ${MDC(LogKeys.URI, ctx.getIssuer)})")
+
+ val (credentials, earliestExpiry) = resolveCredentials(ctx)
+ val serialized = serializeUserCredentials(credentials)
+
+ // Propagate initial credentials
+ onCredentialsUpdate(serialized)
+
+ // Create the renewal executor only after successful initial acquisition.
+ // This avoids leaking a daemon thread if the fail-fast path throws, and
+ // keeps the require() guard valid for retry scenarios.
+ renewalExecutor =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("user-credential-renewal")
+
+ // Schedule first renewal
+ val renewalDelay = computeRenewalDelay(ctx, earliestExpiry)
+ scheduleRenewal(renewalDelay)
+
+ logInfo(log"Credential acquisition successful. Next renewal in " +
+ log"${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(renewalDelay))}.")
+ serialized
+ }
+
+ def stop(): Unit = {
+ if (renewalExecutor != null) {
+ renewalExecutor.shutdownNow()
+ }
+ }
+
+ /**
+ * Scheduled renewal task: load identity token, resolve credentials, propagate,
+ * and schedule the next renewal. Failures are retried with exponential backoff.
+ */
+ private def renewCredentialsTask(): Unit = {
+ try {
+ val userContext = tokenIngestor.load()
+ if (!userContext.isPresent) {
+ throw new IllegalStateException(
+ "TokenIngestor returned empty - identity token file may be missing or malformed")
+ }
+
+ val ctx = userContext.get()
+ logInfo(log"Loaded identity token for principal " +
+ log"${MDC(LogKeys.PRINCIPAL, ctx.getPrincipal)} " +
+ log"(issuer: ${MDC(LogKeys.URI, ctx.getIssuer)})")
+
+ val (credentials, earliestExpiry) = resolveCredentials(ctx)
+ val serialized = serializeUserCredentials(credentials)
+
+ // Propagate credentials to executors. Errors here are logged separately
+ // so that credential-fetch success is not conflated with distribution failure.
+ try {
+ onCredentialsUpdate(serialized)
+ } catch {
+ case e: Exception =>
+ logWarning(log"Credentials were resolved successfully but failed to propagate " +
+ log"to executors. Executors will receive updated credentials on next renewal.", e)
+ }
+
+ // Reset backoff on successful credential resolution. This is intentionally done
+ // even if onCredentialsUpdate failed above: the backoff counter tracks credential
+ // *resolution* failures (STS/provider issues), not propagation failures.
+ // Propagation failures are transient and will be retried on next scheduled renewal.
+ consecutiveFailures.set(0)
+
+ // Schedule next renewal
+ val renewalDelay = computeRenewalDelay(ctx, earliestExpiry)
+ scheduleRenewal(renewalDelay)
+
+ logInfo(log"Credential renewal successful. Next renewal in " +
+ log"${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(renewalDelay))}.")
+ } catch {
+ case _: InterruptedException =>
+ // Shutting down, ignore
+ case e: Exception =>
+ val failures = consecutiveFailures.incrementAndGet()
+ val delay = computeBackoffDelay()
+ logWarning(log"Failed to renew user credentials (attempt " +
+ log"${MDC(LogKeys.NUM_RETRY, failures)}), " +
+ log"will retry in ${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(delay))}.", e)
+ scheduleRenewal(delay)
+ }
+ }
+
+ /**
+ * Resolve credentials for all schemes that have registered providers.
+ *
+ * @return Tuple of (UserCredentials, earliest expiry across all service credentials)
+ */
+ private def resolveCredentials(
+ ctx: UserContext): (UserCredentials, Option[Instant]) = {
+ val schemes = discoverSchemes()
+
+ val credentialMap = new mutable.HashMap[String, ServiceCredential]()
+ var earliestExpiry: Option[Instant] = None
+
+ for (scheme <- schemes) {
+ try {
+ val providerOpt = CredentialProviderLoader.providerFor(scheme, credentialConfMap)
+ if (providerOpt.isPresent) {
+ val provider = providerOpt.get()
+ // Use a synthetic target URI with just the scheme for initial resolution.
+ // The full URI is passed in future versions when path-specific resolution is needed.
+ // The "synthetic" authority signals this is not a real endpoint but a placeholder
+ // for scheme-based provider selection.
+ val target = new URI(scheme, UserCredentialManager.SYNTHETIC_TARGET_AUTHORITY,
+ "/", null, null)
+ val credential = provider.resolve(ctx, target)
+ credentialMap.put(scheme, credential)
+
+ val expiry = credential.getExpiresAt
+ if (expiry != null) {
+ earliestExpiry = earliestExpiry match {
+ case Some(existing) if existing.isBefore(expiry) => Some(existing)
+ case _ => Some(expiry)
+ }
+ }
+ } else {
+ logWarning(log"No credential provider found for scheme " +
+ log"${MDC(LogKeys.URI, scheme)}. Skipping.")
+ }
+ } catch {
+ case e: Exception =>
+ logWarning(log"Failed to resolve credentials for scheme " +
+ log"${MDC(LogKeys.URI, scheme)}. Skipping this provider.", e)
+ }
+ }
+
+ if (credentialMap.isEmpty) {
+ throw new IllegalStateException(
+ "No credential providers resolved any credentials. " +
+ "Check that providers are on the classpath and configured correctly.")
+ }
+
+ (new UserCredentials(credentialMap.asJava), earliestExpiry)
+ }
+
+ /**
+ * Discover all schemes that have at least one registered provider.
+ *
+ * Schemes are determined by explicit configuration keys of the form
+ * `spark.security.credentials.provider.<scheme>`. If no explicit configuration
+ * exists, the method queries `CredentialProviderLoader` to discover all providers
+ * registered via ServiceLoader and collects their supported schemes.
+ *
+ * Note: When using auto-discovery (no explicit config), multiple providers may
+ * support the same scheme. In that case, `CredentialProviderLoader.providerFor`
+ * will throw an `IllegalArgumentException` for that scheme. The caller
+ * (`resolveCredentials`) handles this gracefully via per-provider exception catching,
+ * logging a warning and continuing with remaining schemes.
+ */
+ private def discoverSchemes(): Set[String] = {
+ // Extract explicitly configured scheme -> provider mappings
+ val explicitSchemes = credentialConfMap.asScala
+ .filter { case (k, _) => k.startsWith("spark.security.credentials.provider.") }
+ .map { case (k, _) => k.stripPrefix("spark.security.credentials.provider.") }
+ .toSet
+
+ if (explicitSchemes.nonEmpty) {
+ explicitSchemes
+ } else {
+ // No explicit scheme configuration. Discover all schemes that have a provider
+ // available on the classpath by probing CredentialProviderLoader.
+ // This covers both built-in providers (e.g., connector/credential-aws for "s3a")
+ // and third-party providers registered via ServiceLoader.
+ CredentialProviderLoader.discoverAllSchemes().asScala.toSet
+ }
+ }
+
+ /**
+ * Compute the delay until next renewal.
+ * Uses min(identity token expiry, service credential expiry) - safetyMargin,
+ * bounded below by minInterval.
+ */
+ private[security] def computeRenewalDelay(
+ ctx: UserContext,
+ earliestCredentialExpiry: Option[Instant]): Long = {
+ val now = System.currentTimeMillis()
+
+ // Consider identity token expiry
+ val tokenExpiry: Option[Long] = Option(ctx.getExpiresAt).map(_.toEpochMilli)
+
+ // Consider earliest service credential expiry
+ val credExpiry: Option[Long] = earliestCredentialExpiry.map(_.toEpochMilli)
+
+ // Take the minimum of both
+ val effectiveExpiry: Option[Long] = (tokenExpiry, credExpiry) match {
+ case (Some(t), Some(c)) => Some(math.min(t, c))
+ case (Some(t), None) => Some(t)
+ case (None, Some(c)) => Some(c)
+ case (None, None) => None
+ }
+
+ effectiveExpiry match {
+ case Some(expiry) =>
+ math.max(expiry - now - safetyMargin, minInterval)
+ case None =>
+ // No expiry information available from either the identity token or service
+ // credentials. Use half of the default suggested TTL (15 min / 2 = 7.5 min,
+ // rounded down) as a conservative polling interval. This ensures credentials
+ // are refreshed even when providers don't report expiration times.
+ UserCredentialManager.DEFAULT_RENEWAL_INTERVAL_NO_EXPIRY_MS
+ }
+ }
+
+ /**
+ * Compute backoff delay using exponential backoff with jitter.
+ * Bounded by minInterval (floor) and maxBackoffMs (ceiling).
+ */
+ private[security] def computeBackoffDelay(): Long = {
+ // Guard against negative or zero shift amounts. consecutiveFailures should always
+ // be >= 1 when this is called (incremented before calling), but we protect against
+ // edge cases defensively.
+ val shiftAmount = math.max(0, math.min(consecutiveFailures.get() - 1, 6))
+ val baseDelay = minInterval * (1L << shiftAmount)
+ val cappedDelay = math.min(baseDelay, maxBackoffMs)
+ // Add 10% jitter to avoid thundering herd on recovery
+ val jitter = (cappedDelay * 0.1 * math.random()).toLong
+ math.max(cappedDelay + jitter, minInterval)
+ }
+
+ private def scheduleRenewal(delay: Long): Unit = {
+ try {
+ val renewalTask = new Runnable {
+ override def run(): Unit = {
+ renewCredentialsTask()
+ }
+ }
+ renewalExecutor.schedule(renewalTask, delay, TimeUnit.MILLISECONDS)
+ } catch {
+ case _: RejectedExecutionException =>
+ // Executor has been shut down (e.g., stop() called concurrently). This is expected
+ // during application shutdown -- no further renewals will be scheduled.
+ logDebug(log"Renewal scheduling rejected - executor is shut down.")
+ }
+ }
+
+ /**
+ * Serialize [[UserCredentials]] to a byte array using Java serialization.
+ */
+ private[security] def serializeUserCredentials(credentials: UserCredentials): Array[Byte] = {
+ val bos = new ByteArrayOutputStream()
+ try {
+ val oos = new ObjectOutputStream(bos)
+ try {
+ oos.writeObject(credentials)
+ oos.flush()
+ } finally {
+ oos.close()
+ }
+ bos.toByteArray
+ } finally {
+ bos.close()
+ }
+ }
+}
+
+private[spark] object UserCredentialManager {
+
+ /**
+ * Synthetic authority used in target URIs for scheme-based provider resolution.
+ * Providers should not rely on this value; it signals that no specific endpoint
+ * is targeted and only the URI scheme is meaningful.
+ */
+ private val SYNTHETIC_TARGET_AUTHORITY = "synthetic"
+
+ /**
+ * Maximum backoff delay between credential renewal retry attempts.
+ * Caps the exponential backoff to prevent excessively long gaps between retries.
+ */
+ private val MAX_BACKOFF_MS: Long = TimeUnit.MINUTES.toMillis(10)
+
+ /**
+ * Default renewal interval when neither the identity token nor service credentials
+ * report an expiration time. Set to 7 minutes (half of the default 15-minute
+ * suggested TTL from CredentialProvider.suggestedTtl()).
+ */
+ private val DEFAULT_RENEWAL_INTERVAL_NO_EXPIRY_MS: Long = TimeUnit.MINUTES.toMillis(7)
+
+ /**
+ * ObjectInputFilter pattern restricting deserialization to only the classes needed
+ * for UserCredentials. This prevents deserialization gadget chain attacks while
+ * allowing Java's built-in collection internal classes and arrays that HashMap uses.
+ */
+ private val DESERIALIZATION_FILTER: String =
Review Comment:
Added `maxarray=1000` and `maxrefs=1000` to the filter. These limits prevent resource exhaustion from forged streams while remaining well within the needs of a typical `UserCredentials` graph (a few schemes × a few properties each).
Regarding narrowing `java.util.**` and `java.lang.**` to specific classes. I investigated this but `HashMap.readObject()` internally allocates `Node[]` arrays whose types need to pass the filter's `checkArray` call. Enumerating all internal HashMap implementation classes is fragile across JDK versions, so I kept the package wildcards and rely on `maxarray`/`maxrefs`/`maxdepth` + the `!*` deny-all to bound the attack surface.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement UserCredentialManager [spark]
Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on code in PR #57387:
URL: https://github.com/apache/spark/pull/57387#discussion_r3642992634
##########
core/src/main/java/org/apache/spark/security/CredentialProviderLoader.java:
##########
@@ -221,10 +221,38 @@ private static List<CredentialProvider> loadProviders() {
return result;
}
+ /**
+ * Discovers all URI schemes supported by providers on the classpath.
+ * <p>
+ * This method queries all discovered {@link CredentialProvider} instances and collects
+ * their {@link CredentialProvider#supportedSchemes()} into a single set. Unlike
+ * {@link #providerFor(String, Map)}, this does not initialize providers or apply
+ * explicit selection rules -- it only reports what schemes are potentially available.
+ * <p>
+ * Intended for use by {@code UserCredentialManager} when no explicit scheme configuration
+ * (e.g., {@code spark.security.credentials.provider.<scheme>}) is provided.
+ *
+ * @param conf Spark configuration properties (used for potential future filtering)
+ * @return a set of all supported scheme names (lowercased), possibly empty
+ */
+ public static Set<String> discoverAllSchemes(Map<String, String> conf) {
Review Comment:
Removed the unused `conf`. Will add back with the filtering logic if/when it's needed.
##########
core/src/main/scala/org/apache/spark/deploy/security/UserCredentialManager.scala:
##########
@@ -0,0 +1,433 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.security
+
+import java.io.{ByteArrayOutputStream, ObjectInputFilter, ObjectInputStream, ObjectOutputStream}
+import java.net.URI
+import java.nio.file.Paths
+import java.time.Instant
+import java.util.concurrent.{RejectedExecutionException, ScheduledExecutorService, TimeUnit}
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.LogKeys
+import org.apache.spark.internal.config._
+import org.apache.spark.security._
+import org.apache.spark.ui.UIUtils
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Manager for OIDC-based credential propagation on the driver.
+ *
+ * This class is a sibling of [[HadoopDelegationTokenManager]] and handles the OIDC credential
+ * propagation path independently. Both managers run on independent threads and can both be
+ * active simultaneously (e.g., a cluster accessing HDFS via Kerberos and S3 via OIDC).
+ *
+ * Responsibilities:
+ * 1. Reads the current identity token via a [[TokenIngestor]]
+ * 2. Calls [[CredentialProvider#resolve]] for each configured scheme
+ * 3. Serializes the resulting [[UserCredentials]] and invokes the propagation callback
+ * 4. Schedules renewal based on `min(identity token expiry, service credential expiry) -
+ * safetyMargin`
+ * 5. Retries with exponential backoff on failure
+ *
+ * Started from `CoarseGrainedSchedulerBackend.start()` when
+ * `spark.security.credentials.enabled=true`, independently of
+ * `UserGroupInformation.isSecurityEnabled()`.
+ *
+ * Lifecycle: call `start()` exactly once, then `stop()` to shut down.
+ * Calling `start()` after `stop()` is not supported.
+ */
+private[spark] class UserCredentialManager(
+ sparkConf: SparkConf,
+ tokenIngestor: TokenIngestor,
+ onCredentialsUpdate: Array[Byte] => Unit) extends Logging {
+
+ private val safetyMargin = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_SAFETY_MARGIN)
+ private val minInterval = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_MIN_INTERVAL)
+
+ // Thread-safe counter for exponential backoff calculation.
+ // Accessed from both the caller of start() and the scheduled renewal thread.
+ private val consecutiveFailures = new AtomicInteger(0)
+ private val maxBackoffMs: Long = TimeUnit.MINUTES.toMillis(10)
Review Comment:
Moved to the companion object as `MAX_BACKOFF_MS`.
##########
core/src/main/scala/org/apache/spark/deploy/security/UserCredentialManager.scala:
##########
@@ -0,0 +1,433 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.security
+
+import java.io.{ByteArrayOutputStream, ObjectInputFilter, ObjectInputStream, ObjectOutputStream}
+import java.net.URI
+import java.nio.file.Paths
+import java.time.Instant
+import java.util.concurrent.{RejectedExecutionException, ScheduledExecutorService, TimeUnit}
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.LogKeys
+import org.apache.spark.internal.config._
+import org.apache.spark.security._
+import org.apache.spark.ui.UIUtils
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Manager for OIDC-based credential propagation on the driver.
+ *
+ * This class is a sibling of [[HadoopDelegationTokenManager]] and handles the OIDC credential
+ * propagation path independently. Both managers run on independent threads and can both be
+ * active simultaneously (e.g., a cluster accessing HDFS via Kerberos and S3 via OIDC).
+ *
+ * Responsibilities:
+ * 1. Reads the current identity token via a [[TokenIngestor]]
+ * 2. Calls [[CredentialProvider#resolve]] for each configured scheme
+ * 3. Serializes the resulting [[UserCredentials]] and invokes the propagation callback
+ * 4. Schedules renewal based on `min(identity token expiry, service credential expiry) -
+ * safetyMargin`
+ * 5. Retries with exponential backoff on failure
+ *
+ * Started from `CoarseGrainedSchedulerBackend.start()` when
Review Comment:
Fixed.
##########
core/src/main/java/org/apache/spark/security/CredentialProviderLoader.java:
##########
@@ -221,10 +221,38 @@ private static List<CredentialProvider> loadProviders() {
return result;
}
+ /**
+ * Discovers all URI schemes supported by providers on the classpath.
+ * <p>
+ * This method queries all discovered {@link CredentialProvider} instances and collects
+ * their {@link CredentialProvider#supportedSchemes()} into a single set. Unlike
+ * {@link #providerFor(String, Map)}, this does not initialize providers or apply
+ * explicit selection rules -- it only reports what schemes are potentially available.
+ * <p>
+ * Intended for use by {@code UserCredentialManager} when no explicit scheme configuration
+ * (e.g., {@code spark.security.credentials.provider.<scheme>}) is provided.
+ *
+ * @param conf Spark configuration properties (used for potential future filtering)
+ * @return a set of all supported scheme names (lowercased), possibly empty
+ */
+ public static Set<String> discoverAllSchemes(Map<String, String> conf) {
+ List<CredentialProvider> providers = getProviders();
+ Set<String> schemes = new java.util.HashSet<>();
+ for (CredentialProvider provider : providers) {
+ Set<String> providerSchemes = provider.supportedSchemes();
+ if (providerSchemes != null) {
+ for (String s : providerSchemes) {
+ schemes.add(s.toLowerCase(Locale.ROOT));
+ }
+ }
+ }
+ return schemes;
+ }
+
/**
* Resets the cached provider list and initialization tracking. Intended for testing only.
*/
- static void resetForTesting() {
+ public static void resetForTesting() {
Review Comment:
Added `@VisibleForTesting`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement UserCredentialManager [spark]
Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on code in PR #57387:
URL: https://github.com/apache/spark/pull/57387#discussion_r3665754690
##########
core/src/main/scala/org/apache/spark/deploy/security/UserCredentialManager.scala:
##########
@@ -0,0 +1,443 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.security
+
+import java.io.{ByteArrayOutputStream, ObjectInputFilter, ObjectInputStream, ObjectOutputStream}
+import java.net.URI
+import java.nio.file.Paths
+import java.time.Instant
+import java.util.concurrent.{RejectedExecutionException, ScheduledExecutorService, TimeUnit}
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.LogKeys
+import org.apache.spark.internal.config._
+import org.apache.spark.security._
+import org.apache.spark.ui.UIUtils
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Manager for OIDC-based credential propagation on the driver.
+ *
+ * This class is a sibling of [[HadoopDelegationTokenManager]] and handles the OIDC credential
+ * propagation path independently. Both managers run on independent threads and can both be
+ * active simultaneously (e.g., a cluster accessing HDFS via Kerberos and S3 via OIDC).
+ *
+ * Responsibilities:
+ * 1. Reads the current identity token via a [[TokenIngestor]]
+ * 2. Calls [[CredentialProvider#resolve]] for each configured scheme
+ * 3. Serializes the resulting [[UserCredentials]] and invokes the propagation callback
+ * 4. Schedules renewal based on `min(identity token expiry, service credential expiry) -
+ * safetyMargin`
+ * 5. Retries with exponential backoff on failure
+ *
+ * Intended to be started from `CoarseGrainedSchedulerBackend.start()` when
+ * `spark.security.credentials.enabled=true`, independently of
+ * `UserGroupInformation.isSecurityEnabled()`.
+ *
+ * Lifecycle: call `start()` exactly once, then `stop()` to shut down.
+ * Calling `start()` after `stop()` is not supported.
+ */
+private[spark] class UserCredentialManager(
+ sparkConf: SparkConf,
+ tokenIngestor: TokenIngestor,
+ onCredentialsUpdate: Array[Byte] => Unit) extends Logging {
+
+ private val safetyMargin = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_SAFETY_MARGIN)
+ private val minInterval = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_MIN_INTERVAL)
+
+ // Thread-safe counter for exponential backoff calculation.
+ // Accessed from both the caller of start() and the scheduled renewal thread.
+ private val consecutiveFailures = new AtomicInteger(0)
+ private val maxBackoffMs: Long = UserCredentialManager.MAX_BACKOFF_MS
+
+ @volatile private var renewalExecutor: ScheduledExecutorService = _
+
+ // Pre-compute the filtered credential configuration map (SparkConf is immutable).
+ private val credentialConfMap: java.util.Map[String, String] = sparkConf.getAll
+ .filter(_._1.startsWith("spark.security.credentials."))
+ .toMap.asJava
+
+ /**
+ * Start the credential manager. Acquires initial credentials and schedules renewal.
+ *
+ * The initial credential acquisition is fail-fast: if the token cannot be loaded or
+ * no credentials can be resolved, an exception is thrown. Subsequent renewal failures
+ * are handled with exponential backoff.
+ *
+ * @return The serialized initial [[UserCredentials]].
+ * @throws IllegalStateException if the initial credential acquisition fails.
+ */
+ def start(): Array[Byte] = {
+ require(renewalExecutor == null, "start() must not be called more than once")
+
+ // Initial acquisition is fail-fast (no retry/backoff).
+ // This ensures the application fails to start if credentials cannot be obtained,
+ // rather than running with null credentials.
+ val userContext = tokenIngestor.load()
+ if (!userContext.isPresent) {
+ throw new IllegalStateException(
+ "Failed to start UserCredentialManager: identity token file is missing or malformed. " +
+ s"Check ${SECURITY_CREDENTIALS_IDENTITY_TOKEN_FILE.key} configuration.")
+ }
+
+ val ctx = userContext.get()
+ logInfo(log"Initial identity token loaded for principal " +
+ log"${MDC(LogKeys.PRINCIPAL, ctx.getPrincipal)} " +
+ log"(issuer: ${MDC(LogKeys.URI, ctx.getIssuer)})")
+
+ val (credentials, earliestExpiry) = resolveCredentials(ctx)
+ val serialized = serializeUserCredentials(credentials)
+
+ // Propagate initial credentials
+ onCredentialsUpdate(serialized)
+
+ // Create the renewal executor only after successful initial acquisition.
+ // This avoids leaking a daemon thread if the fail-fast path throws, and
+ // keeps the require() guard valid for retry scenarios.
+ renewalExecutor =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("user-credential-renewal")
+
+ // Schedule first renewal
+ val renewalDelay = computeRenewalDelay(ctx, earliestExpiry)
+ scheduleRenewal(renewalDelay)
+
+ logInfo(log"Credential acquisition successful. Next renewal in " +
+ log"${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(renewalDelay))}.")
+ serialized
+ }
+
+ def stop(): Unit = {
+ if (renewalExecutor != null) {
+ renewalExecutor.shutdownNow()
+ }
+ }
+
+ /**
+ * Scheduled renewal task: load identity token, resolve credentials, propagate,
+ * and schedule the next renewal. Failures are retried with exponential backoff.
+ */
+ private def renewCredentialsTask(): Unit = {
+ try {
+ val userContext = tokenIngestor.load()
+ if (!userContext.isPresent) {
+ throw new IllegalStateException(
+ "TokenIngestor returned empty - identity token file may be missing or malformed")
+ }
+
+ val ctx = userContext.get()
+ logInfo(log"Loaded identity token for principal " +
+ log"${MDC(LogKeys.PRINCIPAL, ctx.getPrincipal)} " +
+ log"(issuer: ${MDC(LogKeys.URI, ctx.getIssuer)})")
+
+ val (credentials, earliestExpiry) = resolveCredentials(ctx)
+ val serialized = serializeUserCredentials(credentials)
+
+ // Propagate credentials to executors. Errors here are logged separately
+ // so that credential-fetch success is not conflated with distribution failure.
+ try {
+ onCredentialsUpdate(serialized)
+ } catch {
+ case e: Exception =>
+ logWarning(log"Credentials were resolved successfully but failed to propagate " +
+ log"to executors. Executors will receive updated credentials on next renewal.", e)
+ }
+
+ // Reset backoff on successful credential resolution. This is intentionally done
+ // even if onCredentialsUpdate failed above: the backoff counter tracks credential
+ // *resolution* failures (STS/provider issues), not propagation failures.
+ // Propagation failures are transient and will be retried on next scheduled renewal.
+ consecutiveFailures.set(0)
+
+ // Schedule next renewal
+ val renewalDelay = computeRenewalDelay(ctx, earliestExpiry)
+ scheduleRenewal(renewalDelay)
+
+ logInfo(log"Credential renewal successful. Next renewal in " +
+ log"${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(renewalDelay))}.")
+ } catch {
+ case _: InterruptedException =>
+ // Shutting down, ignore
+ case e: Exception =>
+ val failures = consecutiveFailures.incrementAndGet()
+ val delay = computeBackoffDelay()
+ logWarning(log"Failed to renew user credentials (attempt " +
+ log"${MDC(LogKeys.NUM_RETRY, failures)}), " +
+ log"will retry in ${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(delay))}.", e)
+ scheduleRenewal(delay)
+ }
+ }
+
+ /**
+ * Resolve credentials for all schemes that have registered providers.
+ *
+ * @return Tuple of (UserCredentials, earliest expiry across all service credentials)
+ */
+ private def resolveCredentials(
+ ctx: UserContext): (UserCredentials, Option[Instant]) = {
+ val schemes = discoverSchemes()
+
+ val credentialMap = new mutable.HashMap[String, ServiceCredential]()
+ var earliestExpiry: Option[Instant] = None
+
+ for (scheme <- schemes) {
+ try {
+ val providerOpt = CredentialProviderLoader.providerFor(scheme, credentialConfMap)
+ if (providerOpt.isPresent) {
+ val provider = providerOpt.get()
+ // Use a synthetic target URI with just the scheme for initial resolution.
+ // The full URI is passed in future versions when path-specific resolution is needed.
+ // The "synthetic" authority signals this is not a real endpoint but a placeholder
+ // for scheme-based provider selection.
+ val target = new URI(scheme, UserCredentialManager.SYNTHETIC_TARGET_AUTHORITY,
+ "/", null, null)
+ val credential = provider.resolve(ctx, target)
+ credentialMap.put(scheme, credential)
+
+ val expiry = credential.getExpiresAt
+ if (expiry != null) {
+ earliestExpiry = earliestExpiry match {
+ case Some(existing) if existing.isBefore(expiry) => Some(existing)
+ case _ => Some(expiry)
+ }
+ }
+ } else {
+ logWarning(log"No credential provider found for scheme " +
+ log"${MDC(LogKeys.URI, scheme)}. Skipping.")
+ }
+ } catch {
+ case e: Exception =>
+ logWarning(log"Failed to resolve credentials for scheme " +
+ log"${MDC(LogKeys.URI, scheme)}. Skipping this provider.", e)
+ }
+ }
+
+ if (credentialMap.isEmpty) {
+ throw new IllegalStateException(
+ "No credential providers resolved any credentials. " +
+ "Check that providers are on the classpath and configured correctly.")
+ }
+
+ (new UserCredentials(credentialMap.asJava), earliestExpiry)
+ }
+
+ /**
+ * Discover all schemes that have at least one registered provider.
+ *
+ * Schemes are determined by explicit configuration keys of the form
+ * `spark.security.credentials.provider.<scheme>`. If no explicit configuration
+ * exists, the method queries `CredentialProviderLoader` to discover all providers
+ * registered via ServiceLoader and collects their supported schemes.
+ *
+ * Note: When using auto-discovery (no explicit config), multiple providers may
+ * support the same scheme. In that case, `CredentialProviderLoader.providerFor`
+ * will throw an `IllegalArgumentException` for that scheme. The caller
+ * (`resolveCredentials`) handles this gracefully via per-provider exception catching,
+ * logging a warning and continuing with remaining schemes.
+ */
+ private def discoverSchemes(): Set[String] = {
+ // Extract explicitly configured scheme -> provider mappings
+ val explicitSchemes = credentialConfMap.asScala
+ .filter { case (k, _) => k.startsWith("spark.security.credentials.provider.") }
+ .map { case (k, _) => k.stripPrefix("spark.security.credentials.provider.") }
+ .toSet
+
+ if (explicitSchemes.nonEmpty) {
+ explicitSchemes
+ } else {
+ // No explicit scheme configuration. Discover all schemes that have a provider
+ // available on the classpath by probing CredentialProviderLoader.
+ // This covers both built-in providers (e.g., connector/credential-aws for "s3a")
+ // and third-party providers registered via ServiceLoader.
+ CredentialProviderLoader.discoverAllSchemes().asScala.toSet
+ }
+ }
+
+ /**
+ * Compute the delay until next renewal.
+ * Uses min(identity token expiry, service credential expiry) - safetyMargin,
+ * bounded below by minInterval.
+ */
+ private[security] def computeRenewalDelay(
+ ctx: UserContext,
+ earliestCredentialExpiry: Option[Instant]): Long = {
+ val now = System.currentTimeMillis()
+
+ // Consider identity token expiry
+ val tokenExpiry: Option[Long] = Option(ctx.getExpiresAt).map(_.toEpochMilli)
+
+ // Consider earliest service credential expiry
+ val credExpiry: Option[Long] = earliestCredentialExpiry.map(_.toEpochMilli)
+
+ // Take the minimum of both
+ val effectiveExpiry: Option[Long] = (tokenExpiry, credExpiry) match {
+ case (Some(t), Some(c)) => Some(math.min(t, c))
+ case (Some(t), None) => Some(t)
+ case (None, Some(c)) => Some(c)
+ case (None, None) => None
+ }
+
+ effectiveExpiry match {
+ case Some(expiry) =>
+ math.max(expiry - now - safetyMargin, minInterval)
+ case None =>
+ // No expiry information available from either the identity token or service
+ // credentials. Use half of the default suggested TTL (15 min / 2 = 7.5 min,
+ // rounded down) as a conservative polling interval. This ensures credentials
+ // are refreshed even when providers don't report expiration times.
+ UserCredentialManager.DEFAULT_RENEWAL_INTERVAL_NO_EXPIRY_MS
+ }
+ }
+
+ /**
+ * Compute backoff delay using exponential backoff with jitter.
+ * Bounded by minInterval (floor) and maxBackoffMs (ceiling).
+ */
+ private[security] def computeBackoffDelay(): Long = {
+ // Guard against negative or zero shift amounts. consecutiveFailures should always
+ // be >= 1 when this is called (incremented before calling), but we protect against
+ // edge cases defensively.
+ val shiftAmount = math.max(0, math.min(consecutiveFailures.get() - 1, 6))
+ val baseDelay = minInterval * (1L << shiftAmount)
+ val cappedDelay = math.min(baseDelay, maxBackoffMs)
+ // Add 10% jitter to avoid thundering herd on recovery
+ val jitter = (cappedDelay * 0.1 * math.random()).toLong
+ math.max(cappedDelay + jitter, minInterval)
+ }
+
+ private def scheduleRenewal(delay: Long): Unit = {
+ try {
+ val renewalTask = new Runnable {
+ override def run(): Unit = {
+ renewCredentialsTask()
+ }
+ }
+ renewalExecutor.schedule(renewalTask, delay, TimeUnit.MILLISECONDS)
+ } catch {
+ case _: RejectedExecutionException =>
+ // Executor has been shut down (e.g., stop() called concurrently). This is expected
+ // during application shutdown -- no further renewals will be scheduled.
+ logDebug(log"Renewal scheduling rejected - executor is shut down.")
+ }
+ }
+
+ /**
+ * Serialize [[UserCredentials]] to a byte array using Java serialization.
+ */
+ private[security] def serializeUserCredentials(credentials: UserCredentials): Array[Byte] = {
+ val bos = new ByteArrayOutputStream()
+ try {
+ val oos = new ObjectOutputStream(bos)
+ try {
+ oos.writeObject(credentials)
+ oos.flush()
+ } finally {
+ oos.close()
+ }
+ bos.toByteArray
+ } finally {
+ bos.close()
+ }
+ }
+}
+
+private[spark] object UserCredentialManager {
+
+ /**
+ * Synthetic authority used in target URIs for scheme-based provider resolution.
+ * Providers should not rely on this value; it signals that no specific endpoint
+ * is targeted and only the URI scheme is meaningful.
+ */
+ private val SYNTHETIC_TARGET_AUTHORITY = "synthetic"
+
+ /**
+ * Maximum backoff delay between credential renewal retry attempts.
+ * Caps the exponential backoff to prevent excessively long gaps between retries.
+ */
+ private val MAX_BACKOFF_MS: Long = TimeUnit.MINUTES.toMillis(10)
+
+ /**
+ * Default renewal interval when neither the identity token nor service credentials
+ * report an expiration time. Set to 7 minutes (half of the default 15-minute
+ * suggested TTL from CredentialProvider.suggestedTtl()).
+ */
+ private val DEFAULT_RENEWAL_INTERVAL_NO_EXPIRY_MS: Long = TimeUnit.MINUTES.toMillis(7)
+
+ /**
+ * ObjectInputFilter pattern restricting deserialization to only the classes needed
+ * for UserCredentials. This prevents deserialization gadget chain attacks while
+ * allowing Java's built-in collection internal classes and arrays that HashMap uses.
+ */
+ private val DESERIALIZATION_FILTER: String =
+ "org.apache.spark.security.**;" +
+ "java.util.**;" +
+ "java.time.**;" +
+ "java.lang.**;" +
+ "maxdepth=10;" +
+ "!*"
+
+ /**
+ * Create a UserCredentialManager if OIDC credential propagation is enabled.
+ *
+ * @param sparkConf The Spark configuration
+ * @param onCredentialsUpdate Callback to propagate credentials to executors
+ * @return Some(manager) if enabled, None otherwise
+ */
+ def create(
+ sparkConf: SparkConf,
+ onCredentialsUpdate: Array[Byte] => Unit): Option[UserCredentialManager] = {
+ if (!sparkConf.get(SECURITY_CREDENTIALS_ENABLED)) {
+ None
+ } else {
+ val tokenFile = sparkConf.get(SECURITY_CREDENTIALS_IDENTITY_TOKEN_FILE).getOrElse {
+ throw new IllegalArgumentException(
+ s"${SECURITY_CREDENTIALS_IDENTITY_TOKEN_FILE.key} must be set when " +
+ s"${SECURITY_CREDENTIALS_ENABLED.key} is true")
+ }
+
+ val tokenIngestor = new FileTokenIngestor(Paths.get(tokenFile))
+ Some(new UserCredentialManager(sparkConf, tokenIngestor, onCredentialsUpdate))
+ }
+ }
+
+ /**
+ * Deserialize [[UserCredentials]] from a byte array.
+ *
+ * Uses an [[ObjectInputFilter]] to restrict deserialized classes to only those
+ * required for [[UserCredentials]], preventing deserialization attacks.
+ */
+ def deserializeUserCredentials(bytes: Array[Byte]): UserCredentials = {
+ val bis = new java.io.ByteArrayInputStream(bytes)
+ try {
+ val ois = new ObjectInputStream(bis)
+ try {
+ ois.setObjectInputFilter(
+ ObjectInputFilter.Config.createFilter(DESERIALIZATION_FILTER))
+ ois.readObject().asInstanceOf[UserCredentials]
+ } finally {
+ ois.close()
+ }
+ } finally {
+ bis.close()
+ }
Review Comment:
Thanks. Applied the suggested change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement `UserCredentialManager` [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #57387:
URL: https://github.com/apache/spark/pull/57387#issuecomment-5106862120
FYI, cc @peter-toth and @parthchandra
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement UserCredentialManager [spark]
Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on code in PR #57387:
URL: https://github.com/apache/spark/pull/57387#discussion_r3665821948
##########
core/src/main/scala/org/apache/spark/deploy/security/UserCredentialManager.scala:
##########
@@ -0,0 +1,443 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.security
+
+import java.io.{ByteArrayOutputStream, ObjectInputFilter, ObjectInputStream, ObjectOutputStream}
+import java.net.URI
+import java.nio.file.Paths
+import java.time.Instant
+import java.util.concurrent.{RejectedExecutionException, ScheduledExecutorService, TimeUnit}
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.LogKeys
+import org.apache.spark.internal.config._
+import org.apache.spark.security._
+import org.apache.spark.ui.UIUtils
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Manager for OIDC-based credential propagation on the driver.
+ *
+ * This class is a sibling of [[HadoopDelegationTokenManager]] and handles the OIDC credential
+ * propagation path independently. Both managers run on independent threads and can both be
+ * active simultaneously (e.g., a cluster accessing HDFS via Kerberos and S3 via OIDC).
+ *
+ * Responsibilities:
+ * 1. Reads the current identity token via a [[TokenIngestor]]
+ * 2. Calls [[CredentialProvider#resolve]] for each configured scheme
+ * 3. Serializes the resulting [[UserCredentials]] and invokes the propagation callback
+ * 4. Schedules renewal based on `min(identity token expiry, service credential expiry) -
+ * safetyMargin`
+ * 5. Retries with exponential backoff on failure
+ *
+ * Intended to be started from `CoarseGrainedSchedulerBackend.start()` when
+ * `spark.security.credentials.enabled=true`, independently of
+ * `UserGroupInformation.isSecurityEnabled()`.
+ *
+ * Lifecycle: call `start()` exactly once, then `stop()` to shut down.
+ * Calling `start()` after `stop()` is not supported.
+ */
+private[spark] class UserCredentialManager(
+ sparkConf: SparkConf,
+ tokenIngestor: TokenIngestor,
+ onCredentialsUpdate: Array[Byte] => Unit) extends Logging {
+
+ private val safetyMargin = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_SAFETY_MARGIN)
+ private val minInterval = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_MIN_INTERVAL)
+
+ // Thread-safe counter for exponential backoff calculation.
+ // Accessed from both the caller of start() and the scheduled renewal thread.
+ private val consecutiveFailures = new AtomicInteger(0)
+ private val maxBackoffMs: Long = UserCredentialManager.MAX_BACKOFF_MS
+
+ @volatile private var renewalExecutor: ScheduledExecutorService = _
+
+ // Pre-compute the filtered credential configuration map (SparkConf is immutable).
+ private val credentialConfMap: java.util.Map[String, String] = sparkConf.getAll
+ .filter(_._1.startsWith("spark.security.credentials."))
+ .toMap.asJava
+
+ /**
+ * Start the credential manager. Acquires initial credentials and schedules renewal.
+ *
+ * The initial credential acquisition is fail-fast: if the token cannot be loaded or
+ * no credentials can be resolved, an exception is thrown. Subsequent renewal failures
+ * are handled with exponential backoff.
+ *
+ * @return The serialized initial [[UserCredentials]].
+ * @throws IllegalStateException if the initial credential acquisition fails.
+ */
+ def start(): Array[Byte] = {
+ require(renewalExecutor == null, "start() must not be called more than once")
+
+ // Initial acquisition is fail-fast (no retry/backoff).
+ // This ensures the application fails to start if credentials cannot be obtained,
+ // rather than running with null credentials.
+ val userContext = tokenIngestor.load()
+ if (!userContext.isPresent) {
+ throw new IllegalStateException(
+ "Failed to start UserCredentialManager: identity token file is missing or malformed. " +
+ s"Check ${SECURITY_CREDENTIALS_IDENTITY_TOKEN_FILE.key} configuration.")
+ }
+
+ val ctx = userContext.get()
+ logInfo(log"Initial identity token loaded for principal " +
+ log"${MDC(LogKeys.PRINCIPAL, ctx.getPrincipal)} " +
+ log"(issuer: ${MDC(LogKeys.URI, ctx.getIssuer)})")
+
+ val (credentials, earliestExpiry) = resolveCredentials(ctx)
+ val serialized = serializeUserCredentials(credentials)
+
+ // Propagate initial credentials
+ onCredentialsUpdate(serialized)
+
+ // Create the renewal executor only after successful initial acquisition.
+ // This avoids leaking a daemon thread if the fail-fast path throws, and
+ // keeps the require() guard valid for retry scenarios.
+ renewalExecutor =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("user-credential-renewal")
+
+ // Schedule first renewal
+ val renewalDelay = computeRenewalDelay(ctx, earliestExpiry)
+ scheduleRenewal(renewalDelay)
+
+ logInfo(log"Credential acquisition successful. Next renewal in " +
+ log"${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(renewalDelay))}.")
+ serialized
+ }
+
+ def stop(): Unit = {
+ if (renewalExecutor != null) {
+ renewalExecutor.shutdownNow()
+ }
+ }
+
+ /**
+ * Scheduled renewal task: load identity token, resolve credentials, propagate,
+ * and schedule the next renewal. Failures are retried with exponential backoff.
+ */
+ private def renewCredentialsTask(): Unit = {
+ try {
+ val userContext = tokenIngestor.load()
+ if (!userContext.isPresent) {
+ throw new IllegalStateException(
+ "TokenIngestor returned empty - identity token file may be missing or malformed")
+ }
+
+ val ctx = userContext.get()
+ logInfo(log"Loaded identity token for principal " +
+ log"${MDC(LogKeys.PRINCIPAL, ctx.getPrincipal)} " +
+ log"(issuer: ${MDC(LogKeys.URI, ctx.getIssuer)})")
+
+ val (credentials, earliestExpiry) = resolveCredentials(ctx)
+ val serialized = serializeUserCredentials(credentials)
+
+ // Propagate credentials to executors. Errors here are logged separately
+ // so that credential-fetch success is not conflated with distribution failure.
+ try {
+ onCredentialsUpdate(serialized)
+ } catch {
+ case e: Exception =>
+ logWarning(log"Credentials were resolved successfully but failed to propagate " +
+ log"to executors. Executors will receive updated credentials on next renewal.", e)
+ }
+
+ // Reset backoff on successful credential resolution. This is intentionally done
+ // even if onCredentialsUpdate failed above: the backoff counter tracks credential
+ // *resolution* failures (STS/provider issues), not propagation failures.
+ // Propagation failures are transient and will be retried on next scheduled renewal.
+ consecutiveFailures.set(0)
+
+ // Schedule next renewal
+ val renewalDelay = computeRenewalDelay(ctx, earliestExpiry)
+ scheduleRenewal(renewalDelay)
+
+ logInfo(log"Credential renewal successful. Next renewal in " +
+ log"${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(renewalDelay))}.")
+ } catch {
+ case _: InterruptedException =>
+ // Shutting down, ignore
+ case e: Exception =>
+ val failures = consecutiveFailures.incrementAndGet()
+ val delay = computeBackoffDelay()
+ logWarning(log"Failed to renew user credentials (attempt " +
+ log"${MDC(LogKeys.NUM_RETRY, failures)}), " +
+ log"will retry in ${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(delay))}.", e)
+ scheduleRenewal(delay)
+ }
+ }
+
+ /**
+ * Resolve credentials for all schemes that have registered providers.
+ *
+ * @return Tuple of (UserCredentials, earliest expiry across all service credentials)
+ */
+ private def resolveCredentials(
+ ctx: UserContext): (UserCredentials, Option[Instant]) = {
+ val schemes = discoverSchemes()
+
+ val credentialMap = new mutable.HashMap[String, ServiceCredential]()
+ var earliestExpiry: Option[Instant] = None
+
+ for (scheme <- schemes) {
+ try {
+ val providerOpt = CredentialProviderLoader.providerFor(scheme, credentialConfMap)
+ if (providerOpt.isPresent) {
+ val provider = providerOpt.get()
+ // Use a synthetic target URI with just the scheme for initial resolution.
+ // The full URI is passed in future versions when path-specific resolution is needed.
+ // The "synthetic" authority signals this is not a real endpoint but a placeholder
+ // for scheme-based provider selection.
+ val target = new URI(scheme, UserCredentialManager.SYNTHETIC_TARGET_AUTHORITY,
+ "/", null, null)
+ val credential = provider.resolve(ctx, target)
+ credentialMap.put(scheme, credential)
Review Comment:
Added a null check before inserting into `credentialMap`. If `resolve()` returns null, we log a warning and skip the scheme. This prevents the `credentialMap.isEmpty` fail-fast bypass and the downstream NPE on `getExpiresAt`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement UserCredentialManager [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #57387:
URL: https://github.com/apache/spark/pull/57387#discussion_r3650925913
##########
core/src/main/scala/org/apache/spark/deploy/security/UserCredentialManager.scala:
##########
@@ -0,0 +1,443 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.security
+
+import java.io.{ByteArrayOutputStream, ObjectInputFilter, ObjectInputStream, ObjectOutputStream}
+import java.net.URI
+import java.nio.file.Paths
+import java.time.Instant
+import java.util.concurrent.{RejectedExecutionException, ScheduledExecutorService, TimeUnit}
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.LogKeys
+import org.apache.spark.internal.config._
+import org.apache.spark.security._
+import org.apache.spark.ui.UIUtils
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Manager for OIDC-based credential propagation on the driver.
+ *
+ * This class is a sibling of [[HadoopDelegationTokenManager]] and handles the OIDC credential
+ * propagation path independently. Both managers run on independent threads and can both be
+ * active simultaneously (e.g., a cluster accessing HDFS via Kerberos and S3 via OIDC).
+ *
+ * Responsibilities:
+ * 1. Reads the current identity token via a [[TokenIngestor]]
+ * 2. Calls [[CredentialProvider#resolve]] for each configured scheme
+ * 3. Serializes the resulting [[UserCredentials]] and invokes the propagation callback
+ * 4. Schedules renewal based on `min(identity token expiry, service credential expiry) -
+ * safetyMargin`
+ * 5. Retries with exponential backoff on failure
+ *
+ * Intended to be started from `CoarseGrainedSchedulerBackend.start()` when
+ * `spark.security.credentials.enabled=true`, independently of
+ * `UserGroupInformation.isSecurityEnabled()`.
+ *
+ * Lifecycle: call `start()` exactly once, then `stop()` to shut down.
+ * Calling `start()` after `stop()` is not supported.
+ */
+private[spark] class UserCredentialManager(
+ sparkConf: SparkConf,
+ tokenIngestor: TokenIngestor,
+ onCredentialsUpdate: Array[Byte] => Unit) extends Logging {
+
+ private val safetyMargin = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_SAFETY_MARGIN)
+ private val minInterval = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_MIN_INTERVAL)
+
+ // Thread-safe counter for exponential backoff calculation.
+ // Accessed from both the caller of start() and the scheduled renewal thread.
+ private val consecutiveFailures = new AtomicInteger(0)
+ private val maxBackoffMs: Long = UserCredentialManager.MAX_BACKOFF_MS
+
+ @volatile private var renewalExecutor: ScheduledExecutorService = _
+
+ // Pre-compute the filtered credential configuration map (SparkConf is immutable).
+ private val credentialConfMap: java.util.Map[String, String] = sparkConf.getAll
+ .filter(_._1.startsWith("spark.security.credentials."))
+ .toMap.asJava
+
+ /**
+ * Start the credential manager. Acquires initial credentials and schedules renewal.
+ *
+ * The initial credential acquisition is fail-fast: if the token cannot be loaded or
+ * no credentials can be resolved, an exception is thrown. Subsequent renewal failures
+ * are handled with exponential backoff.
+ *
+ * @return The serialized initial [[UserCredentials]].
+ * @throws IllegalStateException if the initial credential acquisition fails.
+ */
+ def start(): Array[Byte] = {
+ require(renewalExecutor == null, "start() must not be called more than once")
+
+ // Initial acquisition is fail-fast (no retry/backoff).
+ // This ensures the application fails to start if credentials cannot be obtained,
+ // rather than running with null credentials.
+ val userContext = tokenIngestor.load()
+ if (!userContext.isPresent) {
+ throw new IllegalStateException(
+ "Failed to start UserCredentialManager: identity token file is missing or malformed. " +
+ s"Check ${SECURITY_CREDENTIALS_IDENTITY_TOKEN_FILE.key} configuration.")
+ }
+
+ val ctx = userContext.get()
+ logInfo(log"Initial identity token loaded for principal " +
+ log"${MDC(LogKeys.PRINCIPAL, ctx.getPrincipal)} " +
+ log"(issuer: ${MDC(LogKeys.URI, ctx.getIssuer)})")
+
+ val (credentials, earliestExpiry) = resolveCredentials(ctx)
+ val serialized = serializeUserCredentials(credentials)
+
+ // Propagate initial credentials
+ onCredentialsUpdate(serialized)
+
+ // Create the renewal executor only after successful initial acquisition.
+ // This avoids leaking a daemon thread if the fail-fast path throws, and
+ // keeps the require() guard valid for retry scenarios.
+ renewalExecutor =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("user-credential-renewal")
+
+ // Schedule first renewal
+ val renewalDelay = computeRenewalDelay(ctx, earliestExpiry)
+ scheduleRenewal(renewalDelay)
+
+ logInfo(log"Credential acquisition successful. Next renewal in " +
+ log"${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(renewalDelay))}.")
+ serialized
+ }
+
+ def stop(): Unit = {
+ if (renewalExecutor != null) {
+ renewalExecutor.shutdownNow()
+ }
+ }
+
+ /**
+ * Scheduled renewal task: load identity token, resolve credentials, propagate,
+ * and schedule the next renewal. Failures are retried with exponential backoff.
+ */
+ private def renewCredentialsTask(): Unit = {
+ try {
+ val userContext = tokenIngestor.load()
+ if (!userContext.isPresent) {
+ throw new IllegalStateException(
+ "TokenIngestor returned empty - identity token file may be missing or malformed")
+ }
+
+ val ctx = userContext.get()
+ logInfo(log"Loaded identity token for principal " +
+ log"${MDC(LogKeys.PRINCIPAL, ctx.getPrincipal)} " +
+ log"(issuer: ${MDC(LogKeys.URI, ctx.getIssuer)})")
+
+ val (credentials, earliestExpiry) = resolveCredentials(ctx)
+ val serialized = serializeUserCredentials(credentials)
+
+ // Propagate credentials to executors. Errors here are logged separately
+ // so that credential-fetch success is not conflated with distribution failure.
+ try {
+ onCredentialsUpdate(serialized)
+ } catch {
+ case e: Exception =>
+ logWarning(log"Credentials were resolved successfully but failed to propagate " +
+ log"to executors. Executors will receive updated credentials on next renewal.", e)
+ }
+
+ // Reset backoff on successful credential resolution. This is intentionally done
+ // even if onCredentialsUpdate failed above: the backoff counter tracks credential
+ // *resolution* failures (STS/provider issues), not propagation failures.
+ // Propagation failures are transient and will be retried on next scheduled renewal.
+ consecutiveFailures.set(0)
+
+ // Schedule next renewal
+ val renewalDelay = computeRenewalDelay(ctx, earliestExpiry)
+ scheduleRenewal(renewalDelay)
+
+ logInfo(log"Credential renewal successful. Next renewal in " +
+ log"${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(renewalDelay))}.")
+ } catch {
+ case _: InterruptedException =>
+ // Shutting down, ignore
+ case e: Exception =>
+ val failures = consecutiveFailures.incrementAndGet()
+ val delay = computeBackoffDelay()
+ logWarning(log"Failed to renew user credentials (attempt " +
+ log"${MDC(LogKeys.NUM_RETRY, failures)}), " +
+ log"will retry in ${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(delay))}.", e)
+ scheduleRenewal(delay)
+ }
+ }
+
+ /**
+ * Resolve credentials for all schemes that have registered providers.
+ *
+ * @return Tuple of (UserCredentials, earliest expiry across all service credentials)
+ */
+ private def resolveCredentials(
+ ctx: UserContext): (UserCredentials, Option[Instant]) = {
+ val schemes = discoverSchemes()
+
+ val credentialMap = new mutable.HashMap[String, ServiceCredential]()
+ var earliestExpiry: Option[Instant] = None
+
+ for (scheme <- schemes) {
+ try {
+ val providerOpt = CredentialProviderLoader.providerFor(scheme, credentialConfMap)
+ if (providerOpt.isPresent) {
+ val provider = providerOpt.get()
+ // Use a synthetic target URI with just the scheme for initial resolution.
+ // The full URI is passed in future versions when path-specific resolution is needed.
+ // The "synthetic" authority signals this is not a real endpoint but a placeholder
+ // for scheme-based provider selection.
+ val target = new URI(scheme, UserCredentialManager.SYNTHETIC_TARGET_AUTHORITY,
+ "/", null, null)
+ val credential = provider.resolve(ctx, target)
+ credentialMap.put(scheme, credential)
+
+ val expiry = credential.getExpiresAt
+ if (expiry != null) {
+ earliestExpiry = earliestExpiry match {
+ case Some(existing) if existing.isBefore(expiry) => Some(existing)
+ case _ => Some(expiry)
+ }
+ }
+ } else {
+ logWarning(log"No credential provider found for scheme " +
+ log"${MDC(LogKeys.URI, scheme)}. Skipping.")
+ }
+ } catch {
+ case e: Exception =>
+ logWarning(log"Failed to resolve credentials for scheme " +
+ log"${MDC(LogKeys.URI, scheme)}. Skipping this provider.", e)
+ }
+ }
+
+ if (credentialMap.isEmpty) {
+ throw new IllegalStateException(
+ "No credential providers resolved any credentials. " +
+ "Check that providers are on the classpath and configured correctly.")
+ }
+
+ (new UserCredentials(credentialMap.asJava), earliestExpiry)
+ }
+
+ /**
+ * Discover all schemes that have at least one registered provider.
+ *
+ * Schemes are determined by explicit configuration keys of the form
+ * `spark.security.credentials.provider.<scheme>`. If no explicit configuration
+ * exists, the method queries `CredentialProviderLoader` to discover all providers
+ * registered via ServiceLoader and collects their supported schemes.
+ *
+ * Note: When using auto-discovery (no explicit config), multiple providers may
+ * support the same scheme. In that case, `CredentialProviderLoader.providerFor`
+ * will throw an `IllegalArgumentException` for that scheme. The caller
+ * (`resolveCredentials`) handles this gracefully via per-provider exception catching,
+ * logging a warning and continuing with remaining schemes.
+ */
+ private def discoverSchemes(): Set[String] = {
+ // Extract explicitly configured scheme -> provider mappings
+ val explicitSchemes = credentialConfMap.asScala
+ .filter { case (k, _) => k.startsWith("spark.security.credentials.provider.") }
+ .map { case (k, _) => k.stripPrefix("spark.security.credentials.provider.") }
+ .toSet
Review Comment:
This extraction logic seems to have corner cases.
For example, any key like `spark.security.credentials.provider.s3a.roleArn` yields the "scheme" `s3a.roleArn`, which then fails provider lookup and logs a `No credential provider found` warning on every renewal cycle.
Could you double-check this kind of wrong schema extraction?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement UserCredentialManager [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #57387:
URL: https://github.com/apache/spark/pull/57387#discussion_r3650916705
##########
core/src/main/scala/org/apache/spark/internal/config/package.scala:
##########
@@ -1679,6 +1679,44 @@ package object config {
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("1h")
+ private[spark] val SECURITY_CREDENTIALS_ENABLED =
+ ConfigBuilder("spark.security.credentials.enabled")
+ .doc("Whether to enable OIDC credential propagation. When enabled, the driver reads an " +
+ "identity token from a file, exchanges it for short-lived service credentials via " +
+ "CredentialProvider implementations, and propagates those credentials to executors.")
+ .version("4.3.0")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+ .booleanConf
+ .createWithDefault(false)
+
+ private[spark] val SECURITY_CREDENTIALS_IDENTITY_TOKEN_FILE =
+ ConfigBuilder("spark.security.credentials.identityToken.file")
+ .doc("Path to the OIDC identity token file on the driver. Required when " +
+ "spark.security.credentials.enabled is true. The file should contain a JWT token " +
+ "(e.g., a Kubernetes projected service account token).")
+ .version("4.3.0")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+ .stringConf
+ .createOptional
+
+ private[spark] val SECURITY_CREDENTIALS_RENEWAL_SAFETY_MARGIN =
+ ConfigBuilder("spark.security.credentials.renewal.safetyMargin")
+ .doc("How long before credential expiry to trigger renewal. Credentials are refreshed " +
+ "at min(identity token expiry, service credential expiry) minus this margin.")
+ .version("4.3.0")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+ .timeConf(TimeUnit.MILLISECONDS)
+ .createWithDefaultString("60s")
+
+ private[spark] val SECURITY_CREDENTIALS_RENEWAL_MIN_INTERVAL =
+ ConfigBuilder("spark.security.credentials.renewal.minInterval")
+ .doc("Minimum interval between credential renewal attempts. This prevents tight renewal " +
+ "loops when credentials have very short TTLs or when failures cause rapid retries.")
+ .version("4.3.0")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+ .timeConf(TimeUnit.MILLISECONDS)
+ .createWithDefaultString("30s")
Review Comment:
`computeBackoffDelay()` uses `minInterval` as the base of the exponential backoff, and `computeRenewalDelay()` uses it as the lower bound of the renewal delay. If a user sets this to `0`, both can return `0ms`, so a persistent resolution failure (or an already-expired token) degenerates into a tight retry loop that hammers the token file and the credential providers with no delay. Rejecting non-positive values at config time prevents this.
```suggestion
.checkValue(_ > 0, "The minimum renewal interval must be a positive time value.")
.createWithDefaultString("30s")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement UserCredentialManager [spark]
Posted by "shrirangmhalgi (via GitHub)" <gi...@apache.org>.
shrirangmhalgi commented on code in PR #57387:
URL: https://github.com/apache/spark/pull/57387#discussion_r3627526537
##########
core/src/main/scala/org/apache/spark/deploy/security/UserCredentialManager.scala:
##########
@@ -0,0 +1,433 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.security
+
+import java.io.{ByteArrayOutputStream, ObjectInputFilter, ObjectInputStream, ObjectOutputStream}
+import java.net.URI
+import java.nio.file.Paths
+import java.time.Instant
+import java.util.concurrent.{RejectedExecutionException, ScheduledExecutorService, TimeUnit}
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.LogKeys
+import org.apache.spark.internal.config._
+import org.apache.spark.security._
+import org.apache.spark.ui.UIUtils
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Manager for OIDC-based credential propagation on the driver.
+ *
+ * This class is a sibling of [[HadoopDelegationTokenManager]] and handles the OIDC credential
+ * propagation path independently. Both managers run on independent threads and can both be
+ * active simultaneously (e.g., a cluster accessing HDFS via Kerberos and S3 via OIDC).
+ *
+ * Responsibilities:
+ * 1. Reads the current identity token via a [[TokenIngestor]]
+ * 2. Calls [[CredentialProvider#resolve]] for each configured scheme
+ * 3. Serializes the resulting [[UserCredentials]] and invokes the propagation callback
+ * 4. Schedules renewal based on `min(identity token expiry, service credential expiry) -
+ * safetyMargin`
+ * 5. Retries with exponential backoff on failure
+ *
+ * Started from `CoarseGrainedSchedulerBackend.start()` when
+ * `spark.security.credentials.enabled=true`, independently of
+ * `UserGroupInformation.isSecurityEnabled()`.
+ *
+ * Lifecycle: call `start()` exactly once, then `stop()` to shut down.
+ * Calling `start()` after `stop()` is not supported.
+ */
+private[spark] class UserCredentialManager(
+ sparkConf: SparkConf,
+ tokenIngestor: TokenIngestor,
+ onCredentialsUpdate: Array[Byte] => Unit) extends Logging {
+
+ private val safetyMargin = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_SAFETY_MARGIN)
+ private val minInterval = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_MIN_INTERVAL)
+
+ // Thread-safe counter for exponential backoff calculation.
+ // Accessed from both the caller of start() and the scheduled renewal thread.
+ private val consecutiveFailures = new AtomicInteger(0)
+ private val maxBackoffMs: Long = TimeUnit.MINUTES.toMillis(10)
+
+ @volatile private var renewalExecutor: ScheduledExecutorService = _
+
+ // Pre-compute the filtered credential configuration map (SparkConf is immutable).
+ private val credentialConfMap: java.util.Map[String, String] = sparkConf.getAll
+ .filter(_._1.startsWith("spark.security.credentials."))
+ .toMap.asJava
+
+ /**
+ * Start the credential manager. Acquires initial credentials and schedules renewal.
+ *
+ * The initial credential acquisition is fail-fast: if the token cannot be loaded or
+ * no credentials can be resolved, an exception is thrown. Subsequent renewal failures
+ * are handled with exponential backoff.
+ *
+ * @return The serialized initial [[UserCredentials]].
+ * @throws IllegalStateException if the initial credential acquisition fails.
+ */
+ def start(): Array[Byte] = {
+ require(renewalExecutor == null, "start() must not be called more than once")
+ renewalExecutor =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("user-credential-renewal")
+
+ // Initial acquisition is fail-fast (no retry/backoff).
+ // This ensures the application fails to start if credentials cannot be obtained,
+ // rather than running with null credentials.
+ val userContext = tokenIngestor.load()
+ if (!userContext.isPresent) {
+ throw new IllegalStateException(
+ "Failed to start UserCredentialManager: identity token file is missing or malformed. " +
+ s"Check ${SECURITY_CREDENTIALS_IDENTITY_TOKEN_FILE.key} configuration.")
+ }
+
+ val ctx = userContext.get()
+ logInfo(log"Initial identity token loaded for principal " +
+ log"${MDC(LogKeys.PRINCIPAL, ctx.getPrincipal)} " +
+ log"(issuer: ${MDC(LogKeys.URI, ctx.getIssuer)})")
+
+ val (credentials, earliestExpiry) = resolveCredentials(ctx)
+ val serialized = serializeUserCredentials(credentials)
+
+ // Propagate initial credentials
+ onCredentialsUpdate(serialized)
+
+ // Schedule first renewal
+ val renewalDelay = computeRenewalDelay(ctx, earliestExpiry)
+ scheduleRenewal(renewalDelay)
+
+ logInfo(log"Credential acquisition successful. Next renewal in " +
+ log"${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(renewalDelay))}.")
+ serialized
+ }
+
+ def stop(): Unit = {
+ if (renewalExecutor != null) {
+ renewalExecutor.shutdownNow()
+ }
+ }
+
+ /**
+ * Scheduled renewal task: load identity token, resolve credentials, propagate,
+ * and schedule the next renewal. Failures are retried with exponential backoff.
+ */
+ private def renewCredentialsTask(): Unit = {
+ try {
+ val userContext = tokenIngestor.load()
Review Comment:
(non-blocking): `tokenIngestor.load()` here does `Files.readAllBytes` on the renewal thread with no timeout. In Kubernetes environments with projected service account tokens, the token volume can become temporarily unresponsive under kubelet pressure - blocking the single renewal thread indefinitely and causing credentials to go stale with no retry path.
HDTM has the same pattern (no timeout on `checkTGTAndReloginFromKeytab`), but OIDC deployments are more likely to use network-mounted token files (FUSE, CSI drivers) than Kerberos keytabs. Would a `Future`-based timeout be worth tracking as a follow-up?
##########
core/src/main/scala/org/apache/spark/deploy/security/UserCredentialManager.scala:
##########
@@ -0,0 +1,433 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.security
+
+import java.io.{ByteArrayOutputStream, ObjectInputFilter, ObjectInputStream, ObjectOutputStream}
+import java.net.URI
+import java.nio.file.Paths
+import java.time.Instant
+import java.util.concurrent.{RejectedExecutionException, ScheduledExecutorService, TimeUnit}
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.LogKeys
+import org.apache.spark.internal.config._
+import org.apache.spark.security._
+import org.apache.spark.ui.UIUtils
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Manager for OIDC-based credential propagation on the driver.
+ *
+ * This class is a sibling of [[HadoopDelegationTokenManager]] and handles the OIDC credential
+ * propagation path independently. Both managers run on independent threads and can both be
+ * active simultaneously (e.g., a cluster accessing HDFS via Kerberos and S3 via OIDC).
+ *
+ * Responsibilities:
+ * 1. Reads the current identity token via a [[TokenIngestor]]
+ * 2. Calls [[CredentialProvider#resolve]] for each configured scheme
+ * 3. Serializes the resulting [[UserCredentials]] and invokes the propagation callback
+ * 4. Schedules renewal based on `min(identity token expiry, service credential expiry) -
+ * safetyMargin`
+ * 5. Retries with exponential backoff on failure
+ *
+ * Started from `CoarseGrainedSchedulerBackend.start()` when
+ * `spark.security.credentials.enabled=true`, independently of
+ * `UserGroupInformation.isSecurityEnabled()`.
+ *
+ * Lifecycle: call `start()` exactly once, then `stop()` to shut down.
+ * Calling `start()` after `stop()` is not supported.
+ */
+private[spark] class UserCredentialManager(
+ sparkConf: SparkConf,
+ tokenIngestor: TokenIngestor,
+ onCredentialsUpdate: Array[Byte] => Unit) extends Logging {
+
+ private val safetyMargin = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_SAFETY_MARGIN)
+ private val minInterval = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_MIN_INTERVAL)
+
+ // Thread-safe counter for exponential backoff calculation.
+ // Accessed from both the caller of start() and the scheduled renewal thread.
+ private val consecutiveFailures = new AtomicInteger(0)
+ private val maxBackoffMs: Long = TimeUnit.MINUTES.toMillis(10)
+
+ @volatile private var renewalExecutor: ScheduledExecutorService = _
+
+ // Pre-compute the filtered credential configuration map (SparkConf is immutable).
+ private val credentialConfMap: java.util.Map[String, String] = sparkConf.getAll
+ .filter(_._1.startsWith("spark.security.credentials."))
+ .toMap.asJava
+
+ /**
+ * Start the credential manager. Acquires initial credentials and schedules renewal.
+ *
+ * The initial credential acquisition is fail-fast: if the token cannot be loaded or
+ * no credentials can be resolved, an exception is thrown. Subsequent renewal failures
+ * are handled with exponential backoff.
+ *
+ * @return The serialized initial [[UserCredentials]].
+ * @throws IllegalStateException if the initial credential acquisition fails.
+ */
+ def start(): Array[Byte] = {
+ require(renewalExecutor == null, "start() must not be called more than once")
+ renewalExecutor =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("user-credential-renewal")
+
+ // Initial acquisition is fail-fast (no retry/backoff).
+ // This ensures the application fails to start if credentials cannot be obtained,
+ // rather than running with null credentials.
+ val userContext = tokenIngestor.load()
+ if (!userContext.isPresent) {
+ throw new IllegalStateException(
+ "Failed to start UserCredentialManager: identity token file is missing or malformed. " +
+ s"Check ${SECURITY_CREDENTIALS_IDENTITY_TOKEN_FILE.key} configuration.")
+ }
+
+ val ctx = userContext.get()
+ logInfo(log"Initial identity token loaded for principal " +
+ log"${MDC(LogKeys.PRINCIPAL, ctx.getPrincipal)} " +
+ log"(issuer: ${MDC(LogKeys.URI, ctx.getIssuer)})")
+
+ val (credentials, earliestExpiry) = resolveCredentials(ctx)
+ val serialized = serializeUserCredentials(credentials)
+
+ // Propagate initial credentials
+ onCredentialsUpdate(serialized)
+
+ // Schedule first renewal
+ val renewalDelay = computeRenewalDelay(ctx, earliestExpiry)
+ scheduleRenewal(renewalDelay)
+
+ logInfo(log"Credential acquisition successful. Next renewal in " +
+ log"${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(renewalDelay))}.")
+ serialized
+ }
+
+ def stop(): Unit = {
+ if (renewalExecutor != null) {
+ renewalExecutor.shutdownNow()
+ }
+ }
+
+ /**
+ * Scheduled renewal task: load identity token, resolve credentials, propagate,
+ * and schedule the next renewal. Failures are retried with exponential backoff.
+ */
+ private def renewCredentialsTask(): Unit = {
+ try {
+ val userContext = tokenIngestor.load()
+ if (!userContext.isPresent) {
+ throw new IllegalStateException(
+ "TokenIngestor returned empty - identity token file may be missing or malformed")
+ }
+
+ val ctx = userContext.get()
+ logInfo(log"Loaded identity token for principal " +
+ log"${MDC(LogKeys.PRINCIPAL, ctx.getPrincipal)} " +
+ log"(issuer: ${MDC(LogKeys.URI, ctx.getIssuer)})")
+
+ val (credentials, earliestExpiry) = resolveCredentials(ctx)
+ val serialized = serializeUserCredentials(credentials)
+
+ // Propagate credentials to executors. Errors here are logged separately
+ // so that credential-fetch success is not conflated with distribution failure.
+ try {
+ onCredentialsUpdate(serialized)
+ } catch {
+ case e: Exception =>
+ logWarning(log"Credentials were resolved successfully but failed to propagate " +
+ log"to executors. Executors will receive updated credentials on next renewal.", e)
+ }
+
+ // Reset backoff on successful credential resolution. This is intentionally done
+ // even if onCredentialsUpdate failed above: the backoff counter tracks credential
+ // *resolution* failures (STS/provider issues), not propagation failures.
+ // Propagation failures are transient and will be retried on next scheduled renewal.
+ consecutiveFailures.set(0)
Review Comment:
nit (non-blocking): When `onCredentialsUpdate` throws here, `consecutiveFailures` is reset to 0 and the next renewal is at the full scheduled interval. This means executors could run with stale credentials for the entire renewal period even though fresh credentials are already resolved and sitting in memory.
Would a faster propagation-only retry (shorter delay, skip `tokenIngestor.load()` + `resolveCredentials()`) be worth adding for the case where resolution succeeds but distribution fails?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement UserCredentialManager [spark]
Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on code in PR #57387:
URL: https://github.com/apache/spark/pull/57387#discussion_r3629871761
##########
core/src/main/scala/org/apache/spark/deploy/security/UserCredentialManager.scala:
##########
@@ -0,0 +1,433 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.security
+
+import java.io.{ByteArrayOutputStream, ObjectInputFilter, ObjectInputStream, ObjectOutputStream}
+import java.net.URI
+import java.nio.file.Paths
+import java.time.Instant
+import java.util.concurrent.{RejectedExecutionException, ScheduledExecutorService, TimeUnit}
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.LogKeys
+import org.apache.spark.internal.config._
+import org.apache.spark.security._
+import org.apache.spark.ui.UIUtils
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Manager for OIDC-based credential propagation on the driver.
+ *
+ * This class is a sibling of [[HadoopDelegationTokenManager]] and handles the OIDC credential
+ * propagation path independently. Both managers run on independent threads and can both be
+ * active simultaneously (e.g., a cluster accessing HDFS via Kerberos and S3 via OIDC).
+ *
+ * Responsibilities:
+ * 1. Reads the current identity token via a [[TokenIngestor]]
+ * 2. Calls [[CredentialProvider#resolve]] for each configured scheme
+ * 3. Serializes the resulting [[UserCredentials]] and invokes the propagation callback
+ * 4. Schedules renewal based on `min(identity token expiry, service credential expiry) -
+ * safetyMargin`
+ * 5. Retries with exponential backoff on failure
+ *
+ * Started from `CoarseGrainedSchedulerBackend.start()` when
+ * `spark.security.credentials.enabled=true`, independently of
+ * `UserGroupInformation.isSecurityEnabled()`.
+ *
+ * Lifecycle: call `start()` exactly once, then `stop()` to shut down.
+ * Calling `start()` after `stop()` is not supported.
+ */
+private[spark] class UserCredentialManager(
+ sparkConf: SparkConf,
+ tokenIngestor: TokenIngestor,
+ onCredentialsUpdate: Array[Byte] => Unit) extends Logging {
+
+ private val safetyMargin = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_SAFETY_MARGIN)
+ private val minInterval = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_MIN_INTERVAL)
+
+ // Thread-safe counter for exponential backoff calculation.
+ // Accessed from both the caller of start() and the scheduled renewal thread.
+ private val consecutiveFailures = new AtomicInteger(0)
+ private val maxBackoffMs: Long = TimeUnit.MINUTES.toMillis(10)
+
+ @volatile private var renewalExecutor: ScheduledExecutorService = _
+
+ // Pre-compute the filtered credential configuration map (SparkConf is immutable).
+ private val credentialConfMap: java.util.Map[String, String] = sparkConf.getAll
+ .filter(_._1.startsWith("spark.security.credentials."))
+ .toMap.asJava
+
+ /**
+ * Start the credential manager. Acquires initial credentials and schedules renewal.
+ *
+ * The initial credential acquisition is fail-fast: if the token cannot be loaded or
+ * no credentials can be resolved, an exception is thrown. Subsequent renewal failures
+ * are handled with exponential backoff.
+ *
+ * @return The serialized initial [[UserCredentials]].
+ * @throws IllegalStateException if the initial credential acquisition fails.
+ */
+ def start(): Array[Byte] = {
+ require(renewalExecutor == null, "start() must not be called more than once")
Review Comment:
Good catch. I've fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement UserCredentialManager [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #57387:
URL: https://github.com/apache/spark/pull/57387#discussion_r3650928697
##########
core/src/main/scala/org/apache/spark/deploy/security/UserCredentialManager.scala:
##########
@@ -0,0 +1,443 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.security
+
+import java.io.{ByteArrayOutputStream, ObjectInputFilter, ObjectInputStream, ObjectOutputStream}
+import java.net.URI
+import java.nio.file.Paths
+import java.time.Instant
+import java.util.concurrent.{RejectedExecutionException, ScheduledExecutorService, TimeUnit}
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.LogKeys
+import org.apache.spark.internal.config._
+import org.apache.spark.security._
+import org.apache.spark.ui.UIUtils
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Manager for OIDC-based credential propagation on the driver.
+ *
+ * This class is a sibling of [[HadoopDelegationTokenManager]] and handles the OIDC credential
+ * propagation path independently. Both managers run on independent threads and can both be
+ * active simultaneously (e.g., a cluster accessing HDFS via Kerberos and S3 via OIDC).
+ *
+ * Responsibilities:
+ * 1. Reads the current identity token via a [[TokenIngestor]]
+ * 2. Calls [[CredentialProvider#resolve]] for each configured scheme
+ * 3. Serializes the resulting [[UserCredentials]] and invokes the propagation callback
+ * 4. Schedules renewal based on `min(identity token expiry, service credential expiry) -
+ * safetyMargin`
+ * 5. Retries with exponential backoff on failure
+ *
+ * Intended to be started from `CoarseGrainedSchedulerBackend.start()` when
+ * `spark.security.credentials.enabled=true`, independently of
+ * `UserGroupInformation.isSecurityEnabled()`.
+ *
+ * Lifecycle: call `start()` exactly once, then `stop()` to shut down.
+ * Calling `start()` after `stop()` is not supported.
+ */
+private[spark] class UserCredentialManager(
+ sparkConf: SparkConf,
+ tokenIngestor: TokenIngestor,
+ onCredentialsUpdate: Array[Byte] => Unit) extends Logging {
+
+ private val safetyMargin = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_SAFETY_MARGIN)
+ private val minInterval = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_MIN_INTERVAL)
+
+ // Thread-safe counter for exponential backoff calculation.
+ // Accessed from both the caller of start() and the scheduled renewal thread.
+ private val consecutiveFailures = new AtomicInteger(0)
+ private val maxBackoffMs: Long = UserCredentialManager.MAX_BACKOFF_MS
+
+ @volatile private var renewalExecutor: ScheduledExecutorService = _
+
+ // Pre-compute the filtered credential configuration map (SparkConf is immutable).
Review Comment:
Technically, `SparkConf` is mutable. Please remove this and revise this comment by saying like this is a snapshot of creation time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement UserCredentialManager [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #57387:
URL: https://github.com/apache/spark/pull/57387#discussion_r3650952528
##########
core/src/main/java/org/apache/spark/security/CredentialProviderLoader.java:
##########
@@ -221,10 +223,38 @@ private static List<CredentialProvider> loadProviders() {
return result;
}
+ /**
+ * Discovers all URI schemes supported by providers on the classpath.
+ * <p>
+ * This method queries all discovered {@link CredentialProvider} instances and collects
+ * their {@link CredentialProvider#supportedSchemes()} into a single set. Unlike
+ * {@link #providerFor(String, Map)}, this does not initialize providers or apply
+ * explicit selection rules -- it only reports what schemes are potentially available.
+ * <p>
+ * Intended for use by {@code UserCredentialManager} when no explicit scheme configuration
+ * (e.g., {@code spark.security.credentials.provider.<scheme>}) is provided.
+ *
+ * @return a set of all supported scheme names (lowercased), possibly empty
+ */
+ public static Set<String> discoverAllSchemes() {
+ List<CredentialProvider> providers = getProviders();
+ Set<String> schemes = new java.util.HashSet<>();
+ for (CredentialProvider provider : providers) {
+ Set<String> providerSchemes = provider.supportedSchemes();
+ if (providerSchemes != null) {
Review Comment:
`providerFor` throws an `IllegalStateException` when a provider returns `null` from `supportedSchemes()`, while this method silently skips such providers. The javadoc explains the divergence on initialization and selection rules, but not this one. If the leniency is intentional (a misbehaving third-party provider shouldn't break auto-discovery), consider mentioning it in the javadoc — e.g. "Providers returning null from supportedSchemes() are skipped" — so the asymmetry with `providerFor` reads as deliberate rather than accidental.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement UserCredentialManager [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #57387:
URL: https://github.com/apache/spark/pull/57387#discussion_r3650946749
##########
core/src/main/java/org/apache/spark/security/CredentialProviderLoader.java:
##########
@@ -221,10 +223,38 @@ private static List<CredentialProvider> loadProviders() {
return result;
}
+ /**
+ * Discovers all URI schemes supported by providers on the classpath.
+ * <p>
+ * This method queries all discovered {@link CredentialProvider} instances and collects
+ * their {@link CredentialProvider#supportedSchemes()} into a single set. Unlike
+ * {@link #providerFor(String, Map)}, this does not initialize providers or apply
+ * explicit selection rules -- it only reports what schemes are potentially available.
+ * <p>
+ * Intended for use by {@code UserCredentialManager} when no explicit scheme configuration
+ * (e.g., {@code spark.security.credentials.provider.<scheme>}) is provided.
+ *
+ * @return a set of all supported scheme names (lowercased), possibly empty
+ */
+ public static Set<String> discoverAllSchemes() {
+ List<CredentialProvider> providers = getProviders();
+ Set<String> schemes = new java.util.HashSet<>();
Review Comment:
Please add an import statement, `import java.util.HashSet;` and use it.
```suggestion
Set<String> schemes = new HashSet<>();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement `UserCredentialManager` [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #57387:
URL: https://github.com/apache/spark/pull/57387#discussion_r3666995988
##########
core/src/main/scala/org/apache/spark/deploy/security/UserCredentialManager.scala:
##########
@@ -0,0 +1,441 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.security
+
+import java.io.{ByteArrayOutputStream, ObjectInputFilter, ObjectInputStream, ObjectOutputStream}
+import java.net.URI
+import java.nio.file.Paths
+import java.time.Instant
+import java.util.concurrent.{RejectedExecutionException, ScheduledExecutorService, TimeUnit}
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.LogKeys
+import org.apache.spark.internal.config._
+import org.apache.spark.security._
+import org.apache.spark.ui.UIUtils
+import org.apache.spark.util.{ThreadUtils, Utils}
+
+/**
+ * Manager for OIDC-based credential propagation on the driver.
+ *
+ * This class is a sibling of [[HadoopDelegationTokenManager]] and handles the OIDC credential
+ * propagation path independently. Both managers run on independent threads and can both be
+ * active simultaneously (e.g., a cluster accessing HDFS via Kerberos and S3 via OIDC).
+ *
+ * Responsibilities:
+ * 1. Reads the current identity token via a [[TokenIngestor]]
+ * 2. Calls [[CredentialProvider#resolve]] for each configured scheme
+ * 3. Serializes the resulting [[UserCredentials]] and invokes the propagation callback
+ * 4. Schedules renewal based on `min(identity token expiry, service credential expiry) -
+ * safetyMargin`
+ * 5. Retries with exponential backoff on failure
+ *
+ * Intended to be started from `CoarseGrainedSchedulerBackend.start()` when
+ * `spark.security.oidc.enabled=true`, independently of
+ * `UserGroupInformation.isSecurityEnabled()`.
+ *
+ * Lifecycle: call `start()` exactly once, then `stop()` to shut down.
+ * Calling `start()` after `stop()` is not supported.
+ */
+private[spark] class UserCredentialManager(
+ sparkConf: SparkConf,
+ tokenIngestor: TokenIngestor,
+ onCredentialsUpdate: Array[Byte] => Unit) extends Logging {
+
+ private val safetyMargin = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_SAFETY_MARGIN)
+ private val minInterval = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_MIN_INTERVAL)
+
+ // Counter for exponential backoff calculation.
+ // Only accessed from the single-thread renewal executor.
+ private var consecutiveFailures: Int = 0
+ private val maxBackoffMs: Long = UserCredentialManager.MAX_BACKOFF_MS
+
+ private var renewalExecutor: ScheduledExecutorService = _
+
+ // Snapshot of credential-related configuration at construction time.
+ private val credentialConfMap: java.util.Map[String, String] = sparkConf.getAll
+ .filter(_._1.startsWith("spark.security.oidc."))
+ .toMap.asJava
+
+ /**
+ * Start the credential manager. Acquires initial credentials and schedules renewal.
+ *
+ * The initial credential acquisition is fail-fast: if the token cannot be loaded or
+ * no credentials can be resolved, an exception is thrown. Subsequent renewal failures
+ * are handled with exponential backoff.
+ *
+ * @return The serialized initial [[UserCredentials]].
+ * @throws IllegalStateException if the initial credential acquisition fails.
+ */
+ def start(): Array[Byte] = {
+ require(renewalExecutor == null, "start() must not be called more than once")
+
+ // Initial acquisition is fail-fast (no retry/backoff).
+ // This ensures the application fails to start if credentials cannot be obtained,
+ // rather than running with null credentials.
+ val userContext = tokenIngestor.load()
+ if (!userContext.isPresent) {
+ throw new IllegalStateException(
+ "Failed to start UserCredentialManager: identity token file is missing or malformed. " +
+ s"Check ${SECURITY_CREDENTIALS_IDENTITY_TOKEN_FILE.key} configuration.")
+ }
+
+ val ctx = userContext.get()
+ logInfo(log"Initial identity token loaded for principal " +
+ log"${MDC(LogKeys.PRINCIPAL, ctx.getPrincipal)} " +
+ log"(issuer: ${MDC(LogKeys.URI, ctx.getIssuer)})")
+
+ val (credentials, earliestExpiry) = resolveCredentials(ctx)
+ val serialized = UserCredentialManager.serializeUserCredentials(credentials)
+
+ // Propagate initial credentials
+ onCredentialsUpdate(serialized)
+
+ // Create the renewal executor only after successful initial acquisition.
+ // This avoids leaking a daemon thread if the fail-fast path throws, and
+ // keeps the require() guard valid for retry scenarios.
+ renewalExecutor =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("user-credential-renewal")
+
+ // Schedule first renewal
+ val renewalDelay = computeRenewalDelay(ctx, earliestExpiry)
+ scheduleRenewal(renewalDelay)
+
+ logInfo(log"Credential acquisition successful. Next renewal in " +
+ log"${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(renewalDelay))}.")
+ serialized
+ }
+
+ def stop(): Unit = {
+ if (renewalExecutor != null) {
+ renewalExecutor.shutdownNow()
+ }
+ }
+
+ /**
+ * Scheduled renewal task: load identity token, resolve credentials, propagate,
+ * and schedule the next renewal. Failures are retried with exponential backoff.
+ */
+ private def renewCredentialsTask(): Unit = {
+ try {
+ val userContext = tokenIngestor.load()
+ if (!userContext.isPresent) {
+ throw new IllegalStateException(
+ "TokenIngestor returned empty - identity token file may be missing or malformed")
+ }
+
+ val ctx = userContext.get()
+ logInfo(log"Loaded identity token for principal " +
+ log"${MDC(LogKeys.PRINCIPAL, ctx.getPrincipal)} " +
+ log"(issuer: ${MDC(LogKeys.URI, ctx.getIssuer)})")
+
+ val (credentials, earliestExpiry) = resolveCredentials(ctx)
+ val serialized = UserCredentialManager.serializeUserCredentials(credentials)
+
+ // Propagate credentials to executors. Errors here are logged separately
+ // so that credential-fetch success is not conflated with distribution failure.
+ try {
+ onCredentialsUpdate(serialized)
+ } catch {
+ case e: Exception =>
+ logWarning(log"Credentials were resolved successfully but failed to propagate " +
+ log"to executors. Executors will receive updated credentials on next renewal.", e)
+ }
+
+ // Reset backoff on successful credential resolution. This is intentionally done
+ // even if onCredentialsUpdate failed above: the backoff counter tracks credential
+ // *resolution* failures (STS/provider issues), not propagation failures.
+ // Propagation failures are transient and will be retried on next scheduled renewal.
+ consecutiveFailures = 0
+
+ // Schedule next renewal
+ val renewalDelay = computeRenewalDelay(ctx, earliestExpiry)
+ scheduleRenewal(renewalDelay)
+
+ logInfo(log"Credential renewal successful. Next renewal in " +
+ log"${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(renewalDelay))}.")
+ } catch {
+ case _: InterruptedException =>
+ // Shutting down, ignore
+ case e: Exception =>
+ consecutiveFailures += 1
+ val failures = consecutiveFailures
+ val delay = computeBackoffDelay()
+ logWarning(log"Failed to renew user credentials (attempt " +
+ log"${MDC(LogKeys.NUM_RETRY, failures)}), " +
+ log"will retry in ${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(delay))}.", e)
+ scheduleRenewal(delay)
+ }
Review Comment:
At least, logging?
```suggestion
case t: Throwable =>
// A fatal error would otherwise be swallowed by the executor, silently stopping
// all future renewals. Log it before propagating.
logError(log"Fatal error during credential renewal; no further renewals will be " +
log"scheduled.", t)
throw t
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement `UserCredentialManager` [spark]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #57387:
URL: https://github.com/apache/spark/pull/57387#issuecomment-5107358012
> FYI, cc @peter-toth and @parthchandra
Thanks for flagging me on this @dongjoon-hyun. I believe there is a small overlap between this and https://github.com/apache/spark/pull/57285, but as discussed in the SPIP, both are solving different cases and can exist side by side. Also, it doesn't look like there will be any merge conflicts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement UserCredentialManager [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #57387:
URL: https://github.com/apache/spark/pull/57387#discussion_r3650943579
##########
core/src/main/scala/org/apache/spark/deploy/security/UserCredentialManager.scala:
##########
@@ -0,0 +1,443 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.security
+
+import java.io.{ByteArrayOutputStream, ObjectInputFilter, ObjectInputStream, ObjectOutputStream}
+import java.net.URI
+import java.nio.file.Paths
+import java.time.Instant
+import java.util.concurrent.{RejectedExecutionException, ScheduledExecutorService, TimeUnit}
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.LogKeys
+import org.apache.spark.internal.config._
+import org.apache.spark.security._
+import org.apache.spark.ui.UIUtils
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Manager for OIDC-based credential propagation on the driver.
+ *
+ * This class is a sibling of [[HadoopDelegationTokenManager]] and handles the OIDC credential
+ * propagation path independently. Both managers run on independent threads and can both be
+ * active simultaneously (e.g., a cluster accessing HDFS via Kerberos and S3 via OIDC).
+ *
+ * Responsibilities:
+ * 1. Reads the current identity token via a [[TokenIngestor]]
+ * 2. Calls [[CredentialProvider#resolve]] for each configured scheme
+ * 3. Serializes the resulting [[UserCredentials]] and invokes the propagation callback
+ * 4. Schedules renewal based on `min(identity token expiry, service credential expiry) -
+ * safetyMargin`
+ * 5. Retries with exponential backoff on failure
+ *
+ * Intended to be started from `CoarseGrainedSchedulerBackend.start()` when
+ * `spark.security.credentials.enabled=true`, independently of
+ * `UserGroupInformation.isSecurityEnabled()`.
+ *
+ * Lifecycle: call `start()` exactly once, then `stop()` to shut down.
+ * Calling `start()` after `stop()` is not supported.
+ */
+private[spark] class UserCredentialManager(
+ sparkConf: SparkConf,
+ tokenIngestor: TokenIngestor,
+ onCredentialsUpdate: Array[Byte] => Unit) extends Logging {
+
+ private val safetyMargin = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_SAFETY_MARGIN)
+ private val minInterval = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_MIN_INTERVAL)
+
+ // Thread-safe counter for exponential backoff calculation.
+ // Accessed from both the caller of start() and the scheduled renewal thread.
+ private val consecutiveFailures = new AtomicInteger(0)
+ private val maxBackoffMs: Long = UserCredentialManager.MAX_BACKOFF_MS
+
+ @volatile private var renewalExecutor: ScheduledExecutorService = _
+
+ // Pre-compute the filtered credential configuration map (SparkConf is immutable).
+ private val credentialConfMap: java.util.Map[String, String] = sparkConf.getAll
+ .filter(_._1.startsWith("spark.security.credentials."))
+ .toMap.asJava
+
+ /**
+ * Start the credential manager. Acquires initial credentials and schedules renewal.
+ *
+ * The initial credential acquisition is fail-fast: if the token cannot be loaded or
+ * no credentials can be resolved, an exception is thrown. Subsequent renewal failures
+ * are handled with exponential backoff.
+ *
+ * @return The serialized initial [[UserCredentials]].
+ * @throws IllegalStateException if the initial credential acquisition fails.
+ */
+ def start(): Array[Byte] = {
+ require(renewalExecutor == null, "start() must not be called more than once")
+
+ // Initial acquisition is fail-fast (no retry/backoff).
+ // This ensures the application fails to start if credentials cannot be obtained,
+ // rather than running with null credentials.
+ val userContext = tokenIngestor.load()
+ if (!userContext.isPresent) {
+ throw new IllegalStateException(
+ "Failed to start UserCredentialManager: identity token file is missing or malformed. " +
+ s"Check ${SECURITY_CREDENTIALS_IDENTITY_TOKEN_FILE.key} configuration.")
+ }
+
+ val ctx = userContext.get()
+ logInfo(log"Initial identity token loaded for principal " +
+ log"${MDC(LogKeys.PRINCIPAL, ctx.getPrincipal)} " +
+ log"(issuer: ${MDC(LogKeys.URI, ctx.getIssuer)})")
+
+ val (credentials, earliestExpiry) = resolveCredentials(ctx)
+ val serialized = serializeUserCredentials(credentials)
+
+ // Propagate initial credentials
+ onCredentialsUpdate(serialized)
+
+ // Create the renewal executor only after successful initial acquisition.
+ // This avoids leaking a daemon thread if the fail-fast path throws, and
+ // keeps the require() guard valid for retry scenarios.
+ renewalExecutor =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("user-credential-renewal")
+
+ // Schedule first renewal
+ val renewalDelay = computeRenewalDelay(ctx, earliestExpiry)
+ scheduleRenewal(renewalDelay)
+
+ logInfo(log"Credential acquisition successful. Next renewal in " +
+ log"${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(renewalDelay))}.")
+ serialized
+ }
+
+ def stop(): Unit = {
Review Comment:
It seems that there is a small lifecycle race condition: if `stop()` is called while `start()` is still running (before `renewalExecutor` is assigned), `stop()` is a no-op, and `start()` then goes on to create the executor and schedule renewals — leaking a daemon thread that keeps renewing credentials after shutdown. A `stopped` flag makes it airtight since both fields are volatile (each thread is guaranteed to observe the other's write). Could you consider that?
```scala
@volatile private var stopped = false
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement UserCredentialManager [spark]
Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on PR #57387:
URL: https://github.com/apache/spark/pull/57387#issuecomment-5105098461
Thank you for the feedback, @dongjoon-hyun and @yadavay-amzn.
I've fixed all of them except [one](https://github.com/apache/spark/pull/57387#discussion_r3665812339).
I also found one issue that the type of `consecutiveFailures` was unnecessarily set to `AtomicInteger` and that the comment about the field was incorrect(f6b0d38, 1782165).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement UserCredentialManager [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #57387:
URL: https://github.com/apache/spark/pull/57387#discussion_r3666872518
##########
core/src/main/java/org/apache/spark/security/CredentialProviderLoader.java:
##########
@@ -221,10 +223,38 @@ private static List<CredentialProvider> loadProviders() {
return result;
}
+ /**
+ * Discovers all URI schemes supported by providers on the classpath.
+ * <p>
+ * This method queries all discovered {@link CredentialProvider} instances and collects
+ * their {@link CredentialProvider#supportedSchemes()} into a single set. Unlike
+ * {@link #providerFor(String, Map)}, this does not initialize providers or apply
+ * explicit selection rules -- it only reports what schemes are potentially available.
+ * <p>
+ * Intended for use by {@code UserCredentialManager} when no explicit scheme configuration
+ * (e.g., {@code spark.security.credentials.provider.<scheme>}) is provided.
Review Comment:
The config key prefix here should be `spark.security.oidc.provider.<scheme>`.
```suggestion
* (e.g., {@code spark.security.oidc.provider.<scheme>}) is provided.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement `UserCredentialManager` [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #57387:
URL: https://github.com/apache/spark/pull/57387#discussion_r3666985143
##########
core/src/main/scala/org/apache/spark/deploy/security/UserCredentialManager.scala:
##########
@@ -0,0 +1,441 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.security
+
+import java.io.{ByteArrayOutputStream, ObjectInputFilter, ObjectInputStream, ObjectOutputStream}
+import java.net.URI
+import java.nio.file.Paths
+import java.time.Instant
+import java.util.concurrent.{RejectedExecutionException, ScheduledExecutorService, TimeUnit}
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.LogKeys
+import org.apache.spark.internal.config._
+import org.apache.spark.security._
+import org.apache.spark.ui.UIUtils
+import org.apache.spark.util.{ThreadUtils, Utils}
+
+/**
+ * Manager for OIDC-based credential propagation on the driver.
+ *
+ * This class is a sibling of [[HadoopDelegationTokenManager]] and handles the OIDC credential
+ * propagation path independently. Both managers run on independent threads and can both be
+ * active simultaneously (e.g., a cluster accessing HDFS via Kerberos and S3 via OIDC).
+ *
+ * Responsibilities:
+ * 1. Reads the current identity token via a [[TokenIngestor]]
+ * 2. Calls [[CredentialProvider#resolve]] for each configured scheme
+ * 3. Serializes the resulting [[UserCredentials]] and invokes the propagation callback
+ * 4. Schedules renewal based on `min(identity token expiry, service credential expiry) -
+ * safetyMargin`
+ * 5. Retries with exponential backoff on failure
+ *
+ * Intended to be started from `CoarseGrainedSchedulerBackend.start()` when
+ * `spark.security.oidc.enabled=true`, independently of
+ * `UserGroupInformation.isSecurityEnabled()`.
+ *
+ * Lifecycle: call `start()` exactly once, then `stop()` to shut down.
+ * Calling `start()` after `stop()` is not supported.
+ */
+private[spark] class UserCredentialManager(
+ sparkConf: SparkConf,
+ tokenIngestor: TokenIngestor,
+ onCredentialsUpdate: Array[Byte] => Unit) extends Logging {
+
+ private val safetyMargin = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_SAFETY_MARGIN)
+ private val minInterval = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_MIN_INTERVAL)
+
+ // Counter for exponential backoff calculation.
+ // Only accessed from the single-thread renewal executor.
+ private var consecutiveFailures: Int = 0
+ private val maxBackoffMs: Long = UserCredentialManager.MAX_BACKOFF_MS
+
+ private var renewalExecutor: ScheduledExecutorService = _
+
+ // Snapshot of credential-related configuration at construction time.
+ private val credentialConfMap: java.util.Map[String, String] = sparkConf.getAll
+ .filter(_._1.startsWith("spark.security.oidc."))
+ .toMap.asJava
+
+ /**
+ * Start the credential manager. Acquires initial credentials and schedules renewal.
+ *
+ * The initial credential acquisition is fail-fast: if the token cannot be loaded or
+ * no credentials can be resolved, an exception is thrown. Subsequent renewal failures
+ * are handled with exponential backoff.
+ *
+ * @return The serialized initial [[UserCredentials]].
+ * @throws IllegalStateException if the initial credential acquisition fails.
+ */
+ def start(): Array[Byte] = {
+ require(renewalExecutor == null, "start() must not be called more than once")
+
+ // Initial acquisition is fail-fast (no retry/backoff).
+ // This ensures the application fails to start if credentials cannot be obtained,
+ // rather than running with null credentials.
+ val userContext = tokenIngestor.load()
+ if (!userContext.isPresent) {
+ throw new IllegalStateException(
+ "Failed to start UserCredentialManager: identity token file is missing or malformed. " +
+ s"Check ${SECURITY_CREDENTIALS_IDENTITY_TOKEN_FILE.key} configuration.")
+ }
+
+ val ctx = userContext.get()
+ logInfo(log"Initial identity token loaded for principal " +
+ log"${MDC(LogKeys.PRINCIPAL, ctx.getPrincipal)} " +
+ log"(issuer: ${MDC(LogKeys.URI, ctx.getIssuer)})")
+
+ val (credentials, earliestExpiry) = resolveCredentials(ctx)
+ val serialized = UserCredentialManager.serializeUserCredentials(credentials)
+
+ // Propagate initial credentials
+ onCredentialsUpdate(serialized)
+
+ // Create the renewal executor only after successful initial acquisition.
+ // This avoids leaking a daemon thread if the fail-fast path throws, and
+ // keeps the require() guard valid for retry scenarios.
+ renewalExecutor =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("user-credential-renewal")
+
+ // Schedule first renewal
+ val renewalDelay = computeRenewalDelay(ctx, earliestExpiry)
+ scheduleRenewal(renewalDelay)
+
+ logInfo(log"Credential acquisition successful. Next renewal in " +
+ log"${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(renewalDelay))}.")
+ serialized
+ }
+
+ def stop(): Unit = {
+ if (renewalExecutor != null) {
+ renewalExecutor.shutdownNow()
+ }
+ }
+
+ /**
+ * Scheduled renewal task: load identity token, resolve credentials, propagate,
+ * and schedule the next renewal. Failures are retried with exponential backoff.
+ */
+ private def renewCredentialsTask(): Unit = {
+ try {
+ val userContext = tokenIngestor.load()
+ if (!userContext.isPresent) {
+ throw new IllegalStateException(
+ "TokenIngestor returned empty - identity token file may be missing or malformed")
+ }
+
+ val ctx = userContext.get()
+ logInfo(log"Loaded identity token for principal " +
+ log"${MDC(LogKeys.PRINCIPAL, ctx.getPrincipal)} " +
+ log"(issuer: ${MDC(LogKeys.URI, ctx.getIssuer)})")
+
+ val (credentials, earliestExpiry) = resolveCredentials(ctx)
+ val serialized = UserCredentialManager.serializeUserCredentials(credentials)
+
+ // Propagate credentials to executors. Errors here are logged separately
+ // so that credential-fetch success is not conflated with distribution failure.
+ try {
+ onCredentialsUpdate(serialized)
+ } catch {
+ case e: Exception =>
+ logWarning(log"Credentials were resolved successfully but failed to propagate " +
+ log"to executors. Executors will receive updated credentials on next renewal.", e)
+ }
+
+ // Reset backoff on successful credential resolution. This is intentionally done
+ // even if onCredentialsUpdate failed above: the backoff counter tracks credential
+ // *resolution* failures (STS/provider issues), not propagation failures.
+ // Propagation failures are transient and will be retried on next scheduled renewal.
+ consecutiveFailures = 0
+
+ // Schedule next renewal
+ val renewalDelay = computeRenewalDelay(ctx, earliestExpiry)
+ scheduleRenewal(renewalDelay)
+
+ logInfo(log"Credential renewal successful. Next renewal in " +
+ log"${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(renewalDelay))}.")
+ } catch {
+ case _: InterruptedException =>
+ // Shutting down, ignore
+ case e: Exception =>
Review Comment:
It seems that we ignore `Error`. Shall we use `NonFatal` here and catch `Throwable` too like the following (and [subsequent comment](https://github.com/apache/spark/pull/57387#discussion_r3666995988)).
```suggestion
case NonFatal(e) =>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement `UserCredentialManager` [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #57387:
URL: https://github.com/apache/spark/pull/57387#discussion_r3667035666
##########
core/src/main/java/org/apache/spark/security/CredentialProviderLoader.java:
##########
@@ -221,10 +223,38 @@ private static List<CredentialProvider> loadProviders() {
return result;
}
+ /**
+ * Discovers all URI schemes supported by providers on the classpath.
+ * <p>
+ * This method queries all discovered {@link CredentialProvider} instances and collects
+ * their {@link CredentialProvider#supportedSchemes()} into a single set. Unlike
+ * {@link #providerFor(String, Map)}, this does not initialize providers or apply
+ * explicit selection rules -- it only reports what schemes are potentially available.
+ * <p>
+ * Intended for use by {@code UserCredentialManager} when no explicit scheme configuration
+ * (e.g., {@code spark.security.credentials.provider.<scheme>}) is provided.
+ *
+ * @return a set of all supported scheme names (lowercased), possibly empty
+ */
+ public static Set<String> discoverAllSchemes() {
+ List<CredentialProvider> providers = getProviders();
+ Set<String> schemes = new java.util.HashSet<>();
Review Comment:
> Added `import java.util.HashSet;` and changed to `new HashSet<>()`.
To @sarutak , the suggestion is still not accept actually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement UserCredentialManager [spark]
Posted by "yadavay-amzn (via GitHub)" <gi...@apache.org>.
yadavay-amzn commented on code in PR #57387:
URL: https://github.com/apache/spark/pull/57387#discussion_r3660437045
##########
core/src/main/scala/org/apache/spark/deploy/security/UserCredentialManager.scala:
##########
@@ -0,0 +1,443 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.security
+
+import java.io.{ByteArrayOutputStream, ObjectInputFilter, ObjectInputStream, ObjectOutputStream}
+import java.net.URI
+import java.nio.file.Paths
+import java.time.Instant
+import java.util.concurrent.{RejectedExecutionException, ScheduledExecutorService, TimeUnit}
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.LogKeys
+import org.apache.spark.internal.config._
+import org.apache.spark.security._
+import org.apache.spark.ui.UIUtils
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Manager for OIDC-based credential propagation on the driver.
+ *
+ * This class is a sibling of [[HadoopDelegationTokenManager]] and handles the OIDC credential
+ * propagation path independently. Both managers run on independent threads and can both be
+ * active simultaneously (e.g., a cluster accessing HDFS via Kerberos and S3 via OIDC).
+ *
+ * Responsibilities:
+ * 1. Reads the current identity token via a [[TokenIngestor]]
+ * 2. Calls [[CredentialProvider#resolve]] for each configured scheme
+ * 3. Serializes the resulting [[UserCredentials]] and invokes the propagation callback
+ * 4. Schedules renewal based on `min(identity token expiry, service credential expiry) -
+ * safetyMargin`
+ * 5. Retries with exponential backoff on failure
+ *
+ * Intended to be started from `CoarseGrainedSchedulerBackend.start()` when
+ * `spark.security.credentials.enabled=true`, independently of
+ * `UserGroupInformation.isSecurityEnabled()`.
+ *
+ * Lifecycle: call `start()` exactly once, then `stop()` to shut down.
+ * Calling `start()` after `stop()` is not supported.
+ */
+private[spark] class UserCredentialManager(
+ sparkConf: SparkConf,
+ tokenIngestor: TokenIngestor,
+ onCredentialsUpdate: Array[Byte] => Unit) extends Logging {
+
+ private val safetyMargin = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_SAFETY_MARGIN)
+ private val minInterval = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_MIN_INTERVAL)
+
+ // Thread-safe counter for exponential backoff calculation.
+ // Accessed from both the caller of start() and the scheduled renewal thread.
+ private val consecutiveFailures = new AtomicInteger(0)
+ private val maxBackoffMs: Long = UserCredentialManager.MAX_BACKOFF_MS
+
+ @volatile private var renewalExecutor: ScheduledExecutorService = _
+
+ // Pre-compute the filtered credential configuration map (SparkConf is immutable).
+ private val credentialConfMap: java.util.Map[String, String] = sparkConf.getAll
+ .filter(_._1.startsWith("spark.security.credentials."))
+ .toMap.asJava
+
+ /**
+ * Start the credential manager. Acquires initial credentials and schedules renewal.
+ *
+ * The initial credential acquisition is fail-fast: if the token cannot be loaded or
+ * no credentials can be resolved, an exception is thrown. Subsequent renewal failures
+ * are handled with exponential backoff.
+ *
+ * @return The serialized initial [[UserCredentials]].
+ * @throws IllegalStateException if the initial credential acquisition fails.
+ */
+ def start(): Array[Byte] = {
+ require(renewalExecutor == null, "start() must not be called more than once")
+
+ // Initial acquisition is fail-fast (no retry/backoff).
+ // This ensures the application fails to start if credentials cannot be obtained,
+ // rather than running with null credentials.
+ val userContext = tokenIngestor.load()
+ if (!userContext.isPresent) {
+ throw new IllegalStateException(
+ "Failed to start UserCredentialManager: identity token file is missing or malformed. " +
+ s"Check ${SECURITY_CREDENTIALS_IDENTITY_TOKEN_FILE.key} configuration.")
+ }
+
+ val ctx = userContext.get()
+ logInfo(log"Initial identity token loaded for principal " +
+ log"${MDC(LogKeys.PRINCIPAL, ctx.getPrincipal)} " +
+ log"(issuer: ${MDC(LogKeys.URI, ctx.getIssuer)})")
+
+ val (credentials, earliestExpiry) = resolveCredentials(ctx)
+ val serialized = serializeUserCredentials(credentials)
+
+ // Propagate initial credentials
+ onCredentialsUpdate(serialized)
+
+ // Create the renewal executor only after successful initial acquisition.
+ // This avoids leaking a daemon thread if the fail-fast path throws, and
+ // keeps the require() guard valid for retry scenarios.
+ renewalExecutor =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("user-credential-renewal")
+
+ // Schedule first renewal
+ val renewalDelay = computeRenewalDelay(ctx, earliestExpiry)
+ scheduleRenewal(renewalDelay)
+
+ logInfo(log"Credential acquisition successful. Next renewal in " +
+ log"${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(renewalDelay))}.")
+ serialized
+ }
+
+ def stop(): Unit = {
+ if (renewalExecutor != null) {
+ renewalExecutor.shutdownNow()
+ }
+ }
+
+ /**
+ * Scheduled renewal task: load identity token, resolve credentials, propagate,
+ * and schedule the next renewal. Failures are retried with exponential backoff.
+ */
+ private def renewCredentialsTask(): Unit = {
+ try {
+ val userContext = tokenIngestor.load()
+ if (!userContext.isPresent) {
+ throw new IllegalStateException(
+ "TokenIngestor returned empty - identity token file may be missing or malformed")
+ }
+
+ val ctx = userContext.get()
+ logInfo(log"Loaded identity token for principal " +
+ log"${MDC(LogKeys.PRINCIPAL, ctx.getPrincipal)} " +
+ log"(issuer: ${MDC(LogKeys.URI, ctx.getIssuer)})")
+
+ val (credentials, earliestExpiry) = resolveCredentials(ctx)
+ val serialized = serializeUserCredentials(credentials)
+
+ // Propagate credentials to executors. Errors here are logged separately
+ // so that credential-fetch success is not conflated with distribution failure.
+ try {
+ onCredentialsUpdate(serialized)
+ } catch {
+ case e: Exception =>
+ logWarning(log"Credentials were resolved successfully but failed to propagate " +
+ log"to executors. Executors will receive updated credentials on next renewal.", e)
+ }
+
+ // Reset backoff on successful credential resolution. This is intentionally done
+ // even if onCredentialsUpdate failed above: the backoff counter tracks credential
+ // *resolution* failures (STS/provider issues), not propagation failures.
+ // Propagation failures are transient and will be retried on next scheduled renewal.
+ consecutiveFailures.set(0)
+
+ // Schedule next renewal
+ val renewalDelay = computeRenewalDelay(ctx, earliestExpiry)
+ scheduleRenewal(renewalDelay)
+
+ logInfo(log"Credential renewal successful. Next renewal in " +
+ log"${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(renewalDelay))}.")
+ } catch {
+ case _: InterruptedException =>
+ // Shutting down, ignore
+ case e: Exception =>
+ val failures = consecutiveFailures.incrementAndGet()
+ val delay = computeBackoffDelay()
+ logWarning(log"Failed to renew user credentials (attempt " +
+ log"${MDC(LogKeys.NUM_RETRY, failures)}), " +
+ log"will retry in ${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(delay))}.", e)
+ scheduleRenewal(delay)
+ }
+ }
+
+ /**
+ * Resolve credentials for all schemes that have registered providers.
+ *
+ * @return Tuple of (UserCredentials, earliest expiry across all service credentials)
+ */
+ private def resolveCredentials(
+ ctx: UserContext): (UserCredentials, Option[Instant]) = {
+ val schemes = discoverSchemes()
+
+ val credentialMap = new mutable.HashMap[String, ServiceCredential]()
+ var earliestExpiry: Option[Instant] = None
+
+ for (scheme <- schemes) {
+ try {
+ val providerOpt = CredentialProviderLoader.providerFor(scheme, credentialConfMap)
+ if (providerOpt.isPresent) {
+ val provider = providerOpt.get()
+ // Use a synthetic target URI with just the scheme for initial resolution.
+ // The full URI is passed in future versions when path-specific resolution is needed.
+ // The "synthetic" authority signals this is not a real endpoint but a placeholder
+ // for scheme-based provider selection.
+ val target = new URI(scheme, UserCredentialManager.SYNTHETIC_TARGET_AUTHORITY,
+ "/", null, null)
+ val credential = provider.resolve(ctx, target)
+ credentialMap.put(scheme, credential)
Review Comment:
I think there might be an issue if a provider returns `null` from `resolve()`. The `put` happens on this line before the `getExpiresAt` read just below, so a `null` credential gets inserted into `credentialMap` first, then `credential.getExpiresAt` throws an NPE that the per-scheme `catch` swallows, but the `(scheme -> null)` entry stays in the map.
Two effects I noticed: the `credentialMap.isEmpty` fail-fast check below sees a non-empty map, so if every provider returned `null` the startup wouldn't fail as intended, and `UserCredentials` would carry a null value since it only null-checks keys. I think a possible fix is to read the credential and null-check before inserting:
```scala
val credential = provider.resolve(ctx, target)
if (credential == null) {
logWarning(log"Provider for scheme ${MDC(LogKeys.URI, scheme)} returned null; skipping.")
} else {
credentialMap.put(scheme, credential)
val expiry = credential.getExpiresAt
...
}
```
I can add a small test for the null-return case as part of task 9, since my provider is a consumer of this exact contract, and I'd document on the SPI that `resolve()` should throw rather than return null.
##########
core/src/main/scala/org/apache/spark/deploy/security/UserCredentialManager.scala:
##########
@@ -0,0 +1,443 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.security
+
+import java.io.{ByteArrayOutputStream, ObjectInputFilter, ObjectInputStream, ObjectOutputStream}
+import java.net.URI
+import java.nio.file.Paths
+import java.time.Instant
+import java.util.concurrent.{RejectedExecutionException, ScheduledExecutorService, TimeUnit}
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.LogKeys
+import org.apache.spark.internal.config._
+import org.apache.spark.security._
+import org.apache.spark.ui.UIUtils
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Manager for OIDC-based credential propagation on the driver.
+ *
+ * This class is a sibling of [[HadoopDelegationTokenManager]] and handles the OIDC credential
+ * propagation path independently. Both managers run on independent threads and can both be
+ * active simultaneously (e.g., a cluster accessing HDFS via Kerberos and S3 via OIDC).
+ *
+ * Responsibilities:
+ * 1. Reads the current identity token via a [[TokenIngestor]]
+ * 2. Calls [[CredentialProvider#resolve]] for each configured scheme
+ * 3. Serializes the resulting [[UserCredentials]] and invokes the propagation callback
+ * 4. Schedules renewal based on `min(identity token expiry, service credential expiry) -
+ * safetyMargin`
+ * 5. Retries with exponential backoff on failure
+ *
+ * Intended to be started from `CoarseGrainedSchedulerBackend.start()` when
+ * `spark.security.credentials.enabled=true`, independently of
+ * `UserGroupInformation.isSecurityEnabled()`.
+ *
+ * Lifecycle: call `start()` exactly once, then `stop()` to shut down.
+ * Calling `start()` after `stop()` is not supported.
+ */
+private[spark] class UserCredentialManager(
+ sparkConf: SparkConf,
+ tokenIngestor: TokenIngestor,
+ onCredentialsUpdate: Array[Byte] => Unit) extends Logging {
+
+ private val safetyMargin = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_SAFETY_MARGIN)
+ private val minInterval = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_MIN_INTERVAL)
+
+ // Thread-safe counter for exponential backoff calculation.
+ // Accessed from both the caller of start() and the scheduled renewal thread.
+ private val consecutiveFailures = new AtomicInteger(0)
+ private val maxBackoffMs: Long = UserCredentialManager.MAX_BACKOFF_MS
+
+ @volatile private var renewalExecutor: ScheduledExecutorService = _
+
+ // Pre-compute the filtered credential configuration map (SparkConf is immutable).
+ private val credentialConfMap: java.util.Map[String, String] = sparkConf.getAll
+ .filter(_._1.startsWith("spark.security.credentials."))
+ .toMap.asJava
+
+ /**
+ * Start the credential manager. Acquires initial credentials and schedules renewal.
+ *
+ * The initial credential acquisition is fail-fast: if the token cannot be loaded or
+ * no credentials can be resolved, an exception is thrown. Subsequent renewal failures
+ * are handled with exponential backoff.
+ *
+ * @return The serialized initial [[UserCredentials]].
+ * @throws IllegalStateException if the initial credential acquisition fails.
+ */
+ def start(): Array[Byte] = {
+ require(renewalExecutor == null, "start() must not be called more than once")
+
+ // Initial acquisition is fail-fast (no retry/backoff).
+ // This ensures the application fails to start if credentials cannot be obtained,
+ // rather than running with null credentials.
+ val userContext = tokenIngestor.load()
+ if (!userContext.isPresent) {
+ throw new IllegalStateException(
+ "Failed to start UserCredentialManager: identity token file is missing or malformed. " +
+ s"Check ${SECURITY_CREDENTIALS_IDENTITY_TOKEN_FILE.key} configuration.")
+ }
+
+ val ctx = userContext.get()
+ logInfo(log"Initial identity token loaded for principal " +
+ log"${MDC(LogKeys.PRINCIPAL, ctx.getPrincipal)} " +
+ log"(issuer: ${MDC(LogKeys.URI, ctx.getIssuer)})")
+
+ val (credentials, earliestExpiry) = resolveCredentials(ctx)
+ val serialized = serializeUserCredentials(credentials)
+
+ // Propagate initial credentials
+ onCredentialsUpdate(serialized)
+
+ // Create the renewal executor only after successful initial acquisition.
+ // This avoids leaking a daemon thread if the fail-fast path throws, and
+ // keeps the require() guard valid for retry scenarios.
+ renewalExecutor =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("user-credential-renewal")
+
+ // Schedule first renewal
+ val renewalDelay = computeRenewalDelay(ctx, earliestExpiry)
+ scheduleRenewal(renewalDelay)
+
+ logInfo(log"Credential acquisition successful. Next renewal in " +
+ log"${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(renewalDelay))}.")
+ serialized
+ }
+
+ def stop(): Unit = {
+ if (renewalExecutor != null) {
+ renewalExecutor.shutdownNow()
+ }
+ }
+
+ /**
+ * Scheduled renewal task: load identity token, resolve credentials, propagate,
+ * and schedule the next renewal. Failures are retried with exponential backoff.
+ */
+ private def renewCredentialsTask(): Unit = {
+ try {
+ val userContext = tokenIngestor.load()
+ if (!userContext.isPresent) {
+ throw new IllegalStateException(
+ "TokenIngestor returned empty - identity token file may be missing or malformed")
+ }
+
+ val ctx = userContext.get()
+ logInfo(log"Loaded identity token for principal " +
+ log"${MDC(LogKeys.PRINCIPAL, ctx.getPrincipal)} " +
+ log"(issuer: ${MDC(LogKeys.URI, ctx.getIssuer)})")
+
+ val (credentials, earliestExpiry) = resolveCredentials(ctx)
+ val serialized = serializeUserCredentials(credentials)
+
+ // Propagate credentials to executors. Errors here are logged separately
+ // so that credential-fetch success is not conflated with distribution failure.
+ try {
+ onCredentialsUpdate(serialized)
+ } catch {
+ case e: Exception =>
+ logWarning(log"Credentials were resolved successfully but failed to propagate " +
+ log"to executors. Executors will receive updated credentials on next renewal.", e)
+ }
+
+ // Reset backoff on successful credential resolution. This is intentionally done
+ // even if onCredentialsUpdate failed above: the backoff counter tracks credential
+ // *resolution* failures (STS/provider issues), not propagation failures.
+ // Propagation failures are transient and will be retried on next scheduled renewal.
+ consecutiveFailures.set(0)
+
+ // Schedule next renewal
+ val renewalDelay = computeRenewalDelay(ctx, earliestExpiry)
+ scheduleRenewal(renewalDelay)
+
+ logInfo(log"Credential renewal successful. Next renewal in " +
+ log"${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(renewalDelay))}.")
+ } catch {
+ case _: InterruptedException =>
+ // Shutting down, ignore
+ case e: Exception =>
+ val failures = consecutiveFailures.incrementAndGet()
+ val delay = computeBackoffDelay()
+ logWarning(log"Failed to renew user credentials (attempt " +
+ log"${MDC(LogKeys.NUM_RETRY, failures)}), " +
+ log"will retry in ${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(delay))}.", e)
+ scheduleRenewal(delay)
+ }
+ }
+
+ /**
+ * Resolve credentials for all schemes that have registered providers.
+ *
+ * @return Tuple of (UserCredentials, earliest expiry across all service credentials)
+ */
+ private def resolveCredentials(
+ ctx: UserContext): (UserCredentials, Option[Instant]) = {
+ val schemes = discoverSchemes()
+
+ val credentialMap = new mutable.HashMap[String, ServiceCredential]()
+ var earliestExpiry: Option[Instant] = None
+
+ for (scheme <- schemes) {
+ try {
+ val providerOpt = CredentialProviderLoader.providerFor(scheme, credentialConfMap)
+ if (providerOpt.isPresent) {
+ val provider = providerOpt.get()
+ // Use a synthetic target URI with just the scheme for initial resolution.
+ // The full URI is passed in future versions when path-specific resolution is needed.
+ // The "synthetic" authority signals this is not a real endpoint but a placeholder
+ // for scheme-based provider selection.
+ val target = new URI(scheme, UserCredentialManager.SYNTHETIC_TARGET_AUTHORITY,
+ "/", null, null)
+ val credential = provider.resolve(ctx, target)
+ credentialMap.put(scheme, credential)
+
+ val expiry = credential.getExpiresAt
+ if (expiry != null) {
+ earliestExpiry = earliestExpiry match {
+ case Some(existing) if existing.isBefore(expiry) => Some(existing)
+ case _ => Some(expiry)
+ }
+ }
+ } else {
+ logWarning(log"No credential provider found for scheme " +
+ log"${MDC(LogKeys.URI, scheme)}. Skipping.")
+ }
+ } catch {
+ case e: Exception =>
+ logWarning(log"Failed to resolve credentials for scheme " +
+ log"${MDC(LogKeys.URI, scheme)}. Skipping this provider.", e)
+ }
+ }
+
+ if (credentialMap.isEmpty) {
+ throw new IllegalStateException(
+ "No credential providers resolved any credentials. " +
+ "Check that providers are on the classpath and configured correctly.")
+ }
+
+ (new UserCredentials(credentialMap.asJava), earliestExpiry)
+ }
+
+ /**
+ * Discover all schemes that have at least one registered provider.
+ *
+ * Schemes are determined by explicit configuration keys of the form
+ * `spark.security.credentials.provider.<scheme>`. If no explicit configuration
+ * exists, the method queries `CredentialProviderLoader` to discover all providers
+ * registered via ServiceLoader and collects their supported schemes.
+ *
+ * Note: When using auto-discovery (no explicit config), multiple providers may
+ * support the same scheme. In that case, `CredentialProviderLoader.providerFor`
+ * will throw an `IllegalArgumentException` for that scheme. The caller
+ * (`resolveCredentials`) handles this gracefully via per-provider exception catching,
+ * logging a warning and continuing with remaining schemes.
+ */
+ private def discoverSchemes(): Set[String] = {
+ // Extract explicitly configured scheme -> provider mappings
+ val explicitSchemes = credentialConfMap.asScala
+ .filter { case (k, _) => k.startsWith("spark.security.credentials.provider.") }
+ .map { case (k, _) => k.stripPrefix("spark.security.credentials.provider.") }
+ .toSet
+
+ if (explicitSchemes.nonEmpty) {
+ explicitSchemes
+ } else {
+ // No explicit scheme configuration. Discover all schemes that have a provider
+ // available on the classpath by probing CredentialProviderLoader.
+ // This covers both built-in providers (e.g., connector/credential-aws for "s3a")
+ // and third-party providers registered via ServiceLoader.
+ CredentialProviderLoader.discoverAllSchemes().asScala.toSet
+ }
+ }
+
+ /**
+ * Compute the delay until next renewal.
+ * Uses min(identity token expiry, service credential expiry) - safetyMargin,
+ * bounded below by minInterval.
+ */
+ private[security] def computeRenewalDelay(
+ ctx: UserContext,
+ earliestCredentialExpiry: Option[Instant]): Long = {
+ val now = System.currentTimeMillis()
+
+ // Consider identity token expiry
+ val tokenExpiry: Option[Long] = Option(ctx.getExpiresAt).map(_.toEpochMilli)
+
+ // Consider earliest service credential expiry
+ val credExpiry: Option[Long] = earliestCredentialExpiry.map(_.toEpochMilli)
+
+ // Take the minimum of both
+ val effectiveExpiry: Option[Long] = (tokenExpiry, credExpiry) match {
+ case (Some(t), Some(c)) => Some(math.min(t, c))
+ case (Some(t), None) => Some(t)
+ case (None, Some(c)) => Some(c)
+ case (None, None) => None
+ }
+
+ effectiveExpiry match {
+ case Some(expiry) =>
+ math.max(expiry - now - safetyMargin, minInterval)
+ case None =>
+ // No expiry information available from either the identity token or service
+ // credentials. Use half of the default suggested TTL (15 min / 2 = 7.5 min,
+ // rounded down) as a conservative polling interval. This ensures credentials
+ // are refreshed even when providers don't report expiration times.
+ UserCredentialManager.DEFAULT_RENEWAL_INTERVAL_NO_EXPIRY_MS
+ }
+ }
+
+ /**
+ * Compute backoff delay using exponential backoff with jitter.
+ * Bounded by minInterval (floor) and maxBackoffMs (ceiling).
+ */
+ private[security] def computeBackoffDelay(): Long = {
+ // Guard against negative or zero shift amounts. consecutiveFailures should always
+ // be >= 1 when this is called (incremented before calling), but we protect against
+ // edge cases defensively.
+ val shiftAmount = math.max(0, math.min(consecutiveFailures.get() - 1, 6))
+ val baseDelay = minInterval * (1L << shiftAmount)
+ val cappedDelay = math.min(baseDelay, maxBackoffMs)
+ // Add 10% jitter to avoid thundering herd on recovery
+ val jitter = (cappedDelay * 0.1 * math.random()).toLong
+ math.max(cappedDelay + jitter, minInterval)
+ }
+
+ private def scheduleRenewal(delay: Long): Unit = {
+ try {
+ val renewalTask = new Runnable {
+ override def run(): Unit = {
+ renewCredentialsTask()
+ }
+ }
+ renewalExecutor.schedule(renewalTask, delay, TimeUnit.MILLISECONDS)
+ } catch {
+ case _: RejectedExecutionException =>
+ // Executor has been shut down (e.g., stop() called concurrently). This is expected
+ // during application shutdown -- no further renewals will be scheduled.
+ logDebug(log"Renewal scheduling rejected - executor is shut down.")
+ }
+ }
+
+ /**
+ * Serialize [[UserCredentials]] to a byte array using Java serialization.
+ */
+ private[security] def serializeUserCredentials(credentials: UserCredentials): Array[Byte] = {
+ val bos = new ByteArrayOutputStream()
+ try {
+ val oos = new ObjectOutputStream(bos)
+ try {
+ oos.writeObject(credentials)
+ oos.flush()
+ } finally {
+ oos.close()
+ }
+ bos.toByteArray
+ } finally {
+ bos.close()
+ }
+ }
+}
+
+private[spark] object UserCredentialManager {
+
+ /**
+ * Synthetic authority used in target URIs for scheme-based provider resolution.
+ * Providers should not rely on this value; it signals that no specific endpoint
+ * is targeted and only the URI scheme is meaningful.
+ */
+ private val SYNTHETIC_TARGET_AUTHORITY = "synthetic"
+
+ /**
+ * Maximum backoff delay between credential renewal retry attempts.
+ * Caps the exponential backoff to prevent excessively long gaps between retries.
+ */
+ private val MAX_BACKOFF_MS: Long = TimeUnit.MINUTES.toMillis(10)
+
+ /**
+ * Default renewal interval when neither the identity token nor service credentials
+ * report an expiration time. Set to 7 minutes (half of the default 15-minute
+ * suggested TTL from CredentialProvider.suggestedTtl()).
+ */
+ private val DEFAULT_RENEWAL_INTERVAL_NO_EXPIRY_MS: Long = TimeUnit.MINUTES.toMillis(7)
+
+ /**
+ * ObjectInputFilter pattern restricting deserialization to only the classes needed
+ * for UserCredentials. This prevents deserialization gadget chain attacks while
+ * allowing Java's built-in collection internal classes and arrays that HashMap uses.
+ */
+ private val DESERIALIZATION_FILTER: String =
Review Comment:
Two things I noticed that may be worth addressing, non-blocking.
First, `java.util.**` and `java.lang.**` allow the whole subtrees, which is broader than the `UserCredentials` graph actually needs (`HashMap`, `String`, `Instant`, plus the `org.apache.spark.security.*` types). Narrowing to the specific classes would shrink the surface. Second, there's no `maxarray` or `maxrefs` limit, so a forged stream with a large array-length header could drive allocation before the per-element filter helps. Something like adding `maxarray=<n>;maxrefs=<n>` alongside the existing `maxdepth=10` would cover that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement UserCredentialManager [spark]
Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on PR #57387:
URL: https://github.com/apache/spark/pull/57387#issuecomment-5045308098
@HyukjinKwon @shrirangmhalgi Thank you for reviewing!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement UserCredentialManager [spark]
Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on PR #57387:
URL: https://github.com/apache/spark/pull/57387#issuecomment-5080298080
Could you look into this too, @dongjoon-hyun?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement UserCredentialManager [spark]
Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on code in PR #57387:
URL: https://github.com/apache/spark/pull/57387#discussion_r3665742863
##########
core/src/main/scala/org/apache/spark/deploy/security/UserCredentialManager.scala:
##########
@@ -0,0 +1,443 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.security
+
+import java.io.{ByteArrayOutputStream, ObjectInputFilter, ObjectInputStream, ObjectOutputStream}
+import java.net.URI
+import java.nio.file.Paths
+import java.time.Instant
+import java.util.concurrent.{RejectedExecutionException, ScheduledExecutorService, TimeUnit}
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.LogKeys
+import org.apache.spark.internal.config._
+import org.apache.spark.security._
+import org.apache.spark.ui.UIUtils
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Manager for OIDC-based credential propagation on the driver.
+ *
+ * This class is a sibling of [[HadoopDelegationTokenManager]] and handles the OIDC credential
+ * propagation path independently. Both managers run on independent threads and can both be
+ * active simultaneously (e.g., a cluster accessing HDFS via Kerberos and S3 via OIDC).
+ *
+ * Responsibilities:
+ * 1. Reads the current identity token via a [[TokenIngestor]]
+ * 2. Calls [[CredentialProvider#resolve]] for each configured scheme
+ * 3. Serializes the resulting [[UserCredentials]] and invokes the propagation callback
+ * 4. Schedules renewal based on `min(identity token expiry, service credential expiry) -
+ * safetyMargin`
+ * 5. Retries with exponential backoff on failure
+ *
+ * Intended to be started from `CoarseGrainedSchedulerBackend.start()` when
+ * `spark.security.credentials.enabled=true`, independently of
+ * `UserGroupInformation.isSecurityEnabled()`.
+ *
+ * Lifecycle: call `start()` exactly once, then `stop()` to shut down.
+ * Calling `start()` after `stop()` is not supported.
+ */
+private[spark] class UserCredentialManager(
+ sparkConf: SparkConf,
+ tokenIngestor: TokenIngestor,
+ onCredentialsUpdate: Array[Byte] => Unit) extends Logging {
+
+ private val safetyMargin = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_SAFETY_MARGIN)
+ private val minInterval = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_MIN_INTERVAL)
+
+ // Thread-safe counter for exponential backoff calculation.
+ // Accessed from both the caller of start() and the scheduled renewal thread.
+ private val consecutiveFailures = new AtomicInteger(0)
+ private val maxBackoffMs: Long = UserCredentialManager.MAX_BACKOFF_MS
+
+ @volatile private var renewalExecutor: ScheduledExecutorService = _
+
+ // Pre-compute the filtered credential configuration map (SparkConf is immutable).
+ private val credentialConfMap: java.util.Map[String, String] = sparkConf.getAll
+ .filter(_._1.startsWith("spark.security.credentials."))
+ .toMap.asJava
+
+ /**
+ * Start the credential manager. Acquires initial credentials and schedules renewal.
+ *
+ * The initial credential acquisition is fail-fast: if the token cannot be loaded or
+ * no credentials can be resolved, an exception is thrown. Subsequent renewal failures
+ * are handled with exponential backoff.
+ *
+ * @return The serialized initial [[UserCredentials]].
+ * @throws IllegalStateException if the initial credential acquisition fails.
+ */
+ def start(): Array[Byte] = {
+ require(renewalExecutor == null, "start() must not be called more than once")
+
+ // Initial acquisition is fail-fast (no retry/backoff).
+ // This ensures the application fails to start if credentials cannot be obtained,
+ // rather than running with null credentials.
+ val userContext = tokenIngestor.load()
+ if (!userContext.isPresent) {
+ throw new IllegalStateException(
+ "Failed to start UserCredentialManager: identity token file is missing or malformed. " +
+ s"Check ${SECURITY_CREDENTIALS_IDENTITY_TOKEN_FILE.key} configuration.")
+ }
+
+ val ctx = userContext.get()
+ logInfo(log"Initial identity token loaded for principal " +
+ log"${MDC(LogKeys.PRINCIPAL, ctx.getPrincipal)} " +
+ log"(issuer: ${MDC(LogKeys.URI, ctx.getIssuer)})")
+
+ val (credentials, earliestExpiry) = resolveCredentials(ctx)
+ val serialized = serializeUserCredentials(credentials)
+
+ // Propagate initial credentials
+ onCredentialsUpdate(serialized)
+
+ // Create the renewal executor only after successful initial acquisition.
+ // This avoids leaking a daemon thread if the fail-fast path throws, and
+ // keeps the require() guard valid for retry scenarios.
+ renewalExecutor =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("user-credential-renewal")
+
+ // Schedule first renewal
+ val renewalDelay = computeRenewalDelay(ctx, earliestExpiry)
+ scheduleRenewal(renewalDelay)
+
+ logInfo(log"Credential acquisition successful. Next renewal in " +
+ log"${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(renewalDelay))}.")
+ serialized
+ }
+
+ def stop(): Unit = {
+ if (renewalExecutor != null) {
+ renewalExecutor.shutdownNow()
+ }
+ }
+
+ /**
+ * Scheduled renewal task: load identity token, resolve credentials, propagate,
+ * and schedule the next renewal. Failures are retried with exponential backoff.
+ */
+ private def renewCredentialsTask(): Unit = {
+ try {
+ val userContext = tokenIngestor.load()
+ if (!userContext.isPresent) {
+ throw new IllegalStateException(
+ "TokenIngestor returned empty - identity token file may be missing or malformed")
+ }
+
+ val ctx = userContext.get()
+ logInfo(log"Loaded identity token for principal " +
+ log"${MDC(LogKeys.PRINCIPAL, ctx.getPrincipal)} " +
+ log"(issuer: ${MDC(LogKeys.URI, ctx.getIssuer)})")
+
+ val (credentials, earliestExpiry) = resolveCredentials(ctx)
+ val serialized = serializeUserCredentials(credentials)
+
+ // Propagate credentials to executors. Errors here are logged separately
+ // so that credential-fetch success is not conflated with distribution failure.
+ try {
+ onCredentialsUpdate(serialized)
+ } catch {
+ case e: Exception =>
+ logWarning(log"Credentials were resolved successfully but failed to propagate " +
+ log"to executors. Executors will receive updated credentials on next renewal.", e)
+ }
+
+ // Reset backoff on successful credential resolution. This is intentionally done
+ // even if onCredentialsUpdate failed above: the backoff counter tracks credential
+ // *resolution* failures (STS/provider issues), not propagation failures.
+ // Propagation failures are transient and will be retried on next scheduled renewal.
+ consecutiveFailures.set(0)
+
+ // Schedule next renewal
+ val renewalDelay = computeRenewalDelay(ctx, earliestExpiry)
+ scheduleRenewal(renewalDelay)
+
+ logInfo(log"Credential renewal successful. Next renewal in " +
+ log"${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(renewalDelay))}.")
+ } catch {
+ case _: InterruptedException =>
+ // Shutting down, ignore
+ case e: Exception =>
+ val failures = consecutiveFailures.incrementAndGet()
+ val delay = computeBackoffDelay()
+ logWarning(log"Failed to renew user credentials (attempt " +
+ log"${MDC(LogKeys.NUM_RETRY, failures)}), " +
+ log"will retry in ${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(delay))}.", e)
+ scheduleRenewal(delay)
+ }
+ }
+
+ /**
+ * Resolve credentials for all schemes that have registered providers.
+ *
+ * @return Tuple of (UserCredentials, earliest expiry across all service credentials)
+ */
+ private def resolveCredentials(
+ ctx: UserContext): (UserCredentials, Option[Instant]) = {
+ val schemes = discoverSchemes()
+
+ val credentialMap = new mutable.HashMap[String, ServiceCredential]()
+ var earliestExpiry: Option[Instant] = None
+
+ for (scheme <- schemes) {
+ try {
+ val providerOpt = CredentialProviderLoader.providerFor(scheme, credentialConfMap)
+ if (providerOpt.isPresent) {
+ val provider = providerOpt.get()
+ // Use a synthetic target URI with just the scheme for initial resolution.
+ // The full URI is passed in future versions when path-specific resolution is needed.
+ // The "synthetic" authority signals this is not a real endpoint but a placeholder
+ // for scheme-based provider selection.
+ val target = new URI(scheme, UserCredentialManager.SYNTHETIC_TARGET_AUTHORITY,
+ "/", null, null)
+ val credential = provider.resolve(ctx, target)
+ credentialMap.put(scheme, credential)
+
+ val expiry = credential.getExpiresAt
+ if (expiry != null) {
+ earliestExpiry = earliestExpiry match {
+ case Some(existing) if existing.isBefore(expiry) => Some(existing)
+ case _ => Some(expiry)
+ }
+ }
+ } else {
+ logWarning(log"No credential provider found for scheme " +
+ log"${MDC(LogKeys.URI, scheme)}. Skipping.")
+ }
+ } catch {
+ case e: Exception =>
+ logWarning(log"Failed to resolve credentials for scheme " +
+ log"${MDC(LogKeys.URI, scheme)}. Skipping this provider.", e)
+ }
+ }
+
+ if (credentialMap.isEmpty) {
+ throw new IllegalStateException(
+ "No credential providers resolved any credentials. " +
+ "Check that providers are on the classpath and configured correctly.")
+ }
+
+ (new UserCredentials(credentialMap.asJava), earliestExpiry)
+ }
+
+ /**
+ * Discover all schemes that have at least one registered provider.
+ *
+ * Schemes are determined by explicit configuration keys of the form
+ * `spark.security.credentials.provider.<scheme>`. If no explicit configuration
+ * exists, the method queries `CredentialProviderLoader` to discover all providers
+ * registered via ServiceLoader and collects their supported schemes.
+ *
+ * Note: When using auto-discovery (no explicit config), multiple providers may
+ * support the same scheme. In that case, `CredentialProviderLoader.providerFor`
+ * will throw an `IllegalArgumentException` for that scheme. The caller
+ * (`resolveCredentials`) handles this gracefully via per-provider exception catching,
+ * logging a warning and continuing with remaining schemes.
+ */
+ private def discoverSchemes(): Set[String] = {
+ // Extract explicitly configured scheme -> provider mappings
+ val explicitSchemes = credentialConfMap.asScala
+ .filter { case (k, _) => k.startsWith("spark.security.credentials.provider.") }
+ .map { case (k, _) => k.stripPrefix("spark.security.credentials.provider.") }
+ .toSet
Review Comment:
Good catch. Fixed this to extract only the first segment after the prefix (split on `.` and take the first element), so `spark.security.oidc.provider.s3a.roleArn` correctly yields `s3a`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement UserCredentialManager [spark]
Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on code in PR #57387:
URL: https://github.com/apache/spark/pull/57387#discussion_r3665814173
##########
core/src/main/java/org/apache/spark/security/CredentialProviderLoader.java:
##########
@@ -221,10 +223,38 @@ private static List<CredentialProvider> loadProviders() {
return result;
}
+ /**
+ * Discovers all URI schemes supported by providers on the classpath.
+ * <p>
+ * This method queries all discovered {@link CredentialProvider} instances and collects
+ * their {@link CredentialProvider#supportedSchemes()} into a single set. Unlike
+ * {@link #providerFor(String, Map)}, this does not initialize providers or apply
+ * explicit selection rules -- it only reports what schemes are potentially available.
+ * <p>
+ * Intended for use by {@code UserCredentialManager} when no explicit scheme configuration
+ * (e.g., {@code spark.security.credentials.provider.<scheme>}) is provided.
+ *
+ * @return a set of all supported scheme names (lowercased), possibly empty
+ */
+ public static Set<String> discoverAllSchemes() {
+ List<CredentialProvider> providers = getProviders();
+ Set<String> schemes = new java.util.HashSet<>();
Review Comment:
Added `import java.util.HashSet;` and changed to `new HashSet<>()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement UserCredentialManager [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #57387:
URL: https://github.com/apache/spark/pull/57387#discussion_r3666910656
##########
core/src/main/scala/org/apache/spark/internal/config/package.scala:
##########
@@ -1690,6 +1690,46 @@ package object config {
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("1h")
+ private[spark] val SECURITY_CREDENTIALS_ENABLED =
+ ConfigBuilder("spark.security.oidc.enabled")
+ .doc("Whether to enable OIDC credential propagation. When enabled, the driver reads an " +
+ "identity token from a file, exchanges it for short-lived service credentials via " +
+ "CredentialProvider implementations, and propagates those credentials to executors.")
+ .version("4.3.0")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+ .booleanConf
+ .createWithDefault(false)
+
+ private[spark] val SECURITY_CREDENTIALS_IDENTITY_TOKEN_FILE =
+ ConfigBuilder("spark.security.oidc.identityToken.file")
+ .doc("Path to the OIDC identity token file on the driver. Required when " +
+ "spark.security.oidc.enabled is true. The file should contain a JWT token " +
+ "(e.g., a Kubernetes projected service account token).")
+ .version("4.3.0")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+ .stringConf
+ .createOptional
+
+ private[spark] val SECURITY_CREDENTIALS_RENEWAL_SAFETY_MARGIN =
+ ConfigBuilder("spark.security.oidc.renewal.safetyMargin")
+ .doc("How long before credential expiry to trigger renewal. Credentials are refreshed " +
+ "at min(identity token expiry, service credential expiry) minus this margin.")
+ .version("4.3.0")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+ .timeConf(TimeUnit.MILLISECONDS)
+ .checkValue(_ > 0, "The safety margin must be a positive time value.")
+ .createWithDefaultString("60s")
+
+ private[spark] val SECURITY_CREDENTIALS_RENEWAL_MIN_INTERVAL =
Review Comment:
```suggestion
private[spark] val SECURITY_OIDC_RENEWAL_MIN_INTERVAL =
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement `UserCredentialManager` [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #57387:
URL: https://github.com/apache/spark/pull/57387#discussion_r3666985143
##########
core/src/main/scala/org/apache/spark/deploy/security/UserCredentialManager.scala:
##########
@@ -0,0 +1,441 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.security
+
+import java.io.{ByteArrayOutputStream, ObjectInputFilter, ObjectInputStream, ObjectOutputStream}
+import java.net.URI
+import java.nio.file.Paths
+import java.time.Instant
+import java.util.concurrent.{RejectedExecutionException, ScheduledExecutorService, TimeUnit}
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.LogKeys
+import org.apache.spark.internal.config._
+import org.apache.spark.security._
+import org.apache.spark.ui.UIUtils
+import org.apache.spark.util.{ThreadUtils, Utils}
+
+/**
+ * Manager for OIDC-based credential propagation on the driver.
+ *
+ * This class is a sibling of [[HadoopDelegationTokenManager]] and handles the OIDC credential
+ * propagation path independently. Both managers run on independent threads and can both be
+ * active simultaneously (e.g., a cluster accessing HDFS via Kerberos and S3 via OIDC).
+ *
+ * Responsibilities:
+ * 1. Reads the current identity token via a [[TokenIngestor]]
+ * 2. Calls [[CredentialProvider#resolve]] for each configured scheme
+ * 3. Serializes the resulting [[UserCredentials]] and invokes the propagation callback
+ * 4. Schedules renewal based on `min(identity token expiry, service credential expiry) -
+ * safetyMargin`
+ * 5. Retries with exponential backoff on failure
+ *
+ * Intended to be started from `CoarseGrainedSchedulerBackend.start()` when
+ * `spark.security.oidc.enabled=true`, independently of
+ * `UserGroupInformation.isSecurityEnabled()`.
+ *
+ * Lifecycle: call `start()` exactly once, then `stop()` to shut down.
+ * Calling `start()` after `stop()` is not supported.
+ */
+private[spark] class UserCredentialManager(
+ sparkConf: SparkConf,
+ tokenIngestor: TokenIngestor,
+ onCredentialsUpdate: Array[Byte] => Unit) extends Logging {
+
+ private val safetyMargin = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_SAFETY_MARGIN)
+ private val minInterval = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_MIN_INTERVAL)
+
+ // Counter for exponential backoff calculation.
+ // Only accessed from the single-thread renewal executor.
+ private var consecutiveFailures: Int = 0
+ private val maxBackoffMs: Long = UserCredentialManager.MAX_BACKOFF_MS
+
+ private var renewalExecutor: ScheduledExecutorService = _
+
+ // Snapshot of credential-related configuration at construction time.
+ private val credentialConfMap: java.util.Map[String, String] = sparkConf.getAll
+ .filter(_._1.startsWith("spark.security.oidc."))
+ .toMap.asJava
+
+ /**
+ * Start the credential manager. Acquires initial credentials and schedules renewal.
+ *
+ * The initial credential acquisition is fail-fast: if the token cannot be loaded or
+ * no credentials can be resolved, an exception is thrown. Subsequent renewal failures
+ * are handled with exponential backoff.
+ *
+ * @return The serialized initial [[UserCredentials]].
+ * @throws IllegalStateException if the initial credential acquisition fails.
+ */
+ def start(): Array[Byte] = {
+ require(renewalExecutor == null, "start() must not be called more than once")
+
+ // Initial acquisition is fail-fast (no retry/backoff).
+ // This ensures the application fails to start if credentials cannot be obtained,
+ // rather than running with null credentials.
+ val userContext = tokenIngestor.load()
+ if (!userContext.isPresent) {
+ throw new IllegalStateException(
+ "Failed to start UserCredentialManager: identity token file is missing or malformed. " +
+ s"Check ${SECURITY_CREDENTIALS_IDENTITY_TOKEN_FILE.key} configuration.")
+ }
+
+ val ctx = userContext.get()
+ logInfo(log"Initial identity token loaded for principal " +
+ log"${MDC(LogKeys.PRINCIPAL, ctx.getPrincipal)} " +
+ log"(issuer: ${MDC(LogKeys.URI, ctx.getIssuer)})")
+
+ val (credentials, earliestExpiry) = resolveCredentials(ctx)
+ val serialized = UserCredentialManager.serializeUserCredentials(credentials)
+
+ // Propagate initial credentials
+ onCredentialsUpdate(serialized)
+
+ // Create the renewal executor only after successful initial acquisition.
+ // This avoids leaking a daemon thread if the fail-fast path throws, and
+ // keeps the require() guard valid for retry scenarios.
+ renewalExecutor =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("user-credential-renewal")
+
+ // Schedule first renewal
+ val renewalDelay = computeRenewalDelay(ctx, earliestExpiry)
+ scheduleRenewal(renewalDelay)
+
+ logInfo(log"Credential acquisition successful. Next renewal in " +
+ log"${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(renewalDelay))}.")
+ serialized
+ }
+
+ def stop(): Unit = {
+ if (renewalExecutor != null) {
+ renewalExecutor.shutdownNow()
+ }
+ }
+
+ /**
+ * Scheduled renewal task: load identity token, resolve credentials, propagate,
+ * and schedule the next renewal. Failures are retried with exponential backoff.
+ */
+ private def renewCredentialsTask(): Unit = {
+ try {
+ val userContext = tokenIngestor.load()
+ if (!userContext.isPresent) {
+ throw new IllegalStateException(
+ "TokenIngestor returned empty - identity token file may be missing or malformed")
+ }
+
+ val ctx = userContext.get()
+ logInfo(log"Loaded identity token for principal " +
+ log"${MDC(LogKeys.PRINCIPAL, ctx.getPrincipal)} " +
+ log"(issuer: ${MDC(LogKeys.URI, ctx.getIssuer)})")
+
+ val (credentials, earliestExpiry) = resolveCredentials(ctx)
+ val serialized = UserCredentialManager.serializeUserCredentials(credentials)
+
+ // Propagate credentials to executors. Errors here are logged separately
+ // so that credential-fetch success is not conflated with distribution failure.
+ try {
+ onCredentialsUpdate(serialized)
+ } catch {
+ case e: Exception =>
+ logWarning(log"Credentials were resolved successfully but failed to propagate " +
+ log"to executors. Executors will receive updated credentials on next renewal.", e)
+ }
+
+ // Reset backoff on successful credential resolution. This is intentionally done
+ // even if onCredentialsUpdate failed above: the backoff counter tracks credential
+ // *resolution* failures (STS/provider issues), not propagation failures.
+ // Propagation failures are transient and will be retried on next scheduled renewal.
+ consecutiveFailures = 0
+
+ // Schedule next renewal
+ val renewalDelay = computeRenewalDelay(ctx, earliestExpiry)
+ scheduleRenewal(renewalDelay)
+
+ logInfo(log"Credential renewal successful. Next renewal in " +
+ log"${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(renewalDelay))}.")
+ } catch {
+ case _: InterruptedException =>
+ // Shutting down, ignore
+ case e: Exception =>
Review Comment:
It seems that we ignore `Error`. Shall we use `NonFatal` here and catch `Throwable` too like the following (and subsequent comment).
```suggestion
case NonFatal(e) =>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement `UserCredentialManager` [spark]
Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on PR #57387:
URL: https://github.com/apache/spark/pull/57387#issuecomment-5106759215
Thank you for additional feedback, @dongjoon-hyun. I've addressed new comments and [the remaining one](https://github.com/apache/spark/pull/57387#discussion_r3667035666).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement UserCredentialManager [spark]
Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on code in PR #57387:
URL: https://github.com/apache/spark/pull/57387#discussion_r3665816857
##########
core/src/main/java/org/apache/spark/security/CredentialProviderLoader.java:
##########
@@ -221,10 +223,38 @@ private static List<CredentialProvider> loadProviders() {
return result;
}
+ /**
+ * Discovers all URI schemes supported by providers on the classpath.
+ * <p>
+ * This method queries all discovered {@link CredentialProvider} instances and collects
+ * their {@link CredentialProvider#supportedSchemes()} into a single set. Unlike
+ * {@link #providerFor(String, Map)}, this does not initialize providers or apply
+ * explicit selection rules -- it only reports what schemes are potentially available.
+ * <p>
+ * Intended for use by {@code UserCredentialManager} when no explicit scheme configuration
+ * (e.g., {@code spark.security.credentials.provider.<scheme>}) is provided.
+ *
+ * @return a set of all supported scheme names (lowercased), possibly empty
+ */
+ public static Set<String> discoverAllSchemes() {
+ List<CredentialProvider> providers = getProviders();
+ Set<String> schemes = new java.util.HashSet<>();
+ for (CredentialProvider provider : providers) {
+ Set<String> providerSchemes = provider.supportedSchemes();
+ if (providerSchemes != null) {
Review Comment:
Added a note to the `discoverAllSchemes()` Javadoc: "Providers returning {@code null} from {@code supportedSchemes()} are silently skipped (unlike {@code providerFor}, which throws)."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement UserCredentialManager [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #57387:
URL: https://github.com/apache/spark/pull/57387#discussion_r3666903703
##########
core/src/main/scala/org/apache/spark/internal/config/package.scala:
##########
@@ -1690,6 +1690,46 @@ package object config {
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("1h")
+ private[spark] val SECURITY_CREDENTIALS_ENABLED =
+ ConfigBuilder("spark.security.oidc.enabled")
+ .doc("Whether to enable OIDC credential propagation. When enabled, the driver reads an " +
+ "identity token from a file, exchanges it for short-lived service credentials via " +
+ "CredentialProvider implementations, and propagates those credentials to executors.")
+ .version("4.3.0")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+ .booleanConf
+ .createWithDefault(false)
+
+ private[spark] val SECURITY_CREDENTIALS_IDENTITY_TOKEN_FILE =
+ ConfigBuilder("spark.security.oidc.identityToken.file")
+ .doc("Path to the OIDC identity token file on the driver. Required when " +
+ "spark.security.oidc.enabled is true. The file should contain a JWT token " +
+ "(e.g., a Kubernetes projected service account token).")
+ .version("4.3.0")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+ .stringConf
+ .createOptional
+
+ private[spark] val SECURITY_CREDENTIALS_RENEWAL_SAFETY_MARGIN =
Review Comment:
```suggestion
private[spark] val SECURITY_OIDC_RENEWAL_SAFETY_MARGIN =
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement UserCredentialManager [spark]
Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on PR #57387:
URL: https://github.com/apache/spark/pull/57387#issuecomment-5080435185
Thank you so much, @dongjoon-hyun! I'll check them out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement UserCredentialManager [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #57387:
URL: https://github.com/apache/spark/pull/57387#discussion_r3650936646
##########
core/src/main/scala/org/apache/spark/deploy/security/UserCredentialManager.scala:
##########
@@ -0,0 +1,443 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.security
+
+import java.io.{ByteArrayOutputStream, ObjectInputFilter, ObjectInputStream, ObjectOutputStream}
+import java.net.URI
+import java.nio.file.Paths
+import java.time.Instant
+import java.util.concurrent.{RejectedExecutionException, ScheduledExecutorService, TimeUnit}
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.LogKeys
+import org.apache.spark.internal.config._
+import org.apache.spark.security._
+import org.apache.spark.ui.UIUtils
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Manager for OIDC-based credential propagation on the driver.
+ *
+ * This class is a sibling of [[HadoopDelegationTokenManager]] and handles the OIDC credential
+ * propagation path independently. Both managers run on independent threads and can both be
+ * active simultaneously (e.g., a cluster accessing HDFS via Kerberos and S3 via OIDC).
+ *
+ * Responsibilities:
+ * 1. Reads the current identity token via a [[TokenIngestor]]
+ * 2. Calls [[CredentialProvider#resolve]] for each configured scheme
+ * 3. Serializes the resulting [[UserCredentials]] and invokes the propagation callback
+ * 4. Schedules renewal based on `min(identity token expiry, service credential expiry) -
+ * safetyMargin`
+ * 5. Retries with exponential backoff on failure
+ *
+ * Intended to be started from `CoarseGrainedSchedulerBackend.start()` when
+ * `spark.security.credentials.enabled=true`, independently of
+ * `UserGroupInformation.isSecurityEnabled()`.
+ *
+ * Lifecycle: call `start()` exactly once, then `stop()` to shut down.
+ * Calling `start()` after `stop()` is not supported.
+ */
+private[spark] class UserCredentialManager(
+ sparkConf: SparkConf,
+ tokenIngestor: TokenIngestor,
+ onCredentialsUpdate: Array[Byte] => Unit) extends Logging {
+
+ private val safetyMargin = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_SAFETY_MARGIN)
+ private val minInterval = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_MIN_INTERVAL)
+
+ // Thread-safe counter for exponential backoff calculation.
+ // Accessed from both the caller of start() and the scheduled renewal thread.
+ private val consecutiveFailures = new AtomicInteger(0)
+ private val maxBackoffMs: Long = UserCredentialManager.MAX_BACKOFF_MS
+
+ @volatile private var renewalExecutor: ScheduledExecutorService = _
+
+ // Pre-compute the filtered credential configuration map (SparkConf is immutable).
+ private val credentialConfMap: java.util.Map[String, String] = sparkConf.getAll
+ .filter(_._1.startsWith("spark.security.credentials."))
+ .toMap.asJava
+
+ /**
+ * Start the credential manager. Acquires initial credentials and schedules renewal.
+ *
+ * The initial credential acquisition is fail-fast: if the token cannot be loaded or
+ * no credentials can be resolved, an exception is thrown. Subsequent renewal failures
+ * are handled with exponential backoff.
+ *
+ * @return The serialized initial [[UserCredentials]].
+ * @throws IllegalStateException if the initial credential acquisition fails.
+ */
+ def start(): Array[Byte] = {
+ require(renewalExecutor == null, "start() must not be called more than once")
+
+ // Initial acquisition is fail-fast (no retry/backoff).
+ // This ensures the application fails to start if credentials cannot be obtained,
+ // rather than running with null credentials.
+ val userContext = tokenIngestor.load()
+ if (!userContext.isPresent) {
+ throw new IllegalStateException(
+ "Failed to start UserCredentialManager: identity token file is missing or malformed. " +
+ s"Check ${SECURITY_CREDENTIALS_IDENTITY_TOKEN_FILE.key} configuration.")
+ }
+
+ val ctx = userContext.get()
+ logInfo(log"Initial identity token loaded for principal " +
+ log"${MDC(LogKeys.PRINCIPAL, ctx.getPrincipal)} " +
+ log"(issuer: ${MDC(LogKeys.URI, ctx.getIssuer)})")
+
+ val (credentials, earliestExpiry) = resolveCredentials(ctx)
+ val serialized = serializeUserCredentials(credentials)
+
+ // Propagate initial credentials
+ onCredentialsUpdate(serialized)
+
+ // Create the renewal executor only after successful initial acquisition.
+ // This avoids leaking a daemon thread if the fail-fast path throws, and
+ // keeps the require() guard valid for retry scenarios.
+ renewalExecutor =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("user-credential-renewal")
+
+ // Schedule first renewal
+ val renewalDelay = computeRenewalDelay(ctx, earliestExpiry)
+ scheduleRenewal(renewalDelay)
+
+ logInfo(log"Credential acquisition successful. Next renewal in " +
+ log"${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(renewalDelay))}.")
+ serialized
+ }
+
+ def stop(): Unit = {
+ if (renewalExecutor != null) {
+ renewalExecutor.shutdownNow()
+ }
+ }
+
+ /**
+ * Scheduled renewal task: load identity token, resolve credentials, propagate,
+ * and schedule the next renewal. Failures are retried with exponential backoff.
+ */
+ private def renewCredentialsTask(): Unit = {
+ try {
+ val userContext = tokenIngestor.load()
+ if (!userContext.isPresent) {
+ throw new IllegalStateException(
+ "TokenIngestor returned empty - identity token file may be missing or malformed")
+ }
+
+ val ctx = userContext.get()
+ logInfo(log"Loaded identity token for principal " +
+ log"${MDC(LogKeys.PRINCIPAL, ctx.getPrincipal)} " +
+ log"(issuer: ${MDC(LogKeys.URI, ctx.getIssuer)})")
+
+ val (credentials, earliestExpiry) = resolveCredentials(ctx)
+ val serialized = serializeUserCredentials(credentials)
+
+ // Propagate credentials to executors. Errors here are logged separately
+ // so that credential-fetch success is not conflated with distribution failure.
+ try {
+ onCredentialsUpdate(serialized)
+ } catch {
+ case e: Exception =>
+ logWarning(log"Credentials were resolved successfully but failed to propagate " +
+ log"to executors. Executors will receive updated credentials on next renewal.", e)
+ }
+
+ // Reset backoff on successful credential resolution. This is intentionally done
+ // even if onCredentialsUpdate failed above: the backoff counter tracks credential
+ // *resolution* failures (STS/provider issues), not propagation failures.
+ // Propagation failures are transient and will be retried on next scheduled renewal.
+ consecutiveFailures.set(0)
+
+ // Schedule next renewal
+ val renewalDelay = computeRenewalDelay(ctx, earliestExpiry)
+ scheduleRenewal(renewalDelay)
+
+ logInfo(log"Credential renewal successful. Next renewal in " +
+ log"${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(renewalDelay))}.")
+ } catch {
+ case _: InterruptedException =>
+ // Shutting down, ignore
+ case e: Exception =>
+ val failures = consecutiveFailures.incrementAndGet()
+ val delay = computeBackoffDelay()
+ logWarning(log"Failed to renew user credentials (attempt " +
+ log"${MDC(LogKeys.NUM_RETRY, failures)}), " +
+ log"will retry in ${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(delay))}.", e)
+ scheduleRenewal(delay)
+ }
+ }
+
+ /**
+ * Resolve credentials for all schemes that have registered providers.
+ *
+ * @return Tuple of (UserCredentials, earliest expiry across all service credentials)
+ */
+ private def resolveCredentials(
+ ctx: UserContext): (UserCredentials, Option[Instant]) = {
+ val schemes = discoverSchemes()
+
+ val credentialMap = new mutable.HashMap[String, ServiceCredential]()
+ var earliestExpiry: Option[Instant] = None
+
+ for (scheme <- schemes) {
+ try {
+ val providerOpt = CredentialProviderLoader.providerFor(scheme, credentialConfMap)
+ if (providerOpt.isPresent) {
+ val provider = providerOpt.get()
+ // Use a synthetic target URI with just the scheme for initial resolution.
+ // The full URI is passed in future versions when path-specific resolution is needed.
+ // The "synthetic" authority signals this is not a real endpoint but a placeholder
+ // for scheme-based provider selection.
+ val target = new URI(scheme, UserCredentialManager.SYNTHETIC_TARGET_AUTHORITY,
+ "/", null, null)
+ val credential = provider.resolve(ctx, target)
+ credentialMap.put(scheme, credential)
+
+ val expiry = credential.getExpiresAt
+ if (expiry != null) {
+ earliestExpiry = earliestExpiry match {
+ case Some(existing) if existing.isBefore(expiry) => Some(existing)
+ case _ => Some(expiry)
+ }
+ }
+ } else {
+ logWarning(log"No credential provider found for scheme " +
+ log"${MDC(LogKeys.URI, scheme)}. Skipping.")
+ }
+ } catch {
+ case e: Exception =>
+ logWarning(log"Failed to resolve credentials for scheme " +
+ log"${MDC(LogKeys.URI, scheme)}. Skipping this provider.", e)
+ }
+ }
+
+ if (credentialMap.isEmpty) {
+ throw new IllegalStateException(
+ "No credential providers resolved any credentials. " +
+ "Check that providers are on the classpath and configured correctly.")
+ }
+
+ (new UserCredentials(credentialMap.asJava), earliestExpiry)
+ }
+
+ /**
+ * Discover all schemes that have at least one registered provider.
+ *
+ * Schemes are determined by explicit configuration keys of the form
+ * `spark.security.credentials.provider.<scheme>`. If no explicit configuration
+ * exists, the method queries `CredentialProviderLoader` to discover all providers
+ * registered via ServiceLoader and collects their supported schemes.
+ *
+ * Note: When using auto-discovery (no explicit config), multiple providers may
+ * support the same scheme. In that case, `CredentialProviderLoader.providerFor`
+ * will throw an `IllegalArgumentException` for that scheme. The caller
+ * (`resolveCredentials`) handles this gracefully via per-provider exception catching,
+ * logging a warning and continuing with remaining schemes.
+ */
+ private def discoverSchemes(): Set[String] = {
+ // Extract explicitly configured scheme -> provider mappings
+ val explicitSchemes = credentialConfMap.asScala
+ .filter { case (k, _) => k.startsWith("spark.security.credentials.provider.") }
+ .map { case (k, _) => k.stripPrefix("spark.security.credentials.provider.") }
+ .toSet
+
+ if (explicitSchemes.nonEmpty) {
+ explicitSchemes
+ } else {
+ // No explicit scheme configuration. Discover all schemes that have a provider
+ // available on the classpath by probing CredentialProviderLoader.
+ // This covers both built-in providers (e.g., connector/credential-aws for "s3a")
+ // and third-party providers registered via ServiceLoader.
+ CredentialProviderLoader.discoverAllSchemes().asScala.toSet
+ }
+ }
+
+ /**
+ * Compute the delay until next renewal.
+ * Uses min(identity token expiry, service credential expiry) - safetyMargin,
+ * bounded below by minInterval.
+ */
+ private[security] def computeRenewalDelay(
+ ctx: UserContext,
+ earliestCredentialExpiry: Option[Instant]): Long = {
+ val now = System.currentTimeMillis()
+
+ // Consider identity token expiry
+ val tokenExpiry: Option[Long] = Option(ctx.getExpiresAt).map(_.toEpochMilli)
+
+ // Consider earliest service credential expiry
+ val credExpiry: Option[Long] = earliestCredentialExpiry.map(_.toEpochMilli)
+
+ // Take the minimum of both
+ val effectiveExpiry: Option[Long] = (tokenExpiry, credExpiry) match {
+ case (Some(t), Some(c)) => Some(math.min(t, c))
+ case (Some(t), None) => Some(t)
+ case (None, Some(c)) => Some(c)
+ case (None, None) => None
+ }
+
+ effectiveExpiry match {
+ case Some(expiry) =>
+ math.max(expiry - now - safetyMargin, minInterval)
+ case None =>
+ // No expiry information available from either the identity token or service
+ // credentials. Use half of the default suggested TTL (15 min / 2 = 7.5 min,
+ // rounded down) as a conservative polling interval. This ensures credentials
+ // are refreshed even when providers don't report expiration times.
+ UserCredentialManager.DEFAULT_RENEWAL_INTERVAL_NO_EXPIRY_MS
+ }
+ }
+
+ /**
+ * Compute backoff delay using exponential backoff with jitter.
+ * Bounded by minInterval (floor) and maxBackoffMs (ceiling).
+ */
+ private[security] def computeBackoffDelay(): Long = {
+ // Guard against negative or zero shift amounts. consecutiveFailures should always
+ // be >= 1 when this is called (incremented before calling), but we protect against
+ // edge cases defensively.
+ val shiftAmount = math.max(0, math.min(consecutiveFailures.get() - 1, 6))
+ val baseDelay = minInterval * (1L << shiftAmount)
+ val cappedDelay = math.min(baseDelay, maxBackoffMs)
+ // Add 10% jitter to avoid thundering herd on recovery
+ val jitter = (cappedDelay * 0.1 * math.random()).toLong
+ math.max(cappedDelay + jitter, minInterval)
+ }
+
+ private def scheduleRenewal(delay: Long): Unit = {
+ try {
+ val renewalTask = new Runnable {
+ override def run(): Unit = {
+ renewCredentialsTask()
+ }
+ }
+ renewalExecutor.schedule(renewalTask, delay, TimeUnit.MILLISECONDS)
+ } catch {
+ case _: RejectedExecutionException =>
+ // Executor has been shut down (e.g., stop() called concurrently). This is expected
+ // during application shutdown -- no further renewals will be scheduled.
+ logDebug(log"Renewal scheduling rejected - executor is shut down.")
+ }
+ }
+
+ /**
+ * Serialize [[UserCredentials]] to a byte array using Java serialization.
+ */
+ private[security] def serializeUserCredentials(credentials: UserCredentials): Array[Byte] = {
+ val bos = new ByteArrayOutputStream()
+ try {
+ val oos = new ObjectOutputStream(bos)
+ try {
+ oos.writeObject(credentials)
+ oos.flush()
+ } finally {
+ oos.close()
+ }
+ bos.toByteArray
+ } finally {
+ bos.close()
+ }
+ }
+}
+
+private[spark] object UserCredentialManager {
+
+ /**
+ * Synthetic authority used in target URIs for scheme-based provider resolution.
+ * Providers should not rely on this value; it signals that no specific endpoint
+ * is targeted and only the URI scheme is meaningful.
+ */
+ private val SYNTHETIC_TARGET_AUTHORITY = "synthetic"
+
+ /**
+ * Maximum backoff delay between credential renewal retry attempts.
+ * Caps the exponential backoff to prevent excessively long gaps between retries.
+ */
+ private val MAX_BACKOFF_MS: Long = TimeUnit.MINUTES.toMillis(10)
+
+ /**
+ * Default renewal interval when neither the identity token nor service credentials
+ * report an expiration time. Set to 7 minutes (half of the default 15-minute
+ * suggested TTL from CredentialProvider.suggestedTtl()).
+ */
+ private val DEFAULT_RENEWAL_INTERVAL_NO_EXPIRY_MS: Long = TimeUnit.MINUTES.toMillis(7)
+
+ /**
+ * ObjectInputFilter pattern restricting deserialization to only the classes needed
+ * for UserCredentials. This prevents deserialization gadget chain attacks while
+ * allowing Java's built-in collection internal classes and arrays that HashMap uses.
+ */
+ private val DESERIALIZATION_FILTER: String =
+ "org.apache.spark.security.**;" +
+ "java.util.**;" +
+ "java.time.**;" +
+ "java.lang.**;" +
+ "maxdepth=10;" +
+ "!*"
+
+ /**
+ * Create a UserCredentialManager if OIDC credential propagation is enabled.
+ *
+ * @param sparkConf The Spark configuration
+ * @param onCredentialsUpdate Callback to propagate credentials to executors
+ * @return Some(manager) if enabled, None otherwise
+ */
+ def create(
+ sparkConf: SparkConf,
+ onCredentialsUpdate: Array[Byte] => Unit): Option[UserCredentialManager] = {
+ if (!sparkConf.get(SECURITY_CREDENTIALS_ENABLED)) {
+ None
+ } else {
+ val tokenFile = sparkConf.get(SECURITY_CREDENTIALS_IDENTITY_TOKEN_FILE).getOrElse {
+ throw new IllegalArgumentException(
+ s"${SECURITY_CREDENTIALS_IDENTITY_TOKEN_FILE.key} must be set when " +
+ s"${SECURITY_CREDENTIALS_ENABLED.key} is true")
+ }
+
+ val tokenIngestor = new FileTokenIngestor(Paths.get(tokenFile))
+ Some(new UserCredentialManager(sparkConf, tokenIngestor, onCredentialsUpdate))
+ }
+ }
+
+ /**
+ * Deserialize [[UserCredentials]] from a byte array.
+ *
+ * Uses an [[ObjectInputFilter]] to restrict deserialized classes to only those
+ * required for [[UserCredentials]], preventing deserialization attacks.
+ */
+ def deserializeUserCredentials(bytes: Array[Byte]): UserCredentials = {
+ val bis = new java.io.ByteArrayInputStream(bytes)
+ try {
+ val ois = new ObjectInputStream(bis)
+ try {
+ ois.setObjectInputFilter(
+ ObjectInputFilter.Config.createFilter(DESERIALIZATION_FILTER))
+ ois.readObject().asInstanceOf[UserCredentials]
+ } finally {
+ ois.close()
+ }
+ } finally {
+ bis.close()
+ }
Review Comment:
This looks a little verbose. IIUC, the nested `try/finally` can be simplified with `Utils.tryWithResource`. Closing the `ObjectInputStream` also closes the underlying `ByteArrayInputStream` (whose `close()` is a
no-op anyway), so a single resource block is sufficient. BTW, this also needs the import updated to `import org.apache.spark.util.{ThreadUtils, Utils}`.
```suggestion
val bis = new java.io.ByteArrayInputStream(bytes)
Utils.tryWithResource(new ObjectInputStream(bis)) { ois =>
ois.setObjectInputFilter(
ObjectInputFilter.Config.createFilter(DESERIALIZATION_FILTER))
ois.readObject().asInstanceOf[UserCredentials]
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement UserCredentialManager [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #57387:
URL: https://github.com/apache/spark/pull/57387#discussion_r3650931550
##########
core/src/main/scala/org/apache/spark/deploy/security/UserCredentialManager.scala:
##########
@@ -0,0 +1,443 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.security
+
+import java.io.{ByteArrayOutputStream, ObjectInputFilter, ObjectInputStream, ObjectOutputStream}
+import java.net.URI
+import java.nio.file.Paths
+import java.time.Instant
+import java.util.concurrent.{RejectedExecutionException, ScheduledExecutorService, TimeUnit}
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.LogKeys
+import org.apache.spark.internal.config._
+import org.apache.spark.security._
+import org.apache.spark.ui.UIUtils
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Manager for OIDC-based credential propagation on the driver.
+ *
+ * This class is a sibling of [[HadoopDelegationTokenManager]] and handles the OIDC credential
+ * propagation path independently. Both managers run on independent threads and can both be
+ * active simultaneously (e.g., a cluster accessing HDFS via Kerberos and S3 via OIDC).
+ *
+ * Responsibilities:
+ * 1. Reads the current identity token via a [[TokenIngestor]]
+ * 2. Calls [[CredentialProvider#resolve]] for each configured scheme
+ * 3. Serializes the resulting [[UserCredentials]] and invokes the propagation callback
+ * 4. Schedules renewal based on `min(identity token expiry, service credential expiry) -
+ * safetyMargin`
+ * 5. Retries with exponential backoff on failure
+ *
+ * Intended to be started from `CoarseGrainedSchedulerBackend.start()` when
+ * `spark.security.credentials.enabled=true`, independently of
+ * `UserGroupInformation.isSecurityEnabled()`.
+ *
+ * Lifecycle: call `start()` exactly once, then `stop()` to shut down.
+ * Calling `start()` after `stop()` is not supported.
+ */
+private[spark] class UserCredentialManager(
+ sparkConf: SparkConf,
+ tokenIngestor: TokenIngestor,
+ onCredentialsUpdate: Array[Byte] => Unit) extends Logging {
+
+ private val safetyMargin = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_SAFETY_MARGIN)
+ private val minInterval = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_MIN_INTERVAL)
+
+ // Thread-safe counter for exponential backoff calculation.
+ // Accessed from both the caller of start() and the scheduled renewal thread.
+ private val consecutiveFailures = new AtomicInteger(0)
+ private val maxBackoffMs: Long = UserCredentialManager.MAX_BACKOFF_MS
+
+ @volatile private var renewalExecutor: ScheduledExecutorService = _
+
+ // Pre-compute the filtered credential configuration map (SparkConf is immutable).
+ private val credentialConfMap: java.util.Map[String, String] = sparkConf.getAll
+ .filter(_._1.startsWith("spark.security.credentials."))
+ .toMap.asJava
+
+ /**
+ * Start the credential manager. Acquires initial credentials and schedules renewal.
+ *
+ * The initial credential acquisition is fail-fast: if the token cannot be loaded or
+ * no credentials can be resolved, an exception is thrown. Subsequent renewal failures
+ * are handled with exponential backoff.
+ *
+ * @return The serialized initial [[UserCredentials]].
+ * @throws IllegalStateException if the initial credential acquisition fails.
+ */
+ def start(): Array[Byte] = {
+ require(renewalExecutor == null, "start() must not be called more than once")
+
+ // Initial acquisition is fail-fast (no retry/backoff).
+ // This ensures the application fails to start if credentials cannot be obtained,
+ // rather than running with null credentials.
+ val userContext = tokenIngestor.load()
+ if (!userContext.isPresent) {
+ throw new IllegalStateException(
+ "Failed to start UserCredentialManager: identity token file is missing or malformed. " +
+ s"Check ${SECURITY_CREDENTIALS_IDENTITY_TOKEN_FILE.key} configuration.")
+ }
+
+ val ctx = userContext.get()
+ logInfo(log"Initial identity token loaded for principal " +
+ log"${MDC(LogKeys.PRINCIPAL, ctx.getPrincipal)} " +
+ log"(issuer: ${MDC(LogKeys.URI, ctx.getIssuer)})")
+
+ val (credentials, earliestExpiry) = resolveCredentials(ctx)
+ val serialized = serializeUserCredentials(credentials)
+
+ // Propagate initial credentials
+ onCredentialsUpdate(serialized)
+
+ // Create the renewal executor only after successful initial acquisition.
+ // This avoids leaking a daemon thread if the fail-fast path throws, and
+ // keeps the require() guard valid for retry scenarios.
+ renewalExecutor =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("user-credential-renewal")
+
+ // Schedule first renewal
+ val renewalDelay = computeRenewalDelay(ctx, earliestExpiry)
+ scheduleRenewal(renewalDelay)
+
+ logInfo(log"Credential acquisition successful. Next renewal in " +
+ log"${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(renewalDelay))}.")
+ serialized
+ }
+
+ def stop(): Unit = {
+ if (renewalExecutor != null) {
+ renewalExecutor.shutdownNow()
+ }
+ }
+
+ /**
+ * Scheduled renewal task: load identity token, resolve credentials, propagate,
+ * and schedule the next renewal. Failures are retried with exponential backoff.
+ */
+ private def renewCredentialsTask(): Unit = {
+ try {
+ val userContext = tokenIngestor.load()
+ if (!userContext.isPresent) {
+ throw new IllegalStateException(
+ "TokenIngestor returned empty - identity token file may be missing or malformed")
+ }
+
+ val ctx = userContext.get()
+ logInfo(log"Loaded identity token for principal " +
+ log"${MDC(LogKeys.PRINCIPAL, ctx.getPrincipal)} " +
+ log"(issuer: ${MDC(LogKeys.URI, ctx.getIssuer)})")
+
+ val (credentials, earliestExpiry) = resolveCredentials(ctx)
+ val serialized = serializeUserCredentials(credentials)
+
+ // Propagate credentials to executors. Errors here are logged separately
+ // so that credential-fetch success is not conflated with distribution failure.
+ try {
+ onCredentialsUpdate(serialized)
+ } catch {
+ case e: Exception =>
+ logWarning(log"Credentials were resolved successfully but failed to propagate " +
+ log"to executors. Executors will receive updated credentials on next renewal.", e)
+ }
+
+ // Reset backoff on successful credential resolution. This is intentionally done
+ // even if onCredentialsUpdate failed above: the backoff counter tracks credential
+ // *resolution* failures (STS/provider issues), not propagation failures.
+ // Propagation failures are transient and will be retried on next scheduled renewal.
+ consecutiveFailures.set(0)
+
+ // Schedule next renewal
+ val renewalDelay = computeRenewalDelay(ctx, earliestExpiry)
+ scheduleRenewal(renewalDelay)
+
+ logInfo(log"Credential renewal successful. Next renewal in " +
+ log"${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(renewalDelay))}.")
+ } catch {
+ case _: InterruptedException =>
+ // Shutting down, ignore
+ case e: Exception =>
+ val failures = consecutiveFailures.incrementAndGet()
+ val delay = computeBackoffDelay()
+ logWarning(log"Failed to renew user credentials (attempt " +
+ log"${MDC(LogKeys.NUM_RETRY, failures)}), " +
+ log"will retry in ${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(delay))}.", e)
+ scheduleRenewal(delay)
+ }
+ }
+
+ /**
+ * Resolve credentials for all schemes that have registered providers.
+ *
+ * @return Tuple of (UserCredentials, earliest expiry across all service credentials)
+ */
+ private def resolveCredentials(
+ ctx: UserContext): (UserCredentials, Option[Instant]) = {
+ val schemes = discoverSchemes()
+
+ val credentialMap = new mutable.HashMap[String, ServiceCredential]()
+ var earliestExpiry: Option[Instant] = None
+
+ for (scheme <- schemes) {
+ try {
+ val providerOpt = CredentialProviderLoader.providerFor(scheme, credentialConfMap)
+ if (providerOpt.isPresent) {
+ val provider = providerOpt.get()
+ // Use a synthetic target URI with just the scheme for initial resolution.
+ // The full URI is passed in future versions when path-specific resolution is needed.
+ // The "synthetic" authority signals this is not a real endpoint but a placeholder
+ // for scheme-based provider selection.
+ val target = new URI(scheme, UserCredentialManager.SYNTHETIC_TARGET_AUTHORITY,
+ "/", null, null)
+ val credential = provider.resolve(ctx, target)
+ credentialMap.put(scheme, credential)
+
+ val expiry = credential.getExpiresAt
+ if (expiry != null) {
+ earliestExpiry = earliestExpiry match {
+ case Some(existing) if existing.isBefore(expiry) => Some(existing)
+ case _ => Some(expiry)
+ }
+ }
+ } else {
+ logWarning(log"No credential provider found for scheme " +
+ log"${MDC(LogKeys.URI, scheme)}. Skipping.")
+ }
+ } catch {
+ case e: Exception =>
+ logWarning(log"Failed to resolve credentials for scheme " +
+ log"${MDC(LogKeys.URI, scheme)}. Skipping this provider.", e)
+ }
+ }
+
+ if (credentialMap.isEmpty) {
+ throw new IllegalStateException(
+ "No credential providers resolved any credentials. " +
+ "Check that providers are on the classpath and configured correctly.")
+ }
+
+ (new UserCredentials(credentialMap.asJava), earliestExpiry)
+ }
+
+ /**
+ * Discover all schemes that have at least one registered provider.
+ *
+ * Schemes are determined by explicit configuration keys of the form
+ * `spark.security.credentials.provider.<scheme>`. If no explicit configuration
+ * exists, the method queries `CredentialProviderLoader` to discover all providers
+ * registered via ServiceLoader and collects their supported schemes.
+ *
+ * Note: When using auto-discovery (no explicit config), multiple providers may
+ * support the same scheme. In that case, `CredentialProviderLoader.providerFor`
+ * will throw an `IllegalArgumentException` for that scheme. The caller
+ * (`resolveCredentials`) handles this gracefully via per-provider exception catching,
+ * logging a warning and continuing with remaining schemes.
+ */
+ private def discoverSchemes(): Set[String] = {
+ // Extract explicitly configured scheme -> provider mappings
+ val explicitSchemes = credentialConfMap.asScala
+ .filter { case (k, _) => k.startsWith("spark.security.credentials.provider.") }
+ .map { case (k, _) => k.stripPrefix("spark.security.credentials.provider.") }
+ .toSet
+
+ if (explicitSchemes.nonEmpty) {
+ explicitSchemes
+ } else {
+ // No explicit scheme configuration. Discover all schemes that have a provider
+ // available on the classpath by probing CredentialProviderLoader.
+ // This covers both built-in providers (e.g., connector/credential-aws for "s3a")
+ // and third-party providers registered via ServiceLoader.
+ CredentialProviderLoader.discoverAllSchemes().asScala.toSet
+ }
+ }
+
+ /**
+ * Compute the delay until next renewal.
+ * Uses min(identity token expiry, service credential expiry) - safetyMargin,
+ * bounded below by minInterval.
+ */
+ private[security] def computeRenewalDelay(
+ ctx: UserContext,
+ earliestCredentialExpiry: Option[Instant]): Long = {
+ val now = System.currentTimeMillis()
+
+ // Consider identity token expiry
+ val tokenExpiry: Option[Long] = Option(ctx.getExpiresAt).map(_.toEpochMilli)
+
+ // Consider earliest service credential expiry
+ val credExpiry: Option[Long] = earliestCredentialExpiry.map(_.toEpochMilli)
+
+ // Take the minimum of both
+ val effectiveExpiry: Option[Long] = (tokenExpiry, credExpiry) match {
+ case (Some(t), Some(c)) => Some(math.min(t, c))
+ case (Some(t), None) => Some(t)
+ case (None, Some(c)) => Some(c)
+ case (None, None) => None
+ }
+
+ effectiveExpiry match {
+ case Some(expiry) =>
+ math.max(expiry - now - safetyMargin, minInterval)
+ case None =>
+ // No expiry information available from either the identity token or service
+ // credentials. Use half of the default suggested TTL (15 min / 2 = 7.5 min,
+ // rounded down) as a conservative polling interval. This ensures credentials
+ // are refreshed even when providers don't report expiration times.
+ UserCredentialManager.DEFAULT_RENEWAL_INTERVAL_NO_EXPIRY_MS
+ }
+ }
+
+ /**
+ * Compute backoff delay using exponential backoff with jitter.
+ * Bounded by minInterval (floor) and maxBackoffMs (ceiling).
+ */
+ private[security] def computeBackoffDelay(): Long = {
+ // Guard against negative or zero shift amounts. consecutiveFailures should always
+ // be >= 1 when this is called (incremented before calling), but we protect against
+ // edge cases defensively.
+ val shiftAmount = math.max(0, math.min(consecutiveFailures.get() - 1, 6))
+ val baseDelay = minInterval * (1L << shiftAmount)
+ val cappedDelay = math.min(baseDelay, maxBackoffMs)
+ // Add 10% jitter to avoid thundering herd on recovery
+ val jitter = (cappedDelay * 0.1 * math.random()).toLong
+ math.max(cappedDelay + jitter, minInterval)
+ }
+
+ private def scheduleRenewal(delay: Long): Unit = {
+ try {
+ val renewalTask = new Runnable {
+ override def run(): Unit = {
+ renewCredentialsTask()
+ }
+ }
+ renewalExecutor.schedule(renewalTask, delay, TimeUnit.MILLISECONDS)
+ } catch {
+ case _: RejectedExecutionException =>
+ // Executor has been shut down (e.g., stop() called concurrently). This is expected
+ // during application shutdown -- no further renewals will be scheduled.
+ logDebug(log"Renewal scheduling rejected - executor is shut down.")
+ }
+ }
+
+ /**
+ * Serialize [[UserCredentials]] to a byte array using Java serialization.
+ */
+ private[security] def serializeUserCredentials(credentials: UserCredentials): Array[Byte] = {
Review Comment:
Why `serializeUserCredentials` is designed as an instance method while `deserializeUserCredentials` is a companion method?
Given the code body, please move `serializeUserCredentials` to `object UserCredentialManager` consistently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement UserCredentialManager [spark]
Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on code in PR #57387:
URL: https://github.com/apache/spark/pull/57387#discussion_r3665750800
##########
core/src/main/scala/org/apache/spark/deploy/security/UserCredentialManager.scala:
##########
@@ -0,0 +1,443 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.security
+
+import java.io.{ByteArrayOutputStream, ObjectInputFilter, ObjectInputStream, ObjectOutputStream}
+import java.net.URI
+import java.nio.file.Paths
+import java.time.Instant
+import java.util.concurrent.{RejectedExecutionException, ScheduledExecutorService, TimeUnit}
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.LogKeys
+import org.apache.spark.internal.config._
+import org.apache.spark.security._
+import org.apache.spark.ui.UIUtils
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Manager for OIDC-based credential propagation on the driver.
+ *
+ * This class is a sibling of [[HadoopDelegationTokenManager]] and handles the OIDC credential
+ * propagation path independently. Both managers run on independent threads and can both be
+ * active simultaneously (e.g., a cluster accessing HDFS via Kerberos and S3 via OIDC).
+ *
+ * Responsibilities:
+ * 1. Reads the current identity token via a [[TokenIngestor]]
+ * 2. Calls [[CredentialProvider#resolve]] for each configured scheme
+ * 3. Serializes the resulting [[UserCredentials]] and invokes the propagation callback
+ * 4. Schedules renewal based on `min(identity token expiry, service credential expiry) -
+ * safetyMargin`
+ * 5. Retries with exponential backoff on failure
+ *
+ * Intended to be started from `CoarseGrainedSchedulerBackend.start()` when
+ * `spark.security.credentials.enabled=true`, independently of
+ * `UserGroupInformation.isSecurityEnabled()`.
+ *
+ * Lifecycle: call `start()` exactly once, then `stop()` to shut down.
+ * Calling `start()` after `stop()` is not supported.
+ */
+private[spark] class UserCredentialManager(
+ sparkConf: SparkConf,
+ tokenIngestor: TokenIngestor,
+ onCredentialsUpdate: Array[Byte] => Unit) extends Logging {
+
+ private val safetyMargin = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_SAFETY_MARGIN)
+ private val minInterval = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_MIN_INTERVAL)
+
+ // Thread-safe counter for exponential backoff calculation.
+ // Accessed from both the caller of start() and the scheduled renewal thread.
+ private val consecutiveFailures = new AtomicInteger(0)
+ private val maxBackoffMs: Long = UserCredentialManager.MAX_BACKOFF_MS
+
+ @volatile private var renewalExecutor: ScheduledExecutorService = _
+
+ // Pre-compute the filtered credential configuration map (SparkConf is immutable).
+ private val credentialConfMap: java.util.Map[String, String] = sparkConf.getAll
+ .filter(_._1.startsWith("spark.security.credentials."))
+ .toMap.asJava
+
+ /**
+ * Start the credential manager. Acquires initial credentials and schedules renewal.
+ *
+ * The initial credential acquisition is fail-fast: if the token cannot be loaded or
+ * no credentials can be resolved, an exception is thrown. Subsequent renewal failures
+ * are handled with exponential backoff.
+ *
+ * @return The serialized initial [[UserCredentials]].
+ * @throws IllegalStateException if the initial credential acquisition fails.
+ */
+ def start(): Array[Byte] = {
+ require(renewalExecutor == null, "start() must not be called more than once")
+
+ // Initial acquisition is fail-fast (no retry/backoff).
+ // This ensures the application fails to start if credentials cannot be obtained,
+ // rather than running with null credentials.
+ val userContext = tokenIngestor.load()
+ if (!userContext.isPresent) {
+ throw new IllegalStateException(
+ "Failed to start UserCredentialManager: identity token file is missing or malformed. " +
+ s"Check ${SECURITY_CREDENTIALS_IDENTITY_TOKEN_FILE.key} configuration.")
+ }
+
+ val ctx = userContext.get()
+ logInfo(log"Initial identity token loaded for principal " +
+ log"${MDC(LogKeys.PRINCIPAL, ctx.getPrincipal)} " +
+ log"(issuer: ${MDC(LogKeys.URI, ctx.getIssuer)})")
+
+ val (credentials, earliestExpiry) = resolveCredentials(ctx)
+ val serialized = serializeUserCredentials(credentials)
+
+ // Propagate initial credentials
+ onCredentialsUpdate(serialized)
+
+ // Create the renewal executor only after successful initial acquisition.
+ // This avoids leaking a daemon thread if the fail-fast path throws, and
+ // keeps the require() guard valid for retry scenarios.
+ renewalExecutor =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("user-credential-renewal")
+
+ // Schedule first renewal
+ val renewalDelay = computeRenewalDelay(ctx, earliestExpiry)
+ scheduleRenewal(renewalDelay)
+
+ logInfo(log"Credential acquisition successful. Next renewal in " +
+ log"${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(renewalDelay))}.")
+ serialized
+ }
+
+ def stop(): Unit = {
+ if (renewalExecutor != null) {
+ renewalExecutor.shutdownNow()
+ }
+ }
+
+ /**
+ * Scheduled renewal task: load identity token, resolve credentials, propagate,
+ * and schedule the next renewal. Failures are retried with exponential backoff.
+ */
+ private def renewCredentialsTask(): Unit = {
+ try {
+ val userContext = tokenIngestor.load()
+ if (!userContext.isPresent) {
+ throw new IllegalStateException(
+ "TokenIngestor returned empty - identity token file may be missing or malformed")
+ }
+
+ val ctx = userContext.get()
+ logInfo(log"Loaded identity token for principal " +
+ log"${MDC(LogKeys.PRINCIPAL, ctx.getPrincipal)} " +
+ log"(issuer: ${MDC(LogKeys.URI, ctx.getIssuer)})")
+
+ val (credentials, earliestExpiry) = resolveCredentials(ctx)
+ val serialized = serializeUserCredentials(credentials)
+
+ // Propagate credentials to executors. Errors here are logged separately
+ // so that credential-fetch success is not conflated with distribution failure.
+ try {
+ onCredentialsUpdate(serialized)
+ } catch {
+ case e: Exception =>
+ logWarning(log"Credentials were resolved successfully but failed to propagate " +
+ log"to executors. Executors will receive updated credentials on next renewal.", e)
+ }
+
+ // Reset backoff on successful credential resolution. This is intentionally done
+ // even if onCredentialsUpdate failed above: the backoff counter tracks credential
+ // *resolution* failures (STS/provider issues), not propagation failures.
+ // Propagation failures are transient and will be retried on next scheduled renewal.
+ consecutiveFailures.set(0)
+
+ // Schedule next renewal
+ val renewalDelay = computeRenewalDelay(ctx, earliestExpiry)
+ scheduleRenewal(renewalDelay)
+
+ logInfo(log"Credential renewal successful. Next renewal in " +
+ log"${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(renewalDelay))}.")
+ } catch {
+ case _: InterruptedException =>
+ // Shutting down, ignore
+ case e: Exception =>
+ val failures = consecutiveFailures.incrementAndGet()
+ val delay = computeBackoffDelay()
+ logWarning(log"Failed to renew user credentials (attempt " +
+ log"${MDC(LogKeys.NUM_RETRY, failures)}), " +
+ log"will retry in ${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(delay))}.", e)
+ scheduleRenewal(delay)
+ }
+ }
+
+ /**
+ * Resolve credentials for all schemes that have registered providers.
+ *
+ * @return Tuple of (UserCredentials, earliest expiry across all service credentials)
+ */
+ private def resolveCredentials(
+ ctx: UserContext): (UserCredentials, Option[Instant]) = {
+ val schemes = discoverSchemes()
+
+ val credentialMap = new mutable.HashMap[String, ServiceCredential]()
+ var earliestExpiry: Option[Instant] = None
+
+ for (scheme <- schemes) {
+ try {
+ val providerOpt = CredentialProviderLoader.providerFor(scheme, credentialConfMap)
+ if (providerOpt.isPresent) {
+ val provider = providerOpt.get()
+ // Use a synthetic target URI with just the scheme for initial resolution.
+ // The full URI is passed in future versions when path-specific resolution is needed.
+ // The "synthetic" authority signals this is not a real endpoint but a placeholder
+ // for scheme-based provider selection.
+ val target = new URI(scheme, UserCredentialManager.SYNTHETIC_TARGET_AUTHORITY,
+ "/", null, null)
+ val credential = provider.resolve(ctx, target)
+ credentialMap.put(scheme, credential)
+
+ val expiry = credential.getExpiresAt
+ if (expiry != null) {
+ earliestExpiry = earliestExpiry match {
+ case Some(existing) if existing.isBefore(expiry) => Some(existing)
+ case _ => Some(expiry)
+ }
+ }
+ } else {
+ logWarning(log"No credential provider found for scheme " +
+ log"${MDC(LogKeys.URI, scheme)}. Skipping.")
+ }
+ } catch {
+ case e: Exception =>
+ logWarning(log"Failed to resolve credentials for scheme " +
+ log"${MDC(LogKeys.URI, scheme)}. Skipping this provider.", e)
+ }
+ }
+
+ if (credentialMap.isEmpty) {
+ throw new IllegalStateException(
+ "No credential providers resolved any credentials. " +
+ "Check that providers are on the classpath and configured correctly.")
+ }
+
+ (new UserCredentials(credentialMap.asJava), earliestExpiry)
+ }
+
+ /**
+ * Discover all schemes that have at least one registered provider.
+ *
+ * Schemes are determined by explicit configuration keys of the form
+ * `spark.security.credentials.provider.<scheme>`. If no explicit configuration
+ * exists, the method queries `CredentialProviderLoader` to discover all providers
+ * registered via ServiceLoader and collects their supported schemes.
+ *
+ * Note: When using auto-discovery (no explicit config), multiple providers may
+ * support the same scheme. In that case, `CredentialProviderLoader.providerFor`
+ * will throw an `IllegalArgumentException` for that scheme. The caller
+ * (`resolveCredentials`) handles this gracefully via per-provider exception catching,
+ * logging a warning and continuing with remaining schemes.
+ */
+ private def discoverSchemes(): Set[String] = {
+ // Extract explicitly configured scheme -> provider mappings
+ val explicitSchemes = credentialConfMap.asScala
+ .filter { case (k, _) => k.startsWith("spark.security.credentials.provider.") }
+ .map { case (k, _) => k.stripPrefix("spark.security.credentials.provider.") }
+ .toSet
+
+ if (explicitSchemes.nonEmpty) {
+ explicitSchemes
+ } else {
+ // No explicit scheme configuration. Discover all schemes that have a provider
+ // available on the classpath by probing CredentialProviderLoader.
+ // This covers both built-in providers (e.g., connector/credential-aws for "s3a")
+ // and third-party providers registered via ServiceLoader.
+ CredentialProviderLoader.discoverAllSchemes().asScala.toSet
+ }
+ }
+
+ /**
+ * Compute the delay until next renewal.
+ * Uses min(identity token expiry, service credential expiry) - safetyMargin,
+ * bounded below by minInterval.
+ */
+ private[security] def computeRenewalDelay(
+ ctx: UserContext,
+ earliestCredentialExpiry: Option[Instant]): Long = {
+ val now = System.currentTimeMillis()
+
+ // Consider identity token expiry
+ val tokenExpiry: Option[Long] = Option(ctx.getExpiresAt).map(_.toEpochMilli)
+
+ // Consider earliest service credential expiry
+ val credExpiry: Option[Long] = earliestCredentialExpiry.map(_.toEpochMilli)
+
+ // Take the minimum of both
+ val effectiveExpiry: Option[Long] = (tokenExpiry, credExpiry) match {
+ case (Some(t), Some(c)) => Some(math.min(t, c))
+ case (Some(t), None) => Some(t)
+ case (None, Some(c)) => Some(c)
+ case (None, None) => None
+ }
+
+ effectiveExpiry match {
+ case Some(expiry) =>
+ math.max(expiry - now - safetyMargin, minInterval)
+ case None =>
+ // No expiry information available from either the identity token or service
+ // credentials. Use half of the default suggested TTL (15 min / 2 = 7.5 min,
+ // rounded down) as a conservative polling interval. This ensures credentials
+ // are refreshed even when providers don't report expiration times.
+ UserCredentialManager.DEFAULT_RENEWAL_INTERVAL_NO_EXPIRY_MS
+ }
+ }
+
+ /**
+ * Compute backoff delay using exponential backoff with jitter.
+ * Bounded by minInterval (floor) and maxBackoffMs (ceiling).
+ */
+ private[security] def computeBackoffDelay(): Long = {
+ // Guard against negative or zero shift amounts. consecutiveFailures should always
+ // be >= 1 when this is called (incremented before calling), but we protect against
+ // edge cases defensively.
+ val shiftAmount = math.max(0, math.min(consecutiveFailures.get() - 1, 6))
+ val baseDelay = minInterval * (1L << shiftAmount)
+ val cappedDelay = math.min(baseDelay, maxBackoffMs)
+ // Add 10% jitter to avoid thundering herd on recovery
+ val jitter = (cappedDelay * 0.1 * math.random()).toLong
+ math.max(cappedDelay + jitter, minInterval)
+ }
+
+ private def scheduleRenewal(delay: Long): Unit = {
+ try {
+ val renewalTask = new Runnable {
+ override def run(): Unit = {
+ renewCredentialsTask()
+ }
+ }
+ renewalExecutor.schedule(renewalTask, delay, TimeUnit.MILLISECONDS)
+ } catch {
+ case _: RejectedExecutionException =>
+ // Executor has been shut down (e.g., stop() called concurrently). This is expected
+ // during application shutdown -- no further renewals will be scheduled.
+ logDebug(log"Renewal scheduling rejected - executor is shut down.")
+ }
+ }
+
+ /**
+ * Serialize [[UserCredentials]] to a byte array using Java serialization.
+ */
+ private[security] def serializeUserCredentials(credentials: UserCredentials): Array[Byte] = {
Review Comment:
Moved `serializeUserCredentials` to `object UserCredentialManager` alongside `deserializeUserCredentials` for consistency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement `UserCredentialManager` [spark]
Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on PR #57387:
URL: https://github.com/apache/spark/pull/57387#issuecomment-5110425186
Merged. Thank you all for reviewing!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57893][CORE] Implement UserCredentialManager [spark]
Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on code in PR #57387:
URL: https://github.com/apache/spark/pull/57387#discussion_r3629896137
##########
core/src/main/scala/org/apache/spark/deploy/security/UserCredentialManager.scala:
##########
@@ -0,0 +1,433 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.security
+
+import java.io.{ByteArrayOutputStream, ObjectInputFilter, ObjectInputStream, ObjectOutputStream}
+import java.net.URI
+import java.nio.file.Paths
+import java.time.Instant
+import java.util.concurrent.{RejectedExecutionException, ScheduledExecutorService, TimeUnit}
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.LogKeys
+import org.apache.spark.internal.config._
+import org.apache.spark.security._
+import org.apache.spark.ui.UIUtils
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * Manager for OIDC-based credential propagation on the driver.
+ *
+ * This class is a sibling of [[HadoopDelegationTokenManager]] and handles the OIDC credential
+ * propagation path independently. Both managers run on independent threads and can both be
+ * active simultaneously (e.g., a cluster accessing HDFS via Kerberos and S3 via OIDC).
+ *
+ * Responsibilities:
+ * 1. Reads the current identity token via a [[TokenIngestor]]
+ * 2. Calls [[CredentialProvider#resolve]] for each configured scheme
+ * 3. Serializes the resulting [[UserCredentials]] and invokes the propagation callback
+ * 4. Schedules renewal based on `min(identity token expiry, service credential expiry) -
+ * safetyMargin`
+ * 5. Retries with exponential backoff on failure
+ *
+ * Started from `CoarseGrainedSchedulerBackend.start()` when
+ * `spark.security.credentials.enabled=true`, independently of
+ * `UserGroupInformation.isSecurityEnabled()`.
+ *
+ * Lifecycle: call `start()` exactly once, then `stop()` to shut down.
+ * Calling `start()` after `stop()` is not supported.
+ */
+private[spark] class UserCredentialManager(
+ sparkConf: SparkConf,
+ tokenIngestor: TokenIngestor,
+ onCredentialsUpdate: Array[Byte] => Unit) extends Logging {
+
+ private val safetyMargin = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_SAFETY_MARGIN)
+ private val minInterval = sparkConf.get(SECURITY_CREDENTIALS_RENEWAL_MIN_INTERVAL)
+
+ // Thread-safe counter for exponential backoff calculation.
+ // Accessed from both the caller of start() and the scheduled renewal thread.
+ private val consecutiveFailures = new AtomicInteger(0)
+ private val maxBackoffMs: Long = TimeUnit.MINUTES.toMillis(10)
+
+ @volatile private var renewalExecutor: ScheduledExecutorService = _
+
+ // Pre-compute the filtered credential configuration map (SparkConf is immutable).
+ private val credentialConfMap: java.util.Map[String, String] = sparkConf.getAll
+ .filter(_._1.startsWith("spark.security.credentials."))
+ .toMap.asJava
+
+ /**
+ * Start the credential manager. Acquires initial credentials and schedules renewal.
+ *
+ * The initial credential acquisition is fail-fast: if the token cannot be loaded or
+ * no credentials can be resolved, an exception is thrown. Subsequent renewal failures
+ * are handled with exponential backoff.
+ *
+ * @return The serialized initial [[UserCredentials]].
+ * @throws IllegalStateException if the initial credential acquisition fails.
+ */
+ def start(): Array[Byte] = {
+ require(renewalExecutor == null, "start() must not be called more than once")
+ renewalExecutor =
+ ThreadUtils.newDaemonSingleThreadScheduledExecutor("user-credential-renewal")
+
+ // Initial acquisition is fail-fast (no retry/backoff).
+ // This ensures the application fails to start if credentials cannot be obtained,
+ // rather than running with null credentials.
+ val userContext = tokenIngestor.load()
+ if (!userContext.isPresent) {
+ throw new IllegalStateException(
+ "Failed to start UserCredentialManager: identity token file is missing or malformed. " +
+ s"Check ${SECURITY_CREDENTIALS_IDENTITY_TOKEN_FILE.key} configuration.")
+ }
+
+ val ctx = userContext.get()
+ logInfo(log"Initial identity token loaded for principal " +
+ log"${MDC(LogKeys.PRINCIPAL, ctx.getPrincipal)} " +
+ log"(issuer: ${MDC(LogKeys.URI, ctx.getIssuer)})")
+
+ val (credentials, earliestExpiry) = resolveCredentials(ctx)
+ val serialized = serializeUserCredentials(credentials)
+
+ // Propagate initial credentials
+ onCredentialsUpdate(serialized)
+
+ // Schedule first renewal
+ val renewalDelay = computeRenewalDelay(ctx, earliestExpiry)
+ scheduleRenewal(renewalDelay)
+
+ logInfo(log"Credential acquisition successful. Next renewal in " +
+ log"${MDC(LogKeys.TIME_UNITS, UIUtils.formatDuration(renewalDelay))}.")
+ serialized
+ }
+
+ def stop(): Unit = {
+ if (renewalExecutor != null) {
+ renewalExecutor.shutdownNow()
+ }
+ }
+
+ /**
+ * Scheduled renewal task: load identity token, resolve credentials, propagate,
+ * and schedule the next renewal. Failures are retried with exponential backoff.
+ */
+ private def renewCredentialsTask(): Unit = {
+ try {
+ val userContext = tokenIngestor.load()
+ if (!userContext.isPresent) {
+ throw new IllegalStateException(
+ "TokenIngestor returned empty - identity token file may be missing or malformed")
+ }
+
+ val ctx = userContext.get()
+ logInfo(log"Loaded identity token for principal " +
+ log"${MDC(LogKeys.PRINCIPAL, ctx.getPrincipal)} " +
+ log"(issuer: ${MDC(LogKeys.URI, ctx.getIssuer)})")
+
+ val (credentials, earliestExpiry) = resolveCredentials(ctx)
+ val serialized = serializeUserCredentials(credentials)
+
+ // Propagate credentials to executors. Errors here are logged separately
+ // so that credential-fetch success is not conflated with distribution failure.
+ try {
+ onCredentialsUpdate(serialized)
+ } catch {
+ case e: Exception =>
+ logWarning(log"Credentials were resolved successfully but failed to propagate " +
+ log"to executors. Executors will receive updated credentials on next renewal.", e)
+ }
+
+ // Reset backoff on successful credential resolution. This is intentionally done
+ // even if onCredentialsUpdate failed above: the backoff counter tracks credential
+ // *resolution* failures (STS/provider issues), not propagation failures.
+ // Propagation failures are transient and will be retried on next scheduled renewal.
+ consecutiveFailures.set(0)
Review Comment:
For now, the design relies on Sub-task 5's `TaskDescription`-based credential delivery, which guarantees credentials are available before task execution regardless of RPC broadcast timing. This makes the broadcast a best-effort optimization for credential refresh during long-running tasks rather than a critical delivery path.
If we observe propagation failures causing stale credentials in practice, a shorter propagation-only retry (skipping `tokenIngestor.load()` + `resolveCredentials()`) would be straightforward to add as a follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org