You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2026/09/16 20:48:10 UTC
(spark) branch branch-4.x updated: [SPARK-57901][DOC] Add documentation about OIDC credential propagation
This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 2c37b90a8a87 [SPARK-57901][DOC] Add documentation about OIDC credential propagation
2c37b90a8a87 is described below
commit 2c37b90a8a877ecd29edf4341dbf58ac848099e6
Author: Kousuke Saruta <sa...@apache.org>
AuthorDate: Wed Sep 16 13:47:49 2026 -0700
[SPARK-57901][DOC] Add documentation about OIDC credential propagation
### What changes were proposed in this pull request?
Add user-facing documentation for the OIDC credential propagation feature (umbrella SPARK-57703). This is the documentation sub-task (SPARK-57901). It documents the cloud-agnostic framework, its configuration, the security model, the AWS reference provider, and Kubernetes usage, including how provider-declared properties take effect on the driver as well as the executors.
- `docs/security.md`:
- Overview of OIDC credential propagation and its security model: the raw identity token stays on the driver; only the short-lived, delegated service credentials are propagated to executors. TLS/RPC-encryption recommendations.
- "How it works" walkthrough (token ingestion, per-scheme `CredentialProvider` resolution, versioned distribution to executors, renewal). Supported on Kubernetes.
- Configuration table for the core `spark.security.oidc.*` keys (type, default, description).
- "Custom CredentialProvider" and "AWS reference provider" sections (how to enable and configure).
- "Provider-declared configuration and driver-side access": provider-declared properties (`CredentialProvider.additionalSparkProperties()`) are applied on both the driver and the executors (only for keys the user has not set), so with no explicit provider wiring both access storage using the propagated OIDC credentials. Documents two driver-side, early-startup limitations -- access during `SparkContext` construction (`spark.jars` / `spark.files` / `spark.archives` / `spark.checkpoin [...]
- `docs/running-on-kubernetes.md`: an OIDC section with two complete configuration examples -- workload-level ServiceAccount tokens and per-user identity tokens (Kubernetes Secret mount) -- plus a cluster-mode caveat about the `SparkSubmit` `FileSystem` cache.
- `docs/configuration.md` and `docs/cloud-integration.md`: cross-references to the OIDC section.
### Why are the changes needed?
Users need documentation to configure and use OIDC credential propagation: the configuration keys, the security model (the raw token is not sent to executors), complete Kubernetes examples, and the driver-side behavior and its early-startup limitations. Without this the feature is undocumented.
### Does this PR introduce _any_ user-facing change?
Yes -- documentation only (no behavior or API change).
### How was this patch tested?
Built the docs and reviewed the rendered Markdown; verified internal links (e.g. the `running-on-kubernetes.md` caveat links to the `security.md` subsection) and that all core `spark.security.oidc.*` keys are documented with default and description.
### Was this patch authored or co-authored using generative AI tooling?
Kiro CLI / Claude
Closes #58871 from sarutak/oidc-propagation/docs.
Authored-by: Kousuke Saruta <sa...@apache.org>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
(cherry picked from commit 3c25a8bc153dd1170d06335c536a99964b852109)
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
docs/cloud-integration.md | 14 ++
docs/configuration.md | 4 +
docs/running-on-kubernetes.md | 134 ++++++++++++++++
docs/security.md | 347 ++++++++++++++++++++++++++++++++++++++++++
4 files changed, 499 insertions(+)
diff --git a/docs/cloud-integration.md b/docs/cloud-integration.md
index 7afbfef0b393..e53d8fd1e027 100644
--- a/docs/cloud-integration.md
+++ b/docs/cloud-integration.md
@@ -135,6 +135,20 @@ especially public ones*
Consult [the Hadoop documentation](https://hadoop.apache.org/docs/current/) for the relevant
configuration and security options.
+## OIDC Credential Propagation
+
+The authentication methods above use credentials that are either static (keys in configuration) or
+shared by all jobs running on a host (an instance profile or a pod's service account). As an
+alternative, Spark can access object storage using short-lived credentials derived from a
+per-workload or per-user OIDC identity, so that authorization and audit happen at the granularity
+of that identity rather than a shared role. Spark ships a reference provider for Amazon S3 and any
+STS-compatible endpoint (such as MinIO or Ceph) that supplies `fs.s3a.*` credentials to the S3A
+connector.
+
+This is described in full, including how to enable the reference provider and its configuration
+options, under [OIDC Credential Propagation](security.html#oidc-credential-propagation) on the
+security page.
+
## Configuring
Each cloud connector has its own set of configuration parameters, again,
diff --git a/docs/configuration.md b/docs/configuration.md
index 683d1048dca6..963738953bda 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -3787,6 +3787,10 @@ Command types in proto.</td>
Please refer to the [Security](security.html) page for available options on how to secure different
Spark subsystems.
+For OIDC credential propagation (obtaining and distributing short-lived, identity-derived
+credentials to executors), the `spark.security.oidc.*` configuration keys are documented under
+[OIDC Credential Propagation](security.html#oidc-credential-propagation).
+
### Spark SQL
diff --git a/docs/running-on-kubernetes.md b/docs/running-on-kubernetes.md
index 20d96857b3c6..4a2037a6c850 100644
--- a/docs/running-on-kubernetes.md
+++ b/docs/running-on-kubernetes.md
@@ -290,6 +290,140 @@ To use a secret through an environment variable use the following options to the
--conf spark.kubernetes.executor.secretKeyRef.ENV_NAME=name:key
```
+## OIDC Credential Propagation
+
+Spark can propagate short-lived, identity-derived credentials to executors so that a job accesses
+cloud storage as a specific workload or user rather than as the pod's shared service account. On
+Kubernetes this pairs naturally with projected ServiceAccount tokens, which the kubelet issues as
+OIDC JWTs and rotates automatically.
+
+The mechanism, security model, and core configuration keys are described under
+[OIDC Credential Propagation](security.html#oidc-credential-propagation) on the security page. The
+AWS S3 / STS reference provider and its options are described under
+[AWS reference provider](security.html#aws-reference-provider) on the same page. This section shows
+two complete Kubernetes configurations that use the AWS reference provider.
+
+In both examples, the driver reads the identity token, exchanges it for temporary credentials via
+`sts:AssumeRoleWithWebIdentity`, and propagates those credentials to executors. The raw token stays
+on the driver; executors only ever receive the short-lived S3 credentials. Because credentials are
+carried over Spark's RPC channels, enable [RPC encryption](security.html#network-encryption)
+whenever you enable credential propagation.
+
+The examples below enable AES-based RPC encryption with `spark.authenticate=true` and
+`spark.network.crypto.enabled=true`. In general Spark documents SSL-based RPC encryption
+(`spark.ssl.rpc.enabled=true`) as the preferred method and AES-based encryption as the legacy one
+(see [Network Encryption](security.html#network-encryption)). The examples use the AES-based method
+because on Kubernetes Spark automatically generates and distributes the authentication secret, so it
+needs no additional key material, whereas the SSL-based method requires configuring a key store and
+related settings; choose SSL if your environment already standardizes on it.
+
+With credential propagation enabled and no `spark.hadoop.fs.s3a.aws.credentials.provider` set in
+your Spark configuration (a value in `core-site.xml` is not detected), both the
+driver and the executors access S3 using the propagated OIDC credentials. One caveat is specific to
+`cluster` mode: if you pass application dependencies from the same object store the job reads or
+writes (for example `--jars s3a://BUCKET/app.jar` where output also goes to `s3a://BUCKET/...`),
+`spark-submit` fetches those dependencies before the credentials are wired up and caches a
+`FileSystem` for that bucket using the pod's service account. The driver's later access to the same
+bucket then reuses that cached `FileSystem` and runs under the service account rather than the OIDC
+identity (the job succeeds, but under a mixed identity). Prefer `local://` for `--jars` / `--files`,
+or set `spark.hadoop.fs.s3a.impl.disable.cache=true`. See
+[Provider-declared configuration and driver-side access](security.html#provider-declared-configuration-and-driver-side-access)
+for the full discussion.
+
+These examples assume a Spark image built with both the `credential-aws` and `hadoop-cloud`
+modules (`-Pcredential-aws -Phadoop-cloud`), so that the reference provider and the S3A connector
+are both present. See
+[Enabling the AWS reference provider](security.html#enabling-the-aws-reference-provider)
+for details.
+
+### Example: workload-level ServiceAccount token
+
+This is the simplest setup. Kubernetes automatically mounts a projected ServiceAccount token, and
+the driver uses it directly as the OIDC identity. The IAM role's trust policy must trust the
+cluster's OIDC issuer and the driver's ServiceAccount.
+
+Add a projected token volume to the driver via a pod template (`driver-pod-template.yaml`):
+
+```yaml
+apiVersion: v1
+kind: Pod
+spec:
+ containers:
+ - name: spark-kubernetes-driver
+ volumeMounts:
+ - name: oidc-token
+ mountPath: /var/run/secrets/oidc
+ readOnly: true
+ volumes:
+ - name: oidc-token
+ projected:
+ sources:
+ - serviceAccountToken:
+ path: token
+ expirationSeconds: 3600
+ # The audience must match the audience your IAM role's trust policy expects.
+ audience: sts.amazonaws.com
+```
+
+Then submit with credential propagation enabled:
+
+```bash
+/opt/spark/bin/spark-submit \
+ --deploy-mode cluster \
+ --master k8s://<KUBERNETES_MASTER_ENDPOINT> \
+ --conf spark.kubernetes.container.image=<spark-image-with-credential-aws> \
+ --conf spark.kubernetes.driver.podTemplateFile=driver-pod-template.yaml \
+ --conf spark.authenticate=true \
+ --conf spark.network.crypto.enabled=true \
+ --conf spark.security.oidc.enabled=true \
+ --conf spark.security.oidc.identityToken.file=/var/run/secrets/oidc/token \
+ --conf spark.security.oidc.aws.roleArn=arn:aws:iam::123456789012:role/spark-data-access \
+ <application-jar> <args>
+```
+
+Only the driver needs the projected token volume; executors receive the derived credentials over
+RPC and do not read the token themselves.
+
+### Example: per-user identity token
+
+For per-user access control, the identity token represents an individual user rather than the
+workload. Delivering such a token to the driver pod is outside Spark's scope and is handled by
+existing Kubernetes mechanisms, for example by mounting a
+[Secret](https://kubernetes.io/docs/concepts/configuration/secret/) that an external system (such
+as a sidecar or an admission webhook) populates with the user's token. Spark simply reads the token
+from the configured file path.
+
+Create a Secret holding the user's identity token and mount it into the driver using Spark's
+[secret management](#secret-management) options:
+
+```bash
+kubectl create secret generic user-oidc-token \
+ --from-file=token=/path/to/user-identity-token.jwt
+```
+
+```bash
+/opt/spark/bin/spark-submit \
+ --deploy-mode cluster \
+ --master k8s://<KUBERNETES_MASTER_ENDPOINT> \
+ --conf spark.kubernetes.container.image=<spark-image-with-credential-aws> \
+ --conf spark.kubernetes.driver.secrets.user-oidc-token=/etc/oidc \
+ --conf spark.authenticate=true \
+ --conf spark.network.crypto.enabled=true \
+ --conf spark.security.oidc.enabled=true \
+ --conf spark.security.oidc.identityToken.file=/etc/oidc/token \
+ --conf spark.security.oidc.aws.roleArn=arn:aws:iam::123456789012:role/spark-user-access \
+ <application-jar> <args>
+```
+
+Here `spark.kubernetes.driver.secrets.user-oidc-token=/etc/oidc` mounts the `user-oidc-token`
+Secret at `/etc/oidc`, so its `token` key becomes the file `/etc/oidc/token` that
+`spark.security.oidc.identityToken.file` points to. The Secret is mounted only on the driver; as
+before, executors receive only the derived S3 credentials.
+
+When the mounted token is rotated (the external system updates the Secret), Spark detects the change
+and re-exchanges it on the next renewal cycle, so long-running applications continue to work across
+token rotation.
+
## Pod Template
Kubernetes allows defining pods from [template files](https://kubernetes.io/docs/concepts/workloads/pods/pod-overview/#pod-templates).
Spark users can similarly use template files to define the driver or executor pod configurations that Spark configurations do not support.
diff --git a/docs/security.md b/docs/security.md
index 15b8e498617f..2affb692603c 100644
--- a/docs/security.md
+++ b/docs/security.md
@@ -1146,6 +1146,353 @@ achieved by setting `spark.kubernetes.hadoop.configMapName` to a pre-existing Co
local:///opt/spark/examples/jars/spark-examples_<VERSION>.jar \
<HDFS_FILE_LOCATION>
```
+
+# OIDC Credential Propagation
+
+Spark supports propagating short-lived, per-workload or per-user credentials to executors in
+environments that use OIDC / OAuth 2.0 for authentication, such as Spark on Kubernetes accessing
+cloud object storage. This generalizes the delegation-token propagation model described in the
+[Kerberos](#kerberos) section above to OIDC-based systems: instead of obtaining Hadoop delegation
+tokens from a KDC, the driver reads an OIDC identity token (a JWT), exchanges it for short-lived
+service credentials, and distributes those credentials to executors, refreshing them automatically
+for long-running applications.
+
+This feature is disabled by default. It is enabled by setting `spark.security.oidc.enabled=true`
+and providing an identity token file via `spark.security.oidc.identityToken.file`. When disabled,
+none of the components described below are started and existing applications are unaffected.
+
+As with the Kerberos path, Spark's role here is limited to propagating delegated credentials, not
+to authenticating users or enforcing access control. The downstream service (for example, a cloud
+IAM system or an object store) performs authorization using the credential Spark propagates. The
+raw identity token is consumed only on the driver; executors receive only the short-lived service
+credentials derived from it.
+
+## How it works
+
+When enabled, credential propagation runs on independent threads alongside the Kerberos delegation
+token machinery, and the two coexist without interfering with each other. A cluster that accesses
+HDFS via Kerberos and cloud storage via OIDC at the same time is a supported configuration.
+
+Credential propagation targets Spark on Kubernetes accessing cloud object storage, which is the
+deployment this feature is designed and documented for. See
+[Running Spark on Kubernetes](running-on-kubernetes.html#oidc-credential-propagation) for complete
+configuration examples.
+
+1. On the driver, a token ingestor reads the OIDC identity token from the configured file and
+ produces a driver-only user context. Kubernetes projected ServiceAccount tokens (which are
+ rotated automatically by the kubelet) and externally-injected per-user tokens are both supported,
+ because the ingestor detects file rotation and reloads the token.
+1. A driver-side credential manager (a sibling of the Kerberos delegation token manager) passes the
+ user context to a `CredentialProvider` for each configured scheme, obtaining a short-lived service
+ credential.
+1. The service credentials are serialized and distributed to all registered executors via an RPC
+ message. Newly-registered executors (for example, those created by dynamic allocation) receive
+ the current credentials as part of their registration response, and each launched task also
+ carries the current credentials in its `TaskDescription`, so executors have valid credentials
+ before running any task. All of these delivery paths travel over Spark's RPC channels and are
+ covered by the same RPC encryption.
+1. The manager schedules the next renewal ahead of the earliest expiry (the minimum of the identity
+ token expiry and the service credential expiry, minus a configurable safety margin), and retries
+ with exponential backoff on failure.
+1. On the executor, connector-specific code reads the current service credential from an in-memory
+ store on each access, so credential refreshes are picked up without restarting tasks or
+ invalidating filesystem caches.
+
+The components mirror the existing Kerberos delegation token path:
+
+| OIDC path | Kerberos path |
+|---|---|
+| OIDC identity token file | Keytab / ticket cache |
+| Driver-side credential manager | `HadoopDelegationTokenManager` |
+| `CredentialProvider.resolve()` | `HadoopDelegationTokenProvider.obtainDelegationTokens()` |
+| `UpdateUserCredentials` RPC | `UpdateDelegationTokens` RPC |
+| Executor credential store | `SparkEnv` delegation credentials |
+
+## Security model
+
+The security model closely follows the Kerberos delegation token model, where executors receive
+delegation tokens rather than the original ticket-granting ticket.
+
+* The raw identity token never leaves the driver. Executors receive only short-lived, scoped
+ service credentials produced by a `CredentialProvider`. A service credential is narrower and
+ shorter-lived than the original identity token, but should still be treated as sensitive.
+* The driver-side user context that holds the raw token is never serialized or transmitted (it is
+ not `Serializable`), and it redacts the raw token in its `toString()` representation, so the token
+ is not written to logs.
+* Service credentials are not written to shuffle files, event logs, or checkpoints.
+* Credentials are carried over Spark's RPC channels and therefore rely on Spark's existing RPC
+ encryption configuration. Operators are strongly encouraged to enable RPC encryption whenever
+ credential propagation is enabled. See [Network Encryption](#network-encryption) for details.
+
+## Custom CredentialProvider
+
+The credential exchange step is pluggable. Spark discovers implementations of
+[`CredentialProvider`](api/java/org/apache/spark/security/CredentialProvider.html) using the Java
+Services mechanism (see `java.util.ServiceLoader`): an implementation is made available to Spark by
+listing its fully-qualified class name in the corresponding file in the jar's `META-INF/services`
+directory, exactly as with custom `HadoopDelegationTokenProvider` implementations for the Kerberos
+path.
+
+A `CredentialProvider` declares the URI schemes it supports (for example, `s3a`) and exchanges the
+driver-side user context for a
+[`ServiceCredential`](api/java/org/apache/spark/security/ServiceCredential.html) scoped to a target
+URI. When more than one provider is registered for the same scheme, the provider used for that
+scheme is selected via `spark.security.oidc.provider.<scheme>`. The configuration passed to a
+provider is scoped to keys starting with `spark.security.oidc.`, so unrelated configuration is not
+exposed to third-party providers.
+
+The `CredentialProvider` SPI and its related types
+([`UserContext`](api/java/org/apache/spark/security/UserContext.html),
+[`ServiceCredential`](api/java/org/apache/spark/security/ServiceCredential.html), and
+[`UserCredentials`](api/java/org/apache/spark/security/UserCredentials.html)) are annotated
+`@DeveloperApi` and may evolve in minor releases.
+
+Spark ships a reference implementation for AWS S3 / STS-compatible endpoints in an optional module.
+See [AWS reference provider](#aws-reference-provider) below for how to enable and configure it, and
+[Running Spark on Kubernetes](running-on-kubernetes.html#oidc-credential-propagation) for complete
+configuration examples using Kubernetes projected ServiceAccount tokens and per-user identity tokens.
+
+## Provider-declared configuration and driver-side access
+
+A `CredentialProvider` may declare additional Spark configuration properties that should be set
+whenever it is selected (via `CredentialProvider.additionalSparkProperties()`), for example the
+Hadoop credentials-provider class for a filesystem scheme. Spark applies these properties on both
+the driver and the executors -- only for keys the user has not already set explicitly -- before the
+components that consume them are initialized. This means that, with the feature enabled and no
+explicit provider wiring configured, both the driver and the executors access storage using the
+propagated OIDC credentials.
+
+There are two situations, both specific to accessing storage from the *driver* very early in
+startup, where the propagated credentials are not yet available. Both only matter when the driver
+itself accesses cloud storage (a common case in cluster mode: output-path existence checks, the
+commit protocol, and schema inference all run on the driver).
+
+* **Access during `SparkContext` construction.** The provider wiring is applied early in
+ `SparkContext` initialization, but the credentials it points at are not resolved until the
+ scheduler backend starts, slightly later in the same initialization. Any driver-side storage
+ access that happens in between runs before the credentials exist and therefore cannot use them.
+ This affects resources fetched or validated during construction from a scheme served by a
+ `CredentialProvider` (for example `s3a://`): `spark.jars`, `spark.files`, `spark.archives`, and
+ `spark.checkpoint.dir`. Note the two failure shapes: `spark.files` / `spark.archives` and
+ `spark.checkpoint.dir` propagate the error and fail `SparkContext` construction, while
+ `spark.jars` swallows it (the jar is silently dropped, and tasks later fail with
+ `ClassNotFoundException`). Prefer `local://` for such resources, or stage them through a location
+ that does not require the propagated credentials.
+
+* **Hadoop `FileSystem` cache in Kubernetes cluster mode.** In cluster mode on Kubernetes the driver
+ pod runs `SparkSubmit` in the same JVM as `SparkContext`. When `SparkSubmit` downloads the
+ application resource, `--jars`, or `--files` from a bucket (for example
+ `--jars s3a://BUCKET/app.jar`), it does so before provider selection runs, and the resulting
+ `FileSystem` instance is cached (the cache key is scheme + authority + user). A later driver-side
+ access to the *same* bucket -- such as writing job output to it -- reuses that cached
+ `FileSystem`, which still uses the default credential chain (typically the pod's service account)
+ rather than the propagated OIDC identity. The job does not fail; instead the driver's access to
+ that bucket runs under a different identity than the executors, which is easy to overlook. To
+ avoid this, use `local://` for `--jars` / `--files`, or disable the S3A filesystem cache (for
+ example, `spark.hadoop.fs.s3a.impl.disable.cache=true`), so the driver builds a fresh
+ `FileSystem` that picks up the propagated provider. Note that this disables the cache for the
+ entire `s3a` scheme (not just one bucket), which has a performance cost because a new
+ `FileSystem` is created per access instead of being reused.
+
+## Configuration
+
+The following options control the core (cloud-agnostic) credential propagation framework.
+Provider-specific options (such as those for the AWS reference provider) are documented alongside
+each provider.
+
+<table class="spark-config">
+<thead><tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr></thead>
+<tr>
+ <td><code>spark.security.oidc.enabled</code></td>
+ <td><code>false</code></td>
+ <td>
+ Whether to enable OIDC credential propagation. When enabled, the driver reads an identity
+ token from a file, exchanges it for short-lived service credentials via
+ <code>CredentialProvider</code> implementations, and propagates those credentials to executors.
+ When disabled (the default), the feature's components are not started.
+ </td>
+ <td>4.4.0</td>
+</tr>
+<tr>
+ <td><code>spark.security.oidc.identityToken.file</code></td>
+ <td>(none)</td>
+ <td>
+ Path to the OIDC identity token file on the driver. Required when
+ <code>spark.security.oidc.enabled</code> is <code>true</code>. The file should contain a JWT
+ (for example, a Kubernetes projected ServiceAccount token). The file is re-read when it is
+ rotated, so automatically-rotated tokens are supported.
+ </td>
+ <td>4.4.0</td>
+</tr>
+<tr>
+ <td><code>spark.security.oidc.provider.<scheme></code></td>
+ <td>(none)</td>
+ <td>
+ Selects the <code>CredentialProvider</code> to use for a given URI scheme (for example,
+ <code>s3a</code>) when more than one provider is registered for that scheme. The value is the
+ fully-qualified class name of the provider. When only one provider is registered for a scheme,
+ this option is not required.
+ </td>
+ <td>4.4.0</td>
+</tr>
+<tr>
+ <td><code>spark.security.oidc.renewal.safetyMargin</code></td>
+ <td><code>60s</code></td>
+ <td>
+ How long before credential expiry to trigger renewal. Credentials are refreshed at
+ <code>min(identity token expiry, service credential expiry)</code> minus this margin.
+ </td>
+ <td>4.4.0</td>
+</tr>
+<tr>
+ <td><code>spark.security.oidc.renewal.minInterval</code></td>
+ <td><code>30s</code></td>
+ <td>
+ Minimum interval between credential renewal attempts. This prevents tight renewal loops when
+ credentials have very short lifetimes or when repeated failures would otherwise cause rapid
+ retries.
+ </td>
+ <td>4.4.0</td>
+</tr>
+</table>
+
+## AWS reference provider
+
+Spark ships a reference `CredentialProvider` implementation for Amazon S3 and any STS-compatible
+endpoint (such as MinIO or Ceph) in the optional `credential-aws` module. It calls
+`sts:AssumeRoleWithWebIdentity` with the identity token and a configured IAM role, and returns
+temporary credentials as the `fs.s3a.access.key`, `fs.s3a.secret.key`, and `fs.s3a.session.token`
+properties consumed by the S3A connector.
+
+### Enabling the AWS reference provider
+
+The reference provider is packaged in the `credential-aws` module, which is not part of the default
+Spark distribution (it is kept separate so that the AWS SDK is not added to the core classpath).
+
+The recommended way to make it available (especially for cluster deployments such as Spark on
+Kubernetes) is to build a custom Spark distribution or container image that includes the module,
+using the `-Pcredential-aws` profile. Reading and writing S3 data additionally requires the S3A
+connector (`hadoop-aws`), provided by the `hadoop-cloud` module (the `-Phadoop-cloud` profile); the
+two modules are designed to be used together and pin the AWS SDK to the same version, so build with
+both:
+
+```
+./dev/make-distribution.sh --pip -Pkubernetes -Pcredential-aws -Phadoop-cloud
+```
+
+Baking the module into the image keeps executor startup self-contained and avoids each executor
+resolving artifacts from a remote repository at launch time, which matters when many executors
+start (for example, with dynamic allocation).
+
+Alternatively, for interactive use or one-off jobs, the module can be fetched from Maven Central at
+submit time with `--packages` (this also works in Kubernetes cluster mode):
+
+```
+--packages org.apache.spark:spark-credential-aws_{{site.SCALA_BINARY_VERSION}}:{{site.SPARK_VERSION_SHORT}}
+```
+
+Once the module is on the classpath, the provider is discovered automatically via `ServiceLoader`.
+Enable credential propagation and point Spark at the identity token file (on Kubernetes this is
+typically a projected ServiceAccount token), then configure the IAM role to assume:
+
+```
+spark.security.oidc.enabled true
+spark.security.oidc.identityToken.file /var/run/secrets/oidc/token
+spark.security.oidc.aws.roleArn arn:aws:iam::123456789012:role/spark-data-access
+```
+
+When credential propagation is enabled and you have not set
+`spark.hadoop.fs.s3a.aws.credentials.provider` in your Spark configuration, Spark automatically
+configures the S3A connector on the driver and executors to read the propagated credentials by
+setting:
+
+```
+spark.hadoop.fs.s3a.aws.credentials.provider org.apache.spark.security.aws.SparkOidcAwsCredentialsProvider
+```
+
+Only the `spark.hadoop.fs.s3a.aws.credentials.provider` key in the Spark configuration is
+considered. If you set it yourself, Spark does not override it. A value set elsewhere -- for
+example `fs.s3a.aws.credentials.provider` in `core-site.xml` -- is not detected, and the injected
+`spark.hadoop.*` value takes precedence over it.
+
+For complete, deployment-ready examples on Kubernetes (including how to mount a projected
+ServiceAccount token for workload-level identity, or an externally-injected per-user identity token),
+see [OIDC Credential Propagation](running-on-kubernetes.html#oidc-credential-propagation) on the
+Kubernetes page.
+
+### Using a non-AWS STS-compatible endpoint
+
+The provider works with any STS-compatible endpoint by setting
+`spark.security.oidc.aws.stsEndpoint` (and, if needed, `spark.security.oidc.aws.region`). For
+example, to target a MinIO STS endpoint:
+
+```
+spark.security.oidc.aws.stsEndpoint https://minio.example.com
+spark.security.oidc.aws.region us-east-1
+```
+
+### AWS reference provider configuration
+
+These options are read by the AWS reference provider (from the `credential-aws` module) and are
+only relevant when it is on the classpath and credential propagation is enabled.
+
+<table class="spark-config">
+<thead><tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr></thead>
+<tr>
+ <td><code>spark.security.oidc.aws.roleArn</code></td>
+ <td>(none)</td>
+ <td>
+ The ARN of the IAM role to assume via <code>AssumeRoleWithWebIdentity</code>. Required when the
+ AWS reference provider is used. The role's trust policy must allow the identity provider that
+ issued the token (for example, the Kubernetes cluster's OIDC issuer).
+ </td>
+ <td>4.4.0</td>
+</tr>
+<tr>
+ <td><code>spark.security.oidc.aws.sessionName</code></td>
+ <td>(derived from the token principal)</td>
+ <td>
+ The role session name attached to the assumed-role session. It appears in CloudTrail records
+ and can be used for auditing. Must match <code>[a-zA-Z0-9_+=,.@-]{2,64}</code>. When unset, the
+ session name is derived from the identity token's principal (sanitized to satisfy the allowed
+ character set and length); if no usable principal is available, it falls back to
+ <code>spark-oidc</code>.
+ </td>
+ <td>4.4.0</td>
+</tr>
+<tr>
+ <td><code>spark.security.oidc.aws.durationSeconds</code></td>
+ <td>(STS default, 3600)</td>
+ <td>
+ The requested lifetime, in seconds, of the temporary credentials. Must be between
+ <code>900</code> (15 minutes) and <code>43200</code> (12 hours), subject to the maximum
+ session duration configured on the IAM role. When unset, the STS default (1 hour) applies.
+ </td>
+ <td>4.4.0</td>
+</tr>
+<tr>
+ <td><code>spark.security.oidc.aws.region</code></td>
+ <td>(SDK default)</td>
+ <td>
+ The AWS region for the STS client. When unset and no custom endpoint is configured, the AWS
+ SDK's default region resolution applies (for example, the <code>AWS_REGION</code> environment
+ variable or the shared config profile). When a custom endpoint is set without a region,
+ <code>us-east-1</code> is used as a placeholder.
+ </td>
+ <td>4.4.0</td>
+</tr>
+<tr>
+ <td><code>spark.security.oidc.aws.stsEndpoint</code></td>
+ <td>(none)</td>
+ <td>
+ A custom STS endpoint URL. Set this to target an STS-compatible endpoint other than AWS STS,
+ such as MinIO or Ceph. When unset, the AWS SDK's default STS endpoint for the resolved region
+ is used.
+ </td>
+ <td>4.4.0</td>
+</tr>
+</table>
+
# Event Logging
If your applications are using event logging, the directory where the event logs go
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org