You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Kousuke Saruta <sa...@apache.org> on 2026/06/04 20:12:01 UTC
[DISCUSS] SPIP: OIDC Credential Propagation
Hi all,
I'd like to propose a SPIP for adding delegated credential propagation to
Spark on K8s, generalizing the existing delegation-token model to
OIDC/OAuth-based environments.
SPIP Doc:
https://docs.google.com/document/d/1usJKncCPMiyFUg7aIdpZ0HQsklXIHow_sU_6dfFMjN0/edit?usp=sharing
## Problem
Spark on K8s has no mechanism to take a user/session identity available at
thedriver, exchange it for short-lived storage credentials, and propagate
those credentials to dynamically created executors. All Jobs access cloud
storage as the pod's service account,making per-user authorization and
audit logging impossible without workarounds.
This is the equivalent gap to what Kerberos + allegation tokens solve on
YARN + HDFS but for K8s + cloud storage.
## Proposal
Introduce a CredentialProvider SPI and propagation mechanism that:
1. Reads an OIDC identity token from a configured file path on the driver
2. Exchange it for short-lived service credentials (via STS or compatible)
3. Distributes only those credential (not the raw token) to executors via a
new RPC
4. Automatically refreshes credentials for long-running jobs
The raw identity token never leaves the driver. Executors receive only
short-lived delegated service credentials - mirroring how kerberos
propagates delegation tokens rather than the TGT.
The design mirrors the existing HadoopDelegationTokenManager /
UpdateDelegationTokens pattern, coexists with Kerberos, and is gated by
spark.security.credentials.enabled=false (default).
A reference provider for S3/STS-compatible storage (AWS, MinIO, Ceph) is
included. Azure/GCP providers and Spark Connect integration are out of
scope but the SPI is designed to accommodate them without changes.
Both workload-level SA tokens and per-user identity tokens are supported.
With per-user tokens, STS trust policies can enforce access control based
on the user's identity - enabling true per-user authorization that is
impossible with IRSA/Pod Identity alone.
## Key design decisions
- Core SPI is cloud-agnostic (no AWS/Azure/GCP SDK in core)
- Reference provider lives in connector/credential-aws
- Raw identity token stays on the driver; executors get only delegated
service credentials
- Works with any STS-compatible endpoint (not just AWS)
- @DeveloperApi annotation allows SPI evolution
- Platform-agnostic core, with K8s as the primary target
Full details, architecture diagram, and sequence diagram are in the design
document linked above. I welcome any feedback on the approach.
Thanks,
Kousuke
Re: [DISCUSS] SPIP: OIDC Credential Propagation
Posted by Kousuke Saruta <sa...@apache.org>.
Hi all,
Thank you for your interest in the SPIP documentation and for all your
comments so far.
I plan to hold a vote on SPIP next week.
Thanks,
Kousuke
2026年6月9日(火) 2:18 Parth Chandra <pa...@apache.org>:
> Thanks Kousuke. Let's proceed independently. I'll get something ready
> while we wait for the SPIP review.
>
>
> Parth
>
> On Sat, Jun 6, 2026 at 9:38 AM Kousuke Saruta <fr...@gmail.com>
> wrote:
>
>> Hi Parth,
>>
>> Thank you for the thoughtful response. I think the incremental approach
>> (your path1) might be feasible. Our proposals are complementary and
>> independent. They address different problems and can proceed in parallel
>> without blocking each other.
>> Your DirectTokenProvider unblocks non-Kerberos credential providers in
>> the existing HadoopDelegationTokenManager mechanism. This solves the
>> immediate gate problem for environments where pod-level identity is
>> unavailable or insufficient.
>> My SPIP introduces per-user/per-session identity propagation with a
>> separate manager and RPC, targeting the case where executors need
>> credentials derived from a user identity they cannot obtain themselves.
>> Neither depends on the other landing first. They share no code paths
>> (Different manager, different RPC, and different SPI).
>>
>> Regarding the 2022 review feedback (sorry, I didn't know about that), the
>> constraints have shifted since then. Per-user identity propagation and
>> Spark Connect multi-tenancy require expressing per-session identity, but
>> Spark's current use of UGI is process-wide so per-session scoping would
>> require fundamental changes to how Spark interacts with UGI. In the
>> Appendix C in my SPIP doc, the rejection of UGI reflects these new
>> requirements.
>> Regarding binary payloads in ServiceCredential, for the initial
>> implementation, Base64 encoding within Map[String, String] is sufficient
>> for the S3A use case. Since the SPI is annotated @DeveloperApi, we can add
>> a byte[] field or richer payload type in a future release if concrete
>> integrations require binary credentials. I'd prefer to keep the initial
>> surface small and evolve based on real demand.
>>
>> Best,
>> Kousuke
>>
>> 2026年6月6日(土) 2:40 Parth Chandra <pa...@apache.org>:
>>
>>> Subject: Re: [DISCUSS] SPIP: OIDC Credential Propagation
>>>
>>> Hi Kousuke,
>>>
>>> Thanks for putting this together. As the author of the
>>> original SPARK-38954 [3] and PR #37558 [4], I'm glad to see this problem
>>> getting formal attention — it's been a real gap for cloud-native Spark
>>> deployments.
>>>
>>> I think we're aligned on the problem but differ on scope. Your
>>> proposal addresses identity-aware credential propagation (per-user
>>> authorization, audit trails, Spark Connect multi-tenancy). That's a
>>> compelling long-term direction. The problem I was trying to solve in PR
>>> #37558 [4] is narrower: enable non-Kerberos credential providers to
>>> participate in the existing distribution mechanism, which is already
>>> provider-agnostic (as the Kafka provider demonstrates) but gated on
>>> Kerberos activation.
>>>
>>> After the review feedback on PR #37558 [4] — specifically the
>>> direction that we should use a single auth-agnostic manager and UGI as the
>>> container — I've been working on a minimal approach SPARK-27252 [1][2]: a
>>> DirectTokenProvider sub-trait of the existing
>>> HadoopDelegationTokenProvider, with routing logic inside the existing
>>> HadoopDelegationTokenManager to call direct providers without doAs(). This
>>> requires ~80 lines of changes to existing
>>> code, no new manager, no new RPC message, and no new credential store.
>>> It follows the review feedback from PR #37558 [4] exactly.
>>>
>>> I see two paths forward and am happy with either:
>>>
>>> 1. *Incremental*: The minimal DirectTokenProvider change SPARK-27252
>>> [1][2] lands first, unblocking the immediate use case (driver-mediated
>>> credential refresh without Kerberos). Your UserCredentialManager and
>>> identity-aware architecture can then build on top — or alongside — when
>>> the broader scope (Spark Connect, per-user identity, multi-cloud) is ready.
>>> The two aren't mutually exclusive.
>>> 2. *Unified*: If the community prefers to solve the full identity
>>> propagation problem in one shot, I'd be glad to collaborate on your
>>> proposal. In that case I'd suggest we address the relationship to the 2022
>>> review feedback explicitly — specifically the preference for a single
>>> manager and UGI as a container. Your Appendix C rejects that direction; it
>>> would strengthen the proposal to explain why the constraints have changed
>>> (Spark Connect multi-tenancy, per-user identity requirements that UGI
>>> cannot express).
>>>
>>> One technical observation: your proposal's
>>> CredentialProvider.resolve() returns a ServiceCredential with Map[String,
>>> String] properties. For the S3A case this works well (access key, secret
>>> key, session token are strings). But some credential systems return binary
>>> payloads (signed SAML assertions, serialized protobuf tokens). Worth
>>> considering whether Map[String, byte[]] or an opaque byte[] field alongside
>>> the properties map would future-proof the SPI.
>>>
>>> Happy to discuss further.
>>>
>>> Best,
>>> Parth
>>>
>>> [1] https://issues.apache.org/jira/browse/SPARK-57252
>>> [2]
>>> https://docs.google.com/document/d/1PPqAoJAj48MdjMJNc7DlytXi745z-imFpVaFDnt18Xg/edit?tab=t.0#heading=h.21tncge82jbl
>>> [3] https://issues.apache.org/jira/browse/SPARK-38954
>>> [4] https://github.com/apache/spark/pull/37558
>>>
>>>>
Re: [DISCUSS] SPIP: OIDC Credential Propagation
Posted by Parth Chandra <pa...@apache.org>.
Thanks Kousuke. Let's proceed independently. I'll get something ready while
we wait for the SPIP review.
Parth
On Sat, Jun 6, 2026 at 9:38 AM Kousuke Saruta <fr...@gmail.com> wrote:
> Hi Parth,
>
> Thank you for the thoughtful response. I think the incremental approach
> (your path1) might be feasible. Our proposals are complementary and
> independent. They address different problems and can proceed in parallel
> without blocking each other.
> Your DirectTokenProvider unblocks non-Kerberos credential providers in the
> existing HadoopDelegationTokenManager mechanism. This solves the
> immediate gate problem for environments where pod-level identity is
> unavailable or insufficient.
> My SPIP introduces per-user/per-session identity propagation with a
> separate manager and RPC, targeting the case where executors need
> credentials derived from a user identity they cannot obtain themselves.
> Neither depends on the other landing first. They share no code paths
> (Different manager, different RPC, and different SPI).
>
> Regarding the 2022 review feedback (sorry, I didn't know about that), the
> constraints have shifted since then. Per-user identity propagation and
> Spark Connect multi-tenancy require expressing per-session identity, but
> Spark's current use of UGI is process-wide so per-session scoping would
> require fundamental changes to how Spark interacts with UGI. In the
> Appendix C in my SPIP doc, the rejection of UGI reflects these new
> requirements.
> Regarding binary payloads in ServiceCredential, for the initial
> implementation, Base64 encoding within Map[String, String] is sufficient
> for the S3A use case. Since the SPI is annotated @DeveloperApi, we can add
> a byte[] field or richer payload type in a future release if concrete
> integrations require binary credentials. I'd prefer to keep the initial
> surface small and evolve based on real demand.
>
> Best,
> Kousuke
>
> 2026年6月6日(土) 2:40 Parth Chandra <pa...@apache.org>:
>
>> Subject: Re: [DISCUSS] SPIP: OIDC Credential Propagation
>>
>> Hi Kousuke,
>>
>> Thanks for putting this together. As the author of the
>> original SPARK-38954 [3] and PR #37558 [4], I'm glad to see this problem
>> getting formal attention — it's been a real gap for cloud-native Spark
>> deployments.
>>
>> I think we're aligned on the problem but differ on scope. Your proposal
>> addresses identity-aware credential propagation (per-user authorization,
>> audit trails, Spark Connect multi-tenancy). That's a compelling long-term
>> direction. The problem I was trying to solve in PR #37558 [4] is narrower:
>> enable non-Kerberos credential providers to participate in the existing
>> distribution mechanism, which is already provider-agnostic (as the Kafka
>> provider demonstrates) but gated on Kerberos activation.
>>
>> After the review feedback on PR #37558 [4] — specifically the direction
>> that we should use a single auth-agnostic manager and UGI as the container
>> — I've been working on a minimal approach SPARK-27252 [1][2]: a
>> DirectTokenProvider sub-trait of the existing
>> HadoopDelegationTokenProvider, with routing logic inside the existing
>> HadoopDelegationTokenManager to call direct providers without doAs(). This
>> requires ~80 lines of changes to existing
>> code, no new manager, no new RPC message, and no new credential store.
>> It follows the review feedback from PR #37558 [4] exactly.
>>
>> I see two paths forward and am happy with either:
>>
>> 1. *Incremental*: The minimal DirectTokenProvider change SPARK-27252
>> [1][2] lands first, unblocking the immediate use case (driver-mediated
>> credential refresh without Kerberos). Your UserCredentialManager and
>> identity-aware architecture can then build on top — or alongside — when
>> the broader scope (Spark Connect, per-user identity, multi-cloud) is ready.
>> The two aren't mutually exclusive.
>> 2. *Unified*: If the community prefers to solve the full identity
>> propagation problem in one shot, I'd be glad to collaborate on your
>> proposal. In that case I'd suggest we address the relationship to the 2022
>> review feedback explicitly — specifically the preference for a single
>> manager and UGI as a container. Your Appendix C rejects that direction; it
>> would strengthen the proposal to explain why the constraints have changed
>> (Spark Connect multi-tenancy, per-user identity requirements that UGI
>> cannot express).
>>
>> One technical observation: your proposal's CredentialProvider.resolve()
>> returns a ServiceCredential with Map[String, String] properties. For the
>> S3A case this works well (access key, secret key, session token are
>> strings). But some credential systems return binary payloads (signed SAML
>> assertions, serialized protobuf tokens). Worth considering whether
>> Map[String, byte[]] or an opaque byte[] field alongside the properties map
>> would future-proof the SPI.
>>
>> Happy to discuss further.
>>
>> Best,
>> Parth
>>
>> [1] https://issues.apache.org/jira/browse/SPARK-57252
>> [2]
>> https://docs.google.com/document/d/1PPqAoJAj48MdjMJNc7DlytXi745z-imFpVaFDnt18Xg/edit?tab=t.0#heading=h.21tncge82jbl
>> [3] https://issues.apache.org/jira/browse/SPARK-38954
>> [4] https://github.com/apache/spark/pull/37558
>>
>>>
Re: [DISCUSS] SPIP: OIDC Credential Propagation
Posted by Kousuke Saruta <fr...@gmail.com>.
Hi Parth,
Thank you for the thoughtful response. I think the incremental approach
(your path1) might be feasible. Our proposals are complementary and
independent. They address different problems and can proceed in parallel
without blocking each other.
Your DirectTokenProvider unblocks non-Kerberos credential providers in the
existing HadoopDelegationTokenManager mechanism. This solves the
immediate gate problem for environments where pod-level identity is
unavailable or insufficient.
My SPIP introduces per-user/per-session identity propagation with a
separate manager and RPC, targeting the case where executors need
credentials derived from a user identity they cannot obtain themselves.
Neither depends on the other landing first. They share no code paths
(Different manager, different RPC, and different SPI).
Regarding the 2022 review feedback (sorry, I didn't know about that), the
constraints have shifted since then. Per-user identity propagation and
Spark Connect multi-tenancy require expressing per-session identity, but
Spark's current use of UGI is process-wide so per-session scoping would
require fundamental changes to how Spark interacts with UGI. In the
Appendix C in my SPIP doc, the rejection of UGI reflects these new
requirements.
Regarding binary payloads in ServiceCredential, for the initial
implementation, Base64 encoding within Map[String, String] is sufficient
for the S3A use case. Since the SPI is annotated @DeveloperApi, we can add
a byte[] field or richer payload type in a future release if concrete
integrations require binary credentials. I'd prefer to keep the initial
surface small and evolve based on real demand.
Best,
Kousuke
2026年6月6日(土) 2:40 Parth Chandra <pa...@apache.org>:
> Subject: Re: [DISCUSS] SPIP: OIDC Credential Propagation
>
> Hi Kousuke,
>
> Thanks for putting this together. As the author of the
> original SPARK-38954 [3] and PR #37558 [4], I'm glad to see this problem
> getting formal attention — it's been a real gap for cloud-native Spark
> deployments.
>
> I think we're aligned on the problem but differ on scope. Your proposal
> addresses identity-aware credential propagation (per-user authorization,
> audit trails, Spark Connect multi-tenancy). That's a compelling long-term
> direction. The problem I was trying to solve in PR #37558 [4] is narrower:
> enable non-Kerberos credential providers to participate in the existing
> distribution mechanism, which is already provider-agnostic (as the Kafka
> provider demonstrates) but gated on Kerberos activation.
>
> After the review feedback on PR #37558 [4] — specifically the direction
> that we should use a single auth-agnostic manager and UGI as the container
> — I've been working on a minimal approach SPARK-27252 [1][2]: a
> DirectTokenProvider sub-trait of the existing
> HadoopDelegationTokenProvider, with routing logic inside the existing
> HadoopDelegationTokenManager to call direct providers without doAs(). This
> requires ~80 lines of changes to existing
> code, no new manager, no new RPC message, and no new credential store.
> It follows the review feedback from PR #37558 [4] exactly.
>
> I see two paths forward and am happy with either:
>
> 1. *Incremental*: The minimal DirectTokenProvider change SPARK-27252
> [1][2] lands first, unblocking the immediate use case (driver-mediated
> credential refresh without Kerberos). Your UserCredentialManager and
> identity-aware architecture can then build on top — or alongside — when
> the broader scope (Spark Connect, per-user identity, multi-cloud) is ready.
> The two aren't mutually exclusive.
> 2. *Unified*: If the community prefers to solve the full identity
> propagation problem in one shot, I'd be glad to collaborate on your
> proposal. In that case I'd suggest we address the relationship to the 2022
> review feedback explicitly — specifically the preference for a single
> manager and UGI as a container. Your Appendix C rejects that direction; it
> would strengthen the proposal to explain why the constraints have changed
> (Spark Connect multi-tenancy, per-user identity requirements that UGI
> cannot express).
>
> One technical observation: your proposal's CredentialProvider.resolve()
> returns a ServiceCredential with Map[String, String] properties. For the
> S3A case this works well (access key, secret key, session token are
> strings). But some credential systems return binary payloads (signed SAML
> assertions, serialized protobuf tokens). Worth considering whether
> Map[String, byte[]] or an opaque byte[] field alongside the properties map
> would future-proof the SPI.
>
> Happy to discuss further.
>
> Best,
> Parth
>
> [1] https://issues.apache.org/jira/browse/SPARK-57252
> [2]
> https://docs.google.com/document/d/1PPqAoJAj48MdjMJNc7DlytXi745z-imFpVaFDnt18Xg/edit?tab=t.0#heading=h.21tncge82jbl
> [3] https://issues.apache.org/jira/browse/SPARK-38954
> [4] https://github.com/apache/spark/pull/37558
>
>>
Re: [DISCUSS] SPIP: OIDC Credential Propagation
Posted by Parth Chandra <pa...@apache.org>.
Subject: Re: [DISCUSS] SPIP: OIDC Credential Propagation
Hi Kousuke,
Thanks for putting this together. As the author of the
original SPARK-38954 [3] and PR #37558 [4], I'm glad to see this problem
getting formal attention — it's been a real gap for cloud-native Spark
deployments.
I think we're aligned on the problem but differ on scope. Your proposal
addresses identity-aware credential propagation (per-user authorization,
audit trails, Spark Connect multi-tenancy). That's a compelling long-term
direction. The problem I was trying to solve in PR #37558 [4] is narrower:
enable non-Kerberos credential providers to participate in the existing
distribution mechanism, which is already provider-agnostic (as the Kafka
provider demonstrates) but gated on Kerberos activation.
After the review feedback on PR #37558 [4] — specifically the direction
that we should use a single auth-agnostic manager and UGI as the container
— I've been working on a minimal approach SPARK-27252 [1][2]: a
DirectTokenProvider sub-trait of the existing
HadoopDelegationTokenProvider, with routing logic inside the existing
HadoopDelegationTokenManager to call direct providers without doAs(). This
requires ~80 lines of changes to existing
code, no new manager, no new RPC message, and no new credential store. It
follows the review feedback from PR #37558 [4] exactly.
I see two paths forward and am happy with either:
1. *Incremental*: The minimal DirectTokenProvider change SPARK-27252
[1][2] lands first, unblocking the immediate use case (driver-mediated
credential refresh without Kerberos). Your UserCredentialManager and
identity-aware architecture can then build on top — or alongside — when
the broader scope (Spark Connect, per-user identity, multi-cloud) is ready.
The two aren't mutually exclusive.
2. *Unified*: If the community prefers to solve the full identity
propagation problem in one shot, I'd be glad to collaborate on your
proposal. In that case I'd suggest we address the relationship to the 2022
review feedback explicitly — specifically the preference for a single
manager and UGI as a container. Your Appendix C rejects that direction; it
would strengthen the proposal to explain why the constraints have changed
(Spark Connect multi-tenancy, per-user identity requirements that UGI
cannot express).
One technical observation: your proposal's CredentialProvider.resolve()
returns a ServiceCredential with Map[String, String] properties. For the
S3A case this works well (access key, secret key, session token are
strings). But some credential systems return binary payloads (signed SAML
assertions, serialized protobuf tokens). Worth considering whether
Map[String, byte[]] or an opaque byte[] field alongside the properties map
would future-proof the SPI.
Happy to discuss further.
Best,
Parth
[1] https://issues.apache.org/jira/browse/SPARK-57252
[2]
https://docs.google.com/document/d/1PPqAoJAj48MdjMJNc7DlytXi745z-imFpVaFDnt18Xg/edit?tab=t.0#heading=h.21tncge82jbl
[3] https://issues.apache.org/jira/browse/SPARK-38954
[4] https://github.com/apache/spark/pull/37558
>