You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@polaris.apache.org by "michal-kuzak (via GitHub)" <gi...@apache.org> on 2025/11/13 10:21:22 UTC
[I] [FEATURE REQUEST] - Support S3 STS AssumeRoleWithWebIdentity for on‑prem S3 providers [polaris]
michal-kuzak opened a new issue, #3038:
URL: https://github.com/apache/polaris/issues/3038
### Is your feature request related to a problem? Please describe.
Polaris currently supports S3 access via STS AssumeRole (role ARN + optional external ID). Please add first‑class support for STS AssumeRoleWithWebIdentity (ARWI). Many enterprise S3‑compatible object stores deployed on‑prem rely on OIDC federation and expose an STS endpoint that accepts ARWI, rather than operating a native IAM service. Examples include Pure Storage FlashBlade, or Dell S3 storage among others. Adding ARWI unlocks secure, short‑lived, IdP‑federated access to S3‑compatible stores without long‑lived keys.
On‑prem S3 providers often delegate authentication to external IdPs (Okta, Microsoft Entra ID, etc.) and expect clients to obtain credentials via AssumeRoleWithWebIdentity to their STS endpoint. Polaris today documents role‑arn/region/external‑id and custom S3/STS endpoints, but not ARWI. Supporting ARWI would let Polaris run in OIDC‑first environments (Kubernetes, corporate SSO) and still vend short‑lived credentials to engines.
### Describe the solution you'd like
Add a storage auth selector for S3 catalogs, e.g. stsAuthMethod with values:
- assumeRole (current default/behavior)
- webIdentity (new)
When stsAuthMethod=webIdentity:
- Use AWS SDK’s ARWI credentials provider (e.g., StsAssumeRoleWithWebIdentity… in AWS SDK v2) against the configured sts-endpoint (already supported). ([polaris.incubator.apache.org](https://polaris.incubator.apache.org/in-dev/unreleased/command-line-interface/?utm_source=openai))
- Accept inputs via CLI and REST model:
- role-arn (unchanged)
- role-session-name (optional)
- web-identity-token-file (path to OIDC/JWT file)
- region (optional; also allow via env)
- sts-endpoint (already documented)
### Describe alternatives you've considered
_No response_
### Additional context
Default remains stsAuthMethod=assumeRole; existing catalogs and clients are unaffected.
IMPLICIT mode continues to work; ARWI simply adds another supported path to obtain base credentials.
Ref:
https://support.purestorage.com/bundle/FlashBlade_Admin_Guide_4.6.4/resource/FlashBlade_Admin_Guide_4.6.4.pdf (p.462)
https://support.purestorage.com/bundle/m_purityfb_rest_api/page/FlashBlade/Purity_FB/PurityFB_REST_API/S3_Object_Store_REST_API/topics/concept/c_flashblade_object_store_documentation_s3_api.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [FEATURE REQUEST] - Support S3 STS AssumeRoleWithWebIdentity for on‑prem S3 providers [polaris]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on issue #3038:
URL: https://github.com/apache/polaris/issues/3038#issuecomment-4799571456
@guillaumBrisard thanks for the update. I think it's "more" than just Polaris (Iceberg client is probably involved). We did several updates on Iceberg client recently thanks to @adutra.
Let me take a look on this one to see if we can do a little move in Polaris 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [FEATURE REQUEST] - Support S3 STS AssumeRoleWithWebIdentity for on‑prem S3 providers [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on issue #3038:
URL: https://github.com/apache/polaris/issues/3038#issuecomment-3558711658
The user who deploys Polaris would have to provide the initial token in a file accessible to Polaris servers. It is going to be a "service" account effectively.
@cccs-cat001 : by "user" do you mean a user of the Polaris API? Do you mean that user's identity to carry forward to the S3 requests made as a result of the Polaris REST API request? That would be an interesting feature, doable, I think, but it would require a lot of preparatory work, I'm sure 😅
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [FEATURE REQUEST] - Support S3 STS AssumeRoleWithWebIdentity for on‑prem S3 providers [polaris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #3038:
URL: https://github.com/apache/polaris/issues/3038#issuecomment-3900461241
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment if you think it's still relevant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [FEATURE REQUEST] - Support S3 STS AssumeRoleWithWebIdentity for on‑prem S3 providers [polaris]
Posted by "guillaumBrisard (via GitHub)" <gi...@apache.org>.
guillaumBrisard commented on issue #3038:
URL: https://github.com/apache/polaris/issues/3038#issuecomment-4799543142
To had documentation for this issue from Everpure, they say:
Apache Polaris works on FlashBlade today. The REST API, OAuth2 access control, multi-engine support, and clean metadata separation are real, tangible improvements over Hive Metastore for on-prem Iceberg deployments. Credential vending is not yet available because Polaris uses AssumeRole rather than the OIDC federation that FlashBlade supports, but the static credential model that replaces it is already familiar to most on-prem teams.
The path to full credential vending is clear: FlashBlade already speaks OIDC, and Polaris just needs to learn to as well. We’re looking forward to contributing to that effort and making Polaris a first-class catalog for every Iceberg deployment, cloud and on-prem alike.
https://blog.everpuredata.com/purely-technical/using-apache-polaris-with-flashblade/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [FEATURE REQUEST] - Support S3 STS AssumeRoleWithWebIdentity for on‑prem S3 providers [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on issue #3038:
URL: https://github.com/apache/polaris/issues/3038#issuecomment-3559405459
@dparent1 :
The current state of the server codebase is that Polaris uses fixed base credentials for its own access to storage in all requests. It can use external IdP for authenticating API requests, but will switch to the single "service" credential for storage access.
> I'd like to pass the jwt token provided to polaris to the s3 appliance so it can allow / deny access based on the claims in the jwt token.
I fully support implementing this use case... however, I do believe it will require a lot of code changes in Polaris 😅 just a fair warning 😉 Ultimately, this is about feeding the right input credentials into [AwsCredentialsStorageIntegration](https://github.com/apache/polaris/blob/aa72157a8433c98f1de19b4237d31188cc6c3c83/polaris-core/src/main/java/org/apache/polaris/core/storage/aws/AwsCredentialsStorageIntegration.java#L73).
I'd propose to move this discussion to the `dev` [ML](https://polaris.apache.org/community/) for visibility and because I believe email is a better tool than GH comments for design discussions :) As for me, I do not have a solid plan for this in my mind ATM, so I suppose it will have to be a collaborative and iterative process... as it's supposed to be at ASF :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [FEATURE REQUEST] - Support S3 STS AssumeRoleWithWebIdentity for on‑prem S3 providers [polaris]
Posted by "cccs-cat001 (via GitHub)" <gi...@apache.org>.
cccs-cat001 commented on issue #3038:
URL: https://github.com/apache/polaris/issues/3038#issuecomment-3554544733
I just came to the same conclusion that we need this! I'd be willing to make a PR if someone can help with the design 😅 I've found right here is where we need to make this change: https://github.com/apache/polaris/blob/main/polaris-core/src/main/java/org/apache/polaris/core/storage/aws/AwsCredentialsStorageIntegration.java#L107
But I'm not sure how we can go about getting the user token into this class. From what I can tell, the `AwsCredentislStorageIntegration` is an application scoped class, and we'd need to somehow give it the users request in there to be able to pass in the web token.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [FEATURE REQUEST] - Support S3 STS AssumeRoleWithWebIdentity for on‑prem S3 providers [polaris]
Posted by "dparent1 (via GitHub)" <gi...@apache.org>.
dparent1 commented on issue #3038:
URL: https://github.com/apache/polaris/issues/3038#issuecomment-3558786689
Hi @dimas-b, appreciate your participation here, I'm working with @cccs-cat001 on this. Based on your last response:
Would that not mean that all access to the s3 buckets are essentially done by the service account? I'd like to pass the jwt token provided to polaris to the s3 appliance so it can allow / deny access based on the claims in the jwt token. It would also allow the s3 appliance to log which user is trying to interact with the buckets and the objects in the buckets. Would that be at all possible or would that be a big break from how polaris was designed to function?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [FEATURE REQUEST] - Support S3 STS AssumeRoleWithWebIdentity for on‑prem S3 providers [polaris]
Posted by "xcontreras94 (via GitHub)" <gi...@apache.org>.
xcontreras94 commented on issue #3038:
URL: https://github.com/apache/polaris/issues/3038#issuecomment-4432066578
Hi @dimas-b
> [@michal-kuzak](https://github.com/michal-kuzak) : I wonder if you had a chance to look at [#3170](https://github.com/apache/polaris/pull/3170) ... Does passing the API user's token to STS fit your use cases as far as this issue is concerned? If not, what kind of workflow did you have in mind?
Not quite. We are looking to integrate with FlashBlade which supports STS through federation-based flows like AssumeRoleWithSAML and AssumeRoleWithWebIdentity (OIDC). It seems that due to Polaris hardcoding the AssumeRole path, FlashBlade's STS cannot be used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [FEATURE REQUEST] - Support S3 STS AssumeRoleWithWebIdentity for on‑prem S3 providers [polaris]
Posted by "cccs-cat001 (via GitHub)" <gi...@apache.org>.
cccs-cat001 commented on issue #3038:
URL: https://github.com/apache/polaris/issues/3038#issuecomment-3559465048
I'll write that up :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [FEATURE REQUEST] - Support S3 STS AssumeRoleWithWebIdentity for on‑prem S3 providers [polaris]
Posted by "cccs-cat001 (via GitHub)" <gi...@apache.org>.
cccs-cat001 commented on issue #3038:
URL: https://github.com/apache/polaris/issues/3038#issuecomment-3557921975
I'm not sure I understand, how does that get a token from the user?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [FEATURE REQUEST] - Support S3 STS AssumeRoleWithWebIdentity for on‑prem S3 providers [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on issue #3038:
URL: https://github.com/apache/polaris/issues/3038#issuecomment-4435818587
@xcontreras94 : Thanks for the context! Are you planning to contribute a PR for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [FEATURE REQUEST] - Support S3 STS AssumeRoleWithWebIdentity for on‑prem S3 providers [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on issue #3038:
URL: https://github.com/apache/polaris/issues/3038#issuecomment-3543208967
@michal-kuzak : Do you have capacity to make a PR for this feature? Contributions are welcome 😉
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [FEATURE REQUEST] - Support S3 STS AssumeRoleWithWebIdentity for on‑prem S3 providers [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on issue #3038:
URL: https://github.com/apache/polaris/issues/3038#issuecomment-3554552825
Prior art for getting the initial token is mostly around local files, AFAIK:
https://github.com/apache/polaris/blob/3882c039afb27eecef19118d0890c46cb70d6e84/extensions/auth/opa/impl/src/main/java/org/apache/polaris/extension/auth/opa/token/FileBearerTokenProvider.java#L58
If that works for you @cccs-cat001 , I suppose the class could be moved to a sharable module and reused for `AwsCredentislStorageIntegration`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [FEATURE REQUEST] - Support S3 STS AssumeRoleWithWebIdentity for on‑prem S3 providers [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on issue #3038:
URL: https://github.com/apache/polaris/issues/3038#issuecomment-3554561877
I suppose we could also support implicit (workload identity) authentication too, in case Polaris runs in EC2.
Cross-cloud `AssumeRoleWithWebIdentity` may be a bit too complicated for the initial PR, although it is certainly possible with some extra code. I'd propose to handle that as a follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [FEATURE REQUEST] - Support S3 STS AssumeRoleWithWebIdentity for on‑prem S3 providers [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on issue #3038:
URL: https://github.com/apache/polaris/issues/3038#issuecomment-3617958412
@michal-kuzak : I wonder if you have a chance to look at #3170 ... Does passing the API user's token to STS fit your use cases as far as this issue is concerned? If not, what kind of workflow did you have in mind?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org