You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@polaris.apache.org by "ericyangliu (via GitHub)" <gi...@apache.org> on 2026/06/12 05:10:30 UTC

[I] High volume of AssumeRoleWithWebIdentity calls from Polaris on EKS/IRSA [polaris]

ericyangliu opened a new issue, #4711:
URL: https://github.com/apache/polaris/issues/4711

### Describe the bug

We're running Polaris on EKS with IRSA, using the Kubernetes service account role as the Polaris AWS identity.

While investigating STS usage, we noticed Polaris is making a lot of `AssumeRoleWithWebIdentity` calls for its own IRSA role. The calls are coming from the Java AWS SDK user agent and each call has a new `aws-sdk-java-<timestamp>` session name.

The volume looks higher than expected for normal AWS SDK credential refresh behavior, so it seems like the WebIdentity credential provider/cache may not be getting reused across Polaris requests.

### To Reproduce

1. Run Polaris on EKS with IRSA.
2. Do not configure static AWS credentials for Polaris.
3. Send normal catalog traffic to an S3-backed catalog.
4. Query CloudTrail for `AssumeRoleWithWebIdentity` events on the Polaris service account role.

I don't have a small local reproducer yet; the observations below are from two EKS deployments.

### Actual Behavior

In one deployment, CloudTrail showed this pattern for the Polaris IRSA role:

```text
Polaris version: 1.3.0-incubating
Window: 2026-06-11T18:19:00Z to 2026-06-11T18:20:00Z
Event: AssumeRoleWithWebIdentity

Total events for the Polaris role: 1,261

By user agent:
- aws-sdk-java/2.39.2 ... api/STS#2.39.x ... lang/java#21.0.9: 1,259
- aws-sdk-go-v2 ... app/external-secrets: 2
```

A sanitized event looks like:

```json
{
"eventTime": "2026-06-11T18:19:00Z",
"eventName": "AssumeRoleWithWebIdentity",
"userAgent": "aws-sdk-java/2.39.2 ... api/STS#2.39.x ... lang/java#21.0.9",
"roleArn": "arn:aws:iam::<account-id>:role/<polaris-irsa-role>",
"roleSessionName": "aws-sdk-java-<timestamp>",
"sourceIPAddress": "<redacted>"
}
```

A second deployment showed the same shape at lower volume:

```text
Window: 2026-06-11T18:10:00Z to 2026-06-11T18:20:00Z
Total events for the Polaris role: 434

By user agent:
- aws-sdk-java/2.39.2 ... api/STS#2.39.x ... lang/java#21.0.9: 412
- aws-sdk-go/...: 22
```

### Expected Behavior

Polaris should reuse AWS WebIdentity credentials according to the normal AWS SDK refresh behavior.

I would not expect ordinary catalog traffic to produce hundreds or thousands of `AssumeRoleWithWebIdentity` calls for the Polaris pod role in a short window.

### Additional context

One possible area to check is the service identity path. In 1.3.0-incubating, `AwsIamServiceIdentityConfiguration.awsCredentialsProvider()` builds a `DefaultCredentialsProvider` when static credentials are not configured. If that path runs per request, the AWS SDK's WebIdentity credential cache may not be reused as intended.

### System information

OS: Linux on EKS
Java: 21
Polaris version: 1.3.0-incubating
AWS SDK user agent observed: aws-sdk-java/2.39.2
Storage: AWS S3
Auth setup: EKS IRSA, no static AWS access keys configured for Polaris

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] AssumeRoleWithWebIdentity appears to run per SigV4 AssumeRole request on EKS/IRSA [polaris]

Posted by "harshitaajoshi (via GitHub)" <gi...@apache.org>.

harshitaajoshi commented on issue #4711:
URL: https://github.com/apache/polaris/issues/4711#issuecomment-5877966958

Hi, I'd like to work on this. I traced the path on `main` and can confirm @vigneshio's analysis, with one addition.

**Root cause**
- `SigV4ConnectionCredentialVendor` resolves the service identity credential on every call through the request scoped `DefaultServiceIdentityProvider`, and nothing caches it in between.
- `AwsIamServiceIdentityConfiguration.awsCredentialsProvider()` then builds a new `DefaultCredentialsProvider` each time.
- Under IRSA, each instance creates its own STS client and credential cache for the web identity exchange. So every call triggers a fresh `AssumeRoleWithWebIdentity`, and the per request provider and its STS client are never closed.

**Proposed fix**
- Add an application scoped holder that caches the resolved `AwsIamServiceIdentityCredential` per identity reference. The existing reference URN already includes the realm and identity type.
- Close the cached providers on shutdown.
- No SPI or interface changes. Static key configurations behave as before, and the SDK provider continues to handle IRSA refresh.
- Tests will cover reuse across requests, isolation between realms, static credentials, and closing on shutdown.

**Questions before I start**
1. @XJDKC, you mentioned a local branch for credential caching. Does it cover this base credential reuse, or only caching of the downstream `AssumeRole` session? I'd like to avoid overlapping, and I'm happy to leave session caching to you.
2. The storage path has the same pattern: `StorageConfiguration.stsCredentials()` builds a new `DefaultCredentialsProvider` for each STS call, though `StorageCredentialCache` limits it to cache misses. Should I include it in the same PR or handle it as a follow-up?
3. @ericyangliu, once there is a PR, would you be able to check CloudTrail on a build with the fix?

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] AssumeRoleWithWebIdentity appears to run per SigV4 AssumeRole request on EKS/IRSA [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.

dimas-b commented on issue #4711:
URL: https://github.com/apache/polaris/issues/4711#issuecomment-4746652244

   @vigneshio : your assessment sounds reasonable to me, although I never personally used catalog federation between Polaris and Glue 😅 
   
   Unifying the base AWS credential lookup between the catalog federation and storage access call paths sounds good to me. Ideally the base credential used for Glue connections should be the same as the base credential used for vending storage access credentials to the Polaris client accessing the federated catalog. WDYT?
   
   CC: @XJDKC 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] AssumeRoleWithWebIdentity appears to run per SigV4 AssumeRole request on EKS/IRSA [polaris]

Posted by "adutra (via GitHub)" <gi...@apache.org>.

adutra commented on issue #4711:
URL: https://github.com/apache/polaris/issues/4711#issuecomment-4689391313

   Hi @ericyangliu thanks for reporting this. Did you really intend to close this issue or was that accidental?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] AssumeRoleWithWebIdentity appears to run per SigV4 AssumeRole request on EKS/IRSA [polaris]

Posted by "vigneshio (via GitHub)" <gi...@apache.org>.

vigneshio commented on issue #4711:
URL: https://github.com/apache/polaris/issues/4711#issuecomment-4751425681

   Thanks @XJDKC and @dimas-b
   
   Got it on not sharing the same creds between catalog access and storage. Makes sense because of different roles and the subscoping we do for storage.
   
   I looked at the code againn. For SigV4 catalog federation (like Glue), every time we ask for creds we create a fresh `DefaultCredentialsProvider`. The service identity provider is request scoped, so it doesn't reuse the base `IRSA` creds. That's why we keep seeing new `AssumeRoleWithWebIdentity `calls.
   
   On the storage side we have caching, so the base creds _don't get hit on every request._
   
   You mentioned you have a local branch for making the catalog federation and storage paths behave more the same. Would it be better to look at that or should we first just make the base provider long-lived so the SDK can cache the WebIdentity part..
   
   WDYT ?? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] AssumeRoleWithWebIdentity appears to run per SigV4 AssumeRole request on EKS/IRSA [polaris]

Posted by "ericyangliu (via GitHub)" <gi...@apache.org>.

ericyangliu commented on issue #4711:
URL: https://github.com/apache/polaris/issues/4711#issuecomment-4726350970

   We have just upgraded Polaris to 1.5.0 and the issue still exists. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] AssumeRoleWithWebIdentity appears to run per SigV4 AssumeRole request on EKS/IRSA [polaris]

Posted by "ericyangliu (via GitHub)" <gi...@apache.org>.

ericyangliu commented on issue #4711:
URL: https://github.com/apache/polaris/issues/4711#issuecomment-4693959631

   Hi @adutra ! Thanks for checking. I closed it because I wanted to do more verifications and then reopen it, but I wasn't able to reopen it after closing it... I just checked again, and I was able to do it now. Thank you! 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] AssumeRoleWithWebIdentity appears to run per SigV4 AssumeRole request on EKS/IRSA [polaris]

Posted by "XJDKC (via GitHub)" <gi...@apache.org>.

XJDKC commented on issue #4711:
URL: https://github.com/apache/polaris/issues/4711#issuecomment-4746744897

   1. Catalog Federation doesn't leverage the creds cache for now, if we cache the session credentials, we won't send many sts calls to fetch the creds, I have a local branch for it (unifying the behavior of catalog access and storage access), but hasn't got a chance to clean it up
   2. We should not use same credential for accessing the catalog and storage, several reasons:
       * End users may use different role for catalog access
       * When accessing the storage, we subscope the creds to table level based on the RBAC rules, this credential only has access to the storage.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] AssumeRoleWithWebIdentity appears to run per SigV4 AssumeRole request on EKS/IRSA [polaris]

Posted by "vigneshio (via GitHub)" <gi...@apache.org>.

vigneshio commented on issue #4711:
URL: https://github.com/apache/polaris/issues/4711#issuecomment-4729482522

I reviewed the code and the main concern seems valid.

What I found:

1. `AwsIamServiceIdentityConfiguration.awsCredentialsProvider()` creates a new `DefaultCredentialsProvider` every time when static AWS credentials are not configured.

2. This provider is used through `asServiceIdentityCredential()`, which is called by `DefaultServiceIdentityProvider.getServiceIdentityCredential()`.

3. `SigV4ConnectionCredentialVendor` requests credentials on every `getConnectionCredentials()` call, and there is no caching in this path.

4. Since AWS SDK v2 caches WebIdentity credentials per provider instance, creating a new provider each time can trigger a fresh `AssumeRoleWithWebIdentity` call before the downstream `AssumeRole`. This matches the different `aws-sdk-java-<timestamp>` session names seen in CloudTrail.

One thing worth noting: the storage credential path also creates a new `DefaultCredentialsProvider` through `stsCredentials()`. However, `AwsCredentialsStorageIntegration` caches the assumed-role credentials in `StorageCredentialCache`, so the base identity is not resolved for every request. The SigV4 connection credential path does not have similar caching, which is likely why the repeated WebIdentity calls are showing up there.

From what I can see, this behavior still exists on the current `main` branch.

A possible fix would be to create a single long-lived base credentials provider and reuse it instead of creating a new one for every credential resolution.

Does that sound like the right approach @dimas-b @adutra ?? or is there any other way... WDYT?

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] High volume of AssumeRoleWithWebIdentity calls from Polaris on EKS/IRSA [polaris]

Posted by "ericyangliu (via GitHub)" <gi...@apache.org>.

ericyangliu closed issue #4711: High volume of AssumeRoleWithWebIdentity calls from Polaris on EKS/IRSA
URL: https://github.com/apache/polaris/issues/4711


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] AssumeRoleWithWebIdentity appears to run per SigV4 AssumeRole request on EKS/IRSA [polaris]

Posted by "dimas-b (via GitHub)" <gi...@apache.org>.

dimas-b commented on issue #4711:
URL: https://github.com/apache/polaris/issues/4711#issuecomment-4753266586

   @XJDKC : 
   
   > End users may use different role for catalog access [...]
   
   With Apache Polaris binaries I do not see how that is possible. AFAIK, the base S3/STS credential is loaded from the environment, so it must be the same for federated catalog and direct storage access. WDYT?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] AssumeRoleWithWebIdentity appears to run per SigV4 AssumeRole request on EKS/IRSA [polaris]

Posted by "ericyangliu (via GitHub)" <gi...@apache.org>.

ericyangliu commented on issue #4711:
URL: https://github.com/apache/polaris/issues/4711#issuecomment-4693937278

   Hi @adutra! Thanks for checking. I closed 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] AssumeRoleWithWebIdentity appears to run per SigV4 AssumeRole request on EKS/IRSA [polaris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.

github-actions[bot] commented on issue #4711:
URL: https://github.com/apache/polaris/issues/4711#issuecomment-5029449213

   This issue is stale because it has been open 30 days with no activity. Remove stale label or comment if you think it's still relevant.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] AssumeRoleWithWebIdentity appears to run per SigV4 AssumeRole request on EKS/IRSA [polaris]

Posted by "harshitaajoshi (via GitHub)" <gi...@apache.org>.

harshitaajoshi commented on issue #4711:
URL: https://github.com/apache/polaris/issues/4711#issuecomment-6403308663

   Since there was no overlap concern raised, I went ahead with the base credential reuse in #5869. It caches the resolved service identity credential per realm and identity type in an application scoped bean and closes the providers on shutdown, so IRSA credentials are resolved once and refreshed by the SDK instead of on every request.
   
   I kept the storage path and caching of the downstream `AssumeRole` session out of scope so this stays small. Happy to pick either up as a follow up if that's useful.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org