You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@polaris.apache.org by "pingtimeout (via GitHub)" <gi...@apache.org> on 2025/12/09 17:22:52 UTC
[I] Polaris fails to create a Table because of an STS policy size limit [polaris]
pingtimeout opened a new issue, #3243:
URL: https://github.com/apache/polaris/issues/3243
### Describe the bug
While running benchmarks with much longer namespace names, I got the following error while creating a table:
```
Failed to get subscoped credentials: Packed policy consumes 118% of allotted space, please use smaller policy.
```
The path that caused this issue is composed by 14 nested namespaces, each with a 32 characters name. The problem is located in https://github.com/apache/polaris/blob/31c7784630889d8fb6f0156610b8e59a4021f1c3/polaris-core/src/main/java/org/apache/polaris/core/storage/aws/AwsCredentialsStorageIntegration.java#L175. It results in the following STS policy:
```
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:DeleteObject"
],
"Resource": "arn:aws:s3:::my-very-long-s3-bucket/catalog/f5a4d86558ed1f7fddec42ce11d8ee3a/e3fa7fd44796b949fce728cb334f3c15/f031eb9dc709150a3e1e9a76e9af550a/5598924229adee97260cda483d70674c/0857633d48470f538d1fc4cdc789c753/c4c06b9b673a710f7ce865690ff8797b/9594c7f9e8a1cd28054ff71b933fdc3b/0018c223e978d13aeec87488bc333c70/d4c484ca41745b69286372286f3f30aa/362f8d6579892af8a9c5f38b4e664b07/f28f3a8b846271c68bdbc09992d88d28/915107073e394e3471173ec4633137fa/5eac9dac6dd84afcb35f4805af9d8b34/e1a24630cbf9ced9dffcc123b70b2e43/ab4ffa55f688360e0c12aef543c18351/*"
},
{
"Effect": "Allow",
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::my-very-long-s3-bucket",
"Condition": {
"StringLike": {
"s3:prefix": "catalog/f5a4d86558ed1f7fddec42ce11d8ee3a/e3fa7fd44796b949fce728cb334f3c15/f031eb9dc709150a3e1e9a76e9af550a/5598924229adee97260cda483d70674c/0857633d48470f538d1fc4cdc789c753/c4c06b9b673a710f7ce865690ff8797b/9594c7f9e8a1cd28054ff71b933fdc3b/0018c223e978d13aeec87488bc333c70/d4c484ca41745b69286372286f3f30aa/362f8d6579892af8a9c5f38b4e664b07/f28f3a8b846271c68bdbc09992d88d28/915107073e394e3471173ec4633137fa/5eac9dac6dd84afcb35f4805af9d8b34/e1a24630cbf9ced9dffcc123b70b2e43/ab4ffa55f688360e0c12aef543c18351/*"
}
}
},
{
"Effect": "Allow",
"Action": "s3:GetBucketLocation",
"Resource": "arn:aws:s3:::my-very-long-s3-bucket"
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:GetObjectVersion"
],
"Resource": "arn:aws:s3:::my-very-long-s3-bucket/catalog/f5a4d86558ed1f7fddec42ce11d8ee3a/e3fa7fd44796b949fce728cb334f3c15/f031eb9dc709150a3e1e9a76e9af550a/5598924229adee97260cda483d70674c/0857633d48470f538d1fc4cdc789c753/c4c06b9b673a710f7ce865690ff8797b/9594c7f9e8a1cd28054ff71b933fdc3b/0018c223e978d13aeec87488bc333c70/d4c484ca41745b69286372286f3f30aa/362f8d6579892af8a9c5f38b4e664b07/f28f3a8b846271c68bdbc09992d88d28/915107073e394e3471173ec4633137fa/5eac9dac6dd84afcb35f4805af9d8b34/e1a24630cbf9ced9dffcc123b70b2e43/ab4ffa55f688360e0c12aef543c18351/*"
}
]
}
```
I don't think we can eliminate the STS limitation at all. But I think Polaris could be improved to better deal with this error. For example:
* The returned HTTP code (400) is correct (Bad Request) but the associated error message should be clearer for the user. The current one may not be understandable for everyone.
* During namespace creation, no check is performed on the total length of the path. It might be worth adding so that the failure happens earlier.
* Parts of the STS policy could be merged together (AFAICT): the `s3:PutObject` and `s3:DeleteObject` statements could be merged with the `s3:GetObject` and `s3:GetObjectVersion` statements. This would allow Polaris to support longer paths, while not removing the maximum path limitation entirely though.
### To Reproduce
* Create a deeply nested namespace structure with long length (e.g. `f5a4d86558ed1f7fddec42ce11d8ee3a/e3fa7fd44796b949fce728cb334f3c15/f031eb9dc709150a3e1e9a76e9af550a/5598924229adee97260cda483d70674c/0857633d48470f538d1fc4cdc789c753/c4c06b9b673a710f7ce865690ff8797b/9594c7f9e8a1cd28054ff71b933fdc3b/0018c223e978d13aeec87488bc333c70/d4c484ca41745b69286372286f3f30aa/362f8d6579892af8a9c5f38b4e664b07/f28f3a8b846271c68bdbc09992d88d28/915107073e394e3471173ec4633137fa/5eac9dac6dd84afcb35f4805af9d8b34/e1a24630cbf9ced9dffcc123b70b2e43/ab4ffa55f688360e0c12aef543c18351`)
* Create a table in the last namespace
### Actual Behavior
_No response_
### Expected Behavior
_No response_
### Additional context
_No response_
### System information
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Polaris fails to create a Table because of an STS policy size limit [polaris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #3243:
URL: https://github.com/apache/polaris/issues/3243#issuecomment-4020654540
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment if you think it's still relevant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Polaris fails to create a Table because of an STS policy size limit [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on issue #3243:
URL: https://github.com/apache/polaris/issues/3243#issuecomment-4390073047
Ultimately, the inline STS policy side is affected by user inputs (such as namespace and table names), which Polaris cannot control.
I believe #4228 improve this situation a bit, yet is it still worth considering improvements suggested by @pingtimeout in this issue's description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Polaris fails to create a Table because of an STS policy size limit [polaris]
Posted by "rishii-19-works (via GitHub)" <gi...@apache.org>.
rishii-19-works commented on issue #3243:
URL: https://github.com/apache/polaris/issues/3243#issuecomment-3671153052
Hi @pingtimeout thanks for raising this issue.
I’ve gone through the flow and understand that the failure occurs because the STS policy generated by Polaris becomes too large when namespaces are deeply nested, eventually exceeding AWS policy size limits during table creation.
I’d like to take this up and work on a fix. My current thinking is to reduce the size of the generated policy (for example, by consolidating policy statements) and/or add early validation so the failure is caught with a clearer, Polaris-level error instead of an AWS exception.
Please let me know if there’s a preferred direction or any design constraints I should be aware of before I start implementing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Polaris fails to create a Table because of an STS policy size limit [polaris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #3243:
URL: https://github.com/apache/polaris/issues/3243#issuecomment-3857490864
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Polaris fails to create a Table because of an STS policy size limit [polaris]
Posted by "pingtimeout (via GitHub)" <gi...@apache.org>.
pingtimeout commented on issue #3243:
URL: https://github.com/apache/polaris/issues/3243#issuecomment-3671415635
Thanks @rishii-19-works , just in case you want a repro, you can checkout the branch from https://github.com/apache/polaris-tools/pull/107 and use a configuration with a depth≥12 and width=1, it reproduces the issue within seconds from an empty polaris server
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Polaris fails to create a Table because of an STS policy size limit [polaris]
Posted by "rishii-19-works (via GitHub)" <gi...@apache.org>.
rishii-19-works commented on issue #3243:
URL: https://github.com/apache/polaris/issues/3243#issuecomment-3693764986
Thanks for sharing the repro, that’s very helpful. I’ll use this setup to reproduce the issue locally and validate the fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org