You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@polaris.apache.org by "fivetran-ashokborra (via GitHub)" <gi...@apache.org> on 2026/09/24 05:08:19 UTC

[PR] Support cross-account KMS keys in credential policies [polaris]

fivetran-ashokborra opened a new pull request, #5598:
URL: https://github.com/apache/polaris/pull/5598

   ## Context
   
   When AWS credential vending is used with an S3 bucket whose SSE-KMS key belongs to a different AWS account, the generated STS session policy restricts KMS access to keys in the S3 bucket’s account. This prevents valid cross-account KMS configurations from working.
   
   ## Change
   
   Add the default-disabled `ALLOW_CROSS_ACCOUNT_KMS_KEYS` realm feature flag.
   
   When enabled, read-only credential policies use:
   
   `arn:aws:kms:<region>:*:key/*`
   
   When disabled, the existing account-scoped KMS ARN remains unchanged.
   
   ## Testing
   
   - Added coverage for the feature flag enabled and disabled states.
   - Ran `./gradlew spotlessApply`
   - Ran `./gradlew :polaris-core:test --tests "org.apache.polaris.core.storage.aws.AwsCredentialsStorageIntegrationTest"`
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support cross-account KMS keys in credential policies [polaris]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #5598:
URL: https://github.com/apache/polaris/pull/5598#discussion_r4090157381


##########
polaris-core/src/main/java/org/apache/polaris/core/config/FeatureConfiguration.java:
##########
@@ -89,6 +89,14 @@ public static void enforceFeatureEnabledOrThrow(
           .defaultValue(false)
           .buildFeatureConfiguration();
 
+  public static final FeatureConfiguration<Boolean> ALLOW_CROSS_ACCOUNT_KMS_KEYS =
+      PolarisConfiguration.<Boolean>builder()
+          .key("ALLOW_CROSS_ACCOUNT_KMS_KEYS")
+          .description(
+              "If set to true, allow AWS credential vending for S3 buckets whose SSE-KMS keys are owned by a different account.")

Review Comment:
   This adds a stable user-facing flag and, when enabled, broadens every read-only S3 session policy from the bucket account to all KMS keys in the region. Please add the corresponding entry to `site/content/in-dev/unreleased/configuration/config-sections/flags-polaris_features.md` and `CHANGELOG.md`, including the default and the security impact of granting `kms:Decrypt`/`kms:DescribeKey` against a cross-account wildcard; this is an important operator-facing trust-boundary change.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support cross-account KMS decryption for S3 [polaris]

Posted by "fivetran-ashokborra (via GitHub) via issues" <is...@polaris.apache.org>.
fivetran-ashokborra commented on PR #5598:
URL: https://github.com/apache/polaris/pull/5598#issuecomment-7010818774

   > Sorry about the delayed review, @fivetran-ashokborra . The feature looks reasonable to me.
   > 
   > Could you respond to comments?
   
   I was away. Sorry about that. Checking the comments


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support cross-account KMS decryption for S3 [polaris]

Posted by "fivetran-ashokborra (via GitHub)" <gi...@apache.org>.
fivetran-ashokborra commented on code in PR #5598:
URL: https://github.com/apache/polaris/pull/5598#discussion_r4130064138


##########
polaris-core/src/main/java/org/apache/polaris/core/storage/aws/AwsCredentialsStorageIntegration.java:
##########
@@ -521,8 +525,14 @@ private static void addKmsKeyResources(
   }
 
   private static void addAllKeysResource(
-      String region, String accountId, IamStatement.Builder allowKms) {
-    String allKeysArn = arnKeyAll(region, accountId);
+      String region,
+      String accountId,
+      boolean allowCrossAccountKmsKeys,
+      IamStatement.Builder allowKms) {
+    String allKeysArn =
+        allowCrossAccountKmsKeys
+            ? String.format("arn:aws:kms:%s:*:key/*", region)

Review Comment:
   Thanks @flyingImer for the comment. Addressed 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support cross-account KMS decryption for S3 [polaris]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #5598:
URL: https://github.com/apache/polaris/pull/5598#discussion_r4090980321


##########
polaris-core/src/main/java/org/apache/polaris/core/storage/aws/AwsCredentialsStorageIntegration.java:
##########
@@ -521,8 +525,14 @@ private static void addKmsKeyResources(
   }
 
   private static void addAllKeysResource(
-      String region, String accountId, IamStatement.Builder allowKms) {
-    String allKeysArn = arnKeyAll(region, accountId);
+      String region,
+      String accountId,
+      boolean allowCrossAccountKmsKeys,
+      IamStatement.Builder allowKms) {
+    String allKeysArn =
+        allowCrossAccountKmsKeys
+            ? String.format("arn:aws:kms:%s:*:key/*", region)

Review Comment:
   The new cross-account branch hardcodes the commercial `aws` partition, but this integration supports `aws-cn` and `aws-us-gov` (and derives the partition from the role ARN). With the flag enabled in those partitions, the session policy contains an `arn:aws` KMS resource that cannot match keys in the configured partition, so decryption still fails. Please pass the storage partition into this helper and format `arn:<partition>:kms:<region>:*:key/*`, with coverage for the supported partitions.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support cross-account KMS decryption for S3 [polaris]

Posted by "dimas-b (via GitHub) via issues" <is...@polaris.apache.org>.
dimas-b commented on code in PR #5598:
URL: https://github.com/apache/polaris/pull/5598#discussion_r4429500225


##########
polaris-core/src/main/java/org/apache/polaris/core/storage/aws/AwsCredentialsStorageIntegration.java:
##########
@@ -229,7 +229,8 @@ static StorageAccessConfig compute(AwsStorageCredentialCacheKey key) {
                           key.allowedReadLocations(),
                           key.allowedListLocations(),
                           key.allowedWriteLocations(),
-                          region)
+                          region,
+                          realmConfig.getConfig(FeatureConfiguration.ALLOW_CROSS_ACCOUNT_KMS_KEYS))

Review Comment:
   Good point about per-catalog config. @fivetran-ashokborra WDYT?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support cross-account KMS decryption for S3 [polaris]

Posted by "orca-review-bot (via GitHub)" <gi...@apache.org>.
orca-review-bot commented on code in PR #5598:
URL: https://github.com/apache/polaris/pull/5598#discussion_r4420809405


##########
polaris-core/src/main/java/org/apache/polaris/core/storage/aws/AwsCredentialsStorageIntegration.java:
##########
@@ -229,7 +229,8 @@ static StorageAccessConfig compute(AwsStorageCredentialCacheKey key) {
                           key.allowedReadLocations(),
                           key.allowedListLocations(),
                           key.allowedWriteLocations(),
-                          region)
+                          region,
+                          realmConfig.getConfig(FeatureConfiguration.ALLOW_CROSS_ACCOUNT_KMS_KEYS))

Review Comment:
   This is realm-wide, so turning it on widens the KMS wildcard for every S3 catalog in the realm, even if only one bucket uses a cross-account key. `RealmConfig` already supports `getConfig(flag, catalogProperties)`. Could the flag get a `catalogConfig` key so it can be scoped per catalog, or is realm-only intentional?
   
   <sub>🤖 AI-generated review comment — agents make mistakes, so please verify before acting on it.</sub>



##########
CHANGELOG.md:
##########
@@ -113,6 +113,7 @@ request adding CHANGELOG notes for breaking (!) changes and possibly other secti
 - The database schema used by the Relational JDBC persistence backend is now configurable through standard datasource configuration: the JDBC driver's `currentSchema` connection property (defaulted to `POLARIS_SCHEMA` via `quarkus.datasource.jdbc.additional-jdbc-properties.currentSchema`) selects the schema, and the persistence layer is agnostic of the schema name. Also exposed as `persistence.relationalJdbc.additionalProperties.currentSchema` in the Helm chart.
 - Python CLI: `catalogs create` and `catalogs update` now support `--storage-name` to set an optional name referencing a server-side storage configuration.
 - Python CLI: added `register` support for both `tables` and `views` commands
+- Added the default-disabled `ALLOW_CROSS_ACCOUNT_KMS_KEYS` feature flag for AWS credential vending. When enabled, read-only policies may reference SSE-KMS keys owned by another AWS account in the same region for decryption. This broadens `kms:Decrypt` and `kms:DescribeKey` access to a cross-account regional wildcard, so operators should enable it only with intentional cross-account KMS permissions and trust relationships.

Review Comment:
   Can we say here that the flag only applies when no `encryptionKeys`/`decryptionKeys` are configured, and that listing the specific cross-account key in `decryptionKeys` is the narrower option? Otherwise operators may reach for the regional wildcard first.
   
   <sub>🤖 AI-generated review comment — agents make mistakes, so please verify before acting on it.</sub>



##########
polaris-core/src/test/java/org/apache/polaris/core/storage/aws/AwsCredentialsStorageIntegrationTest.java:
##########
@@ -1245,6 +1246,89 @@ public void testKmsKeyPolicyLogic() {
             CredentialVendingContext.empty());
   }
 
+  @ParameterizedTest
+  @CsvSource({
+    "aws, us-east-2, false",
+    "aws, us-east-2, true",
+    "aws-cn, cn-north-1, false",
+    "aws-cn, cn-north-1, true",
+    "aws-us-gov, us-gov-west-1, false",
+    "aws-us-gov, us-gov-west-1, true"
+  })
+  public void testCrossAccountKmsFeatureFlag(String partition, String region, boolean enabled) {
+    StsClient stsClient = Mockito.mock(StsClient.class);
+    String accountId = "012345678901";
+    String bucket = "bucket";
+    String location = s3Path(bucket, "path/to/warehouse/table");
+    Mockito.when(stsClient.assumeRole(Mockito.isA(AssumeRoleRequest.class)))
+        .thenAnswer(
+            invocation -> {
+              IamPolicy policy =
+                  IamPolicy.fromJson(invocation.<AssumeRoleRequest>getArgument(0).policy());
+              assertThat(policy.statements())
+                  .anySatisfy(
+                      statement ->
+                          assertThat(statement.resources())
+                              .containsExactly(
+                                  IamResource.create(
+                                      String.format(
+                                          "arn:%s:kms:%s:%s:key/*",
+                                          partition, region, enabled ? "*" : accountId))));
+              return ASSUME_ROLE_RESPONSE;
+            });
+
+    AwsStorageConfigurationInfo config =
+        AwsStorageConfigurationInfo.builder()
+            .addAllowedLocation(location)
+            .roleARN(String.format("arn:%s:iam::%s:role/jdoe", partition, accountId))
+            .externalId("externalId")
+            .region(region)
+            .build();
+    RealmConfig realmConfig =
+        enabled ? enabledFeatures(ALLOW_CROSS_ACCOUNT_KMS_KEYS) : EMPTY_REALM_CONFIG;
+
+    new AwsCredentialsStorageIntegration(stsClient, config, realmConfig)
+        .getStorageAccessConfig(
+            toGrants(Set.of(location), Set.of(location), Set.of()),
+            Optional.empty(),
+            CredentialVendingContext.empty());
+  }
+
+  @Test
+  public void testCrossAccountKmsFeatureFlagDoesNotAffectWritePolicy() {
+    StsClient stsClient = Mockito.mock(StsClient.class);
+    String location = s3Path("bucket", "path/to/warehouse/table");
+    Mockito.when(stsClient.assumeRole(Mockito.isA(AssumeRoleRequest.class)))
+        .thenAnswer(
+            invocation -> {
+              IamPolicy policy =
+                  IamPolicy.fromJson(invocation.<AssumeRoleRequest>getArgument(0).policy());
+              assertThat(policy.statements())
+                  .noneSatisfy(
+                      statement ->
+                          assertThat(statement.resources())
+                              .anySatisfy(
+                                  resource ->
+                                      assertThat(resource.value()).startsWith("arn:aws:kms:")));

Review Comment:
   This only rules out `arn:aws:kms:` resources. Can we assert no `:kms:` resource at all, so the test doesn't depend on the partition?
   
   <sub>🤖 AI-generated review comment — agents make mistakes, so please verify before acting on it.</sub>



##########
polaris-core/src/main/java/org/apache/polaris/core/config/FeatureConfiguration.java:
##########
@@ -89,6 +89,16 @@ public static void enforceFeatureEnabledOrThrow(
           .defaultValue(false)
           .buildFeatureConfiguration();
 
+  public static final FeatureConfiguration<Boolean> ALLOW_CROSS_ACCOUNT_KMS_KEYS =
+      PolarisConfiguration.<Boolean>builder()
+          .key("ALLOW_CROSS_ACCOUNT_KMS_KEYS")
+          .description(
+              "If set to true, read-only AWS credential vending policies may reference SSE-KMS keys owned by a different AWS account in the same region. "

Review Comment:
   Same as the CHANGELOG: can we mention in the description that the flag only affects the no-keys fallback and that `decryptionKeys` is the scoped alternative?
   
   <sub>🤖 AI-generated review comment — agents make mistakes, so please verify before acting on it.</sub>



##########
polaris-core/src/main/java/org/apache/polaris/core/storage/aws/AwsCredentialsStorageIntegration.java:
##########
@@ -521,14 +527,19 @@ private static void addKmsKeyResources(
   }
 
   private static void addAllKeysResource(
-      String region, String accountId, IamStatement.Builder allowKms) {
-    String allKeysArn = arnKeyAll(region, accountId);
+      String awsPartition,
+      String region,
+      String accountId,
+      boolean allowCrossAccountKmsKeys,
+      IamStatement.Builder allowKms) {
+    String allKeysArn = arnKeyAll(awsPartition, region, allowCrossAccountKmsKeys ? "*" : accountId);
     allowKms.addResource(IamResource.create(allKeysArn));
     LOGGER.debug("Adding KMS key policy for all keys in account {}", accountId);
   }
 
-  private static String arnKeyAll(String region, String accountId) {
-    return String.format("arn:aws:kms:%s:%s:key/*", region, accountId);
+  private static String arnKeyAll(String awsPartition, String region, String accountId) {
+    return String.format(
+        "arn:%s:kms:%s:%s:key/*", awsPartition != null ? awsPartition : "aws", region, accountId);

Review Comment:
   This partition fix changes the generated policy for aws-cn and aws-us-gov even with the flag off. Those used to get `arn:aws:kms:...`, which would never match. It's a good fix, but can we give it its own line in the CHANGELOG so it isn't hidden behind the flag entry?
   
   <sub>🤖 AI-generated review comment — agents make mistakes, so please verify before acting on it.</sub>



##########
polaris-core/src/main/java/org/apache/polaris/core/storage/aws/AwsCredentialsStorageIntegration.java:
##########
@@ -521,14 +527,19 @@ private static void addKmsKeyResources(
   }
 
   private static void addAllKeysResource(
-      String region, String accountId, IamStatement.Builder allowKms) {
-    String allKeysArn = arnKeyAll(region, accountId);
+      String awsPartition,
+      String region,
+      String accountId,
+      boolean allowCrossAccountKmsKeys,
+      IamStatement.Builder allowKms) {
+    String allKeysArn = arnKeyAll(awsPartition, region, allowCrossAccountKmsKeys ? "*" : accountId);
     allowKms.addResource(IamResource.create(allKeysArn));
     LOGGER.debug("Adding KMS key policy for all keys in account {}", accountId);

Review Comment:
   With the flag on, this still says "all keys in account <bucket account>" even though the resource is `*`. Can we log `allKeysArn` instead?
   
   <sub>🤖 AI-generated review comment — agents make mistakes, so please verify before acting on it.</sub>



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support cross-account KMS decryption for S3 [polaris]

Posted by "flyingImer (via GitHub)" <gi...@apache.org>.
flyingImer commented on code in PR #5598:
URL: https://github.com/apache/polaris/pull/5598#discussion_r4128163185


##########
polaris-core/src/main/java/org/apache/polaris/core/storage/aws/AwsCredentialsStorageIntegration.java:
##########
@@ -521,8 +525,14 @@ private static void addKmsKeyResources(
   }
 
   private static void addAllKeysResource(
-      String region, String accountId, IamStatement.Builder allowKms) {
-    String allKeysArn = arnKeyAll(region, accountId);
+      String region,
+      String accountId,
+      boolean allowCrossAccountKmsKeys,
+      IamStatement.Builder allowKms) {
+    String allKeysArn =
+        allowCrossAccountKmsKeys
+            ? String.format("arn:aws:kms:%s:*:key/*", region)

Review Comment:
   Agreed on this partition issue. The existing account-scoped helper also hardcodes `aws`, so this is an inherited limitation, but the new opt-in still won't work for GovCloud or China because its KMS resource cannot match keys in those partitions. Could we use the partition already extracted from the role ARN and cover the generated policy for `aws`, `aws-cn`, and `aws-us-gov` before merging?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support cross-account KMS decryption for S3 [polaris]

Posted by "dimas-b (via GitHub) via issues" <is...@polaris.apache.org>.
dimas-b commented on code in PR #5598:
URL: https://github.com/apache/polaris/pull/5598#discussion_r4429500225


##########
polaris-core/src/main/java/org/apache/polaris/core/storage/aws/AwsCredentialsStorageIntegration.java:
##########
@@ -229,7 +229,8 @@ static StorageAccessConfig compute(AwsStorageCredentialCacheKey key) {
                           key.allowedReadLocations(),
                           key.allowedListLocations(),
                           key.allowedWriteLocations(),
-                          region)
+                          region,
+                          realmConfig.getConfig(FeatureConfiguration.ALLOW_CROSS_ACCOUNT_KMS_KEYS))

Review Comment:
   Good point about per-catalog config. @fivetran-ashokborra WDYT?
   
   I might be preferable to add it to `AwsStorageConfigurationInfo` rather than a feature flag.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support cross-account KMS decryption for S3 [polaris]

Posted by "adutra (via GitHub) via issues" <is...@polaris.apache.org>.
adutra commented on PR #5598:
URL: https://github.com/apache/polaris/pull/5598#issuecomment-6943675463

   Heads-up: #5967 has been merged, so `CHANGELOG.md` is no longer edited directly. Changelog entries now go in a fragment file under `changelog.d/`, which avoids merge conflicts on `CHANGELOG.md`. See [changelog.d/README.md](https://github.com/apache/polaris/blob/main/changelog.d/README.md) for the format.
   
   Could you rebase onto `main`, revert your changes to `CHANGELOG.md`, and add a fragment file instead if your change needs a changelog entry? A fragment is only needed for things worth calling out in release notes, such as breaking changes, metastore schema changes, or changes that require manual steps during an upgrade. Thanks!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org