You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@polaris.apache.org by "adutra (via GitHub)" <gi...@apache.org> on 2025/08/06 17:26:52 UTC
[PR] Add support for S3 request signing [polaris]
adutra opened a new pull request, #2280:
URL: https://github.com/apache/polaris/pull/2280
WIP
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2503704682
##########
runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java:
##########
@@ -801,6 +806,33 @@ private LoadTableResponse.Builder buildLoadTableResponseWithDelegationCredential
return responseBuilder;
}
+ AccessDelegationMode delegationMode = selectAccessDelegationMode(delegationModes);
+
+ if (delegationMode == REMOTE_SIGNING) {
+
+ S3RemoteSigningCatalogHandler.throwIfRemoteSigningNotEnabled(
+ callContext.getRealmConfig(), getResolvedCatalogEntity());
+
+ AccessConfig accessConfig =
+ accessConfigProvider.getAccessConfigForRemoteSigning(
+ callContext, catalogName, tableIdentifier, resolvedStoragePath);
+
+ Map<String, String> credentialConfig = accessConfig.credentials();
+
+ if (!credentialConfig.isEmpty()) {
+ responseBuilder.addAllConfig(credentialConfig);
+ responseBuilder.addCredential(
Review Comment:
Good catch, this doesn't make sense 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2263164640
##########
runtime/service/src/main/java/org/apache/polaris/service/storage/aws/signer/S3RemoteSigningCatalogHandler.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.service.storage.aws.signer;
+
+import jakarta.ws.rs.core.SecurityContext;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.apache.polaris.core.auth.PolarisAuthorizableOperation;
+import org.apache.polaris.core.auth.PolarisAuthorizer;
+import org.apache.polaris.core.config.FeatureConfiguration;
+import org.apache.polaris.core.config.RealmConfig;
+import org.apache.polaris.core.context.CallContext;
+import org.apache.polaris.core.entity.CatalogEntity;
+import org.apache.polaris.core.entity.PolarisEntitySubType;
+import org.apache.polaris.core.persistence.resolver.ResolutionManifestFactory;
+import org.apache.polaris.service.aws.sign.model.PolarisS3SignRequest;
+import org.apache.polaris.service.aws.sign.model.PolarisS3SignResponse;
+import org.apache.polaris.service.catalog.common.CatalogHandler;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class S3RemoteSigningCatalogHandler extends CatalogHandler implements AutoCloseable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(S3RemoteSigningCatalogHandler.class);
+
+ private final S3RequestSigner s3RequestSigner;
+
+ private CatalogEntity catalogEntity;
+
+ public S3RemoteSigningCatalogHandler(
+ CallContext callContext,
+ ResolutionManifestFactory resolutionManifestFactory,
+ SecurityContext securityContext,
+ String catalogName,
+ PolarisAuthorizer authorizer,
+ S3RequestSigner s3RequestSigner) {
+ super(callContext, resolutionManifestFactory, securityContext, catalogName, authorizer);
+ this.s3RequestSigner = s3RequestSigner;
+ }
+
+ @Override
+ protected void initializeCatalog() {
+ catalogEntity =
+ CatalogEntity.of(resolutionManifest.getResolvedReferenceCatalogEntity().getRawLeafEntity());
+ if (catalogEntity.isExternal()) {
+ throw new ForbiddenException("Cannot use S3 remote signing with federated catalogs.");
+ }
+ // no need to materialize the catalog here, as we only need the catalog entity
+ }
+
+ public PolarisS3SignResponse signS3Request(
+ PolarisS3SignRequest s3SignRequest, TableIdentifier tableIdentifier) {
+
+ LOGGER.debug("Requesting s3 signing for {}: {}", tableIdentifier, s3SignRequest);
+
+ // TODO authorize based on the request's method?
+
+ try {
+ authorizeBasicTableLikeOperationOrThrow(
Review Comment:
The RBAC checks are admittedly very coarse-grained for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2263140745
##########
integration-tests/src/main/java/org/apache/polaris/service/it/test/PolarisRestCatalogIntegrationBase.java:
##########
@@ -146,11 +148,13 @@ public abstract class PolarisRestCatalogIntegrationBase extends CatalogTests<RES
private RESTCatalog restCatalog;
private String currentCatalogName;
private Map<String, String> restCatalogConfig;
- private URI externalCatalogBase;
+ private URI externalCatalogBaseLocation;
Review Comment:
FYI I refactored this class a bit, notably to facilitate subclassing, but there is no functional change in this class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2284671753
##########
runtime/service/src/main/java/org/apache/polaris/service/storage/aws/signer/S3RemoteSigningCatalogHandler.java:
##########
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.service.storage.aws.signer;
+
+import jakarta.ws.rs.core.SecurityContext;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.apache.polaris.core.auth.PolarisAuthorizableOperation;
+import org.apache.polaris.core.auth.PolarisAuthorizer;
+import org.apache.polaris.core.config.FeatureConfiguration;
+import org.apache.polaris.core.config.RealmConfig;
+import org.apache.polaris.core.context.CallContext;
+import org.apache.polaris.core.entity.CatalogEntity;
+import org.apache.polaris.core.entity.PolarisEntitySubType;
+import org.apache.polaris.core.persistence.resolver.ResolutionManifestFactory;
+import org.apache.polaris.service.aws.sign.model.PolarisS3SignRequest;
+import org.apache.polaris.service.aws.sign.model.PolarisS3SignResponse;
+import org.apache.polaris.service.catalog.common.CatalogHandler;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class S3RemoteSigningCatalogHandler extends CatalogHandler implements AutoCloseable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(S3RemoteSigningCatalogHandler.class);
+
+ private final S3RequestSigner s3RequestSigner;
+
+ private CatalogEntity catalogEntity;
+
+ public S3RemoteSigningCatalogHandler(
+ CallContext callContext,
+ ResolutionManifestFactory resolutionManifestFactory,
+ SecurityContext securityContext,
+ String catalogName,
+ PolarisAuthorizer authorizer,
+ S3RequestSigner s3RequestSigner) {
+ super(callContext, resolutionManifestFactory, securityContext, catalogName, authorizer);
+ this.s3RequestSigner = s3RequestSigner;
+ }
+
+ @Override
+ protected void initializeCatalog() {
+ catalogEntity =
+ CatalogEntity.of(resolutionManifest.getResolvedReferenceCatalogEntity().getRawLeafEntity());
+ if (catalogEntity.isExternal()) {
+ throw new ForbiddenException("Cannot use S3 remote signing with federated catalogs.");
+ }
+ // no need to materialize the catalog here, as we only need the catalog entity
+ }
+
+ public PolarisS3SignResponse signS3Request(
+ PolarisS3SignRequest s3SignRequest, TableIdentifier tableIdentifier) {
+
+ LOGGER.debug("Requesting s3 signing for {}: {}", tableIdentifier, s3SignRequest);
+
+ // TODO authorize based on the request's method?
Review Comment:
See my previous comment. For a `DELETE` method you now need `TABLE_REMOTE_SIGN` + `TABLE_WRITE_DATA`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2260479484
##########
runtime/service/src/main/java/org/apache/polaris/service/storage/aws/signer/S3RequestSignerImpl.java:
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.service.storage.aws.signer;
+
+import java.net.URI;
+import java.util.Set;
+import org.apache.iceberg.aws.s3.signer.ImmutableS3SignResponse;
+import org.apache.iceberg.aws.s3.signer.S3SignResponse;
+import org.apache.polaris.core.config.RealmConfig;
+import org.apache.polaris.core.storage.AccessConfig;
+import org.apache.polaris.core.storage.StorageAccessProperty;
+import org.apache.polaris.core.storage.aws.AwsCredentialsStorageIntegration;
+import org.apache.polaris.core.storage.aws.AwsStorageConfigurationInfo;
+import org.apache.polaris.core.storage.cache.StorageCredentialCache;
+import org.apache.polaris.service.types.S3SignRequest;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.AwsCredentials;
+import software.amazon.awssdk.auth.credentials.AwsSessionCredentials;
+import software.amazon.awssdk.http.ContentStreamProvider;
+import software.amazon.awssdk.http.SdkHttpFullRequest;
+import software.amazon.awssdk.http.SdkHttpMethod;
+import software.amazon.awssdk.http.SdkHttpRequest;
+import software.amazon.awssdk.http.auth.aws.signer.AwsV4HttpSigner;
+import software.amazon.awssdk.http.auth.spi.signer.SignRequest;
+import software.amazon.awssdk.http.auth.spi.signer.SignedRequest;
+import software.amazon.awssdk.services.s3.S3Client;
+
+/**
+ * S3 request signer that creates presigned URLs for S3 operations using AWS credentials obtained
+ * from the storage integration.
+ */
+public class S3RequestSignerImpl implements S3RequestSigner {
+
+ private final AwsV4HttpSigner signer = AwsV4HttpSigner.create();
+
+ private final AwsCredentialsStorageIntegration storageIntegration;
+ private final StorageCredentialCache storageCredentialCache;
+
+ public S3RequestSignerImpl(
+ AwsCredentialsStorageIntegration storageIntegration,
+ StorageCredentialCache storageCredentialCache) {
+ this.storageIntegration = storageIntegration;
+ this.storageCredentialCache = storageCredentialCache;
+ }
+
+ @Override
+ public S3SignResponse signRequest(
+ RealmConfig realmConfig,
+ AwsStorageConfigurationInfo storageConfig,
+ S3SignRequest signingRequest) {
+
+ SdkHttpMethod method = SdkHttpMethod.valueOf(signingRequest.method());
+ URI uri = signingRequest.uri();
+ String body = signingRequest.body();
+
+ SdkHttpFullRequest.Builder requestToSign =
+ SdkHttpFullRequest.builder()
Review Comment:
Aren't they included in the request URI?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2283331272
##########
runtime/server/build.gradle.kts:
##########
@@ -39,9 +39,6 @@ val distributionElements by
}
dependencies {
- implementation(project(":polaris-core"))
Review Comment:
This is another change I would try to pull out to another PR, it's not related to this changeset correct?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2263148586
##########
runtime/server/build.gradle.kts:
##########
@@ -39,9 +39,6 @@ val distributionElements by
}
dependencies {
- implementation(project(":polaris-core"))
Review Comment:
Not directly referenced in this module.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2503673391
##########
runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogAdapter.java:
##########
@@ -828,6 +830,9 @@ public Response getConfig(
.addAll(VIEW_ENDPOINTS)
.addAll(PolarisEndpoints.getSupportedGenericTableEndpoints(realmConfig))
.addAll(PolarisEndpoints.getSupportedPolicyEndpoints(realmConfig))
+ .addAll(
+ PolarisEndpoints.getSupportedRemoteSigningEndpoints(
Review Comment:
OK let's remove.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2511871535
##########
runtime/service/src/main/java/org/apache/polaris/service/storage/s3/sign/S3RemoteSigningCatalogHandler.java:
##########
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.service.storage.s3.sign;
+
+import java.util.Collection;
+import java.util.EnumSet;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.auth.PolarisAuthorizableOperation;
+import org.apache.polaris.core.auth.PolarisAuthorizer;
+import org.apache.polaris.core.auth.PolarisPrincipal;
+import org.apache.polaris.core.config.FeatureConfiguration;
+import org.apache.polaris.core.config.RealmConfig;
+import org.apache.polaris.core.context.CallContext;
+import org.apache.polaris.core.entity.CatalogEntity;
+import org.apache.polaris.core.entity.PolarisEntity;
+import org.apache.polaris.core.entity.PolarisEntityConstants;
+import org.apache.polaris.core.persistence.PolarisResolvedPathWrapper;
+import org.apache.polaris.core.persistence.resolver.ResolutionManifestFactory;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisStorageActions;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.apache.polaris.service.catalog.common.CatalogHandler;
+import org.apache.polaris.service.catalog.common.CatalogUtils;
+import org.apache.polaris.service.catalog.io.FileIOUtil;
+import org.apache.polaris.service.context.catalog.CallContextCatalogFactory;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignRequest;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignResponse;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class S3RemoteSigningCatalogHandler extends CatalogHandler implements AutoCloseable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(S3RemoteSigningCatalogHandler.class);
+
+ private final CallContextCatalogFactory catalogFactory;
+ private final S3RequestSigner s3RequestSigner;
+
+ private CatalogEntity catalogEntity;
+ private Catalog baseCatalog;
+
+ public S3RemoteSigningCatalogHandler(
+ PolarisDiagnostics diagnostics,
+ CallContext callContext,
+ ResolutionManifestFactory resolutionManifestFactory,
+ CallContextCatalogFactory catalogFactory,
+ PolarisPrincipal polarisPrincipal,
+ String catalogName,
+ PolarisAuthorizer authorizer,
+ S3RequestSigner s3RequestSigner) {
+ super(
+ diagnostics,
+ callContext,
+ resolutionManifestFactory,
+ polarisPrincipal,
+ catalogName,
+ authorizer,
+ // external catalogs are not supported for S3 remote signing
+ null,
+ null);
+ this.catalogFactory = catalogFactory;
+ this.s3RequestSigner = s3RequestSigner;
+ }
+
+ @Override
+ protected void initializeCatalog() {
+ catalogEntity =
+ CatalogEntity.of(resolutionManifest.getResolvedReferenceCatalogEntity().getRawLeafEntity());
+ if (catalogEntity.isExternal()) {
+ throw new ForbiddenException("Cannot use S3 remote signing with federated catalogs.");
+ }
+ baseCatalog =
+ catalogFactory.createCallContextCatalog(callContext, polarisPrincipal, resolutionManifest);
+ }
+
+ public PolarisS3SignResponse signS3Request(
+ PolarisS3SignRequest s3SignRequest, TableIdentifier tableIdentifier) {
+
+ LOGGER.debug("Requesting s3 signing for {}: {}", tableIdentifier, s3SignRequest);
+
+ PolarisAuthorizableOperation authzOp =
+ s3SignRequest.write()
+ ? PolarisAuthorizableOperation.SIGN_S3_WRITE_REQUEST
+ : PolarisAuthorizableOperation.SIGN_S3_READ_REQUEST;
+
+ authorizeRemoteSigningOrThrow(EnumSet.of(authzOp), tableIdentifier);
+
+ // Must be done after the authorization check, as the auth check creates the catalog entity
+ throwIfRemoteSigningNotEnabled(callContext.getRealmConfig(), catalogEntity);
+
+ var result =
+ InMemoryStorageIntegration.validateAllowedLocations(
+ callContext.getRealmConfig(),
+ getAllowedLocations(tableIdentifier),
+ getStorageActions(s3SignRequest),
+ getTargetLocations(s3SignRequest));
+
+ if (result.values().stream().anyMatch(r -> r.values().stream().anyMatch(v -> !v.isSuccess()))) {
+ throw new ForbiddenException("Requested S3 location is not allowed.");
+ }
+
+ PolarisS3SignResponse s3SignResponse = s3RequestSigner.signRequest(s3SignRequest);
+ LOGGER.debug("S3 signing response: {}", s3SignResponse);
+
+ return s3SignResponse;
+ }
+
+ private Collection<String> getAllowedLocations(TableIdentifier tableIdentifier) {
+
+ if (baseCatalog.tableExists(tableIdentifier)) {
+
+ // If the table exists, get allowed locations from the table metadata
+ Table table = baseCatalog.loadTable(tableIdentifier);
+ if (table instanceof BaseTable baseTable) {
+ return StorageUtil.getLocationsUsedByTable(baseTable.operations().current());
+ }
+
+ throw new ForbiddenException("No storage configuration found for table.");
+
+ } else {
+
+ // If the table or view doesn't exist, the engine might be writing the manifests before the
+ // table creation is committed. In this case, we still need to check allowed locations from
+ // the parent entities.
+
+ PolarisResolvedPathWrapper resolvedPath =
+ CatalogUtils.findResolvedStorageEntity(resolutionManifest, tableIdentifier);
+
+ Optional<PolarisEntity> storageInfo = FileIOUtil.findStorageInfoFromHierarchy(resolvedPath);
+
+ var configurationInfo =
+ storageInfo
+ .map(PolarisEntity::getInternalPropertiesAsMap)
+ .map(info -> info.get(PolarisEntityConstants.getStorageConfigInfoPropertyName()))
+ .map(PolarisStorageConfigurationInfo::deserialize);
+
+ if (configurationInfo.isEmpty()) {
+ throw new ForbiddenException("No storage configuration found for table.");
+ }
+
+ return configurationInfo.get().getAllowedLocations();
+ }
+ }
+
+ private Set<PolarisStorageActions> getStorageActions(PolarisS3SignRequest s3SignRequest) {
+ // TODO M2: better handling of DELETE and LIST
Review Comment:
By just looking at the request URI, it's not easy to know if the request is a write or a delete, or if it is a read or a list.
For example: the [ListObjects](https://redocly.github.io/redoc/?url=https://api.apis.guru/v2/specs/amazonaws.com/s3/2006-03-01/openapi.json#operation/ListObjects) operation is conceptually a LIST operation, and [GetObject](https://redocly.github.io/redoc/?url=https://api.apis.guru/v2/specs/amazonaws.com/s3/2006-03-01/openapi.json#operation/GetObject) is conceptually a READ. But both requests use the GET method, so it's not easy to disambiguate.
Another example: the [DeleteObject](https://redocly.github.io/redoc/?url=https://api.apis.guru/v2/specs/amazonaws.com/s3/2006-03-01/openapi.json#operation/DeleteObject) operation uses the DELETE method, but the [DeleteObjects](https://redocly.github.io/redoc/?url=https://api.apis.guru/v2/specs/amazonaws.com/s3/2006-03-01/openapi.json#operation/DeleteObjects) operation uses... the POST method 🤷♂️
I will add some comments to clarify.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2281920428
##########
runtime/service/src/main/java/org/apache/polaris/service/storage/aws/signer/S3RequestSignerImpl.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.service.storage.aws.signer;
+
+import jakarta.enterprise.context.ApplicationScoped;
+import jakarta.inject.Inject;
+import java.net.URI;
+import org.apache.iceberg.aws.s3.signer.S3SignRequest;
+import org.apache.polaris.service.aws.sign.model.ImmutablePolarisS3SignResponse;
+import org.apache.polaris.service.aws.sign.model.PolarisS3SignResponse;
+import org.apache.polaris.service.storage.StorageConfiguration;
+import software.amazon.awssdk.auth.credentials.AwsCredentials;
+import software.amazon.awssdk.http.ContentStreamProvider;
+import software.amazon.awssdk.http.SdkHttpFullRequest;
+import software.amazon.awssdk.http.SdkHttpMethod;
+import software.amazon.awssdk.http.SdkHttpRequest;
+import software.amazon.awssdk.http.auth.aws.signer.AwsV4HttpSigner;
+import software.amazon.awssdk.http.auth.spi.signer.SignRequest;
+import software.amazon.awssdk.http.auth.spi.signer.SignedRequest;
+import software.amazon.awssdk.services.s3.S3Client;
+
+@ApplicationScoped
+class S3RequestSignerImpl implements S3RequestSigner {
+
+ private final AwsV4HttpSigner signer = AwsV4HttpSigner.create();
+
+ @Inject StorageConfiguration storageConfiguration;
+
+ @Override
+ public PolarisS3SignResponse signRequest(S3SignRequest signingRequest) {
+
+ URI uri = signingRequest.uri();
+ SdkHttpMethod method = SdkHttpMethod.valueOf(signingRequest.method());
+
+ SdkHttpFullRequest.Builder requestToSign =
+ SdkHttpFullRequest.builder()
+ .uri(uri)
+ .protocol(uri.getScheme())
+ .method(method)
+ .headers(signingRequest.headers());
+
+ AwsCredentials credentials = storageConfiguration.awsSystemCredentials().resolveCredentials();
Review Comment:
These are the server credentials – they must not need refreshing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2276745804
##########
runtime/service/src/main/java/org/apache/polaris/service/storage/aws/signer/S3RemoteSigningCatalogHandler.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.service.storage.aws.signer;
+
+import jakarta.ws.rs.core.SecurityContext;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.apache.polaris.core.auth.PolarisAuthorizableOperation;
+import org.apache.polaris.core.auth.PolarisAuthorizer;
+import org.apache.polaris.core.config.FeatureConfiguration;
+import org.apache.polaris.core.config.RealmConfig;
+import org.apache.polaris.core.context.CallContext;
+import org.apache.polaris.core.entity.CatalogEntity;
+import org.apache.polaris.core.entity.PolarisEntitySubType;
+import org.apache.polaris.core.persistence.resolver.ResolutionManifestFactory;
+import org.apache.polaris.service.aws.sign.model.PolarisS3SignRequest;
+import org.apache.polaris.service.aws.sign.model.PolarisS3SignResponse;
+import org.apache.polaris.service.catalog.common.CatalogHandler;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class S3RemoteSigningCatalogHandler extends CatalogHandler implements AutoCloseable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(S3RemoteSigningCatalogHandler.class);
+
+ private final S3RequestSigner s3RequestSigner;
+
+ private CatalogEntity catalogEntity;
+
+ public S3RemoteSigningCatalogHandler(
+ CallContext callContext,
+ ResolutionManifestFactory resolutionManifestFactory,
+ SecurityContext securityContext,
+ String catalogName,
+ PolarisAuthorizer authorizer,
+ S3RequestSigner s3RequestSigner) {
+ super(callContext, resolutionManifestFactory, securityContext, catalogName, authorizer);
+ this.s3RequestSigner = s3RequestSigner;
+ }
+
+ @Override
+ protected void initializeCatalog() {
+ catalogEntity =
+ CatalogEntity.of(resolutionManifest.getResolvedReferenceCatalogEntity().getRawLeafEntity());
+ if (catalogEntity.isExternal()) {
+ throw new ForbiddenException("Cannot use S3 remote signing with federated catalogs.");
+ }
+ // no need to materialize the catalog here, as we only need the catalog entity
+ }
+
+ public PolarisS3SignResponse signS3Request(
+ PolarisS3SignRequest s3SignRequest, TableIdentifier tableIdentifier) {
+
+ LOGGER.debug("Requesting s3 signing for {}: {}", tableIdentifier, s3SignRequest);
+
+ // TODO authorize based on the request's method?
+
+ try {
+ authorizeBasicTableLikeOperationOrThrow(
+ PolarisAuthorizableOperation.SIGN_S3_REQUEST,
+ PolarisEntitySubType.ICEBERG_TABLE,
+ tableIdentifier);
+ } catch (NoSuchTableException e) {
+ // FIXME can we do better here?
+ authorizeCreateTableLikeUnderNamespaceOperationOrThrow(
+ PolarisAuthorizableOperation.SIGN_S3_REQUEST, tableIdentifier);
+ }
+
+ throwIfRemoteSigningNotEnabled(callContext.getRealmConfig(), catalogEntity);
Review Comment:
Actually, not possible: you need to perform RBAC first, so that the catalog entity is fetched.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2284737402
##########
runtime/server/build.gradle.kts:
##########
@@ -39,9 +39,6 @@ val distributionElements by
}
dependencies {
- implementation(project(":polaris-core"))
Review Comment:
Correct:
https://github.com/apache/polaris/pull/2385
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on PR #2280:
URL: https://github.com/apache/polaris/pull/2280#issuecomment-3491950266
> If yes, I am bit confused, if we know M2 is the one we eventually wanna make and M2 is a bit orthogonal to M1 why are making / investing on M1, a couple questions that would be really helpful to get your take :
It's not orthogonal, M2 builds on top of M1.
> 1. How long it will take for us to get M2 post M1 ? (you mentioned M2 is already in works ?)
It depends if we want all the features in M2 or not. I would actually personally break M2 into smaller pieces. The most important piece is "Access checks based on the S3 location", which can be done in a few man-days. The "Nessie style" signed parameters would require a bit more than that, imo.
> 2. why are we making M1 if M1 is not production recommended and based on my understanding it will never be.
For the same reason we released the Events feature in incomplete state, with "beta" status: to get users feedback, and to work incrementally.
> 3. when both M2 and M1 are there when will a user use M1 over M2 ?
These are not "flavors" of remote signing. It's just one same feature, in two levels of maturity M1: experimental, M2: production-ready.
> are we gonna introduce a new endpoint for M2 ?
Yes, if we adopt Nessie's strategy for fast remote signing as proposed in the doc (it's optional). Note: this isn't a breaking change, as it's not user facing.
Happy to discuss this topic more in depth in a Slack channel or during the next sync!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2542043256
##########
runtime/service/src/main/java/org/apache/polaris/service/storage/s3/sign/S3RemoteSigningCatalogHandler.java:
##########
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.service.storage.s3.sign;
+
+import java.util.Collection;
+import java.util.EnumSet;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.auth.PolarisAuthorizableOperation;
+import org.apache.polaris.core.auth.PolarisAuthorizer;
+import org.apache.polaris.core.auth.PolarisPrincipal;
+import org.apache.polaris.core.config.FeatureConfiguration;
+import org.apache.polaris.core.config.RealmConfig;
+import org.apache.polaris.core.context.CallContext;
+import org.apache.polaris.core.entity.CatalogEntity;
+import org.apache.polaris.core.entity.PolarisEntity;
+import org.apache.polaris.core.entity.PolarisEntityConstants;
+import org.apache.polaris.core.persistence.PolarisResolvedPathWrapper;
+import org.apache.polaris.core.persistence.resolver.ResolutionManifestFactory;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisStorageActions;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.apache.polaris.service.catalog.common.CatalogHandler;
+import org.apache.polaris.service.catalog.common.CatalogUtils;
+import org.apache.polaris.service.catalog.io.FileIOUtil;
+import org.apache.polaris.service.context.catalog.CallContextCatalogFactory;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignRequest;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignResponse;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class S3RemoteSigningCatalogHandler extends CatalogHandler implements AutoCloseable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(S3RemoteSigningCatalogHandler.class);
+
+ private final CallContextCatalogFactory catalogFactory;
+ private final S3RequestSigner s3RequestSigner;
+
+ private CatalogEntity catalogEntity;
+ private Catalog baseCatalog;
+
+ public S3RemoteSigningCatalogHandler(
+ PolarisDiagnostics diagnostics,
+ CallContext callContext,
+ ResolutionManifestFactory resolutionManifestFactory,
+ CallContextCatalogFactory catalogFactory,
+ PolarisPrincipal polarisPrincipal,
+ String catalogName,
+ PolarisAuthorizer authorizer,
+ S3RequestSigner s3RequestSigner) {
+ super(
+ diagnostics,
+ callContext,
+ resolutionManifestFactory,
+ polarisPrincipal,
+ catalogName,
+ authorizer,
+ // external catalogs are not supported for S3 remote signing
+ null,
+ null);
+ this.catalogFactory = catalogFactory;
+ this.s3RequestSigner = s3RequestSigner;
+ }
+
+ @Override
+ protected void initializeCatalog() {
+ catalogEntity =
+ CatalogEntity.of(resolutionManifest.getResolvedReferenceCatalogEntity().getRawLeafEntity());
+ if (catalogEntity.isExternal()) {
+ throw new ForbiddenException("Cannot use S3 remote signing with federated catalogs.");
+ }
+ baseCatalog =
+ catalogFactory.createCallContextCatalog(callContext, polarisPrincipal, resolutionManifest);
+ }
+
+ public PolarisS3SignResponse signS3Request(
+ PolarisS3SignRequest s3SignRequest, TableIdentifier tableIdentifier) {
+
+ LOGGER.debug("Requesting s3 signing for {}: {}", tableIdentifier, s3SignRequest);
+
+ PolarisAuthorizableOperation authzOp =
+ s3SignRequest.write()
+ ? PolarisAuthorizableOperation.SIGN_S3_WRITE_REQUEST
+ : PolarisAuthorizableOperation.SIGN_S3_READ_REQUEST;
+
+ authorizeRemoteSigningOrThrow(EnumSet.of(authzOp), tableIdentifier);
+
+ // Must be done after the authorization check, as the auth check creates the catalog entity
+ throwIfRemoteSigningNotEnabled(callContext.getRealmConfig(), catalogEntity);
+
+ var result =
+ InMemoryStorageIntegration.validateAllowedLocations(
+ callContext.getRealmConfig(),
+ getAllowedLocations(tableIdentifier),
+ getStorageActions(s3SignRequest),
+ getTargetLocations(s3SignRequest));
+
+ if (result.values().stream().anyMatch(r -> r.values().stream().anyMatch(v -> !v.isSuccess()))) {
+ throw new ForbiddenException("Requested S3 location is not allowed.");
+ }
+
+ PolarisS3SignResponse s3SignResponse = s3RequestSigner.signRequest(s3SignRequest);
+ LOGGER.debug("S3 signing response: {}", s3SignResponse);
+
+ return s3SignResponse;
+ }
+
+ private Collection<String> getAllowedLocations(TableIdentifier tableIdentifier) {
+
+ if (baseCatalog.tableExists(tableIdentifier)) {
+
+ // If the table exists, get allowed locations from the table metadata
+ Table table = baseCatalog.loadTable(tableIdentifier);
Review Comment:
Sorry, which 3 properties? I'm not sure I understand your question 😅
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2544216972
##########
runtime/service/src/main/java/org/apache/polaris/service/storage/s3/sign/S3RemoteSigningCatalogHandler.java:
##########
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.service.storage.s3.sign;
+
+import java.util.Collection;
+import java.util.EnumSet;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.auth.PolarisAuthorizableOperation;
+import org.apache.polaris.core.auth.PolarisAuthorizer;
+import org.apache.polaris.core.auth.PolarisPrincipal;
+import org.apache.polaris.core.config.FeatureConfiguration;
+import org.apache.polaris.core.config.RealmConfig;
+import org.apache.polaris.core.context.CallContext;
+import org.apache.polaris.core.entity.CatalogEntity;
+import org.apache.polaris.core.entity.PolarisEntity;
+import org.apache.polaris.core.entity.PolarisEntityConstants;
+import org.apache.polaris.core.persistence.PolarisResolvedPathWrapper;
+import org.apache.polaris.core.persistence.resolver.ResolutionManifestFactory;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisStorageActions;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.apache.polaris.service.catalog.common.CatalogHandler;
+import org.apache.polaris.service.catalog.common.CatalogUtils;
+import org.apache.polaris.service.catalog.io.FileIOUtil;
+import org.apache.polaris.service.context.catalog.CallContextCatalogFactory;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignRequest;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignResponse;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class S3RemoteSigningCatalogHandler extends CatalogHandler implements AutoCloseable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(S3RemoteSigningCatalogHandler.class);
+
+ private final CallContextCatalogFactory catalogFactory;
+ private final S3RequestSigner s3RequestSigner;
+
+ private CatalogEntity catalogEntity;
+ private Catalog baseCatalog;
+
+ public S3RemoteSigningCatalogHandler(
+ PolarisDiagnostics diagnostics,
+ CallContext callContext,
+ ResolutionManifestFactory resolutionManifestFactory,
+ CallContextCatalogFactory catalogFactory,
+ PolarisPrincipal polarisPrincipal,
+ String catalogName,
+ PolarisAuthorizer authorizer,
+ S3RequestSigner s3RequestSigner) {
+ super(
+ diagnostics,
+ callContext,
+ resolutionManifestFactory,
+ polarisPrincipal,
+ catalogName,
+ authorizer,
+ // external catalogs are not supported for S3 remote signing
+ null,
+ null);
+ this.catalogFactory = catalogFactory;
+ this.s3RequestSigner = s3RequestSigner;
+ }
+
+ @Override
+ protected void initializeCatalog() {
+ catalogEntity =
+ CatalogEntity.of(resolutionManifest.getResolvedReferenceCatalogEntity().getRawLeafEntity());
+ if (catalogEntity.isExternal()) {
+ throw new ForbiddenException("Cannot use S3 remote signing with federated catalogs.");
+ }
+ baseCatalog =
+ catalogFactory.createCallContextCatalog(callContext, polarisPrincipal, resolutionManifest);
+ }
+
+ public PolarisS3SignResponse signS3Request(
+ PolarisS3SignRequest s3SignRequest, TableIdentifier tableIdentifier) {
+
+ LOGGER.debug("Requesting s3 signing for {}: {}", tableIdentifier, s3SignRequest);
Review Comment:
TBH its fine, I just wanted to know whats your preference :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2511644077
##########
runtime/service/src/main/java/org/apache/polaris/service/catalog/common/CatalogHandler.java:
##########
@@ -394,6 +394,47 @@ protected void authorizeRenameTableLikeOperationOrThrow(
initializeCatalog();
}
+ protected void authorizeRemoteSigningOrThrow(
+ EnumSet<PolarisAuthorizableOperation> ops, TableIdentifier identifier) {
+
+ ensureResolutionManifestForTable(identifier);
+ PolarisResolvedPathWrapper target =
+ resolutionManifest.getResolvedPath(
+ identifier, PolarisEntityType.TABLE_LIKE, PolarisEntitySubType.ICEBERG_TABLE, true);
+
+ // If the table doesn't exist, we still need to check allowed locations from the parent
+ // namespace.
Review Comment:
Yes. When creating a new table, engines write the manifest files first, then commit the table. The manifest writes must be signed too. Thus, a request to sign can arrive at the server before the table is created.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2536244871
##########
CHANGELOG.md:
##########
@@ -36,6 +36,14 @@ request adding CHANGELOG notes for breaking (!) changes and possibly other secti
[Iceberg Metrics Reporting]: https://iceberg.apache.org/docs/latest/metrics-reporting/
+- **S3 remote request signing** has been added, allowing Polaris to work with S3-compatible object storage systems.
+ *Remote signing is currently experimental and not enabled by default*. In particular, RBAC checks are currently not
+ production-ready. One new table privilege was introduced: `TABLE_REMOTE_SIGN`. To enable remote signing:
+ 1. Set the system-wide property `REMOTE_SIGNING_ENABLED` or the catalog-level `polaris.request-signing.enabled`
+ property to `true`.
+ 2. Grant the `TABLE_REMOTE_SIGN` privilege to a catalog role. The role must also be granted the `TABLE_READ_DATA`
+ and `TABLE_WRITE_DATA` privileges.
Review Comment:
Are we then failing during Runtime ? is this done mostly like at the time of granting there was both READ & WRITE but then admin revoked one of them so this grant needs to be revoked too ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2263153060
##########
runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java:
##########
@@ -297,12 +298,6 @@ public CreateNamespaceResponse createNamespace(CreateNamespaceRequest request) {
}
}
- private static boolean isStaticFacade(CatalogEntity catalog) {
Review Comment:
Refactored to become a method of `CatalogEntity`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2276616845
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/StorageAccessProperty.java:
##########
@@ -39,6 +42,21 @@ public enum StorageAccessProperty {
Boolean.class, "s3.path-style-access", "whether to use S3 path style access", false),
CLIENT_REGION(
String.class, "client.region", "region to configure client for making requests to AWS"),
+ AWS_REMOTE_SIGNING_ENABLED(
+ Boolean.class,
+ S3FileIOProperties.REMOTE_SIGNING_ENABLED,
+ "whether to enable remote signing for S3 requests",
Review Comment:
Actually I don't think that's how it works. Looking at this logic:
https://github.com/apache/polaris/blob/15f23cae8e678f8a13c20c5012919565288525a0/polaris-core/src/main/java/org/apache/polaris/core/config/PolarisConfigurationStore.java#L116-L142
It seems that remote signing could be _globally_ disabled, but _enabled_ for a specific catalog.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on PR #2280:
URL: https://github.com/apache/polaris/pull/2280#issuecomment-3201251553
FYI I just rebased and synchronized this PR with the design doc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2260480360
##########
runtime/service/src/main/java/org/apache/polaris/service/storage/aws/signer/S3RequestSignerImpl.java:
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.service.storage.aws.signer;
+
+import java.net.URI;
+import java.util.Set;
+import org.apache.iceberg.aws.s3.signer.ImmutableS3SignResponse;
+import org.apache.iceberg.aws.s3.signer.S3SignResponse;
+import org.apache.polaris.core.config.RealmConfig;
+import org.apache.polaris.core.storage.AccessConfig;
+import org.apache.polaris.core.storage.StorageAccessProperty;
+import org.apache.polaris.core.storage.aws.AwsCredentialsStorageIntegration;
+import org.apache.polaris.core.storage.aws.AwsStorageConfigurationInfo;
+import org.apache.polaris.core.storage.cache.StorageCredentialCache;
+import org.apache.polaris.service.types.S3SignRequest;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.AwsCredentials;
+import software.amazon.awssdk.auth.credentials.AwsSessionCredentials;
+import software.amazon.awssdk.http.ContentStreamProvider;
+import software.amazon.awssdk.http.SdkHttpFullRequest;
+import software.amazon.awssdk.http.SdkHttpMethod;
+import software.amazon.awssdk.http.SdkHttpRequest;
+import software.amazon.awssdk.http.auth.aws.signer.AwsV4HttpSigner;
+import software.amazon.awssdk.http.auth.spi.signer.SignRequest;
+import software.amazon.awssdk.http.auth.spi.signer.SignedRequest;
+import software.amazon.awssdk.services.s3.S3Client;
+
+/**
+ * S3 request signer that creates presigned URLs for S3 operations using AWS credentials obtained
+ * from the storage integration.
+ */
+public class S3RequestSignerImpl implements S3RequestSigner {
+
+ private final AwsV4HttpSigner signer = AwsV4HttpSigner.create();
+
+ private final AwsCredentialsStorageIntegration storageIntegration;
+ private final StorageCredentialCache storageCredentialCache;
+
+ public S3RequestSignerImpl(
+ AwsCredentialsStorageIntegration storageIntegration,
+ StorageCredentialCache storageCredentialCache) {
+ this.storageIntegration = storageIntegration;
+ this.storageCredentialCache = storageCredentialCache;
+ }
+
+ @Override
+ public S3SignResponse signRequest(
+ RealmConfig realmConfig,
+ AwsStorageConfigurationInfo storageConfig,
+ S3SignRequest signingRequest) {
+
+ SdkHttpMethod method = SdkHttpMethod.valueOf(signingRequest.method());
+ URI uri = signingRequest.uri();
+ String body = signingRequest.body();
+
+ SdkHttpFullRequest.Builder requestToSign =
+ SdkHttpFullRequest.builder()
+ .uri(uri)
+ .protocol(uri.getScheme())
+ .method(method)
+ .headers(signingRequest.headers());
+
+ // FIXME is this correct?
+ boolean allowListOperation = isReadMethod(method);
+ Set<String> allowedReadLocations = isReadMethod(method) ? Set.of(uri.toString()) : Set.of();
+ Set<String> allowedWriteLocations = isWriteMethod(method) ? Set.of(uri.toString()) : Set.of();
+
+ // TODO use the storageCredentialCache to get or generate subscoped credentials
Review Comment:
Yep I gave up on the idea, the code has changed significantly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2263157118
##########
runtime/service/src/main/java/org/apache/polaris/service/storage/StorageConfiguration.java:
##########
@@ -72,21 +71,19 @@ public interface StorageConfiguration extends S3AccessConfig {
Optional<Duration> gcpAccessTokenLifespan();
default Supplier<StsClient> stsClientSupplier() {
- return stsClientSupplier(true);
- }
-
- default Supplier<StsClient> stsClientSupplier(boolean withCredentials) {
return Suppliers.memoize(
- () -> {
- StsClientBuilder stsClientBuilder = StsClient.builder();
- if (withCredentials) {
- stsClientBuilder.credentialsProvider(stsCredentials());
- }
- return stsClientBuilder.build();
- });
+ () -> StsClient.builder().credentialsProvider(awsSystemCredentials()).build());
}
- default AwsCredentialsProvider stsCredentials() {
+ /**
+ * Returns an {@link AwsCredentialsProvider} that provides system-wide AWS credentials. If both
+ * access key and secret key are present, it uses them directly; otherwise, it uses the default
+ * credentials provider chain.
+ *
+ * <p>The returned provider is not meant to be vended directly to clients, but rather used with
+ * STS, unless credential subscoping is disabled.
+ */
+ default AwsCredentialsProvider awsSystemCredentials() {
Review Comment:
Renamed to `awsSystemCredentials` because these are not "just" credentials for STS, they are server credentials.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2263150328
##########
runtime/service/src/intTest/java/org/apache/polaris/service/it/RestCatalogMinIOSpecialIT.java:
##########
@@ -86,7 +85,6 @@
@QuarkusIntegrationTest
@TestProfile(RestCatalogMinIOSpecialIT.Profile.class)
@ExtendWith(MinioExtension.class)
-@ExtendWith(PolarisIntegrationTestExtension.class)
Review Comment:
This extension should not be enabled in IT tests annotated with `@QuarkusIntegrationTest`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2511819630
##########
runtime/service/src/main/java/org/apache/polaris/service/storage/s3/sign/S3RemoteSigningCatalogHandler.java:
##########
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.service.storage.s3.sign;
+
+import java.util.Collection;
+import java.util.EnumSet;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.auth.PolarisAuthorizableOperation;
+import org.apache.polaris.core.auth.PolarisAuthorizer;
+import org.apache.polaris.core.auth.PolarisPrincipal;
+import org.apache.polaris.core.config.FeatureConfiguration;
+import org.apache.polaris.core.config.RealmConfig;
+import org.apache.polaris.core.context.CallContext;
+import org.apache.polaris.core.entity.CatalogEntity;
+import org.apache.polaris.core.entity.PolarisEntity;
+import org.apache.polaris.core.entity.PolarisEntityConstants;
+import org.apache.polaris.core.persistence.PolarisResolvedPathWrapper;
+import org.apache.polaris.core.persistence.resolver.ResolutionManifestFactory;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisStorageActions;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.apache.polaris.service.catalog.common.CatalogHandler;
+import org.apache.polaris.service.catalog.common.CatalogUtils;
+import org.apache.polaris.service.catalog.io.FileIOUtil;
+import org.apache.polaris.service.context.catalog.CallContextCatalogFactory;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignRequest;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignResponse;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class S3RemoteSigningCatalogHandler extends CatalogHandler implements AutoCloseable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(S3RemoteSigningCatalogHandler.class);
+
+ private final CallContextCatalogFactory catalogFactory;
+ private final S3RequestSigner s3RequestSigner;
+
+ private CatalogEntity catalogEntity;
+ private Catalog baseCatalog;
+
+ public S3RemoteSigningCatalogHandler(
+ PolarisDiagnostics diagnostics,
+ CallContext callContext,
+ ResolutionManifestFactory resolutionManifestFactory,
+ CallContextCatalogFactory catalogFactory,
+ PolarisPrincipal polarisPrincipal,
+ String catalogName,
+ PolarisAuthorizer authorizer,
+ S3RequestSigner s3RequestSigner) {
+ super(
+ diagnostics,
+ callContext,
+ resolutionManifestFactory,
+ polarisPrincipal,
+ catalogName,
+ authorizer,
+ // external catalogs are not supported for S3 remote signing
+ null,
+ null);
+ this.catalogFactory = catalogFactory;
+ this.s3RequestSigner = s3RequestSigner;
+ }
+
+ @Override
+ protected void initializeCatalog() {
+ catalogEntity =
+ CatalogEntity.of(resolutionManifest.getResolvedReferenceCatalogEntity().getRawLeafEntity());
+ if (catalogEntity.isExternal()) {
+ throw new ForbiddenException("Cannot use S3 remote signing with federated catalogs.");
+ }
+ baseCatalog =
+ catalogFactory.createCallContextCatalog(callContext, polarisPrincipal, resolutionManifest);
+ }
+
+ public PolarisS3SignResponse signS3Request(
+ PolarisS3SignRequest s3SignRequest, TableIdentifier tableIdentifier) {
+
+ LOGGER.debug("Requesting s3 signing for {}: {}", tableIdentifier, s3SignRequest);
+
+ PolarisAuthorizableOperation authzOp =
+ s3SignRequest.write()
+ ? PolarisAuthorizableOperation.SIGN_S3_WRITE_REQUEST
+ : PolarisAuthorizableOperation.SIGN_S3_READ_REQUEST;
+
+ authorizeRemoteSigningOrThrow(EnumSet.of(authzOp), tableIdentifier);
+
+ // Must be done after the authorization check, as the auth check creates the catalog entity
+ throwIfRemoteSigningNotEnabled(callContext.getRealmConfig(), catalogEntity);
+
+ var result =
+ InMemoryStorageIntegration.validateAllowedLocations(
+ callContext.getRealmConfig(),
+ getAllowedLocations(tableIdentifier),
+ getStorageActions(s3SignRequest),
+ getTargetLocations(s3SignRequest));
+
+ if (result.values().stream().anyMatch(r -> r.values().stream().anyMatch(v -> !v.isSuccess()))) {
+ throw new ForbiddenException("Requested S3 location is not allowed.");
+ }
+
+ PolarisS3SignResponse s3SignResponse = s3RequestSigner.signRequest(s3SignRequest);
+ LOGGER.debug("S3 signing response: {}", s3SignResponse);
+
+ return s3SignResponse;
+ }
+
+ private Collection<String> getAllowedLocations(TableIdentifier tableIdentifier) {
+
+ if (baseCatalog.tableExists(tableIdentifier)) {
+
+ // If the table exists, get allowed locations from the table metadata
+ Table table = baseCatalog.loadTable(tableIdentifier);
Review Comment:
> for each SIGN request a loadTable call is super expensive
Indeed, that's where M2 will help because it will save us this `loadTable` call.
I will add a FIXME for now.
> [suggestion / optional to address] wondering for now if we should add a prod readiness check to keep this disabled for prod ?
Good idea, will do 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2544211734
##########
runtime/service/src/main/java/org/apache/polaris/service/storage/s3/sign/S3RemoteSigningCatalogHandler.java:
##########
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.service.storage.s3.sign;
+
+import java.util.Collection;
+import java.util.EnumSet;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.auth.PolarisAuthorizableOperation;
+import org.apache.polaris.core.auth.PolarisAuthorizer;
+import org.apache.polaris.core.auth.PolarisPrincipal;
+import org.apache.polaris.core.config.FeatureConfiguration;
+import org.apache.polaris.core.config.RealmConfig;
+import org.apache.polaris.core.context.CallContext;
+import org.apache.polaris.core.entity.CatalogEntity;
+import org.apache.polaris.core.entity.PolarisEntity;
+import org.apache.polaris.core.entity.PolarisEntityConstants;
+import org.apache.polaris.core.persistence.PolarisResolvedPathWrapper;
+import org.apache.polaris.core.persistence.resolver.ResolutionManifestFactory;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisStorageActions;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.apache.polaris.service.catalog.common.CatalogHandler;
+import org.apache.polaris.service.catalog.common.CatalogUtils;
+import org.apache.polaris.service.catalog.io.FileIOUtil;
+import org.apache.polaris.service.context.catalog.CallContextCatalogFactory;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignRequest;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignResponse;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class S3RemoteSigningCatalogHandler extends CatalogHandler implements AutoCloseable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(S3RemoteSigningCatalogHandler.class);
+
+ private final CallContextCatalogFactory catalogFactory;
+ private final S3RequestSigner s3RequestSigner;
+
+ private CatalogEntity catalogEntity;
+ private Catalog baseCatalog;
+
+ public S3RemoteSigningCatalogHandler(
+ PolarisDiagnostics diagnostics,
+ CallContext callContext,
+ ResolutionManifestFactory resolutionManifestFactory,
+ CallContextCatalogFactory catalogFactory,
+ PolarisPrincipal polarisPrincipal,
+ String catalogName,
+ PolarisAuthorizer authorizer,
+ S3RequestSigner s3RequestSigner) {
+ super(
+ diagnostics,
+ callContext,
+ resolutionManifestFactory,
+ polarisPrincipal,
+ catalogName,
+ authorizer,
+ // external catalogs are not supported for S3 remote signing
+ null,
+ null);
+ this.catalogFactory = catalogFactory;
+ this.s3RequestSigner = s3RequestSigner;
+ }
+
+ @Override
+ protected void initializeCatalog() {
+ catalogEntity =
+ CatalogEntity.of(resolutionManifest.getResolvedReferenceCatalogEntity().getRawLeafEntity());
+ if (catalogEntity.isExternal()) {
+ throw new ForbiddenException("Cannot use S3 remote signing with federated catalogs.");
+ }
+ baseCatalog =
+ catalogFactory.createCallContextCatalog(callContext, polarisPrincipal, resolutionManifest);
+ }
+
+ public PolarisS3SignResponse signS3Request(
+ PolarisS3SignRequest s3SignRequest, TableIdentifier tableIdentifier) {
+
+ LOGGER.debug("Requesting s3 signing for {}: {}", tableIdentifier, s3SignRequest);
+
+ PolarisAuthorizableOperation authzOp =
+ s3SignRequest.write()
+ ? PolarisAuthorizableOperation.SIGN_S3_WRITE_REQUEST
+ : PolarisAuthorizableOperation.SIGN_S3_READ_REQUEST;
+
+ authorizeRemoteSigningOrThrow(EnumSet.of(authzOp), tableIdentifier);
+
+ // Must be done after the authorization check, as the auth check creates the catalog entity
+ throwIfRemoteSigningNotEnabled(callContext.getRealmConfig(), catalogEntity);
+
+ var result =
+ InMemoryStorageIntegration.validateAllowedLocations(
+ callContext.getRealmConfig(),
+ getAllowedLocations(tableIdentifier),
+ getStorageActions(s3SignRequest),
+ getTargetLocations(s3SignRequest));
+
+ if (result.values().stream().anyMatch(r -> r.values().stream().anyMatch(v -> !v.isSuccess()))) {
+ throw new ForbiddenException("Requested S3 location is not allowed.");
+ }
+
+ PolarisS3SignResponse s3SignResponse = s3RequestSigner.signRequest(s3SignRequest);
+ LOGGER.debug("S3 signing response: {}", s3SignResponse);
+
+ return s3SignResponse;
+ }
+
+ private Collection<String> getAllowedLocations(TableIdentifier tableIdentifier) {
+
+ if (baseCatalog.tableExists(tableIdentifier)) {
+
+ // If the table exists, get allowed locations from the table metadata
+ Table table = baseCatalog.loadTable(tableIdentifier);
Review Comment:
iceberg properties location, write.data.path, write.metadata.path, which is used for credential vending, I am not sure we persist them Entity, as the function StorageUtil#getLocationsUsedByTable has always worked in TableMetadata and not on Entity properties
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2542026212
##########
runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java:
##########
@@ -844,6 +869,37 @@ ALLOW_FEDERATED_CATALOGS_CREDENTIAL_VENDING, getResolvedCatalogEntity())) {
return responseBuilder;
}
+ private AccessDelegationMode selectAccessDelegationMode(
+ Set<AccessDelegationMode> delegationModes) {
+
+ if (delegationModes.isEmpty()) {
+ return UNKNOWN;
+ }
+
+ if (delegationModes.size() == 1) {
+ return delegationModes.iterator().next();
+ }
+
+ if (delegationModes.contains(VENDED_CREDENTIALS) && delegationModes.contains(REMOTE_SIGNING)) {
+
+ boolean skipCredIndirection =
+ realmConfig.getConfig(FeatureConfiguration.SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION);
+
+ boolean credentialSubscopingAllowed =
+ baseCatalog instanceof IcebergCatalog
Review Comment:
https://github.com/apache/polaris/issues/3090
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2283324619
##########
integration-tests/src/main/java/org/apache/polaris/service/it/test/PolarisRestCatalogIntegrationBase.java:
##########
@@ -146,11 +148,13 @@ public abstract class PolarisRestCatalogIntegrationBase extends CatalogTests<RES
private RESTCatalog restCatalog;
private String currentCatalogName;
private Map<String, String> restCatalogConfig;
- private URI externalCatalogBase;
+ private URI externalCatalogBaseLocation;
Review Comment:
Minor request here, but you could pull this and the new IntegerationTest Helpers out into a separate PR and reduce the size of this PR by a bit. I think that would also be a very fast review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on PR #2280:
URL: https://github.com/apache/polaris/pull/2280#issuecomment-3168106882
Ready for review. The test failures are unrelated to this change.
\cc @singhpk234 @snazy @dimas-b @metadaddy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "snazy (via GitHub)" <gi...@apache.org>.
snazy commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2272725103
##########
api/iceberg-aws-sign-service/src/main/java/org/apache/polaris/service/aws/sign/model/PolarisS3SignRequest.java:
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.service.aws.sign.model;
+
+import com.fasterxml.jackson.databind.annotation.JsonDeserialize;
+import com.fasterxml.jackson.databind.annotation.JsonSerialize;
+import jakarta.annotation.Nullable;
+import org.apache.iceberg.aws.s3.signer.S3SignRequest;
+import org.apache.polaris.immutables.PolarisImmutable;
+import org.immutables.value.Value;
+
+/**
+ * Request for S3 signing requests.
+ *
+ * <p>Copy of {@link S3SignRequest}, because the original does not have Jackson annotations.
+ */
+@PolarisImmutable
+@JsonDeserialize(as = ImmutablePolarisS3SignRequest.class)
+@JsonSerialize(as = ImmutablePolarisS3SignRequest.class)
+public interface PolarisS3SignRequest extends S3SignRequest {
+
+ @Value.Default
+ @Nullable // Replace javax.annotation.Nullable with jakarta.annotation.Nullable
Review Comment:
```suggestion
@Nullable // Replace javax.annotation.Nullable from 'S3SignRequest` with jakarta.annotation.Nullable
```
##########
polaris-core/src/main/java/org/apache/polaris/core/config/FeatureConfiguration.java:
##########
@@ -349,4 +349,20 @@ public static void enforceFeatureEnabledOrThrow(
+ "it is still possible to enforce the uniqueness of table locations within a catalog.")
.defaultValue(false)
.buildFeatureConfiguration();
+
+ public static final FeatureConfiguration<Boolean> REMOTE_SIGNING_ENABLED =
+ PolarisConfiguration.<Boolean>builder()
+ .key("REMOTE_SIGNING_ENABLED")
+ .catalogConfig("polaris.config.remote-signing.enabled")
+ .description(
+ "If true, the remote signing endpoints are enabled either globally, or for a specific catalog.")
Review Comment:
Maybe add a note that the feature is experimental, elaborating why on the `CHANGELOG.md?
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/StorageAccessProperty.java:
##########
@@ -39,6 +42,21 @@ public enum StorageAccessProperty {
Boolean.class, "s3.path-style-access", "whether to use S3 path style access", false),
CLIENT_REGION(
String.class, "client.region", "region to configure client for making requests to AWS"),
+ AWS_REMOTE_SIGNING_ENABLED(
+ Boolean.class,
+ S3FileIOProperties.REMOTE_SIGNING_ENABLED,
+ "whether to enable remote signing for S3 requests",
Review Comment:
Maybe add some text saying that this depends on the "global" flag.
##########
polaris-core/src/main/java/org/apache/polaris/core/rest/PolarisEndpoints.java:
##########
@@ -94,4 +101,18 @@ public static Set<Endpoint> getSupportedPolicyEndpoints(CallContext callContext)
callContext.getRealmConfig().getConfig(FeatureConfiguration.ENABLE_POLICY_STORE);
return policyStoreEnabled ? POLICY_STORE_ENDPOINTS : ImmutableSet.of();
}
+
+ /**
+ * Get the remote signing endpoints. Returns {@link #REMOTE_SIGNING_ENDPOINTS} if {@link
+ * FeatureConfiguration#REMOTE_SIGNING_ENABLED} is set globally to true or if the catalog enables
+ * remote signing; otherwise, returns an empty set.
+ */
+ public static Set<Endpoint> getSupportedRemoteSigningEndpoints(
+ CallContext callContext, CatalogEntity catalogEntity) {
+ boolean remoteSigningEnabled =
+ callContext
+ .getRealmConfig()
Review Comment:
```suggestion
RealmConfig realmConfig, CatalogEntity catalogEntity) {
boolean remoteSigningEnabled =
realmConfig
```
##########
runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java:
##########
@@ -774,21 +757,55 @@ public Optional<LoadTableResponse> loadTableWithAccessDelegationIfStale(
throw new IllegalStateException("Cannot wrap catalog that does not produce BaseTable");
}
+ private CatalogEntity getCatalogEntity() {
+ PolarisResolvedPathWrapper catalogPath = resolutionManifest.getResolvedReferenceCatalogEntity();
+ callContext
+ .getPolarisCallContext()
+ .getDiagServices()
+ .checkNotNull(catalogPath, "No catalog available for loadTable request");
+ CatalogEntity catalogEntity = CatalogEntity.of(catalogPath.getRawLeafEntity());
+ LOGGER.info("Catalog type: {}", catalogEntity.getCatalogType());
+ return catalogEntity;
+ }
+
private LoadTableResponse.Builder buildLoadTableResponseWithDelegationCredentials(
TableIdentifier tableIdentifier,
TableMetadata tableMetadata,
Set<PolarisStorageActions> actions,
- String snapshots) {
+ Set<AccessDelegationMode> delegationModes,
+ CatalogEntity catalogEntity) {
LoadTableResponse.Builder responseBuilder =
LoadTableResponse.builder().withTableMetadata(tableMetadata);
- if (baseCatalog instanceof SupportsCredentialDelegation credentialDelegation) {
+ if (baseCatalog instanceof SupportsCredentialDelegation credentialDelegation
+ && delegationModes.contains(AccessDelegationMode.VENDED_CREDENTIALS)) {
LOGGER
.atDebug()
.addKeyValue("tableIdentifier", tableIdentifier)
.addKeyValue("tableLocation", tableMetadata.location())
.log("Fetching client credentials for table");
AccessConfig accessConfig =
- credentialDelegation.getAccessConfig(tableIdentifier, tableMetadata, actions);
+ credentialDelegation.getAccessConfigForCredentialDelegation(
+ tableIdentifier, tableMetadata, actions);
+ Map<String, String> credentialConfig = accessConfig.credentials();
+ responseBuilder.addAllConfig(credentialConfig);
+ responseBuilder.addAllConfig(accessConfig.extraProperties());
+ if (!credentialConfig.isEmpty()) {
+ responseBuilder.addCredential(
+ ImmutableCredential.builder()
+ .prefix(tableMetadata.location())
+ .config(credentialConfig)
+ .build());
+ }
Review Comment:
Looks like this part is duplicated. Any chance to remove the duplciated code?
##########
runtime/service/src/main/java/org/apache/polaris/service/context/catalog/PolarisCallContextCatalogFactory.java:
##########
@@ -92,7 +104,10 @@ public Catalog createCallContextCatalog(
securityContext,
taskExecutor,
fileIOFactory,
- polarisEventListener);
+ polarisEventListener,
+ storageIntegrationProvider,
+ prefixParser,
+ uriInfo);
Review Comment:
(Not related to this PR)
The consistently increasing amount of parameters passed to these types worries me.
##########
runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalog.java:
##########
@@ -866,13 +888,52 @@ public AccessConfig getAccessConfig(
return AccessConfig.builder().build();
}
return FileIOUtil.refreshAccessConfig(
- callContext,
- storageCredentialCache,
- getCredentialVendor(),
- tableIdentifier,
- getLocationsAllowedToBeAccessed(tableMetadata),
- storageActions,
- storageInfo.get());
+ callContext,
+ storageCredentialCache,
+ getCredentialVendor(),
+ tableIdentifier,
+ getLocationsAllowedToBeAccessed(tableMetadata),
+ storageActions,
+ storageInfo.get())
+ .orElse(AccessConfig.EMPTY);
+ }
+
+ @Override
+ public AccessConfig getAccessConfigForRemoteSigning(TableIdentifier tableIdentifier) {
+
+ Optional<PolarisStorageConfigurationInfo> configurationInfo =
+ findStorageInfo(tableIdentifier)
+ .map(
+ info ->
+ deserialize(
Review Comment:
Nit: move the deserialze() to a separate .map() for readability.
##########
integration-tests/src/main/java/org/apache/polaris/service/it/test/PolarisS3RemoteSigningIntegrationTest.java:
##########
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.service.it.test;
+
+import static org.apache.polaris.core.storage.StorageAccessProperty.AWS_ENDPOINT;
+import static org.apache.polaris.core.storage.StorageAccessProperty.AWS_PATH_STYLE_ACCESS;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import com.google.common.collect.ImmutableMap;
+import java.util.Map;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.TableMetadata;
+import org.apache.iceberg.TableMetadataParser;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.iceberg.io.ResolvingFileIO;
+import org.apache.iceberg.rest.RESTCatalog;
+import org.apache.polaris.core.admin.model.Catalog;
+import org.apache.polaris.core.config.FeatureConfiguration;
+import org.apache.polaris.service.it.env.CatalogConfig;
+import org.apache.polaris.service.it.env.RestCatalogConfig;
+import org.junit.jupiter.api.Disabled;
+import org.junit.jupiter.api.Test;
+
+/** Runs PolarisRestCatalogIntegrationBase test on S3 with remote signing enabled. */
+@CatalogConfig(properties = {"header.X-Iceberg-Access-Delegation", "remote-signing"})
+@RestCatalogConfig({
+ // The default client file IO implementation is InMemoryFileIO,
+ // which does not support remote signing.
+ org.apache.iceberg.CatalogProperties.FILE_IO_IMPL,
+ "org.apache.iceberg.io.ResolvingFileIO",
+})
+public abstract class PolarisS3RemoteSigningIntegrationTest
+ extends PolarisRestCatalogAwsIntegrationTest {
+
+ @Override
+ protected ImmutableMap.Builder<String, String> clientFileIOProperties() {
+ ImmutableMap.Builder<String, String> builder =
+ super.clientFileIOProperties()
+ .put(AWS_PATH_STYLE_ACCESS.getPropertyName(), String.valueOf(pathStyleAccess()));
+ endpoint().ifPresent(endpoint -> builder.put(AWS_ENDPOINT.getPropertyName(), endpoint));
+ return builder;
+ }
+
+ /**
+ * Override this method to return true if the S3 client should use path-style access. Default is
+ * false, which means virtual-hosted-style access will be used.
+ */
+ protected boolean pathStyleAccess() {
+ return false;
+ }
+
+ @CatalogConfig(properties = {"polaris.config.remote-signing.enabled", "false"})
+ @Test
+ public void testInternalCatalogRemoteSigningDisabled() {
+ @SuppressWarnings("resource")
+ RESTCatalog catalog = catalog();
+ Namespace ns1 = Namespace.of("ns1");
+ catalog.createNamespace(ns1);
+ TableIdentifier tableIdentifier = TableIdentifier.of(ns1, "my_table");
+ assertThatThrownBy(() -> catalog.createTable(tableIdentifier, SCHEMA))
+ .isInstanceOf(ForbiddenException.class)
+ .hasMessageContaining("Remote signing is not enabled for this catalog")
+ .hasMessageContaining(FeatureConfiguration.REMOTE_SIGNING_ENABLED.key())
+ .hasMessageContaining(FeatureConfiguration.REMOTE_SIGNING_ENABLED.catalogConfig());
+ }
+
+ @CatalogConfig(Catalog.TypeEnum.EXTERNAL)
+ @Test
+ public void testExternalCatalogRemoteSigningDisabled() {
+ @SuppressWarnings("resource")
+ RESTCatalog catalog = catalog();
+ Namespace ns1 = Namespace.of("ns1");
+ catalog.createNamespace(ns1);
+ TableMetadata tableMetadata =
+ TableMetadata.newTableMetadata(
+ SCHEMA,
+ PartitionSpec.unpartitioned(),
+ externalCatalogBaseLocation() + "/ns1/my_table",
+ Map.of());
+ try (ResolvingFileIO resolvingFileIO = initializeClientFileIO(new ResolvingFileIO())) {
+ String fileLocation =
+ externalCatalogBaseLocation() + "/ns1/my_table/metadata/v1.metadata.json";
+ TableMetadataParser.write(tableMetadata, resolvingFileIO.newOutputFile(fileLocation));
+ catalog.registerTable(TableIdentifier.of(ns1, "my_table"), fileLocation);
+ try {
+ assertThatThrownBy(() -> catalog.loadTable(TableIdentifier.of(ns1, "my_table")))
+ .isInstanceOf(ForbiddenException.class)
+ .hasMessageContaining("Remote signing is not enabled for external catalogs")
+ .hasMessageContaining(
+ FeatureConfiguration.REMOTE_SIGNING_EXTERNAL_CATALOGS_ENABLED.key());
+ } finally {
+ resolvingFileIO.deleteFile(fileLocation);
+ }
+ }
+ }
+
+ @Test
+ @Override
+ @Disabled("It's not possible to request an access delegation mode when registering a table.")
+ public void testRegisterTable() {
Review Comment:
I _think_ it's worth to try out whether the register-table endpoint receives the `X-Iceberg-Access-Delegation` header. In Nessie, we accept that header in the register-table endpoint and haven't ignored this test.
##########
runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java:
##########
@@ -710,30 +697,26 @@ public Optional<LoadTableResponse> loadTableWithAccessDelegationIfStale(
read, PolarisEntitySubType.ICEBERG_TABLE, tableIdentifier);
}
- PolarisResolvedPathWrapper catalogPath = resolutionManifest.getResolvedReferenceCatalogEntity();
- callContext
- .getPolarisCallContext()
- .getDiagServices()
- .checkNotNull(catalogPath, "No catalog available for loadTable request");
- CatalogEntity catalogEntity = CatalogEntity.of(catalogPath.getRawLeafEntity());
- LOGGER.info("Catalog type: {}", catalogEntity.getCatalogType());
- LOGGER.info(
- "allow external catalog credential vending: {}",
- callContext
- .getRealmConfig()
- .getConfig(
- FeatureConfiguration.ALLOW_EXTERNAL_CATALOG_CREDENTIAL_VENDING, catalogEntity));
- if (catalogEntity
- .getCatalogType()
- .equals(org.apache.polaris.core.admin.model.Catalog.TypeEnum.EXTERNAL)
- && !callContext
- .getRealmConfig()
- .getConfig(
- FeatureConfiguration.ALLOW_EXTERNAL_CATALOG_CREDENTIAL_VENDING, catalogEntity)) {
- throw new ForbiddenException(
- "Access Delegation is not enabled for this catalog. Please consult applicable "
- + "documentation for the catalog config property '%s' to enable this feature",
- FeatureConfiguration.ALLOW_EXTERNAL_CATALOG_CREDENTIAL_VENDING.catalogConfig());
+ CatalogEntity catalogEntity = getCatalogEntity();
+
+ if (delegationModes.contains(AccessDelegationMode.VENDED_CREDENTIALS)) {
+
+ LOGGER.info(
+ "allow external catalog credential vending: {}",
+ callContext
+ .getRealmConfig()
+ .getConfig(
+ FeatureConfiguration.ALLOW_EXTERNAL_CATALOG_CREDENTIAL_VENDING, catalogEntity));
Review Comment:
Nit: extract this value (duplicated) to a variable.
##########
polaris-core/src/main/java/org/apache/polaris/core/entity/CatalogEntity.java:
##########
@@ -205,11 +206,19 @@ public Catalog.TypeEnum getCatalogType() {
.orElse(null);
}
+ public boolean isExternal() {
+ return Objects.equals(getCatalogType(), Catalog.TypeEnum.EXTERNAL);
Review Comment:
```suggestion
return getCatalogType() == Catalog.TypeEnum.EXTERNAL;
```
enum values have object-identity ;)
##########
runtime/service/src/main/java/org/apache/polaris/service/storage/aws/signer/S3RemoteSigningCatalogHandler.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.service.storage.aws.signer;
+
+import jakarta.ws.rs.core.SecurityContext;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.apache.polaris.core.auth.PolarisAuthorizableOperation;
+import org.apache.polaris.core.auth.PolarisAuthorizer;
+import org.apache.polaris.core.config.FeatureConfiguration;
+import org.apache.polaris.core.config.RealmConfig;
+import org.apache.polaris.core.context.CallContext;
+import org.apache.polaris.core.entity.CatalogEntity;
+import org.apache.polaris.core.entity.PolarisEntitySubType;
+import org.apache.polaris.core.persistence.resolver.ResolutionManifestFactory;
+import org.apache.polaris.service.aws.sign.model.PolarisS3SignRequest;
+import org.apache.polaris.service.aws.sign.model.PolarisS3SignResponse;
+import org.apache.polaris.service.catalog.common.CatalogHandler;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class S3RemoteSigningCatalogHandler extends CatalogHandler implements AutoCloseable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(S3RemoteSigningCatalogHandler.class);
+
+ private final S3RequestSigner s3RequestSigner;
+
+ private CatalogEntity catalogEntity;
+
+ public S3RemoteSigningCatalogHandler(
+ CallContext callContext,
+ ResolutionManifestFactory resolutionManifestFactory,
+ SecurityContext securityContext,
+ String catalogName,
+ PolarisAuthorizer authorizer,
+ S3RequestSigner s3RequestSigner) {
+ super(callContext, resolutionManifestFactory, securityContext, catalogName, authorizer);
+ this.s3RequestSigner = s3RequestSigner;
+ }
+
+ @Override
+ protected void initializeCatalog() {
+ catalogEntity =
+ CatalogEntity.of(resolutionManifest.getResolvedReferenceCatalogEntity().getRawLeafEntity());
+ if (catalogEntity.isExternal()) {
+ throw new ForbiddenException("Cannot use S3 remote signing with federated catalogs.");
+ }
+ // no need to materialize the catalog here, as we only need the catalog entity
+ }
+
+ public PolarisS3SignResponse signS3Request(
+ PolarisS3SignRequest s3SignRequest, TableIdentifier tableIdentifier) {
+
+ LOGGER.debug("Requesting s3 signing for {}: {}", tableIdentifier, s3SignRequest);
+
+ // TODO authorize based on the request's method?
+
+ try {
+ authorizeBasicTableLikeOperationOrThrow(
+ PolarisAuthorizableOperation.SIGN_S3_REQUEST,
+ PolarisEntitySubType.ICEBERG_TABLE,
+ tableIdentifier);
+ } catch (NoSuchTableException e) {
+ // FIXME can we do better here?
+ authorizeCreateTableLikeUnderNamespaceOperationOrThrow(
+ PolarisAuthorizableOperation.SIGN_S3_REQUEST, tableIdentifier);
+ }
+
+ throwIfRemoteSigningNotEnabled(callContext.getRealmConfig(), catalogEntity);
Review Comment:
Nit: move before the `try {`
##########
runtime/service/src/main/java/org/apache/polaris/service/storage/aws/signer/S3RemoteSigningCatalogAdapter.java:
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.service.storage.aws.signer;
+
+import com.google.common.annotations.VisibleForTesting;
+import jakarta.enterprise.context.RequestScoped;
+import jakarta.inject.Inject;
+import jakarta.ws.rs.core.Response;
+import jakarta.ws.rs.core.SecurityContext;
+import java.util.function.Function;
+import org.apache.iceberg.aws.s3.signer.S3SignResponse;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.rest.RESTUtil;
+import org.apache.polaris.core.auth.PolarisAuthorizer;
+import org.apache.polaris.core.context.CallContext;
+import org.apache.polaris.core.context.RealmContext;
+import org.apache.polaris.core.persistence.resolver.ResolutionManifestFactory;
+import org.apache.polaris.service.aws.sign.api.IcebergRestS3SignerApiService;
+import org.apache.polaris.service.aws.sign.model.PolarisS3SignRequest;
+import org.apache.polaris.service.catalog.CatalogPrefixParser;
+import org.apache.polaris.service.catalog.common.CatalogAdapter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * An adapter between generated service types like `IcebergRestCatalogApiService` and
+ * `IcebergCatalogHandler`.
+ */
+@RequestScoped
+public class S3RemoteSigningCatalogAdapter
+ implements IcebergRestS3SignerApiService, CatalogAdapter {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(S3RemoteSigningCatalogAdapter.class);
+
+ @Inject CallContext callContext;
+ @Inject ResolutionManifestFactory resolutionManifestFactory;
+ @Inject PolarisAuthorizer polarisAuthorizer;
+ @Inject CatalogPrefixParser prefixParser;
+ @Inject S3RequestSigner s3RequestSigner;
+
+ /**
+ * Execute operations on a catalog wrapper and ensure we close the BaseCatalog afterward. This
+ * will typically ensure the underlying FileIO is closed.
+ */
+ private Response withCatalog(
+ SecurityContext securityContext,
+ String prefix,
+ Function<S3RemoteSigningCatalogHandler, Response> action) {
+ String catalogName = prefixParser.prefixToCatalogName(callContext.getRealmContext(), prefix);
+ try (S3RemoteSigningCatalogHandler handler = newHandlerWrapper(securityContext, catalogName)) {
+ return action.apply(handler);
+ } catch (RuntimeException e) {
+ LOGGER.debug("RuntimeException while operating on catalog. Propagating to caller.", e);
+ throw e;
+ } catch (Exception e) {
+ LOGGER.error("Error while operating on catalog", e);
+ throw new RuntimeException(e);
+ }
+ }
+
+ @VisibleForTesting
+ S3RemoteSigningCatalogHandler newHandlerWrapper(
+ SecurityContext securityContext, String catalogName) {
+ validatePrincipal(securityContext);
+ return new S3RemoteSigningCatalogHandler(
+ callContext,
+ resolutionManifestFactory,
+ securityContext,
+ catalogName,
+ polarisAuthorizer,
+ s3RequestSigner);
+ }
+
+ @Override
+ public Response signS3Request(
+ String prefix,
+ String namespace,
+ String table,
+ PolarisS3SignRequest s3SignRequest,
+ RealmContext realmContext,
+ SecurityContext securityContext) {
+ Namespace ns = decodeNamespace(namespace);
+ TableIdentifier tableIdentifier = TableIdentifier.of(ns, RESTUtil.decodeString(table));
+ return withCatalog(
+ securityContext,
+ prefix,
+ catalog -> {
+ S3SignResponse response = catalog.signS3Request(s3SignRequest, tableIdentifier);
+ return Response.status(Response.Status.OK).entity(response).build();
Review Comment:
... but that would require clients to respect those 🤷
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on PR #2280:
URL: https://github.com/apache/polaris/pull/2280#issuecomment-3188856023
Hey @adutra thank you the change, i was reading the description and it turns out we are introducing new priviledges, IMHO this requires broader set of eyes and feedbacks as its just no longer implementation of the specification of IRC which can be contained in the PR.
Can you please send your Proposal in the dev-list (if you have already have one, can you please link this to the PR)
I only see this thread presently - https://lists.apache.org/thread/qvzwc3qxlfrk9vr7yfbx6zxfhz9lhlbc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2519564798
##########
runtime/service/src/main/java/org/apache/polaris/service/storage/s3/sign/S3RemoteSigningCatalogHandler.java:
##########
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.service.storage.s3.sign;
+
+import java.util.Collection;
+import java.util.EnumSet;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.auth.PolarisAuthorizableOperation;
+import org.apache.polaris.core.auth.PolarisAuthorizer;
+import org.apache.polaris.core.auth.PolarisPrincipal;
+import org.apache.polaris.core.config.FeatureConfiguration;
+import org.apache.polaris.core.config.RealmConfig;
+import org.apache.polaris.core.context.CallContext;
+import org.apache.polaris.core.entity.CatalogEntity;
+import org.apache.polaris.core.entity.PolarisEntity;
+import org.apache.polaris.core.entity.PolarisEntityConstants;
+import org.apache.polaris.core.persistence.PolarisResolvedPathWrapper;
+import org.apache.polaris.core.persistence.resolver.ResolutionManifestFactory;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisStorageActions;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.apache.polaris.service.catalog.common.CatalogHandler;
+import org.apache.polaris.service.catalog.common.CatalogUtils;
+import org.apache.polaris.service.catalog.io.FileIOUtil;
+import org.apache.polaris.service.context.catalog.CallContextCatalogFactory;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignRequest;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignResponse;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class S3RemoteSigningCatalogHandler extends CatalogHandler implements AutoCloseable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(S3RemoteSigningCatalogHandler.class);
+
+ private final CallContextCatalogFactory catalogFactory;
+ private final S3RequestSigner s3RequestSigner;
+
+ private CatalogEntity catalogEntity;
+ private Catalog baseCatalog;
+
+ public S3RemoteSigningCatalogHandler(
+ PolarisDiagnostics diagnostics,
+ CallContext callContext,
+ ResolutionManifestFactory resolutionManifestFactory,
+ CallContextCatalogFactory catalogFactory,
+ PolarisPrincipal polarisPrincipal,
+ String catalogName,
+ PolarisAuthorizer authorizer,
+ S3RequestSigner s3RequestSigner) {
+ super(
+ diagnostics,
+ callContext,
+ resolutionManifestFactory,
+ polarisPrincipal,
+ catalogName,
+ authorizer,
+ // external catalogs are not supported for S3 remote signing
+ null,
+ null);
+ this.catalogFactory = catalogFactory;
+ this.s3RequestSigner = s3RequestSigner;
+ }
+
+ @Override
+ protected void initializeCatalog() {
+ catalogEntity =
+ CatalogEntity.of(resolutionManifest.getResolvedReferenceCatalogEntity().getRawLeafEntity());
+ if (catalogEntity.isExternal()) {
+ throw new ForbiddenException("Cannot use S3 remote signing with federated catalogs.");
+ }
+ baseCatalog =
+ catalogFactory.createCallContextCatalog(callContext, polarisPrincipal, resolutionManifest);
+ }
+
+ public PolarisS3SignResponse signS3Request(
+ PolarisS3SignRequest s3SignRequest, TableIdentifier tableIdentifier) {
+
+ LOGGER.debug("Requesting s3 signing for {}: {}", tableIdentifier, s3SignRequest);
+
+ PolarisAuthorizableOperation authzOp =
+ s3SignRequest.write()
+ ? PolarisAuthorizableOperation.SIGN_S3_WRITE_REQUEST
+ : PolarisAuthorizableOperation.SIGN_S3_READ_REQUEST;
+
+ authorizeRemoteSigningOrThrow(EnumSet.of(authzOp), tableIdentifier);
+
+ // Must be done after the authorization check, as the auth check creates the catalog entity
+ throwIfRemoteSigningNotEnabled(callContext.getRealmConfig(), catalogEntity);
+
+ var result =
+ InMemoryStorageIntegration.validateAllowedLocations(
+ callContext.getRealmConfig(),
+ getAllowedLocations(tableIdentifier),
+ getStorageActions(s3SignRequest),
+ getTargetLocations(s3SignRequest));
+
+ if (result.values().stream().anyMatch(r -> r.values().stream().anyMatch(v -> !v.isSuccess()))) {
+ throw new ForbiddenException("Requested S3 location is not allowed.");
+ }
+
+ PolarisS3SignResponse s3SignResponse = s3RequestSigner.signRequest(s3SignRequest);
+ LOGGER.debug("S3 signing response: {}", s3SignResponse);
+
+ return s3SignResponse;
+ }
+
+ private Collection<String> getAllowedLocations(TableIdentifier tableIdentifier) {
+
+ if (baseCatalog.tableExists(tableIdentifier)) {
+
+ // If the table exists, get allowed locations from the table metadata
+ Table table = baseCatalog.loadTable(tableIdentifier);
+ if (table instanceof BaseTable baseTable) {
+ return StorageUtil.getLocationsUsedByTable(baseTable.operations().current());
+ }
+
+ throw new ForbiddenException("No storage configuration found for table.");
+
+ } else {
+
+ // If the table or view doesn't exist, the engine might be writing the manifests before the
+ // table creation is committed. In this case, we still need to check allowed locations from
+ // the parent entities.
Review Comment:
A few updates on this. I added a few more tests and verified that all the above flags are working with remote signing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2541904748
##########
runtime/service/src/main/java/org/apache/polaris/service/storage/s3/sign/S3RemoteSigningCatalogHandler.java:
##########
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.service.storage.s3.sign;
+
+import java.util.Collection;
+import java.util.EnumSet;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.auth.PolarisAuthorizableOperation;
+import org.apache.polaris.core.auth.PolarisAuthorizer;
+import org.apache.polaris.core.auth.PolarisPrincipal;
+import org.apache.polaris.core.config.FeatureConfiguration;
+import org.apache.polaris.core.config.RealmConfig;
+import org.apache.polaris.core.context.CallContext;
+import org.apache.polaris.core.entity.CatalogEntity;
+import org.apache.polaris.core.entity.PolarisEntity;
+import org.apache.polaris.core.entity.PolarisEntityConstants;
+import org.apache.polaris.core.persistence.PolarisResolvedPathWrapper;
+import org.apache.polaris.core.persistence.resolver.ResolutionManifestFactory;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisStorageActions;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.apache.polaris.service.catalog.common.CatalogHandler;
+import org.apache.polaris.service.catalog.common.CatalogUtils;
+import org.apache.polaris.service.catalog.io.FileIOUtil;
+import org.apache.polaris.service.context.catalog.CallContextCatalogFactory;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignRequest;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignResponse;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class S3RemoteSigningCatalogHandler extends CatalogHandler implements AutoCloseable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(S3RemoteSigningCatalogHandler.class);
+
+ private final CallContextCatalogFactory catalogFactory;
+ private final S3RequestSigner s3RequestSigner;
+
+ private CatalogEntity catalogEntity;
+ private Catalog baseCatalog;
+
+ public S3RemoteSigningCatalogHandler(
+ PolarisDiagnostics diagnostics,
+ CallContext callContext,
+ ResolutionManifestFactory resolutionManifestFactory,
+ CallContextCatalogFactory catalogFactory,
+ PolarisPrincipal polarisPrincipal,
+ String catalogName,
+ PolarisAuthorizer authorizer,
+ S3RequestSigner s3RequestSigner) {
+ super(
+ diagnostics,
+ callContext,
+ resolutionManifestFactory,
+ polarisPrincipal,
+ catalogName,
+ authorizer,
+ // external catalogs are not supported for S3 remote signing
+ null,
+ null);
+ this.catalogFactory = catalogFactory;
+ this.s3RequestSigner = s3RequestSigner;
+ }
+
+ @Override
+ protected void initializeCatalog() {
+ catalogEntity =
+ CatalogEntity.of(resolutionManifest.getResolvedReferenceCatalogEntity().getRawLeafEntity());
+ if (catalogEntity.isExternal()) {
+ throw new ForbiddenException("Cannot use S3 remote signing with federated catalogs.");
+ }
+ baseCatalog =
+ catalogFactory.createCallContextCatalog(callContext, polarisPrincipal, resolutionManifest);
+ }
+
+ public PolarisS3SignResponse signS3Request(
+ PolarisS3SignRequest s3SignRequest, TableIdentifier tableIdentifier) {
+
+ LOGGER.debug("Requesting s3 signing for {}: {}", tableIdentifier, s3SignRequest);
Review Comment:
This call costs almost nothing: all parameters are constants, so the call will immediately return if the DEBUG level is not active for this logger. The JVM can generally inline this call.
But, if you prefer, I will remove the call.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2536245770
##########
runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java:
##########
@@ -844,6 +869,37 @@ ALLOW_FEDERATED_CATALOGS_CREDENTIAL_VENDING, getResolvedCatalogEntity())) {
return responseBuilder;
}
+ private AccessDelegationMode selectAccessDelegationMode(
+ Set<AccessDelegationMode> delegationModes) {
+
+ if (delegationModes.isEmpty()) {
+ return UNKNOWN;
+ }
+
+ if (delegationModes.size() == 1) {
+ return delegationModes.iterator().next();
+ }
+
+ if (delegationModes.contains(VENDED_CREDENTIALS) && delegationModes.contains(REMOTE_SIGNING)) {
+
+ boolean skipCredIndirection =
+ realmConfig.getConfig(FeatureConfiguration.SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION);
+
+ boolean credentialSubscopingAllowed =
+ baseCatalog instanceof IcebergCatalog
Review Comment:
should we add a ticket for this so we can tackle this later, WDYT ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "binarycat0 (via GitHub)" <gi...@apache.org>.
binarycat0 commented on PR #2280:
URL: https://github.com/apache/polaris/pull/2280#issuecomment-3553234640
Hello everyone 👋 I would like to propose my help in implementing M2 ([from proposed milestones](https://docs.google.com/document/d/1ygdia7u4bUHUt6n8XhZo48aKoIyyrCvKqan3XP25iB8/edit?tab=t.0#heading=h.76hegagb5gud)).
As I see M2 can be split into 3 independent features: locations-check, nessir-style-signed-params, support-proxies.
I already have a solution for check-locations and ready to create a follow-up PR after this will be merged.
Could we agree on a timeline to merge?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on PR #2280:
URL: https://github.com/apache/polaris/pull/2280#issuecomment-3487093398
This is now ready again for review!
\cc @snazy @singhpk234 @flyrain @dimas-b
This PR is equivalent to Milestone 1 in the original design doc. Let's try to work incrementally and get this in, we are already working on getting a new PR for Milestone 2 and especially the authz checks that are not yet implemented. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2263155414
##########
runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java:
##########
@@ -758,21 +741,55 @@ public Optional<LoadTableResponse> loadTableWithAccessDelegationIfStale(
throw new IllegalStateException("Cannot wrap catalog that does not produce BaseTable");
}
+ private CatalogEntity getCatalogEntity() {
+ PolarisResolvedPathWrapper catalogPath = resolutionManifest.getResolvedReferenceCatalogEntity();
+ callContext
+ .getPolarisCallContext()
+ .getDiagServices()
+ .checkNotNull(catalogPath, "No catalog available for loadTable request");
+ CatalogEntity catalogEntity = CatalogEntity.of(catalogPath.getRawLeafEntity());
+ LOGGER.info("Catalog type: {}", catalogEntity.getCatalogType());
+ return catalogEntity;
+ }
+
private LoadTableResponse.Builder buildLoadTableResponseWithDelegationCredentials(
TableIdentifier tableIdentifier,
TableMetadata tableMetadata,
Set<PolarisStorageActions> actions,
- String snapshots) {
+ Set<AccessDelegationMode> delegationModes,
+ CatalogEntity catalogEntity) {
LoadTableResponse.Builder responseBuilder =
LoadTableResponse.builder().withTableMetadata(tableMetadata);
- if (baseCatalog instanceof SupportsCredentialDelegation credentialDelegation) {
+ if (baseCatalog instanceof SupportsCredentialDelegation credentialDelegation
+ && delegationModes.contains(AccessDelegationMode.VENDED_CREDENTIALS)) {
LOGGER
.atDebug()
.addKeyValue("tableIdentifier", tableIdentifier)
.addKeyValue("tableLocation", tableMetadata.location())
.log("Fetching client credentials for table");
AccessConfig accessConfig =
- credentialDelegation.getAccessConfig(tableIdentifier, tableMetadata, actions);
+ credentialDelegation.getAccessConfigForCredentialDelegation(
+ tableIdentifier, tableMetadata, actions);
+ Map<String, String> credentialConfig = accessConfig.credentials();
+ responseBuilder.addAllConfig(credentialConfig);
+ responseBuilder.addAllConfig(accessConfig.extraProperties());
+ if (!credentialConfig.isEmpty()) {
+ responseBuilder.addCredential(
+ ImmutableCredential.builder()
+ .prefix(tableMetadata.location())
+ .config(credentialConfig)
+ .build());
+ }
+ } else if (baseCatalog instanceof SupportsRemoteSigning remoteSigning
+ && delegationModes.contains(AccessDelegationMode.REMOTE_SIGNING)) {
+ S3RemoteSigningCatalogHandler.throwIfRemoteSigningNotEnabled(
+ callContext.getRealmConfig(), catalogEntity);
+ LOGGER
+ .atDebug()
+ .addKeyValue("tableIdentifier", tableIdentifier)
+ .addKeyValue("tableLocation", tableMetadata.location())
+ .log("Enabling remote signing for table");
+ AccessConfig accessConfig = remoteSigning.getAccessConfigForRemoteSigning(tableIdentifier);
Review Comment:
This is where all `LoadTableResponse`s get remote signing enablement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "snazy (via GitHub)" <gi...@apache.org>.
snazy commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2276793008
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/StorageAccessProperty.java:
##########
@@ -39,6 +42,21 @@ public enum StorageAccessProperty {
Boolean.class, "s3.path-style-access", "whether to use S3 path style access", false),
CLIENT_REGION(
String.class, "client.region", "region to configure client for making requests to AWS"),
+ AWS_REMOTE_SIGNING_ENABLED(
+ Boolean.class,
+ S3FileIOProperties.REMOTE_SIGNING_ENABLED,
+ "whether to enable remote signing for S3 requests",
Review Comment:
Yea - so you can enable it, but it mustn't be disabled globally. Anyway, not a must-have.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2285663035
##########
runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java:
##########
@@ -297,12 +298,6 @@ public CreateNamespaceResponse createNamespace(CreateNamespaceRequest request) {
}
}
- private static boolean isStaticFacade(CatalogEntity catalog) {
Review Comment:
Thanks! It definitely helps to keep this sort of refactor out of the larger functional change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2284794627
##########
runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java:
##########
@@ -297,12 +298,6 @@ public CreateNamespaceResponse createNamespace(CreateNamespaceRequest request) {
}
}
- private static boolean isStaticFacade(CatalogEntity catalog) {
Review Comment:
@RussellSpitzer FYI:
#2386
#2387
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2267254496
##########
runtime/service/src/main/java/org/apache/polaris/service/catalog/io/DefaultFileIOFactory.java:
##########
@@ -101,6 +110,32 @@ public FileIO loadFileIO(
properties.putAll(accessConfig.get().credentials());
properties.putAll(accessConfig.get().extraProperties());
properties.putAll(accessConfig.get().internalProperties());
+ } else {
+ // If no subscoped creds were produced, use system-wide AWS or GCP credentials if available.
Review Comment:
This bit is not directly related to remote signing, but I realized that, when credentials subscoping is disabled, the server is unable to use the credentials configured in application.properties.
This change allows that.
That being said, the server would still be unable to use e.g. a custom S3 endpoint, simply because this property is not exposed in application.properties.
IMHO each property that can be configured in `AccessConfig` for a specific catalog should also be configurable by a system-wide property in application.properties.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on PR #2280:
URL: https://github.com/apache/polaris/pull/2280#issuecomment-3487401124
Hey Alex,
Thank a ton for all the work !
Is this understanding of M1 and M2 correct ? (take design doc above)
```
M1 - Simple, slow implementation with basic RBAC
- Not production ready – marked experimental
- Does not implement access checks based on S3 location
- Does not support HTTP proxies
- Corresponds to the current PR: https://github.com/apache/polaris/pull/2280
M2 - Fast implementation with efficient RBAC
- Production-ready
- Implements [Access checks based on the S3 location](https://docs.google.com/document/d/1ygdia7u4bUHUt6n8XhZo48aKoIyyrCvKqan3XP25iB8/edit?tab=t.0#bookmark=id.hs05149qp3eo)
- Supports HTTP proxies
- Implements "Nessie style" signed parameters
```
If yes, I am bit confused, if we know M2 is the one we eventually wanna make and M2 is a bit orthogonal to M1 why are making / investing on M1, a couple questions that would be really helpful to get your take :
1. How long it will take for us to get M2 post M1 ? (you mentioned M2 is already in works ?)
2. why are we making M1 if M1 is not production recommended and based on my understanding it will never be.
3. when both M2 and M1 are there when will a user use M1 over M2 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2511708475
##########
runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java:
##########
@@ -844,6 +869,37 @@ ALLOW_FEDERATED_CATALOGS_CREDENTIAL_VENDING, getResolvedCatalogEntity())) {
return responseBuilder;
}
+ private AccessDelegationMode selectAccessDelegationMode(
+ Set<AccessDelegationMode> delegationModes) {
+
+ if (delegationModes.isEmpty()) {
+ return UNKNOWN;
+ }
+
+ if (delegationModes.size() == 1) {
+ return delegationModes.iterator().next();
+ }
+
+ if (delegationModes.contains(VENDED_CREDENTIALS) && delegationModes.contains(REMOTE_SIGNING)) {
+
+ boolean skipCredIndirection =
+ realmConfig.getConfig(FeatureConfiguration.SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION);
+
+ boolean credentialSubscopingAllowed =
+ baseCatalog instanceof IcebergCatalog
Review Comment:
That's a good question!
It's complex to check this at this level because you need to find the `AwsStorageConfigurationInfo` corresponding to the resolved path in order to know if STS is unavailable.
I think we shouldn't make this logic too complex, because it's not common for a client to request more than one access delegation mode anyways.
I can though add a simple check to check if remote signing is enabled. If it's disabled there is no point in selecting `REMOTE_SIGNING`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on PR #2280:
URL: https://github.com/apache/polaris/pull/2280#issuecomment-3504202125
Hey Alex,
Thanks for the response, TBH I am still unclear why do we wanna introduce an endpoint for M1 only to introduce a new one in M2 ? How can an **public** endpoint not be a user facing change ?
Apologies but I am not able to follow, As you suggested, definitely sync will be really helpful to discuss, meanwhile i request you to please bump this in the thread https://lists.apache.org/thread/xpf60ko823c7mlg5bhnddq3btvdsq5s1 too for more eyes, in case I am missing something.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2519568503
##########
runtime/service/src/main/java/org/apache/polaris/service/storage/s3/sign/S3RemoteSigningCatalogHandler.java:
##########
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.service.storage.s3.sign;
+
+import java.util.Collection;
+import java.util.EnumSet;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.auth.PolarisAuthorizableOperation;
+import org.apache.polaris.core.auth.PolarisAuthorizer;
+import org.apache.polaris.core.auth.PolarisPrincipal;
+import org.apache.polaris.core.config.FeatureConfiguration;
+import org.apache.polaris.core.config.RealmConfig;
+import org.apache.polaris.core.context.CallContext;
+import org.apache.polaris.core.entity.CatalogEntity;
+import org.apache.polaris.core.entity.PolarisEntity;
+import org.apache.polaris.core.entity.PolarisEntityConstants;
+import org.apache.polaris.core.persistence.PolarisResolvedPathWrapper;
+import org.apache.polaris.core.persistence.resolver.ResolutionManifestFactory;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisStorageActions;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.apache.polaris.service.catalog.common.CatalogHandler;
+import org.apache.polaris.service.catalog.common.CatalogUtils;
+import org.apache.polaris.service.catalog.io.FileIOUtil;
+import org.apache.polaris.service.context.catalog.CallContextCatalogFactory;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignRequest;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignResponse;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class S3RemoteSigningCatalogHandler extends CatalogHandler implements AutoCloseable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(S3RemoteSigningCatalogHandler.class);
+
+ private final CallContextCatalogFactory catalogFactory;
+ private final S3RequestSigner s3RequestSigner;
+
+ private CatalogEntity catalogEntity;
+ private Catalog baseCatalog;
+
+ public S3RemoteSigningCatalogHandler(
+ PolarisDiagnostics diagnostics,
+ CallContext callContext,
+ ResolutionManifestFactory resolutionManifestFactory,
+ CallContextCatalogFactory catalogFactory,
+ PolarisPrincipal polarisPrincipal,
+ String catalogName,
+ PolarisAuthorizer authorizer,
+ S3RequestSigner s3RequestSigner) {
+ super(
+ diagnostics,
+ callContext,
+ resolutionManifestFactory,
+ polarisPrincipal,
+ catalogName,
+ authorizer,
+ // external catalogs are not supported for S3 remote signing
+ null,
+ null);
+ this.catalogFactory = catalogFactory;
+ this.s3RequestSigner = s3RequestSigner;
+ }
+
+ @Override
+ protected void initializeCatalog() {
+ catalogEntity =
+ CatalogEntity.of(resolutionManifest.getResolvedReferenceCatalogEntity().getRawLeafEntity());
+ if (catalogEntity.isExternal()) {
+ throw new ForbiddenException("Cannot use S3 remote signing with federated catalogs.");
+ }
+ baseCatalog =
+ catalogFactory.createCallContextCatalog(callContext, polarisPrincipal, resolutionManifest);
+ }
+
+ public PolarisS3SignResponse signS3Request(
+ PolarisS3SignRequest s3SignRequest, TableIdentifier tableIdentifier) {
+
+ LOGGER.debug("Requesting s3 signing for {}: {}", tableIdentifier, s3SignRequest);
+
+ PolarisAuthorizableOperation authzOp =
+ s3SignRequest.write()
+ ? PolarisAuthorizableOperation.SIGN_S3_WRITE_REQUEST
+ : PolarisAuthorizableOperation.SIGN_S3_READ_REQUEST;
+
+ authorizeRemoteSigningOrThrow(EnumSet.of(authzOp), tableIdentifier);
+
+ // Must be done after the authorization check, as the auth check creates the catalog entity
+ throwIfRemoteSigningNotEnabled(callContext.getRealmConfig(), catalogEntity);
+
+ var result =
+ InMemoryStorageIntegration.validateAllowedLocations(
+ callContext.getRealmConfig(),
+ getAllowedLocations(tableIdentifier),
+ getStorageActions(s3SignRequest),
+ getTargetLocations(s3SignRequest));
+
+ if (result.values().stream().anyMatch(r -> r.values().stream().anyMatch(v -> !v.isSuccess()))) {
+ throw new ForbiddenException("Requested S3 location is not allowed.");
+ }
+
+ PolarisS3SignResponse s3SignResponse = s3RequestSigner.signRequest(s3SignRequest);
+ LOGGER.debug("S3 signing response: {}", s3SignResponse);
+
+ return s3SignResponse;
+ }
+
+ private Collection<String> getAllowedLocations(TableIdentifier tableIdentifier) {
+
+ if (baseCatalog.tableExists(tableIdentifier)) {
+
+ // If the table exists, get allowed locations from the table metadata
+ Table table = baseCatalog.loadTable(tableIdentifier);
Review Comment:
Update on this topic: I was able to remove the `loadTable` call, replaced with a direct check of the `PolarisEntity` properties. That's going to be faster imho.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2263136841
##########
api/iceberg-aws-sign-service/build.gradle.kts:
##########
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import org.openapitools.generator.gradle.plugin.tasks.GenerateTask
+
+plugins {
+ alias(libs.plugins.openapi.generator)
+ id("polaris-client")
+ alias(libs.plugins.jandex)
+}
+
+dependencies {
+ implementation(project(":polaris-core"))
+
+ implementation(platform(libs.iceberg.bom))
+ implementation("org.apache.iceberg:iceberg-api")
+ implementation("org.apache.iceberg:iceberg-core")
+ implementation("org.apache.iceberg:iceberg-aws")
+
+ implementation(libs.jakarta.annotation.api)
+ implementation(libs.jakarta.inject.api)
+ implementation(libs.jakarta.validation.api)
+ implementation(libs.swagger.annotations)
+
+ implementation(libs.jakarta.servlet.api)
+ implementation(libs.jakarta.ws.rs.api)
+
+ implementation(platform(libs.micrometer.bom))
+ implementation("io.micrometer:micrometer-core")
+
+ implementation(platform(libs.jackson.bom))
+ implementation("com.fasterxml.jackson.core:jackson-annotations")
+ implementation("com.fasterxml.jackson.core:jackson-core")
+ implementation("com.fasterxml.jackson.core:jackson-databind")
+
+ compileOnly(libs.microprofile.fault.tolerance.api)
+
+ compileOnly(project(":polaris-immutables"))
+ annotationProcessor(project(":polaris-immutables", configuration = "processor"))
+}
+
+val rootDir = rootProject.layout.projectDirectory
+val specsDir = rootDir.dir("spec")
+val templatesDir = rootDir.dir("server-templates")
+// Use a different directory than 'generated/', because OpenAPI generator's `GenerateTask` adds the
+// whole directory to its task output, but 'generated/' is not exclusive to that task and in turn
+// breaks Gradle's caching.
+val generatedDir = project.layout.buildDirectory.dir("generated-openapi")
+val generatedOpenApiSrcDir = project.layout.buildDirectory.dir("generated-openapi/src/main/java")
+
+openApiGenerate {
+ // The OpenAPI generator does NOT resolve relative paths correctly against the Gradle project
+ // directory
+ inputSpec = provider { specsDir.file("polaris-catalog-service.yaml").asFile.absolutePath }
+ generatorName = "jaxrs-resteasy"
+ outputDir = provider { generatedDir.get().asFile.absolutePath }
+ apiPackage = "org.apache.polaris.service.aws.sign.api"
+ ignoreFileOverride.set(provider { rootDir.file(".openapi-generator-ignore").asFile.absolutePath })
+ templateDir.set(provider { templatesDir.asFile.absolutePath })
+ removeOperationIdPrefix.set(true)
+ globalProperties.put("apis", "S3SignerApi")
+ globalProperties.put("models", "false")
+ globalProperties.put("apiDocs", "false")
+ globalProperties.put("modelTests", "false")
+ configOptions.put("resourceName", "catalog")
+ configOptions.put("useTags", "true")
+ configOptions.put("useBeanValidation", "false")
+ configOptions.put("sourceFolder", "src/main/java")
+ configOptions.put("useJakartaEe", "true")
+ configOptions.put("hideGenerationTimestamp", "true")
+ additionalProperties.put("apiNamePrefix", "IcebergRest")
+ additionalProperties.put("apiNameSuffix", "")
+ additionalProperties.put("metricsPrefix", "polaris")
+ serverVariables.put("basePath", "api/catalog")
+ typeMappings =
+ mapOf("S3SignRequest" to "org.apache.polaris.service.aws.sign.model.PolarisS3SignRequest")
+ importMappings =
+ mapOf(
+ "IcebergErrorResponse" to "org.apache.iceberg.rest.responses.ErrorResponse",
+ "S3SignRequest" to "org.apache.polaris.service.aws.sign.model.PolarisS3SignRequest",
+ "SignS3Request200Response" to
Review Comment:
FYI The name of the logical response type is a bit weird because it is inlined, instead of being a `$ref` in the OpanAPI spec.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on PR #2280:
URL: https://github.com/apache/polaris/pull/2280#issuecomment-3168329634
Re: integration test failure. The test passes locally. There is some classloading snafu going on, I will investigate later:
```
Caused by: java.lang.IllegalStateException: PolarisServerManager not found
at org.apache.polaris.service.it.ext.PolarisServerManagerLoader.lambda$static$0(PolarisServerManagerLoader.java:28)
at java.base/java.util.Optional.orElseThrow(Optional.java:403)
at org.apache.polaris.service.it.ext.PolarisServerManagerLoader.<clinit>(PolarisServerManagerLoader.java:28)
... 4 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2276583859
##########
integration-tests/src/main/java/org/apache/polaris/service/it/test/PolarisS3RemoteSigningIntegrationTest.java:
##########
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.service.it.test;
+
+import static org.apache.polaris.core.storage.StorageAccessProperty.AWS_ENDPOINT;
+import static org.apache.polaris.core.storage.StorageAccessProperty.AWS_PATH_STYLE_ACCESS;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import com.google.common.collect.ImmutableMap;
+import java.util.Map;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.TableMetadata;
+import org.apache.iceberg.TableMetadataParser;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.iceberg.io.ResolvingFileIO;
+import org.apache.iceberg.rest.RESTCatalog;
+import org.apache.polaris.core.admin.model.Catalog;
+import org.apache.polaris.core.config.FeatureConfiguration;
+import org.apache.polaris.service.it.env.CatalogConfig;
+import org.apache.polaris.service.it.env.RestCatalogConfig;
+import org.junit.jupiter.api.Disabled;
+import org.junit.jupiter.api.Test;
+
+/** Runs PolarisRestCatalogIntegrationBase test on S3 with remote signing enabled. */
+@CatalogConfig(properties = {"header.X-Iceberg-Access-Delegation", "remote-signing"})
+@RestCatalogConfig({
+ // The default client file IO implementation is InMemoryFileIO,
+ // which does not support remote signing.
+ org.apache.iceberg.CatalogProperties.FILE_IO_IMPL,
+ "org.apache.iceberg.io.ResolvingFileIO",
+})
+public abstract class PolarisS3RemoteSigningIntegrationTest
+ extends PolarisRestCatalogAwsIntegrationTest {
+
+ @Override
+ protected ImmutableMap.Builder<String, String> clientFileIOProperties() {
+ ImmutableMap.Builder<String, String> builder =
+ super.clientFileIOProperties()
+ .put(AWS_PATH_STYLE_ACCESS.getPropertyName(), String.valueOf(pathStyleAccess()));
+ endpoint().ifPresent(endpoint -> builder.put(AWS_ENDPOINT.getPropertyName(), endpoint));
+ return builder;
+ }
+
+ /**
+ * Override this method to return true if the S3 client should use path-style access. Default is
+ * false, which means virtual-hosted-style access will be used.
+ */
+ protected boolean pathStyleAccess() {
+ return false;
+ }
+
+ @CatalogConfig(properties = {"polaris.config.remote-signing.enabled", "false"})
+ @Test
+ public void testInternalCatalogRemoteSigningDisabled() {
+ @SuppressWarnings("resource")
+ RESTCatalog catalog = catalog();
+ Namespace ns1 = Namespace.of("ns1");
+ catalog.createNamespace(ns1);
+ TableIdentifier tableIdentifier = TableIdentifier.of(ns1, "my_table");
+ assertThatThrownBy(() -> catalog.createTable(tableIdentifier, SCHEMA))
+ .isInstanceOf(ForbiddenException.class)
+ .hasMessageContaining("Remote signing is not enabled for this catalog")
+ .hasMessageContaining(FeatureConfiguration.REMOTE_SIGNING_ENABLED.key())
+ .hasMessageContaining(FeatureConfiguration.REMOTE_SIGNING_ENABLED.catalogConfig());
+ }
+
+ @CatalogConfig(Catalog.TypeEnum.EXTERNAL)
+ @Test
+ public void testExternalCatalogRemoteSigningDisabled() {
+ @SuppressWarnings("resource")
+ RESTCatalog catalog = catalog();
+ Namespace ns1 = Namespace.of("ns1");
+ catalog.createNamespace(ns1);
+ TableMetadata tableMetadata =
+ TableMetadata.newTableMetadata(
+ SCHEMA,
+ PartitionSpec.unpartitioned(),
+ externalCatalogBaseLocation() + "/ns1/my_table",
+ Map.of());
+ try (ResolvingFileIO resolvingFileIO = initializeClientFileIO(new ResolvingFileIO())) {
+ String fileLocation =
+ externalCatalogBaseLocation() + "/ns1/my_table/metadata/v1.metadata.json";
+ TableMetadataParser.write(tableMetadata, resolvingFileIO.newOutputFile(fileLocation));
+ catalog.registerTable(TableIdentifier.of(ns1, "my_table"), fileLocation);
+ try {
+ assertThatThrownBy(() -> catalog.loadTable(TableIdentifier.of(ns1, "my_table")))
+ .isInstanceOf(ForbiddenException.class)
+ .hasMessageContaining("Remote signing is not enabled for external catalogs")
+ .hasMessageContaining(
+ FeatureConfiguration.REMOTE_SIGNING_EXTERNAL_CATALOGS_ENABLED.key());
+ } finally {
+ resolvingFileIO.deleteFile(fileLocation);
+ }
+ }
+ }
+
+ @Test
+ @Override
+ @Disabled("It's not possible to request an access delegation mode when registering a table.")
+ public void testRegisterTable() {
Review Comment:
Created an issue: https://github.com/apache/polaris/issues/2351
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2284668981
##########
runtime/service/src/main/java/org/apache/polaris/service/storage/aws/signer/S3RemoteSigningCatalogHandler.java:
##########
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.service.storage.aws.signer;
+
+import jakarta.ws.rs.core.SecurityContext;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.apache.polaris.core.auth.PolarisAuthorizableOperation;
+import org.apache.polaris.core.auth.PolarisAuthorizer;
+import org.apache.polaris.core.config.FeatureConfiguration;
+import org.apache.polaris.core.config.RealmConfig;
+import org.apache.polaris.core.context.CallContext;
+import org.apache.polaris.core.entity.CatalogEntity;
+import org.apache.polaris.core.entity.PolarisEntitySubType;
+import org.apache.polaris.core.persistence.resolver.ResolutionManifestFactory;
+import org.apache.polaris.service.aws.sign.model.PolarisS3SignRequest;
+import org.apache.polaris.service.aws.sign.model.PolarisS3SignResponse;
+import org.apache.polaris.service.catalog.common.CatalogHandler;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class S3RemoteSigningCatalogHandler extends CatalogHandler implements AutoCloseable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(S3RemoteSigningCatalogHandler.class);
+
+ private final S3RequestSigner s3RequestSigner;
+
+ private CatalogEntity catalogEntity;
+
+ public S3RemoteSigningCatalogHandler(
+ CallContext callContext,
+ ResolutionManifestFactory resolutionManifestFactory,
+ SecurityContext securityContext,
+ String catalogName,
+ PolarisAuthorizer authorizer,
+ S3RequestSigner s3RequestSigner) {
+ super(callContext, resolutionManifestFactory, securityContext, catalogName, authorizer);
+ this.s3RequestSigner = s3RequestSigner;
+ }
+
+ @Override
+ protected void initializeCatalog() {
+ catalogEntity =
+ CatalogEntity.of(resolutionManifest.getResolvedReferenceCatalogEntity().getRawLeafEntity());
+ if (catalogEntity.isExternal()) {
+ throw new ForbiddenException("Cannot use S3 remote signing with federated catalogs.");
+ }
+ // no need to materialize the catalog here, as we only need the catalog entity
+ }
+
+ public PolarisS3SignResponse signS3Request(
Review Comment:
Not anymore, I refactored this bit to comply with the design doc.
For a read request you need `TABLE_REMOTE_SIGN` + `TABLE_READ_DATA`.
For a write request you need `TABLE_REMOTE_SIGN` + `TABLE_WRITE_DATA`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2503692507
##########
polaris-core/src/main/java/org/apache/polaris/core/rest/PolarisResourcePaths.java:
##########
@@ -78,4 +82,15 @@ public String genericTable(TableIdentifier ident) {
"generic-tables",
RESTUtil.encodeString(ident.name()));
}
+
+ public String s3RemoteSigning(TableIdentifier ident) {
Review Comment:
Well the same could be said for all the methods in this class 😄
The package is `org.apache.polaris.core.rest` btw - so I guess REST is a core concern 🤷♂️
I think `PolarisResourcePaths` and `PolarisEndpoints` should not live in `core` ideally. But that's the way it is today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2504125992
##########
polaris-core/src/main/java/org/apache/polaris/core/rest/PolarisResourcePaths.java:
##########
@@ -78,4 +82,15 @@ public String genericTable(TableIdentifier ident) {
"generic-tables",
RESTUtil.encodeString(ident.name()));
}
+
+ public String s3RemoteSigning(TableIdentifier ident) {
Review Comment:
oops, I did not actually look at the whole class, only at the diff :sweat_smile:
Let's keep it for now and maybe refactor later.... In general, I'd like to untangle core from REST :sweat_smile:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on PR #2280:
URL: https://github.com/apache/polaris/pull/2280#issuecomment-3499097022
The proposed phased approach (M1/M2) SGTM :+1:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on PR #2280:
URL: https://github.com/apache/polaris/pull/2280#issuecomment-3474272840
Switching back to draft, sorry.
I finally managed to adapt this PR to the AccessConfig refactoring, but I still need to solve a few FIXMEs and wrap my head around the new design.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2541982321
##########
CHANGELOG.md:
##########
@@ -36,6 +36,14 @@ request adding CHANGELOG notes for breaking (!) changes and possibly other secti
[Iceberg Metrics Reporting]: https://iceberg.apache.org/docs/latest/metrics-reporting/
+- **S3 remote request signing** has been added, allowing Polaris to work with S3-compatible object storage systems.
+ *Remote signing is currently experimental and not enabled by default*. In particular, RBAC checks are currently not
+ production-ready. One new table privilege was introduced: `TABLE_REMOTE_SIGN`. To enable remote signing:
+ 1. Set the system-wide property `REMOTE_SIGNING_ENABLED` or the catalog-level `polaris.request-signing.enabled`
+ property to `true`.
+ 2. Grant the `TABLE_REMOTE_SIGN` privilege to a catalog role. The role must also be granted the `TABLE_READ_DATA`
+ and `TABLE_WRITE_DATA` privileges.
Review Comment:
This is done by Polaris RBAC enforcement. The two `PoalrisAuthorizableOperation` instances used in remote signing are declared as follows:
```java
SIGN_S3_READ_REQUEST(EnumSet.of(TABLE_REMOTE_SIGN, TABLE_READ_DATA)),
SIGN_S3_WRITE_REQUEST(EnumSet.of(TABLE_REMOTE_SIGN, TABLE_WRITE_DATA)),
```
Thus as you can see to authorize these operations, the role must have been granted 2 privileges: `TABLE_REMOTE_SIGN` and either `TABLE_READ_DATA` (for reads) or `TABLE_WRITE_DATA` (for writes).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2511648804
##########
CHANGELOG.md:
##########
@@ -36,6 +36,14 @@ request adding CHANGELOG notes for breaking (!) changes and possibly other secti
[Iceberg Metrics Reporting]: https://iceberg.apache.org/docs/latest/metrics-reporting/
+- **S3 remote request signing** has been added, allowing Polaris to work with S3-compatible object storage systems.
+ *Remote signing is currently experimental and not enabled by default*. In particular, RBAC checks are currently not
+ production-ready. One new table privilege was introduced: `TABLE_REMOTE_SIGN`. To enable remote signing:
+ 1. Set the system-wide property `REMOTE_SIGNING_ENABLED` or the catalog-level `polaris.request-signing.enabled`
+ property to `true`.
+ 2. Grant the `TABLE_REMOTE_SIGN` privilege to a catalog role. The role must also be granted the `TABLE_READ_DATA`
+ and `TABLE_WRITE_DATA` privileges.
Review Comment:
> is there an assertion when even TABLE_REMOTE_SIGN is granted these two privieldges would be there ?
Currently no.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2511874553
##########
runtime/service/src/main/java/org/apache/polaris/service/storage/s3/sign/S3RemoteSigningCatalogHandler.java:
##########
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.service.storage.s3.sign;
+
+import java.util.Collection;
+import java.util.EnumSet;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.auth.PolarisAuthorizableOperation;
+import org.apache.polaris.core.auth.PolarisAuthorizer;
+import org.apache.polaris.core.auth.PolarisPrincipal;
+import org.apache.polaris.core.config.FeatureConfiguration;
+import org.apache.polaris.core.config.RealmConfig;
+import org.apache.polaris.core.context.CallContext;
+import org.apache.polaris.core.entity.CatalogEntity;
+import org.apache.polaris.core.entity.PolarisEntity;
+import org.apache.polaris.core.entity.PolarisEntityConstants;
+import org.apache.polaris.core.persistence.PolarisResolvedPathWrapper;
+import org.apache.polaris.core.persistence.resolver.ResolutionManifestFactory;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisStorageActions;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.apache.polaris.service.catalog.common.CatalogHandler;
+import org.apache.polaris.service.catalog.common.CatalogUtils;
+import org.apache.polaris.service.catalog.io.FileIOUtil;
+import org.apache.polaris.service.context.catalog.CallContextCatalogFactory;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignRequest;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignResponse;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class S3RemoteSigningCatalogHandler extends CatalogHandler implements AutoCloseable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(S3RemoteSigningCatalogHandler.class);
+
+ private final CallContextCatalogFactory catalogFactory;
+ private final S3RequestSigner s3RequestSigner;
+
+ private CatalogEntity catalogEntity;
+ private Catalog baseCatalog;
+
+ public S3RemoteSigningCatalogHandler(
+ PolarisDiagnostics diagnostics,
+ CallContext callContext,
+ ResolutionManifestFactory resolutionManifestFactory,
+ CallContextCatalogFactory catalogFactory,
+ PolarisPrincipal polarisPrincipal,
+ String catalogName,
+ PolarisAuthorizer authorizer,
+ S3RequestSigner s3RequestSigner) {
+ super(
+ diagnostics,
+ callContext,
+ resolutionManifestFactory,
+ polarisPrincipal,
+ catalogName,
+ authorizer,
+ // external catalogs are not supported for S3 remote signing
+ null,
+ null);
+ this.catalogFactory = catalogFactory;
+ this.s3RequestSigner = s3RequestSigner;
+ }
+
+ @Override
+ protected void initializeCatalog() {
+ catalogEntity =
+ CatalogEntity.of(resolutionManifest.getResolvedReferenceCatalogEntity().getRawLeafEntity());
+ if (catalogEntity.isExternal()) {
+ throw new ForbiddenException("Cannot use S3 remote signing with federated catalogs.");
+ }
+ baseCatalog =
+ catalogFactory.createCallContextCatalog(callContext, polarisPrincipal, resolutionManifest);
+ }
+
+ public PolarisS3SignResponse signS3Request(
+ PolarisS3SignRequest s3SignRequest, TableIdentifier tableIdentifier) {
+
+ LOGGER.debug("Requesting s3 signing for {}: {}", tableIdentifier, s3SignRequest);
+
+ PolarisAuthorizableOperation authzOp =
+ s3SignRequest.write()
+ ? PolarisAuthorizableOperation.SIGN_S3_WRITE_REQUEST
+ : PolarisAuthorizableOperation.SIGN_S3_READ_REQUEST;
+
+ authorizeRemoteSigningOrThrow(EnumSet.of(authzOp), tableIdentifier);
+
+ // Must be done after the authorization check, as the auth check creates the catalog entity
+ throwIfRemoteSigningNotEnabled(callContext.getRealmConfig(), catalogEntity);
+
+ var result =
+ InMemoryStorageIntegration.validateAllowedLocations(
+ callContext.getRealmConfig(),
+ getAllowedLocations(tableIdentifier),
+ getStorageActions(s3SignRequest),
+ getTargetLocations(s3SignRequest));
+
+ if (result.values().stream().anyMatch(r -> r.values().stream().anyMatch(v -> !v.isSuccess()))) {
+ throw new ForbiddenException("Requested S3 location is not allowed.");
+ }
+
+ PolarisS3SignResponse s3SignResponse = s3RequestSigner.signRequest(s3SignRequest);
+ LOGGER.debug("S3 signing response: {}", s3SignResponse);
+
+ return s3SignResponse;
+ }
+
+ private Collection<String> getAllowedLocations(TableIdentifier tableIdentifier) {
+
+ if (baseCatalog.tableExists(tableIdentifier)) {
+
+ // If the table exists, get allowed locations from the table metadata
+ Table table = baseCatalog.loadTable(tableIdentifier);
+ if (table instanceof BaseTable baseTable) {
+ return StorageUtil.getLocationsUsedByTable(baseTable.operations().current());
+ }
+
+ throw new ForbiddenException("No storage configuration found for table.");
+
+ } else {
+
+ // If the table or view doesn't exist, the engine might be writing the manifests before the
+ // table creation is committed. In this case, we still need to check allowed locations from
+ // the parent entities.
Review Comment:
That's a very good question, let me check if we have tests for staged table creation + remote signing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2504129488
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/StorageAccessProperty.java:
##########
@@ -52,6 +54,21 @@ public enum StorageAccessProperty {
"the endpoint to load vended credentials for a table from the catalog",
false,
false),
+ AWS_REMOTE_SIGNING_ENABLED(
+ Boolean.class,
+ S3FileIOProperties.REMOTE_SIGNING_ENABLED,
+ "whether to enable remote signing for S3 requests",
+ false),
+ AWS_REMOTE_SIGNER_URI(
+ String.class,
+ S3V4RestSignerClient.S3_SIGNER_URI,
+ "the base URI for the remote signer service, used for signing S3 requests",
Review Comment:
That's fine... I was worries it might have been more recent ;)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2511782042
##########
runtime/service/src/main/java/org/apache/polaris/service/storage/s3/sign/S3RemoteSigningCatalogHandler.java:
##########
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.service.storage.s3.sign;
+
+import java.util.Collection;
+import java.util.EnumSet;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.auth.PolarisAuthorizableOperation;
+import org.apache.polaris.core.auth.PolarisAuthorizer;
+import org.apache.polaris.core.auth.PolarisPrincipal;
+import org.apache.polaris.core.config.FeatureConfiguration;
+import org.apache.polaris.core.config.RealmConfig;
+import org.apache.polaris.core.context.CallContext;
+import org.apache.polaris.core.entity.CatalogEntity;
+import org.apache.polaris.core.entity.PolarisEntity;
+import org.apache.polaris.core.entity.PolarisEntityConstants;
+import org.apache.polaris.core.persistence.PolarisResolvedPathWrapper;
+import org.apache.polaris.core.persistence.resolver.ResolutionManifestFactory;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisStorageActions;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.apache.polaris.service.catalog.common.CatalogHandler;
+import org.apache.polaris.service.catalog.common.CatalogUtils;
+import org.apache.polaris.service.catalog.io.FileIOUtil;
+import org.apache.polaris.service.context.catalog.CallContextCatalogFactory;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignRequest;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignResponse;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class S3RemoteSigningCatalogHandler extends CatalogHandler implements AutoCloseable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(S3RemoteSigningCatalogHandler.class);
+
+ private final CallContextCatalogFactory catalogFactory;
+ private final S3RequestSigner s3RequestSigner;
+
+ private CatalogEntity catalogEntity;
+ private Catalog baseCatalog;
+
+ public S3RemoteSigningCatalogHandler(
+ PolarisDiagnostics diagnostics,
+ CallContext callContext,
+ ResolutionManifestFactory resolutionManifestFactory,
+ CallContextCatalogFactory catalogFactory,
+ PolarisPrincipal polarisPrincipal,
+ String catalogName,
+ PolarisAuthorizer authorizer,
+ S3RequestSigner s3RequestSigner) {
+ super(
+ diagnostics,
+ callContext,
+ resolutionManifestFactory,
+ polarisPrincipal,
+ catalogName,
+ authorizer,
+ // external catalogs are not supported for S3 remote signing
+ null,
+ null);
+ this.catalogFactory = catalogFactory;
+ this.s3RequestSigner = s3RequestSigner;
+ }
+
+ @Override
+ protected void initializeCatalog() {
+ catalogEntity =
+ CatalogEntity.of(resolutionManifest.getResolvedReferenceCatalogEntity().getRawLeafEntity());
+ if (catalogEntity.isExternal()) {
+ throw new ForbiddenException("Cannot use S3 remote signing with federated catalogs.");
+ }
+ baseCatalog =
+ catalogFactory.createCallContextCatalog(callContext, polarisPrincipal, resolutionManifest);
+ }
+
+ public PolarisS3SignResponse signS3Request(
+ PolarisS3SignRequest s3SignRequest, TableIdentifier tableIdentifier) {
+
+ LOGGER.debug("Requesting s3 signing for {}: {}", tableIdentifier, s3SignRequest);
+
+ PolarisAuthorizableOperation authzOp =
+ s3SignRequest.write()
+ ? PolarisAuthorizableOperation.SIGN_S3_WRITE_REQUEST
+ : PolarisAuthorizableOperation.SIGN_S3_READ_REQUEST;
+
+ authorizeRemoteSigningOrThrow(EnumSet.of(authzOp), tableIdentifier);
+
+ // Must be done after the authorization check, as the auth check creates the catalog entity
+ throwIfRemoteSigningNotEnabled(callContext.getRealmConfig(), catalogEntity);
+
+ var result =
+ InMemoryStorageIntegration.validateAllowedLocations(
+ callContext.getRealmConfig(),
+ getAllowedLocations(tableIdentifier),
+ getStorageActions(s3SignRequest),
+ getTargetLocations(s3SignRequest));
+
+ if (result.values().stream().anyMatch(r -> r.values().stream().anyMatch(v -> !v.isSuccess()))) {
+ throw new ForbiddenException("Requested S3 location is not allowed.");
+ }
+
+ PolarisS3SignResponse s3SignResponse = s3RequestSigner.signRequest(s3SignRequest);
+ LOGGER.debug("S3 signing response: {}", s3SignResponse);
+
+ return s3SignResponse;
+ }
+
+ private Collection<String> getAllowedLocations(TableIdentifier tableIdentifier) {
+
+ if (baseCatalog.tableExists(tableIdentifier)) {
+
+ // If the table exists, get allowed locations from the table metadata
+ Table table = baseCatalog.loadTable(tableIdentifier);
+ if (table instanceof BaseTable baseTable) {
+ return StorageUtil.getLocationsUsedByTable(baseTable.operations().current());
+ }
+
+ throw new ForbiddenException("No storage configuration found for table.");
+
+ } else {
+
+ // If the table or view doesn't exist, the engine might be writing the manifests before the
+ // table creation is committed. In this case, we still need to check allowed locations from
+ // the parent entities.
+
+ PolarisResolvedPathWrapper resolvedPath =
+ CatalogUtils.findResolvedStorageEntity(resolutionManifest, tableIdentifier);
+
+ Optional<PolarisEntity> storageInfo = FileIOUtil.findStorageInfoFromHierarchy(resolvedPath);
+
+ var configurationInfo =
+ storageInfo
+ .map(PolarisEntity::getInternalPropertiesAsMap)
+ .map(info -> info.get(PolarisEntityConstants.getStorageConfigInfoPropertyName()))
+ .map(PolarisStorageConfigurationInfo::deserialize);
+
+ if (configurationInfo.isEmpty()) {
+ throw new ForbiddenException("No storage configuration found for table.");
+ }
+
+ return configurationInfo.get().getAllowedLocations();
+ }
+ }
+
+ private Set<PolarisStorageActions> getStorageActions(PolarisS3SignRequest s3SignRequest) {
+ // TODO M2: better handling of DELETE and LIST
+ return s3SignRequest.write()
+ ? Set.of(PolarisStorageActions.WRITE)
+ : Set.of(PolarisStorageActions.READ);
+ }
+
+ private Set<String> getTargetLocations(PolarisS3SignRequest s3SignRequest) {
+ // TODO M2: map http URI to s3 URI
+ return Set.of();
+ }
+
+ public static void throwIfRemoteSigningNotEnabled(
+ RealmConfig realmConfig, CatalogEntity catalogEntity) {
+ if (catalogEntity.isExternal()) {
+ throw new ForbiddenException("Remote signing is not enabled for external catalogs.");
+ }
Review Comment:
Totally agree, I think we could discuss this possibility later on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2503703127
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/StorageAccessProperty.java:
##########
@@ -52,6 +54,21 @@ public enum StorageAccessProperty {
"the endpoint to load vended credentials for a table from the catalog",
false,
false),
+ AWS_REMOTE_SIGNING_ENABLED(
+ Boolean.class,
+ S3FileIOProperties.REMOTE_SIGNING_ENABLED,
+ "whether to enable remote signing for S3 requests",
+ false),
+ AWS_REMOTE_SIGNER_URI(
+ String.class,
+ S3V4RestSignerClient.S3_SIGNER_URI,
+ "the base URI for the remote signer service, used for signing S3 requests",
Review Comment:
Iceberg 1.2.0. That's more than 2 years old, I'm not sure we need to mention it here. Wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2503670005
##########
runtime/service/src/main/java/org/apache/polaris/service/catalog/io/AccessConfigProvider.java:
##########
@@ -101,4 +120,58 @@ public AccessConfig getAccessConfig(
storageInfo.get(),
refreshCredentialsEndpoint);
}
+
+ /**
+ * Generates a remote signing configuration for accessing table storage at explicit locations.
+ *
+ * @param callContext the call context containing realm, principal, and security context
+ * @param catalogName the name of the catalog
+ * @param tableIdentifier the table identifier, used for logging and refresh endpoint construction
+ * @param resolvedPath the entity hierarchy to search for storage configuration
+ * @return {@link AccessConfig} with scoped credentials and metadata; empty if no storage config
+ * found
+ */
+ public AccessConfig getAccessConfigForRemoteSigning(
+ @Nonnull CallContext callContext,
+ @Nonnull String catalogName,
+ @Nonnull TableIdentifier tableIdentifier,
+ @Nonnull PolarisResolvedPathWrapper resolvedPath) {
+ LOGGER
+ .atDebug()
+ .addKeyValue("tableIdentifier", tableIdentifier)
+ .log("Fetching remote signing config for table");
+
+ Optional<PolarisEntity> storageInfo = FileIOUtil.findStorageInfoFromHierarchy(resolvedPath);
+ Optional<PolarisStorageConfigurationInfo> configurationInfo =
+ storageInfo
+ .map(PolarisEntity::getInternalPropertiesAsMap)
+ .map(info -> info.get(PolarisEntityConstants.getStorageConfigInfoPropertyName()))
+ .map(PolarisStorageConfigurationInfo::deserialize);
+
+ if (configurationInfo.isEmpty()) {
+ LOGGER
+ .atWarn()
+ .addKeyValue("tableIdentifier", tableIdentifier)
+ .log("Table entity has no storage configuration in its hierarchy");
+ return AccessConfig.EMPTY;
+ }
+
+ PolarisStorageIntegration<AwsStorageConfigurationInfo> storageIntegration =
+ storageIntegrationProvider.getStorageIntegrationForConfig(configurationInfo.get());
+
+ if (!(storageIntegration
+ instanceof AwsCredentialsStorageIntegration awsCredentialsStorageIntegration)) {
+ LOGGER
+ .atWarn()
+ .addKeyValue("tableIdentifier", tableIdentifier)
+ .log("Table entity storage integration is not an AWS credentials storage integration");
+ return AccessConfig.EMPTY;
+ }
+
+ String prefix = prefixParser.catalogNameToPrefix(callContext.getRealmContext(), catalogName);
+ URI signerUri = uriInfo.getBaseUri().resolve("api/");
Review Comment:
I agree, but I'm not sure a constant in `s3-sign-service` is the best solution. The `api/` path segment doesn't seem to be a first-class citizen in our APIs, rather a bi-product. It's only declared as a server variable in the OpenAPI generator config, for example:
https://github.com/apache/polaris/blob/1f4e748a96b926cd1639d6ed463997d2f9c455c4/api/iceberg-service/build.gradle.kts#L86
https://github.com/apache/polaris/blob/b6e247deda8a618136ec16c75b86eb2a2d12642b/api/polaris-catalog-service/build.gradle.kts#L116
The management APi is actually a bit different, there is not `basePath` variable in polaris-management-service.yml, so the base path is hard-coded:
https://github.com/apache/polaris/blob/04c8ee6972df948beb15401e8dc905a37b27c33d/spec/polaris-management-service.yml#L26
But the common characteristic is that `api/`is the first path segment of all Polaris APIs, but that is not enforced by any constraint.
I would say the right place for this segment would be in `PolarisResourcePaths`, but you will notice again that none of the paths include the `api/` segment. Same for Iceberg's `ResourcePaths`. So it's really implicit 😅
All of this to say, let's declare a constant in `PolarisResourcePaths` and call it a day, wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "snazy (via GitHub)" <gi...@apache.org>.
snazy commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2259531408
##########
runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogAdapter.java:
##########
@@ -797,4 +812,24 @@ public Response getConfig(
.build())
.build();
}
+
+ @Override
+ public Response signS3Request(
Review Comment:
I think it makes sense to have this in a separate class to avoid the overhead of having unneeded things just for signing purposes. We can emit a separate signing-endpoint in a load-table, if necessary.
##########
runtime/service/src/main/java/org/apache/polaris/service/storage/aws/signer/S3RequestSignerImpl.java:
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.service.storage.aws.signer;
+
+import java.net.URI;
+import java.util.Set;
+import org.apache.iceberg.aws.s3.signer.ImmutableS3SignResponse;
+import org.apache.iceberg.aws.s3.signer.S3SignResponse;
+import org.apache.polaris.core.config.RealmConfig;
+import org.apache.polaris.core.storage.AccessConfig;
+import org.apache.polaris.core.storage.StorageAccessProperty;
+import org.apache.polaris.core.storage.aws.AwsCredentialsStorageIntegration;
+import org.apache.polaris.core.storage.aws.AwsStorageConfigurationInfo;
+import org.apache.polaris.core.storage.cache.StorageCredentialCache;
+import org.apache.polaris.service.types.S3SignRequest;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.AwsCredentials;
+import software.amazon.awssdk.auth.credentials.AwsSessionCredentials;
+import software.amazon.awssdk.http.ContentStreamProvider;
+import software.amazon.awssdk.http.SdkHttpFullRequest;
+import software.amazon.awssdk.http.SdkHttpMethod;
+import software.amazon.awssdk.http.SdkHttpRequest;
+import software.amazon.awssdk.http.auth.aws.signer.AwsV4HttpSigner;
+import software.amazon.awssdk.http.auth.spi.signer.SignRequest;
+import software.amazon.awssdk.http.auth.spi.signer.SignedRequest;
+import software.amazon.awssdk.services.s3.S3Client;
+
+/**
+ * S3 request signer that creates presigned URLs for S3 operations using AWS credentials obtained
+ * from the storage integration.
+ */
+public class S3RequestSignerImpl implements S3RequestSigner {
Review Comment:
```suggestion
class S3RequestSignerImpl implements S3RequestSigner {
```
##########
runtime/service/src/main/java/org/apache/polaris/service/storage/aws/signer/S3RequestSignerImpl.java:
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.service.storage.aws.signer;
+
+import java.net.URI;
+import java.util.Set;
+import org.apache.iceberg.aws.s3.signer.ImmutableS3SignResponse;
+import org.apache.iceberg.aws.s3.signer.S3SignResponse;
+import org.apache.polaris.core.config.RealmConfig;
+import org.apache.polaris.core.storage.AccessConfig;
+import org.apache.polaris.core.storage.StorageAccessProperty;
+import org.apache.polaris.core.storage.aws.AwsCredentialsStorageIntegration;
+import org.apache.polaris.core.storage.aws.AwsStorageConfigurationInfo;
+import org.apache.polaris.core.storage.cache.StorageCredentialCache;
+import org.apache.polaris.service.types.S3SignRequest;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.AwsCredentials;
+import software.amazon.awssdk.auth.credentials.AwsSessionCredentials;
+import software.amazon.awssdk.http.ContentStreamProvider;
+import software.amazon.awssdk.http.SdkHttpFullRequest;
+import software.amazon.awssdk.http.SdkHttpMethod;
+import software.amazon.awssdk.http.SdkHttpRequest;
+import software.amazon.awssdk.http.auth.aws.signer.AwsV4HttpSigner;
+import software.amazon.awssdk.http.auth.spi.signer.SignRequest;
+import software.amazon.awssdk.http.auth.spi.signer.SignedRequest;
+import software.amazon.awssdk.services.s3.S3Client;
+
+/**
+ * S3 request signer that creates presigned URLs for S3 operations using AWS credentials obtained
+ * from the storage integration.
+ */
+public class S3RequestSignerImpl implements S3RequestSigner {
+
+ private final AwsV4HttpSigner signer = AwsV4HttpSigner.create();
+
+ private final AwsCredentialsStorageIntegration storageIntegration;
+ private final StorageCredentialCache storageCredentialCache;
+
+ public S3RequestSignerImpl(
+ AwsCredentialsStorageIntegration storageIntegration,
+ StorageCredentialCache storageCredentialCache) {
+ this.storageIntegration = storageIntegration;
+ this.storageCredentialCache = storageCredentialCache;
+ }
+
+ @Override
+ public S3SignResponse signRequest(
+ RealmConfig realmConfig,
+ AwsStorageConfigurationInfo storageConfig,
+ S3SignRequest signingRequest) {
+
+ SdkHttpMethod method = SdkHttpMethod.valueOf(signingRequest.method());
+ URI uri = signingRequest.uri();
+ String body = signingRequest.body();
+
+ SdkHttpFullRequest.Builder requestToSign =
+ SdkHttpFullRequest.builder()
+ .uri(uri)
+ .protocol(uri.getScheme())
+ .method(method)
+ .headers(signingRequest.headers());
+
+ // FIXME is this correct?
+ boolean allowListOperation = isReadMethod(method);
+ Set<String> allowedReadLocations = isReadMethod(method) ? Set.of(uri.toString()) : Set.of();
+ Set<String> allowedWriteLocations = isWriteMethod(method) ? Set.of(uri.toString()) : Set.of();
+
+ // TODO use the storageCredentialCache to get or generate subscoped credentials
Review Comment:
how? request-signing is (mostly) used in environments that don't have sts.
##########
runtime/service/src/main/java/org/apache/polaris/service/storage/aws/signer/S3RequestSignerImpl.java:
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.service.storage.aws.signer;
+
+import java.net.URI;
+import java.util.Set;
+import org.apache.iceberg.aws.s3.signer.ImmutableS3SignResponse;
+import org.apache.iceberg.aws.s3.signer.S3SignResponse;
+import org.apache.polaris.core.config.RealmConfig;
+import org.apache.polaris.core.storage.AccessConfig;
+import org.apache.polaris.core.storage.StorageAccessProperty;
+import org.apache.polaris.core.storage.aws.AwsCredentialsStorageIntegration;
+import org.apache.polaris.core.storage.aws.AwsStorageConfigurationInfo;
+import org.apache.polaris.core.storage.cache.StorageCredentialCache;
+import org.apache.polaris.service.types.S3SignRequest;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.AwsCredentials;
+import software.amazon.awssdk.auth.credentials.AwsSessionCredentials;
+import software.amazon.awssdk.http.ContentStreamProvider;
+import software.amazon.awssdk.http.SdkHttpFullRequest;
+import software.amazon.awssdk.http.SdkHttpMethod;
+import software.amazon.awssdk.http.SdkHttpRequest;
+import software.amazon.awssdk.http.auth.aws.signer.AwsV4HttpSigner;
+import software.amazon.awssdk.http.auth.spi.signer.SignRequest;
+import software.amazon.awssdk.http.auth.spi.signer.SignedRequest;
+import software.amazon.awssdk.services.s3.S3Client;
+
+/**
+ * S3 request signer that creates presigned URLs for S3 operations using AWS credentials obtained
+ * from the storage integration.
+ */
+public class S3RequestSignerImpl implements S3RequestSigner {
+
+ private final AwsV4HttpSigner signer = AwsV4HttpSigner.create();
+
+ private final AwsCredentialsStorageIntegration storageIntegration;
+ private final StorageCredentialCache storageCredentialCache;
+
+ public S3RequestSignerImpl(
Review Comment:
```suggestion
S3RequestSignerImpl(
```
##########
runtime/service/src/main/java/org/apache/polaris/service/storage/aws/signer/S3RequestSignerImpl.java:
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.service.storage.aws.signer;
+
+import java.net.URI;
+import java.util.Set;
+import org.apache.iceberg.aws.s3.signer.ImmutableS3SignResponse;
+import org.apache.iceberg.aws.s3.signer.S3SignResponse;
+import org.apache.polaris.core.config.RealmConfig;
+import org.apache.polaris.core.storage.AccessConfig;
+import org.apache.polaris.core.storage.StorageAccessProperty;
+import org.apache.polaris.core.storage.aws.AwsCredentialsStorageIntegration;
+import org.apache.polaris.core.storage.aws.AwsStorageConfigurationInfo;
+import org.apache.polaris.core.storage.cache.StorageCredentialCache;
+import org.apache.polaris.service.types.S3SignRequest;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.AwsCredentials;
+import software.amazon.awssdk.auth.credentials.AwsSessionCredentials;
+import software.amazon.awssdk.http.ContentStreamProvider;
+import software.amazon.awssdk.http.SdkHttpFullRequest;
+import software.amazon.awssdk.http.SdkHttpMethod;
+import software.amazon.awssdk.http.SdkHttpRequest;
+import software.amazon.awssdk.http.auth.aws.signer.AwsV4HttpSigner;
+import software.amazon.awssdk.http.auth.spi.signer.SignRequest;
+import software.amazon.awssdk.http.auth.spi.signer.SignedRequest;
+import software.amazon.awssdk.services.s3.S3Client;
+
+/**
+ * S3 request signer that creates presigned URLs for S3 operations using AWS credentials obtained
+ * from the storage integration.
+ */
+public class S3RequestSignerImpl implements S3RequestSigner {
+
+ private final AwsV4HttpSigner signer = AwsV4HttpSigner.create();
+
+ private final AwsCredentialsStorageIntegration storageIntegration;
+ private final StorageCredentialCache storageCredentialCache;
+
+ public S3RequestSignerImpl(
+ AwsCredentialsStorageIntegration storageIntegration,
+ StorageCredentialCache storageCredentialCache) {
+ this.storageIntegration = storageIntegration;
+ this.storageCredentialCache = storageCredentialCache;
+ }
+
+ @Override
+ public S3SignResponse signRequest(
+ RealmConfig realmConfig,
+ AwsStorageConfigurationInfo storageConfig,
+ S3SignRequest signingRequest) {
+
+ SdkHttpMethod method = SdkHttpMethod.valueOf(signingRequest.method());
+ URI uri = signingRequest.uri();
+ String body = signingRequest.body();
+
+ SdkHttpFullRequest.Builder requestToSign =
+ SdkHttpFullRequest.builder()
+ .uri(uri)
+ .protocol(uri.getScheme())
+ .method(method)
+ .headers(signingRequest.headers());
+
+ // FIXME is this correct?
Review Comment:
nope.
See [ListObjects](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html) + [ListObjectsV2](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html) REST spec.
*Be careful:* Not all `GET /{bucket/` requests are list-objects requests!
The "v1" list-objects has no required parameter, so you cannot just check for the presence of a req param. It could also be a get-bucket-cors or get-bucket-encryption or ...-policy or ...-website or ......
Nice, no?
_Maybe_ it's okay to assume that every client uses the v2 api, which has a unique, required request param.
##########
runtime/service/src/main/java/org/apache/polaris/service/storage/aws/signer/S3RequestSignerImpl.java:
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.service.storage.aws.signer;
+
+import java.net.URI;
+import java.util.Set;
+import org.apache.iceberg.aws.s3.signer.ImmutableS3SignResponse;
+import org.apache.iceberg.aws.s3.signer.S3SignResponse;
+import org.apache.polaris.core.config.RealmConfig;
+import org.apache.polaris.core.storage.AccessConfig;
+import org.apache.polaris.core.storage.StorageAccessProperty;
+import org.apache.polaris.core.storage.aws.AwsCredentialsStorageIntegration;
+import org.apache.polaris.core.storage.aws.AwsStorageConfigurationInfo;
+import org.apache.polaris.core.storage.cache.StorageCredentialCache;
+import org.apache.polaris.service.types.S3SignRequest;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.AwsCredentials;
+import software.amazon.awssdk.auth.credentials.AwsSessionCredentials;
+import software.amazon.awssdk.http.ContentStreamProvider;
+import software.amazon.awssdk.http.SdkHttpFullRequest;
+import software.amazon.awssdk.http.SdkHttpMethod;
+import software.amazon.awssdk.http.SdkHttpRequest;
+import software.amazon.awssdk.http.auth.aws.signer.AwsV4HttpSigner;
+import software.amazon.awssdk.http.auth.spi.signer.SignRequest;
+import software.amazon.awssdk.http.auth.spi.signer.SignedRequest;
+import software.amazon.awssdk.services.s3.S3Client;
+
+/**
+ * S3 request signer that creates presigned URLs for S3 operations using AWS credentials obtained
+ * from the storage integration.
+ */
+public class S3RequestSignerImpl implements S3RequestSigner {
+
+ private final AwsV4HttpSigner signer = AwsV4HttpSigner.create();
+
+ private final AwsCredentialsStorageIntegration storageIntegration;
+ private final StorageCredentialCache storageCredentialCache;
+
+ public S3RequestSignerImpl(
+ AwsCredentialsStorageIntegration storageIntegration,
+ StorageCredentialCache storageCredentialCache) {
+ this.storageIntegration = storageIntegration;
+ this.storageCredentialCache = storageCredentialCache;
+ }
+
+ @Override
+ public S3SignResponse signRequest(
+ RealmConfig realmConfig,
+ AwsStorageConfigurationInfo storageConfig,
+ S3SignRequest signingRequest) {
+
+ SdkHttpMethod method = SdkHttpMethod.valueOf(signingRequest.method());
+ URI uri = signingRequest.uri();
+ String body = signingRequest.body();
+
+ SdkHttpFullRequest.Builder requestToSign =
+ SdkHttpFullRequest.builder()
Review Comment:
also need the request parameters here
##########
runtime/service/src/main/java/org/apache/polaris/service/storage/aws/signer/S3RequestSignerImpl.java:
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.service.storage.aws.signer;
+
+import java.net.URI;
+import java.util.Set;
+import org.apache.iceberg.aws.s3.signer.ImmutableS3SignResponse;
+import org.apache.iceberg.aws.s3.signer.S3SignResponse;
+import org.apache.polaris.core.config.RealmConfig;
+import org.apache.polaris.core.storage.AccessConfig;
+import org.apache.polaris.core.storage.StorageAccessProperty;
+import org.apache.polaris.core.storage.aws.AwsCredentialsStorageIntegration;
+import org.apache.polaris.core.storage.aws.AwsStorageConfigurationInfo;
+import org.apache.polaris.core.storage.cache.StorageCredentialCache;
+import org.apache.polaris.service.types.S3SignRequest;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.AwsCredentials;
+import software.amazon.awssdk.auth.credentials.AwsSessionCredentials;
+import software.amazon.awssdk.http.ContentStreamProvider;
+import software.amazon.awssdk.http.SdkHttpFullRequest;
+import software.amazon.awssdk.http.SdkHttpMethod;
+import software.amazon.awssdk.http.SdkHttpRequest;
+import software.amazon.awssdk.http.auth.aws.signer.AwsV4HttpSigner;
+import software.amazon.awssdk.http.auth.spi.signer.SignRequest;
+import software.amazon.awssdk.http.auth.spi.signer.SignedRequest;
+import software.amazon.awssdk.services.s3.S3Client;
+
+/**
+ * S3 request signer that creates presigned URLs for S3 operations using AWS credentials obtained
+ * from the storage integration.
+ */
+public class S3RequestSignerImpl implements S3RequestSigner {
+
+ private final AwsV4HttpSigner signer = AwsV4HttpSigner.create();
+
+ private final AwsCredentialsStorageIntegration storageIntegration;
+ private final StorageCredentialCache storageCredentialCache;
+
+ public S3RequestSignerImpl(
+ AwsCredentialsStorageIntegration storageIntegration,
+ StorageCredentialCache storageCredentialCache) {
+ this.storageIntegration = storageIntegration;
+ this.storageCredentialCache = storageCredentialCache;
+ }
+
+ @Override
+ public S3SignResponse signRequest(
+ RealmConfig realmConfig,
+ AwsStorageConfigurationInfo storageConfig,
+ S3SignRequest signingRequest) {
+
+ SdkHttpMethod method = SdkHttpMethod.valueOf(signingRequest.method());
+ URI uri = signingRequest.uri();
+ String body = signingRequest.body();
+
+ SdkHttpFullRequest.Builder requestToSign =
+ SdkHttpFullRequest.builder()
+ .uri(uri)
+ .protocol(uri.getScheme())
+ .method(method)
+ .headers(signingRequest.headers());
+
+ // FIXME is this correct?
+ boolean allowListOperation = isReadMethod(method);
+ Set<String> allowedReadLocations = isReadMethod(method) ? Set.of(uri.toString()) : Set.of();
+ Set<String> allowedWriteLocations = isWriteMethod(method) ? Set.of(uri.toString()) : Set.of();
+
+ // TODO use the storageCredentialCache to get or generate subscoped credentials
+ // AccessConfig accessConfig1 = storageCredentialCache.getOrGenerateSubScopeCreds();
+
+ AccessConfig accessConfig =
+ storageIntegration.getSubscopedCreds(
+ realmConfig, allowListOperation, allowedReadLocations, allowedWriteLocations);
+
+ AwsCredentials credentials = extractCredentials(accessConfig);
+
+ // FIXME is this correct?
Review Comment:
clients may request a different region (hello path-style vs hostname bucket resolution)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2284636564
##########
runtime/service/src/main/java/org/apache/polaris/service/storage/s3/sign/S3RemoteSigningCatalogHandler.java:
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.service.storage.s3.sign;
+
+import jakarta.ws.rs.core.SecurityContext;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.polaris.core.auth.PolarisAuthorizableOperation;
+import org.apache.polaris.core.auth.PolarisAuthorizer;
+import org.apache.polaris.core.config.FeatureConfiguration;
+import org.apache.polaris.core.config.RealmConfig;
+import org.apache.polaris.core.context.CallContext;
+import org.apache.polaris.core.entity.CatalogEntity;
+import org.apache.polaris.core.persistence.resolver.ResolutionManifestFactory;
+import org.apache.polaris.service.catalog.common.CatalogHandler;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignRequest;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignResponse;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class S3RemoteSigningCatalogHandler extends CatalogHandler implements AutoCloseable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(S3RemoteSigningCatalogHandler.class);
+
+ private final S3RequestSigner s3RequestSigner;
+
+ private CatalogEntity catalogEntity;
+
+ public S3RemoteSigningCatalogHandler(
+ CallContext callContext,
+ ResolutionManifestFactory resolutionManifestFactory,
+ SecurityContext securityContext,
+ String catalogName,
+ PolarisAuthorizer authorizer,
+ S3RequestSigner s3RequestSigner) {
+ super(callContext, resolutionManifestFactory, securityContext, catalogName, authorizer);
+ this.s3RequestSigner = s3RequestSigner;
+ }
+
+ @Override
+ protected void initializeCatalog() {
+ catalogEntity =
+ CatalogEntity.of(resolutionManifest.getResolvedReferenceCatalogEntity().getRawLeafEntity());
+ if (catalogEntity.isExternal()) {
+ throw new ForbiddenException("Cannot use S3 remote signing with federated catalogs.");
+ }
+ // no need to materialize the catalog here, as we only need the catalog entity
+ }
+
+ public PolarisS3SignResponse signS3Request(
+ PolarisS3SignRequest s3SignRequest, TableIdentifier tableIdentifier) {
+
+ LOGGER.debug("Requesting s3 signing for {}: {}", tableIdentifier, s3SignRequest);
+
+ PolarisAuthorizableOperation authzOp =
+ s3SignRequest.write()
+ ? PolarisAuthorizableOperation.SIGN_S3_WRITE_REQUEST
+ : PolarisAuthorizableOperation.SIGN_S3_READ_REQUEST;
+
+ authorizeRemoteSigningOperationOrThrow(authzOp, tableIdentifier);
+
+ // Must be done after the authorization check, as the auth check creates the catalog entity
+ throwIfRemoteSigningNotEnabled(callContext.getRealmConfig(), catalogEntity);
+
+ // TODO S3 location access checks
Review Comment:
FYI, this is left for a follow-up PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2536339663
##########
runtime/service/src/main/java/org/apache/polaris/service/storage/s3/sign/S3RemoteSigningCatalogHandler.java:
##########
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.service.storage.s3.sign;
+
+import java.util.Collection;
+import java.util.EnumSet;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.auth.PolarisAuthorizableOperation;
+import org.apache.polaris.core.auth.PolarisAuthorizer;
+import org.apache.polaris.core.auth.PolarisPrincipal;
+import org.apache.polaris.core.config.FeatureConfiguration;
+import org.apache.polaris.core.config.RealmConfig;
+import org.apache.polaris.core.context.CallContext;
+import org.apache.polaris.core.entity.CatalogEntity;
+import org.apache.polaris.core.entity.PolarisEntity;
+import org.apache.polaris.core.entity.PolarisEntityConstants;
+import org.apache.polaris.core.persistence.PolarisResolvedPathWrapper;
+import org.apache.polaris.core.persistence.resolver.ResolutionManifestFactory;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisStorageActions;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.apache.polaris.service.catalog.common.CatalogHandler;
+import org.apache.polaris.service.catalog.common.CatalogUtils;
+import org.apache.polaris.service.catalog.io.FileIOUtil;
+import org.apache.polaris.service.context.catalog.CallContextCatalogFactory;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignRequest;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignResponse;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class S3RemoteSigningCatalogHandler extends CatalogHandler implements AutoCloseable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(S3RemoteSigningCatalogHandler.class);
+
+ private final CallContextCatalogFactory catalogFactory;
+ private final S3RequestSigner s3RequestSigner;
+
+ private CatalogEntity catalogEntity;
+ private Catalog baseCatalog;
+
+ public S3RemoteSigningCatalogHandler(
+ PolarisDiagnostics diagnostics,
+ CallContext callContext,
+ ResolutionManifestFactory resolutionManifestFactory,
+ CallContextCatalogFactory catalogFactory,
+ PolarisPrincipal polarisPrincipal,
+ String catalogName,
+ PolarisAuthorizer authorizer,
+ S3RequestSigner s3RequestSigner) {
+ super(
+ diagnostics,
+ callContext,
+ resolutionManifestFactory,
+ polarisPrincipal,
+ catalogName,
+ authorizer,
+ // external catalogs are not supported for S3 remote signing
+ null,
+ null);
+ this.catalogFactory = catalogFactory;
+ this.s3RequestSigner = s3RequestSigner;
+ }
+
+ @Override
+ protected void initializeCatalog() {
+ catalogEntity =
+ CatalogEntity.of(resolutionManifest.getResolvedReferenceCatalogEntity().getRawLeafEntity());
+ if (catalogEntity.isExternal()) {
+ throw new ForbiddenException("Cannot use S3 remote signing with federated catalogs.");
+ }
+ baseCatalog =
+ catalogFactory.createCallContextCatalog(callContext, polarisPrincipal, resolutionManifest);
+ }
+
+ public PolarisS3SignResponse signS3Request(
+ PolarisS3SignRequest s3SignRequest, TableIdentifier tableIdentifier) {
+
+ LOGGER.debug("Requesting s3 signing for {}: {}", tableIdentifier, s3SignRequest);
+
+ PolarisAuthorizableOperation authzOp =
+ s3SignRequest.write()
+ ? PolarisAuthorizableOperation.SIGN_S3_WRITE_REQUEST
+ : PolarisAuthorizableOperation.SIGN_S3_READ_REQUEST;
+
+ authorizeRemoteSigningOrThrow(EnumSet.of(authzOp), tableIdentifier);
+
+ // Must be done after the authorization check, as the auth check creates the catalog entity
+ throwIfRemoteSigningNotEnabled(callContext.getRealmConfig(), catalogEntity);
+
+ var result =
+ InMemoryStorageIntegration.validateAllowedLocations(
+ callContext.getRealmConfig(),
+ getAllowedLocations(tableIdentifier),
+ getStorageActions(s3SignRequest),
+ getTargetLocations(s3SignRequest));
+
+ if (result.values().stream().anyMatch(r -> r.values().stream().anyMatch(v -> !v.isSuccess()))) {
+ throw new ForbiddenException("Requested S3 location is not allowed.");
+ }
+
+ PolarisS3SignResponse s3SignResponse = s3RequestSigner.signRequest(s3SignRequest);
+ LOGGER.debug("S3 signing response: {}", s3SignResponse);
+
+ return s3SignResponse;
+ }
+
+ private Collection<String> getAllowedLocations(TableIdentifier tableIdentifier) {
+
+ if (baseCatalog.tableExists(tableIdentifier)) {
+
+ // If the table exists, get allowed locations from the table metadata
+ Table table = baseCatalog.loadTable(tableIdentifier);
Review Comment:
do we store the 3 properties in the entity properties asking because
i only see the StorageUtil#getLocationsUsedByTable passing TableMetada#properties and also not seeing it actually persisted in StorageUtil, am i missing something ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on PR #2280:
URL: https://github.com/apache/polaris/pull/2280#issuecomment-3837519661
A few updates on remote signing:
The community decided to push some changes down to the Iceberg code, which resulted in two PRs:
* https://github.com/apache/iceberg/pull/15112
* https://github.com/apache/iceberg/pull/15168
I have been working actively on a new version that incorporates the above changes, and also provides the full feature, including the "M2" topics:
* Reverse proxy support
* Encrypted signing token
* Full authz checks including table locations
I think it makes sense to close this PR and open a new one later, not only because it is stale, but because the authorization model will change significantly (it will go from a model that checks on every signing request to a model that checks authz only once at load table).
I'm confident we could include the Iceberg changes in the upcoming Iceberg 1.11 release, then have remote signing released (as beta of course) for Polaris 1.5. **But this requires upgrading Polaris to Java 17**.
\cc @singhpk234
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra closed pull request #2280: Add support for S3 request signing
URL: https://github.com/apache/polaris/pull/2280
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on PR #2280:
URL: https://github.com/apache/polaris/pull/2280#issuecomment-3552757966
Heads up: since this PR creates constant merge conflicts, I squashed the older commits to facilitate conflict resolution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2542036319
##########
runtime/service/src/main/java/org/apache/polaris/service/storage/s3/sign/S3RemoteSigningCatalogHandler.java:
##########
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.service.storage.s3.sign;
+
+import java.util.Collection;
+import java.util.EnumSet;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.auth.PolarisAuthorizableOperation;
+import org.apache.polaris.core.auth.PolarisAuthorizer;
+import org.apache.polaris.core.auth.PolarisPrincipal;
+import org.apache.polaris.core.config.FeatureConfiguration;
+import org.apache.polaris.core.config.RealmConfig;
+import org.apache.polaris.core.context.CallContext;
+import org.apache.polaris.core.entity.CatalogEntity;
+import org.apache.polaris.core.entity.PolarisEntity;
+import org.apache.polaris.core.entity.PolarisEntityConstants;
+import org.apache.polaris.core.persistence.PolarisResolvedPathWrapper;
+import org.apache.polaris.core.persistence.resolver.ResolutionManifestFactory;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisStorageActions;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.apache.polaris.service.catalog.common.CatalogHandler;
+import org.apache.polaris.service.catalog.common.CatalogUtils;
+import org.apache.polaris.service.catalog.io.FileIOUtil;
+import org.apache.polaris.service.context.catalog.CallContextCatalogFactory;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignRequest;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignResponse;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class S3RemoteSigningCatalogHandler extends CatalogHandler implements AutoCloseable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(S3RemoteSigningCatalogHandler.class);
+
+ private final CallContextCatalogFactory catalogFactory;
+ private final S3RequestSigner s3RequestSigner;
+
+ private CatalogEntity catalogEntity;
+ private Catalog baseCatalog;
+
+ public S3RemoteSigningCatalogHandler(
+ PolarisDiagnostics diagnostics,
+ CallContext callContext,
+ ResolutionManifestFactory resolutionManifestFactory,
+ CallContextCatalogFactory catalogFactory,
+ PolarisPrincipal polarisPrincipal,
+ String catalogName,
+ PolarisAuthorizer authorizer,
+ S3RequestSigner s3RequestSigner) {
+ super(
+ diagnostics,
+ callContext,
+ resolutionManifestFactory,
+ polarisPrincipal,
+ catalogName,
+ authorizer,
+ // external catalogs are not supported for S3 remote signing
+ null,
+ null);
+ this.catalogFactory = catalogFactory;
+ this.s3RequestSigner = s3RequestSigner;
+ }
+
+ @Override
+ protected void initializeCatalog() {
+ catalogEntity =
+ CatalogEntity.of(resolutionManifest.getResolvedReferenceCatalogEntity().getRawLeafEntity());
+ if (catalogEntity.isExternal()) {
+ throw new ForbiddenException("Cannot use S3 remote signing with federated catalogs.");
+ }
+ baseCatalog =
+ catalogFactory.createCallContextCatalog(callContext, polarisPrincipal, resolutionManifest);
+ }
+
+ public PolarisS3SignResponse signS3Request(
+ PolarisS3SignRequest s3SignRequest, TableIdentifier tableIdentifier) {
+
+ LOGGER.debug("Requesting s3 signing for {}: {}", tableIdentifier, s3SignRequest);
+
+ PolarisAuthorizableOperation authzOp =
+ s3SignRequest.write()
+ ? PolarisAuthorizableOperation.SIGN_S3_WRITE_REQUEST
+ : PolarisAuthorizableOperation.SIGN_S3_READ_REQUEST;
+
+ authorizeRemoteSigningOrThrow(EnumSet.of(authzOp), tableIdentifier);
+
+ // Must be done after the authorization check, as the auth check creates the catalog entity
+ throwIfRemoteSigningNotEnabled(callContext.getRealmConfig(), catalogEntity);
+
+ var result =
+ InMemoryStorageIntegration.validateAllowedLocations(
+ callContext.getRealmConfig(),
+ getAllowedLocations(tableIdentifier),
+ getStorageActions(s3SignRequest),
+ getTargetLocations(s3SignRequest));
+
+ if (result.values().stream().anyMatch(r -> r.values().stream().anyMatch(v -> !v.isSuccess()))) {
+ throw new ForbiddenException("Requested S3 location is not allowed.");
+ }
+
+ PolarisS3SignResponse s3SignResponse = s3RequestSigner.signRequest(s3SignRequest);
+ LOGGER.debug("S3 signing response: {}", s3SignResponse);
+
+ return s3SignResponse;
+ }
+
+ private Collection<String> getAllowedLocations(TableIdentifier tableIdentifier) {
+
+ if (baseCatalog.tableExists(tableIdentifier)) {
+
+ // If the table exists, get allowed locations from the table metadata
+ Table table = baseCatalog.loadTable(tableIdentifier);
+ if (table instanceof BaseTable baseTable) {
+ return StorageUtil.getLocationsUsedByTable(baseTable.operations().current());
+ }
+
+ throw new ForbiddenException("No storage configuration found for table.");
+
+ } else {
+
+ // If the table or view doesn't exist, the engine might be writing the manifests before the
+ // table creation is committed. In this case, we still need to check allowed locations from
+ // the parent entities.
+
+ PolarisResolvedPathWrapper resolvedPath =
+ CatalogUtils.findResolvedStorageEntity(resolutionManifest, tableIdentifier);
+
+ Optional<PolarisEntity> storageInfo = FileIOUtil.findStorageInfoFromHierarchy(resolvedPath);
+
+ var configurationInfo =
+ storageInfo
+ .map(PolarisEntity::getInternalPropertiesAsMap)
+ .map(info -> info.get(PolarisEntityConstants.getStorageConfigInfoPropertyName()))
+ .map(PolarisStorageConfigurationInfo::deserialize);
+
+ if (configurationInfo.isEmpty()) {
+ throw new ForbiddenException("No storage configuration found for table.");
+ }
+
+ return configurationInfo.get().getAllowedLocations();
+ }
+ }
+
+ private Set<PolarisStorageActions> getStorageActions(PolarisS3SignRequest s3SignRequest) {
+ // TODO M2: better handling of DELETE and LIST
+ return s3SignRequest.write()
+ ? Set.of(PolarisStorageActions.WRITE)
+ : Set.of(PolarisStorageActions.READ);
+ }
+
+ private Set<String> getTargetLocations(PolarisS3SignRequest s3SignRequest) {
+ // TODO M2: map http URI to s3 URI
+ return Set.of();
+ }
+
+ public static void throwIfRemoteSigningNotEnabled(
+ RealmConfig realmConfig, CatalogEntity catalogEntity) {
+ if (catalogEntity.isExternal()) {
+ throw new ForbiddenException("Remote signing is not enabled for external catalogs.");
+ }
Review Comment:
https://github.com/apache/polaris/issues/3091
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2544215906
##########
CHANGELOG.md:
##########
@@ -36,6 +36,14 @@ request adding CHANGELOG notes for breaking (!) changes and possibly other secti
[Iceberg Metrics Reporting]: https://iceberg.apache.org/docs/latest/metrics-reporting/
+- **S3 remote request signing** has been added, allowing Polaris to work with S3-compatible object storage systems.
+ *Remote signing is currently experimental and not enabled by default*. In particular, RBAC checks are currently not
+ production-ready. One new table privilege was introduced: `TABLE_REMOTE_SIGN`. To enable remote signing:
+ 1. Set the system-wide property `REMOTE_SIGNING_ENABLED` or the catalog-level `polaris.request-signing.enabled`
+ property to `true`.
+ 2. Grant the `TABLE_REMOTE_SIGN` privilege to a catalog role. The role must also be granted the `TABLE_READ_DATA`
+ and `TABLE_WRITE_DATA` privileges.
Review Comment:
It seems like either READ or WRITE works but the CHANGELOG gives an impression both is required ? should we update the wording ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2536247701
##########
runtime/service/src/main/java/org/apache/polaris/service/storage/s3/sign/S3RemoteSigningCatalogHandler.java:
##########
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.service.storage.s3.sign;
+
+import java.util.Collection;
+import java.util.EnumSet;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.auth.PolarisAuthorizableOperation;
+import org.apache.polaris.core.auth.PolarisAuthorizer;
+import org.apache.polaris.core.auth.PolarisPrincipal;
+import org.apache.polaris.core.config.FeatureConfiguration;
+import org.apache.polaris.core.config.RealmConfig;
+import org.apache.polaris.core.context.CallContext;
+import org.apache.polaris.core.entity.CatalogEntity;
+import org.apache.polaris.core.entity.PolarisEntity;
+import org.apache.polaris.core.entity.PolarisEntityConstants;
+import org.apache.polaris.core.persistence.PolarisResolvedPathWrapper;
+import org.apache.polaris.core.persistence.resolver.ResolutionManifestFactory;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisStorageActions;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.apache.polaris.service.catalog.common.CatalogHandler;
+import org.apache.polaris.service.catalog.common.CatalogUtils;
+import org.apache.polaris.service.catalog.io.FileIOUtil;
+import org.apache.polaris.service.context.catalog.CallContextCatalogFactory;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignRequest;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignResponse;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class S3RemoteSigningCatalogHandler extends CatalogHandler implements AutoCloseable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(S3RemoteSigningCatalogHandler.class);
+
+ private final CallContextCatalogFactory catalogFactory;
+ private final S3RequestSigner s3RequestSigner;
+
+ private CatalogEntity catalogEntity;
+ private Catalog baseCatalog;
+
+ public S3RemoteSigningCatalogHandler(
+ PolarisDiagnostics diagnostics,
+ CallContext callContext,
+ ResolutionManifestFactory resolutionManifestFactory,
+ CallContextCatalogFactory catalogFactory,
+ PolarisPrincipal polarisPrincipal,
+ String catalogName,
+ PolarisAuthorizer authorizer,
+ S3RequestSigner s3RequestSigner) {
+ super(
+ diagnostics,
+ callContext,
+ resolutionManifestFactory,
+ polarisPrincipal,
+ catalogName,
+ authorizer,
+ // external catalogs are not supported for S3 remote signing
+ null,
+ null);
+ this.catalogFactory = catalogFactory;
+ this.s3RequestSigner = s3RequestSigner;
+ }
+
+ @Override
+ protected void initializeCatalog() {
+ catalogEntity =
+ CatalogEntity.of(resolutionManifest.getResolvedReferenceCatalogEntity().getRawLeafEntity());
+ if (catalogEntity.isExternal()) {
+ throw new ForbiddenException("Cannot use S3 remote signing with federated catalogs.");
+ }
+ baseCatalog =
+ catalogFactory.createCallContextCatalog(callContext, polarisPrincipal, resolutionManifest);
+ }
+
+ public PolarisS3SignResponse signS3Request(
+ PolarisS3SignRequest s3SignRequest, TableIdentifier tableIdentifier) {
+
+ LOGGER.debug("Requesting s3 signing for {}: {}", tableIdentifier, s3SignRequest);
+
+ PolarisAuthorizableOperation authzOp =
+ s3SignRequest.write()
+ ? PolarisAuthorizableOperation.SIGN_S3_WRITE_REQUEST
+ : PolarisAuthorizableOperation.SIGN_S3_READ_REQUEST;
+
+ authorizeRemoteSigningOrThrow(EnumSet.of(authzOp), tableIdentifier);
+
+ // Must be done after the authorization check, as the auth check creates the catalog entity
+ throwIfRemoteSigningNotEnabled(callContext.getRealmConfig(), catalogEntity);
+
+ var result =
+ InMemoryStorageIntegration.validateAllowedLocations(
+ callContext.getRealmConfig(),
+ getAllowedLocations(tableIdentifier),
+ getStorageActions(s3SignRequest),
+ getTargetLocations(s3SignRequest));
+
+ if (result.values().stream().anyMatch(r -> r.values().stream().anyMatch(v -> !v.isSuccess()))) {
+ throw new ForbiddenException("Requested S3 location is not allowed.");
+ }
+
+ PolarisS3SignResponse s3SignResponse = s3RequestSigner.signRequest(s3SignRequest);
+ LOGGER.debug("S3 signing response: {}", s3SignResponse);
+
+ return s3SignResponse;
+ }
+
+ private Collection<String> getAllowedLocations(TableIdentifier tableIdentifier) {
+
+ if (baseCatalog.tableExists(tableIdentifier)) {
+
+ // If the table exists, get allowed locations from the table metadata
+ Table table = baseCatalog.loadTable(tableIdentifier);
+ if (table instanceof BaseTable baseTable) {
+ return StorageUtil.getLocationsUsedByTable(baseTable.operations().current());
+ }
+
+ throw new ForbiddenException("No storage configuration found for table.");
+
+ } else {
+
+ // If the table or view doesn't exist, the engine might be writing the manifests before the
+ // table creation is committed. In this case, we still need to check allowed locations from
+ // the parent entities.
+
+ PolarisResolvedPathWrapper resolvedPath =
+ CatalogUtils.findResolvedStorageEntity(resolutionManifest, tableIdentifier);
+
+ Optional<PolarisEntity> storageInfo = FileIOUtil.findStorageInfoFromHierarchy(resolvedPath);
+
+ var configurationInfo =
+ storageInfo
+ .map(PolarisEntity::getInternalPropertiesAsMap)
+ .map(info -> info.get(PolarisEntityConstants.getStorageConfigInfoPropertyName()))
+ .map(PolarisStorageConfigurationInfo::deserialize);
+
+ if (configurationInfo.isEmpty()) {
+ throw new ForbiddenException("No storage configuration found for table.");
+ }
+
+ return configurationInfo.get().getAllowedLocations();
+ }
+ }
+
+ private Set<PolarisStorageActions> getStorageActions(PolarisS3SignRequest s3SignRequest) {
+ // TODO M2: better handling of DELETE and LIST
+ return s3SignRequest.write()
+ ? Set.of(PolarisStorageActions.WRITE)
+ : Set.of(PolarisStorageActions.READ);
+ }
+
+ private Set<String> getTargetLocations(PolarisS3SignRequest s3SignRequest) {
+ // TODO M2: map http URI to s3 URI
+ return Set.of();
+ }
+
+ public static void throwIfRemoteSigningNotEnabled(
+ RealmConfig realmConfig, CatalogEntity catalogEntity) {
+ if (catalogEntity.isExternal()) {
+ throw new ForbiddenException("Remote signing is not enabled for external catalogs.");
+ }
Review Comment:
should we file a ticket so we can tackle this later as a community, WDYT ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2536246671
##########
runtime/service/src/main/java/org/apache/polaris/service/storage/s3/sign/S3RemoteSigningCatalogHandler.java:
##########
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.service.storage.s3.sign;
+
+import java.util.Collection;
+import java.util.EnumSet;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.auth.PolarisAuthorizableOperation;
+import org.apache.polaris.core.auth.PolarisAuthorizer;
+import org.apache.polaris.core.auth.PolarisPrincipal;
+import org.apache.polaris.core.config.FeatureConfiguration;
+import org.apache.polaris.core.config.RealmConfig;
+import org.apache.polaris.core.context.CallContext;
+import org.apache.polaris.core.entity.CatalogEntity;
+import org.apache.polaris.core.entity.PolarisEntity;
+import org.apache.polaris.core.entity.PolarisEntityConstants;
+import org.apache.polaris.core.persistence.PolarisResolvedPathWrapper;
+import org.apache.polaris.core.persistence.resolver.ResolutionManifestFactory;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisStorageActions;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.apache.polaris.service.catalog.common.CatalogHandler;
+import org.apache.polaris.service.catalog.common.CatalogUtils;
+import org.apache.polaris.service.catalog.io.FileIOUtil;
+import org.apache.polaris.service.context.catalog.CallContextCatalogFactory;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignRequest;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignResponse;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class S3RemoteSigningCatalogHandler extends CatalogHandler implements AutoCloseable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(S3RemoteSigningCatalogHandler.class);
+
+ private final CallContextCatalogFactory catalogFactory;
+ private final S3RequestSigner s3RequestSigner;
+
+ private CatalogEntity catalogEntity;
+ private Catalog baseCatalog;
+
+ public S3RemoteSigningCatalogHandler(
+ PolarisDiagnostics diagnostics,
+ CallContext callContext,
+ ResolutionManifestFactory resolutionManifestFactory,
+ CallContextCatalogFactory catalogFactory,
+ PolarisPrincipal polarisPrincipal,
+ String catalogName,
+ PolarisAuthorizer authorizer,
+ S3RequestSigner s3RequestSigner) {
+ super(
+ diagnostics,
+ callContext,
+ resolutionManifestFactory,
+ polarisPrincipal,
+ catalogName,
+ authorizer,
+ // external catalogs are not supported for S3 remote signing
+ null,
+ null);
+ this.catalogFactory = catalogFactory;
+ this.s3RequestSigner = s3RequestSigner;
+ }
+
+ @Override
+ protected void initializeCatalog() {
+ catalogEntity =
+ CatalogEntity.of(resolutionManifest.getResolvedReferenceCatalogEntity().getRawLeafEntity());
+ if (catalogEntity.isExternal()) {
+ throw new ForbiddenException("Cannot use S3 remote signing with federated catalogs.");
+ }
+ baseCatalog =
+ catalogFactory.createCallContextCatalog(callContext, polarisPrincipal, resolutionManifest);
+ }
+
+ public PolarisS3SignResponse signS3Request(
+ PolarisS3SignRequest s3SignRequest, TableIdentifier tableIdentifier) {
+
+ LOGGER.debug("Requesting s3 signing for {}: {}", tableIdentifier, s3SignRequest);
Review Comment:
Totally Agree, so is the conclusion not to log this in the debug anymore ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2558529860
##########
runtime/service/src/main/java/org/apache/polaris/service/storage/s3/sign/S3RemoteSigningCatalogHandler.java:
##########
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.service.storage.s3.sign;
+
+import java.util.Collection;
+import java.util.EnumSet;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.auth.PolarisAuthorizableOperation;
+import org.apache.polaris.core.auth.PolarisAuthorizer;
+import org.apache.polaris.core.auth.PolarisPrincipal;
+import org.apache.polaris.core.config.FeatureConfiguration;
+import org.apache.polaris.core.config.RealmConfig;
+import org.apache.polaris.core.context.CallContext;
+import org.apache.polaris.core.entity.CatalogEntity;
+import org.apache.polaris.core.entity.PolarisEntity;
+import org.apache.polaris.core.entity.PolarisEntityConstants;
+import org.apache.polaris.core.persistence.PolarisResolvedPathWrapper;
+import org.apache.polaris.core.persistence.resolver.ResolutionManifestFactory;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisStorageActions;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.apache.polaris.service.catalog.common.CatalogHandler;
+import org.apache.polaris.service.catalog.common.CatalogUtils;
+import org.apache.polaris.service.catalog.io.FileIOUtil;
+import org.apache.polaris.service.context.catalog.CallContextCatalogFactory;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignRequest;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignResponse;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class S3RemoteSigningCatalogHandler extends CatalogHandler implements AutoCloseable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(S3RemoteSigningCatalogHandler.class);
+
+ private final CallContextCatalogFactory catalogFactory;
+ private final S3RequestSigner s3RequestSigner;
+
+ private CatalogEntity catalogEntity;
+ private Catalog baseCatalog;
+
+ public S3RemoteSigningCatalogHandler(
+ PolarisDiagnostics diagnostics,
+ CallContext callContext,
+ ResolutionManifestFactory resolutionManifestFactory,
+ CallContextCatalogFactory catalogFactory,
+ PolarisPrincipal polarisPrincipal,
+ String catalogName,
+ PolarisAuthorizer authorizer,
+ S3RequestSigner s3RequestSigner) {
+ super(
+ diagnostics,
+ callContext,
+ resolutionManifestFactory,
+ polarisPrincipal,
+ catalogName,
+ authorizer,
+ // external catalogs are not supported for S3 remote signing
+ null,
+ null);
+ this.catalogFactory = catalogFactory;
+ this.s3RequestSigner = s3RequestSigner;
+ }
+
+ @Override
+ protected void initializeCatalog() {
+ catalogEntity =
+ CatalogEntity.of(resolutionManifest.getResolvedReferenceCatalogEntity().getRawLeafEntity());
+ if (catalogEntity.isExternal()) {
+ throw new ForbiddenException("Cannot use S3 remote signing with federated catalogs.");
+ }
+ baseCatalog =
+ catalogFactory.createCallContextCatalog(callContext, polarisPrincipal, resolutionManifest);
+ }
+
+ public PolarisS3SignResponse signS3Request(
+ PolarisS3SignRequest s3SignRequest, TableIdentifier tableIdentifier) {
+
+ LOGGER.debug("Requesting s3 signing for {}: {}", tableIdentifier, s3SignRequest);
+
+ PolarisAuthorizableOperation authzOp =
+ s3SignRequest.write()
+ ? PolarisAuthorizableOperation.SIGN_S3_WRITE_REQUEST
+ : PolarisAuthorizableOperation.SIGN_S3_READ_REQUEST;
+
+ authorizeRemoteSigningOrThrow(EnumSet.of(authzOp), tableIdentifier);
+
+ // Must be done after the authorization check, as the auth check creates the catalog entity
+ throwIfRemoteSigningNotEnabled(callContext.getRealmConfig(), catalogEntity);
+
+ var result =
+ InMemoryStorageIntegration.validateAllowedLocations(
+ callContext.getRealmConfig(),
+ getAllowedLocations(tableIdentifier),
+ getStorageActions(s3SignRequest),
+ getTargetLocations(s3SignRequest));
+
+ if (result.values().stream().anyMatch(r -> r.values().stream().anyMatch(v -> !v.isSuccess()))) {
+ throw new ForbiddenException("Requested S3 location is not allowed.");
+ }
+
+ PolarisS3SignResponse s3SignResponse = s3RequestSigner.signRequest(s3SignRequest);
+ LOGGER.debug("S3 signing response: {}", s3SignResponse);
+
+ return s3SignResponse;
+ }
+
+ private Collection<String> getAllowedLocations(TableIdentifier tableIdentifier) {
+
+ if (baseCatalog.tableExists(tableIdentifier)) {
+
+ // If the table exists, get allowed locations from the table metadata
+ Table table = baseCatalog.loadTable(tableIdentifier);
Review Comment:
I would personally prefer 2 over 3 as IMHO 3 is not quite correct ?
one thing i was thinking is every sign request will make a corresponding loadTable to get signer configs, should be backfill with an updateEntity if these props are missing i.e during read, wdyt ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2511948780
##########
runtime/service/src/main/java/org/apache/polaris/service/storage/s3/sign/S3RemoteSigningCatalogHandler.java:
##########
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.service.storage.s3.sign;
+
+import java.util.Collection;
+import java.util.EnumSet;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.auth.PolarisAuthorizableOperation;
+import org.apache.polaris.core.auth.PolarisAuthorizer;
+import org.apache.polaris.core.auth.PolarisPrincipal;
+import org.apache.polaris.core.config.FeatureConfiguration;
+import org.apache.polaris.core.config.RealmConfig;
+import org.apache.polaris.core.context.CallContext;
+import org.apache.polaris.core.entity.CatalogEntity;
+import org.apache.polaris.core.entity.PolarisEntity;
+import org.apache.polaris.core.entity.PolarisEntityConstants;
+import org.apache.polaris.core.persistence.PolarisResolvedPathWrapper;
+import org.apache.polaris.core.persistence.resolver.ResolutionManifestFactory;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisStorageActions;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.apache.polaris.service.catalog.common.CatalogHandler;
+import org.apache.polaris.service.catalog.common.CatalogUtils;
+import org.apache.polaris.service.catalog.io.FileIOUtil;
+import org.apache.polaris.service.context.catalog.CallContextCatalogFactory;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignRequest;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignResponse;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class S3RemoteSigningCatalogHandler extends CatalogHandler implements AutoCloseable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(S3RemoteSigningCatalogHandler.class);
+
+ private final CallContextCatalogFactory catalogFactory;
+ private final S3RequestSigner s3RequestSigner;
+
+ private CatalogEntity catalogEntity;
+ private Catalog baseCatalog;
+
+ public S3RemoteSigningCatalogHandler(
+ PolarisDiagnostics diagnostics,
+ CallContext callContext,
+ ResolutionManifestFactory resolutionManifestFactory,
+ CallContextCatalogFactory catalogFactory,
+ PolarisPrincipal polarisPrincipal,
+ String catalogName,
+ PolarisAuthorizer authorizer,
+ S3RequestSigner s3RequestSigner) {
+ super(
+ diagnostics,
+ callContext,
+ resolutionManifestFactory,
+ polarisPrincipal,
+ catalogName,
+ authorizer,
+ // external catalogs are not supported for S3 remote signing
+ null,
+ null);
+ this.catalogFactory = catalogFactory;
+ this.s3RequestSigner = s3RequestSigner;
+ }
+
+ @Override
+ protected void initializeCatalog() {
+ catalogEntity =
+ CatalogEntity.of(resolutionManifest.getResolvedReferenceCatalogEntity().getRawLeafEntity());
+ if (catalogEntity.isExternal()) {
+ throw new ForbiddenException("Cannot use S3 remote signing with federated catalogs.");
+ }
+ baseCatalog =
+ catalogFactory.createCallContextCatalog(callContext, polarisPrincipal, resolutionManifest);
+ }
+
+ public PolarisS3SignResponse signS3Request(
+ PolarisS3SignRequest s3SignRequest, TableIdentifier tableIdentifier) {
+
+ LOGGER.debug("Requesting s3 signing for {}: {}", tableIdentifier, s3SignRequest);
+
+ PolarisAuthorizableOperation authzOp =
+ s3SignRequest.write()
+ ? PolarisAuthorizableOperation.SIGN_S3_WRITE_REQUEST
+ : PolarisAuthorizableOperation.SIGN_S3_READ_REQUEST;
+
+ authorizeRemoteSigningOrThrow(EnumSet.of(authzOp), tableIdentifier);
+
+ // Must be done after the authorization check, as the auth check creates the catalog entity
+ throwIfRemoteSigningNotEnabled(callContext.getRealmConfig(), catalogEntity);
+
+ var result =
+ InMemoryStorageIntegration.validateAllowedLocations(
+ callContext.getRealmConfig(),
+ getAllowedLocations(tableIdentifier),
+ getStorageActions(s3SignRequest),
+ getTargetLocations(s3SignRequest));
+
+ if (result.values().stream().anyMatch(r -> r.values().stream().anyMatch(v -> !v.isSuccess()))) {
+ throw new ForbiddenException("Requested S3 location is not allowed.");
+ }
+
+ PolarisS3SignResponse s3SignResponse = s3RequestSigner.signRequest(s3SignRequest);
+ LOGGER.debug("S3 signing response: {}", s3SignResponse);
+
+ return s3SignResponse;
+ }
+
+ private Collection<String> getAllowedLocations(TableIdentifier tableIdentifier) {
+
+ if (baseCatalog.tableExists(tableIdentifier)) {
+
+ // If the table exists, get allowed locations from the table metadata
+ Table table = baseCatalog.loadTable(tableIdentifier);
+ if (table instanceof BaseTable baseTable) {
+ return StorageUtil.getLocationsUsedByTable(baseTable.operations().current());
+ }
+
+ throw new ForbiddenException("No storage configuration found for table.");
+
+ } else {
+
+ // If the table or view doesn't exist, the engine might be writing the manifests before the
+ // table creation is committed. In this case, we still need to check allowed locations from
+ // the parent entities.
Review Comment:
After investigating a bit, it seems that staged vs direct doesn't affect remote signing.
However I found a few feature flags that could interfere with remote signing:
- `ALLOW_UNSTRUCTURED_TABLE_LOCATION`
- `ALLOW_EXTERNAL_TABLE_LOCATION`
- `ALLOW_EXTERNAL_METADATA_FILE_LOCATION`
- `DEFAULT_LOCATION_OBJECT_STORAGE_PREFIX_ENABLED`
I will try to add tests for these flags + remote signing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2263146662
##########
integration-tests/src/main/java/org/apache/polaris/service/it/test/PolarisS3RemoteSigningIntegrationTest.java:
##########
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.service.it.test;
+
+import static org.apache.polaris.core.storage.StorageAccessProperty.AWS_ENDPOINT;
+import static org.apache.polaris.core.storage.StorageAccessProperty.AWS_PATH_STYLE_ACCESS;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import com.google.common.collect.ImmutableMap;
+import java.util.Map;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.TableMetadata;
+import org.apache.iceberg.TableMetadataParser;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.iceberg.io.ResolvingFileIO;
+import org.apache.iceberg.rest.RESTCatalog;
+import org.apache.polaris.core.admin.model.Catalog;
+import org.apache.polaris.core.config.FeatureConfiguration;
+import org.apache.polaris.service.it.env.CatalogConfig;
+import org.apache.polaris.service.it.env.RestCatalogConfig;
+import org.junit.jupiter.api.Disabled;
+import org.junit.jupiter.api.Test;
+
+/** Runs PolarisRestCatalogIntegrationBase test on S3 with remote signing enabled. */
+@CatalogConfig(properties = {"header.X-Iceberg-Access-Delegation", "remote-signing"})
+@RestCatalogConfig({
+ // The default client file IO implementation is InMemoryFileIO,
+ // which does not support remote signing.
+ org.apache.iceberg.CatalogProperties.FILE_IO_IMPL,
+ "org.apache.iceberg.io.ResolvingFileIO",
+})
+public abstract class PolarisS3RemoteSigningIntegrationTest
+ extends PolarisRestCatalogAwsIntegrationTest {
+
+ @Override
+ protected ImmutableMap.Builder<String, String> clientFileIOProperties() {
+ ImmutableMap.Builder<String, String> builder =
+ super.clientFileIOProperties()
+ .put(AWS_PATH_STYLE_ACCESS.getPropertyName(), String.valueOf(pathStyleAccess()));
+ endpoint().ifPresent(endpoint -> builder.put(AWS_ENDPOINT.getPropertyName(), endpoint));
+ return builder;
+ }
+
+ /**
+ * Override this method to return true if the S3 client should use path-style access. Default is
+ * false, which means virtual-hosted-style access will be used.
+ */
+ protected boolean pathStyleAccess() {
+ return false;
+ }
+
+ @CatalogConfig(properties = {"polaris.config.remote-signing.enabled", "false"})
+ @Test
+ public void testInternalCatalogRemoteSigningDisabled() {
+ @SuppressWarnings("resource")
+ RESTCatalog catalog = catalog();
+ Namespace ns1 = Namespace.of("ns1");
+ catalog.createNamespace(ns1);
+ TableIdentifier tableIdentifier = TableIdentifier.of(ns1, "my_table");
+ assertThatThrownBy(() -> catalog.createTable(tableIdentifier, SCHEMA))
+ .isInstanceOf(ForbiddenException.class)
+ .hasMessageContaining("Remote signing is not enabled for this catalog")
+ .hasMessageContaining(FeatureConfiguration.REMOTE_SIGNING_ENABLED.key())
+ .hasMessageContaining(FeatureConfiguration.REMOTE_SIGNING_ENABLED.catalogConfig());
+ }
+
+ @CatalogConfig(Catalog.TypeEnum.EXTERNAL)
+ @Test
+ public void testExternalCatalogRemoteSigningDisabled() {
+ @SuppressWarnings("resource")
+ RESTCatalog catalog = catalog();
+ Namespace ns1 = Namespace.of("ns1");
+ catalog.createNamespace(ns1);
+ TableMetadata tableMetadata =
+ TableMetadata.newTableMetadata(
+ SCHEMA,
+ PartitionSpec.unpartitioned(),
+ externalCatalogBaseLocation() + "/ns1/my_table",
+ Map.of());
+ try (ResolvingFileIO resolvingFileIO = initializeClientFileIO(new ResolvingFileIO())) {
+ String fileLocation =
+ externalCatalogBaseLocation() + "/ns1/my_table/metadata/v1.metadata.json";
+ TableMetadataParser.write(tableMetadata, resolvingFileIO.newOutputFile(fileLocation));
+ catalog.registerTable(TableIdentifier.of(ns1, "my_table"), fileLocation);
+ try {
+ assertThatThrownBy(() -> catalog.loadTable(TableIdentifier.of(ns1, "my_table")))
+ .isInstanceOf(ForbiddenException.class)
+ .hasMessageContaining("Remote signing is not enabled for external catalogs")
+ .hasMessageContaining(
+ FeatureConfiguration.REMOTE_SIGNING_EXTERNAL_CATALOGS_ENABLED.key());
+ } finally {
+ resolvingFileIO.deleteFile(fileLocation);
+ }
+ }
+ }
+
+ @Test
+ @Override
+ @Disabled("It's not possible to request an access delegation mode when registering a table.")
+ public void testRegisterTable() {
Review Comment:
FYI what is happening here is:
As per the Iceberg REST spec, the register-table endpoint does not accept access delegation headers.
As a consequence, the returned `LoadTableResult` does NOT contain any access delegation enablement, e.g. it won't contain vended credentials nor remote signing enablement.
I'm not sure if this is an issue or something that engines know how to deal with.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2263150328
##########
runtime/service/src/intTest/java/org/apache/polaris/service/it/RestCatalogMinIOSpecialIT.java:
##########
@@ -86,7 +85,6 @@
@QuarkusIntegrationTest
@TestProfile(RestCatalogMinIOSpecialIT.Profile.class)
@ExtendWith(MinioExtension.class)
-@ExtendWith(PolarisIntegrationTestExtension.class)
Review Comment:
This extension should not be enabled in IT tests annotated with `@QuarkusIntegrationTest`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2263155414
##########
runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java:
##########
@@ -758,21 +741,55 @@ public Optional<LoadTableResponse> loadTableWithAccessDelegationIfStale(
throw new IllegalStateException("Cannot wrap catalog that does not produce BaseTable");
}
+ private CatalogEntity getCatalogEntity() {
+ PolarisResolvedPathWrapper catalogPath = resolutionManifest.getResolvedReferenceCatalogEntity();
+ callContext
+ .getPolarisCallContext()
+ .getDiagServices()
+ .checkNotNull(catalogPath, "No catalog available for loadTable request");
+ CatalogEntity catalogEntity = CatalogEntity.of(catalogPath.getRawLeafEntity());
+ LOGGER.info("Catalog type: {}", catalogEntity.getCatalogType());
+ return catalogEntity;
+ }
+
private LoadTableResponse.Builder buildLoadTableResponseWithDelegationCredentials(
TableIdentifier tableIdentifier,
TableMetadata tableMetadata,
Set<PolarisStorageActions> actions,
- String snapshots) {
+ Set<AccessDelegationMode> delegationModes,
+ CatalogEntity catalogEntity) {
LoadTableResponse.Builder responseBuilder =
LoadTableResponse.builder().withTableMetadata(tableMetadata);
- if (baseCatalog instanceof SupportsCredentialDelegation credentialDelegation) {
+ if (baseCatalog instanceof SupportsCredentialDelegation credentialDelegation
+ && delegationModes.contains(AccessDelegationMode.VENDED_CREDENTIALS)) {
LOGGER
.atDebug()
.addKeyValue("tableIdentifier", tableIdentifier)
.addKeyValue("tableLocation", tableMetadata.location())
.log("Fetching client credentials for table");
AccessConfig accessConfig =
- credentialDelegation.getAccessConfig(tableIdentifier, tableMetadata, actions);
+ credentialDelegation.getAccessConfigForCredentialDelegation(
+ tableIdentifier, tableMetadata, actions);
+ Map<String, String> credentialConfig = accessConfig.credentials();
+ responseBuilder.addAllConfig(credentialConfig);
+ responseBuilder.addAllConfig(accessConfig.extraProperties());
+ if (!credentialConfig.isEmpty()) {
+ responseBuilder.addCredential(
+ ImmutableCredential.builder()
+ .prefix(tableMetadata.location())
+ .config(credentialConfig)
+ .build());
+ }
+ } else if (baseCatalog instanceof SupportsRemoteSigning remoteSigning
+ && delegationModes.contains(AccessDelegationMode.REMOTE_SIGNING)) {
+ S3RemoteSigningCatalogHandler.throwIfRemoteSigningNotEnabled(
+ callContext.getRealmConfig(), catalogEntity);
+ LOGGER
+ .atDebug()
+ .addKeyValue("tableIdentifier", tableIdentifier)
+ .addKeyValue("tableLocation", tableMetadata.location())
+ .log("Enabling remote signing for table");
+ AccessConfig accessConfig = remoteSigning.getAccessConfigForRemoteSigning(tableIdentifier);
Review Comment:
This is where all `LoadTableResponse`s get remote signing enablement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2276774113
##########
api/iceberg-aws-sign-service/src/main/java/org/apache/polaris/service/aws/sign/model/PolarisS3SignRequest.java:
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.service.aws.sign.model;
+
+import com.fasterxml.jackson.databind.annotation.JsonDeserialize;
+import com.fasterxml.jackson.databind.annotation.JsonSerialize;
+import jakarta.annotation.Nullable;
+import org.apache.iceberg.aws.s3.signer.S3SignRequest;
+import org.apache.polaris.immutables.PolarisImmutable;
+import org.immutables.value.Value;
+
+/**
+ * Request for S3 signing requests.
Review Comment:
```suggestion
* Request for S3 signing.
```
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/aws/AwsCredentialsStorageIntegration.java:
##########
@@ -143,6 +143,37 @@ public AccessConfig getSubscopedCreds(
return accessConfig.build();
}
+ public AccessConfig getRemoteSigningAccessConfig(URI signerUri, String signerEndpoint) {
+
+ AwsStorageConfigurationInfo storageConfig = config();
+ String region = storageConfig.getRegion();
+
+ if (storageConfig.getAwsPartition().equals("aws-us-gov") && region == null) {
+ throw new IllegalArgumentException(
+ String.format(
+ "AWS region must be set when using partition %s", storageConfig.getAwsPartition()));
+ }
+
+ AccessConfig.Builder accessConfig = AccessConfig.builder();
+ if (region != null) {
+ accessConfig.put(StorageAccessProperty.CLIENT_REGION, region);
+ }
+
+ URI endpointUri = storageConfig.getEndpointUri();
+ if (endpointUri != null) {
+ accessConfig.put(StorageAccessProperty.AWS_ENDPOINT, endpointUri.toString());
+ }
+ if (Boolean.TRUE.equals(storageConfig.getPathStyleAccess())) {
+ accessConfig.put(StorageAccessProperty.AWS_PATH_STYLE_ACCESS, Boolean.TRUE.toString());
+ }
Review Comment:
can we put these in a helper, to be used here and the method above ?
##########
runtime/service/src/main/java/org/apache/polaris/service/storage/aws/signer/S3RemoteSigningCatalogHandler.java:
##########
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.service.storage.aws.signer;
+
+import jakarta.ws.rs.core.SecurityContext;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.apache.polaris.core.auth.PolarisAuthorizableOperation;
+import org.apache.polaris.core.auth.PolarisAuthorizer;
+import org.apache.polaris.core.config.FeatureConfiguration;
+import org.apache.polaris.core.config.RealmConfig;
+import org.apache.polaris.core.context.CallContext;
+import org.apache.polaris.core.entity.CatalogEntity;
+import org.apache.polaris.core.entity.PolarisEntitySubType;
+import org.apache.polaris.core.persistence.resolver.ResolutionManifestFactory;
+import org.apache.polaris.service.aws.sign.model.PolarisS3SignRequest;
+import org.apache.polaris.service.aws.sign.model.PolarisS3SignResponse;
+import org.apache.polaris.service.catalog.common.CatalogHandler;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class S3RemoteSigningCatalogHandler extends CatalogHandler implements AutoCloseable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(S3RemoteSigningCatalogHandler.class);
+
+ private final S3RequestSigner s3RequestSigner;
+
+ private CatalogEntity catalogEntity;
+
+ public S3RemoteSigningCatalogHandler(
+ CallContext callContext,
+ ResolutionManifestFactory resolutionManifestFactory,
+ SecurityContext securityContext,
+ String catalogName,
+ PolarisAuthorizer authorizer,
+ S3RequestSigner s3RequestSigner) {
+ super(callContext, resolutionManifestFactory, securityContext, catalogName, authorizer);
+ this.s3RequestSigner = s3RequestSigner;
+ }
+
+ @Override
+ protected void initializeCatalog() {
+ catalogEntity =
+ CatalogEntity.of(resolutionManifest.getResolvedReferenceCatalogEntity().getRawLeafEntity());
+ if (catalogEntity.isExternal()) {
+ throw new ForbiddenException("Cannot use S3 remote signing with federated catalogs.");
+ }
+ // no need to materialize the catalog here, as we only need the catalog entity
+ }
+
+ public PolarisS3SignResponse signS3Request(
+ PolarisS3SignRequest s3SignRequest, TableIdentifier tableIdentifier) {
+
+ LOGGER.debug("Requesting s3 signing for {}: {}", tableIdentifier, s3SignRequest);
+
+ // TODO authorize based on the request's method?
Review Comment:
shouldn't we ? SIGN_S3_REQUEST shouldn't be a blanket approval for deletes ?
##########
runtime/service/src/main/java/org/apache/polaris/service/storage/aws/signer/S3RemoteSigningCatalogHandler.java:
##########
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.service.storage.aws.signer;
+
+import jakarta.ws.rs.core.SecurityContext;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.apache.polaris.core.auth.PolarisAuthorizableOperation;
+import org.apache.polaris.core.auth.PolarisAuthorizer;
+import org.apache.polaris.core.config.FeatureConfiguration;
+import org.apache.polaris.core.config.RealmConfig;
+import org.apache.polaris.core.context.CallContext;
+import org.apache.polaris.core.entity.CatalogEntity;
+import org.apache.polaris.core.entity.PolarisEntitySubType;
+import org.apache.polaris.core.persistence.resolver.ResolutionManifestFactory;
+import org.apache.polaris.service.aws.sign.model.PolarisS3SignRequest;
+import org.apache.polaris.service.aws.sign.model.PolarisS3SignResponse;
+import org.apache.polaris.service.catalog.common.CatalogHandler;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class S3RemoteSigningCatalogHandler extends CatalogHandler implements AutoCloseable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(S3RemoteSigningCatalogHandler.class);
+
+ private final S3RequestSigner s3RequestSigner;
+
+ private CatalogEntity catalogEntity;
+
+ public S3RemoteSigningCatalogHandler(
+ CallContext callContext,
+ ResolutionManifestFactory resolutionManifestFactory,
+ SecurityContext securityContext,
+ String catalogName,
+ PolarisAuthorizer authorizer,
+ S3RequestSigner s3RequestSigner) {
+ super(callContext, resolutionManifestFactory, securityContext, catalogName, authorizer);
+ this.s3RequestSigner = s3RequestSigner;
+ }
+
+ @Override
+ protected void initializeCatalog() {
+ catalogEntity =
+ CatalogEntity.of(resolutionManifest.getResolvedReferenceCatalogEntity().getRawLeafEntity());
+ if (catalogEntity.isExternal()) {
+ throw new ForbiddenException("Cannot use S3 remote signing with federated catalogs.");
+ }
+ // no need to materialize the catalog here, as we only need the catalog entity
+ }
+
+ public PolarisS3SignResponse signS3Request(
Review Comment:
can i have a sign priviledge without table read priviledge ? a client may directly contact the sign endpoint
##########
runtime/service/src/main/java/org/apache/polaris/service/storage/aws/signer/S3RequestSignerImpl.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.service.storage.aws.signer;
+
+import jakarta.enterprise.context.ApplicationScoped;
+import jakarta.inject.Inject;
+import java.net.URI;
+import org.apache.iceberg.aws.s3.signer.S3SignRequest;
+import org.apache.polaris.service.aws.sign.model.ImmutablePolarisS3SignResponse;
+import org.apache.polaris.service.aws.sign.model.PolarisS3SignResponse;
+import org.apache.polaris.service.storage.StorageConfiguration;
+import software.amazon.awssdk.auth.credentials.AwsCredentials;
+import software.amazon.awssdk.http.ContentStreamProvider;
+import software.amazon.awssdk.http.SdkHttpFullRequest;
+import software.amazon.awssdk.http.SdkHttpMethod;
+import software.amazon.awssdk.http.SdkHttpRequest;
+import software.amazon.awssdk.http.auth.aws.signer.AwsV4HttpSigner;
+import software.amazon.awssdk.http.auth.spi.signer.SignRequest;
+import software.amazon.awssdk.http.auth.spi.signer.SignedRequest;
+import software.amazon.awssdk.services.s3.S3Client;
+
+@ApplicationScoped
+class S3RequestSignerImpl implements S3RequestSigner {
+
+ private final AwsV4HttpSigner signer = AwsV4HttpSigner.create();
+
+ @Inject StorageConfiguration storageConfiguration;
+
+ @Override
+ public PolarisS3SignResponse signRequest(S3SignRequest signingRequest) {
+
+ URI uri = signingRequest.uri();
+ SdkHttpMethod method = SdkHttpMethod.valueOf(signingRequest.method());
+
+ SdkHttpFullRequest.Builder requestToSign =
+ SdkHttpFullRequest.builder()
+ .uri(uri)
+ .protocol(uri.getScheme())
+ .method(method)
+ .headers(signingRequest.headers());
+
+ AwsCredentials credentials = storageConfiguration.awsSystemCredentials().resolveCredentials();
Review Comment:
How are the credentials refreshed ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2284644758
##########
runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalog.java:
##########
@@ -834,13 +855,50 @@ public AccessConfig getAccessConfig(
return AccessConfig.builder().build();
}
return FileIOUtil.refreshAccessConfig(
- callContext,
- storageCredentialCache,
- getCredentialVendor(),
- tableIdentifier,
- StorageUtil.getLocationsAllowedToBeAccessed(tableMetadata),
- storageActions,
- storageInfo.get());
+ callContext,
+ storageCredentialCache,
+ getCredentialVendor(),
+ tableIdentifier,
+ StorageUtil.getLocationsAllowedToBeAccessed(tableMetadata),
+ storageActions,
+ storageInfo.get())
+ .orElse(AccessConfig.EMPTY);
+ }
+
+ @Override
+ public AccessConfig getAccessConfigForRemoteSigning(TableIdentifier tableIdentifier) {
+
+ Optional<PolarisStorageConfigurationInfo> configurationInfo =
+ findStorageInfo(tableIdentifier)
+ .map(PolarisEntity::getInternalPropertiesAsMap)
+ .map(info -> info.get(PolarisEntityConstants.getStorageConfigInfoPropertyName()))
+ .map(PolarisStorageConfigurationInfo::deserialize);
+
+ if (configurationInfo.isEmpty()) {
+ LOGGER
+ .atWarn()
+ .addKeyValue("tableIdentifier", tableIdentifier)
+ .log("Remote signing: table entity has no storage configuration in its hierarchy");
+ return AccessConfig.builder().build();
+ }
+
+ PolarisStorageIntegration<AwsStorageConfigurationInfo> storageIntegration =
+ storageIntegrationProvider.getStorageIntegrationForConfig(configurationInfo.get());
+
+ if (!(storageIntegration instanceof AwsCredentialsStorageIntegration)) {
+ LOGGER
+ .atWarn()
+ .addKeyValue("tableIdentifier", tableIdentifier)
+ .log("Table entity storage integration is not an AWS credentials storage integration");
+ return AccessConfig.builder().build();
+ }
+
+ String prefix = prefixParser.catalogNameToPrefix(callContext.getRealmContext(), catalogName);
+ URI signerUri = uriInfo.getBaseUri().resolve("api/");
Review Comment:
FYI, resolving the signer URI in case of HTTP proxies is left for a follow-up PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "snazy (via GitHub)" <gi...@apache.org>.
snazy commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2276822566
##########
runtime/service/src/main/java/org/apache/polaris/service/storage/aws/signer/S3RemoteSigningCatalogHandler.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.service.storage.aws.signer;
+
+import jakarta.ws.rs.core.SecurityContext;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.apache.polaris.core.auth.PolarisAuthorizableOperation;
+import org.apache.polaris.core.auth.PolarisAuthorizer;
+import org.apache.polaris.core.config.FeatureConfiguration;
+import org.apache.polaris.core.config.RealmConfig;
+import org.apache.polaris.core.context.CallContext;
+import org.apache.polaris.core.entity.CatalogEntity;
+import org.apache.polaris.core.entity.PolarisEntitySubType;
+import org.apache.polaris.core.persistence.resolver.ResolutionManifestFactory;
+import org.apache.polaris.service.aws.sign.model.PolarisS3SignRequest;
+import org.apache.polaris.service.aws.sign.model.PolarisS3SignResponse;
+import org.apache.polaris.service.catalog.common.CatalogHandler;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class S3RemoteSigningCatalogHandler extends CatalogHandler implements AutoCloseable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(S3RemoteSigningCatalogHandler.class);
+
+ private final S3RequestSigner s3RequestSigner;
+
+ private CatalogEntity catalogEntity;
+
+ public S3RemoteSigningCatalogHandler(
+ CallContext callContext,
+ ResolutionManifestFactory resolutionManifestFactory,
+ SecurityContext securityContext,
+ String catalogName,
+ PolarisAuthorizer authorizer,
+ S3RequestSigner s3RequestSigner) {
+ super(callContext, resolutionManifestFactory, securityContext, catalogName, authorizer);
+ this.s3RequestSigner = s3RequestSigner;
+ }
+
+ @Override
+ protected void initializeCatalog() {
+ catalogEntity =
+ CatalogEntity.of(resolutionManifest.getResolvedReferenceCatalogEntity().getRawLeafEntity());
+ if (catalogEntity.isExternal()) {
+ throw new ForbiddenException("Cannot use S3 remote signing with federated catalogs.");
+ }
+ // no need to materialize the catalog here, as we only need the catalog entity
+ }
+
+ public PolarisS3SignResponse signS3Request(
+ PolarisS3SignRequest s3SignRequest, TableIdentifier tableIdentifier) {
+
+ LOGGER.debug("Requesting s3 signing for {}: {}", tableIdentifier, s3SignRequest);
+
+ // TODO authorize based on the request's method?
+
+ try {
+ authorizeBasicTableLikeOperationOrThrow(
Review Comment:
hm - probably worth to mention in the changelog
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2276668383
##########
runtime/service/src/main/java/org/apache/polaris/service/context/catalog/PolarisCallContextCatalogFactory.java:
##########
@@ -92,7 +104,10 @@ public Catalog createCallContextCatalog(
securityContext,
taskExecutor,
fileIOFactory,
- polarisEventListener);
+ polarisEventListener,
+ storageIntegrationProvider,
+ prefixParser,
+ uriInfo);
Review Comment:
Agreed. We could look into making it a CDI bean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on PR #2280:
URL: https://github.com/apache/polaris/pull/2280#issuecomment-3197495353
FYI: I was requested to produce a design doc for this feature:
https://docs.google.com/document/d/1ygdia7u4bUHUt6n8XhZo48aKoIyyrCvKqan3XP25iB8/edit?usp=sharing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2284625091
##########
integration-tests/src/main/java/org/apache/polaris/service/it/test/PolarisRestCatalogIntegrationBase.java:
##########
@@ -146,11 +148,13 @@ public abstract class PolarisRestCatalogIntegrationBase extends CatalogTests<RES
private RESTCatalog restCatalog;
private String currentCatalogName;
private Map<String, String> restCatalogConfig;
- private URI externalCatalogBase;
+ private URI externalCatalogBaseLocation;
Review Comment:
Yup good idea!
https://github.com/apache/polaris/pull/2384
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2263140745
##########
integration-tests/src/main/java/org/apache/polaris/service/it/test/PolarisRestCatalogIntegrationBase.java:
##########
@@ -146,11 +148,13 @@ public abstract class PolarisRestCatalogIntegrationBase extends CatalogTests<RES
private RESTCatalog restCatalog;
private String currentCatalogName;
private Map<String, String> restCatalogConfig;
- private URI externalCatalogBase;
+ private URI externalCatalogBaseLocation;
Review Comment:
FYI I refactored this class a bit, notably to facilitate subclassing, but there is no functional changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on PR #2280:
URL: https://github.com/apache/polaris/pull/2280#issuecomment-3161899665
Thank you so much for this @adutra !
when ready please let us know (would be really helpful if you can share some description to walk us through your thought process), happy to help with reviews :) !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "metadaddy (via GitHub)" <gi...@apache.org>.
metadaddy commented on PR #2280:
URL: https://github.com/apache/polaris/pull/2280#issuecomment-3175984012
Looks good. When I get an opportunity, I'll try setting this up with Backblaze B2 as S3-compatible storage and Trino as the Iceberg REST catalog client. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2505113398
##########
CHANGELOG.md:
##########
@@ -36,6 +36,14 @@ request adding CHANGELOG notes for breaking (!) changes and possibly other secti
[Iceberg Metrics Reporting]: https://iceberg.apache.org/docs/latest/metrics-reporting/
+- **S3 remote request signing** has been added, allowing Polaris to work with S3-compatible object storage systems.
+ *Remote signing is currently experimental and not enabled by default*. In particular, RBAC checks are currently not
+ production-ready. One new table privilege was introduced: `TABLE_REMOTE_SIGN`. To enable remote signing:
+ 1. Set the system-wide property `REMOTE_SIGNING_ENABLED` or the catalog-level `polaris.request-signing.enabled`
+ property to `true`.
+ 2. Grant the `TABLE_REMOTE_SIGN` privilege to a catalog role. The role must also be granted the `TABLE_READ_DATA`
+ and `TABLE_WRITE_DATA` privileges.
Review Comment:
[doubt] is there an assertion when even TABLE_REMOTE_SIGN is granted these two privieldges would be there ?
nit :
```suggestion
2. Grant the `TABLE_REMOTE_SIGN` privilege to a catalog role. The catalog role must also be granted the `TABLE_READ_DATA`
and `TABLE_WRITE_DATA` privileges.
```
##########
runtime/service/src/main/java/org/apache/polaris/service/catalog/common/CatalogHandler.java:
##########
@@ -394,6 +394,47 @@ protected void authorizeRenameTableLikeOperationOrThrow(
initializeCatalog();
}
+ protected void authorizeRemoteSigningOrThrow(
+ EnumSet<PolarisAuthorizableOperation> ops, TableIdentifier identifier) {
+
+ ensureResolutionManifestForTable(identifier);
+ PolarisResolvedPathWrapper target =
+ resolutionManifest.getResolvedPath(
+ identifier, PolarisEntityType.TABLE_LIKE, PolarisEntitySubType.ICEBERG_TABLE, true);
+
+ // If the table doesn't exist, we still need to check allowed locations from the parent
+ // namespace.
Review Comment:
[doubt] is this a valid state ?
##########
runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java:
##########
@@ -844,6 +869,37 @@ ALLOW_FEDERATED_CATALOGS_CREDENTIAL_VENDING, getResolvedCatalogEntity())) {
return responseBuilder;
}
+ private AccessDelegationMode selectAccessDelegationMode(
+ Set<AccessDelegationMode> delegationModes) {
Review Comment:
does linking this part of spec https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml#L1859 helps in putting some context that server is allowed to decide which one to choose ?
##########
runtime/service/src/main/java/org/apache/polaris/service/catalog/io/AccessConfigProvider.java:
##########
@@ -101,4 +120,58 @@ public AccessConfig getAccessConfig(
storageInfo.get(),
refreshCredentialsEndpoint);
}
+
+ /**
+ * Generates a remote signing configuration for accessing table storage at explicit locations.
+ *
+ * @param callContext the call context containing realm, principal, and security context
+ * @param catalogName the name of the catalog
+ * @param tableIdentifier the table identifier, used for logging and refresh endpoint construction
+ * @param resolvedPath the entity hierarchy to search for storage configuration
+ * @return {@link AccessConfig} with scoped credentials and metadata; empty if no storage config
+ * found
+ */
+ public AccessConfig getAccessConfigForRemoteSigning(
+ @Nonnull CallContext callContext,
+ @Nonnull String catalogName,
+ @Nonnull TableIdentifier tableIdentifier,
+ @Nonnull PolarisResolvedPathWrapper resolvedPath) {
+ LOGGER
+ .atDebug()
+ .addKeyValue("tableIdentifier", tableIdentifier)
+ .log("Fetching remote signing config for table");
+
+ Optional<PolarisEntity> storageInfo = FileIOUtil.findStorageInfoFromHierarchy(resolvedPath);
+ Optional<PolarisStorageConfigurationInfo> configurationInfo =
+ storageInfo
+ .map(PolarisEntity::getInternalPropertiesAsMap)
+ .map(info -> info.get(PolarisEntityConstants.getStorageConfigInfoPropertyName()))
+ .map(PolarisStorageConfigurationInfo::deserialize);
+
+ if (configurationInfo.isEmpty()) {
+ LOGGER
+ .atWarn()
+ .addKeyValue("tableIdentifier", tableIdentifier)
+ .log("Table entity has no storage configuration in its hierarchy");
+ return AccessConfig.EMPTY;
+ }
+
+ PolarisStorageIntegration<AwsStorageConfigurationInfo> storageIntegration =
+ storageIntegrationProvider.getStorageIntegrationForConfig(configurationInfo.get());
+
+ if (!(storageIntegration
+ instanceof AwsCredentialsStorageIntegration awsCredentialsStorageIntegration)) {
+ LOGGER
+ .atWarn()
+ .addKeyValue("tableIdentifier", tableIdentifier)
+ .log("Table entity storage integration is not an AWS credentials storage integration");
+ return AccessConfig.EMPTY;
+ }
+
+ String prefix = prefixParser.catalogNameToPrefix(callContext.getRealmContext(), catalogName);
+ URI signerUri = uriInfo.getBaseUriBuilder().path(PolarisResourcePaths.API_PATH_SEGMENT).build();
Review Comment:
can we add a TODO to handle proxies here ?
##########
runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java:
##########
@@ -844,6 +869,37 @@ ALLOW_FEDERATED_CATALOGS_CREDENTIAL_VENDING, getResolvedCatalogEntity())) {
return responseBuilder;
}
+ private AccessDelegationMode selectAccessDelegationMode(
+ Set<AccessDelegationMode> delegationModes) {
+
+ if (delegationModes.isEmpty()) {
+ return UNKNOWN;
+ }
+
+ if (delegationModes.size() == 1) {
+ return delegationModes.iterator().next();
+ }
+
+ if (delegationModes.contains(VENDED_CREDENTIALS) && delegationModes.contains(REMOTE_SIGNING)) {
+
+ boolean skipCredIndirection =
+ realmConfig.getConfig(FeatureConfiguration.SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION);
+
+ boolean credentialSubscopingAllowed =
+ baseCatalog instanceof IcebergCatalog
Review Comment:
If its an Iceberg catalog but doesn't support sts wouldn't we prefer remote-signing then ?
##########
runtime/service/src/main/java/org/apache/polaris/service/catalog/io/AccessConfigProvider.java:
##########
@@ -101,4 +120,58 @@ public AccessConfig getAccessConfig(
storageInfo.get(),
refreshCredentialsEndpoint);
}
+
+ /**
+ * Generates a remote signing configuration for accessing table storage at explicit locations.
+ *
+ * @param callContext the call context containing realm, principal, and security context
+ * @param catalogName the name of the catalog
+ * @param tableIdentifier the table identifier, used for logging and refresh endpoint construction
+ * @param resolvedPath the entity hierarchy to search for storage configuration
+ * @return {@link AccessConfig} with scoped credentials and metadata; empty if no storage config
+ * found
+ */
+ public AccessConfig getAccessConfigForRemoteSigning(
+ @Nonnull CallContext callContext,
+ @Nonnull String catalogName,
+ @Nonnull TableIdentifier tableIdentifier,
+ @Nonnull PolarisResolvedPathWrapper resolvedPath) {
+ LOGGER
+ .atDebug()
+ .addKeyValue("tableIdentifier", tableIdentifier)
+ .log("Fetching remote signing config for table");
+
+ Optional<PolarisEntity> storageInfo = FileIOUtil.findStorageInfoFromHierarchy(resolvedPath);
+ Optional<PolarisStorageConfigurationInfo> configurationInfo =
+ storageInfo
+ .map(PolarisEntity::getInternalPropertiesAsMap)
+ .map(info -> info.get(PolarisEntityConstants.getStorageConfigInfoPropertyName()))
+ .map(PolarisStorageConfigurationInfo::deserialize);
+
+ if (configurationInfo.isEmpty()) {
+ LOGGER
+ .atWarn()
+ .addKeyValue("tableIdentifier", tableIdentifier)
+ .log("Table entity has no storage configuration in its hierarchy");
+ return AccessConfig.EMPTY;
+ }
+
+ PolarisStorageIntegration<AwsStorageConfigurationInfo> storageIntegration =
+ storageIntegrationProvider.getStorageIntegrationForConfig(configurationInfo.get());
+
+ if (!(storageIntegration
+ instanceof AwsCredentialsStorageIntegration awsCredentialsStorageIntegration)) {
+ LOGGER
+ .atWarn()
+ .addKeyValue("tableIdentifier", tableIdentifier)
+ .log("Table entity storage integration is not an AWS credentials storage integration");
+ return AccessConfig.EMPTY;
+ }
+
+ String prefix = prefixParser.catalogNameToPrefix(callContext.getRealmContext(), catalogName);
+ URI signerUri = uriInfo.getBaseUriBuilder().path(PolarisResourcePaths.API_PATH_SEGMENT).build();
Review Comment:
can we add a TODO in code to handle proxies ?
##########
runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java:
##########
@@ -844,6 +869,37 @@ ALLOW_FEDERATED_CATALOGS_CREDENTIAL_VENDING, getResolvedCatalogEntity())) {
return responseBuilder;
}
+ private AccessDelegationMode selectAccessDelegationMode(
+ Set<AccessDelegationMode> delegationModes) {
+
+ if (delegationModes.isEmpty()) {
+ return UNKNOWN;
+ }
+
+ if (delegationModes.size() == 1) {
+ return delegationModes.iterator().next();
+ }
+
+ if (delegationModes.contains(VENDED_CREDENTIALS) && delegationModes.contains(REMOTE_SIGNING)) {
+
+ boolean skipCredIndirection =
+ realmConfig.getConfig(FeatureConfiguration.SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION);
+
+ boolean credentialSubscopingAllowed =
+ baseCatalog instanceof IcebergCatalog
Review Comment:
what if its an IcebergCatalog but doesn't support STS ?
##########
runtime/service/src/main/java/org/apache/polaris/service/storage/s3/sign/S3RemoteSigningCatalogHandler.java:
##########
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.service.storage.s3.sign;
+
+import java.util.Collection;
+import java.util.EnumSet;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.auth.PolarisAuthorizableOperation;
+import org.apache.polaris.core.auth.PolarisAuthorizer;
+import org.apache.polaris.core.auth.PolarisPrincipal;
+import org.apache.polaris.core.config.FeatureConfiguration;
+import org.apache.polaris.core.config.RealmConfig;
+import org.apache.polaris.core.context.CallContext;
+import org.apache.polaris.core.entity.CatalogEntity;
+import org.apache.polaris.core.entity.PolarisEntity;
+import org.apache.polaris.core.entity.PolarisEntityConstants;
+import org.apache.polaris.core.persistence.PolarisResolvedPathWrapper;
+import org.apache.polaris.core.persistence.resolver.ResolutionManifestFactory;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisStorageActions;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.apache.polaris.service.catalog.common.CatalogHandler;
+import org.apache.polaris.service.catalog.common.CatalogUtils;
+import org.apache.polaris.service.catalog.io.FileIOUtil;
+import org.apache.polaris.service.context.catalog.CallContextCatalogFactory;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignRequest;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignResponse;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class S3RemoteSigningCatalogHandler extends CatalogHandler implements AutoCloseable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(S3RemoteSigningCatalogHandler.class);
+
+ private final CallContextCatalogFactory catalogFactory;
+ private final S3RequestSigner s3RequestSigner;
+
+ private CatalogEntity catalogEntity;
+ private Catalog baseCatalog;
+
+ public S3RemoteSigningCatalogHandler(
+ PolarisDiagnostics diagnostics,
+ CallContext callContext,
+ ResolutionManifestFactory resolutionManifestFactory,
+ CallContextCatalogFactory catalogFactory,
+ PolarisPrincipal polarisPrincipal,
+ String catalogName,
+ PolarisAuthorizer authorizer,
+ S3RequestSigner s3RequestSigner) {
+ super(
+ diagnostics,
+ callContext,
+ resolutionManifestFactory,
+ polarisPrincipal,
+ catalogName,
+ authorizer,
+ // external catalogs are not supported for S3 remote signing
+ null,
+ null);
+ this.catalogFactory = catalogFactory;
+ this.s3RequestSigner = s3RequestSigner;
+ }
+
+ @Override
+ protected void initializeCatalog() {
+ catalogEntity =
+ CatalogEntity.of(resolutionManifest.getResolvedReferenceCatalogEntity().getRawLeafEntity());
+ if (catalogEntity.isExternal()) {
+ throw new ForbiddenException("Cannot use S3 remote signing with federated catalogs.");
+ }
+ baseCatalog =
+ catalogFactory.createCallContextCatalog(callContext, polarisPrincipal, resolutionManifest);
+ }
+
+ public PolarisS3SignResponse signS3Request(
+ PolarisS3SignRequest s3SignRequest, TableIdentifier tableIdentifier) {
+
+ LOGGER.debug("Requesting s3 signing for {}: {}", tableIdentifier, s3SignRequest);
+
+ PolarisAuthorizableOperation authzOp =
+ s3SignRequest.write()
+ ? PolarisAuthorizableOperation.SIGN_S3_WRITE_REQUEST
+ : PolarisAuthorizableOperation.SIGN_S3_READ_REQUEST;
+
+ authorizeRemoteSigningOrThrow(EnumSet.of(authzOp), tableIdentifier);
+
+ // Must be done after the authorization check, as the auth check creates the catalog entity
+ throwIfRemoteSigningNotEnabled(callContext.getRealmConfig(), catalogEntity);
+
+ var result =
+ InMemoryStorageIntegration.validateAllowedLocations(
+ callContext.getRealmConfig(),
+ getAllowedLocations(tableIdentifier),
+ getStorageActions(s3SignRequest),
+ getTargetLocations(s3SignRequest));
+
+ if (result.values().stream().anyMatch(r -> r.values().stream().anyMatch(v -> !v.isSuccess()))) {
+ throw new ForbiddenException("Requested S3 location is not allowed.");
+ }
+
+ PolarisS3SignResponse s3SignResponse = s3RequestSigner.signRequest(s3SignRequest);
+ LOGGER.debug("S3 signing response: {}", s3SignResponse);
+
+ return s3SignResponse;
+ }
+
+ private Collection<String> getAllowedLocations(TableIdentifier tableIdentifier) {
+
+ if (baseCatalog.tableExists(tableIdentifier)) {
+
+ // If the table exists, get allowed locations from the table metadata
+ Table table = baseCatalog.loadTable(tableIdentifier);
+ if (table instanceof BaseTable baseTable) {
+ return StorageUtil.getLocationsUsedByTable(baseTable.operations().current());
+ }
+
+ throw new ForbiddenException("No storage configuration found for table.");
+
+ } else {
+
+ // If the table or view doesn't exist, the engine might be writing the manifests before the
+ // table creation is committed. In this case, we still need to check allowed locations from
+ // the parent entities.
Review Comment:
is this the case of staged table ? i wondering if we should block on this case ? becasue it may happen the table is different from namespace location ?
##########
runtime/service/src/main/java/org/apache/polaris/service/storage/s3/sign/S3RemoteSigningCatalogHandler.java:
##########
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.service.storage.s3.sign;
+
+import java.util.Collection;
+import java.util.EnumSet;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.auth.PolarisAuthorizableOperation;
+import org.apache.polaris.core.auth.PolarisAuthorizer;
+import org.apache.polaris.core.auth.PolarisPrincipal;
+import org.apache.polaris.core.config.FeatureConfiguration;
+import org.apache.polaris.core.config.RealmConfig;
+import org.apache.polaris.core.context.CallContext;
+import org.apache.polaris.core.entity.CatalogEntity;
+import org.apache.polaris.core.entity.PolarisEntity;
+import org.apache.polaris.core.entity.PolarisEntityConstants;
+import org.apache.polaris.core.persistence.PolarisResolvedPathWrapper;
+import org.apache.polaris.core.persistence.resolver.ResolutionManifestFactory;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisStorageActions;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.apache.polaris.service.catalog.common.CatalogHandler;
+import org.apache.polaris.service.catalog.common.CatalogUtils;
+import org.apache.polaris.service.catalog.io.FileIOUtil;
+import org.apache.polaris.service.context.catalog.CallContextCatalogFactory;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignRequest;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignResponse;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class S3RemoteSigningCatalogHandler extends CatalogHandler implements AutoCloseable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(S3RemoteSigningCatalogHandler.class);
+
+ private final CallContextCatalogFactory catalogFactory;
+ private final S3RequestSigner s3RequestSigner;
+
+ private CatalogEntity catalogEntity;
+ private Catalog baseCatalog;
+
+ public S3RemoteSigningCatalogHandler(
+ PolarisDiagnostics diagnostics,
+ CallContext callContext,
+ ResolutionManifestFactory resolutionManifestFactory,
+ CallContextCatalogFactory catalogFactory,
+ PolarisPrincipal polarisPrincipal,
+ String catalogName,
+ PolarisAuthorizer authorizer,
+ S3RequestSigner s3RequestSigner) {
+ super(
+ diagnostics,
+ callContext,
+ resolutionManifestFactory,
+ polarisPrincipal,
+ catalogName,
+ authorizer,
+ // external catalogs are not supported for S3 remote signing
+ null,
+ null);
+ this.catalogFactory = catalogFactory;
+ this.s3RequestSigner = s3RequestSigner;
+ }
+
+ @Override
+ protected void initializeCatalog() {
+ catalogEntity =
+ CatalogEntity.of(resolutionManifest.getResolvedReferenceCatalogEntity().getRawLeafEntity());
+ if (catalogEntity.isExternal()) {
+ throw new ForbiddenException("Cannot use S3 remote signing with federated catalogs.");
+ }
+ baseCatalog =
+ catalogFactory.createCallContextCatalog(callContext, polarisPrincipal, resolutionManifest);
+ }
+
+ public PolarisS3SignResponse signS3Request(
+ PolarisS3SignRequest s3SignRequest, TableIdentifier tableIdentifier) {
+
+ LOGGER.debug("Requesting s3 signing for {}: {}", tableIdentifier, s3SignRequest);
+
+ PolarisAuthorizableOperation authzOp =
+ s3SignRequest.write()
+ ? PolarisAuthorizableOperation.SIGN_S3_WRITE_REQUEST
+ : PolarisAuthorizableOperation.SIGN_S3_READ_REQUEST;
+
+ authorizeRemoteSigningOrThrow(EnumSet.of(authzOp), tableIdentifier);
+
+ // Must be done after the authorization check, as the auth check creates the catalog entity
+ throwIfRemoteSigningNotEnabled(callContext.getRealmConfig(), catalogEntity);
+
+ var result =
+ InMemoryStorageIntegration.validateAllowedLocations(
+ callContext.getRealmConfig(),
+ getAllowedLocations(tableIdentifier),
+ getStorageActions(s3SignRequest),
+ getTargetLocations(s3SignRequest));
+
+ if (result.values().stream().anyMatch(r -> r.values().stream().anyMatch(v -> !v.isSuccess()))) {
+ throw new ForbiddenException("Requested S3 location is not allowed.");
+ }
+
+ PolarisS3SignResponse s3SignResponse = s3RequestSigner.signRequest(s3SignRequest);
+ LOGGER.debug("S3 signing response: {}", s3SignResponse);
+
+ return s3SignResponse;
+ }
+
+ private Collection<String> getAllowedLocations(TableIdentifier tableIdentifier) {
+
+ if (baseCatalog.tableExists(tableIdentifier)) {
+
+ // If the table exists, get allowed locations from the table metadata
+ Table table = baseCatalog.loadTable(tableIdentifier);
+ if (table instanceof BaseTable baseTable) {
+ return StorageUtil.getLocationsUsedByTable(baseTable.operations().current());
+ }
+
+ throw new ForbiddenException("No storage configuration found for table.");
+
+ } else {
+
+ // If the table or view doesn't exist, the engine might be writing the manifests before the
+ // table creation is committed. In this case, we still need to check allowed locations from
+ // the parent entities.
+
+ PolarisResolvedPathWrapper resolvedPath =
+ CatalogUtils.findResolvedStorageEntity(resolutionManifest, tableIdentifier);
+
+ Optional<PolarisEntity> storageInfo = FileIOUtil.findStorageInfoFromHierarchy(resolvedPath);
+
+ var configurationInfo =
+ storageInfo
+ .map(PolarisEntity::getInternalPropertiesAsMap)
+ .map(info -> info.get(PolarisEntityConstants.getStorageConfigInfoPropertyName()))
+ .map(PolarisStorageConfigurationInfo::deserialize);
+
+ if (configurationInfo.isEmpty()) {
+ throw new ForbiddenException("No storage configuration found for table.");
+ }
+
+ return configurationInfo.get().getAllowedLocations();
+ }
+ }
+
+ private Set<PolarisStorageActions> getStorageActions(PolarisS3SignRequest s3SignRequest) {
+ // TODO M2: better handling of DELETE and LIST
+ return s3SignRequest.write()
+ ? Set.of(PolarisStorageActions.WRITE)
+ : Set.of(PolarisStorageActions.READ);
+ }
+
+ private Set<String> getTargetLocations(PolarisS3SignRequest s3SignRequest) {
+ // TODO M2: map http URI to s3 URI
+ return Set.of();
+ }
+
+ public static void throwIfRemoteSigningNotEnabled(
+ RealmConfig realmConfig, CatalogEntity catalogEntity) {
+ if (catalogEntity.isExternal()) {
+ throw new ForbiddenException("Remote signing is not enabled for external catalogs.");
+ }
Review Comment:
I understand we discussed this before for federated catalog we should disable remote signing but during that time we didn;t support cred vending for them either, i wonder if its a good time to reconsider that remote signing would be possible too (ofc in a new PR)
##########
runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java:
##########
@@ -844,6 +869,37 @@ ALLOW_FEDERATED_CATALOGS_CREDENTIAL_VENDING, getResolvedCatalogEntity())) {
return responseBuilder;
}
+ private AccessDelegationMode selectAccessDelegationMode(
Review Comment:
would it be helpful to add a comment here and link irc spec here to pick the delegation mode they prefer ?
##########
runtime/service/src/main/java/org/apache/polaris/service/storage/s3/sign/S3RemoteSigningCatalogHandler.java:
##########
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.service.storage.s3.sign;
+
+import java.util.Collection;
+import java.util.EnumSet;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.auth.PolarisAuthorizableOperation;
+import org.apache.polaris.core.auth.PolarisAuthorizer;
+import org.apache.polaris.core.auth.PolarisPrincipal;
+import org.apache.polaris.core.config.FeatureConfiguration;
+import org.apache.polaris.core.config.RealmConfig;
+import org.apache.polaris.core.context.CallContext;
+import org.apache.polaris.core.entity.CatalogEntity;
+import org.apache.polaris.core.entity.PolarisEntity;
+import org.apache.polaris.core.entity.PolarisEntityConstants;
+import org.apache.polaris.core.persistence.PolarisResolvedPathWrapper;
+import org.apache.polaris.core.persistence.resolver.ResolutionManifestFactory;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisStorageActions;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.apache.polaris.service.catalog.common.CatalogHandler;
+import org.apache.polaris.service.catalog.common.CatalogUtils;
+import org.apache.polaris.service.catalog.io.FileIOUtil;
+import org.apache.polaris.service.context.catalog.CallContextCatalogFactory;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignRequest;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignResponse;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class S3RemoteSigningCatalogHandler extends CatalogHandler implements AutoCloseable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(S3RemoteSigningCatalogHandler.class);
+
+ private final CallContextCatalogFactory catalogFactory;
+ private final S3RequestSigner s3RequestSigner;
+
+ private CatalogEntity catalogEntity;
+ private Catalog baseCatalog;
+
+ public S3RemoteSigningCatalogHandler(
+ PolarisDiagnostics diagnostics,
+ CallContext callContext,
+ ResolutionManifestFactory resolutionManifestFactory,
+ CallContextCatalogFactory catalogFactory,
+ PolarisPrincipal polarisPrincipal,
+ String catalogName,
+ PolarisAuthorizer authorizer,
+ S3RequestSigner s3RequestSigner) {
+ super(
+ diagnostics,
+ callContext,
+ resolutionManifestFactory,
+ polarisPrincipal,
+ catalogName,
+ authorizer,
+ // external catalogs are not supported for S3 remote signing
+ null,
+ null);
+ this.catalogFactory = catalogFactory;
+ this.s3RequestSigner = s3RequestSigner;
+ }
+
+ @Override
+ protected void initializeCatalog() {
+ catalogEntity =
+ CatalogEntity.of(resolutionManifest.getResolvedReferenceCatalogEntity().getRawLeafEntity());
+ if (catalogEntity.isExternal()) {
+ throw new ForbiddenException("Cannot use S3 remote signing with federated catalogs.");
+ }
+ baseCatalog =
+ catalogFactory.createCallContextCatalog(callContext, polarisPrincipal, resolutionManifest);
+ }
+
+ public PolarisS3SignResponse signS3Request(
+ PolarisS3SignRequest s3SignRequest, TableIdentifier tableIdentifier) {
+
+ LOGGER.debug("Requesting s3 signing for {}: {}", tableIdentifier, s3SignRequest);
+
+ PolarisAuthorizableOperation authzOp =
+ s3SignRequest.write()
+ ? PolarisAuthorizableOperation.SIGN_S3_WRITE_REQUEST
+ : PolarisAuthorizableOperation.SIGN_S3_READ_REQUEST;
+
+ authorizeRemoteSigningOrThrow(EnumSet.of(authzOp), tableIdentifier);
+
+ // Must be done after the authorization check, as the auth check creates the catalog entity
+ throwIfRemoteSigningNotEnabled(callContext.getRealmConfig(), catalogEntity);
+
+ var result =
+ InMemoryStorageIntegration.validateAllowedLocations(
+ callContext.getRealmConfig(),
+ getAllowedLocations(tableIdentifier),
+ getStorageActions(s3SignRequest),
+ getTargetLocations(s3SignRequest));
+
+ if (result.values().stream().anyMatch(r -> r.values().stream().anyMatch(v -> !v.isSuccess()))) {
+ throw new ForbiddenException("Requested S3 location is not allowed.");
+ }
+
+ PolarisS3SignResponse s3SignResponse = s3RequestSigner.signRequest(s3SignRequest);
+ LOGGER.debug("S3 signing response: {}", s3SignResponse);
+
+ return s3SignResponse;
+ }
+
+ private Collection<String> getAllowedLocations(TableIdentifier tableIdentifier) {
+
+ if (baseCatalog.tableExists(tableIdentifier)) {
+
+ // If the table exists, get allowed locations from the table metadata
+ Table table = baseCatalog.loadTable(tableIdentifier);
Review Comment:
for each SIGN request a loadTable call is super expensive, if someone enables this in prod by accident for a table million of files the service might become unresponsive ?
[suggestion / optional to address] wondering for now if we should add a prod readiness check to keep this disabled for prod ?
##########
runtime/service/src/main/java/org/apache/polaris/service/storage/s3/sign/S3RemoteSigningCatalogHandler.java:
##########
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.service.storage.s3.sign;
+
+import java.util.Collection;
+import java.util.EnumSet;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.auth.PolarisAuthorizableOperation;
+import org.apache.polaris.core.auth.PolarisAuthorizer;
+import org.apache.polaris.core.auth.PolarisPrincipal;
+import org.apache.polaris.core.config.FeatureConfiguration;
+import org.apache.polaris.core.config.RealmConfig;
+import org.apache.polaris.core.context.CallContext;
+import org.apache.polaris.core.entity.CatalogEntity;
+import org.apache.polaris.core.entity.PolarisEntity;
+import org.apache.polaris.core.entity.PolarisEntityConstants;
+import org.apache.polaris.core.persistence.PolarisResolvedPathWrapper;
+import org.apache.polaris.core.persistence.resolver.ResolutionManifestFactory;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisStorageActions;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.apache.polaris.service.catalog.common.CatalogHandler;
+import org.apache.polaris.service.catalog.common.CatalogUtils;
+import org.apache.polaris.service.catalog.io.FileIOUtil;
+import org.apache.polaris.service.context.catalog.CallContextCatalogFactory;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignRequest;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignResponse;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class S3RemoteSigningCatalogHandler extends CatalogHandler implements AutoCloseable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(S3RemoteSigningCatalogHandler.class);
+
+ private final CallContextCatalogFactory catalogFactory;
+ private final S3RequestSigner s3RequestSigner;
+
+ private CatalogEntity catalogEntity;
+ private Catalog baseCatalog;
+
+ public S3RemoteSigningCatalogHandler(
+ PolarisDiagnostics diagnostics,
+ CallContext callContext,
+ ResolutionManifestFactory resolutionManifestFactory,
+ CallContextCatalogFactory catalogFactory,
+ PolarisPrincipal polarisPrincipal,
+ String catalogName,
+ PolarisAuthorizer authorizer,
+ S3RequestSigner s3RequestSigner) {
+ super(
+ diagnostics,
+ callContext,
+ resolutionManifestFactory,
+ polarisPrincipal,
+ catalogName,
+ authorizer,
+ // external catalogs are not supported for S3 remote signing
+ null,
+ null);
+ this.catalogFactory = catalogFactory;
+ this.s3RequestSigner = s3RequestSigner;
+ }
+
+ @Override
+ protected void initializeCatalog() {
+ catalogEntity =
+ CatalogEntity.of(resolutionManifest.getResolvedReferenceCatalogEntity().getRawLeafEntity());
+ if (catalogEntity.isExternal()) {
+ throw new ForbiddenException("Cannot use S3 remote signing with federated catalogs.");
+ }
+ baseCatalog =
+ catalogFactory.createCallContextCatalog(callContext, polarisPrincipal, resolutionManifest);
+ }
+
+ public PolarisS3SignResponse signS3Request(
+ PolarisS3SignRequest s3SignRequest, TableIdentifier tableIdentifier) {
+
+ LOGGER.debug("Requesting s3 signing for {}: {}", tableIdentifier, s3SignRequest);
Review Comment:
do we need to explicity log it here or quarkus supports Access log, asking because i don't see such logging in Iceberg Catalog Handler
##########
runtime/service/src/main/java/org/apache/polaris/service/storage/s3/sign/S3RemoteSigningCatalogHandler.java:
##########
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.service.storage.s3.sign;
+
+import java.util.Collection;
+import java.util.EnumSet;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.auth.PolarisAuthorizableOperation;
+import org.apache.polaris.core.auth.PolarisAuthorizer;
+import org.apache.polaris.core.auth.PolarisPrincipal;
+import org.apache.polaris.core.config.FeatureConfiguration;
+import org.apache.polaris.core.config.RealmConfig;
+import org.apache.polaris.core.context.CallContext;
+import org.apache.polaris.core.entity.CatalogEntity;
+import org.apache.polaris.core.entity.PolarisEntity;
+import org.apache.polaris.core.entity.PolarisEntityConstants;
+import org.apache.polaris.core.persistence.PolarisResolvedPathWrapper;
+import org.apache.polaris.core.persistence.resolver.ResolutionManifestFactory;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisStorageActions;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.apache.polaris.service.catalog.common.CatalogHandler;
+import org.apache.polaris.service.catalog.common.CatalogUtils;
+import org.apache.polaris.service.catalog.io.FileIOUtil;
+import org.apache.polaris.service.context.catalog.CallContextCatalogFactory;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignRequest;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignResponse;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class S3RemoteSigningCatalogHandler extends CatalogHandler implements AutoCloseable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(S3RemoteSigningCatalogHandler.class);
+
+ private final CallContextCatalogFactory catalogFactory;
+ private final S3RequestSigner s3RequestSigner;
+
+ private CatalogEntity catalogEntity;
+ private Catalog baseCatalog;
+
+ public S3RemoteSigningCatalogHandler(
+ PolarisDiagnostics diagnostics,
+ CallContext callContext,
+ ResolutionManifestFactory resolutionManifestFactory,
+ CallContextCatalogFactory catalogFactory,
+ PolarisPrincipal polarisPrincipal,
+ String catalogName,
+ PolarisAuthorizer authorizer,
+ S3RequestSigner s3RequestSigner) {
+ super(
+ diagnostics,
+ callContext,
+ resolutionManifestFactory,
+ polarisPrincipal,
+ catalogName,
+ authorizer,
+ // external catalogs are not supported for S3 remote signing
+ null,
+ null);
+ this.catalogFactory = catalogFactory;
+ this.s3RequestSigner = s3RequestSigner;
+ }
+
+ @Override
+ protected void initializeCatalog() {
+ catalogEntity =
+ CatalogEntity.of(resolutionManifest.getResolvedReferenceCatalogEntity().getRawLeafEntity());
+ if (catalogEntity.isExternal()) {
+ throw new ForbiddenException("Cannot use S3 remote signing with federated catalogs.");
+ }
+ baseCatalog =
+ catalogFactory.createCallContextCatalog(callContext, polarisPrincipal, resolutionManifest);
+ }
+
+ public PolarisS3SignResponse signS3Request(
+ PolarisS3SignRequest s3SignRequest, TableIdentifier tableIdentifier) {
+
+ LOGGER.debug("Requesting s3 signing for {}: {}", tableIdentifier, s3SignRequest);
+
+ PolarisAuthorizableOperation authzOp =
+ s3SignRequest.write()
+ ? PolarisAuthorizableOperation.SIGN_S3_WRITE_REQUEST
+ : PolarisAuthorizableOperation.SIGN_S3_READ_REQUEST;
+
+ authorizeRemoteSigningOrThrow(EnumSet.of(authzOp), tableIdentifier);
+
+ // Must be done after the authorization check, as the auth check creates the catalog entity
+ throwIfRemoteSigningNotEnabled(callContext.getRealmConfig(), catalogEntity);
+
+ var result =
+ InMemoryStorageIntegration.validateAllowedLocations(
+ callContext.getRealmConfig(),
+ getAllowedLocations(tableIdentifier),
+ getStorageActions(s3SignRequest),
+ getTargetLocations(s3SignRequest));
+
+ if (result.values().stream().anyMatch(r -> r.values().stream().anyMatch(v -> !v.isSuccess()))) {
+ throw new ForbiddenException("Requested S3 location is not allowed.");
+ }
+
+ PolarisS3SignResponse s3SignResponse = s3RequestSigner.signRequest(s3SignRequest);
+ LOGGER.debug("S3 signing response: {}", s3SignResponse);
+
+ return s3SignResponse;
+ }
+
+ private Collection<String> getAllowedLocations(TableIdentifier tableIdentifier) {
+
+ if (baseCatalog.tableExists(tableIdentifier)) {
+
+ // If the table exists, get allowed locations from the table metadata
+ Table table = baseCatalog.loadTable(tableIdentifier);
+ if (table instanceof BaseTable baseTable) {
+ return StorageUtil.getLocationsUsedByTable(baseTable.operations().current());
+ }
+
+ throw new ForbiddenException("No storage configuration found for table.");
+
+ } else {
+
+ // If the table or view doesn't exist, the engine might be writing the manifests before the
+ // table creation is committed. In this case, we still need to check allowed locations from
+ // the parent entities.
+
+ PolarisResolvedPathWrapper resolvedPath =
+ CatalogUtils.findResolvedStorageEntity(resolutionManifest, tableIdentifier);
+
+ Optional<PolarisEntity> storageInfo = FileIOUtil.findStorageInfoFromHierarchy(resolvedPath);
+
+ var configurationInfo =
+ storageInfo
+ .map(PolarisEntity::getInternalPropertiesAsMap)
+ .map(info -> info.get(PolarisEntityConstants.getStorageConfigInfoPropertyName()))
+ .map(PolarisStorageConfigurationInfo::deserialize);
+
+ if (configurationInfo.isEmpty()) {
+ throw new ForbiddenException("No storage configuration found for table.");
+ }
+
+ return configurationInfo.get().getAllowedLocations();
+ }
+ }
+
+ private Set<PolarisStorageActions> getStorageActions(PolarisS3SignRequest s3SignRequest) {
+ // TODO M2: better handling of DELETE and LIST
Review Comment:
[for my understanding] can you please elaborate this case more ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2505969332
##########
runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java:
##########
@@ -844,6 +869,37 @@ ALLOW_FEDERATED_CATALOGS_CREDENTIAL_VENDING, getResolvedCatalogEntity())) {
return responseBuilder;
}
+ private AccessDelegationMode selectAccessDelegationMode(
Review Comment:
would it be helpful to add a comment here and link irc spec here to pick the delegation mode they prefer ?
##########
runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java:
##########
@@ -844,6 +869,37 @@ ALLOW_FEDERATED_CATALOGS_CREDENTIAL_VENDING, getResolvedCatalogEntity())) {
return responseBuilder;
}
+ private AccessDelegationMode selectAccessDelegationMode(
+ Set<AccessDelegationMode> delegationModes) {
+
+ if (delegationModes.isEmpty()) {
+ return UNKNOWN;
+ }
+
+ if (delegationModes.size() == 1) {
+ return delegationModes.iterator().next();
+ }
+
+ if (delegationModes.contains(VENDED_CREDENTIALS) && delegationModes.contains(REMOTE_SIGNING)) {
+
+ boolean skipCredIndirection =
+ realmConfig.getConfig(FeatureConfiguration.SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION);
+
+ boolean credentialSubscopingAllowed =
+ baseCatalog instanceof IcebergCatalog
Review Comment:
If its an Iceberg catalog but doesn't support sts wouldn't we prefer remote-signing then ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2504118201
##########
runtime/service/src/main/java/org/apache/polaris/service/catalog/io/AccessConfigProvider.java:
##########
@@ -101,4 +120,58 @@ public AccessConfig getAccessConfig(
storageInfo.get(),
refreshCredentialsEndpoint);
}
+
+ /**
+ * Generates a remote signing configuration for accessing table storage at explicit locations.
+ *
+ * @param callContext the call context containing realm, principal, and security context
+ * @param catalogName the name of the catalog
+ * @param tableIdentifier the table identifier, used for logging and refresh endpoint construction
+ * @param resolvedPath the entity hierarchy to search for storage configuration
+ * @return {@link AccessConfig} with scoped credentials and metadata; empty if no storage config
+ * found
+ */
+ public AccessConfig getAccessConfigForRemoteSigning(
+ @Nonnull CallContext callContext,
+ @Nonnull String catalogName,
+ @Nonnull TableIdentifier tableIdentifier,
+ @Nonnull PolarisResolvedPathWrapper resolvedPath) {
+ LOGGER
+ .atDebug()
+ .addKeyValue("tableIdentifier", tableIdentifier)
+ .log("Fetching remote signing config for table");
+
+ Optional<PolarisEntity> storageInfo = FileIOUtil.findStorageInfoFromHierarchy(resolvedPath);
+ Optional<PolarisStorageConfigurationInfo> configurationInfo =
+ storageInfo
+ .map(PolarisEntity::getInternalPropertiesAsMap)
+ .map(info -> info.get(PolarisEntityConstants.getStorageConfigInfoPropertyName()))
+ .map(PolarisStorageConfigurationInfo::deserialize);
+
+ if (configurationInfo.isEmpty()) {
+ LOGGER
+ .atWarn()
+ .addKeyValue("tableIdentifier", tableIdentifier)
+ .log("Table entity has no storage configuration in its hierarchy");
+ return AccessConfig.EMPTY;
+ }
+
+ PolarisStorageIntegration<AwsStorageConfigurationInfo> storageIntegration =
+ storageIntegrationProvider.getStorageIntegrationForConfig(configurationInfo.get());
+
+ if (!(storageIntegration
+ instanceof AwsCredentialsStorageIntegration awsCredentialsStorageIntegration)) {
+ LOGGER
+ .atWarn()
+ .addKeyValue("tableIdentifier", tableIdentifier)
+ .log("Table entity storage integration is not an AWS credentials storage integration");
+ return AccessConfig.EMPTY;
+ }
+
+ String prefix = prefixParser.catalogNameToPrefix(callContext.getRealmContext(), catalogName);
+ URI signerUri = uriInfo.getBaseUri().resolve("api/");
Review Comment:
+1 to `PolarisResourcePaths`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2500660568
##########
runtime/service/src/main/java/org/apache/polaris/service/catalog/io/AccessConfigProvider.java:
##########
@@ -101,4 +120,58 @@ public AccessConfig getAccessConfig(
storageInfo.get(),
refreshCredentialsEndpoint);
}
+
+ /**
+ * Generates a remote signing configuration for accessing table storage at explicit locations.
+ *
+ * @param callContext the call context containing realm, principal, and security context
+ * @param catalogName the name of the catalog
+ * @param tableIdentifier the table identifier, used for logging and refresh endpoint construction
+ * @param resolvedPath the entity hierarchy to search for storage configuration
+ * @return {@link AccessConfig} with scoped credentials and metadata; empty if no storage config
+ * found
+ */
+ public AccessConfig getAccessConfigForRemoteSigning(
+ @Nonnull CallContext callContext,
+ @Nonnull String catalogName,
+ @Nonnull TableIdentifier tableIdentifier,
+ @Nonnull PolarisResolvedPathWrapper resolvedPath) {
+ LOGGER
+ .atDebug()
+ .addKeyValue("tableIdentifier", tableIdentifier)
+ .log("Fetching remote signing config for table");
+
+ Optional<PolarisEntity> storageInfo = FileIOUtil.findStorageInfoFromHierarchy(resolvedPath);
+ Optional<PolarisStorageConfigurationInfo> configurationInfo =
+ storageInfo
+ .map(PolarisEntity::getInternalPropertiesAsMap)
+ .map(info -> info.get(PolarisEntityConstants.getStorageConfigInfoPropertyName()))
+ .map(PolarisStorageConfigurationInfo::deserialize);
+
+ if (configurationInfo.isEmpty()) {
+ LOGGER
+ .atWarn()
+ .addKeyValue("tableIdentifier", tableIdentifier)
+ .log("Table entity has no storage configuration in its hierarchy");
+ return AccessConfig.EMPTY;
+ }
+
+ PolarisStorageIntegration<AwsStorageConfigurationInfo> storageIntegration =
+ storageIntegrationProvider.getStorageIntegrationForConfig(configurationInfo.get());
+
+ if (!(storageIntegration
+ instanceof AwsCredentialsStorageIntegration awsCredentialsStorageIntegration)) {
+ LOGGER
+ .atWarn()
+ .addKeyValue("tableIdentifier", tableIdentifier)
+ .log("Table entity storage integration is not an AWS credentials storage integration");
+ return AccessConfig.EMPTY;
+ }
+
+ String prefix = prefixParser.catalogNameToPrefix(callContext.getRealmContext(), catalogName);
+ URI signerUri = uriInfo.getBaseUri().resolve("api/");
Review Comment:
I'm not sure about the `api/` constant... It related to our `@Path` annotations in the generated REST API classes, so it's usage is a bit obscure here. Could we make it a constant in the `s3-sign-service` module?
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/StorageAccessProperty.java:
##########
@@ -52,6 +54,21 @@ public enum StorageAccessProperty {
"the endpoint to load vended credentials for a table from the catalog",
false,
false),
+ AWS_REMOTE_SIGNING_ENABLED(
+ Boolean.class,
+ S3FileIOProperties.REMOTE_SIGNING_ENABLED,
+ "whether to enable remote signing for S3 requests",
+ false),
+ AWS_REMOTE_SIGNER_URI(
+ String.class,
+ S3V4RestSignerClient.S3_SIGNER_URI,
+ "the base URI for the remote signer service, used for signing S3 requests",
Review Comment:
Does this have a reference Iceberg version number where the property becomes respected?
##########
polaris-core/src/main/java/org/apache/polaris/core/rest/PolarisResourcePaths.java:
##########
@@ -78,4 +82,15 @@ public String genericTable(TableIdentifier ident) {
"generic-tables",
RESTUtil.encodeString(ident.name()));
}
+
+ public String s3RemoteSigning(TableIdentifier ident) {
Review Comment:
Is this a `core` concern? I suppose it is only relevant to the REST layer 🤔
##########
runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogAdapter.java:
##########
@@ -828,6 +830,9 @@ public Response getConfig(
.addAll(VIEW_ENDPOINTS)
.addAll(PolarisEndpoints.getSupportedGenericTableEndpoints(realmConfig))
.addAll(PolarisEndpoints.getSupportedPolicyEndpoints(realmConfig))
+ .addAll(
+ PolarisEndpoints.getSupportedRemoteSigningEndpoints(
Review Comment:
I'm not sure it's worth advertising in `ConfigResponse` because it's not a standard endpoint, for which clients might want to check the endpoints list. Clients learn about it via `loadTable` responses, I guess 🤔
##########
runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java:
##########
@@ -801,6 +806,33 @@ private LoadTableResponse.Builder buildLoadTableResponseWithDelegationCredential
return responseBuilder;
}
+ AccessDelegationMode delegationMode = selectAccessDelegationMode(delegationModes);
+
+ if (delegationMode == REMOTE_SIGNING) {
+
+ S3RemoteSigningCatalogHandler.throwIfRemoteSigningNotEnabled(
+ callContext.getRealmConfig(), getResolvedCatalogEntity());
+
+ AccessConfig accessConfig =
+ accessConfigProvider.getAccessConfigForRemoteSigning(
+ callContext, catalogName, tableIdentifier, resolvedStoragePath);
+
+ Map<String, String> credentialConfig = accessConfig.credentials();
+
+ if (!credentialConfig.isEmpty()) {
+ responseBuilder.addAllConfig(credentialConfig);
+ responseBuilder.addCredential(
Review Comment:
Why `addCredential()` when we're remote-signing?
##########
spec/README.md:
##########
@@ -17,14 +17,40 @@
under the License.
-->
-# Polaris API Specifications
-
-Polaris provides two sets of OpenAPI specifications:
-- `polaris-management-service.yml` - Defines the management APIs for using Polaris to create and manage Iceberg catalogs and their principals
-- `polaris-catalog-service.yaml` - Defines the specification for the Polaris Catalog API, which encompasses both the Iceberg REST Catalog API
- and Polaris-native API.
- - `polaris-catalog-apis` - Contains the specification for Polaris-specific Catalog APIs
- - `iceberg-rest-catalog-open-api.yaml` - Contains the specification for Iceberg Rest Catalog API
+# Apache Polaris API Specifications
+
+Apache Polaris provides the following OpenAPI specifications:
+
+- [polaris-management-service.yml](polaris-management-service.yml) - Defines the management APIs for using Apache
+ Polaris to create and manage Apache Iceberg catalogs and their principals
+
+- [polaris-catalog-service.yaml](polaris-catalog-service.yaml) - Defines the specification for the Apache Polaris
+ Catalog API, which encompasses both the Apache Iceberg REST Catalog API and Apache Polaris-native APIs:
+
+ - [iceberg-rest-catalog-open-api.yaml](iceberg-rest-catalog-open-api.yaml) - Contains the specification for
+ Apache Iceberg Rest Catalog API.
+
+ - [polaris-catalog-apis](polaris-catalog-apis) - This folder contains the specifications for Apache
+ Polaris-specific Catalog APIs:
+
+ - [generic-tables-api.yaml](polaris-catalog-apis/generic-tables-api.yaml) - Contains the specification for
+ the Generic Tables API
+
+ - [notifications-api.yaml](polaris-catalog-apis/notifications-api.yaml) - Contains the specification for
+ the Notifications API
+
+ - [oauth-tokens-api.yaml](polaris-catalog-apis/oauth-tokens-api.yaml) - Contains the specification for the
+ internal OAuth Token endpoint, extracted from the Apache Iceberg REST Catalog API.
+
+ - [policy-apis.yaml](polaris-catalog-apis/policy-apis.yaml) - Contains the specification for the Policy APIs.
+
+- [s3-sign](s3-sign) - This folder contains the specifications for S3 remote signing:
+
+ - [iceberg-s3-signer-open-api.yaml](s3-sign/iceberg-s3-signer-open-api.yaml) - Contains the specification of the
Review Comment:
nit: maybe clarify that Polaris uses only type definitions from the spec file?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2263163189
##########
runtime/service/src/main/java/org/apache/polaris/service/storage/aws/signer/S3RemoteSigningCatalogAdapter.java:
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.service.storage.aws.signer;
+
+import com.google.common.annotations.VisibleForTesting;
+import jakarta.enterprise.context.RequestScoped;
+import jakarta.inject.Inject;
+import jakarta.ws.rs.core.Response;
+import jakarta.ws.rs.core.SecurityContext;
+import java.util.function.Function;
+import org.apache.iceberg.aws.s3.signer.S3SignResponse;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.rest.RESTUtil;
+import org.apache.polaris.core.auth.PolarisAuthorizer;
+import org.apache.polaris.core.context.CallContext;
+import org.apache.polaris.core.context.RealmContext;
+import org.apache.polaris.core.persistence.resolver.ResolutionManifestFactory;
+import org.apache.polaris.service.aws.sign.api.IcebergRestS3SignerApiService;
+import org.apache.polaris.service.aws.sign.model.PolarisS3SignRequest;
+import org.apache.polaris.service.catalog.CatalogPrefixParser;
+import org.apache.polaris.service.catalog.common.CatalogAdapter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * An adapter between generated service types like `IcebergRestCatalogApiService` and
+ * `IcebergCatalogHandler`.
+ */
+@RequestScoped
+public class S3RemoteSigningCatalogAdapter
+ implements IcebergRestS3SignerApiService, CatalogAdapter {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(S3RemoteSigningCatalogAdapter.class);
+
+ @Inject CallContext callContext;
+ @Inject ResolutionManifestFactory resolutionManifestFactory;
+ @Inject PolarisAuthorizer polarisAuthorizer;
+ @Inject CatalogPrefixParser prefixParser;
+ @Inject S3RequestSigner s3RequestSigner;
+
+ /**
+ * Execute operations on a catalog wrapper and ensure we close the BaseCatalog afterward. This
+ * will typically ensure the underlying FileIO is closed.
+ */
+ private Response withCatalog(
+ SecurityContext securityContext,
+ String prefix,
+ Function<S3RemoteSigningCatalogHandler, Response> action) {
+ String catalogName = prefixParser.prefixToCatalogName(callContext.getRealmContext(), prefix);
+ try (S3RemoteSigningCatalogHandler handler = newHandlerWrapper(securityContext, catalogName)) {
+ return action.apply(handler);
+ } catch (RuntimeException e) {
+ LOGGER.debug("RuntimeException while operating on catalog. Propagating to caller.", e);
+ throw e;
+ } catch (Exception e) {
+ LOGGER.error("Error while operating on catalog", e);
+ throw new RuntimeException(e);
+ }
+ }
+
+ @VisibleForTesting
+ S3RemoteSigningCatalogHandler newHandlerWrapper(
+ SecurityContext securityContext, String catalogName) {
+ validatePrincipal(securityContext);
+ return new S3RemoteSigningCatalogHandler(
+ callContext,
+ resolutionManifestFactory,
+ securityContext,
+ catalogName,
+ polarisAuthorizer,
+ s3RequestSigner);
+ }
+
+ @Override
+ public Response signS3Request(
+ String prefix,
+ String namespace,
+ String table,
+ PolarisS3SignRequest s3SignRequest,
+ RealmContext realmContext,
+ SecurityContext securityContext) {
+ Namespace ns = decodeNamespace(namespace);
+ TableIdentifier tableIdentifier = TableIdentifier.of(ns, RESTUtil.decodeString(table));
+ return withCatalog(
+ securityContext,
+ prefix,
+ catalog -> {
+ S3SignResponse response = catalog.signS3Request(s3SignRequest, tableIdentifier);
+ return Response.status(Response.Status.OK).entity(response).build();
Review Comment:
In theory the response should include Cache-Control headers. This is left for a later improvement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2549728523
##########
runtime/service/src/main/java/org/apache/polaris/service/storage/s3/sign/S3RemoteSigningCatalogHandler.java:
##########
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.service.storage.s3.sign;
+
+import java.util.Collection;
+import java.util.EnumSet;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.auth.PolarisAuthorizableOperation;
+import org.apache.polaris.core.auth.PolarisAuthorizer;
+import org.apache.polaris.core.auth.PolarisPrincipal;
+import org.apache.polaris.core.config.FeatureConfiguration;
+import org.apache.polaris.core.config.RealmConfig;
+import org.apache.polaris.core.context.CallContext;
+import org.apache.polaris.core.entity.CatalogEntity;
+import org.apache.polaris.core.entity.PolarisEntity;
+import org.apache.polaris.core.entity.PolarisEntityConstants;
+import org.apache.polaris.core.persistence.PolarisResolvedPathWrapper;
+import org.apache.polaris.core.persistence.resolver.ResolutionManifestFactory;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisStorageActions;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.apache.polaris.service.catalog.common.CatalogHandler;
+import org.apache.polaris.service.catalog.common.CatalogUtils;
+import org.apache.polaris.service.catalog.io.FileIOUtil;
+import org.apache.polaris.service.context.catalog.CallContextCatalogFactory;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignRequest;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignResponse;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class S3RemoteSigningCatalogHandler extends CatalogHandler implements AutoCloseable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(S3RemoteSigningCatalogHandler.class);
+
+ private final CallContextCatalogFactory catalogFactory;
+ private final S3RequestSigner s3RequestSigner;
+
+ private CatalogEntity catalogEntity;
+ private Catalog baseCatalog;
+
+ public S3RemoteSigningCatalogHandler(
+ PolarisDiagnostics diagnostics,
+ CallContext callContext,
+ ResolutionManifestFactory resolutionManifestFactory,
+ CallContextCatalogFactory catalogFactory,
+ PolarisPrincipal polarisPrincipal,
+ String catalogName,
+ PolarisAuthorizer authorizer,
+ S3RequestSigner s3RequestSigner) {
+ super(
+ diagnostics,
+ callContext,
+ resolutionManifestFactory,
+ polarisPrincipal,
+ catalogName,
+ authorizer,
+ // external catalogs are not supported for S3 remote signing
+ null,
+ null);
+ this.catalogFactory = catalogFactory;
+ this.s3RequestSigner = s3RequestSigner;
+ }
+
+ @Override
+ protected void initializeCatalog() {
+ catalogEntity =
+ CatalogEntity.of(resolutionManifest.getResolvedReferenceCatalogEntity().getRawLeafEntity());
+ if (catalogEntity.isExternal()) {
+ throw new ForbiddenException("Cannot use S3 remote signing with federated catalogs.");
+ }
+ baseCatalog =
+ catalogFactory.createCallContextCatalog(callContext, polarisPrincipal, resolutionManifest);
+ }
+
+ public PolarisS3SignResponse signS3Request(
+ PolarisS3SignRequest s3SignRequest, TableIdentifier tableIdentifier) {
+
+ LOGGER.debug("Requesting s3 signing for {}: {}", tableIdentifier, s3SignRequest);
+
+ PolarisAuthorizableOperation authzOp =
+ s3SignRequest.write()
+ ? PolarisAuthorizableOperation.SIGN_S3_WRITE_REQUEST
+ : PolarisAuthorizableOperation.SIGN_S3_READ_REQUEST;
+
+ authorizeRemoteSigningOrThrow(EnumSet.of(authzOp), tableIdentifier);
+
+ // Must be done after the authorization check, as the auth check creates the catalog entity
+ throwIfRemoteSigningNotEnabled(callContext.getRealmConfig(), catalogEntity);
+
+ var result =
+ InMemoryStorageIntegration.validateAllowedLocations(
+ callContext.getRealmConfig(),
+ getAllowedLocations(tableIdentifier),
+ getStorageActions(s3SignRequest),
+ getTargetLocations(s3SignRequest));
+
+ if (result.values().stream().anyMatch(r -> r.values().stream().anyMatch(v -> !v.isSuccess()))) {
+ throw new ForbiddenException("Requested S3 location is not allowed.");
+ }
+
+ PolarisS3SignResponse s3SignResponse = s3RequestSigner.signRequest(s3SignRequest);
+ LOGGER.debug("S3 signing response: {}", s3SignResponse);
+
+ return s3SignResponse;
+ }
+
+ private Collection<String> getAllowedLocations(TableIdentifier tableIdentifier) {
+
+ if (baseCatalog.tableExists(tableIdentifier)) {
+
+ // If the table exists, get allowed locations from the table metadata
+ Table table = baseCatalog.loadTable(tableIdentifier);
Review Comment:
Ah now I get it! Good question! I think there is a problem indeed.
For the base location, we're good, it is stored in the properties map:
https://github.com/apache/polaris/blob/07edd301f6cbb87ff2cf875bc5ea0b99875acded/polaris-core/src/main/java/org/apache/polaris/core/entity/table/IcebergTableLikeEntity.java#L119
But afaict `write.data.path` and `write.metadata.path` are not stored in the entity 😞
I see a few options:
1. We could store `write.data.path` and `write.metadata.path` in the entity, but that will be a problem for catalogs created before the change.
2. We could go back to `baseCatalog.loadTable(tableIdentifier)` but that would be very slow.
3. We could just add a FIXME here + create a GitHub issue and wait for a better solution with M2.
Wdyt?
I would go with 3) since we know that this code path will be improved, but I'm fine with 2) as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2549747609
##########
CHANGELOG.md:
##########
@@ -36,6 +36,14 @@ request adding CHANGELOG notes for breaking (!) changes and possibly other secti
[Iceberg Metrics Reporting]: https://iceberg.apache.org/docs/latest/metrics-reporting/
+- **S3 remote request signing** has been added, allowing Polaris to work with S3-compatible object storage systems.
+ *Remote signing is currently experimental and not enabled by default*. In particular, RBAC checks are currently not
+ production-ready. One new table privilege was introduced: `TABLE_REMOTE_SIGN`. To enable remote signing:
+ 1. Set the system-wide property `REMOTE_SIGNING_ENABLED` or the catalog-level `polaris.request-signing.enabled`
+ property to `true`.
+ 2. Grant the `TABLE_REMOTE_SIGN` privilege to a catalog role. The role must also be granted the `TABLE_READ_DATA`
+ and `TABLE_WRITE_DATA` privileges.
Review Comment:
OK I updated the changelog!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2511708475
##########
runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java:
##########
@@ -844,6 +869,37 @@ ALLOW_FEDERATED_CATALOGS_CREDENTIAL_VENDING, getResolvedCatalogEntity())) {
return responseBuilder;
}
+ private AccessDelegationMode selectAccessDelegationMode(
+ Set<AccessDelegationMode> delegationModes) {
+
+ if (delegationModes.isEmpty()) {
+ return UNKNOWN;
+ }
+
+ if (delegationModes.size() == 1) {
+ return delegationModes.iterator().next();
+ }
+
+ if (delegationModes.contains(VENDED_CREDENTIALS) && delegationModes.contains(REMOTE_SIGNING)) {
+
+ boolean skipCredIndirection =
+ realmConfig.getConfig(FeatureConfiguration.SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION);
+
+ boolean credentialSubscopingAllowed =
+ baseCatalog instanceof IcebergCatalog
Review Comment:
That's a good question!
It's complex to check this at this level because you need to find the `AwsStorageConfigurationInfo` corresponding to the resolved path in order to know if STS is unavailable.
I think we shouldn't make this logic too complex, because it's not common for a client to request more than one access delegation mode anyways.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for S3 request signing [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #2280:
URL: https://github.com/apache/polaris/pull/2280#discussion_r2511752330
##########
runtime/service/src/main/java/org/apache/polaris/service/storage/s3/sign/S3RemoteSigningCatalogHandler.java:
##########
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.service.storage.s3.sign;
+
+import java.util.Collection;
+import java.util.EnumSet;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.iceberg.BaseTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.exceptions.ForbiddenException;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.auth.PolarisAuthorizableOperation;
+import org.apache.polaris.core.auth.PolarisAuthorizer;
+import org.apache.polaris.core.auth.PolarisPrincipal;
+import org.apache.polaris.core.config.FeatureConfiguration;
+import org.apache.polaris.core.config.RealmConfig;
+import org.apache.polaris.core.context.CallContext;
+import org.apache.polaris.core.entity.CatalogEntity;
+import org.apache.polaris.core.entity.PolarisEntity;
+import org.apache.polaris.core.entity.PolarisEntityConstants;
+import org.apache.polaris.core.persistence.PolarisResolvedPathWrapper;
+import org.apache.polaris.core.persistence.resolver.ResolutionManifestFactory;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisStorageActions;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.apache.polaris.service.catalog.common.CatalogHandler;
+import org.apache.polaris.service.catalog.common.CatalogUtils;
+import org.apache.polaris.service.catalog.io.FileIOUtil;
+import org.apache.polaris.service.context.catalog.CallContextCatalogFactory;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignRequest;
+import org.apache.polaris.service.s3.sign.model.PolarisS3SignResponse;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class S3RemoteSigningCatalogHandler extends CatalogHandler implements AutoCloseable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(S3RemoteSigningCatalogHandler.class);
+
+ private final CallContextCatalogFactory catalogFactory;
+ private final S3RequestSigner s3RequestSigner;
+
+ private CatalogEntity catalogEntity;
+ private Catalog baseCatalog;
+
+ public S3RemoteSigningCatalogHandler(
+ PolarisDiagnostics diagnostics,
+ CallContext callContext,
+ ResolutionManifestFactory resolutionManifestFactory,
+ CallContextCatalogFactory catalogFactory,
+ PolarisPrincipal polarisPrincipal,
+ String catalogName,
+ PolarisAuthorizer authorizer,
+ S3RequestSigner s3RequestSigner) {
+ super(
+ diagnostics,
+ callContext,
+ resolutionManifestFactory,
+ polarisPrincipal,
+ catalogName,
+ authorizer,
+ // external catalogs are not supported for S3 remote signing
+ null,
+ null);
+ this.catalogFactory = catalogFactory;
+ this.s3RequestSigner = s3RequestSigner;
+ }
+
+ @Override
+ protected void initializeCatalog() {
+ catalogEntity =
+ CatalogEntity.of(resolutionManifest.getResolvedReferenceCatalogEntity().getRawLeafEntity());
+ if (catalogEntity.isExternal()) {
+ throw new ForbiddenException("Cannot use S3 remote signing with federated catalogs.");
+ }
+ baseCatalog =
+ catalogFactory.createCallContextCatalog(callContext, polarisPrincipal, resolutionManifest);
+ }
+
+ public PolarisS3SignResponse signS3Request(
+ PolarisS3SignRequest s3SignRequest, TableIdentifier tableIdentifier) {
+
+ LOGGER.debug("Requesting s3 signing for {}: {}", tableIdentifier, s3SignRequest);
Review Comment:
Quarkus indeed supports access logs. We must be careful with logging here because this endpoint will be called very, very frequently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org