You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@polaris.apache.org by "lefebsy (via GitHub)" <gi...@apache.org> on 2024/10/21 20:13:16 UTC
[PR] add s3 compatible storage - first commit [polaris]
lefebsy opened a new pull request, #389:
URL: https://github.com/apache/polaris/pull/389
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1827006689
##########
spec/polaris-management-service.yml:
##########
@@ -901,6 +903,58 @@ components:
required:
- roleArn
+ S3StorageConfigInfo:
+ type: object
+ description: S3 compatible storage configuration info (MinIO, Dell ECS, Netapp StorageGRID, ...)
+ allOf:
+ - $ref: '#/components/schemas/StorageConfigInfo'
+ properties:
+ credsCatalogAndClientStrategy:
+ type: string
+ enum:
+ - VALUE
+ - ENV_VAR_NAME
+ default: ENV_VAR_NAME
+ example: "ACCESS_KEY"
+ description: When you send key VALUE directly via this command, they should apear in logs. By ENV_VAR_NAME without dollar, only a reference will appear in logs, but the value have to be available as environnement variable in the context where Polaris is running
+ credsVendingStrategy:
+ type: string
+ enum:
+ - TOKEN_WITH_ASSUME_ROLE
+ - KEYS_SAME_AS_CATALOG
+ - KEYS_DEDICATED_TO_CLIENT
Review Comment:
Ok.
It'd simply one key for catalog itself, then another unique key for any clients whoever they are. I Let client distinction to the _principal/role/privilege_ level. I think it is hard at the class _storage/credential_ level to stick a pair of keys to each different clients.
It is a basic way, when SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION is True and there is not a temporary token, to not divulge internal catalog key and serve a key that can be deactivated for security concerns without breaking catalog itself.
After discussing with MonkeyCanCode here
[Prod Deployment credentials](https://github.com/apache/polaris/discussions/405) the main advantage in this proposal is that you have not to rely on the main credentials provided at the global Polaris service level.
Today if you revoke the Polaris service credentials for AWS, all catalogs with AWS storages are instantly KO.
In this implementation each catalog is independent. It is the same idea about clients keys, to not breaking catalog when clients keys are revoked or rotated fo security reasons.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1881121885
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.s3compatible.S3CompatibleStorageConfigurationInfo.CredsCatalogAndClientStrategyEnum;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** Credential vendor that supports generating */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+ private StsClient stsClient;
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ public void createStsClient(S3CompatibleStorageConfigurationInfo s3storageConfig) {
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+ stsBuilder.region(
+ Region
+ .US_WEST_1); // default region to avoid bug, because most (all?) S3 compatible softwares
+ // do not care about regions
+ stsBuilder.endpointOverride(
+ URI.create(
+ s3storageConfig
+ .getS3Endpoint())); // S3Compatible - AWS STS endpoint is unique and different from
Review Comment:
Yes it is the same endpoint. No difference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1881121885
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.s3compatible.S3CompatibleStorageConfigurationInfo.CredsCatalogAndClientStrategyEnum;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** Credential vendor that supports generating */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+ private StsClient stsClient;
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ public void createStsClient(S3CompatibleStorageConfigurationInfo s3storageConfig) {
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+ stsBuilder.region(
+ Region
+ .US_WEST_1); // default region to avoid bug, because most (all?) S3 compatible softwares
+ // do not care about regions
+ stsBuilder.endpointOverride(
+ URI.create(
+ s3storageConfig
+ .getS3Endpoint())); // S3Compatible - AWS STS endpoint is unique and different from
Review Comment:
Yes it is the same endpoint. No difference.
(at least for MinIo, Ceph, dell ECS, ...)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2744464274
> @lefebsy : I appreciate your work on this PR, but I'd like to submit an alternative implementation for consideration by the community. Will mention it here, when it's ready.
Hi @dimas-b , great to know that you will contribute to this as well. Given the PR is WIP for a quite long time, can we merge this first, and make changes later? We can always improve on it and make changes as needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2843008319
No worries @lefebsy! I'm sad to hear that you can't carry the PR forward but I'm thankful for your contributions so far. This is a feature I know a lot of users are excited about and it's unfortunate that it turned out to be so contentious.
@dimas-b, I'll keep an eye out for your PR, thanks for offering to pick this up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2814019442
Hey @lefebsy, do you think we'll be able to revive this? If you're not available we can also copy this into another PR. I think the changes are pretty much ready to go
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1994092260
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import static org.apache.polaris.core.config.FeatureConfiguration.STORAGE_CREDENTIAL_DURATION_SECONDS;
+import static org.apache.polaris.core.config.PolarisConfiguration.loadConfig;
+import static org.apache.polaris.core.storage.PolarisCredentialProperty.AWS_KEY_ID;
+import static org.apache.polaris.core.storage.PolarisCredentialProperty.AWS_SECRET_KEY;
+import static org.apache.polaris.core.storage.PolarisCredentialProperty.AWS_TOKEN;
+
+import jakarta.annotation.Nonnull;
+import jakarta.ws.rs.NotAuthorizedException;
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.profiles.ProfileFileSupplier;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** S3 compatible implementation of PolarisStorageIntegration */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ @Override
+ public EnumMap<PolarisCredentialProperty, String> getSubscopedCreds(
+ @Nonnull PolarisDiagnostics diagnostics,
+ @Nonnull S3CompatibleStorageConfigurationInfo storageConfig,
+ boolean allowListOperation,
+ @Nonnull Set<String> allowedReadLocations,
+ @Nonnull Set<String> allowedWriteLocations) {
+
+ String caI = System.getenv(storageConfig.getS3CredentialsCatalogAccessKeyId());
Review Comment:
I'm strongly opposed to storing credential env. variable name is the storage config (essentially in the catalog config).
The environment and catalog config are different management domains with different concerns.
Using a static env. variable name (e.g. `AWS_ACCESS_KEY`) is ok in the short term, IMHO.
Is it strictly necessary to support different secrets for different catalogs in this case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1813201796
##########
spec/polaris-management-service.yml:
##########
@@ -901,6 +903,58 @@ components:
required:
- roleArn
+ S3StorageConfigInfo:
+ type: object
+ description: S3 compatible storage configuration info (MinIO, Dell ECS, Netapp StorageGRID, ...)
+ allOf:
+ - $ref: '#/components/schemas/StorageConfigInfo'
+ properties:
+ credsCatalogAndClientStrategy:
+ type: string
+ enum:
+ - VALUE
+ - ENV_VAR_NAME
+ default: ENV_VAR_NAME
+ example: "ACCESS_KEY"
+ description: When you send key VALUE directly via this command, they should apear in logs. By ENV_VAR_NAME without dollar, only a reference will appear in logs, but the value have to be available as environnement variable in the context where Polaris is running
+ credsVendingStrategy:
+ type: string
+ enum:
+ - TOKEN_WITH_ASSUME_ROLE
+ - KEYS_SAME_AS_CATALOG
Review Comment:
Exactly, explained like that it's much clearer.
I was trying also to do a distinction with the third option where it is possible to serve another pair of keys, not the one used by the catalog. In this last case, the keys served to the 'client' can be revoke by S3 admin without touching the keys used internaly by this catalog.
It should not be used in production but for dev mode or some context where security is not an issue it is an easy solution
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "mmgaggle (via GitHub)" <gi...@apache.org>.
mmgaggle commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2585594212
I have tested this with Ceph and it is working properly. Very cool!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1953525177
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import jakarta.annotation.Nonnull;
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Stream;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.policybuilder.iam.IamConditionOperator;
+import software.amazon.awssdk.policybuilder.iam.IamEffect;
+import software.amazon.awssdk.policybuilder.iam.IamPolicy;
+import software.amazon.awssdk.policybuilder.iam.IamResource;
+import software.amazon.awssdk.policybuilder.iam.IamStatement;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** S3 compatible implementation of PolarisStorageIntegration */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+ private StsClient stsClient;
+ private String caI, caS; // catalogAccessKeyId & catalogSecretAccessKey
+ private String clI, clS; // clientAccessKeyId & clientSecretAccessKey
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public EnumMap<PolarisCredentialProperty, String> getSubscopedCreds(
+ @NotNull PolarisDiagnostics diagnostics,
+ @NotNull S3CompatibleStorageConfigurationInfo storageConfig,
+ boolean allowListOperation,
+ @NotNull Set<String> allowedReadLocations,
+ @NotNull Set<String> allowedWriteLocations) {
+
+ caI = System.getenv(storageConfig.getS3CredentialsCatalogAccessKeyId());
+ caS = System.getenv(storageConfig.getS3CredentialsCatalogSecretAccessKey());
+
+ EnumMap<PolarisCredentialProperty, String> propertiesMap =
+ new EnumMap<>(PolarisCredentialProperty.class);
+ propertiesMap.put(PolarisCredentialProperty.AWS_ENDPOINT, storageConfig.getS3Endpoint());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_PATH_STYLE_ACCESS,
+ storageConfig.getS3PathStyleAccess().toString());
+ if (storageConfig.getS3Region() != null) {
+ propertiesMap.put(PolarisCredentialProperty.CLIENT_REGION, storageConfig.getS3Region());
+ }
+
+ if (storageConfig.getSkipCredentialSubscopingIndirection() == true) {
+ LOGGER.debug("S3Compatible - skipCredentialSubscopingIndirection !");
+ clI = System.getenv(storageConfig.getS3CredentialsClientAccessKeyId());
+ clS = System.getenv(storageConfig.getS3CredentialsClientSecretAccessKey());
+ if (clI != null && clS != null) {
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, clI);
+ propertiesMap.put(PolarisCredentialProperty.AWS_SECRET_KEY, clS);
+ } else {
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, caI);
+ propertiesMap.put(PolarisCredentialProperty.AWS_SECRET_KEY, caS);
+ }
+ } else {
+ LOGGER.debug("S3Compatible - assumeRole !");
+ createStsClient(storageConfig);
+ AssumeRoleResponse response =
+ stsClient.assumeRole(
+ AssumeRoleRequest.builder()
+ .roleSessionName("PolarisCredentialsSTS")
+ .roleArn(
+ (storageConfig.getS3RoleArn() == null) ? "" : storageConfig.getS3RoleArn())
+ .policy(
+ policyString(allowListOperation, allowedReadLocations, allowedWriteLocations)
+ .toJson())
+ .build());
+
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, response.credentials().accessKeyId());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_SECRET_KEY, response.credentials().secretAccessKey());
+ propertiesMap.put(PolarisCredentialProperty.AWS_TOKEN, response.credentials().sessionToken());
+ LOGGER.debug(
+ "S3Compatible - assumeRole - Token Expiration at : {}",
+ response.credentials().expiration().toString());
+ }
+
+ return propertiesMap;
+ }
+
+ public void createStsClient(S3CompatibleStorageConfigurationInfo storageConfig) {
+ LOGGER.debug("S3Compatible - createStsClient()");
+ try {
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+ stsBuilder.endpointOverride(URI.create(storageConfig.getS3Endpoint()));
Review Comment:
Almost yes.
If catalog creator do not want to rely on default and global Polaris service keys and want dedicated keys for each catalog, adding credential alternative system in this in factory could be useful. No ?
- Security pattern like not putting all eggs in one basket
- Also onPrem S3 buckets can been managed with security pattern where administrators are generating different keys for each s3 user associated with some buckets + assumeRole policy (I have this case in my company, considered as secutity segreggation pattern). In this case Polaris with central and unique keys need to be over instanciated :-( Managed at catalog level it allow to keep one Polaris service and many catalogs
```
//not using provider built credentials from profile or standard AWS env var
stsBuilder.credentialsProvider(
StaticCredentialsProvider.create(AwsBasicCredentials.create(caI, caS)));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1953564742
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleStorageConfigurationInfo.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import java.util.List;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Polaris Storage Configuration information for an S3 Compatible solution, MinIO, Ceph, Dell ECS...
+ */
+public class S3CompatibleStorageConfigurationInfo extends PolarisStorageConfigurationInfo {
+
+ // 5 is the approximate max allowed locations for the size of AccessPolicy when LIST is required
+ // for allowed read and write locations for subscoping creds.
+ @JsonIgnore private static final int MAX_ALLOWED_LOCATIONS = 5;
+ private @NotNull String s3Endpoint;
+ private @Nullable String s3CredentialsCatalogAccessKeyId;
+ private @Nullable String s3CredentialsCatalogSecretAccessKey;
+ private @Nullable Boolean s3PathStyleAccess;
+ private @NotNull Boolean skipCredentialSubscopingIndirection;
+ private @Nullable String s3CredentialsClientAccessKeyId;
+ private @Nullable String s3CredentialsClientSecretAccessKey;
+ private @Nullable String s3Region;
+ private @Nullable String s3RoleArn;
+
+ // Constructor
+ @JsonCreator
+ public S3CompatibleStorageConfigurationInfo(
+ @JsonProperty(value = "storageType", required = true) @NotNull StorageType storageType,
+ @JsonProperty(value = "s3Endpoint", required = true) @NotNull String s3Endpoint,
+ @JsonProperty(value = "s3CredentialsCatalogAccessKeyId", required = true) @NotNull
+ String s3CredentialsCatalogAccessKeyId,
+ @JsonProperty(value = "s3CredentialsCatalogSecretAccessKey", required = true) @NotNull
+ String s3CredentialsCatalogSecretAccessKey,
+ @JsonProperty(value = "SkipCredentialSubscopingIndirection", required = false) @Nullable
+ Boolean skipCredentialSubscopingIndirection,
+ @JsonProperty(value = "s3CredentialsClientAccessKeyId", required = false) @Nullable
+ String s3CredentialsClientAccessKeyId,
+ @JsonProperty(value = "s3CredentialsClientSecretAccessKey", required = false) @Nullable
+ String s3CredentialsClientSecretAccessKey,
+ @JsonProperty(value = "s3PathStyleAccess", required = false) @Nullable
+ Boolean s3PathStyleAccess,
+ @JsonProperty(value = "s3Region", required = false) @Nullable String s3Region,
+ @JsonProperty(value = "s3RoleArn", required = false) @Nullable String s3RoleArn,
+ @JsonProperty(value = "allowedLocations", required = true) @Nullable
+ List<String> allowedLocations) {
+
+ // storing properties
+ super(storageType, allowedLocations);
+ validateMaxAllowedLocations(MAX_ALLOWED_LOCATIONS);
+ this.s3PathStyleAccess = s3PathStyleAccess;
Review Comment:
False is default in API spec...
Need to add a second default level ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1953583638
##########
spec/polaris-management-service.yml:
##########
@@ -905,6 +907,51 @@ components:
required:
- roleArn
+ S3CompatibleStorageConfigInfo:
+ type: object
+ description: S3 compatible storage configuration info (MinIO, Ceph, Dell ECS, Netapp StorageGRID, ...)
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1981867107
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleStorageConfigurationInfo.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import jakarta.annotation.Nonnull;
+import java.util.List;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * S3-Compatible Storage Configuration. This class holds the parameters needed to connect to
+ * S3-compatible storage services such as MinIO, Ceph, Dell ECS, etc.
+ */
+public class S3CompatibleStorageConfigurationInfo extends PolarisStorageConfigurationInfo {
+
+ // 5 is the approximate max allowed locations for the size of AccessPolicy when LIST is required
+ // for allowed read and write locations for sub-scoping credentials.
+ @JsonIgnore private static final int MAX_ALLOWED_LOCATIONS = 5;
+ private final @NotNull String s3Endpoint;
+ private final @Nullable String s3ProfileName;
+ private final @Nullable String s3CredentialsCatalogAccessKeyId;
+ private final @Nullable String s3CredentialsCatalogSecretAccessKey;
+ private final @NotNull Boolean s3PathStyleAccess;
+ private final @Nullable String s3Region;
+ private final @Nullable String s3RoleArn;
+
+ @JsonCreator
+ public S3CompatibleStorageConfigurationInfo(
+ @JsonProperty(value = "storageType", required = true) @NotNull StorageType storageType,
Review Comment:
I fully agree !!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "omarsmak (via GitHub)" <gi...@apache.org>.
omarsmak commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2719171198
> @omarsmak
>
> => Hello, the conflict should be resolved since rebased :-)
Thanks @lefebsy . Can you please address this last comment so we have this PR wrapped up? https://github.com/apache/polaris/pull/389/files#r1980741611
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2715352268
> Thanks a lot @lefebsy for working on it. I agreed that we don't have to put Helm change in, so that the github pipeline can pass. We can always add it back as a followup PR. It should be ready to merge as long as we fix this https://github.com/apache/polaris/pull/389/files#r1980741611.
Yes. I have restored helm from main branch. I have probably done a small test modification and forget it. I'm really confused.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "collado-mike (via GitHub)" <gi...@apache.org>.
collado-mike commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1950073614
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/PolarisCredentialProperty.java:
##########
@@ -23,6 +23,8 @@ public enum PolarisCredentialProperty {
AWS_KEY_ID(String.class, "s3.access-key-id", "the aws access key id"),
AWS_SECRET_KEY(String.class, "s3.secret-access-key", "the aws access key secret"),
AWS_TOKEN(String.class, "s3.session-token", "the aws scoped access token"),
+ AWS_ENDPOINT(String.class, "s3.endpoint", "the aws s3 endpoint"),
+ AWS_PATH_STYLE_ACCESS(Boolean.class, "s3.path-style-access", "the aws s3 path style access"),
Review Comment:
Typically, these are catalog-level properties, not credentials. I think it's best to avoid overloading the credential providers with the need to populate these configuration properties.
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleStorageConfigurationInfo.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import java.util.List;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Polaris Storage Configuration information for an S3 Compatible solution, MinIO, Ceph, Dell ECS...
+ */
+public class S3CompatibleStorageConfigurationInfo extends PolarisStorageConfigurationInfo {
+
+ // 5 is the approximate max allowed locations for the size of AccessPolicy when LIST is required
+ // for allowed read and write locations for subscoping creds.
+ @JsonIgnore private static final int MAX_ALLOWED_LOCATIONS = 5;
+ private @NotNull String s3Endpoint;
+ private @Nullable String s3CredentialsCatalogAccessKeyId;
+ private @Nullable String s3CredentialsCatalogSecretAccessKey;
+ private @Nullable Boolean s3PathStyleAccess;
+ private @NotNull Boolean skipCredentialSubscopingIndirection;
+ private @Nullable String s3CredentialsClientAccessKeyId;
+ private @Nullable String s3CredentialsClientSecretAccessKey;
Review Comment:
What is the distinction between `s3CredentialsCatalogAccessKeyId` and `s3CredentialsClientAccessKeyId`? Rather than having users set the names of environment variables, can't we use AWS profiles? That's a heck of a lot safer than allowing users to put the name of _any_ environment variable and read it directly from the server.
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import jakarta.annotation.Nonnull;
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Stream;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.policybuilder.iam.IamConditionOperator;
+import software.amazon.awssdk.policybuilder.iam.IamEffect;
+import software.amazon.awssdk.policybuilder.iam.IamPolicy;
+import software.amazon.awssdk.policybuilder.iam.IamResource;
+import software.amazon.awssdk.policybuilder.iam.IamStatement;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** S3 compatible implementation of PolarisStorageIntegration */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+ private StsClient stsClient;
+ private String caI, caS; // catalogAccessKeyId & catalogSecretAccessKey
+ private String clI, clS; // clientAccessKeyId & clientSecretAccessKey
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public EnumMap<PolarisCredentialProperty, String> getSubscopedCreds(
+ @NotNull PolarisDiagnostics diagnostics,
+ @NotNull S3CompatibleStorageConfigurationInfo storageConfig,
+ boolean allowListOperation,
+ @NotNull Set<String> allowedReadLocations,
+ @NotNull Set<String> allowedWriteLocations) {
+
+ caI = System.getenv(storageConfig.getS3CredentialsCatalogAccessKeyId());
+ caS = System.getenv(storageConfig.getS3CredentialsCatalogSecretAccessKey());
+
+ EnumMap<PolarisCredentialProperty, String> propertiesMap =
+ new EnumMap<>(PolarisCredentialProperty.class);
+ propertiesMap.put(PolarisCredentialProperty.AWS_ENDPOINT, storageConfig.getS3Endpoint());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_PATH_STYLE_ACCESS,
+ storageConfig.getS3PathStyleAccess().toString());
+ if (storageConfig.getS3Region() != null) {
+ propertiesMap.put(PolarisCredentialProperty.CLIENT_REGION, storageConfig.getS3Region());
+ }
+
+ if (storageConfig.getSkipCredentialSubscopingIndirection() == true) {
+ LOGGER.debug("S3Compatible - skipCredentialSubscopingIndirection !");
+ clI = System.getenv(storageConfig.getS3CredentialsClientAccessKeyId());
+ clS = System.getenv(storageConfig.getS3CredentialsClientSecretAccessKey());
Review Comment:
This is not safe. Now, if I have privilege to create a catalog, I can simply set this `skipCredentialSubcopingIndirection` flag and I immediately get access to the raw credentials used to assume any role for any other catalog in the service?
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.s3compatible.S3CompatibleStorageConfigurationInfo.CredsCatalogAndClientStrategyEnum;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** Credential vendor that supports generating */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+ private StsClient stsClient;
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ public void createStsClient(S3CompatibleStorageConfigurationInfo s3storageConfig) {
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+ stsBuilder.region(
+ Region
+ .US_WEST_1); // default region to avoid bug, because most (all?) S3 compatible softwares
+ // do not care about regions
+ stsBuilder.endpointOverride(
+ URI.create(
+ s3storageConfig
+ .getS3Endpoint())); // S3Compatible - AWS STS endpoint is unique and different from
+ // the S3 Endpoint
+ stsBuilder.credentialsProvider(
+ StaticCredentialsProvider.create(
+ AwsBasicCredentials.create(
+ s3storageConfig.getS3CredentialsCatalogAccessKeyId(),
+ s3storageConfig.getS3CredentialsCatalogSecretAccessKey())));
+ this.stsClient = stsBuilder.build();
+ LOGGER.debug("S3Compatible - stsClient successfully built");
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public EnumMap<PolarisCredentialProperty, String> getSubscopedCreds(
+ @NotNull PolarisDiagnostics diagnostics,
+ @NotNull S3CompatibleStorageConfigurationInfo storageConfig,
+ boolean allowListOperation,
+ @NotNull Set<String> allowedReadLocations,
+ @NotNull Set<String> allowedWriteLocations) {
+
+ LOGGER.debug("S3Compatible - getSubscopedCreds - applying credential strategy");
+ EnumMap<PolarisCredentialProperty, String> propertiesMap =
+ new EnumMap<>(PolarisCredentialProperty.class);
+ propertiesMap.put(PolarisCredentialProperty.AWS_ENDPOINT, storageConfig.getS3Endpoint());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_PATH_STYLE_ACCESS,
+ storageConfig.getS3PathStyleAccess().toString());
Review Comment:
I don't understand why you need them in this class at all. The catalog properties are used to initialize the FileIO in `BasePolarisCatalog`. The properties from this class are appended to those properties.
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import jakarta.annotation.Nonnull;
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Stream;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.policybuilder.iam.IamConditionOperator;
+import software.amazon.awssdk.policybuilder.iam.IamEffect;
+import software.amazon.awssdk.policybuilder.iam.IamPolicy;
+import software.amazon.awssdk.policybuilder.iam.IamResource;
+import software.amazon.awssdk.policybuilder.iam.IamStatement;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** S3 compatible implementation of PolarisStorageIntegration */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+ private StsClient stsClient;
+ private String caI, caS; // catalogAccessKeyId & catalogSecretAccessKey
+ private String clI, clS; // clientAccessKeyId & clientSecretAccessKey
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public EnumMap<PolarisCredentialProperty, String> getSubscopedCreds(
+ @NotNull PolarisDiagnostics diagnostics,
+ @NotNull S3CompatibleStorageConfigurationInfo storageConfig,
+ boolean allowListOperation,
+ @NotNull Set<String> allowedReadLocations,
+ @NotNull Set<String> allowedWriteLocations) {
+
+ caI = System.getenv(storageConfig.getS3CredentialsCatalogAccessKeyId());
+ caS = System.getenv(storageConfig.getS3CredentialsCatalogSecretAccessKey());
+
+ EnumMap<PolarisCredentialProperty, String> propertiesMap =
+ new EnumMap<>(PolarisCredentialProperty.class);
+ propertiesMap.put(PolarisCredentialProperty.AWS_ENDPOINT, storageConfig.getS3Endpoint());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_PATH_STYLE_ACCESS,
+ storageConfig.getS3PathStyleAccess().toString());
+ if (storageConfig.getS3Region() != null) {
+ propertiesMap.put(PolarisCredentialProperty.CLIENT_REGION, storageConfig.getS3Region());
+ }
+
+ if (storageConfig.getSkipCredentialSubscopingIndirection() == true) {
+ LOGGER.debug("S3Compatible - skipCredentialSubscopingIndirection !");
+ clI = System.getenv(storageConfig.getS3CredentialsClientAccessKeyId());
+ clS = System.getenv(storageConfig.getS3CredentialsClientSecretAccessKey());
+ if (clI != null && clS != null) {
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, clI);
+ propertiesMap.put(PolarisCredentialProperty.AWS_SECRET_KEY, clS);
+ } else {
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, caI);
+ propertiesMap.put(PolarisCredentialProperty.AWS_SECRET_KEY, caS);
+ }
+ } else {
+ LOGGER.debug("S3Compatible - assumeRole !");
+ createStsClient(storageConfig);
+ AssumeRoleResponse response =
+ stsClient.assumeRole(
+ AssumeRoleRequest.builder()
+ .roleSessionName("PolarisCredentialsSTS")
+ .roleArn(
+ (storageConfig.getS3RoleArn() == null) ? "" : storageConfig.getS3RoleArn())
+ .policy(
+ policyString(allowListOperation, allowedReadLocations, allowedWriteLocations)
+ .toJson())
+ .build());
+
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, response.credentials().accessKeyId());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_SECRET_KEY, response.credentials().secretAccessKey());
+ propertiesMap.put(PolarisCredentialProperty.AWS_TOKEN, response.credentials().sessionToken());
+ LOGGER.debug(
+ "S3Compatible - assumeRole - Token Expiration at : {}",
+ response.credentials().expiration().toString());
+ }
+
+ return propertiesMap;
+ }
+
+ public void createStsClient(S3CompatibleStorageConfigurationInfo storageConfig) {
+ LOGGER.debug("S3Compatible - createStsClient()");
+ try {
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+ stsBuilder.endpointOverride(URI.create(storageConfig.getS3Endpoint()));
Review Comment:
If I read this right, this is the only code that really needs to deviate from the default S3 credential vending process, right? We could realistically refactor the default STS client to be provided by a factory that takes in the storage config. Then we don't need all this duplication
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "richban (via GitHub)" <gi...@apache.org>.
richban commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2817282177
Without this PR, I can't configure the already S3 compatible storage to use Minio Server?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy closed pull request #389: [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible...
URL: https://github.com/apache/polaris/pull/389
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1953507226
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import jakarta.annotation.Nonnull;
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Stream;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.policybuilder.iam.IamConditionOperator;
+import software.amazon.awssdk.policybuilder.iam.IamEffect;
+import software.amazon.awssdk.policybuilder.iam.IamPolicy;
+import software.amazon.awssdk.policybuilder.iam.IamResource;
+import software.amazon.awssdk.policybuilder.iam.IamStatement;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** S3 compatible implementation of PolarisStorageIntegration */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+ private StsClient stsClient;
+ private String caI, caS; // catalogAccessKeyId & catalogSecretAccessKey
+ private String clI, clS; // clientAccessKeyId & clientSecretAccessKey
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public EnumMap<PolarisCredentialProperty, String> getSubscopedCreds(
+ @NotNull PolarisDiagnostics diagnostics,
+ @NotNull S3CompatibleStorageConfigurationInfo storageConfig,
+ boolean allowListOperation,
+ @NotNull Set<String> allowedReadLocations,
+ @NotNull Set<String> allowedWriteLocations) {
+
+ caI = System.getenv(storageConfig.getS3CredentialsCatalogAccessKeyId());
+ caS = System.getenv(storageConfig.getS3CredentialsCatalogSecretAccessKey());
+
+ EnumMap<PolarisCredentialProperty, String> propertiesMap =
+ new EnumMap<>(PolarisCredentialProperty.class);
+ propertiesMap.put(PolarisCredentialProperty.AWS_ENDPOINT, storageConfig.getS3Endpoint());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_PATH_STYLE_ACCESS,
+ storageConfig.getS3PathStyleAccess().toString());
+ if (storageConfig.getS3Region() != null) {
+ propertiesMap.put(PolarisCredentialProperty.CLIENT_REGION, storageConfig.getS3Region());
+ }
+
+ if (storageConfig.getSkipCredentialSubscopingIndirection() == true) {
+ LOGGER.debug("S3Compatible - skipCredentialSubscopingIndirection !");
+ clI = System.getenv(storageConfig.getS3CredentialsClientAccessKeyId());
+ clS = System.getenv(storageConfig.getS3CredentialsClientSecretAccessKey());
Review Comment:
Not it is not related
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1953525177
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import jakarta.annotation.Nonnull;
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Stream;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.policybuilder.iam.IamConditionOperator;
+import software.amazon.awssdk.policybuilder.iam.IamEffect;
+import software.amazon.awssdk.policybuilder.iam.IamPolicy;
+import software.amazon.awssdk.policybuilder.iam.IamResource;
+import software.amazon.awssdk.policybuilder.iam.IamStatement;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** S3 compatible implementation of PolarisStorageIntegration */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+ private StsClient stsClient;
+ private String caI, caS; // catalogAccessKeyId & catalogSecretAccessKey
+ private String clI, clS; // clientAccessKeyId & clientSecretAccessKey
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public EnumMap<PolarisCredentialProperty, String> getSubscopedCreds(
+ @NotNull PolarisDiagnostics diagnostics,
+ @NotNull S3CompatibleStorageConfigurationInfo storageConfig,
+ boolean allowListOperation,
+ @NotNull Set<String> allowedReadLocations,
+ @NotNull Set<String> allowedWriteLocations) {
+
+ caI = System.getenv(storageConfig.getS3CredentialsCatalogAccessKeyId());
+ caS = System.getenv(storageConfig.getS3CredentialsCatalogSecretAccessKey());
+
+ EnumMap<PolarisCredentialProperty, String> propertiesMap =
+ new EnumMap<>(PolarisCredentialProperty.class);
+ propertiesMap.put(PolarisCredentialProperty.AWS_ENDPOINT, storageConfig.getS3Endpoint());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_PATH_STYLE_ACCESS,
+ storageConfig.getS3PathStyleAccess().toString());
+ if (storageConfig.getS3Region() != null) {
+ propertiesMap.put(PolarisCredentialProperty.CLIENT_REGION, storageConfig.getS3Region());
+ }
+
+ if (storageConfig.getSkipCredentialSubscopingIndirection() == true) {
+ LOGGER.debug("S3Compatible - skipCredentialSubscopingIndirection !");
+ clI = System.getenv(storageConfig.getS3CredentialsClientAccessKeyId());
+ clS = System.getenv(storageConfig.getS3CredentialsClientSecretAccessKey());
+ if (clI != null && clS != null) {
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, clI);
+ propertiesMap.put(PolarisCredentialProperty.AWS_SECRET_KEY, clS);
+ } else {
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, caI);
+ propertiesMap.put(PolarisCredentialProperty.AWS_SECRET_KEY, caS);
+ }
+ } else {
+ LOGGER.debug("S3Compatible - assumeRole !");
+ createStsClient(storageConfig);
+ AssumeRoleResponse response =
+ stsClient.assumeRole(
+ AssumeRoleRequest.builder()
+ .roleSessionName("PolarisCredentialsSTS")
+ .roleArn(
+ (storageConfig.getS3RoleArn() == null) ? "" : storageConfig.getS3RoleArn())
+ .policy(
+ policyString(allowListOperation, allowedReadLocations, allowedWriteLocations)
+ .toJson())
+ .build());
+
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, response.credentials().accessKeyId());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_SECRET_KEY, response.credentials().secretAccessKey());
+ propertiesMap.put(PolarisCredentialProperty.AWS_TOKEN, response.credentials().sessionToken());
+ LOGGER.debug(
+ "S3Compatible - assumeRole - Token Expiration at : {}",
+ response.credentials().expiration().toString());
+ }
+
+ return propertiesMap;
+ }
+
+ public void createStsClient(S3CompatibleStorageConfigurationInfo storageConfig) {
+ LOGGER.debug("S3Compatible - createStsClient()");
+ try {
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+ stsBuilder.endpointOverride(URI.create(storageConfig.getS3Endpoint()));
Review Comment:
Almost yes.
If catalog creator do not want to rely on default and global Polaris service keys and want dedicated keys for each catalog, adding this in factory could be useful. No ?
- Security pattern like not putting all eggs in one basket
- Also onPrem S3 buckets can been managed with security pattern where administrators are generating different keys for each s3 user associated with some buckets (I have this case in my company, considered as secutity segreggation pattern). In this case Polaris with central and unique keys need to be over instanciated :-( Managed at catalog level it allow to keep one Polaris service and many catalogs
```
//not using provider built credentials from profile or standard AWS env var
stsBuilder.credentialsProvider(
StaticCredentialsProvider.create(AwsBasicCredentials.create(caI, caS)));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "snazy (via GitHub)" <gi...@apache.org>.
snazy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1995774089
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/PolarisStorageConfigurationInfo.java:
##########
@@ -62,6 +63,7 @@
@JsonTypeInfo(use = JsonTypeInfo.Id.NAME)
@JsonSubTypes({
@JsonSubTypes.Type(value = AwsStorageConfigurationInfo.class),
+ @JsonSubTypes.Type(value = S3CompatibleStorageConfigurationInfo.class),
Review Comment:
AWS config is an S3 configuration.
I'd not be opposed to rename `AwsStorageConfigurationInfo` to `S3StorageConfigurationInfo` and make that configuration work.
Having two different configurations for the same thing doesn't really make sense to me.
@lefebsy, mind doing this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2711429844
> @lefebsy are we merging this PR soon or there is still some outstanding work?
Hello @omarsmak
Still need 1 review to check the latest modifications requested (regression tests) but except if the merge have to wait for #1068 I think the biggest part of the task is done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1989423703
##########
polaris-core/src/main/java/org/apache/polaris/core/entity/CatalogEntity.java:
##########
@@ -250,6 +268,21 @@ public Builder setStorageConfigurationInfo(
awsConfig.validateArn(awsConfigModel.getRoleArn());
config = awsConfig;
break;
+
+ case S3_COMPATIBLE:
+ S3CompatibleStorageConfigInfo s3ConfigModel =
+ (S3CompatibleStorageConfigInfo) storageConfigModel;
+ config =
+ new S3CompatibleStorageConfigurationInfo(
Review Comment:
This implementation explicitly pass the s3 endpoint to the stsClient.
Apart from AWS which exposes AssumeRole, STS and globally IAM on a separate endpoint from the S3 APIs, other S3 providers seem to use the same endpoint.
In a mixed AWS + other approach, it would be necessary to explicitly specify the STS endpoint and the S3 endpoint.
In the non-AWS approach, it is not mandatory, it is the same endpoint (Minio, Ceph, backblaze, Dell ECS, ...)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1980749790
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleStorageConfigurationInfo.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import jakarta.annotation.Nonnull;
+import java.util.List;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * S3-Compatible Storage Configuration. This class holds the parameters needed to connect to
+ * S3-compatible storage services such as MinIO, Ceph, Dell ECS, etc.
+ */
+public class S3CompatibleStorageConfigurationInfo extends PolarisStorageConfigurationInfo {
+
+ // 5 is the approximate max allowed locations for the size of AccessPolicy when LIST is required
+ // for allowed read and write locations for sub-scoping credentials.
+ @JsonIgnore private static final int MAX_ALLOWED_LOCATIONS = 5;
Review Comment:
I'd suggest remove it per discussion here, https://github.com/apache/polaris/pull/1068#discussion_r1972467647. So that we don't have to introduce extra config later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2708233405
> Thanks a lot for working on it. Getting close. Left some comments. Other than the server side env variable settings, can we also have unit tests for two new classes added?
Is it ok for you if I do UT only for config class and move to dockerized tests for credentials class ? I am really not good in the mock area.
I can easily add minio to regtests docker-compose and enrich regtests/t_spark_sql
My first try just learned me not to name script with "S3" inside otherwise it's trapped by AWS stop pattern and never run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1813159130
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3/S3CredentialsStorageIntegration.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3;
+
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** Credential vendor that supports generating */
+public class S3CredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3StorageConfigurationInfo> {
+
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CredentialsStorageIntegration.class);
+
+ private StsClient stsClient;
+
+ // Constructor
+ public S3CredentialsStorageIntegration() {
+ super(S3CredentialsStorageIntegration.class.getName());
+ }
+
+ public void createStsClient(S3StorageConfigurationInfo s3storageConfig) {
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+
+ LOGGER.info(
+ "S3Compatible - AWS STS endpoint is unique and different from the S3 Endpoint. AWS SDK need to be overided with dedicated Endpoint from S3Compatible, otherwise the AWS STS url is targeted");
+
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+
+ stsBuilder.region(
+ Region
+ .US_WEST_1); // default region to avoid bug, because most (all?) S3 compatible softwares
Review Comment:
Correct. It could logically be used with any S3, AWS included.
The name is maybe a bad compromise, I agree to change for a better one.
At the beginning I have named it s3compat.... but at the end simply s3.
This choice is the result of this two constatations :
- In core/storage folder the classes are named aws, gcp and azure
- But in Enum Storage Type AWS is directly named S3....
So I found in enum this S3_COMPATIBILITY compromise. MinIO use it...
For the class I shortened it to S3 because it could works with AWS too.
I can come back to S3compat everywhere...
If Enum Storage was different and aws implementation named also AWS like GCP & Azure.... but it is not. Modifying the existing values of the Enum can be a breaking change, I do not want to introduce this change in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1813183830
##########
spec/polaris-management-service.yml:
##########
@@ -901,6 +903,58 @@ components:
required:
- roleArn
+ S3StorageConfigInfo:
Review Comment:
This implementation is focused OnPrem S3 because there is already the AWS class... however I can try to let it also works transparently with AWS :
- I think that roleArn is not mandatory for AWS S3, so let it to the existing implementation
- Using access and secret key should work with AWS S3 too
- I have overrided the AWS STS endpoint with S3 endpoint. I could add a modification, maybe with a STS endpoint property... something like "if property is empty" -> "STSclient call AWS default STS endpoint" else -> "STS client call the endpoint setted" or a boolean with a clear and explicit description
- I have removed the cross region tweak of the AWS FileIOClientFactory, it can be kept to assure a full compatibility
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1828956822
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleStorageConfigurationInfo.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import java.util.List;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+/** Polaris Storage Configuration information for an S3 Compatible solution, MinIO, Dell ECS... */
+public class S3CompatibleStorageConfigurationInfo extends PolarisStorageConfigurationInfo {
+
+ // 5 is the approximate max allowed locations for the size of AccessPolicy when LIST is required
+ // for allowed read and write locations for subscoping creds.
+ @JsonIgnore private static final int MAX_ALLOWED_LOCATIONS = 5;
+ private @NotNull CredsVendingStrategyEnum credsVendingStrategy;
+ private @NotNull CredsCatalogAndClientStrategyEnum credsCatalogAndClientStrategy;
+ private @NotNull String s3endpoint;
+ private @NotNull Boolean s3pathStyleAccess;
+ private @NotNull String s3CredentialsCatalogAccessKeyId;
+ private @NotNull String s3CredentialsCatalogSecretAccessKey;
+ private @Nullable String s3CredentialsClientAccessKeyId;
+ private @Nullable String s3CredentialsClientSecretAccessKey;
+
+ // Define how and what the catalog client will receive as credentials
+ public static enum CredsVendingStrategyEnum {
+ KEYS_SAME_AS_CATALOG,
+ KEYS_DEDICATED_TO_CLIENT,
+ TOKEN_WITH_ASSUME_ROLE;
+ };
+
+ // Define how the access and secret keys will be receive during the catalo creation, if
Review Comment:
mega nit: catalo -> catalog
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1881134556
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.s3compatible.S3CompatibleStorageConfigurationInfo.CredsCatalogAndClientStrategyEnum;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** Credential vendor that supports generating */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+ private StsClient stsClient;
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ public void createStsClient(S3CompatibleStorageConfigurationInfo s3storageConfig) {
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+ stsBuilder.region(
+ Region
+ .US_WEST_1); // default region to avoid bug, because most (all?) S3 compatible softwares
+ // do not care about regions
+ stsBuilder.endpointOverride(
+ URI.create(
+ s3storageConfig
+ .getS3Endpoint())); // S3Compatible - AWS STS endpoint is unique and different from
+ // the S3 Endpoint
+ stsBuilder.credentialsProvider(
+ StaticCredentialsProvider.create(
+ AwsBasicCredentials.create(
+ s3storageConfig.getS3CredentialsCatalogAccessKeyId(),
+ s3storageConfig.getS3CredentialsCatalogSecretAccessKey())));
+ this.stsClient = stsBuilder.build();
+ LOGGER.debug("S3Compatible - stsClient successfully built");
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public EnumMap<PolarisCredentialProperty, String> getSubscopedCreds(
+ @NotNull PolarisDiagnostics diagnostics,
+ @NotNull S3CompatibleStorageConfigurationInfo storageConfig,
+ boolean allowListOperation,
+ @NotNull Set<String> allowedReadLocations,
+ @NotNull Set<String> allowedWriteLocations) {
+
+ LOGGER.debug("S3Compatible - getSubscopedCreds - applying credential strategy");
+ EnumMap<PolarisCredentialProperty, String> propertiesMap =
+ new EnumMap<>(PolarisCredentialProperty.class);
+ propertiesMap.put(PolarisCredentialProperty.AWS_ENDPOINT, storageConfig.getS3Endpoint());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_PATH_STYLE_ACCESS,
+ storageConfig.getS3PathStyleAccess().toString());
Review Comment:
failed to use catalog porperties, they are not forwarded at this class
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2814040864
I'd prefer not to copy this 1:1 as I have some concerns with some areas of this PR (as commented).
If noone is actively working on this, I'd be willing to open another PR for this feature... maybe next week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1813201796
##########
spec/polaris-management-service.yml:
##########
@@ -901,6 +903,58 @@ components:
required:
- roleArn
+ S3StorageConfigInfo:
+ type: object
+ description: S3 compatible storage configuration info (MinIO, Dell ECS, Netapp StorageGRID, ...)
+ allOf:
+ - $ref: '#/components/schemas/StorageConfigInfo'
+ properties:
+ credsCatalogAndClientStrategy:
+ type: string
+ enum:
+ - VALUE
+ - ENV_VAR_NAME
+ default: ENV_VAR_NAME
+ example: "ACCESS_KEY"
+ description: When you send key VALUE directly via this command, they should apear in logs. By ENV_VAR_NAME without dollar, only a reference will appear in logs, but the value have to be available as environnement variable in the context where Polaris is running
+ credsVendingStrategy:
+ type: string
+ enum:
+ - TOKEN_WITH_ASSUME_ROLE
+ - KEYS_SAME_AS_CATALOG
Review Comment:
Exactly, explained like that it's much clearer.
I was trying also to do a distinction with the third option where it is possible to serve another pair of keys, not the one used by the catalog.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1813183830
##########
spec/polaris-management-service.yml:
##########
@@ -901,6 +903,58 @@ components:
required:
- roleArn
+ S3StorageConfigInfo:
Review Comment:
- I think that roleArn is not mandatory for AWS S3
- Using access and secret key should work with AWS S3 too.
- I have overrided the AWS STS endpoint with S3 endpoint. I could add a modification, maybe with a STS endpoint property... something like "if property is empty" -> "STSclient call AWS default STS endpoint" else -> "STS client call the endpoint setted" or a boolean with a clear and explicit description
- I have removed the cross region tweak of the AWS FileIOClientFactory, it can be kept to assure a full compatibility
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1813159130
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3/S3CredentialsStorageIntegration.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3;
+
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** Credential vendor that supports generating */
+public class S3CredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3StorageConfigurationInfo> {
+
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CredentialsStorageIntegration.class);
+
+ private StsClient stsClient;
+
+ // Constructor
+ public S3CredentialsStorageIntegration() {
+ super(S3CredentialsStorageIntegration.class.getName());
+ }
+
+ public void createStsClient(S3StorageConfigurationInfo s3storageConfig) {
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+
+ LOGGER.info(
+ "S3Compatible - AWS STS endpoint is unique and different from the S3 Endpoint. AWS SDK need to be overided with dedicated Endpoint from S3Compatible, otherwise the AWS STS url is targeted");
+
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+
+ stsBuilder.region(
+ Region
+ .US_WEST_1); // default region to avoid bug, because most (all?) S3 compatible softwares
Review Comment:
Correct. It could logically be used with any S3, AWS included (but I removed the cross region tweak implemented in the AWS FileIOClientFactory and overide the STS endpoint).
The name is maybe a bad compromise, I agree to change for a better one.
At the beginning I have named it s3compat.... but at the end simply s3.
This choice is the result of this two constatations :
- In core/storage folder the classes are named aws, gcp and azure
- But in Enum Storage Type AWS is directly named S3....
So I found in enum this S3_COMPATIBILITY compromise. MinIO use this term...
For the class I shortened it to S3 because S3Compatible was verbose.
I can come back to S3compatible everywhere...
If Enum Storage was different and aws implementation named also AWS like GCP & Azure.... but it is not. Modifying the existing values of the Enum can be a breaking change, I do not want to introduce this change in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1813201796
##########
spec/polaris-management-service.yml:
##########
@@ -901,6 +903,58 @@ components:
required:
- roleArn
+ S3StorageConfigInfo:
+ type: object
+ description: S3 compatible storage configuration info (MinIO, Dell ECS, Netapp StorageGRID, ...)
+ allOf:
+ - $ref: '#/components/schemas/StorageConfigInfo'
+ properties:
+ credsCatalogAndClientStrategy:
+ type: string
+ enum:
+ - VALUE
+ - ENV_VAR_NAME
+ default: ENV_VAR_NAME
+ example: "ACCESS_KEY"
+ description: When you send key VALUE directly via this command, they should apear in logs. By ENV_VAR_NAME without dollar, only a reference will appear in logs, but the value have to be available as environnement variable in the context where Polaris is running
+ credsVendingStrategy:
+ type: string
+ enum:
+ - TOKEN_WITH_ASSUME_ROLE
+ - KEYS_SAME_AS_CATALOG
Review Comment:
Exactly, explained like that it's much clearer.
I was trying also to do a distinction with the third option where it is possible to serve another pair of keys, not the one used by the catalog. In this last case, the keys served to the 'client' can be revoke by S3 admin without touching the keys used internaly by this catalog.
It should not be used in production, but for dev mode, or some context where security subscoping is not an issue, it is an easy solution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1813179181
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3/S3CredentialsStorageIntegration.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3;
+
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** Credential vendor that supports generating */
+public class S3CredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3StorageConfigurationInfo> {
+
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CredentialsStorageIntegration.class);
+
+ private StsClient stsClient;
+
+ // Constructor
+ public S3CredentialsStorageIntegration() {
+ super(S3CredentialsStorageIntegration.class.getName());
+ }
+
+ public void createStsClient(S3StorageConfigurationInfo s3storageConfig) {
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+
+ LOGGER.info(
Review Comment:
OK.
It was helping me to understand behavior at the beginning.
I agree to remove and clean whitespaces and comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1828958095
##########
regtests/minio/miniodata/Readme.md:
##########
@@ -0,0 +1 @@
+# Folder for MiniIo data container volume
Review Comment:
Is `MiniIo` a typo?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1827006689
##########
spec/polaris-management-service.yml:
##########
@@ -901,6 +903,58 @@ components:
required:
- roleArn
+ S3StorageConfigInfo:
+ type: object
+ description: S3 compatible storage configuration info (MinIO, Dell ECS, Netapp StorageGRID, ...)
+ allOf:
+ - $ref: '#/components/schemas/StorageConfigInfo'
+ properties:
+ credsCatalogAndClientStrategy:
+ type: string
+ enum:
+ - VALUE
+ - ENV_VAR_NAME
+ default: ENV_VAR_NAME
+ example: "ACCESS_KEY"
+ description: When you send key VALUE directly via this command, they should apear in logs. By ENV_VAR_NAME without dollar, only a reference will appear in logs, but the value have to be available as environnement variable in the context where Polaris is running
+ credsVendingStrategy:
+ type: string
+ enum:
+ - TOKEN_WITH_ASSUME_ROLE
+ - KEYS_SAME_AS_CATALOG
+ - KEYS_DEDICATED_TO_CLIENT
Review Comment:
Ok.
It's simply one key for catalog itself, then another unique key for any clients whoever they are. I Let client distinction to the _principal/role/privilege_ level. I think it is hard at the class _storage/credential_ level to stick a pair of keys to each different clients.
It is a basic way, when SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION is True and there is not a temporary token, to not divulge internal catalog key and serve a key that can be deactivated for security concerns without breaking catalog itself.
After discussing with MonkeyCanCode here
[Prod Deployment credentials](https://github.com/apache/polaris/discussions/405) the main advantage in this proposal is that you have not to rely on the main credentials provided at the global Polaris service level.
Today if you revoke the Polaris service credentials for AWS, all catalogs with AWS storages are instantly KO.
In this implementation each catalog is independent. It is the same idea about clients keys, to not breaking catalog when clients keys are revoked or rotated fo security reasons.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1953569309
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleStorageConfigurationInfo.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import java.util.List;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Polaris Storage Configuration information for an S3 Compatible solution, MinIO, Ceph, Dell ECS...
+ */
+public class S3CompatibleStorageConfigurationInfo extends PolarisStorageConfigurationInfo {
+
+ // 5 is the approximate max allowed locations for the size of AccessPolicy when LIST is required
+ // for allowed read and write locations for subscoping creds.
+ @JsonIgnore private static final int MAX_ALLOWED_LOCATIONS = 5;
+ private @NotNull String s3Endpoint;
+ private @Nullable String s3CredentialsCatalogAccessKeyId;
+ private @Nullable String s3CredentialsCatalogSecretAccessKey;
+ private @Nullable Boolean s3PathStyleAccess;
+ private @NotNull Boolean skipCredentialSubscopingIndirection;
+ private @Nullable String s3CredentialsClientAccessKeyId;
+ private @Nullable String s3CredentialsClientSecretAccessKey;
+ private @Nullable String s3Region;
+ private @Nullable String s3RoleArn;
+
+ // Constructor
+ @JsonCreator
+ public S3CompatibleStorageConfigurationInfo(
+ @JsonProperty(value = "storageType", required = true) @NotNull StorageType storageType,
+ @JsonProperty(value = "s3Endpoint", required = true) @NotNull String s3Endpoint,
+ @JsonProperty(value = "s3CredentialsCatalogAccessKeyId", required = true) @NotNull
+ String s3CredentialsCatalogAccessKeyId,
+ @JsonProperty(value = "s3CredentialsCatalogSecretAccessKey", required = true) @NotNull
+ String s3CredentialsCatalogSecretAccessKey,
+ @JsonProperty(value = "SkipCredentialSubscopingIndirection", required = false) @Nullable
+ Boolean skipCredentialSubscopingIndirection,
+ @JsonProperty(value = "s3CredentialsClientAccessKeyId", required = false) @Nullable
+ String s3CredentialsClientAccessKeyId,
+ @JsonProperty(value = "s3CredentialsClientSecretAccessKey", required = false) @Nullable
+ String s3CredentialsClientSecretAccessKey,
+ @JsonProperty(value = "s3PathStyleAccess", required = false) @Nullable
+ Boolean s3PathStyleAccess,
+ @JsonProperty(value = "s3Region", required = false) @Nullable String s3Region,
+ @JsonProperty(value = "s3RoleArn", required = false) @Nullable String s3RoleArn,
+ @JsonProperty(value = "allowedLocations", required = true) @Nullable
+ List<String> allowedLocations) {
+
+ // storing properties
+ super(storageType, allowedLocations);
+ validateMaxAllowedLocations(MAX_ALLOWED_LOCATIONS);
+ this.s3PathStyleAccess = s3PathStyleAccess;
+ this.s3Endpoint = s3Endpoint;
+ this.s3CredentialsCatalogAccessKeyId = s3CredentialsCatalogAccessKeyId;
+ this.s3CredentialsCatalogSecretAccessKey = s3CredentialsCatalogSecretAccessKey;
+ this.s3CredentialsClientAccessKeyId = s3CredentialsClientAccessKeyId;
+ this.s3CredentialsClientSecretAccessKey = s3CredentialsClientSecretAccessKey;
+ this.skipCredentialSubscopingIndirection = skipCredentialSubscopingIndirection;
+ this.s3Region = s3Region;
+ this.s3RoleArn = s3RoleArn;
+ }
+
+ public @NotNull String getS3Endpoint() {
+ return this.s3Endpoint;
+ }
+
+ public @NotNull Boolean getS3PathStyleAccess() {
+ return this.s3PathStyleAccess;
+ }
+
+ public @NotNull String getS3CredentialsCatalogAccessKeyId() {
+ return (this.s3CredentialsCatalogAccessKeyId == null)
+ ? ""
+ : this.s3CredentialsCatalogAccessKeyId;
+ }
+
+ public @NotNull String getS3CredentialsCatalogSecretAccessKey() {
+ return (this.s3CredentialsCatalogSecretAccessKey == null)
+ ? ""
+ : this.s3CredentialsCatalogSecretAccessKey;
Review Comment:
correct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1953580815
##########
regtests/minio/Readme.md:
##########
@@ -0,0 +1,42 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# MiniIO Secured
+## Minio and secured buckets with TLS self-signed / custom AC
+
+To be able to test Polaris with buckets in TLS under custom AC or self-signed certificate
+
+## MiniIO generate self-signed certificates designed for docker-compose setup
+
+- Download minio certificate generator : https://github.com/minio/certgen
+- ```./certgen -host "localhost,minio,*"```
Review Comment:
thx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2654495228
> Vending scoped credentials is one of the marquee features for Polaris, and vending a static credential might give users a false sense of security.
We definitely want to avoid that, because handing out raw credentials is pretty much the same as making them public—any Polaris user could easily access them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "sfc-gh-ygu (via GitHub)" <gi...@apache.org>.
sfc-gh-ygu commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1991897894
##########
spec/polaris-management-service.yml:
##########
@@ -905,6 +907,51 @@ components:
required:
- roleArn
+ S3CompatibleStorageConfigInfo:
+ type: object
+ description: S3 compatible storage configuration info (MinIO, Ceph, Dell ECS, Netapp StorageGRID, ...)
+ allOf:
+ - $ref: '#/components/schemas/StorageConfigInfo'
+ properties:
+ s3.endpoint:
+ type: string
+ description: the S3 endpoint, will also be used as STS endpoint
+ example: "http[s]://host:port"
+ s3.credentials.catalog.accessKeyId:
+ type: string
+ description: Default to AWS credentials, otherwise set the environement variable name for the 'ACCESS_KEY_ID' used by the catalog to communicate with S3
+ example: "CATALOG_1_ACCESS_KEY_ENV_VARIABLE_NAME or AWS_ACCESS_KEY_ID"
Review Comment:
Resurface this as a reminder. We still need a fix for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "omarsmak (via GitHub)" <gi...@apache.org>.
omarsmak commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2718467817
@lefebsy you have a conflict by the way
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1989628403
##########
helm/polaris/values.yaml:
##########
@@ -359,209 +359,76 @@ extraEnv:
# name: aws-secret
# key: secret_access_key
-# -- Extra volumes to add to the polaris pod. See https://kubernetes.io/docs/concepts/storage/volumes/.
-extraVolumes: []
- # - name: extra-volume
- # emptyDir: {}
-
-# -- Extra volume mounts to add to the polaris container. See https://kubernetes.io/docs/concepts/storage/volumes/.
-extraVolumeMounts: []
- # - name: extra-volume
- # mountPath: /usr/share/extra-volume
-
-# -- Add additional init containers to the polaris pod(s) See https://kubernetes.io/docs/concepts/workloads/pods/init-containers/.
-extraInitContainers: []
- # - name: your-image-name
- # image: your-image
- # imagePullPolicy: Always
- # command: ['sh', '-c', 'echo "hello world"']
-
-tracing:
- # -- Specifies whether tracing for the polaris server should be enabled.
- enabled: false
- # -- The collector endpoint URL to connect to (required).
- # The endpoint URL must have either the http:// or the https:// scheme.
- # The collector must talk the OpenTelemetry protocol (OTLP) and the port must be its gRPC port (by default 4317).
- # See https://quarkus.io/guides/opentelemetry for more information.
- endpoint: "http://otlp-collector:4317"
- # -- Which requests should be sampled. Valid values are: "all", "none", or a ratio between 0.0 and
- # "1.0d" (inclusive). E.g. "0.5d" means that 50% of the requests will be sampled.
- # Note: avoid entering numbers here, always prefer a string representation of the ratio.
- sample: "1.0d"
- # -- Resource attributes to identify the polaris service among other tracing sources.
- # See https://opentelemetry.io/docs/reference/specification/resource/semantic_conventions/#service.
- # If left empty, traces will be attached to a service named "Apache Polaris"; to change this,
- # provide a service.name attribute here.
- attributes:
- {}
- # service.name: my-polaris
-
-metrics:
- # -- Specifies whether metrics for the polaris server should be enabled.
- enabled: true
- # -- Additional tags (dimensional labels) to add to the metrics.
- tags:
- {}
- # service: polaris
- # environment: production
-
-serviceMonitor:
- # -- Specifies whether a ServiceMonitor for Prometheus operator should be created.
- enabled: true
- # -- The scrape interval; leave empty to let Prometheus decide. Must be a valid duration, e.g. 1d, 1h30m, 5m, 10s.
- interval: ""
- # -- Labels for the created ServiceMonitor so that Prometheus operator can properly pick it up.
- labels:
- {}
- # release: prometheus
- # -- Relabeling rules to apply to metrics. Ref https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config.
- metricRelabelings: []
- # - source_labels: [ __meta_kubernetes_namespace ]
- # separator: ;
- # regex: (.*)
- # target_label: namespace
- # replacement: $1
- # action: replace
-
-# -- Logging configuration.
-logging:
- # -- The log level of the root category, which is used as the default log level for all categories.
- level: INFO
- # -- The header name to use for the request ID.
- requestIdHeaderName: Polaris-Request-Id
- # -- Configuration for the console appender.
- console:
- # -- Whether to enable the console appender.
- enabled: true
- # -- The log level of the console appender.
- threshold: ALL
- # -- Whether to log in JSON format.
- json: false
- # -- The log format to use. Ignored if JSON format is enabled. See
- # https://quarkus.io/guides/logging#logging-format for details.
- format: "%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p [%c{3.}] [%X{requestId},%X{realmId}] [%X{traceId},%X{parentId},%X{spanId},%X{sampled}] (%t) %s%e%n"
- # -- Configuration for the file appender.
- file:
- # -- Whether to enable the file appender.
- enabled: false
- # -- The log level of the file appender.
- threshold: ALL
- # -- Whether to log in JSON format.
- json: false
- # -- The log format to use. Ignored if JSON format is enabled. See
- # https://quarkus.io/guides/logging#logging-format for details.
- format: "%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p [%c{3.}] [%X{requestId},%X{realmId}] [%X{traceId},%X{parentId},%X{spanId},%X{sampled}] (%t) %s%e%n"
- # -- The local directory where log files are stored. The persistent volume claim will be mounted
- # here.
- logsDir: /deployments/logs
- # -- The log file name.
- fileName: polaris.log
- # -- Log rotation configuration.
- rotation:
- # -- The maximum size of the log file before it is rotated. Should be expressed as a Kubernetes quantity.
- maxFileSize: 100Mi
- # -- The maximum number of backup files to keep.
- maxBackupIndex: 5
- # -- An optional suffix to append to the rotated log files. If present, the rotated log files
- # will be grouped in time buckets, and each bucket will contain at most maxBackupIndex files.
- # The suffix must be in a date-time format that is understood by DateTimeFormatter. If the
- # suffix ends with .gz or .zip, the rotated files will also be compressed using the
- # corresponding algorithm.
- fileSuffix: ~ # .yyyy-MM-dd.gz
- # -- The log storage configuration. A persistent volume claim will be created using these
- # settings.
- storage:
- # -- The storage class name of the persistent volume claim to create.
- className: standard
- # -- The size of the persistent volume claim to create.
- size: 512Gi
- # -- Labels to add to the persistent volume claim spec selector; a persistent volume with
- # matching labels must exist. Leave empty if using dynamic provisioning.
- selectorLabels: {}
- # app.kubernetes.io/name: polaris
- # app.kubernetes.io/instance: RELEASE-NAME
- # -- Configuration for specific log categories.
- categories:
- org.apache.polaris: INFO
- org.apache.iceberg.rest: INFO
- # Useful to debug configuration issues:
- # io.smallrye.config: DEBUG
- # -- Configuration for MDC (Mapped Diagnostic Context). Values specified here will be added to the
- # log context of all incoming requests and can be used in log patterns.
- mdc:
- # aid=polaris
- # sid=polaris-service
- {}
-
-# -- Realm context resolver configuration.
-realmContext:
- # -- The type of realm context resolver to use. Two built-in types are supported: default and test;
- # test is not recommended for production as it does not perform any realm validation.
- type: default
- # -- List of valid realms, for use with the default realm context resolver. The first realm in
- # the list is the default realm. Realms not in this list will be rejected.
- realms:
- - POLARIS
-
-# -- Polaris features configuration.
-features:
- # -- Features to enable or disable globally. If a feature is not present in the map, the default
- # built-in value is used.
- defaults: {}
- # ENFORCE_PRINCIPAL_CREDENTIAL_ROTATION_REQUIRED_CHECKING: false
- # SUPPORTED_CATALOG_STORAGE_TYPES:
- # - S3
- # - GCS
- # - AZURE
- # - FILE
- # -- Features to enable or disable per realm. This field is a map of maps. The realm name is the key, and the value is a map of
- # feature names to values. If a feature is not present in the map, the default value from the 'defaults' field is used.
- realmOverrides: {}
- # my-realm:
- # ENFORCE_PRINCIPAL_CREDENTIAL_ROTATION_REQUIRED_CHECKING: true
-
-# -- Polaris persistence configuration.
-persistence:
- # -- The type of persistence to use. Two built-in types are supported: in-memory and eclipse-link.
- type: eclipse-link # in-memory
- # -- The configuration for the eclipse-link persistence manager.
- eclipseLink:
- # -- The secret name to pull persistence.xml from.
- secret:
- # -- The name of the secret to pull persistence.xml from.
- # If not provided, the default built-in persistence.xml will be used. This is probably not what you want.
- name: ~
- # -- The key in the secret to pull persistence.xml from.
- key: persistence.xml
- # -- The persistence unit name to use.
- persistenceUnit: polaris
-
-# -- Polaris FileIO configuration.
-fileIo:
- # -- The type of file IO to use. Two built-in types are supported: default and wasb. The wasb one translates WASB paths to ABFS ones.
- type: default
-
-# -- Storage credentials for the server. If the following properties are unset, default
-# credentials will be used, in which case the pod must have the necessary permissions to access the storage.
-storage:
- # -- The secret to pull storage credentials from.
- secret:
- # -- The name of the secret to pull storage credentials from.
- name: ~
- # -- The key in the secret to pull the AWS access key ID from. Only required when using AWS.
- awsAccessKeyId: ~
- # -- The key in the secret to pull the AWS secret access key from. Only required when using AWS.
- awsSecretAccessKey: ~
- # -- The key in the secret to pull the GCP token from. Only required when using GCP.
- gcpToken: ~
- # -- The key in the secret to pull the GCP token expiration time from. Only required when using GCP. Must be a valid ISO 8601 duration. The default is PT1H (1 hour).
- gcpTokenLifespan: ~
-
-# -- Polaris authentication configuration.
-authentication:
- # -- The type of authentication to use. Two built-in types are supported: default and test;
- # test is not recommended for production.
- authenticator:
+# -- Configures whether to enable the bootstrap metastore manager job
+bootstrapMetastoreManager: false
Review Comment:
This change doesn't look right. You are removing extremely useful stuff from this value, and the added content seems to come from an old state of the Helm chart.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "sfc-gh-ygu (via GitHub)" <gi...@apache.org>.
sfc-gh-ygu commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2715303227
Thanks a lot @lefebsy for working on it. I agreed that we don't have to put Helm change in, so that the github pipeline can pass. We can always add it back as a followup PR. It should be ready to merge as long as we fix this https://github.com/apache/polaris/pull/389/files#r1980741611.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1989423703
##########
polaris-core/src/main/java/org/apache/polaris/core/entity/CatalogEntity.java:
##########
@@ -250,6 +268,21 @@ public Builder setStorageConfigurationInfo(
awsConfig.validateArn(awsConfigModel.getRoleArn());
config = awsConfig;
break;
+
+ case S3_COMPATIBLE:
+ S3CompatibleStorageConfigInfo s3ConfigModel =
+ (S3CompatibleStorageConfigInfo) storageConfigModel;
+ config =
+ new S3CompatibleStorageConfigurationInfo(
Review Comment:
This implementation explicitly pass the s3 endpoint to the stsClient.
Apart from AWS which exposes AssumeRole, STS and globally IAM on a separate endpoint from the S3 APIs, other S3 providers seem to use the same endpoint.
In a mixed AWS + other approach, it would be necessary to explicitly specify the STS endpoint and the S3 endpoint.
In the non-AWS approach, it is not mandatory, it is the same endpoint (Minio, Ceph, blackblaze, Dell ECS, ...)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2842895625
Hello,
I started this PR 6 months ago, and did not realize that the review process would be so time consuming, that each new review would ask to explain the changes required during the previous review or the previous commenter, or add tests on existing code etc..
Unfortunately, since some weeks I no longer have the ability to devote more personal time to these round trips. Mabe it would be preferable to close or remove this PR. I hope not everything is to throw away. If Dimas-b wants to make another better implementation : +1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1815722719
##########
spec/polaris-management-service.yml:
##########
@@ -901,6 +903,58 @@ components:
required:
- roleArn
+ S3StorageConfigInfo:
+ type: object
+ description: S3 compatible storage configuration info (MinIO, Dell ECS, Netapp StorageGRID, ...)
+ allOf:
+ - $ref: '#/components/schemas/StorageConfigInfo'
+ properties:
+ credsCatalogAndClientStrategy:
+ type: string
+ enum:
+ - VALUE
+ - ENV_VAR_NAME
+ default: ENV_VAR_NAME
+ example: "ACCESS_KEY"
+ description: When you send key VALUE directly via this command, they should apear in logs. By ENV_VAR_NAME without dollar, only a reference will appear in logs, but the value have to be available as environnement variable in the context where Polaris is running
+ credsVendingStrategy:
+ type: string
+ enum:
+ - TOKEN_WITH_ASSUME_ROLE
+ - KEYS_SAME_AS_CATALOG
+ - KEYS_DEDICATED_TO_CLIENT
+ default: TOKEN_WITH_ASSUME_ROLE
+ description: The catalog strategy to vend credentials to client. Options possible are same keys than catalog, keys dedicated to clients, or Tokens with STS methods 'assumeRole' for Dell ECS or NetApp StorageGrid solution, 'truc' for MinIo solution)
Review Comment:
cleaned
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "Gerrit-K (via GitHub)" <gi...@apache.org>.
Gerrit-K commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r2014324360
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import static org.apache.polaris.core.config.FeatureConfiguration.STORAGE_CREDENTIAL_DURATION_SECONDS;
+import static org.apache.polaris.core.config.PolarisConfiguration.loadConfig;
+import static org.apache.polaris.core.storage.PolarisCredentialProperty.AWS_KEY_ID;
+import static org.apache.polaris.core.storage.PolarisCredentialProperty.AWS_SECRET_KEY;
+import static org.apache.polaris.core.storage.PolarisCredentialProperty.AWS_TOKEN;
+
+import jakarta.annotation.Nonnull;
+import jakarta.ws.rs.NotAuthorizedException;
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.profiles.ProfileFileSupplier;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** S3 compatible implementation of PolarisStorageIntegration */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ @Override
+ public EnumMap<PolarisCredentialProperty, String> getSubscopedCreds(
+ @Nonnull PolarisDiagnostics diagnostics,
+ @Nonnull S3CompatibleStorageConfigurationInfo storageConfig,
+ boolean allowListOperation,
+ @Nonnull Set<String> allowedReadLocations,
+ @Nonnull Set<String> allowedWriteLocations) {
+
+ String caI = System.getenv(storageConfig.getS3CredentialsCatalogAccessKeyId());
+ String caS = System.getenv(storageConfig.getS3CredentialsCatalogSecretAccessKey());
+
+ EnumMap<PolarisCredentialProperty, String> propertiesMap =
+ new EnumMap<>(PolarisCredentialProperty.class);
+ propertiesMap.put(PolarisCredentialProperty.AWS_ENDPOINT, storageConfig.getS3Endpoint());
Review Comment:
tl;dr: I believe the endpoint is currently not set when the `SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION` feature is activated, but I think it should be (or it should be configurable at the catalog level again).
Long story: I tested this MR successfully in an earlier stage (back in early January, before the Quarkus migration) and now wanted to upgrade our PoC setup to the most recent state of this MR. Our setup involves a Ceph S3 backend that doesn't have the STS configured, which means: we need to be able to adjust the `s3.endpoint` property, but can't use subscoped credentials. This path doesn't seem to be possible anymore. Previously, we had `SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION` disabled (because we weren't aware of it) but `skipCredentialSubscopingIndirection` (on the catalog) enabled. The latter seems to have been removed in one of the updates, so STS is always enforced if not skipped globally. If `SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION` is set, the `getSubscopedCreds` function is not even called by the `FileIOUtil`, so the `S3FileIO` is initialized without region, endpoint etc., which causes it to fall back to environment variables and eventually failing.
For our case, we definitely need to set the endpoint for our Ceph catalog, but also don't want to globally disable STS and/or setting global `AWS_*` variables just to fix the `S3FileIO`. Am I missing something in configuring this or does this not work anymore?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1827006689
##########
spec/polaris-management-service.yml:
##########
@@ -901,6 +903,58 @@ components:
required:
- roleArn
+ S3StorageConfigInfo:
+ type: object
+ description: S3 compatible storage configuration info (MinIO, Dell ECS, Netapp StorageGRID, ...)
+ allOf:
+ - $ref: '#/components/schemas/StorageConfigInfo'
+ properties:
+ credsCatalogAndClientStrategy:
+ type: string
+ enum:
+ - VALUE
+ - ENV_VAR_NAME
+ default: ENV_VAR_NAME
+ example: "ACCESS_KEY"
+ description: When you send key VALUE directly via this command, they should apear in logs. By ENV_VAR_NAME without dollar, only a reference will appear in logs, but the value have to be available as environnement variable in the context where Polaris is running
+ credsVendingStrategy:
+ type: string
+ enum:
+ - TOKEN_WITH_ASSUME_ROLE
+ - KEYS_SAME_AS_CATALOG
+ - KEYS_DEDICATED_TO_CLIENT
Review Comment:
Ok.
It is simply one key for catalog itself, then another unique key for any clients whoever they are. I Let client distinction to the _principal/role/privilege_ level. I think it is hard at the class _storage/credential_ level to stick a pair of keys to each different clients.
It is a basic way, when SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION is True and there is not a temporary token, to not divulge internal catalog key and serve a key that can be deactivated for security concerns without breaking catalog itself.
After discussing with MonkeyCanCode here
[Prod Deployment credentials](https://github.com/apache/polaris/discussions/405) the main advantage in this proposal is that you have not to rely on the main credentials provided at the global Polaris service level.
Today if you revoke the Polaris service credentials for AWS, all catalogs with AWS storages are instantly KO.
In this implementation each catalog is independent. It is the same idea about clients keys, to not breaking catalog when clients keys are revoked or rotated fo security reasons.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1827006689
##########
spec/polaris-management-service.yml:
##########
@@ -901,6 +903,58 @@ components:
required:
- roleArn
+ S3StorageConfigInfo:
+ type: object
+ description: S3 compatible storage configuration info (MinIO, Dell ECS, Netapp StorageGRID, ...)
+ allOf:
+ - $ref: '#/components/schemas/StorageConfigInfo'
+ properties:
+ credsCatalogAndClientStrategy:
+ type: string
+ enum:
+ - VALUE
+ - ENV_VAR_NAME
+ default: ENV_VAR_NAME
+ example: "ACCESS_KEY"
+ description: When you send key VALUE directly via this command, they should apear in logs. By ENV_VAR_NAME without dollar, only a reference will appear in logs, but the value have to be available as environnement variable in the context where Polaris is running
+ credsVendingStrategy:
+ type: string
+ enum:
+ - TOKEN_WITH_ASSUME_ROLE
+ - KEYS_SAME_AS_CATALOG
+ - KEYS_DEDICATED_TO_CLIENT
Review Comment:
Ok.
It's simply one key for catalog itself, then another unique key for any clients whoever they are. I Let client distinction to the _principal/role/privilege_ level. I think it is hard at the class _storage/credential_ level to stick a pair of keys to each different clients.
It is a basic way, when SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION is True and there is not a temporary token, to not divulge internal catalog key and serve a key that can be deactivated or rotated for security concerns without breaking catalog itself.
After discussing with MonkeyCanCode here
[Prod Deployment credentials](https://github.com/apache/polaris/discussions/405) the advantage in this proposal is that you have not to rely on the main credentials provided at the global Polaris service level.
Today if you revoke the Polaris service credentials for AWS, all catalogs with AWS storages are instantly KO.
In this implementation each catalog is independent. It is the same idea about clients keys, to not breaking catalog when clients keys are revoked or rotated fo security reasons.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1824971501
##########
spec/polaris-management-service.yml:
##########
@@ -901,6 +903,58 @@ components:
required:
- roleArn
+ S3StorageConfigInfo:
Review Comment:
If we can unify these, I think that would be ideal. But I don't know enough about how S3 vs S3Compatible are similar/different to say how possible that is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1828953817
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/PolarisCredentialProperty.java:
##########
@@ -23,6 +23,8 @@ public enum PolarisCredentialProperty {
AWS_KEY_ID(String.class, "s3.access-key-id", "the aws access key id"),
AWS_SECRET_KEY(String.class, "s3.secret-access-key", "the aws access key secret"),
AWS_TOKEN(String.class, "s3.session-token", "the aws scoped access token"),
+ AWS_ENDPOINT(String.class, "s3.endpoint", "the aws s3 endpoint"),
+ AWS_PATH_STYLE_ACCESS(Boolean.class, "s3.path-style-access", "the aws s3 path style access"),
Review Comment:
`whether or not to use path-style access`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1829782885
##########
spec/polaris-management-service.yml:
##########
@@ -901,6 +903,58 @@ components:
required:
- roleArn
+ S3StorageConfigInfo:
Review Comment:
I think it will not be too hard to unify, in a next step. I miss AWS access to do tests, but for what I know or seen :
- STS endpoint is a AWS specific endpoint, in S3 compatible solutions, when it is available, it is merged with S3 endpoint.
- Region is available in S3 compatible solutions, but not used a lot, or mostly implemented to be compliant with aws sdk clients.
- about the cross region, it is more related to Polaris and Iceberg with the Iceberg overloaded S3fileIO, but I seen that https://github.com/apache/iceberg/pull/11259 have been merged 3 weeks ago --> so next Iceberg version will be ok soon.
MinIO claim that their product API is 100% compatible with AWS S3 API. Almost the same for many alternatives...
- The S3 Compatible implementation could easily propose an optional parameter "arnRole" like the mandatory one in the existing aws class, with less regexp patern to allow more flexibility for some implementation where "aws" inside the string is replaced by the product name (exemple "ecs" for DELL ECS)... It could help for a smooth transition
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1953563423
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleStorageConfigurationInfo.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import java.util.List;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Polaris Storage Configuration information for an S3 Compatible solution, MinIO, Ceph, Dell ECS...
+ */
+public class S3CompatibleStorageConfigurationInfo extends PolarisStorageConfigurationInfo {
+
+ // 5 is the approximate max allowed locations for the size of AccessPolicy when LIST is required
+ // for allowed read and write locations for subscoping creds.
+ @JsonIgnore private static final int MAX_ALLOWED_LOCATIONS = 5;
+ private @NotNull String s3Endpoint;
+ private @Nullable String s3CredentialsCatalogAccessKeyId;
+ private @Nullable String s3CredentialsCatalogSecretAccessKey;
+ private @Nullable Boolean s3PathStyleAccess;
+ private @NotNull Boolean skipCredentialSubscopingIndirection;
+ private @Nullable String s3CredentialsClientAccessKeyId;
+ private @Nullable String s3CredentialsClientSecretAccessKey;
+ private @Nullable String s3Region;
+ private @Nullable String s3RoleArn;
+
+ // Constructor
+ @JsonCreator
+ public S3CompatibleStorageConfigurationInfo(
+ @JsonProperty(value = "storageType", required = true) @NotNull StorageType storageType,
+ @JsonProperty(value = "s3Endpoint", required = true) @NotNull String s3Endpoint,
+ @JsonProperty(value = "s3CredentialsCatalogAccessKeyId", required = true) @NotNull
+ String s3CredentialsCatalogAccessKeyId,
+ @JsonProperty(value = "s3CredentialsCatalogSecretAccessKey", required = true) @NotNull
+ String s3CredentialsCatalogSecretAccessKey,
+ @JsonProperty(value = "SkipCredentialSubscopingIndirection", required = false) @Nullable
+ Boolean skipCredentialSubscopingIndirection,
+ @JsonProperty(value = "s3CredentialsClientAccessKeyId", required = false) @Nullable
+ String s3CredentialsClientAccessKeyId,
+ @JsonProperty(value = "s3CredentialsClientSecretAccessKey", required = false) @Nullable
+ String s3CredentialsClientSecretAccessKey,
+ @JsonProperty(value = "s3PathStyleAccess", required = false) @Nullable
+ Boolean s3PathStyleAccess,
+ @JsonProperty(value = "s3Region", required = false) @Nullable String s3Region,
+ @JsonProperty(value = "s3RoleArn", required = false) @Nullable String s3RoleArn,
+ @JsonProperty(value = "allowedLocations", required = true) @Nullable
+ List<String> allowedLocations) {
+
+ // storing properties
+ super(storageType, allowedLocations);
Review Comment:
Yes same as in GCP and Azure class, but not in AWS. OK for me I do not mind.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1953556482
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleStorageConfigurationInfo.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import java.util.List;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Polaris Storage Configuration information for an S3 Compatible solution, MinIO, Ceph, Dell ECS...
+ */
+public class S3CompatibleStorageConfigurationInfo extends PolarisStorageConfigurationInfo {
+
+ // 5 is the approximate max allowed locations for the size of AccessPolicy when LIST is required
+ // for allowed read and write locations for subscoping creds.
+ @JsonIgnore private static final int MAX_ALLOWED_LOCATIONS = 5;
+ private @NotNull String s3Endpoint;
+ private @Nullable String s3CredentialsCatalogAccessKeyId;
+ private @Nullable String s3CredentialsCatalogSecretAccessKey;
+ private @Nullable Boolean s3PathStyleAccess;
+ private @NotNull Boolean skipCredentialSubscopingIndirection;
+ private @Nullable String s3CredentialsClientAccessKeyId;
+ private @Nullable String s3CredentialsClientSecretAccessKey;
Review Comment:
Explained in the updated PR summary..
The idea was to use
- 'catalog' key only for polaris s3 access and sts generation
- 'client' keys served to spark or trino when STS are not available.
Replacing variable name by Profile name... I like it !
Yes it could probably works, harder to integrate the file in a kubernetes deployement. Secrets are not allowed in configMap, all the file content should be stored as secret and mounted as volume. Except if profile file is able to interprete variables for secret keys...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1953526960
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import jakarta.annotation.Nonnull;
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Stream;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.policybuilder.iam.IamConditionOperator;
+import software.amazon.awssdk.policybuilder.iam.IamEffect;
+import software.amazon.awssdk.policybuilder.iam.IamPolicy;
+import software.amazon.awssdk.policybuilder.iam.IamResource;
+import software.amazon.awssdk.policybuilder.iam.IamStatement;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** S3 compatible implementation of PolarisStorageIntegration */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+ private StsClient stsClient;
+ private String caI, caS; // catalogAccessKeyId & catalogSecretAccessKey
+ private String clI, clS; // clientAccessKeyId & clientSecretAccessKey
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public EnumMap<PolarisCredentialProperty, String> getSubscopedCreds(
+ @NotNull PolarisDiagnostics diagnostics,
+ @NotNull S3CompatibleStorageConfigurationInfo storageConfig,
+ boolean allowListOperation,
+ @NotNull Set<String> allowedReadLocations,
+ @NotNull Set<String> allowedWriteLocations) {
+
+ caI = System.getenv(storageConfig.getS3CredentialsCatalogAccessKeyId());
+ caS = System.getenv(storageConfig.getS3CredentialsCatalogSecretAccessKey());
+
+ EnumMap<PolarisCredentialProperty, String> propertiesMap =
+ new EnumMap<>(PolarisCredentialProperty.class);
+ propertiesMap.put(PolarisCredentialProperty.AWS_ENDPOINT, storageConfig.getS3Endpoint());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_PATH_STYLE_ACCESS,
+ storageConfig.getS3PathStyleAccess().toString());
+ if (storageConfig.getS3Region() != null) {
+ propertiesMap.put(PolarisCredentialProperty.CLIENT_REGION, storageConfig.getS3Region());
+ }
+
+ if (storageConfig.getSkipCredentialSubscopingIndirection() == true) {
+ LOGGER.debug("S3Compatible - skipCredentialSubscopingIndirection !");
+ clI = System.getenv(storageConfig.getS3CredentialsClientAccessKeyId());
+ clS = System.getenv(storageConfig.getS3CredentialsClientSecretAccessKey());
+ if (clI != null && clS != null) {
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, clI);
+ propertiesMap.put(PolarisCredentialProperty.AWS_SECRET_KEY, clS);
+ } else {
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, caI);
+ propertiesMap.put(PolarisCredentialProperty.AWS_SECRET_KEY, caS);
+ }
+ } else {
+ LOGGER.debug("S3Compatible - assumeRole !");
+ createStsClient(storageConfig);
+ AssumeRoleResponse response =
+ stsClient.assumeRole(
+ AssumeRoleRequest.builder()
+ .roleSessionName("PolarisCredentialsSTS")
+ .roleArn(
+ (storageConfig.getS3RoleArn() == null) ? "" : storageConfig.getS3RoleArn())
+ .policy(
+ policyString(allowListOperation, allowedReadLocations, allowedWriteLocations)
+ .toJson())
+ .build());
+
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, response.credentials().accessKeyId());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_SECRET_KEY, response.credentials().secretAccessKey());
+ propertiesMap.put(PolarisCredentialProperty.AWS_TOKEN, response.credentials().sessionToken());
+ LOGGER.debug(
+ "S3Compatible - assumeRole - Token Expiration at : {}",
+ response.credentials().expiration().toString());
+ }
+
+ return propertiesMap;
+ }
+
+ public void createStsClient(S3CompatibleStorageConfigurationInfo storageConfig) {
+ LOGGER.debug("S3Compatible - createStsClient()");
+ try {
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+ stsBuilder.endpointOverride(URI.create(storageConfig.getS3Endpoint()));
+ if (caI != null && caS != null) {
+ // else default provider build credentials from profile or standard AWS env var
+ stsBuilder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(caI, caS)));
+ }
+ this.stsClient = stsBuilder.build();
+ LOGGER.debug("S3Compatible - stsClient successfully built");
+
+ } catch (Exception e) {
+ System.err.println("S3Compatible - stsClient - build failure : " + e.getMessage());
Review Comment:
Yes...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1953501268
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import jakarta.annotation.Nonnull;
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Stream;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.policybuilder.iam.IamConditionOperator;
+import software.amazon.awssdk.policybuilder.iam.IamEffect;
+import software.amazon.awssdk.policybuilder.iam.IamPolicy;
+import software.amazon.awssdk.policybuilder.iam.IamResource;
+import software.amazon.awssdk.policybuilder.iam.IamStatement;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** S3 compatible implementation of PolarisStorageIntegration */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+ private StsClient stsClient;
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1813201796
##########
spec/polaris-management-service.yml:
##########
@@ -901,6 +903,58 @@ components:
required:
- roleArn
+ S3StorageConfigInfo:
+ type: object
+ description: S3 compatible storage configuration info (MinIO, Dell ECS, Netapp StorageGRID, ...)
+ allOf:
+ - $ref: '#/components/schemas/StorageConfigInfo'
+ properties:
+ credsCatalogAndClientStrategy:
+ type: string
+ enum:
+ - VALUE
+ - ENV_VAR_NAME
+ default: ENV_VAR_NAME
+ example: "ACCESS_KEY"
+ description: When you send key VALUE directly via this command, they should apear in logs. By ENV_VAR_NAME without dollar, only a reference will appear in logs, but the value have to be available as environnement variable in the context where Polaris is running
+ credsVendingStrategy:
+ type: string
+ enum:
+ - TOKEN_WITH_ASSUME_ROLE
+ - KEYS_SAME_AS_CATALOG
Review Comment:
Exactly, explained like that it's much clearer.
I was trying also to do a distinction with the third option where it is possible to use another pair of keys, not the one used by the catalog.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1813183830
##########
spec/polaris-management-service.yml:
##########
@@ -901,6 +903,58 @@ components:
required:
- roleArn
+ S3StorageConfigInfo:
Review Comment:
This implementation focuses on OnPrem S3 because there is already the AWS class.. However, I can try to let it work seamlessly with AWS :
- I think that roleArn is not mandatory for AWS S3, so let it to the existing implementation for this scenario
- Using access and secret key should work with AWS S3 too
- I have overrided the AWS STS endpoint with S3 endpoint. I could add a modification, maybe with a STS endpoint property... something like "if property is empty" -> "STSclient call AWS default STS endpoint" else -> "STS client call the endpoint setted" or a boolean with a clear and explicit description
- Region, (maybe little more reflexion is needed to avoid conflict)
- Add region property
- I have removed the cross region tweak of the AWS FileIOClientFactory, it can be kept to assure a full compatibility
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1813183830
##########
spec/polaris-management-service.yml:
##########
@@ -901,6 +903,58 @@ components:
required:
- roleArn
+ S3StorageConfigInfo:
Review Comment:
This implementation focuses on OnPrem S3 because there is already the AWS class.. However, I can try to let it work seamlessly with AWS :
- I think that roleArn is not mandatory for AWS S3, so let it to the existing implementation for this scenario
- Using access and secret key should work with AWS S3 too
- I have overrided the AWS STS endpoint with S3 endpoint. I could add a modification, maybe with a STS endpoint property... something like "if property is empty" -> "STSclient call AWS default STS endpoint" else -> "STS client call the endpoint setted" or a boolean with a clear and explicit description
- Add region property
- I have removed the cross region tweak of the AWS FileIOClientFactory, it can be kept to assure a full compatibility
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1815722127
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3/S3CredentialsStorageIntegration.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3;
+
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** Credential vendor that supports generating */
+public class S3CredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3StorageConfigurationInfo> {
+
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CredentialsStorageIntegration.class);
+
+ private StsClient stsClient;
+
+ // Constructor
+ public S3CredentialsStorageIntegration() {
+ super(S3CredentialsStorageIntegration.class.getName());
+ }
+
+ public void createStsClient(S3StorageConfigurationInfo s3storageConfig) {
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+
+ LOGGER.info(
+ "S3Compatible - AWS STS endpoint is unique and different from the S3 Endpoint. AWS SDK need to be overided with dedicated Endpoint from S3Compatible, otherwise the AWS STS url is targeted");
+
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+
+ stsBuilder.region(
+ Region
+ .US_WEST_1); // default region to avoid bug, because most (all?) S3 compatible softwares
Review Comment:
renamed in S3compatible everywhere
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1980741611
##########
spec/polaris-management-service.yml:
##########
@@ -905,6 +907,51 @@ components:
required:
- roleArn
+ S3CompatibleStorageConfigInfo:
+ type: object
+ description: S3 compatible storage configuration info (MinIO, Ceph, Dell ECS, Netapp StorageGRID, ...)
+ allOf:
+ - $ref: '#/components/schemas/StorageConfigInfo'
+ properties:
+ s3.endpoint:
+ type: string
+ description: the S3 endpoint, will also be used as STS endpoint
+ example: "http[s]://host:port"
+ s3.credentials.catalog.accessKeyId:
+ type: string
+ description: Default to AWS credentials, otherwise set the environement variable name for the 'ACCESS_KEY_ID' used by the catalog to communicate with S3
+ example: "CATALOG_1_ACCESS_KEY_ENV_VARIABLE_NAME or AWS_ACCESS_KEY_ID"
Review Comment:
> Otherwise the creator let the key parameters empty, "cross fingers", and let default s3 sdk use whatever is available in polaris service environement.
The server side can have a configurable map of `(s3 endpoint, (accessKeyEnvVar, secretAccessKeyEnvVar))`. In that case, clients don't have to provide them, at the same time, Polaris server can always find the correct env variables per the key(s3 endpoint). WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1980749790
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleStorageConfigurationInfo.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import jakarta.annotation.Nonnull;
+import java.util.List;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * S3-Compatible Storage Configuration. This class holds the parameters needed to connect to
+ * S3-compatible storage services such as MinIO, Ceph, Dell ECS, etc.
+ */
+public class S3CompatibleStorageConfigurationInfo extends PolarisStorageConfigurationInfo {
+
+ // 5 is the approximate max allowed locations for the size of AccessPolicy when LIST is required
+ // for allowed read and write locations for sub-scoping credentials.
+ @JsonIgnore private static final int MAX_ALLOWED_LOCATIONS = 5;
Review Comment:
I'd suggest remove it per discussion here, https://github.com/apache/polaris/pull/1068#discussion_r1972467647.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1980741611
##########
spec/polaris-management-service.yml:
##########
@@ -905,6 +907,51 @@ components:
required:
- roleArn
+ S3CompatibleStorageConfigInfo:
+ type: object
+ description: S3 compatible storage configuration info (MinIO, Ceph, Dell ECS, Netapp StorageGRID, ...)
+ allOf:
+ - $ref: '#/components/schemas/StorageConfigInfo'
+ properties:
+ s3.endpoint:
+ type: string
+ description: the S3 endpoint, will also be used as STS endpoint
+ example: "http[s]://host:port"
+ s3.credentials.catalog.accessKeyId:
+ type: string
+ description: Default to AWS credentials, otherwise set the environement variable name for the 'ACCESS_KEY_ID' used by the catalog to communicate with S3
+ example: "CATALOG_1_ACCESS_KEY_ENV_VARIABLE_NAME or AWS_ACCESS_KEY_ID"
Review Comment:
> Otherwise the creator let the key parameters empty, "cross fingers", and let default s3 sdk use whatever is available in polaris service environement.
The server side can have a configurable map of `(s3 endpoint, (accessKeyEnvVar, secretAccessKeyEnvVar))`. In that case, clients don't have to provide these parameters, at the same time, Polaris server can always find the correct env variables per the key(s3 endpoint). WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1989628403
##########
helm/polaris/values.yaml:
##########
@@ -359,209 +359,76 @@ extraEnv:
# name: aws-secret
# key: secret_access_key
-# -- Extra volumes to add to the polaris pod. See https://kubernetes.io/docs/concepts/storage/volumes/.
-extraVolumes: []
- # - name: extra-volume
- # emptyDir: {}
-
-# -- Extra volume mounts to add to the polaris container. See https://kubernetes.io/docs/concepts/storage/volumes/.
-extraVolumeMounts: []
- # - name: extra-volume
- # mountPath: /usr/share/extra-volume
-
-# -- Add additional init containers to the polaris pod(s) See https://kubernetes.io/docs/concepts/workloads/pods/init-containers/.
-extraInitContainers: []
- # - name: your-image-name
- # image: your-image
- # imagePullPolicy: Always
- # command: ['sh', '-c', 'echo "hello world"']
-
-tracing:
- # -- Specifies whether tracing for the polaris server should be enabled.
- enabled: false
- # -- The collector endpoint URL to connect to (required).
- # The endpoint URL must have either the http:// or the https:// scheme.
- # The collector must talk the OpenTelemetry protocol (OTLP) and the port must be its gRPC port (by default 4317).
- # See https://quarkus.io/guides/opentelemetry for more information.
- endpoint: "http://otlp-collector:4317"
- # -- Which requests should be sampled. Valid values are: "all", "none", or a ratio between 0.0 and
- # "1.0d" (inclusive). E.g. "0.5d" means that 50% of the requests will be sampled.
- # Note: avoid entering numbers here, always prefer a string representation of the ratio.
- sample: "1.0d"
- # -- Resource attributes to identify the polaris service among other tracing sources.
- # See https://opentelemetry.io/docs/reference/specification/resource/semantic_conventions/#service.
- # If left empty, traces will be attached to a service named "Apache Polaris"; to change this,
- # provide a service.name attribute here.
- attributes:
- {}
- # service.name: my-polaris
-
-metrics:
- # -- Specifies whether metrics for the polaris server should be enabled.
- enabled: true
- # -- Additional tags (dimensional labels) to add to the metrics.
- tags:
- {}
- # service: polaris
- # environment: production
-
-serviceMonitor:
- # -- Specifies whether a ServiceMonitor for Prometheus operator should be created.
- enabled: true
- # -- The scrape interval; leave empty to let Prometheus decide. Must be a valid duration, e.g. 1d, 1h30m, 5m, 10s.
- interval: ""
- # -- Labels for the created ServiceMonitor so that Prometheus operator can properly pick it up.
- labels:
- {}
- # release: prometheus
- # -- Relabeling rules to apply to metrics. Ref https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config.
- metricRelabelings: []
- # - source_labels: [ __meta_kubernetes_namespace ]
- # separator: ;
- # regex: (.*)
- # target_label: namespace
- # replacement: $1
- # action: replace
-
-# -- Logging configuration.
-logging:
- # -- The log level of the root category, which is used as the default log level for all categories.
- level: INFO
- # -- The header name to use for the request ID.
- requestIdHeaderName: Polaris-Request-Id
- # -- Configuration for the console appender.
- console:
- # -- Whether to enable the console appender.
- enabled: true
- # -- The log level of the console appender.
- threshold: ALL
- # -- Whether to log in JSON format.
- json: false
- # -- The log format to use. Ignored if JSON format is enabled. See
- # https://quarkus.io/guides/logging#logging-format for details.
- format: "%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p [%c{3.}] [%X{requestId},%X{realmId}] [%X{traceId},%X{parentId},%X{spanId},%X{sampled}] (%t) %s%e%n"
- # -- Configuration for the file appender.
- file:
- # -- Whether to enable the file appender.
- enabled: false
- # -- The log level of the file appender.
- threshold: ALL
- # -- Whether to log in JSON format.
- json: false
- # -- The log format to use. Ignored if JSON format is enabled. See
- # https://quarkus.io/guides/logging#logging-format for details.
- format: "%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p [%c{3.}] [%X{requestId},%X{realmId}] [%X{traceId},%X{parentId},%X{spanId},%X{sampled}] (%t) %s%e%n"
- # -- The local directory where log files are stored. The persistent volume claim will be mounted
- # here.
- logsDir: /deployments/logs
- # -- The log file name.
- fileName: polaris.log
- # -- Log rotation configuration.
- rotation:
- # -- The maximum size of the log file before it is rotated. Should be expressed as a Kubernetes quantity.
- maxFileSize: 100Mi
- # -- The maximum number of backup files to keep.
- maxBackupIndex: 5
- # -- An optional suffix to append to the rotated log files. If present, the rotated log files
- # will be grouped in time buckets, and each bucket will contain at most maxBackupIndex files.
- # The suffix must be in a date-time format that is understood by DateTimeFormatter. If the
- # suffix ends with .gz or .zip, the rotated files will also be compressed using the
- # corresponding algorithm.
- fileSuffix: ~ # .yyyy-MM-dd.gz
- # -- The log storage configuration. A persistent volume claim will be created using these
- # settings.
- storage:
- # -- The storage class name of the persistent volume claim to create.
- className: standard
- # -- The size of the persistent volume claim to create.
- size: 512Gi
- # -- Labels to add to the persistent volume claim spec selector; a persistent volume with
- # matching labels must exist. Leave empty if using dynamic provisioning.
- selectorLabels: {}
- # app.kubernetes.io/name: polaris
- # app.kubernetes.io/instance: RELEASE-NAME
- # -- Configuration for specific log categories.
- categories:
- org.apache.polaris: INFO
- org.apache.iceberg.rest: INFO
- # Useful to debug configuration issues:
- # io.smallrye.config: DEBUG
- # -- Configuration for MDC (Mapped Diagnostic Context). Values specified here will be added to the
- # log context of all incoming requests and can be used in log patterns.
- mdc:
- # aid=polaris
- # sid=polaris-service
- {}
-
-# -- Realm context resolver configuration.
-realmContext:
- # -- The type of realm context resolver to use. Two built-in types are supported: default and test;
- # test is not recommended for production as it does not perform any realm validation.
- type: default
- # -- List of valid realms, for use with the default realm context resolver. The first realm in
- # the list is the default realm. Realms not in this list will be rejected.
- realms:
- - POLARIS
-
-# -- Polaris features configuration.
-features:
- # -- Features to enable or disable globally. If a feature is not present in the map, the default
- # built-in value is used.
- defaults: {}
- # ENFORCE_PRINCIPAL_CREDENTIAL_ROTATION_REQUIRED_CHECKING: false
- # SUPPORTED_CATALOG_STORAGE_TYPES:
- # - S3
- # - GCS
- # - AZURE
- # - FILE
- # -- Features to enable or disable per realm. This field is a map of maps. The realm name is the key, and the value is a map of
- # feature names to values. If a feature is not present in the map, the default value from the 'defaults' field is used.
- realmOverrides: {}
- # my-realm:
- # ENFORCE_PRINCIPAL_CREDENTIAL_ROTATION_REQUIRED_CHECKING: true
-
-# -- Polaris persistence configuration.
-persistence:
- # -- The type of persistence to use. Two built-in types are supported: in-memory and eclipse-link.
- type: eclipse-link # in-memory
- # -- The configuration for the eclipse-link persistence manager.
- eclipseLink:
- # -- The secret name to pull persistence.xml from.
- secret:
- # -- The name of the secret to pull persistence.xml from.
- # If not provided, the default built-in persistence.xml will be used. This is probably not what you want.
- name: ~
- # -- The key in the secret to pull persistence.xml from.
- key: persistence.xml
- # -- The persistence unit name to use.
- persistenceUnit: polaris
-
-# -- Polaris FileIO configuration.
-fileIo:
- # -- The type of file IO to use. Two built-in types are supported: default and wasb. The wasb one translates WASB paths to ABFS ones.
- type: default
-
-# -- Storage credentials for the server. If the following properties are unset, default
-# credentials will be used, in which case the pod must have the necessary permissions to access the storage.
-storage:
- # -- The secret to pull storage credentials from.
- secret:
- # -- The name of the secret to pull storage credentials from.
- name: ~
- # -- The key in the secret to pull the AWS access key ID from. Only required when using AWS.
- awsAccessKeyId: ~
- # -- The key in the secret to pull the AWS secret access key from. Only required when using AWS.
- awsSecretAccessKey: ~
- # -- The key in the secret to pull the GCP token from. Only required when using GCP.
- gcpToken: ~
- # -- The key in the secret to pull the GCP token expiration time from. Only required when using GCP. Must be a valid ISO 8601 duration. The default is PT1H (1 hour).
- gcpTokenLifespan: ~
-
-# -- Polaris authentication configuration.
-authentication:
- # -- The type of authentication to use. Two built-in types are supported: default and test;
- # test is not recommended for production.
- authenticator:
+# -- Configures whether to enable the bootstrap metastore manager job
+bootstrapMetastoreManager: false
Review Comment:
This change doesn't look right. You are removing extremely useful stuff from this values.yaml file, and the added content seems to come from an old state of the Helm chart.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2715307234
Thanks a lot @lefebsy for working on it. I agreed that we don't have to put Helm change in, so that the github pipeline can pass. We can always add it back as a followup PR. It should be ready to merge as long as we fix this https://github.com/apache/polaris/pull/389/files#r1980741611.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1994092260
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import static org.apache.polaris.core.config.FeatureConfiguration.STORAGE_CREDENTIAL_DURATION_SECONDS;
+import static org.apache.polaris.core.config.PolarisConfiguration.loadConfig;
+import static org.apache.polaris.core.storage.PolarisCredentialProperty.AWS_KEY_ID;
+import static org.apache.polaris.core.storage.PolarisCredentialProperty.AWS_SECRET_KEY;
+import static org.apache.polaris.core.storage.PolarisCredentialProperty.AWS_TOKEN;
+
+import jakarta.annotation.Nonnull;
+import jakarta.ws.rs.NotAuthorizedException;
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.profiles.ProfileFileSupplier;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** S3 compatible implementation of PolarisStorageIntegration */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ @Override
+ public EnumMap<PolarisCredentialProperty, String> getSubscopedCreds(
+ @Nonnull PolarisDiagnostics diagnostics,
+ @Nonnull S3CompatibleStorageConfigurationInfo storageConfig,
+ boolean allowListOperation,
+ @Nonnull Set<String> allowedReadLocations,
+ @Nonnull Set<String> allowedWriteLocations) {
+
+ String caI = System.getenv(storageConfig.getS3CredentialsCatalogAccessKeyId());
Review Comment:
I'm strongly opposed to storing credential env. variable name is the strage config (essentially in the catalog config).
The environment and catalog config are different management domains with different concerns.
Using a static env. variable name (e.g. `AWS_ACCESS_KEY`) is ok in the short term, IMHO.
Is it strictly necessary to support different secrets for different catalogs in this case?
##########
polaris-core/src/main/java/org/apache/polaris/core/entity/CatalogEntity.java:
##########
@@ -141,6 +143,22 @@ private StorageConfigInfo getStorageInfo(Map<String, String> internalProperties)
.setRegion(awsConfig.getRegion())
.build();
}
+ if (configInfo instanceof S3CompatibleStorageConfigurationInfo) {
+ S3CompatibleStorageConfigurationInfo s3Config =
+ (S3CompatibleStorageConfigurationInfo) configInfo;
+ return S3CompatibleStorageConfigInfo.builder()
+ .setStorageType(StorageConfigInfo.StorageTypeEnum.S3_COMPATIBLE)
+ .setS3Endpoint(s3Config.getS3Endpoint())
+ .setS3ProfileName(s3Config.getS3ProfileName())
Review Comment:
these kind of attribute copy blocks of code are easy to mess up by accidental typos... Could you add a unit test for this conversion?
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import static org.apache.polaris.core.config.FeatureConfiguration.STORAGE_CREDENTIAL_DURATION_SECONDS;
+import static org.apache.polaris.core.config.PolarisConfiguration.loadConfig;
+import static org.apache.polaris.core.storage.PolarisCredentialProperty.AWS_KEY_ID;
+import static org.apache.polaris.core.storage.PolarisCredentialProperty.AWS_SECRET_KEY;
+import static org.apache.polaris.core.storage.PolarisCredentialProperty.AWS_TOKEN;
+
+import jakarta.annotation.Nonnull;
+import jakarta.ws.rs.NotAuthorizedException;
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.profiles.ProfileFileSupplier;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** S3 compatible implementation of PolarisStorageIntegration */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ @Override
+ public EnumMap<PolarisCredentialProperty, String> getSubscopedCreds(
+ @Nonnull PolarisDiagnostics diagnostics,
+ @Nonnull S3CompatibleStorageConfigurationInfo storageConfig,
+ boolean allowListOperation,
+ @Nonnull Set<String> allowedReadLocations,
+ @Nonnull Set<String> allowedWriteLocations) {
+
+ String caI = System.getenv(storageConfig.getS3CredentialsCatalogAccessKeyId());
+ String caS = System.getenv(storageConfig.getS3CredentialsCatalogSecretAccessKey());
+
+ EnumMap<PolarisCredentialProperty, String> propertiesMap =
+ new EnumMap<>(PolarisCredentialProperty.class);
+ propertiesMap.put(PolarisCredentialProperty.AWS_ENDPOINT, storageConfig.getS3Endpoint());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_PATH_STYLE_ACCESS,
+ storageConfig.getS3PathStyleAccess().toString());
+ if (storageConfig.getS3Region() != null) {
+ propertiesMap.put(PolarisCredentialProperty.CLIENT_REGION, storageConfig.getS3Region());
+ }
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+ StsClientBuilder stsBuilder = StsClient.builder();
+ stsBuilder.endpointOverride(URI.create(storageConfig.getS3Endpoint()));
+ if (storageConfig.getS3ProfileName() != null) {
+ stsBuilder.credentialsProvider(
+ ProfileCredentialsProvider.builder()
+ .profileFile(ProfileFileSupplier.defaultSupplier())
+ .profileName(storageConfig.getS3ProfileName())
+ .build());
+ LOGGER.debug("S3Compatible - stsClient using profile from catalog settings");
+ } else if (caI != null && caS != null) {
+ stsBuilder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(caI, caS)));
+ LOGGER.debug("S3Compatible - stsClient using keys from catalog settings");
+ }
+ try (StsClient stsClient = stsBuilder.build()) {
Review Comment:
Creating an StsClient for each request is an overkill, IMHO, but we can probably refactor that later.
##########
polaris-core/src/main/java/org/apache/polaris/core/entity/CatalogEntity.java:
##########
@@ -250,6 +268,21 @@ public Builder setStorageConfigurationInfo(
awsConfig.validateArn(awsConfigModel.getRoleArn());
config = awsConfig;
break;
+
+ case S3_COMPATIBLE:
+ S3CompatibleStorageConfigInfo s3ConfigModel =
+ (S3CompatibleStorageConfigInfo) storageConfigModel;
+ config =
+ new S3CompatibleStorageConfigurationInfo(
+ s3ConfigModel.getS3Endpoint(),
+ s3ConfigModel.getS3ProfileName(),
+ s3ConfigModel.getS3CredentialsCatalogAccessKeyEnvVar(),
+ s3ConfigModel.getS3CredentialsCatalogSecretAccessKeyEnvVar(),
+ s3ConfigModel.getS3PathStyleAccess(),
+ s3ConfigModel.getS3Region(),
+ s3ConfigModel.getS3RoleArn(),
+ new ArrayList<>(allowedLocations));
Review Comment:
nit: why `new` array?
##########
regtests/README.md:
##########
@@ -40,7 +40,7 @@ follows:
```shell
./gradlew clean :polaris-quarkus-server:assemble -Dquarkus.container-image.build=true --no-build-cache
-docker compose -f ./regtests/docker-compose.yml up --build --exit-code-from regtest
+docker-compose -f ./regtests/docker-compose.yml up --build --exit-code-from regtest
Review Comment:
This seems to be unrelated to the purpose of this PR. Please make this change in a separate PR.
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/PolarisStorageConfigurationInfo.java:
##########
@@ -241,6 +243,7 @@ public void validateMaxAllowedLocations(int maxAllowedLocations) {
/** Polaris' storage type, each has a fixed prefix for its location */
public enum StorageType {
S3("s3://"),
+ S3_COMPATIBLE("s3://"),
Review Comment:
I believe `s3` is sufficient for this PR because AWS also maps to only `s3`. Other URIs schemes (like `s3a`) are applicable to AWS too, but I think it is best to handle that in a follow-up PR.
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/StorageUtil.java:
##########
@@ -62,4 +67,136 @@ public class StorageUtil {
public static @Nonnull String getBucket(URI uri) {
return uri.getAuthority();
}
+
+ /**
+ * Given a path, return it without leading slash
+ *
+ * @param path A path to parse
+ * @return Same path without leading slash
+ */
+ private static @Nonnull String trimLeadingSlash(String path) {
+ if (path.startsWith("/")) {
+ path = path.substring(1);
+ }
+ return path;
+ }
+
+ /**
+ * Given an uri, and format an S3 path
+ *
+ * @param uri A path to parse
+ * @return A bucket and a path joined by slash
+ */
+ private static @Nonnull String parseS3Path(URI uri) {
+ String bucket = getBucket(uri);
+ String path = trimLeadingSlash(uri.getPath());
+ return String.join("/", bucket, path);
+ }
+
+ /**
+ * Given a roleArn, return the prefix
+ *
+ * @param roleArn A roleArn to parse
+ * @return The prefix of the roleArn
+ */
+ private static String getArnPrefixFor(String roleArn) {
+ if (roleArn.contains("aws-cn")) {
+ return "arn:aws-cn:s3:::";
+ } else if (roleArn.contains("aws-us-gov")) {
+ return "arn:aws-us-gov:s3:::";
+ } else {
+ return "arn:aws:s3:::";
+ }
+ }
+
+ /**
+ * generate an IamPolicy from the input readLocations and writeLocations, optionally with list
+ * support. Credentials will be scoped to exactly the resources provided. If read and write
+ * locations are empty, a non-empty policy will be generated that grants GetObject and optionally
+ * ListBucket privileges with no resources. This prevents us from sending an empty policy to AWS
+ * and just assuming the role with full privileges.
+ *
+ * @param roleArn A roleArn
+ * @param allowList Allow list or not
+ * @param readLocations A list of input read locations
+ * @param writeLocations A list of input write locations
+ * @return A policy limiting scope access
+ */
+ // TODO - add KMS key access
+ public static IamPolicy policyString(
Review Comment:
Why not reuse the policy from the code the deals with AWS S3?
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/StorageUtil.java:
##########
@@ -62,4 +67,136 @@ public class StorageUtil {
public static @Nonnull String getBucket(URI uri) {
return uri.getAuthority();
}
+
+ /**
+ * Given a path, return it without leading slash
+ *
+ * @param path A path to parse
+ * @return Same path without leading slash
+ */
+ private static @Nonnull String trimLeadingSlash(String path) {
+ if (path.startsWith("/")) {
+ path = path.substring(1);
+ }
+ return path;
+ }
+
+ /**
+ * Given an uri, and format an S3 path
+ *
+ * @param uri A path to parse
+ * @return A bucket and a path joined by slash
+ */
+ private static @Nonnull String parseS3Path(URI uri) {
+ String bucket = getBucket(uri);
+ String path = trimLeadingSlash(uri.getPath());
+ return String.join("/", bucket, path);
+ }
+
+ /**
+ * Given a roleArn, return the prefix
+ *
+ * @param roleArn A roleArn to parse
+ * @return The prefix of the roleArn
+ */
+ private static String getArnPrefixFor(String roleArn) {
+ if (roleArn.contains("aws-cn")) {
+ return "arn:aws-cn:s3:::";
+ } else if (roleArn.contains("aws-us-gov")) {
+ return "arn:aws-us-gov:s3:::";
Review Comment:
I believe this kind of special cases should not be in Polaris code, but it's ok for this PR as I understand it mimics previous AWS-specific logic.
##########
regtests/minio/docker-compose.yml:
##########
@@ -0,0 +1,66 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+services:
+ polaris-minio:
+ image: minio/minio:latest
+ container_name: minio
+ environment:
+ - MINIO_ROOT_USER=admin
+ - MINIO_ROOT_PASSWORD=password
+ - MINIO_DOMAIN=minio
+ networks:
+ minio_net:
+ aliases:
+ - warehouse.minio
+ - warehouse2.minio
+ ports:
+ - 9001:9001
+ - 9000:9000
+ volumes:
+ - ./miniodata:/data
+ - ./certs:/root/.minio/certs/
Review Comment:
Is this strictly necessary for tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2843840123
I started related, but somewhat indirect refactoring in #1504
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "sfc-gh-schen (via GitHub)" <gi...@apache.org>.
sfc-gh-schen commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1884637510
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.s3compatible.S3CompatibleStorageConfigurationInfo.CredsCatalogAndClientStrategyEnum;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** Credential vendor that supports generating */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+ private StsClient stsClient;
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ public void createStsClient(S3CompatibleStorageConfigurationInfo s3storageConfig) {
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+ stsBuilder.region(
+ Region
+ .US_WEST_1); // default region to avoid bug, because most (all?) S3 compatible softwares
+ // do not care about regions
+ stsBuilder.endpointOverride(
+ URI.create(
+ s3storageConfig
+ .getS3Endpoint())); // S3Compatible - AWS STS endpoint is unique and different from
Review Comment:
Okay.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1885226004
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.s3compatible.S3CompatibleStorageConfigurationInfo.CredsCatalogAndClientStrategyEnum;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** Credential vendor that supports generating */
Review Comment:
That was the unchanged comment copied from AWS class. Removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "mmgaggle (via GitHub)" <gi...@apache.org>.
mmgaggle commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2649813953
> Most of the code here seems like it should be delegated to either catalog-level properties (to init the FileIO with the right endpoint) or customizations to the way we construct the STS builder. I don't think a separate `S3_COMPATIBLE` StorageIntegration really needs to exist, with the only distinct feature here being the vending of raw credentials.
Is there a way to define a storage endpoint that is used for S3 and STS per catalog without this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1953598222
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleStorageConfigurationInfo.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import java.util.List;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Polaris Storage Configuration information for an S3 Compatible solution, MinIO, Ceph, Dell ECS...
+ */
+public class S3CompatibleStorageConfigurationInfo extends PolarisStorageConfigurationInfo {
+
+ // 5 is the approximate max allowed locations for the size of AccessPolicy when LIST is required
+ // for allowed read and write locations for subscoping creds.
+ @JsonIgnore private static final int MAX_ALLOWED_LOCATIONS = 5;
+ private @NotNull String s3Endpoint;
+ private @Nullable String s3CredentialsCatalogAccessKeyId;
+ private @Nullable String s3CredentialsCatalogSecretAccessKey;
+ private @Nullable Boolean s3PathStyleAccess;
+ private @NotNull Boolean skipCredentialSubscopingIndirection;
+ private @Nullable String s3CredentialsClientAccessKeyId;
+ private @Nullable String s3CredentialsClientSecretAccessKey;
+ private @Nullable String s3Region;
+ private @Nullable String s3RoleArn;
+
+ // Constructor
+ @JsonCreator
+ public S3CompatibleStorageConfigurationInfo(
+ @JsonProperty(value = "storageType", required = true) @NotNull StorageType storageType,
+ @JsonProperty(value = "s3Endpoint", required = true) @NotNull String s3Endpoint,
+ @JsonProperty(value = "s3CredentialsCatalogAccessKeyId", required = true) @NotNull
+ String s3CredentialsCatalogAccessKeyId,
+ @JsonProperty(value = "s3CredentialsCatalogSecretAccessKey", required = true) @NotNull
+ String s3CredentialsCatalogSecretAccessKey,
+ @JsonProperty(value = "SkipCredentialSubscopingIndirection", required = false) @Nullable
+ Boolean skipCredentialSubscopingIndirection,
+ @JsonProperty(value = "s3CredentialsClientAccessKeyId", required = false) @Nullable
+ String s3CredentialsClientAccessKeyId,
+ @JsonProperty(value = "s3CredentialsClientSecretAccessKey", required = false) @Nullable
+ String s3CredentialsClientSecretAccessKey,
+ @JsonProperty(value = "s3PathStyleAccess", required = false) @Nullable
+ Boolean s3PathStyleAccess,
+ @JsonProperty(value = "s3Region", required = false) @Nullable String s3Region,
+ @JsonProperty(value = "s3RoleArn", required = false) @Nullable String s3RoleArn,
+ @JsonProperty(value = "allowedLocations", required = true) @Nullable
+ List<String> allowedLocations) {
+
+ // storing properties
+ super(storageType, allowedLocations);
+ validateMaxAllowedLocations(MAX_ALLOWED_LOCATIONS);
+ this.s3PathStyleAccess = s3PathStyleAccess;
+ this.s3Endpoint = s3Endpoint;
+ this.s3CredentialsCatalogAccessKeyId = s3CredentialsCatalogAccessKeyId;
+ this.s3CredentialsCatalogSecretAccessKey = s3CredentialsCatalogSecretAccessKey;
+ this.s3CredentialsClientAccessKeyId = s3CredentialsClientAccessKeyId;
+ this.s3CredentialsClientSecretAccessKey = s3CredentialsClientSecretAccessKey;
+ this.skipCredentialSubscopingIndirection = skipCredentialSubscopingIndirection;
+ this.s3Region = s3Region;
+ this.s3RoleArn = s3RoleArn;
+ }
+
+ public @NotNull String getS3Endpoint() {
+ return this.s3Endpoint;
+ }
+
+ public @NotNull Boolean getS3PathStyleAccess() {
+ return this.s3PathStyleAccess;
+ }
+
+ public @NotNull String getS3CredentialsCatalogAccessKeyId() {
+ return (this.s3CredentialsCatalogAccessKeyId == null)
+ ? ""
+ : this.s3CredentialsCatalogAccessKeyId;
+ }
+
+ public @NotNull String getS3CredentialsCatalogSecretAccessKey() {
+ return (this.s3CredentialsCatalogSecretAccessKey == null)
+ ? ""
+ : this.s3CredentialsCatalogSecretAccessKey;
+ }
+
+ public @NotNull String getS3CredentialsClientAccessKeyId() {
+ return (this.s3CredentialsClientAccessKeyId == null) ? "" : this.s3CredentialsClientAccessKeyId;
+ }
+
+ public @NotNull String getS3CredentialsClientSecretAccessKey() {
+ return (this.s3CredentialsClientSecretAccessKey == null)
+ ? ""
+ : this.s3CredentialsClientSecretAccessKey;
+ }
+
+ public @NotNull String getS3RoleArn() {
+ return this.s3RoleArn;
+ }
+
+ public @Nullable String getS3Region() {
+ return this.s3Region;
+ }
+
+ public @Nullable Boolean getSkipCredentialSubscopingIndirection() {
+ return this.skipCredentialSubscopingIndirection;
+ }
+
+ @Override
+ public String toString() {
+ return MoreObjects.toStringHelper(this)
+ .add("storageType", getStorageType().name())
+ .add("allowedLocation", getAllowedLocations())
+ .toString();
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1953512629
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import jakarta.annotation.Nonnull;
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Stream;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.policybuilder.iam.IamConditionOperator;
+import software.amazon.awssdk.policybuilder.iam.IamEffect;
+import software.amazon.awssdk.policybuilder.iam.IamPolicy;
+import software.amazon.awssdk.policybuilder.iam.IamResource;
+import software.amazon.awssdk.policybuilder.iam.IamStatement;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** S3 compatible implementation of PolarisStorageIntegration */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+ private StsClient stsClient;
+ private String caI, caS; // catalogAccessKeyId & catalogSecretAccessKey
+ private String clI, clS; // clientAccessKeyId & clientSecretAccessKey
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public EnumMap<PolarisCredentialProperty, String> getSubscopedCreds(
+ @NotNull PolarisDiagnostics diagnostics,
+ @NotNull S3CompatibleStorageConfigurationInfo storageConfig,
+ boolean allowListOperation,
+ @NotNull Set<String> allowedReadLocations,
+ @NotNull Set<String> allowedWriteLocations) {
+
+ caI = System.getenv(storageConfig.getS3CredentialsCatalogAccessKeyId());
+ caS = System.getenv(storageConfig.getS3CredentialsCatalogSecretAccessKey());
+
+ EnumMap<PolarisCredentialProperty, String> propertiesMap =
+ new EnumMap<>(PolarisCredentialProperty.class);
+ propertiesMap.put(PolarisCredentialProperty.AWS_ENDPOINT, storageConfig.getS3Endpoint());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_PATH_STYLE_ACCESS,
+ storageConfig.getS3PathStyleAccess().toString());
+ if (storageConfig.getS3Region() != null) {
+ propertiesMap.put(PolarisCredentialProperty.CLIENT_REGION, storageConfig.getS3Region());
+ }
+
+ if (storageConfig.getSkipCredentialSubscopingIndirection() == true) {
+ LOGGER.debug("S3Compatible - skipCredentialSubscopingIndirection !");
+ clI = System.getenv(storageConfig.getS3CredentialsClientAccessKeyId());
+ clS = System.getenv(storageConfig.getS3CredentialsClientSecretAccessKey());
+ if (clI != null && clS != null) {
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, clI);
+ propertiesMap.put(PolarisCredentialProperty.AWS_SECRET_KEY, clS);
+ } else {
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, caI);
+ propertiesMap.put(PolarisCredentialProperty.AWS_SECRET_KEY, caS);
+ }
+ } else {
+ LOGGER.debug("S3Compatible - assumeRole !");
+ createStsClient(storageConfig);
+ AssumeRoleResponse response =
+ stsClient.assumeRole(
+ AssumeRoleRequest.builder()
+ .roleSessionName("PolarisCredentialsSTS")
+ .roleArn(
+ (storageConfig.getS3RoleArn() == null) ? "" : storageConfig.getS3RoleArn())
+ .policy(
+ policyString(allowListOperation, allowedReadLocations, allowedWriteLocations)
+ .toJson())
+ .build());
+
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, response.credentials().accessKeyId());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_SECRET_KEY, response.credentials().secretAccessKey());
+ propertiesMap.put(PolarisCredentialProperty.AWS_TOKEN, response.credentials().sessionToken());
+ LOGGER.debug(
+ "S3Compatible - assumeRole - Token Expiration at : {}",
+ response.credentials().expiration().toString());
+ }
+
+ return propertiesMap;
+ }
+
+ public void createStsClient(S3CompatibleStorageConfigurationInfo storageConfig) {
Review Comment:
You have also ask to move stsClient from class to method, so I will simply delete this mehod
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2711427740
Hi @lefebsy, let me know once the PR is ready for another look. As I said here, https://github.com/apache/polaris/pull/389#discussion_r1987787270, this PR isn't blocked by other PRs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "omarsmak (via GitHub)" <gi...@apache.org>.
omarsmak commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1988647044
##########
polaris-core/src/main/java/org/apache/polaris/core/entity/CatalogEntity.java:
##########
@@ -250,6 +268,21 @@ public Builder setStorageConfigurationInfo(
awsConfig.validateArn(awsConfigModel.getRoleArn());
config = awsConfig;
break;
+
+ case S3_COMPATIBLE:
Review Comment:
I am still trying to figure out why we opted to introduce a new source type `S3_COMPATIBLE`? IMO this should be agnostic, is the issue here enforcing the s3 custom endpoint and path style? (they are mandatory with compatible)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1989876050
##########
helm/polaris/values.yaml:
##########
@@ -359,209 +359,76 @@ extraEnv:
# name: aws-secret
# key: secret_access_key
-# -- Extra volumes to add to the polaris pod. See https://kubernetes.io/docs/concepts/storage/volumes/.
-extraVolumes: []
- # - name: extra-volume
- # emptyDir: {}
-
-# -- Extra volume mounts to add to the polaris container. See https://kubernetes.io/docs/concepts/storage/volumes/.
-extraVolumeMounts: []
- # - name: extra-volume
- # mountPath: /usr/share/extra-volume
-
-# -- Add additional init containers to the polaris pod(s) See https://kubernetes.io/docs/concepts/workloads/pods/init-containers/.
-extraInitContainers: []
- # - name: your-image-name
- # image: your-image
- # imagePullPolicy: Always
- # command: ['sh', '-c', 'echo "hello world"']
-
-tracing:
- # -- Specifies whether tracing for the polaris server should be enabled.
- enabled: false
- # -- The collector endpoint URL to connect to (required).
- # The endpoint URL must have either the http:// or the https:// scheme.
- # The collector must talk the OpenTelemetry protocol (OTLP) and the port must be its gRPC port (by default 4317).
- # See https://quarkus.io/guides/opentelemetry for more information.
- endpoint: "http://otlp-collector:4317"
- # -- Which requests should be sampled. Valid values are: "all", "none", or a ratio between 0.0 and
- # "1.0d" (inclusive). E.g. "0.5d" means that 50% of the requests will be sampled.
- # Note: avoid entering numbers here, always prefer a string representation of the ratio.
- sample: "1.0d"
- # -- Resource attributes to identify the polaris service among other tracing sources.
- # See https://opentelemetry.io/docs/reference/specification/resource/semantic_conventions/#service.
- # If left empty, traces will be attached to a service named "Apache Polaris"; to change this,
- # provide a service.name attribute here.
- attributes:
- {}
- # service.name: my-polaris
-
-metrics:
- # -- Specifies whether metrics for the polaris server should be enabled.
- enabled: true
- # -- Additional tags (dimensional labels) to add to the metrics.
- tags:
- {}
- # service: polaris
- # environment: production
-
-serviceMonitor:
- # -- Specifies whether a ServiceMonitor for Prometheus operator should be created.
- enabled: true
- # -- The scrape interval; leave empty to let Prometheus decide. Must be a valid duration, e.g. 1d, 1h30m, 5m, 10s.
- interval: ""
- # -- Labels for the created ServiceMonitor so that Prometheus operator can properly pick it up.
- labels:
- {}
- # release: prometheus
- # -- Relabeling rules to apply to metrics. Ref https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config.
- metricRelabelings: []
- # - source_labels: [ __meta_kubernetes_namespace ]
- # separator: ;
- # regex: (.*)
- # target_label: namespace
- # replacement: $1
- # action: replace
-
-# -- Logging configuration.
-logging:
- # -- The log level of the root category, which is used as the default log level for all categories.
- level: INFO
- # -- The header name to use for the request ID.
- requestIdHeaderName: Polaris-Request-Id
- # -- Configuration for the console appender.
- console:
- # -- Whether to enable the console appender.
- enabled: true
- # -- The log level of the console appender.
- threshold: ALL
- # -- Whether to log in JSON format.
- json: false
- # -- The log format to use. Ignored if JSON format is enabled. See
- # https://quarkus.io/guides/logging#logging-format for details.
- format: "%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p [%c{3.}] [%X{requestId},%X{realmId}] [%X{traceId},%X{parentId},%X{spanId},%X{sampled}] (%t) %s%e%n"
- # -- Configuration for the file appender.
- file:
- # -- Whether to enable the file appender.
- enabled: false
- # -- The log level of the file appender.
- threshold: ALL
- # -- Whether to log in JSON format.
- json: false
- # -- The log format to use. Ignored if JSON format is enabled. See
- # https://quarkus.io/guides/logging#logging-format for details.
- format: "%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p [%c{3.}] [%X{requestId},%X{realmId}] [%X{traceId},%X{parentId},%X{spanId},%X{sampled}] (%t) %s%e%n"
- # -- The local directory where log files are stored. The persistent volume claim will be mounted
- # here.
- logsDir: /deployments/logs
- # -- The log file name.
- fileName: polaris.log
- # -- Log rotation configuration.
- rotation:
- # -- The maximum size of the log file before it is rotated. Should be expressed as a Kubernetes quantity.
- maxFileSize: 100Mi
- # -- The maximum number of backup files to keep.
- maxBackupIndex: 5
- # -- An optional suffix to append to the rotated log files. If present, the rotated log files
- # will be grouped in time buckets, and each bucket will contain at most maxBackupIndex files.
- # The suffix must be in a date-time format that is understood by DateTimeFormatter. If the
- # suffix ends with .gz or .zip, the rotated files will also be compressed using the
- # corresponding algorithm.
- fileSuffix: ~ # .yyyy-MM-dd.gz
- # -- The log storage configuration. A persistent volume claim will be created using these
- # settings.
- storage:
- # -- The storage class name of the persistent volume claim to create.
- className: standard
- # -- The size of the persistent volume claim to create.
- size: 512Gi
- # -- Labels to add to the persistent volume claim spec selector; a persistent volume with
- # matching labels must exist. Leave empty if using dynamic provisioning.
- selectorLabels: {}
- # app.kubernetes.io/name: polaris
- # app.kubernetes.io/instance: RELEASE-NAME
- # -- Configuration for specific log categories.
- categories:
- org.apache.polaris: INFO
- org.apache.iceberg.rest: INFO
- # Useful to debug configuration issues:
- # io.smallrye.config: DEBUG
- # -- Configuration for MDC (Mapped Diagnostic Context). Values specified here will be added to the
- # log context of all incoming requests and can be used in log patterns.
- mdc:
- # aid=polaris
- # sid=polaris-service
- {}
-
-# -- Realm context resolver configuration.
-realmContext:
- # -- The type of realm context resolver to use. Two built-in types are supported: default and test;
- # test is not recommended for production as it does not perform any realm validation.
- type: default
- # -- List of valid realms, for use with the default realm context resolver. The first realm in
- # the list is the default realm. Realms not in this list will be rejected.
- realms:
- - POLARIS
-
-# -- Polaris features configuration.
-features:
- # -- Features to enable or disable globally. If a feature is not present in the map, the default
- # built-in value is used.
- defaults: {}
- # ENFORCE_PRINCIPAL_CREDENTIAL_ROTATION_REQUIRED_CHECKING: false
- # SUPPORTED_CATALOG_STORAGE_TYPES:
- # - S3
- # - GCS
- # - AZURE
- # - FILE
- # -- Features to enable or disable per realm. This field is a map of maps. The realm name is the key, and the value is a map of
- # feature names to values. If a feature is not present in the map, the default value from the 'defaults' field is used.
- realmOverrides: {}
- # my-realm:
- # ENFORCE_PRINCIPAL_CREDENTIAL_ROTATION_REQUIRED_CHECKING: true
-
-# -- Polaris persistence configuration.
-persistence:
- # -- The type of persistence to use. Two built-in types are supported: in-memory and eclipse-link.
- type: eclipse-link # in-memory
- # -- The configuration for the eclipse-link persistence manager.
- eclipseLink:
- # -- The secret name to pull persistence.xml from.
- secret:
- # -- The name of the secret to pull persistence.xml from.
- # If not provided, the default built-in persistence.xml will be used. This is probably not what you want.
- name: ~
- # -- The key in the secret to pull persistence.xml from.
- key: persistence.xml
- # -- The persistence unit name to use.
- persistenceUnit: polaris
-
-# -- Polaris FileIO configuration.
-fileIo:
- # -- The type of file IO to use. Two built-in types are supported: default and wasb. The wasb one translates WASB paths to ABFS ones.
- type: default
-
-# -- Storage credentials for the server. If the following properties are unset, default
-# credentials will be used, in which case the pod must have the necessary permissions to access the storage.
-storage:
- # -- The secret to pull storage credentials from.
- secret:
- # -- The name of the secret to pull storage credentials from.
- name: ~
- # -- The key in the secret to pull the AWS access key ID from. Only required when using AWS.
- awsAccessKeyId: ~
- # -- The key in the secret to pull the AWS secret access key from. Only required when using AWS.
- awsSecretAccessKey: ~
- # -- The key in the secret to pull the GCP token from. Only required when using GCP.
- gcpToken: ~
- # -- The key in the secret to pull the GCP token expiration time from. Only required when using GCP. Must be a valid ISO 8601 duration. The default is PT1H (1 hour).
- gcpTokenLifespan: ~
-
-# -- Polaris authentication configuration.
-authentication:
- # -- The type of authentication to use. Two built-in types are supported: default and test;
- # test is not recommended for production.
- authenticator:
+# -- Configures whether to enable the bootstrap metastore manager job
+bootstrapMetastoreManager: false
Review Comment:
Ok it's a total mistake from my part, I do not know when this occurred, probably during a rebase. I will come back to main branch for this file.
Sorry
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "jean-humann (via GitHub)" <gi...@apache.org>.
jean-humann commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2520270033
Hello everyone this PR seems to be blocked for a month now, is there anything we can do to make it to the end ? 🙏
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1830231421
##########
regtests/minio/miniodata/Readme.md:
##########
@@ -0,0 +1 @@
+# Folder for MiniIo data container volume
Review Comment:
Yes. Hawk eyes !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1811159119
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3/S3CredentialsStorageIntegration.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3;
+
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** Credential vendor that supports generating */
+public class S3CredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3StorageConfigurationInfo> {
+
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CredentialsStorageIntegration.class);
+
+ private StsClient stsClient;
+
+ // Constructor
+ public S3CredentialsStorageIntegration() {
+ super(S3CredentialsStorageIntegration.class.getName());
+ }
+
+ public void createStsClient(S3StorageConfigurationInfo s3storageConfig) {
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+
+ LOGGER.info(
+ "S3Compatible - AWS STS endpoint is unique and different from the S3 Endpoint. AWS SDK need to be overided with dedicated Endpoint from S3Compatible, otherwise the AWS STS url is targeted");
+
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+
+ stsBuilder.region(
+ Region
+ .US_WEST_1); // default region to avoid bug, because most (all?) S3 compatible softwares
Review Comment:
From the name, it sounds like this class is not only used for s3compat
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1811167252
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3/S3CredentialsStorageIntegration.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3;
+
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** Credential vendor that supports generating */
+public class S3CredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3StorageConfigurationInfo> {
+
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CredentialsStorageIntegration.class);
+
+ private StsClient stsClient;
+
+ // Constructor
+ public S3CredentialsStorageIntegration() {
+ super(S3CredentialsStorageIntegration.class.getName());
+ }
+
+ public void createStsClient(S3StorageConfigurationInfo s3storageConfig) {
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+
+ LOGGER.info(
Review Comment:
I think we can do a little cleanup of spurious whitespace, comments, etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2428877476
Hello,
For reviewers, I have a security concern about this proposition of implementation.
- The configurationStorage is communicated and displayed everywhere in Polaris : logs, createCatalog response, getCatalog response...
- This configuration store not a simple role_arn but also secret_keys. It should be possible to filter the sensible data in the logs and in the responses.
I have not find by myself the best place in code to do this.
Maybe it could be also intersting for other storage types to be able to mask some sensible properties.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1813159130
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3/S3CredentialsStorageIntegration.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3;
+
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** Credential vendor that supports generating */
+public class S3CredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3StorageConfigurationInfo> {
+
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CredentialsStorageIntegration.class);
+
+ private StsClient stsClient;
+
+ // Constructor
+ public S3CredentialsStorageIntegration() {
+ super(S3CredentialsStorageIntegration.class.getName());
+ }
+
+ public void createStsClient(S3StorageConfigurationInfo s3storageConfig) {
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+
+ LOGGER.info(
+ "S3Compatible - AWS STS endpoint is unique and different from the S3 Endpoint. AWS SDK need to be overided with dedicated Endpoint from S3Compatible, otherwise the AWS STS url is targeted");
+
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+
+ stsBuilder.region(
+ Region
+ .US_WEST_1); // default region to avoid bug, because most (all?) S3 compatible softwares
Review Comment:
Correct. It could logically be used with any S3, AWS included.
The name is maybe a bad compromise, I agree to change for a better one.
This choice is the result of this two constatations :
- In core/storage folder the classes are named aws, gcp and azure
- But in Enum Storage Type AWS is directly named S3....
So I found this S3_COMPATIBILITY compromise. MinIO use it...
If Enum Storage was different and aws implementation named also AWS like GCP & Azure....
but it is not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1813159130
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3/S3CredentialsStorageIntegration.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3;
+
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** Credential vendor that supports generating */
+public class S3CredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3StorageConfigurationInfo> {
+
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CredentialsStorageIntegration.class);
+
+ private StsClient stsClient;
+
+ // Constructor
+ public S3CredentialsStorageIntegration() {
+ super(S3CredentialsStorageIntegration.class.getName());
+ }
+
+ public void createStsClient(S3StorageConfigurationInfo s3storageConfig) {
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+
+ LOGGER.info(
+ "S3Compatible - AWS STS endpoint is unique and different from the S3 Endpoint. AWS SDK need to be overided with dedicated Endpoint from S3Compatible, otherwise the AWS STS url is targeted");
+
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+
+ stsBuilder.region(
+ Region
+ .US_WEST_1); // default region to avoid bug, because most (all?) S3 compatible softwares
Review Comment:
Correct. It could logically be used with any S3, AWS included (but I removed the cross region tweak implemented in the AWS FileIOClientFactory).
The name is maybe a bad compromise, I agree to change for a better one.
At the beginning I have named it s3compat.... but at the end simply s3.
This choice is the result of this two constatations :
- In core/storage folder the classes are named aws, gcp and azure
- But in Enum Storage Type AWS is directly named S3....
So I found in enum this S3_COMPATIBILITY compromise. MinIO use this term...
For the class I shortened it to S3 because it could works with AWS too.
I can come back to S3compat everywhere...
If Enum Storage was different and aws implementation named also AWS like GCP & Azure.... but it is not. Modifying the existing values of the Enum can be a breaking change, I do not want to introduce this change in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1813159130
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3/S3CredentialsStorageIntegration.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3;
+
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** Credential vendor that supports generating */
+public class S3CredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3StorageConfigurationInfo> {
+
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CredentialsStorageIntegration.class);
+
+ private StsClient stsClient;
+
+ // Constructor
+ public S3CredentialsStorageIntegration() {
+ super(S3CredentialsStorageIntegration.class.getName());
+ }
+
+ public void createStsClient(S3StorageConfigurationInfo s3storageConfig) {
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+
+ LOGGER.info(
+ "S3Compatible - AWS STS endpoint is unique and different from the S3 Endpoint. AWS SDK need to be overided with dedicated Endpoint from S3Compatible, otherwise the AWS STS url is targeted");
+
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+
+ stsBuilder.region(
+ Region
+ .US_WEST_1); // default region to avoid bug, because most (all?) S3 compatible softwares
Review Comment:
Correct. It could logically be used with any S3, AWS included (but I removed the cross region tweak implemented in the AWS FileIOClientFactory and overide the STS endpoint).
The name is maybe a bad compromise, I agree to change for a better one.
At the beginning I have named it s3compat.... but at the end simply s3.
This choice is the result of this two constatations :
- In core/storage folder the classes are named aws, gcp and azure
- But in Enum Storage Type AWS is directly named S3....
So I found in enum this S3_COMPATIBILITY compromise. MinIO use this term...
For the class I shortened it to S3 because it could works with AWS too.
I can come back to S3compat everywhere...
If Enum Storage was different and aws implementation named also AWS like GCP & Azure.... but it is not. Modifying the existing values of the Enum can be a breaking change, I do not want to introduce this change in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1889844516
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.s3compatible.S3CompatibleStorageConfigurationInfo.CredsCatalogAndClientStrategyEnum;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** Credential vendor that supports generating */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+ private StsClient stsClient;
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ public void createStsClient(S3CompatibleStorageConfigurationInfo s3storageConfig) {
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+ stsBuilder.region(
+ Region
+ .US_WEST_1); // default region to avoid bug, because most (all?) S3 compatible softwares
+ // do not care about regions
+ stsBuilder.endpointOverride(
+ URI.create(
+ s3storageConfig
+ .getS3Endpoint())); // S3Compatible - AWS STS endpoint is unique and different from
+ // the S3 Endpoint
+ stsBuilder.credentialsProvider(
+ StaticCredentialsProvider.create(
+ AwsBasicCredentials.create(
+ s3storageConfig.getS3CredentialsCatalogAccessKeyId(),
+ s3storageConfig.getS3CredentialsCatalogSecretAccessKey())));
+ this.stsClient = stsBuilder.build();
+ LOGGER.debug("S3Compatible - stsClient successfully built");
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public EnumMap<PolarisCredentialProperty, String> getSubscopedCreds(
+ @NotNull PolarisDiagnostics diagnostics,
+ @NotNull S3CompatibleStorageConfigurationInfo storageConfig,
+ boolean allowListOperation,
+ @NotNull Set<String> allowedReadLocations,
+ @NotNull Set<String> allowedWriteLocations) {
+
+ LOGGER.debug("S3Compatible - getSubscopedCreds - applying credential strategy");
+ EnumMap<PolarisCredentialProperty, String> propertiesMap =
+ new EnumMap<>(PolarisCredentialProperty.class);
+ propertiesMap.put(PolarisCredentialProperty.AWS_ENDPOINT, storageConfig.getS3Endpoint());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_PATH_STYLE_ACCESS,
+ storageConfig.getS3PathStyleAccess().toString());
+
+ switch (storageConfig.getCredsVendingStrategy()) {
+ case KEYS_SAME_AS_CATALOG:
+ if (storageConfig.getCredsCatalogAndClientStrategy()
+ == CredsCatalogAndClientStrategyEnum.ENV_VAR_NAME) {
+ String cai = System.getenv(storageConfig.getS3CredentialsCatalogAccessKeyId());
+ String cas = System.getenv(storageConfig.getS3CredentialsCatalogSecretAccessKey());
+ if (cai == null || cas == null) {
+ throw new IllegalArgumentException(
+ "Expect valid environment variable for catalog keys");
+ } else {
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, cai);
+ propertiesMap.put(PolarisCredentialProperty.AWS_SECRET_KEY, cas);
+ }
+ } else {
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_KEY_ID,
+ storageConfig.getS3CredentialsCatalogAccessKeyId());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_SECRET_KEY,
+ storageConfig.getS3CredentialsCatalogSecretAccessKey());
+ }
+ break;
+
+ case KEYS_DEDICATED_TO_CLIENT:
+ if (storageConfig.getCredsCatalogAndClientStrategy()
+ == CredsCatalogAndClientStrategyEnum.ENV_VAR_NAME) {
+ String cli = System.getenv(storageConfig.getS3CredentialsClientAccessKeyId());
+ String cls = System.getenv(storageConfig.getS3CredentialsClientSecretAccessKey());
Review Comment:
DefaultCredentialsProvider : done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2707171388
> I refer to the field(s3PathStyleAccess) in this class, which can be set to null. In that case, storageConfig.getS3PathStyleAccess().toString() will throw a NPE
There is a default "false" in rest spec. But ok 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1827006689
##########
spec/polaris-management-service.yml:
##########
@@ -901,6 +903,58 @@ components:
required:
- roleArn
+ S3StorageConfigInfo:
+ type: object
+ description: S3 compatible storage configuration info (MinIO, Dell ECS, Netapp StorageGRID, ...)
+ allOf:
+ - $ref: '#/components/schemas/StorageConfigInfo'
+ properties:
+ credsCatalogAndClientStrategy:
+ type: string
+ enum:
+ - VALUE
+ - ENV_VAR_NAME
+ default: ENV_VAR_NAME
+ example: "ACCESS_KEY"
+ description: When you send key VALUE directly via this command, they should apear in logs. By ENV_VAR_NAME without dollar, only a reference will appear in logs, but the value have to be available as environnement variable in the context where Polaris is running
+ credsVendingStrategy:
+ type: string
+ enum:
+ - TOKEN_WITH_ASSUME_ROLE
+ - KEYS_SAME_AS_CATALOG
+ - KEYS_DEDICATED_TO_CLIENT
Review Comment:
Ok.
It's simply one key for catalog itself, then another unique key for any clients whoever they are. I Let client distinction to the _principal/role/privilege_ level. I think it is hard at the class _storage/credential_ level to stick a pair of keys to each different clients.
It is a basic way, when SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION is True and there is not a temporary token, to not divulge internal catalog key and serve a key that can be deactivated or rotated for security concerns without breaking catalog itself.
After discussing with MonkeyCanCode here
[Prod Deployment credentials](https://github.com/apache/polaris/discussions/405) the main advantage in this proposal is that you have not to rely on the main credentials provided at the global Polaris service level.
Today if you revoke the Polaris service credentials for AWS, all catalogs with AWS storages are instantly KO.
In this implementation each catalog is independent. It is the same idea about clients keys, to not breaking catalog when clients keys are revoked or rotated fo security reasons.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1826552138
##########
spec/polaris-management-service.yml:
##########
@@ -901,6 +903,58 @@ components:
required:
- roleArn
+ S3StorageConfigInfo:
+ type: object
+ description: S3 compatible storage configuration info (MinIO, Dell ECS, Netapp StorageGRID, ...)
+ allOf:
+ - $ref: '#/components/schemas/StorageConfigInfo'
+ properties:
+ credsCatalogAndClientStrategy:
+ type: string
+ enum:
+ - VALUE
+ - ENV_VAR_NAME
+ default: ENV_VAR_NAME
+ example: "ACCESS_KEY"
+ description: When you send key VALUE directly via this command, they should apear in logs. By ENV_VAR_NAME without dollar, only a reference will appear in logs, but the value have to be available as environnement variable in the context where Polaris is running
+ credsVendingStrategy:
+ type: string
+ enum:
+ - TOKEN_WITH_ASSUME_ROLE
+ - KEYS_SAME_AS_CATALOG
Review Comment:
Correct. Default value False. With a clear description to explain users not familiar with concept of "subScoping" implemented in Polaris. I will refactor this way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1828956270
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.s3compatible.S3CompatibleStorageConfigurationInfo.CredsCatalogAndClientStrategyEnum;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** Credential vendor that supports generating */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+ private StsClient stsClient;
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ public void createStsClient(S3CompatibleStorageConfigurationInfo s3storageConfig) {
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+ stsBuilder.region(
+ Region
+ .US_WEST_1); // default region to avoid bug, because most (all?) S3 compatible softwares
+ // do not care about regions
+ stsBuilder.endpointOverride(
+ URI.create(
+ s3storageConfig
+ .getS3Endpoint())); // S3Compatible - AWS STS endpoint is unique and different from
+ // the S3 Endpoint
+ stsBuilder.credentialsProvider(
+ StaticCredentialsProvider.create(
+ AwsBasicCredentials.create(
+ s3storageConfig.getS3CredentialsCatalogAccessKeyId(),
+ s3storageConfig.getS3CredentialsCatalogSecretAccessKey())));
+ this.stsClient = stsBuilder.build();
+ LOGGER.debug("S3Compatible - stsClient successfully built");
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public EnumMap<PolarisCredentialProperty, String> getSubscopedCreds(
+ @NotNull PolarisDiagnostics diagnostics,
+ @NotNull S3CompatibleStorageConfigurationInfo storageConfig,
+ boolean allowListOperation,
+ @NotNull Set<String> allowedReadLocations,
+ @NotNull Set<String> allowedWriteLocations) {
+
+ LOGGER.debug("S3Compatible - getSubscopedCreds - applying credential strategy");
+ EnumMap<PolarisCredentialProperty, String> propertiesMap =
+ new EnumMap<>(PolarisCredentialProperty.class);
+ propertiesMap.put(PolarisCredentialProperty.AWS_ENDPOINT, storageConfig.getS3Endpoint());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_PATH_STYLE_ACCESS,
+ storageConfig.getS3PathStyleAccess().toString());
+
+ switch (storageConfig.getCredsVendingStrategy()) {
+ case KEYS_SAME_AS_CATALOG:
+ if (storageConfig.getCredsCatalogAndClientStrategy()
+ == CredsCatalogAndClientStrategyEnum.ENV_VAR_NAME) {
+ String cai = System.getenv(storageConfig.getS3CredentialsCatalogAccessKeyId());
+ String cas = System.getenv(storageConfig.getS3CredentialsCatalogSecretAccessKey());
+ if (cai == null || cas == null) {
+ throw new IllegalArgumentException(
+ "Expect valid environment variable for catalog keys");
+ } else {
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, cai);
+ propertiesMap.put(PolarisCredentialProperty.AWS_SECRET_KEY, cas);
+ }
+ } else {
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_KEY_ID,
+ storageConfig.getS3CredentialsCatalogAccessKeyId());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_SECRET_KEY,
+ storageConfig.getS3CredentialsCatalogSecretAccessKey());
+ }
+ break;
+
+ case KEYS_DEDICATED_TO_CLIENT:
+ if (storageConfig.getCredsCatalogAndClientStrategy()
+ == CredsCatalogAndClientStrategyEnum.ENV_VAR_NAME) {
+ String cli = System.getenv(storageConfig.getS3CredentialsClientAccessKeyId());
+ String cls = System.getenv(storageConfig.getS3CredentialsClientSecretAccessKey());
+ if (cli == null || cls == null) {
+ throw new IllegalArgumentException("Expect valid environment variable for client keys");
+ } else {
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, cli);
+ propertiesMap.put(PolarisCredentialProperty.AWS_SECRET_KEY, cls);
+ }
+ } else {
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_KEY_ID,
+ storageConfig.getS3CredentialsClientAccessKeyId());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_SECRET_KEY,
+ storageConfig.getS3CredentialsClientSecretAccessKey());
+ }
+ break;
+
+ case TOKEN_WITH_ASSUME_ROLE:
+ if (this.stsClient == null) {
+ createStsClient(storageConfig);
+ }
+ LOGGER.debug("S3Compatible - assumeRole !");
+ AssumeRoleResponse response =
+ stsClient.assumeRole(
+ AssumeRoleRequest.builder()
+ .roleSessionName("PolarisCredentialsSTS")
+ // @TODO customize duration
+ // .durationSeconds(1800)
Review Comment:
See:
https://github.com/apache/polaris/pull/408
Let's just make sure to do this in whichever PR merges second :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1830247969
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.s3compatible.S3CompatibleStorageConfigurationInfo.CredsCatalogAndClientStrategyEnum;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** Credential vendor that supports generating */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+ private StsClient stsClient;
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ public void createStsClient(S3CompatibleStorageConfigurationInfo s3storageConfig) {
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+ stsBuilder.region(
+ Region
+ .US_WEST_1); // default region to avoid bug, because most (all?) S3 compatible softwares
+ // do not care about regions
+ stsBuilder.endpointOverride(
+ URI.create(
+ s3storageConfig
+ .getS3Endpoint())); // S3Compatible - AWS STS endpoint is unique and different from
+ // the S3 Endpoint
+ stsBuilder.credentialsProvider(
+ StaticCredentialsProvider.create(
+ AwsBasicCredentials.create(
+ s3storageConfig.getS3CredentialsCatalogAccessKeyId(),
+ s3storageConfig.getS3CredentialsCatalogSecretAccessKey())));
+ this.stsClient = stsBuilder.build();
+ LOGGER.debug("S3Compatible - stsClient successfully built");
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public EnumMap<PolarisCredentialProperty, String> getSubscopedCreds(
+ @NotNull PolarisDiagnostics diagnostics,
+ @NotNull S3CompatibleStorageConfigurationInfo storageConfig,
+ boolean allowListOperation,
+ @NotNull Set<String> allowedReadLocations,
+ @NotNull Set<String> allowedWriteLocations) {
+
+ LOGGER.debug("S3Compatible - getSubscopedCreds - applying credential strategy");
+ EnumMap<PolarisCredentialProperty, String> propertiesMap =
+ new EnumMap<>(PolarisCredentialProperty.class);
+ propertiesMap.put(PolarisCredentialProperty.AWS_ENDPOINT, storageConfig.getS3Endpoint());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_PATH_STYLE_ACCESS,
+ storageConfig.getS3PathStyleAccess().toString());
Review Comment:
To be refactor for satisfying change to requested : boolean "SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION".
I will try to move it in the catalog properties.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1813179181
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3/S3CredentialsStorageIntegration.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3;
+
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** Credential vendor that supports generating */
+public class S3CredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3StorageConfigurationInfo> {
+
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CredentialsStorageIntegration.class);
+
+ private StsClient stsClient;
+
+ // Constructor
+ public S3CredentialsStorageIntegration() {
+ super(S3CredentialsStorageIntegration.class.getName());
+ }
+
+ public void createStsClient(S3StorageConfigurationInfo s3storageConfig) {
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+
+ LOGGER.info(
Review Comment:
OK.
It was helping me to understand behavior at the beginning.
I agree to remove and clean whitespaces and comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1813159130
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3/S3CredentialsStorageIntegration.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3;
+
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** Credential vendor that supports generating */
+public class S3CredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3StorageConfigurationInfo> {
+
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CredentialsStorageIntegration.class);
+
+ private StsClient stsClient;
+
+ // Constructor
+ public S3CredentialsStorageIntegration() {
+ super(S3CredentialsStorageIntegration.class.getName());
+ }
+
+ public void createStsClient(S3StorageConfigurationInfo s3storageConfig) {
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+
+ LOGGER.info(
+ "S3Compatible - AWS STS endpoint is unique and different from the S3 Endpoint. AWS SDK need to be overided with dedicated Endpoint from S3Compatible, otherwise the AWS STS url is targeted");
+
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+
+ stsBuilder.region(
+ Region
+ .US_WEST_1); // default region to avoid bug, because most (all?) S3 compatible softwares
Review Comment:
Correct.
It could potentially be used with any S3, AWS included (in this implementation I removed the cross region tweak implemented in the AWS FileIOClientFactory and overide the STS endpoint).
The name is maybe a bad compromise, I agree to change for a better one.
At the beginning I have named it s3compat.... but at the end simply s3.
This choice is the result of this two constatations :
- In core/storage folder the classes are named aws, gcp and azure
- But in Enum Storage Type AWS is directly named S3....
So I found in enum this S3_COMPATIBILITY compromise. MinIO use this term...
For the class I shortened it to S3 because S3Compatible was verbose.
I can come back to S3compatible everywhere to be consistent...
If Enum Storage was different and aws implementation named also AWS like GCP & Azure.... but it is not. Modifying the existing values of the Enum can be a breaking change, I do not want to introduce this change in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1813159130
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3/S3CredentialsStorageIntegration.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3;
+
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** Credential vendor that supports generating */
+public class S3CredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3StorageConfigurationInfo> {
+
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CredentialsStorageIntegration.class);
+
+ private StsClient stsClient;
+
+ // Constructor
+ public S3CredentialsStorageIntegration() {
+ super(S3CredentialsStorageIntegration.class.getName());
+ }
+
+ public void createStsClient(S3StorageConfigurationInfo s3storageConfig) {
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+
+ LOGGER.info(
+ "S3Compatible - AWS STS endpoint is unique and different from the S3 Endpoint. AWS SDK need to be overided with dedicated Endpoint from S3Compatible, otherwise the AWS STS url is targeted");
+
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+
+ stsBuilder.region(
+ Region
+ .US_WEST_1); // default region to avoid bug, because most (all?) S3 compatible softwares
Review Comment:
Correct. It could logically be used with any S3, AWS included.
The name is maybe a bad compromise, I agree to change for a better one.
At the beginning I have named it s3compat.... but at the end simply s3.
This choice is the result of this two constatations :
- In core/storage folder the classes are named aws, gcp and azure
- But in Enum Storage Type AWS is directly named S3....
So I found in enum this S3_COMPATIBILITY compromise. MinIO use it...
For the class I shortened it to S3 because it could works with AWS too.
I can come back to S3compat everywhere...
If Enum Storage was different and aws implementation named also AWS like GCP & Azure.... but it is not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1813159130
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3/S3CredentialsStorageIntegration.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3;
+
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** Credential vendor that supports generating */
+public class S3CredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3StorageConfigurationInfo> {
+
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CredentialsStorageIntegration.class);
+
+ private StsClient stsClient;
+
+ // Constructor
+ public S3CredentialsStorageIntegration() {
+ super(S3CredentialsStorageIntegration.class.getName());
+ }
+
+ public void createStsClient(S3StorageConfigurationInfo s3storageConfig) {
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+
+ LOGGER.info(
+ "S3Compatible - AWS STS endpoint is unique and different from the S3 Endpoint. AWS SDK need to be overided with dedicated Endpoint from S3Compatible, otherwise the AWS STS url is targeted");
+
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+
+ stsBuilder.region(
+ Region
+ .US_WEST_1); // default region to avoid bug, because most (all?) S3 compatible softwares
Review Comment:
Correct.
It could potentially be used with any S3, AWS included (in this implementation I removed the cross region tweak implemented in the AWS FileIOClientFactory and overide the STS endpoint).
The name is maybe a bad compromise, I agree to change for a better one.
At the beginning I have named it s3compat.... but at the end simply s3.
This choice is the result of this two constatations :
- In core/storage folder the classes are named aws, gcp and azure
- But in Enum Storage Type AWS is directly named S3....
So I found in enum this S3_COMPATIBILITY compromise. MinIO use this term...
For the class I shortened it to S3 because S3Compatible was verbose.
I can come back to S3compatible everywhere...
If Enum Storage was different and aws implementation named also AWS like GCP & Azure.... but it is not. Modifying the existing values of the Enum can be a breaking change, I do not want to introduce this change in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1981903387
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleStorageConfigurationInfo.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import jakarta.annotation.Nonnull;
+import java.util.List;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * S3-Compatible Storage Configuration. This class holds the parameters needed to connect to
+ * S3-compatible storage services such as MinIO, Ceph, Dell ECS, etc.
+ */
+public class S3CompatibleStorageConfigurationInfo extends PolarisStorageConfigurationInfo {
+
+ // 5 is the approximate max allowed locations for the size of AccessPolicy when LIST is required
+ // for allowed read and write locations for sub-scoping credentials.
+ @JsonIgnore private static final int MAX_ALLOWED_LOCATIONS = 5;
Review Comment:
Ok I will rebase after #1068 merged
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "collado-mike (via GitHub)" <gi...@apache.org>.
collado-mike commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1980500751
##########
spec/polaris-management-service.yml:
##########
@@ -905,6 +907,51 @@ components:
required:
- roleArn
+ S3CompatibleStorageConfigInfo:
+ type: object
+ description: S3 compatible storage configuration info (MinIO, Ceph, Dell ECS, Netapp StorageGRID, ...)
+ allOf:
+ - $ref: '#/components/schemas/StorageConfigInfo'
+ properties:
+ s3.endpoint:
+ type: string
+ description: the S3 endpoint, will also be used as STS endpoint
+ example: "http[s]://host:port"
+ s3.credentials.catalog.accessKeyId:
+ type: string
+ description: Default to AWS credentials, otherwise set the environement variable name for the 'ACCESS_KEY_ID' used by the catalog to communicate with S3
+ example: "CATALOG_1_ACCESS_KEY_ENV_VARIABLE_NAME or AWS_ACCESS_KEY_ID"
Review Comment:
Personally, I'm ok with this impl, as the env vars are never exposed to the catalog creator. If the catalog creator submits env vars that don't exist, the STS call fails and that's it. The caller cannot use this approach to read information from the server that would normally be inaccessible
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1981884935
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import static org.apache.polaris.core.PolarisConfiguration.STORAGE_CREDENTIAL_DURATION_SECONDS;
+import static org.apache.polaris.core.PolarisConfiguration.loadConfig;
+
+import jakarta.ws.rs.NotAuthorizedException;
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.profiles.ProfileFileSupplier;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** S3 compatible implementation of PolarisStorageIntegration */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ @Override
+ public EnumMap<PolarisCredentialProperty, String> getSubscopedCreds(
+ @NotNull PolarisDiagnostics diagnostics,
+ @NotNull S3CompatibleStorageConfigurationInfo storageConfig,
+ boolean allowListOperation,
+ @NotNull Set<String> allowedReadLocations,
+ @NotNull Set<String> allowedWriteLocations) {
+
+ String caI = System.getenv(storageConfig.getS3CredentialsCatalogAccessKeyId());
+ String caS = System.getenv(storageConfig.getS3CredentialsCatalogSecretAccessKey());
+
+ EnumMap<PolarisCredentialProperty, String> propertiesMap =
+ new EnumMap<>(PolarisCredentialProperty.class);
+ propertiesMap.put(PolarisCredentialProperty.AWS_ENDPOINT, storageConfig.getS3Endpoint());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_PATH_STYLE_ACCESS,
+ storageConfig.getS3PathStyleAccess().toString());
+ if (storageConfig.getS3Region() != null) {
+ propertiesMap.put(PolarisCredentialProperty.CLIENT_REGION, storageConfig.getS3Region());
+ }
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
Review Comment:
Correct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "metadaddy (via GitHub)" <gi...@apache.org>.
metadaddy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1989564301
##########
polaris-core/src/main/java/org/apache/polaris/core/entity/CatalogEntity.java:
##########
@@ -250,6 +268,21 @@ public Builder setStorageConfigurationInfo(
awsConfig.validateArn(awsConfigModel.getRoleArn());
config = awsConfig;
break;
+
+ case S3_COMPATIBLE:
+ S3CompatibleStorageConfigInfo s3ConfigModel =
+ (S3CompatibleStorageConfigInfo) storageConfigModel;
+ config =
+ new S3CompatibleStorageConfigurationInfo(
Review Comment:
Also, Backblaze (only one 'l'! 🙂) B2, and I'm guessing some other S3-compatible services, doesn't have an STS, so there's no point in specifying an endpoint.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "omarsmak (via GitHub)" <gi...@apache.org>.
omarsmak commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1989656706
##########
helm/polaris/values.yaml:
##########
@@ -359,209 +359,76 @@ extraEnv:
# name: aws-secret
# key: secret_access_key
-# -- Extra volumes to add to the polaris pod. See https://kubernetes.io/docs/concepts/storage/volumes/.
-extraVolumes: []
- # - name: extra-volume
- # emptyDir: {}
-
-# -- Extra volume mounts to add to the polaris container. See https://kubernetes.io/docs/concepts/storage/volumes/.
-extraVolumeMounts: []
- # - name: extra-volume
- # mountPath: /usr/share/extra-volume
-
-# -- Add additional init containers to the polaris pod(s) See https://kubernetes.io/docs/concepts/workloads/pods/init-containers/.
-extraInitContainers: []
- # - name: your-image-name
- # image: your-image
- # imagePullPolicy: Always
- # command: ['sh', '-c', 'echo "hello world"']
-
-tracing:
- # -- Specifies whether tracing for the polaris server should be enabled.
- enabled: false
- # -- The collector endpoint URL to connect to (required).
- # The endpoint URL must have either the http:// or the https:// scheme.
- # The collector must talk the OpenTelemetry protocol (OTLP) and the port must be its gRPC port (by default 4317).
- # See https://quarkus.io/guides/opentelemetry for more information.
- endpoint: "http://otlp-collector:4317"
- # -- Which requests should be sampled. Valid values are: "all", "none", or a ratio between 0.0 and
- # "1.0d" (inclusive). E.g. "0.5d" means that 50% of the requests will be sampled.
- # Note: avoid entering numbers here, always prefer a string representation of the ratio.
- sample: "1.0d"
- # -- Resource attributes to identify the polaris service among other tracing sources.
- # See https://opentelemetry.io/docs/reference/specification/resource/semantic_conventions/#service.
- # If left empty, traces will be attached to a service named "Apache Polaris"; to change this,
- # provide a service.name attribute here.
- attributes:
- {}
- # service.name: my-polaris
-
-metrics:
- # -- Specifies whether metrics for the polaris server should be enabled.
- enabled: true
- # -- Additional tags (dimensional labels) to add to the metrics.
- tags:
- {}
- # service: polaris
- # environment: production
-
-serviceMonitor:
- # -- Specifies whether a ServiceMonitor for Prometheus operator should be created.
- enabled: true
- # -- The scrape interval; leave empty to let Prometheus decide. Must be a valid duration, e.g. 1d, 1h30m, 5m, 10s.
- interval: ""
- # -- Labels for the created ServiceMonitor so that Prometheus operator can properly pick it up.
- labels:
- {}
- # release: prometheus
- # -- Relabeling rules to apply to metrics. Ref https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config.
- metricRelabelings: []
- # - source_labels: [ __meta_kubernetes_namespace ]
- # separator: ;
- # regex: (.*)
- # target_label: namespace
- # replacement: $1
- # action: replace
-
-# -- Logging configuration.
-logging:
- # -- The log level of the root category, which is used as the default log level for all categories.
- level: INFO
- # -- The header name to use for the request ID.
- requestIdHeaderName: Polaris-Request-Id
- # -- Configuration for the console appender.
- console:
- # -- Whether to enable the console appender.
- enabled: true
- # -- The log level of the console appender.
- threshold: ALL
- # -- Whether to log in JSON format.
- json: false
- # -- The log format to use. Ignored if JSON format is enabled. See
- # https://quarkus.io/guides/logging#logging-format for details.
- format: "%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p [%c{3.}] [%X{requestId},%X{realmId}] [%X{traceId},%X{parentId},%X{spanId},%X{sampled}] (%t) %s%e%n"
- # -- Configuration for the file appender.
- file:
- # -- Whether to enable the file appender.
- enabled: false
- # -- The log level of the file appender.
- threshold: ALL
- # -- Whether to log in JSON format.
- json: false
- # -- The log format to use. Ignored if JSON format is enabled. See
- # https://quarkus.io/guides/logging#logging-format for details.
- format: "%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p [%c{3.}] [%X{requestId},%X{realmId}] [%X{traceId},%X{parentId},%X{spanId},%X{sampled}] (%t) %s%e%n"
- # -- The local directory where log files are stored. The persistent volume claim will be mounted
- # here.
- logsDir: /deployments/logs
- # -- The log file name.
- fileName: polaris.log
- # -- Log rotation configuration.
- rotation:
- # -- The maximum size of the log file before it is rotated. Should be expressed as a Kubernetes quantity.
- maxFileSize: 100Mi
- # -- The maximum number of backup files to keep.
- maxBackupIndex: 5
- # -- An optional suffix to append to the rotated log files. If present, the rotated log files
- # will be grouped in time buckets, and each bucket will contain at most maxBackupIndex files.
- # The suffix must be in a date-time format that is understood by DateTimeFormatter. If the
- # suffix ends with .gz or .zip, the rotated files will also be compressed using the
- # corresponding algorithm.
- fileSuffix: ~ # .yyyy-MM-dd.gz
- # -- The log storage configuration. A persistent volume claim will be created using these
- # settings.
- storage:
- # -- The storage class name of the persistent volume claim to create.
- className: standard
- # -- The size of the persistent volume claim to create.
- size: 512Gi
- # -- Labels to add to the persistent volume claim spec selector; a persistent volume with
- # matching labels must exist. Leave empty if using dynamic provisioning.
- selectorLabels: {}
- # app.kubernetes.io/name: polaris
- # app.kubernetes.io/instance: RELEASE-NAME
- # -- Configuration for specific log categories.
- categories:
- org.apache.polaris: INFO
- org.apache.iceberg.rest: INFO
- # Useful to debug configuration issues:
- # io.smallrye.config: DEBUG
- # -- Configuration for MDC (Mapped Diagnostic Context). Values specified here will be added to the
- # log context of all incoming requests and can be used in log patterns.
- mdc:
- # aid=polaris
- # sid=polaris-service
- {}
-
-# -- Realm context resolver configuration.
-realmContext:
- # -- The type of realm context resolver to use. Two built-in types are supported: default and test;
- # test is not recommended for production as it does not perform any realm validation.
- type: default
- # -- List of valid realms, for use with the default realm context resolver. The first realm in
- # the list is the default realm. Realms not in this list will be rejected.
- realms:
- - POLARIS
-
-# -- Polaris features configuration.
-features:
- # -- Features to enable or disable globally. If a feature is not present in the map, the default
- # built-in value is used.
- defaults: {}
- # ENFORCE_PRINCIPAL_CREDENTIAL_ROTATION_REQUIRED_CHECKING: false
- # SUPPORTED_CATALOG_STORAGE_TYPES:
- # - S3
- # - GCS
- # - AZURE
- # - FILE
- # -- Features to enable or disable per realm. This field is a map of maps. The realm name is the key, and the value is a map of
- # feature names to values. If a feature is not present in the map, the default value from the 'defaults' field is used.
- realmOverrides: {}
- # my-realm:
- # ENFORCE_PRINCIPAL_CREDENTIAL_ROTATION_REQUIRED_CHECKING: true
-
-# -- Polaris persistence configuration.
-persistence:
- # -- The type of persistence to use. Two built-in types are supported: in-memory and eclipse-link.
- type: eclipse-link # in-memory
- # -- The configuration for the eclipse-link persistence manager.
- eclipseLink:
- # -- The secret name to pull persistence.xml from.
- secret:
- # -- The name of the secret to pull persistence.xml from.
- # If not provided, the default built-in persistence.xml will be used. This is probably not what you want.
- name: ~
- # -- The key in the secret to pull persistence.xml from.
- key: persistence.xml
- # -- The persistence unit name to use.
- persistenceUnit: polaris
-
-# -- Polaris FileIO configuration.
-fileIo:
- # -- The type of file IO to use. Two built-in types are supported: default and wasb. The wasb one translates WASB paths to ABFS ones.
- type: default
-
-# -- Storage credentials for the server. If the following properties are unset, default
-# credentials will be used, in which case the pod must have the necessary permissions to access the storage.
-storage:
- # -- The secret to pull storage credentials from.
- secret:
- # -- The name of the secret to pull storage credentials from.
- name: ~
- # -- The key in the secret to pull the AWS access key ID from. Only required when using AWS.
- awsAccessKeyId: ~
- # -- The key in the secret to pull the AWS secret access key from. Only required when using AWS.
- awsSecretAccessKey: ~
- # -- The key in the secret to pull the GCP token from. Only required when using GCP.
- gcpToken: ~
- # -- The key in the secret to pull the GCP token expiration time from. Only required when using GCP. Must be a valid ISO 8601 duration. The default is PT1H (1 hour).
- gcpTokenLifespan: ~
-
-# -- Polaris authentication configuration.
-authentication:
- # -- The type of authentication to use. Two built-in types are supported: default and test;
- # test is not recommended for production.
- authenticator:
+# -- Configures whether to enable the bootstrap metastore manager job
+bootstrapMetastoreManager: false
Review Comment:
@lefebsy maybe would be easier if you take the helm stuff outside of this PR and have it as a follow up. Wdyt? (so we don't prolong this PR further :))
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2707233462
> > I refer to the field(s3PathStyleAccess) in this class, which can be set to null. In that case, storageConfig.getS3PathStyleAccess().toString() will throw a NPE
>
> There is a default "false" in rest spec. But ok 👍
I agreed that it's fine if the object is always created from the REST. But the class itself could be constructed internally in the future. I don't want a NPE surprise in that case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "Gerrit-K (via GitHub)" <gi...@apache.org>.
Gerrit-K commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r2014324360
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import static org.apache.polaris.core.config.FeatureConfiguration.STORAGE_CREDENTIAL_DURATION_SECONDS;
+import static org.apache.polaris.core.config.PolarisConfiguration.loadConfig;
+import static org.apache.polaris.core.storage.PolarisCredentialProperty.AWS_KEY_ID;
+import static org.apache.polaris.core.storage.PolarisCredentialProperty.AWS_SECRET_KEY;
+import static org.apache.polaris.core.storage.PolarisCredentialProperty.AWS_TOKEN;
+
+import jakarta.annotation.Nonnull;
+import jakarta.ws.rs.NotAuthorizedException;
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.profiles.ProfileFileSupplier;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** S3 compatible implementation of PolarisStorageIntegration */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ @Override
+ public EnumMap<PolarisCredentialProperty, String> getSubscopedCreds(
+ @Nonnull PolarisDiagnostics diagnostics,
+ @Nonnull S3CompatibleStorageConfigurationInfo storageConfig,
+ boolean allowListOperation,
+ @Nonnull Set<String> allowedReadLocations,
+ @Nonnull Set<String> allowedWriteLocations) {
+
+ String caI = System.getenv(storageConfig.getS3CredentialsCatalogAccessKeyId());
+ String caS = System.getenv(storageConfig.getS3CredentialsCatalogSecretAccessKey());
+
+ EnumMap<PolarisCredentialProperty, String> propertiesMap =
+ new EnumMap<>(PolarisCredentialProperty.class);
+ propertiesMap.put(PolarisCredentialProperty.AWS_ENDPOINT, storageConfig.getS3Endpoint());
Review Comment:
tl;dr: I believe the endpoint is currently not set when the `SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION` feature is activated, but I think it should be (or it should be configurable at the catalog level again).
Long story: I tested this MR successfully in an earlier stage (back in early January, before the Quarkus migration) and now wanted to upgrade our PoC setup to the most recent state of this MR. Our setup involves a Ceph S3 backend that doesn't have the STS configured, which means: we need to be able to adjust the `s3.endpoint` property, but can't use subscoped credentials. This path doesn't seem to be possible anymore. Previously, we had `SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION` disabled (because we weren't aware of it) but `skipCredentialSubscopingIndirection` (on the catalog) enabled. The latter seems to have been removed in one of the updates, so STS is always enforced if not skipped globally. If `SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION` is set, the `getSubscopedCreds` function is not even called by the `FileIOUtil`, so the `S3FileIO` is initialized without region, endpoint etc., which causes it to fall back to environment variables and eventually failing.
For our case, we definitely need to set the endpoint for our Ceph catalog, but also don't want to globally disable STS and/or setting global `AWS_*` variables just to fix the `S3FileIO`, because we want to have AWS S3 and Ceph S3 catalogs in the same Polaris instance. Am I missing something in configuring this or does this not work anymore?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "Gerrit-K (via GitHub)" <gi...@apache.org>.
Gerrit-K commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r2014324360
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import static org.apache.polaris.core.config.FeatureConfiguration.STORAGE_CREDENTIAL_DURATION_SECONDS;
+import static org.apache.polaris.core.config.PolarisConfiguration.loadConfig;
+import static org.apache.polaris.core.storage.PolarisCredentialProperty.AWS_KEY_ID;
+import static org.apache.polaris.core.storage.PolarisCredentialProperty.AWS_SECRET_KEY;
+import static org.apache.polaris.core.storage.PolarisCredentialProperty.AWS_TOKEN;
+
+import jakarta.annotation.Nonnull;
+import jakarta.ws.rs.NotAuthorizedException;
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.profiles.ProfileFileSupplier;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** S3 compatible implementation of PolarisStorageIntegration */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ @Override
+ public EnumMap<PolarisCredentialProperty, String> getSubscopedCreds(
+ @Nonnull PolarisDiagnostics diagnostics,
+ @Nonnull S3CompatibleStorageConfigurationInfo storageConfig,
+ boolean allowListOperation,
+ @Nonnull Set<String> allowedReadLocations,
+ @Nonnull Set<String> allowedWriteLocations) {
+
+ String caI = System.getenv(storageConfig.getS3CredentialsCatalogAccessKeyId());
+ String caS = System.getenv(storageConfig.getS3CredentialsCatalogSecretAccessKey());
+
+ EnumMap<PolarisCredentialProperty, String> propertiesMap =
+ new EnumMap<>(PolarisCredentialProperty.class);
+ propertiesMap.put(PolarisCredentialProperty.AWS_ENDPOINT, storageConfig.getS3Endpoint());
Review Comment:
tl;dr: I believe the endpoint is currently not set when the `SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION` feature is activated, but I think it should be (or it should be configurable at the catalog level again).
Long story: I tested this MR successfully in an earlier stage (back in early January, before the Quarkus migration) and now wanted to upgrade our PoC setup to the most recent state of this MR. Our setup involves a Ceph S3 backend that doesn't have the STS configured, which means: we need to be able to adjust the `s3.endpoint` property, but can't use subscoped credentials. This path doesn't seem to be possible anymore. Previously, we had `SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION` disabled (because we weren't aware of it) but `skipCredentialSubscopingIndirection` (on the catalog) enabled. The latter seems to have been removed in one of the updates, so STS is always enforced if not skipped globally. If `SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION` is set, the `getSubscopedCreds` function is not even called by the `FileIOUtil`, so the `S3FileIO` is initialized without region, endpoint etc., which causes it to fall back to environment variables and eventually failing.
For our case, we definitely need to set the endpoint for our Ceph catalog, but also don't want to globally disable STS and/or setting global `AWS_*` variables just to fix the `S3FileIO`, because we want to have AWS S3 and Ceph S3 catalogs in the same Polaris instance. Am I missing something in configuring this or does this not work anymore?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1948001849
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleStorageConfigurationInfo.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import java.util.List;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Polaris Storage Configuration information for an S3 Compatible solution, MinIO, Ceph, Dell ECS...
+ */
+public class S3CompatibleStorageConfigurationInfo extends PolarisStorageConfigurationInfo {
+
+ // 5 is the approximate max allowed locations for the size of AccessPolicy when LIST is required
+ // for allowed read and write locations for subscoping creds.
+ @JsonIgnore private static final int MAX_ALLOWED_LOCATIONS = 5;
+ private @NotNull String s3Endpoint;
+ private @Nullable String s3CredentialsCatalogAccessKeyId;
+ private @Nullable String s3CredentialsCatalogSecretAccessKey;
+ private @Nullable Boolean s3PathStyleAccess;
+ private @NotNull Boolean skipCredentialSubscopingIndirection;
+ private @Nullable String s3CredentialsClientAccessKeyId;
+ private @Nullable String s3CredentialsClientSecretAccessKey;
+ private @Nullable String s3Region;
+ private @Nullable String s3RoleArn;
+
+ // Constructor
Review Comment:
Nit: remove this comment?
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleStorageConfigurationInfo.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import java.util.List;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Polaris Storage Configuration information for an S3 Compatible solution, MinIO, Ceph, Dell ECS...
+ */
+public class S3CompatibleStorageConfigurationInfo extends PolarisStorageConfigurationInfo {
+
+ // 5 is the approximate max allowed locations for the size of AccessPolicy when LIST is required
+ // for allowed read and write locations for subscoping creds.
+ @JsonIgnore private static final int MAX_ALLOWED_LOCATIONS = 5;
+ private @NotNull String s3Endpoint;
+ private @Nullable String s3CredentialsCatalogAccessKeyId;
+ private @Nullable String s3CredentialsCatalogSecretAccessKey;
+ private @Nullable Boolean s3PathStyleAccess;
+ private @NotNull Boolean skipCredentialSubscopingIndirection;
+ private @Nullable String s3CredentialsClientAccessKeyId;
+ private @Nullable String s3CredentialsClientSecretAccessKey;
+ private @Nullable String s3Region;
+ private @Nullable String s3RoleArn;
Review Comment:
Minor: can we mark them `final` ?
##########
polaris-server.yml:
##########
@@ -69,6 +69,7 @@ featureConfiguration:
ENFORCE_PRINCIPAL_CREDENTIAL_ROTATION_REQUIRED_CHECKING: false
SUPPORTED_CATALOG_STORAGE_TYPES:
- S3
+ - S3_COMPATIBLE
Review Comment:
This will need rebase as Quakus has changed the configurations. You can find it here, https://polaris.apache.org/in-dev/unreleased/configuration/
```
polaris.features.defaults.SUPPORTED_CATALOG_STORAGE_TYPES=["S3","GCS","AZURE"].
Note: this excludes the default FILE storage type, which is not meaningful in a distributed deployment.
```
##########
regtests/minio/Readme.md:
##########
@@ -0,0 +1,42 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# MiniIO Secured
+## Minio and secured buckets with TLS self-signed / custom AC
Review Comment:
Can we give a full name for `AC`?
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import jakarta.annotation.Nonnull;
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Stream;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.policybuilder.iam.IamConditionOperator;
+import software.amazon.awssdk.policybuilder.iam.IamEffect;
+import software.amazon.awssdk.policybuilder.iam.IamPolicy;
+import software.amazon.awssdk.policybuilder.iam.IamResource;
+import software.amazon.awssdk.policybuilder.iam.IamStatement;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** S3 compatible implementation of PolarisStorageIntegration */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+ private StsClient stsClient;
+ private String caI, caS; // catalogAccessKeyId & catalogSecretAccessKey
+ private String clI, clS; // clientAccessKeyId & clientSecretAccessKey
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public EnumMap<PolarisCredentialProperty, String> getSubscopedCreds(
+ @NotNull PolarisDiagnostics diagnostics,
+ @NotNull S3CompatibleStorageConfigurationInfo storageConfig,
+ boolean allowListOperation,
+ @NotNull Set<String> allowedReadLocations,
+ @NotNull Set<String> allowedWriteLocations) {
+
+ caI = System.getenv(storageConfig.getS3CredentialsCatalogAccessKeyId());
+ caS = System.getenv(storageConfig.getS3CredentialsCatalogSecretAccessKey());
+
+ EnumMap<PolarisCredentialProperty, String> propertiesMap =
+ new EnumMap<>(PolarisCredentialProperty.class);
+ propertiesMap.put(PolarisCredentialProperty.AWS_ENDPOINT, storageConfig.getS3Endpoint());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_PATH_STYLE_ACCESS,
+ storageConfig.getS3PathStyleAccess().toString());
+ if (storageConfig.getS3Region() != null) {
+ propertiesMap.put(PolarisCredentialProperty.CLIENT_REGION, storageConfig.getS3Region());
+ }
+
+ if (storageConfig.getSkipCredentialSubscopingIndirection() == true) {
+ LOGGER.debug("S3Compatible - skipCredentialSubscopingIndirection !");
+ clI = System.getenv(storageConfig.getS3CredentialsClientAccessKeyId());
+ clS = System.getenv(storageConfig.getS3CredentialsClientSecretAccessKey());
+ if (clI != null && clS != null) {
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, clI);
+ propertiesMap.put(PolarisCredentialProperty.AWS_SECRET_KEY, clS);
+ } else {
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, caI);
+ propertiesMap.put(PolarisCredentialProperty.AWS_SECRET_KEY, caS);
+ }
+ } else {
+ LOGGER.debug("S3Compatible - assumeRole !");
+ createStsClient(storageConfig);
+ AssumeRoleResponse response =
+ stsClient.assumeRole(
+ AssumeRoleRequest.builder()
+ .roleSessionName("PolarisCredentialsSTS")
+ .roleArn(
+ (storageConfig.getS3RoleArn() == null) ? "" : storageConfig.getS3RoleArn())
+ .policy(
+ policyString(allowListOperation, allowedReadLocations, allowedWriteLocations)
+ .toJson())
+ .build());
+
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, response.credentials().accessKeyId());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_SECRET_KEY, response.credentials().secretAccessKey());
+ propertiesMap.put(PolarisCredentialProperty.AWS_TOKEN, response.credentials().sessionToken());
+ LOGGER.debug(
+ "S3Compatible - assumeRole - Token Expiration at : {}",
+ response.credentials().expiration().toString());
+ }
+
+ return propertiesMap;
+ }
+
+ public void createStsClient(S3CompatibleStorageConfigurationInfo storageConfig) {
+ LOGGER.debug("S3Compatible - createStsClient()");
+ try {
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+ stsBuilder.endpointOverride(URI.create(storageConfig.getS3Endpoint()));
+ if (caI != null && caS != null) {
+ // else default provider build credentials from profile or standard AWS env var
+ stsBuilder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(caI, caS)));
+ }
+ this.stsClient = stsBuilder.build();
+ LOGGER.debug("S3Compatible - stsClient successfully built");
+
+ } catch (Exception e) {
+ System.err.println("S3Compatible - stsClient - build failure : " + e.getMessage());
+ }
+ }
+
+ /*
+ * function from AwsCredentialsStorageIntegration but without roleArn parameter
+ */
+ private IamPolicy policyString(
Review Comment:
Same here, I'd avoid duplicating such a long method. Can we refactor it a bit?
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import jakarta.annotation.Nonnull;
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Stream;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.policybuilder.iam.IamConditionOperator;
+import software.amazon.awssdk.policybuilder.iam.IamEffect;
+import software.amazon.awssdk.policybuilder.iam.IamPolicy;
+import software.amazon.awssdk.policybuilder.iam.IamResource;
+import software.amazon.awssdk.policybuilder.iam.IamStatement;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** S3 compatible implementation of PolarisStorageIntegration */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+ private StsClient stsClient;
+ private String caI, caS; // catalogAccessKeyId & catalogSecretAccessKey
+ private String clI, clS; // clientAccessKeyId & clientSecretAccessKey
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public EnumMap<PolarisCredentialProperty, String> getSubscopedCreds(
+ @NotNull PolarisDiagnostics diagnostics,
+ @NotNull S3CompatibleStorageConfigurationInfo storageConfig,
+ boolean allowListOperation,
+ @NotNull Set<String> allowedReadLocations,
+ @NotNull Set<String> allowedWriteLocations) {
+
+ caI = System.getenv(storageConfig.getS3CredentialsCatalogAccessKeyId());
+ caS = System.getenv(storageConfig.getS3CredentialsCatalogSecretAccessKey());
+
+ EnumMap<PolarisCredentialProperty, String> propertiesMap =
+ new EnumMap<>(PolarisCredentialProperty.class);
+ propertiesMap.put(PolarisCredentialProperty.AWS_ENDPOINT, storageConfig.getS3Endpoint());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_PATH_STYLE_ACCESS,
+ storageConfig.getS3PathStyleAccess().toString());
+ if (storageConfig.getS3Region() != null) {
+ propertiesMap.put(PolarisCredentialProperty.CLIENT_REGION, storageConfig.getS3Region());
+ }
+
+ if (storageConfig.getSkipCredentialSubscopingIndirection() == true) {
+ LOGGER.debug("S3Compatible - skipCredentialSubscopingIndirection !");
+ clI = System.getenv(storageConfig.getS3CredentialsClientAccessKeyId());
+ clS = System.getenv(storageConfig.getS3CredentialsClientSecretAccessKey());
+ if (clI != null && clS != null) {
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, clI);
+ propertiesMap.put(PolarisCredentialProperty.AWS_SECRET_KEY, clS);
+ } else {
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, caI);
+ propertiesMap.put(PolarisCredentialProperty.AWS_SECRET_KEY, caS);
+ }
+ } else {
+ LOGGER.debug("S3Compatible - assumeRole !");
+ createStsClient(storageConfig);
+ AssumeRoleResponse response =
+ stsClient.assumeRole(
+ AssumeRoleRequest.builder()
+ .roleSessionName("PolarisCredentialsSTS")
+ .roleArn(
+ (storageConfig.getS3RoleArn() == null) ? "" : storageConfig.getS3RoleArn())
+ .policy(
+ policyString(allowListOperation, allowedReadLocations, allowedWriteLocations)
+ .toJson())
+ .build());
+
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, response.credentials().accessKeyId());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_SECRET_KEY, response.credentials().secretAccessKey());
+ propertiesMap.put(PolarisCredentialProperty.AWS_TOKEN, response.credentials().sessionToken());
+ LOGGER.debug(
+ "S3Compatible - assumeRole - Token Expiration at : {}",
+ response.credentials().expiration().toString());
+ }
+
+ return propertiesMap;
+ }
+
+ public void createStsClient(S3CompatibleStorageConfigurationInfo storageConfig) {
Review Comment:
Can we make it `private` as it is invoked only within this class?
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleStorageConfigurationInfo.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import java.util.List;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Polaris Storage Configuration information for an S3 Compatible solution, MinIO, Ceph, Dell ECS...
+ */
+public class S3CompatibleStorageConfigurationInfo extends PolarisStorageConfigurationInfo {
+
+ // 5 is the approximate max allowed locations for the size of AccessPolicy when LIST is required
+ // for allowed read and write locations for subscoping creds.
+ @JsonIgnore private static final int MAX_ALLOWED_LOCATIONS = 5;
+ private @NotNull String s3Endpoint;
+ private @Nullable String s3CredentialsCatalogAccessKeyId;
+ private @Nullable String s3CredentialsCatalogSecretAccessKey;
+ private @Nullable Boolean s3PathStyleAccess;
+ private @NotNull Boolean skipCredentialSubscopingIndirection;
+ private @Nullable String s3CredentialsClientAccessKeyId;
+ private @Nullable String s3CredentialsClientSecretAccessKey;
+ private @Nullable String s3Region;
+ private @Nullable String s3RoleArn;
+
+ // Constructor
+ @JsonCreator
+ public S3CompatibleStorageConfigurationInfo(
+ @JsonProperty(value = "storageType", required = true) @NotNull StorageType storageType,
+ @JsonProperty(value = "s3Endpoint", required = true) @NotNull String s3Endpoint,
+ @JsonProperty(value = "s3CredentialsCatalogAccessKeyId", required = true) @NotNull
+ String s3CredentialsCatalogAccessKeyId,
+ @JsonProperty(value = "s3CredentialsCatalogSecretAccessKey", required = true) @NotNull
+ String s3CredentialsCatalogSecretAccessKey,
+ @JsonProperty(value = "SkipCredentialSubscopingIndirection", required = false) @Nullable
+ Boolean skipCredentialSubscopingIndirection,
+ @JsonProperty(value = "s3CredentialsClientAccessKeyId", required = false) @Nullable
+ String s3CredentialsClientAccessKeyId,
+ @JsonProperty(value = "s3CredentialsClientSecretAccessKey", required = false) @Nullable
+ String s3CredentialsClientSecretAccessKey,
+ @JsonProperty(value = "s3PathStyleAccess", required = false) @Nullable
+ Boolean s3PathStyleAccess,
+ @JsonProperty(value = "s3Region", required = false) @Nullable String s3Region,
+ @JsonProperty(value = "s3RoleArn", required = false) @Nullable String s3RoleArn,
+ @JsonProperty(value = "allowedLocations", required = true) @Nullable
+ List<String> allowedLocations) {
+
+ // storing properties
+ super(storageType, allowedLocations);
+ validateMaxAllowedLocations(MAX_ALLOWED_LOCATIONS);
+ this.s3PathStyleAccess = s3PathStyleAccess;
+ this.s3Endpoint = s3Endpoint;
+ this.s3CredentialsCatalogAccessKeyId = s3CredentialsCatalogAccessKeyId;
+ this.s3CredentialsCatalogSecretAccessKey = s3CredentialsCatalogSecretAccessKey;
+ this.s3CredentialsClientAccessKeyId = s3CredentialsClientAccessKeyId;
+ this.s3CredentialsClientSecretAccessKey = s3CredentialsClientSecretAccessKey;
+ this.skipCredentialSubscopingIndirection = skipCredentialSubscopingIndirection;
+ this.s3Region = s3Region;
+ this.s3RoleArn = s3RoleArn;
+ }
+
+ public @NotNull String getS3Endpoint() {
+ return this.s3Endpoint;
+ }
+
+ public @NotNull Boolean getS3PathStyleAccess() {
+ return this.s3PathStyleAccess;
+ }
+
+ public @NotNull String getS3CredentialsCatalogAccessKeyId() {
+ return (this.s3CredentialsCatalogAccessKeyId == null)
+ ? ""
+ : this.s3CredentialsCatalogAccessKeyId;
+ }
+
+ public @NotNull String getS3CredentialsCatalogSecretAccessKey() {
+ return (this.s3CredentialsCatalogSecretAccessKey == null)
+ ? ""
+ : this.s3CredentialsCatalogSecretAccessKey;
Review Comment:
Can we move these logic to constructor? I also suggest builder style, which seems a better fit here.
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleStorageConfigurationInfo.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import java.util.List;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Polaris Storage Configuration information for an S3 Compatible solution, MinIO, Ceph, Dell ECS...
+ */
+public class S3CompatibleStorageConfigurationInfo extends PolarisStorageConfigurationInfo {
+
+ // 5 is the approximate max allowed locations for the size of AccessPolicy when LIST is required
+ // for allowed read and write locations for subscoping creds.
+ @JsonIgnore private static final int MAX_ALLOWED_LOCATIONS = 5;
+ private @NotNull String s3Endpoint;
+ private @Nullable String s3CredentialsCatalogAccessKeyId;
+ private @Nullable String s3CredentialsCatalogSecretAccessKey;
+ private @Nullable Boolean s3PathStyleAccess;
+ private @NotNull Boolean skipCredentialSubscopingIndirection;
+ private @Nullable String s3CredentialsClientAccessKeyId;
+ private @Nullable String s3CredentialsClientSecretAccessKey;
+ private @Nullable String s3Region;
+ private @Nullable String s3RoleArn;
+
+ // Constructor
+ @JsonCreator
+ public S3CompatibleStorageConfigurationInfo(
+ @JsonProperty(value = "storageType", required = true) @NotNull StorageType storageType,
+ @JsonProperty(value = "s3Endpoint", required = true) @NotNull String s3Endpoint,
+ @JsonProperty(value = "s3CredentialsCatalogAccessKeyId", required = true) @NotNull
+ String s3CredentialsCatalogAccessKeyId,
+ @JsonProperty(value = "s3CredentialsCatalogSecretAccessKey", required = true) @NotNull
+ String s3CredentialsCatalogSecretAccessKey,
+ @JsonProperty(value = "SkipCredentialSubscopingIndirection", required = false) @Nullable
+ Boolean skipCredentialSubscopingIndirection,
+ @JsonProperty(value = "s3CredentialsClientAccessKeyId", required = false) @Nullable
+ String s3CredentialsClientAccessKeyId,
+ @JsonProperty(value = "s3CredentialsClientSecretAccessKey", required = false) @Nullable
+ String s3CredentialsClientSecretAccessKey,
+ @JsonProperty(value = "s3PathStyleAccess", required = false) @Nullable
+ Boolean s3PathStyleAccess,
+ @JsonProperty(value = "s3Region", required = false) @Nullable String s3Region,
+ @JsonProperty(value = "s3RoleArn", required = false) @Nullable String s3RoleArn,
+ @JsonProperty(value = "allowedLocations", required = true) @Nullable
+ List<String> allowedLocations) {
+
+ // storing properties
+ super(storageType, allowedLocations);
Review Comment:
Can we use the new type directly here? I think we don't need to pass it from the constructor. Do we?
```suggestion
super(StorageType.S3_COMPATIBLE, allowedLocations);
```
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleStorageConfigurationInfo.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import java.util.List;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Polaris Storage Configuration information for an S3 Compatible solution, MinIO, Ceph, Dell ECS...
+ */
+public class S3CompatibleStorageConfigurationInfo extends PolarisStorageConfigurationInfo {
+
+ // 5 is the approximate max allowed locations for the size of AccessPolicy when LIST is required
+ // for allowed read and write locations for subscoping creds.
+ @JsonIgnore private static final int MAX_ALLOWED_LOCATIONS = 5;
+ private @NotNull String s3Endpoint;
+ private @Nullable String s3CredentialsCatalogAccessKeyId;
+ private @Nullable String s3CredentialsCatalogSecretAccessKey;
+ private @Nullable Boolean s3PathStyleAccess;
+ private @NotNull Boolean skipCredentialSubscopingIndirection;
+ private @Nullable String s3CredentialsClientAccessKeyId;
+ private @Nullable String s3CredentialsClientSecretAccessKey;
+ private @Nullable String s3Region;
+ private @Nullable String s3RoleArn;
+
+ // Constructor
+ @JsonCreator
+ public S3CompatibleStorageConfigurationInfo(
+ @JsonProperty(value = "storageType", required = true) @NotNull StorageType storageType,
+ @JsonProperty(value = "s3Endpoint", required = true) @NotNull String s3Endpoint,
+ @JsonProperty(value = "s3CredentialsCatalogAccessKeyId", required = true) @NotNull
+ String s3CredentialsCatalogAccessKeyId,
+ @JsonProperty(value = "s3CredentialsCatalogSecretAccessKey", required = true) @NotNull
+ String s3CredentialsCatalogSecretAccessKey,
+ @JsonProperty(value = "SkipCredentialSubscopingIndirection", required = false) @Nullable
+ Boolean skipCredentialSubscopingIndirection,
+ @JsonProperty(value = "s3CredentialsClientAccessKeyId", required = false) @Nullable
+ String s3CredentialsClientAccessKeyId,
+ @JsonProperty(value = "s3CredentialsClientSecretAccessKey", required = false) @Nullable
+ String s3CredentialsClientSecretAccessKey,
+ @JsonProperty(value = "s3PathStyleAccess", required = false) @Nullable
+ Boolean s3PathStyleAccess,
+ @JsonProperty(value = "s3Region", required = false) @Nullable String s3Region,
+ @JsonProperty(value = "s3RoleArn", required = false) @Nullable String s3RoleArn,
+ @JsonProperty(value = "allowedLocations", required = true) @Nullable
+ List<String> allowedLocations) {
+
+ // storing properties
+ super(storageType, allowedLocations);
+ validateMaxAllowedLocations(MAX_ALLOWED_LOCATIONS);
+ this.s3PathStyleAccess = s3PathStyleAccess;
+ this.s3Endpoint = s3Endpoint;
+ this.s3CredentialsCatalogAccessKeyId = s3CredentialsCatalogAccessKeyId;
+ this.s3CredentialsCatalogSecretAccessKey = s3CredentialsCatalogSecretAccessKey;
+ this.s3CredentialsClientAccessKeyId = s3CredentialsClientAccessKeyId;
+ this.s3CredentialsClientSecretAccessKey = s3CredentialsClientSecretAccessKey;
+ this.skipCredentialSubscopingIndirection = skipCredentialSubscopingIndirection;
+ this.s3Region = s3Region;
+ this.s3RoleArn = s3RoleArn;
+ }
+
+ public @NotNull String getS3Endpoint() {
+ return this.s3Endpoint;
+ }
+
+ public @NotNull Boolean getS3PathStyleAccess() {
+ return this.s3PathStyleAccess;
+ }
+
+ public @NotNull String getS3CredentialsCatalogAccessKeyId() {
+ return (this.s3CredentialsCatalogAccessKeyId == null)
+ ? ""
+ : this.s3CredentialsCatalogAccessKeyId;
+ }
+
+ public @NotNull String getS3CredentialsCatalogSecretAccessKey() {
+ return (this.s3CredentialsCatalogSecretAccessKey == null)
+ ? ""
+ : this.s3CredentialsCatalogSecretAccessKey;
+ }
+
+ public @NotNull String getS3CredentialsClientAccessKeyId() {
+ return (this.s3CredentialsClientAccessKeyId == null) ? "" : this.s3CredentialsClientAccessKeyId;
+ }
+
+ public @NotNull String getS3CredentialsClientSecretAccessKey() {
+ return (this.s3CredentialsClientSecretAccessKey == null)
+ ? ""
+ : this.s3CredentialsClientSecretAccessKey;
+ }
+
+ public @NotNull String getS3RoleArn() {
+ return this.s3RoleArn;
+ }
+
+ public @Nullable String getS3Region() {
+ return this.s3Region;
+ }
+
+ public @Nullable Boolean getSkipCredentialSubscopingIndirection() {
+ return this.skipCredentialSubscopingIndirection;
+ }
+
+ @Override
+ public String toString() {
+ return MoreObjects.toStringHelper(this)
+ .add("storageType", getStorageType().name())
+ .add("allowedLocation", getAllowedLocations())
+ .toString();
Review Comment:
Can we include all fields?
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleStorageConfigurationInfo.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import java.util.List;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Polaris Storage Configuration information for an S3 Compatible solution, MinIO, Ceph, Dell ECS...
+ */
+public class S3CompatibleStorageConfigurationInfo extends PolarisStorageConfigurationInfo {
+
+ // 5 is the approximate max allowed locations for the size of AccessPolicy when LIST is required
+ // for allowed read and write locations for subscoping creds.
+ @JsonIgnore private static final int MAX_ALLOWED_LOCATIONS = 5;
+ private @NotNull String s3Endpoint;
+ private @Nullable String s3CredentialsCatalogAccessKeyId;
+ private @Nullable String s3CredentialsCatalogSecretAccessKey;
+ private @Nullable Boolean s3PathStyleAccess;
+ private @NotNull Boolean skipCredentialSubscopingIndirection;
+ private @Nullable String s3CredentialsClientAccessKeyId;
+ private @Nullable String s3CredentialsClientSecretAccessKey;
+ private @Nullable String s3Region;
+ private @Nullable String s3RoleArn;
+
+ // Constructor
+ @JsonCreator
+ public S3CompatibleStorageConfigurationInfo(
+ @JsonProperty(value = "storageType", required = true) @NotNull StorageType storageType,
+ @JsonProperty(value = "s3Endpoint", required = true) @NotNull String s3Endpoint,
+ @JsonProperty(value = "s3CredentialsCatalogAccessKeyId", required = true) @NotNull
+ String s3CredentialsCatalogAccessKeyId,
+ @JsonProperty(value = "s3CredentialsCatalogSecretAccessKey", required = true) @NotNull
+ String s3CredentialsCatalogSecretAccessKey,
+ @JsonProperty(value = "SkipCredentialSubscopingIndirection", required = false) @Nullable
+ Boolean skipCredentialSubscopingIndirection,
+ @JsonProperty(value = "s3CredentialsClientAccessKeyId", required = false) @Nullable
+ String s3CredentialsClientAccessKeyId,
+ @JsonProperty(value = "s3CredentialsClientSecretAccessKey", required = false) @Nullable
+ String s3CredentialsClientSecretAccessKey,
+ @JsonProperty(value = "s3PathStyleAccess", required = false) @Nullable
+ Boolean s3PathStyleAccess,
+ @JsonProperty(value = "s3Region", required = false) @Nullable String s3Region,
+ @JsonProperty(value = "s3RoleArn", required = false) @Nullable String s3RoleArn,
+ @JsonProperty(value = "allowedLocations", required = true) @Nullable
+ List<String> allowedLocations) {
+
+ // storing properties
Review Comment:
Nit: can we remove the comment?
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleStorageConfigurationInfo.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import java.util.List;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Polaris Storage Configuration information for an S3 Compatible solution, MinIO, Ceph, Dell ECS...
Review Comment:
```suggestion
* S3-Compatible Storage Configuration. This class holds the parameters needed to connect to S3-compatible storage services such as MinIO, Ceph, Dell ECS, etc.
```
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleStorageConfigurationInfo.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import java.util.List;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Polaris Storage Configuration information for an S3 Compatible solution, MinIO, Ceph, Dell ECS...
+ */
+public class S3CompatibleStorageConfigurationInfo extends PolarisStorageConfigurationInfo {
+
+ // 5 is the approximate max allowed locations for the size of AccessPolicy when LIST is required
+ // for allowed read and write locations for subscoping creds.
+ @JsonIgnore private static final int MAX_ALLOWED_LOCATIONS = 5;
+ private @NotNull String s3Endpoint;
+ private @Nullable String s3CredentialsCatalogAccessKeyId;
+ private @Nullable String s3CredentialsCatalogSecretAccessKey;
+ private @Nullable Boolean s3PathStyleAccess;
+ private @NotNull Boolean skipCredentialSubscopingIndirection;
+ private @Nullable String s3CredentialsClientAccessKeyId;
+ private @Nullable String s3CredentialsClientSecretAccessKey;
+ private @Nullable String s3Region;
+ private @Nullable String s3RoleArn;
+
+ // Constructor
+ @JsonCreator
+ public S3CompatibleStorageConfigurationInfo(
+ @JsonProperty(value = "storageType", required = true) @NotNull StorageType storageType,
+ @JsonProperty(value = "s3Endpoint", required = true) @NotNull String s3Endpoint,
+ @JsonProperty(value = "s3CredentialsCatalogAccessKeyId", required = true) @NotNull
+ String s3CredentialsCatalogAccessKeyId,
+ @JsonProperty(value = "s3CredentialsCatalogSecretAccessKey", required = true) @NotNull
+ String s3CredentialsCatalogSecretAccessKey,
+ @JsonProperty(value = "SkipCredentialSubscopingIndirection", required = false) @Nullable
+ Boolean skipCredentialSubscopingIndirection,
+ @JsonProperty(value = "s3CredentialsClientAccessKeyId", required = false) @Nullable
+ String s3CredentialsClientAccessKeyId,
+ @JsonProperty(value = "s3CredentialsClientSecretAccessKey", required = false) @Nullable
+ String s3CredentialsClientSecretAccessKey,
+ @JsonProperty(value = "s3PathStyleAccess", required = false) @Nullable
+ Boolean s3PathStyleAccess,
+ @JsonProperty(value = "s3Region", required = false) @Nullable String s3Region,
+ @JsonProperty(value = "s3RoleArn", required = false) @Nullable String s3RoleArn,
+ @JsonProperty(value = "allowedLocations", required = true) @Nullable
+ List<String> allowedLocations) {
+
+ // storing properties
+ super(storageType, allowedLocations);
+ validateMaxAllowedLocations(MAX_ALLOWED_LOCATIONS);
+ this.s3PathStyleAccess = s3PathStyleAccess;
Review Comment:
Do we need to set a default value, like `false` when s3PathStyleAccess is null?
##########
regtests/minio/Readme.md:
##########
@@ -0,0 +1,42 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# MiniIO Secured
+## Minio and secured buckets with TLS self-signed / custom AC
+
+To be able to test Polaris with buckets in TLS under custom AC or self-signed certificate
+
+## MiniIO generate self-signed certificates designed for docker-compose setup
+
+- Download minio certificate generator : https://github.com/minio/certgen
+- ```./certgen -host "localhost,minio,*"```
Review Comment:
```suggestion
- Generate certifications: ```./certgen -host "localhost,minio,*"```
```
##########
regtests/minio/Readme.md:
##########
@@ -0,0 +1,42 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# MiniIO Secured
+## Minio and secured buckets with TLS self-signed / custom AC
+
+To be able to test Polaris with buckets in TLS under custom AC or self-signed certificate
Review Comment:
```suggestion
Here are steps to test Polaris with buckets in TLS under custom AC or self-signed certificate
```
##########
spec/polaris-management-service.yml:
##########
@@ -905,6 +907,51 @@ components:
required:
- roleArn
+ S3CompatibleStorageConfigInfo:
+ type: object
+ description: S3 compatible storage configuration info (MinIO, Ceph, Dell ECS, Netapp StorageGRID, ...)
Review Comment:
```suggestion
description: s3-compatible storage configuration info (MinIO, Ceph, Dell ECS, Netapp StorageGRID, ...)
```
##########
spec/polaris-management-service.yml:
##########
@@ -905,6 +907,51 @@ components:
required:
- roleArn
+ S3CompatibleStorageConfigInfo:
+ type: object
+ description: S3 compatible storage configuration info (MinIO, Ceph, Dell ECS, Netapp StorageGRID, ...)
+ allOf:
+ - $ref: '#/components/schemas/StorageConfigInfo'
+ properties:
+ s3.endpoint:
+ type: string
+ description: the S3 endpoint, will also be used as STS endpoint
Review Comment:
I guess we don't have to mention `STS endpoint`, since there is no separated configure for that, and `s3.endpoint` will always be used as the STS endpoint. Can you confirm?
##########
regtests/minio/Readme.md:
##########
@@ -0,0 +1,42 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# MiniIO Secured
+## Minio and secured buckets with TLS self-signed / custom AC
+
+To be able to test Polaris with buckets in TLS under custom AC or self-signed certificate
+
+## MiniIO generate self-signed certificates designed for docker-compose setup
+
+- Download minio certificate generator : https://github.com/minio/certgen
+- ```./certgen -host "localhost,minio,*"```
+- put them in ./certs and ./certs/CAs
+- they will be mounted in default minio container placeholder
Review Comment:
```suggestion
- Put them in ./certs and ./certs/CAs. They will be mounted in the default MinIO container placeholder.
```
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import jakarta.annotation.Nonnull;
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Stream;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.policybuilder.iam.IamConditionOperator;
+import software.amazon.awssdk.policybuilder.iam.IamEffect;
+import software.amazon.awssdk.policybuilder.iam.IamPolicy;
+import software.amazon.awssdk.policybuilder.iam.IamResource;
+import software.amazon.awssdk.policybuilder.iam.IamStatement;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** S3 compatible implementation of PolarisStorageIntegration */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+ private StsClient stsClient;
+ private String caI, caS; // catalogAccessKeyId & catalogSecretAccessKey
+ private String clI, clS; // clientAccessKeyId & clientSecretAccessKey
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public EnumMap<PolarisCredentialProperty, String> getSubscopedCreds(
+ @NotNull PolarisDiagnostics diagnostics,
+ @NotNull S3CompatibleStorageConfigurationInfo storageConfig,
+ boolean allowListOperation,
+ @NotNull Set<String> allowedReadLocations,
+ @NotNull Set<String> allowedWriteLocations) {
+
+ caI = System.getenv(storageConfig.getS3CredentialsCatalogAccessKeyId());
+ caS = System.getenv(storageConfig.getS3CredentialsCatalogSecretAccessKey());
+
+ EnumMap<PolarisCredentialProperty, String> propertiesMap =
+ new EnumMap<>(PolarisCredentialProperty.class);
+ propertiesMap.put(PolarisCredentialProperty.AWS_ENDPOINT, storageConfig.getS3Endpoint());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_PATH_STYLE_ACCESS,
+ storageConfig.getS3PathStyleAccess().toString());
+ if (storageConfig.getS3Region() != null) {
+ propertiesMap.put(PolarisCredentialProperty.CLIENT_REGION, storageConfig.getS3Region());
+ }
+
+ if (storageConfig.getSkipCredentialSubscopingIndirection() == true) {
+ LOGGER.debug("S3Compatible - skipCredentialSubscopingIndirection !");
+ clI = System.getenv(storageConfig.getS3CredentialsClientAccessKeyId());
+ clS = System.getenv(storageConfig.getS3CredentialsClientSecretAccessKey());
+ if (clI != null && clS != null) {
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, clI);
+ propertiesMap.put(PolarisCredentialProperty.AWS_SECRET_KEY, clS);
+ } else {
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, caI);
+ propertiesMap.put(PolarisCredentialProperty.AWS_SECRET_KEY, caS);
+ }
+ } else {
+ LOGGER.debug("S3Compatible - assumeRole !");
+ createStsClient(storageConfig);
+ AssumeRoleResponse response =
+ stsClient.assumeRole(
+ AssumeRoleRequest.builder()
+ .roleSessionName("PolarisCredentialsSTS")
+ .roleArn(
+ (storageConfig.getS3RoleArn() == null) ? "" : storageConfig.getS3RoleArn())
+ .policy(
+ policyString(allowListOperation, allowedReadLocations, allowedWriteLocations)
+ .toJson())
+ .build());
+
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, response.credentials().accessKeyId());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_SECRET_KEY, response.credentials().secretAccessKey());
+ propertiesMap.put(PolarisCredentialProperty.AWS_TOKEN, response.credentials().sessionToken());
+ LOGGER.debug(
+ "S3Compatible - assumeRole - Token Expiration at : {}",
+ response.credentials().expiration().toString());
+ }
+
+ return propertiesMap;
+ }
+
+ public void createStsClient(S3CompatibleStorageConfigurationInfo storageConfig) {
+ LOGGER.debug("S3Compatible - createStsClient()");
+ try {
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+ stsBuilder.endpointOverride(URI.create(storageConfig.getS3Endpoint()));
+ if (caI != null && caS != null) {
+ // else default provider build credentials from profile or standard AWS env var
+ stsBuilder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(caI, caS)));
+ }
+ this.stsClient = stsBuilder.build();
+ LOGGER.debug("S3Compatible - stsClient successfully built");
+
+ } catch (Exception e) {
+ System.err.println("S3Compatible - stsClient - build failure : " + e.getMessage());
+ }
+ }
+
+ /*
+ * function from AwsCredentialsStorageIntegration but without roleArn parameter
+ */
+ private IamPolicy policyString(
+ boolean allowList, Set<String> readLocations, Set<String> writeLocations) {
+ IamPolicy.Builder policyBuilder = IamPolicy.builder();
+ IamStatement.Builder allowGetObjectStatementBuilder =
+ IamStatement.builder()
+ .effect(IamEffect.ALLOW)
+ .addAction("s3:GetObject")
+ .addAction("s3:GetObjectVersion");
+ Map<String, IamStatement.Builder> bucketListStatementBuilder = new HashMap<>();
+ Map<String, IamStatement.Builder> bucketGetLocationStatementBuilder = new HashMap<>();
+
+ String arnPrefix = "arn:aws:s3:::";
+ Stream.concat(readLocations.stream(), writeLocations.stream())
+ .distinct()
+ .forEach(
+ location -> {
+ URI uri = URI.create(location);
+ allowGetObjectStatementBuilder.addResource(
+ IamResource.create(
+ arnPrefix + StorageUtil.concatFilePrefixes(parseS3Path(uri), "*", "/")));
+ final var bucket = arnPrefix + StorageUtil.getBucket(uri);
+ if (allowList) {
+ bucketListStatementBuilder
+ .computeIfAbsent(
+ bucket,
+ (String key) ->
+ IamStatement.builder()
+ .effect(IamEffect.ALLOW)
+ .addAction("s3:ListBucket")
+ .addResource(key))
+ .addCondition(
+ IamConditionOperator.STRING_LIKE,
+ "s3:prefix",
+ StorageUtil.concatFilePrefixes(trimLeadingSlash(uri.getPath()), "*", "/"));
+ }
+ bucketGetLocationStatementBuilder.computeIfAbsent(
+ bucket,
+ key ->
+ IamStatement.builder()
+ .effect(IamEffect.ALLOW)
+ .addAction("s3:GetBucketLocation")
+ .addResource(key));
+ });
+
+ if (!writeLocations.isEmpty()) {
+ IamStatement.Builder allowPutObjectStatementBuilder =
+ IamStatement.builder()
+ .effect(IamEffect.ALLOW)
+ .addAction("s3:PutObject")
+ .addAction("s3:DeleteObject");
+ writeLocations.forEach(
+ location -> {
+ URI uri = URI.create(location);
+ allowPutObjectStatementBuilder.addResource(
+ IamResource.create(
+ arnPrefix + StorageUtil.concatFilePrefixes(parseS3Path(uri), "*", "/")));
+ });
+ policyBuilder.addStatement(allowPutObjectStatementBuilder.build());
+ }
+ if (!bucketListStatementBuilder.isEmpty()) {
+ bucketListStatementBuilder
+ .values()
+ .forEach(statementBuilder -> policyBuilder.addStatement(statementBuilder.build()));
+ } else if (allowList) {
+ // add list privilege with 0 resources
+ policyBuilder.addStatement(
+ IamStatement.builder().effect(IamEffect.ALLOW).addAction("s3:ListBucket").build());
+ }
+
+ bucketGetLocationStatementBuilder
+ .values()
+ .forEach(statementBuilder -> policyBuilder.addStatement(statementBuilder.build()));
+ return policyBuilder.addStatement(allowGetObjectStatementBuilder.build()).build();
+ }
+
+ /* function from AwsCredentialsStorageIntegration */
+ private static @Nonnull String parseS3Path(URI uri) {
+ String bucket = StorageUtil.getBucket(uri);
+ String path = trimLeadingSlash(uri.getPath());
+ return String.join("/", bucket, path);
+ }
+
+ /* function from AwsCredentialsStorageIntegration */
+ private static @Nonnull String trimLeadingSlash(String path) {
+ if (path.startsWith("/")) {
+ path = path.substring(1);
+ }
+ return path;
+ }
Review Comment:
Here are options to avoid duplication:
1. Make the method public in the original class(`AwsCredentialsStorageIntegration`), and invoke them here.
2. Move them from `AwsCredentialsStorageIntegration` to a util class like `StorageUtil`
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import jakarta.annotation.Nonnull;
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Stream;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.policybuilder.iam.IamConditionOperator;
+import software.amazon.awssdk.policybuilder.iam.IamEffect;
+import software.amazon.awssdk.policybuilder.iam.IamPolicy;
+import software.amazon.awssdk.policybuilder.iam.IamResource;
+import software.amazon.awssdk.policybuilder.iam.IamStatement;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** S3 compatible implementation of PolarisStorageIntegration */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+ private StsClient stsClient;
+ private String caI, caS; // catalogAccessKeyId & catalogSecretAccessKey
+ private String clI, clS; // clientAccessKeyId & clientSecretAccessKey
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public EnumMap<PolarisCredentialProperty, String> getSubscopedCreds(
+ @NotNull PolarisDiagnostics diagnostics,
+ @NotNull S3CompatibleStorageConfigurationInfo storageConfig,
+ boolean allowListOperation,
+ @NotNull Set<String> allowedReadLocations,
+ @NotNull Set<String> allowedWriteLocations) {
+
+ caI = System.getenv(storageConfig.getS3CredentialsCatalogAccessKeyId());
+ caS = System.getenv(storageConfig.getS3CredentialsCatalogSecretAccessKey());
+
+ EnumMap<PolarisCredentialProperty, String> propertiesMap =
+ new EnumMap<>(PolarisCredentialProperty.class);
+ propertiesMap.put(PolarisCredentialProperty.AWS_ENDPOINT, storageConfig.getS3Endpoint());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_PATH_STYLE_ACCESS,
+ storageConfig.getS3PathStyleAccess().toString());
+ if (storageConfig.getS3Region() != null) {
+ propertiesMap.put(PolarisCredentialProperty.CLIENT_REGION, storageConfig.getS3Region());
+ }
+
+ if (storageConfig.getSkipCredentialSubscopingIndirection() == true) {
+ LOGGER.debug("S3Compatible - skipCredentialSubscopingIndirection !");
+ clI = System.getenv(storageConfig.getS3CredentialsClientAccessKeyId());
+ clS = System.getenv(storageConfig.getS3CredentialsClientSecretAccessKey());
+ if (clI != null && clS != null) {
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, clI);
+ propertiesMap.put(PolarisCredentialProperty.AWS_SECRET_KEY, clS);
+ } else {
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, caI);
+ propertiesMap.put(PolarisCredentialProperty.AWS_SECRET_KEY, caS);
+ }
+ } else {
+ LOGGER.debug("S3Compatible - assumeRole !");
+ createStsClient(storageConfig);
+ AssumeRoleResponse response =
+ stsClient.assumeRole(
+ AssumeRoleRequest.builder()
+ .roleSessionName("PolarisCredentialsSTS")
+ .roleArn(
+ (storageConfig.getS3RoleArn() == null) ? "" : storageConfig.getS3RoleArn())
+ .policy(
+ policyString(allowListOperation, allowedReadLocations, allowedWriteLocations)
+ .toJson())
+ .build());
+
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, response.credentials().accessKeyId());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_SECRET_KEY, response.credentials().secretAccessKey());
+ propertiesMap.put(PolarisCredentialProperty.AWS_TOKEN, response.credentials().sessionToken());
+ LOGGER.debug(
+ "S3Compatible - assumeRole - Token Expiration at : {}",
+ response.credentials().expiration().toString());
+ }
+
+ return propertiesMap;
+ }
+
+ public void createStsClient(S3CompatibleStorageConfigurationInfo storageConfig) {
+ LOGGER.debug("S3Compatible - createStsClient()");
+ try {
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+ stsBuilder.endpointOverride(URI.create(storageConfig.getS3Endpoint()));
+ if (caI != null && caS != null) {
+ // else default provider build credentials from profile or standard AWS env var
+ stsBuilder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(caI, caS)));
+ }
+ this.stsClient = stsBuilder.build();
+ LOGGER.debug("S3Compatible - stsClient successfully built");
+
+ } catch (Exception e) {
+ System.err.println("S3Compatible - stsClient - build failure : " + e.getMessage());
Review Comment:
If creating sts client fails, should we propagate the error to the caller? I believe we should. We can either let the exception bubble up naturally or catch it here and wrap it in a custom runtime exception with a specific, informative error message to provide better context.
##########
spec/polaris-management-service.yml:
##########
@@ -905,6 +907,51 @@ components:
required:
- roleArn
+ S3CompatibleStorageConfigInfo:
+ type: object
+ description: S3 compatible storage configuration info (MinIO, Ceph, Dell ECS, Netapp StorageGRID, ...)
+ allOf:
+ - $ref: '#/components/schemas/StorageConfigInfo'
+ properties:
+ s3.endpoint:
+ type: string
+ description: the S3 endpoint, will also be used as STS endpoint
+ example: "http[s]://host:port"
+ s3.credentials.catalog.accessKeyId:
Review Comment:
Can we use a name like `s3.credentials.catalog.accessKeyEnvVar` to explicitly state it's for an environment variable? Same for others(`s3.credentials.catalog.secretAccessKey`, etc).
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import jakarta.annotation.Nonnull;
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Stream;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.policybuilder.iam.IamConditionOperator;
+import software.amazon.awssdk.policybuilder.iam.IamEffect;
+import software.amazon.awssdk.policybuilder.iam.IamPolicy;
+import software.amazon.awssdk.policybuilder.iam.IamResource;
+import software.amazon.awssdk.policybuilder.iam.IamStatement;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** S3 compatible implementation of PolarisStorageIntegration */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+ private StsClient stsClient;
Review Comment:
Can we move it to the method `getSubscopedCreds()` if a new client is created every time `getSubscopedCreds()` is called?
##########
spec/polaris-management-service.yml:
##########
@@ -905,6 +907,51 @@ components:
required:
- roleArn
+ S3CompatibleStorageConfigInfo:
+ type: object
+ description: S3 compatible storage configuration info (MinIO, Ceph, Dell ECS, Netapp StorageGRID, ...)
+ allOf:
+ - $ref: '#/components/schemas/StorageConfigInfo'
+ properties:
+ s3.endpoint:
+ type: string
+ description: the S3 endpoint, will also be used as STS endpoint
+ example: "http[s]://host:port"
+ s3.credentials.catalog.accessKeyId:
+ type: string
+ description: Default to AWS credentials, otherwise set the environement variable name for the 'ACCESS_KEY_ID' used by the catalog to communicate with S3
+ example: "CATALOG_1_ACCESS_KEY_ENV_VARIABLE_NAME or AWS_ACCESS_KEY_ID"
Review Comment:
Does this means every time a Polaris client creates a new catalog, it has to know env variables in the server side?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1953579662
##########
regtests/minio/Readme.md:
##########
@@ -0,0 +1,42 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# MiniIO Secured
+## Minio and secured buckets with TLS self-signed / custom AC
Review Comment:
Yes. And AC is french for CA. Corrected in Certificate Authority
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1953507226
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import jakarta.annotation.Nonnull;
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Stream;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.policybuilder.iam.IamConditionOperator;
+import software.amazon.awssdk.policybuilder.iam.IamEffect;
+import software.amazon.awssdk.policybuilder.iam.IamPolicy;
+import software.amazon.awssdk.policybuilder.iam.IamResource;
+import software.amazon.awssdk.policybuilder.iam.IamStatement;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** S3 compatible implementation of PolarisStorageIntegration */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+ private StsClient stsClient;
+ private String caI, caS; // catalogAccessKeyId & catalogSecretAccessKey
+ private String clI, clS; // clientAccessKeyId & clientSecretAccessKey
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public EnumMap<PolarisCredentialProperty, String> getSubscopedCreds(
+ @NotNull PolarisDiagnostics diagnostics,
+ @NotNull S3CompatibleStorageConfigurationInfo storageConfig,
+ boolean allowListOperation,
+ @NotNull Set<String> allowedReadLocations,
+ @NotNull Set<String> allowedWriteLocations) {
+
+ caI = System.getenv(storageConfig.getS3CredentialsCatalogAccessKeyId());
+ caS = System.getenv(storageConfig.getS3CredentialsCatalogSecretAccessKey());
+
+ EnumMap<PolarisCredentialProperty, String> propertiesMap =
+ new EnumMap<>(PolarisCredentialProperty.class);
+ propertiesMap.put(PolarisCredentialProperty.AWS_ENDPOINT, storageConfig.getS3Endpoint());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_PATH_STYLE_ACCESS,
+ storageConfig.getS3PathStyleAccess().toString());
+ if (storageConfig.getS3Region() != null) {
+ propertiesMap.put(PolarisCredentialProperty.CLIENT_REGION, storageConfig.getS3Region());
+ }
+
+ if (storageConfig.getSkipCredentialSubscopingIndirection() == true) {
+ LOGGER.debug("S3Compatible - skipCredentialSubscopingIndirection !");
+ clI = System.getenv(storageConfig.getS3CredentialsClientAccessKeyId());
+ clS = System.getenv(storageConfig.getS3CredentialsClientSecretAccessKey());
Review Comment:
No, it is not related
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1953525177
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import jakarta.annotation.Nonnull;
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Stream;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.policybuilder.iam.IamConditionOperator;
+import software.amazon.awssdk.policybuilder.iam.IamEffect;
+import software.amazon.awssdk.policybuilder.iam.IamPolicy;
+import software.amazon.awssdk.policybuilder.iam.IamResource;
+import software.amazon.awssdk.policybuilder.iam.IamStatement;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** S3 compatible implementation of PolarisStorageIntegration */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+ private StsClient stsClient;
+ private String caI, caS; // catalogAccessKeyId & catalogSecretAccessKey
+ private String clI, clS; // clientAccessKeyId & clientSecretAccessKey
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public EnumMap<PolarisCredentialProperty, String> getSubscopedCreds(
+ @NotNull PolarisDiagnostics diagnostics,
+ @NotNull S3CompatibleStorageConfigurationInfo storageConfig,
+ boolean allowListOperation,
+ @NotNull Set<String> allowedReadLocations,
+ @NotNull Set<String> allowedWriteLocations) {
+
+ caI = System.getenv(storageConfig.getS3CredentialsCatalogAccessKeyId());
+ caS = System.getenv(storageConfig.getS3CredentialsCatalogSecretAccessKey());
+
+ EnumMap<PolarisCredentialProperty, String> propertiesMap =
+ new EnumMap<>(PolarisCredentialProperty.class);
+ propertiesMap.put(PolarisCredentialProperty.AWS_ENDPOINT, storageConfig.getS3Endpoint());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_PATH_STYLE_ACCESS,
+ storageConfig.getS3PathStyleAccess().toString());
+ if (storageConfig.getS3Region() != null) {
+ propertiesMap.put(PolarisCredentialProperty.CLIENT_REGION, storageConfig.getS3Region());
+ }
+
+ if (storageConfig.getSkipCredentialSubscopingIndirection() == true) {
+ LOGGER.debug("S3Compatible - skipCredentialSubscopingIndirection !");
+ clI = System.getenv(storageConfig.getS3CredentialsClientAccessKeyId());
+ clS = System.getenv(storageConfig.getS3CredentialsClientSecretAccessKey());
+ if (clI != null && clS != null) {
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, clI);
+ propertiesMap.put(PolarisCredentialProperty.AWS_SECRET_KEY, clS);
+ } else {
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, caI);
+ propertiesMap.put(PolarisCredentialProperty.AWS_SECRET_KEY, caS);
+ }
+ } else {
+ LOGGER.debug("S3Compatible - assumeRole !");
+ createStsClient(storageConfig);
+ AssumeRoleResponse response =
+ stsClient.assumeRole(
+ AssumeRoleRequest.builder()
+ .roleSessionName("PolarisCredentialsSTS")
+ .roleArn(
+ (storageConfig.getS3RoleArn() == null) ? "" : storageConfig.getS3RoleArn())
+ .policy(
+ policyString(allowListOperation, allowedReadLocations, allowedWriteLocations)
+ .toJson())
+ .build());
+
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, response.credentials().accessKeyId());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_SECRET_KEY, response.credentials().secretAccessKey());
+ propertiesMap.put(PolarisCredentialProperty.AWS_TOKEN, response.credentials().sessionToken());
+ LOGGER.debug(
+ "S3Compatible - assumeRole - Token Expiration at : {}",
+ response.credentials().expiration().toString());
+ }
+
+ return propertiesMap;
+ }
+
+ public void createStsClient(S3CompatibleStorageConfigurationInfo storageConfig) {
+ LOGGER.debug("S3Compatible - createStsClient()");
+ try {
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+ stsBuilder.endpointOverride(URI.create(storageConfig.getS3Endpoint()));
Review Comment:
Almost yes.
If catalog creator do not want to rely on default and global Polaris service keys and want dedicated keys for each catalog, adding credential alternative system in this in factory could be useful. No ?
- Security pattern like not putting all eggs in one basket
- Also onPrem S3 buckets can been managed with security pattern where administrators are generating different keys for each s3 user associated with some buckets + assumeRole policy (I have this case in my company, considered as secutity segreggation pattern). In this case Polaris with central and unique keys need to be over instanciated :-( Managed at catalog level it allow to keep one Polaris service and many catalogs
```
//not using provider built credentials from standard AWS env var
stsBuilder.credentialsProvider(
StaticCredentialsProvider.create(AwsBasicCredentials.create(caI, caS)));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1953585821
##########
spec/polaris-management-service.yml:
##########
@@ -905,6 +907,51 @@ components:
required:
- roleArn
+ S3CompatibleStorageConfigInfo:
+ type: object
+ description: S3 compatible storage configuration info (MinIO, Ceph, Dell ECS, Netapp StorageGRID, ...)
+ allOf:
+ - $ref: '#/components/schemas/StorageConfigInfo'
+ properties:
+ s3.endpoint:
+ type: string
+ description: the S3 endpoint, will also be used as STS endpoint
Review Comment:
I confirm. I've written it mostly because it is not well or explicitly documented in s3 softs documentation.
I will remove it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2537372134
> Hello everyone this PR seems to be blocked for a month now, is there anything we can do to make it to the end ? 🙏
Sorry for the one-month break. I tried the approaches proposed in the comments.
Refactoring performed ;)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1881134556
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.s3compatible.S3CompatibleStorageConfigurationInfo.CredsCatalogAndClientStrategyEnum;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** Credential vendor that supports generating */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+ private StsClient stsClient;
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ public void createStsClient(S3CompatibleStorageConfigurationInfo s3storageConfig) {
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+ stsBuilder.region(
+ Region
+ .US_WEST_1); // default region to avoid bug, because most (all?) S3 compatible softwares
+ // do not care about regions
+ stsBuilder.endpointOverride(
+ URI.create(
+ s3storageConfig
+ .getS3Endpoint())); // S3Compatible - AWS STS endpoint is unique and different from
+ // the S3 Endpoint
+ stsBuilder.credentialsProvider(
+ StaticCredentialsProvider.create(
+ AwsBasicCredentials.create(
+ s3storageConfig.getS3CredentialsCatalogAccessKeyId(),
+ s3storageConfig.getS3CredentialsCatalogSecretAccessKey())));
+ this.stsClient = stsBuilder.build();
+ LOGGER.debug("S3Compatible - stsClient successfully built");
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public EnumMap<PolarisCredentialProperty, String> getSubscopedCreds(
+ @NotNull PolarisDiagnostics diagnostics,
+ @NotNull S3CompatibleStorageConfigurationInfo storageConfig,
+ boolean allowListOperation,
+ @NotNull Set<String> allowedReadLocations,
+ @NotNull Set<String> allowedWriteLocations) {
+
+ LOGGER.debug("S3Compatible - getSubscopedCreds - applying credential strategy");
+ EnumMap<PolarisCredentialProperty, String> propertiesMap =
+ new EnumMap<>(PolarisCredentialProperty.class);
+ propertiesMap.put(PolarisCredentialProperty.AWS_ENDPOINT, storageConfig.getS3Endpoint());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_PATH_STYLE_ACCESS,
+ storageConfig.getS3PathStyleAccess().toString());
Review Comment:
failed to use catalog porperties, they are not forwarded to this class
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1828954734
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.s3compatible.S3CompatibleStorageConfigurationInfo.CredsCatalogAndClientStrategyEnum;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** Credential vendor that supports generating */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+ private StsClient stsClient;
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ public void createStsClient(S3CompatibleStorageConfigurationInfo s3storageConfig) {
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+ stsBuilder.region(
+ Region
+ .US_WEST_1); // default region to avoid bug, because most (all?) S3 compatible softwares
+ // do not care about regions
Review Comment:
nit: can we fix this comment?
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.s3compatible.S3CompatibleStorageConfigurationInfo.CredsCatalogAndClientStrategyEnum;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** Credential vendor that supports generating */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+ private StsClient stsClient;
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ public void createStsClient(S3CompatibleStorageConfigurationInfo s3storageConfig) {
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+ stsBuilder.region(
+ Region
+ .US_WEST_1); // default region to avoid bug, because most (all?) S3 compatible softwares
+ // do not care about regions
+ stsBuilder.endpointOverride(
+ URI.create(
+ s3storageConfig
+ .getS3Endpoint())); // S3Compatible - AWS STS endpoint is unique and different from
+ // the S3 Endpoint
Review Comment:
nit: ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1830237327
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.s3compatible.S3CompatibleStorageConfigurationInfo.CredsCatalogAndClientStrategyEnum;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** Credential vendor that supports generating */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+ private StsClient stsClient;
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ public void createStsClient(S3CompatibleStorageConfigurationInfo s3storageConfig) {
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
Review Comment:
Like AWS class ?
Find it weird to put something related to a storage type outside the class and provided by the constructor. Seems GCP and Azure classes are keeping it inside the class too.
No ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1811169674
##########
spec/polaris-management-service.yml:
##########
@@ -901,6 +903,58 @@ components:
required:
- roleArn
+ S3StorageConfigInfo:
+ type: object
+ description: S3 compatible storage configuration info (MinIO, Dell ECS, Netapp StorageGRID, ...)
+ allOf:
+ - $ref: '#/components/schemas/StorageConfigInfo'
+ properties:
+ credsCatalogAndClientStrategy:
+ type: string
+ enum:
+ - VALUE
+ - ENV_VAR_NAME
+ default: ENV_VAR_NAME
+ example: "ACCESS_KEY"
+ description: When you send key VALUE directly via this command, they should apear in logs. By ENV_VAR_NAME without dollar, only a reference will appear in logs, but the value have to be available as environnement variable in the context where Polaris is running
+ credsVendingStrategy:
+ type: string
+ enum:
+ - TOKEN_WITH_ASSUME_ROLE
+ - KEYS_SAME_AS_CATALOG
+ - KEYS_DEDICATED_TO_CLIENT
+ default: TOKEN_WITH_ASSUME_ROLE
+ description: The catalog strategy to vend credentials to client. Options possible are same keys than catalog, keys dedicated to clients, or Tokens with STS methods 'assumeRole' for Dell ECS or NetApp StorageGrid solution, 'truc' for MinIo solution)
Review Comment:
`truc`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1811169014
##########
spec/polaris-management-service.yml:
##########
@@ -901,6 +903,58 @@ components:
required:
- roleArn
+ S3StorageConfigInfo:
Review Comment:
Is this specific to s3compat, or is it also meant to be used for s3 itself?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1813209421
##########
spec/polaris-management-service.yml:
##########
@@ -901,6 +903,58 @@ components:
required:
- roleArn
+ S3StorageConfigInfo:
+ type: object
+ description: S3 compatible storage configuration info (MinIO, Dell ECS, Netapp StorageGRID, ...)
+ allOf:
+ - $ref: '#/components/schemas/StorageConfigInfo'
+ properties:
+ credsCatalogAndClientStrategy:
+ type: string
+ enum:
+ - VALUE
+ - ENV_VAR_NAME
+ default: ENV_VAR_NAME
+ example: "ACCESS_KEY"
+ description: When you send key VALUE directly via this command, they should apear in logs. By ENV_VAR_NAME without dollar, only a reference will appear in logs, but the value have to be available as environnement variable in the context where Polaris is running
+ credsVendingStrategy:
+ type: string
+ enum:
+ - TOKEN_WITH_ASSUME_ROLE
+ - KEYS_SAME_AS_CATALOG
+ - KEYS_DEDICATED_TO_CLIENT
Review Comment:
Here client is anything trying to obtain keys (or security token service) from this catalog (spark, trino,...).
This is not the right term to use in the context of Polaris?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1813159130
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3/S3CredentialsStorageIntegration.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3;
+
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** Credential vendor that supports generating */
+public class S3CredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3StorageConfigurationInfo> {
+
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CredentialsStorageIntegration.class);
+
+ private StsClient stsClient;
+
+ // Constructor
+ public S3CredentialsStorageIntegration() {
+ super(S3CredentialsStorageIntegration.class.getName());
+ }
+
+ public void createStsClient(S3StorageConfigurationInfo s3storageConfig) {
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+
+ LOGGER.info(
+ "S3Compatible - AWS STS endpoint is unique and different from the S3 Endpoint. AWS SDK need to be overided with dedicated Endpoint from S3Compatible, otherwise the AWS STS url is targeted");
+
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+
+ stsBuilder.region(
+ Region
+ .US_WEST_1); // default region to avoid bug, because most (all?) S3 compatible softwares
Review Comment:
Correct.
It could potentially be used with any S3, AWS included (in this implementation I removed the cross region tweak implemented in the AWS FileIOClientFactory and overide the STS endpoint).
The name is maybe a bad compromise, I agree to change for a better one.
At the beginning I have named it s3compatible but at the end simply s3.
This choice is the result of this two constatations :
- In core/storage folder the classes are named aws, gcp and azure
- But in Enum Storage Type AWS is directly named S3....
So I found in enum this S3_COMPATIBILITY compromise. MinIO use this term...
For the class I shortened it to S3 because S3Compatible was verbose.
I can come back to S3compatible everywhere to be consistent...
If Enum Storage was different and aws implementation named also AWS like GCP & Azure.... but it is not. Modifying the existing values of the Enum can be a breaking change, I do not want to introduce this change in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "Gerrit-K (via GitHub)" <gi...@apache.org>.
Gerrit-K commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2589277031
> I have tested this with Ceph and it is working properly. Very cool!
@mmgaggle Nice! Have you also tested the STS path (i.e. with `skipCredentialSubscopingIndirection: false`)? I've also done some testing against ceph (deployed using rook), but haven't been able to test this particular path due to the way our ceph is currently configured. I would be interested to know if this works. Thanks in advance!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "mmgaggle (via GitHub)" <gi...@apache.org>.
mmgaggle commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2597353722
> > I have tested this with Ceph and it is working properly. Very cool!
>
>
>
> @mmgaggle Nice! Have you also tested the STS path (i.e. with `skipCredentialSubscopingIndirection: false`)? I've also done some testing against ceph (deployed using rook), but haven't been able to test this particular path due to the way our ceph is currently configured. I would be interested to know if this works. Thanks in advance!
Yes, I can give it a go tomorrow and report back. This is the mode where it vends a static client credential?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1949671835
##########
spec/polaris-management-service.yml:
##########
@@ -877,6 +878,7 @@ components:
propertyName: storageType
mapping:
S3: "#/components/schemas/AwsStorageConfigInfo"
+ S3_COMPATIBLE: "#/components/schemas/S3CompatibleStorageConfigInfo"
Review Comment:
Do we need to make accordingly change to Polaris cli? cc @eric-maynard. I’m OK to do that in a follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "ashokvengala1990 (via GitHub)" <gi...@apache.org>.
ashokvengala1990 commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2842173863
@lefebsy
We are waiting for this PR to be merged. One of our customers wants to use MinIO or Dell ECS storage for Polaris. Please let me know if I can contribute to this if it is taking longer than expected. Looking forward to your response. Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "mmgaggle (via GitHub)" <gi...@apache.org>.
mmgaggle commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2699232825
Thank you for working on this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2708621197
> Is it ok for you if I do UT only for config class and move to dockerized tests for credentials class ? I am really not good in the mock code area.
>
> I can easily add minio to regtests docker-compose and enrich "regtests/t_spark_sql/"
>
> My first try just learned me not to name script with "S3" inside otherwise it's trapped by AWS stop pattern and never run.
I'm OK with a followup PR to resolve the mock related unit tests. We can get some help from the community. Can you file an issue to track that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1987787270
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleStorageConfigurationInfo.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import jakarta.annotation.Nonnull;
+import java.util.List;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * S3-Compatible Storage Configuration. This class holds the parameters needed to connect to
+ * S3-compatible storage services such as MinIO, Ceph, Dell ECS, etc.
+ */
+public class S3CompatibleStorageConfigurationInfo extends PolarisStorageConfigurationInfo {
+
+ // 5 is the approximate max allowed locations for the size of AccessPolicy when LIST is required
+ // for allowed read and write locations for sub-scoping credentials.
+ @JsonIgnore private static final int MAX_ALLOWED_LOCATIONS = 5;
Review Comment:
We don't have to wait for #1068. #1068 is required only because the limits are there already, removing them potentially impact existing users. So we are safe to remove the limit here in the first place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1987779379
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleStorageConfigurationInfo.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import jakarta.annotation.Nonnull;
+import java.util.List;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * S3-Compatible Storage Configuration. This class holds the parameters needed to connect to
+ * S3-compatible storage services such as MinIO, Ceph, Dell ECS, etc.
+ */
+public class S3CompatibleStorageConfigurationInfo extends PolarisStorageConfigurationInfo {
+
+ // 5 is the approximate max allowed locations for the size of AccessPolicy when LIST is required
+ // for allowed read and write locations for sub-scoping credentials.
+ @JsonIgnore private static final int MAX_ALLOWED_LOCATIONS = 5;
Review Comment:
#1068 and #1124 seems in progress, so I've just pushed the rest of the modifications requested without waiting for a futur rebase on #1068
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2729376750
@lefebsy : I appreciate your work on this PR, but I'd like to submit an alternative implementation for consideration by the community. Will mention it here, when it's ready.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "omarsmak (via GitHub)" <gi...@apache.org>.
omarsmak commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2715653246
> > Thanks a lot @lefebsy for working on it. I agreed that we don't have to put Helm change in, so that the github pipeline can pass. We can always add it back as a followup PR. It should be ready to merge as long as we fix this https://github.com/apache/polaris/pull/389/files#r1980741611.
>
> Yes. I have restored helm from main branch. I have probably done a small test modification and forget it. I'm really confused.
Thanks @lefebsy . The PR looks now a lot better from helm standpoint. We just need to address last comment from @flyrain to get this PR going 🙂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2719167733
@omarsmak
=> Hello, the conflict should be resolved since rebased :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "collado-mike (via GitHub)" <gi...@apache.org>.
collado-mike commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2654801231
> Is there a way to define a storage endpoint that is used for S3 and STS per catalog without this?
We do this in one of the integ tests that spins up a mock S3 container - see https://github.com/apache/polaris/blob/main/integration-tests/src/main/java/org/apache/polaris/service/it/test/PolarisSparkIntegrationTest.java#L116-L134
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1953525177
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import jakarta.annotation.Nonnull;
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Stream;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.policybuilder.iam.IamConditionOperator;
+import software.amazon.awssdk.policybuilder.iam.IamEffect;
+import software.amazon.awssdk.policybuilder.iam.IamPolicy;
+import software.amazon.awssdk.policybuilder.iam.IamResource;
+import software.amazon.awssdk.policybuilder.iam.IamStatement;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** S3 compatible implementation of PolarisStorageIntegration */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+ private StsClient stsClient;
+ private String caI, caS; // catalogAccessKeyId & catalogSecretAccessKey
+ private String clI, clS; // clientAccessKeyId & clientSecretAccessKey
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public EnumMap<PolarisCredentialProperty, String> getSubscopedCreds(
+ @NotNull PolarisDiagnostics diagnostics,
+ @NotNull S3CompatibleStorageConfigurationInfo storageConfig,
+ boolean allowListOperation,
+ @NotNull Set<String> allowedReadLocations,
+ @NotNull Set<String> allowedWriteLocations) {
+
+ caI = System.getenv(storageConfig.getS3CredentialsCatalogAccessKeyId());
+ caS = System.getenv(storageConfig.getS3CredentialsCatalogSecretAccessKey());
+
+ EnumMap<PolarisCredentialProperty, String> propertiesMap =
+ new EnumMap<>(PolarisCredentialProperty.class);
+ propertiesMap.put(PolarisCredentialProperty.AWS_ENDPOINT, storageConfig.getS3Endpoint());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_PATH_STYLE_ACCESS,
+ storageConfig.getS3PathStyleAccess().toString());
+ if (storageConfig.getS3Region() != null) {
+ propertiesMap.put(PolarisCredentialProperty.CLIENT_REGION, storageConfig.getS3Region());
+ }
+
+ if (storageConfig.getSkipCredentialSubscopingIndirection() == true) {
+ LOGGER.debug("S3Compatible - skipCredentialSubscopingIndirection !");
+ clI = System.getenv(storageConfig.getS3CredentialsClientAccessKeyId());
+ clS = System.getenv(storageConfig.getS3CredentialsClientSecretAccessKey());
+ if (clI != null && clS != null) {
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, clI);
+ propertiesMap.put(PolarisCredentialProperty.AWS_SECRET_KEY, clS);
+ } else {
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, caI);
+ propertiesMap.put(PolarisCredentialProperty.AWS_SECRET_KEY, caS);
+ }
+ } else {
+ LOGGER.debug("S3Compatible - assumeRole !");
+ createStsClient(storageConfig);
+ AssumeRoleResponse response =
+ stsClient.assumeRole(
+ AssumeRoleRequest.builder()
+ .roleSessionName("PolarisCredentialsSTS")
+ .roleArn(
+ (storageConfig.getS3RoleArn() == null) ? "" : storageConfig.getS3RoleArn())
+ .policy(
+ policyString(allowListOperation, allowedReadLocations, allowedWriteLocations)
+ .toJson())
+ .build());
+
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, response.credentials().accessKeyId());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_SECRET_KEY, response.credentials().secretAccessKey());
+ propertiesMap.put(PolarisCredentialProperty.AWS_TOKEN, response.credentials().sessionToken());
+ LOGGER.debug(
+ "S3Compatible - assumeRole - Token Expiration at : {}",
+ response.credentials().expiration().toString());
+ }
+
+ return propertiesMap;
+ }
+
+ public void createStsClient(S3CompatibleStorageConfigurationInfo storageConfig) {
+ LOGGER.debug("S3Compatible - createStsClient()");
+ try {
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+ stsBuilder.endpointOverride(URI.create(storageConfig.getS3Endpoint()));
Review Comment:
Almost yes.
If catalog creator do not want to rely on default and global Polaris service keys and want dedicated keys for each catalog, adding this in factory could be useful. No ?
- Security pattern like not putting all eggs in one basket
- Also onPrem S3 buckets can been managed with security pattern where administrators are generatings different keys for each s3 user associated with some buckets (I have this case in my company, considered as secutity segreggation pattern). In this case Polaris with central and unique keys need to be over instanciated :-( Managed at catalog level it allow to keep one Polaris service and many catalogs
```
//not using provider built credentials from profile or standard AWS env var
stsBuilder.credentialsProvider(
StaticCredentialsProvider.create(AwsBasicCredentials.create(caI, caS)));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1954008620
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/PolarisStorageConfigurationInfo.java:
##########
@@ -241,6 +243,7 @@ public void validateMaxAllowedLocations(int maxAllowedLocations) {
/** Polaris' storage type, each has a fixed prefix for its location */
public enum StorageType {
S3("s3://"),
+ S3_COMPATIBLE("s3://"),
Review Comment:
It could be added in the list, but do not know how to check with unit test if it is working.
Wich type of *hadoop client* can be considered for test ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1813159130
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3/S3CredentialsStorageIntegration.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3;
+
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** Credential vendor that supports generating */
+public class S3CredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3StorageConfigurationInfo> {
+
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CredentialsStorageIntegration.class);
+
+ private StsClient stsClient;
+
+ // Constructor
+ public S3CredentialsStorageIntegration() {
+ super(S3CredentialsStorageIntegration.class.getName());
+ }
+
+ public void createStsClient(S3StorageConfigurationInfo s3storageConfig) {
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+
+ LOGGER.info(
+ "S3Compatible - AWS STS endpoint is unique and different from the S3 Endpoint. AWS SDK need to be overided with dedicated Endpoint from S3Compatible, otherwise the AWS STS url is targeted");
+
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+
+ stsBuilder.region(
+ Region
+ .US_WEST_1); // default region to avoid bug, because most (all?) S3 compatible softwares
Review Comment:
Correct the name give this idea.
The name is maybe a bad compromise, I agree to change for a better one.
At the beginning I have named it s3compatible but at the end simply s3.
This choice is the result of this two constatations :
- In core/storage folder the classes are named aws, gcp and azure
- But in Enum Storage Type AWS is directly named S3....
So I found in enum this S3_COMPATIBILITY compromise. MinIO use this term...
For the class I shortened it to S3 because S3Compatible was verbose.
I can come back to S3compatible everywhere to be consistent...
If Enum Storage was different and aws implementation named also AWS like GCP & Azure.... but it is not. Modifying the existing values of the Enum can be a breaking change, I do not want to introduce this change in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1813201796
##########
spec/polaris-management-service.yml:
##########
@@ -901,6 +903,58 @@ components:
required:
- roleArn
+ S3StorageConfigInfo:
+ type: object
+ description: S3 compatible storage configuration info (MinIO, Dell ECS, Netapp StorageGRID, ...)
+ allOf:
+ - $ref: '#/components/schemas/StorageConfigInfo'
+ properties:
+ credsCatalogAndClientStrategy:
+ type: string
+ enum:
+ - VALUE
+ - ENV_VAR_NAME
+ default: ENV_VAR_NAME
+ example: "ACCESS_KEY"
+ description: When you send key VALUE directly via this command, they should apear in logs. By ENV_VAR_NAME without dollar, only a reference will appear in logs, but the value have to be available as environnement variable in the context where Polaris is running
+ credsVendingStrategy:
+ type: string
+ enum:
+ - TOKEN_WITH_ASSUME_ROLE
+ - KEYS_SAME_AS_CATALOG
Review Comment:
Exactly, explained like that it's much clearer.
I was trying also to do a distinction with the third option where it is possible to serve another pair of keys, not the one used by the catalog. In this last case, the keys served to the 'client' can be revoke by S3 admin without touching the keys used internaly by this catalog.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1813159130
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3/S3CredentialsStorageIntegration.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3;
+
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** Credential vendor that supports generating */
+public class S3CredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3StorageConfigurationInfo> {
+
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CredentialsStorageIntegration.class);
+
+ private StsClient stsClient;
+
+ // Constructor
+ public S3CredentialsStorageIntegration() {
+ super(S3CredentialsStorageIntegration.class.getName());
+ }
+
+ public void createStsClient(S3StorageConfigurationInfo s3storageConfig) {
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+
+ LOGGER.info(
+ "S3Compatible - AWS STS endpoint is unique and different from the S3 Endpoint. AWS SDK need to be overided with dedicated Endpoint from S3Compatible, otherwise the AWS STS url is targeted");
+
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+
+ stsBuilder.region(
+ Region
+ .US_WEST_1); // default region to avoid bug, because most (all?) S3 compatible softwares
Review Comment:
Correct. It could logically be used with any S3, AWS included.
The name is maybe a bad compromise, I agree to change for a better one.
At the beginning I have named it s3compat.... but at the end simply s3.
This choice is the result of this two constatations :
- In core/storage folder the classes are named aws, gcp and azure
- But in Enum Storage Type AWS is directly named S3....
So I found in enum this S3_COMPATIBILITY compromise. MinIO use it...
For the class I shortened it to S3 because it could works with AWS too.
I can come back to S3compat everywhere...
If Enum Storage was different and aws implementation named also AWS like GCP & Azure.... but it is not. Modify the existing values of the Enum can be a breaking change, I do not want to introduce this change in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "sfc-gh-schen (via GitHub)" <gi...@apache.org>.
sfc-gh-schen commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1876785065
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.s3compatible.S3CompatibleStorageConfigurationInfo.CredsCatalogAndClientStrategyEnum;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** Credential vendor that supports generating */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+ private StsClient stsClient;
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ public void createStsClient(S3CompatibleStorageConfigurationInfo s3storageConfig) {
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+ stsBuilder.region(
+ Region
+ .US_WEST_1); // default region to avoid bug, because most (all?) S3 compatible softwares
+ // do not care about regions
+ stsBuilder.endpointOverride(
+ URI.create(
+ s3storageConfig
+ .getS3Endpoint())); // S3Compatible - AWS STS endpoint is unique and different from
Review Comment:
The sts endpoint is the same as its customized s3 endpoint? No difference like <endpoint>/sts or s3storageConfig.getStsEndpoint()?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2797993777
@thotz, I'm also waiting for the original author to make the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "richban (via GitHub)" <gi...@apache.org>.
richban commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2833750594
@lefebsy Would love this PR to be merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "ryanovas (via GitHub)" <gi...@apache.org>.
ryanovas commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2843102251
@eric-maynard I am one of those users, I would be incredibly disappointed to see this PR be lost. I have been following it for weeks if not months at this point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2781233548
> There are unaddressed concerns with this PR (as commented on specific changes). I believe they need to be resolved before merging.
Agreed. I meant merge it after comments are addressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2655229076
> Thanks a lot, @lefebsy, for adding this! I believe this is a really useful feature—appreciate your effort. Apologies for the delay.
>
> I agree with @collado-mike that we should avoid vending raw credentials.
You're all welcome.
I can uderstand the motivation behind not vending raw credentials. Even if I will miss this in my personal company context.
I've prepared the quarkus rebase, will add any possible suggestions of this review and of course keep only STS.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1953561349
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleStorageConfigurationInfo.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import java.util.List;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Polaris Storage Configuration information for an S3 Compatible solution, MinIO, Ceph, Dell ECS...
+ */
+public class S3CompatibleStorageConfigurationInfo extends PolarisStorageConfigurationInfo {
+
+ // 5 is the approximate max allowed locations for the size of AccessPolicy when LIST is required
+ // for allowed read and write locations for subscoping creds.
+ @JsonIgnore private static final int MAX_ALLOWED_LOCATIONS = 5;
+ private @NotNull String s3Endpoint;
+ private @Nullable String s3CredentialsCatalogAccessKeyId;
+ private @Nullable String s3CredentialsCatalogSecretAccessKey;
+ private @Nullable Boolean s3PathStyleAccess;
+ private @NotNull Boolean skipCredentialSubscopingIndirection;
+ private @Nullable String s3CredentialsClientAccessKeyId;
+ private @Nullable String s3CredentialsClientSecretAccessKey;
+ private @Nullable String s3Region;
+ private @Nullable String s3RoleArn;
+
+ // Constructor
+ @JsonCreator
+ public S3CompatibleStorageConfigurationInfo(
+ @JsonProperty(value = "storageType", required = true) @NotNull StorageType storageType,
+ @JsonProperty(value = "s3Endpoint", required = true) @NotNull String s3Endpoint,
+ @JsonProperty(value = "s3CredentialsCatalogAccessKeyId", required = true) @NotNull
+ String s3CredentialsCatalogAccessKeyId,
+ @JsonProperty(value = "s3CredentialsCatalogSecretAccessKey", required = true) @NotNull
+ String s3CredentialsCatalogSecretAccessKey,
+ @JsonProperty(value = "SkipCredentialSubscopingIndirection", required = false) @Nullable
+ Boolean skipCredentialSubscopingIndirection,
+ @JsonProperty(value = "s3CredentialsClientAccessKeyId", required = false) @Nullable
+ String s3CredentialsClientAccessKeyId,
+ @JsonProperty(value = "s3CredentialsClientSecretAccessKey", required = false) @Nullable
+ String s3CredentialsClientSecretAccessKey,
+ @JsonProperty(value = "s3PathStyleAccess", required = false) @Nullable
+ Boolean s3PathStyleAccess,
+ @JsonProperty(value = "s3Region", required = false) @Nullable String s3Region,
+ @JsonProperty(value = "s3RoleArn", required = false) @Nullable String s3RoleArn,
+ @JsonProperty(value = "allowedLocations", required = true) @Nullable
+ List<String> allowedLocations) {
+
+ // storing properties
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1953597985
##########
spec/polaris-management-service.yml:
##########
@@ -905,6 +907,51 @@ components:
required:
- roleArn
+ S3CompatibleStorageConfigInfo:
+ type: object
+ description: S3 compatible storage configuration info (MinIO, Ceph, Dell ECS, Netapp StorageGRID, ...)
+ allOf:
+ - $ref: '#/components/schemas/StorageConfigInfo'
+ properties:
+ s3.endpoint:
+ type: string
+ description: the S3 endpoint, will also be used as STS endpoint
+ example: "http[s]://host:port"
+ s3.credentials.catalog.accessKeyId:
+ type: string
+ description: Default to AWS credentials, otherwise set the environement variable name for the 'ACCESS_KEY_ID' used by the catalog to communicate with S3
+ example: "CATALOG_1_ACCESS_KEY_ENV_VARIABLE_NAME or AWS_ACCESS_KEY_ID"
Review Comment:
Yes if catalog creator want to use specific keys he have to be aware of the name of the variables inside Polaris service environnement.
Otherwise the creator let the key parameters empty, "cross fingers", and let default s3 sdk use whatever is available in polaris service environement.
I have not find a better solution. Passing values will leak secret in logs and in catalog "get response" :-(
Upper in this code review Collado-Mike suggest to switch to AWS Profile. But the catalog creator still have to be aware of the name of the profile...
=> Need agreement between Polaris service deployment administrator and catalog creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2546185853
Refactored after many comments :
- SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION is replacing the initial 'Strategy options' as requested by @eric-maynard
- Come back of RoleARN as optional parameter, because requested in the issue #32
- Added 'Region for client' to be aligned with last AWS implementation modifications
- Removed the credentials parameters 'by value', only env var name are possible. If parameters are empty, it will fallback to default AWS variables
- @collado-mike comments
- about 'at least a session token' :
seems to be not available in the S3 compatible softwares I've tested, or I missed the information - only AssumeRole (with empty role or not) is often implemented
- about the SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION should be in the catalog properties and not in the storage porperties :
Polaris is not forwarding the catalog properties where it is needed. Maybe it can be adressed later in other PR ?
- STS client created inside the core/storage classes, not pass via constructor :
Looks similar to the Azure implementation, not the AWS one. I have avoid to modify the 'Service' code in this PR, modifications are limited to REST specs, core Storage and regTest...
Thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2699276840
Hello,
Last refactoring vending only STS.
- Support of **profile** to manage credentials used by catalog to communicate with S3
- Refactor the **duplicated functions** used by 'Aws' and also by 'S3Compatible'. They have been moved to _StorageUtil_, here they should be imported and used by 'S3Compatible' or either by 'Aws' if this place is adopted.
```
curl -X POST -H "Authorization: Bearer ${SPARK_BEARER_TOKEN}" \
-H 'Accept: application/json' -H 'Content-Type: application/json' \
http://${POLARIS_HOST}:8181/api/management/v1/catalogs -d \
'{
"name": "my-s3compatible-catalog-1",
"id": 100,
"type": "INTERNAL",
"readOnly": false,
"properties": {
"default-base-location": "${S3_LOCATION}"
},
"storageConfigInfo": {
"storageType": "S3_COMPATIBLE",
"allowedLocations": ["${S3_LOCATION}/"],
"s3.endpoint": "https://localhost:9000"
}
}'
```
As is AWS SDK will use all default values and settings available in Polaris service, to build catalog communication to S3 endpoint
Otherwise indications can be given
```
# optional - 'Indicate a AWS Profile name'
"s3.profileName": "minio-catalog-1",
```
or
```
# optional - 'Indicate Env variable name'
"s3.credentials.catalog.accessKeyEnvVar": "CATALOG_S3_KEY_ID_FOR_CATALOG_1",
"s3.credentials.catalog.secretAccessKeyEnvVar": "CATALOG_S3_KEY_SECRET_FOR_CATALOG_1",
```
and if helpful
```
# optional
"s3.region": "region-1",
"s3.pathStyleAccess": true,
"s3.roleArn": "arn:xxx:xxx:xxx:xxx:xxx"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1987840778
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleStorageConfigurationInfo.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import jakarta.annotation.Nonnull;
+import java.util.List;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * S3-Compatible Storage Configuration. This class holds the parameters needed to connect to
+ * S3-compatible storage services such as MinIO, Ceph, Dell ECS, etc.
+ */
+public class S3CompatibleStorageConfigurationInfo extends PolarisStorageConfigurationInfo {
+
+ // 5 is the approximate max allowed locations for the size of AccessPolicy when LIST is required
+ // for allowed read and write locations for sub-scoping credentials.
+ @JsonIgnore private static final int MAX_ALLOWED_LOCATIONS = 5;
Review Comment:
ok fine !
removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1989398173
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/PolarisStorageConfigurationInfo.java:
##########
@@ -62,6 +63,7 @@
@JsonTypeInfo(use = JsonTypeInfo.Id.NAME)
@JsonSubTypes({
@JsonSubTypes.Type(value = AwsStorageConfigurationInfo.class),
+ @JsonSubTypes.Type(value = S3CompatibleStorageConfigurationInfo.class),
Review Comment:
The question of modifying the current implementation arose at the beginning of the PR. The direction given was to propose a variant in order not to disrupt the existing. Possibly in a second step, unify.
So this PR follows the instructions and avoids modifying the existing and proposes a variant adapted to the S3 alternatives.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1989454133
##########
polaris-core/src/main/java/org/apache/polaris/core/entity/CatalogEntity.java:
##########
@@ -250,6 +268,21 @@ public Builder setStorageConfigurationInfo(
awsConfig.validateArn(awsConfigModel.getRoleArn());
config = awsConfig;
break;
+
+ case S3_COMPATIBLE:
Review Comment:
Correct.
To complete the response given to @dimas-b few a few lines higher :
Before the security arbitrage favoring vending only STS there were more options to be used by onPrem ecosystem habits (use of keys that do not expire quickly).
Since then, the difference between the AWS class and the S3compatible class is smaller.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1989454133
##########
polaris-core/src/main/java/org/apache/polaris/core/entity/CatalogEntity.java:
##########
@@ -250,6 +268,21 @@ public Builder setStorageConfigurationInfo(
awsConfig.validateArn(awsConfigModel.getRoleArn());
config = awsConfig;
break;
+
+ case S3_COMPATIBLE:
Review Comment:
Correct.
To complete the response given to @dimas-b a few lines higher :
Before the security arbitrage favoring vending only STS there were more options to be used by onPrem ecosystems habits (keys that do not expire quickly by exemple).
Since then, the difference between the AWS class and the S3compatible class is smaller.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2458490912
> I guess I don't understand what isn't supported today with catalog-properties. E.g., in https://github.com/apache/polaris/blob/main/polaris-service/src/test/java/org/apache/polaris/service/catalog/PolarisSparkIntegrationTest.java , we use `S3MockContainer` as an S3 endpoint, which requires the same path-style access and custom enpdoint configuration as what's included here. Can we not follow the same pattern for minio?
Not found how endpoint or other properties can be set from REST API create or update catalog. Only arnRole is accepted as mandatory parameter today in AWS storage type.
>
> As a rule, I think vending static credentials is _not_ a good idea. Some customization for how the STS client is instantiated, possibly with support for custom profiles for different catalogs could make sense. But I think, ultimately, the credentials returned should _always_ be a temporary session token. Even if we just call [GetSessionToken](https://docs.aws.amazon.com/STS/latest/APIReference/API_GetSessionToken.html) without requiring an IAM role, it would vastly more secure than sending raw credentials.
I agree, by default it is STS without role. Raw credentials are poor fallback scenario when STS are not available. There is enterprise context where STS, assumeRole etc are not allowed. Only pair of keys are available. By example, Dell ECS require additional policy to enable STS and assumeRole. Not activated out of the box. I tried to be explicit about this degraded security pattern.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1813209421
##########
spec/polaris-management-service.yml:
##########
@@ -901,6 +903,58 @@ components:
required:
- roleArn
+ S3StorageConfigInfo:
+ type: object
+ description: S3 compatible storage configuration info (MinIO, Dell ECS, Netapp StorageGRID, ...)
+ allOf:
+ - $ref: '#/components/schemas/StorageConfigInfo'
+ properties:
+ credsCatalogAndClientStrategy:
+ type: string
+ enum:
+ - VALUE
+ - ENV_VAR_NAME
+ default: ENV_VAR_NAME
+ example: "ACCESS_KEY"
+ description: When you send key VALUE directly via this command, they should apear in logs. By ENV_VAR_NAME without dollar, only a reference will appear in logs, but the value have to be available as environnement variable in the context where Polaris is running
+ credsVendingStrategy:
+ type: string
+ enum:
+ - TOKEN_WITH_ASSUME_ROLE
+ - KEYS_SAME_AS_CATALOG
+ - KEYS_DEDICATED_TO_CLIENT
Review Comment:
Here client is anything trying to obtain keys (or security token service) from this catalog (spark, trino,...). There is no particular distinction of identity.
This is not the right term to use in the context of Polaris?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1830237327
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.s3compatible.S3CompatibleStorageConfigurationInfo.CredsCatalogAndClientStrategyEnum;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** Credential vendor that supports generating */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+ private StsClient stsClient;
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ public void createStsClient(S3CompatibleStorageConfigurationInfo s3storageConfig) {
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
Review Comment:
Like AWS class ?
Find it weird to put something related to a storage type outside the class and provided by the constructor. Seems Azure class is keeping it inside the class too.
No ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1830244325
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.s3compatible.S3CompatibleStorageConfigurationInfo.CredsCatalogAndClientStrategyEnum;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** Credential vendor that supports generating */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+ private StsClient stsClient;
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ public void createStsClient(S3CompatibleStorageConfigurationInfo s3storageConfig) {
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+ stsBuilder.region(
+ Region
+ .US_WEST_1); // default region to avoid bug, because most (all?) S3 compatible softwares
+ // do not care about regions
+ stsBuilder.endpointOverride(
+ URI.create(
+ s3storageConfig
+ .getS3Endpoint())); // S3Compatible - AWS STS endpoint is unique and different from
+ // the S3 Endpoint
+ stsBuilder.credentialsProvider(
+ StaticCredentialsProvider.create(
+ AwsBasicCredentials.create(
+ s3storageConfig.getS3CredentialsCatalogAccessKeyId(),
+ s3storageConfig.getS3CredentialsCatalogSecretAccessKey())));
+ this.stsClient = stsBuilder.build();
+ LOGGER.debug("S3Compatible - stsClient successfully built");
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public EnumMap<PolarisCredentialProperty, String> getSubscopedCreds(
+ @NotNull PolarisDiagnostics diagnostics,
+ @NotNull S3CompatibleStorageConfigurationInfo storageConfig,
+ boolean allowListOperation,
+ @NotNull Set<String> allowedReadLocations,
+ @NotNull Set<String> allowedWriteLocations) {
+
+ LOGGER.debug("S3Compatible - getSubscopedCreds - applying credential strategy");
+ EnumMap<PolarisCredentialProperty, String> propertiesMap =
+ new EnumMap<>(PolarisCredentialProperty.class);
+ propertiesMap.put(PolarisCredentialProperty.AWS_ENDPOINT, storageConfig.getS3Endpoint());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_PATH_STYLE_ACCESS,
+ storageConfig.getS3PathStyleAccess().toString());
+
+ switch (storageConfig.getCredsVendingStrategy()) {
+ case KEYS_SAME_AS_CATALOG:
+ if (storageConfig.getCredsCatalogAndClientStrategy()
+ == CredsCatalogAndClientStrategyEnum.ENV_VAR_NAME) {
+ String cai = System.getenv(storageConfig.getS3CredentialsCatalogAccessKeyId());
+ String cas = System.getenv(storageConfig.getS3CredentialsCatalogSecretAccessKey());
+ if (cai == null || cas == null) {
+ throw new IllegalArgumentException(
+ "Expect valid environment variable for catalog keys");
+ } else {
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, cai);
+ propertiesMap.put(PolarisCredentialProperty.AWS_SECRET_KEY, cas);
+ }
+ } else {
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_KEY_ID,
+ storageConfig.getS3CredentialsCatalogAccessKeyId());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_SECRET_KEY,
+ storageConfig.getS3CredentialsCatalogSecretAccessKey());
+ }
+ break;
+
+ case KEYS_DEDICATED_TO_CLIENT:
+ if (storageConfig.getCredsCatalogAndClientStrategy()
+ == CredsCatalogAndClientStrategyEnum.ENV_VAR_NAME) {
+ String cli = System.getenv(storageConfig.getS3CredentialsClientAccessKeyId());
+ String cls = System.getenv(storageConfig.getS3CredentialsClientSecretAccessKey());
Review Comment:
Do not find how to conciliate this with createCatalog() REST API... And not happy with the idea to let this credentials catalog's fully and exclusively at Polaris service level.
Bad compromise ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1811170303
##########
spec/polaris-management-service.yml:
##########
@@ -901,6 +903,58 @@ components:
required:
- roleArn
+ S3StorageConfigInfo:
+ type: object
+ description: S3 compatible storage configuration info (MinIO, Dell ECS, Netapp StorageGRID, ...)
+ allOf:
+ - $ref: '#/components/schemas/StorageConfigInfo'
+ properties:
+ credsCatalogAndClientStrategy:
+ type: string
+ enum:
+ - VALUE
+ - ENV_VAR_NAME
+ default: ENV_VAR_NAME
+ example: "ACCESS_KEY"
+ description: When you send key VALUE directly via this command, they should apear in logs. By ENV_VAR_NAME without dollar, only a reference will appear in logs, but the value have to be available as environnement variable in the context where Polaris is running
+ credsVendingStrategy:
+ type: string
+ enum:
+ - TOKEN_WITH_ASSUME_ROLE
+ - KEYS_SAME_AS_CATALOG
Review Comment:
It seems like `KEYS_SAME_AS_CATALOG` is the same as just disabling subscoping, is that right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1826552138
##########
spec/polaris-management-service.yml:
##########
@@ -901,6 +903,58 @@ components:
required:
- roleArn
+ S3StorageConfigInfo:
+ type: object
+ description: S3 compatible storage configuration info (MinIO, Dell ECS, Netapp StorageGRID, ...)
+ allOf:
+ - $ref: '#/components/schemas/StorageConfigInfo'
+ properties:
+ credsCatalogAndClientStrategy:
+ type: string
+ enum:
+ - VALUE
+ - ENV_VAR_NAME
+ default: ENV_VAR_NAME
+ example: "ACCESS_KEY"
+ description: When you send key VALUE directly via this command, they should apear in logs. By ENV_VAR_NAME without dollar, only a reference will appear in logs, but the value have to be available as environnement variable in the context where Polaris is running
+ credsVendingStrategy:
+ type: string
+ enum:
+ - TOKEN_WITH_ASSUME_ROLE
+ - KEYS_SAME_AS_CATALOG
Review Comment:
Correct. Default value False. With a clear description to explain users not familiar with concept of subScoping" implemented in Polaris. I will refactor this way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1829782885
##########
spec/polaris-management-service.yml:
##########
@@ -901,6 +903,58 @@ components:
required:
- roleArn
+ S3StorageConfigInfo:
Review Comment:
I think it will not be too hard to unify, in a next step. I miss AWS access to do tests, but for what I know or seen :
- STS endpoint is a AWS IAM endpoint, in S3 compatible solutions, when it is available, it is merged with S3 endpoint.
- Region is available in S3 compatible solutions, but not used a lot, or mostly implemented to be compliant with aws sdk clients.
- about the cross region, it is more related to Polaris and Iceberg with the Iceberg overloaded S3fileIO, but I seen that https://github.com/apache/iceberg/pull/11259 have been merged 3 weeks ago --> so next Iceberg version will be ok soon.
MinIO claim that their product API is 100% compatible with AWS S3 API. Almost the same for many alternatives...
- The S3 Compatible implementation could easily propose an optional parameter "arnRole" like the existing aws class, with less regexp patern to allow more flexibility for some implementation where "aws" inside the string is replaced by the product name (exemple "ecs" for DELL ECS)... It could help for a smooth transition
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1829782885
##########
spec/polaris-management-service.yml:
##########
@@ -901,6 +903,58 @@ components:
required:
- roleArn
+ S3StorageConfigInfo:
Review Comment:
I think it will not be too hard to unify, in a next step. I miss AWS access to do tests, but for what I know or seen :
- STS endpoint is a AWS IAM endpoint, in S3 compatible solutions, when it is available, it is merged with S3 endpoint.
- Region is available in S3 compatible solutions, but not used a lot, or mostly implemented to be compliant with aws sdk clients.
- about the cross region, it is more related to Polaris and Iceberg with the Iceberg overloaded S3fileIO, but I seen that https://github.com/apache/iceberg/pull/11259 have been merged 3 weeks ago --> so next Iceberg version will be ok soon.
MinIO claim that their product API is 100% compatible with AWS S3 API. Almost the same for many alternatives...
- The S3 Compatible implementation could easily propose an optional parameter "arnRole" like the mandatory one in the existing aws class, with less regexp patern to allow more flexibility for some implementation where "aws" inside the string is replaced by the product name (exemple "ecs" for DELL ECS)... It could help for a smooth transition
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "collado-mike (via GitHub)" <gi...@apache.org>.
collado-mike commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1829890935
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.s3compatible.S3CompatibleStorageConfigurationInfo.CredsCatalogAndClientStrategyEnum;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** Credential vendor that supports generating */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+ private StsClient stsClient;
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ public void createStsClient(S3CompatibleStorageConfigurationInfo s3storageConfig) {
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+ stsBuilder.region(
+ Region
+ .US_WEST_1); // default region to avoid bug, because most (all?) S3 compatible softwares
+ // do not care about regions
+ stsBuilder.endpointOverride(
+ URI.create(
+ s3storageConfig
+ .getS3Endpoint())); // S3Compatible - AWS STS endpoint is unique and different from
+ // the S3 Endpoint
+ stsBuilder.credentialsProvider(
+ StaticCredentialsProvider.create(
+ AwsBasicCredentials.create(
+ s3storageConfig.getS3CredentialsCatalogAccessKeyId(),
+ s3storageConfig.getS3CredentialsCatalogSecretAccessKey())));
+ this.stsClient = stsBuilder.build();
+ LOGGER.debug("S3Compatible - stsClient successfully built");
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public EnumMap<PolarisCredentialProperty, String> getSubscopedCreds(
+ @NotNull PolarisDiagnostics diagnostics,
+ @NotNull S3CompatibleStorageConfigurationInfo storageConfig,
+ boolean allowListOperation,
+ @NotNull Set<String> allowedReadLocations,
+ @NotNull Set<String> allowedWriteLocations) {
+
+ LOGGER.debug("S3Compatible - getSubscopedCreds - applying credential strategy");
+ EnumMap<PolarisCredentialProperty, String> propertiesMap =
+ new EnumMap<>(PolarisCredentialProperty.class);
+ propertiesMap.put(PolarisCredentialProperty.AWS_ENDPOINT, storageConfig.getS3Endpoint());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_PATH_STYLE_ACCESS,
+ storageConfig.getS3PathStyleAccess().toString());
+
+ switch (storageConfig.getCredsVendingStrategy()) {
+ case KEYS_SAME_AS_CATALOG:
+ if (storageConfig.getCredsCatalogAndClientStrategy()
+ == CredsCatalogAndClientStrategyEnum.ENV_VAR_NAME) {
+ String cai = System.getenv(storageConfig.getS3CredentialsCatalogAccessKeyId());
+ String cas = System.getenv(storageConfig.getS3CredentialsCatalogSecretAccessKey());
+ if (cai == null || cas == null) {
+ throw new IllegalArgumentException(
+ "Expect valid environment variable for catalog keys");
+ } else {
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, cai);
+ propertiesMap.put(PolarisCredentialProperty.AWS_SECRET_KEY, cas);
+ }
+ } else {
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_KEY_ID,
+ storageConfig.getS3CredentialsCatalogAccessKeyId());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_SECRET_KEY,
+ storageConfig.getS3CredentialsCatalogSecretAccessKey());
+ }
+ break;
+
+ case KEYS_DEDICATED_TO_CLIENT:
+ if (storageConfig.getCredsCatalogAndClientStrategy()
+ == CredsCatalogAndClientStrategyEnum.ENV_VAR_NAME) {
+ String cli = System.getenv(storageConfig.getS3CredentialsClientAccessKeyId());
+ String cls = System.getenv(storageConfig.getS3CredentialsClientSecretAccessKey());
Review Comment:
Seems you could rely on the `DefaultCredentialsProvider` and maybe allow profiles to be specified? This would allow for env variables, but also file configuration or other means of retrieving credentials.
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.s3compatible.S3CompatibleStorageConfigurationInfo.CredsCatalogAndClientStrategyEnum;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** Credential vendor that supports generating */
Review Comment:
comment seems to just ...
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.s3compatible.S3CompatibleStorageConfigurationInfo.CredsCatalogAndClientStrategyEnum;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** Credential vendor that supports generating */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+ private StsClient stsClient;
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ public void createStsClient(S3CompatibleStorageConfigurationInfo s3storageConfig) {
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
Review Comment:
Why is this constructed here rather than being passed in as a constructor parameter?
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.s3compatible.S3CompatibleStorageConfigurationInfo.CredsCatalogAndClientStrategyEnum;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** Credential vendor that supports generating */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+ private StsClient stsClient;
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ public void createStsClient(S3CompatibleStorageConfigurationInfo s3storageConfig) {
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+ stsBuilder.region(
+ Region
+ .US_WEST_1); // default region to avoid bug, because most (all?) S3 compatible softwares
+ // do not care about regions
+ stsBuilder.endpointOverride(
+ URI.create(
+ s3storageConfig
+ .getS3Endpoint())); // S3Compatible - AWS STS endpoint is unique and different from
+ // the S3 Endpoint
+ stsBuilder.credentialsProvider(
+ StaticCredentialsProvider.create(
+ AwsBasicCredentials.create(
+ s3storageConfig.getS3CredentialsCatalogAccessKeyId(),
+ s3storageConfig.getS3CredentialsCatalogSecretAccessKey())));
+ this.stsClient = stsBuilder.build();
+ LOGGER.debug("S3Compatible - stsClient successfully built");
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public EnumMap<PolarisCredentialProperty, String> getSubscopedCreds(
+ @NotNull PolarisDiagnostics diagnostics,
+ @NotNull S3CompatibleStorageConfigurationInfo storageConfig,
+ boolean allowListOperation,
+ @NotNull Set<String> allowedReadLocations,
+ @NotNull Set<String> allowedWriteLocations) {
+
+ LOGGER.debug("S3Compatible - getSubscopedCreds - applying credential strategy");
+ EnumMap<PolarisCredentialProperty, String> propertiesMap =
+ new EnumMap<>(PolarisCredentialProperty.class);
+ propertiesMap.put(PolarisCredentialProperty.AWS_ENDPOINT, storageConfig.getS3Endpoint());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_PATH_STYLE_ACCESS,
+ storageConfig.getS3PathStyleAccess().toString());
Review Comment:
These are catalog properties, not credential-vending properties. These should be set at the catalog-level when it is created. Those properties would then be passed into the FileIO when it is constructed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1830227895
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/PolarisCredentialProperty.java:
##########
@@ -23,6 +23,8 @@ public enum PolarisCredentialProperty {
AWS_KEY_ID(String.class, "s3.access-key-id", "the aws access key id"),
AWS_SECRET_KEY(String.class, "s3.secret-access-key", "the aws access key secret"),
AWS_TOKEN(String.class, "s3.session-token", "the aws scoped access token"),
+ AWS_ENDPOINT(String.class, "s3.endpoint", "the aws s3 endpoint"),
+ AWS_PATH_STYLE_ACCESS(Boolean.class, "s3.path-style-access", "the aws s3 path style access"),
Review Comment:
Many S3COMPATIBLE solutions are deployed without network devices or configurations in front of them allowing support of dynamic hosts names including buckets.
TLS certificate with private AC could also be a challenge for dynamic host name. "*. domain" can also be forbidden by some enterprise security policy
Path style is useful in many cases. In ideal world I agree it should stay deprecated...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1813179181
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3/S3CredentialsStorageIntegration.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3;
+
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** Credential vendor that supports generating */
+public class S3CredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3StorageConfigurationInfo> {
+
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CredentialsStorageIntegration.class);
+
+ private StsClient stsClient;
+
+ // Constructor
+ public S3CredentialsStorageIntegration() {
+ super(S3CredentialsStorageIntegration.class.getName());
+ }
+
+ public void createStsClient(S3StorageConfigurationInfo s3storageConfig) {
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+
+ LOGGER.info(
Review Comment:
OK.
It was helping me to understand global behavior at the beginning.
I agree to remove and clean whitespaces and comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1813196246
##########
spec/polaris-management-service.yml:
##########
@@ -901,6 +903,58 @@ components:
required:
- roleArn
+ S3StorageConfigInfo:
+ type: object
+ description: S3 compatible storage configuration info (MinIO, Dell ECS, Netapp StorageGRID, ...)
+ allOf:
+ - $ref: '#/components/schemas/StorageConfigInfo'
+ properties:
+ credsCatalogAndClientStrategy:
+ type: string
+ enum:
+ - VALUE
+ - ENV_VAR_NAME
+ default: ENV_VAR_NAME
+ example: "ACCESS_KEY"
+ description: When you send key VALUE directly via this command, they should apear in logs. By ENV_VAR_NAME without dollar, only a reference will appear in logs, but the value have to be available as environnement variable in the context where Polaris is running
+ credsVendingStrategy:
+ type: string
+ enum:
+ - TOKEN_WITH_ASSUME_ROLE
+ - KEYS_SAME_AS_CATALOG
+ - KEYS_DEDICATED_TO_CLIENT
+ default: TOKEN_WITH_ASSUME_ROLE
+ description: The catalog strategy to vend credentials to client. Options possible are same keys than catalog, keys dedicated to clients, or Tokens with STS methods 'assumeRole' for Dell ECS or NetApp StorageGrid solution, 'truc' for MinIo solution)
Review Comment:
Sorry, typo.
I have forget to delete the end of the sentence.
( I was thinking to list possible other MinIO supported functions who could be implemented https://min.io/docs/minio/linux/developers/security-token-service.html?ref=docs-redirect#id1
- AssumeRoleWithWebIdentity for OIDC external identity provider
- AssumeRoleWithLDAPIdentity for LDAP
- ...
)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1813159130
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3/S3CredentialsStorageIntegration.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3;
+
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** Credential vendor that supports generating */
+public class S3CredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3StorageConfigurationInfo> {
+
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CredentialsStorageIntegration.class);
+
+ private StsClient stsClient;
+
+ // Constructor
+ public S3CredentialsStorageIntegration() {
+ super(S3CredentialsStorageIntegration.class.getName());
+ }
+
+ public void createStsClient(S3StorageConfigurationInfo s3storageConfig) {
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+
+ LOGGER.info(
+ "S3Compatible - AWS STS endpoint is unique and different from the S3 Endpoint. AWS SDK need to be overided with dedicated Endpoint from S3Compatible, otherwise the AWS STS url is targeted");
+
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+
+ stsBuilder.region(
+ Region
+ .US_WEST_1); // default region to avoid bug, because most (all?) S3 compatible softwares
Review Comment:
Correct. It could logically be used with any S3, AWS included.
The name is maybe a bad compromise, I agree to change for a better one.
At the beginning I have named it s3compat.... but at the end simply s3.
This choice is the result of this two constatations :
- In core/storage folder the classes are named aws, gcp and azure
- But in Enum Storage Type AWS is directly named S3....
So I found in enum this S3_COMPATIBILITY compromise. MinIO use this term...
For the class I shortened it to S3 because it could works with AWS too.
I can come back to S3compat everywhere...
If Enum Storage was different and aws implementation named also AWS like GCP & Azure.... but it is not. Modifying the existing values of the Enum can be a breaking change, I do not want to introduce this change in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1813183830
##########
spec/polaris-management-service.yml:
##########
@@ -901,6 +903,58 @@ components:
required:
- roleArn
+ S3StorageConfigInfo:
Review Comment:
This implementation focuses on OnPrem S3 because there is already the AWS class.. However, I can try to let it work seamlessly with AWS :
- I think that roleArn is not mandatory for AWS S3, so let it to the existing implementation for this scenario
- Using access and secret key should work with AWS S3 too
- I have overrided the AWS STS endpoint with S3 endpoint. I could add a modification, maybe with a STS endpoint property... something like "if property is empty" -> "STSclient call AWS default STS endpoint" else -> "STS client call the endpoint setted" or a boolean with a clear and explicit description
- I have removed the cross region tweak of the AWS FileIOClientFactory, it can be kept to assure a full compatibility
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1813183830
##########
spec/polaris-management-service.yml:
##########
@@ -901,6 +903,58 @@ components:
required:
- roleArn
+ S3StorageConfigInfo:
Review Comment:
This implementation focuses on OnPrem S3 because there is already the AWS class..
However, in a next step I can try to let it work seamlessly with AWS too :
- I think that roleArn is not mandatory for AWS S3, so let it to the existing implementation for this scenario
- Using access and secret key should work with AWS S3 too
- I have overrided the AWS STS endpoint with S3 endpoint. I could add a modification, maybe with a STS endpoint property... something like "if property is empty" -> "STSclient call AWS default STS endpoint" else -> "STS client call the endpoint setted" or a boolean with a clear and explicit description
- Region, (maybe little more reflexion is needed to avoid conflict)
- Add region property
- I have removed the cross region tweak of the AWS FileIOClientFactory, it can be kept to assure a full compatibility
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1813201796
##########
spec/polaris-management-service.yml:
##########
@@ -901,6 +903,58 @@ components:
required:
- roleArn
+ S3StorageConfigInfo:
+ type: object
+ description: S3 compatible storage configuration info (MinIO, Dell ECS, Netapp StorageGRID, ...)
+ allOf:
+ - $ref: '#/components/schemas/StorageConfigInfo'
+ properties:
+ credsCatalogAndClientStrategy:
+ type: string
+ enum:
+ - VALUE
+ - ENV_VAR_NAME
+ default: ENV_VAR_NAME
+ example: "ACCESS_KEY"
+ description: When you send key VALUE directly via this command, they should apear in logs. By ENV_VAR_NAME without dollar, only a reference will appear in logs, but the value have to be available as environnement variable in the context where Polaris is running
+ credsVendingStrategy:
+ type: string
+ enum:
+ - TOKEN_WITH_ASSUME_ROLE
+ - KEYS_SAME_AS_CATALOG
Review Comment:
Exactly, explained like that it's much clearer.
I was trying also to do a distinction with the third option where it is possible to serve another pair of keys, not the one used by the catalog. In this last case, the keys served to the 'client' can be revoke by S3 admin without touching the keys used internaly by this catalog.
It should not be used in production but for dev mode, or some context where security subscoping is not an issue, it is an easy solution
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1813159130
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3/S3CredentialsStorageIntegration.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3;
+
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** Credential vendor that supports generating */
+public class S3CredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3StorageConfigurationInfo> {
+
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CredentialsStorageIntegration.class);
+
+ private StsClient stsClient;
+
+ // Constructor
+ public S3CredentialsStorageIntegration() {
+ super(S3CredentialsStorageIntegration.class.getName());
+ }
+
+ public void createStsClient(S3StorageConfigurationInfo s3storageConfig) {
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+
+ LOGGER.info(
+ "S3Compatible - AWS STS endpoint is unique and different from the S3 Endpoint. AWS SDK need to be overided with dedicated Endpoint from S3Compatible, otherwise the AWS STS url is targeted");
+
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+
+ stsBuilder.region(
+ Region
+ .US_WEST_1); // default region to avoid bug, because most (all?) S3 compatible softwares
Review Comment:
Correct the name give this idea.
The name is maybe a bad compromise, I agree to change for a better one.
At the beginning I have named it s3compatible but at the end simply s3.
This choice is the result of this two constatations :
- In core/storage folder the classes are named aws, gcp and azure
- But in Enum Storage Type AWS is directly named S3....
So I found in enum this S3_COMPATIBILITY compromise. MinIO use this term...
For the class I shortened it to S3 because S3Compatible was verbose.
I can come back to S3compatible everywhere to be consistent...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "omarsmak (via GitHub)" <gi...@apache.org>.
omarsmak commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1988516970
##########
polaris-core/src/main/java/org/apache/polaris/core/entity/CatalogEntity.java:
##########
@@ -250,6 +268,21 @@ public Builder setStorageConfigurationInfo(
awsConfig.validateArn(awsConfigModel.getRoleArn());
config = awsConfig;
break;
+
+ case S3_COMPATIBLE:
+ S3CompatibleStorageConfigInfo s3ConfigModel =
+ (S3CompatibleStorageConfigInfo) storageConfigModel;
+ config =
+ new S3CompatibleStorageConfigurationInfo(
Review Comment:
Don't we as well need pass a custom STS endpoint in order to initialize STSClient with a custom endpoint that points to MinIO as an example?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1994110373
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import static org.apache.polaris.core.config.FeatureConfiguration.STORAGE_CREDENTIAL_DURATION_SECONDS;
+import static org.apache.polaris.core.config.PolarisConfiguration.loadConfig;
+import static org.apache.polaris.core.storage.PolarisCredentialProperty.AWS_KEY_ID;
+import static org.apache.polaris.core.storage.PolarisCredentialProperty.AWS_SECRET_KEY;
+import static org.apache.polaris.core.storage.PolarisCredentialProperty.AWS_TOKEN;
+
+import jakarta.annotation.Nonnull;
+import jakarta.ws.rs.NotAuthorizedException;
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.profiles.ProfileFileSupplier;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** S3 compatible implementation of PolarisStorageIntegration */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ @Override
+ public EnumMap<PolarisCredentialProperty, String> getSubscopedCreds(
+ @Nonnull PolarisDiagnostics diagnostics,
+ @Nonnull S3CompatibleStorageConfigurationInfo storageConfig,
+ boolean allowListOperation,
+ @Nonnull Set<String> allowedReadLocations,
+ @Nonnull Set<String> allowedWriteLocations) {
+
+ String caI = System.getenv(storageConfig.getS3CredentialsCatalogAccessKeyId());
Review Comment:
I shared the same concern, see here https://github.com/apache/polaris/pull/389/files#r1980741611
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2779941852
@flyrain :
> Given the PR is WIP for a quite long time, can we merge this first, and make changes later? We can always improve on it and make changes as needed.
There are unaddressed concerns with this PR. I believe they need to be resolved before merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "Gerrit-K (via GitHub)" <gi...@apache.org>.
Gerrit-K commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1895896396
##########
polaris-core/src/main/java/org/apache/polaris/core/entity/CatalogEntity.java:
##########
@@ -141,6 +143,25 @@ private StorageConfigInfo getStorageInfo(Map<String, String> internalProperties)
.setRegion(awsConfig.getRegion())
.build();
}
+ if (configInfo instanceof S3CompatibleStorageConfigurationInfo) {
+ S3CompatibleStorageConfigurationInfo s3Config =
+ (S3CompatibleStorageConfigurationInfo) configInfo;
+ return S3CompatibleStorageConfigInfo.builder()
+ .setStorageType(StorageConfigInfo.StorageTypeEnum.S3_COMPATIBLE)
+ .setS3Endpoint(s3Config.getS3Endpoint())
+ .setS3PathStyleAccess(s3Config.getS3PathStyleAccess())
+ .setSkipCredentialSubscopingIndirection(
+ s3Config.getSkipCredentialSubscopingIndirection())
+ .setAllowedLocations(s3Config.getAllowedLocations())
+ .setS3CredentialsCatalogAccessKeyId(s3Config.getS3CredentialsCatalogAccessKeyId())
+ .setS3CredentialsCatalogSecretAccessKey(
+ s3Config.getS3CredentialsCatalogSecretAccessKey())
+ .setS3CredentialsClientAccessKeyId(s3Config.getS3CredentialsClientSecretAccessKey())
+ .setS3CredentialsClientSecretAccessKey(s3Config.getS3CredentialsClientAccessKeyId())
Review Comment:
```suggestion
.setS3CredentialsClientAccessKeyId(s3Config.getS3CredentialsClientAccessKeyId())
.setS3CredentialsClientSecretAccessKey(s3Config.getS3CredentialsClientSecretAccessKey())
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1896751195
##########
polaris-core/src/main/java/org/apache/polaris/core/entity/CatalogEntity.java:
##########
@@ -141,6 +143,25 @@ private StorageConfigInfo getStorageInfo(Map<String, String> internalProperties)
.setRegion(awsConfig.getRegion())
.build();
}
+ if (configInfo instanceof S3CompatibleStorageConfigurationInfo) {
+ S3CompatibleStorageConfigurationInfo s3Config =
+ (S3CompatibleStorageConfigurationInfo) configInfo;
+ return S3CompatibleStorageConfigInfo.builder()
+ .setStorageType(StorageConfigInfo.StorageTypeEnum.S3_COMPATIBLE)
+ .setS3Endpoint(s3Config.getS3Endpoint())
+ .setS3PathStyleAccess(s3Config.getS3PathStyleAccess())
+ .setSkipCredentialSubscopingIndirection(
+ s3Config.getSkipCredentialSubscopingIndirection())
+ .setAllowedLocations(s3Config.getAllowedLocations())
+ .setS3CredentialsCatalogAccessKeyId(s3Config.getS3CredentialsCatalogAccessKeyId())
+ .setS3CredentialsCatalogSecretAccessKey(
+ s3Config.getS3CredentialsCatalogSecretAccessKey())
+ .setS3CredentialsClientAccessKeyId(s3Config.getS3CredentialsClientSecretAccessKey())
+ .setS3CredentialsClientSecretAccessKey(s3Config.getS3CredentialsClientAccessKeyId())
Review Comment:
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1830247969
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.s3compatible.S3CompatibleStorageConfigurationInfo.CredsCatalogAndClientStrategyEnum;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** Credential vendor that supports generating */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+ private StsClient stsClient;
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ public void createStsClient(S3CompatibleStorageConfigurationInfo s3storageConfig) {
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+ stsBuilder.region(
+ Region
+ .US_WEST_1); // default region to avoid bug, because most (all?) S3 compatible softwares
+ // do not care about regions
+ stsBuilder.endpointOverride(
+ URI.create(
+ s3storageConfig
+ .getS3Endpoint())); // S3Compatible - AWS STS endpoint is unique and different from
+ // the S3 Endpoint
+ stsBuilder.credentialsProvider(
+ StaticCredentialsProvider.create(
+ AwsBasicCredentials.create(
+ s3storageConfig.getS3CredentialsCatalogAccessKeyId(),
+ s3storageConfig.getS3CredentialsCatalogSecretAccessKey())));
+ this.stsClient = stsBuilder.build();
+ LOGGER.debug("S3Compatible - stsClient successfully built");
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public EnumMap<PolarisCredentialProperty, String> getSubscopedCreds(
+ @NotNull PolarisDiagnostics diagnostics,
+ @NotNull S3CompatibleStorageConfigurationInfo storageConfig,
+ boolean allowListOperation,
+ @NotNull Set<String> allowedReadLocations,
+ @NotNull Set<String> allowedWriteLocations) {
+
+ LOGGER.debug("S3Compatible - getSubscopedCreds - applying credential strategy");
+ EnumMap<PolarisCredentialProperty, String> propertiesMap =
+ new EnumMap<>(PolarisCredentialProperty.class);
+ propertiesMap.put(PolarisCredentialProperty.AWS_ENDPOINT, storageConfig.getS3Endpoint());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_PATH_STYLE_ACCESS,
+ storageConfig.getS3PathStyleAccess().toString());
Review Comment:
To be refactor for satisfying change to requested : boolean "SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION".
I will try to find a way to move it in the catalog properties. But catalog properties are not forwarded to "S3CompatibleCredentialsStorageIntegration.java", only storage properties by default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "yikuanlee (via GitHub)" <gi...@apache.org>.
yikuanlee commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1919403665
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/PolarisStorageConfigurationInfo.java:
##########
@@ -241,6 +243,7 @@ public void validateMaxAllowedLocations(int maxAllowedLocations) {
/** Polaris' storage type, each has a fixed prefix for its location */
public enum StorageType {
S3("s3://"),
+ S3_COMPATIBLE("s3://"),
Review Comment:
wondered if we only support "s3://", can we also support "s3a://" for location
this way table metadata can support location s3a://... for hadoop client read/write data
like S3_COMPATIBLE(List.of("s3://", "s3a://")),
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1948001113
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/PolarisCredentialProperty.java:
##########
@@ -23,6 +23,8 @@ public enum PolarisCredentialProperty {
AWS_KEY_ID(String.class, "s3.access-key-id", "the aws access key id"),
AWS_SECRET_KEY(String.class, "s3.secret-access-key", "the aws access key secret"),
AWS_TOKEN(String.class, "s3.session-token", "the aws scoped access token"),
+ AWS_ENDPOINT(String.class, "s3.endpoint", "the aws s3 endpoint"),
+ AWS_PATH_STYLE_ACCESS(Boolean.class, "s3.path-style-access", "the aws s3 path style access"),
Review Comment:
I think @eric-maynard was asking if we can change the description to something like this:
```suggestion
AWS_PATH_STYLE_ACCESS(Boolean.class, "s3.path-style-access", "whether or not to use path-style access"),
```
I also agreed that we should make sure it `false` by default
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "mmgaggle (via GitHub)" <gi...@apache.org>.
mmgaggle commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1952780004
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/PolarisCredentialProperty.java:
##########
@@ -23,6 +23,8 @@ public enum PolarisCredentialProperty {
AWS_KEY_ID(String.class, "s3.access-key-id", "the aws access key id"),
AWS_SECRET_KEY(String.class, "s3.secret-access-key", "the aws access key secret"),
AWS_TOKEN(String.class, "s3.session-token", "the aws scoped access token"),
+ AWS_ENDPOINT(String.class, "s3.endpoint", "the aws s3 endpoint"),
+ AWS_PATH_STYLE_ACCESS(Boolean.class, "s3.path-style-access", "the aws s3 path style access"),
Review Comment:
For IBM's watsonx.data product it is set to `true` by default for Minio and Ceph bucket types, reason being that it's more likely to work. Path style will work regardless of whether the customer has setup wildcard DNS, a TLS certificate with a subject-alternate-name (the wildcard), and the hostname in the zonegroup (for Ceph). Virtual host style will only work if all of those things are done.
It's not a hill I would die on, but it's worthy of consideration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1953509365
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.s3compatible.S3CompatibleStorageConfigurationInfo.CredsCatalogAndClientStrategyEnum;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** Credential vendor that supports generating */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+ private StsClient stsClient;
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ public void createStsClient(S3CompatibleStorageConfigurationInfo s3storageConfig) {
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+ stsBuilder.region(
+ Region
+ .US_WEST_1); // default region to avoid bug, because most (all?) S3 compatible softwares
+ // do not care about regions
+ stsBuilder.endpointOverride(
+ URI.create(
+ s3storageConfig
+ .getS3Endpoint())); // S3Compatible - AWS STS endpoint is unique and different from
+ // the S3 Endpoint
+ stsBuilder.credentialsProvider(
+ StaticCredentialsProvider.create(
+ AwsBasicCredentials.create(
+ s3storageConfig.getS3CredentialsCatalogAccessKeyId(),
+ s3storageConfig.getS3CredentialsCatalogSecretAccessKey())));
+ this.stsClient = stsBuilder.build();
+ LOGGER.debug("S3Compatible - stsClient successfully built");
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public EnumMap<PolarisCredentialProperty, String> getSubscopedCreds(
+ @NotNull PolarisDiagnostics diagnostics,
+ @NotNull S3CompatibleStorageConfigurationInfo storageConfig,
+ boolean allowListOperation,
+ @NotNull Set<String> allowedReadLocations,
+ @NotNull Set<String> allowedWriteLocations) {
+
+ LOGGER.debug("S3Compatible - getSubscopedCreds - applying credential strategy");
+ EnumMap<PolarisCredentialProperty, String> propertiesMap =
+ new EnumMap<>(PolarisCredentialProperty.class);
+ propertiesMap.put(PolarisCredentialProperty.AWS_ENDPOINT, storageConfig.getS3Endpoint());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_PATH_STYLE_ACCESS,
+ storageConfig.getS3PathStyleAccess().toString());
Review Comment:
I trust you that there is probably a simple way to do this.
But I just failed to find how to access this stuff, even after hours of inspecting context properties in debug session.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1953531402
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import jakarta.annotation.Nonnull;
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Stream;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.policybuilder.iam.IamConditionOperator;
+import software.amazon.awssdk.policybuilder.iam.IamEffect;
+import software.amazon.awssdk.policybuilder.iam.IamPolicy;
+import software.amazon.awssdk.policybuilder.iam.IamResource;
+import software.amazon.awssdk.policybuilder.iam.IamStatement;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** S3 compatible implementation of PolarisStorageIntegration */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+ private StsClient stsClient;
+ private String caI, caS; // catalogAccessKeyId & catalogSecretAccessKey
+ private String clI, clS; // clientAccessKeyId & clientSecretAccessKey
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public EnumMap<PolarisCredentialProperty, String> getSubscopedCreds(
+ @NotNull PolarisDiagnostics diagnostics,
+ @NotNull S3CompatibleStorageConfigurationInfo storageConfig,
+ boolean allowListOperation,
+ @NotNull Set<String> allowedReadLocations,
+ @NotNull Set<String> allowedWriteLocations) {
+
+ caI = System.getenv(storageConfig.getS3CredentialsCatalogAccessKeyId());
+ caS = System.getenv(storageConfig.getS3CredentialsCatalogSecretAccessKey());
+
+ EnumMap<PolarisCredentialProperty, String> propertiesMap =
+ new EnumMap<>(PolarisCredentialProperty.class);
+ propertiesMap.put(PolarisCredentialProperty.AWS_ENDPOINT, storageConfig.getS3Endpoint());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_PATH_STYLE_ACCESS,
+ storageConfig.getS3PathStyleAccess().toString());
+ if (storageConfig.getS3Region() != null) {
+ propertiesMap.put(PolarisCredentialProperty.CLIENT_REGION, storageConfig.getS3Region());
+ }
+
+ if (storageConfig.getSkipCredentialSubscopingIndirection() == true) {
+ LOGGER.debug("S3Compatible - skipCredentialSubscopingIndirection !");
+ clI = System.getenv(storageConfig.getS3CredentialsClientAccessKeyId());
+ clS = System.getenv(storageConfig.getS3CredentialsClientSecretAccessKey());
+ if (clI != null && clS != null) {
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, clI);
+ propertiesMap.put(PolarisCredentialProperty.AWS_SECRET_KEY, clS);
+ } else {
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, caI);
+ propertiesMap.put(PolarisCredentialProperty.AWS_SECRET_KEY, caS);
+ }
+ } else {
+ LOGGER.debug("S3Compatible - assumeRole !");
+ createStsClient(storageConfig);
+ AssumeRoleResponse response =
+ stsClient.assumeRole(
+ AssumeRoleRequest.builder()
+ .roleSessionName("PolarisCredentialsSTS")
+ .roleArn(
+ (storageConfig.getS3RoleArn() == null) ? "" : storageConfig.getS3RoleArn())
+ .policy(
+ policyString(allowListOperation, allowedReadLocations, allowedWriteLocations)
+ .toJson())
+ .build());
+
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, response.credentials().accessKeyId());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_SECRET_KEY, response.credentials().secretAccessKey());
+ propertiesMap.put(PolarisCredentialProperty.AWS_TOKEN, response.credentials().sessionToken());
+ LOGGER.debug(
+ "S3Compatible - assumeRole - Token Expiration at : {}",
+ response.credentials().expiration().toString());
+ }
+
+ return propertiesMap;
+ }
+
+ public void createStsClient(S3CompatibleStorageConfigurationInfo storageConfig) {
+ LOGGER.debug("S3Compatible - createStsClient()");
+ try {
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+ stsBuilder.endpointOverride(URI.create(storageConfig.getS3Endpoint()));
+ if (caI != null && caS != null) {
+ // else default provider build credentials from profile or standard AWS env var
+ stsBuilder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(caI, caS)));
+ }
+ this.stsClient = stsBuilder.build();
+ LOGGER.debug("S3Compatible - stsClient successfully built");
+
+ } catch (Exception e) {
+ System.err.println("S3Compatible - stsClient - build failure : " + e.getMessage());
+ }
+ }
+
+ /*
+ * function from AwsCredentialsStorageIntegration but without roleArn parameter
+ */
+ private IamPolicy policyString(
Review Comment:
Any suggestion ?
Something like calling the awsCredentialsStorageIntegration one ? (I have failed to deal with roleArn pattern onPrem not allowed by this method i.e : "arn:ecs:s3:...." )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1953574473
##########
polaris-server.yml:
##########
@@ -69,6 +69,7 @@ featureConfiguration:
ENFORCE_PRINCIPAL_CREDENTIAL_ROTATION_REQUIRED_CHECKING: false
SUPPORTED_CATALOG_STORAGE_TYPES:
- S3
+ - S3_COMPATIBLE
Review Comment:
Thank you very much.
(My old laptop have a lot of pain to run live debug since quarkus is there... a lot harder to setup and sloooooooooower, I do not know if I miss something or if quarkus require much more dev ressources..)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1813183830
##########
spec/polaris-management-service.yml:
##########
@@ -901,6 +903,58 @@ components:
required:
- roleArn
+ S3StorageConfigInfo:
Review Comment:
This implementation is focused OnPrem S3 because there is already the AWS class... however I can try to let it also working transparently with AWS :
- I think that roleArn is not mandatory for AWS S3, so let it to the existing implementation
- Using access and secret key should work with AWS S3 too
- I have overrided the AWS STS endpoint with S3 endpoint. I could add a modification, maybe with a STS endpoint property... something like "if property is empty" -> "STSclient call AWS default STS endpoint" else -> "STS client call the endpoint setted" or a boolean with a clear and explicit description
- I have removed the cross region tweak of the AWS FileIOClientFactory, it can be kept to assure a full compatibility
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1816354209
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3/S3CredentialsStorageIntegration.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3;
+
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** Credential vendor that supports generating */
+public class S3CredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3StorageConfigurationInfo> {
+
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CredentialsStorageIntegration.class);
+
+ private StsClient stsClient;
+
+ // Constructor
+ public S3CredentialsStorageIntegration() {
+ super(S3CredentialsStorageIntegration.class.getName());
+ }
+
+ public void createStsClient(S3StorageConfigurationInfo s3storageConfig) {
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+
+ LOGGER.info(
Review Comment:
Removed some whitespaces and reduced a little bit comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1811171015
##########
spec/polaris-management-service.yml:
##########
@@ -901,6 +903,58 @@ components:
required:
- roleArn
+ S3StorageConfigInfo:
+ type: object
+ description: S3 compatible storage configuration info (MinIO, Dell ECS, Netapp StorageGRID, ...)
+ allOf:
+ - $ref: '#/components/schemas/StorageConfigInfo'
+ properties:
+ credsCatalogAndClientStrategy:
+ type: string
+ enum:
+ - VALUE
+ - ENV_VAR_NAME
+ default: ENV_VAR_NAME
+ example: "ACCESS_KEY"
+ description: When you send key VALUE directly via this command, they should apear in logs. By ENV_VAR_NAME without dollar, only a reference will appear in logs, but the value have to be available as environnement variable in the context where Polaris is running
+ credsVendingStrategy:
+ type: string
+ enum:
+ - TOKEN_WITH_ASSUME_ROLE
+ - KEYS_SAME_AS_CATALOG
+ - KEYS_DEDICATED_TO_CLIENT
Review Comment:
How does this work? What identifies a client?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1813201796
##########
spec/polaris-management-service.yml:
##########
@@ -901,6 +903,58 @@ components:
required:
- roleArn
+ S3StorageConfigInfo:
+ type: object
+ description: S3 compatible storage configuration info (MinIO, Dell ECS, Netapp StorageGRID, ...)
+ allOf:
+ - $ref: '#/components/schemas/StorageConfigInfo'
+ properties:
+ credsCatalogAndClientStrategy:
+ type: string
+ enum:
+ - VALUE
+ - ENV_VAR_NAME
+ default: ENV_VAR_NAME
+ example: "ACCESS_KEY"
+ description: When you send key VALUE directly via this command, they should apear in logs. By ENV_VAR_NAME without dollar, only a reference will appear in logs, but the value have to be available as environnement variable in the context where Polaris is running
+ credsVendingStrategy:
+ type: string
+ enum:
+ - TOKEN_WITH_ASSUME_ROLE
+ - KEYS_SAME_AS_CATALOG
Review Comment:
Exactly, explained like that it's much clearer.
I was trying also to do a distinction with the third option where it is possible to serve another pair of keys, not the one used by the catalog. In this last case, the keys served to the 'client' can be revoke by S3 admin without touching the keys used internaly by this catalog.
It should not be used in production but for dev mode or some context where security subscoping is not an issue it is an easy solution
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1813183830
##########
spec/polaris-management-service.yml:
##########
@@ -901,6 +903,58 @@ components:
required:
- roleArn
+ S3StorageConfigInfo:
Review Comment:
I think that roleArn is not mandatory for AWS S3.
Using access and secret key should work with AWS S3 too.
Same naming bad compromise explained previously. I agree to modify in a better suited way and be aligned with Polaris conventions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1824969729
##########
spec/polaris-management-service.yml:
##########
@@ -901,6 +903,58 @@ components:
required:
- roleArn
+ S3StorageConfigInfo:
+ type: object
+ description: S3 compatible storage configuration info (MinIO, Dell ECS, Netapp StorageGRID, ...)
+ allOf:
+ - $ref: '#/components/schemas/StorageConfigInfo'
+ properties:
+ credsCatalogAndClientStrategy:
+ type: string
+ enum:
+ - VALUE
+ - ENV_VAR_NAME
+ default: ENV_VAR_NAME
+ example: "ACCESS_KEY"
+ description: When you send key VALUE directly via this command, they should apear in logs. By ENV_VAR_NAME without dollar, only a reference will appear in logs, but the value have to be available as environnement variable in the context where Polaris is running
+ credsVendingStrategy:
+ type: string
+ enum:
+ - TOKEN_WITH_ASSUME_ROLE
+ - KEYS_SAME_AS_CATALOG
+ - KEYS_DEDICATED_TO_CLIENT
Review Comment:
I think the term is correct, I was just stuck trying to understand how the service will track which keys are dedicated to which client.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1824968986
##########
spec/polaris-management-service.yml:
##########
@@ -901,6 +903,58 @@ components:
required:
- roleArn
+ S3StorageConfigInfo:
+ type: object
+ description: S3 compatible storage configuration info (MinIO, Dell ECS, Netapp StorageGRID, ...)
+ allOf:
+ - $ref: '#/components/schemas/StorageConfigInfo'
+ properties:
+ credsCatalogAndClientStrategy:
+ type: string
+ enum:
+ - VALUE
+ - ENV_VAR_NAME
+ default: ENV_VAR_NAME
+ example: "ACCESS_KEY"
+ description: When you send key VALUE directly via this command, they should apear in logs. By ENV_VAR_NAME without dollar, only a reference will appear in logs, but the value have to be available as environnement variable in the context where Polaris is running
+ credsVendingStrategy:
+ type: string
+ enum:
+ - TOKEN_WITH_ASSUME_ROLE
+ - KEYS_SAME_AS_CATALOG
Review Comment:
Maybe instead of this, we should expect the user to just disable subscoping via `SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3... [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1828951565
##########
spec/polaris-management-service.yml:
##########
@@ -901,6 +903,58 @@ components:
required:
- roleArn
+ S3StorageConfigInfo:
+ type: object
+ description: S3 compatible storage configuration info (MinIO, Dell ECS, Netapp StorageGRID, ...)
+ allOf:
+ - $ref: '#/components/schemas/StorageConfigInfo'
+ properties:
+ credsCatalogAndClientStrategy:
+ type: string
+ enum:
+ - VALUE
+ - ENV_VAR_NAME
+ default: ENV_VAR_NAME
+ example: "ACCESS_KEY"
+ description: When you send key VALUE directly via this command, they should apear in logs. By ENV_VAR_NAME without dollar, only a reference will appear in logs, but the value have to be available as environnement variable in the context where Polaris is running
+ credsVendingStrategy:
+ type: string
+ enum:
+ - TOKEN_WITH_ASSUME_ROLE
+ - KEYS_SAME_AS_CATALOG
+ - KEYS_DEDICATED_TO_CLIENT
Review Comment:
> you have not to rely on the main credentials provided at the global Polaris service level.
I think this is the key point here. I agree that the experience you describe is bad, but I'm not sure that fixing it should be a blocker for s3compat support (or that this is the right fix).
Would you be okay saving this for later, or carving it out into a different PR? In my view relying on the global credentials in production is universally a bad idea, regardless of what STORAGE_TYPE you're using.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2553376743
Ready for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "Gerrit-K (via GitHub)" <gi...@apache.org>.
Gerrit-K commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2565348736
Not sure if this is out of scope for this PoC(?), but this PR doesn't contain the corresponding changes for the CLI, yet. This could break some CLI commands. For example, if a catalog with type `S3_COMPATIBLE` is created, the command `polaris catalogs list` won't work anymore, even if there are other catalogs with regular storage types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2843839469
Thanks for your contribution @lefebsy !.. even if this PR does not get merged
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "yikuanlee (via GitHub)" <gi...@apache.org>.
yikuanlee commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1919403665
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/PolarisStorageConfigurationInfo.java:
##########
@@ -241,6 +243,7 @@ public void validateMaxAllowedLocations(int maxAllowedLocations) {
/** Polaris' storage type, each has a fixed prefix for its location */
public enum StorageType {
S3("s3://"),
+ S3_COMPATIBLE("s3://"),
Review Comment:
wondered if we only support "s3://", can we also support "s3a://" for location
like S3_COMPATIBLE(List.of("s3://", "s3a://")),
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1953163646
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/PolarisCredentialProperty.java:
##########
@@ -23,6 +23,8 @@ public enum PolarisCredentialProperty {
AWS_KEY_ID(String.class, "s3.access-key-id", "the aws access key id"),
AWS_SECRET_KEY(String.class, "s3.secret-access-key", "the aws access key secret"),
AWS_TOKEN(String.class, "s3.session-token", "the aws scoped access token"),
+ AWS_ENDPOINT(String.class, "s3.endpoint", "the aws s3 endpoint"),
+ AWS_PATH_STYLE_ACCESS(Boolean.class, "s3.path-style-access", "the aws s3 path style access"),
Review Comment:
To clarify, I was referring to the Polaris scope default. Users always have the option to set it to true when creating a new catalog.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1953509365
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.s3compatible.S3CompatibleStorageConfigurationInfo.CredsCatalogAndClientStrategyEnum;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** Credential vendor that supports generating */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+ private StsClient stsClient;
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ public void createStsClient(S3CompatibleStorageConfigurationInfo s3storageConfig) {
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+ stsBuilder.region(
+ Region
+ .US_WEST_1); // default region to avoid bug, because most (all?) S3 compatible softwares
+ // do not care about regions
+ stsBuilder.endpointOverride(
+ URI.create(
+ s3storageConfig
+ .getS3Endpoint())); // S3Compatible - AWS STS endpoint is unique and different from
+ // the S3 Endpoint
+ stsBuilder.credentialsProvider(
+ StaticCredentialsProvider.create(
+ AwsBasicCredentials.create(
+ s3storageConfig.getS3CredentialsCatalogAccessKeyId(),
+ s3storageConfig.getS3CredentialsCatalogSecretAccessKey())));
+ this.stsClient = stsBuilder.build();
+ LOGGER.debug("S3Compatible - stsClient successfully built");
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public EnumMap<PolarisCredentialProperty, String> getSubscopedCreds(
+ @NotNull PolarisDiagnostics diagnostics,
+ @NotNull S3CompatibleStorageConfigurationInfo storageConfig,
+ boolean allowListOperation,
+ @NotNull Set<String> allowedReadLocations,
+ @NotNull Set<String> allowedWriteLocations) {
+
+ LOGGER.debug("S3Compatible - getSubscopedCreds - applying credential strategy");
+ EnumMap<PolarisCredentialProperty, String> propertiesMap =
+ new EnumMap<>(PolarisCredentialProperty.class);
+ propertiesMap.put(PolarisCredentialProperty.AWS_ENDPOINT, storageConfig.getS3Endpoint());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_PATH_STYLE_ACCESS,
+ storageConfig.getS3PathStyleAccess().toString());
Review Comment:
I trust you that there is probably a way to do this.
But I just failed to find how to access this stuff, even after hours of inspecting context properties in debug session.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1953533589
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import jakarta.annotation.Nonnull;
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Stream;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.policybuilder.iam.IamConditionOperator;
+import software.amazon.awssdk.policybuilder.iam.IamEffect;
+import software.amazon.awssdk.policybuilder.iam.IamPolicy;
+import software.amazon.awssdk.policybuilder.iam.IamResource;
+import software.amazon.awssdk.policybuilder.iam.IamStatement;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** S3 compatible implementation of PolarisStorageIntegration */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+ private StsClient stsClient;
+ private String caI, caS; // catalogAccessKeyId & catalogSecretAccessKey
+ private String clI, clS; // clientAccessKeyId & clientSecretAccessKey
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public EnumMap<PolarisCredentialProperty, String> getSubscopedCreds(
+ @NotNull PolarisDiagnostics diagnostics,
+ @NotNull S3CompatibleStorageConfigurationInfo storageConfig,
+ boolean allowListOperation,
+ @NotNull Set<String> allowedReadLocations,
+ @NotNull Set<String> allowedWriteLocations) {
+
+ caI = System.getenv(storageConfig.getS3CredentialsCatalogAccessKeyId());
+ caS = System.getenv(storageConfig.getS3CredentialsCatalogSecretAccessKey());
+
+ EnumMap<PolarisCredentialProperty, String> propertiesMap =
+ new EnumMap<>(PolarisCredentialProperty.class);
+ propertiesMap.put(PolarisCredentialProperty.AWS_ENDPOINT, storageConfig.getS3Endpoint());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_PATH_STYLE_ACCESS,
+ storageConfig.getS3PathStyleAccess().toString());
+ if (storageConfig.getS3Region() != null) {
+ propertiesMap.put(PolarisCredentialProperty.CLIENT_REGION, storageConfig.getS3Region());
+ }
+
+ if (storageConfig.getSkipCredentialSubscopingIndirection() == true) {
+ LOGGER.debug("S3Compatible - skipCredentialSubscopingIndirection !");
+ clI = System.getenv(storageConfig.getS3CredentialsClientAccessKeyId());
+ clS = System.getenv(storageConfig.getS3CredentialsClientSecretAccessKey());
+ if (clI != null && clS != null) {
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, clI);
+ propertiesMap.put(PolarisCredentialProperty.AWS_SECRET_KEY, clS);
+ } else {
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, caI);
+ propertiesMap.put(PolarisCredentialProperty.AWS_SECRET_KEY, caS);
+ }
+ } else {
+ LOGGER.debug("S3Compatible - assumeRole !");
+ createStsClient(storageConfig);
+ AssumeRoleResponse response =
+ stsClient.assumeRole(
+ AssumeRoleRequest.builder()
+ .roleSessionName("PolarisCredentialsSTS")
+ .roleArn(
+ (storageConfig.getS3RoleArn() == null) ? "" : storageConfig.getS3RoleArn())
+ .policy(
+ policyString(allowListOperation, allowedReadLocations, allowedWriteLocations)
+ .toJson())
+ .build());
+
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, response.credentials().accessKeyId());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_SECRET_KEY, response.credentials().secretAccessKey());
+ propertiesMap.put(PolarisCredentialProperty.AWS_TOKEN, response.credentials().sessionToken());
+ LOGGER.debug(
+ "S3Compatible - assumeRole - Token Expiration at : {}",
+ response.credentials().expiration().toString());
+ }
+
+ return propertiesMap;
+ }
+
+ public void createStsClient(S3CompatibleStorageConfigurationInfo storageConfig) {
+ LOGGER.debug("S3Compatible - createStsClient()");
+ try {
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+ stsBuilder.endpointOverride(URI.create(storageConfig.getS3Endpoint()));
+ if (caI != null && caS != null) {
+ // else default provider build credentials from profile or standard AWS env var
+ stsBuilder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(caI, caS)));
+ }
+ this.stsClient = stsBuilder.build();
+ LOGGER.debug("S3Compatible - stsClient successfully built");
+
+ } catch (Exception e) {
+ System.err.println("S3Compatible - stsClient - build failure : " + e.getMessage());
+ }
+ }
+
+ /*
+ * function from AwsCredentialsStorageIntegration but without roleArn parameter
+ */
+ private IamPolicy policyString(
+ boolean allowList, Set<String> readLocations, Set<String> writeLocations) {
+ IamPolicy.Builder policyBuilder = IamPolicy.builder();
+ IamStatement.Builder allowGetObjectStatementBuilder =
+ IamStatement.builder()
+ .effect(IamEffect.ALLOW)
+ .addAction("s3:GetObject")
+ .addAction("s3:GetObjectVersion");
+ Map<String, IamStatement.Builder> bucketListStatementBuilder = new HashMap<>();
+ Map<String, IamStatement.Builder> bucketGetLocationStatementBuilder = new HashMap<>();
+
+ String arnPrefix = "arn:aws:s3:::";
+ Stream.concat(readLocations.stream(), writeLocations.stream())
+ .distinct()
+ .forEach(
+ location -> {
+ URI uri = URI.create(location);
+ allowGetObjectStatementBuilder.addResource(
+ IamResource.create(
+ arnPrefix + StorageUtil.concatFilePrefixes(parseS3Path(uri), "*", "/")));
+ final var bucket = arnPrefix + StorageUtil.getBucket(uri);
+ if (allowList) {
+ bucketListStatementBuilder
+ .computeIfAbsent(
+ bucket,
+ (String key) ->
+ IamStatement.builder()
+ .effect(IamEffect.ALLOW)
+ .addAction("s3:ListBucket")
+ .addResource(key))
+ .addCondition(
+ IamConditionOperator.STRING_LIKE,
+ "s3:prefix",
+ StorageUtil.concatFilePrefixes(trimLeadingSlash(uri.getPath()), "*", "/"));
+ }
+ bucketGetLocationStatementBuilder.computeIfAbsent(
+ bucket,
+ key ->
+ IamStatement.builder()
+ .effect(IamEffect.ALLOW)
+ .addAction("s3:GetBucketLocation")
+ .addResource(key));
+ });
+
+ if (!writeLocations.isEmpty()) {
+ IamStatement.Builder allowPutObjectStatementBuilder =
+ IamStatement.builder()
+ .effect(IamEffect.ALLOW)
+ .addAction("s3:PutObject")
+ .addAction("s3:DeleteObject");
+ writeLocations.forEach(
+ location -> {
+ URI uri = URI.create(location);
+ allowPutObjectStatementBuilder.addResource(
+ IamResource.create(
+ arnPrefix + StorageUtil.concatFilePrefixes(parseS3Path(uri), "*", "/")));
+ });
+ policyBuilder.addStatement(allowPutObjectStatementBuilder.build());
+ }
+ if (!bucketListStatementBuilder.isEmpty()) {
+ bucketListStatementBuilder
+ .values()
+ .forEach(statementBuilder -> policyBuilder.addStatement(statementBuilder.build()));
+ } else if (allowList) {
+ // add list privilege with 0 resources
+ policyBuilder.addStatement(
+ IamStatement.builder().effect(IamEffect.ALLOW).addAction("s3:ListBucket").build());
+ }
+
+ bucketGetLocationStatementBuilder
+ .values()
+ .forEach(statementBuilder -> policyBuilder.addStatement(statementBuilder.build()));
+ return policyBuilder.addStatement(allowGetObjectStatementBuilder.build()).build();
+ }
+
+ /* function from AwsCredentialsStorageIntegration */
+ private static @Nonnull String parseS3Path(URI uri) {
+ String bucket = StorageUtil.getBucket(uri);
+ String path = trimLeadingSlash(uri.getPath());
+ return String.join("/", bucket, path);
+ }
+
+ /* function from AwsCredentialsStorageIntegration */
+ private static @Nonnull String trimLeadingSlash(String path) {
+ if (path.startsWith("/")) {
+ path = path.substring(1);
+ }
+ return path;
+ }
Review Comment:
I have avoided to touch aws storage code... Maybe to refactor after the PR merge ? (If merged)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1953560872
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleStorageConfigurationInfo.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import java.util.List;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Polaris Storage Configuration information for an S3 Compatible solution, MinIO, Ceph, Dell ECS...
+ */
+public class S3CompatibleStorageConfigurationInfo extends PolarisStorageConfigurationInfo {
+
+ // 5 is the approximate max allowed locations for the size of AccessPolicy when LIST is required
+ // for allowed read and write locations for subscoping creds.
+ @JsonIgnore private static final int MAX_ALLOWED_LOCATIONS = 5;
+ private @NotNull String s3Endpoint;
+ private @Nullable String s3CredentialsCatalogAccessKeyId;
+ private @Nullable String s3CredentialsCatalogSecretAccessKey;
+ private @Nullable Boolean s3PathStyleAccess;
+ private @NotNull Boolean skipCredentialSubscopingIndirection;
+ private @Nullable String s3CredentialsClientAccessKeyId;
+ private @Nullable String s3CredentialsClientSecretAccessKey;
+ private @Nullable String s3Region;
+ private @Nullable String s3RoleArn;
Review Comment:
ok
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleStorageConfigurationInfo.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import java.util.List;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Polaris Storage Configuration information for an S3 Compatible solution, MinIO, Ceph, Dell ECS...
+ */
+public class S3CompatibleStorageConfigurationInfo extends PolarisStorageConfigurationInfo {
+
+ // 5 is the approximate max allowed locations for the size of AccessPolicy when LIST is required
+ // for allowed read and write locations for subscoping creds.
+ @JsonIgnore private static final int MAX_ALLOWED_LOCATIONS = 5;
+ private @NotNull String s3Endpoint;
+ private @Nullable String s3CredentialsCatalogAccessKeyId;
+ private @Nullable String s3CredentialsCatalogSecretAccessKey;
+ private @Nullable Boolean s3PathStyleAccess;
+ private @NotNull Boolean skipCredentialSubscopingIndirection;
+ private @Nullable String s3CredentialsClientAccessKeyId;
+ private @Nullable String s3CredentialsClientSecretAccessKey;
+ private @Nullable String s3Region;
+ private @Nullable String s3RoleArn;
+
+ // Constructor
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1953589374
##########
spec/polaris-management-service.yml:
##########
@@ -905,6 +907,51 @@ components:
required:
- roleArn
+ S3CompatibleStorageConfigInfo:
+ type: object
+ description: S3 compatible storage configuration info (MinIO, Ceph, Dell ECS, Netapp StorageGRID, ...)
+ allOf:
+ - $ref: '#/components/schemas/StorageConfigInfo'
+ properties:
+ s3.endpoint:
+ type: string
+ description: the S3 endpoint, will also be used as STS endpoint
+ example: "http[s]://host:port"
+ s3.credentials.catalog.accessKeyId:
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "Gerrit-K (via GitHub)" <gi...@apache.org>.
Gerrit-K commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2598433602
> This is the mode where it vends a static client credential?
Yes, when `skipCredentialSubscopingIndirection: false` is issues temporary STS tokens and when `skipCredentialSubscopingIndirection: true` it just responds with the credentials that are set in the catalog configuration, i.e. `s3.credentials.client.*` or `s3.credentials.catalog.*` if the former is not set. I was unfortunately only able to validate the latter case due to our Ceph setup that currently doesn't enable STS. Appreciate your insights, but no rush! 🙏
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1953597985
##########
spec/polaris-management-service.yml:
##########
@@ -905,6 +907,51 @@ components:
required:
- roleArn
+ S3CompatibleStorageConfigInfo:
+ type: object
+ description: S3 compatible storage configuration info (MinIO, Ceph, Dell ECS, Netapp StorageGRID, ...)
+ allOf:
+ - $ref: '#/components/schemas/StorageConfigInfo'
+ properties:
+ s3.endpoint:
+ type: string
+ description: the S3 endpoint, will also be used as STS endpoint
+ example: "http[s]://host:port"
+ s3.credentials.catalog.accessKeyId:
+ type: string
+ description: Default to AWS credentials, otherwise set the environement variable name for the 'ACCESS_KEY_ID' used by the catalog to communicate with S3
+ example: "CATALOG_1_ACCESS_KEY_ENV_VARIABLE_NAME or AWS_ACCESS_KEY_ID"
Review Comment:
Yes if catalog creator want to use specific keys he have to be aware of the name of the variables inside Polaris service environnement.
Otherwise the creator let the key parameters empty, "cross fingers", and let default s3 sdk use whatever is available in polaris service environement.
I have not find a better solution. Passing values will leak secret in logs and in catalog "get response" :-(
Upper in this code review Collado-Mike suggest to switch to AWS Profile (I like this proposition). But the catalog creator still have to be aware of the name of the profile...
=> Need agreement between Polaris service deployment administrator and catalog creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "omarsmak (via GitHub)" <gi...@apache.org>.
omarsmak commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1989653032
##########
polaris-core/src/main/java/org/apache/polaris/core/entity/CatalogEntity.java:
##########
@@ -250,6 +268,21 @@ public Builder setStorageConfigurationInfo(
awsConfig.validateArn(awsConfigModel.getRoleArn());
config = awsConfig;
break;
+
+ case S3_COMPATIBLE:
+ S3CompatibleStorageConfigInfo s3ConfigModel =
+ (S3CompatibleStorageConfigInfo) storageConfigModel;
+ config =
+ new S3CompatibleStorageConfigurationInfo(
Review Comment:
got it. Thanks for the info :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "omarsmak (via GitHub)" <gi...@apache.org>.
omarsmak commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1989654008
##########
polaris-core/src/main/java/org/apache/polaris/core/entity/CatalogEntity.java:
##########
@@ -250,6 +268,21 @@ public Builder setStorageConfigurationInfo(
awsConfig.validateArn(awsConfigModel.getRoleArn());
config = awsConfig;
break;
+
+ case S3_COMPATIBLE:
Review Comment:
Thanks for thew info. I am fine with it as it is as long we do have a plan to unify it at some point
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "omarsmak (via GitHub)" <gi...@apache.org>.
omarsmak commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2711404557
@lefebsy are we merging this PR soon or there is still some outstanding work?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "lefebsy (via GitHub)" <gi...@apache.org>.
lefebsy commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2711530567
> Hi @lefebsy, let me know once the PR is ready for another look. As I said here, [#389 (comment)](https://github.com/apache/polaris/pull/389#discussion_r1987787270), this PR isn't blocked by other PRs.
It is ready for review :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1987893831
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/PolarisStorageConfigurationInfo.java:
##########
@@ -62,6 +63,7 @@
@JsonTypeInfo(use = JsonTypeInfo.Id.NAME)
@JsonSubTypes({
@JsonSubTypes.Type(value = AwsStorageConfigurationInfo.class),
+ @JsonSubTypes.Type(value = S3CompatibleStorageConfigurationInfo.class),
Review Comment:
What is the reason to model non-AWS S3 storage as a separate config info class as opposed to adding a few optional settings to `AwsStorageConfigurationInfo`?
My understanding is that the same AWS SDK is used for actually access anyway, so storage config info basically goes into the same AWS client, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1989558277
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/PolarisStorageConfigurationInfo.java:
##########
@@ -62,6 +63,7 @@
@JsonTypeInfo(use = JsonTypeInfo.Id.NAME)
@JsonSubTypes({
@JsonSubTypes.Type(value = AwsStorageConfigurationInfo.class),
+ @JsonSubTypes.Type(value = S3CompatibleStorageConfigurationInfo.class),
Review Comment:
Fair enough... I'll review closer soon :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1980789090
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import static org.apache.polaris.core.PolarisConfiguration.STORAGE_CREDENTIAL_DURATION_SECONDS;
+import static org.apache.polaris.core.PolarisConfiguration.loadConfig;
+
+import jakarta.ws.rs.NotAuthorizedException;
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.profiles.ProfileFileSupplier;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** S3 compatible implementation of PolarisStorageIntegration */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ @Override
+ public EnumMap<PolarisCredentialProperty, String> getSubscopedCreds(
+ @NotNull PolarisDiagnostics diagnostics,
+ @NotNull S3CompatibleStorageConfigurationInfo storageConfig,
+ boolean allowListOperation,
+ @NotNull Set<String> allowedReadLocations,
+ @NotNull Set<String> allowedWriteLocations) {
+
+ String caI = System.getenv(storageConfig.getS3CredentialsCatalogAccessKeyId());
+ String caS = System.getenv(storageConfig.getS3CredentialsCatalogSecretAccessKey());
+
+ EnumMap<PolarisCredentialProperty, String> propertiesMap =
+ new EnumMap<>(PolarisCredentialProperty.class);
+ propertiesMap.put(PolarisCredentialProperty.AWS_ENDPOINT, storageConfig.getS3Endpoint());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_PATH_STYLE_ACCESS,
+ storageConfig.getS3PathStyleAccess().toString());
+ if (storageConfig.getS3Region() != null) {
+ propertiesMap.put(PolarisCredentialProperty.CLIENT_REGION, storageConfig.getS3Region());
+ }
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+ stsBuilder.endpointOverride(URI.create(storageConfig.getS3Endpoint()));
+ if (storageConfig.getS3ProfileName() != null) {
+ stsBuilder.credentialsProvider(
+ ProfileCredentialsProvider.builder()
+ .profileFile(ProfileFileSupplier.defaultSupplier())
+ .profileName(storageConfig.getS3ProfileName())
+ .build());
+ LOGGER.debug("S3Compatible - stsClient using profile from catalog settings");
+ } else if (caI != null && caS != null) {
+ stsBuilder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(caI, caS)));
+ LOGGER.debug("S3Compatible - stsClient using keys from catalog settings");
+ }
+ try (StsClient stsClient = stsBuilder.build()) {
+ LOGGER.debug("S3Compatible - stsClient successfully built");
+ AssumeRoleResponse response =
+ stsClient.assumeRole(
+ AssumeRoleRequest.builder()
+ .roleSessionName("PolarisCredentialsSTS")
+ .roleArn(storageConfig.getS3RoleArn())
+ .policy(
+ StorageUtil.policyString(
+ storageConfig.getS3RoleArn(),
+ allowListOperation,
+ allowedReadLocations,
+ allowedWriteLocations)
+ .toJson())
+ .durationSeconds(loadConfig(STORAGE_CREDENTIAL_DURATION_SECONDS))
+ .build());
+
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, response.credentials().accessKeyId());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_SECRET_KEY, response.credentials().secretAccessKey());
+ propertiesMap.put(PolarisCredentialProperty.AWS_TOKEN, response.credentials().sessionToken());
+ LOGGER.debug(
+ "S3Compatible - assumeRole - Obtained token expiration : {}",
+ response.credentials().expiration().toString());
+ } catch (Exception e) {
+ throw new NotAuthorizedException(
+ "Unable to build S3 Security Token Service client - " + e.getMessage());
Review Comment:
```suggestion
throw new NotAuthorizedException(
"Unable to build S3 Security Token Service client", e);
```
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleStorageConfigurationInfo.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import java.util.List;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Polaris Storage Configuration information for an S3 Compatible solution, MinIO, Ceph, Dell ECS...
+ */
+public class S3CompatibleStorageConfigurationInfo extends PolarisStorageConfigurationInfo {
+
+ // 5 is the approximate max allowed locations for the size of AccessPolicy when LIST is required
+ // for allowed read and write locations for subscoping creds.
+ @JsonIgnore private static final int MAX_ALLOWED_LOCATIONS = 5;
+ private @NotNull String s3Endpoint;
+ private @Nullable String s3CredentialsCatalogAccessKeyId;
+ private @Nullable String s3CredentialsCatalogSecretAccessKey;
+ private @Nullable Boolean s3PathStyleAccess;
+ private @NotNull Boolean skipCredentialSubscopingIndirection;
+ private @Nullable String s3CredentialsClientAccessKeyId;
+ private @Nullable String s3CredentialsClientSecretAccessKey;
+ private @Nullable String s3Region;
+ private @Nullable String s3RoleArn;
+
+ // Constructor
+ @JsonCreator
+ public S3CompatibleStorageConfigurationInfo(
+ @JsonProperty(value = "storageType", required = true) @NotNull StorageType storageType,
+ @JsonProperty(value = "s3Endpoint", required = true) @NotNull String s3Endpoint,
+ @JsonProperty(value = "s3CredentialsCatalogAccessKeyId", required = true) @NotNull
+ String s3CredentialsCatalogAccessKeyId,
+ @JsonProperty(value = "s3CredentialsCatalogSecretAccessKey", required = true) @NotNull
+ String s3CredentialsCatalogSecretAccessKey,
+ @JsonProperty(value = "SkipCredentialSubscopingIndirection", required = false) @Nullable
+ Boolean skipCredentialSubscopingIndirection,
+ @JsonProperty(value = "s3CredentialsClientAccessKeyId", required = false) @Nullable
+ String s3CredentialsClientAccessKeyId,
+ @JsonProperty(value = "s3CredentialsClientSecretAccessKey", required = false) @Nullable
+ String s3CredentialsClientSecretAccessKey,
+ @JsonProperty(value = "s3PathStyleAccess", required = false) @Nullable
+ Boolean s3PathStyleAccess,
+ @JsonProperty(value = "s3Region", required = false) @Nullable String s3Region,
+ @JsonProperty(value = "s3RoleArn", required = false) @Nullable String s3RoleArn,
+ @JsonProperty(value = "allowedLocations", required = true) @Nullable
+ List<String> allowedLocations) {
+
+ // storing properties
+ super(storageType, allowedLocations);
+ validateMaxAllowedLocations(MAX_ALLOWED_LOCATIONS);
+ this.s3PathStyleAccess = s3PathStyleAccess;
Review Comment:
I refer to the field(`s3PathStyleAccess`) in this class, which can be set to `null`. In that case, `storageConfig.getS3PathStyleAccess().toString()` will throw a NPE.
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import static org.apache.polaris.core.PolarisConfiguration.STORAGE_CREDENTIAL_DURATION_SECONDS;
+import static org.apache.polaris.core.PolarisConfiguration.loadConfig;
+
+import jakarta.ws.rs.NotAuthorizedException;
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.profiles.ProfileFileSupplier;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** S3 compatible implementation of PolarisStorageIntegration */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ @Override
+ public EnumMap<PolarisCredentialProperty, String> getSubscopedCreds(
+ @NotNull PolarisDiagnostics diagnostics,
+ @NotNull S3CompatibleStorageConfigurationInfo storageConfig,
+ boolean allowListOperation,
+ @NotNull Set<String> allowedReadLocations,
+ @NotNull Set<String> allowedWriteLocations) {
+
+ String caI = System.getenv(storageConfig.getS3CredentialsCatalogAccessKeyId());
+ String caS = System.getenv(storageConfig.getS3CredentialsCatalogSecretAccessKey());
+
+ EnumMap<PolarisCredentialProperty, String> propertiesMap =
+ new EnumMap<>(PolarisCredentialProperty.class);
+ propertiesMap.put(PolarisCredentialProperty.AWS_ENDPOINT, storageConfig.getS3Endpoint());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_PATH_STYLE_ACCESS,
+ storageConfig.getS3PathStyleAccess().toString());
+ if (storageConfig.getS3Region() != null) {
+ propertiesMap.put(PolarisCredentialProperty.CLIENT_REGION, storageConfig.getS3Region());
+ }
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
+ stsBuilder.endpointOverride(URI.create(storageConfig.getS3Endpoint()));
+ if (storageConfig.getS3ProfileName() != null) {
+ stsBuilder.credentialsProvider(
+ ProfileCredentialsProvider.builder()
+ .profileFile(ProfileFileSupplier.defaultSupplier())
+ .profileName(storageConfig.getS3ProfileName())
+ .build());
+ LOGGER.debug("S3Compatible - stsClient using profile from catalog settings");
+ } else if (caI != null && caS != null) {
+ stsBuilder.credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create(caI, caS)));
+ LOGGER.debug("S3Compatible - stsClient using keys from catalog settings");
+ }
+ try (StsClient stsClient = stsBuilder.build()) {
+ LOGGER.debug("S3Compatible - stsClient successfully built");
+ AssumeRoleResponse response =
+ stsClient.assumeRole(
+ AssumeRoleRequest.builder()
+ .roleSessionName("PolarisCredentialsSTS")
+ .roleArn(storageConfig.getS3RoleArn())
+ .policy(
+ StorageUtil.policyString(
+ storageConfig.getS3RoleArn(),
+ allowListOperation,
+ allowedReadLocations,
+ allowedWriteLocations)
+ .toJson())
+ .durationSeconds(loadConfig(STORAGE_CREDENTIAL_DURATION_SECONDS))
+ .build());
+
+ propertiesMap.put(PolarisCredentialProperty.AWS_KEY_ID, response.credentials().accessKeyId());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_SECRET_KEY, response.credentials().secretAccessKey());
+ propertiesMap.put(PolarisCredentialProperty.AWS_TOKEN, response.credentials().sessionToken());
Review Comment:
Nit: using static import so that we can keep them in one line each.
```suggestion
propertiesMap.put(AWS_KEY_ID, response.credentials().accessKeyId());
propertiesMap.put(AWS_SECRET_KEY, response.credentials().secretAccessKey());
propertiesMap.put(AWS_TOKEN, response.credentials().sessionToken());
```
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleStorageConfigurationInfo.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import jakarta.annotation.Nonnull;
+import java.util.List;
+import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
+import org.jetbrains.annotations.NotNull;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * S3-Compatible Storage Configuration. This class holds the parameters needed to connect to
+ * S3-compatible storage services such as MinIO, Ceph, Dell ECS, etc.
+ */
+public class S3CompatibleStorageConfigurationInfo extends PolarisStorageConfigurationInfo {
+
+ // 5 is the approximate max allowed locations for the size of AccessPolicy when LIST is required
+ // for allowed read and write locations for sub-scoping credentials.
+ @JsonIgnore private static final int MAX_ALLOWED_LOCATIONS = 5;
+ private final @NotNull String s3Endpoint;
+ private final @Nullable String s3ProfileName;
+ private final @Nullable String s3CredentialsCatalogAccessKeyId;
+ private final @Nullable String s3CredentialsCatalogSecretAccessKey;
+ private final @NotNull Boolean s3PathStyleAccess;
+ private final @Nullable String s3Region;
+ private final @Nullable String s3RoleArn;
+
+ @JsonCreator
+ public S3CompatibleStorageConfigurationInfo(
+ @JsonProperty(value = "storageType", required = true) @NotNull StorageType storageType,
Review Comment:
This is not necessary, as line 64 hard codes the `StorageType`.
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.storage.s3compatible;
+
+import static org.apache.polaris.core.PolarisConfiguration.STORAGE_CREDENTIAL_DURATION_SECONDS;
+import static org.apache.polaris.core.PolarisConfiguration.loadConfig;
+
+import jakarta.ws.rs.NotAuthorizedException;
+import java.net.URI;
+import java.util.EnumMap;
+import java.util.Set;
+import org.apache.polaris.core.PolarisDiagnostics;
+import org.apache.polaris.core.storage.InMemoryStorageIntegration;
+import org.apache.polaris.core.storage.PolarisCredentialProperty;
+import org.apache.polaris.core.storage.StorageUtil;
+import org.jetbrains.annotations.NotNull;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.profiles.ProfileFileSupplier;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** S3 compatible implementation of PolarisStorageIntegration */
+public class S3CompatibleCredentialsStorageIntegration
+ extends InMemoryStorageIntegration<S3CompatibleStorageConfigurationInfo> {
+
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(S3CompatibleCredentialsStorageIntegration.class);
+
+ public S3CompatibleCredentialsStorageIntegration() {
+ super(S3CompatibleCredentialsStorageIntegration.class.getName());
+ }
+
+ @Override
+ public EnumMap<PolarisCredentialProperty, String> getSubscopedCreds(
+ @NotNull PolarisDiagnostics diagnostics,
+ @NotNull S3CompatibleStorageConfigurationInfo storageConfig,
+ boolean allowListOperation,
+ @NotNull Set<String> allowedReadLocations,
+ @NotNull Set<String> allowedWriteLocations) {
+
+ String caI = System.getenv(storageConfig.getS3CredentialsCatalogAccessKeyId());
+ String caS = System.getenv(storageConfig.getS3CredentialsCatalogSecretAccessKey());
+
+ EnumMap<PolarisCredentialProperty, String> propertiesMap =
+ new EnumMap<>(PolarisCredentialProperty.class);
+ propertiesMap.put(PolarisCredentialProperty.AWS_ENDPOINT, storageConfig.getS3Endpoint());
+ propertiesMap.put(
+ PolarisCredentialProperty.AWS_PATH_STYLE_ACCESS,
+ storageConfig.getS3PathStyleAccess().toString());
+ if (storageConfig.getS3Region() != null) {
+ propertiesMap.put(PolarisCredentialProperty.CLIENT_REGION, storageConfig.getS3Region());
+ }
+
+ LOGGER.debug("S3Compatible - createStsClient()");
+ StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();
Review Comment:
Nit: `software.amazon.awssdk.services.sts.StsClient.builder();` -> `StsClient.builder();`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "thotz (via GitHub)" <gi...@apache.org>.
thotz commented on PR #389:
URL: https://github.com/apache/polaris/pull/389#issuecomment-2791794488
@lefebsy @flyrain I can see this feature is targeted for the 1.0 release on April 15th. Can we expect this change to be available by then?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #389:
URL: https://github.com/apache/polaris/pull/389#discussion_r1987893831
##########
polaris-core/src/main/java/org/apache/polaris/core/storage/PolarisStorageConfigurationInfo.java:
##########
@@ -62,6 +63,7 @@
@JsonTypeInfo(use = JsonTypeInfo.Id.NAME)
@JsonSubTypes({
@JsonSubTypes.Type(value = AwsStorageConfigurationInfo.class),
+ @JsonSubTypes.Type(value = S3CompatibleStorageConfigurationInfo.class),
Review Comment:
What is the reason to model non-AWS S3 storage as a separate config info class as opposed to adding a few optional settings to `AwsStorageConfigurationInfo`?
My understanding is that the same AWS SDK is used for actual access anyway, so storage config info basically goes into the same AWS client, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org