You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "CalvinKirs (via GitHub)" <gi...@apache.org> on 2026/05/14 10:08:14 UTC
[PR] [refactor](fe) Add typed filesystem properties for S3 [doris]
CalvinKirs opened a new pull request, #63252:
URL: https://github.com/apache/doris/pull/63252
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Add provider-owned typed filesystem properties and bind S3 runtime paths through S3FileSystemProperties, preparing for S3Properties removal from fe-core.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- mvn -Dmaven.build.cache.enabled=false -f fe/pom.xml -pl fe-filesystem/fe-filesystem-s3 -am -DfailIfNoTests=false test
- mvn -Dmaven.build.cache.enabled=false -f fe/pom.xml -pl fe-filesystem/fe-filesystem-oss,fe-filesystem/fe-filesystem-cos,fe-filesystem/fe-filesystem-obs -am -DfailIfNoTests=false test
- mvn -Dmaven.build.cache.enabled=false -f fe/pom.xml -pl fe-filesystem/fe-filesystem-s3,fe-filesystem/fe-filesystem-oss,fe-filesystem/fe-filesystem-cos,fe-filesystem/fe-filesystem-obs -am -DfailIfNoTests=false test
- mvn -Dmaven.build.cache.enabled=false -f fe/pom.xml -pl fe-filesystem/fe-filesystem-s3 -am -DskipTests validate
- env MAVEN_OPTS=-Xmx5g MVN_OPT=-Dmaven.build.cache.enabled=false FE_MAVEN_THREADS=1 FE_MAVEN_RETRY_THREADS=1 CUSTOM_UI_DIST=/mnt/disk1/gq/idea/incubator-doris/ui/dist ./build.sh --fe
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](fe) Add typed filesystem properties for S3 [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63252:
URL: https://github.com/apache/doris/pull/63252#issuecomment-4450651148
# FE Regression Coverage Report
Increment line coverage ` 0.00% (0/125)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63252_bf360bb94b28d58c61db4accf1741dc1378fb21e_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63252_bf360bb94b28d58c61db4accf1741dc1378fb21e_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](fe) Add typed filesystem properties for S3 [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63252:
URL: https://github.com/apache/doris/pull/63252#issuecomment-4458227013
<details>
<summary>TPC-H: <b>Total hot run time: 31065 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 28929d52cbeaa73d69f4a741de1177c3cf922ffb, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17630 3871 3861 3861
q2 q3 10767 1336 811 811
q4 4686 476 352 352
q5 7586 2258 2126 2126
q6 233 176 138 138
q7 936 777 624 624
q8 9401 1781 1712 1712
q9 5099 4886 4935 4886
q10 6393 2087 1781 1781
q11 437 282 243 243
q12 635 420 293 293
q13 18144 3366 2781 2781
q14 259 258 232 232
q15 q16 820 786 706 706
q17 917 937 963 937
q18 7077 5779 5580 5580
q19 1278 1303 1068 1068
q20 541 400 264 264
q21 5857 2503 2377 2377
q22 450 355 293 293
Total cold run time: 99146 ms
Total hot run time: 31065 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4187 4081 4087 4081
q2 q3 4493 4931 4314 4314
q4 2143 2225 1378 1378
q5 4388 4279 4293 4279
q6 232 171 129 129
q7 1734 2121 1774 1774
q8 2561 2150 2125 2125
q9 7823 8142 7663 7663
q10 4570 4512 4098 4098
q11 553 398 359 359
q12 743 729 542 542
q13 3344 3599 2961 2961
q14 288 295 277 277
q15 q16 724 732 654 654
q17 1313 1288 1302 1288
q18 8174 7274 7330 7274
q19 1161 1129 1138 1129
q20 2217 2224 1940 1940
q21 5286 4547 4379 4379
q22 520 449 396 396
Total cold run time: 56454 ms
Total hot run time: 51040 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](fe) Add typed filesystem properties for S3 [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63252:
URL: https://github.com/apache/doris/pull/63252#discussion_r3245372588
##########
fe/fe-filesystem/fe-filesystem-s3/src/main/java/org/apache/doris/filesystem/s3/S3CredentialsProviderFactory.java:
##########
@@ -0,0 +1,198 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.filesystem.s3;
+
+import org.apache.commons.lang3.StringUtils;
+import software.amazon.awssdk.auth.credentials.AnonymousCredentialsProvider;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
+import software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain;
+import software.amazon.awssdk.auth.credentials.AwsSessionCredentials;
+import software.amazon.awssdk.auth.credentials.ContainerCredentialsProvider;
+import software.amazon.awssdk.auth.credentials.EnvironmentVariableCredentialsProvider;
+import software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.auth.credentials.SystemPropertyCredentialsProvider;
+import software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sts.StsClient;
+import software.amazon.awssdk.services.sts.auth.StsAssumeRoleCredentialsProvider;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.UUID;
+import java.util.function.BiFunction;
+
+/**
+ * Credential provider factory for S3 filesystem AWS SDK v2 clients.
+ */
+public final class S3CredentialsProviderFactory {
+
+ private S3CredentialsProviderFactory() {
+ }
+
+ public static AwsCredentialsProvider createClientProvider(S3FileSystemProperties properties) {
+ return createClientProvider(properties, S3CredentialsProviderFactory::buildStsClient);
+ }
+
+ public static AwsCredentialsProvider createClientProvider(S3FileSystemProperties properties,
+ BiFunction<AwsCredentialsProvider, String, StsClient> stsClientFactory) {
+ if (properties.hasAssumeRole()) {
+ return createAssumeRoleProvider(properties, stsClientFactory);
+ }
+ return createBaseProvider(properties, true);
+ }
+
+ public static AwsCredentialsProvider createStsSourceProvider(S3FileSystemProperties properties) {
+ return createBaseProvider(properties, false);
+ }
+
+ public static AwsCredentialsProvider create(
+ S3CredentialsProviderType type, boolean includeAnonymousInDefault) {
+ switch (type) {
+ case ENV:
+ return EnvironmentVariableCredentialsProvider.create();
+ case SYSTEM_PROPERTIES:
+ return SystemPropertyCredentialsProvider.create();
+ case WEB_IDENTITY:
+ return WebIdentityTokenFileCredentialsProvider.create();
+ case CONTAINER:
+ return ContainerCredentialsProvider.create();
+ case INSTANCE_PROFILE:
+ return InstanceProfileCredentialsProvider.create();
+ case ANONYMOUS:
+ return AnonymousCredentialsProvider.create();
+ case DEFAULT:
+ return createDefault(includeAnonymousInDefault);
+ default:
+ throw new UnsupportedOperationException(
+ "AWS SDK V2 does not support credentials provider mode: " + type);
+ }
+ }
+
+ public static String hadoopClassName(
+ S3CredentialsProviderType type, boolean includeAnonymousInDefault) {
+ switch (type) {
+ case ENV:
+ return EnvironmentVariableCredentialsProvider.class.getName();
+ case SYSTEM_PROPERTIES:
+ return SystemPropertyCredentialsProvider.class.getName();
+ case WEB_IDENTITY:
+ return WebIdentityTokenFileCredentialsProvider.class.getName();
+ case CONTAINER:
+ return ContainerCredentialsProvider.class.getName();
+ case INSTANCE_PROFILE:
+ return InstanceProfileCredentialsProvider.class.getName();
+ case ANONYMOUS:
+ return AnonymousCredentialsProvider.class.getName();
+ case DEFAULT:
+ List<String> providers = new ArrayList<>();
+ providers.add(EnvironmentVariableCredentialsProvider.class.getName());
+ providers.add(SystemPropertyCredentialsProvider.class.getName());
+ providers.add(InstanceProfileCredentialsProvider.class.getName());
+ if (isWebIdentityConfigured()) {
+ providers.add(WebIdentityTokenFileCredentialsProvider.class.getName());
+ }
+ if (isContainerCredentialsConfigured()) {
+ providers.add(ContainerCredentialsProvider.class.getName());
+ }
+ if (includeAnonymousInDefault) {
+ providers.add(AnonymousCredentialsProvider.class.getName());
+ }
+ return String.join(",", providers);
+ default:
+ throw new UnsupportedOperationException(
+ "AWS SDK V2 does not support credentials provider mode: " + type);
+ }
+ }
+
+ private static AwsCredentialsProvider createAssumeRoleProvider(S3FileSystemProperties properties,
+ BiFunction<AwsCredentialsProvider, String, StsClient> stsClientFactory) {
+ StsClient stsClient = stsClientFactory.apply(createStsSourceProvider(properties), properties.getRegion());
+ return StsAssumeRoleCredentialsProvider.builder()
+ .stsClient(stsClient)
+ .refreshRequest(builder -> {
+ builder.roleArn(properties.getRoleArn())
+ .roleSessionName("doris_" + UUID.randomUUID().toString().replace("-", ""));
+ String externalId = properties.getExternalId();
+ if (StringUtils.isNotBlank(externalId)) {
+ builder.externalId(externalId);
+ }
+ }).build();
+ }
+
+ private static StsClient buildStsClient(AwsCredentialsProvider credentialsProvider, String region) {
+ return StsClient.builder()
+ .credentialsProvider(credentialsProvider)
+ .region(Region.of(region))
+ .build();
+ }
+
+ private static AwsCredentialsProvider createBaseProvider(S3FileSystemProperties properties,
+ boolean includeAnonymousInDefault) {
+ AwsCredentialsProvider staticProvider = createStaticProvider(
+ properties.getAccessKey(),
+ properties.getSecretKey(),
+ properties.getSessionToken());
+ if (staticProvider != null) {
+ return staticProvider;
+ }
+ return create(properties.getCredentialsProviderType(), includeAnonymousInDefault);
+ }
+
+ private static AwsCredentialsProvider createStaticProvider(String accessKey, String secretKey,
+ String sessionToken) {
+ if (StringUtils.isBlank(accessKey) || StringUtils.isBlank(secretKey)) {
+ return null;
+ }
+ if (StringUtils.isNotBlank(sessionToken)) {
+ return StaticCredentialsProvider.create(
+ AwsSessionCredentials.create(accessKey, secretKey, sessionToken));
+ }
+ return StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey));
+ }
+
+ private static AwsCredentialsProvider createDefault(boolean includeAnonymous) {
Review Comment:
This changes the meaning of the default credentials mode. The old S3ObjStorage fallback used `DefaultCredentialsProvider.create()` before anonymous fallback, so AWS profile credentials, process credentials, and any other providers in the SDK's default chain worked. This hand-built chain omits those providers, so an existing deployment that relies on `~/.aws/credentials` / `AWS_PROFILE` and does not set AK/SK will now fail to resolve credentials. Please keep the SDK default provider in the DEFAULT path (and only append anonymous when requested) instead of replacing it with a partial chain.
##########
fe/fe-filesystem/fe-filesystem-s3/src/main/java/org/apache/doris/filesystem/s3/S3FileSystemProperties.java:
##########
@@ -0,0 +1,398 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.filesystem.s3;
+
+import org.apache.doris.filesystem.FileSystemType;
+import org.apache.doris.filesystem.properties.BackendStorageKind;
+import org.apache.doris.filesystem.properties.BackendStorageProperties;
+import org.apache.doris.filesystem.properties.FileSystemProperties;
+import org.apache.doris.filesystem.properties.HadoopStorageProperties;
+import org.apache.doris.filesystem.properties.S3CompatibleFileSystemProperties;
+import org.apache.doris.filesystem.properties.StorageKind;
+import org.apache.doris.foundation.property.ConnectorPropertiesUtils;
+import org.apache.doris.foundation.property.ConnectorProperty;
+import org.apache.doris.foundation.property.ParamRules;
+
+import lombok.Getter;
+import org.apache.commons.lang3.StringUtils;
+
+import java.lang.reflect.Field;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * Provider-owned typed properties for AWS S3 and S3-compatible object storage.
+ * Binding uses {@link ConnectorProperty} aliases so legacy key names can continue
+ * to work while callers migrate to canonical s3.* names.
+ */
+public final class S3FileSystemProperties
+ implements FileSystemProperties, BackendStorageProperties, HadoopStorageProperties,
+ S3CompatibleFileSystemProperties {
+
+ public static final String ENDPOINT = "s3.endpoint";
+ public static final String REGION = "s3.region";
+ public static final String ACCESS_KEY = "s3.access_key";
+ public static final String SECRET_KEY = "s3.secret_key";
+ public static final String SESSION_TOKEN = "s3.session_token";
+ public static final String ROLE_ARN = "s3.role_arn";
+ public static final String EXTERNAL_ID = "s3.external_id";
+ public static final String BUCKET = "s3.bucket";
+ public static final String ROOT_PATH = "s3.root.path";
+ public static final String MAX_CONNECTIONS = "s3.connection.maximum";
+ public static final String REQUEST_TIMEOUT_MS = "s3.connection.request.timeout";
+ public static final String CONNECTION_TIMEOUT_MS = "s3.connection.timeout";
+ public static final String USE_PATH_STYLE = "use_path_style";
+ public static final String CREDENTIALS_PROVIDER_TYPE = "s3.credentials_provider_type";
+
+ public static final String DEFAULT_MAX_CONNECTIONS = "50";
+ public static final String DEFAULT_REQUEST_TIMEOUT_MS = "3000";
+ public static final String DEFAULT_CONNECTION_TIMEOUT_MS = "1000";
+ public static final String DEFAULT_CREDENTIALS_PROVIDER_TYPE = "DEFAULT";
+
+ private static final Pattern[] ENDPOINT_PATTERNS = new Pattern[] {
+ Pattern.compile(
+ "^(?:https?://)?(?:"
+ + "s3(?:[-.]fips)?(?:[-.]dualstack)?[-.]([a-z0-9-]+)|"
+ + "s3express-control\\.([a-z0-9-]+)|"
+ + "s3express-[a-z0-9-]+\\.([a-z0-9-]+)"
+ + ")\\.amazonaws\\.com(?:/.*)?$",
+ Pattern.CASE_INSENSITIVE),
+ Pattern.compile(
+ "^(?:https?://)?glue(?:-fips)?\\.([a-z0-9-]+)\\.(amazonaws\\.com(?:\\.cn)?|api\\.aws)$",
+ Pattern.CASE_INSENSITIVE)
+ };
+
+ @Getter
+ @ConnectorProperty(names = {ENDPOINT, "AWS_ENDPOINT", "endpoint", "ENDPOINT", "aws.endpoint",
+ "glue.endpoint", "aws.glue.endpoint"},
+ required = false,
+ description = "The endpoint of S3.")
+ private String endpoint = "";
+
+ @Getter
+ @ConnectorProperty(names = {REGION, "AWS_REGION", "region", "REGION", "aws.region", "glue.region",
+ "aws.glue.region", "iceberg.rest.signing-region", "rest.signing-region", "client.region"},
+ required = false,
+ isRegionField = true,
+ description = "The region of S3.")
+ private String region = "";
+
+ @Getter
+ @ConnectorProperty(names = {ACCESS_KEY, "AWS_ACCESS_KEY", "access_key", "ACCESS_KEY",
+ "glue.access_key", "aws.glue.access-key",
+ "client.credentials-provider.glue.access_key", "iceberg.rest.access-key-id",
+ "s3.access-key-id"},
+ required = false,
+ sensitive = true,
+ description = "The access key of S3.")
+ private String accessKey = "";
+
+ @Getter
+ @ConnectorProperty(names = {SECRET_KEY, "AWS_SECRET_KEY", "secret_key", "SECRET_KEY",
+ "glue.secret_key", "aws.glue.secret-key",
+ "client.credentials-provider.glue.secret_key", "iceberg.rest.secret-access-key",
+ "s3.secret-access-key"},
+ required = false,
+ sensitive = true,
+ description = "The secret key of S3.")
+ private String secretKey = "";
+
+ @Getter
+ @ConnectorProperty(names = {SESSION_TOKEN, "AWS_TOKEN", "session_token",
+ "s3.session-token", "iceberg.rest.session-token"},
+ required = false,
+ description = "The session token of S3.")
+ private String sessionToken = "";
+
+ @Getter
+ @ConnectorProperty(names = {ROLE_ARN, "AWS_ROLE_ARN", "glue.role_arn"},
+ required = false,
+ description = "The IAM role ARN for AssumeRole-based access.")
+ private String roleArn = "";
+
+ @Getter
+ @ConnectorProperty(names = {EXTERNAL_ID, "AWS_EXTERNAL_ID", "glue.external_id"},
+ required = false,
+ description = "The external ID for AssumeRole trust policy.")
+ private String externalId = "";
+
+ @Getter
+ @ConnectorProperty(names = {BUCKET, "AWS_BUCKET"},
+ required = false,
+ description = "The default bucket name.")
+ private String bucket = "";
+
+ @Getter
+ @ConnectorProperty(names = {ROOT_PATH, "AWS_ROOT_PATH"},
+ required = false,
+ description = "The root path prefix inside the bucket.")
+ private String rootPath = "";
+
+ @Getter
+ @ConnectorProperty(names = {MAX_CONNECTIONS, "AWS_MAX_CONNECTIONS"},
+ required = false,
+ description = "The maximum number of connections to S3.")
+ private String maxConnections = DEFAULT_MAX_CONNECTIONS;
+
+ @Getter
+ @ConnectorProperty(names = {REQUEST_TIMEOUT_MS, "AWS_REQUEST_TIMEOUT_MS"},
+ required = false,
+ description = "The request timeout of S3 in milliseconds.")
+ private String requestTimeoutMs = DEFAULT_REQUEST_TIMEOUT_MS;
+
+ @Getter
+ @ConnectorProperty(names = {CONNECTION_TIMEOUT_MS, "AWS_CONNECTION_TIMEOUT_MS"},
+ required = false,
+ description = "The connection timeout of S3 in milliseconds.")
+ private String connectionTimeoutMs = DEFAULT_CONNECTION_TIMEOUT_MS;
+
+ @Getter
+ @ConnectorProperty(names = {USE_PATH_STYLE, "s3.path-style-access"},
+ required = false,
+ description = "Whether to use path-style bucket addressing.")
+ private String usePathStyle = "false";
+
+ @ConnectorProperty(names = {CREDENTIALS_PROVIDER_TYPE, "AWS_CREDENTIALS_PROVIDER_TYPE",
+ "glue.credentials_provider_type", "iceberg.rest.credentials_provider_type"},
+ required = false,
+ description = "The credentials provider type.")
+ private String credentialsProviderType = DEFAULT_CREDENTIALS_PROVIDER_TYPE;
+
+ private final Map<String, String> rawProperties;
+ private final Map<String, String> matchedProperties;
+
+ private S3FileSystemProperties(Map<String, String> rawProperties) {
+ this.rawProperties = Collections.unmodifiableMap(new HashMap<>(rawProperties));
+ this.matchedProperties = Collections.unmodifiableMap(collectMatchedProperties(rawProperties));
+ ConnectorPropertiesUtils.bindConnectorProperties(this, rawProperties);
+ normalizeForLegacyS3Compatibility();
+ }
+
+ /** Binds and validates raw properties. */
+ public static S3FileSystemProperties of(Map<String, String> properties) {
+ S3FileSystemProperties props = new S3FileSystemProperties(properties);
+ props.validate();
+ return props;
+ }
+
+ private void validate() {
+ new ParamRules()
+ .requireTogether(new String[] {accessKey, secretKey},
+ "s3.access_key and s3.secret_key must be set together")
+ .requireAllIfPresent(sessionToken, new String[] {accessKey, secretKey},
+ "s3.session_token requires s3.access_key and s3.secret_key")
+ .requireAllIfPresent(externalId, new String[] {roleArn},
+ "s3.external_id must be used together with s3.role_arn")
+ .check(this::hasUnsupportedCredentialsProviderType,
+ "Unsupported s3.credentials_provider_type: " + credentialsProviderType)
+ .check(() -> StringUtils.isBlank(endpoint) && StringUtils.isBlank(region),
+ "Either s3.endpoint or s3.region must be set")
+ .check(() -> StringUtils.isBlank(region),
Review Comment:
This validation rejects existing custom S3-compatible configurations that set only an endpoint. Before this migration, `S3ObjStorage.buildClient()` explicitly allowed a missing region, logged a warning, and used `us-east-1` as the SigV4 placeholder when an endpoint override was present; the removed `buildClient_missingRegionLogsWarnAndFallsBack` test covered that compatibility path. Now `new S3ObjStorage(Map.of("AWS_ENDPOINT", "https://minio.local"))` fails during binding before the client can be built. Since the PR says there is no behavior change, please preserve the previous endpoint-only fallback for the runtime filesystem path or document and gate this as an intentional incompatible change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](fe) Add typed filesystem properties for S3 [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #63252:
URL: https://github.com/apache/doris/pull/63252#issuecomment-4487220003
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](fe) Add typed filesystem properties for S3 [doris]
Posted by "CalvinKirs (via GitHub)" <gi...@apache.org>.
CalvinKirs merged PR #63252:
URL: https://github.com/apache/doris/pull/63252
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](fe) Add typed filesystem properties for S3 [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63252:
URL: https://github.com/apache/doris/pull/63252#issuecomment-4450097853
<details>
<summary>TPC-DS: <b>Total hot run time: 172570 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bf360bb94b28d58c61db4accf1741dc1378fb21e, data reload: false
query5 4309 645 507 507
query6 340 213 199 199
query7 4266 524 297 297
query8 328 234 213 213
query9 8797 4000 3962 3962
query10 464 336 311 311
query11 5810 2387 2204 2204
query12 188 128 129 128
query13 1302 629 423 423
query14 6771 5391 5050 5050
query14_1 4379 4397 4355 4355
query15 207 204 193 193
query16 1014 476 432 432
query17 1349 735 606 606
query18 2756 472 348 348
query19 226 203 155 155
query20 137 135 128 128
query21 212 138 118 118
query22 13641 14173 14429 14173
query23 17275 16488 16300 16300
query23_1 16461 16352 16373 16352
query24 7905 1753 1348 1348
query24_1 1357 1332 1353 1332
query25 584 479 419 419
query26 818 311 164 164
query27 2713 613 338 338
query28 4332 1940 1928 1928
query29 964 617 503 503
query30 296 232 196 196
query31 1092 1071 947 947
query32 88 74 66 66
query33 529 348 287 287
query34 1168 1134 646 646
query35 759 783 724 724
query36 1349 1314 1156 1156
query37 143 96 86 86
query38 3178 3110 3062 3062
query39 918 925 925 925
query39_1 878 878 872 872
query40 223 151 134 134
query41 62 60 59 59
query42 110 106 105 105
query43 326 317 277 277
query44
query45 206 196 192 192
query46 1105 1162 755 755
query47 2287 2312 2146 2146
query48 398 418 282 282
query49 615 522 424 424
query50 693 298 232 232
query51 4492 4272 4280 4272
query52 103 106 99 99
query53 254 291 209 209
query54 327 284 262 262
query55 94 96 86 86
query56 315 307 296 296
query57 1434 1388 1317 1317
query58 286 272 267 267
query59 1504 1658 1385 1385
query60 346 331 327 327
query61 167 154 154 154
query62 662 628 559 559
query63 238 192 206 192
query64 1638 838 672 672
query65
query66 1663 518 407 407
query67 30143 30028 29860 29860
query68
query69 471 349 307 307
query70 1013 959 946 946
query71 310 270 277 270
query72 3184 2861 2598 2598
query73 825 746 427 427
query74 5064 4942 4742 4742
query75 2784 2686 2350 2350
query76 2306 1114 770 770
query77 429 438 350 350
query78 13058 13056 12474 12474
query79 2890 990 722 722
query80 1770 583 488 488
query81 529 292 241 241
query82 936 153 121 121
query83 353 281 246 246
query84 291 135 111 111
query85 918 526 438 438
query86 458 335 311 311
query87 3398 3321 3187 3187
query88 3568 2666 2630 2630
query89 454 385 333 333
query90 1906 183 176 176
query91 177 169 143 143
query92 82 75 76 75
query93 1280 966 555 555
query94 743 360 303 303
query95 676 390 437 390
query96 990 767 339 339
query97 2668 2745 2539 2539
query98 241 231 231 231
query99 1122 1119 958 958
Total cold run time: 256853 ms
Total hot run time: 172570 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](fe) Add typed filesystem properties for S3 [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #63252:
URL: https://github.com/apache/doris/pull/63252#issuecomment-4487219673
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](fe) Add typed filesystem properties for S3 [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on PR #63252:
URL: https://github.com/apache/doris/pull/63252#issuecomment-4486782588
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](fe) Add typed filesystem properties for S3 [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63252:
URL: https://github.com/apache/doris/pull/63252#issuecomment-4449714657
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](fe) Add typed filesystem properties for S3 [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63252:
URL: https://github.com/apache/doris/pull/63252#issuecomment-4450007454
<details>
<summary>TPC-H: <b>Total hot run time: 29180 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bf360bb94b28d58c61db4accf1741dc1378fb21e, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17724 4135 3842 3842
q2 q3 10683 874 618 618
q4 4665 453 352 352
q5 7438 1319 1145 1145
q6 205 170 137 137
q7 933 939 766 766
q8 9697 1423 1211 1211
q9 6166 5411 5299 5299
q10 6299 2098 1792 1792
q11 476 275 249 249
q12 686 422 302 302
q13 18184 3307 2739 2739
q14 298 287 268 268
q15 q16 897 862 783 783
q17 1154 1040 727 727
q18 6458 5694 5539 5539
q19 1650 1250 969 969
q20 535 407 265 265
q21 4587 2295 1876 1876
q22 420 357 301 301
Total cold run time: 99155 ms
Total hot run time: 29180 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4210 4067 4072 4067
q2 q3 4627 4786 4178 4178
q4 2092 2164 1382 1382
q5 4933 4950 5236 4950
q6 203 164 133 133
q7 2016 1981 1733 1733
q8 3436 3226 3182 3182
q9 8450 8431 8369 8369
q10 4472 4517 4251 4251
q11 662 447 405 405
q12 755 757 539 539
q13 3222 3628 2895 2895
q14 301 311 273 273
q15 q16 767 815 677 677
q17 1320 1316 1272 1272
q18 8177 7142 7077 7077
q19 1119 1139 1177 1139
q20 2371 2248 1943 1943
q21 6171 5446 4925 4925
q22 570 511 441 441
Total cold run time: 59874 ms
Total hot run time: 53831 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](fe) Add typed filesystem properties for S3 [doris]
Posted by "CalvinKirs (via GitHub)" <gi...@apache.org>.
CalvinKirs commented on PR #63252:
URL: https://github.com/apache/doris/pull/63252#issuecomment-4456236517
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](fe) Add typed filesystem properties for S3 [doris]
Posted by "CalvinKirs (via GitHub)" <gi...@apache.org>.
CalvinKirs commented on PR #63252:
URL: https://github.com/apache/doris/pull/63252#issuecomment-4449714696
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](fe) Add typed filesystem properties for S3 [doris]
Posted by "CalvinKirs (via GitHub)" <gi...@apache.org>.
CalvinKirs commented on PR #63252:
URL: https://github.com/apache/doris/pull/63252#issuecomment-4458128164
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](fe) Add typed filesystem properties for S3 [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63252:
URL: https://github.com/apache/doris/pull/63252#issuecomment-4459152901
# FE Regression Coverage Report
Increment line coverage ` 0.00% (0/33)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63252_28929d52cbeaa73d69f4a741de1177c3cf922ffb_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63252_28929d52cbeaa73d69f4a741de1177c3cf922ffb_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](fe) Add typed filesystem properties for S3 [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63252:
URL: https://github.com/apache/doris/pull/63252#issuecomment-4458291284
<details>
<summary>TPC-DS: <b>Total hot run time: 169106 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 28929d52cbeaa73d69f4a741de1177c3cf922ffb, data reload: false
query5 4313 648 525 525
query6 331 216 207 207
query7 4224 562 332 332
query8 343 230 219 219
query9 8828 4029 4041 4029
query10 452 357 297 297
query11 5822 2427 2186 2186
query12 186 128 125 125
query13 1268 607 429 429
query14 5943 5405 5048 5048
query14_1 4371 4413 4364 4364
query15 211 204 191 191
query16 995 491 438 438
query17 984 760 606 606
query18 2460 487 363 363
query19 217 218 164 164
query20 143 135 135 135
query21 216 146 115 115
query22 13560 13495 13367 13367
query23 17197 16418 16056 16056
query23_1 16213 16552 16439 16439
query24 7479 1734 1292 1292
query24_1 1310 1342 1308 1308
query25 573 502 434 434
query26 1332 330 177 177
query27 2713 561 345 345
query28 4469 1955 1934 1934
query29 1031 657 519 519
query30 306 265 203 203
query31 1115 1078 947 947
query32 94 79 74 74
query33 560 368 310 310
query34 1188 1184 655 655
query35 811 774 681 681
query36 1340 1314 1193 1193
query37 150 108 90 90
query38 3201 3131 3070 3070
query39 943 923 891 891
query39_1 884 897 874 874
query40 237 150 127 127
query41 66 64 62 62
query42 113 110 115 110
query43 320 329 286 286
query44
query45 211 199 195 195
query46 1043 1202 713 713
query47 2331 2333 2256 2256
query48 411 405 292 292
query49 621 489 386 386
query50 1035 360 272 272
query51 4465 4322 4254 4254
query52 109 105 96 96
query53 254 289 207 207
query54 312 274 256 256
query55 94 90 87 87
query56 327 305 312 305
query57 1411 1415 1307 1307
query58 296 263 268 263
query59 1531 1639 1447 1447
query60 319 320 309 309
query61 160 171 147 147
query62 667 628 556 556
query63 245 205 207 205
query64 2446 790 655 655
query65
query66 1711 471 357 357
query67 30046 30006 29277 29277
query68
query69 476 331 307 307
query70 1073 921 981 921
query71 315 272 256 256
query72 2968 2672 2427 2427
query73 855 769 419 419
query74 5068 4941 4741 4741
query75 2646 2606 2266 2266
query76 2285 1150 773 773
query77 397 413 319 319
query78 12028 12094 11623 11623
query79 1363 1053 759 759
query80 669 556 450 450
query81 459 283 241 241
query82 1384 162 132 132
query83 362 286 253 253
query84 303 143 111 111
query85 878 534 441 441
query86 387 338 306 306
query87 3409 3403 3198 3198
query88 3506 2635 2620 2620
query89 430 396 337 337
query90 1971 182 178 178
query91 181 177 141 141
query92 80 81 71 71
query93 1489 1442 917 917
query94 527 354 304 304
query95 670 474 359 359
query96 1088 741 343 343
query97 2724 2712 2561 2561
query98 239 235 232 232
query99 1112 1100 992 992
Total cold run time: 252955 ms
Total hot run time: 169106 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](fe) Add typed filesystem properties for S3 [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63252:
URL: https://github.com/apache/doris/pull/63252#issuecomment-4458660916
# FE UT Coverage Report
Increment line coverage `` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63252_28929d52cbeaa73d69f4a741de1177c3cf922ffb/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63252_28929d52cbeaa73d69f4a741de1177c3cf922ffb/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org