You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "CalvinKirs (via GitHub)" <gi...@apache.org> on 2026/05/19 09:04:18 UTC
[PR] Fe obs native sdk spike [doris]
CalvinKirs opened a new pull request, #63400:
URL: https://github.com/apache/doris/pull/63400
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fs)Add native filesystem SPI for object storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63400:
URL: https://github.com/apache/doris/pull/63400#issuecomment-4706376627
<details>
<summary>TPC-DS: <b>Total hot run time: 168172 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5dc7212667a13d1d88807c324488dcc3d3f225b0, data reload: false
query5 4334 619 479 479
query6 447 187 164 164
query7 4804 546 307 307
query8 379 211 190 190
query9 8727 3969 3976 3969
query10 442 314 258 258
query11 5979 2335 2182 2182
query12 155 101 97 97
query13 1377 604 428 428
query14 6395 5357 5108 5108
query14_1 4327 4324 4333 4324
query15 202 191 169 169
query16 986 436 420 420
query17 1089 671 540 540
query18 2411 456 332 332
query19 197 175 136 136
query20 112 103 102 102
query21 213 135 111 111
query22 13665 13619 13370 13370
query23 17400 16533 16161 16161
query23_1 16162 16308 16175 16175
query24 7723 1743 1288 1288
query24_1 1297 1308 1304 1304
query25 535 433 373 373
query26 1302 306 158 158
query27 2684 563 349 349
query28 4451 2014 1990 1990
query29 1086 634 489 489
query30 311 236 198 198
query31 1113 1082 958 958
query32 111 61 58 58
query33 545 330 264 264
query34 1206 1139 647 647
query35 771 798 672 672
query36 1449 1386 1266 1266
query37 151 111 93 93
query38 3217 3158 3066 3066
query39 934 932 883 883
query39_1 880 887 866 866
query40 223 127 104 104
query41 69 97 61 61
query42 92 89 90 89
query43 312 311 276 276
query44
query45 195 184 176 176
query46 1131 1218 734 734
query47 2376 2417 2264 2264
query48 394 414 279 279
query49 616 460 347 347
query50 1055 364 253 253
query51 4374 4295 4210 4210
query52 85 87 73 73
query53 238 260 183 183
query54 255 207 203 203
query55 76 76 68 68
query56 220 215 198 198
query57 1440 1401 1314 1314
query58 235 203 208 203
query59 1567 1676 1478 1478
query60 271 239 228 228
query61 151 147 151 147
query62 703 653 585 585
query63 234 180 186 180
query64 2543 785 586 586
query65
query66 1790 460 334 334
query67 29750 29626 29435 29435
query68
query69 399 295 264 264
query70 979 954 956 954
query71 293 224 208 208
query72 3033 2623 2307 2307
query73 825 742 436 436
query74 5106 4991 4755 4755
query75 2653 2558 2221 2221
query76 2335 1149 752 752
query77 346 376 290 290
query78 12361 12520 11840 11840
query79 1232 1065 745 745
query80 511 460 379 379
query81 445 278 247 247
query82 232 158 121 121
query83 266 279 240 240
query84
query85 843 500 402 402
query86 314 297 289 289
query87 3394 3337 3227 3227
query88 3581 2693 2711 2693
query89 402 380 329 329
query90 2175 175 172 172
query91 172 155 130 130
query92 64 55 57 55
query93 1489 1402 902 902
query94 550 344 300 300
query95 669 371 333 333
query96 1081 818 344 344
query97 2735 2682 2587 2587
query98 213 204 201 201
query99 1150 1174 1035 1035
Total cold run time: 250035 ms
Total hot run time: 168172 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fs)Add native filesystem SPI for object storage [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #63400:
URL: https://github.com/apache/doris/pull/63400#issuecomment-4704089408
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fs)Add native filesystem SPI for object storage [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63400:
URL: https://github.com/apache/doris/pull/63400#discussion_r3412863803
##########
fe/fe-filesystem/fe-filesystem-oss/src/main/java/org/apache/doris/filesystem/oss/OssObjStorage.java:
##########
@@ -34,128 +61,328 @@
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
+import java.io.FileNotFoundException;
import java.io.IOException;
+import java.io.InputStream;
import java.net.URL;
+import java.util.Collections;
import java.util.Date;
-import java.util.HashMap;
+import java.util.List;
import java.util.Map;
-import java.util.UUID;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
/**
- * Alibaba Cloud OSS implementation of {@link org.apache.doris.filesystem.spi.ObjStorage}.
+ * Alibaba Cloud OSS implementation backed by the native OSS SDK.
*
- * <p>Extends {@link S3ObjStorage} so all core I/O operations (list, head, put, delete, copy,
- * multipart upload) delegate to the parent using S3-compatible APIs — matching the reference pattern
- * where {@code OssRemote extends DefaultRemote} (the S3-backed base class).
- *
- * <p>The two cloud-specific extension methods that <em>require</em> a native SDK are overridden:
- * <ul>
- * <li>{@link #getPresignedUrl(String)} — uses Alibaba OSS SDK
- * ({@code OSS.generatePresignedUrl}) to produce the correct OSS signature format.</li>
- * <li>{@link #getStsToken()} — uses Alibaba RAM/STS SDK
- * ({@code DefaultAcsClient}) to call {@code sts.aliyuncs.com}.</li>
- * </ul>
- *
- * <p>Recognized property keys (OSS-specific; AWS_* equivalents accepted as fallback):
- * <ul>
- * <li>{@code OSS_ENDPOINT} / {@code AWS_ENDPOINT} — OSS endpoint URL</li>
- * <li>{@code OSS_ACCESS_KEY} / {@code AWS_ACCESS_KEY} — access key ID</li>
- * <li>{@code OSS_SECRET_KEY} / {@code AWS_SECRET_KEY} — secret access key</li>
- * <li>{@code OSS_TOKEN} / {@code AWS_TOKEN} — STS session token (optional)</li>
- * <li>{@code OSS_BUCKET} / {@code AWS_BUCKET} — bucket name (required for cloud extensions)</li>
- * <li>{@code OSS_REGION} / {@code AWS_REGION} — region for STS calls</li>
- * <li>{@code OSS_ROLE_ARN} / {@code AWS_ROLE_ARN} — role ARN for STS assumption</li>
- * </ul>
+ * <p>This class consumes typed OSS properties. Raw key aliases are resolved by
+ * {@link OssFileSystemProperties}; client construction and authentication do not
+ * translate through AWS-compatible keys.
*/
-public class OssObjStorage extends S3ObjStorage {
+public class OssObjStorage implements ObjStorage<OSS> {
private static final Logger LOG = LogManager.getLogger(OssObjStorage.class);
- /** Validity period for presigned URLs and STS tokens, in seconds. */
private static final int SESSION_EXPIRE_SECONDS = 3600;
+ private static final int DELETE_BATCH_SIZE = 1000;
+ private static final Credentials ANONYMOUS_CREDENTIALS =
+ new DefaultCredentials("anonymous", "anonymous");
+ // A bucket name that can serve as a virtual-hosted subdomain: 3-63 chars, lowercase
+ // letters/digits/hyphens, must start and end with an alphanumeric.
+ private static final java.util.regex.Pattern VIRTUAL_HOST_BUCKET_NAME =
+ java.util.regex.Pattern.compile("[a-z0-9][a-z0-9-]{1,61}[a-z0-9]");
- private final Map<String, String> ossProperties;
+ private final OssFileSystemProperties properties;
+ private final AtomicBoolean closed = new AtomicBoolean(false);
private volatile OSS ossClient;
public OssObjStorage(Map<String, String> properties) {
- super(toS3Props(properties));
- this.ossProperties = properties;
+ this(OssFileSystemProperties.of(properties));
+ }
+
+ public OssObjStorage(OssFileSystemProperties properties) {
+ this.properties = properties;
+ }
+
+ /** Whether path-style (vs virtual-hosted-style) bucket access is explicitly configured. */
+ public boolean isUsePathStyle() {
+ return properties.isUsePathStyle();
+ }
+
+ @Override
+ public OSS getClient() throws IOException {
+ return getClient(properties.getBucket());
}
/**
- * Translates OSS-specific property keys to the AWS keys expected by {@link S3ObjStorage}.
- * If both forms are present, the AWS_* key takes precedence.
+ * Returns the lazily-built OSS client, choosing path-style vs virtual-hosted addressing
+ * based on {@code bucket} (see {@link #resolvePathStyle(String)}).
+ *
+ * <p>The addressing decision is made once, when the client is first built, from the first
+ * bucket accessed through this instance. An {@code OssObjStorage} is scoped to a single
+ * endpoint/bucket in practice (e.g. one backup repository), so this matches the per-bucket
+ * behavior the legacy AWS-SDK-based client relied on for that case.
*/
- static Map<String, String> toS3Props(Map<String, String> ossProps) {
- Map<String, String> s3Props = new HashMap<>(ossProps);
- if (ossProps.containsKey("OSS_ENDPOINT") && !ossProps.containsKey("AWS_ENDPOINT")) {
- s3Props.put("AWS_ENDPOINT", ossProps.get("OSS_ENDPOINT"));
+ private OSS getClient(String bucket) throws IOException {
+ if (closed.get()) {
+ throw new IOException("OssObjStorage is already closed");
}
- if (ossProps.containsKey("OSS_ACCESS_KEY") && !ossProps.containsKey("AWS_ACCESS_KEY")) {
- s3Props.put("AWS_ACCESS_KEY", ossProps.get("OSS_ACCESS_KEY"));
+ if (ossClient == null) {
+ synchronized (this) {
+ if (ossClient == null) {
+ ossClient = buildOssClient(resolvePathStyle(bucket));
+ }
+ }
}
- if (ossProps.containsKey("OSS_SECRET_KEY") && !ossProps.containsKey("AWS_SECRET_KEY")) {
- s3Props.put("AWS_SECRET_KEY", ossProps.get("OSS_SECRET_KEY"));
+ return ossClient;
+ }
+
+ /**
+ * Decides whether to use path-style (SLD) addressing for {@code bucket}.
+ *
+ * <p>Honors an explicit {@code use_path_style=true}, and otherwise falls back to path-style
+ * when the bucket name cannot be expressed as a virtual-hosted DNS label (for example it
+ * contains an underscore, which is illegal in a hostname). This mirrors the AWS SDK v2
+ * behavior the legacy S3-compatible client relied on, so buckets with non-DNS-safe names
+ * keep working instead of failing with a virtual-hosted {@code NoSuchBucket}. The native OSS
+ * SDK does no such fallback on its own.
+ *
+ * <p>Note: the OSS SDK's own {@code validateBucketName} is intentionally not used here — it
+ * accepts underscores (a legal object-storage name) even though such a name is not a legal
+ * DNS host label, which is precisely the case that must trigger path-style.
+ */
+ boolean resolvePathStyle(String bucket) {
+ if (properties.isUsePathStyle()) {
+ return true;
}
- if (ossProps.containsKey("OSS_TOKEN") && !ossProps.containsKey("AWS_TOKEN")) {
- s3Props.put("AWS_TOKEN", ossProps.get("OSS_TOKEN"));
+ return hasText(bucket) && !isVirtualHostCompatible(bucket);
+ }
+
+ /**
+ * Returns true when {@code bucket} is a valid virtual-hosted DNS label: 3-63 characters of
+ * lowercase letters, digits and hyphens, starting and ending with an alphanumeric. Names
+ * with underscores, uppercase letters or dots cannot be safely used as a virtual-hosted
+ * subdomain and therefore require path-style addressing.
+ */
+ private static boolean isVirtualHostCompatible(String bucket) {
+ return VIRTUAL_HOST_BUCKET_NAME.matcher(bucket).matches();
+ }
+
+ protected OSS buildOssClient(boolean pathStyle) throws IOException {
+ String endpoint = properties.getEndpoint();
+ String accessKey = properties.getAccessKey();
+ String secretKey = properties.getSecretKey();
+ if (!hasText(accessKey)) {
+ return new OSSClientBuilder().build(endpoint, anonymousCredentialsProvider(),
+ anonymousClientConfiguration(pathStyle));
}
- if (ossProps.containsKey("OSS_BUCKET") && !ossProps.containsKey("AWS_BUCKET")) {
- s3Props.put("AWS_BUCKET", ossProps.get("OSS_BUCKET"));
+ ClientBuilderConfiguration config = clientConfiguration(pathStyle);
+ String token = properties.getSessionToken();
+ if (hasText(token)) {
+ return new OSSClientBuilder().build(endpoint, accessKey, secretKey, token, config);
}
- if (ossProps.containsKey("OSS_REGION") && !ossProps.containsKey("AWS_REGION")) {
- s3Props.put("AWS_REGION", ossProps.get("OSS_REGION"));
+ return new OSSClientBuilder().build(endpoint, accessKey, secretKey, config);
+ }
+
+ @Override
+ public RemoteObjects listObjects(String remotePath, String continuationToken) throws IOException {
+ return listObjectsWithOptions(remotePath, ObjectListOptions.builder()
+ .continuationToken(continuationToken)
+ .build());
+ }
+
+ @Override
+ public RemoteObjects listObjectsWithOptions(String remotePath, ObjectListOptions options) throws IOException {
+ ObjectStorageUri uri = ObjectStorageUri.parse(remotePath, false);
+ ListObjectsRequest request = new ListObjectsRequest(uri.bucket());
+ request.setPrefix(uri.key());
+ if (options != null) {
+ String marker = hasText(options.continuationToken())
+ ? options.continuationToken() : options.startAfter();
+ if (hasText(marker)) {
+ request.setMarker(marker);
+ }
+ if (options.maxKeys() > 0) {
+ request.setMaxKeys(options.maxKeys());
+ }
+ if (hasText(options.delimiter())) {
+ request.setDelimiter(options.delimiter());
+ }
}
- if (ossProps.containsKey("OSS_ROLE_ARN") && !ossProps.containsKey("AWS_ROLE_ARN")) {
- s3Props.put("AWS_ROLE_ARN", ossProps.get("OSS_ROLE_ARN"));
+ try {
+ ObjectListing listing = getClient(uri.bucket()).listObjects(request);
+ List<RemoteObject> objects = listing.getObjectSummaries().stream()
+ .map(obj -> toRemoteObject(uri.key(), obj))
+ .collect(Collectors.toList());
+ return new RemoteObjects(objects, listing.isTruncated(),
+ listing.isTruncated() ? listing.getNextMarker() : null);
+ } catch (OSSException e) {
+ // OSSException (server-side errors such as NoSuchBucket) is a sibling of
+ // ClientException, not a subclass, so it must be caught explicitly or it would
+ // propagate as an unwrapped runtime exception.
+ throw new IOException("Failed to list objects at " + remotePath + ": " + e.getMessage(), e);
+ } catch (ClientException e) {
+ throw new IOException("Failed to list objects at " + remotePath + ": " + e.getMessage(), e);
}
- s3Props.put("use_path_style", "false");
- return s3Props;
}
- // -----------------------------------------------------------------------
- // Cloud-specific extension overrides
- // -----------------------------------------------------------------------
+ @Override
+ public RemoteObject headObject(String remotePath) throws IOException {
+ ObjectStorageUri uri = ObjectStorageUri.parse(remotePath, false);
+ try {
+ ObjectMetadata metadata = getClient(uri.bucket()).getObjectMetadata(uri.bucket(), uri.key());
+ return new RemoteObject(uri.key(), uri.key(), metadata.getETag(), metadata.getContentLength(),
+ lastModifiedMs(metadata.getLastModified()));
+ } catch (OSSException e) {
+ if (isNotFound(e)) {
+ throw new FileNotFoundException("Object not found: " + remotePath);
+ }
+ throw new IOException("headObject failed for " + remotePath + ": " + e.getMessage(), e);
+ } catch (ClientException e) {
+ throw new IOException("headObject failed for " + remotePath + ": " + e.getMessage(), e);
+ }
+ }
- /**
- * Generates a pre-signed PUT URL using the Alibaba Cloud OSS native SDK.
- *
- * @param objectKey the bare object key (no scheme or bucket prefix)
- */
@Override
- public String getPresignedUrl(String objectKey) throws IOException {
- String bucket = resolveRequired("OSS_BUCKET", "AWS_BUCKET", "OSS bucket for presigned URL");
+ public void putObject(String remotePath, RequestBody requestBody) throws IOException {
+ ObjectStorageUri uri = ObjectStorageUri.parse(remotePath, false);
+ ObjectMetadata metadata = new ObjectMetadata();
+ metadata.setContentLength(requestBody.contentLength());
+ try (InputStream content = requestBody.content()) {
+ getClient(uri.bucket()).putObject(new PutObjectRequest(uri.bucket(), uri.key(), content, metadata));
Review Comment:
This catches only `ClientException`, but this same class already notes in `listObjectsWithOptions` that `OSSException` is a sibling and otherwise propagates as an unwrapped runtime exception. Server-side OSS failures for this write path, such as `NoSuchBucket` or `AccessDenied`, will therefore escape `ObjStorage.putObject` without being converted to the declared `IOException`; the same pattern appears in copy, multipart upload, presign, and batch delete below. Callers of the filesystem API commonly handle `IOException` to surface a `UserException` or `Status`, so native OSS ends up with different failure behavior from the other backends. Please catch and wrap `OSSException` consistently for all SDK calls that can return server-side errors, preserving `FileNotFoundException` where not-found is special.
##########
fe/fe-filesystem/fe-filesystem-cos/src/main/java/org/apache/doris/filesystem/cos/CosObjStorage.java:
##########
@@ -36,122 +60,285 @@
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
+import java.io.FileNotFoundException;
import java.io.IOException;
+import java.io.InputStream;
import java.net.URL;
+import java.util.Collections;
import java.util.Date;
import java.util.HashMap;
+import java.util.List;
import java.util.Map;
-import java.util.UUID;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
/**
- * Tencent Cloud COS implementation of {@link org.apache.doris.filesystem.spi.ObjStorage}.
+ * Tencent Cloud COS implementation backed by the native COS SDK.
*
- * <p>Extends {@link S3ObjStorage} so all core I/O operations (list, head, put, delete, copy,
- * multipart upload) delegate to the parent using S3-compatible APIs — matching the reference pattern
- * where {@code CosRemote extends DefaultRemote} (the S3-backed base class).
- *
- * <p>The two cloud-specific extension methods that <em>require</em> a native SDK are overridden:
- * <ul>
- * <li>{@link #getPresignedUrl(String)} — uses Tencent COS SDK
- * ({@code COSClient.generatePresignedUrl}) to produce the correct COS signature format.</li>
- * <li>{@link #getStsToken()} — uses Tencent Cloud STS SDK
- * ({@code StsClient.AssumeRole}) to call {@code sts.tencentcloudapi.com}.</li>
- * </ul>
- *
- * <p>Recognized property keys (COS-specific; AWS_* equivalents accepted as fallback):
- * <ul>
- * <li>{@code COS_ENDPOINT} / {@code AWS_ENDPOINT} — COS endpoint URL</li>
- * <li>{@code COS_ACCESS_KEY} / {@code AWS_ACCESS_KEY} — SecretId</li>
- * <li>{@code COS_SECRET_KEY} / {@code AWS_SECRET_KEY} — SecretKey</li>
- * <li>{@code COS_BUCKET} / {@code AWS_BUCKET} — bucket name (required for cloud extensions)</li>
- * <li>{@code COS_REGION} / {@code AWS_REGION} — region (e.g. {@code ap-guangzhou})</li>
- * <li>{@code COS_ROLE_ARN} / {@code AWS_ROLE_ARN} — role ARN for STS assumption</li>
- * </ul>
+ * <p>This class consumes typed COS properties. Raw key aliases are resolved by
+ * {@link CosFileSystemProperties}; client construction and authentication do not
+ * translate through AWS-compatible keys.
*/
-public class CosObjStorage extends S3ObjStorage {
+public class CosObjStorage implements ObjStorage<COSClient> {
private static final Logger LOG = LogManager.getLogger(CosObjStorage.class);
- /** Validity period for presigned URLs and STS tokens, in seconds. */
private static final int SESSION_EXPIRE_SECONDS = 3600;
+ private static final int DELETE_BATCH_SIZE = 1000;
- private final Map<String, String> cosProperties;
+ private final CosFileSystemProperties properties;
+ private final AtomicBoolean closed = new AtomicBoolean(false);
private volatile COSClient cosClient;
public CosObjStorage(Map<String, String> properties) {
- super(toS3Props(properties));
- this.cosProperties = properties;
+ this(CosFileSystemProperties.of(properties));
+ }
+
+ public CosObjStorage(CosFileSystemProperties properties) {
+ this.properties = properties;
+ }
+
+ /** Whether path-style (vs virtual-hosted-style) bucket access is configured. */
+ public boolean isUsePathStyle() {
+ return properties.isUsePathStyle();
}
- /**
- * Translates COS-specific property keys to the AWS keys expected by {@link S3ObjStorage}.
- * If both forms are present, the AWS_* key takes precedence.
- */
- static Map<String, String> toS3Props(Map<String, String> cosProps) {
- Map<String, String> s3Props = new HashMap<>(cosProps);
- if (cosProps.containsKey("COS_ENDPOINT") && !cosProps.containsKey("AWS_ENDPOINT")) {
- s3Props.put("AWS_ENDPOINT", cosProps.get("COS_ENDPOINT"));
+ @Override
+ public COSClient getClient() throws IOException {
+ if (closed.get()) {
+ throw new IOException("CosObjStorage is already closed");
}
- if (cosProps.containsKey("COS_ACCESS_KEY") && !cosProps.containsKey("AWS_ACCESS_KEY")) {
- s3Props.put("AWS_ACCESS_KEY", cosProps.get("COS_ACCESS_KEY"));
+ if (cosClient == null) {
+ synchronized (this) {
+ if (cosClient == null) {
+ cosClient = buildCosClient(properties.getRegion());
+ }
+ }
}
- if (cosProps.containsKey("COS_SECRET_KEY") && !cosProps.containsKey("AWS_SECRET_KEY")) {
- s3Props.put("AWS_SECRET_KEY", cosProps.get("COS_SECRET_KEY"));
+ return cosClient;
+ }
+
+ protected COSClient buildCosClient(String region) throws IOException {
+ COSCredentials cred = buildCredentials();
+ ClientConfig clientConfig = new ClientConfig();
+ clientConfig.setRegion(new Region(region));
+ clientConfig.setHttpProtocol(HttpProtocol.https);
Review Comment:
`CosFileSystem` passes `objStorage.isUsePathStyle()` into `S3CompatibleFileSystem`, so `use_path_style=true` makes the shared layer parse HTTPS URLs as path-style (`endpoint/bucket/key`). The native COS client built here never applies that flag, while S3, OBS, and OSS configure the underlying SDK for path-style. For COS endpoints or bucket names that require path-style addressing, Doris will parse the path as if path-style is enabled but the SDK will still build virtual-hosted COS requests, so the configured native COS filesystem fails despite accepting the option. Please either configure COS SDK endpoint building for path-style here or reject/ignore the flag consistently for COS instead of advertising it through the shared filesystem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Fe obs native sdk spike [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63400:
URL: https://github.com/apache/doris/pull/63400#issuecomment-4486150920
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Fe obs native sdk spike [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63400:
URL: https://github.com/apache/doris/pull/63400#issuecomment-4487457424
# FE Regression Coverage Report
Increment line coverage ` 0.00% (0/21)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63400_212b24aa50a212791fa028b77ec38cd6a3d0358a_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63400_212b24aa50a212791fa028b77ec38cd6a3d0358a_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fs)Add native filesystem SPI for object storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63400:
URL: https://github.com/apache/doris/pull/63400#issuecomment-4630671390
<details>
<summary>TPC-H: <b>Total hot run time: 29380 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8f382cf1d00d81c0e65947e3a0fcfa6a873300a3, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17701 4018 3987 3987
q2 q3 10770 1424 826 826
q4 4684 480 346 346
q5 7557 926 593 593
q6 186 173 139 139
q7 769 869 628 628
q8 9348 1545 1659 1545
q9 5870 4521 4525 4521
q10 6756 1813 1544 1544
q11 443 273 255 255
q12 634 440 290 290
q13 18135 3329 2739 2739
q14 270 258 237 237
q15 q16 825 795 727 727
q17 971 936 920 920
q18 6867 5764 5652 5652
q19 1297 1375 1059 1059
q20 525 396 275 275
q21 6371 2768 2843 2768
q22 449 368 329 329
Total cold run time: 100428 ms
Total hot run time: 29380 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 5077 4739 4672 4672
q2 q3 4853 5291 4682 4682
q4 2151 2233 1404 1404
q5 4864 4816 4655 4655
q6 229 178 127 127
q7 1845 1804 1536 1536
q8 2418 2096 2102 2096
q9 7912 7612 7392 7392
q10 4711 4673 4234 4234
q11 526 382 352 352
q12 727 737 522 522
q13 3088 3508 2815 2815
q14 268 276 250 250
q15 q16 672 696 617 617
q17 1267 1260 1241 1241
q18 7179 6905 6631 6631
q19 1112 1122 1097 1097
q20 2214 2207 1944 1944
q21 5285 4592 4468 4468
q22 506 468 408 408
Total cold run time: 56904 ms
Total hot run time: 51143 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fs)Add native filesystem SPI for object storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63400:
URL: https://github.com/apache/doris/pull/63400#issuecomment-4532204253
<details>
<summary>TPC-DS: <b>Total hot run time: 171603 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9a283b39f0034b331578533fce6bae2bf53770b4, data reload: false
query5 4316 664 537 537
query6 340 218 210 210
query7 4216 567 316 316
query8 324 226 231 226
query9 8857 4111 4150 4111
query10 452 362 301 301
query11 5811 2434 2230 2230
query12 179 129 127 127
query13 1272 615 451 451
query14 6148 5551 5290 5290
query14_1 4536 4565 4531 4531
query15 211 208 206 206
query16 1010 478 458 458
query17 1169 707 582 582
query18 2540 477 351 351
query19 208 197 158 158
query20 138 131 130 130
query21 213 143 116 116
query22 13609 13616 13389 13389
query23 17353 16537 16366 16366
query23_1 16319 16442 16270 16270
query24 7491 1768 1315 1315
query24_1 1332 1327 1327 1327
query25 556 472 420 420
query26 1297 335 179 179
query27 2676 573 361 361
query28 4431 2019 1978 1978
query29 945 609 493 493
query30 306 238 202 202
query31 1137 1080 965 965
query32 82 77 75 75
query33 529 358 303 303
query34 1196 1129 670 670
query35 759 794 701 701
query36 1412 1401 1210 1210
query37 159 101 95 95
query38 3209 3167 3128 3128
query39 947 945 915 915
query39_1 867 897 881 881
query40 227 147 126 126
query41 67 66 68 66
query42 113 112 111 111
query43 346 340 301 301
query44
query45 222 213 200 200
query46 1093 1236 803 803
query47 2322 2429 2266 2266
query48 412 417 312 312
query49 650 506 420 420
query50 1015 353 269 269
query51 4419 4309 4270 4270
query52 107 106 97 97
query53 270 293 212 212
query54 330 304 291 291
query55 95 95 89 89
query56 313 336 317 317
query57 1439 1410 1359 1359
query58 323 285 281 281
query59 1619 1699 1464 1464
query60 336 344 324 324
query61 186 180 185 180
query62 708 657 596 596
query63 248 208 217 208
query64 2464 844 708 708
query65
query66 1770 478 362 362
query67 30076 29936 29165 29165
query68
query69 455 339 317 317
query70 1069 998 1012 998
query71 312 276 273 273
query72 2987 2701 2187 2187
query73 841 744 451 451
query74 5097 4982 4789 4789
query75 2700 2604 2294 2294
query76 2279 1121 774 774
query77 408 414 330 330
query78 12534 12570 11936 11936
query79 1456 1044 764 764
query80 652 546 450 450
query81 454 285 248 248
query82 1372 159 120 120
query83 356 277 251 251
query84 258 142 116 116
query85 921 555 460 460
query86 387 345 323 323
query87 3483 3390 3259 3259
query88 3672 2791 2750 2750
query89 454 395 346 346
query90 1889 191 193 191
query91 179 171 138 138
query92 80 79 75 75
query93 1492 1459 811 811
query94 570 352 316 316
query95 676 478 359 359
query96 1070 763 335 335
query97 2778 2749 2674 2674
query98 233 225 241 225
query99 1179 1160 1039 1039
Total cold run time: 254849 ms
Total hot run time: 171603 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fs)Add native filesystem SPI for object storage [doris]
Posted by "CalvinKirs (via GitHub)" <gi...@apache.org>.
CalvinKirs commented on PR #63400:
URL: https://github.com/apache/doris/pull/63400#issuecomment-4705865313
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fs)Add native filesystem SPI for object storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63400:
URL: https://github.com/apache/doris/pull/63400#issuecomment-4707090592
# FE Regression Coverage Report
Increment line coverage ` 1.35% (12/886)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63400_5dc7212667a13d1d88807c324488dcc3d3f225b0_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63400_5dc7212667a13d1d88807c324488dcc3d3f225b0_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fs)Add native filesystem SPI for object storage [doris]
Posted by "CalvinKirs (via GitHub)" <gi...@apache.org>.
CalvinKirs commented on PR #63400:
URL: https://github.com/apache/doris/pull/63400#issuecomment-4620915965
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fs)Add native filesystem SPI for object storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63400:
URL: https://github.com/apache/doris/pull/63400#issuecomment-4531569268
<details>
<summary>TPC-DS: <b>Total hot run time: 172202 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f5e9d05b3ae0bc07ba8a1d55aa622a9e7078c6e4, data reload: false
query5 4330 655 536 536
query6 400 214 197 197
query7 4254 530 313 313
query8 334 232 216 216
query9 8836 4145 4128 4128
query10 454 345 297 297
query11 5791 2561 2236 2236
query12 192 132 123 123
query13 1348 631 449 449
query14 6333 5498 5215 5215
query14_1 4551 4547 4512 4512
query15 219 203 188 188
query16 1035 459 453 453
query17 1163 738 600 600
query18 2563 489 365 365
query19 225 206 177 177
query20 138 133 127 127
query21 216 143 118 118
query22 13656 13614 13277 13277
query23 17335 16487 16269 16269
query23_1 16402 16340 16360 16340
query24 7804 1787 1325 1325
query24_1 1325 1350 1335 1335
query25 558 472 402 402
query26 1307 328 171 171
query27 2670 566 339 339
query28 4413 2005 2027 2005
query29 983 634 495 495
query30 317 240 204 204
query31 1129 1092 964 964
query32 83 78 71 71
query33 562 348 313 313
query34 1212 1129 664 664
query35 785 790 700 700
query36 1391 1433 1287 1287
query37 156 99 87 87
query38 3260 3154 3076 3076
query39 932 946 909 909
query39_1 880 883 863 863
query40 237 154 125 125
query41 66 62 62 62
query42 107 109 106 106
query43 334 334 292 292
query44
query45 232 209 207 207
query46 1143 1241 767 767
query47 2407 2468 2307 2307
query48 406 411 295 295
query49 646 509 388 388
query50 1001 363 253 253
query51 4347 4326 4284 4284
query52 104 104 92 92
query53 256 280 201 201
query54 314 280 254 254
query55 96 93 82 82
query56 292 313 297 297
query57 1442 1404 1316 1316
query58 300 267 286 267
query59 1612 1674 1482 1482
query60 330 331 326 326
query61 179 178 176 176
query62 711 656 594 594
query63 239 207 212 207
query64 2449 850 679 679
query65
query66 1716 531 353 353
query67 29994 30014 29964 29964
query68
query69 452 339 309 309
query70 1057 977 1011 977
query71 313 282 271 271
query72 3023 2709 2359 2359
query73 856 792 451 451
query74 5134 5001 4830 4830
query75 2675 2618 2280 2280
query76 2286 1153 764 764
query77 418 401 322 322
query78 12377 12400 11918 11918
query79 1510 1027 754 754
query80 641 533 442 442
query81 533 278 248 248
query82 1334 153 125 125
query83 360 273 246 246
query84 263 141 111 111
query85 1161 584 446 446
query86 407 335 350 335
query87 3468 3391 3265 3265
query88 3623 2761 2735 2735
query89 449 408 347 347
query90 1916 170 182 170
query91 181 171 143 143
query92 89 84 80 80
query93 1454 1473 908 908
query94 522 346 328 328
query95 686 372 356 356
query96 1026 784 350 350
query97 2736 2758 2597 2597
query98 238 228 233 228
query99 1189 1155 1033 1033
Total cold run time: 255716 ms
Total hot run time: 172202 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fs)Add native filesystem SPI for object storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63400:
URL: https://github.com/apache/doris/pull/63400#issuecomment-4705223710
# FE Regression Coverage Report
Increment line coverage ` 19.44% (7/36)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63400_7ffa22bbb469a317eb5512959a32b686da2822f6_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63400_7ffa22bbb469a317eb5512959a32b686da2822f6_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fs)Add native filesystem SPI for object storage [doris]
Posted by "CalvinKirs (via GitHub)" <gi...@apache.org>.
CalvinKirs commented on code in PR #63400:
URL: https://github.com/apache/doris/pull/63400#discussion_r3361936262
##########
fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/ObjectStorageUri.java:
##########
@@ -0,0 +1,97 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.filesystem.spi;
+
+/**
+ * Parsed object-storage URI. Extracts bucket and key without any fe-core dependency.
+ */
+public final class ObjectStorageUri {
+
+ private final String bucket;
+ private final String key;
+
+ private ObjectStorageUri(String bucket, String key) {
+ this.bucket = bucket;
+ this.key = key;
+ }
+
+ /**
+ * Parses URIs of the form: scheme://bucket/key.
+ *
+ * <p>When {@code pathStyleAccess} is {@code true} <strong>and</strong> the scheme is
+ * {@code http} or {@code https}, the URI is treated as path-style:
+ * {@code scheme://endpoint/bucket/key}. The first path segment after the host is the
+ * bucket, the remainder is the key. For all other schemes the first authority
+ * component is always treated as the bucket, regardless of {@code pathStyleAccess}.
+ */
+ public static ObjectStorageUri parse(String path, boolean pathStyleAccess) {
Review Comment:
The interface is actually the common low-level abstraction implemented by all backends, including AzureObjStorage (Azure Blob), which
is not an S3-style API — so a provider-neutral name is intentional. The "S3-style API" wording in the Javadoc I'll reword it to "common object-storage abstraction" to avoid implying it's S3-specific. Let me know if you'd still prefer a rename.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fs)Add native filesystem SPI for object storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63400:
URL: https://github.com/apache/doris/pull/63400#issuecomment-4630058463
<details>
<summary>TPC-H: <b>Total hot run time: 29244 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6f6e5d9db34d01d22f4d4c575d5f75ace5621d1b, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17595 4014 4022 4014
q2 q3 10759 1401 799 799
q4 4683 493 358 358
q5 7582 854 585 585
q6 186 170 140 140
q7 773 845 647 647
q8 9343 1559 1641 1559
q9 5792 4528 4563 4528
q10 6783 1819 1537 1537
q11 434 277 248 248
q12 632 416 300 300
q13 18144 3346 2766 2766
q14 261 254 242 242
q15 q16 827 773 711 711
q17 970 961 921 921
q18 6903 5697 5491 5491
q19 1306 1185 1171 1171
q20 495 390 257 257
q21 6220 2744 2648 2648
q22 458 373 322 322
Total cold run time: 100146 ms
Total hot run time: 29244 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 5079 4773 4845 4773
q2 q3 4906 5237 4733 4733
q4 2110 2178 1410 1410
q5 4832 4828 4645 4645
q6 227 178 130 130
q7 1918 1742 1536 1536
q8 2369 2103 2113 2103
q9 7809 7329 7400 7329
q10 4744 4661 4206 4206
q11 529 379 350 350
q12 719 730 528 528
q13 3003 3341 2770 2770
q14 268 286 249 249
q15 q16 681 687 612 612
q17 1263 1249 1237 1237
q18 7236 6929 6805 6805
q19 1115 1117 1131 1117
q20 2200 2224 1936 1936
q21 5254 4534 4427 4427
q22 521 468 412 412
Total cold run time: 56783 ms
Total hot run time: 51308 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Fe obs native sdk spike [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63400:
URL: https://github.com/apache/doris/pull/63400#issuecomment-4486626924
<details>
<summary>TPC-DS: <b>Total hot run time: 169617 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 212b24aa50a212791fa028b77ec38cd6a3d0358a, data reload: false
query5 4352 656 535 535
query6 325 221 207 207
query7 4219 542 320 320
query8 322 228 215 215
query9 8838 3997 3946 3946
query10 463 329 299 299
query11 5662 2393 2215 2215
query12 177 125 123 123
query13 1260 596 443 443
query14 5986 5339 5045 5045
query14_1 4332 4350 4324 4324
query15 208 197 180 180
query16 1007 455 440 440
query17 975 749 610 610
query18 2457 491 366 366
query19 214 215 164 164
query20 140 137 134 134
query21 217 136 125 125
query22 13604 13487 13320 13320
query23 17147 16411 16033 16033
query23_1 16191 16117 16183 16117
query24 7417 1726 1311 1311
query24_1 1315 1303 1308 1303
query25 588 496 445 445
query26 1311 328 173 173
query27 2690 582 342 342
query28 4455 1967 1926 1926
query29 1021 641 524 524
query30 306 243 203 203
query31 1120 1067 939 939
query32 96 83 74 74
query33 543 365 307 307
query34 1160 1138 653 653
query35 773 821 685 685
query36 1395 1362 1227 1227
query37 154 103 91 91
query38 3236 3165 3047 3047
query39 926 940 894 894
query39_1 865 883 895 883
query40 233 156 133 133
query41 74 69 69 69
query42 112 114 109 109
query43 326 328 286 286
query44
query45 223 218 209 209
query46 1096 1186 719 719
query47 2275 2360 2223 2223
query48 400 409 295 295
query49 648 495 399 399
query50 986 361 259 259
query51 4315 4328 4301 4301
query52 106 109 98 98
query53 260 283 208 208
query54 327 282 268 268
query55 94 92 90 90
query56 326 309 375 309
query57 1438 1372 1332 1332
query58 298 270 261 261
query59 1568 1635 1426 1426
query60 325 325 298 298
query61 153 160 148 148
query62 664 636 550 550
query63 237 201 204 201
query64 2448 803 640 640
query65
query66 1737 464 349 349
query67 29395 29983 29848 29848
query68
query69 466 348 297 297
query70 1021 918 992 918
query71 299 275 269 269
query72 2954 2682 2410 2410
query73 805 771 414 414
query74 5059 4946 4755 4755
query75 2670 2583 2290 2290
query76 2269 1100 736 736
query77 393 399 336 336
query78 12181 12371 11666 11666
query79 1444 1027 759 759
query80 664 563 442 442
query81 459 277 239 239
query82 1370 154 124 124
query83 361 276 246 246
query84 297 137 112 112
query85 867 543 434 434
query86 407 361 354 354
query87 3400 3386 3184 3184
query88 3500 2650 2611 2611
query89 436 385 338 338
query90 1967 193 173 173
query91 173 170 141 141
query92 81 77 71 71
query93 1448 1456 921 921
query94 549 349 302 302
query95 671 487 342 342
query96 963 748 349 349
query97 2705 2695 2562 2562
query98 236 232 232 232
query99 1127 1098 990 990
Total cold run time: 251698 ms
Total hot run time: 169617 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fs)Add native filesystem SPI for object storage [doris]
Posted by "CalvinKirs (via GitHub)" <gi...@apache.org>.
CalvinKirs commented on code in PR #63400:
URL: https://github.com/apache/doris/pull/63400#discussion_r3361954816
##########
fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/ObjectStorageFileSystem.java:
##########
@@ -0,0 +1,1086 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.filesystem.spi;
+
+import org.apache.doris.filesystem.DorisInputFile;
+import org.apache.doris.filesystem.DorisInputStream;
+import org.apache.doris.filesystem.DorisOutputFile;
+import org.apache.doris.filesystem.FileEntry;
+import org.apache.doris.filesystem.FileIterator;
+import org.apache.doris.filesystem.GlobListing;
+import org.apache.doris.filesystem.Location;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.file.FileAlreadyExistsException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Pattern;
+
+/* Object-storage-backed FileSystem implementation for the Doris FE filesystem SPI.
+ * Does not depend on fe-core, fe-common, or fe-catalog.
+ */
+public class ObjectStorageFileSystem extends ObjFileSystem {
Review Comment:
On abstract: agreed — it's never instantiated directly (only extended by S3/OSS/OBS/COS), so I've made it abstract. On the rename to S3CompatibleFileSystem: I'd prefer to keep ObjectStorageFileSystem to avoid baking "S3" into a type that several non-S3-branded backends share, but I've clarified the role in the class Javadoc per the thread above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fs)Add native filesystem SPI for object storage [doris]
Posted by "CalvinKirs (via GitHub)" <gi...@apache.org>.
CalvinKirs commented on PR #63400:
URL: https://github.com/apache/doris/pull/63400#issuecomment-4531365968
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fs)Add native filesystem SPI for object storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63400:
URL: https://github.com/apache/doris/pull/63400#issuecomment-4706284239
<details>
<summary>TPC-H: <b>Total hot run time: 28723 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5dc7212667a13d1d88807c324488dcc3d3f225b0, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17672 4004 3987 3987
q2 q3 10840 1410 785 785
q4 4687 473 344 344
q5 7536 856 581 581
q6 178 169 135 135
q7 757 852 625 625
q8 9343 1557 1539 1539
q9 5742 4516 4443 4443
q10 6779 1789 1538 1538
q11 448 271 251 251
q12 636 428 283 283
q13 18103 3358 2798 2798
q14 265 259 243 243
q15 q16 813 777 707 707
q17 956 951 939 939
q18 7014 5825 5504 5504
q19 1366 1273 1067 1067
q20 501 409 255 255
q21 5824 2565 2399 2399
q22 435 354 300 300
Total cold run time: 99895 ms
Total hot run time: 28723 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4429 4246 4387 4246
q2 q3 4561 4930 4365 4365
q4 2089 2204 1389 1389
q5 4411 4285 4301 4285
q6 231 174 132 132
q7 1747 1622 1464 1464
q8 3038 2269 2198 2198
q9 8168 8269 8093 8093
q10 4850 4736 4231 4231
q11 578 423 389 389
q12 764 745 542 542
q13 3363 3639 3006 3006
q14 300 306 280 280
q15 q16 708 737 661 661
q17 1343 1315 1309 1309
q18 7861 7308 7358 7308
q19 1195 1190 1132 1132
q20 2208 2209 1941 1941
q21 5250 4559 4427 4427
q22 523 450 404 404
Total cold run time: 57617 ms
Total hot run time: 51802 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fs)Add native filesystem SPI for object storage [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #63400:
URL: https://github.com/apache/doris/pull/63400#issuecomment-4706810885
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fs)Add native filesystem SPI for object storage [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou merged PR #63400:
URL: https://github.com/apache/doris/pull/63400
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fs)Add native filesystem SPI for object storage [doris]
Posted by "CalvinKirs (via GitHub)" <gi...@apache.org>.
CalvinKirs commented on PR #63400:
URL: https://github.com/apache/doris/pull/63400#issuecomment-4531124078
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fs)Add native filesystem SPI for object storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63400:
URL: https://github.com/apache/doris/pull/63400#issuecomment-4531484799
<details>
<summary>TPC-H: <b>Total hot run time: 31485 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f5e9d05b3ae0bc07ba8a1d55aa622a9e7078c6e4, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 18137 4061 3990 3990
q2 q3 10798 1350 817 817
q4 4695 472 341 341
q5 7552 2267 2100 2100
q6 256 173 136 136
q7 953 785 628 628
q8 9341 1701 1606 1606
q9 6955 4902 4909 4902
q10 6436 2220 1891 1891
q11 435 277 247 247
q12 693 418 293 293
q13 18194 3357 2780 2780
q14 260 257 242 242
q15 q16 816 781 705 705
q17 976 935 877 877
q18 6780 5781 5527 5527
q19 1223 1415 1186 1186
q20 545 417 288 288
q21 6057 2751 2614 2614
q22 471 375 315 315
Total cold run time: 101573 ms
Total hot run time: 31485 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4756 4667 5004 4667
q2 q3 4853 5329 4698 4698
q4 2145 2195 1419 1419
q5 5188 4754 4700 4700
q6 226 177 129 129
q7 1859 1761 1530 1530
q8 2253 1976 1954 1954
q9 7397 7450 7450 7450
q10 4728 4672 4261 4261
q11 553 394 358 358
q12 737 751 546 546
q13 3063 3292 2809 2809
q14 271 287 257 257
q15 q16 676 701 617 617
q17 1299 1272 1264 1264
q18 7315 6842 6737 6737
q19 1084 1096 1105 1096
q20 2202 2228 1959 1959
q21 5267 4600 4481 4481
q22 524 482 412 412
Total cold run time: 56396 ms
Total hot run time: 51344 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fs)Add native filesystem SPI for object storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63400:
URL: https://github.com/apache/doris/pull/63400#issuecomment-4622161327
<details>
<summary>TPC-H: <b>Total hot run time: 29129 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4114b20c466abfb34c93be1ebe90a629566b627d, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17893 4033 4038 4033
q2 q3 10830 1409 799 799
q4 4693 482 355 355
q5 7546 871 596 596
q6 180 172 135 135
q7 762 839 654 654
q8 9397 1516 1540 1516
q9 5809 4479 4526 4479
q10 6845 1805 1524 1524
q11 436 271 250 250
q12 634 426 290 290
q13 18110 3356 2816 2816
q14 270 257 242 242
q15 q16 821 779 716 716
q17 993 860 909 860
q18 6764 5772 5490 5490
q19 1354 1281 1076 1076
q20 529 401 260 260
q21 6075 2843 2725 2725
q22 463 379 313 313
Total cold run time: 100404 ms
Total hot run time: 29129 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 5146 4711 4796 4711
q2 q3 4856 5263 4666 4666
q4 2128 2199 1407 1407
q5 4769 4849 4707 4707
q6 260 173 133 133
q7 1847 1704 1633 1633
q8 2465 2153 2079 2079
q9 7998 7801 7429 7429
q10 4732 4704 4277 4277
q11 537 383 352 352
q12 724 744 522 522
q13 3083 3398 2765 2765
q14 273 281 256 256
q15 q16 681 703 612 612
q17 1294 1265 1250 1250
q18 7234 6902 6764 6764
q19 1088 1085 1092 1085
q20 2216 2217 1945 1945
q21 5288 4590 4438 4438
q22 541 466 408 408
Total cold run time: 57160 ms
Total hot run time: 51439 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fs)Add native filesystem SPI for object storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63400:
URL: https://github.com/apache/doris/pull/63400#issuecomment-4622245814
<details>
<summary>TPC-DS: <b>Total hot run time: 170567 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4114b20c466abfb34c93be1ebe90a629566b627d, data reload: false
query5 4328 658 469 469
query6 446 206 189 189
query7 4820 562 300 300
query8 397 238 216 216
query9 8803 4024 4061 4024
query10 443 316 264 264
query11 5927 2350 2129 2129
query12 157 101 97 97
query13 1244 602 435 435
query14 6446 5442 5075 5075
query14_1 4431 4362 4381 4362
query15 204 199 175 175
query16 1031 421 429 421
query17 1106 687 556 556
query18 2500 471 338 338
query19 206 179 145 145
query20 119 112 106 106
query21 242 138 120 120
query22 13733 13560 13372 13372
query23 17440 16558 16181 16181
query23_1 16224 16261 16259 16259
query24 7493 1763 1277 1277
query24_1 1324 1300 1295 1295
query25 609 453 378 378
query26 1303 334 164 164
query27 2657 536 353 353
query28 4462 2037 2017 2017
query29 1081 637 503 503
query30 313 238 219 219
query31 1133 1091 962 962
query32 133 67 65 65
query33 539 330 257 257
query34 1181 1146 654 654
query35 770 794 697 697
query36 1404 1409 1214 1214
query37 154 115 96 96
query38 3205 3170 3070 3070
query39 945 928 909 909
query39_1 902 893 883 883
query40 229 132 109 109
query41 77 69 69 69
query42 105 100 98 98
query43 322 331 281 281
query44
query45 201 200 177 177
query46 1120 1270 757 757
query47 2343 2392 2260 2260
query48 403 413 309 309
query49 646 494 377 377
query50 1023 367 268 268
query51 4317 4377 4260 4260
query52 90 94 81 81
query53 256 280 201 201
query54 306 252 217 217
query55 82 80 72 72
query56 261 253 244 244
query57 1458 1440 1337 1337
query58 275 232 226 226
query59 1658 1701 1510 1510
query60 302 259 249 249
query61 183 183 183 183
query62 707 676 597 597
query63 241 192 187 187
query64 2634 796 636 636
query65
query66 1807 462 344 344
query67 29824 29810 29671 29671
query68
query69 426 290 273 273
query70 944 960 953 953
query71 299 223 215 215
query72 2983 2651 2359 2359
query73 854 785 437 437
query74 5124 4922 4792 4792
query75 2671 2596 2246 2246
query76 2349 1154 781 781
query77 355 381 291 291
query78 12563 12456 11942 11942
query79 1422 1053 746 746
query80 990 476 384 384
query81 505 286 245 245
query82 867 156 120 120
query83 360 276 249 249
query84 265 141 108 108
query85 936 547 446 446
query86 440 317 299 299
query87 3375 3279 3188 3188
query88 3640 2759 2744 2744
query89 425 378 330 330
query90 1789 178 178 178
query91 179 168 138 138
query92 67 65 60 60
query93 1487 1369 863 863
query94 658 355 304 304
query95 687 469 354 354
query96 1106 858 356 356
query97 2739 2695 2569 2569
query98 216 210 206 206
query99 1179 1200 1020 1020
Total cold run time: 252861 ms
Total hot run time: 170567 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fs)Add native filesystem SPI for object storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63400:
URL: https://github.com/apache/doris/pull/63400#issuecomment-4622334042
# FE Regression Coverage Report
Increment line coverage ` 63.64% (7/11)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63400_4114b20c466abfb34c93be1ebe90a629566b627d_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63400_4114b20c466abfb34c93be1ebe90a629566b627d_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fs)Add native filesystem SPI for object storage [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #63400:
URL: https://github.com/apache/doris/pull/63400#issuecomment-4707022492
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Fe obs native sdk spike [doris]
Posted by "CalvinKirs (via GitHub)" <gi...@apache.org>.
CalvinKirs commented on PR #63400:
URL: https://github.com/apache/doris/pull/63400#issuecomment-4486151114
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Fe obs native sdk spike [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63400:
URL: https://github.com/apache/doris/pull/63400#issuecomment-4486541019
<details>
<summary>TPC-H: <b>Total hot run time: 31393 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 212b24aa50a212791fa028b77ec38cd6a3d0358a, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17721 3981 3953 3953
q2 q3 10764 1403 818 818
q4 4687 477 342 342
q5 7614 2266 2096 2096
q6 238 178 139 139
q7 937 780 631 631
q8 9426 1683 1621 1621
q9 5231 4891 4917 4891
q10 6404 2083 1803 1803
q11 426 274 252 252
q12 632 414 286 286
q13 18117 3355 2738 2738
q14 267 255 239 239
q15 q16 823 780 707 707
q17 982 906 965 906
q18 6884 5764 5492 5492
q19 1309 1287 1108 1108
q20 538 401 365 365
q21 6241 2850 2633 2633
q22 450 373 442 373
Total cold run time: 99691 ms
Total hot run time: 31393 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4602 4576 4613 4576
q2 q3 4871 5339 4623 4623
q4 2181 2206 1422 1422
q5 4998 4674 4624 4624
q6 239 186 140 140
q7 1857 1684 1509 1509
q8 2399 2021 2081 2021
q9 7643 7231 7194 7194
q10 4471 4402 4003 4003
q11 520 378 368 368
q12 697 721 512 512
q13 3066 3455 2787 2787
q14 271 280 262 262
q15 q16 676 695 610 610
q17 1268 1241 1238 1238
q18 7175 6626 6583 6583
q19 1090 1101 1078 1078
q20 2219 2207 1928 1928
q21 5301 4604 4467 4467
q22 503 448 397 397
Total cold run time: 56047 ms
Total hot run time: 50342 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fs)Add native filesystem SPI for object storage [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #63400:
URL: https://github.com/apache/doris/pull/63400#discussion_r3322797937
##########
fe/fe-filesystem/fe-filesystem-s3/src/main/java/org/apache/doris/filesystem/s3/S3OutputStream.java:
##########
@@ -15,48 +15,39 @@
// specific language governing permissions and limitations
// under the License.
-package org.apache.doris.filesystem.s3;
-
-import org.apache.doris.filesystem.spi.RequestBody;
+package org.apache.doris.filesystem.spi;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.OutputStream;
/**
- * OutputStream that buffers writes in memory and uploads to S3 on close.
+ * OutputStream that buffers writes in memory and uploads to object storage on close.
*
- * <p>This implementation is intentionally simple and suitable for small metadata files (manifests,
- * snapshots, job info, etc.). Writes are rejected when the in-memory buffer would exceed
- * {@link #MAX_SINGLE_UPLOAD_BYTES} to prevent OOM on large payloads.
- * For large file writes (Hive data files, Backup archives), multipart upload must be used instead.
+ * <p>This implementation is intentionally simple and suitable for small metadata files
+ * (manifests, snapshots, job info, etc.). Writes are rejected when the in-memory buffer
+ * would exceed {@link #MAX_SINGLE_UPLOAD_BYTES} to prevent OOM on large payloads.
+ * For large file writes (Hive data files, Backup archives), multipart upload must be
+ * used instead.
*
- * <p><strong>Empty-close semantics (#22):</strong> if {@link #close()} is called without any
- * preceding {@code write(...)} call, NO object is uploaded. This avoids polluting the bucket
- * with phantom 0-byte placeholders when the caller opens an output stream and aborts before
- * writing. To explicitly create a zero-byte object, call {@code write(new byte[0])} (or
- * {@code write(b, off, 0)}) prior to {@link #close()} — any write call, even of length 0,
- * marks the stream as "written" and triggers an upload of the empty buffer.
+ * <p><strong>Empty-close semantics (#22):</strong> if {@link #close()} is called without
+ * any preceding {@code write(...)} call, no object is uploaded. This avoids polluting the
+ * bucket with phantom 0-byte placeholders when the caller opens an output stream and
+ * aborts before writing. To explicitly create a zero-byte object, call
+ * {@code write(new byte[0])} or {@code write(b, off, 0)} before {@link #close()}.
*/
-class S3OutputStream extends OutputStream {
+public class ObjectStorageOutputStream extends OutputStream {
- /**
- * Maximum in-memory buffer size before writes are rejected.
- * S3 single-PUT limit is 5 GB, but we enforce a much smaller guard (256 MB) so that an
- * accidental large-file write fails early with a clear message rather than OOMing silently.
- */
Review Comment:
why remove this comment?
##########
fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/capability/BatchDeleteCapability.java:
##########
@@ -0,0 +1,31 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.filesystem.capability;
+
+import org.apache.doris.filesystem.Location;
+
+import java.io.IOException;
+import java.util.List;
+
+/**
+ * Capability for deleting multiple file locations through a provider-optimized path.
+ */
+public interface BatchDeleteCapability {
Review Comment:
maybe all Capability interface should extends from a interface named `Capability` and update `requireCapability`'s paramter to `Class<? extends Capability>` in `FileSystem`.
##########
fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/ObjectStorageFileSystem.java:
##########
@@ -0,0 +1,1086 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.filesystem.spi;
+
+import org.apache.doris.filesystem.DorisInputFile;
+import org.apache.doris.filesystem.DorisInputStream;
+import org.apache.doris.filesystem.DorisOutputFile;
+import org.apache.doris.filesystem.FileEntry;
+import org.apache.doris.filesystem.FileIterator;
+import org.apache.doris.filesystem.GlobListing;
+import org.apache.doris.filesystem.Location;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.file.FileAlreadyExistsException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Pattern;
+
+/* Object-storage-backed FileSystem implementation for the Doris FE filesystem SPI.
+ * Does not depend on fe-core, fe-common, or fe-catalog.
+ */
+public class ObjectStorageFileSystem extends ObjFileSystem {
Review Comment:
The names "ObjectStorageFileSystem" and "ObjFileSystem" are too similar. More distinct and descriptive names that reflect their actual functions are needed.
##########
fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/ObjectStorageOutputStream.java:
##########
@@ -84,7 +75,6 @@ public void close() throws IOException {
}
closed = true;
if (!writeCalled) {
- // No write call was ever made: skip upload to avoid creating a phantom 0-byte object.
Review Comment:
why remove this comment?
##########
fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/ObjectStorageFileSystem.java:
##########
@@ -0,0 +1,1086 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.filesystem.spi;
+
+import org.apache.doris.filesystem.DorisInputFile;
+import org.apache.doris.filesystem.DorisInputStream;
+import org.apache.doris.filesystem.DorisOutputFile;
+import org.apache.doris.filesystem.FileEntry;
+import org.apache.doris.filesystem.FileIterator;
+import org.apache.doris.filesystem.GlobListing;
+import org.apache.doris.filesystem.Location;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.file.FileAlreadyExistsException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Pattern;
+
+/* Object-storage-backed FileSystem implementation for the Doris FE filesystem SPI.
+ * Does not depend on fe-core, fe-common, or fe-catalog.
+ */
+public class ObjectStorageFileSystem extends ObjFileSystem {
Review Comment:
maybe need to rename to S3CompatibleFileSystem. it should be abstract
##########
fe/fe-filesystem/fe-filesystem-s3/src/main/java/org/apache/doris/filesystem/s3/S3OutputStream.java:
##########
@@ -15,48 +15,39 @@
// specific language governing permissions and limitations
// under the License.
-package org.apache.doris.filesystem.s3;
-
-import org.apache.doris.filesystem.spi.RequestBody;
+package org.apache.doris.filesystem.spi;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.OutputStream;
/**
- * OutputStream that buffers writes in memory and uploads to S3 on close.
+ * OutputStream that buffers writes in memory and uploads to object storage on close.
*
- * <p>This implementation is intentionally simple and suitable for small metadata files (manifests,
- * snapshots, job info, etc.). Writes are rejected when the in-memory buffer would exceed
- * {@link #MAX_SINGLE_UPLOAD_BYTES} to prevent OOM on large payloads.
- * For large file writes (Hive data files, Backup archives), multipart upload must be used instead.
+ * <p>This implementation is intentionally simple and suitable for small metadata files
+ * (manifests, snapshots, job info, etc.). Writes are rejected when the in-memory buffer
+ * would exceed {@link #MAX_SINGLE_UPLOAD_BYTES} to prevent OOM on large payloads.
+ * For large file writes (Hive data files, Backup archives), multipart upload must be
+ * used instead.
*
- * <p><strong>Empty-close semantics (#22):</strong> if {@link #close()} is called without any
- * preceding {@code write(...)} call, NO object is uploaded. This avoids polluting the bucket
- * with phantom 0-byte placeholders when the caller opens an output stream and aborts before
- * writing. To explicitly create a zero-byte object, call {@code write(new byte[0])} (or
- * {@code write(b, off, 0)}) prior to {@link #close()} — any write call, even of length 0,
- * marks the stream as "written" and triggers an upload of the empty buffer.
+ * <p><strong>Empty-close semantics (#22):</strong> if {@link #close()} is called without
+ * any preceding {@code write(...)} call, no object is uploaded. This avoids polluting the
+ * bucket with phantom 0-byte placeholders when the caller opens an output stream and
+ * aborts before writing. To explicitly create a zero-byte object, call
+ * {@code write(new byte[0])} or {@code write(b, off, 0)} before {@link #close()}.
*/
-class S3OutputStream extends OutputStream {
+public class ObjectStorageOutputStream extends OutputStream {
- /**
- * Maximum in-memory buffer size before writes are rejected.
- * S3 single-PUT limit is 5 GB, but we enforce a much smaller guard (256 MB) so that an
- * accidental large-file write fails early with a clear message rather than OOMing silently.
- */
private static final long MAX_SINGLE_UPLOAD_BYTES = 256L * 1024 * 1024; // 256 MB
private final String remotePath;
- private final S3ObjStorage objStorage;
+ private final ObjStorage<?> objStorage;
private final ByteArrayOutputStream buffer = new ByteArrayOutputStream();
private boolean closed = false;
- // Tracks whether write(...) was called at least once. close() skips the upload entirely
- // when this is false to avoid creating phantom 0-byte objects on accidental empty close.
Review Comment:
why remove this comment?
##########
fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/ObjStorage.java:
##########
Review Comment:
if this is a S3 style API interface, it should reflacted on the interface's name
##########
fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/ObjectListOptions.java:
##########
@@ -0,0 +1,87 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.filesystem.spi;
+
+/**
+ * Optional controls for object listing.
+ */
+public final class ObjectListOptions {
+
+ private final String continuationToken;
+ private final String startAfter;
+ private final int maxKeys;
+ private final String delimiter;
Review Comment:
add comment to these attribute
##########
fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/ObjectStorageFileSystem.java:
##########
@@ -0,0 +1,1086 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.filesystem.spi;
+
+import org.apache.doris.filesystem.DorisInputFile;
+import org.apache.doris.filesystem.DorisInputStream;
+import org.apache.doris.filesystem.DorisOutputFile;
+import org.apache.doris.filesystem.FileEntry;
+import org.apache.doris.filesystem.FileIterator;
+import org.apache.doris.filesystem.GlobListing;
+import org.apache.doris.filesystem.Location;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.file.FileAlreadyExistsException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Pattern;
+
+/* Object-storage-backed FileSystem implementation for the Doris FE filesystem SPI.
+ * Does not depend on fe-core, fe-common, or fe-catalog.
+ */
+public class ObjectStorageFileSystem extends ObjFileSystem {
+
+ // Object stores do not have real directories; use a zero-byte marker with trailing slash.
+ private static final String DIR_MARKER_SUFFIX = "/";
+
+ private final boolean usePathStyle;
+
+ protected ObjectStorageFileSystem(String name, ObjStorage<?> objStorage, boolean usePathStyle) {
+ super(name, objStorage);
+ this.usePathStyle = usePathStyle;
+ }
+
+ @Override
+ protected boolean isNotFoundError(IOException e) {
+ return e instanceof java.io.FileNotFoundException;
+ }
Review Comment:
same as super class
##########
fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/ObjectStorageUri.java:
##########
@@ -0,0 +1,97 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.filesystem.spi;
+
+/**
+ * Parsed object-storage URI. Extracts bucket and key without any fe-core dependency.
+ */
+public final class ObjectStorageUri {
+
+ private final String bucket;
+ private final String key;
+
+ private ObjectStorageUri(String bucket, String key) {
+ this.bucket = bucket;
+ this.key = key;
+ }
+
+ /**
+ * Parses URIs of the form: scheme://bucket/key.
+ *
+ * <p>When {@code pathStyleAccess} is {@code true} <strong>and</strong> the scheme is
+ * {@code http} or {@code https}, the URI is treated as path-style:
+ * {@code scheme://endpoint/bucket/key}. The first path segment after the host is the
+ * bucket, the remainder is the key. For all other schemes the first authority
+ * component is always treated as the bucket, regardless of {@code pathStyleAccess}.
+ */
+ public static ObjectStorageUri parse(String path, boolean pathStyleAccess) {
Review Comment:
There is an implicit dependency here, which requires that the path be escaped. Please note this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fs)Add native filesystem SPI for object storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63400:
URL: https://github.com/apache/doris/pull/63400#issuecomment-4704760670
<details>
<summary>TPC-DS: <b>Total hot run time: 168363 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7ffa22bbb469a317eb5512959a32b686da2822f6, data reload: false
query5 4316 613 470 470
query6 444 188 167 167
query7 4856 534 312 312
query8 357 209 187 187
query9 8734 3973 3982 3973
query10 436 297 246 246
query11 5925 2387 2142 2142
query12 153 102 99 99
query13 1248 589 401 401
query14 6314 5402 5038 5038
query14_1 4388 4381 4376 4376
query15 205 200 177 177
query16 1010 455 429 429
query17 1150 713 574 574
query18 2560 465 350 350
query19 204 191 140 140
query20 116 108 107 107
query21 212 137 114 114
query22 13544 13542 13521 13521
query23 17309 16586 16110 16110
query23_1 16275 16425 16192 16192
query24 7420 1775 1298 1298
query24_1 1317 1314 1310 1310
query25 563 492 393 393
query26 1301 332 165 165
query27 2662 568 341 341
query28 4450 1992 2016 1992
query29 1109 625 494 494
query30 308 235 195 195
query31 1137 1076 955 955
query32 107 61 58 58
query33 520 328 260 260
query34 1165 1146 643 643
query35 757 822 680 680
query36 1402 1429 1237 1237
query37 146 101 90 90
query38 3219 3154 3057 3057
query39 928 926 887 887
query39_1 868 873 877 873
query40 233 115 97 97
query41 63 62 59 59
query42 93 93 91 91
query43 313 326 273 273
query44
query45 193 182 180 180
query46 1103 1208 755 755
query47 2344 2346 2281 2281
query48 386 427 284 284
query49 606 471 358 358
query50 983 340 259 259
query51 4307 4266 4296 4266
query52 94 91 75 75
query53 243 251 190 190
query54 260 206 206 206
query55 76 77 69 69
query56 248 226 228 226
query57 1427 1400 1334 1334
query58 234 209 205 205
query59 1539 1651 1393 1393
query60 271 239 228 228
query61 157 149 146 146
query62 699 663 588 588
query63 221 188 187 187
query64 2520 734 584 584
query65
query66 1758 457 337 337
query67 29645 29668 29468 29468
query68
query69 426 297 255 255
query70 951 907 956 907
query71 293 224 209 209
query72 2912 2634 2374 2374
query73 845 797 404 404
query74 5131 4917 4833 4833
query75 2658 2569 2239 2239
query76 2322 1155 780 780
query77 347 371 283 283
query78 12231 12633 11930 11930
query79 1409 1013 759 759
query80 573 456 391 391
query81 450 285 241 241
query82 579 159 118 118
query83 349 273 253 253
query84
query85 863 517 412 412
query86 360 306 280 280
query87 3390 3339 3274 3274
query88 3592 2735 2723 2723
query89 425 383 332 332
query90 1984 179 168 168
query91 171 151 130 130
query92 64 58 55 55
query93 1407 1526 866 866
query94 541 346 324 324
query95 665 475 332 332
query96 1014 831 369 369
query97 2689 2707 2580 2580
query98 212 201 198 198
query99 1172 1178 1028 1028
Total cold run time: 249604 ms
Total hot run time: 168363 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fs)Add native filesystem SPI for object storage [doris]
Posted by "CalvinKirs (via GitHub)" <gi...@apache.org>.
CalvinKirs commented on code in PR #63400:
URL: https://github.com/apache/doris/pull/63400#discussion_r3361927896
##########
fe/fe-filesystem/fe-filesystem-s3/src/main/java/org/apache/doris/filesystem/s3/S3OutputStream.java:
##########
@@ -15,48 +15,39 @@
// specific language governing permissions and limitations
// under the License.
-package org.apache.doris.filesystem.s3;
-
-import org.apache.doris.filesystem.spi.RequestBody;
+package org.apache.doris.filesystem.spi;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.OutputStream;
/**
- * OutputStream that buffers writes in memory and uploads to S3 on close.
+ * OutputStream that buffers writes in memory and uploads to object storage on close.
*
- * <p>This implementation is intentionally simple and suitable for small metadata files (manifests,
- * snapshots, job info, etc.). Writes are rejected when the in-memory buffer would exceed
- * {@link #MAX_SINGLE_UPLOAD_BYTES} to prevent OOM on large payloads.
- * For large file writes (Hive data files, Backup archives), multipart upload must be used instead.
+ * <p>This implementation is intentionally simple and suitable for small metadata files
+ * (manifests, snapshots, job info, etc.). Writes are rejected when the in-memory buffer
+ * would exceed {@link #MAX_SINGLE_UPLOAD_BYTES} to prevent OOM on large payloads.
+ * For large file writes (Hive data files, Backup archives), multipart upload must be
+ * used instead.
*
- * <p><strong>Empty-close semantics (#22):</strong> if {@link #close()} is called without any
- * preceding {@code write(...)} call, NO object is uploaded. This avoids polluting the bucket
- * with phantom 0-byte placeholders when the caller opens an output stream and aborts before
- * writing. To explicitly create a zero-byte object, call {@code write(new byte[0])} (or
- * {@code write(b, off, 0)}) prior to {@link #close()} — any write call, even of length 0,
- * marks the stream as "written" and triggers an upload of the empty buffer.
+ * <p><strong>Empty-close semantics (#22):</strong> if {@link #close()} is called without
+ * any preceding {@code write(...)} call, no object is uploaded. This avoids polluting the
+ * bucket with phantom 0-byte placeholders when the caller opens an output stream and
+ * aborts before writing. To explicitly create a zero-byte object, call
+ * {@code write(new byte[0])} or {@code write(b, off, 0)} before {@link #close()}.
*/
-class S3OutputStream extends OutputStream {
+public class ObjectStorageOutputStream extends OutputStream {
- /**
- * Maximum in-memory buffer size before writes are rejected.
- * S3 single-PUT limit is 5 GB, but we enforce a much smaller guard (256 MB) so that an
- * accidental large-file write fails early with a clear message rather than OOMing silently.
- */
private static final long MAX_SINGLE_UPLOAD_BYTES = 256L * 1024 * 1024; // 256 MB
private final String remotePath;
- private final S3ObjStorage objStorage;
+ private final ObjStorage<?> objStorage;
private final ByteArrayOutputStream buffer = new ByteArrayOutputStream();
private boolean closed = false;
- // Tracks whether write(...) was called at least once. close() skips the upload entirely
- // when this is false to avoid creating phantom 0-byte objects on accidental empty close.
Review Comment:
The class was moved/renamed to ObjectStorageOutputStream (in fe-filesystem-spi); the comment was not intentionally dropped. The
5GB-vs-256MB rationale is restored there as an inline comment on MAX_SINGLE_UPLOAD_BYTES.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fs)Add native filesystem SPI for object storage [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #63400:
URL: https://github.com/apache/doris/pull/63400#discussion_r3385917677
##########
fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/properties/S3CompatibleFileSystemProperties.java:
##########
@@ -62,9 +62,14 @@ public interface S3CompatibleFileSystemProperties extends FileSystemProperties {
/** Returns the connection timeout in milliseconds as a provider property value. */
String getConnectionTimeoutMs();
- /** Returns whether path-style bucket addressing is enabled. */
+ /** Returns whether path-style bucket addressing is enabled, as a raw provider property value. */
String getUsePathStyle();
+ /** Returns path-style bucket addressing as a parsed boolean (single conversion point). */
+ default boolean isUsePathStyle() {
+ return Boolean.parseBoolean(getUsePathStyle());
Review Comment:
This operation is risky. What methods can ensure that getUsePathStyle returns a valid boolean string? What happens if parsing fails here?
##########
fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/FileSystem.java:
##########
@@ -30,6 +34,28 @@
*/
public interface FileSystem extends AutoCloseable {
+ /**
+ * Returns a provider-specific optional capability exposed through a stable API type.
+ *
+ * <p>Core filesystem operations must remain on {@link FileSystem}. Optional operations
+ * such as presigned URLs or temporary credentials should be exposed as capability
+ * interfaces so callers do not depend on concrete filesystem implementations.
+ */
+ default <T extends Capability> Optional<T> capability(Class<T> capabilityType) {
+ Objects.requireNonNull(capabilityType, "capabilityType");
Review Comment:
optimize this error message
##########
fe/fe-filesystem/fe-filesystem-azure/src/main/java/org/apache/doris/filesystem/azure/AzureFileSystemProperties.java:
##########
@@ -0,0 +1,404 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.filesystem.azure;
+
+import org.apache.doris.filesystem.FileSystemType;
+import org.apache.doris.filesystem.properties.BackendStorageKind;
+import org.apache.doris.filesystem.properties.BackendStorageProperties;
+import org.apache.doris.filesystem.properties.FileSystemProperties;
+import org.apache.doris.filesystem.properties.HadoopStorageProperties;
+import org.apache.doris.filesystem.properties.StorageKind;
+import org.apache.doris.foundation.property.ConnectorPropertiesUtils;
+import org.apache.doris.foundation.property.ConnectorProperty;
+import org.apache.doris.foundation.property.ParamRules;
+
+import org.apache.commons.lang3.StringUtils;
+
+import java.lang.reflect.Field;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedHashSet;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Provider-owned Azure Blob Storage properties.
+ *
+ * <p>The public aliases, endpoint formatting, shared-key validation, and backend
+ * map match fe-core AzureProperties. Legacy uppercase aliases remain accepted so
+ * existing FE filesystem callers can migrate through {@link AzureFileSystemProvider#bind(Map)}.
+ */
+public final class AzureFileSystemProperties
+ implements FileSystemProperties, BackendStorageProperties, HadoopStorageProperties {
+
+ public static final String ENDPOINT = "azure.endpoint";
+ public static final String ACCOUNT_NAME = "azure.account_name";
+ public static final String ACCOUNT_KEY = "azure.account_key";
+ public static final String CLIENT_ID = "azure.oauth2_client_id";
+ public static final String CLIENT_SECRET = "azure.oauth2_client_secret";
+ public static final String OAUTH_SERVER_URI = "azure.oauth2_server_uri";
+ public static final String OAUTH_ACCOUNT_HOST = "azure.oauth2_account_host";
+ public static final String TENANT_ID = "azure.oauth2_client_tenant_id";
+ public static final String AUTH_TYPE = "azure.auth_type";
+ public static final String CONTAINER = "container";
+ public static final String USE_PATH_STYLE = "use_path_style";
+ public static final String FORCE_PARSING_BY_STANDARD_URI = "force_parsing_by_standard_uri";
+
+ public static final String SHARED_KEY_AUTH = "SharedKey";
+ public static final String OAUTH2_AUTH = "OAuth2";
+ public static final String AZURE_ENDPOINT_TEMPLATE = "https://%s.blob.core.windows.net";
+
+ private static final String[] AZURE_BLOB_HOST_SUFFIXES = {
+ "blob.core.windows.net",
+ "blob.core.chinacloudapi.cn",
+ "blob.core.usgovcloudapi.net",
+ "blob.core.cloudapi.de"
+ };
+
+ @ConnectorProperty(names = {ENDPOINT, "s3.endpoint", "AWS_ENDPOINT", "endpoint", "ENDPOINT",
Review Comment:
Why use both static variables and strings together?
##########
fe/fe-filesystem/fe-filesystem-azure/src/main/java/org/apache/doris/filesystem/azure/AzureFileSystemProperties.java:
##########
@@ -0,0 +1,404 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.filesystem.azure;
+
+import org.apache.doris.filesystem.FileSystemType;
+import org.apache.doris.filesystem.properties.BackendStorageKind;
+import org.apache.doris.filesystem.properties.BackendStorageProperties;
+import org.apache.doris.filesystem.properties.FileSystemProperties;
+import org.apache.doris.filesystem.properties.HadoopStorageProperties;
+import org.apache.doris.filesystem.properties.StorageKind;
+import org.apache.doris.foundation.property.ConnectorPropertiesUtils;
+import org.apache.doris.foundation.property.ConnectorProperty;
+import org.apache.doris.foundation.property.ParamRules;
+
+import org.apache.commons.lang3.StringUtils;
+
+import java.lang.reflect.Field;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedHashSet;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Provider-owned Azure Blob Storage properties.
+ *
+ * <p>The public aliases, endpoint formatting, shared-key validation, and backend
+ * map match fe-core AzureProperties. Legacy uppercase aliases remain accepted so
+ * existing FE filesystem callers can migrate through {@link AzureFileSystemProvider#bind(Map)}.
+ */
+public final class AzureFileSystemProperties
+ implements FileSystemProperties, BackendStorageProperties, HadoopStorageProperties {
+
+ public static final String ENDPOINT = "azure.endpoint";
+ public static final String ACCOUNT_NAME = "azure.account_name";
+ public static final String ACCOUNT_KEY = "azure.account_key";
+ public static final String CLIENT_ID = "azure.oauth2_client_id";
+ public static final String CLIENT_SECRET = "azure.oauth2_client_secret";
+ public static final String OAUTH_SERVER_URI = "azure.oauth2_server_uri";
+ public static final String OAUTH_ACCOUNT_HOST = "azure.oauth2_account_host";
+ public static final String TENANT_ID = "azure.oauth2_client_tenant_id";
+ public static final String AUTH_TYPE = "azure.auth_type";
+ public static final String CONTAINER = "container";
+ public static final String USE_PATH_STYLE = "use_path_style";
+ public static final String FORCE_PARSING_BY_STANDARD_URI = "force_parsing_by_standard_uri";
+
+ public static final String SHARED_KEY_AUTH = "SharedKey";
+ public static final String OAUTH2_AUTH = "OAuth2";
+ public static final String AZURE_ENDPOINT_TEMPLATE = "https://%s.blob.core.windows.net";
+
+ private static final String[] AZURE_BLOB_HOST_SUFFIXES = {
+ "blob.core.windows.net",
+ "blob.core.chinacloudapi.cn",
+ "blob.core.usgovcloudapi.net",
+ "blob.core.cloudapi.de"
+ };
+
+ @ConnectorProperty(names = {ENDPOINT, "s3.endpoint", "AWS_ENDPOINT", "endpoint", "ENDPOINT",
+ "AZURE_ENDPOINT"},
+ required = false,
+ description = "The endpoint of Azure Blob Storage.")
+ private String endpoint = "";
+
+ @ConnectorProperty(names = {ACCOUNT_NAME, "azure.access_key", "s3.access_key",
+ "AWS_ACCESS_KEY", "ACCESS_KEY", "access_key", "AZURE_ACCOUNT_NAME"},
+ required = false,
+ description = "The Azure storage account name.")
+ private String accountName = "";
+
+ @ConnectorProperty(names = {ACCOUNT_KEY, "azure.secret_key", "s3.secret_key",
+ "AWS_SECRET_KEY", "secret_key", "SECRET_KEY", "AZURE_ACCOUNT_KEY"},
+ required = false,
+ sensitive = true,
+ description = "The Azure storage account key.")
+ private String accountKey = "";
+
+ @ConnectorProperty(names = {CLIENT_ID, "AZURE_CLIENT_ID"},
+ required = false,
+ description = "The client id of Azure AD application.")
+ private String clientId = "";
+
+ @ConnectorProperty(names = {CLIENT_SECRET, "AZURE_CLIENT_SECRET"},
+ required = false,
+ sensitive = true,
+ description = "The client secret of Azure AD application.")
+ private String clientSecret = "";
+
+ @ConnectorProperty(names = {OAUTH_SERVER_URI},
+ required = false,
+ description = "The Azure OAuth2 token endpoint.")
+ private String oauthServerUri = "";
+
+ @ConnectorProperty(names = {OAUTH_ACCOUNT_HOST},
+ required = false,
+ description = "The Azure account host used by Hadoop OAuth2 config.")
+ private String oauthAccountHost = "";
+
+ @ConnectorProperty(names = {TENANT_ID, "AZURE_TENANT_ID"},
+ required = false,
+ description = "The Azure AD tenant id used by the native Azure SDK.")
+ private String tenantId = "";
+
+ @ConnectorProperty(names = {AUTH_TYPE},
+ required = false,
+ description = "The auth type of Azure Blob Storage.")
+ private String azureAuthType = SHARED_KEY_AUTH;
+
+ @ConnectorProperty(names = {CONTAINER, "azure.bucket", "azure.container", "s3.bucket",
+ "AZURE_CONTAINER", "AZURE_BUCKET", "AWS_BUCKET"},
+ required = false,
+ description = "The Azure container name.")
+ private String container = "";
+
+ @ConnectorProperty(names = {USE_PATH_STYLE, "s3.path-style-access"},
+ required = false,
+ description = "Whether to use path style URL for the storage.")
+ private String usePathStyle = "false";
+
+ @ConnectorProperty(names = {FORCE_PARSING_BY_STANDARD_URI},
+ required = false,
+ description = "Whether to force standard URI parsing.")
+ private String forceParsingByStandardUrl = "false";
+
+ private final Map<String, String> rawProperties;
+ private final Map<String, String> matchedProperties;
+
+ private AzureFileSystemProperties(Map<String, String> rawProperties) {
+ this.rawProperties = Collections.unmodifiableMap(new HashMap<>(rawProperties));
Review Comment:
Why did you add another layer of hashmap?
##########
fe/fe-filesystem/fe-filesystem-azure/src/main/java/org/apache/doris/filesystem/azure/AzureFileSystemProperties.java:
##########
@@ -0,0 +1,404 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.filesystem.azure;
+
+import org.apache.doris.filesystem.FileSystemType;
+import org.apache.doris.filesystem.properties.BackendStorageKind;
+import org.apache.doris.filesystem.properties.BackendStorageProperties;
+import org.apache.doris.filesystem.properties.FileSystemProperties;
+import org.apache.doris.filesystem.properties.HadoopStorageProperties;
+import org.apache.doris.filesystem.properties.StorageKind;
+import org.apache.doris.foundation.property.ConnectorPropertiesUtils;
+import org.apache.doris.foundation.property.ConnectorProperty;
+import org.apache.doris.foundation.property.ParamRules;
+
+import org.apache.commons.lang3.StringUtils;
+
+import java.lang.reflect.Field;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedHashSet;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Provider-owned Azure Blob Storage properties.
+ *
+ * <p>The public aliases, endpoint formatting, shared-key validation, and backend
+ * map match fe-core AzureProperties. Legacy uppercase aliases remain accepted so
+ * existing FE filesystem callers can migrate through {@link AzureFileSystemProvider#bind(Map)}.
+ */
+public final class AzureFileSystemProperties
+ implements FileSystemProperties, BackendStorageProperties, HadoopStorageProperties {
+
+ public static final String ENDPOINT = "azure.endpoint";
+ public static final String ACCOUNT_NAME = "azure.account_name";
+ public static final String ACCOUNT_KEY = "azure.account_key";
+ public static final String CLIENT_ID = "azure.oauth2_client_id";
+ public static final String CLIENT_SECRET = "azure.oauth2_client_secret";
+ public static final String OAUTH_SERVER_URI = "azure.oauth2_server_uri";
+ public static final String OAUTH_ACCOUNT_HOST = "azure.oauth2_account_host";
+ public static final String TENANT_ID = "azure.oauth2_client_tenant_id";
+ public static final String AUTH_TYPE = "azure.auth_type";
+ public static final String CONTAINER = "container";
+ public static final String USE_PATH_STYLE = "use_path_style";
+ public static final String FORCE_PARSING_BY_STANDARD_URI = "force_parsing_by_standard_uri";
+
+ public static final String SHARED_KEY_AUTH = "SharedKey";
+ public static final String OAUTH2_AUTH = "OAuth2";
+ public static final String AZURE_ENDPOINT_TEMPLATE = "https://%s.blob.core.windows.net";
+
+ private static final String[] AZURE_BLOB_HOST_SUFFIXES = {
+ "blob.core.windows.net",
+ "blob.core.chinacloudapi.cn",
+ "blob.core.usgovcloudapi.net",
+ "blob.core.cloudapi.de"
+ };
+
+ @ConnectorProperty(names = {ENDPOINT, "s3.endpoint", "AWS_ENDPOINT", "endpoint", "ENDPOINT",
+ "AZURE_ENDPOINT"},
+ required = false,
+ description = "The endpoint of Azure Blob Storage.")
+ private String endpoint = "";
+
+ @ConnectorProperty(names = {ACCOUNT_NAME, "azure.access_key", "s3.access_key",
+ "AWS_ACCESS_KEY", "ACCESS_KEY", "access_key", "AZURE_ACCOUNT_NAME"},
+ required = false,
+ description = "The Azure storage account name.")
+ private String accountName = "";
+
+ @ConnectorProperty(names = {ACCOUNT_KEY, "azure.secret_key", "s3.secret_key",
+ "AWS_SECRET_KEY", "secret_key", "SECRET_KEY", "AZURE_ACCOUNT_KEY"},
+ required = false,
+ sensitive = true,
+ description = "The Azure storage account key.")
+ private String accountKey = "";
+
+ @ConnectorProperty(names = {CLIENT_ID, "AZURE_CLIENT_ID"},
+ required = false,
+ description = "The client id of Azure AD application.")
+ private String clientId = "";
+
+ @ConnectorProperty(names = {CLIENT_SECRET, "AZURE_CLIENT_SECRET"},
+ required = false,
+ sensitive = true,
+ description = "The client secret of Azure AD application.")
+ private String clientSecret = "";
+
+ @ConnectorProperty(names = {OAUTH_SERVER_URI},
+ required = false,
+ description = "The Azure OAuth2 token endpoint.")
+ private String oauthServerUri = "";
+
+ @ConnectorProperty(names = {OAUTH_ACCOUNT_HOST},
+ required = false,
+ description = "The Azure account host used by Hadoop OAuth2 config.")
+ private String oauthAccountHost = "";
+
+ @ConnectorProperty(names = {TENANT_ID, "AZURE_TENANT_ID"},
+ required = false,
+ description = "The Azure AD tenant id used by the native Azure SDK.")
+ private String tenantId = "";
+
+ @ConnectorProperty(names = {AUTH_TYPE},
+ required = false,
+ description = "The auth type of Azure Blob Storage.")
+ private String azureAuthType = SHARED_KEY_AUTH;
+
+ @ConnectorProperty(names = {CONTAINER, "azure.bucket", "azure.container", "s3.bucket",
+ "AZURE_CONTAINER", "AZURE_BUCKET", "AWS_BUCKET"},
+ required = false,
+ description = "The Azure container name.")
+ private String container = "";
+
+ @ConnectorProperty(names = {USE_PATH_STYLE, "s3.path-style-access"},
+ required = false,
+ description = "Whether to use path style URL for the storage.")
+ private String usePathStyle = "false";
+
+ @ConnectorProperty(names = {FORCE_PARSING_BY_STANDARD_URI},
+ required = false,
+ description = "Whether to force standard URI parsing.")
+ private String forceParsingByStandardUrl = "false";
+
+ private final Map<String, String> rawProperties;
+ private final Map<String, String> matchedProperties;
+
+ private AzureFileSystemProperties(Map<String, String> rawProperties) {
+ this.rawProperties = Collections.unmodifiableMap(new HashMap<>(rawProperties));
+ this.matchedProperties = Collections.unmodifiableMap(collectMatchedProperties(rawProperties));
+ ConnectorPropertiesUtils.bindConnectorProperties(this, rawProperties);
+ normalize();
+ }
+
+ public static AzureFileSystemProperties of(Map<String, String> properties) {
+ AzureFileSystemProperties props = new AzureFileSystemProperties(properties);
+ props.validate();
+ return props;
+ }
+
+ @Override
+ public void validate() {
+ new ParamRules()
+ .check(() -> !isSharedKeyAuth() && !isOauth2Auth(),
+ "Unsupported Azure auth_type: " + azureAuthType)
+ .check(() -> isSharedKeyAuth()
+ && (StringUtils.isBlank(accountName) || StringUtils.isBlank(accountKey)),
+ "When auth_type is SharedKey, account_name and account_key are required.")
+ .check(() -> isOauth2Auth()
+ && (StringUtils.isBlank(oauthAccountHost)
+ || StringUtils.isBlank(clientId)
+ || StringUtils.isBlank(clientSecret)
+ || StringUtils.isBlank(oauthServerUri)),
+ "When auth_type is OAuth2, oauth2_account_host, oauth2_client_id, "
+ + "oauth2_client_secret, and oauth2_server_uri are required.")
+ .validate("Invalid Azure filesystem properties");
+ }
+
+ @Override
+ public String providerName() {
+ return "AZURE";
+ }
+
+ @Override
+ public StorageKind kind() {
+ return StorageKind.OBJECT_STORAGE;
+ }
+
+ @Override
+ public FileSystemType type() {
+ return FileSystemType.AZURE;
+ }
+
+ @Override
+ public Map<String, String> rawProperties() {
+ return rawProperties;
+ }
+
+ @Override
+ public Map<String, String> matchedProperties() {
+ return matchedProperties;
+ }
+
+ @Override
+ public Optional<BackendStorageProperties> toBackendProperties() {
+ return Optional.of(this);
+ }
+
+ @Override
+ public Optional<HadoopStorageProperties> toHadoopProperties() {
+ return Optional.of(this);
+ }
+
+ @Override
+ public BackendStorageKind backendKind() {
+ return BackendStorageKind.S3_COMPATIBLE;
+ }
+
+ @Override
+ public Map<String, String> toMap() {
+ if (isOauth2Auth()) {
+ return toHadoopConfigurationMap();
+ }
Review Comment:
add comment to explain why
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fs)Add native filesystem SPI for object storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63400:
URL: https://github.com/apache/doris/pull/63400#issuecomment-4630131514
<details>
<summary>TPC-DS: <b>Total hot run time: 169068 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6f6e5d9db34d01d22f4d4c575d5f75ace5621d1b, data reload: false
query5 4316 617 472 472
query6 446 195 178 178
query7 4898 576 297 297
query8 376 228 207 207
query9 8723 3992 4039 3992
query10 464 312 260 260
query11 5891 2336 2216 2216
query12 155 101 100 100
query13 1269 609 428 428
query14 6428 5405 5091 5091
query14_1 4394 4420 4318 4318
query15 205 193 173 173
query16 1004 447 387 387
query17 1099 680 566 566
query18 2487 466 330 330
query19 198 177 134 134
query20 112 107 108 107
query21 223 138 113 113
query22 13700 13587 13296 13296
query23 17191 16460 16248 16248
query23_1 16213 16312 16259 16259
query24 7580 1762 1297 1297
query24_1 1302 1326 1323 1323
query25 540 448 380 380
query26 1314 323 168 168
query27 2678 566 332 332
query28 4442 2030 1995 1995
query29 1097 596 482 482
query30 309 236 204 204
query31 1124 1080 959 959
query32 109 65 60 60
query33 531 332 263 263
query34 1184 1109 650 650
query35 748 794 685 685
query36 1410 1402 1237 1237
query37 153 107 94 94
query38 3202 3145 3045 3045
query39 941 938 913 913
query39_1 871 874 871 871
query40 230 130 107 107
query41 71 67 69 67
query42 100 97 101 97
query43 321 320 282 282
query44
query45 196 199 179 179
query46 1070 1172 713 713
query47 2403 2450 2273 2273
query48 411 411 304 304
query49 643 489 375 375
query50 982 354 262 262
query51 4333 4332 4260 4260
query52 90 90 78 78
query53 248 269 197 197
query54 283 232 217 217
query55 83 79 69 69
query56 251 244 237 237
query57 1436 1404 1363 1363
query58 260 221 226 221
query59 1581 1618 1393 1393
query60 303 257 252 252
query61 186 184 181 181
query62 713 673 584 584
query63 236 188 189 188
query64 2601 838 627 627
query65
query66 1769 462 339 339
query67 29885 29683 29429 29429
query68
query69 415 301 255 255
query70 953 956 939 939
query71 319 227 214 214
query72 2854 2662 2386 2386
query73 873 746 400 400
query74 5183 4901 4800 4800
query75 2674 2608 2241 2241
query76 2313 1139 742 742
query77 347 378 283 283
query78 12397 12550 11880 11880
query79 1370 977 750 750
query80 704 458 386 386
query81 473 277 240 240
query82 586 160 127 127
query83 341 273 245 245
query84 254 142 112 112
query85 924 520 434 434
query86 392 308 266 266
query87 3384 3355 3182 3182
query88 3623 2757 2750 2750
query89 425 370 333 333
query90 1875 186 181 181
query91 176 164 136 136
query92 63 65 57 57
query93 1495 1431 878 878
query94 576 365 296 296
query95 671 469 338 338
query96 1123 758 339 339
query97 2688 2736 2554 2554
query98 207 204 204 204
query99 1138 1177 1036 1036
Total cold run time: 251273 ms
Total hot run time: 169068 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fs)Add native filesystem SPI for object storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63400:
URL: https://github.com/apache/doris/pull/63400#issuecomment-4630762545
<details>
<summary>TPC-DS: <b>Total hot run time: 169882 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8f382cf1d00d81c0e65947e3a0fcfa6a873300a3, data reload: false
query5 4367 634 477 477
query6 442 195 202 195
query7 4843 561 297 297
query8 397 216 201 201
query9 8764 4062 4048 4048
query10 454 316 260 260
query11 5817 2355 2150 2150
query12 160 106 102 102
query13 1270 592 440 440
query14 6402 5482 5093 5093
query14_1 4438 4443 4433 4433
query15 207 199 178 178
query16 1032 466 440 440
query17 1146 716 608 608
query18 2557 498 350 350
query19 210 192 153 153
query20 117 119 108 108
query21 221 148 123 123
query22 13674 13648 13408 13408
query23 17392 16510 16227 16227
query23_1 16240 16348 16336 16336
query24 7505 1806 1337 1337
query24_1 1333 1330 1301 1301
query25 594 481 412 412
query26 1311 323 180 180
query27 2613 522 357 357
query28 4436 2064 2047 2047
query29 1089 640 512 512
query30 317 238 199 199
query31 1114 1076 981 981
query32 113 63 64 63
query33 530 363 260 260
query34 1190 1134 668 668
query35 748 791 678 678
query36 1385 1372 1200 1200
query37 157 111 85 85
query38 3209 3162 3041 3041
query39 939 929 892 892
query39_1 882 877 873 873
query40 219 144 122 122
query41 82 62 62 62
query42 97 96 94 94
query43 326 327 279 279
query44
query45 193 189 183 183
query46 1084 1201 724 724
query47 2323 2406 2283 2283
query48 406 412 277 277
query49 618 468 360 360
query50 1052 367 265 265
query51 4319 4292 4170 4170
query52 91 93 78 78
query53 255 270 193 193
query54 276 217 204 204
query55 81 77 70 70
query56 236 237 215 215
query57 1422 1414 1320 1320
query58 248 223 210 210
query59 1573 1705 1450 1450
query60 288 254 229 229
query61 168 153 160 153
query62 701 660 575 575
query63 230 187 192 187
query64 2559 811 635 635
query65
query66 1765 464 346 346
query67 29854 29778 29620 29620
query68
query69 426 302 270 270
query70 974 935 970 935
query71 300 225 212 212
query72 3066 2741 2459 2459
query73 848 787 450 450
query74 5116 5017 4768 4768
query75 2662 2603 2251 2251
query76 2358 1179 796 796
query77 353 363 271 271
query78 12336 12328 11817 11817
query79 1481 1086 779 779
query80 710 466 394 394
query81 468 283 243 243
query82 569 160 117 117
query83 353 272 254 254
query84 262 142 117 117
query85 911 554 452 452
query86 417 313 290 290
query87 3444 3396 3204 3204
query88 3668 2757 2711 2711
query89 436 385 327 327
query90 1879 194 184 184
query91 177 173 144 144
query92 65 63 54 54
query93 1468 1484 907 907
query94 620 367 319 319
query95 722 394 359 359
query96 1123 833 352 352
query97 2697 2683 2554 2554
query98 215 209 209 209
query99 1177 1183 1045 1045
Total cold run time: 251664 ms
Total hot run time: 169882 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fs)Add native filesystem SPI for object storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63400:
URL: https://github.com/apache/doris/pull/63400#issuecomment-4621834841
# FE UT Coverage Report
Increment line coverage ` 54.55% (6/11)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63400_4114b20c466abfb34c93be1ebe90a629566b627d/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63400_4114b20c466abfb34c93be1ebe90a629566b627d/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fs)Add native filesystem SPI for object storage [doris]
Posted by "CalvinKirs (via GitHub)" <gi...@apache.org>.
CalvinKirs commented on PR #63400:
URL: https://github.com/apache/doris/pull/63400#issuecomment-4628008067
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fs)Add native filesystem SPI for object storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63400:
URL: https://github.com/apache/doris/pull/63400#issuecomment-4518048525
<details>
<summary>TPC-H: <b>Total hot run time: 31544 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 814a8ad08ed9f8bc4f0df8564c67bc4121570cca, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17670 3835 3826 3826
q2 q3 10792 1400 827 827
q4 4716 482 348 348
q5 8093 2286 2092 2092
q6 305 175 142 142
q7 959 777 661 661
q8 9366 1732 1639 1639
q9 7088 4875 4948 4875
q10 6450 2130 1833 1833
q11 438 272 245 245
q12 694 429 294 294
q13 18229 3447 2793 2793
q14 260 250 233 233
q15 q16 829 763 710 710
q17 907 966 963 963
q18 7004 5932 5761 5761
q19 1201 1446 1131 1131
q20 527 420 264 264
q21 6134 2816 2591 2591
q22 462 380 316 316
Total cold run time: 102124 ms
Total hot run time: 31544 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4597 4519 4721 4519
q2 q3 4924 5166 4625 4625
q4 2130 2201 1441 1441
q5 4878 4715 4620 4620
q6 232 184 128 128
q7 1878 1713 1535 1535
q8 2364 1909 1905 1905
q9 7277 7245 7224 7224
q10 4482 4444 3996 3996
q11 526 382 347 347
q12 719 720 505 505
q13 3027 3449 2830 2830
q14 275 280 246 246
q15 q16 678 714 604 604
q17 1274 1245 1243 1243
q18 7368 6709 6794 6709
q19 1150 1099 1107 1099
q20 2211 2204 1923 1923
q21 5308 4619 4496 4496
q22 518 452 443 443
Total cold run time: 55816 ms
Total hot run time: 50438 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fs)Add native filesystem SPI for object storage [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #63400:
URL: https://github.com/apache/doris/pull/63400#issuecomment-4706810377
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fs)Add native filesystem SPI for object storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63400:
URL: https://github.com/apache/doris/pull/63400#issuecomment-4704716172
<details>
<summary>TPC-H: <b>Total hot run time: 28872 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7ffa22bbb469a317eb5512959a32b686da2822f6, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17786 4069 4075 4069
q2 q3 10767 1388 798 798
q4 4678 475 341 341
q5 7563 866 563 563
q6 174 164 136 136
q7 772 849 610 610
q8 9345 1599 1649 1599
q9 5746 4486 4565 4486
q10 6772 1781 1532 1532
q11 431 272 247 247
q12 625 429 295 295
q13 18114 3439 2741 2741
q14 263 258 243 243
q15 q16 825 761 707 707
q17 939 957 903 903
q18 7201 5862 5597 5597
q19 1296 1199 1121 1121
q20 514 391 264 264
q21 5831 2574 2315 2315
q22 433 367 305 305
Total cold run time: 100075 ms
Total hot run time: 28872 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4303 4235 4241 4235
q2 q3 4550 4944 4334 4334
q4 2072 2195 1357 1357
q5 4447 4316 4327 4316
q6 228 175 127 127
q7 1731 1615 1445 1445
q8 2956 2230 2131 2131
q9 8111 8346 7998 7998
q10 4817 4753 4263 4263
q11 585 417 389 389
q12 740 751 543 543
q13 3248 3662 2959 2959
q14 300 296 283 283
q15 q16 733 718 629 629
q17 1347 1325 1309 1309
q18 7978 7292 7232 7232
q19 1188 1124 1113 1113
q20 2212 2206 1930 1930
q21 5233 4528 4392 4392
q22 501 442 407 407
Total cold run time: 57280 ms
Total hot run time: 51392 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fs)Add native filesystem SPI for object storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63400:
URL: https://github.com/apache/doris/pull/63400#issuecomment-4532984413
# FE Regression Coverage Report
Increment line coverage ` 0.00% (0/68)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63400_9a283b39f0034b331578533fce6bae2bf53770b4_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63400_9a283b39f0034b331578533fce6bae2bf53770b4_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fs)Add native filesystem SPI for object storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63400:
URL: https://github.com/apache/doris/pull/63400#issuecomment-4532139099
<details>
<summary>TPC-H: <b>Total hot run time: 31770 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9a283b39f0034b331578533fce6bae2bf53770b4, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17607 4065 4032 4032
q2 q3 10825 1366 831 831
q4 4685 480 343 343
q5 7582 2227 2070 2070
q6 226 178 137 137
q7 984 778 636 636
q8 9486 1736 1666 1666
q9 5204 4974 4966 4966
q10 6370 2175 1874 1874
q11 424 275 240 240
q12 625 417 296 296
q13 18093 3325 2728 2728
q14 263 262 239 239
q15 q16 822 774 710 710
q17 996 1003 920 920
q18 7042 5828 5544 5544
q19 1292 1347 1137 1137
q20 596 516 276 276
q21 6432 2857 2756 2756
q22 470 369 448 369
Total cold run time: 100024 ms
Total hot run time: 31770 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4756 4857 4799 4799
q2 q3 4866 5365 4681 4681
q4 2149 2213 1425 1425
q5 4779 4983 4744 4744
q6 232 180 126 126
q7 1892 1824 1576 1576
q8 2518 2161 2224 2161
q9 8129 7980 7469 7469
q10 4736 4664 4267 4267
q11 533 384 356 356
q12 740 738 545 545
q13 3069 3310 2812 2812
q14 275 296 263 263
q15 q16 685 704 610 610
q17 1319 1290 1296 1290
q18 7193 6847 6734 6734
q19 1134 1102 1127 1102
q20 2235 2216 1966 1966
q21 5310 4603 4505 4505
q22 538 465 415 415
Total cold run time: 57088 ms
Total hot run time: 51846 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fs)Add native filesystem SPI for object storage [doris]
Posted by "CalvinKirs (via GitHub)" <gi...@apache.org>.
CalvinKirs commented on PR #63400:
URL: https://github.com/apache/doris/pull/63400#issuecomment-4532015333
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fs)Add native filesystem SPI for object storage [doris]
Posted by "CalvinKirs (via GitHub)" <gi...@apache.org>.
CalvinKirs commented on PR #63400:
URL: https://github.com/apache/doris/pull/63400#issuecomment-4517491320
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fs)Add native filesystem SPI for object storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63400:
URL: https://github.com/apache/doris/pull/63400#issuecomment-4518115827
<details>
<summary>TPC-DS: <b>Total hot run time: 169054 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 814a8ad08ed9f8bc4f0df8564c67bc4121570cca, data reload: false
query5 4306 662 516 516
query6 336 223 199 199
query7 4225 559 305 305
query8 351 232 220 220
query9 8840 3942 4009 3942
query10 452 370 321 321
query11 5821 2527 2204 2204
query12 182 129 122 122
query13 1311 578 419 419
query14 5915 5330 5051 5051
query14_1 4313 4322 4305 4305
query15 211 204 184 184
query16 1015 450 438 438
query17 942 742 610 610
query18 2450 496 368 368
query19 226 212 171 171
query20 134 136 132 132
query21 219 143 121 121
query22 13520 13563 13332 13332
query23 17194 16405 16081 16081
query23_1 16212 16163 16251 16163
query24 7474 1771 1301 1301
query24_1 1299 1322 1312 1312
query25 553 499 456 456
query26 1313 307 182 182
query27 2705 565 353 353
query28 4486 1943 1950 1943
query29 1002 666 526 526
query30 309 246 199 199
query31 1113 1079 939 939
query32 91 73 79 73
query33 573 363 317 317
query34 1158 1167 660 660
query35 771 793 676 676
query36 1323 1337 1189 1189
query37 159 108 99 99
query38 3192 3138 3055 3055
query39 944 940 901 901
query39_1 877 889 881 881
query40 234 149 130 130
query41 72 76 71 71
query42 112 109 111 109
query43 322 328 280 280
query44
query45 218 206 196 196
query46 1070 1178 748 748
query47 2384 2358 2244 2244
query48 406 443 303 303
query49 653 506 402 402
query50 970 338 249 249
query51 4316 4234 4262 4234
query52 104 106 97 97
query53 250 275 201 201
query54 327 293 266 266
query55 96 94 88 88
query56 316 328 325 325
query57 1437 1426 1281 1281
query58 318 286 283 283
query59 1563 1587 1384 1384
query60 315 324 307 307
query61 163 155 161 155
query62 662 624 564 564
query63 235 198 198 198
query64 2376 795 634 634
query65
query66 1723 473 362 362
query67 29887 29892 29851 29851
query68
query69 472 344 303 303
query70 1030 1020 1022 1020
query71 303 279 268 268
query72 3059 2742 2426 2426
query73 866 751 414 414
query74 5065 4982 4702 4702
query75 2664 2586 2257 2257
query76 2292 1132 771 771
query77 394 413 338 338
query78 12165 12055 11680 11680
query79 1499 1105 740 740
query80 1334 549 461 461
query81 500 285 239 239
query82 1332 160 123 123
query83 361 277 250 250
query84 302 145 109 109
query85 937 529 444 444
query86 431 335 319 319
query87 3412 3398 3197 3197
query88 3579 2675 2668 2668
query89 443 380 336 336
query90 1797 178 178 178
query91 178 171 145 145
query92 86 76 75 75
query93 1431 1566 836 836
query94 675 352 311 311
query95 692 368 347 347
query96 1067 761 333 333
query97 2695 2705 2548 2548
query98 235 228 224 224
query99 1119 1106 1010 1010
Total cold run time: 253296 ms
Total hot run time: 169054 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fs)Add native filesystem SPI for object storage [doris]
Posted by "CalvinKirs (via GitHub)" <gi...@apache.org>.
CalvinKirs commented on PR #63400:
URL: https://github.com/apache/doris/pull/63400#issuecomment-4628857072
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fs)Add native filesystem SPI for object storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63400:
URL: https://github.com/apache/doris/pull/63400#issuecomment-4630096678
# FE Regression Coverage Report
Increment line coverage ` 9.72% (7/72)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63400_8f382cf1d00d81c0e65947e3a0fcfa6a873300a3_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63400_8f382cf1d00d81c0e65947e3a0fcfa6a873300a3_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org