You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by mo...@apache.org on 2022/12/24 09:09:56 UTC
[doris] 04/15: [fix](s3 outfile) Add the`use_path_style` parameter for s3 outfile (#15288)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 5c4a99dff9ebda02434f45c54db440ff7f9d6ed2
Author: Tiewei Fang <43...@users.noreply.github.com>
AuthorDate: Fri Dec 23 16:22:06 2022 +0800
[fix](s3 outfile) Add the`use_path_style` parameter for s3 outfile (#15288)
Currently, `outfile` did not support `use_path_style` parameter and use `virtual-host style` by default,
however some Object-storage may only support `use_path_style` access mode.
This pr add the`use_path_style` parameter for s3 outfile, so that different object-storage can use different access mode.
---
.../sql-manual/sql-reference/Data-Manipulation-Statements/OUTFILE.md | 1 +
.../sql-manual/sql-reference/Data-Manipulation-Statements/OUTFILE.md | 1 +
fe/fe-core/src/main/java/org/apache/doris/analysis/OutFileClause.java | 4 ++++
3 files changed, 6 insertions(+)
diff --git a/docs/en/docs/sql-manual/sql-reference/Data-Manipulation-Statements/OUTFILE.md b/docs/en/docs/sql-manual/sql-reference/Data-Manipulation-Statements/OUTFILE.md
index b094fcdd3f..d46b2164d7 100644
--- a/docs/en/docs/sql-manual/sql-reference/Data-Manipulation-Statements/OUTFILE.md
+++ b/docs/en/docs/sql-manual/sql-reference/Data-Manipulation-Statements/OUTFILE.md
@@ -101,6 +101,7 @@ illustrate:
AWS_ACCESS_KEY
AWS_SECRET_KEY
AWS_REGION
+ use_path_stype: (optional) default false . The S3 SDK uses the virtual-hosted style by default. However, some object storage systems may not be enabled or support virtual-hosted style access. At this time, we can add the use_path_style parameter to force the use of path style access method.
```
### example
diff --git a/docs/zh-CN/docs/sql-manual/sql-reference/Data-Manipulation-Statements/OUTFILE.md b/docs/zh-CN/docs/sql-manual/sql-reference/Data-Manipulation-Statements/OUTFILE.md
index 82f8ecec33..25b35f5894 100644
--- a/docs/zh-CN/docs/sql-manual/sql-reference/Data-Manipulation-Statements/OUTFILE.md
+++ b/docs/zh-CN/docs/sql-manual/sql-reference/Data-Manipulation-Statements/OUTFILE.md
@@ -104,6 +104,7 @@ INTO OUTFILE "file_path"
AWS_ACCESS_KEY
AWS_SECRET_KEY
AWS_REGION
+ use_path_stype: (选填) 默认为false 。S3 SDK 默认使用 virtual-hosted style 方式。但某些对象存储系统可能没开启或不支持virtual-hosted style 方式的访问，此时可以添加 use_path_style 参数来强制使用 path style 访问方式。
```
### example
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/OutFileClause.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/OutFileClause.java
index 3d3f82b74b..78f57cff9c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/OutFileClause.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/OutFileClause.java
@@ -648,6 +648,10 @@ public class OutFileClause {
}
}
if (storageType == StorageBackend.StorageType.S3) {
+ if (properties.containsKey(S3Resource.USE_PATH_STYLE)) {
+ brokerProps.put(S3Resource.USE_PATH_STYLE, properties.get(S3Resource.USE_PATH_STYLE));
+ processedPropKeys.add(S3Resource.USE_PATH_STYLE);
+ }
S3Storage.checkS3(brokerProps);
} else if (storageType == StorageBackend.StorageType.HDFS) {
if (!brokerProps.containsKey(HdfsResource.HADOOP_FS_NAME)) {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org