You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by mo...@apache.org on 2021/03/27 11:00:06 UTC
[incubator-doris] branch master updated: [DOC] Add docs of load and
export using S3 protocol (#5551)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new 86af8c7 [DOC] Add docs of load and export using S3 protocol (#5551)
86af8c7 is described below
commit 86af8c76a36843a99c1fb95c4c4e6355792f4066
Author: Zhengguo Yang <ya...@gmail.com>
AuthorDate: Sat Mar 27 18:58:29 2021 +0800
[DOC] Add docs of load and export using S3 protocol (#5551)
Add docs of load and export using S3 protocol
---
be/src/common/config.h | 11 +++
be/src/common/daemon.cpp | 2 +-
docs/.vuepress/public/images/aws_ak_sk.png | Bin 0 -> 702853 bytes
docs/.vuepress/sidebar/en.js | 1 +
docs/.vuepress/sidebar/zh-CN.js | 1 +
docs/en/administrator-guide/config/be_config.md | 15 ++++
docs/en/administrator-guide/export-manual.md | 2 +-
.../administrator-guide/load-data/load-manual.md | 6 +-
.../load-data/s3-load-manual.md | 78 ++++++++++++++++++++
.../Data Manipulation/BROKER LOAD.md | 20 +++++-
.../sql-statements/Data Manipulation/EXPORT.md | 10 +--
.../sql-statements/Data Manipulation/LOAD.md | 1 +
docs/zh-CN/administrator-guide/backup-restore.md | 2 +-
docs/zh-CN/administrator-guide/config/be_config.md | 15 ++++
docs/zh-CN/administrator-guide/export-manual.md | 2 +-
.../administrator-guide/load-data/load-manual.md | 6 +-
.../load-data/s3-load-manual.md | 79 +++++++++++++++++++++
.../Data Manipulation/BROKER LOAD.md | 14 +++-
.../sql-statements/Data Manipulation/EXPORT.md | 10 +--
.../sql-statements/Data Manipulation/LOAD.md | 1 +
20 files changed, 254 insertions(+), 22 deletions(-)
diff --git a/be/src/common/config.h b/be/src/common/config.h
index 8d2736b..f1547f1 100644
--- a/be/src/common/config.h
+++ b/be/src/common/config.h
@@ -574,8 +574,19 @@ CONF_String(thrift_server_type_of_fe, "THREAD_POOL");
// disable zone map index when page row is too few
CONF_mInt32(zone_map_row_num_threshold, "20");
+// aws sdk log level
+// Off = 0,
+// Fatal = 1,
+// Error = 2,
+// Warn = 3,
+// Info = 4,
+// Debug = 5,
+// Trace = 6
+CONF_Int32(aws_log_level, "3");
+
} // namespace config
} // namespace doris
#endif // DORIS_BE_SRC_COMMON_CONFIG_H
+
diff --git a/be/src/common/daemon.cpp b/be/src/common/daemon.cpp
index 07b7a5b..5b21ea0 100644
--- a/be/src/common/daemon.cpp
+++ b/be/src/common/daemon.cpp
@@ -270,7 +270,7 @@ void Daemon::init(int argc, char** argv, const std::vector<StorePath>& paths) {
TopNFunctions::init();
// disable EC2 metadata service
setenv("AWS_EC2_METADATA_DISABLED", "true", false);
- Aws::Utils::Logging::LogLevel logLevel = Aws::Utils::Logging::LogLevel::Info;
+ Aws::Utils::Logging::LogLevel logLevel = static_cast<Aws::Utils::Logging::LogLevel>(config::aws_log_level);
aws_options.loggingOptions.logLevel = logLevel;
aws_options.loggingOptions.logger_create_fn = [logLevel] {
return std::make_shared<DorisAWSLogger>(logLevel);
diff --git a/docs/.vuepress/public/images/aws_ak_sk.png b/docs/.vuepress/public/images/aws_ak_sk.png
new file mode 100644
index 0000000..821c9e8
Binary files /dev/null and b/docs/.vuepress/public/images/aws_ak_sk.png differ
diff --git a/docs/.vuepress/sidebar/en.js b/docs/.vuepress/sidebar/en.js
index 6cb6d7a..b12ff2e 100644
--- a/docs/.vuepress/sidebar/en.js
+++ b/docs/.vuepress/sidebar/en.js
@@ -59,6 +59,7 @@ module.exports = [
"sequence-column-manual",
"spark-load-manual",
"stream-load-manual",
+ "s3-load-manual",
"delete-manual",
"insert-into-manual",
"load-json-format",
diff --git a/docs/.vuepress/sidebar/zh-CN.js b/docs/.vuepress/sidebar/zh-CN.js
index 1a40a59..7eaa21d 100644
--- a/docs/.vuepress/sidebar/zh-CN.js
+++ b/docs/.vuepress/sidebar/zh-CN.js
@@ -58,6 +58,7 @@ module.exports = [
"sequence-column-manual",
"spark-load-manual",
"stream-load-manual",
+ "s3-load-manual",
"delete-manual",
"insert-into-manual",
"load-json-format",
diff --git a/docs/en/administrator-guide/config/be_config.md b/docs/en/administrator-guide/config/be_config.md
index f2d1ce0..8dd31c1 100644
--- a/docs/en/administrator-guide/config/be_config.md
+++ b/docs/en/administrator-guide/config/be_config.md
@@ -870,3 +870,18 @@ If the parameter is `THREAD_POOL`, the model is a blocking I/O model.
* Type: int32
* Description: If the number of rows in a page is less than this value, no zonemap will be created to reduce data expansion
* Default: 20
+
+### `aws_log_level`
+
+* Type: int32
+* Description: log level of AWS SDK,
+ ```
+ Off = 0,
+ Fatal = 1,
+ Error = 2,
+ Warn = 3,
+ Info = 4,
+ Debug = 5,
+ Trace = 6
+ ```
+* Default: 3
\ No newline at end of file
diff --git a/docs/en/administrator-guide/export-manual.md b/docs/en/administrator-guide/export-manual.md
index 79c2680..0a04432 100644
--- a/docs/en/administrator-guide/export-manual.md
+++ b/docs/en/administrator-guide/export-manual.md
@@ -39,7 +39,7 @@ This document mainly introduces the basic principles, usage, best practices and
## Principle
-After the user submits an Export job. Doris counts all Tablets involved in this job. These tablets are then grouped to generate a special query plan for each group. The query plan reads the data on the included tablet and then writes the data to the specified path of the remote storage through Broker.
+After the user submits an Export job. Doris counts all Tablets involved in this job. These tablets are then grouped to generate a special query plan for each group. The query plan reads the data on the included tablet and then writes the data to the specified path of the remote storage through Broker. It can also be directly exported to the remote storage that supports S3 protocol through S3 protocol.
The overall mode of dispatch is as follows:
diff --git a/docs/en/administrator-guide/load-data/load-manual.md b/docs/en/administrator-guide/load-data/load-manual.md
index f5514e6..8d813a7 100644
--- a/docs/en/administrator-guide/load-data/load-manual.md
+++ b/docs/en/administrator-guide/load-data/load-manual.md
@@ -41,7 +41,7 @@ Doris supports multiple imports. It is recommended to read this document in full
## Load mode
-To adapt to different data import requirements, Doris system provides five different import methods. Each import mode supports different data sources and has different usage modes (asynchronous, synchronous).
+To adapt to different data import requirements, Doris system provides 6 different import methods. Each import mode supports different data sources and has different usage modes (asynchronous, synchronous).
All import methods support CSV data format. Broker load also supports parquet and orc data format.
@@ -67,6 +67,10 @@ For instructions on each import mode, please refer to the operation manual for a
Users submit routine import jobs through MySQL protocol, generate a resident thread, read and import data from data sources (such as Kafka) uninterruptedly into Doris.
+* Load through S3 protocol
+
+ Users directly load data through the S3 protocol, and the usage is similar to Broker Load
+
## Basic Principles
### Import execution process
diff --git a/docs/en/administrator-guide/load-data/s3-load-manual.md b/docs/en/administrator-guide/load-data/s3-load-manual.md
new file mode 100644
index 0000000..7e3aa28
--- /dev/null
+++ b/docs/en/administrator-guide/load-data/s3-load-manual.md
@@ -0,0 +1,78 @@
+---
+{
+"title": "S3 Load",
+"language": "zh-CN"
+}
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# S3 Load
+
+Starting from version 0.14, Doris supports the direct import of data from online storage systems that support the S3 protocol through the S3 protocol.
+
+This document mainly introduces how to import data stored in AWS S3. It also supports the import of other object storage systems that support the S3 protocol, such as Baidu Cloud’s BOS, Alibaba Cloud’s OSS and Tencent Cloud’s COS, etc.
+## Applicable scenarios
+
+* Source data in S3 protocol accessible storage systems, such as S3, BOS.
+* Data volumes range from tens to hundreds of GB.
+
+## Preparing
+1. Standard AK and SK
+ First, you need to find or regenerate AWS `Access keys`, you can find the generation method in `My Security Credentials` of AWS console, as shown in the following figure:
+ [AK_SK](/images/aws_ak_sk.png)
+ Select `Create New Access Key` and pay attention to save and generate AK and SK.
+2. Prepare REGION and ENDPOINT
+ REGION can be selected when creating the bucket or can be viewed in the bucket list. ENDPOINT can be found through REGION on the following page [AWS Documentation](https://docs.aws.amazon.com/general/latest/gr/s3.html#s3_region)
+
+Other cloud storage systems can find relevant information compatible with S3 in corresponding documents
+
+## Start Loading
+Like Broker Load just replace `WITH BROKER broker_name ()` with
+```
+ WITH S3
+ (
+ "AWS_ENDPOINT" = "AWS_ENDPOINT",
+ "AWS_ACCESS_KEY" = "AWS_ACCESS_KEY",
+ "AWS_SECRET_KEY"="AWS_SECRET_KEY",
+ "AWS_REGION" = "AWS_REGION"
+ )
+```
+
+example:
+```
+ LOAD LABEL example_db.exmpale_label_1
+ (
+ DATA INFILE("s3://your_bucket_name/your_file.txt")
+ INTO TABLE load_test
+ COLUMNS TERMINATED BY ","
+ )
+ WITH S3
+ (
+ "AWS_ENDPOINT" = "AWS_ENDPOINT",
+ "AWS_ACCESS_KEY" = "AWS_ACCESS_KEY",
+ "AWS_SECRET_KEY"="AWS_SECRET_KEY",
+ "AWS_REGION" = "AWS_REGION"
+ )
+ PROPERTIES
+ (
+ "timeout" = "3600"
+ );
+```
\ No newline at end of file
diff --git a/docs/en/sql-reference/sql-statements/Data Manipulation/BROKER LOAD.md b/docs/en/sql-reference/sql-statements/Data Manipulation/BROKER LOAD.md
index b77b5b4..bb05272 100644
--- a/docs/en/sql-reference/sql-statements/Data Manipulation/BROKER LOAD.md
+++ b/docs/en/sql-reference/sql-statements/Data Manipulation/BROKER LOAD.md
@@ -36,6 +36,7 @@ under the License.
2. Baidu AFS: afs for Baidu. Only be used inside Baidu.
3. Baidu Object Storage(BOS): BOS on Baidu Cloud.
4. Apache HDFS.
+ 5. Amazon S3：Amazon S3。
### Syntax:
@@ -43,8 +44,8 @@ under the License.
(
data_desc1[, data_desc2, ...]
)
- WITH BROKER broker_name
- [broker_properties]
+ WITH [BROKER broker_name | S3]
+ [load_properties]
[opt_properties];
1. load_label
@@ -136,7 +137,7 @@ under the License.
The name of the Broker used can be viewed through the `show broker` command.
- 4. broker_properties
+ 4. load_properties
Used to provide Broker access to data sources. Different brokers, and different access methods, need to provide different information.
@@ -174,6 +175,19 @@ under the License.
dfs.ha.namenodes.xxx: Customize the name of a namenode, separated by commas. XXX is a custom name in dfs. name services, such as "dfs. ha. namenodes. my_ha" = "my_nn"
dfs.namenode.rpc-address.xxx.nn: Specify RPC address information for namenode, where NN denotes the name of the namenode configured in dfs.ha.namenodes.xxxx, such as: "dfs.namenode.rpc-address.my_ha.my_nn"= "host:port"
dfs.client.failover.proxy.provider: Specify the provider that client connects to namenode by default: org. apache. hadoop. hdfs. server. namenode. ha. Configured Failover ProxyProvider.
+ 4. Amazon S3
+
+ fs.s3a.access.key：AmazonS3的access key
+ fs.s3a.secret.key：AmazonS3的secret key
+ fs.s3a.endpoint：AmazonS3的endpoint
+ 5. If using the S3 protocol to directly connect to the remote storage, you need to specify the following attributes
+
+ (
+ "AWS_ENDPOINT" = "",
+ "AWS_ACCESS_KEY" = "",
+ "AWS_SECRET_KEY"="",
+ "AWS_REGION" = ""
+ )
4. opt_properties
diff --git a/docs/en/sql-reference/sql-statements/Data Manipulation/EXPORT.md b/docs/en/sql-reference/sql-statements/Data Manipulation/EXPORT.md
index 6da41ec..0038827 100644
--- a/docs/en/sql-reference/sql-statements/Data Manipulation/EXPORT.md
+++ b/docs/en/sql-reference/sql-statements/Data Manipulation/EXPORT.md
@@ -37,7 +37,7 @@ under the License.
[WHERE [expr]]
TO export_path
[opt_properties]
- [broker];
+ [broker｜S3];
1. table_name
The table names to be exported currently support the export of tables with engine as OLAP and mysql.
@@ -63,11 +63,11 @@ under the License.
timeout: The time-out for importing jobs is 1 day by default, in seconds.
tablet_num_per_task: The maximum number of tablets that each subtask can allocate.
- 6. broker
- Broker used to specify export usage
+ 6. broker|S3
+ Specify to use broker export or export through S3 protocol
Grammar:
- WITH BROKER broker_name ("key"="value"[,...])
- Here you need to specify the specific broker name and the required broker attributes
+ WITH [BROKER broker_name| S3] ("key"="value"[,...])
+ Here you need to specify the specific broker name and the required broker attributes, If you use the S3 protocol, you do not need to specify the broker name
For brokers corresponding to different storage systems, the input parameters are different. Specific parameters can be referred to: `help broker load', broker required properties.
When exporting to local, you do not need to fill in this part.
diff --git a/docs/en/sql-reference/sql-statements/Data Manipulation/LOAD.md b/docs/en/sql-reference/sql-statements/Data Manipulation/LOAD.md
index 358df15..230bc67 100644
--- a/docs/en/sql-reference/sql-statements/Data Manipulation/LOAD.md
+++ b/docs/en/sql-reference/sql-statements/Data Manipulation/LOAD.md
@@ -33,6 +33,7 @@ Palo currently supports the following four import methods:
2. Broker Load: Use broker to import data.
3. Mini Load: Upload files through HTTP protocol for batch data import.
4. Stream Load: Stream data import through HTTP protocol.
+5. S3 Load: Directly access the storage system supporting the S3 protocol for data import through the S3 protocol. The import syntax is basically the same as that of Broker Load.
This help mainly describes the first import method, namely Hadoop Load related help information. The rest of the import methods can use the following commands to view help:
diff --git a/docs/zh-CN/administrator-guide/backup-restore.md b/docs/zh-CN/administrator-guide/backup-restore.md
index 2102e77..f3e094f 100644
--- a/docs/zh-CN/administrator-guide/backup-restore.md
+++ b/docs/zh-CN/administrator-guide/backup-restore.md
@@ -94,7 +94,7 @@ Doris 支持将当前数据以文件的形式，通过 broker 备份到远端存
1. CREATE REPOSITORY
- 创建一个远端仓库路径，用于备份或恢复。该命令需要借助 Broker 进程访问远端存储，不同的 Broker 需要提供不同的参数，具体请参阅 [Broker文档](broker.md)
+ 创建一个远端仓库路径，用于备份或恢复。该命令需要借助 Broker 进程访问远端存储，不同的 Broker 需要提供不同的参数，具体请参阅 [Broker文档](broker.md)，也可以直接通过S3 协议备份到支持AWS S3协议的远程存储上去，具体参考 [创建远程仓库文档](../sql-reference/sql-statements/Data Definition/CREATE REPOSITORY.md)
1. BACKUP
diff --git a/docs/zh-CN/administrator-guide/config/be_config.md b/docs/zh-CN/administrator-guide/config/be_config.md
index 289e208..8f97567 100644
--- a/docs/zh-CN/administrator-guide/config/be_config.md
+++ b/docs/zh-CN/administrator-guide/config/be_config.md
@@ -871,3 +871,18 @@ Stream Load 一般适用于导入几个GB以内的数据，不适合导入过大
* 描述: 如果一个page中的行数小于这个值就不会创建zonemap，用来减少数据膨胀
* 默认值: 20
+### `aws_log_level`
+
+* 类型: int32
+* 描述: AWS SDK 的日志级别
+ ```
+ Off = 0,
+ Fatal = 1,
+ Error = 2,
+ Warn = 3,
+ Info = 4,
+ Debug = 5,
+ Trace = 6
+ ```
+* 默认值: 3
+
diff --git a/docs/zh-CN/administrator-guide/export-manual.md b/docs/zh-CN/administrator-guide/export-manual.md
index 64306ce..a3f43be 100644
--- a/docs/zh-CN/administrator-guide/export-manual.md
+++ b/docs/zh-CN/administrator-guide/export-manual.md
@@ -39,7 +39,7 @@ under the License.
## 原理
-用户提交一个 Export 作业后。Doris 会统计这个作业涉及的所有 Tablet。然后对这些 Tablet 进行分组，每组生成一个特殊的查询计划。该查询计划会读取所包含的 Tablet 上的数据，然后通过 Broker 将数据写到远端存储指定的路径中。
+用户提交一个 Export 作业后。Doris 会统计这个作业涉及的所有 Tablet。然后对这些 Tablet 进行分组，每组生成一个特殊的查询计划。该查询计划会读取所包含的 Tablet 上的数据，然后通过 Broker 将数据写到远端存储指定的路径中，也可以通过S3协议直接导出到支持S3协议的远端存储上。
总体的调度方式如下:
diff --git a/docs/zh-CN/administrator-guide/load-data/load-manual.md b/docs/zh-CN/administrator-guide/load-data/load-manual.md
index eebc167..0b877a4 100644
--- a/docs/zh-CN/administrator-guide/load-data/load-manual.md
+++ b/docs/zh-CN/administrator-guide/load-data/load-manual.md
@@ -41,7 +41,7 @@ Doris 支持多种导入方式。建议先完整阅读本文档，再根据所
## 导入方式
-为适配不同的数据导入需求，Doris 系统提供了5种不同的导入方式。每种导入方式支持不同的数据源，存在不同的使用方式（异步，同步）。
+为适配不同的数据导入需求，Doris 系统提供了6种不同的导入方式。每种导入方式支持不同的数据源，存在不同的使用方式（异步，同步）。
所有导入方式都支持 csv 数据格式。其中 Broker load 还支持 parquet 和 orc 数据格式。
@@ -67,6 +67,10 @@ Doris 支持多种导入方式。建议先完整阅读本文档，再根据所
用户通过 MySQL 协议提交例行导入作业，生成一个常驻线程，不间断的从数据源（如 Kafka）中读取数据并导入到 Doris 中。
+* 通过S3协议直接导入
+
+ 用户通过S3协议直接导入数据，用法和Broker Load 类似
+
## 基本原理
### 导入执行流程
diff --git a/docs/zh-CN/administrator-guide/load-data/s3-load-manual.md b/docs/zh-CN/administrator-guide/load-data/s3-load-manual.md
new file mode 100644
index 0000000..4bf52ce
--- /dev/null
+++ b/docs/zh-CN/administrator-guide/load-data/s3-load-manual.md
@@ -0,0 +1,79 @@
+---
+{
+"title": "S3 Load",
+"language": "zh-CN"
+}
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# S3 Load
+
+从0.14 版本开始，Doris 支持通过S3协议直接从支持S3协议的在线存储系统导入数据。
+
+本文档主要介绍如何导入 AWS S3 中存储的数据。也支持导入其他支持S3协议的对象存储系统导入，如果百度云的BOS，阿里云的OSS和腾讯云的COS等、
+
+## 适用场景
+
+* 源数据在 支持S3协议的存储系统中，如 S3,BOS 等。
+* 数据量在 几十到百GB 级别。
+
+## 准备工作
+1. 准本AK 和 SK
+ 首先需要找到或者重新生成 AWS `Access keys`，可以在AWS console 的 `My Security Credentials` 找到生成方式， 如下图所示：
+ [AK_SK](/images/aws_ak_sk.png)
+ 选择 `Create New Access Key` 注意保存生成 AK和SK.
+2. 准备 REGION 和 ENDPOINT
+ REGION 可以在创建桶的时候选择也可以在桶列表中查看到。ENDPOINT 可以通过如下页面通过REGION查到 [AWS 文档](https://docs.aws.amazon.com/general/latest/gr/s3.html#s3_region)
+
+其他云存储系统可以相应的文档找到与S3兼容的相关信息
+
+## 开始导入
+导入方式和Broker Load 基本相同，只需要将 `WITH BROKER broker_name ()` 语句替换成如下部分
+```
+ WITH S3
+ (
+ "AWS_ENDPOINT" = "AWS_ENDPOINT",
+ "AWS_ACCESS_KEY" = "AWS_ACCESS_KEY",
+ "AWS_SECRET_KEY"="AWS_SECRET_KEY",
+ "AWS_REGION" = "AWS_REGION"
+ )
+```
+
+完整示例如下
+```
+ LOAD LABEL example_db.exmpale_label_1
+ (
+ DATA INFILE("s3://your_bucket_name/your_file.txt")
+ INTO TABLE load_test
+ COLUMNS TERMINATED BY ","
+ )
+ WITH S3
+ (
+ "AWS_ENDPOINT" = "AWS_ENDPOINT",
+ "AWS_ACCESS_KEY" = "AWS_ACCESS_KEY",
+ "AWS_SECRET_KEY"="AWS_SECRET_KEY",
+ "AWS_REGION" = "AWS_REGION"
+ )
+ PROPERTIES
+ (
+ "timeout" = "3600"
+ );
+```
\ No newline at end of file
diff --git a/docs/zh-CN/sql-reference/sql-statements/Data Manipulation/BROKER LOAD.md b/docs/zh-CN/sql-reference/sql-statements/Data Manipulation/BROKER LOAD.md
index 57f1bff..56dfeb5 100644
--- a/docs/zh-CN/sql-reference/sql-statements/Data Manipulation/BROKER LOAD.md
+++ b/docs/zh-CN/sql-reference/sql-statements/Data Manipulation/BROKER LOAD.md
@@ -43,8 +43,8 @@ under the License.
(
data_desc1[, data_desc2, ...]
)
- WITH BROKER broker_name
- [broker_properties]
+ WITH [BROKER broker_name | S3]
+ [load_properties]
[opt_properties];
1. load_label
@@ -136,7 +136,7 @@ under the License.
所使用的 broker 名称，可以通过 show broker 命令查看。
- 4. broker_properties
+ 4. load_properties
用于提供通过 broker 访问数据源的信息。不同的 broker，以及不同的访问方式，需要提供的信息不同。
@@ -180,6 +180,14 @@ under the License.
fs.s3a.access.key：AmazonS3的access key
fs.s3a.secret.key：AmazonS3的secret key
fs.s3a.endpoint：AmazonS3的endpoint
+ 5. 如果使用S3协议直接连接远程存储时需要指定如下属性
+
+ (
+ "AWS_ENDPOINT" = "",
+ "AWS_ACCESS_KEY" = "",
+ "AWS_SECRET_KEY"="",
+ "AWS_REGION" = ""
+ )
4. opt_properties
diff --git a/docs/zh-CN/sql-reference/sql-statements/Data Manipulation/EXPORT.md b/docs/zh-CN/sql-reference/sql-statements/Data Manipulation/EXPORT.md
index f41be07..e1a08c9 100644
--- a/docs/zh-CN/sql-reference/sql-statements/Data Manipulation/EXPORT.md
+++ b/docs/zh-CN/sql-reference/sql-statements/Data Manipulation/EXPORT.md
@@ -37,7 +37,7 @@ under the License.
[WHERE [expr]]
TO export_path
[opt_properties]
- [broker];
+ [broker|S3];
1. table_name
当前要导出的表的表名，目前支持engine为olap和mysql的表的导出。
@@ -63,11 +63,11 @@ under the License.
timeout：导入作业的超时时间，默认为1天，单位是秒。
tablet_num_per_task：每个子任务能分配的最大 Tablet 数量。
- 6. broker
- 用于指定导出使用的broker
+ 6. broker|s3
+ 指定使用broker导出或者通过S3协议导出
语法：
- WITH BROKER broker_name ("key"="value"[,...])
- 这里需要指定具体的broker name, 以及所需的broker属性
+ WITH [BROKER broker_name | S3] ("key"="value"[,...])
+ 这里需要指定具体的broker name, 以及所需的broker属性, 如果使用S3协议则无需指定broker name
对于不同存储系统对应的 broker，这里需要输入的参数不同。具体参数可以参阅：`help broker load` 中 broker 所需属性。
导出到本地时，不需要填写这部分。
diff --git a/docs/zh-CN/sql-reference/sql-statements/Data Manipulation/LOAD.md b/docs/zh-CN/sql-reference/sql-statements/Data Manipulation/LOAD.md
index 98f25c4..64fcc5c 100644
--- a/docs/zh-CN/sql-reference/sql-statements/Data Manipulation/LOAD.md
+++ b/docs/zh-CN/sql-reference/sql-statements/Data Manipulation/LOAD.md
@@ -33,6 +33,7 @@ under the License.
2. Broker Load：使用 broker 进行进行数据导入。
3. Mini Load：通过 http 协议上传文件进行批量数据导入。
4. Stream Load：通过 http 协议进行流式数据导入。
+ 5. S3 Load: 通过S3协议直接访问支持S3协议的存储系统进行数据导入, 导入的语法与Broker Load 基本相同。
本帮助主要描述第一种导入方式，即 Hadoop Load 相关帮助信息。其余导入方式可以使用以下命令查看帮助：
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org