You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by li...@apache.org on 2026/08/01 07:20:04 UTC
(seatunnel) branch dev updated: [Docs][Connector-V2] Improve S3File SftpFile and StarRocks docs (#11326)
This is an automated email from the ASF dual-hosted git repository.
davidzollo pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/seatunnel.git
The following commit(s) were added to refs/heads/dev by this push:
new 7fdab176bd [Docs][Connector-V2] Improve S3File SftpFile and StarRocks docs (#11326)
7fdab176bd is described below
commit 7fdab176bdfcaa1ff05c59fa26d87a930192b072
Author: Daniel Carter <80...@qq.com>
AuthorDate: Sat Aug 1 15:19:58 2026 +0800
[Docs][Connector-V2] Improve S3File SftpFile and StarRocks docs (#11326)
Co-authored-by: DanielCarter-stack <Da...@users.noreply.github.com>
---
docs/en/connectors/sink/S3File.md | 13 +++++----
docs/en/connectors/sink/SftpFile.md | 18 +++++++-----
docs/en/connectors/source/StarRocks.md | 51 +++++++++++++++++---------------
docs/zh/connectors/sink/S3File.md | 17 ++++++-----
docs/zh/connectors/sink/SftpFile.md | 19 +++++++-----
docs/zh/connectors/sink/StarRocks.md | 48 +++++++++++++++---------------
docs/zh/connectors/source/StarRocks.md | 53 ++++++++++++++++++----------------
7 files changed, 117 insertions(+), 102 deletions(-)
diff --git a/docs/en/connectors/sink/S3File.md b/docs/en/connectors/sink/S3File.md
index a254445da4..2a3de2607a 100644
--- a/docs/en/connectors/sink/S3File.md
+++ b/docs/en/connectors/sink/S3File.md
@@ -141,12 +141,12 @@ If write to `csv`, `text` file type, All column will be string.
| parquet_avro_write_timestamp_as_int96 | boolean | no | false | Only used when file_format is parquet. |
| parquet_avro_write_fixed_as_int96 | array | no | - | Only used when file_format is parquet. |
| hadoop_s3_properties | map | no | | If you need to add a other option, you could add it here and refer to this [link](https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html) |
+| schema_evolution_enabled | boolean | no | false | Enable schema evolution support for CDC pipelines. When true, ADD/DROP/RENAME/MODIFY column events from the source are applied to the sink without a job restart. Not supported for binary format. |
| schema_save_mode | Enum | no | CREATE_SCHEMA_WHEN_NOT_EXIST | Before turning on the synchronous task, do different treatment of the target path |
| data_save_mode | Enum | no | APPEND_DATA | Before opening the synchronous task, the data file in the target path is differently processed |
| enable_header_write | boolean | no | false | Only used when file_format_type is text,csv.<br/> false:don't write header,true:write header. |
| encoding | string | no | "UTF-8" | Only used when file_format_type is json,text,csv,xml. |
| merge_update_event | boolean | no | false | Only used when file_format_type is canal_json,debezium_json or maxwell_json. When value is true, the UPDATE_AFTER and UPDATE_BEFORE event will be merged into UPDATE event data |
-| schema_evolution_enabled | boolean | no | false | Enable schema evolution support for CDC pipelines. When true, ADD/DROP/RENAME/MODIFY column events from the source are applied to the sink without a job restart. Not supported for binary format. |
### path [string]
@@ -530,7 +530,6 @@ sink {
Only used when file_format_type is text,csv.false:don't write header,true:write header.
-
### schema_evolution_enabled [boolean]
When set to `true`, the file sink handles CDC schema change events (ADD COLUMN, DROP COLUMN, RENAME COLUMN, MODIFY COLUMN type) at runtime without requiring a job restart. On each schema change the current output file is closed and a new file is opened with the updated schema.
@@ -549,15 +548,17 @@ Users on the default CDC source config (`schema-changes.enabled = false`) are co
Example usage in a CDC pipeline:
```hocon
-LocalFile {
- path = "/tmp/cdc/${table_name}"
+S3File {
+ path = "/test/cdc/${table_name}"
+ fs.s3a.endpoint = "s3.cn-north-1.amazonaws.com.cn"
+ access_key = "xxxxxxxxxxxxxxxxx"
+ secret_key = "xxxxxxxxxxxxxxxxx"
file_format_type = "parquet"
schema_evolution_enabled = true
- have_partition = true
- partition_by = ["updated_at_month"]
}
```
+For production jobs, avoid hardcoding long-lived keys in job files. Prefer an IAM-based provider such as `fs.s3a.aws.credentials.provider = com.amazonaws.auth.InstanceProfileCredentialsProvider`, or inject `access_key` and `secret_key` with SeaTunnel variable substitution.
## Changelog
diff --git a/docs/en/connectors/sink/SftpFile.md b/docs/en/connectors/sink/SftpFile.md
index 4d6a0592a1..25378cf860 100644
--- a/docs/en/connectors/sink/SftpFile.md
+++ b/docs/en/connectors/sink/SftpFile.md
@@ -26,6 +26,10 @@ If you use SeaTunnel Engine, It automatically integrated the hadoop jar when you
By default, we use 2PC commit to ensure `exactly-once`
+- [ ] [cdc](../../introduction/concepts/connector-v2-features.md)
+- [x] [support multiple table write](../../introduction/concepts/connector-v2-features.md)
+- [ ] [timer flush](../../introduction/concepts/connector-v2-features.md)
+
- [x] file format type
- [x] text
- [x] csv
@@ -79,10 +83,10 @@ If you use SeaTunnel Engine, It automatically integrated the hadoop jar when you
| enable_header_write | boolean | no | false | Only used when file_format_type is text,csv.<br/> false:don't write header,true:write header. |
| parquet_avro_write_fixed_as_int96 | array | no | - | Only used when file_format is parquet. |
| encoding | string | no | "UTF-8" | Only used when file_format_type is json,text,csv,xml. |
+| schema_evolution_enabled | boolean | no | false | Enable schema evolution support for CDC pipelines. When true, ADD/DROP/RENAME/MODIFY column events from the source are applied to the sink without a job restart. Not supported for binary format. |
| schema_save_mode | string | no | CREATE_SCHEMA_WHEN_NOT_EXIST | Existing dir processing method |
| data_save_mode | string | no | APPEND_DATA | Existing data processing method |
| merge_update_event | boolean | no | false | Only used when file_format_type is canal_json,debezium_json or maxwell_json. When value is true, the UPDATE_AFTER and UPDATE_BEFORE event will be merged into UPDATE event data |
-| schema_evolution_enabled | boolean | no | false | Enable schema evolution support for CDC pipelines. When true, ADD/DROP/RENAME/MODIFY column events from the source are applied to the sink without a job restart. Not supported for binary format. |
### host [string]
@@ -341,7 +345,6 @@ SftpFile {
```
-
### schema_evolution_enabled [boolean]
When set to `true`, the file sink handles CDC schema change events (ADD COLUMN, DROP COLUMN, RENAME COLUMN, MODIFY COLUMN type) at runtime without requiring a job restart. On each schema change the current output file is closed and a new file is opened with the updated schema.
@@ -360,16 +363,17 @@ Users on the default CDC source config (`schema-changes.enabled = false`) are co
Example usage in a CDC pipeline:
```hocon
-LocalFile {
- path = "/tmp/cdc/${table_name}"
+SftpFile {
+ host = "xxx.xxx.xxx.xxx"
+ port = 22
+ user = "username"
+ password = "xxxxxxxxxxxxxxxxx"
+ path = "/data/sftp/cdc/${table_name}"
file_format_type = "parquet"
schema_evolution_enabled = true
- have_partition = true
- partition_by = ["updated_at_month"]
}
```
-
## Changelog
<ChangeLog />
diff --git a/docs/en/connectors/source/StarRocks.md b/docs/en/connectors/source/StarRocks.md
index 86b543e36d..6c5a9a95e4 100644
--- a/docs/en/connectors/source/StarRocks.md
+++ b/docs/en/connectors/source/StarRocks.md
@@ -21,24 +21,24 @@ delivers the query plan as a parameter to BE nodes, and then obtains data result
## Options
-| name | type | required | default value |
-|-------------------------|---------|----------|-------------------|
-| nodeUrls | list | yes | - |
-| username | string | yes | - |
-| password | string | yes | - |
-| database | string | yes | - |
-| table | string | no | - |
-| scan_filter | string | no | - |
-| schema | config | yes | - |
-| table_list | array | no | - |
-| request_tablet_size | int | no | Integer.MAX_VALUE |
-| scan_connect_timeout_ms | int | no | 30000 |
-| scan_query_timeout_sec | int | no | 3600 |
-| scan_keep_alive_min | int | no | 10 |
-| scan_batch_rows | int | no | 1024 |
-| scan_mem_limit | long | no | 2147483648 |
-| max_retries | int | no | 3 |
-| scan.params.* | string | no | - |
+| name | type | required | default value | description |
+|-------------------------|--------|----------|-------------------|---------------------------------------------------------------------------------------------------------|
+| nodeUrls | list | yes | - | StarRocks FE HTTP addresses, format: `["fe_ip:fe_http_port", ...]`. |
+| username | string | yes | - | StarRocks username. |
+| password | string | yes | - | StarRocks password. |
+| database | string | yes | - | StarRocks database name. |
+| table | string | no | - | StarRocks table name. Required when `table_list` is not configured. |
+| table_list | array | no | - | Tables to read. Required when `table` is not configured. Each entry can define its own `schema` and filter. |
+| schema | config | no | - | Output schema. Configure it at the top level for `table`, or inside each `table_list` entry for multi-table reads. |
+| scan_filter | string | no | "" | Source-side filter expression passed to StarRocks. |
+| request_tablet_size | int | no | Integer.MAX_VALUE | Maximum tablets in one SeaTunnel split. Smaller values can create more splits. |
+| scan_connect_timeout_ms | int | no | 1000 | Timeout in milliseconds when connecting to StarRocks BE for scan. |
+| scan_query_timeout_sec | int | no | 3600 | Query timeout in seconds. `-1` means no timeout. |
+| scan_keep_alive_min | int | no | 10 | Keep-alive time of the query task, in minutes. |
+| scan_batch_rows | int | no | 1024 | Maximum rows read from BE in one batch. |
+| scan_mem_limit | long | no | 1073741824 | Maximum memory allowed for a single BE query, in bytes. |
+| max_retries | int | no | 3 | Number of retry requests sent to StarRocks. |
+| scan.params.* | string | no | - | Extra BE scan parameters. The prefix `scan.params.` is removed before sending parameters to StarRocks. |
### nodeUrls [list]
@@ -58,7 +58,9 @@ The name of StarRocks database
### table [string]
-The name of StarRocks table
+The name of StarRocks table. Configure either `table` or `table_list`.
+
+When `table` is used, configure the source schema at the same level. When `table_list` is used, configure `schema` inside each table item.
### scan_filter [string]
@@ -74,7 +76,7 @@ e.g.
#### fields [Config]
-The schema of the starRocks that you want to generate. For more details, please refer to [Schema Feature](../../introduction/concepts/schema-feature.md).
+The schema of the StarRocks rows that SeaTunnel will emit. For more details, please refer to [Schema Feature](../../introduction/concepts/schema-feature.md).
e.g.
@@ -89,7 +91,8 @@ schema {
### table_list [array]
-The list of tables to be read, you can use this configuration instead of `table`
+The list of tables to be read. Use this configuration instead of `table` when one job needs to read multiple StarRocks tables.
+Each table item supports `table`, `schema`, and `scan_filter`.
### request_tablet_size [int]
@@ -122,7 +125,7 @@ partition[5] read data of tablet[14, 15] from be_node_3
### scan_connect_timeout_ms [int]
-requests connection timeout sent to StarRocks
+Connection timeout in milliseconds for requests sent to StarRocks. See the options table for the default value.
### scan_query_timeout_sec [int]
@@ -138,7 +141,7 @@ The maximum number of data rows to read from BE at a time. Increasing this value
### scan_mem_limit [long]
-The maximum memory space allowed for a single query in the BE node, in bytes. The default value is 2147483648 (2 GB).
+The maximum memory space allowed for a single query in the BE node, in bytes. See the options table for the default value.
### max_retries [int]
@@ -246,4 +249,4 @@ source {
## Changelog
-<ChangeLog />
\ No newline at end of file
+<ChangeLog />
diff --git a/docs/zh/connectors/sink/S3File.md b/docs/zh/connectors/sink/S3File.md
index b06abaa3a2..7536854aa1 100644
--- a/docs/zh/connectors/sink/S3File.md
+++ b/docs/zh/connectors/sink/S3File.md
@@ -20,7 +20,7 @@ import ChangeLog from '../changelog/connector-file-s3.md';
默认情况下,我们使用 2PC 提交来确保 `精确一次`。
-- [ ] [cdc](../../introduction/concepts/connector-v2-features.md)
+- [ ] [CDC](../../introduction/concepts/connector-v2-features.md)
- [x] [支持多表写入](../../introduction/concepts/connector-v2-features.md)
- [x] 文件格式类型
- [x] text
@@ -34,7 +34,7 @@ import ChangeLog from '../changelog/connector-file-s3.md';
- [x] canal_json
- [x] debezium_json
- [x] maxwell_json
-- [ ] [timer flush](../../introduction/concepts/connector-v2-features.md)
+- [ ] [定时刷新](../../introduction/concepts/connector-v2-features.md)
## 描述
@@ -138,12 +138,12 @@ import ChangeLog from '../changelog/connector-file-s3.md';
| parquet_avro_write_timestamp_as_int96 | boolean | 否 | false | 仅当 file_format 为 parquet 时使用 |
| parquet_avro_write_fixed_as_int96 | array | 否 | - | 仅当 file_format 为 parquet 时使用 |
| hadoop_s3_properties | map | 否 | | 如果您需要添加其他选项,可以在此处添加,并参考此[链接](https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html) |
+| schema_evolution_enabled | boolean | 否 | false | 开启 Schema 演变支持,适用于 CDC 管道。为 true 时,来自上游的 ADD/DROP/RENAME/MODIFY 列事件无需重启作业即可应用到 Sink。不支持 binary 格式。 |
| schema_save_mode | Enum | 否 | CREATE_SCHEMA_WHEN_NOT_EXIST | 在开启同步任务之前,对目标路径进行不同的处理 |
| data_save_mode | Enum | 否 | APPEND_DATA | 在开启同步任务之前,对目标路径中的数据文件进行不同的处理 |
| enable_header_write | boolean | 否 | false | 仅当 file_format_type 为 text,csv 时使用。<br/> false: 不写入表头, true: 写入表头。 |
| encoding | string | 否 | "UTF-8" | 仅当 file_format_type 为 json,text,csv,xml 时使用。 |
| merge_update_event | boolean | 否 | false | 仅当file_format_type为canal_json、debezium_json、maxwell_json. |
-| schema_evolution_enabled | boolean | 否 | false | 开启 Schema 演变支持,适用于 CDC 管道。为 true 时,来自上游的 ADD/DROP/RENAME/MODIFY 列事件无需重启作业即可应用到 Sink。不支持 binary 格式。 |
### path [string]
@@ -518,7 +518,6 @@ sink {
### enable_header_write [boolean]
仅在 file_format_type 为 text 或 csv 时使用。false:不写入表头,true:写入表头。
-
### schema_evolution_enabled [boolean]
设置为 `true` 时,文件 Sink 可在运行时处理 CDC Schema 变更事件(ADD COLUMN、DROP COLUMN、RENAME COLUMN、MODIFY COLUMN 类型),无需重启作业。每次 Schema 变更时,当前输出文件会被关闭,并以新 Schema 打开一个新文件。
@@ -537,15 +536,17 @@ sink {
CDC 管道中的使用示例:
```hocon
-LocalFile {
- path = "/tmp/cdc/${table_name}"
+S3File {
+ path = "/test/cdc/${table_name}"
+ fs.s3a.endpoint = "s3.cn-north-1.amazonaws.com.cn"
+ access_key = "xxxxxxxxxxxxxxxxx"
+ secret_key = "xxxxxxxxxxxxxxxxx"
file_format_type = "parquet"
schema_evolution_enabled = true
- have_partition = true
- partition_by = ["updated_at_month"]
}
```
+生产作业中不建议把长期有效的密钥直接写入任务文件。优先使用 IAM 类认证方式,例如 `fs.s3a.aws.credentials.provider = com.amazonaws.auth.InstanceProfileCredentialsProvider`,或通过 SeaTunnel 变量替换注入 `access_key` 和 `secret_key`。
## 变更日志
diff --git a/docs/zh/connectors/sink/SftpFile.md b/docs/zh/connectors/sink/SftpFile.md
index 78b83ee92f..b55c7ca6e0 100644
--- a/docs/zh/connectors/sink/SftpFile.md
+++ b/docs/zh/connectors/sink/SftpFile.md
@@ -25,6 +25,10 @@ import ChangeLog from '../changelog/connector-file-sftp.md';
默认情况下,我们使用2PC commit来确保`精确一次`
+- [ ] [CDC](../../introduction/concepts/connector-v2-features.md)
+- [x] [支持多表写入](../../introduction/concepts/connector-v2-features.md)
+- [ ] [定时刷新](../../introduction/concepts/connector-v2-features.md)
+
- [x] 文件格式类型
- [x] text
- [x] csv
@@ -37,7 +41,6 @@ import ChangeLog from '../changelog/connector-file-sftp.md';
- [x] canal_json
- [x] debezium_json
- [x] maxwell_json
-- [ ] [timer flush](../../introduction/concepts/connector-v2-features.md)
## 参数
@@ -78,10 +81,10 @@ import ChangeLog from '../changelog/connector-file-sftp.md';
| enable_header_write | boolean | 否 | false | 仅当file_format_type为text、csv时使用<br/>false:不写标头,true:写标头。 |
| parquet_avro_write_fixed_as_int96 | array | 否 | - | 仅当file_format_type为parquet时使用 |
| encoding | string | 否 | "UTF-8" | 仅当file_format_type为json、text、csv、xml时使用。 |
+| schema_evolution_enabled | boolean | 否 | false | 开启 Schema 演变支持,适用于 CDC 管道。为 true 时,来自上游的 ADD/DROP/RENAME/MODIFY 列事件无需重启作业即可应用到 Sink。不支持 binary 格式。 |
| schema_save_mode | string | 否 | CREATE_SCHEMA_WHEN_NOT_EXIST | 现有目录处理方式 |
| data_save_mode | string | 否 | APPEND_DATA | 现有数据处理方式 |
| merge_update_event | boolean | 否 | false | 仅当file_format_type为canal_json、debezium_json、maxwell_json. |
-| schema_evolution_enabled | boolean | 否 | false | 开启 Schema 演变支持,适用于 CDC 管道。为 true 时,来自上游的 ADD/DROP/RENAME/MODIFY 列事件无需重启作业即可应用到 Sink。不支持 binary 格式。 |
### host [string]
@@ -339,7 +342,6 @@ SftpFile {
```
-
### schema_evolution_enabled [boolean]
设置为 `true` 时,文件 Sink 可在运行时处理 CDC Schema 变更事件(ADD COLUMN、DROP COLUMN、RENAME COLUMN、MODIFY COLUMN 类型),无需重启作业。每次 Schema 变更时,当前输出文件会被关闭,并以新 Schema 打开一个新文件。
@@ -358,16 +360,17 @@ SftpFile {
CDC 管道中的使用示例:
```hocon
-LocalFile {
- path = "/tmp/cdc/${table_name}"
+SftpFile {
+ host = "xxx.xxx.xxx.xxx"
+ port = 22
+ user = "username"
+ password = "xxxxxxxxxxxxxxxxx"
+ path = "/data/sftp/cdc/${table_name}"
file_format_type = "parquet"
schema_evolution_enabled = true
- have_partition = true
- partition_by = ["updated_at_month"]
}
```
-
## 变更日志
<ChangeLog />
diff --git a/docs/zh/connectors/sink/StarRocks.md b/docs/zh/connectors/sink/StarRocks.md
index 32726cfff7..a760cb7709 100644
--- a/docs/zh/connectors/sink/StarRocks.md
+++ b/docs/zh/connectors/sink/StarRocks.md
@@ -12,9 +12,10 @@ import ChangeLog from '../changelog/connector-starrocks.md';
## 主要特性
-- [ ] [精准一次](../../introduction/concepts/connector-v2-features.md)
-- [x] [cdc](../../introduction/concepts/connector-v2-features.md)
-- [x] [timer flush](../../introduction/concepts/connector-v2-features.md)
+- [ ] [精确一次](../../introduction/concepts/connector-v2-features.md)
+- [x] [CDC](../../introduction/concepts/connector-v2-features.md)
+- [x] [支持多表写入](../../introduction/concepts/connector-v2-features.md)
+- [ ] [定时刷新](../../introduction/concepts/connector-v2-features.md)
## 描述
@@ -33,28 +34,27 @@ StarRocks数据接收器内部实现采用了缓存,通过stream load将数据
## 接收器选项
-| 名称 | 类型 | 是否必须 | 默认值 | Description |
+| 名称 | 类型 | 是否必须 | 默认值 | 说明 |
|-----------------------------|---------|------|------------------------------|---------------------------------------------------------------------------------------------------------------------|
-| nodeUrls | list | yes | - | `StarRocks`集群地址, 格式为 `["fe_ip:fe_http_port", ...]` |
-| base-url | string | yes | - | JDBC URL样式的连接信息。如:`jdbc:mysql://localhost:9030/` 或 `jdbc:mysql://localhost:9030` 或 `jdbc:mysql://localhost:9030/db` |
-| username | string | yes | - | 目标`StarRocks` 用户名 |
-| password | string | yes | - | 目标`StarRocks` 密码 |
-| database | string | yes | - | 指定目标 StarRocks 表所在的数据库的名称 |
-| table | string | no | - | 指定目标 StarRocks 表的名称, 如果没有设置该值,则表名与上游表名相同 |
-| labelPrefix | string | no | - | StarRocks stream load作业标签前缀 |
-| batch_max_rows | long | no | 1024 | 在批写情况下,当缓冲区数量达到`batch_max_rows`数量或`batch_max_bytes`字节大小或者时间达到`checkpoint.interval`时,数据会被刷新到StarRocks |
-| batch_max_bytes | int | no | 5 * 1024 * 1024 | 在批写情况下,当缓冲区数量达到`batch_max_rows`数量或`batch_max_bytes`字节大小或者时间达到`checkpoint.interval`时,数据会被刷新到StarRocks |
-| max_retries | int | no | - | 数据写入StarRocks失败后的重试次数 |
-| retry_backoff_multiplier_ms | int | no | - | 用作生成下一个退避延迟的乘数 |
-| max_retry_backoff_ms | int | no | - | 向StarRocks发送重试请求之前的等待时长 |
-| enable_upsert_delete | boolean | no | false | 是否开启upsert/delete事件的同步,仅仅支持主键模型的表 |
-| save_mode_create_template | string | no | 参见表下方的说明 | 参见表下方的说明 |
-| starrocks.config | map | no | - | stream load `data_desc`参数 |
-| http_socket_timeout_ms | int | no | 180000 | http socket超时时间,默认为3分钟 |
-| schema_save_mode | Enum | no | CREATE_SCHEMA_WHEN_NOT_EXIST | 在同步任务打开之前,针对目标端已存在的表结构选择不同的处理方法 |
-| data_save_mode | Enum | no | APPEND_DATA | 在同步任务打开之前,针对目标端已存在的数据选择不同的处理方法 |
-| table_options | Map | no | - | SaveMode 自动建表时合并进 CREATE TABLE PROPERTIES 的表级属性,详见下文 |
-| custom_sql | String | no | - | 当data_save_mode设置为CUSTOM_PROCESSING时,必须同时设置CUSTOM_SQL参数。CUSTOM_SQL的值为可执行的SQL语句,在同步任务开启前SQL将会被执行 |
+| nodeUrls | list | 是 | - | `StarRocks` 集群地址,格式为 `["fe_ip:fe_http_port", ...]` |
+| base-url | string | 是 | - | JDBC URL 样式的连接信息。如:`jdbc:mysql://localhost:9030/`、`jdbc:mysql://localhost:9030` 或 `jdbc:mysql://localhost:9030/db` |
+| username | string | 是 | - | 目标 `StarRocks` 用户名 |
+| password | string | 是 | - | 目标 `StarRocks` 密码 |
+| database | string | 是 | - | 目标 StarRocks 表所在的数据库名称 |
+| table | string | 否 | - | 目标 StarRocks 表名。如果没有设置,则表名与上游表名相同 |
+| labelPrefix | string | 否 | - | StarRocks Stream Load 作业标签前缀 |
+| batch_max_rows | long | 否 | 1024 | 批量写入时,当缓存行数达到 `batch_max_rows`、字节数达到 `batch_max_bytes`,或时间达到 `checkpoint.interval` 时,数据会刷新到 StarRocks |
+| batch_max_bytes | int | 否 | 5 * 1024 * 1024 | 批量写入时,当缓存行数达到 `batch_max_rows`、字节数达到 `batch_max_bytes`,或时间达到 `checkpoint.interval` 时,数据会刷新到 StarRocks |
+| max_retries | int | 否 | - | 数据写入 StarRocks 失败后的重试次数 |
+| retry_backoff_multiplier_ms | int | 否 | - | 用作生成下一次退避延迟的乘数 |
+| max_retry_backoff_ms | int | 否 | - | 向 StarRocks 发送重试请求前的等待时长 |
+| enable_upsert_delete | boolean | 否 | false | 是否开启 upsert/delete 事件同步,仅支持主键模型表 |
+| save_mode_create_template | string | 否 | 参见表下方的说明 | 自动建表模板,详见表下方说明 |
+| starrocks.config | map | 否 | - | Stream Load `data_desc` 参数 |
+| http_socket_timeout_ms | int | 否 | 180000 | HTTP socket 超时时间,默认为 3 分钟 |
+| schema_save_mode | Enum | 否 | CREATE_SCHEMA_WHEN_NOT_EXIST | 同步任务启动前,针对目标端已存在的表结构选择不同处理方式 |
+| data_save_mode | Enum | 否 | APPEND_DATA | 同步任务启动前,针对目标端已存在的数据选择不同处理方式 |
+| custom_sql | String | 否 | - | 当 `data_save_mode` 设置为 `CUSTOM_PROCESSING` 时必须配置。该 SQL 会在同步任务启动前执行 |
### save_mode_create_template
diff --git a/docs/zh/connectors/source/StarRocks.md b/docs/zh/connectors/source/StarRocks.md
index b9bc3230b9..38a3a69c32 100644
--- a/docs/zh/connectors/source/StarRocks.md
+++ b/docs/zh/connectors/source/StarRocks.md
@@ -10,7 +10,7 @@ import ChangeLog from '../changelog/connector-starrocks.md';
`StarRocks`源连接器的内部实现是从`FE`获取查询计划,
将查询计划作为参数传递给`BE`节点,然后从`BE`节点获取数据结果。
-## 主要功能
+## 主要特性
- [x] [批处理](../../introduction/concepts/connector-v2-features.md)
- [ ] [流处理](../../introduction/concepts/connector-v2-features.md)
@@ -21,24 +21,24 @@ import ChangeLog from '../changelog/connector-starrocks.md';
## 配置选项
-| 名称 | 类型 | 是否必须 | 默认值 |
-|-------------------------|-----------|------|-------------------|
-| nodeUrls | list | 是 | - |
-| username | string | 是 | - |
-| password | string | 是 | - |
-| database | string | 是 | - |
-| table | string | 否 | - |
-| scan_filter | string | 否 | - |
-| schema | config | 是 | - |
-| table_list | array | 否 | - |
-| request_tablet_size | int | 否 | Integer.MAX_VALUE |
-| scan_connect_timeout_ms | int | 否 | 30000 |
-| scan_query_timeout_sec | int | 否 | 3600 |
-| scan_keep_alive_min | int | 否 | 10 |
-| scan_batch_rows | int | 否 | 1024 |
-| scan_mem_limit | long | 否 | 2147483648 |
-| max_retries | int | 否 | 3 |
-| scan.params.* | string | 否 | - |
+| 名称 | 类型 | 是否必须 | 默认值 | 说明 |
+|-------------------------|--------|------|-------------------|---------------------------------------------------------------|
+| nodeUrls | list | 是 | - | StarRocks FE HTTP 地址,格式为 `["fe_ip:fe_http_port", ...]`。 |
+| username | string | 是 | - | StarRocks 用户名。 |
+| password | string | 是 | - | StarRocks 密码。 |
+| database | string | 是 | - | StarRocks 数据库名。 |
+| table | string | 否 | - | StarRocks 表名。未配置 `table_list` 时必须配置。 |
+| table_list | array | 否 | - | 要读取的表列表。未配置 `table` 时必须配置,每个表项可单独配置 `schema` 和过滤条件。 |
+| schema | config | 否 | - | 输出数据结构。读取单表时配置在顶层,读取多表时配置在每个 `table_list` 表项中。 |
+| scan_filter | string | 否 | "" | 下推到 StarRocks 源端执行的过滤表达式。 |
+| request_tablet_size | int | 否 | Integer.MAX_VALUE | 一个 SeaTunnel 分片最多包含的 tablet 数量,值越小通常分片越多。 |
+| scan_connect_timeout_ms | int | 否 | 1000 | 连接 StarRocks BE 进行扫描时的超时时间,单位毫秒。 |
+| scan_query_timeout_sec | int | 否 | 3600 | 查询超时时间,单位秒,`-1` 表示不限制。 |
+| scan_keep_alive_min | int | 否 | 10 | 查询任务保持连接时长,单位分钟。 |
+| scan_batch_rows | int | 否 | 1024 | 每次从 BE 节点读取的最大行数。 |
+| scan_mem_limit | long | 否 | 1073741824 | 单个 BE 查询允许使用的最大内存,单位字节。 |
+| max_retries | int | 否 | 3 | 发送到 StarRocks 的重试请求次数。 |
+| scan.params.* | string | 否 | - | 额外 BE 扫描参数,发送到 StarRocks 前会去掉 `scan.params.` 前缀。 |
### nodeUrls [list]
@@ -58,7 +58,9 @@ import ChangeLog from '../changelog/connector-starrocks.md';
### table [string]
-`StarRocks` 表名。
+`StarRocks` 表名。`table` 和 `table_list` 二选一配置。
+
+使用 `table` 时,`schema` 配置在同一层;使用 `table_list` 时,`schema` 配置在每个表项内部。
### scan_filter [string]
@@ -74,7 +76,7 @@ import ChangeLog from '../changelog/connector-starrocks.md';
#### fields [Config]
-要生成的`starRocks`的`schema`。更多详情请参考 [Schema 特性](../../introduction/concepts/schema-feature.md)。
+SeaTunnel 输出的 StarRocks 数据结构。更多详情请参考 [Schema 特性](../../introduction/concepts/schema-feature.md)。
示例
@@ -89,7 +91,8 @@ schema {
### table_list [array]
-`StarRocks` 表名列表,当需要同时读取多表时使用此配置代替 table
+`StarRocks` 表名列表。当一个作业需要同时读取多张 StarRocks 表时,使用此配置代替 `table`。
+每个表项支持配置 `table`、`schema` 和 `scan_filter`。
### request_tablet_size [int]
@@ -122,7 +125,7 @@ partition[5] 从 be_node_3 读取 tablet 数据:tablet[14,15]
### scan_connect_timeout_ms [int]
-发送到 `StarRocks` 的请求连接超时。
+连接 StarRocks BE 进行扫描时的超时时间,单位毫秒。默认值以参数表为准。
### scan_query_timeout_sec [int]
@@ -136,7 +139,7 @@ partition[5] 从 be_node_3 读取 tablet 数据:tablet[14,15]
一次从 `BE` 节点读取的最大数据行数。增加此值可以减少引擎与 `StarRocks` 之间建立的连接数量,从而减轻由网络延迟引起的开销。
### scan_mem_limit [long]
-单个查询在 BE 节点上允许的最大内存空间,单位为字节,默认值为 2147483648 字节(即 2 GB)。
+单个查询在 BE 节点上允许的最大内存空间,单位为字节。默认值以参数表为准。
### max_retries [int]
@@ -244,4 +247,4 @@ source {
## 变更日志
-<ChangeLog />
\ No newline at end of file
+<ChangeLog />