You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "TaoZex (via GitHub)" <gi...@apache.org> on 2023/07/21 02:59:07 UTC
[GitHub] [seatunnel] TaoZex opened a new pull request, #5132: [Docs][Connector-V2][StarRocks]Reconstruct the StarRocks connector document
TaoZex opened a new pull request, #5132:
URL: https://github.com/apache/seatunnel/pull/5132
<!--
Thank you for contributing to SeaTunnel! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
## Contribution Checklist
- Make sure that the pull request corresponds to a [GITHUB issue](https://github.com/apache/seatunnel/issues).
- Name the pull request in the form "[Feature] [component] Title of the pull request", where *Feature* can be replaced by `Hotfix`, `Bug`, etc.
- Minor fixes should be named following this pattern: `[hotfix] [docs] Fix typo in README.md doc`.
-->
## Purpose of this pull request
subtask of https://github.com/apache/seatunnel/issues/4572
Reconstruct the StarRocks connector document
## Check list
* [ ] Code changed are covered with tests, or it does not need tests for reason:
* [ ] If any new Jar binary package adding in your PR, please add License Notice according
[New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md)
* [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
* [ ] If you are contributing the connector code, please check that the following files are updated:
1. Update change log that in connector document. For more details you can refer to [connector-v2](https://github.com/apache/seatunnel/tree/dev/docs/en/connector-v2)
2. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it
3. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
* [ ] Update the [`release-note`](https://github.com/apache/seatunnel/blob/dev/release-note.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] EricJoy2048 commented on a diff in pull request #5132: [Docs][Connector-V2][StarRocks]Reconstruct the StarRocks connector document
Posted by "EricJoy2048 (via GitHub)" <gi...@apache.org>.
EricJoy2048 commented on code in PR #5132:
URL: https://github.com/apache/seatunnel/pull/5132#discussion_r1289515207
##########
docs/en/connector-v2/sink/StarRocks.md:
##########
@@ -2,94 +2,44 @@
> StarRocks sink connector
-## Description
+## Support Those Engines
-Used to send data to StarRocks. Both support streaming and batch mode.
-The internal implementation of StarRocks sink connector is cached and imported by stream load in batches.
+> Spark<br/>
+> Flink<br/>
+> SeaTunnel Zeta<br/>
-## Key features
+## Key Features
- [ ] [exactly-once](../../concept/connector-v2-features.md)
- [x] [cdc](../../concept/connector-v2-features.md)
-## Options
-
-| name | type | required | default value |
-|-----------------------------|---------|----------|-----------------|
-| nodeUrls | list | yes | - |
-| base-url | string | yes | - |
-| username | string | yes | - |
-| password | string | yes | - |
-| database | string | yes | - |
-| table | string | no | - |
-| labelPrefix | string | no | - |
-| batch_max_rows | long | no | 1024 |
-| batch_max_bytes | int | no | 5 * 1024 * 1024 |
-| batch_interval_ms | int | no | - |
-| max_retries | int | no | - |
-| retry_backoff_multiplier_ms | int | no | - |
-| max_retry_backoff_ms | int | no | - |
-| enable_upsert_delete | boolean | no | false |
-| save_mode_create_template | string | no | see below |
-| starrocks.config | map | no | - |
-
-### nodeUrls [list]
-
-`StarRocks` cluster address, the format is `["fe_ip:fe_http_port", ...]`
-
-### base-url [string]
-
-The JDBC URL like `jdbc:mysql://localhost:9030/` or `jdbc:mysql://localhost:9030` or `jdbc:mysql://localhost:9030/db`
-
-### username [string]
-
-`StarRocks` user username
-
-### password [string]
-
-`StarRocks` user password
-
-### database [string]
-
-The name of StarRocks database
-
-### table [string]
-
-The name of StarRocks table, If not set, the table name will be the name of the upstream table
-
-### labelPrefix [string]
-
-The prefix of StarRocks stream load label
-
-### batch_max_rows [long]
-
-For batch writing, when the number of buffers reaches the number of `batch_max_rows` or the byte size of `batch_max_bytes` or the time reaches `batch_interval_ms`, the data will be flushed into the StarRocks
-
-### batch_max_bytes [int]
-
-For batch writing, when the number of buffers reaches the number of `batch_max_rows` or the byte size of `batch_max_bytes` or the time reaches `batch_interval_ms`, the data will be flushed into the StarRocks
-
-### batch_interval_ms [int]
-
-For batch writing, when the number of buffers reaches the number of `batch_max_rows` or the byte size of `batch_max_bytes` or the time reaches `batch_interval_ms`, the data will be flushed into the StarRocks
-
-### max_retries [int]
-
-The number of retries to flush failed
-
-### retry_backoff_multiplier_ms [int]
-
-Using as a multiplier for generating the next delay for backoff
-
-### max_retry_backoff_ms [int]
-
-The amount of time to wait before attempting to retry a request to `StarRocks`
-
-### enable_upsert_delete [boolean]
+## Description
-Whether to enable upsert/delete, only supports PrimaryKey model.
+Used to send data to StarRocks. Both support streaming and batch mode.
+The internal implementation of StarRocks sink connector is cached and imported by stream load in batches.
-### save_mode_create_template [string]
+## Sink Options
+
+| Name | Type | Required | Default | Description |
+|------------------------------|---------|----------|-----------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| nodeUrls | list | yes | - | `StarRocks` cluster address, the format is `["fe_ip:fe_http_port", ...]` |
+| base-url | string | yes | - | The JDBC URL like `jdbc:mysql://localhost:9030/` or `jdbc:mysql://localhost:9030` or `jdbc:mysql://localhost:9030/db` |
+| username | string | yes | - | `StarRocks` user username |
+| password | string | yes | - | `StarRocks` user password |
+| database | string | yes | - | The name of StarRocks database |
+| table | string | no | - | The name of StarRocks table, If not set, the table name will be the name of the upstream table |
+| labelPrefix | string | no | - | The prefix of StarRocks stream load label |
Review Comment:
From Doris Sink Connector I found use stream load label can implement 2PC and make sink support `exactly-once`. What does this look like in Starlocks?
##########
docs/en/connector-v2/sink/StarRocks.md:
##########
@@ -158,12 +161,29 @@ sink {
}
}
}
-
```
-Use CSV format to import data
+### Support write cdc changelog event(INSERT/UPDATE/DELETE)
Review Comment:
Can you add an example for `save_mode_create_template ` option?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] ic4y merged pull request #5132: [Docs][Connector-V2][StarRocks]Reconstruct the StarRocks connector document
Posted by "ic4y (via GitHub)" <gi...@apache.org>.
ic4y merged PR #5132:
URL: https://github.com/apache/seatunnel/pull/5132
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] zhilinli123 commented on pull request #5132: [Docs][Connector-V2][StarRocks]Reconstruct the StarRocks connector document
Posted by "zhilinli123 (via GitHub)" <gi...@apache.org>.
zhilinli123 commented on PR #5132:
URL: https://github.com/apache/seatunnel/pull/5132#issuecomment-1670617133
Add a connector version for this
https://github.com/apache/seatunnel/issues/4572 @TaoZex thks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] liugddx commented on pull request #5132: [Docs][Connector-V2][StarRocks]Reconstruct the StarRocks connector document
Posted by "liugddx (via GitHub)" <gi...@apache.org>.
liugddx commented on PR #5132:
URL: https://github.com/apache/seatunnel/pull/5132#issuecomment-1671604869
> Add a connector version for this #4572 @TaoZex thks
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] davidzollo commented on pull request #5132: [Docs][Connector-V2][StarRocks]Reconstruct the StarRocks connector document
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #5132:
URL: https://github.com/apache/seatunnel/pull/5132#issuecomment-1646863034
good job
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] TyrantLucifer commented on pull request #5132: [Docs][Connector-V2][StarRocks]Reconstruct the StarRocks connector document
Posted by "TyrantLucifer (via GitHub)" <gi...@apache.org>.
TyrantLucifer commented on PR #5132:
URL: https://github.com/apache/seatunnel/pull/5132#issuecomment-1646454186
@Hisoka-X @EricJoy2048 PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] TaoZex commented on a diff in pull request #5132: [Docs][Connector-V2][StarRocks]Reconstruct the StarRocks connector document
Posted by "TaoZex (via GitHub)" <gi...@apache.org>.
TaoZex commented on code in PR #5132:
URL: https://github.com/apache/seatunnel/pull/5132#discussion_r1289636755
##########
docs/en/connector-v2/sink/StarRocks.md:
##########
@@ -2,94 +2,44 @@
> StarRocks sink connector
-## Description
+## Support Those Engines
-Used to send data to StarRocks. Both support streaming and batch mode.
-The internal implementation of StarRocks sink connector is cached and imported by stream load in batches.
+> Spark<br/>
+> Flink<br/>
+> SeaTunnel Zeta<br/>
-## Key features
+## Key Features
- [ ] [exactly-once](../../concept/connector-v2-features.md)
- [x] [cdc](../../concept/connector-v2-features.md)
-## Options
-
-| name | type | required | default value |
-|-----------------------------|---------|----------|-----------------|
-| nodeUrls | list | yes | - |
-| base-url | string | yes | - |
-| username | string | yes | - |
-| password | string | yes | - |
-| database | string | yes | - |
-| table | string | no | - |
-| labelPrefix | string | no | - |
-| batch_max_rows | long | no | 1024 |
-| batch_max_bytes | int | no | 5 * 1024 * 1024 |
-| batch_interval_ms | int | no | - |
-| max_retries | int | no | - |
-| retry_backoff_multiplier_ms | int | no | - |
-| max_retry_backoff_ms | int | no | - |
-| enable_upsert_delete | boolean | no | false |
-| save_mode_create_template | string | no | see below |
-| starrocks.config | map | no | - |
-
-### nodeUrls [list]
-
-`StarRocks` cluster address, the format is `["fe_ip:fe_http_port", ...]`
-
-### base-url [string]
-
-The JDBC URL like `jdbc:mysql://localhost:9030/` or `jdbc:mysql://localhost:9030` or `jdbc:mysql://localhost:9030/db`
-
-### username [string]
-
-`StarRocks` user username
-
-### password [string]
-
-`StarRocks` user password
-
-### database [string]
-
-The name of StarRocks database
-
-### table [string]
-
-The name of StarRocks table, If not set, the table name will be the name of the upstream table
-
-### labelPrefix [string]
-
-The prefix of StarRocks stream load label
-
-### batch_max_rows [long]
-
-For batch writing, when the number of buffers reaches the number of `batch_max_rows` or the byte size of `batch_max_bytes` or the time reaches `batch_interval_ms`, the data will be flushed into the StarRocks
-
-### batch_max_bytes [int]
-
-For batch writing, when the number of buffers reaches the number of `batch_max_rows` or the byte size of `batch_max_bytes` or the time reaches `batch_interval_ms`, the data will be flushed into the StarRocks
-
-### batch_interval_ms [int]
-
-For batch writing, when the number of buffers reaches the number of `batch_max_rows` or the byte size of `batch_max_bytes` or the time reaches `batch_interval_ms`, the data will be flushed into the StarRocks
-
-### max_retries [int]
-
-The number of retries to flush failed
-
-### retry_backoff_multiplier_ms [int]
-
-Using as a multiplier for generating the next delay for backoff
-
-### max_retry_backoff_ms [int]
-
-The amount of time to wait before attempting to retry a request to `StarRocks`
-
-### enable_upsert_delete [boolean]
+## Description
-Whether to enable upsert/delete, only supports PrimaryKey model.
+Used to send data to StarRocks. Both support streaming and batch mode.
+The internal implementation of StarRocks sink connector is cached and imported by stream load in batches.
-### save_mode_create_template [string]
+## Sink Options
+
+| Name | Type | Required | Default | Description |
+|------------------------------|---------|----------|-----------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| nodeUrls | list | yes | - | `StarRocks` cluster address, the format is `["fe_ip:fe_http_port", ...]` |
+| base-url | string | yes | - | The JDBC URL like `jdbc:mysql://localhost:9030/` or `jdbc:mysql://localhost:9030` or `jdbc:mysql://localhost:9030/db` |
+| username | string | yes | - | `StarRocks` user username |
+| password | string | yes | - | `StarRocks` user password |
+| database | string | yes | - | The name of StarRocks database |
+| table | string | no | - | The name of StarRocks table, If not set, the table name will be the name of the upstream table |
+| labelPrefix | string | no | - | The prefix of StarRocks stream load label |
Review Comment:
Currently, starrocks does not support exactly-once
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] EricJoy2048 commented on a diff in pull request #5132: [Docs][Connector-V2][StarRocks]Reconstruct the StarRocks connector document
Posted by "EricJoy2048 (via GitHub)" <gi...@apache.org>.
EricJoy2048 commented on code in PR #5132:
URL: https://github.com/apache/seatunnel/pull/5132#discussion_r1289999196
##########
docs/en/connector-v2/sink/StarRocks.md:
##########
@@ -2,94 +2,44 @@
> StarRocks sink connector
-## Description
+## Support Those Engines
-Used to send data to StarRocks. Both support streaming and batch mode.
-The internal implementation of StarRocks sink connector is cached and imported by stream load in batches.
+> Spark<br/>
+> Flink<br/>
+> SeaTunnel Zeta<br/>
-## Key features
+## Key Features
- [ ] [exactly-once](../../concept/connector-v2-features.md)
- [x] [cdc](../../concept/connector-v2-features.md)
-## Options
-
-| name | type | required | default value |
-|-----------------------------|---------|----------|-----------------|
-| nodeUrls | list | yes | - |
-| base-url | string | yes | - |
-| username | string | yes | - |
-| password | string | yes | - |
-| database | string | yes | - |
-| table | string | no | - |
-| labelPrefix | string | no | - |
-| batch_max_rows | long | no | 1024 |
-| batch_max_bytes | int | no | 5 * 1024 * 1024 |
-| batch_interval_ms | int | no | - |
-| max_retries | int | no | - |
-| retry_backoff_multiplier_ms | int | no | - |
-| max_retry_backoff_ms | int | no | - |
-| enable_upsert_delete | boolean | no | false |
-| save_mode_create_template | string | no | see below |
-| starrocks.config | map | no | - |
-
-### nodeUrls [list]
-
-`StarRocks` cluster address, the format is `["fe_ip:fe_http_port", ...]`
-
-### base-url [string]
-
-The JDBC URL like `jdbc:mysql://localhost:9030/` or `jdbc:mysql://localhost:9030` or `jdbc:mysql://localhost:9030/db`
-
-### username [string]
-
-`StarRocks` user username
-
-### password [string]
-
-`StarRocks` user password
-
-### database [string]
-
-The name of StarRocks database
-
-### table [string]
-
-The name of StarRocks table, If not set, the table name will be the name of the upstream table
-
-### labelPrefix [string]
-
-The prefix of StarRocks stream load label
-
-### batch_max_rows [long]
-
-For batch writing, when the number of buffers reaches the number of `batch_max_rows` or the byte size of `batch_max_bytes` or the time reaches `batch_interval_ms`, the data will be flushed into the StarRocks
-
-### batch_max_bytes [int]
-
-For batch writing, when the number of buffers reaches the number of `batch_max_rows` or the byte size of `batch_max_bytes` or the time reaches `batch_interval_ms`, the data will be flushed into the StarRocks
-
-### batch_interval_ms [int]
-
-For batch writing, when the number of buffers reaches the number of `batch_max_rows` or the byte size of `batch_max_bytes` or the time reaches `batch_interval_ms`, the data will be flushed into the StarRocks
-
-### max_retries [int]
-
-The number of retries to flush failed
-
-### retry_backoff_multiplier_ms [int]
-
-Using as a multiplier for generating the next delay for backoff
-
-### max_retry_backoff_ms [int]
-
-The amount of time to wait before attempting to retry a request to `StarRocks`
-
-### enable_upsert_delete [boolean]
+## Description
-Whether to enable upsert/delete, only supports PrimaryKey model.
+Used to send data to StarRocks. Both support streaming and batch mode.
+The internal implementation of StarRocks sink connector is cached and imported by stream load in batches.
-### save_mode_create_template [string]
+## Sink Options
+
+| Name | Type | Required | Default | Description |
+|------------------------------|---------|----------|-----------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| nodeUrls | list | yes | - | `StarRocks` cluster address, the format is `["fe_ip:fe_http_port", ...]` |
+| base-url | string | yes | - | The JDBC URL like `jdbc:mysql://localhost:9030/` or `jdbc:mysql://localhost:9030` or `jdbc:mysql://localhost:9030/db` |
+| username | string | yes | - | `StarRocks` user username |
+| password | string | yes | - | `StarRocks` user password |
+| database | string | yes | - | The name of StarRocks database |
+| table | string | no | - | The name of StarRocks table, If not set, the table name will be the name of the upstream table |
+| labelPrefix | string | no | - | The prefix of StarRocks stream load label |
Review Comment:
Ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org