You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by ji...@apache.org on 2022/07/21 08:18:07 UTC
[doris-website] 01/03: flink connector fqa
This is an automated email from the ASF dual-hosted git repository.
jiafengzheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
commit 420fb7e0ad2dce1241428ba461cc8b6168b12ed8
Author: jiafeng.zhang <zh...@gmail.com>
AuthorDate: Thu Jul 21 16:14:59 2022 +0800
flink connector fqa
flink connector fqa
---
docs/ecosystem/flink-doris-connector.md | 11 +++++++++--
.../current/ecosystem/flink-doris-connector.md | 8 +++++++-
2 files changed, 16 insertions(+), 3 deletions(-)
diff --git a/docs/ecosystem/flink-doris-connector.md b/docs/ecosystem/flink-doris-connector.md
index ce6b6eb861..b63362c4df 100644
--- a/docs/ecosystem/flink-doris-connector.md
+++ b/docs/ecosystem/flink-doris-connector.md
@@ -356,9 +356,12 @@ source.sinkTo(builder.build());
| doris.read.field | -- | N | List of column names in the Doris table, separated by commas |
| doris.filter.query | -- | N | Filter expression of the query, which is transparently transmitted to Doris. Doris uses this expression to complete source-side data filtering. |
| sink.label-prefix | -- | Y | The label prefix used by stream load imports. In the 2pc scenario, global uniqueness is required to ensure the EOS semantics of Flink. |
-| sink.properties.* | -- | N | The stream load parameters.<br /> <br /> eg:<br /> sink.properties.column_separator' = ','<br /> <br /> Setting 'sink.properties.escape_delimiters' = 'true' if you want to use a control char as a separator, so that such as '\\x01' will translate to binary 0x01<br /><br /> Support JSON format import, you need to enable both 'sink.properties.format' ='json' and 'sink.properties.strip_outer_array' ='true'|
+| sink.properties.* | -- | N | The stream load parameters.<br /> <br /> eg:<br /> sink.properties.column_separator' = ','<br /> <br /> Setting 'sink.properties.escape_delimiters' = 'true' if you want to use a control char as a separator, so that such as '\\x01' will translate to binary 0x01<br /><br />Support JSON format import, you need to enable both 'sink.properties.format' ='json' and 'sink.properties.strip_outer_array' ='true' |
| sink.enable-delete | true | N | Whether to enable deletion. This option requires Doris table to enable batch delete function (0.15+ version is enabled by default), and only supports Uniq model.|
| sink.enable-2pc | true | N | Whether to enable two-phase commit (2pc), the default is true, to ensure Exactly-Once semantics. For two-phase commit, please refer to [here](../data-operate/import/import-way/stream-load-manual.md). |
+| sink.max-retries | 1 | N | In the 2pc scenario, the number of retries after the commit phase fails. |
+| sink.buffer-size | 1048576(1MB) | N | Write data cache buffer size, in bytes. It is not recommended to modify, the default configuration is sufficient. |
+| sink.buffer-count | 3 | N | The number of write data cache buffers, it is not recommended to modify, the default configuration is sufficient.
@@ -438,7 +441,7 @@ The most suitable scenario for using Flink Doris Connector is to synchronize sou
### common problem
-1. Bitmap type write
+1. **Bitmap type write**
```sql
CREATE TABLE bitmap_sink (
@@ -456,3 +459,7 @@ WITH (
'sink.properties.columns' = 'dt,page,user_id,user_id=to_bitmap(user_id)'
)
````
+2. **errCode = 2, detailMessage = Label [label_0_1] has already been used, relate to txn [19650]**
+
+In the Exactly-Once scenario, the Flink Job must be restarted from the latest Checkpoint/Savepoint, otherwise the above error will be reported.
+When Exactly-Once is not required, it can also be solved by turning off 2PC commits (sink.enable-2pc=false) or changing to a different sink.label-prefix.
\ No newline at end of file
diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/ecosystem/flink-doris-connector.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/ecosystem/flink-doris-connector.md
index b0c7899d7b..77a1301ec8 100644
--- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/ecosystem/flink-doris-connector.md
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/ecosystem/flink-doris-connector.md
@@ -361,6 +361,8 @@ source.sinkTo(builder.build());
| sink.enable-delete | TRUE | N | 是否启用删除。此选项需要 Doris 表开启批量删除功能(Doris0.15+版本默认开启)，只支持 Unique 模型。 |
| sink.enable-2pc | TRUE | N | 是否开启两阶段提交(2pc)，默认为true，保证Exactly-Once语义。关于两阶段提交可参考[这里](../data-operate/import/import-way/stream-load-manual.md)。 |
+
+
## Doris 和 Flink 列类型映射关系
| Doris Type | Flink Type |
@@ -436,7 +438,7 @@ insert into doris_sink select id,name from cdc_mysql_source;
### 常见问题
-1. Bitmap类型写入
+1. **Bitmap类型写入**
```sql
CREATE TABLE bitmap_sink (
@@ -454,3 +456,7 @@ WITH (
'sink.properties.columns' = 'dt,page,user_id,user_id=to_bitmap(user_id)'
)
```
+2. **errCode = 2, detailMessage = Label [label_0_1] has already been used, relate to txn [19650]**
+
+Exactly-Once场景下，Flink Job重启时必须从最新的Checkpoint/Savepoint启动，否则会报如上错误。
+不要求Exactly-Once时，也可通过关闭2PC提交（sink.enable-2pc=false） 或更换不同的sink.label-prefix解决。
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org