You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by ji...@apache.org on 2022/07/29 09:23:05 UTC
[doris] branch master updated: update flink connecotr problem (#11075)
This is an automated email from the ASF dual-hosted git repository.
jiafengzheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 4cbfd7822a update flink connecotr problem (#11075)
4cbfd7822a is described below
commit 4cbfd7822a1b1d2f45e27973ea865d500a0d1285
Author: wudi <67...@qq.com>
AuthorDate: Fri Jul 29 17:22:59 2022 +0800
update flink connecotr problem (#11075)
update flink connecotr problem
---
docs/en/docs/ecosystem/flink-doris-connector.md | 9 +++++++--
docs/zh-CN/docs/ecosystem/flink-doris-connector.md | 11 ++++++++---
2 files changed, 15 insertions(+), 5 deletions(-)
diff --git a/docs/en/docs/ecosystem/flink-doris-connector.md b/docs/en/docs/ecosystem/flink-doris-connector.md
index 66896544ef..1a504cdba1 100644
--- a/docs/en/docs/ecosystem/flink-doris-connector.md
+++ b/docs/en/docs/ecosystem/flink-doris-connector.md
@@ -365,7 +365,7 @@ source.sinkTo(builder.build());
| doris.read.field | -- | N | List of column names in the Doris table, separated by commas |
| doris.filter.query | -- | N | Filter expression of the query, which is transparently transmitted to Doris. Doris uses this expression to complete source-side data filtering. |
| sink.label-prefix | -- | Y | The label prefix used by stream load imports. In the 2pc scenario, global uniqueness is required to ensure the EOS semantics of Flink. |
-| sink.properties.* | -- | N | The stream load parameters.<br /> <br /> eg:<br /> sink.properties.column_separator' = ','<br /> <br /> Setting 'sink.properties.escape_delimiters' = 'true' if you want to use a control char as a separator, so that such as '\\x01' will translate to binary 0x01<br /><br /> Support JSON format import, you need to enable both 'sink.properties.format' ='json' and 'sink.properties.strip_outer_array' ='true'|
+| sink.properties.* | -- | N | The stream load parameters.<br /> <br /> eg:<br /> `sink.properties.column_separator' = ','`<br /> <br /> Setting `'sink.properties.escape_delimiters' = 'true'` if you want to use a control char as a separator, so that such as '\\x01' will translate to binary 0x01<br /><br /> Support JSON format import, you need to enable both `'sink.properties.format' ='json'` and `'sink.properties.read_json_by_line' = 'true'` |
| sink.enable-delete | true | N | Whether to enable deletion. This option requires Doris table to enable batch delete function (0.15+ version is enabled by default), and only supports Uniq model.|
| sink.enable-2pc | true | N | Whether to enable two-phase commit (2pc), the default is true, to ensure Exactly-Once semantics. For two-phase commit, please refer to [here](../data-operate/import/import-way/stream-load-manual.md). |
| sink.max-retries | 1 | N | In the 2pc scenario, the number of retries after the commit phase fails. |
@@ -450,7 +450,7 @@ The most suitable scenario for using Flink Doris Connector is to synchronize sou
### common problem
-1. Bitmap type write
+**1. Bitmap type write**
```sql
CREATE TABLE bitmap_sink (
@@ -468,3 +468,8 @@ WITH (
'sink.properties.columns' = 'dt,page,user_id,user_id=to_bitmap(user_id)'
)
````
+
+**2. errCode = 2, detailMessage = Label [label_0_1] has already been used, relate to txn [19650]**
+
+In the Exactly-Once scenario, the Flink Job must be restarted from the latest Checkpoint/Savepoint, otherwise the above error will be reported. </br>
+When Exactly-Once is not required, it can also be solved by turning off 2PC commits (`sink.enable-2pc=false`) or changing to a different `sink.label-prefix`.
\ No newline at end of file
diff --git a/docs/zh-CN/docs/ecosystem/flink-doris-connector.md b/docs/zh-CN/docs/ecosystem/flink-doris-connector.md
index 83e8102447..55ba3bc4cf 100644
--- a/docs/zh-CN/docs/ecosystem/flink-doris-connector.md
+++ b/docs/zh-CN/docs/ecosystem/flink-doris-connector.md
@@ -364,8 +364,8 @@ source.sinkTo(builder.build());
| doris.deserialize.queue.size | 64 | N | 异步转换 Arrow 格式的内部处理队列，当 doris.deserialize.arrow.async 为 true 时生效 |
| doris.read.field | -- | N | 读取 Doris 表的列名列表，多列之间使用逗号分隔 |
| doris.filter.query | -- | N | 过滤读取数据的表达式，此表达式透传给 Doris。Doris 使用此表达式完成源端数据过滤。 |
-| sink.label-prefix | -- | Y | Stream load导入使用的label前缀。2pc场景下要求全局唯一 ，用来保证Flink的EOS语义。 |
-| sink.properties.* | -- | N | Stream Load 的导入参数。<br/>例如: 'sink.properties.column_separator' = ', ' 定义列分隔符， 'sink.properties.escape_delimiters' = 'true' 特殊字符作为分隔符,'\x01'会被转换为二进制的0x01 <br/><br/>JSON格式导入<br/>'sink.properties.format' = 'json' 'sink.properties.read_json_by_line' = 'true' |
+| sink.label-prefix | -- | Y | Stream load导入使用的label前缀。2pc场景下要求全局唯一 ，用来保证Flink的EOS语义。 |
+| sink.properties.* | -- | N | Stream Load 的导入参数。<br/>例如: `'sink.properties.column_separator' = ', '` 定义列分隔符， `'sink.properties.escape_delimiters' = 'true'` 特殊字符作为分隔符,'\x01'会被转换为二进制的0x01 <br/><br/>JSON格式导入<br/>`'sink.properties.format' = 'json'` <br/> `'sink.properties.read_json_by_line' = 'true'` |
| sink.enable-delete | TRUE | N | 是否启用删除。此选项需要 Doris 表开启批量删除功能(Doris0.15+版本默认开启)，只支持 Unique 模型。 |
| sink.enable-2pc | TRUE | N | 是否开启两阶段提交(2pc)，默认为true，保证Exactly-Once语义。关于两阶段提交可参考[这里](../data-operate/import/import-way/stream-load-manual.md)。 |
| sink.max-retries | 1 | N | 2pc场景下，commit阶段失败后的重试次数。 |
@@ -449,7 +449,7 @@ insert into doris_sink select id,name from cdc_mysql_source;
### 常见问题
-1. Bitmap类型写入
+1. **Bitmap类型写入**
```sql
CREATE TABLE bitmap_sink (
@@ -467,3 +467,8 @@ WITH (
'sink.properties.columns' = 'dt,page,user_id,user_id=to_bitmap(user_id)'
)
```
+
+2. **errCode = 2, detailMessage = Label [label_0_1] has already been used, relate to txn [19650]**
+
+Exactly-Once场景下，Flink Job重启时必须从最新的Checkpoint/Savepoint启动，否则会报如上错误。</br>
+不要求Exactly-Once时，也可通过关闭2PC提交（`sink.enable-2pc=false`） 或更换不同的`sink.label-prefix`解决。
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org