You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@paimon.apache.org by lz...@apache.org on 2023/05/25 08:06:01 UTC
[incubator-paimon] 07/20: [doc] Improve doc of cdc ingestion (#1138)
This is an automated email from the ASF dual-hosted git repository.
lzljs3620320 pushed a commit to branch release-0.4
in repository https://gitbox.apache.org/repos/asf/incubator-paimon.git
commit 91a4667804a708703f1bfb66bdeb948e78d68d5b
Author: s7monk <34...@users.noreply.github.com>
AuthorDate: Wed May 17 14:22:45 2023 +0800
[doc] Improve doc of cdc ingestion (#1138)
---
docs/content/how-to/cdc-ingestion.md | 39 ++++-------------
.../shortcodes/generated/mysql_sync_database.html | 51 ++++++++++++++++++++++
.../shortcodes/generated/mysql_sync_table.html | 47 ++++++++++++++++++++
3 files changed, 106 insertions(+), 31 deletions(-)
diff --git a/docs/content/how-to/cdc-ingestion.md b/docs/content/how-to/cdc-ingestion.md
index a905d144d..9f425aae1 100644
--- a/docs/content/how-to/cdc-ingestion.md
+++ b/docs/content/how-to/cdc-ingestion.md
@@ -30,6 +30,12 @@ Paimon supports synchronizing changes from different databases using change data
## MySQL
+### Prepare CDC Bundled Jar
+
+```
+flink-sql-connector-mysql-cdc-*.jar
+```
+
### Synchronizing Tables
By using [MySqlSyncTableAction](/docs/{{< param Branch >}}/api/java/org/apache/paimon/flink/action/cdc/mysql/MySqlSyncTableAction) in a Flink DataStream job or directly through `flink run`, users can synchronize one or multiple tables from MySQL into one Paimon table.
@@ -52,22 +58,7 @@ To use this feature through `flink run`, run the following shell command.
[--table-conf <paimon-table-sink-conf> [--table-conf <paimon-table-sink-conf> ...]]
```
-* `--warehouse` is the path to Paimon warehouse.
-* `--database` is the database name in Paimon catalog.
-* `--table` is the Paimon table name.
-* `--partition-keys` are the partition keys for Paimon table. If there are multiple partition keys, connect them with comma, for example `dt,hh,mm`.
-* `--primary-keys` are the primary keys for Paimon table. If there are multiple primary keys, connect them with comma, for example `buyer_id,seller_id`.
-* `--computed-column` are the definitions of computed columns. The argument field is from MySQL table field name. Supported expressions are:
- * year(date-column): Extract year from a DATE, DATETIME or TIMESTAMP. Output is an INT value represent the year.
- * substring(column,beginInclusive): Get column.substring(beginInclusive). Output is a STRING.
- * substring(column,beginInclusive,endExclusive): Get column.substring(beginInclusive,endExclusive). Output is a STRING.
- * truncate(column,width): truncate column by width. Output type is same with column.
- * If the column is a STRING, truncate(column,width) will truncate the string to width characters, namely `value.substring(0, width)`.
- * If the column is an INT or LONG, truncate(column,width) will truncate the number with the algorithm `v - (((v % W) + W) % W)`. The `redundant` compute part is to keep the result always positive.
- * If the column is a DECIMAL, truncate(column,width) will truncate the decimal with the algorithm: let `scaled_W = decimal(W, scale(v))`, then return `v - (v % scaled_W)`.
-* `--mysql-conf` is the configuration for Flink CDC MySQL table sources. Each configuration should be specified in the format `key=value`. `hostname`, `username`, `password`, `database-name` and `table-name` are required configurations, others are optional. See its [document](https://ververica.github.io/flink-cdc-connectors/master/content/connectors/mysql-cdc.html#connector-options) for a complete list of configurations.
-* `--catalog-conf` is the configuration for Paimon catalog. Each configuration should be specified in the format `key=value`. See [here]({{< ref "maintenance/configurations" >}}) for a complete list of catalog configurations.
-* `--table-conf` is the configuration for Paimon table sink. Each configuration should be specified in the format `key=value`. See [here]({{< ref "maintenance/configurations" >}}) for a complete list of table configurations.
+{{< generated/mysql_sync_table >}}
If the Paimon table you specify does not exist, this action will automatically create the table. Its schema will be derived from all specified MySQL tables. If the Paimon table already exists, its schema will be compared against the schema of all specified MySQL tables.
@@ -132,21 +123,7 @@ To use this feature through `flink run`, run the following shell command.
[--table-conf <paimon-table-sink-conf> [--table-conf <paimon-table-sink-conf> ...]]
```
-* `--warehouse` is the path to Paimon warehouse.
-* `--database` is the database name in Paimon catalog.
-* `--ignore-incompatible` is default false, in this case, if MySQL table name exists in Paimon and their schema is incompatible,
-an exception will be thrown. You can specify it to true explicitly to ignore the incompatible tables and exception.
-* `--table-prefix` is the prefix of all Paimon tables to be synchronized. For example, if you want all synchronized tables
-to have "ods_" as prefix, you can specify `--table-prefix ods_`.
-* `--table-suffix` is the suffix of all Paimon tables to be synchronized. The usage is same as `--table-prefix`.
-* `--including-tables` is used to specify which source tables are to be synchronized. You must use '|' to separate multiple
-tables. Regular expression is supported, for example, specifying `--including-tables test|paimon.*` means to synchronize
-table 'test' and all tables start with 'paimon'.
-* `--excluding-tables` is used to specify which source tables are not to be synchronized. The usage is same as `--including-tables`.
-`--excluding-tables` has higher priority than `--including-tables` if you specified both.
-* `--mysql-conf` is the configuration for Flink CDC MySQL table sources. Each configuration should be specified in the format `key=value`. `hostname`, `username`, `password` and `database-name` are required configurations, others are optional. Note that `database-name` should be the exact name of the MySQL databse you want to synchronize. It can't be a regular expression. See its [document](https://ververica.github.io/flink-cdc-connectors/master/content/connectors/mysql-cdc.html#connecto [...]
-* `--catalog-conf` is the configuration for Paimon catalog. Each configuration should be specified in the format `key=value`. See [here]({{< ref "maintenance/configurations" >}}) for a complete list of catalog configurations.
-* `--table-conf` is the configuration for Paimon table sink. Each configuration should be specified in the format `key=value`. All Paimon sink table will be applied the same set of configurations. See [here]({{< ref "maintenance/configurations" >}}) for a complete list of table configurations.
+{{< generated/mysql_sync_database >}}
Only tables with primary keys will be synchronized.
diff --git a/docs/layouts/shortcodes/generated/mysql_sync_database.html b/docs/layouts/shortcodes/generated/mysql_sync_database.html
new file mode 100644
index 000000000..c57ca35d2
--- /dev/null
+++ b/docs/layouts/shortcodes/generated/mysql_sync_database.html
@@ -0,0 +1,51 @@
+{{ $ref := ref . "maintenance/configurations.md" }}
+<table class="configuration table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 15%">Configuration</th>
+ <th class="text-left" style="width: 85%">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><h5>--warehouse</h5></td>
+ <td>The path to Paimon warehouse.</td>
+ </tr>
+ <tr>
+ <td><h5>--database</h5></td>
+ <td>The database name in Paimon catalog.</td>
+ </tr>
+ <tr>
+ <td><h5>--ignore-incompatible</h5></td>
+ <td>It is default false, in this case, if MySQL table name exists in Paimon and their schema is incompatible,an exception will be thrown. You can specify it to true explicitly to ignore the incompatible tables and exception.</td>
+ </tr>
+ <tr>
+ <td><h5>--table-prefix</h5></td>
+ <td>The prefix of all Paimon tables to be synchronized. For example, if you want all synchronized tables to have "ods_" as prefix, you can specify "--table-prefix ods_".</td>
+ </tr>
+ <tr>
+ <td><h5>--table-suffix</h5></td>
+ <td>The suffix of all Paimon tables to be synchronized. The usage is same as "--table-prefix".</td>
+ </tr>
+ <tr>
+ <td><h5>--including-tables</h5></td>
+ <td>It is used to specify which source tables are to be synchronized. You must use '|' to separate multiple tables.Because '|' is a special character, a comma is required, for example: 'a|b|c'.Regular expression is supported, for example, specifying "--including-tables test|paimon.*" means to synchronize table 'test' and all tables start with 'paimon'.</td>
+ </tr>
+ <tr>
+ <td><h5>--excluding-tables</h5></td>
+ <td>It is used to specify which source tables are not to be synchronized. The usage is same as "--including-tables". "--excluding-tables" has higher priority than "--including-tables" if you specified both.</td>
+ </tr>
+ <tr>
+ <td><h5>--mysql-conf</h5></td>
+ <td>The configuration for Flink CDC MySQL table sources. Each configuration should be specified in the format "key=value". hostname, username, password, database-name and table-name are required configurations, others are optional. See its <a href="https://ververica.github.io/flink-cdc-connectors/master/content/connectors/mysql-cdc.html#connector-options">document</a> for a complete list of configurations.</td>
+ </tr>
+ <tr>
+ <td><h5>--catalog-conf</h5></td>
+ <td>The configuration for Paimon catalog. Each configuration should be specified in the format "key=value". See <a href="{{ $ref }}">here</a> for a complete list of catalog configurations.</td>
+ </tr>
+ <tr>
+ <td><h5>--table-conf</h5></td>
+ <td>The configuration for Paimon table sink. Each configuration should be specified in the format "key=value". See <a href="{{ $ref }}">here</a> for a complete list of table configurations.</td>
+ </tr>
+ </tbody>
+</table>
\ No newline at end of file
diff --git a/docs/layouts/shortcodes/generated/mysql_sync_table.html b/docs/layouts/shortcodes/generated/mysql_sync_table.html
new file mode 100644
index 000000000..12ff50f61
--- /dev/null
+++ b/docs/layouts/shortcodes/generated/mysql_sync_table.html
@@ -0,0 +1,47 @@
+{{ $ref := ref . "maintenance/configurations.md" }}
+<table class="configuration table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 15%">Configuration</th>
+ <th class="text-left" style="width: 85%">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><h5>--warehouse</h5></td>
+ <td>The path to Paimon warehouse.</td>
+ </tr>
+ <tr>
+ <td><h5>--database</h5></td>
+ <td>The database name in Paimon catalog.</td>
+ </tr>
+ <tr>
+ <td><h5>--table</h5></td>
+ <td>The Paimon table name.</td>
+ </tr>
+ <tr>
+ <td><h5>--partition-keys</h5></td>
+ <td>The partition keys for Paimon table. If there are multiple partition keys, connect them with comma, for example "dt,hh,mm".</td>
+ </tr>
+ <tr>
+ <td><h5>--primary-keys</h5></td>
+ <td>The primary keys for Paimon table. If there are multiple primary keys, connect them with comma, for example "buyer_id,seller_id".</td>
+ </tr>
+ <tr>
+ <td><h5>--computed-column</h5></td>
+ <td>The definitions of computed columns. The argument field is from MySQL table field name. <br /><br />Supported expressions are: <ul><li>year(date-column): Extract year from a DATE, DATETIME or TIMESTAMP. Output is an INT value represent the year.</li><li>substring(column,beginInclusive): Get column.substring(beginInclusive). Output is a STRING.</li><li>substring(column,beginInclusive,endExclusive): Get column.substring(beginInclusive,endExclusive). Output is a STRING.</li><li> [...]
+ </tr>
+ <tr>
+ <td><h5>--mysql-conf</h5></td>
+ <td>The configuration for Flink CDC MySQL table sources. Each configuration should be specified in the format "key=value". hostname, username, password, database-name and table-name are required configurations, others are optional. See its <a href="https://ververica.github.io/flink-cdc-connectors/master/content/connectors/mysql-cdc.html#connector-options">document</a> for a complete list of configurations.</td>
+ </tr>
+ <tr>
+ <td><h5>--catalog-conf</h5></td>
+ <td>The configuration for Paimon catalog. Each configuration should be specified in the format "key=value". See <a href="{{ $ref }}">here</a> for a complete list of catalog configurations.</td>
+ </tr>
+ <tr>
+ <td><h5>--table-conf</h5></td>
+ <td>The configuration for Paimon table sink. Each configuration should be specified in the format "key=value". See <a href="{{ $ref }}">here</a> for a complete list of table configurations.</td>
+ </tr>
+ </tbody>
+</table>
\ No newline at end of file