You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by le...@apache.org on 2024/03/18 11:06:12 UTC
(flink-cdc) branch master updated: [FLINK-34680][cdc][docs] Add overview page for CDC connectors and optimize connector docs
This is an automated email from the ASF dual-hosted git repository.
leonard pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink-cdc.git
The following commit(s) were added to refs/heads/master by this push:
new 8fd28f178 [FLINK-34680][cdc][docs] Add overview page for CDC connectors and optimize connector docs
8fd28f178 is described below
commit 8fd28f1788f35ef6a8a2560811520170ff70a10f
Author: Qingsheng Ren <re...@gmail.com>
AuthorDate: Mon Mar 18 19:06:08 2024 +0800
[FLINK-34680][cdc][docs] Add overview page for CDC connectors and optimize connector docs
This closes #3160.
---
docs/content/docs/connectors/doris.md | 19 +++++-------
.../legacy-flink-cdc-sources/overview.md | 8 +++--
docs/content/docs/connectors/mysql.md | 26 +++++++----------
docs/content/docs/connectors/overview.md | 34 +++++++++++++++-------
docs/content/docs/connectors/starrocks.md | 22 +++++++-------
5 files changed, 57 insertions(+), 52 deletions(-)
diff --git a/docs/content/docs/connectors/doris.md b/docs/content/docs/connectors/doris.md
index 101cae7d6..61b88d0df 100644
--- a/docs/content/docs/connectors/doris.md
+++ b/docs/content/docs/connectors/doris.md
@@ -24,13 +24,12 @@ specific language governing permissions and limitations
under the License.
-->
-# Doris Pipeline Connector
-
-This article introduces of Doris Pipeline Connector
+# Doris Connector
+This article introduces of Doris Connector
## Example
-----------------
+
```yaml
source:
@@ -50,8 +49,7 @@ pipeline:
```
-## Pipeline options
-----------------
+## Connector Options
<div class="highlight">
<table class="colwidths-auto docutils">
@@ -180,17 +178,16 @@ pipeline:
</tbody>
</table>
</div>
-## Data Type Mapping
-----------------
+## Data Type Mapping
<div class="wy-table-responsive">
<table class="colwidths-auto docutils">
<thead>
<tr>
- <th class="text-left" style="width:10%;">CDC type</th>
- <th class="text-left" style="width:30%;">Doris type<a href="https://doris.apache.org/docs/dev/sql-manual/sql-reference/Data-Types/BOOLEAN/"></a></th>
- <th class="text-left" style="width:60%;">NOTE</th>
+ <th class="text-left" style="width:10%;">Flink CDC Type</th>
+ <th class="text-left" style="width:30%;"><a href="https://doris.apache.org/docs/dev/sql-manual/sql-reference/Data-Types/BOOLEAN/">Doris Type</a></th>
+ <th class="text-left" style="width:60%;">Note</th>
</tr>
</thead>
<tbody>
diff --git a/docs/content/docs/connectors/legacy-flink-cdc-sources/overview.md b/docs/content/docs/connectors/legacy-flink-cdc-sources/overview.md
index babdb7931..f09ca2879 100644
--- a/docs/content/docs/connectors/legacy-flink-cdc-sources/overview.md
+++ b/docs/content/docs/connectors/legacy-flink-cdc-sources/overview.md
@@ -24,10 +24,12 @@ specific language governing permissions and limitations
under the License.
-->
-# CDC Connectors for Apache Flink
+# Legacy CDC sources for Apache Flink
-CDC Connectors for Apache Flink<sup>®</sup> is a set of source connectors for <a href="https://flink.apache.org/">Apache Flink<sup>®</sup></a>, ingesting changes from different databases using change data capture (CDC).
-The CDC Connectors for Apache Flink<sup>®</sup> integrate Debezium as the engine to capture data changes. So it can fully leverage the ability of Debezium. See more about what is [Debezium](https://github.com/debezium/debezium).
+Flink CDC sources is a set of source connectors for <a href="https://flink.apache.org/">Apache Flink<sup>®</sup></a>, ingesting changes from different databases using change data capture (CDC).
+Some CDC sources integrate Debezium as the engine to capture data changes. So it can fully leverage the ability of Debezium. See more about what is [Debezium](https://github.com/debezium/debezium).
+
+You can also read [tutorials]({{< ref "docs/connectors/legacy-flink-cdc-sources/tutorials/mysql-postgres-tutorial" >}}) about how to use these sources.
{{< img src="/fig/cdc-flow.png" width="600px" alt="Flink CDC" >}}
diff --git a/docs/content/docs/connectors/mysql.md b/docs/content/docs/connectors/mysql.md
index 3f9d27c64..455e22edf 100644
--- a/docs/content/docs/connectors/mysql.md
+++ b/docs/content/docs/connectors/mysql.md
@@ -24,16 +24,15 @@ specific language governing permissions and limitations
under the License.
-->
-# MySQL CDC Pipeline Connector
+# MySQL Connector
-The MySQL CDC Pipeline Connector allows for reading snapshot data and incremental data from MySQL database and provides end-to-end full-database data synchronization capabilities.
-This document describes how to setup the MySQL CDC Pipeline connector.
+MySQL connector allows reading snapshot data and incremental data from MySQL database and provides end-to-end full-database data synchronization capabilities.
+This document describes how to setup the MySQL connector.
-How to create Pipeline
-----------------
+## Example
-The pipeline for reading data from MySQL and sink to Doris can be defined as follows:
+An example of the pipeline for reading data from MySQL and sink to Doris can be defined as follows:
```yaml
source:
@@ -58,8 +57,7 @@ pipeline:
parallelism: 4
```
-Pipeline Connector Options
-----------------
+## Connector Options
<div class="highlight">
<table class="colwidths-auto docutils">
@@ -245,8 +243,7 @@ Pipeline Connector Options
</table>
</div>
-Startup Reading Position
---------
+## Startup Reading Position
The config option `scan.startup.mode` specifies the startup mode for MySQL CDC consumer. The valid enumerations are:
@@ -259,16 +256,15 @@ The config option `scan.startup.mode` specifies the startup mode for MySQL CDC c
- `timestamp`: Skip snapshot phase and start reading binlog events from a specific timestamp.
-Data Type Mapping
-----------------
+## Data Type Mapping
<div class="wy-table-responsive">
<table class="colwidths-auto docutils">
<thead>
<tr>
- <th class="text-left" style="width:30%;">MySQL type<a href="https://dev.mysql.com/doc/man/8.0/en/data-types.html"></a></th>
- <th class="text-left" style="width:10%;">CDC type</th>
- <th class="text-left" style="width:60%;">NOTE</th>
+ <th class="text-left" style="width:30%;"><a href="https://dev.mysql.com/doc/man/8.0/en/data-types.html">MySQL type</a></th>
+ <th class="text-left" style="width:10%;">Flink CDC type</th>
+ <th class="text-left" style="width:60%;">Note</th>
</tr>
</thead>
<tbody>
diff --git a/docs/content/docs/connectors/overview.md b/docs/content/docs/connectors/overview.md
index 00b2bf2a3..0eee7bdfa 100644
--- a/docs/content/docs/connectors/overview.md
+++ b/docs/content/docs/connectors/overview.md
@@ -24,21 +24,33 @@ specific language governing permissions and limitations
under the License.
-->
-# Pipeline Connectors Of CDC Streaming ELT Framework
+# Connectors
+
+Flink CDC provides several source and sink connectors to interact with external
+systems. You can use these connectors out-of-box, by adding released JARs to
+your Flink CDC environment, and specifying the connector in your YAML pipeline
+definition.
## Supported Connectors
-| Connector | Database |
-|---------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| [doris-pipeline](doris-pipeline.md) | <li> [Doris](https://doris.apache.org/): 1.2.x, 2.x.x |
-| [mysql-pipeline](mysql-pipeline.md) | <li> [MySQL](https://dev.mysql.com/doc): 5.6, 5.7, 8.0.x <li> [RDS MySQL](https://www.aliyun.com/product/rds/mysql): 5.6, 5.7, 8.0.x <li> [PolarDB MySQL](https://www.aliyun.com/product/polardb): 5.6, 5.7, 8.0.x <li> [Aurora MySQL](https://aws.amazon.com/cn/rds/aurora): 5.6, 5.7, 8.0.x <li> [MariaDB](https://mariadb.org): 10.x <li> [PolarDB X](https://github.com/ApsaraDB/galaxysql): 2.0.1 |
-| [starrocks-pipeline](starrocks-pipeline.md) | <li> [StarRocks](https://www.starrocks.io/): 2.x, 3.x |
+| Connector | Supported Type | External System |
+|------------------------------------------------------|----------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| [Apache Doris]({{< ref "docs/connectors/doris" >}}) | Sink | <li> [Apache Doris](https://doris.apache.org/): 1.2.x, 2.x.x |
+| [MySQL]({{< ref "docs/connectors/mysql" >}}) | Source | <li> [MySQL](https://dev.mysql.com/doc): 5.6, 5.7, 8.0.x <li> [RDS MySQL](https://www.aliyun.com/product/rds/mysql): 5.6, 5.7, 8.0.x <li> [PolarDB MySQL](https://www.aliyun.com/product/polardb): 5.6, 5.7, 8.0.x <li> [Aurora MySQL](https://aws.amazon.com/cn/rds/aurora): 5.6, 5.7, 8.0.x <li> [MariaDB](https://mariadb.org): 10.x <li> [PolarDB X](https://github.com/ApsaraDB/galaxysql): 2.0.1 |
+| [StarRocks]({{< ref "docs/connectors/starrocks" >}}) | Sink | <li> [StarRocks](https://www.starrocks.io/): 2.x, 3.x |
+
+## Develop Your Own Connector
+
+If provided connectors cannot fulfill your requirement, you can always develop
+your own connector to get your external system involved in Flink CDC pipelines.
+Check out [Flink CDC APIs]({{< ref "docs/developer-guide/understand-flink-cdc-api" >}})
+to learn how to develop your own connectors.
-## Supported Flink Versions
-The following table shows the version mapping between Flink<sup>®</sup> CDC Pipeline and Flink<sup>®</sup>:
+## Legacy Flink CDC Sources
-| Flink<sup>®</sup> CDC Version | Flink<sup>®</sup> Version |
-|:-----------------------------------:|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|
-| <font color="DarkCyan">3.0.*</font> | <font color="MediumVioletRed">1.14.\*</font>, <font color="MediumVioletRed">1.15.\*</font>, <font color="MediumVioletRed">1.16.\*</font>, <font color="MediumVioletRed">1.17.\*</font>, <font color="MediumVioletRed">1.18.\*</font> |
+Flink CDC sources introduces before 3.0 are still available as normal Flink
+connector sources. You can find more details in the
+[overview page]({{< ref "docs/connectors/legacy-flink-cdc-sources/overview" >}})
+of legacy Flink CDC sources.
{{< top >}}
diff --git a/docs/content/docs/connectors/starrocks.md b/docs/content/docs/connectors/starrocks.md
index 64dff6689..84a4a99d1 100644
--- a/docs/content/docs/connectors/starrocks.md
+++ b/docs/content/docs/connectors/starrocks.md
@@ -24,17 +24,16 @@ specific language governing permissions and limitations
under the License.
-->
-# StarRocks Pipeline Connector
+# StarRocks Connector
-The StarRocks Pipeline connector can be used as the *Data Sink* of the pipeline, and write data to [StarRocks](https://github.com/StarRocks/starrocks). This document describes how to set up the StarRocks Pipeline connector.
+StarRocks connector can be used as the *Data Sink* of the pipeline, and write data to [StarRocks](https://github.com/StarRocks/starrocks). This document describes how to set up the StarRocks connector.
## What can the connector do?
* Create table automatically if not exist
* Schema change synchronization
* Data synchronization
-How to create Pipeline
-----------------
+## Example
The pipeline for reading data from MySQL and sink to StarRocks can be defined as follows:
@@ -62,8 +61,8 @@ pipeline:
parallelism: 2
```
-Pipeline Connector Options
-----------------
+## Connector Options
+
<div class="highlight">
<table class="colwidths-auto docutils">
<thead>
@@ -220,8 +219,8 @@ Pipeline Connector Options
</table>
</div>
-Usage Notes
---------
+
+## Usage Notes
* Only support StarRocks primary key table, so the source table must have primary keys.
@@ -244,15 +243,14 @@ Usage Notes
to write data to StarRocks. You can see [sink documentation](https://github.com/StarRocks/starrocks-connector-for-apache-flink/blob/main/docs/content/connector-sink.md)
for how it works.
-Data Type Mapping
-----------------
+## Data Type Mapping
<div class="wy-table-responsive">
<table class="colwidths-auto docutils">
<thead>
<tr>
- <th class="text-left">CDC type</th>
+ <th class="text-left">Flink CDC type</th>
<th class="text-left">StarRocks type</th>
- <th class="text-left" style="width:60%;">NOTE</th>
+ <th class="text-left" style="width:60%;">Note</th>
</tr>
</thead>
<tbody>