You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "yuxiqian (via GitHub)" <gi...@apache.org> on 2026/03/27 04:57:45 UTC
[PR] [docs] Polish README.md [flink-cdc]
yuxiqian opened a new pull request, #4349:
URL: https://github.com/apache/flink-cdc/pull/4349
This PR updates README.md to clarify YAML / SQL / DataStream API usages, added connector downloading link and version support overview.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [docs] Polish README.md [flink-cdc]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #4349:
URL: https://github.com/apache/flink-cdc/pull/4349#discussion_r2999047314
##########
README.md:
##########
@@ -15,132 +15,145 @@
<img src="https://img.shields.io/github/actions/workflow/status/apache/flink-cdc/flink_cdc_ci_nightly.yml?branch=master&label=nightly" alt="Nightly Build">
</a>
<a href="https://github.com/apache/flink-cdc/tree/master/LICENSE" target="_blank">
- <img src="https://img.shields.io/static/v1?label=license&message=Apache License 2.0&color=white" alt="License">
+ <img src="https://img.shields.io/github/license/apache/flink-cdc?color=white" alt="License">
</a>
</p>
-Flink CDC is a distributed data integration tool for real time data and batch data. Flink CDC brings the simplicity
-and elegance of data integration via YAML to describe the data movement and transformation in a
-[Data Pipeline](docs/content/docs/core-concept/data-pipeline.md).
+Flink CDC is a distributed data integration tool for real time data and batch data, built on top of Apache Flink. It prioritizes efficient end-to-end data integration and offers enhanced functionalities such as full database synchronization, sharding table synchronization, schema evolution and data transformation.
+
-The Flink CDC prioritizes efficient end-to-end data integration and offers enhanced functionalities such as
-full database synchronization, sharding table synchronization, schema evolution and data transformation.
+## API Layers
+
+Flink CDC provides three API layers for different usage scenarios:
+
+### 1. YAML API (Pipeline API)
+
+The YAML API provides a declarative, zero-code approach to define data pipelines. Users describe the source, sink, [routing](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/core-concept/route/), [transformation](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/core-concept/transform/), and [schema evolution](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/core-concept/schema-evolution/) rules in a YAML file and submit it via the `flink-cdc.sh` CLI.
+
+Please refer to the [Quickstart Guide](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/get-started/introduction/) for detailed setup instructions.
+
+```yaml
+source:
+ type: mysql
+ hostname: localhost
+ port: 3306
+ username: root
+ password: 123456
+ tables: app_db\.\.*
Review Comment:
In the YAML example, `tables: app_db\.\.\.*` doesn’t match the table selector format used in the project docs (e.g., `app_db.\.*` in docs/core-concept/data-pipeline.md). As written, the escaped `\.` will be treated differently and can prevent matching all tables under `app_db` as intended; please align this example with the documented pattern syntax.
```suggestion
tables: app_db.\.*
```
##########
README.md:
##########
@@ -15,132 +15,145 @@
<img src="https://img.shields.io/github/actions/workflow/status/apache/flink-cdc/flink_cdc_ci_nightly.yml?branch=master&label=nightly" alt="Nightly Build">
</a>
<a href="https://github.com/apache/flink-cdc/tree/master/LICENSE" target="_blank">
- <img src="https://img.shields.io/static/v1?label=license&message=Apache License 2.0&color=white" alt="License">
+ <img src="https://img.shields.io/github/license/apache/flink-cdc?color=white" alt="License">
</a>
</p>
-Flink CDC is a distributed data integration tool for real time data and batch data. Flink CDC brings the simplicity
-and elegance of data integration via YAML to describe the data movement and transformation in a
-[Data Pipeline](docs/content/docs/core-concept/data-pipeline.md).
+Flink CDC is a distributed data integration tool for real time data and batch data, built on top of Apache Flink. It prioritizes efficient end-to-end data integration and offers enhanced functionalities such as full database synchronization, sharding table synchronization, schema evolution and data transformation.
Review Comment:
Minor wording: "real time" should be hyphenated as "real-time" when used as a compound modifier (e.g., "real-time data").
```suggestion
Flink CDC is a distributed data integration tool for real-time data and batch data, built on top of Apache Flink. It prioritizes efficient end-to-end data integration and offers enhanced functionalities such as full database synchronization, sharding table synchronization, schema evolution and data transformation.
```
##########
README.md:
##########
@@ -15,132 +15,145 @@
<img src="https://img.shields.io/github/actions/workflow/status/apache/flink-cdc/flink_cdc_ci_nightly.yml?branch=master&label=nightly" alt="Nightly Build">
</a>
<a href="https://github.com/apache/flink-cdc/tree/master/LICENSE" target="_blank">
- <img src="https://img.shields.io/static/v1?label=license&message=Apache License 2.0&color=white" alt="License">
+ <img src="https://img.shields.io/github/license/apache/flink-cdc?color=white" alt="License">
</a>
</p>
-Flink CDC is a distributed data integration tool for real time data and batch data. Flink CDC brings the simplicity
-and elegance of data integration via YAML to describe the data movement and transformation in a
-[Data Pipeline](docs/content/docs/core-concept/data-pipeline.md).
+Flink CDC is a distributed data integration tool for real time data and batch data, built on top of Apache Flink. It prioritizes efficient end-to-end data integration and offers enhanced functionalities such as full database synchronization, sharding table synchronization, schema evolution and data transformation.
+
-The Flink CDC prioritizes efficient end-to-end data integration and offers enhanced functionalities such as
-full database synchronization, sharding table synchronization, schema evolution and data transformation.
+## API Layers
+
+Flink CDC provides three API layers for different usage scenarios:
+
+### 1. YAML API (Pipeline API)
+
+The YAML API provides a declarative, zero-code approach to define data pipelines. Users describe the source, sink, [routing](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/core-concept/route/), [transformation](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/core-concept/transform/), and [schema evolution](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/core-concept/schema-evolution/) rules in a YAML file and submit it via the `flink-cdc.sh` CLI.
+
+Please refer to the [Quickstart Guide](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/get-started/introduction/) for detailed setup instructions.
+
+```yaml
+source:
+ type: mysql
+ hostname: localhost
+ port: 3306
+ username: root
+ password: 123456
+ tables: app_db\.\.*
+
+sink:
+ type: doris
+ fenodes: 127.0.0.1:8030
+ username: root
+ password: ""
+
+# Transform data on-the-fly
+transform:
+ - source-table: app_db.orders
+ projection: id, order_id, UPPER(product_name) as product_name
+ filter: id > 10 AND order_id > 100
+
+# Route source tables to different sink tables
+route:
+ - source-table: app_db.orders
+ sink-table: ods_db.ods_orders
+ - source-table: app_db.shipments
+ sink-table: ods_db.ods_shipments
+ - source-table: app_db.\.*
+ sink-table: ods_db.others
+
+pipeline:
+ name: Sync MySQL Database to Doris
+ parallelism: 2
+ schema.change.behavior: evolve # Support schema evolution
+```
-
+**Pipeline connectors:**
-### Quickstart Guide
+[Doris](https://mvnrepository.com/artifact/org.apache.flink/flink-cdc-pipeline-connector-doris) | [Elasticsearch](https://mvnrepository.com/artifact/org.apache.flink/flink-cdc-pipeline-connector-elasticsearch) | [Fluss](https://mvnrepository.com/artifact/org.apache.flink/flink-cdc-pipeline-connector-fluss) | [Hudi](https://mvnrepository.com/artifact/org.apache.flink/flink-cdc-pipeline-connector-hudi) | [Iceberg](https://mvnrepository.com/artifact/org.apache.flink/flink-cdc-pipeline-connector-iceberg) | [Kafka](https://mvnrepository.com/artifact/org.apache.flink/flink-cdc-pipeline-connector-kafka) | [MaxCompute](https://mvnrepository.com/artifact/org.apache.flink/flink-cdc-pipeline-connector-maxcompute) | [MySQL](https://mvnrepository.com/artifact/org.apache.flink/flink-cdc-pipeline-connector-mysql) | [OceanBase](https://mvnrepository.com/artifact/org.apache.flink/flink-cdc-pipeline-connector-oceanbase) | [Oracle](https://mvnrepository.com/artifact/org.apache.flink/flink-cdc-pipeline
-connector-oracle) | [Paimon](https://mvnrepository.com/artifact/org.apache.flink/flink-cdc-pipeline-connector-paimon) | [PostgreSQL](https://mvnrepository.com/artifact/org.apache.flink/flink-cdc-pipeline-connector-postgres) | [StarRocks](https://mvnrepository.com/artifact/org.apache.flink/flink-cdc-pipeline-connector-starrocks)
-Flink CDC provides a CdcUp CLI utility to start a playground environment and run Flink CDC jobs.
-You will need to have a working Docker and Docker compose environment to use it.
+See the [connector overview](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/connectors/pipeline-connectors/overview/) for a full list and configurations.
-1. Run `git clone https://github.com/apache/flink-cdc.git --depth=1` to retrieve a copy of Flink CDC source code.
-2. Run `cd tools/cdcup/ && ./cdcup.sh init` to use the CdcUp tool to start a playground environment.
-3. Run `./cdcup.sh up` to boot-up docker containers, and wait for them to be ready.
-4. Run `./cdcup.sh mysql` to open a MySQL session, and create at least one table.
+### 2. SQL API (Table/SQL API)
+
+The SQL API integrates with Flink SQL, allowing users to define CDC sources using SQL DDL statements. Deploy the SQL connector JAR to `FLINK_HOME/lib/` and use it directly in Flink SQL Client:
```sql
--- initialize db and table
-CREATE DATABASE cdc_playground;
-USE cdc_playground;
-CREATE TABLE test_table (id INT PRIMARY KEY, name VARCHAR(32));
+CREATE TABLE mysql_binlog (
+ id INT NOT NULL,
+ name STRING,
+ description STRING,
+ weight DECIMAL(10,3),
+ PRIMARY KEY(id) NOT ENFORCED
+) WITH (
+ 'connector' = 'mysql-cdc',
+ 'hostname' = 'localhost',
+ 'port' = '3306',
+ 'username' = 'flinkuser',
+ 'password' = 'flinkpw',
+ 'database-name' = 'inventory',
+ 'table-name' = 'products'
+);
+
+SELECT id, UPPER(name), description, weight FROM mysql_binlog;
+```
--- insert test data
-INSERT INTO test_table VALUES (1, 'alice'), (2, 'bob'), (3, 'cicada'), (4, 'derrida');
+**Available SQL connectors** (dependencies bundled):
--- verify if it has been successfully inserted
-SELECT * FROM test_table;
-```
+[MySQL](https://mvnrepository.com/artifact/org.apache.flink/flink-sql-connector-mysql-cdc) | [PostgreSQL](https://mvnrepository.com/artifact/org.apache.flink/flink-sql-connector-postgres-cdc) | [Oracle](https://mvnrepository.com/artifact/org.apache.flink/flink-sql-connector-oracle-cdc) | [SQL Server](https://mvnrepository.com/artifact/org.apache.flink/flink-sql-connector-sqlserver-cdc) | [MongoDB](https://mvnrepository.com/artifact/org.apache.flink/flink-sql-connector-mongodb-cdc) | [OceanBase](https://mvnrepository.com/artifact/org.apache.flink/flink-sql-connector-oceanbase-cdc) | [TiDB](https://mvnrepository.com/artifact/org.apache.flink/flink-sql-connector-tidb-cdc) | [Db2](https://mvnrepository.com/artifact/org.apache.flink/flink-sql-connector-db2-cdc) | [Vitess](https://mvnrepository.com/artifact/org.apache.flink/flink-sql-connector-vitess-cdc)
-5. Run `./cdcup.sh pipeline pipeline-definition.yaml` to submit the pipeline job. You may also edit the pipeline definition file for further configurations.
-6. Run `./cdcup.sh flink` to access the Flink Web UI.
-
-### Getting Started
-
-1. Prepare a [Apache Flink](https://nightlies.apache.org/flink/flink-docs-master/docs/try-flink/local_installation/#starting-and-stopping-a-local-cluster) cluster and set up `FLINK_HOME` environment variable.
-2. [Download](https://github.com/apache/flink-cdc/releases) Flink CDC tar, unzip it and put jars of pipeline connector to Flink `lib` directory.
-
-> If you're using macOS or Linux, you may use `brew install apache-flink-cdc` to install Flink CDC and compatible connectors quickly.
-
-3. Create a **YAML** file to describe the data source and data sink, the following example synchronizes all tables under MySQL app_db database to Doris :
- ```yaml
- source:
- type: mysql
- hostname: localhost
- port: 3306
- username: root
- password: 123456
- tables: app_db.\.*
-
- sink:
- type: doris
- fenodes: 127.0.0.1:8030
- username: root
- password: ""
-
- transform:
- - source-table: adb.web_order01
- projection: \*, format('%S', product_name) as product_name
- filter: addone(id) > 10 AND order_id > 100
- description: project fields and filter
- - source-table: adb.web_order02
- projection: \*, format('%S', product_name) as product_name
- filter: addone(id) > 20 AND order_id > 200
- description: project fields and filter
-
- route:
- - source-table: app_db.orders
- sink-table: ods_db.ods_orders
- - source-table: app_db.shipments
- sink-table: ods_db.ods_shipments
- - source-table: app_db.products
- sink-table: ods_db.ods_products
-
- pipeline:
- name: Sync MySQL Database to Doris
- parallelism: 2
- route-mode: ALL_MATCH
- user-defined-function:
- - name: addone
- classpath: com.example.functions.AddOneFunctionClass
- - name: format
- classpath: com.example.functions.FormatFunctionClass
- ```
-4. Submit pipeline job using `flink-cdc.sh` script.
- ```shell
- bash bin/flink-cdc.sh /path/mysql-to-doris.yaml
- ```
-5. View job execution status through Flink WebUI or downstream database.
-
-Try it out yourself with our more detailed [tutorial](docs/content/docs/get-started/quickstart/mysql-to-doris.md).
-You can also see [connector overview](docs/content/docs/connectors/pipeline-connectors/overview.md) to view a comprehensive catalog of the
-connectors currently provided and understand more detailed configurations.
-
-### Join the Community
+See the [source connector overview](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/connectors/flink-sources/overview/) for a full list and configurations.
-There are many ways to participate in the Apache Flink CDC community. The
-[mailing lists](https://flink.apache.org/what-is-flink/community/#mailing-lists) are the primary place where all Flink
-committers are present. For user support and questions use the user mailing list. If you've found a problem of Flink CDC,
-please create a [Flink jira](https://issues.apache.org/jira/projects/FLINK/summary) and tag it with the `Flink CDC` tag.
-Bugs and feature requests can either be discussed on the dev mailing list or on Jira.
+### 3. DataStream API
+The DataStream API provides programmatic access for building custom Flink streaming applications. Add the corresponding connector as a Maven dependency:
+```xml
+<dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-connector-mysql-cdc</artifactId>
+ <version>${flink-cdc.version}</version>
+</dependency>
+```
-### Contributing
+**Available source connectors:**
-Welcome to contribute to Flink CDC, please see our [Developer Guide](docs/content/docs/developer-guide/contribute-to-flink-cdc.md)
-and [APIs Guide](docs/content/docs/developer-guide/understand-flink-cdc-api.md).
+[MySQL](https://mvnrepository.com/artifact/org.apache.flink/flink-connector-mysql-cdc) | [PostgreSQL](https://mvnrepository.com/artifact/org.apache.flink/flink-connector-postgres-cdc) | [Oracle](https://mvnrepository.com/artifact/org.apache.flink/flink-connector-oracle-cdc) | [SQL Server](https://mvnrepository.com/artifact/org.apache.flink/flink-connector-sqlserver-cdc) | [MongoDB](https://mvnrepository.com/artifact/org.apache.flink/flink-connector-mongodb-cdc) | [OceanBase](https://mvnrepository.com/artifact/org.apache.flink/flink-connector-oceanbase-cdc) | [TiDB](https://mvnrepository.com/artifact/org.apache.flink/flink-connector-tidb-cdc) | [Db2](https://mvnrepository.com/artifact/org.apache.flink/flink-connector-db2-cdc) | [Vitess](https://mvnrepository.com/artifact/org.apache.flink/flink-connector-vitess-cdc)
+All artifacts use group ID `org.apache.flink`. See the [DataStream API packaging guide](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/connectors/flink-sources/datastream-api-package-guidance/) for a complete `pom.xml` example.
+## Flink Version Compatibility
-### License
+| Flink CDC Version | Flink Version | Notes |
+|-------------------|---------------------------|-----------------------------------|
+| 3.6.x | 1.20.\*, 2.2.\* | |
+| 3.5.x | 1.19.\*, 1.20.\* | |
+| 3.4.x | 1.19.\*, 1.20.\* | |
+| 3.3.x | 1.19.\*, 1.20.\* | |
+| 3.2.x | 1.17.\*, 1.18.\*, 1.19.\* | |
+| 3.1.x | 1.17.\*, 1.18.\*, 1.19.\* | Only 3.1.1 supports Flink 1.19.\* |
Review Comment:
The Flink version compatibility table is ambiguous about which API layer it applies to. In the docs, DataStream/SQL connectors list 3.2.* as supporting Flink 1.20.*, while pipeline connectors list 3.2.x only up to 1.19.*; the README table currently presents a single mapping without clarifying this distinction, which can mislead users. Consider splitting the matrix by API layer (Pipeline vs Sources/SQL) or adding a clear note on scope.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [docs] Polish README.md [flink-cdc]
Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian merged PR #4349:
URL: https://github.com/apache/flink-cdc/pull/4349
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org