[jira] [Commented] (FLINK-34851) [Feature][Pipeline] Flink CDC pipeline supports transform

["Flink CDC Issue Import (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-34851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17828836#comment-17828836 ] 

Flink CDC Issue Import commented on FLINK-34851:
------------------------------------------------

Date: Mon Feb 26 23:42:17 CST 2024, Author: [aiwenmo|https://github.com/aiwenmo]

> Completed features:

1. Support projection.
2. Support filter.
3. Support automatic evolution of projection's column.
4. Support SQL operators, functions, constants, and metadata columns.
5. Support restoring jobs from save points.

> Todo:

- [ ] ([#3081|https://github.com/apache/flink-cdc/issues/3081] | [FLINK-3081|https://issues.apache.org/jira/browse/FLINK-3081]) 
- [ ] ([#3077|https://github.com/apache/flink-cdc/issues/3077] | [FLINK-3077|https://issues.apache.org/jira/browse/FLINK-3077]) 
- [ ] ([#3078|https://github.com/apache/flink-cdc/issues/3078] | [FLINK-3078|https://issues.apache.org/jira/browse/FLINK-3078]) 
- [ ] ([#3079|https://github.com/apache/flink-cdc/issues/3079] | [FLINK-3079|https://issues.apache.org/jira/browse/FLINK-3079]) 
- [ ] ([#3080|https://github.com/apache/flink-cdc/issues/3080] | [FLINK-3080|https://issues.apache.org/jira/browse/FLINK-3080]) 

> [Feature][Pipeline] Flink CDC pipeline supports transform
> ---------------------------------------------------------
>
>                 Key: FLINK-34851
>                 URL: https://issues.apache.org/jira/browse/FLINK-34851
>             Project: Flink
>          Issue Type: Improvement
>          Components: Flink CDC
>            Reporter: Flink CDC Issue Import
>            Priority: Major
>              Labels: github-import
>
> ### Search before asking
> - [X] I searched in the [issues|https://github.com/ververica/flink-cdc-connectors/issues] and found nothing similar.
> ### Motivation
> ```
> source:
>   type: mysql
>   name: source-database
>   host: localhost
>   port: 3306
>   username: admin
>   password: pass
>   tables: adb.*, bdb.user_table_[0-9]+, [app|web]_order_.*
>   chunk-column: app_order_.*:id,web_order:product_id
>   capture-new-tables: true
> sink:
>   type: kafka
>   name: sink-queue
>   bootstrap-servers: localhost:9092
>   auto-create-table: true
> route:
>   - source-table: mydb.default.app_order_.*
>     sink-table: odsdb.default.app_order
>     description: sync all sharding tables to one
>   - source-table: mydb.default.web_order
>     sink-table: odsdb.default.ods_web_order
>     description: sync table to with given prefix ods_
> transform:
>   - source-table: mydb.app_order_.*
>     projection: id, order_id, TO_UPPER(product_name)
>     filter: id > 10 AND order_id > 100
>     description: project fields from source table
>   - source-table: mydb.web_order_.*
>     projection: CONCAT(id, order_id) as uniq_id, *
>     filter: uniq_id > 10
>     description: add new uniq_id for each row
> pipeline:
>   name: source-database-sync-pipe
>   parallelism: 4
>   enable-schema-evolution: false
> ```
> ### Solution
> I am currently working on the implementation of this feature, and the relevant content will be added gradually in the following text.
> ### Alternatives
> None.
> ### Anything else?
> _No response_
> ### Are you willing to submit a PR?
> - [X] I'm willing to submit a PR!
> ---------------- Imported from GitHub ----------------
> Url: https://github.com/apache/flink-cdc/issues/2932
> Created by: [aiwenmo|https://github.com/aiwenmo]
> Labels: enhancement, 
> Created at: Wed Dec 27 00:22:36 CST 2023
> State: open



--
This message was sent by Atlassian Jira
(v8.20.10#820010)