You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "qyfftf (via GitHub)" <gi...@apache.org> on 2025/02/05 08:03:50 UTC
[PR] [FLINK-37156] [cdc-composer/cli] pipeline supports collecting data once and writing it to multiple sinks. [flink-cdc]
qyfftf opened a new pull request, #3906:
URL: https://github.com/apache/flink-cdc/pull/3906
I've implemented a solution where data is collected from MySQL once and then sent to multiple target data sources, such as Paimon, Kafka, and StarRocks, which reduces redundant data collection.
pipeline yaml :
```
source:
type: mysql
name: MySQL Source
hostname: 127.0.0.1
port: 3306
username:root
password: 123456
tables: test.order
server-id: 5401-5404
jdbc.properties.useSSL: false
sink:
- type: paimon
name: Paimon Sink
catalog.properties.metastore: filesystem
catalog.properties.warehouse: /tmp/path/warehouse
- type: kafka
name: Kafka Sink
properties.bootstrap.servers: PLAINTEXT://localhost:9092
- type: starrocks
jdbc-url: jdbc:mysql://127.0.0.1:9030
load-url: 127.0.0.1:8030
username: root
password: ""
table.create.properties.replication_num: 1
route:
- source-table: test.order
sink-table: test.order
pipeline:
name: MySQL to Paimon Pipeline
parallelism: 2
```
<img width="1611" alt="image" src="https://github.com/user-attachments/assets/00ff0811-d682-4cfe-b21e-18ca56bbf53f" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37156] [cdc-composer/cli] pipeline supports collecting data once and writing it to multiple sinks. [flink-cdc]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3906:
URL: https://github.com/apache/flink-cdc/pull/3906#issuecomment-2964584614
This pull request has been automatically marked as stale because it has not had recent activity for 120 days. It will be closed in 60 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37156] [cdc-composer/cli] pipeline supports collecting data once and writing it to multiple sinks. [flink-cdc]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #3906: [FLINK-37156] [cdc-composer/cli] pipeline supports collecting data once and writing it to multiple sinks.
URL: https://github.com/apache/flink-cdc/pull/3906
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37156] [cdc-composer/cli] pipeline supports collecting data once and writing it to multiple sinks. [flink-cdc]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3906:
URL: https://github.com/apache/flink-cdc/pull/3906#issuecomment-3172984422
This pull request has been closed because it has not had recent activity. You could reopen it if you try to continue your work, and anyone who are interested in it are encouraged to continue work on this pull request.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37156] [cdc-composer/cli] pipeline supports collecting data once and writing it to multiple sinks. [flink-cdc]
Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on PR #3906:
URL: https://github.com/apache/flink-cdc/pull/3906#issuecomment-2649616085
Thanks for @qyfftf's contribution! I have some concerns over the topology design.
According to the new topology graph you've posted, seems there's only one `SchemaOperator` in the upstream while there are multiple downstream sinks. How can a single `SchemaOperator` maintain and evolve schemas in multiple sinks? What if downstream schemas differ (presumably caused by tolerated errors)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org