You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "qyfftf (via GitHub)" <gi...@apache.org> on 2025/02/05 08:03:50 UTC

[PR] [FLINK-37156] [cdc-composer/cli] pipeline supports collecting data once and writing it to multiple sinks. [flink-cdc]

qyfftf opened a new pull request, #3906:
URL: https://github.com/apache/flink-cdc/pull/3906

   I've implemented a solution where data is collected from MySQL once and then sent to multiple target data sources, such as Paimon, Kafka, and StarRocks, which reduces redundant data collection.
   pipeline yaml :
   ```
   source:
     type: mysql
     name: MySQL Source
     hostname: 127.0.0.1
     port: 3306
     username:root
     password: 123456
     tables: test.order
     server-id: 5401-5404
     jdbc.properties.useSSL: false
   
   sink:
      - type: paimon
        name: Paimon Sink
        catalog.properties.metastore: filesystem
        catalog.properties.warehouse: /tmp/path/warehouse
      - type: kafka
        name: Kafka Sink
        properties.bootstrap.servers: PLAINTEXT://localhost:9092
      - type: starrocks
        jdbc-url: jdbc:mysql://127.0.0.1:9030
        load-url: 127.0.0.1:8030
        username: root
        password: ""
        table.create.properties.replication_num: 1
   
   route:
     - source-table:  test.order
       sink-table:  test.order
   
   pipeline:
     name: MySQL to Paimon Pipeline
     parallelism: 2
   ```
   
   
   <img width="1611" alt="image" src="https://github.com/user-attachments/assets/00ff0811-d682-4cfe-b21e-18ca56bbf53f" />
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37156] [cdc-composer/cli] pipeline supports collecting data once and writing it to multiple sinks. [flink-cdc]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3906:
URL: https://github.com/apache/flink-cdc/pull/3906#issuecomment-2964584614

   This pull request has been automatically marked as stale because it has not had recent activity for 120 days. It will be closed in 60 days if no further activity occurs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37156] [cdc-composer/cli] pipeline supports collecting data once and writing it to multiple sinks. [flink-cdc]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #3906: [FLINK-37156] [cdc-composer/cli] pipeline supports collecting data once and writing it to multiple sinks.
URL: https://github.com/apache/flink-cdc/pull/3906


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37156] [cdc-composer/cli] pipeline supports collecting data once and writing it to multiple sinks. [flink-cdc]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3906:
URL: https://github.com/apache/flink-cdc/pull/3906#issuecomment-3172984422

   This pull request has been closed because it has not had recent activity. You could reopen it if you try to continue your work, and anyone who are interested in it are encouraged to continue work on this pull request.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37156] [cdc-composer/cli] pipeline supports collecting data once and writing it to multiple sinks. [flink-cdc]

Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on PR #3906:
URL: https://github.com/apache/flink-cdc/pull/3906#issuecomment-2649616085

   Thanks for @qyfftf's contribution! I have some concerns over the topology design.
   
   According to the new topology graph you've posted, seems there's only one `SchemaOperator` in the upstream while there are multiple downstream sinks. How can a single `SchemaOperator` maintain and evolve schemas in multiple sinks? What if downstream schemas differ (presumably caused by tolerated errors)?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org