You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "yuxiqian (via GitHub)" <gi...@apache.org> on 2024/05/09 09:03:44 UTC
[PR] [FLINK-35317][cli] Supports submitting multiple pipeline jobs at once [flink-cdc]
yuxiqian opened a new pull request, #3306:
URL: https://github.com/apache/flink-cdc/pull/3306
This closes [FLINK-35317](https://issues.apache.org/jira/browse/FLINK-35317).
Currently, Flink CDC CLI only allows submitting one YAML pipeline job each time. This PR allows submitting multiple .yml files at once like this:
`./bin/flink-cdc.sh job1.yml job2.yml job3.yml --flink-home /opt/flink ...`
prints the following output:
```
Pipeline has been submitted to cluster.
Job ID: 60f6e58bd3c0f54ee51b703d7e59b9ca
Job Description: Dummy Pipeline 1
Pipeline has been submitted to cluster.
Job ID: e4403683aaacf185f87db4a769a1db74
Job Description: Dummy Pipeline 2
Pipeline has been submitted to cluster.
Job ID: 350737f6e5e3d7cfaa0b17f77e5a5035
Job Description: Dummy Pipeline 3
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-35317][cli] Supports submitting multiple pipeline jobs at once [flink-cdc]
Posted by "loserwang1024 (via GitHub)" <gi...@apache.org>.
loserwang1024 commented on PR #3306:
URL: https://github.com/apache/flink-cdc/pull/3306#issuecomment-2103714999
I don't know whether we need to do it? Foe example, what happened if fist job is submitted success but the later is fail. Will we cancel the first one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-35317][cli] Supports submitting multiple pipeline jobs at once [flink-cdc]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3306:
URL: https://github.com/apache/flink-cdc/pull/3306#issuecomment-2477662338
This pull request has been automatically marked as stale because it has not had recent activity for 60 days. It will be closed in 30 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-35317][cli] Supports submitting multiple pipeline jobs at once [flink-cdc]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3306:
URL: https://github.com/apache/flink-cdc/pull/3306#issuecomment-2351233922
This pull request has been automatically marked as stale because it has not had recent activity for 60 days. It will be closed in 30 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-35317][cli] Supports submitting multiple pipeline jobs at once [flink-cdc]
Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian closed pull request #3306: [FLINK-35317][cli] Supports submitting multiple pipeline jobs at once
URL: https://github.com/apache/flink-cdc/pull/3306
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-35317][cli] Supports submitting multiple pipeline jobs at once [flink-cdc]
Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on PR #3306:
URL: https://github.com/apache/flink-cdc/pull/3306#issuecomment-2103734362
Hi @loserwang1024, the original idea is pipeline definition does not allow defining source & sink with different connectors for now, and user must write separated YAML files to define hetero-source / hetero-sink pipelines. For now CLI just submits all of them as individual jobs without caring if they run successfully or fail, just like a convenient alternative to executing `./flink-cdc.sh` for each YAML file.
In the future, CDC Composer might be able to do some optimization by reusing sources / sinks and compose merged operator topologies without changing the CLI frontend interface.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org