You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/06/04 08:14:36 UTC
[GitHub] [airflow] SteNicholas opened a new issue #9134: Support Flink operator
SteNicholas opened a new issue #9134:
URL: https://github.com/apache/airflow/issues/9134
**Description**
Airflow currently supports Spark operators for kicking off a spark-submit job. In real-time computing or online machine learning scenarios, Flink operator could be designed to submit flink job in streaming workflow.
**Use case / motivation**
Airflow coud supports definition of `FlinkSubmitOperator` for DAG composed of multiple Flink jobs. `FlinkSubmitOperator` is designed to introduce the operator that kicks off a flink-submit task, and provides corresponding to JDBC and SQL operators.
**Related Issues**
N/A
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi commented on issue #9134: Support Flink operator
Posted by GitBox <gi...@apache.org>.
subkanthi commented on issue #9134:
URL: https://github.com/apache/airflow/issues/9134#issuecomment-659608175
@mik-laj , @zhongjiajie is it OK if i just work on the simple flinksubmit operator first, maybe we can have another issue for the streaming part?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] SteNicholas commented on issue #9134: Support Flink operator
Posted by GitBox <gi...@apache.org>.
SteNicholas commented on issue #9134:
URL: https://github.com/apache/airflow/issues/9134#issuecomment-667779853
Do you mean that the implemenation of FlinkOperator is based on `flink run` command? But If you want to kill Flink Job, you must persist with the job id. Therefore, how do you control with this presistence?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] SteNicholas edited a comment on issue #9134: Support Flink operator
Posted by GitBox <gi...@apache.org>.
SteNicholas edited a comment on issue #9134:
URL: https://github.com/apache/airflow/issues/9134#issuecomment-654724805
@subkanthi I thought that you currently couldn't complete this issue. Because Airflow couldn't support the streaming semantic, how do you work on this issue? @mik-laj @zhongjiajie What do we think about?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi commented on issue #9134: Support Flink operator
Posted by GitBox <gi...@apache.org>.
subkanthi commented on issue #9134:
URL: https://github.com/apache/airflow/issues/9134#issuecomment-653909511
If this is not being worked on, I can take it on, please let me know. @zhongjiajie
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #9134: Support Flink operator
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #9134:
URL: https://github.com/apache/airflow/issues/9134#issuecomment-653931976
@subkanthi I assigned you to this ticket. 🐈
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] SteNicholas commented on issue #9134: Support Flink operator
Posted by GitBox <gi...@apache.org>.
SteNicholas commented on issue #9134:
URL: https://github.com/apache/airflow/issues/9134#issuecomment-654724805
@subkanthi I thought that you currently couldn't complete this issue. Because the streaming semantic couldn't support, how do you work on this issue?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] SteNicholas commented on issue #9134: Support Flink operator
Posted by GitBox <gi...@apache.org>.
SteNicholas commented on issue #9134:
URL: https://github.com/apache/airflow/issues/9134#issuecomment-683379870
@sreenath-kamath Submitting Flink job always uses Flink command, and using the flink REST API's to submit the job isn't friendly to Airflow's users.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ChethanUK edited a comment on issue #9134: Support Flink operator
Posted by GitBox <gi...@apache.org>.
ChethanUK edited a comment on issue #9134:
URL: https://github.com/apache/airflow/issues/9134#issuecomment-867418283
Working on FlinkOnK8sOperator, will send PR soon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] SteNicholas commented on issue #9134: Support Flink operator
Posted by GitBox <gi...@apache.org>.
SteNicholas commented on issue #9134:
URL: https://github.com/apache/airflow/issues/9134#issuecomment-661752411
@subkanthi How does the operator teminate? The flink submit for streaming application doesn't teminate.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mdianjun commented on issue #9134: Support Flink operator
Posted by GitBox <gi...@apache.org>.
mdianjun commented on issue #9134:
URL: https://github.com/apache/airflow/issues/9134#issuecomment-867290450
@subkanthi Hello，is there any progress on this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ChethanUK commented on issue #9134: Support Flink operator
Posted by GitBox <gi...@apache.org>.
ChethanUK commented on issue #9134:
URL: https://github.com/apache/airflow/issues/9134#issuecomment-867418283
Working on this, will send PR soon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] zhongjiajie commented on issue #9134: Support Flink operator
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on issue #9134:
URL: https://github.com/apache/airflow/issues/9134#issuecomment-654976434
I think it depend on what we could add streaming to airflow, if it will task many times, maybe support parts of flink function on flink operator is acceptable.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #9134: Support Flink operator
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #9134:
URL: https://github.com/apache/airflow/issues/9134#issuecomment-638687114
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi commented on issue #9134: Support Flink operator
Posted by GitBox <gi...@apache.org>.
subkanthi commented on issue #9134:
URL: https://github.com/apache/airflow/issues/9134#issuecomment-655501248
I thought this was very similar to the SparkSubmit operator where it expects spark-submit to be in the path and takes the jar file as a parameter.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #9134: Support Flink operator
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #9134:
URL: https://github.com/apache/airflow/issues/9134#issuecomment-654820970
@SteNicholas I think we can add two operators. One that starts streaming operations on a remote cluster, return ID to XCOM and the other that stops these operations. However, I have no experience with Flink. Do you think this is possible?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] SteNicholas edited a comment on issue #9134: Support Flink operator
Posted by GitBox <gi...@apache.org>.
SteNicholas edited a comment on issue #9134:
URL: https://github.com/apache/airflow/issues/9134#issuecomment-667779853
@subkanthi Do you mean that the implemenation of FlinkOperator is based on `flink run` command? But If you want to kill Flink Job, you must persist with the job id. Therefore, how do you control with this presistence?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] zhongjiajie commented on issue #9134: Support Flink operator
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on issue #9134:
URL: https://github.com/apache/airflow/issues/9134#issuecomment-656665534
> was
But that parts of flink function, nor full of it. maybe you could see https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-35+Add+Signal+Based+Scheduling+To+Airflow for more detail.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] zhongjiajie commented on issue #9134: Support Flink operator
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on issue #9134:
URL: https://github.com/apache/airflow/issues/9134#issuecomment-638729228
Sound good! looking forward your contribute. BTW, Airflow open to all new and useful operators.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi commented on issue #9134: Support Flink operator
Posted by GitBox <gi...@apache.org>.
subkanthi commented on issue #9134:
URL: https://github.com/apache/airflow/issues/9134#issuecomment-661862683
Is it not the same as spark-submit kill command, we use the flink cancel command.
./bin/flink cancel <jobID>
Are we all referring to starting and cancelling jobs using the CLI?
https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/cli.html
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sreenath-kamath commented on issue #9134: Support Flink operator
Posted by GitBox <gi...@apache.org>.
sreenath-kamath commented on issue #9134:
URL: https://github.com/apache/airflow/issues/9134#issuecomment-683371059
Why not use the flink REST API's to submit the job and store the job_id in the xcom variable and exit the process, https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/rest_api.html. Incase the user wants to poll for the status of the job in the dag a `Sensor` can also be written which takes the job id as an argument and ensures that the job is still running. wdyt ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] punit-zeta commented on issue #9134: Support Flink operator
Posted by GitBox <gi...@apache.org>.
punit-zeta commented on issue #9134:
URL: https://github.com/apache/airflow/issues/9134#issuecomment-993182963
Hi, @ChethanUK any update on this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] zhongjiajie edited a comment on issue #9134: Support Flink operator
Posted by GitBox <gi...@apache.org>.
zhongjiajie edited a comment on issue #9134:
URL: https://github.com/apache/airflow/issues/9134#issuecomment-656665534
But that parts of flink function, nor full of it. maybe you could see https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-35+Add+Signal+Based+Scheduling+To+Airflow for more detail.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org