You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/09/25 21:20:45 UTC
[GitHub] [airflow] chethanuk-plutoflume opened a new pull request, #26664: Adding Flink on K8s Operator #9134
chethanuk-plutoflume opened a new pull request, #26664:
URL: https://github.com/apache/airflow/pull/26664
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
related: #9134
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
Adding new Flink on K8s Operator to cncf provider
NOTE: The [Flink Kubernetes Operator](https://github.com/apache/flink-kubernetes-operator) extends the [Kubernetes](https://kubernetes.io/) API with the ability to manage and operate Flink Deployments. The operator features the following amongst others:
Basically allows users to manage Flink applications and their lifecycle through native k8s tooling
<img alt="Operator Overview" width="100%" src="https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/img/overview.svg">
- Deploy and monitor Flink Application and Session deployments
- Upgrade, suspend and delete deployments
- Full logging and metrics integration
- Flexible deployments and native integration with Kubernetes tooling
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #26664: Adding Flink on K8s Operator #9134
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #26664:
URL: https://github.com/apache/airflow/pull/26664#issuecomment-1264541472
I think this shoudl be rather part of a new "apache.flink" provider (depending on cncf.kubernetes) than part of cncf.kubernetes provider. I wonder what others think, but IMHO adding flink to cncf.kubernetes (even if it uses K8S to run) is somewhat counter-intuitive.
@jedcunningham @dstandish @ashb @ephraimbuddy - WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] chethanuk-plutoflume commented on a diff in pull request #26664: Adding Flink on K8s Operator #9134
Posted by GitBox <gi...@apache.org>.
chethanuk-plutoflume commented on code in PR #26664:
URL: https://github.com/apache/airflow/pull/26664#discussion_r981120236
##########
airflow/providers/cncf/kubernetes/sensors/spark_kubernetes.py:
##########
@@ -101,10 +101,11 @@ def poke(self, context: Context) -> bool:
response = self.hook.get_custom_object(
group=self.api_group,
version=self.api_version,
- plural="sparkapplications",
+ plural="flinkDeployments",
Review Comment:
Indeed, while trying to fix one test, messed up other. So pushed another change and resolving this conversation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] xinbinhuang closed pull request #26664: Adding Flink on K8s Operator #9134
Posted by GitBox <gi...@apache.org>.
xinbinhuang closed pull request #26664: Adding Flink on K8s Operator #9134
URL: https://github.com/apache/airflow/pull/26664
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] chethanuk-plutoflume commented on a diff in pull request #26664: Adding Flink on K8s Operator #9134
Posted by GitBox <gi...@apache.org>.
chethanuk-plutoflume commented on code in PR #26664:
URL: https://github.com/apache/airflow/pull/26664#discussion_r981120236
##########
airflow/providers/cncf/kubernetes/sensors/spark_kubernetes.py:
##########
@@ -101,10 +101,11 @@ def poke(self, context: Context) -> bool:
response = self.hook.get_custom_object(
group=self.api_group,
version=self.api_version,
- plural="sparkapplications",
+ plural="flinkDeployments",
Review Comment:
Indeed, while trying to fix one test, messed up other
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #26664: Adding Flink on K8s Operator #9134
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #26664:
URL: https://github.com/apache/airflow/pull/26664#issuecomment-1288303423
Conflicts need resolving.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] GezimSejdiu commented on a diff in pull request #26664: Adding Flink on K8s Operator #9134
Posted by GitBox <gi...@apache.org>.
GezimSejdiu commented on code in PR #26664:
URL: https://github.com/apache/airflow/pull/26664#discussion_r981024827
##########
airflow/providers/cncf/kubernetes/sensors/spark_kubernetes.py:
##########
@@ -101,10 +101,11 @@ def poke(self, context: Context) -> bool:
response = self.hook.get_custom_object(
group=self.api_group,
version=self.api_version,
- plural="sparkapplications",
+ plural="flinkDeployments",
Review Comment:
I assume this was done by mistake. It should still stay `sparkapplications`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] GezimSejdiu commented on pull request #26664: Adding Flink on K8s Operator #9134
Posted by GitBox <gi...@apache.org>.
GezimSejdiu commented on PR #26664:
URL: https://github.com/apache/airflow/pull/26664#issuecomment-1322105675
Hey @chethanuk-plutoflume ,
are you aware of any blockers that this has to be pushed? Do you need more tests from the community so that this also gets priority to be pushed for the next release? I will try to find some time later this week and test that out.
Just wanted to check it, as I find it quite useful for flink-deployments. I did some tests with argo-workflow and directly using kubectl, btu this is much, much better as it is natively integrated with Airflow. I know that this makes sense indeed when the only batch mode is used, but as a start, it would be great to push it and look for how to handle streaming mode.
Just my two cents.
Once again, thanks a lot for your hard work. b.t.w I also had in mind to push this to Airflow :) and was so happy to see it here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #26664: Adding Flink on K8s Operator #9134
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #26664:
URL: https://github.com/apache/airflow/pull/26664#issuecomment-1296565188
still :(
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] chethanuk-plutoflume commented on a diff in pull request #26664: Adding Flink on K8s Operator #9134
Posted by GitBox <gi...@apache.org>.
chethanuk-plutoflume commented on code in PR #26664:
URL: https://github.com/apache/airflow/pull/26664#discussion_r981396773
##########
airflow/providers/cncf/kubernetes/sensors/spark_kubernetes.py:
##########
@@ -101,10 +101,11 @@ def poke(self, context: Context) -> bool:
response = self.hook.get_custom_object(
group=self.api_group,
version=self.api_version,
- plural="sparkapplications",
+ plural="flinkDeployments",
Review Comment:
I think was old commit - closing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] xinbinhuang commented on pull request #26664: Adding Flink on K8s Operator #9134
Posted by GitBox <gi...@apache.org>.
xinbinhuang commented on PR #26664:
URL: https://github.com/apache/airflow/pull/26664#issuecomment-1362198639
fixed in #28512
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] chethanuk-plutoflume commented on a diff in pull request #26664: Adding Flink on K8s Operator #9134
Posted by GitBox <gi...@apache.org>.
chethanuk-plutoflume commented on code in PR #26664:
URL: https://github.com/apache/airflow/pull/26664#discussion_r981396773
##########
airflow/providers/cncf/kubernetes/sensors/spark_kubernetes.py:
##########
@@ -101,10 +101,11 @@ def poke(self, context: Context) -> bool:
response = self.hook.get_custom_object(
group=self.api_group,
version=self.api_version,
- plural="sparkapplications",
+ plural="flinkDeployments",
Review Comment:
I think was old commit - closing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org