You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/12/21 00:19:44 UTC

[GitHub] [airflow] xinbinhuang opened a new pull request, #28512: Port https://github.com/apache/airflow/pull/26664

xinbinhuang opened a new pull request, #28512:
URL: https://github.com/apache/airflow/pull/28512

   <!--
   Thank you for contributing! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   In case of an existing issue, reference it using one of the following:
   
   closes: #ISSUE
   related: #ISSUE
   
   How to write a good git commit message:
   http://chris.beams.io/posts/git-commit/
   -->
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] xinbinhuang commented on a diff in pull request #28512: [Fixed] "Adding Flink on K8s Operator"

Posted by GitBox <gi...@apache.org>.
xinbinhuang commented on code in PR #28512:
URL: https://github.com/apache/airflow/pull/28512#discussion_r1054803429


##########
airflow/providers/apache/flink/operators/flink_kubernetes.py:
##########
@@ -0,0 +1,143 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Sequence
+
+from kubernetes.client import CoreV1Api
+
+from airflow.compat.functools import cached_property
+from airflow.configuration import conf
+from airflow.models import BaseOperator
+from airflow.providers.cncf.kubernetes.hooks.kubernetes import KubernetesHook
+
+if TYPE_CHECKING:
+    from airflow.utils.context import Context
+
+
+class FlinkKubernetesOperator(BaseOperator):
+    """
+    Creates flinkDeployment object in kubernetes cluster:
+
+    .. seealso::
+        For more detail about Flink Deployment Object have a look at the reference:
+        https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/reference/#flinkdeployment
+
+    :param application_file: Defines Kubernetes 'custom_resource_definition' of 'flinkDeployment' as either a
+        path to a '.yaml' file, '.json' file, YAML string or JSON string.
+    :param namespace: kubernetes namespace to put flinkDeployment
+    :param kubernetes_conn_id: The :ref:`kubernetes connection id <howto/connection:kubernetes>`
+        for the to Kubernetes cluster.
+    :param api_group: kubernetes api group of flinkDeployment
+    :param api_version: kubernetes api version of flinkDeployment
+    :param in_cluster: run kubernetes client with in_cluster configuration.
+    :param cluster_context: context that points to kubernetes cluster.
+        Ignored when in_cluster is True. If None, current-context is used.
+    :param config_file: The path to the Kubernetes config file. (templated)
+        If not specified, default value is ``~/.kube/config``
+    """
+
+    template_fields: Sequence[str] = ("application_file", "namespace")
+    template_ext: Sequence[str] = (".yaml", ".yml", ".json")
+    ui_color = "#f4a460"
+
+    def __init__(
+        self,
+        *,
+        application_file: str,
+        namespace: str | None = None,
+        kubernetes_conn_id: str = "kubernetes_default",
+        api_group: str = "flink.apache.org",
+        api_version: str = "v1beta1",
+        in_cluster: bool | None = None,
+        cluster_context: str | None = None,
+        config_file: str | None = None,
+        plural: str = "flinkdeployments",
+        **kwargs,
+    ) -> None:
+        super().__init__(**kwargs)
+        self.application_file = application_file
+        self.namespace = namespace
+        self.kubernetes_conn_id = kubernetes_conn_id
+        self.api_group = api_group
+        self.api_version = api_version
+        self.plural = plural
+        self.in_cluster = in_cluster
+        self.cluster_context = cluster_context
+        self.config_file = config_file
+
+    @cached_property
+    def hook(self) -> KubernetesHook:
+        hook = KubernetesHook(
+            conn_id=self.kubernetes_conn_id,
+            in_cluster=self.in_cluster,
+            config_file=self.config_file,
+            cluster_context=self.cluster_context,
+        )
+        self._patch_deprecated_k8s_settings(hook)
+        return hook
+
+    def _patch_deprecated_k8s_settings(self, hook: KubernetesHook):
+        """
+        Here we read config from core Airflow config [kubernetes] section.
+        In a future release we will stop looking at this section and require users
+        to use Airflow connections to configure KPO.
+
+        When we find values there that we need to apply on the hook, we patch special
+        hook attributes here.
+        """
+        # default for enable_tcp_keepalive is True; patch if False
+        if conf.getboolean("kubernetes", "enable_tcp_keepalive") is False:
+            hook._deprecated_core_disable_tcp_keepalive = True  # noqa
+
+        # default verify_ssl is True; patch if False.
+        if conf.getboolean("kubernetes", "verify_ssl") is False:
+            hook._deprecated_core_disable_verify_ssl = True  # noqa
+
+        # default for in_cluster is True; patch if False and no KPO param.
+        conf_in_cluster = conf.getboolean("kubernetes", "in_cluster")
+        if self.in_cluster is None and conf_in_cluster is False:
+            hook._deprecated_core_in_cluster = conf_in_cluster  # noqa
+
+        # there's no default for cluster context; if we get something (and no KPO param) patch it.
+        conf_cluster_context = conf.get("kubernetes", "cluster_context", fallback=None)
+        if not self.cluster_context and conf_cluster_context:
+            hook._deprecated_core_cluster_context = conf_cluster_context  # noqa
+

Review Comment:
   Got it. I think all these internal patching are not needed. I've removed them all



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] kaxil merged pull request #28512: [Fixed] "Adding Flink on K8s Operator"

Posted by GitBox <gi...@apache.org>.
kaxil merged PR #28512:
URL: https://github.com/apache/airflow/pull/28512


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] xinbinhuang commented on pull request #28512: [Fixed] "Adding Flink on K8s Operator"

Posted by GitBox <gi...@apache.org>.
xinbinhuang commented on PR #28512:
URL: https://github.com/apache/airflow/pull/28512#issuecomment-1362189007

   @potiuk @jedcunningham CI green. Wanna take a look?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] xinbinhuang commented on pull request #28512: [Fixed] "Adding Flink on K8s Operator"

Posted by GitBox <gi...@apache.org>.
xinbinhuang commented on PR #28512:
URL: https://github.com/apache/airflow/pull/28512#issuecomment-1360628421

   @potiuk I think the CI has some issue reporting back the error ([CI log](https://github.com/apache/airflow/actions/runs/3745083722/jobs/6359128624))
   
   when I ran it locally, I got the traceback:
   
   ```Traceback (most recent call last):
     File "/Users/binh/.local/bin/breeze", line 8, in <module>
       sys.exit(main())
     File "/Users/binh/.local/pipx/venvs/apache-airflow-breeze/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
       return self.main(*args, **kwargs)
     File "/Users/binh/.local/pipx/venvs/apache-airflow-breeze/lib/python3.9/site-packages/rich_click/rich_group.py", line 21, in main
       rv = super().main(*args, standalone_mode=False, **kwargs)
     File "/Users/binh/.local/pipx/venvs/apache-airflow-breeze/lib/python3.9/site-packages/click/core.py", line 1055, in main
       rv = self.invoke(ctx)
     File "/Users/binh/.local/pipx/venvs/apache-airflow-breeze/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
       return _process_result(sub_ctx.command.invoke(sub_ctx))
     File "/Users/binh/.local/pipx/venvs/apache-airflow-breeze/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
       return _process_result(sub_ctx.command.invoke(sub_ctx))
     File "/Users/binh/.local/pipx/venvs/apache-airflow-breeze/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
       return ctx.invoke(self.callback, **ctx.params)
     File "/Users/binh/.local/pipx/venvs/apache-airflow-breeze/lib/python3.9/site-packages/click/core.py", line 760, in invoke
       return __callback(*args, **kwargs)
     File "/Users/binh/projects/airflow/dev/breeze/src/airflow_breeze/commands/ci_commands.py", line 253, in selective_check
       print(str(sc), file=sys.stderr)
     File "/Users/binh/projects/airflow/dev/breeze/src/airflow_breeze/utils/selective_checks.py", line 275, in __str__
       output.append(get_ga_output(field_name, getattr(self, field_name)))
     File "/Users/binh/.pyenv/versions/3.9.11/lib/python3.9/functools.py", line 993, in __get__
       val = self.func(instance)
     File "/Users/binh/projects/airflow/dev/breeze/src/airflow_breeze/utils/selective_checks.py", line 558, in test_types
       current_test_types = set(self._get_test_types_to_run())
     File "/Users/binh/projects/airflow/dev/breeze/src/airflow_breeze/utils/selective_checks.py", line 539, in _get_test_types_to_run
       affected_providers = find_all_providers_affected(changed_files=self._files)
     File "/Users/binh/projects/airflow/dev/breeze/src/airflow_breeze/utils/selective_checks.py", line 237, in find_all_providers_affected
       add_dependent_providers(all_providers, provider, dependencies)
     File "/Users/binh/projects/airflow/dev/breeze/src/airflow_breeze/utils/selective_checks.py", line 223, in add_dependent_providers
       for dep_name in dependencies[provider_to_check]["cross-providers-deps"]:
   KeyError: 'apache.flink'
   ```
   
   Any clues if we can fix the CI to report back the traceback properly?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #28512: [Fixed] "Adding Flink on K8s Operator"

Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #28512:
URL: https://github.com/apache/airflow/pull/28512#issuecomment-1360653166

   Good one. Unforeseen interaction of stderr redirection to GITHUB_OUTPUT.
   
   Fix in https://github.com/apache/airflow/pull/28514 (explanation in the commit descripton - and you can see the changes in the PR :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] xinbinhuang commented on a diff in pull request #28512: [Fixed] "Adding Flink on K8s Operator"

Posted by GitBox <gi...@apache.org>.
xinbinhuang commented on code in PR #28512:
URL: https://github.com/apache/airflow/pull/28512#discussion_r1054695655


##########
airflow/providers/apache/flink/operators/flink_kubernetes.py:
##########
@@ -0,0 +1,143 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Sequence
+
+from kubernetes.client import CoreV1Api
+
+from airflow.compat.functools import cached_property
+from airflow.configuration import conf
+from airflow.models import BaseOperator
+from airflow.providers.cncf.kubernetes.hooks.kubernetes import KubernetesHook
+
+if TYPE_CHECKING:
+    from airflow.utils.context import Context
+
+
+class FlinkKubernetesOperator(BaseOperator):
+    """
+    Creates flinkDeployment object in kubernetes cluster:
+
+    .. seealso::
+        For more detail about Flink Deployment Object have a look at the reference:
+        https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/reference/#flinkdeployment
+
+    :param application_file: Defines Kubernetes 'custom_resource_definition' of 'flinkDeployment' as either a
+        path to a '.yaml' file, '.json' file, YAML string or JSON string.
+    :param namespace: kubernetes namespace to put flinkDeployment
+    :param kubernetes_conn_id: The :ref:`kubernetes connection id <howto/connection:kubernetes>`
+        for the to Kubernetes cluster.
+    :param api_group: kubernetes api group of flinkDeployment
+    :param api_version: kubernetes api version of flinkDeployment
+    :param in_cluster: run kubernetes client with in_cluster configuration.
+    :param cluster_context: context that points to kubernetes cluster.
+        Ignored when in_cluster is True. If None, current-context is used.
+    :param config_file: The path to the Kubernetes config file. (templated)
+        If not specified, default value is ``~/.kube/config``
+    """
+
+    template_fields: Sequence[str] = ("application_file", "namespace")
+    template_ext: Sequence[str] = (".yaml", ".yml", ".json")
+    ui_color = "#f4a460"
+
+    def __init__(
+        self,
+        *,
+        application_file: str,
+        namespace: str | None = None,
+        kubernetes_conn_id: str = "kubernetes_default",
+        api_group: str = "flink.apache.org",
+        api_version: str = "v1beta1",
+        in_cluster: bool | None = None,
+        cluster_context: str | None = None,
+        config_file: str | None = None,
+        plural: str = "flinkdeployments",
+        **kwargs,
+    ) -> None:
+        super().__init__(**kwargs)
+        self.application_file = application_file
+        self.namespace = namespace
+        self.kubernetes_conn_id = kubernetes_conn_id
+        self.api_group = api_group
+        self.api_version = api_version
+        self.plural = plural
+        self.in_cluster = in_cluster
+        self.cluster_context = cluster_context
+        self.config_file = config_file
+
+    @cached_property
+    def hook(self) -> KubernetesHook:
+        hook = KubernetesHook(
+            conn_id=self.kubernetes_conn_id,
+            in_cluster=self.in_cluster,
+            config_file=self.config_file,
+            cluster_context=self.cluster_context,
+        )
+        self._patch_deprecated_k8s_settings(hook)
+        return hook
+
+    def _patch_deprecated_k8s_settings(self, hook: KubernetesHook):
+        """
+        Here we read config from core Airflow config [kubernetes] section.
+        In a future release we will stop looking at this section and require users
+        to use Airflow connections to configure KPO.
+
+        When we find values there that we need to apply on the hook, we patch special
+        hook attributes here.
+        """
+        # default for enable_tcp_keepalive is True; patch if False
+        if conf.getboolean("kubernetes", "enable_tcp_keepalive") is False:
+            hook._deprecated_core_disable_tcp_keepalive = True  # noqa
+
+        # default verify_ssl is True; patch if False.
+        if conf.getboolean("kubernetes", "verify_ssl") is False:
+            hook._deprecated_core_disable_verify_ssl = True  # noqa
+
+        # default for in_cluster is True; patch if False and no KPO param.
+        conf_in_cluster = conf.getboolean("kubernetes", "in_cluster")
+        if self.in_cluster is None and conf_in_cluster is False:
+            hook._deprecated_core_in_cluster = conf_in_cluster  # noqa
+
+        # there's no default for cluster context; if we get something (and no KPO param) patch it.
+        conf_cluster_context = conf.get("kubernetes", "cluster_context", fallback=None)
+        if not self.cluster_context and conf_cluster_context:
+            hook._deprecated_core_cluster_context = conf_cluster_context  # noqa
+

Review Comment:
   Yeah I saw the function is being called, but I can't find the patched attributes being used anywhere (i.e. `hook._deprecated_core_cluster_context`)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] chethanuk-plutoflume commented on a diff in pull request #28512: [Fixed] "Adding Flink on K8s Operator"

Posted by GitBox <gi...@apache.org>.
chethanuk-plutoflume commented on code in PR #28512:
URL: https://github.com/apache/airflow/pull/28512#discussion_r1054099178


##########
airflow/providers/apache/flink/operators/flink_kubernetes.py:
##########
@@ -0,0 +1,143 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Sequence
+
+from kubernetes.client import CoreV1Api
+
+from airflow.compat.functools import cached_property
+from airflow.configuration import conf
+from airflow.models import BaseOperator
+from airflow.providers.cncf.kubernetes.hooks.kubernetes import KubernetesHook
+
+if TYPE_CHECKING:
+    from airflow.utils.context import Context
+
+
+class FlinkKubernetesOperator(BaseOperator):
+    """
+    Creates flinkDeployment object in kubernetes cluster:
+
+    .. seealso::
+        For more detail about Flink Deployment Object have a look at the reference:
+        https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/reference/#flinkdeployment
+
+    :param application_file: Defines Kubernetes 'custom_resource_definition' of 'flinkDeployment' as either a
+        path to a '.yaml' file, '.json' file, YAML string or JSON string.
+    :param namespace: kubernetes namespace to put flinkDeployment
+    :param kubernetes_conn_id: The :ref:`kubernetes connection id <howto/connection:kubernetes>`
+        for the to Kubernetes cluster.
+    :param api_group: kubernetes api group of flinkDeployment
+    :param api_version: kubernetes api version of flinkDeployment
+    :param in_cluster: run kubernetes client with in_cluster configuration.
+    :param cluster_context: context that points to kubernetes cluster.
+        Ignored when in_cluster is True. If None, current-context is used.
+    :param config_file: The path to the Kubernetes config file. (templated)
+        If not specified, default value is ``~/.kube/config``
+    """
+
+    template_fields: Sequence[str] = ("application_file", "namespace")
+    template_ext: Sequence[str] = (".yaml", ".yml", ".json")
+    ui_color = "#f4a460"
+
+    def __init__(
+        self,
+        *,
+        application_file: str,
+        namespace: str | None = None,
+        kubernetes_conn_id: str = "kubernetes_default",
+        api_group: str = "flink.apache.org",
+        api_version: str = "v1beta1",
+        in_cluster: bool | None = None,
+        cluster_context: str | None = None,
+        config_file: str | None = None,
+        plural: str = "flinkdeployments",
+        **kwargs,
+    ) -> None:
+        super().__init__(**kwargs)
+        self.application_file = application_file
+        self.namespace = namespace
+        self.kubernetes_conn_id = kubernetes_conn_id
+        self.api_group = api_group
+        self.api_version = api_version
+        self.plural = plural
+        self.in_cluster = in_cluster
+        self.cluster_context = cluster_context
+        self.config_file = config_file
+
+    @cached_property
+    def hook(self) -> KubernetesHook:
+        hook = KubernetesHook(
+            conn_id=self.kubernetes_conn_id,
+            in_cluster=self.in_cluster,
+            config_file=self.config_file,
+            cluster_context=self.cluster_context,
+        )
+        self._patch_deprecated_k8s_settings(hook)
+        return hook
+
+    def _patch_deprecated_k8s_settings(self, hook: KubernetesHook):
+        """
+        Here we read config from core Airflow config [kubernetes] section.
+        In a future release we will stop looking at this section and require users
+        to use Airflow connections to configure KPO.
+
+        When we find values there that we need to apply on the hook, we patch special
+        hook attributes here.
+        """
+        # default for enable_tcp_keepalive is True; patch if False
+        if conf.getboolean("kubernetes", "enable_tcp_keepalive") is False:
+            hook._deprecated_core_disable_tcp_keepalive = True  # noqa
+
+        # default verify_ssl is True; patch if False.
+        if conf.getboolean("kubernetes", "verify_ssl") is False:
+            hook._deprecated_core_disable_verify_ssl = True  # noqa
+
+        # default for in_cluster is True; patch if False and no KPO param.
+        conf_in_cluster = conf.getboolean("kubernetes", "in_cluster")
+        if self.in_cluster is None and conf_in_cluster is False:
+            hook._deprecated_core_in_cluster = conf_in_cluster  # noqa
+
+        # there's no default for cluster context; if we get something (and no KPO param) patch it.
+        conf_cluster_context = conf.get("kubernetes", "cluster_context", fallback=None)
+        if not self.cluster_context and conf_cluster_context:
+            hook._deprecated_core_cluster_context = conf_cluster_context  # noqa
+

Review Comment:
   https://github.com/apache/airflow/pull/28512/files#diff-5de054c1bea5c3b53b30769de1d806cd1d6075cc13732d984fbea70ab1501d6bR96
   We are patching the hook



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] chethanuk-plutoflume commented on a diff in pull request #28512: [Fixed] "Adding Flink on K8s Operator"

Posted by GitBox <gi...@apache.org>.
chethanuk-plutoflume commented on code in PR #28512:
URL: https://github.com/apache/airflow/pull/28512#discussion_r1054709492


##########
airflow/providers/apache/flink/operators/flink_kubernetes.py:
##########
@@ -0,0 +1,143 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Sequence
+
+from kubernetes.client import CoreV1Api
+
+from airflow.compat.functools import cached_property
+from airflow.configuration import conf
+from airflow.models import BaseOperator
+from airflow.providers.cncf.kubernetes.hooks.kubernetes import KubernetesHook
+
+if TYPE_CHECKING:
+    from airflow.utils.context import Context
+
+
+class FlinkKubernetesOperator(BaseOperator):
+    """
+    Creates flinkDeployment object in kubernetes cluster:
+
+    .. seealso::
+        For more detail about Flink Deployment Object have a look at the reference:
+        https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/reference/#flinkdeployment
+
+    :param application_file: Defines Kubernetes 'custom_resource_definition' of 'flinkDeployment' as either a
+        path to a '.yaml' file, '.json' file, YAML string or JSON string.
+    :param namespace: kubernetes namespace to put flinkDeployment
+    :param kubernetes_conn_id: The :ref:`kubernetes connection id <howto/connection:kubernetes>`
+        for the to Kubernetes cluster.
+    :param api_group: kubernetes api group of flinkDeployment
+    :param api_version: kubernetes api version of flinkDeployment
+    :param in_cluster: run kubernetes client with in_cluster configuration.
+    :param cluster_context: context that points to kubernetes cluster.
+        Ignored when in_cluster is True. If None, current-context is used.
+    :param config_file: The path to the Kubernetes config file. (templated)
+        If not specified, default value is ``~/.kube/config``
+    """
+
+    template_fields: Sequence[str] = ("application_file", "namespace")
+    template_ext: Sequence[str] = (".yaml", ".yml", ".json")
+    ui_color = "#f4a460"
+
+    def __init__(
+        self,
+        *,
+        application_file: str,
+        namespace: str | None = None,
+        kubernetes_conn_id: str = "kubernetes_default",
+        api_group: str = "flink.apache.org",
+        api_version: str = "v1beta1",
+        in_cluster: bool | None = None,
+        cluster_context: str | None = None,
+        config_file: str | None = None,
+        plural: str = "flinkdeployments",
+        **kwargs,
+    ) -> None:
+        super().__init__(**kwargs)
+        self.application_file = application_file
+        self.namespace = namespace
+        self.kubernetes_conn_id = kubernetes_conn_id
+        self.api_group = api_group
+        self.api_version = api_version
+        self.plural = plural
+        self.in_cluster = in_cluster
+        self.cluster_context = cluster_context
+        self.config_file = config_file
+
+    @cached_property
+    def hook(self) -> KubernetesHook:
+        hook = KubernetesHook(
+            conn_id=self.kubernetes_conn_id,
+            in_cluster=self.in_cluster,
+            config_file=self.config_file,
+            cluster_context=self.cluster_context,
+        )
+        self._patch_deprecated_k8s_settings(hook)
+        return hook
+
+    def _patch_deprecated_k8s_settings(self, hook: KubernetesHook):
+        """
+        Here we read config from core Airflow config [kubernetes] section.
+        In a future release we will stop looking at this section and require users
+        to use Airflow connections to configure KPO.
+
+        When we find values there that we need to apply on the hook, we patch special
+        hook attributes here.
+        """
+        # default for enable_tcp_keepalive is True; patch if False
+        if conf.getboolean("kubernetes", "enable_tcp_keepalive") is False:
+            hook._deprecated_core_disable_tcp_keepalive = True  # noqa
+
+        # default verify_ssl is True; patch if False.
+        if conf.getboolean("kubernetes", "verify_ssl") is False:
+            hook._deprecated_core_disable_verify_ssl = True  # noqa
+
+        # default for in_cluster is True; patch if False and no KPO param.
+        conf_in_cluster = conf.getboolean("kubernetes", "in_cluster")
+        if self.in_cluster is None and conf_in_cluster is False:
+            hook._deprecated_core_in_cluster = conf_in_cluster  # noqa
+
+        # there's no default for cluster context; if we get something (and no KPO param) patch it.
+        conf_cluster_context = conf.get("kubernetes", "cluster_context", fallback=None)
+        if not self.cluster_context and conf_cluster_context:
+            hook._deprecated_core_cluster_context = conf_cluster_context  # noqa
+

Review Comment:
   I think I was using _deprecated_core_cluster_context and old API in one of the internal tests, while I didn't open source. We can safely remove 119 and 120 in this case



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] xinbinhuang commented on a diff in pull request #28512: [Fixed] "Adding Flink on K8s Operator"

Posted by GitBox <gi...@apache.org>.
xinbinhuang commented on code in PR #28512:
URL: https://github.com/apache/airflow/pull/28512#discussion_r1053911449


##########
airflow/providers/apache/flink/operators/flink_kubernetes.py:
##########
@@ -0,0 +1,143 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Sequence
+
+from kubernetes.client import CoreV1Api
+
+from airflow.compat.functools import cached_property
+from airflow.configuration import conf
+from airflow.models import BaseOperator
+from airflow.providers.cncf.kubernetes.hooks.kubernetes import KubernetesHook
+
+if TYPE_CHECKING:
+    from airflow.utils.context import Context
+
+
+class FlinkKubernetesOperator(BaseOperator):
+    """
+    Creates flinkDeployment object in kubernetes cluster:
+
+    .. seealso::
+        For more detail about Flink Deployment Object have a look at the reference:
+        https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/reference/#flinkdeployment
+
+    :param application_file: Defines Kubernetes 'custom_resource_definition' of 'flinkDeployment' as either a
+        path to a '.yaml' file, '.json' file, YAML string or JSON string.
+    :param namespace: kubernetes namespace to put flinkDeployment
+    :param kubernetes_conn_id: The :ref:`kubernetes connection id <howto/connection:kubernetes>`
+        for the to Kubernetes cluster.
+    :param api_group: kubernetes api group of flinkDeployment
+    :param api_version: kubernetes api version of flinkDeployment
+    :param in_cluster: run kubernetes client with in_cluster configuration.
+    :param cluster_context: context that points to kubernetes cluster.
+        Ignored when in_cluster is True. If None, current-context is used.
+    :param config_file: The path to the Kubernetes config file. (templated)
+        If not specified, default value is ``~/.kube/config``
+    """
+
+    template_fields: Sequence[str] = ("application_file", "namespace")
+    template_ext: Sequence[str] = (".yaml", ".yml", ".json")
+    ui_color = "#f4a460"
+
+    def __init__(
+        self,
+        *,
+        application_file: str,
+        namespace: str | None = None,
+        kubernetes_conn_id: str = "kubernetes_default",
+        api_group: str = "flink.apache.org",
+        api_version: str = "v1beta1",
+        in_cluster: bool | None = None,
+        cluster_context: str | None = None,
+        config_file: str | None = None,
+        plural: str = "flinkdeployments",
+        **kwargs,
+    ) -> None:
+        super().__init__(**kwargs)
+        self.application_file = application_file
+        self.namespace = namespace
+        self.kubernetes_conn_id = kubernetes_conn_id
+        self.api_group = api_group
+        self.api_version = api_version
+        self.plural = plural
+        self.in_cluster = in_cluster
+        self.cluster_context = cluster_context
+        self.config_file = config_file
+
+    @cached_property
+    def hook(self) -> KubernetesHook:
+        hook = KubernetesHook(
+            conn_id=self.kubernetes_conn_id,
+            in_cluster=self.in_cluster,
+            config_file=self.config_file,
+            cluster_context=self.cluster_context,
+        )
+        self._patch_deprecated_k8s_settings(hook)
+        return hook
+
+    def _patch_deprecated_k8s_settings(self, hook: KubernetesHook):
+        """
+        Here we read config from core Airflow config [kubernetes] section.
+        In a future release we will stop looking at this section and require users
+        to use Airflow connections to configure KPO.
+
+        When we find values there that we need to apply on the hook, we patch special
+        hook attributes here.
+        """
+        # default for enable_tcp_keepalive is True; patch if False
+        if conf.getboolean("kubernetes", "enable_tcp_keepalive") is False:
+            hook._deprecated_core_disable_tcp_keepalive = True  # noqa
+
+        # default verify_ssl is True; patch if False.
+        if conf.getboolean("kubernetes", "verify_ssl") is False:
+            hook._deprecated_core_disable_verify_ssl = True  # noqa
+
+        # default for in_cluster is True; patch if False and no KPO param.
+        conf_in_cluster = conf.getboolean("kubernetes", "in_cluster")
+        if self.in_cluster is None and conf_in_cluster is False:
+            hook._deprecated_core_in_cluster = conf_in_cluster  # noqa
+
+        # there's no default for cluster context; if we get something (and no KPO param) patch it.
+        conf_cluster_context = conf.get("kubernetes", "cluster_context", fallback=None)
+        if not self.cluster_context and conf_cluster_context:
+            hook._deprecated_core_cluster_context = conf_cluster_context  # noqa
+

Review Comment:
   @chethanuk-plutoflume Are these actually needed ? Not see them being used/referenced anywhere?



##########
airflow/providers/apache/flink/operators/flink_kubernetes.py:
##########
@@ -0,0 +1,143 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Sequence
+
+from kubernetes.client import CoreV1Api
+
+from airflow.compat.functools import cached_property
+from airflow.configuration import conf
+from airflow.models import BaseOperator
+from airflow.providers.cncf.kubernetes.hooks.kubernetes import KubernetesHook
+
+if TYPE_CHECKING:
+    from airflow.utils.context import Context
+
+
+class FlinkKubernetesOperator(BaseOperator):
+    """
+    Creates flinkDeployment object in kubernetes cluster:
+
+    .. seealso::
+        For more detail about Flink Deployment Object have a look at the reference:
+        https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/reference/#flinkdeployment
+
+    :param application_file: Defines Kubernetes 'custom_resource_definition' of 'flinkDeployment' as either a
+        path to a '.yaml' file, '.json' file, YAML string or JSON string.
+    :param namespace: kubernetes namespace to put flinkDeployment
+    :param kubernetes_conn_id: The :ref:`kubernetes connection id <howto/connection:kubernetes>`
+        for the to Kubernetes cluster.
+    :param api_group: kubernetes api group of flinkDeployment
+    :param api_version: kubernetes api version of flinkDeployment
+    :param in_cluster: run kubernetes client with in_cluster configuration.
+    :param cluster_context: context that points to kubernetes cluster.
+        Ignored when in_cluster is True. If None, current-context is used.
+    :param config_file: The path to the Kubernetes config file. (templated)
+        If not specified, default value is ``~/.kube/config``
+    """
+
+    template_fields: Sequence[str] = ("application_file", "namespace")
+    template_ext: Sequence[str] = (".yaml", ".yml", ".json")
+    ui_color = "#f4a460"
+
+    def __init__(
+        self,
+        *,
+        application_file: str,
+        namespace: str | None = None,
+        kubernetes_conn_id: str = "kubernetes_default",
+        api_group: str = "flink.apache.org",
+        api_version: str = "v1beta1",
+        in_cluster: bool | None = None,
+        cluster_context: str | None = None,
+        config_file: str | None = None,
+        plural: str = "flinkdeployments",
+        **kwargs,
+    ) -> None:
+        super().__init__(**kwargs)
+        self.application_file = application_file
+        self.namespace = namespace
+        self.kubernetes_conn_id = kubernetes_conn_id
+        self.api_group = api_group
+        self.api_version = api_version
+        self.plural = plural
+        self.in_cluster = in_cluster
+        self.cluster_context = cluster_context
+        self.config_file = config_file
+
+    @cached_property
+    def hook(self) -> KubernetesHook:
+        hook = KubernetesHook(
+            conn_id=self.kubernetes_conn_id,
+            in_cluster=self.in_cluster,
+            config_file=self.config_file,
+            cluster_context=self.cluster_context,
+        )
+        self._patch_deprecated_k8s_settings(hook)
+        return hook
+
+    def _patch_deprecated_k8s_settings(self, hook: KubernetesHook):
+        """
+        Here we read config from core Airflow config [kubernetes] section.
+        In a future release we will stop looking at this section and require users
+        to use Airflow connections to configure KPO.
+
+        When we find values there that we need to apply on the hook, we patch special
+        hook attributes here.
+        """
+        # default for enable_tcp_keepalive is True; patch if False
+        if conf.getboolean("kubernetes", "enable_tcp_keepalive") is False:
+            hook._deprecated_core_disable_tcp_keepalive = True  # noqa
+
+        # default verify_ssl is True; patch if False.
+        if conf.getboolean("kubernetes", "verify_ssl") is False:
+            hook._deprecated_core_disable_verify_ssl = True  # noqa
+
+        # default for in_cluster is True; patch if False and no KPO param.
+        conf_in_cluster = conf.getboolean("kubernetes", "in_cluster")
+        if self.in_cluster is None and conf_in_cluster is False:
+            hook._deprecated_core_in_cluster = conf_in_cluster  # noqa
+
+        # there's no default for cluster context; if we get something (and no KPO param) patch it.
+        conf_cluster_context = conf.get("kubernetes", "cluster_context", fallback=None)
+        if not self.cluster_context and conf_cluster_context:
+            hook._deprecated_core_cluster_context = conf_cluster_context  # noqa
+

Review Comment:
   @chethanuk-plutoflume Are these actually needed ? Not see them being used/referenced anywhere.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] chethanuk-plutoflume commented on pull request #28512: [Fixed] "Adding Flink on K8s Operator"

Posted by GitBox <gi...@apache.org>.
chethanuk-plutoflume commented on PR #28512:
URL: https://github.com/apache/airflow/pull/28512#issuecomment-1360995776

   Awesome thanks for fixing breeze cross-providers-deps @xinbinhuang 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org