You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "akashsriv07 (via GitHub)" <gi...@apache.org> on 2024/02/07 14:00:25 UTC
[PR] Adding airflow operator to create and monitor Spark Apps/App Env on DPGDC clusters [airflow]
akashsriv07 opened a new pull request, #37223:
URL: https://github.com/apache/airflow/pull/37223
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
Adding airflow operator to create and monitor Spark Apps/App Env on DPGDC clusters
---
GDC extends Google Cloud’s infrastructure and services to customer edge locations and data centers. We intend to bring [Dataproc](https://cloud.google.com/dataproc?hl=en) as a managed service offering to GDC.
As part of this, we need to integrate Airflow with Dataproc on GDC API resources to trigger Apache Spark workloads. The integration needs to cover two distinct API paths: local execution through the KRM API and remote execution through an equivalent One Platform API. We're targeting the former one in this PR i.e. Airflow operator leveraging KRM APIs.
This PR contains two feature implementations :
1. An Airflow operator can submit a SparkApplication via the KRM API, enabling customers to use Airflow to orchestrate Dataproc on GDC workloads locally.
2. An Airflow operator can create an ApplicationEnvironment via both the KRM API.
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding airflow operator to submit and monitor Spark Apps/App Env on DPGDC clusters [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #37223:
URL: https://github.com/apache/airflow/pull/37223#issuecomment-1995201673
> Hey, Michal from Cloud Composer team here.
+1 for this code living in a separate, GDC provider - GDC is based on slightly different infra than GCP, and the operators won't necessarily work in the same way (some might be very similar, some very different).
Given the above statemant and the fact that we are going to seprate google to several providers we should treat this PR like adding a new provider which means it nreds to go through the approval cycle like all other providers.
https://github.com/apache/airflow/blob/main/PROVIDERS.rst#accepting-new-community-providers
> If there's shared code, I propose having a util module/provider between these two.
That will not work and this is one of the reason why seperating Google provider is not an easy task. You can't have utill shared between two providers. Utill belongs to one provider. Maybe Google can intoduce google.common provider but that would require separate discussion.
I really advise against adding more completely to Google provider it will make things harder for the decouple project
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding airflow operator to submit and monitor Spark Apps/App Env on DPGDC clusters [airflow]
Posted by "molcay (via GitHub)" <gi...@apache.org>.
molcay commented on PR #37223:
URL: https://github.com/apache/airflow/pull/37223#issuecomment-1961153709
Hi @akashsriv07,
I will propose a different approach for the folder/module structure. Since GDC might have different APIs, or different inner working mechanisms, it can be good to separate the GDC related things (like `hooks`, `operators`, `sensors`) from the `cloud` one. In the future, if you want to add more things (like `hooks`, `operators`, `sensors`), it can be easily extended in a more structured way. Also, you will have access for the existing Google Cloud `operators`, `sensors`, etc.
Hence, I am suggesting creating a new module inside a `google` provider (at the same level with the `cloud` module). Something like the following:
```
airflow/providers/google
├── ads
├── cloud
├── common
├── firebase
├── gdc # <-- this is the new one
├── leveldb
├── marketing_platform
├── suite
├── CHANGELOG.rst
├── go_module_utils.py
├── __init__.py
└── provider.yaml
```
What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding airflow operator to submit and monitor Spark Apps/App Env on DPGDC clusters [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37223:
URL: https://github.com/apache/airflow/pull/37223#issuecomment-2322788894
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding airflow operator to submit and monitor Spark Apps/App Env on DPGDC clusters [airflow]
Posted by "akashsriv07 (via GitHub)" <gi...@apache.org>.
akashsriv07 commented on code in PR #37223:
URL: https://github.com/apache/airflow/pull/37223#discussion_r1499362176
##########
airflow/providers/google/cloud/sensors/dataprocgdc.py:
##########
@@ -0,0 +1,127 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from functools import cached_property
+from typing import TYPE_CHECKING, Sequence
+
+from kubernetes import client
+
+from airflow.exceptions import AirflowException, AirflowSkipException
+from airflow.providers.cncf.kubernetes.hooks.kubernetes import KubernetesHook
+from airflow.sensors.base import BaseSensorOperator
+
+if TYPE_CHECKING:
+ from airflow.utils.context import Context
+
+
+class DataprocGDCKrmSensor(BaseSensorOperator):
Review Comment:
I thought around that, however the response states in GDC is different from the ones in spark Kubernetes application mentioned by you.
##########
tests/system/providers/google/cloud/dataprocgdc/example_dataprocgdc.py:
##########
@@ -0,0 +1,77 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Example Airflow DAG that shows how to create and monitor a Spark Application or App Environment
+ in a DPGDC cluster
+"""
+
+from __future__ import annotations
+
+from datetime import datetime
+
+from airflow import DAG
+from airflow.providers.google.cloud.operators.dataprocgdc import (
+ DataprocGdcCreateAppEnvironmentKrmOperator,
+ DataprocGDCSubmitSparkJobKrmOperator,
+)
+from airflow.providers.google.cloud.sensors.dataprocgdc import DataprocGDCKrmSensor
+
+DAG_ID = "example_dag_dpgdc-krm"
+
+with DAG(
Review Comment:
Yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding airflow operator to submit and monitor Spark Apps/App Env on DPGDC clusters [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37223:
URL: https://github.com/apache/airflow/pull/37223#issuecomment-2083907076
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding airflow operator to submit and monitor Spark Apps/App Env on DPGDC clusters [airflow]
Posted by "akashsriv07 (via GitHub)" <gi...@apache.org>.
akashsriv07 commented on PR #37223:
URL: https://github.com/apache/airflow/pull/37223#issuecomment-1963317023
> Hi @akashsriv07,
>
> I will propose a different approach for the folder/module structure. Since GDC might have different APIs, or different inner working mechanisms, it can be good to separate the GDC related things (like `hooks`, `operators`, `sensors`) from the `cloud` one. In the future, if you want to add more things (like `hooks`, `operators`, `sensors`), it can be easily extended in a more structured way. Also, you will have access for the existing Google Cloud `operators`, `sensors`, etc.
>
> Hence, I am suggesting creating a new module inside a `google` provider (at the same level with the `cloud` module). Something like the following:
>
> ```
> airflow/providers/google
> ├── ads
> ├── cloud
> ├── common
> ├── firebase
> ├── gdc # <-- this is the new one
> ├── leveldb
> ├── marketing_platform
> ├── suite
> ├── CHANGELOG.rst
> ├── go_module_utils.py
> ├── __init__.py
> └── provider.yaml
> ```
>
> What do you think?
Hey @molcay , GDC is part of Google Cloud only. Shouldn't it be a part of it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding airflow operator to submit and monitor Spark Apps/App Env on DPGDC clusters [airflow]
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on PR #37223:
URL: https://github.com/apache/airflow/pull/37223#issuecomment-1939739020
> Hey Hussein, Dataproc on GDC is yet to go GA and this is one of the critical features to add as part of the same. Hence we don't have a public doc on the same. You can take a look at this blog post though: https://cloud.google.com/blog/products/infrastructure-modernization/google-distributed-cloud-new-ai-and-data-services
It will be a bit complicated to review this PR without a doc, I will try to review the syntax and check if our conventions are respected. (cc: @eladkal could you take a look?)
> The CRD for DPGDC is completely different than the sparkoperator.k8s.io/v1beta2, which stops us from leveraging the same operator/sensor. using KRM APIs is one of the ways to interact with a DPGDC cluster. In future, we plan to add the operator similar to dataproc.py(ex: DataprocSubmitSparkJobOperator) which will leverage Google's internal One Platform API mechanism.
No problem with that, my goal was to reduce code duplication if possible, but since they are different, we can keep building your operator from scratch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding airflow operator to submit and monitor Spark Apps/App Env on DPGDC clusters [airflow]
Posted by "MaksYermak (via GitHub)" <gi...@apache.org>.
MaksYermak commented on code in PR #37223:
URL: https://github.com/apache/airflow/pull/37223#discussion_r1499352165
##########
tests/system/providers/google/cloud/dataprocgdc/example_dataprocgdc.py:
##########
@@ -0,0 +1,77 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Example Airflow DAG that shows how to create and monitor a Spark Application or App Environment
+ in a DPGDC cluster
+"""
+
+from __future__ import annotations
+
+from datetime import datetime
+
+from airflow import DAG
+from airflow.providers.google.cloud.operators.dataprocgdc import (
+ DataprocGdcCreateAppEnvironmentKrmOperator,
+ DataprocGDCSubmitSparkJobKrmOperator,
+)
+from airflow.providers.google.cloud.sensors.dataprocgdc import DataprocGDCKrmSensor
+
+DAG_ID = "example_dag_dpgdc-krm"
+
+with DAG(
+ DAG_ID, schedule="@once", catchup=False, start_date=datetime(2024, 2, 5), tags=["example", "dataprocgdc"]
+) as dag:
+ submitSparkJobOperator = DataprocGDCSubmitSparkJobKrmOperator(
+ task_id="example-dataprocgdc-submitspark-operator",
+ trigger_rule="all_success",
+ depends_on_past=False,
+ retries=1,
+ application_file="sparkpi.yaml",
+ namespace="default",
+ kubernetes_conn_id="myk8s",
Review Comment:
Does this operator work with GKE cluster configuration?
##########
airflow/providers/google/cloud/sensors/dataprocgdc.py:
##########
@@ -0,0 +1,127 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from functools import cached_property
+from typing import TYPE_CHECKING, Sequence
+
+from kubernetes import client
+
+from airflow.exceptions import AirflowException, AirflowSkipException
+from airflow.providers.cncf.kubernetes.hooks.kubernetes import KubernetesHook
+from airflow.sensors.base import BaseSensorOperator
+
+if TYPE_CHECKING:
+ from airflow.utils.context import Context
+
+
+class DataprocGDCKrmSensor(BaseSensorOperator):
Review Comment:
Is it possible to use this `SparkKubernetesSensor` (https://github.com/apache/airflow/blob/main/airflow/providers/cncf/kubernetes/sensors/spark_kubernetes.py#L33) for this purpose or maybe you can adopt this sensor for GDC cluster?
##########
tests/system/providers/google/cloud/dataprocgdc/example_dataprocgdc.py:
##########
@@ -0,0 +1,77 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Example Airflow DAG that shows how to create and monitor a Spark Application or App Environment
+ in a DPGDC cluster
+"""
+
+from __future__ import annotations
+
+from datetime import datetime
+
+from airflow import DAG
+from airflow.providers.google.cloud.operators.dataprocgdc import (
+ DataprocGdcCreateAppEnvironmentKrmOperator,
+ DataprocGDCSubmitSparkJobKrmOperator,
+)
+from airflow.providers.google.cloud.sensors.dataprocgdc import DataprocGDCKrmSensor
+
+DAG_ID = "example_dag_dpgdc-krm"
+
+with DAG(
Review Comment:
Did you check this DAG on your local env?
##########
airflow/providers/google/cloud/operators/dataprocgdc.py:
##########
@@ -0,0 +1,272 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+"""This module contains Dataproc GDC operators using KRM APIs."""
+
+from __future__ import annotations
+
+import datetime
+from functools import cached_property
+from typing import TYPE_CHECKING, Sequence
+
+from kubernetes.client import ApiException, CoreV1Api
+from kubernetes.watch import Watch
+
+from airflow.exceptions import AirflowException
+from airflow.models import BaseOperator
+from airflow.providers.cncf.kubernetes.hooks.kubernetes import KubernetesHook, _load_body_to_dict
+
+if TYPE_CHECKING:
+ from kubernetes.client.models import CoreV1EventList
+
+ from airflow.utils.context import Context
+
+
+class DataprocGDCSubmitSparkJobKrmOperator(BaseOperator):
+ """
+ Creates sparkApplication object in dpgdc cluster using KRM APIs.
+
+ :param application_file: Defines gdc 'custom_resource_definition' of
+ 'sparkApplication'
+ path to a '.yaml' file, '.json' file, YAML string or python dictionary.
+ :param namespace: kubernetes namespace to put sparkApplication
+ :param kubernetes_conn_id: The :ref:`kubernetes connection id
+ <howto/connection:kubernetes>` for the GDC cluster.
+ :param api_group: DPGDC api group
+ :param api_version: DPGDC api version
+ :param watch: whether to watch the job status and logs or not
+ """
+
+ template_fields: Sequence[str] = "application_file"
+ template_ext: Sequence[str] = (".yaml", ".yml", ".json")
+ ui_color = "#f4a460"
+
+ def __init__(
+ self,
+ *,
+ application_file: str | dict,
+ kubernetes_conn_id: str = "myk8s",
+ config_file: str | None = None,
+ cluster_context: str | None = None,
+ namespace: str = "default",
+ api_group: str = "dataprocgdc.cloud.google.com",
+ api_version: str = "v1alpha1",
+ in_cluster: bool = False,
+ watch: bool = True,
+ **kwargs,
+ ) -> None:
+ super().__init__(**kwargs)
+ self.application_file = application_file
+ self.kubernetes_conn_id = kubernetes_conn_id
+ self.config_file = config_file
+ self.cluster_context = cluster_context
+ self.namespace = namespace
+ self.api_group = api_group
+ self.api_version = api_version
+ self.in_cluster = in_cluster
+ self.watch = watch
+ self.plural = "sparkapplications"
+
+ @cached_property
+ def hook(self) -> KubernetesHook:
+ hook = KubernetesHook(
+ conn_id=self.kubernetes_conn_id,
+ in_cluster=self.in_cluster,
+ config_file=self.config_file,
+ cluster_context=self.cluster_context,
+ )
+ return hook
+
+ @cached_property
+ def client(self) -> CoreV1Api:
+ return self.hook.core_v1_client
+
+ def _get_namespace_event_stream(self, namespace, query_kwargs=None):
+ try:
+ return Watch().stream(
+ self.hook.core_v1_client.list_namespaced_event,
+ namespace=namespace,
+ watch=True,
+ **(query_kwargs or {}),
+ )
+ except ApiException as e:
+ if e.status == 410: # Resource version is too old
+ events: CoreV1EventList = self.hook.core_v1_client.list_namespaced_event(
+ namespace=namespace, watch=False
+ )
+ resource_version = events.metadata.resource_version
+ query_kwargs["resource_version"] = resource_version
+ return self._get_namespace_event_stream(namespace, query_kwargs)
+ else:
+ raise
+
+ def execute(self, context: Context):
+ if isinstance(self.application_file, str):
+ body = _load_body_to_dict(self.application_file)
+ else:
+ body = self.application_file
+ name = body["metadata"]["name"]
+ namespace = self.namespace or self.hook.get_namespace()
+
+ response = None
+ is_job_created = False
+ if self.watch:
+ try:
+ namespace_event_stream = self._get_namespace_event_stream(
+ namespace=namespace,
+ query_kwargs={
+ "field_selector": f"involvedObject.kind=SparkApplication,involvedObject.name={name}"
+ },
+ )
+
+ response = self.hook.create_custom_object(
+ group=self.api_group,
+ version=self.api_version,
+ plural=self.plural,
+ body=body,
+ namespace=namespace,
+ )
+
+ is_job_created = True
+ for event in namespace_event_stream:
+ obj = event["object"]
+ if event["object"].last_timestamp >= datetime.datetime.strptime(
+ response["metadata"]["creationTimestamp"], "%Y-%m-%dT%H:%M:%S%z"
+ ):
+ self.log.info(obj.message)
+ if obj.reason == "SparkDriverRunning":
+ pod_log_stream = Watch().stream(
+ self.hook.core_v1_client.read_namespaced_pod_log,
+ name=f"{name}-driver",
+ namespace=namespace,
+ timestamps=True,
+ )
+ for line in pod_log_stream:
+ self.log.info(line)
+ elif obj.reason in [
+ "SparkApplicationSubmissionFailed",
+ "SparkApplicationFailed",
+ "SparkApplicationDeleted",
+ ]:
+ is_job_created = False
+ raise AirflowException(obj.message)
+ elif obj.reason == "SparkApplicationCompleted":
+ break
+ else:
+ continue
+ except Exception:
+ if is_job_created:
+ self.on_kill()
+ raise
+ else:
+ response = self.hook.create_custom_object(
+ group=self.api_group,
+ version=self.api_version,
+ plural=self.plural,
+ body=body,
+ namespace=namespace,
+ )
+
+ return response
+
+ def on_kill(self) -> None:
+ if isinstance(self.application_file, str):
+ body = _load_body_to_dict(self.application_file)
+ else:
+ body = self.application_file
+ name = body["metadata"]["name"]
+ namespace = self.namespace or self.hook.get_namespace()
+ self.hook.delete_custom_object(
+ group=self.api_group, version=self.api_version, plural=self.plural, namespace=namespace, name=name
+ )
+
+
+class DataprocGdcCreateAppEnvironmentKrmOperator(BaseOperator):
Review Comment:
For me these operators `DataprocGDCSubmitSparkJobKrmOperator` and `DataprocGdcCreateAppEnvironmentKrmOperator` look quite similar to `KubernetesCreateResourceOperator` operator(https://github.com/apache/airflow/blob/main/airflow/providers/cncf/kubernetes/operators/resource.py#L107). Is it possible to use `KubernetesCreateResourceOperator` for your purpose or maybe you can be inherited from this operator?
Also, in this PR(https://github.com/apache/airflow/pull/37616) I have added ability to create namespaced 'custom_resource_definition' object and create this object on GKE cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding airflow operator to submit and monitor Spark Apps/App Env on DPGDC clusters [airflow]
Posted by "dirrao (via GitHub)" <gi...@apache.org>.
dirrao commented on code in PR #37223:
URL: https://github.com/apache/airflow/pull/37223#discussion_r1482409590
##########
airflow/providers/google/cloud/operators/dataprocgdc.py:
##########
@@ -0,0 +1,300 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+"""This module contains Dataproc GDC operators using KRM APIs."""
+
+from __future__ import annotations
+
+import datetime
+from functools import cached_property
+from typing import Sequence, TYPE_CHECKING
+
+from airflow.exceptions import AirflowException
+from airflow.models import BaseOperator
+from airflow.providers.cncf.kubernetes.hooks.kubernetes import _load_body_to_dict
+from airflow.providers.cncf.kubernetes.hooks.kubernetes import KubernetesHook
+from kubernetes.client import CoreV1Api, ApiException
+from kubernetes.watch import Watch
+
+
+if TYPE_CHECKING:
+ from kubernetes.client.models import CoreV1EventList
+ from airflow.utils.context import Context
+
+
+class DataprocGDCSubmitSparkJobKrmOperator(BaseOperator):
Review Comment:
Can't we extend the functionality of existing operator `DataprocSubmitSparkJobOperator` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding airflow operator to submit and monitor Spark Apps/App Env on DPGDC clusters [airflow]
Posted by "akashsriv07 (via GitHub)" <gi...@apache.org>.
akashsriv07 commented on PR #37223:
URL: https://github.com/apache/airflow/pull/37223#issuecomment-1935231856
Hey Hussein,
Dataproc on GDC is yet to go GA and this is one of the critical features to add as part of the same. Hence we don't have a public doc on the same.
You can take a look at this blog post though: https://cloud.google.com/blog/products/infrastructure-modernization/google-distributed-cloud-new-ai-and-data-services
The CRD for DPGDC is completely different than the sparkoperator.k8s.io/v1beta2, which stops us from leveraging the same operator/sensor.
using KRM APIs is one of the ways to interact with a DPGDC cluster.
In future, we plan to add the operator similar to dataproc.py(ex: DataprocSubmitSparkJobOperator) which will leverage Google's internal One Platform API mechanism.
> I didn't find anything about this in the GCP documentation, could you please add the documentation link?
>
> Is the CRD `dataprocgdc.cloud.google.com/v1alpha1` based on on `sparkoperator.k8s.io/v1beta2`? I'm asking because we already have two operators for the `spark-on-k8s-operator`, so maybe we can use one of them as a superclass to your operator to avoid code duplication and implementing everything from scratch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding airflow operator to submit and monitor Spark Apps/App Env on DPGDC clusters [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #37223: Adding airflow operator to submit and monitor Spark Apps/App Env on DPGDC clusters
URL: https://github.com/apache/airflow/pull/37223
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding airflow operator to create and monitor Spark Apps/App Env on DPGDC clusters [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #37223:
URL: https://github.com/apache/airflow/pull/37223#issuecomment-1932111641
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:
- Pay attention to the quality of your code (ruff, mypy and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/main/contributing-docs/08_static_code_checks.rst#prerequisites-for-pre-commit-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/dev/breeze/doc/README.rst) for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#coding-style-and-best-practices).
- Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding airflow operator to submit and monitor Spark Apps/App Env on DPGDC clusters [airflow]
Posted by "MaksYermak (via GitHub)" <gi...@apache.org>.
MaksYermak commented on code in PR #37223:
URL: https://github.com/apache/airflow/pull/37223#discussion_r1504363088
##########
airflow/providers/google/cloud/operators/dataprocgdc.py:
##########
@@ -0,0 +1,272 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+"""This module contains Dataproc GDC operators using KRM APIs."""
+
+from __future__ import annotations
+
+import datetime
+from functools import cached_property
+from typing import TYPE_CHECKING, Sequence
+
+from kubernetes.client import ApiException, CoreV1Api
+from kubernetes.watch import Watch
+
+from airflow.exceptions import AirflowException
+from airflow.models import BaseOperator
+from airflow.providers.cncf.kubernetes.hooks.kubernetes import KubernetesHook, _load_body_to_dict
+
+if TYPE_CHECKING:
+ from kubernetes.client.models import CoreV1EventList
+
+ from airflow.utils.context import Context
+
+
+class DataprocGDCSubmitSparkJobKrmOperator(BaseOperator):
+ """
+ Creates sparkApplication object in dpgdc cluster using KRM APIs.
+
+ :param application_file: Defines gdc 'custom_resource_definition' of
+ 'sparkApplication'
+ path to a '.yaml' file, '.json' file, YAML string or python dictionary.
+ :param namespace: kubernetes namespace to put sparkApplication
+ :param kubernetes_conn_id: The :ref:`kubernetes connection id
+ <howto/connection:kubernetes>` for the GDC cluster.
+ :param api_group: DPGDC api group
+ :param api_version: DPGDC api version
+ :param watch: whether to watch the job status and logs or not
+ """
+
+ template_fields: Sequence[str] = "application_file"
+ template_ext: Sequence[str] = (".yaml", ".yml", ".json")
+ ui_color = "#f4a460"
+
+ def __init__(
+ self,
+ *,
+ application_file: str | dict,
+ kubernetes_conn_id: str = "myk8s",
+ config_file: str | None = None,
+ cluster_context: str | None = None,
+ namespace: str = "default",
+ api_group: str = "dataprocgdc.cloud.google.com",
+ api_version: str = "v1alpha1",
+ in_cluster: bool = False,
+ watch: bool = True,
+ **kwargs,
+ ) -> None:
+ super().__init__(**kwargs)
+ self.application_file = application_file
+ self.kubernetes_conn_id = kubernetes_conn_id
+ self.config_file = config_file
+ self.cluster_context = cluster_context
+ self.namespace = namespace
+ self.api_group = api_group
+ self.api_version = api_version
+ self.in_cluster = in_cluster
+ self.watch = watch
+ self.plural = "sparkapplications"
+
+ @cached_property
+ def hook(self) -> KubernetesHook:
+ hook = KubernetesHook(
+ conn_id=self.kubernetes_conn_id,
+ in_cluster=self.in_cluster,
+ config_file=self.config_file,
+ cluster_context=self.cluster_context,
+ )
+ return hook
+
+ @cached_property
+ def client(self) -> CoreV1Api:
+ return self.hook.core_v1_client
+
+ def _get_namespace_event_stream(self, namespace, query_kwargs=None):
+ try:
+ return Watch().stream(
+ self.hook.core_v1_client.list_namespaced_event,
+ namespace=namespace,
+ watch=True,
+ **(query_kwargs or {}),
+ )
+ except ApiException as e:
+ if e.status == 410: # Resource version is too old
+ events: CoreV1EventList = self.hook.core_v1_client.list_namespaced_event(
+ namespace=namespace, watch=False
+ )
+ resource_version = events.metadata.resource_version
+ query_kwargs["resource_version"] = resource_version
+ return self._get_namespace_event_stream(namespace, query_kwargs)
+ else:
+ raise
+
+ def execute(self, context: Context):
+ if isinstance(self.application_file, str):
+ body = _load_body_to_dict(self.application_file)
+ else:
+ body = self.application_file
+ name = body["metadata"]["name"]
+ namespace = self.namespace or self.hook.get_namespace()
+
+ response = None
+ is_job_created = False
+ if self.watch:
+ try:
+ namespace_event_stream = self._get_namespace_event_stream(
+ namespace=namespace,
+ query_kwargs={
+ "field_selector": f"involvedObject.kind=SparkApplication,involvedObject.name={name}"
+ },
+ )
+
+ response = self.hook.create_custom_object(
+ group=self.api_group,
+ version=self.api_version,
+ plural=self.plural,
+ body=body,
+ namespace=namespace,
+ )
+
+ is_job_created = True
+ for event in namespace_event_stream:
+ obj = event["object"]
+ if event["object"].last_timestamp >= datetime.datetime.strptime(
+ response["metadata"]["creationTimestamp"], "%Y-%m-%dT%H:%M:%S%z"
+ ):
+ self.log.info(obj.message)
+ if obj.reason == "SparkDriverRunning":
+ pod_log_stream = Watch().stream(
+ self.hook.core_v1_client.read_namespaced_pod_log,
+ name=f"{name}-driver",
+ namespace=namespace,
+ timestamps=True,
+ )
+ for line in pod_log_stream:
+ self.log.info(line)
+ elif obj.reason in [
+ "SparkApplicationSubmissionFailed",
+ "SparkApplicationFailed",
+ "SparkApplicationDeleted",
+ ]:
+ is_job_created = False
+ raise AirflowException(obj.message)
+ elif obj.reason == "SparkApplicationCompleted":
+ break
+ else:
+ continue
+ except Exception:
+ if is_job_created:
+ self.on_kill()
+ raise
+ else:
+ response = self.hook.create_custom_object(
+ group=self.api_group,
+ version=self.api_version,
+ plural=self.plural,
+ body=body,
+ namespace=namespace,
+ )
+
+ return response
+
+ def on_kill(self) -> None:
+ if isinstance(self.application_file, str):
+ body = _load_body_to_dict(self.application_file)
+ else:
+ body = self.application_file
+ name = body["metadata"]["name"]
+ namespace = self.namespace or self.hook.get_namespace()
+ self.hook.delete_custom_object(
+ group=self.api_group, version=self.api_version, plural=self.plural, namespace=namespace, name=name
+ )
+
+
+class DataprocGdcCreateAppEnvironmentKrmOperator(BaseOperator):
Review Comment:
@akashsriv07 could you please answer my question above?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding airflow operator to submit and monitor Spark Apps/App Env on DPGDC clusters [airflow]
Posted by "michalmodras (via GitHub)" <gi...@apache.org>.
michalmodras commented on code in PR #37223:
URL: https://github.com/apache/airflow/pull/37223#discussion_r1518845852
##########
tests/system/providers/google/gdc/dataprocgdc/example_dataprocgdc.py:
##########
@@ -0,0 +1,77 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Example Airflow DAG that shows how to create and monitor a Spark Application or App Environment
+ in a DPGDC cluster
+"""
+
+from __future__ import annotations
+
+from datetime import datetime
+
+from airflow import DAG
+from airflow.providers.google.gdc.operators.dataprocgdc import (
+ DataprocGdcCreateAppEnvironmentKrmOperator,
+ DataprocGDCSubmitSparkJobKrmOperator,
+)
+from airflow.providers.google.gdc.sensors.dataprocgdc import DataprocGDCKrmSensor
+
+DAG_ID = "example_dag_dpgdc-krm"
+
+with DAG(
Review Comment:
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding airflow operator to submit and monitor Spark Apps/App Env on DPGDC clusters [airflow]
Posted by "akashsriv07 (via GitHub)" <gi...@apache.org>.
akashsriv07 commented on PR #37223:
URL: https://github.com/apache/airflow/pull/37223#issuecomment-2084416950
> Setting request changes to avoid accedental merge
Will explore the feasibility on creating the provider and the process to be followed. Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding airflow operator to submit and monitor Spark Apps/App Env on DPGDC clusters [airflow]
Posted by "michalmodras (via GitHub)" <gi...@apache.org>.
michalmodras commented on PR #37223:
URL: https://github.com/apache/airflow/pull/37223#issuecomment-1964943387
Hey, Michal from Cloud Composer team here.
+1 for this code living in a separate, GDC provider - GDC is based on slightly different infra than GCP, and the operators won't necessarily work in the same way (some might be very similar, some very different). If there's shared code, I propose having a util module/provider between these two.
FWIW we plan to separate out more Google providers (e.g. taking out ads to a separate one) to decouple support of Google services a bit, instead of having them all in one provider, at times unnecessarily tangling e.g. dependencies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding airflow operator to submit and monitor Spark Apps/App Env on DPGDC clusters [airflow]
Posted by "molcay (via GitHub)" <gi...@apache.org>.
molcay commented on PR #37223:
URL: https://github.com/apache/airflow/pull/37223#issuecomment-1964082744
Hey @akashsriv07,
When I read the following sentence on [the product page](https://cloud.google.com/distributed-cloud#section-1), I thought that this product **is not directly in the GCP**, rather it is equvienlt to the GCP but it is for on-premise infrastructure:
> Build, deploy, and scale modern industry & public sector applications on-premise with AI ready modern infrastructure, ensure data security, and enable agile developer workflows across edge locations & data centers.
But of course, I am not entirely sure on the topic. This suggestion folds on a grey area I guess.
But at the base level, maybe we can group the changes related to GDC in a single module and put it to inside of the cloud module.
```bash
airflow/providers/google/cloud
├── example_dags
├── fs
├── gdc # <-- this is the new one
├── hooks
├── __init__.py
├── _internal_client
├── links
├── log
├── operators
├── __pycache__
├── secrets
├── sensors
├── transfers
├── triggers
└── utils
```
Also, maybe it is too early to have this module structure right now. We can also wait and see how it will extend
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding airflow operator to submit and monitor Spark Apps/App Env on DPGDC clusters [airflow]
Posted by "MaksYermak (via GitHub)" <gi...@apache.org>.
MaksYermak commented on code in PR #37223:
URL: https://github.com/apache/airflow/pull/37223#discussion_r1504336354
##########
tests/system/providers/google/gdc/dataprocgdc/example_dataprocgdc.py:
##########
@@ -0,0 +1,77 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Example Airflow DAG that shows how to create and monitor a Spark Application or App Environment
+ in a DPGDC cluster
+"""
+
+from __future__ import annotations
+
+from datetime import datetime
+
+from airflow import DAG
+from airflow.providers.google.gdc.operators.dataprocgdc import (
+ DataprocGdcCreateAppEnvironmentKrmOperator,
+ DataprocGDCSubmitSparkJobKrmOperator,
+)
+from airflow.providers.google.gdc.sensors.dataprocgdc import DataprocGDCKrmSensor
+
+DAG_ID = "example_dag_dpgdc-krm"
+
+with DAG(
+ DAG_ID, schedule="@once", catchup=False, start_date=datetime(2024, 2, 5), tags=["example", "dataprocgdc"]
+) as dag:
+ submitSparkJobOperator = DataprocGDCSubmitSparkJobKrmOperator(
+ task_id="example-dataprocgdc-submitspark-operator",
+ trigger_rule="all_success",
+ depends_on_past=False,
+ retries=1,
+ application_file="sparkpi.yaml",
+ namespace="default",
+ kubernetes_conn_id="myk8s",
Review Comment:
Does this operator work with GKE cluster configuration?
##########
tests/system/providers/google/gdc/dataprocgdc/example_dataprocgdc.py:
##########
@@ -0,0 +1,77 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Example Airflow DAG that shows how to create and monitor a Spark Application or App Environment
+ in a DPGDC cluster
+"""
+
+from __future__ import annotations
+
+from datetime import datetime
+
+from airflow import DAG
+from airflow.providers.google.gdc.operators.dataprocgdc import (
+ DataprocGdcCreateAppEnvironmentKrmOperator,
+ DataprocGDCSubmitSparkJobKrmOperator,
+)
+from airflow.providers.google.gdc.sensors.dataprocgdc import DataprocGDCKrmSensor
+
+DAG_ID = "example_dag_dpgdc-krm"
+
+with DAG(
Review Comment:
Could you please update this system test using project codestyle e.g. https://github.com/apache/airflow/blob/main/tests/system/providers/google/cloud/kubernetes_engine/example_kubernetes_engine.py ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding airflow operator to submit and monitor Spark Apps/App Env on DPGDC clusters [airflow]
Posted by "akashsriv07 (via GitHub)" <gi...@apache.org>.
akashsriv07 commented on code in PR #37223:
URL: https://github.com/apache/airflow/pull/37223#discussion_r1482416871
##########
airflow/providers/google/cloud/operators/dataprocgdc.py:
##########
@@ -0,0 +1,300 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+"""This module contains Dataproc GDC operators using KRM APIs."""
+
+from __future__ import annotations
+
+import datetime
+from functools import cached_property
+from typing import Sequence, TYPE_CHECKING
+
+from airflow.exceptions import AirflowException
+from airflow.models import BaseOperator
+from airflow.providers.cncf.kubernetes.hooks.kubernetes import _load_body_to_dict
+from airflow.providers.cncf.kubernetes.hooks.kubernetes import KubernetesHook
+from kubernetes.client import CoreV1Api, ApiException
+from kubernetes.watch import Watch
+
+
+if TYPE_CHECKING:
+ from kubernetes.client.models import CoreV1EventList
+ from airflow.utils.context import Context
+
+
+class DataprocGDCSubmitSparkJobKrmOperator(BaseOperator):
Review Comment:
There are two ways we need to go for implementing Airflow operator to trigger workloads on GDC cluster.
1 is via KRM API and another is via Google cloud API(where we could use the DataprocSubmitSparkJobOperator, however need to check if that's possible as well).
This PR targets the KRM APIs exposed in a GDC cluster. So this cannot leverage the existing operator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org