You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "nailo2c (via GitHub)" <gi...@apache.org> on 2026/02/24 00:58:47 UTC
[PR] Add Azure Virtual Machines operators, sensor, and trigger (#49796) [airflow]
nailo2c opened a new pull request, #62391:
URL: https://github.com/apache/airflow/pull/62391
closes: #49796
# Why
This request is from issue #49796. `ec2` and `gce` support these features, so I think it would be great if Azure Virtual Machines also supported them.
# How
Implemented `hooks`, `operators`, `sensors`, and `triggers` following the same pattern as the `ec2` provider, with deferrable mode support.
The whole process is shown in sequence diagram below:
```mermaid
sequenceDiagram
participant DAG as DAG
participant StartOp as StartOperator
participant Sensor as StateSensor
participant Trigger as StateTrigger
participant Hook as AzureComputeHook
participant Azure as Azure Compute API
DAG->>StartOp: execute()
StartOp->>Hook: start_instance(resource_group, vm_name)
Hook->>Azure: begin_start(...)
Hook->>Azure: poller.result() (when wait_for_completion=True)
StartOp-->>DAG: success
DAG->>Sensor: execute()
Sensor->>Hook: poke() -> get_power_state()
Hook->>Azure: instance_view(...)
alt state already reached
Sensor-->>DAG: success
else state not reached + deferrable=True
Sensor-->>Trigger: defer(trigger=AzureVirtualMachineStateTrigger)
loop poll until target state
Trigger->>Hook: async_get_power_state() [native async]
Hook->>Azure: instance_view(...)
end
Trigger-->>Sensor: TriggerEvent(status=success)
Sensor-->>DAG: execute_complete()
end
```
# What
### System Test
1. I prepared Azure cloud resources and DAGs from my local Airflow environment.
<img width="2534" height="724" alt="af_49796_1_before_start" src="https://github.com/user-attachments/assets/3803c546-662a-47cb-96a3-b8dd17d714d6" />
<br>
2. Starting
<img width="2528" height="789" alt="af_49796_2_starting" src="https://github.com/user-attachments/assets/c8cfbe3c-821a-4e2c-b932-4ac6df968f22" />
<br>
3. Running
<img width="2532" height="751" alt="af_49796_3_running" src="https://github.com/user-attachments/assets/0b8d0c30-a2d7-4650-847f-f01614f51b1e" />
<br>
4. Restarting
<img width="2539" height="717" alt="af_49796_4_restarting" src="https://github.com/user-attachments/assets/ce20243f-c584-40f3-aa35-56b9273c0994" />
<br>
5. Deallocating
<img width="2543" height="748" alt="af_49796_5_deallocating" src="https://github.com/user-attachments/assets/21bcce1f-99a4-4c27-b6ab-ded0864641d1" />
<br>
6. Stopped
<img width="2537" height="782" alt="af_49796_6_stopped" src="https://github.com/user-attachments/assets/1be497f7-1a49-4b45-a4e9-30fcacdd880f" />
### Doc
<img width="2509" height="1320" alt="af_49796_7_doc" src="https://github.com/user-attachments/assets/2fb7fe47-5168-45dc-98d9-610a0f71348f" />
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR, please
change below checkbox to `[X]` followed by the name of the tool, uncomment the "Generated-by".
-->
- [x] Yes (please specify the tool below)
Claude ( I've thoroughly reviewed all AI-generated code :) )
<!--
Generated-by: [Tool Name] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
---
* Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment: `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Azure Virtual Machines operators, sensor, and trigger (#49796) [airflow]
Posted by "nailo2c (via GitHub)" <gi...@apache.org>.
nailo2c commented on code in PR #62391:
URL: https://github.com/apache/airflow/pull/62391#discussion_r2849774639
##########
providers/microsoft/azure/src/airflow/providers/microsoft/azure/triggers/compute.py:
##########
@@ -0,0 +1,90 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import asyncio
+from collections.abc import AsyncIterator
+from typing import Any
+
+from airflow.triggers.base import BaseTrigger, TriggerEvent
+
+
+class AzureVirtualMachineStateTrigger(BaseTrigger):
+ """
+ Poll the Azure VM power state and yield a TriggerEvent once it matches the target.
+
+ Uses the native async Azure SDK client (``azure.mgmt.compute.aio``) so that
+ the triggerer event loop is never blocked.
+
+ :param resource_group_name: Name of the Azure resource group.
+ :param vm_name: Name of the virtual machine.
+ :param target_state: Desired power state, e.g. ``running``, ``deallocated``.
+ :param azure_conn_id: Azure connection id.
+ :param poke_interval: Polling interval in seconds.
+ """
+
+ def __init__(
+ self,
+ resource_group_name: str,
+ vm_name: str,
+ target_state: str,
+ azure_conn_id: str = "azure_default",
+ poke_interval: float = 30.0,
+ ) -> None:
+ super().__init__()
+ self.resource_group_name = resource_group_name
+ self.vm_name = vm_name
+ self.target_state = target_state
+ self.azure_conn_id = azure_conn_id
+ self.poke_interval = poke_interval
+
+ def serialize(self) -> tuple[str, dict[str, Any]]:
+ """Serialize AzureVirtualMachineStateTrigger arguments and classpath."""
+ return (
+ "airflow.providers.microsoft.azure.triggers.compute.AzureVirtualMachineStateTrigger",
Review Comment:
Got it-- indeed, that will be easier to refactor in the future
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Azure Virtual Machines operators, sensor, and trigger (#49796) [airflow]
Posted by "sjyangkevin (via GitHub)" <gi...@apache.org>.
sjyangkevin commented on code in PR #62391:
URL: https://github.com/apache/airflow/pull/62391#discussion_r2844047170
##########
providers/microsoft/azure/src/airflow/providers/microsoft/azure/hooks/compute.py:
##########
@@ -0,0 +1,233 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from functools import cached_property
+from typing import Any, cast
+
+from azure.common.client_factory import get_client_from_auth_file, get_client_from_json_dict
+from azure.identity import ClientSecretCredential, DefaultAzureCredential
+from azure.identity.aio import ClientSecretCredential as AsyncClientSecretCredential
+from azure.mgmt.compute import ComputeManagementClient
+from azure.mgmt.compute.aio import ComputeManagementClient as AsyncComputeManagementClient
+
+from airflow.providers.common.compat.connection import get_async_connection
+from airflow.providers.common.compat.sdk import AirflowException
+from airflow.providers.microsoft.azure.hooks.base_azure import AzureBaseHook
+from airflow.providers.microsoft.azure.utils import (
+ get_async_default_azure_credential,
+ get_sync_default_azure_credential,
+)
+
+
+class AzureComputeHook(AzureBaseHook):
+ """
+ A hook to interact with Azure Compute to manage Virtual Machines.
+
+ :param azure_conn_id: :ref:`Azure connection id<howto/connection:azure>` of
+ a service principal which will be used to manage virtual machines.
+ """
+
+ conn_name_attr = "azure_conn_id"
+ default_conn_name = "azure_default"
+ conn_type = "azure_compute"
+ hook_name = "Azure Compute"
+
+ def __init__(self, azure_conn_id: str = default_conn_name) -> None:
+ super().__init__(sdk_client=ComputeManagementClient, conn_id=azure_conn_id)
+
+ @cached_property
+ def connection(self) -> ComputeManagementClient:
+ return self.get_conn()
+
+ def get_conn(self) -> Any:
+ """
+ Authenticate the resource using the connection id passed during init.
+
+ :return: the authenticated ComputeManagementClient.
+ """
+ conn = self.get_connection(self.conn_id)
+ tenant = conn.extra_dejson.get("tenantId")
+
+ key_path = conn.extra_dejson.get("key_path")
+ if key_path:
+ if not key_path.endswith(".json"):
+ raise AirflowException("Unrecognised extension for key file.")
Review Comment:
Thanks for implementing it!
Just a small feedback here, can we avoid using `AirflowException`, and use a better way suggested in contribution guide https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#dont-raise-airflowexception-directly.
Context:
https://lists.apache.org/thread/t8bnhyqy77kq4fk7fj3fmjd5wo9kv6w0
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Azure Virtual Machines operators, sensor, and trigger (#49796) [airflow]
Posted by "nailo2c (via GitHub)" <gi...@apache.org>.
nailo2c commented on code in PR #62391:
URL: https://github.com/apache/airflow/pull/62391#discussion_r2844060574
##########
providers/microsoft/azure/src/airflow/providers/microsoft/azure/hooks/compute.py:
##########
@@ -0,0 +1,233 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from functools import cached_property
+from typing import Any, cast
+
+from azure.common.client_factory import get_client_from_auth_file, get_client_from_json_dict
+from azure.identity import ClientSecretCredential, DefaultAzureCredential
+from azure.identity.aio import ClientSecretCredential as AsyncClientSecretCredential
+from azure.mgmt.compute import ComputeManagementClient
+from azure.mgmt.compute.aio import ComputeManagementClient as AsyncComputeManagementClient
+
+from airflow.providers.common.compat.connection import get_async_connection
+from airflow.providers.common.compat.sdk import AirflowException
+from airflow.providers.microsoft.azure.hooks.base_azure import AzureBaseHook
+from airflow.providers.microsoft.azure.utils import (
+ get_async_default_azure_credential,
+ get_sync_default_azure_credential,
+)
+
+
+class AzureComputeHook(AzureBaseHook):
+ """
+ A hook to interact with Azure Compute to manage Virtual Machines.
+
+ :param azure_conn_id: :ref:`Azure connection id<howto/connection:azure>` of
+ a service principal which will be used to manage virtual machines.
+ """
+
+ conn_name_attr = "azure_conn_id"
+ default_conn_name = "azure_default"
+ conn_type = "azure_compute"
+ hook_name = "Azure Compute"
+
+ def __init__(self, azure_conn_id: str = default_conn_name) -> None:
+ super().__init__(sdk_client=ComputeManagementClient, conn_id=azure_conn_id)
+
+ @cached_property
+ def connection(self) -> ComputeManagementClient:
+ return self.get_conn()
+
+ def get_conn(self) -> Any:
+ """
+ Authenticate the resource using the connection id passed during init.
+
+ :return: the authenticated ComputeManagementClient.
+ """
+ conn = self.get_connection(self.conn_id)
+ tenant = conn.extra_dejson.get("tenantId")
+
+ key_path = conn.extra_dejson.get("key_path")
+ if key_path:
+ if not key_path.endswith(".json"):
+ raise AirflowException("Unrecognised extension for key file.")
Review Comment:
Nice, I didn't know that before. I'll update it accordingly. Thanks for the review! :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Azure Virtual Machines operators, sensor, and trigger (#49796) [airflow]
Posted by "dabla (via GitHub)" <gi...@apache.org>.
dabla commented on PR #62391:
URL: https://github.com/apache/airflow/pull/62391#issuecomment-4023927359
> Hi, thanks for the review, I learned a lot from it! 😄 If there's anything else I should improve, pleas let me know.
Just need to fix the spelling wordlist conflict, then I think we can merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Azure Virtual Machines operators, sensor, and trigger (#49796) [airflow]
Posted by "nailo2c (via GitHub)" <gi...@apache.org>.
nailo2c commented on code in PR #62391:
URL: https://github.com/apache/airflow/pull/62391#discussion_r2849798519
##########
providers/microsoft/azure/tests/unit/microsoft/azure/triggers/test_compute.py:
##########
@@ -0,0 +1,121 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from unittest import mock
+
+import pytest
+
+from airflow.providers.microsoft.azure.triggers.compute import AzureVirtualMachineStateTrigger
+from airflow.triggers.base import TriggerEvent
+
+RESOURCE_GROUP = "test-rg"
+VM_NAME = "test-vm"
+TARGET_STATE = "running"
+CONN_ID = "azure_default"
+POKE_INTERVAL = 10.0
+
+
+class TestAzureVirtualMachineStateTrigger:
+ def test_serialize(self):
+ trigger = AzureVirtualMachineStateTrigger(
+ resource_group_name=RESOURCE_GROUP,
+ vm_name=VM_NAME,
+ target_state=TARGET_STATE,
+ azure_conn_id=CONN_ID,
+ poke_interval=POKE_INTERVAL,
+ )
+
+ class_path, args = trigger.serialize()
Review Comment:
good point
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Azure Virtual Machines operators, sensor, and trigger (#49796) [airflow]
Posted by "nailo2c (via GitHub)" <gi...@apache.org>.
nailo2c commented on PR #62391:
URL: https://github.com/apache/airflow/pull/62391#issuecomment-3955092789
Hi, thanks for the review, I learned a lot from it! 😄
If there's anything else I should improve, pleas let me know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Azure Virtual Machines operators, sensor, and trigger (#49796) [airflow]
Posted by "nailo2c (via GitHub)" <gi...@apache.org>.
nailo2c commented on code in PR #62391:
URL: https://github.com/apache/airflow/pull/62391#discussion_r2849756177
##########
providers/microsoft/azure/src/airflow/providers/microsoft/azure/operators/compute.py:
##########
@@ -0,0 +1,158 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from collections.abc import Sequence
+from typing import TYPE_CHECKING
+
+from airflow.providers.common.compat.sdk import BaseOperator
+from airflow.providers.microsoft.azure.hooks.compute import AzureComputeHook
+
+if TYPE_CHECKING:
+ from airflow.sdk import Context
+
+
+class AzureVirtualMachineStartOperator(BaseOperator):
Review Comment:
Nice suggestion, the code is cleaner with this change!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Azure Virtual Machines operators, sensor, and trigger (#49796) [airflow]
Posted by "dabla (via GitHub)" <gi...@apache.org>.
dabla commented on code in PR #62391:
URL: https://github.com/apache/airflow/pull/62391#discussion_r2848983340
##########
providers/microsoft/azure/src/airflow/providers/microsoft/azure/operators/compute.py:
##########
@@ -0,0 +1,158 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from collections.abc import Sequence
+from typing import TYPE_CHECKING
+
+from airflow.providers.common.compat.sdk import BaseOperator
+from airflow.providers.microsoft.azure.hooks.compute import AzureComputeHook
+
+if TYPE_CHECKING:
+ from airflow.sdk import Context
+
+
+class AzureVirtualMachineStartOperator(BaseOperator):
Review Comment:
I would add a BaseAzureVirtualMachineOperator class which has following hook property:
```
@cached_property
def hook(self) -> AzureComputeHook:
return AzureComputeHook(azure_conn_id=self.azure_conn_id)
```
That way all underlying operators inherit from BaseAzureVirtualMachineOperator and just can call self.hook and you have a more DRY approach.
##########
providers/microsoft/azure/src/airflow/providers/microsoft/azure/triggers/compute.py:
##########
@@ -0,0 +1,90 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import asyncio
+from collections.abc import AsyncIterator
+from typing import Any
+
+from airflow.triggers.base import BaseTrigger, TriggerEvent
+
+
+class AzureVirtualMachineStateTrigger(BaseTrigger):
+ """
+ Poll the Azure VM power state and yield a TriggerEvent once it matches the target.
+
+ Uses the native async Azure SDK client (``azure.mgmt.compute.aio``) so that
+ the triggerer event loop is never blocked.
+
+ :param resource_group_name: Name of the Azure resource group.
+ :param vm_name: Name of the virtual machine.
+ :param target_state: Desired power state, e.g. ``running``, ``deallocated``.
+ :param azure_conn_id: Azure connection id.
+ :param poke_interval: Polling interval in seconds.
+ """
+
+ def __init__(
+ self,
+ resource_group_name: str,
+ vm_name: str,
+ target_state: str,
+ azure_conn_id: str = "azure_default",
+ poke_interval: float = 30.0,
+ ) -> None:
+ super().__init__()
+ self.resource_group_name = resource_group_name
+ self.vm_name = vm_name
+ self.target_state = target_state
+ self.azure_conn_id = azure_conn_id
+ self.poke_interval = poke_interval
+
+ def serialize(self) -> tuple[str, dict[str, Any]]:
+ """Serialize AzureVirtualMachineStateTrigger arguments and classpath."""
+ return (
+ "airflow.providers.microsoft.azure.triggers.compute.AzureVirtualMachineStateTrigger",
Review Comment:
I would do it like this which is more refactor prone/safe:
f"{self.__class__.__module__}.{self.__class__.__name__}",
Instead of:
"airflow.providers.microsoft.azure.triggers.compute.AzureVirtualMachineStateTrigger",
##########
providers/microsoft/azure/src/airflow/providers/microsoft/azure/triggers/compute.py:
##########
@@ -0,0 +1,90 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import asyncio
+from collections.abc import AsyncIterator
+from typing import Any
+
+from airflow.triggers.base import BaseTrigger, TriggerEvent
+
+
+class AzureVirtualMachineStateTrigger(BaseTrigger):
+ """
+ Poll the Azure VM power state and yield a TriggerEvent once it matches the target.
+
+ Uses the native async Azure SDK client (``azure.mgmt.compute.aio``) so that
+ the triggerer event loop is never blocked.
+
+ :param resource_group_name: Name of the Azure resource group.
+ :param vm_name: Name of the virtual machine.
+ :param target_state: Desired power state, e.g. ``running``, ``deallocated``.
+ :param azure_conn_id: Azure connection id.
+ :param poke_interval: Polling interval in seconds.
+ """
+
+ def __init__(
+ self,
+ resource_group_name: str,
+ vm_name: str,
+ target_state: str,
+ azure_conn_id: str = "azure_default",
+ poke_interval: float = 30.0,
+ ) -> None:
+ super().__init__()
+ self.resource_group_name = resource_group_name
+ self.vm_name = vm_name
+ self.target_state = target_state
+ self.azure_conn_id = azure_conn_id
+ self.poke_interval = poke_interval
+
+ def serialize(self) -> tuple[str, dict[str, Any]]:
+ """Serialize AzureVirtualMachineStateTrigger arguments and classpath."""
+ return (
+ "airflow.providers.microsoft.azure.triggers.compute.AzureVirtualMachineStateTrigger",
+ {
+ "resource_group_name": self.resource_group_name,
+ "vm_name": self.vm_name,
+ "target_state": self.target_state,
+ "azure_conn_id": self.azure_conn_id,
+ "poke_interval": self.poke_interval,
+ },
+ )
+
+ async def run(self) -> AsyncIterator[TriggerEvent]:
+ """Poll VM power state asynchronously until it matches the target state."""
+ from airflow.providers.microsoft.azure.hooks.compute import AzureComputeHook
+
+ try:
+ async with AzureComputeHook(azure_conn_id=self.azure_conn_id) as hook:
+ while True:
+ power_state = await hook.async_get_power_state(self.resource_group_name, self.vm_name)
+ if power_state == self.target_state:
+ message = f"VM {self.vm_name} reached state '{self.target_state}'."
+ yield TriggerEvent({"status": "success", "message": message})
+ return
+ self.log.info(
+ "VM %s power state: %s. Waiting for %s. Sleeping for %s seconds.",
+ self.vm_name,
+ power_state,
+ self.target_state,
+ self.poke_interval,
+ )
+ await asyncio.sleep(self.poke_interval)
+ except Exception as e:
+ yield TriggerEvent({"status": "error", "message": str(e)})
Review Comment:
Probably best to also add return statement here
##########
providers/microsoft/azure/tests/unit/microsoft/azure/triggers/test_compute.py:
##########
@@ -0,0 +1,121 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from unittest import mock
+
+import pytest
+
+from airflow.providers.microsoft.azure.triggers.compute import AzureVirtualMachineStateTrigger
+from airflow.triggers.base import TriggerEvent
+
+RESOURCE_GROUP = "test-rg"
+VM_NAME = "test-vm"
+TARGET_STATE = "running"
+CONN_ID = "azure_default"
+POKE_INTERVAL = 10.0
+
+
+class TestAzureVirtualMachineStateTrigger:
+ def test_serialize(self):
+ trigger = AzureVirtualMachineStateTrigger(
+ resource_group_name=RESOURCE_GROUP,
+ vm_name=VM_NAME,
+ target_state=TARGET_STATE,
+ azure_conn_id=CONN_ID,
+ poke_interval=POKE_INTERVAL,
+ )
+
+ class_path, args = trigger.serialize()
Review Comment:
I would change assert here:
```
actual = trigger.serialize()
assert isinstance(actual, tuple)
assert actual[0] == f"{AzureVirtualMachineStateTrigger.__module__}.{AzureVirtualMachineStateTrigger.__name__}"
assert actual[1] == {
"resource_group_name": RESOURCE_GROUP,
"vm_name": VM_NAME,
"target_state": TARGET_STATE,
"azure_conn_id": CONN_ID,
"poke_interval": POKE_INTERVAL,
}
```
##########
providers/microsoft/azure/src/airflow/providers/microsoft/azure/hooks/compute.py:
##########
@@ -0,0 +1,237 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from functools import cached_property
+from typing import Any, cast
+
+from azure.common.client_factory import get_client_from_auth_file, get_client_from_json_dict
+from azure.identity import ClientSecretCredential, DefaultAzureCredential
+from azure.identity.aio import (
+ ClientSecretCredential as AsyncClientSecretCredential,
+ DefaultAzureCredential as AsyncDefaultAzureCredential,
+)
+from azure.mgmt.compute import ComputeManagementClient
+from azure.mgmt.compute.aio import ComputeManagementClient as AsyncComputeManagementClient
+
+from airflow.providers.common.compat.connection import get_async_connection
+from airflow.providers.microsoft.azure.hooks.base_azure import AzureBaseHook
+from airflow.providers.microsoft.azure.utils import (
+ get_async_default_azure_credential,
+ get_sync_default_azure_credential,
+)
+
+
+class AzureComputeHook(AzureBaseHook):
+ """
+ A hook to interact with Azure Compute to manage Virtual Machines.
+
+ :param azure_conn_id: :ref:`Azure connection id<howto/connection:azure>` of
+ a service principal which will be used to manage virtual machines.
+ """
+
+ conn_name_attr = "azure_conn_id"
+ default_conn_name = "azure_default"
+ conn_type = "azure_compute"
+ hook_name = "Azure Compute"
+
+ def __init__(self, azure_conn_id: str = default_conn_name) -> None:
+ super().__init__(sdk_client=ComputeManagementClient, conn_id=azure_conn_id)
+ self._async_conn: AsyncComputeManagementClient | None = None
+
+ @cached_property
+ def connection(self) -> ComputeManagementClient:
+ return self.get_conn()
+
+ def get_conn(self) -> Any:
Review Comment:
would return ComputeManagementClient as type here instead of Any
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Azure Virtual Machines operators, sensor, and trigger (#49796) [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #62391:
URL: https://github.com/apache/airflow/pull/62391
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Azure Virtual Machines operators, sensor, and trigger (#49796) [airflow]
Posted by "sjyangkevin (via GitHub)" <gi...@apache.org>.
sjyangkevin commented on code in PR #62391:
URL: https://github.com/apache/airflow/pull/62391#discussion_r2844047170
##########
providers/microsoft/azure/src/airflow/providers/microsoft/azure/hooks/compute.py:
##########
@@ -0,0 +1,233 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from functools import cached_property
+from typing import Any, cast
+
+from azure.common.client_factory import get_client_from_auth_file, get_client_from_json_dict
+from azure.identity import ClientSecretCredential, DefaultAzureCredential
+from azure.identity.aio import ClientSecretCredential as AsyncClientSecretCredential
+from azure.mgmt.compute import ComputeManagementClient
+from azure.mgmt.compute.aio import ComputeManagementClient as AsyncComputeManagementClient
+
+from airflow.providers.common.compat.connection import get_async_connection
+from airflow.providers.common.compat.sdk import AirflowException
+from airflow.providers.microsoft.azure.hooks.base_azure import AzureBaseHook
+from airflow.providers.microsoft.azure.utils import (
+ get_async_default_azure_credential,
+ get_sync_default_azure_credential,
+)
+
+
+class AzureComputeHook(AzureBaseHook):
+ """
+ A hook to interact with Azure Compute to manage Virtual Machines.
+
+ :param azure_conn_id: :ref:`Azure connection id<howto/connection:azure>` of
+ a service principal which will be used to manage virtual machines.
+ """
+
+ conn_name_attr = "azure_conn_id"
+ default_conn_name = "azure_default"
+ conn_type = "azure_compute"
+ hook_name = "Azure Compute"
+
+ def __init__(self, azure_conn_id: str = default_conn_name) -> None:
+ super().__init__(sdk_client=ComputeManagementClient, conn_id=azure_conn_id)
+
+ @cached_property
+ def connection(self) -> ComputeManagementClient:
+ return self.get_conn()
+
+ def get_conn(self) -> Any:
+ """
+ Authenticate the resource using the connection id passed during init.
+
+ :return: the authenticated ComputeManagementClient.
+ """
+ conn = self.get_connection(self.conn_id)
+ tenant = conn.extra_dejson.get("tenantId")
+
+ key_path = conn.extra_dejson.get("key_path")
+ if key_path:
+ if not key_path.endswith(".json"):
+ raise AirflowException("Unrecognised extension for key file.")
Review Comment:
Thanks for implementing the it!
Just a small feedback here, can we avoid using `AirflowException`, and use a better way suggested in contribution guide https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#dont-raise-airflowexception-directly.
Context:
https://lists.apache.org/thread/t8bnhyqy77kq4fk7fj3fmjd5wo9kv6w0
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org