You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Vamsi-klu (via GitHub)" <gi...@apache.org> on 2026/08/18 00:58:34 UTC
[PR] Add deferrable mode to Databricks SQL warehouse operators [airflow]
Vamsi-klu opened a new pull request, #71752:
URL: https://github.com/apache/airflow/pull/71752
Phase 1 start/stop operators from #70088 hold a worker for the whole warehouse wait. This adds the agreed additive deferrable path so those waits can run on the triggerer without changing `wait_for_termination`, `polling_period_seconds`, or `timeout`.
- `a_get_warehouse_state` on `DatabricksHook`
- `DatabricksWarehouseStateTrigger` with serialize, monotonic timeout, and retry_args validation
- `deferrable` on both start and stop operators (`default_deferrable` pattern)
- Restored `WarehouseState.to_json` / `from_json` for trigger events
- Phase 1 wait semantics preserved (stale `STOPPED`, deletion, in-flight poll after deadline)
related: #70088
related: #21377
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Cursor Grok 4.6
Generated-by: Cursor Grok 4.6 following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
Made with [Cursor](https://cursor.com)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add deferrable mode to Databricks SQL warehouse operators [airflow]
Posted by "Vamsi-klu (via GitHub)" <gi...@apache.org>.
Vamsi-klu commented on code in PR #71752:
URL: https://github.com/apache/airflow/pull/71752#discussion_r3813938306
##########
providers/databricks/src/airflow/providers/databricks/triggers/databricks.py:
##########
@@ -336,3 +336,128 @@ async def run(self):
}
)
return
+
+
+class DatabricksWarehouseStateTrigger(BaseTrigger):
+ """
+ Poll a Databricks SQL warehouse until it reaches a target lifecycle state.
+
+ :param warehouse_id: ID of the Databricks SQL warehouse.
+ :param target_state: Lifecycle state to wait for (``RUNNING`` or ``STOPPED``).
+ :param databricks_conn_id: Reference to the :ref:`Databricks connection <howto/connection:databricks>`.
+ :param timeout: Maximum number of seconds to wait after the trigger starts polling.
+ The deadline uses ``time.monotonic()`` locally so it survives trigger serialization
+ without depending on wall-clock time.
+ :param polling_period_seconds: Controls the rate of the poll for the warehouse state.
+ By default, the trigger will poll every 30 seconds.
+ :param retry_limit: The number of times to retry the connection in case of service outages.
+ :param retry_delay: Minimum wait in seconds between retryable attempts when using the
+ default retry strategy. The wait uses exponential backoff (doubling after each
+ failure, capped at ``2 ** retry_limit`` seconds). May be a floating point number.
+ :param retry_args: An optional dictionary with arguments passed to ``tenacity.Retrying`` class.
+ :param caller: The name of the operator that is calling the hook.
+ """
+
+ def __init__(
+ self,
+ warehouse_id: str,
+ target_state: str,
+ databricks_conn_id: str,
+ timeout: float,
+ polling_period_seconds: int = 30,
+ retry_limit: int = 3,
+ retry_delay: int = 10,
+ retry_args: dict[Any, Any] | None = None,
+ caller: str = "DatabricksWarehouseStateTrigger",
+ ) -> None:
+ super().__init__()
+ # Trigger kwargs cross Airflow's serialization boundary, so fail before storing invalid
+ # trigger state or surfacing a generic serializer error without Databricks-specific guidance.
+ validate_deferrable_databricks_retry_args(retry_args, owner=caller)
+ self.warehouse_id = warehouse_id
+ self.target_state = target_state
+ self.databricks_conn_id = databricks_conn_id
+ self.timeout = timeout
+ self.polling_period_seconds = polling_period_seconds
+ self.retry_limit = retry_limit
+ self.retry_delay = retry_delay
+ self.retry_args = retry_args
+ self.caller = caller
+ self.hook = DatabricksHook(
+ databricks_conn_id,
+ retry_limit=self.retry_limit,
+ retry_delay=self.retry_delay,
+ retry_args=retry_args,
+ caller=caller,
+ )
+
+ def serialize(self) -> tuple[str, dict[str, Any]]:
+ return (
+ "airflow.providers.databricks.triggers.databricks.DatabricksWarehouseStateTrigger",
+ {
+ "warehouse_id": self.warehouse_id,
+ "target_state": self.target_state,
+ "databricks_conn_id": self.databricks_conn_id,
+ "timeout": self.timeout,
+ "polling_period_seconds": self.polling_period_seconds,
+ "retry_limit": self.retry_limit,
+ "retry_delay": self.retry_delay,
+ "retry_args": self.retry_args,
+ "caller": self.caller,
+ },
+ )
+
+ async def on_kill(self) -> None:
+ # Warehouses have no cancel-start/stop API. Clearing a deferred start must not stop the
+ # warehouse — the Dag author may still want it running after the wait is abandoned.
+ self.log.info(
+ "Databricks SQL warehouse %s wait cancelled; leaving warehouse state unchanged.",
+ self.warehouse_id,
+ )
+
+ def _build_trigger_event(
+ self, *, status: str, last_state: str, state: WarehouseState | None = None
+ ) -> TriggerEvent:
+ payload: dict[str, Any] = {
+ "status": status,
+ "warehouse_id": self.warehouse_id,
+ "target_state": self.target_state,
+ "last_state": last_state,
+ }
+ if state is not None:
+ payload["state"] = state.to_json()
Review Comment:
Kept the serialized state key so the payload matches the RunState and SQLStatementState triggers, which is also why WarehouseState.to_json and from_json came back after the Phase 1 removal. last_state stays as the plain string used in the timeout message. The PR description now calls out that restore from 70088.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add deferrable mode to Databricks SQL warehouse operators [airflow]
Posted by "Vamsi-klu (via GitHub)" <gi...@apache.org>.
Vamsi-klu commented on code in PR #71752:
URL: https://github.com/apache/airflow/pull/71752#discussion_r3813935655
##########
providers/databricks/src/airflow/providers/databricks/hooks/databricks.py:
##########
@@ -799,6 +806,15 @@ def get_warehouse_state(self, warehouse_id: str) -> WarehouseState:
"""
return WarehouseState(self.get_warehouse(warehouse_id)["state"])
+ async def a_get_warehouse_state(self, warehouse_id: str) -> WarehouseState:
Review Comment:
Added a_get_warehouse as the async mirror of get_warehouse. a_get_warehouse_state now goes through that method so the GET path stays in one place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add deferrable mode to Databricks SQL warehouse operators [airflow]
Posted by "Vamsi-klu (via GitHub)" <gi...@apache.org>.
Vamsi-klu commented on PR #71752:
URL: https://github.com/apache/airflow/pull/71752#issuecomment-5343646150
> Thanks — the Phase 1 contract is preserved carefully here, and the state-machine edge cases we went through on #70088 (stale `STOPPED` not terminal, deletion mid-wait, the deadline boundary) are all covered by deterministic tests rather than sleeps.
>
> I validated the deferrable paths against a real workspace: Airflow 3.2.2, this branch's provider installed from source, a dedicated throwaway serverless warehouse (2X-Small), triggerer doing the waiting.
>
> | scenario | task state | evidence |
> | --- | --- | --- |
> | deferrable start from `STOPPED` | deferred -> success | trigger logged `is STARTING; waiting for RUNNING`, warehouse reached `RUNNING` |
> | deferrable stop | deferred -> success | warehouse reached `STOPPED` |
> | start against an already-`RUNNING` warehouse | success, never deferred | no `DeferTask` for the task |
> | `wait_for_termination=False` stop | success, never deferred | transition still requested, warehouse stopped afterwards |
> | `timeout=1` start | deferred -> failed | `DatabricksWarehouseError: ... did not reach RUNNING within 1s; last state: STARTING.` |
>
> So the behavioural claims in the description hold up where I could exercise them. Unit tests reproduce your numbers as well (43 + 14 + 13, and the full provider suite at 922 passed / 12 skipped).
>
> Two things I would like settled before merge, both left inline:
>
> - The changelog entry is inserted under the `7.18.1` header, which is an already-released version, so a shipped release would advertise a feature it does not contain. Per `providers/AGENTS.md` and the `NOTE TO CONTRIBUTORS` block at the top of that file, routine feature entries are collected by the release manager from commit messages anyway — the five lines can just go.
> - The deferrable `timeout` is a duration recomputed inside every `run()`, so it restarts from zero on each triggerer restart or HA rebalance, while the synchronous path fails deterministically. `DatabricksSQLStatementExecutionTrigger` next door serializes an absolute `end_time` for exactly this reason. Either follow that pattern or state the per-run semantics explicitly — right now the docstring and the PR description claim the opposite of what the code does.
>
> The rest is nits, also inline. One out-of-scope note worth being aware of rather than fixing here: these operators' first deferrable surface inherits #71525, so on core 3.0.x the trigger cannot fetch the connection at all (3.1.0+ is fine). Whether that caveat belongs in this page or centrally is a call for the docs discussion happening on #71667.
>
Thanks for the live workspace check.
The follow-up is on the branch. Changelog Features under 7.18.1 is gone. The trigger now serializes an absolute end_time the same way DatabricksSQLStatementExecutionTrigger does. a_get_warehouse shares the GET path with get_warehouse. The on_kill override and the unpinned test are gone. The typed state JSON stays for sibling trigger symmetry, and the description now mentions the Phase 1 to_json restore.
The 3.0.x connection fetch caveat from 71525 is still out of scope here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add deferrable mode to Databricks SQL warehouse operators [airflow]
Posted by "Vamsi-klu (via GitHub)" <gi...@apache.org>.
Vamsi-klu commented on code in PR #71752:
URL: https://github.com/apache/airflow/pull/71752#discussion_r3813930229
##########
providers/databricks/src/airflow/providers/databricks/triggers/databricks.py:
##########
@@ -336,3 +336,128 @@ async def run(self):
}
)
return
+
+
+class DatabricksWarehouseStateTrigger(BaseTrigger):
+ """
+ Poll a Databricks SQL warehouse until it reaches a target lifecycle state.
+
+ :param warehouse_id: ID of the Databricks SQL warehouse.
+ :param target_state: Lifecycle state to wait for (``RUNNING`` or ``STOPPED``).
+ :param databricks_conn_id: Reference to the :ref:`Databricks connection <howto/connection:databricks>`.
+ :param timeout: Maximum number of seconds to wait after the trigger starts polling.
+ The deadline uses ``time.monotonic()`` locally so it survives trigger serialization
+ without depending on wall-clock time.
+ :param polling_period_seconds: Controls the rate of the poll for the warehouse state.
+ By default, the trigger will poll every 30 seconds.
+ :param retry_limit: The number of times to retry the connection in case of service outages.
+ :param retry_delay: Minimum wait in seconds between retryable attempts when using the
+ default retry strategy. The wait uses exponential backoff (doubling after each
+ failure, capped at ``2 ** retry_limit`` seconds). May be a floating point number.
+ :param retry_args: An optional dictionary with arguments passed to ``tenacity.Retrying`` class.
+ :param caller: The name of the operator that is calling the hook.
+ """
+
+ def __init__(
+ self,
+ warehouse_id: str,
+ target_state: str,
+ databricks_conn_id: str,
+ timeout: float,
+ polling_period_seconds: int = 30,
+ retry_limit: int = 3,
+ retry_delay: int = 10,
+ retry_args: dict[Any, Any] | None = None,
+ caller: str = "DatabricksWarehouseStateTrigger",
+ ) -> None:
+ super().__init__()
+ # Trigger kwargs cross Airflow's serialization boundary, so fail before storing invalid
+ # trigger state or surfacing a generic serializer error without Databricks-specific guidance.
+ validate_deferrable_databricks_retry_args(retry_args, owner=caller)
+ self.warehouse_id = warehouse_id
+ self.target_state = target_state
+ self.databricks_conn_id = databricks_conn_id
+ self.timeout = timeout
+ self.polling_period_seconds = polling_period_seconds
+ self.retry_limit = retry_limit
+ self.retry_delay = retry_delay
+ self.retry_args = retry_args
+ self.caller = caller
+ self.hook = DatabricksHook(
+ databricks_conn_id,
+ retry_limit=self.retry_limit,
+ retry_delay=self.retry_delay,
+ retry_args=retry_args,
+ caller=caller,
+ )
+
+ def serialize(self) -> tuple[str, dict[str, Any]]:
+ return (
+ "airflow.providers.databricks.triggers.databricks.DatabricksWarehouseStateTrigger",
+ {
+ "warehouse_id": self.warehouse_id,
+ "target_state": self.target_state,
+ "databricks_conn_id": self.databricks_conn_id,
+ "timeout": self.timeout,
+ "polling_period_seconds": self.polling_period_seconds,
+ "retry_limit": self.retry_limit,
+ "retry_delay": self.retry_delay,
+ "retry_args": self.retry_args,
+ "caller": self.caller,
+ },
+ )
+
+ async def on_kill(self) -> None:
+ # Warehouses have no cancel-start/stop API. Clearing a deferred start must not stop the
+ # warehouse — the Dag author may still want it running after the wait is abandoned.
+ self.log.info(
+ "Databricks SQL warehouse %s wait cancelled; leaving warehouse state unchanged.",
+ self.warehouse_id,
+ )
+
+ def _build_trigger_event(
+ self, *, status: str, last_state: str, state: WarehouseState | None = None
+ ) -> TriggerEvent:
+ payload: dict[str, Any] = {
+ "status": status,
+ "warehouse_id": self.warehouse_id,
+ "target_state": self.target_state,
+ "last_state": last_state,
+ }
+ if state is not None:
+ payload["state"] = state.to_json()
+ return TriggerEvent(payload)
+
+ async def run(self):
+ async with self.hook:
+ deadline = time.monotonic() + self.timeout
Review Comment:
Followed DatabricksSQLStatementExecutionTrigger. The operator now records an absolute end_time from timeout when it defers, serialize carries that timestamp, and run() compares it with time.time(). A triggerer restart no longer resets the wait. The how-to and the trigger docstring now describe that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add deferrable mode to Databricks SQL warehouse operators [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #71752:
URL: https://github.com/apache/airflow/pull/71752#issuecomment-5348470493
can you please update the comment in https://github.com/apache/airflow/issues/21377#issuecomment-5014011201 with checklist to the actual steps in the plan and which PR solved which step? That way we can easily track what is left open
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add deferrable mode to Databricks SQL warehouse operators [airflow]
Posted by "Vamsi-klu (via GitHub)" <gi...@apache.org>.
Vamsi-klu commented on PR #71752:
URL: https://github.com/apache/airflow/pull/71752#issuecomment-5322168384
@eladkal @moomidaani this is the #70088 deferrable follow-up — could you review when you have a chance?
---
Drafted-by: Cursor Grok 4.6 (no human review before posting)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add deferrable mode to Databricks SQL warehouse operators [airflow]
Posted by "Vamsi-klu (via GitHub)" <gi...@apache.org>.
Vamsi-klu commented on PR #71752:
URL: https://github.com/apache/airflow/pull/71752#issuecomment-5389749437
Updated the tracking comment: https://github.com/apache/airflow/issues/21377#issuecomment-5014011201
#70088 is Phase 1. This PR is the deferrable follow-up. Create/delete, name lookup, edit, and a live system-test Dag are still open.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add deferrable mode to Databricks SQL warehouse operators [airflow]
Posted by "Vamsi-klu (via GitHub)" <gi...@apache.org>.
Vamsi-klu commented on code in PR #71752:
URL: https://github.com/apache/airflow/pull/71752#discussion_r3813932727
##########
providers/databricks/src/airflow/providers/databricks/triggers/databricks.py:
##########
@@ -336,3 +336,128 @@ async def run(self):
}
)
return
+
+
+class DatabricksWarehouseStateTrigger(BaseTrigger):
+ """
+ Poll a Databricks SQL warehouse until it reaches a target lifecycle state.
+
+ :param warehouse_id: ID of the Databricks SQL warehouse.
+ :param target_state: Lifecycle state to wait for (``RUNNING`` or ``STOPPED``).
+ :param databricks_conn_id: Reference to the :ref:`Databricks connection <howto/connection:databricks>`.
+ :param timeout: Maximum number of seconds to wait after the trigger starts polling.
+ The deadline uses ``time.monotonic()`` locally so it survives trigger serialization
+ without depending on wall-clock time.
+ :param polling_period_seconds: Controls the rate of the poll for the warehouse state.
+ By default, the trigger will poll every 30 seconds.
+ :param retry_limit: The number of times to retry the connection in case of service outages.
+ :param retry_delay: Minimum wait in seconds between retryable attempts when using the
+ default retry strategy. The wait uses exponential backoff (doubling after each
+ failure, capped at ``2 ** retry_limit`` seconds). May be a floating point number.
+ :param retry_args: An optional dictionary with arguments passed to ``tenacity.Retrying`` class.
+ :param caller: The name of the operator that is calling the hook.
+ """
+
+ def __init__(
+ self,
+ warehouse_id: str,
+ target_state: str,
+ databricks_conn_id: str,
+ timeout: float,
+ polling_period_seconds: int = 30,
+ retry_limit: int = 3,
+ retry_delay: int = 10,
+ retry_args: dict[Any, Any] | None = None,
+ caller: str = "DatabricksWarehouseStateTrigger",
+ ) -> None:
+ super().__init__()
+ # Trigger kwargs cross Airflow's serialization boundary, so fail before storing invalid
+ # trigger state or surfacing a generic serializer error without Databricks-specific guidance.
+ validate_deferrable_databricks_retry_args(retry_args, owner=caller)
+ self.warehouse_id = warehouse_id
+ self.target_state = target_state
+ self.databricks_conn_id = databricks_conn_id
+ self.timeout = timeout
+ self.polling_period_seconds = polling_period_seconds
+ self.retry_limit = retry_limit
+ self.retry_delay = retry_delay
+ self.retry_args = retry_args
+ self.caller = caller
+ self.hook = DatabricksHook(
+ databricks_conn_id,
+ retry_limit=self.retry_limit,
+ retry_delay=self.retry_delay,
+ retry_args=retry_args,
+ caller=caller,
+ )
+
+ def serialize(self) -> tuple[str, dict[str, Any]]:
+ return (
+ "airflow.providers.databricks.triggers.databricks.DatabricksWarehouseStateTrigger",
+ {
+ "warehouse_id": self.warehouse_id,
+ "target_state": self.target_state,
+ "databricks_conn_id": self.databricks_conn_id,
+ "timeout": self.timeout,
+ "polling_period_seconds": self.polling_period_seconds,
+ "retry_limit": self.retry_limit,
+ "retry_delay": self.retry_delay,
+ "retry_args": self.retry_args,
+ "caller": self.caller,
+ },
+ )
+
+ async def on_kill(self) -> None:
Review Comment:
Removed the on_kill override and the test that only asserted the base no-op. The class docstring now says we leave warehouse state unchanged because Databricks has no cancel API for start or stop.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add deferrable mode to Databricks SQL warehouse operators [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal merged PR #71752:
URL: https://github.com/apache/airflow/pull/71752
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add deferrable mode to Databricks SQL warehouse operators [airflow]
Posted by "Vamsi-klu (via GitHub)" <gi...@apache.org>.
Vamsi-klu commented on code in PR #71752:
URL: https://github.com/apache/airflow/pull/71752#discussion_r3813927139
##########
providers/databricks/docs/changelog.rst:
##########
@@ -29,6 +29,11 @@ Changelog
7.18.1
......
+Features
+~~~~~~~~
+
+* ``Add deferrable mode to Databricks SQL warehouse start and stop operators``
Review Comment:
Dropped the Features block under 7.18.1. The commit message already carries the note for the release manager.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add deferrable mode to Databricks SQL warehouse operators [airflow]
Posted by "Vamsi-klu (via GitHub)" <gi...@apache.org>.
Vamsi-klu commented on PR #71752:
URL: https://github.com/apache/airflow/pull/71752#issuecomment-5347386615
@eladkal as @moomindani approved the PR, Can i get your thoughts on this please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add deferrable mode to Databricks SQL warehouse operators [airflow]
Posted by "moomindani (via GitHub)" <gi...@apache.org>.
moomindani commented on code in PR #71752:
URL: https://github.com/apache/airflow/pull/71752#discussion_r3813498696
##########
providers/databricks/src/airflow/providers/databricks/triggers/databricks.py:
##########
@@ -336,3 +336,128 @@ async def run(self):
}
)
return
+
+
+class DatabricksWarehouseStateTrigger(BaseTrigger):
+ """
+ Poll a Databricks SQL warehouse until it reaches a target lifecycle state.
+
+ :param warehouse_id: ID of the Databricks SQL warehouse.
+ :param target_state: Lifecycle state to wait for (``RUNNING`` or ``STOPPED``).
+ :param databricks_conn_id: Reference to the :ref:`Databricks connection <howto/connection:databricks>`.
+ :param timeout: Maximum number of seconds to wait after the trigger starts polling.
+ The deadline uses ``time.monotonic()`` locally so it survives trigger serialization
+ without depending on wall-clock time.
+ :param polling_period_seconds: Controls the rate of the poll for the warehouse state.
+ By default, the trigger will poll every 30 seconds.
+ :param retry_limit: The number of times to retry the connection in case of service outages.
+ :param retry_delay: Minimum wait in seconds between retryable attempts when using the
+ default retry strategy. The wait uses exponential backoff (doubling after each
+ failure, capped at ``2 ** retry_limit`` seconds). May be a floating point number.
+ :param retry_args: An optional dictionary with arguments passed to ``tenacity.Retrying`` class.
+ :param caller: The name of the operator that is calling the hook.
+ """
+
+ def __init__(
+ self,
+ warehouse_id: str,
+ target_state: str,
+ databricks_conn_id: str,
+ timeout: float,
+ polling_period_seconds: int = 30,
+ retry_limit: int = 3,
+ retry_delay: int = 10,
+ retry_args: dict[Any, Any] | None = None,
+ caller: str = "DatabricksWarehouseStateTrigger",
+ ) -> None:
+ super().__init__()
+ # Trigger kwargs cross Airflow's serialization boundary, so fail before storing invalid
+ # trigger state or surfacing a generic serializer error without Databricks-specific guidance.
+ validate_deferrable_databricks_retry_args(retry_args, owner=caller)
+ self.warehouse_id = warehouse_id
+ self.target_state = target_state
+ self.databricks_conn_id = databricks_conn_id
+ self.timeout = timeout
+ self.polling_period_seconds = polling_period_seconds
+ self.retry_limit = retry_limit
+ self.retry_delay = retry_delay
+ self.retry_args = retry_args
+ self.caller = caller
+ self.hook = DatabricksHook(
+ databricks_conn_id,
+ retry_limit=self.retry_limit,
+ retry_delay=self.retry_delay,
+ retry_args=retry_args,
+ caller=caller,
+ )
+
+ def serialize(self) -> tuple[str, dict[str, Any]]:
+ return (
+ "airflow.providers.databricks.triggers.databricks.DatabricksWarehouseStateTrigger",
+ {
+ "warehouse_id": self.warehouse_id,
+ "target_state": self.target_state,
+ "databricks_conn_id": self.databricks_conn_id,
+ "timeout": self.timeout,
+ "polling_period_seconds": self.polling_period_seconds,
+ "retry_limit": self.retry_limit,
+ "retry_delay": self.retry_delay,
+ "retry_args": self.retry_args,
+ "caller": self.caller,
+ },
+ )
+
+ async def on_kill(self) -> None:
+ # Warehouses have no cancel-start/stop API. Clearing a deferred start must not stop the
+ # warehouse — the Dag author may still want it running after the wait is abandoned.
+ self.log.info(
+ "Databricks SQL warehouse %s wait cancelled; leaving warehouse state unchanged.",
+ self.warehouse_id,
+ )
+
+ def _build_trigger_event(
+ self, *, status: str, last_state: str, state: WarehouseState | None = None
+ ) -> TriggerEvent:
+ payload: dict[str, Any] = {
+ "status": status,
+ "warehouse_id": self.warehouse_id,
+ "target_state": self.target_state,
+ "last_state": last_state,
+ }
+ if state is not None:
+ payload["state"] = state.to_json()
+ return TriggerEvent(payload)
+
+ async def run(self):
+ async with self.hook:
+ deadline = time.monotonic() + self.timeout
Review Comment:
The deadline is computed here, inside `run()`, and `serialize()` carries `timeout` as a duration — so after a triggerer restart, deploy, or HA rebalance the wait starts a fresh full `timeout` window, and a warehouse that never reaches its target can stay deferred well past the configured value. The synchronous path fails deterministically instead.
`DatabricksSQLStatementExecutionTrigger` in this same file takes an absolute `end_time` (serialized at line 281, consumed at line 299) that the operator computes, precisely so the deadline survives serialization. Following that pattern here would make the two consistent; if the per-run semantics are deliberate instead, they are worth stating in the docstring and the how-to.
Related: the docstring above says the monotonic deadline "survives trigger serialization", and the PR description repeats it. It is the other way round — a monotonic value cannot be serialized, which is why this line recomputes it.
##########
providers/databricks/src/airflow/providers/databricks/triggers/databricks.py:
##########
@@ -336,3 +336,128 @@ async def run(self):
}
)
return
+
+
+class DatabricksWarehouseStateTrigger(BaseTrigger):
+ """
+ Poll a Databricks SQL warehouse until it reaches a target lifecycle state.
+
+ :param warehouse_id: ID of the Databricks SQL warehouse.
+ :param target_state: Lifecycle state to wait for (``RUNNING`` or ``STOPPED``).
+ :param databricks_conn_id: Reference to the :ref:`Databricks connection <howto/connection:databricks>`.
+ :param timeout: Maximum number of seconds to wait after the trigger starts polling.
+ The deadline uses ``time.monotonic()`` locally so it survives trigger serialization
+ without depending on wall-clock time.
+ :param polling_period_seconds: Controls the rate of the poll for the warehouse state.
+ By default, the trigger will poll every 30 seconds.
+ :param retry_limit: The number of times to retry the connection in case of service outages.
+ :param retry_delay: Minimum wait in seconds between retryable attempts when using the
+ default retry strategy. The wait uses exponential backoff (doubling after each
+ failure, capped at ``2 ** retry_limit`` seconds). May be a floating point number.
+ :param retry_args: An optional dictionary with arguments passed to ``tenacity.Retrying`` class.
+ :param caller: The name of the operator that is calling the hook.
+ """
+
+ def __init__(
+ self,
+ warehouse_id: str,
+ target_state: str,
+ databricks_conn_id: str,
+ timeout: float,
+ polling_period_seconds: int = 30,
+ retry_limit: int = 3,
+ retry_delay: int = 10,
+ retry_args: dict[Any, Any] | None = None,
+ caller: str = "DatabricksWarehouseStateTrigger",
+ ) -> None:
+ super().__init__()
+ # Trigger kwargs cross Airflow's serialization boundary, so fail before storing invalid
+ # trigger state or surfacing a generic serializer error without Databricks-specific guidance.
+ validate_deferrable_databricks_retry_args(retry_args, owner=caller)
+ self.warehouse_id = warehouse_id
+ self.target_state = target_state
+ self.databricks_conn_id = databricks_conn_id
+ self.timeout = timeout
+ self.polling_period_seconds = polling_period_seconds
+ self.retry_limit = retry_limit
+ self.retry_delay = retry_delay
+ self.retry_args = retry_args
+ self.caller = caller
+ self.hook = DatabricksHook(
+ databricks_conn_id,
+ retry_limit=self.retry_limit,
+ retry_delay=self.retry_delay,
+ retry_args=retry_args,
+ caller=caller,
+ )
+
+ def serialize(self) -> tuple[str, dict[str, Any]]:
+ return (
+ "airflow.providers.databricks.triggers.databricks.DatabricksWarehouseStateTrigger",
+ {
+ "warehouse_id": self.warehouse_id,
+ "target_state": self.target_state,
+ "databricks_conn_id": self.databricks_conn_id,
+ "timeout": self.timeout,
+ "polling_period_seconds": self.polling_period_seconds,
+ "retry_limit": self.retry_limit,
+ "retry_delay": self.retry_delay,
+ "retry_args": self.retry_args,
+ "caller": self.caller,
+ },
+ )
+
+ async def on_kill(self) -> None:
Review Comment:
`BaseTrigger.on_kill` is already a documented no-op, so this override only adds a log line. I deleted it entirely and `test_on_kill_leaves_warehouse_unchanged` still passed, which means the test does not pin the behaviour the description credits it with.
Either drop both the override and the test and keep the reasoning as a comment, or keep the override and stop describing it as tested. Worth noting the base-class method only exists on newer cores while the provider supports `apache-airflow>=2.11.0`.
##########
providers/databricks/docs/changelog.rst:
##########
@@ -29,6 +29,11 @@ Changelog
7.18.1
......
+Features
+~~~~~~~~
+
+* ``Add deferrable mode to Databricks SQL warehouse start and stop operators``
Review Comment:
This block sits under the `7.18.1` header, which the release manager created for the 2026-08-06 wave — a released version, so its changelog would advertise a feature that release does not contain, and it will collide with the entry regenerated from `git log` for the next version.
`providers/AGENTS.md` and the `NOTE TO CONTRIBUTORS` block at the top of this file both say routine feature entries are collected automatically and direct edits are reserved for breaking/important behaviour notes. Simplest fix is to drop all five lines; the commit message already carries the note.
##########
providers/databricks/src/airflow/providers/databricks/hooks/databricks.py:
##########
@@ -799,6 +806,15 @@ def get_warehouse_state(self, warehouse_id: str) -> WarehouseState:
"""
return WarehouseState(self.get_warehouse(warehouse_id)["state"])
+ async def a_get_warehouse_state(self, warehouse_id: str) -> WarehouseState:
Review Comment:
This inlines the endpoint tuple, while the synchronous `get_warehouse_state` goes through `get_warehouse()`. They are identical today — same `2.0` constant, same `WarehouseState(response["state"])` parsing — but that is now two copies, so a later change to `get_warehouse` (query params, error mapping, an API version bump) would silently miss the async path. An `a_get_warehouse` mirror, or a small shared endpoint builder, keeps them in step.
##########
providers/databricks/src/airflow/providers/databricks/triggers/databricks.py:
##########
@@ -336,3 +336,128 @@ async def run(self):
}
)
return
+
+
+class DatabricksWarehouseStateTrigger(BaseTrigger):
+ """
+ Poll a Databricks SQL warehouse until it reaches a target lifecycle state.
+
+ :param warehouse_id: ID of the Databricks SQL warehouse.
+ :param target_state: Lifecycle state to wait for (``RUNNING`` or ``STOPPED``).
+ :param databricks_conn_id: Reference to the :ref:`Databricks connection <howto/connection:databricks>`.
+ :param timeout: Maximum number of seconds to wait after the trigger starts polling.
+ The deadline uses ``time.monotonic()`` locally so it survives trigger serialization
+ without depending on wall-clock time.
+ :param polling_period_seconds: Controls the rate of the poll for the warehouse state.
+ By default, the trigger will poll every 30 seconds.
+ :param retry_limit: The number of times to retry the connection in case of service outages.
+ :param retry_delay: Minimum wait in seconds between retryable attempts when using the
+ default retry strategy. The wait uses exponential backoff (doubling after each
+ failure, capped at ``2 ** retry_limit`` seconds). May be a floating point number.
+ :param retry_args: An optional dictionary with arguments passed to ``tenacity.Retrying`` class.
+ :param caller: The name of the operator that is calling the hook.
+ """
+
+ def __init__(
+ self,
+ warehouse_id: str,
+ target_state: str,
+ databricks_conn_id: str,
+ timeout: float,
+ polling_period_seconds: int = 30,
+ retry_limit: int = 3,
+ retry_delay: int = 10,
+ retry_args: dict[Any, Any] | None = None,
+ caller: str = "DatabricksWarehouseStateTrigger",
+ ) -> None:
+ super().__init__()
+ # Trigger kwargs cross Airflow's serialization boundary, so fail before storing invalid
+ # trigger state or surfacing a generic serializer error without Databricks-specific guidance.
+ validate_deferrable_databricks_retry_args(retry_args, owner=caller)
+ self.warehouse_id = warehouse_id
+ self.target_state = target_state
+ self.databricks_conn_id = databricks_conn_id
+ self.timeout = timeout
+ self.polling_period_seconds = polling_period_seconds
+ self.retry_limit = retry_limit
+ self.retry_delay = retry_delay
+ self.retry_args = retry_args
+ self.caller = caller
+ self.hook = DatabricksHook(
+ databricks_conn_id,
+ retry_limit=self.retry_limit,
+ retry_delay=self.retry_delay,
+ retry_args=retry_args,
+ caller=caller,
+ )
+
+ def serialize(self) -> tuple[str, dict[str, Any]]:
+ return (
+ "airflow.providers.databricks.triggers.databricks.DatabricksWarehouseStateTrigger",
+ {
+ "warehouse_id": self.warehouse_id,
+ "target_state": self.target_state,
+ "databricks_conn_id": self.databricks_conn_id,
+ "timeout": self.timeout,
+ "polling_period_seconds": self.polling_period_seconds,
+ "retry_limit": self.retry_limit,
+ "retry_delay": self.retry_delay,
+ "retry_args": self.retry_args,
+ "caller": self.caller,
+ },
+ )
+
+ async def on_kill(self) -> None:
+ # Warehouses have no cancel-start/stop API. Clearing a deferred start must not stop the
+ # warehouse — the Dag author may still want it running after the wait is abandoned.
+ self.log.info(
+ "Databricks SQL warehouse %s wait cancelled; leaving warehouse state unchanged.",
+ self.warehouse_id,
+ )
+
+ def _build_trigger_event(
+ self, *, status: str, last_state: str, state: WarehouseState | None = None
+ ) -> TriggerEvent:
+ payload: dict[str, Any] = {
+ "status": status,
+ "warehouse_id": self.warehouse_id,
+ "target_state": self.target_state,
+ "last_state": last_state,
+ }
+ if state is not None:
+ payload["state"] = state.to_json()
Review Comment:
`payload["state"]` is always `json.dumps({"state": payload["last_state"]})`, and `execute_complete` then does `WarehouseState.from_json(event["state"]).state` where `event["last_state"]` would do — while hard-indexing `event["state"]` and `.get()`-ing everything else. Worth either dropping the redundant key or keeping it deliberately for symmetry with the sibling triggers and saying so.
On the `to_json`/`from_json` restore: this closes the loop from #70088, where their removal was requested because nothing used them and you noted they would return with this trigger. That is the strongest justification for the change and the description does not mention it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org