You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "raghavan-arvind (via GitHub)" <gi...@apache.org> on 2026/08/14 15:48:22 UTC
[PR] Add DatadogMonitorSensor waiting on a monitor's overall state [airflow]
raghavan-arvind opened a new pull request, #71625:
URL: https://github.com/apache/airflow/pull/71625
## Why
The existing `DatadogSensor` queries the event stream for transition events in a time window. That shape cannot answer "is this monitor healthy right now": a monitor that entered Alert before the window (or recovered before it) emits no event inside it, so gating a pipeline on monitor health requires reconstructing state from transitions.
This adds `DatadogMonitorSensor`, which checks a monitor's current `overall_state` — an already-alerting or already-recovered monitor is observed regardless of when it last changed state.
## What
- `DatadogMonitorSensor`: waits for a monitor to reach one of `target_states` (default `("OK",)`). Supports poke and reschedule modes via `poke()`, and deferrable mode (default from `[operators] default_deferrable`) via a new `DatadogMonitorTrigger`, following the same shape as other deferrable sensors. In deferrable mode it pokes once before deferring, so an already-satisfied condition never round-trips through the triggerer.
- `DatadogMonitorTrigger`: polls the monitor state off the event loop (`asyncio.to_thread`, since the datadog client is sync) and fires when a target state is reached.
- The datadog library returns API errors as a payload instead of raising; both paths surface them as a clear `AirflowException` (e.g. `Datadog monitor 999999999: ['Monitor not found']`).
- provider.yaml gains a `triggers` section; unit tests for sensor (all modes) and trigger (serialize round-trip, poll-until-target, API error).
## Testing
`uv run --project providers/datadog pytest providers/datadog/tests/unit/datadog`: 19 passed. Also validated end to end in a production-like Airflow 3.3 deployment: canary tasks covering pass / timeout / missing-monitor across poke, reschedule and deferrable modes behave as expected.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add DatadogMonitorSensorAsync waiting on a monitor's overall state [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on PR #71625:
URL: https://github.com/apache/airflow/pull/71625#issuecomment-5942545351
Is there a reason this needs a separate `DatadogMonitorSensorAsync` class rather than `deferrable=True` on the existing `DatadogSensor`? That's the pattern the project has moved to (`TimeSensorAsync` is deprecated in favor of `TimeSensor(deferrable=True)`), and a separate `*Async` sensor plus a new `triggers` module are permanent provider surface. The `resume_execution` override that skips the base sensor's soft-fail handling also depends on #71256, which is closed. Is there an issue or a user asking for deferrable Datadog monitors? Converting to draft until there is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add DatadogMonitorSensor waiting on a monitor's overall state [airflow]
Posted by "raghavan-arvind (via GitHub)" <gi...@apache.org>.
raghavan-arvind commented on PR #71625:
URL: https://github.com/apache/airflow/pull/71625#issuecomment-5295863106
Would like to only support `deferrable` for this sensor -- is that okay? Any existing patterns for this? For retrying transient errors, `500`, `429`, the semantics of changing the poll interval temporarily into an exponential backoff are more straightforward that way
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add DatadogMonitorSensor waiting on a monitor's overall state [airflow]
Posted by "raghavan-arvind (via GitHub)" <gi...@apache.org>.
raghavan-arvind commented on code in PR #71625:
URL: https://github.com/apache/airflow/pull/71625#discussion_r3785513236
##########
providers/datadog/src/airflow/providers/datadog/sensors/datadog.py:
##########
@@ -98,3 +119,80 @@ def poke(self, context: Context) -> bool:
# If no check was inserted, assume any event that matched yields true.
return bool(response)
+
+
+class DatadogMonitorSensor(BaseSensorOperator):
+ """
+ Waits for a Datadog monitor to reach one of the target states.
+
+ :param monitor_id: The id of the Datadog monitor to watch.
+ :param target_states: Monitor overall states that complete the wait
+ (e.g. ``("OK",)`` or ``("Alert", "Warn")``).
+ :param datadog_conn_id: The connection to datadog, containing metadata for api keys.
+ :param deferrable: Run sensor in deferrable mode.
+ """
+
+ ui_color = "#66c3dd"
+
+ def __init__(
+ self,
+ *,
+ monitor_id: int,
+ target_states: Sequence[str] = ("OK",),
+ datadog_conn_id: str = "datadog_default",
+ deferrable: bool = conf.getboolean("operators", "default_deferrable", fallback=False),
+ **kwargs,
+ ) -> None:
+ super().__init__(**kwargs)
+ self.monitor_id = monitor_id
+ self.target_states = target_states
+ self.datadog_conn_id = datadog_conn_id
+ self.deferrable = deferrable
+
+ def poke(self, context: Context) -> bool:
+ state = get_monitor_state(self.monitor_id, self.datadog_conn_id)
+ self.log.info("Monitor %s overall_state=%s", self.monitor_id, state)
+ return state in self.target_states
+
+ def execute(self, context: Context) -> None:
+ if not self.deferrable:
+ super().execute(context)
+ return
+ if self.poke(context):
+ return
+ self.defer(
+ trigger=DatadogMonitorTrigger(
+ monitor_id=self.monitor_id,
+ target_states=self.target_states,
+ datadog_conn_id=self.datadog_conn_id,
+ poke_interval=self.poke_interval,
+ ),
+ method_name="execute_complete",
+ timeout=timedelta(seconds=self.timeout),
+ )
+
+ def execute_complete(self, context: Context, event: dict[str, Any]) -> None:
+ if event.get("status") != "success":
+ raise AirflowException(f"DatadogMonitorTrigger failed: {event}")
+ self.log.info("Monitor %s reached state %s", self.monitor_id, event.get("state"))
+
+ def resume_execution(self, next_method: str, next_kwargs: dict[str, Any] | None, context: Context):
+ """
+ Resume from deferral, applying ``soft_fail`` only to timeouts.
+
+ ``BaseSensorOperator.resume_execution`` converts any deferral-path failure into a skip when
+ ``soft_fail`` is set, including trigger crashes (e.g. a bad monitor id or auth failure),
+ which silently skips the downstream branch. This sensor instead keeps parity with poke and
+ reschedule modes, where only timeouts are skippable and other errors fail the task.
+ """
+ try:
+ return super(BaseSensorOperator, self).resume_execution(next_method, next_kwargs, context)
+ except TaskDeferralError as e:
+ timed_out = isinstance(e, TaskDeferralTimeout) or str(e) == "Trigger/execution timeout"
+ if timed_out and self.soft_fail:
+ raise AirflowSkipException(str(e)) from e
+ if getattr(self, "never_fail", False):
+ raise AirflowSkipException(str(e)) from e
+ if timed_out:
+ raise AirflowSensorTimeout(*e.args) from e
+ raise
Review Comment:
Ideally we can remove this and merge some variant of #71256 first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add DatadogMonitorSensor waiting on a monitor's overall state [airflow]
Posted by "raghavan-arvind (via GitHub)" <gi...@apache.org>.
raghavan-arvind commented on PR #71625:
URL: https://github.com/apache/airflow/pull/71625#issuecomment-5295363027
Verified all these sensors return the correct result:
```
# Datadog sensor canaries: every (case x mode) combination, alerting when
# the outcome differs from the expected one encoded in the task name.
DATADOG_CANARY_CASES = {
# name -> (monitor_id, target_states, soft_fail, expected outcome)
'ok_should_pass': (DATADOG_OK_MONITOR_ID, ('OK',), False, 'success'),
'alert_timeout_should_fail': (DATADOG_OK_MONITOR_ID, ('Alert',), False, 'failed'),
'alert_timeout_soft_fail_should_skip': (DATADOG_OK_MONITOR_ID, ('Alert',), True, 'skipped'),
'missing_should_fail': (DATADOG_INVALID_MONITOR_ID, ('OK',), False, 'failed'),
'missing_soft_fail_should_fail': (DATADOG_INVALID_MONITOR_ID, ('OK',), True, 'failed'),
}
for case_name, (monitor_id, target_states, soft_fail, expected) in DATADOG_CANARY_CASES.items():
for sensor_mode in ('poke', 'reschedule', 'deferrable'):
callbacks = {
'success': {},
'failed': {
'on_failure_callback': None,
'on_success_callback': alert_on_failure,
'on_skipped_callback': alert_on_failure,
},
'skipped': {'on_success_callback': alert_on_failure},
}[expected]
DatadogMonitorSensor(
dag=dag,
task_id=f'datadog_monitor_{case_name}_{sensor_mode}',
monitor_id=monitor_id,
target_states=target_states,
soft_fail=soft_fail,
timeout=timedelta(seconds=90),
poke_interval=30,
deferrable=sensor_mode == 'deferrable',
**({'mode': sensor_mode} if sensor_mode != 'deferrable' else {}),
**callbacks,
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add DatadogMonitorSensor waiting on a monitor's overall state [airflow]
Posted by "raghavan-arvind (via GitHub)" <gi...@apache.org>.
raghavan-arvind commented on PR #71625:
URL: https://github.com/apache/airflow/pull/71625#issuecomment-5295611170
The soft_fail tests are reliant on some fix like:
https://github.com/apache/airflow/pull/71256
Would like to discuss that PR first before this one
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add DatadogMonitorSensorAsync waiting on a monitor's overall state [airflow]
Posted by "github-actions[bot] (via GitHub) via commits" <co...@airflow.apache.org>.
github-actions[bot] commented on PR #71625:
URL: https://github.com/apache/airflow/pull/71625#issuecomment-6903866764
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org