You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "kaxil (via GitHub)" <gi...@apache.org> on 2026/09/21 12:47:38 UTC
[PR] Skip `on_kill()` for triggers reassigned to another triggerer [airflow]
kaxil opened a new pull request, #73454:
URL: https://github.com/apache/airflow/pull/73454
When a triggerer's heartbeat lapses for longer than `triggerer_health_check_threshold`, `Trigger.assign_unassigned` hands its triggers to another triggerer. The old triggerer is often still alive: the runner-silence watchdog deliberately skips heartbeats when the runner subprocess stops responding, and a long GC pause, a stopped container or a network partition do the same. When it recovers, `TriggerRunnerSupervisor.update_triggers` sees those triggers missing from its assignment list and queues them for cancellation with the user-action message, so `run_trigger` calls `on_kill()` on every one of them. Nothing in that path distinguishes "the row was deleted because a user cleared the task" from "the row was reassigned".
For any trigger that cancels remote work from `on_kill()` this cancels a job the new owner has just started polling. The batch triggers in the Anthropic and Common AI providers, and the EMR, Redshift Data, Databricks, Dataproc and BigQuery triggers all do. The `BaseTrigger.on_kill` docstring and the deferring docs both promise it does not fire on redistribution, so this is a contract violation, present since `on_kill()` shipped in 3.3.0 (#65590). The `cancel_triggers` docstring claimed redistribution "goes through a separate path"; there was no such path.
## Solution
Before queueing a cancel, the supervisor asks the DB which of the departed ids still have a trigger row and who owns them now (`Trigger.fetch_assignments`, behind an overridable `fetch_trigger_assignments` hook that owns its own session so `update_triggers` stays DB-free for subclasses without a metadata DB). A row that survives after leaving our list was reassigned, so those ids go to a new `releasing_triggers` set and reach the runner as `TriggerStateSync.to_release`; the runner cancels those asyncio tasks with a distinct sentinel, and `run_trigger` drops the trigger without `on_kill()`. Only a row that is gone, which is what a clear or mark-success/failed produces, takes the existing user-action path. The classification runs once per departed id, since ids already queued are excluded from the next loop's difference. The new field defaults to an empty set, so existing `TriggerStateSync` producers are unaffected.
**Why in the supervisor rather than the runner.** The runner subprocess has no DB access; the supervisor already owns the assignment query and the per-loop `create_session()` in `build_trigger_workloads`, so the one extra `SELECT id ... WHERE id IN (...)` sits next to it, and the runner keeps its existing message-based contract.
## Reproduction
Two triggerers started with `--queues afl297` and `triggerer.queues_enabled=True` (so the standalone triggerer stays out of it), a task on that queue deferring to a trigger whose `on_kill()` appends a line to a file, then `SIGSTOP` on the owning triggerer and its runner child until the trigger row moved, then `SIGCONT`.
| | Old owner after `SIGCONT` | `on_kill()` marker |
|---|---|---|
| main | `Trigger cancelled by user action, invoking on_kill` | written by the old owner's runner while the row already belonged to the new owner and the task was still `deferred` |
| this PR | `Triggers were reassigned to another triggerer, releasing them without on_kill new_owners={3: 8}` and, in the runner, `Trigger reassigned to another triggerer, dropping it without on_kill` | not written |
With this PR, marking the deferred task failed through the REST API still produced the marker from the new owner's runner within seconds, so the user-action path is intact.
## Known gap
Row existence proves the row was reassigned, not that the new owner has started it. If a user clears the task in the window after the old owner classified the trigger as reassigned and before the new owner fetched its workload, neither side calls `on_kill()`: the old owner released, and the new owner skips a trigger whose task instance is gone. This is the same outcome main has when the old owner has actually died, so `on_kill()` on a user action was already best-effort while a failover is in flight. The fix errs toward leaving remote work running rather than cancelling work another triggerer is polling.
---
* Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Skip `on_kill()` for triggers reassigned to another triggerer [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil merged PR #73454:
URL: https://github.com/apache/airflow/pull/73454
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Skip `on_kill()` for triggers reassigned to another triggerer [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #73454:
URL: https://github.com/apache/airflow/pull/73454#discussion_r4080345306
##########
airflow-core/src/airflow/jobs/triggerer_job_runner.py:
##########
@@ -995,8 +1021,22 @@ def update_triggers(self, requested_trigger_ids: set[int]):
self.creating_triggers.extend(workloads_to_create)
if cancel_trigger_ids:
- # Enqueue orphaned triggers for cancellation
- self.cancelling_triggers.update(cancel_trigger_ids)
+ # A trigger leaves our assignment list for one of two reasons, and only the DB can tell
+ # them apart:
+ # - The row is gone: the task instance is no longer deferred on it, because a user
+ # cleared or marked it, or the scheduler timed the deferral out. The runner cancels
+ # with the user-action message and run_trigger() decides whether on_kill() applies.
+ # - The row still exists but points at another triggerer: our heartbeat lapsed and
+ # Trigger.assign_unassigned handed the trigger over. The new owner is polling the
+ # remote work, so the runner must drop it locally *without* on_kill().
Review Comment:
```suggestion
# Only the DB tells the two cases apart: a gone row means the task left the
# deferred state (user action), a surviving row means assign_unassigned handed
# the trigger to a triggerer that is now polling the remote work.
```
##########
airflow-core/src/airflow/jobs/triggerer_job_runner.py:
##########
@@ -513,6 +519,10 @@ class TriggerRunnerSupervisor(WatchedSubprocess):
# FinishedTriggers message
cancelling_triggers: set[int] = attrs.field(factory=set, init=False)
+ # Like cancelling_triggers, but for triggers reassigned to another triggerer (see update_triggers);
+ # the async process drops these without invoking on_kill()
Review Comment:
```suggestion
```
We can remove this ig, the attribute name and the block above say it
##########
airflow-core/tests/unit/jobs/test_triggerer_job.py:
##########
@@ -2744,6 +2804,109 @@ def test_update_triggers_skips_when_ti_has_no_dag_version(session, supervisor_bu
supervisor.stdin.write.assert_not_called()
+def _alive_triggerer_job(session) -> Job:
+ other_job = Job(job_type="TriggererJob")
+ other_job.latest_heartbeat = timezone.utcnow()
+ session.add(other_job)
+ session.flush()
+ return other_job
+
+
+def test_load_triggers_releases_reassigned_trigger_without_user_action_cancel(session, supervisor_builder):
+ """
+ A trigger whose row moved to another triggerer must be dropped locally, not queued
+ as a user-action cancel (which would fire ``on_kill()`` and cancel remote work the new
+ owner is still polling).
+ """
+ trigger = TimeDeltaTrigger(datetime.timedelta(days=7))
+ _, _, trigger_orm, _ = create_trigger_in_db(session, trigger)
+ supervisor = supervisor_builder()
+ supervisor.running_triggers = {trigger_orm.id}
+
+ # Our heartbeat lapsed and another (alive) triggerer took the trigger over.
+ other_job = _alive_triggerer_job(session)
+ session.execute(update(Trigger).where(Trigger.id == trigger_orm.id).values(triggerer_id=other_job.id))
+ session.flush()
+
+ supervisor.load_triggers()
+
+ assert supervisor.cancelling_triggers == set()
+ assert supervisor.releasing_triggers == {trigger_orm.id}
+
+
+def test_load_triggers_cancels_deleted_trigger_as_user_action(session, supervisor_builder):
+ """A trigger whose row is gone (task left the deferred state) takes the user-action path."""
+ trigger = TimeDeltaTrigger(datetime.timedelta(days=7))
+ _, _, trigger_orm, task_instance = create_trigger_in_db(session, trigger)
+ supervisor = supervisor_builder()
+ supervisor.running_triggers = {trigger_orm.id}
+
+ session.execute(
+ update(TaskInstance)
+ .where(TaskInstance.id == task_instance.id)
+ .values(trigger_id=None, state="scheduled")
+ )
+ session.execute(delete(Trigger).where(Trigger.id == trigger_orm.id))
+ session.flush()
+
+ supervisor.load_triggers()
+
+ assert supervisor.cancelling_triggers == {trigger_orm.id}
+ assert supervisor.releasing_triggers == set()
+
+
+def test_update_triggers_splits_cancel_set_by_row_existence(supervisor_builder, mocker):
+ supervisor = supervisor_builder()
+ supervisor.running_triggers = {1, 2, 3}
+ fetch_assignments = mocker.patch.object(
+ TriggerRunnerSupervisor, "fetch_trigger_assignments", autospec=True, return_value={2: 99}
+ )
+
+ supervisor.update_triggers({3})
+
+ fetch_assignments.assert_called_once_with(supervisor, {1, 2})
+ assert supervisor.releasing_triggers == {2}
+ assert supervisor.cancelling_triggers == {1}
+
+ # Already-classified triggers are not re-queried on the next loop.
+ supervisor.update_triggers({3})
+ fetch_assignments.assert_called_once()
+
+
+def test_update_triggers_does_not_query_when_nothing_to_cancel(supervisor_builder, mocker):
+ supervisor = supervisor_builder()
+ supervisor.running_triggers = {1}
+ fetch_assignments = mocker.patch.object(
+ TriggerRunnerSupervisor, "fetch_trigger_assignments", autospec=True
+ )
+
+ supervisor.update_triggers({1})
+
+ fetch_assignments.assert_not_called()
Review Comment:
`cancel_trigger_ids` is an empty set here. The query sits under `if cancel_trigger_ids:`. An empty set is falsy. So the test is asserting that Python skips a false if. That part cannot break.
##########
airflow-core/src/airflow/jobs/triggerer_job_runner.py:
##########
@@ -1442,19 +1486,21 @@ async def create_triggers(self):
async def cancel_triggers(self):
"""
- Drain the to_cancel queue and ensure all triggers that are not in the DB are cancelled.
+ Drain the to_cancel and to_release queues and cancel the matching trigger tasks.
- This allows the cleanup job to delete them.
- Passes "user-action" as the cancel message so that run_trigger() knows to invoke
- on_kill(). Triggers in this queue are always removed because this is the path in which
- the user performed some action on the task. Trigger redistribution goes through a separate
- path.
+ The cancel message tells run_trigger() whether on_kill() applies: ``to_cancel`` holds
+ triggers whose row is gone, ``to_release`` holds triggers reassigned to another
+ triggerer. See TriggerRunnerSupervisor.update_triggers for how the split is made.
Review Comment:
```suggestion
"""Drain to_cancel and to_release, cancelling each task with the message that tells run_trigger() whether on_kill() applies."""
```
shorter?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Skip `on_kill()` for triggers reassigned to another triggerer [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on PR #73454:
URL: https://github.com/apache/airflow/pull/73454#issuecomment-6456418435
> Hello @kaxil ,
>
> Thanks for this fix. Is a backport to the Airflow 3.3.x release line planned, and if so, which version would include it? We observed explicit BigQuery CancelJob calls while a BigQueryInsertJobOperator task was deferred on Airflow 3.3.1. We have not confirmed that trigger reassignment caused our incident, but the behavior described in this PR appears relevant. The fix is not present in the 3.3.1 or 3.3.2 tags.
Unfortunately, it will only be part of 3.4.0 since 3.4.0rc1 is about to be created
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Skip `on_kill()` for triggers reassigned to another triggerer [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #73454:
URL: https://github.com/apache/airflow/pull/73454#discussion_r4086161925
##########
airflow-core/src/airflow/jobs/triggerer_job_runner.py:
##########
@@ -995,8 +1021,22 @@ def update_triggers(self, requested_trigger_ids: set[int]):
self.creating_triggers.extend(workloads_to_create)
if cancel_trigger_ids:
- # Enqueue orphaned triggers for cancellation
- self.cancelling_triggers.update(cancel_trigger_ids)
+ # A trigger leaves our assignment list for one of two reasons, and only the DB can tell
+ # them apart:
+ # - The row is gone: the task instance is no longer deferred on it, because a user
+ # cleared or marked it, or the scheduler timed the deferral out. The runner cancels
+ # with the user-action message and run_trigger() decides whether on_kill() applies.
+ # - The row still exists but points at another triggerer: our heartbeat lapsed and
+ # Trigger.assign_unassigned handed the trigger over. The new owner is polling the
+ # remote work, so the runner must drop it locally *without* on_kill().
Review Comment:
Done in fe9fd85a2b.
##########
airflow-core/src/airflow/jobs/triggerer_job_runner.py:
##########
@@ -513,6 +519,10 @@ class TriggerRunnerSupervisor(WatchedSubprocess):
# FinishedTriggers message
cancelling_triggers: set[int] = attrs.field(factory=set, init=False)
+ # Like cancelling_triggers, but for triggers reassigned to another triggerer (see update_triggers);
+ # the async process drops these without invoking on_kill()
Review Comment:
Removed in fe9fd85a2b.
##########
airflow-core/src/airflow/jobs/triggerer_job_runner.py:
##########
@@ -1442,19 +1486,21 @@ async def create_triggers(self):
async def cancel_triggers(self):
"""
- Drain the to_cancel queue and ensure all triggers that are not in the DB are cancelled.
+ Drain the to_cancel and to_release queues and cancel the matching trigger tasks.
- This allows the cleanup job to delete them.
- Passes "user-action" as the cancel message so that run_trigger() knows to invoke
- on_kill(). Triggers in this queue are always removed because this is the path in which
- the user performed some action on the task. Trigger redistribution goes through a separate
- path.
+ The cancel message tells run_trigger() whether on_kill() applies: ``to_cancel`` holds
+ triggers whose row is gone, ``to_release`` holds triggers reassigned to another
+ triggerer. See TriggerRunnerSupervisor.update_triggers for how the split is made.
Review Comment:
Done in fe9fd85a2b. I kept it on one line but swapped "whether" for "if" so it fits within the 110-character limit.
##########
airflow-core/tests/unit/jobs/test_triggerer_job.py:
##########
@@ -2744,6 +2804,109 @@ def test_update_triggers_skips_when_ti_has_no_dag_version(session, supervisor_bu
supervisor.stdin.write.assert_not_called()
+def _alive_triggerer_job(session) -> Job:
+ other_job = Job(job_type="TriggererJob")
+ other_job.latest_heartbeat = timezone.utcnow()
+ session.add(other_job)
+ session.flush()
+ return other_job
+
+
+def test_load_triggers_releases_reassigned_trigger_without_user_action_cancel(session, supervisor_builder):
+ """
+ A trigger whose row moved to another triggerer must be dropped locally, not queued
+ as a user-action cancel (which would fire ``on_kill()`` and cancel remote work the new
+ owner is still polling).
+ """
+ trigger = TimeDeltaTrigger(datetime.timedelta(days=7))
+ _, _, trigger_orm, _ = create_trigger_in_db(session, trigger)
+ supervisor = supervisor_builder()
+ supervisor.running_triggers = {trigger_orm.id}
+
+ # Our heartbeat lapsed and another (alive) triggerer took the trigger over.
+ other_job = _alive_triggerer_job(session)
+ session.execute(update(Trigger).where(Trigger.id == trigger_orm.id).values(triggerer_id=other_job.id))
+ session.flush()
+
+ supervisor.load_triggers()
+
+ assert supervisor.cancelling_triggers == set()
+ assert supervisor.releasing_triggers == {trigger_orm.id}
+
+
+def test_load_triggers_cancels_deleted_trigger_as_user_action(session, supervisor_builder):
+ """A trigger whose row is gone (task left the deferred state) takes the user-action path."""
+ trigger = TimeDeltaTrigger(datetime.timedelta(days=7))
+ _, _, trigger_orm, task_instance = create_trigger_in_db(session, trigger)
+ supervisor = supervisor_builder()
+ supervisor.running_triggers = {trigger_orm.id}
+
+ session.execute(
+ update(TaskInstance)
+ .where(TaskInstance.id == task_instance.id)
+ .values(trigger_id=None, state="scheduled")
+ )
+ session.execute(delete(Trigger).where(Trigger.id == trigger_orm.id))
+ session.flush()
+
+ supervisor.load_triggers()
+
+ assert supervisor.cancelling_triggers == {trigger_orm.id}
+ assert supervisor.releasing_triggers == set()
+
+
+def test_update_triggers_splits_cancel_set_by_row_existence(supervisor_builder, mocker):
+ supervisor = supervisor_builder()
+ supervisor.running_triggers = {1, 2, 3}
+ fetch_assignments = mocker.patch.object(
+ TriggerRunnerSupervisor, "fetch_trigger_assignments", autospec=True, return_value={2: 99}
+ )
+
+ supervisor.update_triggers({3})
+
+ fetch_assignments.assert_called_once_with(supervisor, {1, 2})
+ assert supervisor.releasing_triggers == {2}
+ assert supervisor.cancelling_triggers == {1}
+
+ # Already-classified triggers are not re-queried on the next loop.
+ supervisor.update_triggers({3})
+ fetch_assignments.assert_called_once()
+
+
+def test_update_triggers_does_not_query_when_nothing_to_cancel(supervisor_builder, mocker):
+ supervisor = supervisor_builder()
+ supervisor.running_triggers = {1}
+ fetch_assignments = mocker.patch.object(
+ TriggerRunnerSupervisor, "fetch_trigger_assignments", autospec=True
+ )
+
+ supervisor.update_triggers({1})
+
+ fetch_assignments.assert_not_called()
Review Comment:
Agreed, removed in fe9fd85a2b.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Skip `on_kill()` for triggers reassigned to another triggerer [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #73454:
URL: https://github.com/apache/airflow/pull/73454#issuecomment-6456346151
### Backport failed to create: v3-4-test. View the failure log <a href='https://github.com/apache/airflow/actions/runs/40603865940'> Run details </a>
Note: As of [Merging PRs targeted for Airflow 3.X](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#merging-prs-targeted-for-airflow-3x)
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
In matter of doubt please ask in [#release-management](https://apache-airflow.slack.com/archives/C03G9H97MM2) Slack channel.
<table>
<tr>
<th>Status</th>
<th>Branch</th>
<th>Result</th>
</tr>
<tr>
<td>❌</td>
<td>v3-4-test</td>
<td><a href="https://github.com/apache/airflow/commit/eebf253"><img src='https://img.shields.io/badge/Commit-eebf253-red' alt='Commit Link'></a></td>
</tr>
</table>
You can attempt to backport this manually by running:
```bash
cherry_picker eebf253 v3-4-test
```
This should apply the commit to the v3-4-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.
After you have resolved the conflicts, you can continue the backport process by running:
```bash
cherry_picker --continue
```
If you don't have cherry-picker installed, see the [installation guide](https://github.com/apache/airflow/blob/main/dev/README.md#how-to-backport-pr-with-cherry-picker-cli).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Skip `on_kill()` for triggers reassigned to another triggerer [airflow]
Posted by "flo076 (via GitHub)" <gi...@apache.org>.
flo076 commented on PR #73454:
URL: https://github.com/apache/airflow/pull/73454#issuecomment-6456528778
Thanks for clarifying. The impact for us is broader than BigQuery. We have observed unexpected interruptions affecting multiple deferrable GCP operators, including BigQuery and Dataproc. In the BigQuery case, audit logs confirm explicit CancelJob calls while the task was deferred; two successive attempts of the same task ended in cancellation, so a retry did not reliably recover it.
We cannot yet prove that trigger reassignment caused every incident. Nevertheless, the core failure mode fixed by this PR affects multiple remote-job triggers, and makes deferrable mode effectively unsafe for our affected workloads on Airflow 3.3.1. Falling back to non-deferrable execution consumes workers for the duration of these jobs.
Given this operational impact, could a 3.3.x backport be reconsidered? If that is not possible, is there a supported mitigation for managed environments that cannot move to 3.4.0 immediately?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Skip `on_kill()` for triggers reassigned to another triggerer [airflow]
Posted by "flo076 (via GitHub)" <gi...@apache.org>.
flo076 commented on PR #73454:
URL: https://github.com/apache/airflow/pull/73454#issuecomment-6456091913
Hello,
Thanks for this fix. Is a backport to the Airflow 3.3.x release line planned, and if so, which version would include it? We observed explicit BigQuery CancelJob calls while a BigQueryInsertJobOperator task was deferred on Airflow 3.3.1. We have not confirmed that trigger reassignment caused our incident, but the behavior described in this PR appears relevant. The fix is not present in the 3.3.1 or 3.3.2 tags.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org