You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jinshen-cn (via GitHub)" <gi...@apache.org> on 2026/04/24 19:25:52 UTC
[I] Airflow 3.2.0 scheduler/triggerer deadlock on task_instance due to concurrent updates of deferrable tasks [airflow]
jinshen-cn opened a new issue, #65818:
URL: https://github.com/apache/airflow/issues/65818
### Under which category would you file this issue?
Airflow Core
### Apache Airflow version
3.2.0
### What happened and how to reproduce it?
# Description
After upgrading from Airflow 3.1.7 → 3.2.0, we are consistently observing MySQL deadlocks between the scheduler and triggerer when processing deferrable tasks.
This did not occur in 3.1.7 under the same workload.
# Environment
- Airflow version: 3.2.0
- Previous version (no issue): 3.1.7
- Executor: CeleryExecutor
- DB: MySQL
- Scheduler replicas: 3
- Triggerer: 1 instance
- Workload: heavy use of deferrable operators (sensors / async tasks)
# Symptoms
- Scheduler/Trigger crashes or restarts due to DB deadlocks
- Deadlocks consistently involve task_instance table
- System becomes unstable under load
Example deadlock pattern:
```
UPDATE task_instance
SET updated_at=..., trigger_id=NULL
WHERE task_instance.state != 'deferred'
AND task_instance.trigger_id IS NOT NULL
```
conflicting with:
```
UPDATE task_instance
SET state='scheduled', trigger_id=NULL,
next_method='__fail__', next_kwargs=...
WHERE task_instance.state = 'deferred'
AND task_instance.trigger_timeout < now()
```
# Root Cause Analysis
## Key observation
In Airflow 3.2.0, both scheduler and triggerer mutate task_instance rows for deferrable tasks:
## Triggerer (set-based update)
- Performs bulk UPDATE on deferred tasks that timeout
- Updates:
- state
- trigger_id
- next_method
- next_kwargs
## Scheduler (callback-driven updates)
- Processes executor callbacks via:
```
callback = session.get(Callback, callback_id)
callback.run(session=session)
```
- Inside callback.run():
- Loads TaskInstance
- Mutates:
- state
- trigger_id
- other fields
## Result
Two independent writers:
- Triggerer → bulk UPDATE (set-based)
- Scheduler → row-by-row ORM UPDATE
Both target overlapping `task_instance` rows.
## Why this causes deadlocks
- Both queries scan overlapping row sets (even if predicates are logically disjoint)
- Lock acquisition order differs:
- Triggerer: index scan order
- Scheduler: callback / primary key order
- With multiple scheduler replicas, contention increases significantly
Typical pattern:
```
Scheduler: locks row A → waits for row B
Triggerer: locks row B → waits for row A
→ DEADLOCK
```
### What you think should happen instead?
1. Avoid concurrent writes:
- Scheduler should not mutate task_instance fields owned by triggerer
2. Enforce consistent ordering:
- Ensure both components lock rows in deterministic order
3. Batch updates:
- Avoid large scans or uncontrolled ORM flushes
4. Ownership separation:
- Triggerer handles deferred lifecycle exclusively
- Scheduler only consumes results
### Operating System
Ubuntu 22.04.5 LTS
### Deployment
None
### Apache Airflow Provider(s)
_No response_
### Versions of Apache Airflow Providers
_No response_
### Official Helm Chart version
Not Applicable
### Kubernetes Version
_No response_
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow 3.2.0 scheduler/triggerer deadlock on task_instance due to concurrent updates of deferrable tasks [airflow]
Posted by "fedemgp (via GitHub)" <gi...@apache.org>.
fedemgp commented on issue #65818:
URL: https://github.com/apache/airflow/issues/65818#issuecomment-4833368399
is it possible that this issue will be solved after 3.2.2? due to this changes: https://github.com/apache/airflow/issues/65818
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow 3.2.0 scheduler/triggerer deadlock on task_instance due to concurrent updates of deferrable tasks [airflow]
Posted by "uplsh580 (via GitHub)" <gi...@apache.org>.
uplsh580 commented on issue #65818:
URL: https://github.com/apache/airflow/issues/65818#issuecomment-4525328884
We are seeing the same class of deadlock on Airflow 3.1.8 / MySQL with 2 triggerer replicas.
In our case the deadlock happens on a single-row UPDATE through the `Trigger.submit_event` → `handle_event_submit` path (not the bulk UPDATE paths in `SchedulerJobRunner.check_trigger_timeouts` / `Trigger.clean_unused` that #65920 / #65836 target). The exception is not retried anywhere in the call chain (`triggerer_job_runner.handle_events` → `Trigger.submit_event` → `handle_event_submit`), so it propagates up to `TriggerRunnerSupervisor.run` and the triggerer process exits, after which Kubernetes restarts the pod. No task failure — deferred tasks are picked up by the other triggerer — but the restart shows up as an alert.
Environment
- Airflow 3.1.8
- MySQL (InnoDB)
- triggerer replicas: 2
- Workload: deferrable operators (Spark application completion event in this case)
Failing statement
UPDATE task_instance
SET state='scheduled', scheduled_dttm=..., updated_at=..., trigger_id=NULL, next_kwargs=...
WHERE task_instance.id = '019e511c-c507-74d8-ace0-f4c396cc8ef1'
→ `MySQLdb.OperationalError: (1213, 'Deadlock found when trying to get lock; try restarting transaction')`
Stack
airflow/jobs/triggerer_job_runner.py:534 run
airflow/jobs/triggerer_job_runner.py:561 handle_events
airflow/models/trigger.py:252 submit_event
airflow/models/trigger.py:422 handle_event_submit
→ session.flush()
Looking at `airflow-core/src/airflow/models/trigger.py` on the 3.1.8 tag, neither `submit_event` (L239) nor `handle_event_submit` (L394 / L426) carries `@retry_db_transaction` or any try/except, while several deadlock-sensitive paths on the scheduler side already do. Wrapping these entry points (or the per-event loop in `triggerer_job_runner.handle_events`) with `@retry_db_transaction` would also cover this single-row UPDATE case that the existing PRs do not address.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow 3.2.0 scheduler/triggerer deadlock on task_instance due to concurrent updates of deferrable tasks [airflow]
Posted by "uplsh580 (via GitHub)" <gi...@apache.org>.
uplsh580 commented on issue #65818:
URL: https://github.com/apache/airflow/issues/65818#issuecomment-5212428542
Following up on my [earlier comment](https://github.com/apache/airflow/issues/65818#issuecomment-4525328884): we hit the same deadlock again on 2026-08-06, now on **Airflow 3.2.2** — same single-row UPDATE through the `Trigger.submit_event` → `handle_event_submit` path, and the triggerer process died, causing a container restart (the pod itself stayed up).
@fedemgp to answer your question: no — I think 3.2.2 does not fix this. #66412 addresses the comms-channel race inside the triggerer (lock-based serialization → response multiplexing), which is unrelated to this MySQL row-lock deadlock on `task_instance`. We can confirm it still reproduces on 3.2.2.
Environment
- Airflow 3.2.2
- MySQL (InnoDB)
- triggerer replicas: 2
- Workload: deferrable operators (YARN/Spark application completion event)
Failing statement
UPDATE task_instance
SET state='scheduled', scheduled_dttm=..., updated_at=..., trigger_id=NULL, next_kwargs=...
WHERE task_instance.id = '019fd7e27e377324a5f4536ed6640510'
→ MySQLdb.OperationalError: (1213, 'Deadlock found when trying to get lock; try restarting transaction')
Stack (3.2.2)
airflow/jobs/triggerer_job_runner.py:218 _execute
airflow/jobs/triggerer_job_runner.py:612 run
airflow/jobs/triggerer_job_runner.py:655 handle_events
airflow/models/trigger.py:274 submit_event
airflow/models/trigger.py:505 handle_event_submit
→ session.flush()
One additional detail compared to our previous occuer process did not terminate on SIGINT/SIGTERM afterthe exception and had to be SIGKILLed by the supervisor (`Process exited exit_code=-9 signal_sent=SIGKILL`), so triggers thstill running at that point were interrupted mid-fle other replica picked them up), but the restart isnoisy at the alerting level.On 3.2.2 the call chain (`handle_events` → `submit_mit`) still has no `@retry_db_transaction` orequivalent, so a single transient 1213 kills the whix PRs for the bulk-UPDATE paths (#65836, #65920)have gone stale, I'd propose wrapping this per-evennsaction` (matching the existing scheduler-sidepattern) as an incremental fix — it wouldn't removef, but it would stop a single transient deadlock from taking down the triggerer. I'll put up a PR along t
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow 3.2.0 scheduler/triggerer deadlock on task_instance due to concurrent updates of deferrable tasks [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #65818:
URL: https://github.com/apache/airflow/issues/65818#issuecomment-4315722792
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow 3.2.0 scheduler/triggerer deadlock on task_instance due to concurrent updates of deferrable tasks [airflow]
Posted by "shaealh (via GitHub)" <gi...@apache.org>.
shaealh commented on issue #65818:
URL: https://github.com/apache/airflow/issues/65818#issuecomment-4323785264
Opened [this PR](https://github.com/apache/airflow/pull/65920)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow 3.2.0 scheduler/triggerer deadlock on task_instance due to concurrent updates of deferrable tasks [airflow]
Posted by "Vamsi-klu (via GitHub)" <gi...@apache.org>.
Vamsi-klu commented on issue #65818:
URL: https://github.com/apache/airflow/issues/65818#issuecomment-5435506029
Working on the lock-order / ownership split. Not a second retry.
@uplsh580 #71391 (retry on submit_event/submit_failure) is complementary. I will not duplicate that retry. If 71391 merges first, I will rebase and keep @provide_session outside @retry_db_transaction. If 71391 is growing into this ownership split, say so and I will stop.
Adjacent open work I will not compete with:
- #72062 MySQL clean_unused CASCADE race — absorb the DELETE re-check predicates
- #70961 batch Trigger deletes — that PR keeps the TI UPDATE this one drops; we should not fight over the DELETE shape
This PR:
- stop clean_unused from UPDATEing task_instance
- NULL trigger_id on every DEFERRED-exit, including clear_task_instances (today that leftover FK is what the bulk UPDATE papers over)
- scheduler timeout becomes a bounded PK-ordered fallback for unassigned/dead triggerers only (healthy assigned triggerer + past trigger_timeout must not flip the TI)
- submit_event/submit_failure SELECT FOR UPDATE SKIP LOCKED; if another writer already took the row, no-op
Issue body swaps the writers: the unbounded timeout UPDATE is scheduler check_trigger_timeouts; the state!=deferred trigger_id NULL UPDATE is triggerer clean_unused.
---
Drafted-by: Cursor Grok 4.6 (no human review before posting)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org