You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by "darren-cheung-sa (via GitHub)" <gi...@apache.org> on 2026/07/27 15:08:07 UTC

[I] Scheduler deadlocks in pipe write during task dispatch (LocalExecutor) — runtime variant of #67881 [airflow]

darren-cheung-sa opened a new issue, #70526:
URL: https://github.com/apache/airflow/issues/70526

   ### Under which category would you file this issue?
   
   Airflow Core
   
   ### Apache Airflow version
   
   3.3.0
   
   ### What happened and how to reproduce it?
   
   **Issue Description**
   
   The scheduler permanently deadlocks during **normal task dispatch** (not shutdown), blocked in a pipe `write()` to the LocalExecutor's activity queue. All dispatch halts; the SchedulerJob heartbeat stops while the systemd service stays "active". ~10 minutes later, every running task's supervisor kills its task ("Server indicated the task shouldn't be running anymore"), burning retry attempts.
   
   This appears to be the **runtime/dispatch-path variant of the shutdown deadlock fixed in #67881** (merged 2026-07-02, not yet released — 3.3.0 still ships the pre-fix `end()` and empty `terminate()`).
   
   Occurred **4 times in ~14 hours** on one production host once dispatch volume rose (~150–250 concurrent tasks). Identical signature each time.
   
   **Evidence captured mid-deadlock:**
   - `/proc/<scheduler-pid>/stack` → blocked in `pipe_write`
   - `/proc/<scheduler-pid>/syscall` → `write(fd=9)`; fd 9 → `pipe:[<inode>]`
   - Scanning `/proc/*/fd`: the **same pipe inode held (read fd 7 + write fd 9) by the scheduler and every LocalExecutor worker process**
   - Host healthy: 100+ GB free RAM, idle CPU, no OOM, clean kernel log
   - One occurrence happened with **zero** api-server errors/timeouts, ruling out API-side causes
   
   **Steps to reproduce:**
   
   Race condition — probabilistic and load-correlated, so not minimally deterministic, but reliably triggered in our environment within minutes-to-hours:
   
   1. LocalExecutor, `parallelism = 256`, default `max_tis_per_query` (512), PostgreSQL backend
   2. Generate dispatch bursts of 150–250 mostly lightweight, long-running tasks (ours poll remote Spark jobs), with tasks completing (returning results) concurrently with new dispatch
   3. Scheduler eventually blocks forever in `pipe_write`; heartbeat stops
   
   Notably there is no clean load threshold: the same host survived a 30-minute soak at 256 running tasks, then deadlocked at ~40 running during a later dispatch wave — consistent with a timing race on the shared queues, not resource exhaustion.
   
   Suggested synthetic repro: a DAG with ~300 mapped `time.sleep(300)` tasks, cleared repeatedly to force dispatch bursts while earlier tasks are completing.
   
   **Mitigations tried (none sufficient):**
   
   | Change | Result |
   |---|---|
   | `[api] workers` 4 → 12 | fixed unrelated api saturation; deadlock recurred with 0 timeouts |
   | `default_pool` 300 → 150 | reduced frequency; recurred |
   | `[scheduler] max_tis_per_query` 512 → 32 | reduced frequency; recurred |
   | Pausing the high-volume DAGs | stable (current workaround, at large throughput cost) |
   
   ### What you think should happen instead?
   
   The scheduler should never block indefinitely on executor queue writes — a full result/activity queue should trigger draining (or a bounded/non-blocking put with retry), so a slow consumer can't freeze the entire scheduling loop.
   
   **Suspected mechanism** — same as #67881 describes, but hit from the dispatch side:
   
   1. Workers fill `result_queue`'s OS pipe buffer (~64KB) and block on `put()`
   2. Blocked workers stop draining `activity_queue`
   3. Scheduler blocks in `_process_workloads()` → `self.activity_queue.put(workload)` (`airflow/executors/local_executor.py`)
   4. Being single-threaded at that point, the scheduler never reaches `_read_results()`, so `result_queue` never drains → circular wait, permanent
   
   #67881 adds result-queue draining during `end()`/join, but `_process_workloads()` still calls a blocking `put()` with no draining fallback, so the runtime deadlock remains possible on `main` as well.
   
   ### Operating System
   
   Amazon Linux 2023
   
   ### Deployment
   
   Virtualenv installation
   
   ### Apache Airflow Provider(s)
   
   _No response_
   
   ### Versions of Apache Airflow Providers
   
   n/a — core issue (apache-airflow-task-sdk 1.3.0 for reference)
   
   ### Official Helm Chart version
   
   Not Applicable
   
   ### Kubernetes Version
   
   _No response_
   
   ### Helm Chart configuration
   
   _No response_
   
   ### Docker Image customizations
   
   _No response_
   
   ### Anything else?
   
   Standalone deployment: systemd units for scheduler (LocalExecutor), api-server, and dag-processor on a single EC2 host; PostgreSQL metadata DB.
   
   Happy to provide full `/proc` captures, journald excerpts of the supervisor mass-terminations, and timing data from all 4 occurrences on request.
   
   Mainly using Claude to diagnose & type this out. Looked myself for existing issues, and the one I linked was the closest thing I could find to ours
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Scheduler deadlocks in pipe write during task dispatch (LocalExecutor) — runtime variant of #67881 [airflow]

Posted by "darren-cheung-sa (via GitHub)" <gi...@apache.org>.

darren-cheung-sa commented on issue #70526:
URL: https://github.com/apache/airflow/issues/70526#issuecomment-5093951764

   @SakshamKapoor2911 It would be great if you could take on the PR. I'm new/unfamiliar to this issue and to contributing to this repo (and as you said, also trying to make fixes to our pipelines). After that, I'll try to pull your fixes into my deployment (also new to that, but your steps seem easy enough)
   
   Thanks for all the work & communication!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Scheduler deadlocks in pipe write during task dispatch (LocalExecutor) — runtime variant of #67881 [airflow]

Posted by "darren-cheung-sa (via GitHub)" <gi...@apache.org>.

darren-cheung-sa commented on issue #70526:
URL: https://github.com/apache/airflow/issues/70526#issuecomment-5093187561

   @SakshamKapoor2911 wondering if you could confirm this issue was similar to yours / shed some light on it / suggest solution
   
   I don't think your fix was included in 3.3.0, was it?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Scheduler deadlocks in pipe write during task dispatch (LocalExecutor) — runtime variant of #67881 [airflow]

Posted by "darren-cheung-sa (via GitHub)" <gi...@apache.org>.

darren-cheung-sa commented on issue #70526:
URL: https://github.com/apache/airflow/issues/70526#issuecomment-5094212003

   I will take a look at your design in 70543 once I have a chance. And I will apply the 70545 fix locally if my other temp-fixes fail (obviously ideally I get my workflow working with official Airflow versions, but it might not be possible)
   
   Thank you


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Scheduler deadlocks in pipe write during task dispatch (LocalExecutor) — runtime variant of #67881 [airflow]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.

boring-cyborg[bot] commented on issue #70526:
URL: https://github.com/apache/airflow/issues/70526#issuecomment-5093059479

   Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Scheduler deadlocks in pipe write during task dispatch (LocalExecutor) — runtime variant of #67881 [airflow]

Posted by "SakshamKapoor2911 (via GitHub)" <gi...@apache.org>.

SakshamKapoor2911 commented on issue #70526:
URL: https://github.com/apache/airflow/issues/70526#issuecomment-5093891565

Correct — neither fix is in the published 3.3.0. #67881 was merged to `main` after the 3.3.0 cut and will ship in the next release.

Both can be hot-patched directly into your 3.3.0 installation by editing a single file: `<venv>/lib/python3.x/site-packages/airflow/executors/local_executor.py`.

**Shutdown deadlock (#67881)** — apply the [merged diff](https://github.com/apache/airflow/pull/67881/files), which drains `result_queue` during worker join in `end()` and implements `terminate()` to force-kill workers (currently a no-op).

**Dispatch deadlock (#70526)** — add one line in `_process_workloads()`, inside the `for workload in workload_list:` loop, before `self.activity_queue.put(workload)`:

```python
self._read_results()
```

This drains the result queue before each dispatch, unblocking workers so they can resume consuming the activity queue — breaking the circular wait during normal runtime.

I didn't deploy either in production myself — both were verified against unit tests. My branch with the dispatch fix + tests is [here](https://github.com/SakshamKapoor2911/airflow/tree/fix-local-executor-dispatch-deadlock). Would you like me to open a PR? I think you should be able to drive it — you've already done the hard diagnostic work here — but happy to co-author or take it if you'd rather focus on getting your deployment stable first.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Scheduler deadlocks in pipe write during task dispatch (LocalExecutor) — runtime variant of #67881 [airflow]

Posted by "darren-cheung-sa (via GitHub)" <gi...@apache.org>.

darren-cheung-sa commented on issue #70526:
URL: https://github.com/apache/airflow/issues/70526#issuecomment-5093713031

   Thanks @SakshamKapoor2911 , I will take a look at your changes and try to adapt them for our use case. If successful, I'll open a PR and tag you for review, much appreciated
   
   Could you explain how you proceeded after that / how I should proceed after that? I assume your fix isn't in the published Airflow 3.3.0 version. Are you somehow running your tasks on a different version of Airflow (e.g. the one with your fix), and if so, could you tell me how I could do that?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Scheduler deadlocks in pipe write during task dispatch (LocalExecutor) — runtime variant of #67881 [airflow]

Posted by "SakshamKapoor2911 (via GitHub)" <gi...@apache.org>.

SakshamKapoor2911 commented on issue #70526:
URL: https://github.com/apache/airflow/issues/70526#issuecomment-5094099535

   I opened #70543 for discussion. It asks whether a continuous background drain thread would be a better long-term fix than the spot-drain approach both our PRs use.
   
   The spot-fix from #70526 (drain before each put) is correct and works. The question in #70543 is whether we should prevent the underlying condition altogether so future contributors do not need to keep adding these defensive drains. No strong opinion either way, I wanted to surface it for maintainer guidance.
   
   Would welcome your thoughts there since you have diagnosed the same root cause from the production side.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Scheduler deadlocks in pipe write during task dispatch (LocalExecutor) — runtime variant of #67881 [airflow]

Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.

vatsrahul1001 closed issue #70526: Scheduler deadlocks in pipe write during task dispatch (LocalExecutor) — runtime variant of #67881
URL: https://github.com/apache/airflow/issues/70526


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Scheduler deadlocks in pipe write during task dispatch (LocalExecutor) — runtime variant of #67881 [airflow]

Posted by "SakshamKapoor2911 (via GitHub)" <gi...@apache.org>.

SakshamKapoor2911 commented on issue #70526:
URL: https://github.com/apache/airflow/issues/70526#issuecomment-5094139648

   PR is open at #70545. The fix is the one-line change described above — you can hot-patch your 3.3.0 install right away while CI runs on the PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Scheduler deadlocks in pipe write during task dispatch (LocalExecutor) — runtime variant of #67881 [airflow]

Posted by "SakshamKapoor2911 (via GitHub)" <gi...@apache.org>.

SakshamKapoor2911 commented on issue #70526:
URL: https://github.com/apache/airflow/issues/70526#issuecomment-5093619785

   Great diagnosis — this is indeed the dispatch-side gap that #67881 (shutdown `end()`) did not cover. The same drain-before-write pattern applies: calling `_read_results()` before each `activity_queue.put()` in `_process_workloads` ensures workers blocked on a full result pipe get unblocked and can resume consuming from the activity queue, preventing the circular wait.
   
   Verified locally with existing tests (17 pass, 0 regressions) plus two new ordering tests. Reference branch with fix + tests:
   
   https://github.com/SakshamKapoor2911/airflow/tree/fix-local-executor-dispatch-deadlock
   
   Feel free to cherry-pick, adapt, or ignore — I will not open a competing PR since you have already put in the diagnostic work. Happy to review yours when ready.
   
   ---
   Drafted-by: Claude Code (DeepSeek v4 Pro); reviewed by @SakshamKapoor2911 before posting


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org