You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "vlieven (via GitHub)" <gi...@apache.org> on 2026/10/10 18:47:18 UTC
[I] OpenLineage listener forks a ProcessPoolExecutor in the Airflow 3 API server, leaking orphaned worker processes [airflow]
vlieven opened a new issue, #74544:
URL: https://github.com/apache/airflow/issues/74544
### Under which category would you file this issue?
Providers
### Apache Airflow version
3.3.1
### What happened and how to reproduce it?
The API server leaks processes forked by the OpenLineage listener until the container is OOM-killed.
In Airflow 3, `patch_dag_run_state` (`api_fastapi/core_api/services/public/dag_run.py`) calls `on_dag_run_success` / `on_dag_run_failed` from inside the API server when a DAG run is marked success/failed through the UI or REST API. Manual task instance state changes do the same through `_on_task_instance_manual_state_change`. The OpenLineage listener hands these events to `OpenLineageListener.executor`, a `ProcessPoolExecutor` created without an `mp_context`, so on Linux with Python < 3.14 it **forks** the API server worker.
That causes two problems:
1. **Children deadlock.** The uvicorn worker is multithreaded (AnyIO worker threads, DB pool, etc.), and forking a multithreaded process can leave the child holding a lock that will never be released. Some children hang in `futex_wait` and never send their event, so **lineage events are silently lost**.
2. **Children are never cleaned up.** `before_stopping` is only called by components such as the scheduler, never by API server workers, so the pool is never shut down. Idle pool children block forever in `pipe_read`. They never get EOF because they inherited the write end of their own call-queue pipe. When gunicorn replaces a worker (`worker_refresh_interval`, timeout, crash), its pool children are reparented to PID 1 and live forever. Each one is a full copy of the worker (150-300 MiB PSS).
What we observed in one api-server pod, ~33 h after start:
```
master + 2 live workers ~580 MiB PSS
10 orphaned "gunicorn: worker" procs ~2,290 MiB PSS (ppid=1)
container memory.current 2,898 MiB of 3,272 MiB limit
```
Orphan states (`/proc/<pid>/wchan`): 5 × `pipe_read` (1 thread, idle pool worker), 5 × `futex_wait_queue` (2 threads, deadlocked after fork).
`py-spy dump` of an orphan (the stack the child inherited from its parent at fork time, followed by the pool worker loop):
```
Thread 184 (idle): "AnyIO worker thread"
_recv (multiprocessing/connection.py:395)
_recv_bytes (multiprocessing/connection.py:430)
recv_bytes (multiprocessing/connection.py:216)
get (multiprocessing/queues.py:103)
_process_worker (concurrent/futures/process.py:252)
run (multiprocessing/process.py:108)
_bootstrap (multiprocessing/process.py:314)
_launch (multiprocessing/popen_fork.py:71)
__init__ (multiprocessing/popen_fork.py:19)
_Popen (multiprocessing/context.py:282)
start (multiprocessing/process.py:121)
_spawn_process (concurrent/futures/process.py:799)
_launch_processes (concurrent/futures/process.py:789)
_start_executor_manager_thread (concurrent/futures/process.py:762)
submit (concurrent/futures/process.py:823)
submit_callable (openlineage/plugins/listener.py:971)
on_dag_run_success (openlineage/plugins/listener.py:893)
...pluggy...
patch_dag_run_state (airflow/api_fastapi/core_api/services/public/dag_run.py:204)
patch_dag_run (airflow/api_fastapi/core_api/routes/public/dag_run.py:232)
run (anyio/_backends/_asyncio.py:1100)
```
Even without worker refresh, every API server worker that has handled a manual state change keeps one extra forked process (~250 MiB) for its whole lifetime. Worker refresh, which is meant to limit memory growth, turns that into an unbounded leak.
### What you think should happen instead?
The listener should not fork from the API server. The process pool makes sense in the scheduler (isolating event emission from the scheduler loop, see #39235), but in the API server the work is a few HTTP calls. Possible fixes:
- Use a `ThreadPoolExecutor` when the listener runs outside the scheduler (e.g. when the hook is invoked from the API server).
- Create the pool with a non-fork `mp_context` (`forkserver`/`spawn`) and shut the pool down when an API server worker exits, so children cannot outlive their worker. This avoids the deadlocks, but each child re-imports Airflow.
### Operating System
Debian (official `apache/airflow:slim-3.3.1-python3.12` image), Python 3.12.13
### Deployment
Other 3rd-party Helm chart
### Apache Airflow Provider(s)
openlineage
### Versions of Apache Airflow Providers
apache-airflow-providers-standard==1.11.1
apache-airflow-providers-cncf-kubernetes==10.17.0
apache-airflow-providers-openlineage==2.20.2
openlineage-python==1.53.0
### Official Helm Chart version
Not Applicable
### Kubernetes Version
1.35
### Helm Chart configuration
Not relevant
### Docker Image customizations
Not Applicable
### Anything else?
### Deployment details
- `airflow api-server` with `[api] server_type = gunicorn`, `[api] workers = 2`, `[api] worker_refresh_interval = 10800`
- gunicorn 26.2.0
- OpenLineage enabled with the `http` transport
- Kubernetes, api-server container memory limit ~3.2 GiB
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] OpenLineage listener forks a ProcessPoolExecutor in the Airflow 3 API server, leaking orphaned worker processes [airflow]
Posted by "garvit-arora (via GitHub)" <gi...@apache.org>.
garvit-arora commented on issue #74544:
URL: https://github.com/apache/airflow/issues/74544#issuecomment-6165862867
Thanks @vlieven for checking the workaround in a running deployment. The disappearance of orphaned processes is useful operational evidence. As you noted, the observed event rate cannot establish whether thread dispatch handles the higher-frequency case that motivated #39235.
Thanks @Shoryamishra61 for taking the upstream fix in #74577. The latest PR head limits API-server dispatch to one thread, retains the scheduler's configured process pool, and adds coverage for pool sizes above one, concurrent submissions, and shutdown. That addresses the adapter-ownership concern I raised. I'll follow that PR instead of opening a competing implementation. It still needs the project's normal review and CI before merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] OpenLineage listener forks a ProcessPoolExecutor in the Airflow 3 API server, leaking orphaned worker processes [airflow]
Posted by "vlieven (via GitHub)" <gi...@apache.org>.
vlieven commented on issue #74544:
URL: https://github.com/apache/airflow/issues/74544#issuecomment-6119272797
Hi @garvit-arora, thank you! I am currently working around the issue by patching the provider to use the ThreadPoolExecutor instead of the ProcessPoolExecutor. This essentially reverts the change made in #39235, so I am not quite sure it is the correct upstream fix for the issue, and hence I did not open a PR with those changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] OpenLineage listener forks a ProcessPoolExecutor in the Airflow 3 API server, leaking orphaned worker processes [airflow]
Posted by "Shoryamishra61 (via GitHub)" <gi...@apache.org>.
Shoryamishra61 commented on issue #74544:
URL: https://github.com/apache/airflow/issues/74544#issuecomment-6124271981
Thanks for catching this. Agreed that sharing the lazily initialized adapter across multiple API emission threads introduces a concurrency risk.
I'll constrain the API-server executor to a single worker while preserving the scheduler's existing ProcessPoolExecutor configuration, and add regression coverage for configurations where `dag_state_change_process_pool_size > 1`, concurrent submissions, and worker shutdown.
Appreciate the review -- I'd prefer to keep the fix focused on the original process-leak issue without introducing a separate adapter ownership problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] OpenLineage listener forks a ProcessPoolExecutor in the Airflow 3 API server, leaking orphaned worker processes [airflow]
Posted by "garvit-arora (via GitHub)" <gi...@apache.org>.
garvit-arora commented on issue #74544:
URL: https://github.com/apache/airflow/issues/74544#issuecomment-6122218969
Thanks, this helps clarify the tradeoff. #39235 addressed scheduler hangs with CeleryExecutor and HTTP emission, so switching the listener to threads everywhere could reintroduce that problem.
I have a local, unsubmitted patch that keeps `ProcessPoolExecutor` for the scheduler and uses a single `ThreadPoolExecutor` only in API-server workers. I’m checking the API-server detection and worker-lifecycle behavior before proposing it upstream.
With your thread-pool workaround, have the orphaned processes and missed OpenLineage events both stopped during API-server worker refresh? Any observations from that run would help establish the regression case for a focused PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] OpenLineage listener forks a ProcessPoolExecutor in the Airflow 3 API server, leaking orphaned worker processes [airflow]
Posted by "vlieven (via GitHub)" <gi...@apache.org>.
vlieven commented on issue #74544:
URL: https://github.com/apache/airflow/issues/74544#issuecomment-6146141318
@garvit-arora I don't have orphaned process anymore when swapping to a ThreadPoolExecutor. I've also not observed missed OpenLineage events, but I imagine we're not running it at a frequency that #39235 addressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] OpenLineage listener forks a ProcessPoolExecutor in the Airflow 3 API server, leaking orphaned worker processes [airflow]
Posted by "garvit-arora (via GitHub)" <gi...@apache.org>.
garvit-arora commented on issue #74544:
URL: https://github.com/apache/airflow/issues/74544#issuecomment-6116397200
@vlieven I can work on a focused fix for this. I’ll trace where the OpenLineage listener is invoked from API-server workers and how its executor is created and shut down, then keep any change scoped to avoiding forked, orphaned workers there while preserving scheduler behavior. I did not find an existing PR for #74544. If you already have a patch underway, please let me know so we can coordinate rather than duplicate it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org