You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "PrafullPandey06 (via GitHub)" <gi...@apache.org> on 2026/07/20 08:38:23 UTC
[I] Task supervisor lingers indefinitely on CeleryExecutor with dumb-init (socket_cleanup_timeout never triggers) [airflow]
PrafullPandey06 opened a new issue, #70117:
URL: https://github.com/apache/airflow/issues/70117
### Under which category would you file this issue?
Task SDK
### Apache Airflow version
3.1.8
### What happened and how to reproduce it?
Task supervisor processes (`airflow worker -- <uuid>`) remain alive indefinitely after their task subprocess has finished and exited. The `workers.socket_cleanup_timeout` safety-net (introduced in 3.0.2 via https://github.com/apache/airflow/pull/51180) never triggers because the supervisor never detects the child process exit — `_process_exit_monotonic` is never set.
On our production Celery workers running for ~3 days:
- **worker-0:** 26 stuck supervisors, 357 zombie processes, 0 running tasks
- **worker-1:** 30 stuck supervisors, 417 zombie processes, 0 running tasks
Each stuck supervisor holds ~340 MB of RSS. With 26–30 orphaned supervisors, this accounts for ~8–10 GB of wasted memory per worker pod, causing OOM kills on containers with 6–8 Gi limits.
## How to reproduce
1. Deploy Airflow 3.1.8 with CeleryExecutor using `dumb-init` as container entrypoint (standard Helm chart setup):
```
command: ["/usr/bin/dumb-init", "--"]
args: ["bash", "-c", "exec airflow celery worker -H ${HOSTNAME}"]
```
2. Configure `worker_concurrency=16` (or any value)
3. Run a DAG with dynamically mapped tasks (using `expand()`), e.g. a task that produces many map instances
4. Wait for tasks to complete
5. Check for lingering supervisors:
```bash
# Inside the worker container:
count=0
for pid in /proc/[0-9]*; do
cmd=$(tr '\0' ' ' < "$pid/cmdline" 2>/dev/null)
case "$cmd" in "airflow worker -- "*) count=$((count+1));; esac
done
echo "live supervisors: $count"
```
6. Expected: 0 supervisors when no tasks are running
7. Actual: supervisors accumulate over time, never exit
### What you think should happen instead?
## Root cause analysis
### Environment
- **Airflow:** 3.1.8
- **Executor:** CeleryExecutor with `worker_concurrency=16`
- **Container init:** `dumb-init` (PID 1) — standard in Airflow Helm chart
- **Worker uptime:** ~2 days 18 hours
- **DAG triggering the issue:** `generate_compliance_report` with dynamically mapped task (`map_index=152`)
### The bug chain
1. ForkPoolWorker-3 (PID 71) dispatches a mapped task to a new supervisor (PID 3183)
2. Supervisor creates 28 pipe pairs + 3 epoll instances for task communication
3. Supervisor forks a task subprocess to run the mapped task
4. Task subprocess finishes and exits
5. **`dumb-init` (PID 1) reaps the child via its `waitpid(-1)` loop** — consumes the exit status before the supervisor can
6. Supervisor calls `waitpid()` → gets `ECHILD` ("no such child")
7. `_process_exit_monotonic` is **never set**
8. The socket cleanup timeout check never evaluates to `True`:
```python
# From supervisor.py — this condition is never True because _process_exit_monotonic is None
if (
self._process_exit_monotonic
and time.monotonic() - self._process_exit_monotonic > SOCKET_CLEANUP_TIMEOUT
):
self._cleanup_open_sockets()
```
9. Supervisor sits in `epoll_wait()` forever with 28 dead pipe pairs
10. ~340 MB of memory is permanently wasted
### Evidence: stuck supervisor inspection (PID 3183)
**Process state — sleeping with no children:**
```
State: S (sleeping)
PPid: 71 ← parent is ForkPoolWorker-3
VmRSS: 338696 kB ← ~340 MB held by idle supervisor
Threads: 2
```
**No live or zombie children — dumb-init already reaped them:**
```
Live children: (empty)
Zombie children (PPID=3183): (empty)
```
**96 open file descriptors — 28 pipe pairs to dead subprocess:**
```
3 anon_inode:[eventpoll] ← 3 epoll instances watching dead pipes
2 pipe:[4213839153] ← pipe pair to dead task subprocess
2 pipe:[4213839152]
2 pipe:[4213839151]
... (28 pipe pairs total)
1 /opt/airflow/logs/dag_id=generate_compliance_report/run_id=scheduled__2026-07-18T08:00:00+00:00/task_id=run_sql_for_client/map_index=152/attempt=1.log
```
**Syscall — blocked in recvfrom/epoll_wait:**
```
45 0x0 0x7f9aeb968d10 0x4 0x0 0x0 0x0 0x7ffcda5b8a70 0x7f9b1d184dfe
```
**socket_cleanup_timeout is configured but never fires:**
```
$ airflow config get-value workers socket_cleanup_timeout
60.0
$ kubectl logs ... --since=72h | grep "Process exited with open sockets"
(empty — zero matches in 72 hours)
```
### Evidence: worker-level impact
**kubectl top (no tasks running):**
```
POD NAME CPU(cores) MEMORY(bytes)
sandbox2-airflow-worker-0 airflow-worker 98m 5798Mi
sandbox2-airflow-worker-1 airflow-worker 3m 6487Mi
```
**Airflow DB confirms zero running tasks:**
```
No running tasks
```
**Process counts:**
```
worker-0: pool slots (ForkPoolWorkers): 16, stuck supervisors: 26, zombies: 357
worker-1: pool slots (ForkPoolWorkers): 16, stuck supervisors: 30, zombies: 417
```
**Top memory processes (all idle supervisors, no active tasks):**
```
347056 KB airflow worker -- 019f7c11-7ae5-7819-b15e-128d0d65eb87
346552 KB airflow worker -- 019f7c08-b0ee-791c-901b-4ea4e21371c3
346016 KB airflow worker -- 019f7c00-4d19-7e7e-8878-e82a35d3a8eb
345516 KB airflow worker -- 019f7c10-26bd-7013-b34c-551e753886b4
... (20+ more, all ~340 MB each)
```
### Operating System
RHEL 9.7 (container), Amazon Linux 2 (EKS node), kernel 5.10.236-228.935.amzn2.x86_64
### Deployment
None
### Apache Airflow Provider(s)
amazon, celery, http
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==9.31.0
apache-airflow-providers-celery==3.10.6
apache-airflow-providers-cncf-kubernetes==10.1.0
apache-airflow-providers-common-compat==1.15.0
apache-airflow-providers-common-io==1.7.3
apache-airflow-providers-common-sql==2.0.1
apache-airflow-providers-fab==3.7.0
apache-airflow-providers-http==6.0.4
apache-airflow-providers-postgres==6.0.0
apache-airflow-providers-smtp==3.0.1
apache-airflow-providers-snowflake==6.5.4
apache-airflow-providers-standard==1.15.0
### Official Helm Chart version
Not Applicable
### Kubernetes Version
v1.32.3-eks-473151a
### Helm Chart configuration
Using community chart: airflow-helm/charts v9.0.0
(https://github.com/airflow-helm/charts), not the official Apache Airflow Helm chart.
### Docker Image customizations
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Task supervisor lingers indefinitely on CeleryExecutor when psutil.Process.wait() returns None (socket_cleanup_timeout never triggers) [airflow]
Posted by "hooiv (via GitHub)" <gi...@apache.org>.
hooiv commented on issue #70117:
URL: https://github.com/apache/airflow/issues/70117#issuecomment-5070476938
I've tracked down the actual root cause of the missing exit status
You were completely right that `dumb-init` couldn't be reaping the process while the parent supervisor was still alive. It turns out the culprit was **Celery's `billiard` library**.
Because `CeleryExecutor` runs the supervisor entrypoint directly inside the Celery worker process, it inherits `billiard`'s aggressive global `SIGCHLD` signal handler. Whenever the Airflow task subprocess finishes, `billiard` intercepts the signal and calls `waitpid(-1, WNOHANG)`. This silently eats the exit status before the Task SDK's `psutil.wait(timeout=0)` ever gets a chance to see it, resulting in a return value of `None`.
I've updated PR #70163 with a fix that simply resets `SIGCHLD` to `SIG_DFL` inside `celery_executor_utils.py` right before the supervisor executes. This stops `billiard` from interfering and perfectly resolves the hanging supervisors
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Task supervisor lingers indefinitely on CeleryExecutor with dumb-init (socket_cleanup_timeout never triggers) [airflow]
Posted by "Andrushika (via GitHub)" <gi...@apache.org>.
Andrushika commented on issue #70117:
URL: https://github.com/apache/airflow/issues/70117#issuecomment-5035344622
> I see a fix being raised [#70163](https://github.com/apache/airflow/pull/70163) for the same issue and it looks like init process is reaping the supervisor's child process.
No, that PR is re-describing the error path in this issue, not because another user encountered it. (it does not show that dumb-init actually reaped the task process.)
> I am not sure if this error path is correct, since init should only be able to reap the orphan process.
This is because the mechanism of `init`: the supervisor directly forks the task process, so the task is its child. As long as the supervisor is still alive, PID 1 should not be able to reap it unless the task was somehow reparented.
I agree that these steps reproduce the lingering supervisors, but I do not think they identify the root cause yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Task supervisor lingers indefinitely on CeleryExecutor with dumb-init (socket_cleanup_timeout never triggers) [airflow]
Posted by "Andrushika (via GitHub)" <gi...@apache.org>.
Andrushika commented on issue #70117:
URL: https://github.com/apache/airflow/issues/70117#issuecomment-5032559583
I tried to reproduce this and I think I can confirm the hang mechanism. I could not run the full Celery setup, but I tested what happens when the child's exit status is taken by someone else. Tested on main (16278347df) with psutil 7.2.2, on both macOS and Linux:
```python
import os, time, psutil
pid = os.fork()
if pid == 0:
os._exit(42)
proc = psutil.Process(pid)
time.sleep(0.2)
os.waitpid(pid, 0) # exit status consumed by someone else
print(proc.wait(timeout=0)) # -> None, no exception
```
So `wait(timeout=0)` returns `None` here, it does not raise. But `_check_subprocess_exit` ([supervisor.py:1139](https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L1139)) only handles an int return or `TimeoutExpired`. So `_exit_code` stays `None` forever, and the monitor loop ([supervisor.py:1551](https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L1551)) never exits. I think this matches what you saw: ECHILD, no zombie children, and the socket cleanup never fires.
> dumb-init (PID 1) reaps the child via its waitpid(-1) loop
I am not sure if this error path is correct, since `init` should only be able to reap the orphan process.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Task supervisor lingers indefinitely on CeleryExecutor when psutil.Process.wait() returns None (socket_cleanup_timeout never triggers) [airflow]
Posted by "PrafullPandey06 (via GitHub)" <gi...@apache.org>.
PrafullPandey06 commented on issue #70117:
URL: https://github.com/apache/airflow/issues/70117#issuecomment-5047615518
You're right I shouldn't have stated the dumb-init reaping path as confirmed root cause. I don't have strace evidence showing PID 1 actually calling waitpid on the task PID, and as you pointed out, that shouldn't normally happen while the supervisor is alive. I've updated the issue title and description to reflect this.
That said, the hang is reproducible and the impact is real 26–30 stuck supervisors accumulating ~8–10 GB of wasted memory per worker pod. Whatever is causing wait(timeout=0) to return None, the supervisor doesn't handle that case, and the cleanup timeout has no independent fallback.
Your reproduction script confirms the exact code gap _check_subprocess_exit only handles an int return or TimeoutExpired, so None is silently ignored.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Task supervisor lingers indefinitely on CeleryExecutor with dumb-init (socket_cleanup_timeout never triggers) [airflow]
Posted by "PrafullPandey06 (via GitHub)" <gi...@apache.org>.
PrafullPandey06 commented on issue #70117:
URL: https://github.com/apache/airflow/issues/70117#issuecomment-5034192193
I see a fix being raised https://github.com/apache/airflow/pull/70163 for the same issue and it looks like init process is reaping the supervisor's child process.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org