You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by "richevanscybermyte (via GitHub)" <gi...@apache.org> on 2026/07/28 13:26:01 UTC
[I] iptables_check.py health check deadlocks when iptables output exceeds the pipe buffer, leaking processes until the VR exhausts memory and self-reboots (~9h cycle) [cloudstack]
richevanscybermyte opened a new issue, #13735:
URL: https://github.com/apache/cloudstack/issues/13735
##### ISSUE TYPE
* Bug Report
##### COMPONENT NAME
~~~
SystemVM / Virtual Router health checks (systemvm/debian/root/health_checks/iptables_check.py)
~~~
##### CLOUDSTACK VERSION
~~~
4.22.0.0 and 4.22.1.0 (management), systemvm template 4.22.0.
The affected code is unchanged on current main:
https://github.com/apache/cloudstack/blob/main/systemvm/debian/root/health_checks/iptables_check.py
~~~
##### CONFIGURATION
Redundant VPC virtual routers, KVM. Advanced zone. Default health check settings
(`router.health.checks.enabled=true`, `router.health.checks.advanced.interval=10`).
The affected VPC has ~200 port-forwarding/LB rules (iptables-save output: 77,029 bytes).
##### OS / ENVIRONMENT
KVM hosts on Rocky Linux 9.5/9.8 (qemu-kvm 9.0/10.1 — reproduced on both). Debian-based
system VM template, 512 MB RAM routers.
##### SUMMARY
`iptables_check.py` runs the fetch command with `Popen(cmd, shell=True, stdout=PIPE)` and
then calls `pout.wait()` **before** reading the pipe:
```python
fetchIpTableEntriesCmd = "iptables-save | grep " + destIp
pout = Popen(fetchIpTableEntriesCmd, shell=True, stdout=PIPE)
if pout.wait() != 0: # <-- deadlocks: wait() before the pipe is drained
...
ipTablesMatchingEntries = pout.communicate()[0].decode().strip().split('\n')
```
This is the deadlock explicitly warned about in the Python `subprocess` documentation:
when the command's output exceeds the OS pipe capacity (64 KiB on Linux), the child
blocks writing to the full pipe and the parent blocks forever in `wait()`.
On routers with large rule sets the check therefore hangs **on every run**. Because the
advanced health check fires every `router.health.checks.advanced.interval` (10 min
default), a new stuck process chain (`monitorServices.py advanced` -> `sh` ->
`iptables_check.py`) accumulates every interval — ~13 MB RSS each, roughly 85–90 MB/h.
On a 512 MB router, RAM and swap are exhausted after ~8.5–9 h of uptime. At that point
keepalived's `heartbeat` track script can no longer fork within its timeout
(`VRRP_Script(heartbeat) timed_out` -> `Entering FAULT STATE`), the router demotes,
wipes `/etc/keepalived/keepalived.conf`, and the guest reboots — taking the VPC down.
The impact is amplified on redundant VPCs: both routers of a pair are created together,
so their exhaustion clocks are synchronized and **both routers fail within minutes of
each other**, defeating the redundancy. We experienced a full-VPC outage roughly every
9 hours until the check was excluded.
##### STEPS TO REPRODUCE
~~~
1. Create a VPC with enough port-forwarding/LB rules that the health check's
"iptables-save | grep <ip>" output exceeds 64 KiB inside the router
(our affected router: iptables-save = 77,029 bytes total).
2. Leave router.health.checks.enabled=true with default advanced interval (10 min).
3. Inside the router, watch: ps -eo pid,ppid,etimes,rss,args | grep -E "monitorServices|iptables_check"
4. A new stuck chain appears every 600 s and never exits (parent and child both in
wchan do_wait; the shell child is blocked writing to the full pipe).
5. Watch "free -k": used memory climbs linearly; after ~8.5-9 h the router enters
keepalived FAULT, demotes/wipes, and reboots.
~~~
Observed accumulation (etimes exactly 600 s apart, none ever exiting):
~~~
90114 7551 /usr/bin/python /root/monitorServices.py advanced
90136 7551 /bin/sh -c ./health_checks/iptables_check.py advanced
90137 7551 /usr/bin/python ./health_checks/iptables_check.py advanced
92506 6952 /usr/bin/python /root/monitorServices.py advanced
... 6351 (repeats every 600s)
... 5751
... 5152
... 4552
~~~
Control group: three routers in other VPCs on the same platform with small rule sets
(iptables-save 1.2–9.3 KB) run the identical check to completion in <1 s and have
152-day uptimes. The only variable is iptables output size vs the 64 KiB pipe buffer.
##### EXPECTED RESULTS
~~~
The health check completes (or fails fast) regardless of iptables rule-set size.
~~~
##### ACTUAL RESULTS
~~~
The check hangs permanently on every run once the piped output exceeds 64 KiB.
Stuck process chains accumulate every advanced-check interval until the router
exhausts RAM+swap (~8.5-9 h on a 512 MB router), keepalived's track script starves,
the router demotes/wipes its keepalived config and reboots. Redundant pairs fail
near-simultaneously because their clocks are synchronized at creation, causing a
recurring full-VPC outage.
~~~
##### WORKAROUND
Set `router.health.checks.to.exclude = iptables_check.py` (global setting, dynamic).
Note: in our testing the value had to be the script filename **with** the `.py`
extension — the bare name `iptables_check` was silently ignored and the check kept
running (possibly a second, minor bug in how the exclusion list is matched).
Accumulated stuck processes must be killed manually inside affected routers.
##### SUGGESTED FIX
Do not call `wait()` while `stdout=PIPE` is unread. Use `communicate()` first and
check `returncode` afterwards, e.g.:
```python
pout = Popen(fetchIpTableEntriesCmd, shell=True, stdout=PIPE)
stdout, _ = pout.communicate()
if pout.returncode != 0:
...
ipTablesMatchingEntries = stdout.decode().strip().split('\n')
```
The same `Popen(stdout=PIPE)` + `wait()` pattern exists in other health check scripts
(e.g. `gateways_check.py`, `cpu_usage_check.py`). Their outputs are normally tiny so
they do not currently hang, but they share the same latent hazard and may be worth
fixing in the same pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] iptables_check.py health check deadlocks when iptables output exceeds the pipe buffer, leaking processes until the VR exhausts memory and self-reboots (~9h cycle) [cloudstack]
Posted by "nvazquez (via GitHub) via commits" <co...@cloudstack.apache.org>.
nvazquez closed issue #13735: iptables_check.py health check deadlocks when iptables output exceeds the pipe buffer, leaking processes until the VR exhausts memory and self-reboots (~9h cycle)
URL: https://github.com/apache/cloudstack/issues/13735
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] iptables_check.py health check deadlocks when iptables output exceeds the pipe buffer, leaking processes until the VR exhausts memory and self-reboots (~9h cycle) [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #13735:
URL: https://github.com/apache/cloudstack/issues/13735#issuecomment-5104770076
@richevanscybermyte
can you create a pull request ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] iptables_check.py health check deadlocks when iptables output exceeds the pipe buffer, leaking processes until the VR exhausts memory and self-reboots (~9h cycle) [cloudstack]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #13735:
URL: https://github.com/apache/cloudstack/issues/13735#issuecomment-5356778106
## 🎯 Triage report
`iptables_check.py` (a VR/VPC router health check) uses `Popen(..., stdout=PIPE)` followed by `wait()` before draining the pipe. On routers with large rule sets (>64KB `iptables-save` output), this deadlocks, leaking a stuck process chain every health-check interval until the router exhausts RAM/swap (~9h) and self-reboots — and redundant VPC router pairs fail together, defeating redundancy. A precise root cause, suggested code fix, and workaround are already provided.
### 📊 Assessment
| Dimension | Value | Reasoning |
|---|---|---|
| **Type** | type:bug | Classic subprocess pipe-deadlock bug with clear reproduction and root cause |
| **Component** | component:virtual-router | Affects the systemVM/VPC virtual router health-check scripts |
| **Severity** | Severity:Critical | Causes full VPC outages (both redundant routers fail near-simultaneously) roughly every 9 hours on affected routers until worked around |
| **Labels** | type:bug, component:virtual-router, Severity:Critical | See above |
| **Coding agent** | Suitable | Root cause, exact file/line, and a concrete suggested fix (`communicate()` before checking `returncode`) are already provided; fix is small, self-contained, and testable |
### 🔗 Similar issues
No closely related open duplicates found.
<details><summary>💡 Notes and suggestions</summary>
The maintainer already asked the reporter to submit a PR. The suggested fix (call `communicate()` first, then check `returncode`, instead of `wait()` before draining stdout) is correct and matches Python's documented subprocess deadlock hazard. The reporter also notes the same `Popen(stdout=PIPE)` + `wait()` anti-pattern exists in `gateways_check.py` and `cpu_usage_check.py` — worth auditing and fixing in the same PR since those are dormant but latent hazards. Also flagged: the exclusion setting `router.health.checks.to.exclude` apparently requires the `.py` extension to match — worth verifying/fixing that matching logic too.
</details>
> Generated by [Daily Issue Triage](https://github.com/apache/cloudstack/actions/runs/32375697891) · sonnet50 224.8K · [◷](https://github.com/search?q=repo%3Aapache%2Fcloudstack+%22gh-aw-workflow-call-id%3A+apache%2Fcloudstack%2Fdaily-issue-triage%22&type=issues)
>
<details>
<summary>Add this agentic workflows to your repo</summary>
To install this agentic workflow, run
```
gh aw add githubnext/agentics/workflows/daily-issue-triage.md@d7c1dc4b72b00607a67caaffdcc216cb64379cf9
```
</details>
<!-- gh-aw-agentic-workflow: Daily Issue Triage, engine: copilot, version: 1.0.52, model: claude-sonnet-5, id: 32375697891, workflow_id: daily-issue-triage, run: https://github.com/apache/cloudstack/actions/runs/32375697891 -->
<!-- gh-aw-workflow-call-id: apache/cloudstack/daily-issue-triage -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] iptables_check.py health check deadlocks when iptables output exceeds the pipe buffer, leaking processes until the VR exhausts memory and self-reboots (~9h cycle) [cloudstack]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #13735:
URL: https://github.com/apache/cloudstack/issues/13735#issuecomment-5104735903
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org