You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/07/21 08:39:41 UTC
[GitHub] [airflow] maciejkul opened a new issue, #25211: Airflow scheduler spawn zombie process
maciejkul opened a new issue, #25211:
URL: https://github.com/apache/airflow/issues/25211
### Apache Airflow version
2.3.2
### What happened
Airflow scheduler spawn zombie process. I get more then 5000 zombie in daily interval. As workaround, I restart airflow scheduler container daily.
### Below some information from debug
* i don't have any queued DAGs (I don't see any in webserver).
* i cannot find any Zombie PID in airflow logs
* airflow update from 2.2 to 2.3.2 doesn't change anything
* zombie process just "appear". I don't see any relevance.
* i get same issue when all DAGs are disabled
* issue persist only in scheduler
* i don't test daemon mode
* i have 210 dags file
Non zombie process from scheduler container:
```
deploy@50393bd86f1f:/usr/local/airflow$ ps aux | grep -v Z
deploy+ 1 6.5 0.1 1081432 194652 ? Ssl Jul20 90:49 /usr/local/bin/python3.8 /usr/local/bin/airflow scheduler
deploy+ 83 2.7 0.1 937152 158472 ? S Jul20 38:12 airflow scheduler -- DagFileProcessorManager
deploy+ 1665 1.0 0.0 8060 2288 pts/0 Ss 08:23 0:00 bash
deploy+ 1689 13.0 0.1 967880 169384 ? Sl 08:23 0:00 airflow scheduler - DagFileProcessor /usr/local/airflow/dags/load_009_utlan.py
deploy+ 1696 0.0 0.1 939976 162012 ? S 08:23 0:00 airflow scheduler - DagFileProcessor /usr/local/airflow/dags/confluence_publisher.py
deploy+ 1700 0.0 0.0 11560 1616 pts/0 R+ 08:23 0:00 ps aux
```
Zombie count:
```
deploy@50393bd86f1f:/usr/local/airflow$ ps aux | grep Z | wc -l
13552
```
Zombie process (not all):
```
deploy+ 32704 0.0 0.0 0 0 ? Z 02:25 0:00 [sh] <defunct>
deploy+ 32707 0.0 0.0 0 0 ? Z Jul20 0:00 [sh] <defunct>
deploy+ 32713 0.0 0.0 0 0 ? Z Jul20 0:00 [sh] <defunct>
deploy+ 32714 0.0 0.0 0 0 ? Z Jul20 0:00 [sh] <defunct>
deploy+ 32716 0.0 0.0 0 0 ? Z Jul20 0:00 [sh] <defunct>
deploy+ 32718 0.0 0.0 0 0 ? Z Jul20 0:00 [sh] <defunct>
deploy+ 32720 0.0 0.0 0 0 ? Z Jul20 0:00 [sh] <defunct>
deploy+ 32721 0.0 0.0 0 0 ? Z Jul20 0:00 [sh] <defunct>
deploy+ 32724 0.0 0.0 0 0 ? Z Jul20 0:00 [sh] <defunct>
deploy+ 32727 0.0 0.0 0 0 ? Z Jul20 0:00 [sh] <defunct>
deploy+ 32728 0.0 0.0 0 0 ? Z 03:20 0:00 [sh] <defunct>
deploy+ 32733 0.0 0.0 0 0 ? Z 00:31 0:00 [sh] <defunct>
deploy+ 32736 0.0 0.0 0 0 ? Z 00:31 0:00 [sh] <defunct>
deploy+ 32738 0.0 0.0 0 0 ? Z Jul20 0:00 [sh] <defunct>
deploy+ 32740 0.0 0.0 0 0 ? Z Jul20 0:00 [sh] <defunct>
deploy+ 32741 0.0 0.0 0 0 ? Z 03:20 0:00 [sh] <defunct>
deploy+ 32742 0.0 0.0 0 0 ? Z Jul20 0:00 [sh] <defunct>
deploy+ 32744 0.0 0.0 0 0 ? Z 06:02 0:00 [sh] <defunct>
deploy+ 32746 0.0 0.0 0 0 ? Z Jul20 0:00 [sh] <defunct>
deploy+ 32748 0.0 0.0 0 0 ? Z 02:25 0:00 [sh] <defunct>
deploy+ 32750 0.0 0.0 0 0 ? Z 01:30 0:00 [sh] <defunct>
deploy+ 32751 0.0 0.0 0 0 ? Z Jul20 0:00 [sh] <defunct>
deploy+ 32752 0.0 0.0 0 0 ? Z 06:49 0:00 [sh] <defunct>
deploy+ 32756 0.0 0.0 0 0 ? Z Jul20 0:00 [sh] <defunct>
deploy+ 32757 0.0 0.0 0 0 ? Z 06:02 0:00 [sh] <defunct>
deploy+ 32758 0.0 0.0 0 0 ? Z Jul20 0:00 [sh] <defunct>
deploy+ 32760 0.0 0.0 0 0 ? Z 01:30 0:00 [sh] <defunct>
deploy+ 32761 0.0 0.0 0 0 ? Z Jul20 0:00 [sh] <defunct>
deploy+ 32763 0.0 0.0 0 0 ? Z Jul20 0:00 [sh] <defunct>
deploy+ 32764 0.0 0.0 0 0 ? Z 06:49 0:00 [sh] <defunct>
deploy+ 32767 0.0 0.0 0 0 ? Z Jul20 0:00 [sh] <defunct>
```
Zombie parents (not all again):
```
deploy@50393bd86f1f:/usr/local/airflow$ ps -o ppid= -p 32767
1
deploy@50393bd86f1f:/usr/local/airflow$ ps -o ppid= -p 32757
1
```
### What you think should happen instead
Wrong subprocess handling in airflow scheduler code.
### How to reproduce
_No response_
### Operating System
Debian GNU/Linux 10 (buster) - Docker container
### Versions of Apache Airflow Providers
deploy@50393bd86f1f:/usr/local/airflow/dags$ pip freeze | grep apache-
apache-airflow==2.3.2
apache-airflow-providers-celery==3.0.0
apache-airflow-providers-ftp==3.0.0
apache-airflow-providers-http==3.0.0
apache-airflow-providers-imap==3.0.0
apache-airflow-providers-mongo==3.0.0
apache-airflow-providers-postgres==5.0.0
apache-airflow-providers-sqlite==3.0.0
### Deployment
Docker-Compose
### Deployment details
We use docker-compose with docker swarm. Container is behind proxy.
### Anything else
Issue persist every time
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #25211: Airflow scheduler spawn zombie process
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #25211:
URL: https://github.com/apache/airflow/issues/25211#issuecomment-1191243593
Can you pleasupgrade to 2.3.3 and check if it's fixed there? There were a number of fixes implemented there and the easiest way to check is to upgrade (whch you should do anyway as it contains important bugfixes).
Please let us know if it occurs after you do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #25211: Airflow scheduler spawn zombie process
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #25211:
URL: https://github.com/apache/airflow/issues/25211#issuecomment-1191208829
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #25211: Airflow scheduler spawn zombie process
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #25211: Airflow scheduler spawn zombie process
URL: https://github.com/apache/airflow/issues/25211
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org