You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/05/07 07:51:17 UTC
[GitHub] [airflow] mshalak-nix opened a new issue #8760: Webserver becomes unusable if 100,000 tasks completed
mshalak-nix opened a new issue #8760:
URL: https://github.com/apache/airflow/issues/8760
**Apache Airflow version**: 1.10.9
**Kubernetes version: 1.15.11
**Environment**:
- **Cloud provider or hardware configuration**: Google Cloud
**What happened**:
I started to notice that Airflow webserver performance degrades over time. At one moment all my webserver pods had 100% CPU used, so UI and API became too slow, even unusable.
I started to investigate this issue and looks like it depends on amount of task instances stored in DB. I had 100,000+ of them in the task_instance table. When I checked what process took full CPU on a webserver pods, that was a gunicorn. After I cleaned up task_instance table, CPU usage dropped to nothing. Right now I have 30,000 tasks completed and see CPU spikes again.
Also this issue seems to go away if I disable all my DAGs. Looks like webserver has some query which executes from time to time and consumes all the CPU.
**What you expected to happen**:
Amount of completed tasks do not influence the webserver performance.
**How to reproduce it**:
Generate 100,000 tasks and have 10 enabled DAGs in Airflow. Webserver CPU usage will be high, web becomes unusable.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #8760: Webserver becomes unusable after 100,000 tasks were completed
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #8760:
URL: https://github.com/apache/airflow/issues/8760#issuecomment-625474431
Is there a particular endpoint that is slow in the Webserver ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #8760: Webserver becomes unusable if 100,000 tasks completed
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #8760:
URL: https://github.com/apache/airflow/issues/8760#issuecomment-625091837
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mshalak-nix edited a comment on issue #8760: Webserver becomes unusable after 100,000 tasks were completed
Posted by GitBox <gi...@apache.org>.
mshalak-nix edited a comment on issue #8760:
URL: https://github.com/apache/airflow/issues/8760#issuecomment-625792761
Looks like I figured it out. The issue was with prometheus exporter I used, here is the fix: https://github.com/robinhood/airflow-prometheus-exporter/pull/25
It generated wrong joins, which led to one metric repetition multiple times. Since the Prometheus endpoint was served by a web server, it constantly had to execute DB query which produced thousands of metric duplicates (depending on amount of tasks in DB), that's why web server was so slow.
So closing this issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mshalak-nix commented on issue #8760: Webserver becomes unusable after 100,000 tasks were completed
Posted by GitBox <gi...@apache.org>.
mshalak-nix commented on issue #8760:
URL: https://github.com/apache/airflow/issues/8760#issuecomment-625792761
Looks like I figured it out. The issue was with prometheus exporter I used, here is the fix: https://github.com/robinhood/airflow-prometheus-exporter/pull/25
It generated wrong joins, which lead to one metric repetition multiple times. Since the Prometheus endpoint was server by a web server, it constantly had to execute DB query which produced thousands of metric duplicates (depending on amount of tasks in DB), that's why web server was so slow.
So closing this issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org