You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/03/17 12:51:13 UTC
[GitHub] [airflow] vineethguna opened a new issue #14851: Airflow workers causing 100% CPU on PostgreSQL Database
vineethguna opened a new issue #14851:
URL: https://github.com/apache/airflow/issues/14851
<!--
Welcome to Apache Airflow! For a smooth issue process, try to answer the following questions.
Don't worry if they're not all applicable; just try to include what you can :-)
If you need to include code snippets or logs, please put them in fenced code
blocks. If they're super-long, please use the details tag like
<details><summary>super-long log</summary> lots of stuff </details>
Please delete these comment blocks before submitting the issue.
-->
<!--
IMPORTANT!!!
PLEASE CHECK "SIMILAR TO X EXISTING ISSUES" OPTION IF VISIBLE
NEXT TO "SUBMIT NEW ISSUE" BUTTON!!!
PLEASE CHECK IF THIS ISSUE HAS BEEN REPORTED PREVIOUSLY USING SEARCH!!!
Please complete the next sections or the issue will be closed.
These questions are the first thing we need to know to understand the context.
-->
**Airflow Setup Used**
Airflow Version: 1.10.12
PostgreSQL version: 10
Executor: Celery Executor
Broker: Redis
Result Backend: Redis
Worker Concurrency: 25
Number of workers: 4
Airflow Configuration:
AIRFLOW__CELERY__WORKER_CONCURRENCY: 25
AIRFLOW__CORE__PARALLELISM: 100
AIRFLOW__CORE__DAG_CONCURRENCY: 100
AIRFLOW__CORE__MAX_ACTIVE_RUNS_PER_DAG: 100
AIRFLOW__CELERY__SYNC_PARALLELISM: 5
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`): 1.18.14
**Environment**:
- **Cloud provider or hardware configuration**: Azure
- **OS** (e.g. from /etc/os-release):
- **Kernel** (e.g. `uname -a`):
- **Install tools**:
- **Others**:
**What happened**:
While running 100 parallel tasks using airflow workers with PostgreSQL as metadata database, the CPU on PostgreSQL is hitting 100% consistently even though the database is provisioned for 16 cores.
Due to the above bottleneck on PostgreSQL the time taken to execute the tasks is increasing proportionally to the number of parallel tasks getting executed
If airflow workers only run 1 task it takes 15 seconds to complete the task
If airflow workers are running 100 similar tasks parallelly the task execution time is increasing to 270 seconds on average
The is no CPU, memory bottlenecks on airflow workers
**What you expected to happen**:
The task execution time should not increase with the number of parallel tasks getting executed. And PostgreSQL should not hit 100% CPU
**How to reproduce it**:
- Use the above airflow setup to launch web server, scheduler, workers
- Create a simple DAG with python operator which prints "hello world"
- Trigger 100 DAG runs
- Observe the CPU on PostgreSQL database
**Anything else we need to know**:
All airflow workers use NullPool to execute queries on PostgreSQL, i.e. it opens a connection, executes the query, closes the connection
The above lifecycle happens for each and every query, upon inspecting the query metrics on PostgreSQL there was no latency issues with the query execution, but the PostgreSQL CPU is getting consumed for handling the connects and disconnects from the airflow workers
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #14851: Airflow workers causing 100% CPU on PostgreSQL Database
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #14851:
URL: https://github.com/apache/airflow/issues/14851#issuecomment-830933611
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #14851: Airflow workers causing 100% CPU on PostgreSQL Database
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #14851:
URL: https://github.com/apache/airflow/issues/14851#issuecomment-801274271
have you considered using PGBouncer? See: https://github.com/apache/airflow/issues/13941
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #14851: Airflow workers causing 100% CPU on PostgreSQL Database
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #14851:
URL: https://github.com/apache/airflow/issues/14851#issuecomment-826170430
This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #14851: Airflow workers causing 100% CPU on PostgreSQL Database
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #14851:
URL: https://github.com/apache/airflow/issues/14851#issuecomment-805904138
I think, we should add it to documenentation. Are you willing to submit a PR?
Here is ticket: https://github.com/apache/airflow/issues/13941
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vineethguna commented on issue #14851: Airflow workers causing 100% CPU on PostgreSQL Database
Posted by GitBox <gi...@apache.org>.
vineethguna commented on issue #14851:
URL: https://github.com/apache/airflow/issues/14851#issuecomment-806363085
I will add it to the documentation, Will submit the PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #14851: Airflow workers causing 100% CPU on PostgreSQL Database
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #14851:
URL: https://github.com/apache/airflow/issues/14851#issuecomment-826170430
This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vineethguna commented on issue #14851: Airflow workers causing 100% CPU on PostgreSQL Database
Posted by GitBox <gi...@apache.org>.
vineethguna commented on issue #14851:
URL: https://github.com/apache/airflow/issues/14851#issuecomment-801690194
@mik-laj Thanks for the suggestion, will try PGBouncer and get back with the observations
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vineethguna commented on issue #14851: Airflow workers causing 100% CPU on PostgreSQL Database
Posted by GitBox <gi...@apache.org>.
vineethguna commented on issue #14851:
URL: https://github.com/apache/airflow/issues/14851#issuecomment-805901458
Thanks @mik-laj . After using pgbouncer the bottlenecks on postgresql reduced and there was a huge performance improvement.
Is using PGBouncer documented anywhere in airflow documentation. If not is it good to add it to the documentation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] closed issue #14851: Airflow workers causing 100% CPU on PostgreSQL Database
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed issue #14851:
URL: https://github.com/apache/airflow/issues/14851
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org