You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "rahulgoyal2987 (via GitHub)" <gi...@apache.org> on 2025/07/19 14:28:46 UTC
[D] Airflow slows for 30 min to 1 hour due to db locks in High availability scheduler [airflow]
GitHub user rahulgoyal2987 created a discussion: Airflow slows for 30 min to 1 hour due to db locks in High availability scheduler
### Apache Airflow version
3.0.3
### If "Other Airflow 2 version" selected, which one?
2.10.3
### What happened?
We are using airflow 2.10.3
Airflow slows for 30 min to 1 hour due to db locks in High availability scheduler (we have scheduler 2 replicas on prod) and fixes only after restart.
What we observed from metrics that locks (For Update query) is acquired at task instance level. This task instance for update query is taking lot of time and increasing the DBLoad in aurora postgresql.
Airflow have most of places fetching row with skip_locked function argument however while fetching task instance skip locked is not used. Similar kind of code i found in airflow 3.0.3 (https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/models/taskinstance.py#L1183)
Will not there be lock/slowness issue if task instance state is changed without skip_locked ?
### What you think should happen instead?
Scheduler slowness/restart causes SLA miss in some DAG.
### How to reproduce
Create high load on airflow and increase no of scheduler replicas
### Operating System
Kubernetes helm
### Versions of Apache Airflow Providers
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
GitHub link: https://github.com/apache/airflow/discussions/53538
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow slows for 30 min to 1 hour due to db locks in High availability scheduler [airflow]
Posted by "kyungjunleeme (via GitHub)" <gi...@apache.org>.
GitHub user kyungjunleeme added a comment to the discussion: Airflow slows for 30 min to 1 hour due to db locks in High availability scheduler
Thank you for sharing ur situation. I aslo find soloutuion and deep into pgbounder.
GitHub link: https://github.com/apache/airflow/discussions/53538#discussioncomment-13817130
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow slows for 30 min to 1 hour due to db locks in High availability scheduler [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk added a comment to the discussion: Airflow slows for 30 min to 1 hour due to db locks in High availability scheduler
> 'l @kyungjunleeme right, i want to get airflow slowness solution in airflow 2.10.3
Now we are planning to implement pgbouner on prod airflow. I am not sure pgbouner will help in airflow slowness.
PGBouncer is HIGHLY recommended for Airflow 2 (look at the database documentation in Airflow docs for 2.10.3 if you care about performance and have high load, I am actually quite surprised you do not have it already, this is almost mandatory requirement with high load, because Airflow 2 task instances open - each - new connection and Postgress creates a new process for every connection, which likely causes it use a LOT of memory and that's the reason for slowness (not locking). We also recommend you have monitoring of your database and alerting to be able to diagnose such situations.
In general - there will be no changes nor optimizations any more for Airflow 2 - just critical bugfixes. So you should also seriously consider migrating to Airflow 3 asap. Architecture is changed there and task instance no longer open DB connections directly to database.
Converting to discussion if more discussion needed.
GitHub link: https://github.com/apache/airflow/discussions/53538#discussioncomment-13817131
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow slows for 30 min to 1 hour due to db locks in High availability scheduler [airflow]
Posted by "rahulgoyal2987 (via GitHub)" <gi...@apache.org>.
GitHub user rahulgoyal2987 added a comment to the discussion: Airflow slows for 30 min to 1 hour due to db locks in High availability scheduler
Do you want to get a solution in airflow 2.10.3?
@kyungjunleeme right, i want to get airflow slowness solution in airflow 2.10.3
Now we are planning to implement pgbouner on prod airflow. I am not sure pgbouner will help in airflow slowness.
GitHub link: https://github.com/apache/airflow/discussions/53538#discussioncomment-13817129
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow slows for 30 min to 1 hour due to db locks in High availability scheduler [airflow]
Posted by "kyungjunleeme (via GitHub)" <gi...@apache.org>.
GitHub user kyungjunleeme added a comment to the discussion: Airflow slows for 30 min to 1 hour due to db locks in High availability scheduler
Interesting. Thank you for issuing issue. This is very helpful to understand lock use in airflow.
So what do you want from this issue?
Do you want to get a solution in airflow 2.10.3?
GitHub link: https://github.com/apache/airflow/discussions/53538#discussioncomment-13817128
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org