You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2023/01/02 14:46:32 UTC
[GitHub] [airflow] arjunanan6 opened a new issue, #28678: Support disabling of dag_dir_list_interval
arjunanan6 opened a new issue, #28678:
URL: https://github.com/apache/airflow/issues/28678
### Apache Airflow version
2.5.0
### What happened
We have a unique situation with our Airflow deployment, in that we bake the DAGs into the image pre-deployment, meaning that a deployment is triggered every time there is a new DAG contribution.
During a recent performance troubleshooting, we have uncovered that the scheduler consumes much more memory than it should, and while looking for ways to optimize it, we came across the `dag_dir_list_interval` configuration. It's currently set to the default of 5 minutes. However, it seems like there is no possibility to disable it barring the possibility of setting it to a really high value like 86400.
### What you think should happen instead
It would be nice if the parameter supports a value, say `-1 `that disables the scan altogether.
### How to reproduce
Default configuration, the scan takes place every 5 minutes.
### Operating System
Debian GNU/Linux
### Versions of Apache Airflow Providers
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
Airflow deployed with official helm chart on AKS.
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #28678: Support disabling of dag_dir_list_interval
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #28678:
URL: https://github.com/apache/airflow/issues/28678#issuecomment-1369684594
Assigned you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hussein-awala commented on issue #28678: Support disabling of dag_dir_list_interval
Posted by GitBox <gi...@apache.org>.
hussein-awala commented on issue #28678:
URL: https://github.com/apache/airflow/issues/28678#issuecomment-1369556418
happy to work on this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org