You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/10/10 13:03:32 UTC
[GitHub] [airflow] parisni opened a new issue, #26969: Provide user custom code to trigger listing dags / modules
parisni opened a new issue, #26969:
URL: https://github.com/apache/airflow/issues/26969
### Description
airflow currently scans the dag and module folder to check for updates. On large airflow instances with many dags and module this adds a considerable load on the disks
### Use case/motivation
while this works for any configuration, some users are able to know when this scan should happens (git commit, CD tasks, inotify....) and by the way decrease considerably the disk IO load, and eventually the overall airflow impact.
If we had a way to implement custom code to trigger scan, [currently dag-dir-list-interval](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#dag-dir-list-interval) we as user would be able to decrease disk IO and while also faster the update of dags and module.
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #26969: Provide user custom code to trigger listing dags / modules
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #26969:
URL: https://github.com/apache/airflow/issues/26969#issuecomment-1287939494
This is a basic design of Airlfow. And it is based on a number of premises - such as that parsing a python DAG might create different result every time it is run. If you want to propose a change here, it takes far more than Feature - this calls for writing a proposal how to implement it (https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals) , starting a discussion on devlist https://airflow.apache.org/community/ and following it up (possibly even leading the implementation or doing it yourself).
This is not impossible. I thnk various approaches there are possible, but you need to understand, argue about and discuss all the consequences of any changes if you want to make it happens. This requires deep understanding of Airflow internals - and unless you want do deep-dive, this is mostly a discussion rather than small feature that anyone can implement.
Converting if into discussion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #26969: Provide user custom code to trigger listing dags / modules
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #26969: Provide user custom code to trigger listing dags / modules
URL: https://github.com/apache/airflow/issues/26969
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org