[GitHub] [airflow] adamd01 opened a new issue #21299: Put _deactivate_missing_dags logic behind config

[GitBox <gi...@apache.org>]

adamd01 opened a new issue #21299:
URL: https://github.com/apache/airflow/issues/21299


   ### Description
   
   In Airflow 2.2.0 this PR was added (https://github.com/apache/airflow/pull/17121) which deactivates any dags which are no longer found in the file being processed. 
   
   It would be good to have the option to make this logic configurable.
   
   ### Use case/motivation
   
   We have a set up where we generate DAGs dynamically by hitting an API. (In order to abstract DAG creation from users who can't directly code up an Airflow DAG).
   
   We have 100s of DAGs which means importing all the DAGs each time the dag processor runs is slow. And we want to make sure any new DAGs are added as quickly as possible. 
   
   To make this happens we use some state on the scheduler to record when the last dag parse happened, and then hit our api with that timestamp to only add new dags. Every 5 minutes we do a full refresh to make sure everything is in sync. 
   
   With this change in place we have to go back to doing the full import each time, which means we have a slow process for importing new dags. Right now this stops us from upgrading to 2.2.x
   
   As a potential workaround I tried syncing serialised DAGs from the DB to supplement the incremental API fetches, but this seems quite messy and threw up some other issues. 
   
   I'd be keen to know if anyone has other ideas of a work around or if there is more support for this functionality to be configurable I'm happy to work on a PR. 
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #21299: Put _deactivate_missing_dags logic behind config

[GitBox <gi...@apache.org>]

boring-cyborg[bot] commented on issue #21299:
URL: https://github.com/apache/airflow/issues/21299#issuecomment-1029232154


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org