[GitHub] [airflow] michaelmicheal commented on a diff in pull request #29446: Scheduler, make stale DAG deactivation threshold configurable instead of using dag processing timeout

["michaelmicheal (via GitHub)" <gi...@apache.org>]

michaelmicheal commented on code in PR #29446:
URL: https://github.com/apache/airflow/pull/29446#discussion_r1101929995


##########
airflow/config_templates/default_airflow.cfg:
##########
@@ -1045,8 +1045,16 @@ min_file_process_interval = 30
 # referenced and should be marked as orphaned.
 parsing_cleanup_interval = 60
 
+# How long (in seconds) to wait after we've reparsed a DAG file before deactivating stale
+# DAGs (DAGs which are no longer present in the expected files). The reason why we need
+# this threshold is to account for the time between when the file is parsed and when the
+# DAG is loaded. The absolute maximum that this could take is `dag_file_processor_timeout`,
+# but when you have a long timeout configured, it results in a significant delay in the
+# deactivation of stale dags.
+stale_dag_threshold = 30
+
 # How often (in seconds) to scan the DAGs directory for new files. Default to 5 minutes.
-dag_dir_list_interval = 300
+dag_dir_list_interval = 10

Review Comment:
   Sorry, accident



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org