You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by ka...@apache.org on 2025/06/03 09:42:18 UTC

(airflow) 01/02: Add "disappearing DAGs" section on FAQ doc (#49987)

This is an automated email from the ASF dual-hosted git repository.

kaxilnaik pushed a commit to branch v3-0-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit e06fb664c044f377d2e8f8d6d5da6f5a4a40215d
Author: rtrindvg <82...@users.noreply.github.com>
AuthorDate: Mon Jun 2 05:07:59 2025 -0300

    Add "disappearing DAGs" section on FAQ doc (#49987)
    
    ---------
    
    Co-authored-by: Amogh Desai <am...@gmail.com>
    (cherry picked from commit b89044da97efba3ef43ec6c2441c7026ab61fc69)
---
 airflow-core/docs/faq.rst | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)

diff --git a/airflow-core/docs/faq.rst b/airflow-core/docs/faq.rst
index 8727deac08e..9eb5fa79ad0 100644
--- a/airflow-core/docs/faq.rst
+++ b/airflow-core/docs/faq.rst
@@ -206,6 +206,37 @@ until ``min_file_process_interval`` is reached since DAG Parser will look for mo
 
         return dag
 
+What to do if you see disappearing DAGs on UI?
+----------------------------------------------
+There are several reasons why DAGs might disappear from the UI. Common causes include:
+
+* **Total parsing of all DAGs is too long** - If parsing takes longer than :ref:`config:core__dagbag_import_timeout`,
+  files may not be processed completely. This often occurs when DAGs don't follow
+  :ref:`DAG writing best practices<best_practice:writing_a_dag>` like:
+
+  * Excessive top-level code execution
+  * External system calls during parsing
+  * Complex dynamic DAG generation
+
+* **Inconsistent dynamic DAG generation** - DAGs created through
+  :doc:`dynamic generation </howto/dynamic-dag-generation>` must produce stable DAG IDs across parses.
+  Verify consistency by running ``python your_dag_file.py`` repeatedly.
+
+* **File processing configuration issues** - A certain combination of parameters may lead to scenarios which certain DAGs are less likely to be processed at each loop. Check these parameters:
+
+  * :ref:`config:dag_processor__file_parsing_sort_mode` - Ensure sorting method matches your sync strategy
+  * :ref:`config:dag_processor__parsing_processes` - Number of parallel parsers
+  * :ref:`config:scheduler__parsing_cleanup_interval` - Controls stale DAG cleanup frequency
+  * :ref:`config:scheduler__dag_stale_not_seen_duration` - Time threshold for marking DAGs as stale
+
+* **File synchronization problems** - Common with git-sync setups:
+
+  * Symbolic link swapping delays
+  * Permission changes during sync
+  * `mtime` preservation issues
+
+* **Time synchronization issues** - Ensure all nodes (database, schedulers, workers) use NTP with <1s clock drift.
+
 
 DAG construction
 ^^^^^^^^^^^^^^^^