[I] DagFileProcessorManager silently hangs on DB lock contention during startup — no log, no timeout, no recovery [airflow]

["IggeeTianci (via GitHub)" <gi...@apache.org>]

IggeeTianci opened a new issue, #68101:
URL: https://github.com/apache/airflow/issues/68101

   ### Under which category would you file this issue?
   
   Airflow Core
   
   ### Apache Airflow version
   
   3.2.0
   
   ### What happened and how to reproduce it?
   
   What happened:
   
     The dag-processor silently hangs on startup when its initial DB queries (deactivate_deleted_dags(), _scan_stale_dags()) encounter row-level lock contention. The process logs "Found N files for bundle dags-folder" and then produces no further output — no workers are
     spawned, no heartbeat is emitted, and no error is logged. The liveness probe eventually kills it, but this creates a self-reinforcing crash-loop because SIGKILL leaves the in-flight transaction as an idle in transaction session on PostgreSQL, which continues holding the lock that blocks the next restart.
   
   What I think should happen instead:
   
   1. DB queries in the startup path (_refresh_dag_bundles → deactivate_deleted_dags, deactivate_stale_dags) should have a statement_timeout or use SELECT ... NOWAIT / SET lock_timeout so they fail fast rather than blocking indefinitely. 
     2. If a lock cannot be acquired within a reasonable timeout (e.g., 30s), the processor should log a clear warning (e.g., "Cannot acquire lock on dag table — possible zombie session holding locks").
     3. The heartbeat() call in _run_parsing_loop should happen on a background thread or before blocking DB operations, so the liveness probe doesn't kill a processor that's simply waiting on a lock. Currently, heartbeat only fires once per loop iteration — if the loop is blocked on a DB query, it never heartbeats.
   
     How to reproduce:
   
     1. Deploy Airflow 3 with dag_file_processor_timeout < DB lock hold time
     2. Have a DAG that produces non-deterministic serialization (e.g., fetches a Variable at parse time that changes frequently), causing serialized_dag and dag_version tables to grow
     3. Kill the dag-processor with SIGKILL while it's mid-transaction on dag_version
     4. The orphaned PostgreSQL session holds locks; the next dag-processor instance blocks silently on startup
   
   
     Workaround:
   
     - Manually terminate zombie sessions via SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE state = 'idle in transaction'
     - Set idle_in_transaction_session_timeout to a short value (e.g., 5 minutes) in the PostgreSQL/Aurora parameter group
   
     Suggested fix locations:
   
     - airflow-core/src/airflow/dag_processing/manager.py — _refresh_dag_bundles() (line ~650–720) and _run_parsing_loop() (line ~880)
     - Consider wrapping the startup DB operations with session.execute(text("SET LOCAL lock_timeout = '30s'")) or using SQLAlchemy's execution_options(timeout=30)
     - Consider moving heartbeat to a background timer so it's not gated on the main loop completing
   
     Operating environment:
   
     - Airflow 3.2.0
     - Aurora PostgreSQL 16 (serverless v2)
     - KubernetesExecutor, dedicated dag-processor deployment
     - Liveness probe: airflow jobs check --local --job-type DagProcessorJob with periodSeconds: 90, failureThreshold: 6
   
   ### What you think should happen instead?
   
   _No response_
   
   ### Operating System
   
   _No response_
   
   ### Deployment
   
   None
   
   ### Apache Airflow Provider(s)
   
   _No response_
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Official Helm Chart version
   
   Not Applicable
   
   ### Kubernetes Version
   
   _No response_
   
   ### Helm Chart configuration
   
   _No response_
   
   ### Docker Image customizations
   
   _No response_
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] DagFileProcessorManager silently hangs on DB lock contention during startup — no log, no timeout, no recovery [airflow]

["jscheffl (via GitHub)" <gi...@apache.org>]

jscheffl closed issue #68101: DagFileProcessorManager silently hangs on DB lock contention during startup — no log, no timeout, no recovery
URL: https://github.com/apache/airflow/issues/68101


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] DagFileProcessorManager silently hangs on DB lock contention during startup — no log, no timeout, no recovery [airflow]

["boring-cyborg[bot] (via GitHub)" <gi...@apache.org>]

boring-cyborg[bot] commented on issue #68101:
URL: https://github.com/apache/airflow/issues/68101#issuecomment-4635266559

   Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org