[GitHub] [airflow] potiuk commented on issue #30737: Failures in Celery worker can set task in indefininte queued state

["potiuk (via GitHub)" <gi...@apache.org>]

potiuk commented on issue #30737:
URL: https://github.com/apache/airflow/issues/30737#issuecomment-1516043300

   > We know the cause for the parsing errors and it has since been fixed, but the fact that the executor and the scheduler do not agree about the state of the task is a completely different issue. 
   
   FYI. It's a property of Airflow being distributed system and workers having to execute Python code. There is a DAG versionin  Airflow Improvement proposal (it is in a draft form and has not been approved yet) https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-36+DAG+Versioning - and anyone in the community (your company ? ) is free to volunteer and implement it. 
   
   One of the properties of DAG versioning should be that the version of DAG Python file (and all releated imports) used to parse the structure of the DAG should be somewhat frozen for the whole execution time (including backfill execution of those if you choose so). That SHOUDL solve the problem, as long as you do not do a dynamic DAG generation with using external sources. 
   
   If you are using external sources (database, Pulling configuration etc.) while the DAG is parsed, there is *absolutely nothing* Airflow can do, and this is somethign tht is considered as a very bad practice that you should never, ever do https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html#top-level-python-code, especially that there are ways of doing it that are recommended https://airflow.apache.org/docs/apache-airflow/stable/howto/dynamic-dag-generation.html 
   
   So yeah, this might be another issue, but if you are violating best practices, this is your issue, not Airflow's.
   
   
   > Also, how is "queued" a valid end-state for a task
   
   It shoudl not be. And in Airflow 2.6.0 (that is going to be released in a few days most likely) this has been already fixed and we even have a nice blog explaining it published yestterday:
   
   https://medium.com/apache-airflow/unsticking-airflow-stuck-queued-tasks-are-no-more-in-2-6-0-6f40a1a22835
   
   BTW. Airflow 2.6.0rc will be out in a few days, and subscribing to devlist and testing it and reporting any problems is a great way you can help with getting things improved.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org