You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/09/22 21:56:50 UTC

[GitHub] [airflow] StewartThomson opened a new issue #18446: Task Failure when using dynamic dags

StewartThomson opened a new issue #18446:
URL: https://github.com/apache/airflow/issues/18446


   ### Apache Airflow version
   
   2.1.2
   
   ### Operating System
   
   Docker on Amazon Linux 2
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Docker-Compose
   
   ### Deployment details
   
   _No response_
   
   ### What happened
   
   Dynamic dags sporadically result in a scheduler error of:
   `ERROR - Executor reports task instance <TaskInstance: {task_id} 2021-09-21 00:00:00+00:00 [queued]> finished (failed) although the task says its queued. (Info: None) Was the task killed externally?`
   
   This pretty much breaks the whole DAG. The task instance gets stuck in a queued state, and the DAG is unable to create new DAG runs.
   
   It's worth noting that nothing is killing the task, and this only happens for dynamic dags. No other DAGs in the system are affected by these new dags.
   
   ### What you expected to happen
   
   I expect the tasks to not be stuck queued, and I really expect the DAG to be able to create new dag runs.
   
   ### How to reproduce
   
   1. Create a file that produces multiple dynamic DAGs
   2. Create multiple DAG runs on any of the dynamic dags
   3. Wait. Some number of tasks will become stuck in a queued state, the scheduler will produce the error above, and no new DAG runs will be able to be created.
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] StewartThomson commented on issue #18446: Task Failure when using dynamic dags

Posted by GitBox <gi...@apache.org>.
StewartThomson commented on issue #18446:
URL: https://github.com/apache/airflow/issues/18446#issuecomment-925904494


   I've tracked down the source of my woes.
   
   The dags were being created directly in a loop
   
   Creating and returning the dag in a function as outline here: https://www.astronomer.io/guides/dynamically-generating-dags
   solved my issue.
   
   Closing, thanks for your response.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] StewartThomson closed issue #18446: Task Failure when using dynamic dags

Posted by GitBox <gi...@apache.org>.
StewartThomson closed issue #18446:
URL: https://github.com/apache/airflow/issues/18446


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] Jorricks commented on issue #18446: Task Failure when using dynamic dags

Posted by GitBox <gi...@apache.org>.
Jorricks commented on issue #18446:
URL: https://github.com/apache/airflow/issues/18446#issuecomment-925679850


   Hello @StewartThomson,
   
   I'd love to look into the issue but I'm not sure I fully understand the setup.
   Can you create a minimal example of a Dynamic DAG with this issue?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org