You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/11/10 10:56:44 UTC

[GitHub] [airflow] FreeDemon opened a new issue #19511: All tasks finished but dag runs state is running

FreeDemon opened a new issue #19511:
URL: https://github.com/apache/airflow/issues/19511


   ### Apache Airflow version
   
   2.2.1 (latest released)
   
   ### Operating System
   
   CentOS Linux 7.7
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow-providers-cncf-kubernetes==2.1.0
   apache-airflow-providers-ftp==2.0.1
   apache-airflow-providers-http==2.0.1
   apache-airflow-providers-imap==2.0.1
   apache-airflow-providers-sqlite==2.0.1
   
   ### Deployment
   
   Virtualenv installation
   
   ### Deployment details
   
   Install according to the official installation document
   use mysql as database
   use CeleryExecutor , 3 workers, worker_concurrency=32, parallelism= 96
   
   
   
   ### What happened
   
   I have 1000 DAGs files. Each DAGs has 4 tasks. Each task just outputs Hello word. Each DAGs is executed 10 times. When the task displays success, DAGs run still displays the running status. It takes a long time for the DAGs status to change to success, about 10 minutes
   
   ### What you expected to happen
   
   When all tasks are completed, the DAG run status should also be completed, rather than waiting for a long time
   
   ### How to reproduce
   
   _No response_
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #19511: With large number of DAGS, dag_run state takes long time to update after last task

Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #19511:
URL: https://github.com/apache/airflow/issues/19511#issuecomment-1014924181


   @ShakaibKhan any findings you can share?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ShakaibKhan commented on issue #19511: With large number of DAGS, dag_run state takes long time to update after last task

Posted by GitBox <gi...@apache.org>.
ShakaibKhan commented on issue #19511:
URL: https://github.com/apache/airflow/issues/19511#issuecomment-1001276962


   On v2.3.0.dev0 and I created 1000 DAGs w/ 4 task all printing hello world (Using Sequential Executor and default configs) and am able to see DAG status update pretty much immediately. I will try again in version 2.2 and using CeleryExecutor


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #19511: All tasks finished but dag runs state is running

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #19511:
URL: https://github.com/apache/airflow/issues/19511#issuecomment-965016104


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ShakaibKhan commented on issue #19511: With large number of DAGS, dag_run state takes long time to update after last task

Posted by GitBox <gi...@apache.org>.
ShakaibKhan commented on issue #19511:
URL: https://github.com/apache/airflow/issues/19511#issuecomment-1016519668


   Sorry, I have not made anymore findings. I do not think I can take on this issue at this time. Unassigning myself.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ShakaibKhan commented on issue #19511: With large number of DAGS, dag_run state takes long time to update after last task

Posted by GitBox <gi...@apache.org>.
ShakaibKhan commented on issue #19511:
URL: https://github.com/apache/airflow/issues/19511#issuecomment-982096365


   I would be interested in investigating this and potentially making a PR if this is on no one else's plate


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org