You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/12/06 10:46:48 UTC

[GitHub] [airflow] ohadmata opened a new issue #12849: Scheduler is slowly due to many local log files

ohadmata opened a new issue #12849:
URL: https://github.com/apache/airflow/issues/12849


   Hi, I have a strange behavior in my production airflow:
   The scheduler is VERY slowly, it's takes about five minutes to start a certain task after all of the previous tasks has completed.
   After the deletion of the logs files (local logs) it's become super fast.
   
   I searched in the web with no solution, 
   All of the slowly-scheduler issues are related to many DAG files and not to many log files.
   Why the scheduler is affected by the number of logs?
   
   In addition, I saw about 5k logs directories (each one for a different DAG) while I have only 200 DAGS in my airflow instance.
   The airflow shouldn't remove his old logs for deleted dags?
   
   thanks
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ashb commented on issue #12849: Scheduler is slowly due to many local log files

Posted by GitBox <gi...@apache.org>.
ashb commented on issue #12849:
URL: https://github.com/apache/airflow/issues/12849#issuecomment-749010862


   Please try out 2.0.0 -- the number of files should no longer have an impact on the scheduler performance


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ohadmata commented on issue #12849: Scheduler is slowly due to many local log files

Posted by GitBox <gi...@apache.org>.
ohadmata commented on issue #12849:
URL: https://github.com/apache/airflow/issues/12849#issuecomment-749012355


   @ashb I will try, Thanks!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ohadmata closed issue #12849: Scheduler is slowly due to many local log files

Posted by GitBox <gi...@apache.org>.
ohadmata closed issue #12849:
URL: https://github.com/apache/airflow/issues/12849


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #12849: Scheduler is slowly due to many local log files

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #12849:
URL: https://github.com/apache/airflow/issues/12849#issuecomment-739484989


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org