You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@airflow.apache.org by ra...@gmail.com, ra...@gmail.com on 2018/05/23 16:22:21 UTC

Scheduler gets slow with increasing number of Dags

Hi All,

We have a use case where there are 100s of DAGs in the scheduler's local dag folder but at a time only ~50 dags are active(Other dags are disabled). New dags keep on adding to the local Dag folder.
We are observing that scheduler is taking lot of time(around 20 minutes) in picking those newly created dags.
We are using following conf
Machine config: 8 core, 16 GB RAM
Executor: LocalExecutor
Parallelism: 50
max_thread: 4,
Dir list interval: 60.

Is there any recommended setting or config to optimise the scheduler's dag listing and parsing.

Thanks,
Raman