You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Jake Boggan (JIRA)" <ji...@apache.org> on 2018/03/28 21:28:00 UTC
[jira] [Commented] (AIRFLOW-2201) Scheduler processes dags every
second
[ https://issues.apache.org/jira/browse/AIRFLOW-2201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16418144#comment-16418144 ]
Jake Boggan commented on AIRFLOW-2201:
--------------------------------------
I am having an identical problem with 1.8.2 - setting `worker_refresh_batch_size = 0` doesn't seem to help as some have suggested.
> Scheduler processes dags every second
> -------------------------------------
>
> Key: AIRFLOW-2201
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2201
> Project: Apache Airflow
> Issue Type: Bug
> Affects Versions: 1.9.0
> Reporter: John Arnold
> Priority: Critical
>
> Building latest master.
>
> The scheduler appears to be running continuously, not respecting scheduler_heartbeat_sec = 30
>
> Logs are filled up with messages like:
> Mar8 23:20:36 netdocker1-eastus2 INFO :[2018-03-08 23:20:36,826] \{dag_processing.py:468} INFO - Processor for /var/lib/airflow/dags/test_nt.py finished
> Mar8 23:20:36 netdocker1-eastus2 INFO :[2018-03-08 23:20:36,831] \{dag_processing.py:543} INFO - Started a process (PID: 64) to generate tasks for /var/lib/airflow/dags/test_nt.py
> Mar8 23:20:37 netdocker1-eastus2 INFO :[2018-03-08 23:20:37,834] \{dag_processing.py:468} INFO - Processor for /var/lib/airflow/dags/test_nt.py finished
> Mar8 23:20:37 netdocker1-eastus2 INFO :[2018-03-08 23:20:37,838] \{dag_processing.py:543} INFO - Started a process (PID: 66) to generate tasks for /var/lib/airflow/dags/test_nt.py
> Mar8 23:20:38 netdocker1-eastus2 INFO :[2018-03-08 23:20:38,842] \{dag_processing.py:468} INFO - Processor for /var/lib/airflow/dags/test_nt.py finished
> Mar8 23:20:38 netdocker1-eastus2 INFO :[2018-03-08 23:20:38,846] \{dag_processing.py:543} INFO - Started a process (PID: 68) to generate tasks for /var/lib/airflow/dags/test_nt.py
> Mar8 23:20:39 netdocker1-eastus2 INFO :[2018-03-08 23:20:39,850] \{dag_processing.py:468} INFO - Processor for /var/lib/airflow/dags/test_nt.py finished
> Mar8 23:20:39 netdocker1-eastus2 INFO :[2018-03-08 23:20:39,856] \{dag_processing.py:543} INFO - Started a process (PID: 70) to generate tasks for /var/lib/airflow/dags/test_nt.py
> Mar8 23:20:40 netdocker1-eastus2 INFO :[2018-03-08 23:20:40,863] \{dag_processing.py:468} INFO - Processor for /var/lib/airflow/dags/test_nt.py finished
> Mar8 23:20:40 netdocker1-eastus2 INFO :[2018-03-08 23:20:40,867] \{dag_processing.py:543} INFO - Started a process (PID: 72) to generate tasks for /var/lib/airflow/dags/test_nt.py
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)