[jira] [Assigned] (AIRFLOW-2684) scheduler_heartbeat_sec is not respected

["Ash Berlin-Taylor (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-2684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ash Berlin-Taylor reassigned AIRFLOW-2684:
------------------------------------------

    Assignee: Ash Berlin-Taylor

> scheduler_heartbeat_sec is not respected
> ----------------------------------------
>
>                 Key: AIRFLOW-2684
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2684
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: scheduler
>    Affects Versions: 2.0.0
>            Reporter: James Meickle
>            Assignee: Ash Berlin-Taylor
>            Priority: Major
>
> In our staging environment, we leave Airflow debug logs on. We're generating immense amounts of logs because for both our 1.9 cluster and 1.10 cluster, scheduler_heartbeat_sec is not being respected. It's set to 5 in both environments, but produces logs like this every second:
>  
> ```
> airflow_scheduler-stdout.log: [2018-06-27 13:47:20,272] \{{jobs.py:1685}} DEBUG - Ran scheduling loop in 0.00 seconds
> airflow_scheduler-stdout.log: [2018-06-27 13:47:20,273] \{{jobs.py:1686}} DEBUG - Sleeping for 1.00 seconds
> airflow_scheduler-stdout.log: [2018-06-27 13:47:21,273] \{{jobs.py:1607}} DEBUG - Starting Loop...
> airflow_scheduler-stdout.log: [2018-06-27 13:47:21,274] \{{jobs.py:1627}} INFO - Heartbeating the process manager
> airflow_scheduler-stdout.log: [2018-06-27 13:47:21,274] \{{dag_processing.py:523}} DEBUG - Queuing the following files for processing:
> airflow_scheduler-stdout.log: [2018-06-27 13:47:21,275] \{{jobs.py:1662}} INFO - Heartbeating the executor
> airflow_scheduler-stdout.log: [2018-06-27 13:47:21,275] \{{base_executor.py:103}} DEBUG - 0 running task instances
> airflow_scheduler-stdout.log: [2018-06-27 13:47:21,275] \{{base_executor.py:104}} DEBUG - 0 in queue
> airflow_scheduler-stdout.log: [2018-06-27 13:47:21,276] \{{base_executor.py:105}} DEBUG - 32 open slots
> airflow_scheduler-stdout.log: [2018-06-27 13:47:21,276] \{{base_executor.py:132}} DEBUG - Calling the <class 'airflow.executors.celery_executor.CeleryExecutor'> sync method
> airflow_scheduler-stdout.log: [2018-06-27 13:47:21,276] \{{celery_executor.py:80}} DEBUG - Inquiring about 0 celery task(s)
> airflow_scheduler-stdout.log: [2018-06-27 13:47:21,277] \{{jobs.py:1685}} DEBUG - Ran scheduling loop in 0.00 seconds
> airflow_scheduler-stdout.log: [2018-06-27 13:47:21,277] \{{jobs.py:1686}} DEBUG - Sleeping for 1.00 seconds
> ```



--
This message was sent by Atlassian Jira
(v8.3.4#803005)