You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "James Meickle (JIRA)" <ji...@apache.org> on 2018/06/27 14:20:00 UTC
[jira] [Commented] (AIRFLOW-2684) scheduler_heartbeat_sec is not
respected
[ https://issues.apache.org/jira/browse/AIRFLOW-2684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525133#comment-16525133 ]
James Meickle commented on AIRFLOW-2684:
----------------------------------------
Or perhaps it's `job_heartbeat_sec` that is used here, it's unclear. Both are set to 5, but we still get secondly executor heartbeats.
> scheduler_heartbeat_sec is not respected
> ----------------------------------------
>
> Key: AIRFLOW-2684
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2684
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 1.10
> Reporter: James Meickle
> Priority: Major
>
> In our staging environment, we leave Airflow debug logs on. We're generating immense amounts of logs because for both our 1.9 cluster and 1.10 cluster, scheduler_heartbeat_sec is not being respected. It's set to 5 in both environments, but produces logs like this every second:
>
> ```
> airflow_scheduler-stdout.log: [2018-06-27 13:47:20,272] \{{jobs.py:1685}} DEBUG - Ran scheduling loop in 0.00 seconds
> airflow_scheduler-stdout.log: [2018-06-27 13:47:20,273] \{{jobs.py:1686}} DEBUG - Sleeping for 1.00 seconds
> airflow_scheduler-stdout.log: [2018-06-27 13:47:21,273] \{{jobs.py:1607}} DEBUG - Starting Loop...
> airflow_scheduler-stdout.log: [2018-06-27 13:47:21,274] \{{jobs.py:1627}} INFO - Heartbeating the process manager
> airflow_scheduler-stdout.log: [2018-06-27 13:47:21,274] \{{dag_processing.py:523}} DEBUG - Queuing the following files for processing:
> airflow_scheduler-stdout.log: [2018-06-27 13:47:21,275] \{{jobs.py:1662}} INFO - Heartbeating the executor
> airflow_scheduler-stdout.log: [2018-06-27 13:47:21,275] \{{base_executor.py:103}} DEBUG - 0 running task instances
> airflow_scheduler-stdout.log: [2018-06-27 13:47:21,275] \{{base_executor.py:104}} DEBUG - 0 in queue
> airflow_scheduler-stdout.log: [2018-06-27 13:47:21,276] \{{base_executor.py:105}} DEBUG - 32 open slots
> airflow_scheduler-stdout.log: [2018-06-27 13:47:21,276] \{{base_executor.py:132}} DEBUG - Calling the <class 'airflow.executors.celery_executor.CeleryExecutor'> sync method
> airflow_scheduler-stdout.log: [2018-06-27 13:47:21,276] \{{celery_executor.py:80}} DEBUG - Inquiring about 0 celery task(s)
> airflow_scheduler-stdout.log: [2018-06-27 13:47:21,277] \{{jobs.py:1685}} DEBUG - Ran scheduling loop in 0.00 seconds
> airflow_scheduler-stdout.log: [2018-06-27 13:47:21,277] \{{jobs.py:1686}} DEBUG - Sleeping for 1.00 seconds
> ```
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)