[jira] [Created] (AIRFLOW-5858) Airflow celery worker missing heartbeat

["Nikhil SInghal (Jira)" <ji...@apache.org>]

Nikhil SInghal created AIRFLOW-5858:
---------------------------------------

             Summary: Airflow celery worker missing heartbeat
                 Key: AIRFLOW-5858
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-5858
             Project: Apache Airflow
          Issue Type: Bug
          Components: celery
    Affects Versions: 1.10.2
            Reporter: Nikhil SInghal


Our Airflow setup uses Celery Executors and Redis as a broker. We are facing a issue of missing heartbeat from celery workers. Once this heartbeat is missed the worker stops taking any new task. It also appears as offline in the Celery flower UI. Manual restart of workers fixes this problem.

Wanted to know if this is a known issue or being faced by other users in the community

These are the logs from failure. 
[2019-11-06 02:30:36,368: INFO/MainProcess] missed heartbeat from celery@dp-airflow-worker-6cb4b596f8-nzdrt

worker: Warm shutdown (MainProcess)

 -------------- celery@dp-airflow-worker-6cb4b596f8-4qkfg v4.1.1 (latentcall)
---- **** -----
--- * ***  * -- Linux-4.9.0-9-amd64-x86_64-with-debian-10.1 2019-11-05 15:24:34
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app:         airflow.executors.celery_executor:0x7f2a01250cf8
- ** ---------- .> transport:   redis://:**@redis-11313.internal.c3160.ap-southeast-1-mz.ec2.cloud.rlrcp.com:11313//
- ** ---------- .> results: postgresql://airflow:**@airflowdbprod.ckvce9fjaook.ap-southeast-1.rds.amazonaws.com:5432/airflowdb    
- *** --- * --- .> concurrency: 64 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
 -------------- [queues]
                .> default          exchange=default(direct) key=default


[tasks]
  . airflow.executors.celery_executor.execute_command



--
This message was sent by Atlassian Jira
(v8.3.4#803005)