[jira] [Created] (AIRFLOW-3943) ExternalTaskSensor are randomly not being triggered (staying idle)

["Christophe (JIRA)" <ji...@apache.org>]

Christophe created AIRFLOW-3943:
-----------------------------------

             Summary: ExternalTaskSensor are randomly not being triggered (staying idle)
                 Key: AIRFLOW-3943
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-3943
             Project: Apache Airflow
          Issue Type: Bug
          Components: DagRun, scheduler
    Affects Versions: 1.10.2
            Reporter: Christophe


Hello, I don't know how to reproduce the error but with guidance I can go to log or somewhere else to search more information if needed.

Since we upgraded from 1.10.1 to 1.10.2 it appeared that 3 days out of 10 days — where everything was running fine — a subset of our ExternalTaskSensor stay in idle time and are not triggered / runned.

To explain more, we have daily ingestion jobs that runs a 2' and afterwards we have processing jobs with sensors that waits for ingestion to be run.

If the first Gantt (dag_ingestion.jpg) we see the ingestion part for one table saying `A`, this ingestion ends up around 3:23. 

In the second Gantt (dag_processing.jpg) for the processing we see that the Sensor is waiting for a long time without doing anything — passing green at 9:30 because we manually triggered the DAG with a clear. 

In the logs we see that the task is running once at 2:22 and the nothing is happening until the manual trigger at 8:32 (9:32 UTC for us). 

I don't know how to debug this and the issue is happening randomly on different sensors.

When we where on 1.10.1 everything was running smoothly. 

PS : I can't uplaod screen on Jira, I don't know why I have some issues :(



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)