You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by "Blake Livingston (JIRA)" <ji...@apache.org> on 2019/05/08 13:28:00 UTC

[jira] [Created] (AIRFLOW-4476) Dag runs multiple times if the first node raises an AirflowSkipException

Blake Livingston created AIRFLOW-4476:
-----------------------------------------

             Summary: Dag runs multiple times if the first node raises an AirflowSkipException
                 Key: AIRFLOW-4476
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-4476
             Project: Apache Airflow
          Issue Type: Bug
          Components: scheduler
    Affects Versions: 1.10.3
            Reporter: Blake Livingston


We have a DAG that runs nightly - the first task checks a service to see if there is work to do. If there is no work, it raises an AirflowSkipException to short circuit itself and the rest of the DAG.
What we noticed, though, is that the DAG would keep re-scheduling for multiple times during the 'minute' that its cron-spec specified it should run. 

!image-2019-05-08-09-26-37-391.png!

A workaround is to have the sensor node sleep for a minute before raising the AirflowSkipException.

I'm guessing that when the Scheduler checks for existing Dag runs, it doesn't count a skipped initial task as a valid run, and keeps scheduling.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)