You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by "Blake Livingston (JIRA)" <ji...@apache.org> on 2019/05/08 13:46:00 UTC

[jira] [Closed] (AIRFLOW-4476) Dag runs multiple times if the first node raises an AirflowSkipException

     [ https://issues.apache.org/jira/browse/AIRFLOW-4476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Blake Livingston closed AIRFLOW-4476.
-------------------------------------
    Resolution: Invalid

On further testing, the workaround doesnt seem to work

> Dag runs multiple times if the first node raises an AirflowSkipException
> ------------------------------------------------------------------------
>
>                 Key: AIRFLOW-4476
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-4476
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: scheduler
>    Affects Versions: 1.10.3
>            Reporter: Blake Livingston
>            Priority: Minor
>
> We have a DAG that runs nightly - the first task checks a service to see if there is work to do. If there is no work, it raises an AirflowSkipException to short circuit itself and the rest of the DAG.
>  What we noticed, though, is that the DAG would keep re-scheduling for multiple times during the 'minute' that its cron-spec specified it should run. 
> A workaround is to have the sensor node sleep for a minute before raising the AirflowSkipException.
> I'm guessing that when the Scheduler checks for existing Dag runs, it doesn't count a skipped initial task as a valid run, and keeps scheduling.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)