You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Blake Livingston (JIRA)" <ji...@apache.org> on 2019/05/08 13:29:00 UTC
[jira] [Updated] (AIRFLOW-4476) Dag runs multiple times if the
first node raises an AirflowSkipException
[ https://issues.apache.org/jira/browse/AIRFLOW-4476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Blake Livingston updated AIRFLOW-4476:
--------------------------------------
Description:
We have a DAG that runs nightly - the first task checks a service to see if there is work to do. If there is no work, it raises an AirflowSkipException to short circuit itself and the rest of the DAG.
What we noticed, though, is that the DAG would keep re-scheduling for multiple times during the 'minute' that its cron-spec specified it should run.
A workaround is to have the sensor node sleep for a minute before raising the AirflowSkipException.
I'm guessing that when the Scheduler checks for existing Dag runs, it doesn't count a skipped initial task as a valid run, and keeps scheduling.
was:
We have a DAG that runs nightly - the first task checks a service to see if there is work to do. If there is no work, it raises an AirflowSkipException to short circuit itself and the rest of the DAG.
What we noticed, though, is that the DAG would keep re-scheduling for multiple times during the 'minute' that its cron-spec specified it should run.
!image-2019-05-08-09-26-37-391.png!
A workaround is to have the sensor node sleep for a minute before raising the AirflowSkipException.
I'm guessing that when the Scheduler checks for existing Dag runs, it doesn't count a skipped initial task as a valid run, and keeps scheduling.
> Dag runs multiple times if the first node raises an AirflowSkipException
> ------------------------------------------------------------------------
>
> Key: AIRFLOW-4476
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4476
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 1.10.3
> Reporter: Blake Livingston
> Priority: Minor
>
> We have a DAG that runs nightly - the first task checks a service to see if there is work to do. If there is no work, it raises an AirflowSkipException to short circuit itself and the rest of the DAG.
> What we noticed, though, is that the DAG would keep re-scheduling for multiple times during the 'minute' that its cron-spec specified it should run.
> A workaround is to have the sensor node sleep for a minute before raising the AirflowSkipException.
> I'm guessing that when the Scheduler checks for existing Dag runs, it doesn't count a skipped initial task as a valid run, and keeps scheduling.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)