You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Dan Davydov (JIRA)" <ji...@apache.org> on 2016/11/29 20:21:58 UTC
[jira] [Updated] (AIRFLOW-536) Dags only get one dagrun is
scheduled per scheduler loop
[ https://issues.apache.org/jira/browse/AIRFLOW-536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dan Davydov updated AIRFLOW-536:
--------------------------------
Description:
At the moment Dags only get one dag scheduled per scheduler loop (which can be quite slow, on the order of minutes), but what should happen is that all pending dagruns for a dag should be scheduled.
In addition to dagruns being scheduled more quickly for dags that havem ultiple dag runs queued up (e.g. paused dags), fixing this issue will enable a "only_run_latest" option for dags (so e.g. if the dag is paused and then later unpaused, only the most recent ds will be run), as well as facilitating compatibility between backfills and the scheduler (if you run a backfill over 100 dag runs the backfill will be waiting on 100 scheduler cycles).
was:
At the moment Dags only get one dag scheduled per scheduler loop (which can be quite slow, on the order of minutes), but what should happen is that all pending dagruns for a dagrun should be scheduled.
In addition to dagruns being scheduled more quickly for dags that havem ultiple dag runs queued up (e.g. paused dags), fixing this issue will enable a "only_run_latest" option for dags (so e.g. if the dag is paused and then later unpaused, only the most recent ds will be run), as well as facilitating compatibility between backfills and the scheduler (if you run a backfill over 100 dag runs the backfill will be waiting on 100 scheduler cycles).
> Dags only get one dagrun is scheduled per scheduler loop
> --------------------------------------------------------
>
> Key: AIRFLOW-536
> URL: https://issues.apache.org/jira/browse/AIRFLOW-536
> Project: Apache Airflow
> Issue Type: Bug
> Reporter: Dan Davydov
>
> At the moment Dags only get one dag scheduled per scheduler loop (which can be quite slow, on the order of minutes), but what should happen is that all pending dagruns for a dag should be scheduled.
> In addition to dagruns being scheduled more quickly for dags that havem ultiple dag runs queued up (e.g. paused dags), fixing this issue will enable a "only_run_latest" option for dags (so e.g. if the dag is paused and then later unpaused, only the most recent ds will be run), as well as facilitating compatibility between backfills and the scheduler (if you run a backfill over 100 dag runs the backfill will be waiting on 100 scheduler cycles).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)