[jira] [Commented] (AIRFLOW-910) Parallelize dag runs in backfills

["ASF subversion and git services (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15906263#comment-15906263 ] 

ASF subversion and git services commented on AIRFLOW-910:
---------------------------------------------------------

Commit d79ed74783048b2816a4005ba887c2985e753ecf in incubator-airflow's branch refs/heads/master from [~bolke]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=d79ed74 ]

[AIRFLOW-910] Use parallel task execution for backfills

The refactor to use dag runs in backfills caused a
regression
in task execution performance as dag runs were
executed
sequentially. Next to that, the backfills were non
deterministic
due to the random execution of tasks, causing root
tasks
being added to the non ready list too soon.

This updates the backfill logic as follows:
* Parallelize execution of tasks
* Use a leave first execution model
* Replace state updates from the executor by task
based only

Closes #2107 from bolkedebruin/AIRFLOW-910


> Parallelize dag runs in backfills
> ---------------------------------
>
>                 Key: AIRFLOW-910
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-910
>             Project: Apache Airflow
>          Issue Type: Sub-task
>          Components: backfill
>    Affects Versions: 1.8.0rc4
>            Reporter: Bolke de Bruin
>            Assignee: Bolke de Bruin
>            Priority: Blocker
>             Fix For: 1.8.0
>
>
> Currently dag runs are executed sequentially while backfilling. This is a regression and slows down the processing off tasks.
> [~aoen]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)