[jira] [Updated] (AIRFLOW-193) Allow a series of tasks to be executed on the same worker

["Sergei Iakhnin (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sergei Iakhnin updated AIRFLOW-193:
-----------------------------------
    Affects Version/s: Airflow 1.7.1.2

> Allow a series of tasks to be executed on the same worker
> ---------------------------------------------------------
>
>                 Key: AIRFLOW-193
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-193
>             Project: Apache Airflow
>          Issue Type: New Feature
>    Affects Versions: Airflow 1.7.1.2
>            Reporter: Sergei Iakhnin
>
> Currently the only way to limit the execution of a series of tasks to a single worker is via pools, however this is not a very convenient method when managing hundreds of workers.
> In the context of scientific workflows it is a common desire to be able to retrieve a (possibly large) sample from a data repository (or object store), then progressively elaborate it via a series of transformations, and finally deposit the result back. From a modelling perspective it makes sense to have the series of transformations each be encapsulated in a separate task. From practical considerations (performance, network bandwidth) it would be desirable to retrieve the sample to a single worker's local storage, where it would then be worked upon until completion. This, of course, requires the ability to have a slew of tasks to be bound to a particular worker. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)