You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/03/27 20:57:38 UTC
[GitHub] [airflow] dimberman opened a new issue #7920: Allow a series of
tasks to be executed on the same worker
dimberman opened a new issue #7920: Allow a series of tasks to be executed on the same worker
URL: https://github.com/apache/airflow/issues/7920
**Description**
Currently the only way to limit the execution of a series of tasks to a single worker is via pools, however this is not a very convenient method when managing hundreds of workers.
In the context of scientific workflows it is a common desire to be able to retrieve a (possibly large) sample from a data repository (or object store), then progressively elaborate it via a series of transformations, and finally deposit the result back. From a modelling perspective it makes sense to have the series of transformations each be encapsulated in a separate task. From practical considerations (performance, network bandwidth) it would be desirable to retrieve the sample to a single worker's local storage, where it would then be worked upon until completion. This, of course, requires the ability to have a slew of tasks to be bound to a particular worker.
**Use case / motivation**
**Related Issues**
Moved here from https://issues.apache.org/jira/browse/AIRFLOW-193
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] dimberman closed issue #7920: Allow a series of tasks to
be executed on the same worker
Posted by GitBox <gi...@apache.org>.
dimberman closed issue #7920: Allow a series of tasks to be executed on the same worker
URL: https://github.com/apache/airflow/issues/7920
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services