[jira] [Updated] (AIRFLOW-2317) Support for multiple resource pools

["Aizhamal Nurmamat kyzy (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-2317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aizhamal Nurmamat kyzy updated AIRFLOW-2317:
--------------------------------------------
         Labels: pool  (was: )
    Component/s: scheduler

Adding "scheduler" component and "pool" label to issues for component refactor.

> Support for multiple resource pools
> -----------------------------------
>
>                 Key: AIRFLOW-2317
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2317
>             Project: Apache Airflow
>          Issue Type: Wish
>          Components: pools, scheduler
>            Reporter: James Davidheiser
>            Priority: Minor
>              Labels: pool
>
> We are migrating to Airflow from Luigi, where we have the capability to require multiple pools, and multiple "units" of a pool, for a given task.  This is very useful for a variety of use cases, but the two core examples are:
>  * If a task accessing a data store is extremely resource-intensive, we might define a pool size of 10 resources, but use 5 of those in a single task to denote the extra load.  When smaller tasks are using the same data store, it's completely fine to allow more of them to run at once.
> If a task is connecting to two different data stores, to transfer data between them, we might want to require a resource for each data store, so we can limit concurrency on both simultaneously.
>  
>  
> I know there are a lot of other issues related to capacity scheduling, which are tracked more broadly in https://issues.apache.org/jira/browse/AIRFLOW-72, and using more than one pool slot was suggested in https://issues.apache.org/jira/browse/AIRFLOW-1467, so this seems like it could be a useful thing to consider as part of larger efforts around managing capacity.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)