Assigning multiple pools to a task

[Neelesh <ne...@gmail.com>]

Hello,

Is it possible to assign multiple pools to a task?

I'm wrapping a bash operator around some heavy-weight custom data
processing script that cannot be modified. This script cannot be modified,
and it uses multiple resources. Im trying to use pools as proxies for these
resources.

Thanks

Re: Assigning multiple pools to a task

[Neelesh <ne...@gmail.com>]

Yes,  after looking at the code, it looks like implementing a many to many
between pools and tasks requires significant redesign.

My workaround is to implement it externally by borrowing required resources
from an external pool using a sensor, and returning them back once the task
is done using another sensor.  They are basically entry/exit sensors for a
task

Thx

On Sat, May 12, 2018, 2:28 PM Dennis O'Brien <de...@dennisobrien.net>
wrote:

> A task is assigned to a pool by the task specifying the name of the pool.
> The docs suggest that the pool argument is a string, not a list of strings.
> https://airflow.apache.org/code.html#baseoperator
>
> And looking at the code it does seem like this relationship of one task
> assigned to zero or one pools is strict.
>
> It's an interesting idea of trying to associated a task with multiple
> pools.  I think things would get complicated fast to avoid deadlock
> situations.  Do you know of any other work flow systems that support
> one-to-many relationship?
>
> Is it possible to bundle the resources into a single resource?  I'm
> guessing this is not an option, and that tasks use different combinations
> of these resources.
>
> cheers,
> Dennis
>
>
> On Fri, May 11, 2018 at 1:04 PM Neelesh <ne...@gmail.com> wrote:
>
> > Hello,
> >
> > Is it possible to assign multiple pools to a task?
> >
> > I'm wrapping a bash operator around some heavy-weight custom data
> > processing script that cannot be modified. This script cannot be
> modified,
> > and it uses multiple resources. Im trying to use pools as proxies for
> these
> > resources.
> >
> > Thanks
> >
>

Re: Assigning multiple pools to a task

[Dennis O'Brien <de...@dennisobrien.net>]

A task is assigned to a pool by the task specifying the name of the pool.
The docs suggest that the pool argument is a string, not a list of strings.
https://airflow.apache.org/code.html#baseoperator

And looking at the code it does seem like this relationship of one task
assigned to zero or one pools is strict.

It's an interesting idea of trying to associated a task with multiple
pools.  I think things would get complicated fast to avoid deadlock
situations.  Do you know of any other work flow systems that support
one-to-many relationship?

Is it possible to bundle the resources into a single resource?  I'm
guessing this is not an option, and that tasks use different combinations
of these resources.

cheers,
Dennis


On Fri, May 11, 2018 at 1:04 PM Neelesh <ne...@gmail.com> wrote:

> Hello,
>
> Is it possible to assign multiple pools to a task?
>
> I'm wrapping a bash operator around some heavy-weight custom data
> processing script that cannot be modified. This script cannot be modified,
> and it uses multiple resources. Im trying to use pools as proxies for these
> resources.
>
> Thanks
>