Possible airflow-pool bug

[David Kegley <ke...@b23.io>]

I've been seeing some weird behavior when using Airflow's execution pool feature.  Pools have been massively over-filled leading to failed tasks. I created a bug for this issue, but in the mean time, has anyone else experienced this behavior?

https://issues.apache.org/jira/browse/AIRFLOW-584

Best,
David

Re: Possible airflow-pool bug

[David Kegley <ke...@b23.io>]

Hey Gerard, 

For our use case we have 1 pool per dag and each dag has between 50 and 100 tasks.  We have the pool size set between 3 and 5 in most cases and have seen up to 50 running tasks within the pool

Best,
David

> On Nov 1, 2016, at 11:17 AM, Gerard Toonstra <gt...@gmail.com> wrote:
> 
> David,
> 
> When you say "massive" oversubscribing, are you running a lot of dags in
> parallel that
> use your configured pools? Access to pools is not atomic at the moment.
> 
> Can you also quantify "massive" ?  Not that it matters, but to get a better
> idea.
> 
> Rgds,
> 
> Gerard
> 
> 
> On Tue, Nov 1, 2016 at 4:09 PM, siddharth anand <sa...@apache.org> wrote:
> 
>> Yes.. we have seen the over-subscription of pools. We do need a fix for it
>> --- I don't believe there is one. We need someone to own and fix it.. happy
>> to review a PR.
>> 
>> We use pools at Agari for all of our needs. We are okay with mild
>> oversubscription, so we do see numbers slightly higher, but our pipelines
>> work fine with that level. In your example, the oversubscription is much
>> higher.
>> -s
>> 
>> On Tue, Nov 1, 2016 at 6:38 AM, David Kegley <ke...@b23.io> wrote:
>> 
>>> I've been seeing some weird behavior when using Airflow's execution pool
>>> feature.  Pools have been massively over-filled leading to failed tasks.
>> I
>>> created a bug for this issue, but in the mean time, has anyone else
>>> experienced this behavior?
>>> 
>>> https://issues.apache.org/jira/browse/AIRFLOW-584
>>> 
>>> Best,
>>> David
>>> 
>> 

Re: Possible airflow-pool bug

[Gerard Toonstra <gt...@gmail.com>]

David,

When you say "massive" oversubscribing, are you running a lot of dags in
parallel that
use your configured pools? Access to pools is not atomic at the moment.

Can you also quantify "massive" ?  Not that it matters, but to get a better
idea.

Rgds,

Gerard


On Tue, Nov 1, 2016 at 4:09 PM, siddharth anand <sa...@apache.org> wrote:

> Yes.. we have seen the over-subscription of pools. We do need a fix for it
> --- I don't believe there is one. We need someone to own and fix it.. happy
> to review a PR.
>
> We use pools at Agari for all of our needs. We are okay with mild
> oversubscription, so we do see numbers slightly higher, but our pipelines
> work fine with that level. In your example, the oversubscription is much
> higher.
> -s
>
> On Tue, Nov 1, 2016 at 6:38 AM, David Kegley <ke...@b23.io> wrote:
>
> > I've been seeing some weird behavior when using Airflow's execution pool
> > feature.  Pools have been massively over-filled leading to failed tasks.
> I
> > created a bug for this issue, but in the mean time, has anyone else
> > experienced this behavior?
> >
> > https://issues.apache.org/jira/browse/AIRFLOW-584
> >
> > Best,
> > David
> >
>

Re: Possible airflow-pool bug

[siddharth anand <sa...@apache.org>]

Yes.. we have seen the over-subscription of pools. We do need a fix for it
--- I don't believe there is one. We need someone to own and fix it.. happy
to review a PR.

We use pools at Agari for all of our needs. We are okay with mild
oversubscription, so we do see numbers slightly higher, but our pipelines
work fine with that level. In your example, the oversubscription is much
higher.
-s

On Tue, Nov 1, 2016 at 6:38 AM, David Kegley <ke...@b23.io> wrote:

> I've been seeing some weird behavior when using Airflow's execution pool
> feature.  Pools have been massively over-filled leading to failed tasks. I
> created a bug for this issue, but in the mean time, has anyone else
> experienced this behavior?
>
> https://issues.apache.org/jira/browse/AIRFLOW-584
>
> Best,
> David
>