DAGs not running

[David Klosowski <da...@thinknear.com>]

Hey Airflow Devs:

We noticed an issue with task instances not running sometimes.

Some interesting details are:
> in the database it shows that specific tasks that are "stuck" have an
end_date but no state (null state)
> Airflow has been running for several days just fine and there were no
changes to Airflow in that period.
> it happened for two separate important tasks for us at the same time

We're running:
> Airflow 1.8.0 with a Postgres backend
> Celery with Redis backed by Postgres result_backend
> 3 worker hosts and 1 scheduler host

Let me know if you guys have any suggestions or advice.  Going to look
through the code to see if I can see where/how this might be happening, but
any suggestions or well known issues that might point to the problem might
be helpful.

Thanks.

Regards,
David

Re: DAGs not running

[David Klosowski <da...@thinknear.com>]

Thanks Bolke,

That's good to know.  We were looking into upgrading, so this gives us more
motivation ;-)

Cheers,
David


On Fri, Sep 22, 2017 at 11:57 AM, Bolke de Bruin <bd...@gmail.com> wrote:

> Please upgrade to 1.8.2. The were some ‘gaps’ in the 1.8.0 scheduler.
>
> Cheers
> Bolke
>
> > On 22 Sep 2017, at 19:54, David Klosowski <da...@thinknear.com> wrote:
> >
> > Hey Airflow Devs:
> >
> > We noticed an issue with task instances not running sometimes.
> >
> > Some interesting details are:
> >> in the database it shows that specific tasks that are "stuck" have an
> > end_date but no state (null state)
> >> Airflow has been running for several days just fine and there were no
> > changes to Airflow in that period.
> >> it happened for two separate important tasks for us at the same time
> >
> > We're running:
> >> Airflow 1.8.0 with a Postgres backend
> >> Celery with Redis backed by Postgres result_backend
> >> 3 worker hosts and 1 scheduler host
> >
> > Let me know if you guys have any suggestions or advice.  Going to look
> > through the code to see if I can see where/how this might be happening,
> but
> > any suggestions or well known issues that might point to the problem
> might
> > be helpful.
> >
> > Thanks.
> >
> > Regards,
> > David
>
>

Re: DAGs not running

[Bolke de Bruin <bd...@gmail.com>]

Please upgrade to 1.8.2. The were some ‘gaps’ in the 1.8.0 scheduler.

Cheers
Bolke

> On 22 Sep 2017, at 19:54, David Klosowski <da...@thinknear.com> wrote:
> 
> Hey Airflow Devs:
> 
> We noticed an issue with task instances not running sometimes.
> 
> Some interesting details are:
>> in the database it shows that specific tasks that are "stuck" have an
> end_date but no state (null state)
>> Airflow has been running for several days just fine and there were no
> changes to Airflow in that period.
>> it happened for two separate important tasks for us at the same time
> 
> We're running:
>> Airflow 1.8.0 with a Postgres backend
>> Celery with Redis backed by Postgres result_backend
>> 3 worker hosts and 1 scheduler host
> 
> Let me know if you guys have any suggestions or advice.  Going to look
> through the code to see if I can see where/how this might be happening, but
> any suggestions or well known issues that might point to the problem might
> be helpful.
> 
> Thanks.
> 
> Regards,
> David