You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by ra...@gmail.com,
ra...@gmail.com on 2019/05/29 06:00:58 UTC
Task Getting Stuck in Queued State
Hi All,
We are observing intermittently that Tasks get stuck in queued state and never get executed by Airflow.
On debugging it we found that one of the queued dependency was not met due to which task did not move from queued to running state. So task remained in queued state.
(are_dependencies_met function returned false for QUEUE_DEPS inside _check_and_change_state_before_execution)
By looking into scheduler code it seems that scheduler does not reschedule the queued state tasks due to which task never got added to executor queue again and remained stuck in queued state.
There is a logic inside _check_and_change_state_before_execution function to move the task from queued to None state(which gets picked by scheduler for rescheduling) if RUN_DEPS are not met but this logic seems to be missing for QUEUE_DEPS.
It seems that task should be moved to None state even if QUEUE_DEPS are not met.
Any help/pointer on this would be useful.
Thanks,
Raman Gupta
Re: Task Getting Stuck in Queued State
Posted by Bolke de Bruin <bd...@gmail.com>.
Thanks for the detailed debugging! It gives us a point where to look at.
B.
Verstuurd vanaf mijn iPad
> Op 29 mei 2019 om 08:49 heeft Emmanuel Brard <em...@getyourguide.com> het volgende geschreven:
>
> Hi,
>
> We have noticed the same thing. On way for us to unblock it was to restart
> the scheduler (there is a function that check missing tasks in executor at
> the scheduler start-up phase).
> We implemented a health-check that checks for these kind of stuck queued
> tasks, but obviously fixing it in airflow core would be better.
>
> Cheers,
> E
>
> On Wed, May 29, 2019 at 8:01 AM ramandumcs@gmail.com <ra...@gmail.com>
> wrote:
>
>> Hi All,
>> We are observing intermittently that Tasks get stuck in queued state and
>> never get executed by Airflow.
>> On debugging it we found that one of the queued dependency was not met due
>> to which task did not move from queued to running state. So task remained
>> in queued state.
>> (are_dependencies_met function returned false for QUEUE_DEPS inside
>> _check_and_change_state_before_execution)
>>
>> By looking into scheduler code it seems that scheduler does not reschedule
>> the queued state tasks due to which task never got added to executor queue
>> again and remained stuck in queued state.
>>
>> There is a logic inside _check_and_change_state_before_execution function
>> to move the task from queued to None state(which gets picked by scheduler
>> for rescheduling) if RUN_DEPS are not met but this logic seems to be
>> missing for QUEUE_DEPS.
>>
>> It seems that task should be moved to None state even if QUEUE_DEPS are
>> not met.
>> Any help/pointer on this would be useful.
>>
>> Thanks,
>> Raman Gupta
>>
>>
>>
>
> --
>
>
>
>
>
>
>
>
> GetYourGuide AG
>
> Stampfenbachstrasse 48
>
> 8006 Zürich
>
> Switzerland
>
>
>
> <https://www.facebook.com/GetYourGuide>
> <https://twitter.com/GetYourGuide>
> <https://www.instagram.com/getyourguide/>
> <https://www.linkedin.com/company/getyourguide-ag>
> <http://www.getyourguide.com>
>
>
>
>
>
>
>
Re: Task Getting Stuck in Queued State
Posted by Emmanuel Brard <em...@getyourguide.com>.
Hi,
We have noticed the same thing. On way for us to unblock it was to restart
the scheduler (there is a function that check missing tasks in executor at
the scheduler start-up phase).
We implemented a health-check that checks for these kind of stuck queued
tasks, but obviously fixing it in airflow core would be better.
Cheers,
E
On Wed, May 29, 2019 at 8:01 AM ramandumcs@gmail.com <ra...@gmail.com>
wrote:
> Hi All,
> We are observing intermittently that Tasks get stuck in queued state and
> never get executed by Airflow.
> On debugging it we found that one of the queued dependency was not met due
> to which task did not move from queued to running state. So task remained
> in queued state.
> (are_dependencies_met function returned false for QUEUE_DEPS inside
> _check_and_change_state_before_execution)
>
> By looking into scheduler code it seems that scheduler does not reschedule
> the queued state tasks due to which task never got added to executor queue
> again and remained stuck in queued state.
>
> There is a logic inside _check_and_change_state_before_execution function
> to move the task from queued to None state(which gets picked by scheduler
> for rescheduling) if RUN_DEPS are not met but this logic seems to be
> missing for QUEUE_DEPS.
>
> It seems that task should be moved to None state even if QUEUE_DEPS are
> not met.
> Any help/pointer on this would be useful.
>
> Thanks,
> Raman Gupta
>
>
>
--
GetYourGuide AG
Stampfenbachstrasse 48
8006 Zürich
Switzerland
<https://www.facebook.com/GetYourGuide>
<https://twitter.com/GetYourGuide>
<https://www.instagram.com/getyourguide/>
<https://www.linkedin.com/company/getyourguide-ag>
<http://www.getyourguide.com>