You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by David Capwell <dc...@gmail.com> on 2017/07/29 00:05:16 UTC
Tasks stay queued when they fail in celery
We noticed that in the past few days we keep seeing tasks stay in the
queued state. Looking into celery, we see that the task had failed.
Traceback (most recent call last):
File "/python/lib/python2.7/site-packages/celery/app/trace.py", line
367, in trace_task
R = retval = fun(*args, **kwargs)
File "/python/lib/python2.7/site-packages/celery/app/trace.py", line
622, in __protected_call__
return self.run(*args, **kwargs)
File "/python/lib/python2.7/site-packages/airflow/executors/celery_executor.py",
line 59, in execute_command
raise AirflowException('Celery command failed')
AirflowException: Celery command failed
Why does airflow not learn about this and recover? And what can we do to
prevent this?
Thanks for your time reading this email.
Re: Tasks stay queued when they fail in celery
Posted by David Capwell <dc...@gmail.com>.
Thanks for the details!
On Aug 4, 2017 10:21 AM, "George Leslie-Waksman"
<ge...@cloverhealth.com.invalid> wrote:
> We've seen this before as well, it's a bug in the Celery Executor that has
> a bunch of different manifestations.
>
> There is at least one open issue relating to this bug:
> https://issues.apache.org/jira/browse/AIRFLOW-1463
>
> I have been working on a fix but it's likely to be a few more days before I
> have a chance to make some progress.
>
> --George
>
> On Fri, Jul 28, 2017 at 5:05 PM David Capwell <dc...@gmail.com> wrote:
>
> > We noticed that in the past few days we keep seeing tasks stay in the
> > queued state. Looking into celery, we see that the task had failed.
> >
> > Traceback (most recent call last):
> > File "/python/lib/python2.7/site-packages/celery/app/trace.py", line
> > 367, in trace_task
> > R = retval = fun(*args, **kwargs)
> > File "/python/lib/python2.7/site-packages/celery/app/trace.py", line
> > 622, in __protected_call__
> > return self.run(*args, **kwargs)
> > File
> > "/python/lib/python2.7/site-packages/airflow/executors/
> celery_executor.py",
> > line 59, in execute_command
> > raise AirflowException('Celery command failed')
> > AirflowException: Celery command failed
> >
> >
> > Why does airflow not learn about this and recover? And what can we do to
> > prevent this?
> >
> > Thanks for your time reading this email.
> >
>
Re: Tasks stay queued when they fail in celery
Posted by George Leslie-Waksman <ge...@cloverhealth.com.INVALID>.
We've seen this before as well, it's a bug in the Celery Executor that has
a bunch of different manifestations.
There is at least one open issue relating to this bug:
https://issues.apache.org/jira/browse/AIRFLOW-1463
I have been working on a fix but it's likely to be a few more days before I
have a chance to make some progress.
--George
On Fri, Jul 28, 2017 at 5:05 PM David Capwell <dc...@gmail.com> wrote:
> We noticed that in the past few days we keep seeing tasks stay in the
> queued state. Looking into celery, we see that the task had failed.
>
> Traceback (most recent call last):
> File "/python/lib/python2.7/site-packages/celery/app/trace.py", line
> 367, in trace_task
> R = retval = fun(*args, **kwargs)
> File "/python/lib/python2.7/site-packages/celery/app/trace.py", line
> 622, in __protected_call__
> return self.run(*args, **kwargs)
> File
> "/python/lib/python2.7/site-packages/airflow/executors/celery_executor.py",
> line 59, in execute_command
> raise AirflowException('Celery command failed')
> AirflowException: Celery command failed
>
>
> Why does airflow not learn about this and recover? And what can we do to
> prevent this?
>
> Thanks for your time reading this email.
>