You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Goutham Pratapa <pr...@gmail.com> on 2018/05/15 09:26:53 UTC
Airflow with Celery
Hi all,
I have been using airflow with Celery executor in the background
https://hastebin.com/sipecovomi.ini --> airflow.cfg
https://hastebin.com/urutokuvoq.py --> The dag I have been using
This shows that the dag is always in running state.
Airflow flower shows nothing in the tasks or in the broker.
Did I miss anything can anyone help me in this regard.
--
Cheers !!!
Goutham Pratapa
Re: Airflow with Celery
Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
I had similar issues with Airflow running the Celery executor.
The celery_result_backend should be a persistent database like Postgres or
MySql. What broker are you using? I would recommend using Redis or
RabbitMQ, depending on what you like the most.
Cheers, Fokko
2018-05-15 21:12 GMT+02:00 David Capwell <dc...@gmail.com>:
> What I find is that when celery rejects we hit this. For us we don't do
> work on the hosts so solve by over provisioning tasks in celery
>
> On Tue, May 15, 2018, 6:30 AM Andy Cooper <an...@astronomer.io>
> wrote:
>
>> I have had very similar issues when there was a problem with the
>> connection
>> string pointing to the message broker. Triple check those connection
>> strings and attempt to connect outside of airflow.
>>
>> On Tue, May 15, 2018 at 9:27 AM Goutham Pratapa <pratapagoutham@gmail.com
>> >
>> wrote:
>>
>> > Hi all,
>> >
>> > I have been using airflow with Celery executor in the background
>> >
>> > https://hastebin.com/sipecovomi.ini --> airflow.cfg
>> >
>> > https://hastebin.com/urutokuvoq.py --> The dag I have been using
>> >
>> >
>> >
>> > This shows that the dag is always in running state.
>> >
>> >
>> >
>> >
>> > Airflow flower shows nothing in the tasks or in the broker.
>> >
>> >
>> > Did I miss anything can anyone help me in this regard.
>> >
>> >
>> > --
>> > Cheers !!!
>> > Goutham Pratapa
>> >
>>
>
Re: Airflow with Celery
Posted by David Capwell <dc...@gmail.com>.
What I find is that when celery rejects we hit this. For us we don't do
work on the hosts so solve by over provisioning tasks in celery
On Tue, May 15, 2018, 6:30 AM Andy Cooper <an...@astronomer.io> wrote:
> I have had very similar issues when there was a problem with the connection
> string pointing to the message broker. Triple check those connection
> strings and attempt to connect outside of airflow.
>
> On Tue, May 15, 2018 at 9:27 AM Goutham Pratapa <pr...@gmail.com>
> wrote:
>
> > Hi all,
> >
> > I have been using airflow with Celery executor in the background
> >
> > https://hastebin.com/sipecovomi.ini --> airflow.cfg
> >
> > https://hastebin.com/urutokuvoq.py --> The dag I have been using
> >
> >
> >
> > This shows that the dag is always in running state.
> >
> >
> >
> >
> > Airflow flower shows nothing in the tasks or in the broker.
> >
> >
> > Did I miss anything can anyone help me in this regard.
> >
> >
> > --
> > Cheers !!!
> > Goutham Pratapa
> >
>
Re: Airflow with Celery
Posted by Andy Cooper <an...@astronomer.io>.
I have had very similar issues when there was a problem with the connection
string pointing to the message broker. Triple check those connection
strings and attempt to connect outside of airflow.
On Tue, May 15, 2018 at 9:27 AM Goutham Pratapa <pr...@gmail.com>
wrote:
> Hi all,
>
> I have been using airflow with Celery executor in the background
>
> https://hastebin.com/sipecovomi.ini --> airflow.cfg
>
> https://hastebin.com/urutokuvoq.py --> The dag I have been using
>
>
>
> This shows that the dag is always in running state.
>
>
>
>
> Airflow flower shows nothing in the tasks or in the broker.
>
>
> Did I miss anything can anyone help me in this regard.
>
>
> --
> Cheers !!!
> Goutham Pratapa
>