Airflow RabbitMQ with Celery across EC2 Instances

[Vikas Malhotra <vi...@gmail.com>]

Hello Everyone,

I was wondering if anyone has done setup of Airflow with Celery Executor
with RabbitMQ as backend across EC2 instances and if they are willing to
share their experiences and any pitfalls that I can avoid?

If anyone has some basic steps or guidelines, that you would be great as
well :)

Thanks & Regards,
Vikas

Re: Airflow RabbitMQ with Celery across EC2 Instances

[Yogesh Garg <yo...@gmail.com>]

Hello Vikas,

We have set up CeleryExecutor with RabbitMQ as messaging broker. In our
setup Rabbitmq, airflow scheduler, celery are running on same machine. So
far the setup is working fine. One suggestion is avoid using same rabbitmq
as "celery_result_backend" as it can choke rabbitmq with persistent results
generated by celery. So better to use the airflow datastore for storing
celery results as well.

Thanks.

On Sat, Oct 8, 2016 at 3:21 AM, Vikas Malhotra <vi...@gmail.com>
wrote:

> Hello Everyone,
>
> I was wondering if anyone has done setup of Airflow with Celery Executor
> with RabbitMQ as backend across EC2 instances and if they are willing to
> share their experiences and any pitfalls that I can avoid?
>
> If anyone has some basic steps or guidelines, that you would be great as
> well :)
>
> Thanks & Regards,
> Vikas
>