You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Desiree Cox <dc...@zendesk.com.INVALID> on 2017/03/07 18:22:03 UTC

Is anybody using CeleryExecutor + redis-sentinel message broker?

Hello! I am a very new engineer at my company and I have been asked to set
up Airflow with Celery. We have existing Redis resources for the message
broker, but I am not allowed to connect to redis directly, I must do it
through redis-sentinel. Celery 4 can use redis-sentinel but I don't see how
to pass the configuration through Airflow—you have to send it a dictionary
called 'broker_transport_options' with the name of the Redis service you
are looking for. Has anybody gotten the CeleryExecutor working with
redis-sentinel? My teammate says I should contribute a PR to Airflow (which
I am happy to try) but I want to check in case anybody has suggestions.
Thank you!

-- 
Desiree Cox
Associate Software Engineer
Zendesk

Re: Is anybody using CeleryExecutor + redis-sentinel message broker?

Posted by Desiree Cox <dc...@zendesk.com.INVALID>.
Yes it does help. Thanks Robin! I struggled a little on account of having
to pass this parameter as a dict but I figured it out and I can copy what
you did to create the JIRA ticket and PR. Thanks again!

On Wed, Mar 8, 2017 at 1:15 AM, Miller, Robin <
Robin.Miller@affiliate.oliverwyman.com> wrote:

> Hi Desiree,
>
>
> It sounds to me like you'll probably need to add an additional
> configuration option to pass to Celery. In the file
> airflow/executors/celery_executor.py you can see all of the config
> options that are passed into Airflow in the CeleryConfig class. I've
> currently got a PR open for a change of this nature at the moment:
> https://github.com/apache/incubator-airflow/pull/1912/files
> <https://github.com/apache/incubator-airflow/pull/1912/files>.
>
>
> Hope this helps,
>
> Robin Miller
> OLIVER WYMAN
> robin.miller@affiliate.oliverwyman.com<mailto:robin.
> miller@affiliate.oliverwyman.com>
> www.oliverwyman.com<http://www.oliverwyman.com/
> <http://www.oliverwyman.com/>>
>
> ________________________________
> From: Desiree Cox <dc...@zendesk.com.INVALID>
> Sent: 07 March 2017 18:22:03
> To: dev@airflow.incubator.apache.org
> Subject: Is anybody using CeleryExecutor + redis-sentinel message broker?
>
>
> Hello! I am a very new engineer at my company and I have been asked to set
> up Airflow with Celery. We have existing Redis resources for the message
> broker, but I am not allowed to connect to redis directly, I must do it
> through redis-sentinel. Celery 4 can use redis-sentinel but I don't see how
> to pass the configuration through Airflow—you have to send it a dictionary
> called 'broker_transport_options' with the name of the Redis service you
> are looking for. Has anybody gotten the CeleryExecutor working with
> redis-sentinel? My teammate says I should contribute a PR to Airflow (which
> I am happy to try) but I want to check in case anybody has suggestions.
> Thank you!
>
> --
> Desiree Cox
> Associate Software Engineer
> Zendesk
>
> ________________________________
> This e-mail and any attachments may be confidential or legally privileged.
> If you received this message in error or are not the intended recipient,
> you should destroy the e-mail message and any attachments or copies, and
> you are prohibited from retaining, distributing, disclosing or using any
> information contained herein. Please inform us of the erroneous delivery by
> return e-mail. Thank you for your cooperation.
>



-- 
Desiree Cox
Associate Software Engineer
Zendesk

Re: Is anybody using CeleryExecutor + redis-sentinel message broker?

Posted by "Miller, Robin" <Ro...@affiliate.oliverwyman.com>.
Hi Desiree,


It sounds to me like you'll probably need to add an additional configuration option to pass to Celery. In the file airflow/executors/celery_executor.py you can see all of the config options that are passed into Airflow in the CeleryConfig class. I've currently got a PR open for a change of this nature at the moment: https://github.com/apache/incubator-airflow/pull/1912/files.


Hope this helps,

Robin Miller
OLIVER WYMAN
robin.miller@affiliate.oliverwyman.com<ma...@affiliate.oliverwyman.com>
www.oliverwyman.com<http://www.oliverwyman.com/>

________________________________
From: Desiree Cox <dc...@zendesk.com.INVALID>
Sent: 07 March 2017 18:22:03
To: dev@airflow.incubator.apache.org
Subject: Is anybody using CeleryExecutor + redis-sentinel message broker?

Hello! I am a very new engineer at my company and I have been asked to set
up Airflow with Celery. We have existing Redis resources for the message
broker, but I am not allowed to connect to redis directly, I must do it
through redis-sentinel. Celery 4 can use redis-sentinel but I don't see how
to pass the configuration through Airflow—you have to send it a dictionary
called 'broker_transport_options' with the name of the Redis service you
are looking for. Has anybody gotten the CeleryExecutor working with
redis-sentinel? My teammate says I should contribute a PR to Airflow (which
I am happy to try) but I want to check in case anybody has suggestions.
Thank you!

--
Desiree Cox
Associate Software Engineer
Zendesk

________________________________
This e-mail and any attachments may be confidential or legally privileged. If you received this message in error or are not the intended recipient, you should destroy the e-mail message and any attachments or copies, and you are prohibited from retaining, distributing, disclosing or using any information contained herein. Please inform us of the erroneous delivery by return e-mail. Thank you for your cooperation.