[GitHub] [airflow] dintorf commented on issue #28655: AIRFLOW__CELERY__BROKER_URL not supporting multiple sentinel url, instead throwing error

["dintorf (via GitHub)" <gi...@apache.org>]

dintorf commented on issue #28655:
URL: https://github.com/apache/airflow/issues/28655#issuecomment-1458723703

   I know this issue is closed, however I believe I have gotten Airflow working with Redis Sentinel. Here are the configuration details:
   
   ```bash
   # airflow config for celery using redis sentinel
   
   # note the broker URL password is the redis password, and the redis DB is 0
   AIRFLOW__CELERY__BROKER_URL=sentinel://:str0ng_passw0rd@redis-sentinel1:26379/0;sentinel://:str0ng_passw0rd@redis-sentinel2:26379/0;sentinel://:str0ng_passw0rd@redis-sentinel3:26379/0
   AIRFLOW__CELERY_BROKER_TRANSPORT_OPTIONS__MASTER_NAME=mymaster
   # note this is the password for sentinel
   AIRFLOW__CELERY_BROKER_TRANSPORT_OPTIONS__SENTINEL_KWARGS={"password":"str0ng_passw0rd"}
   AIRFLOW__CELERY__CELERY_CONFIG_OPTIONS=config.celery_config.CELERY_CONFIG
   ```
   
   The broker transport options are put into celery config using `conf.getsection("celery_broker_transport_options")` which does not load json strings as dicts. Celery expects the `sentinel_kwargs` to be a dict, so I've added this python file to update the broker transport options:
   
   ```python
   # /usr/local/airflow/config/celery_config.py
   import json
   
   from airflow.config_templates.default_celery import DEFAULT_CELERY_CONFIG
   
   CELERY_CONFIG = DEFAULT_CELERY_CONFIG
   if "sentinel_kwargs" in CELERY_CONFIG.get("broker_transport_options", {}):
       CELERY_CONFIG["broker_transport_options"]["sentinel_kwargs"] = json.loads(
           CELERY_CONFIG["broker_transport_options"]["sentinel_kwargs"]
       )
   ```
   
   I've only tested this locally. However, I believe this would work anywhere.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org