You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/04/22 13:32:20 UTC

[GitHub] [airflow] jstelzer opened a new issue #15486: Unable to connect to redis in aws that requires encrypted data in flight

jstelzer opened a new issue #15486:
URL: https://github.com/apache/airflow/issues/15486

**Apache Airflow version**: 2.0.1

**Environment**: AWS

- **Cloud provider or hardware configuration**: AWS using redis configured to encrypt data in flight
- **OS** (e.g. from /etc/os-release): docker image based on apache/airflow:2.0.1 running in ECS

**What happened**:

When deploying the flower service, clicking on the broker tab of the ui would just hang. Nothing got logged.

**What you expected to happen**: I expected to see broker status

**How to reproduce it**: Point your airfow cluster at a redis cluster that requires TLS

**Anything else we need to know**:

This bug report is as much a question as it is a bug.

Yesterday I spent a bunch of time debugging why flower in aws would hang every time the broker dashboard was accessed.

The answer: calls to the redis backend would hang indefinitely because redis was configured to encrypt data in flight.

Without the tls negotiation on handshake, any call (ping, get, put) would block forever.

Nothing interesting showed in the cloudwatch logs and at first i thought this was just a tcp connectivity issue.

Evenutally i got frustrated with trying to debug it remotely so i hacked up a docker compose that let me run things locally, but connect to my aws infra.

At that point I identified the problem and confirmed it with a quick program using the python redis module directly.

Next I read your docs for configuring airflow. The ssl options are not clearly documented, and seem to be there for people who are using self singed certs so that they can get their trust store/peers working together.

In aws, the certs are all signed by a 'real' CA that's already in the default trust store. So there's no reason to mess with keys/cacerts/etc.

I ended up reading thru the celery source looking for how it worked when I spotted this:
`
15:27 $ rg 'rediss://'
celery/backends/redis.py
71:is redis://. A Redis SSL connection URL should use the scheme rediss://.
75:A rediss:// URL must have parameter ssl_cert_reqs and this must be set to \

Read up more on the rediss:// prefix on the celery docs and it seemed to be what I was after:

diff --git a/airflow/config_templates/default_celery.py b/airflow/config_templates/default_celery.py
index 7231db54f..6f460e147 100644
--- a/airflow/config_templates/default_celery.py
+++ b/airflow/config_templates/default_celery.py
@@ -65,7 +65,7 @@ try:
'ca_certs': conf.get('celery', 'SSL_CACERT'),
'cert_reqs': ssl.CERT_REQUIRED,
}
- elif 'redis://' in broker_url:
+ elif 'redis://' in broker_url or 'rediss://' in broker_url:
broker_use_ssl = {
'ssl_keyfile': conf.get('celery', 'SSL_KEY'),
'ssl_certfile': conf.get('celery', 'SSL_CERT'),
`

So, I created the trivial patch to airflow above and it worked as desired.

My question is, is this really needed? If there is a better way for me to use the existing config options, please educated me.

At the end of the day, I have docker images in ecs that are operational, I just don't like maintaining one off patches that muddy up the upgrade path.

Celery seems to see the string literal 'rediss://' in the url and it just 'does the right thing'. So I think this is needed for folks who are using redis in aws who also have the encrypt in transit boxes checked on their redis configurations.

So, again to be clear, this isn't the same as configuring ssl keys, cacerts and all that. This is literally just asking celery to negotiate a tls connection to the broker via the usual mechanisms.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org