You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by "SaloniiSharmaa (via GitHub)" <gi...@apache.org> on 2025/09/07 07:24:05 UTC
[PR] docs(redis): add Redis maintenance guide [airflow]
SaloniiSharmaa opened a new pull request, #55344:
URL: https://github.com/apache/airflow/pull/55344
### Summary
This PR adds a Redis maintenance guide for Airflow operators.
- The first commit accidentally duplicated the guide.
- The second commit fixes the duplication and ensures the guide is correct and clear.
### Motivation
Operators using Redis with Airflow have experienced memory growth due to keys without TTLs. This doc helps prevent production issues by providing cleanup strategies and monitoring instructions.
Closes #<55215>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs(redis): add Redis maintenance guide [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #55344:
URL: https://github.com/apache/airflow/pull/55344#discussion_r2329456988
##########
docs/maintenance/redis.md:
##########
@@ -0,0 +1,234 @@
+
+
+\# Redis Maintenance
+
+
+
+\## Introduction
+
+
+
+When running Airflow with the \*\*CeleryExecutor\*\* or \*\*CeleryKubernetesExecutor\*\*, Redis is often used as the message broker.
Review Comment:
> do you think it’s worth adding a one-liner in the doc saying it was deprecated, or should we leave it out completely to avoid confusion?
No, if we do that our docs will be impossible to maintain. When you explain about Redis that is "internal" detail of Celery. You should not be bothered if user is using `CeleryExecutor` or `CeleryKubernetesExecutor`. That is a higher level decision that is not part of what you are trying to solve. So the notion of your doc is "Assuming you are using Celery lets discuss how you maintains Redis"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs(redis): add Redis maintenance guide [airflow]
Posted by "SaloniiSharmaa (via GitHub)" <gi...@apache.org>.
SaloniiSharmaa commented on code in PR #55344:
URL: https://github.com/apache/airflow/pull/55344#discussion_r2329536566
##########
docs/maintenance/redis.md:
##########
@@ -0,0 +1,234 @@
+
+
+\# Redis Maintenance
+
+
+
+\## Introduction
+
+
+
+When running Airflow with the \*\*CeleryExecutor\*\* or \*\*CeleryKubernetesExecutor\*\*, Redis is often used as the message broker.
Review Comment:
Got it, thanks for clarifying!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs(redis): add Redis maintenance guide [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on code in PR #55344:
URL: https://github.com/apache/airflow/pull/55344#discussion_r2328822757
##########
docs/maintenance/redis.md:
##########
@@ -0,0 +1,234 @@
+
+
+\# Redis Maintenance
+
+
+
+\## Introduction
+
+
+
+When running Airflow with the \*\*CeleryExecutor\*\* or \*\*CeleryKubernetesExecutor\*\*, Redis is often used as the message broker.
+
+Redis stores task-related data, queue information, and other temporary state.
+
+
+
+Over time, Redis can grow very large if keys are not cleaned up.
+
+This can lead to performance problems or even production outages if the instance runs out of memory.
+
+
+
+This guide describes how to monitor, clean, and maintain your Redis instance in an Airflow setup.
+
+
+
+---
+
+
+
+\## Why Maintenance Is Needed
+
+
+
+Redis does not automatically remove all keys created by Celery or Kombu (the library Airflow uses for queuing).
+
+If keys are left without a \*\*Time-To-Live (TTL)\*\*, they can accumulate indefinitely.
+
+
+
+\### Common symptoms
+
+
+
+\* Redis memory usage grows continuously.
+
+\* Instance reaches the maximum memory limit and stops accepting writes.
+
+\* Airflow tasks start failing due to Redis errors.
+
+
+
+---
+
+
+
+\## Recommended Cleanup Strategy
+
+
+
+\### 1. Use TTLs on Redis keys
+
+
+
+If you discover keys without TTLs, you can enforce expiry manually.
+
+The example below uses a Lua script to set TTLs for all keys except critical ones.
+
+
+
+```bash
+
+\#!/bin/bash
+
+
+
+\# Usage: ./cleanup.sh -h <host> -p <port>
+
+
+
+HOST=$1
+
+PORT=$2
+
+
+
+\# Inline Lua script to set TTL on keys
+
+lua\_script='
+
+local keys = redis.call("KEYS", "\*")
+
+for \_, key in ipairs(keys) do
+
+ if string.match(key, "^unacked") or string.match(key, "^\_kombu%.binding") then
+
+ -- Skip critical keys
+
+ else
+
+ if redis.call("TTL", key) == -1 then
+
+ redis.call("EXPIRE", key, 54000) -- 15 days
+
+ end
+
+ end
+
+end
+
+'
+
+
+
+redis6-cli -h "$HOST" -p "$PORT" --eval <(echo "$lua\_script")
Review Comment:
```suggestion
redis-cli -h "$HOST" -p "$PORT" --eval <(echo "$lua\_script")
```
##########
docs/maintenance/redis.md:
##########
@@ -0,0 +1,234 @@
+
+
+\# Redis Maintenance
+
+
+
+\## Introduction
+
+
+
+When running Airflow with the \*\*CeleryExecutor\*\* or \*\*CeleryKubernetesExecutor\*\*, Redis is often used as the message broker.
+
+Redis stores task-related data, queue information, and other temporary state.
+
+
+
+Over time, Redis can grow very large if keys are not cleaned up.
Review Comment:
@ashb / @kaxil - isn't there a way today to configure TTL for Celery's broker's messages? I wonder how it has never been reported as an issue before #55344.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs(redis): add Redis maintenance guide [airflow]
Posted by "gudata (via GitHub)" <gi...@apache.org>.
gudata commented on PR #55344:
URL: https://github.com/apache/airflow/pull/55344#issuecomment-3264297719
Can you clarify what information will be gone from the airflow if a key expires? Will all task history / xcom will stay in place?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs(redis): add Redis maintenance guide [airflow]
Posted by "harrold-mussa (via GitHub)" <gi...@apache.org>.
harrold-mussa commented on PR #55344:
URL: https://github.com/apache/airflow/pull/55344#issuecomment-3538928119
Hi @SaloniiSharmaa I would like to know if you are working on this and may require any help? I'm here to contribute and see if you need any assistance !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs(redis): add Redis maintenance guide [airflow]
Posted by "SaloniiSharmaa (via GitHub)" <gi...@apache.org>.
SaloniiSharmaa commented on PR #55344:
URL: https://github.com/apache/airflow/pull/55344#issuecomment-3264710689
Thanks for raising these points! From what I understood:
- If a key expires, it’s mostly temporary runtime stuff (queues/locks) that goes — task history + XCom stay safe in the metadata DB.
- So yeah, task history/XComs aren’t affected.
- TTL should be set long enough to cover your longest DAG runs/retries, so nothing expires mid-flight.
I will update the doc with these clarifications if this makes sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs(redis): add Redis maintenance guide [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #55344:
URL: https://github.com/apache/airflow/pull/55344#issuecomment-3263554843
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:
- Pay attention to the quality of your code (ruff, mypy and type annotations). Our [prek-hooks]( https://github.com/apache/airflow/blob/main/contributing-docs/08_static_code_checks.rst#prerequisites-for-prek-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/airflow-core/docs/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/dev/breeze/doc/README.rst) for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#coding-style-and-best-practices).
- Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs(redis): add Redis maintenance guide [airflow]
Posted by "SaloniiSharmaa (via GitHub)" <gi...@apache.org>.
SaloniiSharmaa commented on code in PR #55344:
URL: https://github.com/apache/airflow/pull/55344#discussion_r2329234850
##########
docs/maintenance/redis.md:
##########
@@ -0,0 +1,234 @@
+
+
+\# Redis Maintenance
+
+
+
+\## Introduction
+
+
+
+When running Airflow with the \*\*CeleryExecutor\*\* or \*\*CeleryKubernetesExecutor\*\*, Redis is often used as the message broker.
Review Comment:
@eladkal
Would you prefer I move the whole guide as-is, or trim it down for the celery_executor.rst and only add a reference to a new file? also — I’ll remove mentions of CeleryKubernetesExecutor. Out of curiosity, since some users might still be on older versions, do you think it’s worth adding a one-liner in the doc saying it was deprecated, or should we leave it out completely to avoid confusion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs(redis): add Redis maintenance guide [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #55344:
URL: https://github.com/apache/airflow/pull/55344#discussion_r2329456988
##########
docs/maintenance/redis.md:
##########
@@ -0,0 +1,234 @@
+
+
+\# Redis Maintenance
+
+
+
+\## Introduction
+
+
+
+When running Airflow with the \*\*CeleryExecutor\*\* or \*\*CeleryKubernetesExecutor\*\*, Redis is often used as the message broker.
Review Comment:
> do you think it’s worth adding a one-liner in the doc saying it was deprecated, or should we leave it out completely to avoid confusion?
No, if we do that our docs will be impossible to maintain. When you explain about Redis that is "internal" detail of Celery. You should not be bothered if user is using `CeleryExecutor` or `CeleryKubernetesExecutor`. That is a higher level decision that is not part of what you are trying to solve. So the notion of your doc is "Assuming you are using Celery and assuming you choose Redis lets discuss how you maintains Redis"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs(redis): add Redis maintenance guide [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #55344:
URL: https://github.com/apache/airflow/pull/55344#discussion_r2329209377
##########
docs/maintenance/redis.md:
##########
Review Comment:
I don't think this is the right place for the file.
This relevant only for CeleryExecutor thus it should be in Celery provider at the executor docs part
##########
docs/maintenance/redis.md:
##########
@@ -0,0 +1,234 @@
+
+
+\# Redis Maintenance
+
+
+
+\## Introduction
+
+
+
+When running Airflow with the \*\*CeleryExecutor\*\* or \*\*CeleryKubernetesExecutor\*\*, Redis is often used as the message broker.
Review Comment:
CeleryKubernetesExecutor is deprecated lets not mention it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs(redis): add Redis maintenance guide [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #55344:
URL: https://github.com/apache/airflow/pull/55344#discussion_r2329211030
##########
docs/maintenance/redis.md:
##########
Review Comment:
check https://airflow.apache.org/docs/apache-airflow-providers-celery/stable/celery_executor.html
it should be part of this doc or if content is too long then a separate doc in the provider but with reference from this file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs(redis): add Redis maintenance guide [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #55344: docs(redis): add Redis maintenance guide
URL: https://github.com/apache/airflow/pull/55344
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs(redis): add Redis maintenance guide [airflow]
Posted by "SaloniiSharmaa (via GitHub)" <gi...@apache.org>.
SaloniiSharmaa commented on PR #55344:
URL: https://github.com/apache/airflow/pull/55344#issuecomment-3264950269
[redis_maintenance.rst.txt](https://github.com/user-attachments/files/22203181/redis_maintenance.rst.txt)
hi , after all the feedbacks , i decided to make a new doc starting from scratch and including all the points that are raised on this pr. before actually updating , i felt it would be better to first share the doc . so here it is .
Do comment , if you feel any improvement scope.
Feedback is always welcomed !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs(redis): add Redis maintenance guide [airflow]
Posted by "gudata (via GitHub)" <gi...@apache.org>.
gudata commented on PR #55344:
URL: https://github.com/apache/airflow/pull/55344#issuecomment-3264298125
How to calculate what value for TTL to set?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs(redis): add Redis maintenance guide [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #55344:
URL: https://github.com/apache/airflow/pull/55344#issuecomment-3397845867
Can you please rebase and fix the issues?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org