You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jason810496 (via GitHub)" <gi...@apache.org> on 2026/08/19 05:54:44 UTC
[PR] Honor the API server Dag cache TTL when no size limit is set [airflow]
jason810496 opened a new pull request, #71814:
URL: https://github.com/apache/airflow/pull/71814
- depends on: #71704
- split from: #71813
`[api] dag_cache_size = 0` reads as "no size limit", but it selected a mapping that never evicted and silently ignored `[api] dag_cache_ttl`. Age-based eviction could only be enabled by also accepting a size cap.
A TTL now applies with or without a size limit; only setting both options to `0` disables eviction. Deployments on `dag_cache_size = 0` start evicting on the default hour-long TTL — set `dag_cache_ttl = 0` to keep the old behaviour.
Also corrects the documented `version_added` for both options: 3.2.2, not 3.3.0.
---
##### Was generative AI tooling used to co-author this PR?
- [x] Yes — Claude Code (Opus 5)
Generated-by: Claude Code (Opus 5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Honor the API server Dag cache TTL when no size limit is set [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 merged PR #71814:
URL: https://github.com/apache/airflow/pull/71814
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Honor the API server Dag cache TTL when no size limit is set [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #71814:
URL: https://github.com/apache/airflow/pull/71814#discussion_r3811943190
##########
airflow-core/docs/administration-and-deployment/web-stack.rst:
##########
@@ -188,9 +188,10 @@ For example, to trigger a rolling restart of the API server pods:
kubectl rollout restart deployment airflow-api-server
-The API server also supports bounded DAG caching via ``dag_cache_size`` and
-``dag_cache_ttl``, which limits memory consumed by cached SerializedDAG objects.
-This reduces memory growth from DAG version accumulation regardless of server type.
+The API server also evicts cached SerializedDAG objects via ``dag_cache_size`` and
+``dag_cache_ttl``, which reduces memory growth from Dag version accumulation regardless of
+server type. Note that only ``dag_cache_size`` caps memory outright: each re-check resets a
+cached entry's expiry, so ``dag_cache_ttl`` reclaims only the versions that stop being requested.
Review Comment:
I rephrased the docs in a21020f68b. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Honor the API server Dag cache TTL when no size limit is set [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #71814:
URL: https://github.com/apache/airflow/pull/71814#discussion_r3811593746
##########
airflow-core/src/airflow/models/dagbag.py:
##########
@@ -79,26 +80,30 @@ def __init__(
Initialize DBDagBag.
:param load_op_links: Should the extra operator link be loaded when de-serializing the DAG?
- :param cache_size: Size of LRU cache. If None or 0, uses unbounded dict (no eviction).
- :param cache_ttl: Time-to-live for cache entries in seconds. If None or 0, no TTL (LRU only).
+ :param cache_size: Max cached entries. 0 or None means no size limit.
+ :param cache_ttl: Seconds until a cached entry expires, applied with or without a size limit.
+ 0 or None disables TTL. With neither a size limit nor a TTL the cache never evicts.
"""
self.load_op_links = load_op_links
self._dags: MutableMapping[UUID | str, _CacheEntry] = {}
self._use_cache = False
self._revalidation_interval = conf.getint("core", "min_serialized_dag_update_interval")
- # Initialize bounded cache if cache_size is provided and > 0
- if cache_size and cache_size > 0:
- if cache_ttl and cache_ttl > 0:
- self._dags = TTLCache(maxsize=cache_size, ttl=cache_ttl)
- else:
- self._dags = LRUCache(maxsize=cache_size)
+ # A TTL applies with or without a size limit: an uncapped TTLCache is what lets
+ # ``dag_cache_size = 0`` mean "no size limit" rather than "no eviction at all".
+ size = max(cache_size or 0, 0)
+ ttl = max(cache_ttl or 0, 0)
Review Comment:
The caller site will raise the warning against the negative value.
https://github.com/apache/airflow/blob/d74fbff4e32108df9276c9a907536d7da710f8dc/airflow-core/src/airflow/api_fastapi/common/dagbag.py#L36-L55
Since this PR change the behavior anyway, so no harm to raise ValueError on negative value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Honor the API server Dag cache TTL when no size limit is set [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #71814:
URL: https://github.com/apache/airflow/pull/71814#discussion_r3811524901
##########
airflow-core/src/airflow/models/dagbag.py:
##########
@@ -79,26 +80,30 @@ def __init__(
Initialize DBDagBag.
:param load_op_links: Should the extra operator link be loaded when de-serializing the DAG?
- :param cache_size: Size of LRU cache. If None or 0, uses unbounded dict (no eviction).
- :param cache_ttl: Time-to-live for cache entries in seconds. If None or 0, no TTL (LRU only).
+ :param cache_size: Max cached entries. 0 or None means no size limit.
+ :param cache_ttl: Seconds until a cached entry expires, applied with or without a size limit.
+ 0 or None disables TTL. With neither a size limit nor a TTL the cache never evicts.
"""
self.load_op_links = load_op_links
self._dags: MutableMapping[UUID | str, _CacheEntry] = {}
self._use_cache = False
self._revalidation_interval = conf.getint("core", "min_serialized_dag_update_interval")
- # Initialize bounded cache if cache_size is provided and > 0
- if cache_size and cache_size > 0:
- if cache_ttl and cache_ttl > 0:
- self._dags = TTLCache(maxsize=cache_size, ttl=cache_ttl)
- else:
- self._dags = LRUCache(maxsize=cache_size)
+ # A TTL applies with or without a size limit: an uncapped TTLCache is what lets
+ # ``dag_cache_size = 0`` mean "no size limit" rather than "no eviction at all".
+ size = max(cache_size or 0, 0)
+ ttl = max(cache_ttl or 0, 0)
Review Comment:
Instead of silently clamping, maybe we should just raise on a negative value? I assume that’s what the previous code does.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Honor the API server Dag cache TTL when no size limit is set [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #71814:
URL: https://github.com/apache/airflow/pull/71814#discussion_r3811509508
##########
airflow-core/docs/administration-and-deployment/web-stack.rst:
##########
@@ -188,9 +188,10 @@ For example, to trigger a rolling restart of the API server pods:
kubectl rollout restart deployment airflow-api-server
-The API server also supports bounded DAG caching via ``dag_cache_size`` and
-``dag_cache_ttl``, which limits memory consumed by cached SerializedDAG objects.
-This reduces memory growth from DAG version accumulation regardless of server type.
+The API server also evicts cached SerializedDAG objects via ``dag_cache_size`` and
+``dag_cache_ttl``, which reduces memory growth from Dag version accumulation regardless of
+server type. Note that only ``dag_cache_size`` caps memory outright: each re-check resets a
+cached entry's expiry, so ``dag_cache_ttl`` reclaims only the versions that stop being requested.
Review Comment:
This is _slightly_ overstating. The TTL is refreshed only at revalidation boundaries, not on every request. If an operator sets `dag_cache_ttl < min_serialized_dag_update_interval`, an actively-requested entry can still expire and reload between revalidations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Honor the API server Dag cache TTL when no size limit is set [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on PR #71814:
URL: https://github.com/apache/airflow/pull/71814#issuecomment-5339818314
We should probably add a news fragment to the behavioral change when `dag_cache_size = 0` since it’s visible to the user. The change is justified as a bug fix, but the fix may in turn cause issues in existing deployments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Honor the API server Dag cache TTL when no size limit is set [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #71814:
URL: https://github.com/apache/airflow/pull/71814#discussion_r3811943190
##########
airflow-core/docs/administration-and-deployment/web-stack.rst:
##########
@@ -188,9 +188,10 @@ For example, to trigger a rolling restart of the API server pods:
kubectl rollout restart deployment airflow-api-server
-The API server also supports bounded DAG caching via ``dag_cache_size`` and
-``dag_cache_ttl``, which limits memory consumed by cached SerializedDAG objects.
-This reduces memory growth from DAG version accumulation regardless of server type.
+The API server also evicts cached SerializedDAG objects via ``dag_cache_size`` and
+``dag_cache_ttl``, which reduces memory growth from Dag version accumulation regardless of
+server type. Note that only ``dag_cache_size`` caps memory outright: each re-check resets a
+cached entry's expiry, so ``dag_cache_ttl`` reclaims only the versions that stop being requested.
Review Comment:
I rephrased the docs in a21020f68b. Thanks.no human review before posting)
##########
airflow-core/src/airflow/models/dagbag.py:
##########
@@ -79,26 +80,30 @@ def __init__(
Initialize DBDagBag.
:param load_op_links: Should the extra operator link be loaded when de-serializing the DAG?
- :param cache_size: Size of LRU cache. If None or 0, uses unbounded dict (no eviction).
- :param cache_ttl: Time-to-live for cache entries in seconds. If None or 0, no TTL (LRU only).
+ :param cache_size: Max cached entries. 0 or None means no size limit.
+ :param cache_ttl: Seconds until a cached entry expires, applied with or without a size limit.
+ 0 or None disables TTL. With neither a size limit nor a TTL the cache never evicts.
"""
self.load_op_links = load_op_links
self._dags: MutableMapping[UUID | str, _CacheEntry] = {}
self._use_cache = False
self._revalidation_interval = conf.getint("core", "min_serialized_dag_update_interval")
- # Initialize bounded cache if cache_size is provided and > 0
- if cache_size and cache_size > 0:
- if cache_ttl and cache_ttl > 0:
- self._dags = TTLCache(maxsize=cache_size, ttl=cache_ttl)
- else:
- self._dags = LRUCache(maxsize=cache_size)
+ # A TTL applies with or without a size limit: an uncapped TTLCache is what lets
+ # ``dag_cache_size = 0`` mean "no size limit" rather than "no eviction at all".
+ size = max(cache_size or 0, 0)
+ ttl = max(cache_ttl or 0, 0)
Review Comment:
Raised `ValueError` when the value is negative. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org