You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jason810496 (via GitHub)" <gi...@apache.org> on 2026/08/19 05:54:49 UTC
[PR] Make the scheduler Dag cache size and TTL configurable [airflow]
jason810496 opened a new pull request, #71816:
URL: https://github.com/apache/airflow/pull/71816
- depends on: #71704
- split from: #71813
The scheduler's cache bound landed in #71704 as a fixed constant, so a deployment whose working set of Dag versions with runs in flight exceeds it cannot raise the ceiling, and a memory-constrained one cannot lower it.
Adds `[scheduler] dag_cache_size` (default 512) and `[scheduler] dag_cache_ttl` (default 0), mirroring the pair the API server already exposes and defaulting to the bound the scheduler already had. Both components resolve their pair through one helper so `0` means the same thing on each side.
---
##### Was generative AI tooling used to co-author this PR?
- [x] Yes — Claude Code (Opus 5)
Generated-by: Claude Code (Opus 5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the scheduler Dag cache size and TTL configurable [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 merged PR #71816:
URL: https://github.com/apache/airflow/pull/71816
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the scheduler Dag cache size and TTL configurable [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #71816:
URL: https://github.com/apache/airflow/pull/71816#discussion_r3954737498
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -301,6 +290,30 @@ def _is_parent_process() -> bool:
return multiprocessing.current_process().name == "MainProcess"
+def _create_scheduler_dag_bag() -> CachedDBDagBag:
+ """
+ Build the scheduler's DagBag from the ``[scheduler]`` cache options.
+
+ Defaults to an LRU cache of 512 versions with no TTL. A size limit is the only hard ceiling:
+ each re-check resets an entry's expiry, so a TTL reclaims a version only once its runs finish
+ and it stops being requested, bounding memory by the concurrently active set rather than
+ outright. With both limits disabled, retain loaded Dags without eviction for the process
+ lifetime.
+ """
+ cache_size = conf.getint("scheduler", "dag_cache_size", fallback=512)
+ cache_ttl = conf.getint("scheduler", "dag_cache_ttl", fallback=0)
Review Comment:
Addressed in c787d97eaa and updated the outdated PR description
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the scheduler Dag cache size and TTL configurable [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #71816:
URL: https://github.com/apache/airflow/pull/71816#issuecomment-6265397159
### Backport successfully created: v3-4-test
Note: As of [Merging PRs targeted for Airflow 3.X](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#merging-prs-targeted-for-airflow-3x)
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
In matter of doubt please ask in [#release-management](https://apache-airflow.slack.com/archives/C03G9H97MM2) Slack channel.
<table>
<tr>
<th>Status</th>
<th>Branch</th>
<th>Result</th>
</tr>
<tr>
<td>✅</td>
<td>v3-4-test</td>
<td><a href="https://github.com/apache/airflow/pull/75087"><img src="https://img.shields.io/badge/PR-75087-blue" alt="PR Link"></a></td>
</tr>
</table>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the scheduler Dag cache size and TTL configurable [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on PR #71816:
URL: https://github.com/apache/airflow/pull/71816#issuecomment-5579333083
> The API server and scheduler read and validate their size and TTL pairs through one helper.
This does not seem to be correct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make the scheduler Dag cache size and TTL configurable [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #71816:
URL: https://github.com/apache/airflow/pull/71816#discussion_r3954406612
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -301,6 +290,30 @@ def _is_parent_process() -> bool:
return multiprocessing.current_process().name == "MainProcess"
+def _create_scheduler_dag_bag() -> CachedDBDagBag:
+ """
+ Build the scheduler's DagBag from the ``[scheduler]`` cache options.
+
+ Defaults to an LRU cache of 512 versions with no TTL. A size limit is the only hard ceiling:
+ each re-check resets an entry's expiry, so a TTL reclaims a version only once its runs finish
+ and it stops being requested, bounding memory by the concurrently active set rather than
+ outright. With both limits disabled, retain loaded Dags without eviction for the process
+ lifetime.
+ """
+ cache_size = conf.getint("scheduler", "dag_cache_size", fallback=512)
+ cache_ttl = conf.getint("scheduler", "dag_cache_ttl", fallback=0)
Review Comment:
These fallbacks shouldn’t be needed since `config.yml` already specifies defaults.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org