You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "wjddn279 (via GitHub)" <gi...@apache.org> on 2026/03/23 10:59:25 UTC
[PR] [DOCS]add guide for dag version inflation and it's checker [airflow]
wjddn279 opened a new pull request, #64100:
URL: https://github.com/apache/airflow/pull/64100
related: https://github.com/apache/airflow/pull/59430
add docs in FAQ to explain about dag version inflation and it's checker
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
<!--
Thank you for contributing!
Please provide above a brief description of the changes made in this pull request.
Write a good git commit message following this guide: http://chris.beams.io/posts/git-commit/
Please make sure that your code changes are covered with tests.
And in case of new features or big changes remember to adjust the documentation.
Feel free to ping (in general) for the review if you do not see reaction for a few days
(72 Hours is the minimum reaction time you can expect from volunteers) - we sometimes miss notifications.
In case of an existing issue, reference it using one of the following:
* closes: #ISSUE
* related: #ISSUE
-->
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR, please
change below checkbox to `[X]` followed by the name of the tool, uncomment the "Generated-by".
-->
- [x] Yes (please specify the tool below)
claude 4.6 opus
<!--
Generated-by: [Tool Name] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
---
* Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DOCS]add guide for dag version inflation and it's checker [airflow]
Posted by "wjddn279 (via GitHub)" <gi...@apache.org>.
wjddn279 commented on code in PR #64100:
URL: https://github.com/apache/airflow/pull/64100#discussion_r2985451886
##########
airflow-core/docs/faq.rst:
##########
@@ -237,6 +237,166 @@ There are several reasons why Dags might disappear from the UI. Common causes in
* **Time synchronization issues** - Ensure all nodes (database, schedulers, workers) use NTP with <1s clock drift.
+.. _faq:dag-version-inflation:
+
+Why does my Dag version keep increasing?
+-----------------------------------------
+
+Every time the Dag processor parses a Dag file, it serializes the Dag and compares the result with the
+version stored in the metadata database. If anything has changed, Airflow creates a new Dag version. This
+mechanism ensures that Dag runs use consistent code throughout their execution, even if the Dag file is
+updated mid-run.
+
+**Dag version inflation** occurs when the version number increases indefinitely without the Dag author
+making any intentional changes.
+
+What goes wrong
+"""""""""""""""
+
+When Dag versions increase without meaningful changes:
+
+* The metadata database accumulates unnecessary Dag version records, increasing storage and query overhead.
+* The UI shows a misleading history of Dag changes, making it harder to identify real modifications.
+* The scheduler and API server may consume more memory as they load and cache a growing number of Dag versions.
+
+Common causes
+"""""""""""""
+
+Version inflation is caused by using values that change at **parse time** — that is, every time the Dag
+processor evaluates the Dag file — as arguments to Dag or Task constructors. The most common patterns are:
+
+**1. Using ``datetime.now()`` or ``pendulum.now()`` as ``start_date``:**
+
+.. code-block:: python
+
+ from datetime import datetime
+
+ from airflow.sdk import DAG
+
+ # BAD: datetime.now() produces a different value on every parse
+ with DAG(
+ dag_id="bad_example",
+ start_date=datetime.now(),
+ schedule="@daily",
+ ):
+ ...
+
+Every parse produces a different ``start_date``, so the serialized Dag is always different from the
+stored version.
+
+**2. Using random values in Dag or Task arguments:**
+
+.. code-block:: python
+
+ import random
+
+ from airflow.sdk import DAG
+ from airflow.providers.standard.operators.python import PythonOperator
+
+ with DAG(dag_id="bad_random", start_date="2024-01-01", schedule="@daily") as dag:
+ # BAD: random value changes every parse
+ PythonOperator(
+ task_id=f"task_{random.randint(1, 1000)}",
+ python_callable=lambda: None,
+ )
+
+**3. Assigning runtime-varying values to variables used in constructors:**
+
+.. code-block:: python
+
+ from datetime import datetime
+
+ from airflow.sdk import DAG
+ from airflow.providers.standard.operators.python import PythonOperator
+
+ # BAD: the variable captures a parse-time value, then is passed to the DAG
+ default_args = {"start_date": datetime.now()}
+
+ with DAG(dag_id="bad_defaults", default_args=default_args, schedule="@daily") as dag:
+ PythonOperator(task_id="my_task", python_callable=lambda: None)
+
+Even though ``datetime.now()`` is not called directly inside the Dag constructor, it flows in through
+``default_args`` and still causes a different serialized Dag on every parse.
+
+**4. Using environment variables or file contents that change between parses:**
+
+.. code-block:: python
+
+ import os
+
+ from airflow.sdk import DAG
+ from airflow.providers.standard.operators.bash import BashOperator
+
+ with DAG(dag_id="bad_env", start_date="2024-01-01", schedule="@daily") as dag:
+ # BAD if BUILD_NUMBER changes on every deployment or parse
+ BashOperator(
+ task_id="echo_build",
+ bash_command=f"echo {os.environ.get('BUILD_NUMBER', 'unknown')}",
+ )
+
+How to avoid version inflation
+""""""""""""""""""""""""""""""
+
+* **Use fixed ``start_date`` values.** Always set ``start_date`` to a static ``datetime`` literal:
+
+ .. code-block:: python
+
+ import datetime
+
+ from airflow.sdk import DAG
+
+ with DAG(
+ dag_id="good_example",
+ start_date=datetime.datetime(2024, 1, 1),
+ schedule="@daily",
+ ):
+ ...
+
+* **Keep all Dag and Task constructor arguments deterministic.** Arguments passed to Dag and Operator
+ constructors must produce the same value on every parse. Move any dynamic computation into the
+ ``execute()`` method or use Jinja templates, which are evaluated at task execution time rather than
+ parse time.
+
+* **Use Jinja templates for dynamic values:**
+
+ .. code-block:: python
+
+ from airflow.providers.standard.operators.bash import BashOperator
+
+ # GOOD: the template is resolved at execution time, not parse time
+ BashOperator(
+ task_id="echo_date",
+ bash_command="echo {{ ds }}",
+ )
+
+* **Use Airflow Variables with templates instead of top-level lookups:**
+
+ .. code-block:: python
+
+ from airflow.providers.standard.operators.bash import BashOperator
+
+ # GOOD: Variable is resolved at execution time via template
+ BashOperator(
+ task_id="echo_var",
+ bash_command="echo {{ var.value.my_variable }}",
+ )
+
+Dag version inflation detection
+""""""""""""""""""""""""""""""""
+
+Starting from Airflow 3.2, the Dag processor performs **AST-based static analysis** on every Dag file
+before parsing to detect runtime-varying values in Dag and Task constructors. When a potential issue is
+found, it is surfaced as a **Dag warning** visible in the UI.
+
+You can control this behavior with the
+:ref:`dag_version_inflation_check_level <config:dag_processor__dag_version_inflation_check_level>`
+configuration option:
+
+* ``off`` — Disables the check entirely. No errors or warnings are generated.
+* ``warning`` (default) — Dags load normally but warnings are displayed in the UI when issues are detected.
+* ``error`` — Treats detected issues as Dag import errors, preventing the Dag from loading.
Review Comment:
Good idea! Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DOCS]add guide for dag version inflation and it's checker [airflow]
Posted by "wjddn279 (via GitHub)" <gi...@apache.org>.
wjddn279 commented on PR #64100:
URL: https://github.com/apache/airflow/pull/64100#issuecomment-4122817670
@Lee-W fixed!
When you have a moment, could you also take a look at this PR? Thanks! https://github.com/apache/airflow/pull/61345
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DOCS]add guide for dag version inflation and it's checker [airflow]
Posted by "wjddn279 (via GitHub)" <gi...@apache.org>.
wjddn279 commented on PR #64100:
URL: https://github.com/apache/airflow/pull/64100#issuecomment-4115695721
@potiuk @Lee-W @Dev-iL
can you check this docs update? thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DOCS]add guide for dag version inflation and it's checker [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #64100:
URL: https://github.com/apache/airflow/pull/64100#issuecomment-4153262070
### Backport successfully created: v3-2-test
Note: As of [Merging PRs targeted for Airflow 3.X](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#merging-prs-targeted-for-airflow-3x)
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
In matter of doubt please ask in [#release-management](https://apache-airflow.slack.com/archives/C03G9H97MM2) Slack channel.
<table>
<tr>
<th>Status</th>
<th>Branch</th>
<th>Result</th>
</tr>
<tr>
<td>✅</td>
<td>v3-2-test</td>
<td><a href="https://github.com/apache/airflow/pull/64461"><img src="https://img.shields.io/badge/PR-64461-blue" alt="PR Link"></a></td>
</tr>
</table>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DOCS]add guide for dag version inflation and it's checker [airflow]
Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 merged PR #64100:
URL: https://github.com/apache/airflow/pull/64100
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DOCS]add guide for dag version inflation and it's checker [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #64100:
URL: https://github.com/apache/airflow/pull/64100#discussion_r2979431833
##########
airflow-core/docs/faq.rst:
##########
@@ -237,6 +237,166 @@ There are several reasons why Dags might disappear from the UI. Common causes in
* **Time synchronization issues** - Ensure all nodes (database, schedulers, workers) use NTP with <1s clock drift.
+.. _faq:dag-version-inflation:
+
+Why does my Dag version keep increasing?
+-----------------------------------------
+
+Every time the Dag processor parses a Dag file, it serializes the Dag and compares the result with the
+version stored in the metadata database. If anything has changed, Airflow creates a new Dag version. This
+mechanism ensures that Dag runs use consistent code throughout their execution, even if the Dag file is
+updated mid-run.
+
+**Dag version inflation** occurs when the version number increases indefinitely without the Dag author
+making any intentional changes.
+
+What goes wrong
+"""""""""""""""
+
+When Dag versions increase without meaningful changes:
+
+* The metadata database accumulates unnecessary Dag version records, increasing storage and query overhead.
+* The UI shows a misleading history of Dag changes, making it harder to identify real modifications.
+* The scheduler and API server may consume more memory as they load and cache a growing number of Dag versions.
+
+Common causes
+"""""""""""""
+
+Version inflation is caused by using values that change at **parse time** — that is, every time the Dag
+processor evaluates the Dag file — as arguments to Dag or Task constructors. The most common patterns are:
+
+**1. Using ``datetime.now()`` or ``pendulum.now()`` as ``start_date``:**
+
+.. code-block:: python
+
+ from datetime import datetime
+
+ from airflow.sdk import DAG
+
+ # BAD: datetime.now() produces a different value on every parse
+ with DAG(
+ dag_id="bad_example",
+ start_date=datetime.now(),
+ schedule="@daily",
+ ):
+ ...
Review Comment:
```suggestion
with DAG(
dag_id="bad_example",
# BAD: datetime.now() produces a different value on every parse
start_date=datetime.now(),
schedule="@daily",
):
...
```
##########
airflow-core/docs/faq.rst:
##########
@@ -237,6 +237,166 @@ There are several reasons why Dags might disappear from the UI. Common causes in
* **Time synchronization issues** - Ensure all nodes (database, schedulers, workers) use NTP with <1s clock drift.
+.. _faq:dag-version-inflation:
+
+Why does my Dag version keep increasing?
+-----------------------------------------
+
+Every time the Dag processor parses a Dag file, it serializes the Dag and compares the result with the
+version stored in the metadata database. If anything has changed, Airflow creates a new Dag version. This
+mechanism ensures that Dag runs use consistent code throughout their execution, even if the Dag file is
+updated mid-run.
+
+**Dag version inflation** occurs when the version number increases indefinitely without the Dag author
+making any intentional changes.
+
+What goes wrong
+"""""""""""""""
+
+When Dag versions increase without meaningful changes:
+
+* The metadata database accumulates unnecessary Dag version records, increasing storage and query overhead.
+* The UI shows a misleading history of Dag changes, making it harder to identify real modifications.
+* The scheduler and API server may consume more memory as they load and cache a growing number of Dag versions.
+
+Common causes
+"""""""""""""
+
+Version inflation is caused by using values that change at **parse time** — that is, every time the Dag
+processor evaluates the Dag file — as arguments to Dag or Task constructors. The most common patterns are:
+
+**1. Using ``datetime.now()`` or ``pendulum.now()`` as ``start_date``:**
+
+.. code-block:: python
+
+ from datetime import datetime
+
+ from airflow.sdk import DAG
+
+ # BAD: datetime.now() produces a different value on every parse
+ with DAG(
+ dag_id="bad_example",
+ start_date=datetime.now(),
+ schedule="@daily",
+ ):
+ ...
+
+Every parse produces a different ``start_date``, so the serialized Dag is always different from the
+stored version.
+
+**2. Using random values in Dag or Task arguments:**
+
+.. code-block:: python
+
+ import random
+
+ from airflow.sdk import DAG
+ from airflow.providers.standard.operators.python import PythonOperator
+
+ with DAG(dag_id="bad_random", start_date="2024-01-01", schedule="@daily") as dag:
+ # BAD: random value changes every parse
+ PythonOperator(
+ task_id=f"task_{random.randint(1, 1000)}",
+ python_callable=lambda: None,
+ )
Review Comment:
```suggestion
with DAG(dag_id="bad_random", start_date="2024-01-01", schedule="@daily") as dag:
PythonOperator(
# BAD: random value changes every parse
task_id=f"task_{random.randint(1, 1000)}",
python_callable=lambda: None,
)
```
##########
airflow-core/docs/faq.rst:
##########
@@ -237,6 +237,166 @@ There are several reasons why Dags might disappear from the UI. Common causes in
* **Time synchronization issues** - Ensure all nodes (database, schedulers, workers) use NTP with <1s clock drift.
+.. _faq:dag-version-inflation:
+
+Why does my Dag version keep increasing?
+-----------------------------------------
+
+Every time the Dag processor parses a Dag file, it serializes the Dag and compares the result with the
+version stored in the metadata database. If anything has changed, Airflow creates a new Dag version. This
+mechanism ensures that Dag runs use consistent code throughout their execution, even if the Dag file is
+updated mid-run.
+
+**Dag version inflation** occurs when the version number increases indefinitely without the Dag author
+making any intentional changes.
+
+What goes wrong
+"""""""""""""""
+
+When Dag versions increase without meaningful changes:
+
+* The metadata database accumulates unnecessary Dag version records, increasing storage and query overhead.
+* The UI shows a misleading history of Dag changes, making it harder to identify real modifications.
+* The scheduler and API server may consume more memory as they load and cache a growing number of Dag versions.
+
+Common causes
+"""""""""""""
+
+Version inflation is caused by using values that change at **parse time** — that is, every time the Dag
+processor evaluates the Dag file — as arguments to Dag or Task constructors. The most common patterns are:
+
+**1. Using ``datetime.now()`` or ``pendulum.now()`` as ``start_date``:**
+
+.. code-block:: python
+
+ from datetime import datetime
+
+ from airflow.sdk import DAG
+
+ # BAD: datetime.now() produces a different value on every parse
+ with DAG(
+ dag_id="bad_example",
+ start_date=datetime.now(),
+ schedule="@daily",
+ ):
+ ...
+
+Every parse produces a different ``start_date``, so the serialized Dag is always different from the
+stored version.
+
+**2. Using random values in Dag or Task arguments:**
+
+.. code-block:: python
+
+ import random
+
+ from airflow.sdk import DAG
+ from airflow.providers.standard.operators.python import PythonOperator
+
+ with DAG(dag_id="bad_random", start_date="2024-01-01", schedule="@daily") as dag:
+ # BAD: random value changes every parse
+ PythonOperator(
+ task_id=f"task_{random.randint(1, 1000)}",
+ python_callable=lambda: None,
+ )
+
+**3. Assigning runtime-varying values to variables used in constructors:**
+
+.. code-block:: python
+
+ from datetime import datetime
+
+ from airflow.sdk import DAG
+ from airflow.providers.standard.operators.python import PythonOperator
+
+ # BAD: the variable captures a parse-time value, then is passed to the DAG
+ default_args = {"start_date": datetime.now()}
+
+ with DAG(dag_id="bad_defaults", default_args=default_args, schedule="@daily") as dag:
+ PythonOperator(task_id="my_task", python_callable=lambda: None)
+
+Even though ``datetime.now()`` is not called directly inside the Dag constructor, it flows in through
+``default_args`` and still causes a different serialized Dag on every parse.
+
+**4. Using environment variables or file contents that change between parses:**
+
+.. code-block:: python
+
+ import os
+
+ from airflow.sdk import DAG
+ from airflow.providers.standard.operators.bash import BashOperator
+
+ with DAG(dag_id="bad_env", start_date="2024-01-01", schedule="@daily") as dag:
+ # BAD if BUILD_NUMBER changes on every deployment or parse
+ BashOperator(
+ task_id="echo_build",
+ bash_command=f"echo {os.environ.get('BUILD_NUMBER', 'unknown')}",
+ )
+
+How to avoid version inflation
+""""""""""""""""""""""""""""""
+
+* **Use fixed ``start_date`` values.** Always set ``start_date`` to a static ``datetime`` literal:
+
+ .. code-block:: python
+
+ import datetime
+
+ from airflow.sdk import DAG
+
+ with DAG(
+ dag_id="good_example",
+ start_date=datetime.datetime(2024, 1, 1),
+ schedule="@daily",
+ ):
+ ...
+
+* **Keep all Dag and Task constructor arguments deterministic.** Arguments passed to Dag and Operator
+ constructors must produce the same value on every parse. Move any dynamic computation into the
+ ``execute()`` method or use Jinja templates, which are evaluated at task execution time rather than
+ parse time.
+
+* **Use Jinja templates for dynamic values:**
+
+ .. code-block:: python
+
+ from airflow.providers.standard.operators.bash import BashOperator
+
+ # GOOD: the template is resolved at execution time, not parse time
+ BashOperator(
+ task_id="echo_date",
+ bash_command="echo {{ ds }}",
+ )
+
+* **Use Airflow Variables with templates instead of top-level lookups:**
+
+ .. code-block:: python
+
+ from airflow.providers.standard.operators.bash import BashOperator
+
+ # GOOD: Variable is resolved at execution time via template
+ BashOperator(
+ task_id="echo_var",
+ bash_command="echo {{ var.value.my_variable }}",
+ )
+
+Dag version inflation detection
+""""""""""""""""""""""""""""""""
+
+Starting from Airflow 3.2, the Dag processor performs **AST-based static analysis** on every Dag file
+before parsing to detect runtime-varying values in Dag and Task constructors. When a potential issue is
+found, it is surfaced as a **Dag warning** visible in the UI.
+
+You can control this behavior with the
+:ref:`dag_version_inflation_check_level <config:dag_processor__dag_version_inflation_check_level>`
+configuration option:
+
+* ``off`` — Disables the check entirely. No errors or warnings are generated.
+* ``warning`` (default) — Dags load normally but warnings are displayed in the UI when issues are detected.
+* ``error`` — Treats detected issues as Dag import errors, preventing the Dag from loading.
Review Comment:
Probably could mention the ruff rule https://docs.astral.sh/ruff/rules/airflow3-dag-dynamic-value/ here?
##########
airflow-core/docs/faq.rst:
##########
@@ -237,6 +237,166 @@ There are several reasons why Dags might disappear from the UI. Common causes in
* **Time synchronization issues** - Ensure all nodes (database, schedulers, workers) use NTP with <1s clock drift.
+.. _faq:dag-version-inflation:
+
+Why does my Dag version keep increasing?
+-----------------------------------------
+
+Every time the Dag processor parses a Dag file, it serializes the Dag and compares the result with the
+version stored in the metadata database. If anything has changed, Airflow creates a new Dag version. This
+mechanism ensures that Dag runs use consistent code throughout their execution, even if the Dag file is
+updated mid-run.
+
+**Dag version inflation** occurs when the version number increases indefinitely without the Dag author
+making any intentional changes.
+
+What goes wrong
+"""""""""""""""
+
+When Dag versions increase without meaningful changes:
+
+* The metadata database accumulates unnecessary Dag version records, increasing storage and query overhead.
+* The UI shows a misleading history of Dag changes, making it harder to identify real modifications.
+* The scheduler and API server may consume more memory as they load and cache a growing number of Dag versions.
+
+Common causes
+"""""""""""""
+
+Version inflation is caused by using values that change at **parse time** — that is, every time the Dag
+processor evaluates the Dag file — as arguments to Dag or Task constructors. The most common patterns are:
+
+**1. Using ``datetime.now()`` or ``pendulum.now()`` as ``start_date``:**
+
+.. code-block:: python
+
+ from datetime import datetime
+
+ from airflow.sdk import DAG
+
+ # BAD: datetime.now() produces a different value on every parse
+ with DAG(
+ dag_id="bad_example",
+ start_date=datetime.now(),
+ schedule="@daily",
+ ):
+ ...
+
+Every parse produces a different ``start_date``, so the serialized Dag is always different from the
+stored version.
+
+**2. Using random values in Dag or Task arguments:**
+
+.. code-block:: python
+
+ import random
+
+ from airflow.sdk import DAG
+ from airflow.providers.standard.operators.python import PythonOperator
+
+ with DAG(dag_id="bad_random", start_date="2024-01-01", schedule="@daily") as dag:
+ # BAD: random value changes every parse
+ PythonOperator(
+ task_id=f"task_{random.randint(1, 1000)}",
+ python_callable=lambda: None,
+ )
+
+**3. Assigning runtime-varying values to variables used in constructors:**
+
+.. code-block:: python
+
+ from datetime import datetime
+
+ from airflow.sdk import DAG
+ from airflow.providers.standard.operators.python import PythonOperator
+
+ # BAD: the variable captures a parse-time value, then is passed to the DAG
+ default_args = {"start_date": datetime.now()}
+
+ with DAG(dag_id="bad_defaults", default_args=default_args, schedule="@daily") as dag:
+ PythonOperator(task_id="my_task", python_callable=lambda: None)
+
+Even though ``datetime.now()`` is not called directly inside the Dag constructor, it flows in through
+``default_args`` and still causes a different serialized Dag on every parse.
+
+**4. Using environment variables or file contents that change between parses:**
+
+.. code-block:: python
+
+ import os
+
+ from airflow.sdk import DAG
+ from airflow.providers.standard.operators.bash import BashOperator
+
+ with DAG(dag_id="bad_env", start_date="2024-01-01", schedule="@daily") as dag:
+ # BAD if BUILD_NUMBER changes on every deployment or parse
+ BashOperator(
+ task_id="echo_build",
+ bash_command=f"echo {os.environ.get('BUILD_NUMBER', 'unknown')}",
+ )
Review Comment:
```suggestion
BashOperator(
task_id="echo_build",
# BAD if BUILD_NUMBER changes on every deployment or parse
bash_command=f"echo {os.environ.get('BUILD_NUMBER', 'unknown')}",
```
##########
airflow-core/docs/faq.rst:
##########
@@ -237,6 +237,166 @@ There are several reasons why Dags might disappear from the UI. Common causes in
* **Time synchronization issues** - Ensure all nodes (database, schedulers, workers) use NTP with <1s clock drift.
+.. _faq:dag-version-inflation:
+
+Why does my Dag version keep increasing?
+-----------------------------------------
+
+Every time the Dag processor parses a Dag file, it serializes the Dag and compares the result with the
+version stored in the metadata database. If anything has changed, Airflow creates a new Dag version. This
+mechanism ensures that Dag runs use consistent code throughout their execution, even if the Dag file is
+updated mid-run.
+
+**Dag version inflation** occurs when the version number increases indefinitely without the Dag author
+making any intentional changes.
+
+What goes wrong
+"""""""""""""""
+
+When Dag versions increase without meaningful changes:
+
+* The metadata database accumulates unnecessary Dag version records, increasing storage and query overhead.
+* The UI shows a misleading history of Dag changes, making it harder to identify real modifications.
+* The scheduler and API server may consume more memory as they load and cache a growing number of Dag versions.
+
+Common causes
+"""""""""""""
+
+Version inflation is caused by using values that change at **parse time** — that is, every time the Dag
+processor evaluates the Dag file — as arguments to Dag or Task constructors. The most common patterns are:
+
+**1. Using ``datetime.now()`` or ``pendulum.now()`` as ``start_date``:**
+
+.. code-block:: python
+
+ from datetime import datetime
+
+ from airflow.sdk import DAG
+
+ # BAD: datetime.now() produces a different value on every parse
+ with DAG(
+ dag_id="bad_example",
+ start_date=datetime.now(),
+ schedule="@daily",
+ ):
+ ...
+
+Every parse produces a different ``start_date``, so the serialized Dag is always different from the
+stored version.
+
+**2. Using random values in Dag or Task arguments:**
+
+.. code-block:: python
+
+ import random
+
+ from airflow.sdk import DAG
+ from airflow.providers.standard.operators.python import PythonOperator
+
+ with DAG(dag_id="bad_random", start_date="2024-01-01", schedule="@daily") as dag:
+ # BAD: random value changes every parse
+ PythonOperator(
+ task_id=f"task_{random.randint(1, 1000)}",
+ python_callable=lambda: None,
+ )
+
+**3. Assigning runtime-varying values to variables used in constructors:**
+
+.. code-block:: python
+
+ from datetime import datetime
+
+ from airflow.sdk import DAG
+ from airflow.providers.standard.operators.python import PythonOperator
+
+ # BAD: the variable captures a parse-time value, then is passed to the DAG
+ default_args = {"start_date": datetime.now()}
+
+ with DAG(dag_id="bad_defaults", default_args=default_args, schedule="@daily") as dag:
+ PythonOperator(task_id="my_task", python_callable=lambda: None)
+
+Even though ``datetime.now()`` is not called directly inside the Dag constructor, it flows in through
+``default_args`` and still causes a different serialized Dag on every parse.
+
+**4. Using environment variables or file contents that change between parses:**
+
+.. code-block:: python
+
+ import os
+
+ from airflow.sdk import DAG
+ from airflow.providers.standard.operators.bash import BashOperator
+
+ with DAG(dag_id="bad_env", start_date="2024-01-01", schedule="@daily") as dag:
+ # BAD if BUILD_NUMBER changes on every deployment or parse
+ BashOperator(
+ task_id="echo_build",
+ bash_command=f"echo {os.environ.get('BUILD_NUMBER', 'unknown')}",
+ )
+
+How to avoid version inflation
+""""""""""""""""""""""""""""""
+
+* **Use fixed ``start_date`` values.** Always set ``start_date`` to a static ``datetime`` literal:
+
+ .. code-block:: python
+
+ import datetime
+
+ from airflow.sdk import DAG
+
+ with DAG(
+ dag_id="good_example",
+ start_date=datetime.datetime(2024, 1, 1),
+ schedule="@daily",
+ ):
+ ...
+
+* **Keep all Dag and Task constructor arguments deterministic.** Arguments passed to Dag and Operator
+ constructors must produce the same value on every parse. Move any dynamic computation into the
+ ``execute()`` method or use Jinja templates, which are evaluated at task execution time rather than
+ parse time.
+
+* **Use Jinja templates for dynamic values:**
+
+ .. code-block:: python
+
+ from airflow.providers.standard.operators.bash import BashOperator
+
+ # GOOD: the template is resolved at execution time, not parse time
+ BashOperator(
+ task_id="echo_date",
+ bash_command="echo {{ ds }}",
+ )
+
+* **Use Airflow Variables with templates instead of top-level lookups:**
+
+ .. code-block:: python
+
+ from airflow.providers.standard.operators.bash import BashOperator
+
+ # GOOD: Variable is resolved at execution time via template
+ BashOperator(
+ task_id="echo_var",
+ bash_command="echo {{ var.value.my_variable }}",
+ )
Review Comment:
```suggestion
BashOperator(
task_id="echo_date",
# GOOD: the template is resolved at execution time, not parse time
bash_command="echo {{ ds }}",
)
* **Use Airflow Variables with templates instead of top-level lookups:**
.. code-block:: python
from airflow.providers.standard.operators.bash import BashOperator
BashOperator(
task_id="echo_var",
# GOOD: Variable is resolved at execution time via template
bash_command="echo {{ var.value.my_variable }}",
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org