You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by js...@apache.org on 2025/08/30 17:08:23 UTC
(airflow) branch main updated: Make term Dag consistent in docs (#55097)
This is an automated email from the ASF dual-hosted git repository.
jscheffl pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 1f4e1565b89 Make term Dag consistent in docs (#55097)
1f4e1565b89 is described below
commit 1f4e1565b899c84fef5fa5c7525441ed05ff7df7
Author: Jens Scheffler <95...@users.noreply.github.com>
AuthorDate: Sat Aug 30 19:08:17 2025 +0200
Make term Dag consistent in docs (#55097)
* Make term Dag consistent in docs
* Fix uups
* Found a couple of leftovers
* Update airflow-core/docs/extra-packages-ref.rst
* Update airflow-core/docs/extra-packages-ref.rst
* Update airflow-core/docs/templates-ref.rst
---
.../cluster-policies.rst | 30 +--
.../administration-and-deployment/dag-bundles.rst | 66 ++---
.../dag-serialization.rst | 64 ++---
.../docs/administration-and-deployment/index.rst | 2 +-
.../administration-and-deployment/listeners.rst | 4 +-
.../logging-monitoring/errors.rst | 4 +-
.../logging-monitoring/metrics.rst | 62 ++---
.../modules_management.rst | 36 +--
.../docs/administration-and-deployment/plugins.rst | 4 +-
.../priority-weight.rst | 14 +-
.../production-deployment.rst | 20 +-
.../administration-and-deployment/scheduler.rst | 46 ++--
.../authoring-and-scheduling/asset-scheduling.rst | 70 +++---
.../docs/authoring-and-scheduling/assets.rst | 8 +-
.../docs/authoring-and-scheduling/cron.rst | 10 +-
.../docs/authoring-and-scheduling/deferring.rst | 10 +-
.../dynamic-task-mapping.rst | 14 +-
.../authoring-and-scheduling/event-scheduling.rst | 20 +-
.../docs/authoring-and-scheduling/index.rst | 2 +-
.../docs/authoring-and-scheduling/serializers.rst | 2 +-
.../docs/authoring-and-scheduling/timetable.rst | 72 +++---
.../docs/authoring-and-scheduling/timezone.rst | 32 +--
airflow-core/docs/best-practices.rst | 266 ++++++++++-----------
airflow-core/docs/cli-and-env-variables-ref.rst | 4 +-
.../docs/core-concepts/auth-manager/index.rst | 30 +--
.../core-concepts/auth-manager/simple/index.rst | 4 +-
airflow-core/docs/core-concepts/backfill.rst | 16 +-
airflow-core/docs/core-concepts/dag-run.rst | 106 ++++----
airflow-core/docs/core-concepts/dags.rst | 68 +++---
airflow-core/docs/core-concepts/debug.rst | 34 +--
airflow-core/docs/core-concepts/executor/index.rst | 16 +-
airflow-core/docs/core-concepts/objectstorage.rst | 6 +-
airflow-core/docs/core-concepts/operators.rst | 16 +-
airflow-core/docs/core-concepts/overview.rst | 69 +++---
airflow-core/docs/core-concepts/params.rst | 42 ++--
airflow-core/docs/core-concepts/taskflow.rst | 10 +-
airflow-core/docs/core-concepts/tasks.rst | 18 +-
airflow-core/docs/core-concepts/variables.rst | 2 +-
airflow-core/docs/core-concepts/xcoms.rst | 2 +-
airflow-core/docs/extra-packages-ref.rst | 4 +-
airflow-core/docs/faq.rst | 146 +++++------
airflow-core/docs/howto/add-dag-tags.rst | 8 +-
airflow-core/docs/howto/add-owner-links.rst | 10 +-
airflow-core/docs/howto/custom-operator.rst | 14 +-
airflow-core/docs/howto/customize-ui.rst | 6 +-
airflow-core/docs/howto/docker-compose/index.rst | 10 +-
airflow-core/docs/howto/dynamic-dag-generation.rst | 92 +++----
airflow-core/docs/howto/listener-plugin.rst | 14 +-
airflow-core/docs/howto/notifications.rst | 2 +-
airflow-core/docs/howto/operator/index.rst | 2 +-
airflow-core/docs/howto/set-config.rst | 2 +-
airflow-core/docs/howto/set-up-database.rst | 2 +-
airflow-core/docs/howto/setup-and-teardown.rst | 22 +-
airflow-core/docs/howto/timetable.rst | 48 ++--
airflow-core/docs/howto/usage-cli.rst | 44 ++--
airflow-core/docs/howto/variable.rst | 2 +-
airflow-core/docs/index.rst | 28 +--
airflow-core/docs/installation/index.rst | 4 +-
.../docs/installation/upgrading_to_airflow3.rst | 26 +-
airflow-core/docs/public-airflow-interface.rst | 48 ++--
airflow-core/docs/security/audit_logs.rst | 10 +-
.../docs/security/deprecated_permissions.rst | 2 +-
airflow-core/docs/security/kerberos.rst | 10 +-
.../security/secrets/mask-sensitive-values.rst | 2 +-
airflow-core/docs/security/security_model.rst | 50 ++--
airflow-core/docs/security/sql.rst | 4 +-
airflow-core/docs/start.rst | 4 +-
airflow-core/docs/templates-ref.rst | 14 +-
airflow-core/docs/troubleshooting.rst | 4 +-
airflow-core/docs/tutorial/fundamentals.rst | 60 ++---
airflow-core/docs/tutorial/objectstorage.rst | 8 +-
airflow-core/docs/tutorial/pipeline.rst | 22 +-
airflow-core/docs/tutorial/taskflow.rst | 36 +--
airflow-core/docs/ui.rst | 202 ++++++++--------
airflow-core/newsfragments/50374.feature.rst | 2 +-
airflow-core/newsfragments/51424.significant.rst | 2 +-
airflow-core/newsfragments/53796.misc.rst | 2 +-
.../newsfragments/template.significant.rst | 2 +-
78 files changed, 1144 insertions(+), 1127 deletions(-)
diff --git a/airflow-core/docs/administration-and-deployment/cluster-policies.rst b/airflow-core/docs/administration-and-deployment/cluster-policies.rst
index b67234c61b5..2d410b5a507 100644
--- a/airflow-core/docs/administration-and-deployment/cluster-policies.rst
+++ b/airflow-core/docs/administration-and-deployment/cluster-policies.rst
@@ -18,17 +18,19 @@
Cluster Policies
================
-If you want to check or mutate dags or Tasks on a cluster-wide level, then a Cluster Policy will let you do
-that. They have three main purposes:
+If you want to check or mutate Dags or Tasks on a cluster-wide level, then a Cluster Policy will let you do
+it. They also let you apply cluster-wide settings to Dags based on the dag_id or other properties.
-* Checking that dags / tasks meet a certain standard
-* Setting default arguments on dags / tasks
+Common use-cases include:
+
+* Checking that Dags / tasks meet a certain standard
+* Setting default arguments on Dags / tasks
* Performing custom routing logic
There are three main types of cluster policy:
* ``dag_policy``: Takes a :class:`~airflow.models.dag.DAG` parameter called ``dag``. Runs at load time of the
- DAG from DagBag :class:`~airflow.models.dagbag.DagBag`.
+ Dag from DagBag :class:`~airflow.models.dagbag.DagBag`.
* ``task_policy``: Takes a :class:`~airflow.models.baseoperator.BaseOperator` parameter called ``task``. The
policy gets executed when the task is created during parsing of the task from DagBag at load time. This
means that the whole task definition can be altered in the task policy. It does not relate to a specific
@@ -36,19 +38,19 @@ There are three main types of cluster policy:
executed in the future.
* ``task_instance_mutation_hook``: Takes a :class:`~airflow.models.taskinstance.TaskInstance` parameter called
``task_instance``. The ``task_instance_mutation_hook`` applies not to a task but to the instance of a task that
- relates to a particular DagRun. It is executed in a "worker", not in the dag file processor, just before the
+ relates to a particular DagRun. It is executed in a "worker", not in the Dag file processor, just before the
task instance is executed. The policy is only applied to the currently executed run (i.e. instance) of that
task.
-The DAG and Task cluster policies can raise the :class:`~airflow.exceptions.AirflowClusterPolicyViolation`
-exception to indicate that the dag/task they were passed is not compliant and should not be loaded.
+The Dag and Task cluster policies can raise the :class:`~airflow.exceptions.AirflowClusterPolicyViolation`
+exception to indicate that the Dag/task they were passed is not compliant and should not be loaded.
They can also raise the :class:`~airflow.exceptions.AirflowClusterPolicySkipDag` exception
-when skipping that DAG is needed intentionally. Unlike :class:`~airflow.exceptions.AirflowClusterPolicyViolation`,
+when skipping that Dag is needed intentionally. Unlike :class:`~airflow.exceptions.AirflowClusterPolicyViolation`,
this exception is not displayed on the Airflow web UI (Internally, it's not recorded on ``import_error`` table on meta database.)
-Any extra attributes set by a cluster policy take priority over those defined in your DAG file; for example,
-if you set an ``sla`` on your Task in the DAG file, and then your cluster policy also sets an ``sla``, the
+Any extra attributes set by a cluster policy take priority over those defined in your Dag file; for example,
+if you set an ``sla`` on your Task in the Dag file, and then your cluster policy also sets an ``sla``, the
cluster policy's value will take precedence.
.. _administration-and-deployment:cluster-policies-define:
@@ -124,10 +126,10 @@ Available Policy Functions
Examples
--------
-DAG policies
+Dag policies
~~~~~~~~~~~~
-This policy checks if each DAG has at least one tag defined:
+This policy checks if each Dag has at least one tag defined:
.. literalinclude:: /../tests/unit/cluster_policies/__init__.py
:language: python
@@ -140,7 +142,7 @@ This policy checks if each DAG has at least one tag defined:
.. note::
- DAG policies are applied after the DAG has been completely loaded, so overriding the ``default_args`` parameter has no effect. If you want to override the default operator settings, use task policies instead.
+ Dag policies are applied after the Dag has been completely loaded, so overriding the ``default_args`` parameter has no effect. If you want to override the default operator settings, use task policies instead.
Task policies
~~~~~~~~~~~~~
diff --git a/airflow-core/docs/administration-and-deployment/dag-bundles.rst b/airflow-core/docs/administration-and-deployment/dag-bundles.rst
index 53bd5e16afb..3e58e1d7585 100644
--- a/airflow-core/docs/administration-and-deployment/dag-bundles.rst
+++ b/airflow-core/docs/administration-and-deployment/dag-bundles.rst
@@ -18,49 +18,49 @@
Dag Bundles
===========
-A dag bundle is a collection of one or more dags, files along with their associated files, such as other
-Python scripts, configuration files, or other resources. Dag bundles can source the dags from various
+A Dag bundle is a collection of one or more Dags, files along with their associated files, such as other
+Python scripts, configuration files, or other resources. Dag bundles can source the Dags from various
locations, such as local directories, Git repositories, or other external systems. Deployment administrators
-can also write their own dag bundle classes to support custom sources. You can also define more than one dag
-bundle in an Airflow deployments, allowing for better organization of your dags. By keeping the bundle at a
-higher level, it allows for versioning everything the dag needs to run.
+can also write their own Dag bundle classes to support custom sources. You can also define more than one Dag
+bundle in an Airflow deployments, allowing for better organization of your Dags. By keeping the bundle at a
+higher level, it allows for versioning everything the Dag needs to run.
-This is similar, but more powerful than the *dags folder* in Airflow 2 or earlier, where dags were required to
-be in one place on the local disk, and getting the dags there was solely the responsibility of the deployment
+This is similar, but more powerful than the *Dags folder* in Airflow 2 or earlier, where Dags were required to
+be in one place on the local disk, and getting the Dags there was solely the responsibility of the deployment
manager.
-Since dag bundles support versioning, they also allow Airflow to run a task using a specific version of the
-dag bundle, allowing for a dag run to use the same code for the whole run, even if the dag is updated mid-way
+Since Dag bundles support versioning, they also allow Airflow to run a task using a specific version of the
+Dag bundle, allowing for a Dag run to use the same code for the whole run, even if the Dag is updated mid-way
through the run.
-Why are dag bundles important?
+Why are Dag bundles important?
------------------------------
-- **Version Control**: By supporting versioning, dag bundles allow dag runs to use the same code for the whole run, even if the dag is updated mid way through the run.
-- **Scalability**: With dag bundles, Airflow can efficiently manage large numbers of DAGs by organizing them into logical units.
-- **Flexibility**: Dag bundles enable seamless integration with external systems, such as Git repositories, to source dags.
+- **Version Control**: By supporting versioning, Dag bundles allow Dag runs to use the same code for the whole run, even if the Dag is updated mid way through the run.
+- **Scalability**: With Dag bundles, Airflow can efficiently manage large numbers of Dags by organizing them into logical units.
+- **Flexibility**: Dag bundles enable seamless integration with external systems, such as Git repositories, to source Dags.
-Types of dag bundles
+Types of Dag bundles
--------------------
-Airflow supports multiple types of dag Bundles, each catering to specific use cases:
+Airflow supports multiple types of Dag Bundles, each catering to specific use cases:
**airflow.dag_processing.bundles.local.LocalDagBundle**
- These bundles reference a local directory containing DAG files. They are ideal for development and testing environments, but do not support versioning of the bundle, meaning tasks always run using the latest code.
+ These bundles reference a local directory containing Dag files. They are ideal for development and testing environments, but do not support versioning of the bundle, meaning tasks always run using the latest code.
**airflow.providers.git.bundles.git.GitDagBundle**
- These bundles integrate with Git repositories, allowing Airflow to fetch dags directly from a repository.
+ These bundles integrate with Git repositories, allowing Airflow to fetch Dags directly from a repository.
**airflow.providers.amazon.aws.bundles.s3.S3DagBundle**
- These bundles reference an S3 bucket containing DAG files. They do not support versioning of the bundle, meaning tasks always run using the latest code.
+ These bundles reference an S3 bucket containing Dag files. They do not support versioning of the bundle, meaning tasks always run using the latest code.
-Configuring dag bundles
+Configuring Dag bundles
-----------------------
-Dag bundles are configured in :ref:`config:dag_processor__dag_bundle_config_list`. You can add one or more dag bundles here.
+Dag bundles are configured in :ref:`config:dag_processor__dag_bundle_config_list`. You can add one or more Dag bundles here.
-By default, Airflow adds a local dag bundle, which is the same as the old dags folder. This is done for backwards compatibility, and you can remove it if you do not want to use it. You can also keep it and add other dag bundles, such as a git dag bundle.
+By default, Airflow adds a local Dag bundle, which is the same as the old Dags folder. This is done for backwards compatibility, and you can remove it if you do not want to use it. You can also keep it and add other Dag bundles, such as a git Dag bundle.
-For example, adding multiple dag bundles to your ``airflow.cfg`` file:
+For example, adding multiple Dag bundles to your ``airflow.cfg`` file:
.. code-block:: ini
@@ -83,8 +83,8 @@ For example, adding multiple dag bundles to your ``airflow.cfg`` file:
The whitespace, particularly on the last line, is important so a multi-line value works properly. More details can be found in the
the `configparser docs <https://docs.python.org/3/library/configparser.html#supported-ini-file-structure>`_.
-If you want a view url different from the default provided by the dag bundle, you can change the url in the kwargs of the dag bundle configuration.
-For example, if you want to use a custom URL for the git dag bundle:
+If you want a view url different from the default provided by the Dag bundle, you can change the url in the kwargs of the Dag bundle configuration.
+For example, if you want to use a custom URL for the git Dag bundle:
.. code-block:: ini
@@ -103,12 +103,12 @@ For example, if you want to use a custom URL for the git dag bundle:
Above, the ``view_url_template`` is set to a custom URL that will be used to view the Dags in the ``my_git_repo`` bundle. The ``{subdir}`` placeholder will be replaced
with the ``subdir`` attribute of the bundle. The placeholders are attributes of the bundle. You cannot use any placeholder outside of the bundle's attributes.
-When you specify a custom URL, it overrides the default URL provided by the dag bundle.
+When you specify a custom URL, it overrides the default URL provided by the Dag bundle.
The url is verified for safety, and if it is not safe, the view url for the bundle will be set to ``None``. This is to prevent any potential security issues with unsafe URLs.
-You can also override the :ref:`config:dag_processor__refresh_interval` per dag bundle by passing it in kwargs.
-This controls how often the dag processor refreshes, or looks for new files, in the dag bundles.
+You can also override the :ref:`config:dag_processor__refresh_interval` per Dag bundle by passing it in kwargs.
+This controls how often the Dag processor refreshes, or looks for new files, in the Dag bundles.
Starting Airflow 3.0.2 git is pre installed in the base image. However, if you are using versions prior 3.0.2, you would need to install git in your docker image.
@@ -119,18 +119,18 @@ Starting Airflow 3.0.2 git is pre installed in the base image. However, if you a
ENV GIT_PYTHON_REFRESH=quiet
-Writing custom dag bundles
+Writing custom Dag bundles
--------------------------
-When implementing your own dag bundle by extending the ``BaseDagBundle`` class, there are several methods you must implement. Below is a guide to help you implement a custom dag bundle.
+When implementing your own Dag bundle by extending the ``BaseDagBundle`` class, there are several methods you must implement. Below is a guide to help you implement a custom Dag bundle.
Abstract Methods
~~~~~~~~~~~~~~~~
The following methods are abstract and must be implemented in your custom bundle class:
**path**
- This property should return a ``Path`` to the directory where the dag files for this bundle are stored.
- Airflow uses this property to locate the DAG files for processing.
+ This property should return a ``Path`` to the directory where the Dag files for this bundle are stored.
+ Airflow uses this property to locate the Dag files for processing.
**get_current_version**
This method should return the current version of the bundle as a string.
@@ -139,7 +139,7 @@ The following methods are abstract and must be implemented in your custom bundle
**refresh**
This method should handle refreshing the bundle's contents from its source (e.g., pulling the latest changes from a remote repository).
- This is used by the dag processor periodically to ensure that the bundle is up-to-date.
+ This is used by the Dag processor periodically to ensure that the bundle is up-to-date.
Optional Methods
~~~~~~~~~~~~~~~~
@@ -152,7 +152,7 @@ In addition to the abstract methods, you may choose to override the following me
in the ``initialize`` method instead.
**initialize**
- This method is called before the bundle is first used in the dag processor or worker. It allows you to perform expensive operations only when the bundle's content is accessed.
+ This method is called before the bundle is first used in the Dag processor or worker. It allows you to perform expensive operations only when the bundle's content is accessed.
**view_url**
This method should return a URL as a string to view the bundle on an external system (e.g., a Git repository's web interface).
diff --git a/airflow-core/docs/administration-and-deployment/dag-serialization.rst b/airflow-core/docs/administration-and-deployment/dag-serialization.rst
index 7c1d9c5435c..89384f3436b 100644
--- a/airflow-core/docs/administration-and-deployment/dag-serialization.rst
+++ b/airflow-core/docs/administration-and-deployment/dag-serialization.rst
@@ -18,38 +18,38 @@
.. _dag-serialization:
-DAG Serialization
+Dag Serialization
=================
In order to make Airflow Webserver stateless, Airflow >=1.10.7 supports
-DAG Serialization and DB Persistence. From Airflow 2.0.0, the Scheduler
-also uses serialized dags for consistency and makes scheduling decisions.
+Dag Serialization and DB Persistence. From Airflow 2.0.0, the Scheduler
+also uses serialized Dags for consistency and makes scheduling decisions.
.. image:: ../img/dag_serialization.png
-Without DAG Serialization & persistence in DB, the Webserver and the Scheduler both
-need access to the DAG files. Both the Scheduler and Webserver parse the DAG files.
+Without Dag Serialization & persistence in DB, the Webserver and the Scheduler both
+need access to the Dag files. Both the Scheduler and Webserver parse the Dag files.
-With **DAG Serialization** we aim to decouple the Webserver from DAG parsing
+With **Dag Serialization** we aim to decouple the Webserver from Dag parsing
which would make the Webserver very light-weight.
As shown in the image above, when using this feature,
the :class:`~airflow.jobs.scheduler_job.DagFileProcessorProcess` in the Scheduler
-parses the DAG files, serializes them in JSON format and saves them in the Metadata DB
+parses the Dag files, serializes them in JSON format and saves them in the Metadata DB
as :class:`~airflow.models.serialized_dag.SerializedDagModel` model.
-The Webserver now instead of having to parse the DAG files again, reads the
-serialized dags in JSON, de-serializes them and creates the DagBag and uses it
-to show in the UI. And the Scheduler does not need the actual dags for making scheduling decisions,
-instead of using the dag files, we use the serialized dags that contain all the information needed to
-schedule the dags from Airflow 2.0.0 (this was done as part of :ref:`Scheduler HA <scheduler:ha>`).
+The Webserver now instead of having to parse the Dag files again, reads the
+serialized Dags in JSON, de-serializes them and creates the DagBag and uses it
+to show in the UI. And the Scheduler does not need the actual Dags for making scheduling decisions,
+instead of using the Dag files, we use the serialized Dags that contain all the information needed to
+schedule the Dags from Airflow 2.0.0 (this was done as part of :ref:`Scheduler HA <scheduler:ha>`).
-One of the key features that is implemented as a part of DAG Serialization is that
-instead of loading an entire DagBag when the WebServer starts we only load each DAG on demand from the
+One of the key features that is implemented as a part of Dag Serialization is that
+instead of loading an entire DagBag when the WebServer starts we only load each Dag on demand from the
Serialized Dag table. It helps to reduce the Webserver startup time and memory. This reduction is notable
-when you have a large number of dags.
+when you have a large number of Dags.
-You can enable the source code to be stored in the database to make the Webserver completely independent of the DAG files.
+You can enable the source code to be stored in the database to make the Webserver completely independent of the Dag files.
This is not necessary if your files are embedded in the Docker image or you can otherwise provide
them to the Webserver. The data is stored in the :class:`~airflow.models.dagcode.DagCode` model.
@@ -60,7 +60,7 @@ To limit the excessive growth of the database, only the most recent entries are
are purged.
.. note::
- DAG Serialization is strictly required and can not be turned off from Airflow 2.0+.
+ Dag Serialization is strictly required and can not be turned off from Airflow 2.0+.
Dag Serialization Settings
@@ -79,14 +79,14 @@ Add the following settings in ``airflow.cfg``:
compress_serialized_dags = False
* ``min_serialized_dag_update_interval``: This flag sets the minimum interval (in seconds) after which
- the serialized dags in the DB should be updated. This helps in reducing database write rate.
-* ``min_serialized_dag_fetch_interval``: This option controls how often the Serialized DAG will be re-fetched
+ the serialized Dags in the DB should be updated. This helps in reducing database write rate.
+* ``min_serialized_dag_fetch_interval``: This option controls how often the Serialized Dag will be re-fetched
from the DB when it is already loaded in the DagBag in the Webserver. Setting this higher will reduce
- load on the DB, but at the expense of displaying a possibly stale cached version of the DAG.
+ load on the DB, but at the expense of displaying a possibly stale cached version of the Dag.
* ``max_num_rendered_ti_fields_per_task``: This option controls the maximum number of Rendered Task Instance
Fields (Template Fields) per task to store in the Database.
-* ``compress_serialized_dags``: This option controls whether to compress the Serialized DAG to the Database.
- It is useful when there are very large dags in your cluster. When ``True``, this will disable the DAG dependencies view.
+* ``compress_serialized_dags``: This option controls whether to compress the Serialized Dag to the Database.
+ It is useful when there are very large Dags in your cluster. When ``True``, this will disable the Dag dependencies view.
If you are updating Airflow from <1.10.7, please do not forget to run ``airflow db migrate``.
@@ -102,7 +102,7 @@ Limitations
.. note::
You need Airflow >= 1.10.10 for completely stateless Webserver.
- Airflow 1.10.7 to 1.10.9 needed access to DAG files in some cases.
+ Airflow 1.10.7 to 1.10.9 needed access to Dag files in some cases.
More Information: https://airflow.apache.org/docs/1.10.9/dag-serialization.html#limitations
Using a different JSON Library
@@ -123,10 +123,10 @@ configure local settings.
.. _dag-serialization-defaults:
-DAG Serialization with Default Values (Airflow 3.1+)
+Dag Serialization with Default Values (Airflow 3.1+)
------------------------------------------------------
-Starting with Airflow 3.1, DAG serialization establishes a versioned contract between Task SDKs
+Starting with Airflow 3.1, Dag serialization establishes a versioned contract between Task SDKs
and Airflow server components (Scheduler & API-Server). Combined with the Task Execution API, this
decouples client and server components, enabling independent deployments and upgrades while maintaining
backward compatibility and automatic default value resolution.
@@ -134,11 +134,11 @@ backward compatibility and automatic default value resolution.
How Default Values Work
~~~~~~~~~~~~~~~~~~~~~~~
-When Airflow processes DAGs, it applies default values in a specific order of precedence for the server:
+When Airflow processes Dags, it applies default values in a specific order of precedence for the server:
1. **Schema defaults**: Built-in Airflow defaults (lowest priority)
2. **Client defaults**: SDK-specific defaults
-3. **DAG default_args**: DAG-level settings (existing behavior)
+3. **Dag default_args**: Dag-level settings (existing behavior)
4. **Partial arguments**: MappedOperator shared values
5. **Task values**: Explicit task settings (highest priority)
@@ -148,7 +148,7 @@ more general ones.
JSON Structure
~~~~~~~~~~~~~~
-Serialized DAGs now include a ``client_defaults`` section that contains common default values:
+Serialized Dags now include a ``client_defaults`` section that contains common default values:
.. code-block:: json
@@ -195,7 +195,7 @@ MappedOperators (dynamic task mapping) also participate in the default value sys
.. code-block:: python
- # DAG Definition
+ # Dag Definition
BashOperator.partial(task_id="mapped_task", retries=2, owner="team_lead").expand(
bash_command=["echo 1", "echo 2", "echo 3"]
)
@@ -214,7 +214,7 @@ Independent Deployment Architecture
The serialization contract, combined with the Task Execution API, enables complete separation between:
- **Server Components** (Scheduler, API-Server): Handle orchestration, don't run user code
-- **Client Components** (Task SDK, DAG processing): Run user code in isolated environments
+- **Client Components** (Task SDK, Dag processor): Run user code in isolated environments
**Key Benefits:**
@@ -228,7 +228,7 @@ The serialization contract, combined with the Task Execution API, enables comple
Any Task SDK implementation must:
1. **Follow published schemas**:
- - DAG serialization: Produce JSON that validates against schema. Example: ``https://airflow.apache.org/schemas/dag-serialization/v2.json``
+ - Dag serialization: Produce JSON that validates against schema. Example: ``https://airflow.apache.org/schemas/dag-serialization/v2.json``
- Task execution: Support runtime communication via Execution API schema. Example: ``https://airflow.apache.org/schemas/execution-api/2025-05-20.json``
2. **Include client_defaults**: Optionally, provide SDK-specific defaults in the ``client_defaults.tasks`` section
3. **Use proper versioning**: Include ``__version`` field to indicate serialization format
@@ -236,7 +236,7 @@ Any Task SDK implementation must:
**Server Guarantees:**
As long as SDKs conform to both schema contracts, Airflow server components will:
-- Correctly deserialize DAGs from any compliant SDK
+- Correctly deserialize Dags from any compliant SDK
- Support task execution communication during runtime
- Apply appropriate default values according to the hierarchy
- Maintain compatibility across SDK versions and languages
diff --git a/airflow-core/docs/administration-and-deployment/index.rst b/airflow-core/docs/administration-and-deployment/index.rst
index 720f5f695eb..812ae95d8bc 100644
--- a/airflow-core/docs/administration-and-deployment/index.rst
+++ b/airflow-core/docs/administration-and-deployment/index.rst
@@ -18,7 +18,7 @@
Administration and Deployment
=====================================
-This section contains information about deploying dags into production and the administration of Airflow deployments.
+This section contains information about deploying Dags into production and the administration of Airflow deployments.
.. toctree::
:maxdepth: 2
diff --git a/airflow-core/docs/administration-and-deployment/listeners.rst b/airflow-core/docs/administration-and-deployment/listeners.rst
index 95eed49273d..75643f9e71d 100644
--- a/airflow-core/docs/administration-and-deployment/listeners.rst
+++ b/airflow-core/docs/administration-and-deployment/listeners.rst
@@ -113,7 +113,7 @@ Dag Import Error Events
- ``on_new_dag_import_error``
- ``on_existing_dag_import_error``
-Dag import error events occur when dag processor finds import error in the Dag code and update the metadata database table.
+Dag import error events occur when Dag processor finds import error in the Dag code and update the metadata database table.
|experimental|
@@ -131,7 +131,7 @@ Airflow defines the specification as `hookspec <https://github.com/apache/airflo
To include the listener in your Airflow installation, include it as a part of an :doc:`Airflow Plugin </administration-and-deployment/plugins>`.
-Listener API is meant to be called across all dags and all operators. You can't listen to events generated by specific dags. For that behavior, try methods like ``on_success_callback`` and ``pre_execute``. These provide callbacks for particular Dag authors or operator creators. The logs and ``print()`` calls will be handled as part of the listeners.
+Listener API is meant to be called across all Dags and all operators. You can't listen to events generated by specific Dags. For that behavior, try methods like ``on_success_callback`` and ``pre_execute``. These provide callbacks for particular Dag authors or operator creators. The logs and ``print()`` calls will be handled as part of the listeners.
Compatibility note
diff --git a/airflow-core/docs/administration-and-deployment/logging-monitoring/errors.rst b/airflow-core/docs/administration-and-deployment/logging-monitoring/errors.rst
index 0ad3fa8c512..f2f04405382 100644
--- a/airflow-core/docs/administration-and-deployment/logging-monitoring/errors.rst
+++ b/airflow-core/docs/administration-and-deployment/logging-monitoring/errors.rst
@@ -64,7 +64,7 @@ Tags
======================================= ==================================================
Name Description
======================================= ==================================================
-``dag_id`` Dag name of the dag that failed
+``dag_id`` Dag name of the Dag that failed
``task_id`` Task name of the task that failed
``data_interval_start`` Start of data interval when the task failed
``data_interval_end`` End of data interval when the task failed
@@ -79,7 +79,7 @@ in favor of ``data_interval_start``.
Breadcrumbs
------------
-When a task fails with an error `breadcrumbs <https://docs.sentry.io/platforms/python/enriching-events/breadcrumbs/>`__ will be added for the other tasks in the current DAG run.
+When a task fails with an error `breadcrumbs <https://docs.sentry.io/platforms/python/enriching-events/breadcrumbs/>`__ will be added for the other tasks in the current Dag run.
======================================= ==============================================================
Name Description
diff --git a/airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst b/airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst
index 56170c4c41f..187fc48dac4 100644
--- a/airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst
+++ b/airflow-core/docs/administration-and-deployment/logging-monitoring/metrics.rst
@@ -146,9 +146,9 @@ Name Descripti
``<job_name>_heartbeat_failure`` Number of failed Heartbeats for a ``<job_name>`` job, ex. ``SchedulerJob``,
``LocalTaskJob``
``local_task_job.task_exit.<job_id>.<dag_id>.<task_id>.<return_code>`` Number of ``LocalTaskJob`` terminations with a ``<return_code>``
- while running a task ``<task_id>`` of a DAG ``<dag_id>``.
+ while running a task ``<task_id>`` of a Dag ``<dag_id>``.
``local_task_job.task_exit`` Number of ``LocalTaskJob`` terminations with a ``<return_code>``
- while running a task ``<task_id>`` of a DAG ``<dag_id>``.
+ while running a task ``<task_id>`` of a Dag ``<dag_id>``.
Metric with job_id, dag_id, task_id and return_code tagging.
``operator_failures_<operator_name>`` Operator ``<operator_name>`` failures
``operator_failures`` Operator ``<operator_name>`` failures. Metric with operator_name tagging.
@@ -159,40 +159,40 @@ Name Descripti
``previously_succeeded`` Number of previously succeeded task instances. Metric with dag_id and task_id tagging.
``task_instances_without_heartbeats_killed`` Task instances without heartbeats killed. Metric with dag_id and task_id tagging.
``scheduler_heartbeat`` Scheduler heartbeats
-``dag_processor_heartbeat`` Standalone DAG processor heartbeats
-``dag_processing.processes`` Relative number of currently running DAG parsing processes (ie this delta
+``dag_processor_heartbeat`` Standalone Dag processor heartbeats
+``dag_processing.processes`` Relative number of currently running Dag parsing processes (ie this delta
is negative when, since the last metric was sent, processes have completed).
Metric with file_path and action tagging.
``dag_processing.processor_timeouts`` Number of file processors that have been killed due to taking too long.
Metric with file_path tagging.
``dag_processing.other_callback_count`` Number of non-SLA callbacks received
-``dag_processing.file_path_queue_update_count`` Number of times we've scanned the filesystem and queued all existing dags
+``dag_processing.file_path_queue_update_count`` Number of times we've scanned the filesystem and queued all existing Dags
``dag_file_processor_timeouts`` (DEPRECATED) same behavior as ``dag_processing.processor_timeouts``
``dag_processing.manager_stalls`` Number of stalled ``DagFileProcessorManager``
-``dag_file_refresh_error`` Number of failures loading any DAG files
+``dag_file_refresh_error`` Number of failures loading any Dag files
``scheduler.tasks.killed_externally`` Number of tasks killed externally. Metric with dag_id and task_id tagging.
``scheduler.orphaned_tasks.cleared`` Number of Orphaned tasks cleared by the Scheduler
``scheduler.orphaned_tasks.adopted`` Number of Orphaned tasks adopted by the Scheduler
``scheduler.critical_section_busy`` Count of times a scheduler process tried to get a lock on the critical
section (needed to send tasks to the executor) and found it locked by
another process.
-``ti.start.<dag_id>.<task_id>`` Number of started task in a given dag. Similar to <job_name>_start but for task
-``ti.start`` Number of started task in a given dag. Similar to <job_name>_start but for task.
+``ti.start.<dag_id>.<task_id>`` Number of started task in a given Dag. Similar to <job_name>_start but for task
+``ti.start`` Number of started task in a given Dag. Similar to <job_name>_start but for task.
Metric with dag_id and task_id tagging.
-``ti.finish.<dag_id>.<task_id>.<state>`` Number of completed task in a given dag. Similar to <job_name>_end but for task
-``ti.finish`` Number of completed task in a given dag. Similar to <job_name>_end but for task
+``ti.finish.<dag_id>.<task_id>.<state>`` Number of completed task in a given Dag. Similar to <job_name>_end but for task
+``ti.finish`` Number of completed task in a given Dag. Similar to <job_name>_end but for task
Metric with dag_id and task_id tagging.
-``dag.callback_exceptions`` Number of exceptions raised from DAG callbacks. When this happens, it
- means DAG callback is not working. Metric with dag_id tagging
+``dag.callback_exceptions`` Number of exceptions raised from Dag callbacks. When this happens, it
+ means Dag callback is not working. Metric with dag_id tagging
``celery.task_timeout_error`` Number of ``AirflowTaskTimeout`` errors raised when publishing Task to Celery Broker.
``celery.execute_command.failure`` Number of non-zero exit code from Celery task.
-``task_removed_from_dag.<dag_id>`` Number of tasks removed for a given dag (i.e. task no longer exists in DAG).
-``task_removed_from_dag`` Number of tasks removed for a given dag (i.e. task no longer exists in DAG).
+``task_removed_from_dag.<dag_id>`` Number of tasks removed for a given Dag (i.e. task no longer exists in Dag).
+``task_removed_from_dag`` Number of tasks removed for a given Dag (i.e. task no longer exists in Dag).
Metric with dag_id and run_type tagging.
-``task_restored_to_dag.<dag_id>`` Number of tasks restored for a given dag (i.e. task instance which was
- previously in REMOVED state in the DB is added to DAG file)
-``task_restored_to_dag.<dag_id>`` Number of tasks restored for a given dag (i.e. task instance which was
- previously in REMOVED state in the DB is added to DAG file).
+``task_restored_to_dag.<dag_id>`` Number of tasks restored for a given Dag (i.e. task instance which was
+ previously in REMOVED state in the DB is added to Dag file)
+``task_restored_to_dag.<dag_id>`` Number of tasks restored for a given Dag (i.e. task instance which was
+ previously in REMOVED state in the DB is added to Dag file).
Metric with dag_id and run_type tagging.
``task_instance_created_<operator_name>`` Number of tasks instances created for a given Operator
``task_instance_created`` Number of tasks instances created for a given Operator.
@@ -203,9 +203,9 @@ Name Descripti
``triggers.failed`` Number of triggers that errored before they could fire an event
``triggers.succeeded`` Number of triggers that have fired at least one event
``asset.updates`` Number of updated assets
-``asset.orphaned`` Number of assets marked as orphans because they are no longer referenced in DAG
+``asset.orphaned`` Number of assets marked as orphans because they are no longer referenced in Dag
schedule parameters or task outlets
-``asset.triggered_dagruns`` Number of DAG runs triggered by an asset update
+``asset.triggered_dagruns`` Number of Dag runs triggered by an asset update
====================================================================== ================================================================
Gauges
@@ -214,16 +214,16 @@ Gauges
==================================================== ========================================================================
Name Description
==================================================== ========================================================================
-``dagbag_size`` Number of dags found when the scheduler ran a scan based on its
+``dagbag_size`` Number of Dags found when the scheduler ran a scan based on its
configuration
-``dag_processing.import_errors`` Number of errors from trying to parse DAG files
-``dag_processing.total_parse_time`` Seconds taken to scan and import ``dag_processing.file_path_queue_size`` DAG files
-``dag_processing.file_path_queue_size`` Number of DAG files to be considered for the next scan
+``dag_processing.import_errors`` Number of errors from trying to parse Dag files
+``dag_processing.total_parse_time`` Seconds taken to scan and import ``dag_processing.file_path_queue_size`` Dag files
+``dag_processing.file_path_queue_size`` Number of Dag files to be considered for the next scan
``dag_processing.last_run.seconds_ago.<dag_file>`` Seconds since ``<dag_file>`` was last processed
``dag_processing.last_num_of_db_queries.<dag_file>`` Number of queries to Airflow database during parsing per ``<dag_file>``
``scheduler.tasks.starving`` Number of tasks that cannot be scheduled because of no open slot in pool
``scheduler.tasks.executable`` Number of tasks that are ready for execution (set to queued)
- with respect to pool limits, DAG concurrency, executor state,
+ with respect to pool limits, Dag concurrency, executor state,
and priority.
``executor.open_slots.<executor_class_name>`` Number of open slots on a specific executor. Only emitted when multiple executors are configured.
``executor.open_slots`` Number of open slots on executor
@@ -249,8 +249,8 @@ Name Description
``triggerer.capacity_left.<hostname>`` Capacity left on a triggerer to run triggers (described by hostname)
``triggerer.capacity_left`` Capacity left on a triggerer to run triggers (described by hostname).
Metric with hostname tagging.
-``ti.running.<queue>.<dag_id>.<task_id>`` Number of running tasks in a given dag. As ti.start and ti.finish can run out of sync this metric shows all running tis.
-``ti.running`` Number of running tasks in a given dag. As ti.start and ti.finish can run out of sync this metric shows all running tis.
+``ti.running.<queue>.<dag_id>.<task_id>`` Number of running tasks in a given Dag. As ti.start and ti.finish can run out of sync this metric shows all running tis.
+``ti.running`` Number of running tasks in a given Dag. As ti.start and ti.finish can run out of sync this metric shows all running tis.
Metric with queue, dag_id and task_id tagging.
==================================================== ========================================================================
@@ -260,8 +260,8 @@ Timers
================================================================ ========================================================================
Name Description
================================================================ ========================================================================
-``dagrun.dependency-check.<dag_id>`` Milliseconds taken to check DAG dependencies
-``dagrun.dependency-check`` Milliseconds taken to check DAG dependencies. Metric with dag_id tagging.
+``dagrun.dependency-check.<dag_id>`` Milliseconds taken to check Dag dependencies
+``dagrun.dependency-check`` Milliseconds taken to check Dag dependencies. Metric with dag_id tagging.
``dag.<dag_id>.<task_id>.duration`` Milliseconds taken to run a task
``task.duration`` Milliseconds taken to run a task. Metric with dag_id and task-id tagging.
``dag.<dag_id>.<task_id>.scheduled_duration`` Milliseconds a task spends in the Scheduled state, before being Queued
@@ -270,8 +270,8 @@ Name Description
``dag.<dag_id>.<task_id>.queued_duration`` Milliseconds a task spends in the Queued state, before being Running
``task.queued_duration`` Milliseconds a task spends in the Queued state, before being Running.
Metric with dag_id and task_id tagging.
-``dag_processing.last_duration.<dag_file>`` Milliseconds taken to load the given DAG file
-``dag_processing.last_duration`` Milliseconds taken to load the given DAG file. Metric with file_name tagging.
+``dag_processing.last_duration.<dag_file>`` Milliseconds taken to load the given Dag file
+``dag_processing.last_duration`` Milliseconds taken to load the given Dag file. Metric with file_name tagging.
``dagrun.duration.success.<dag_id>`` Milliseconds taken for a DagRun to reach success state
``dagrun.duration.success`` Milliseconds taken for a DagRun to reach success state.
Metric with dag_id and run_type tagging.
diff --git a/airflow-core/docs/administration-and-deployment/modules_management.rst b/airflow-core/docs/administration-and-deployment/modules_management.rst
index c593486ac3e..9ba4361e0ff 100644
--- a/airflow-core/docs/administration-and-deployment/modules_management.rst
+++ b/airflow-core/docs/administration-and-deployment/modules_management.rst
@@ -20,14 +20,14 @@
Modules Management
==================
-Airflow allows you to use your own Python modules in the DAG and in the
+Airflow allows you to use your own Python modules in the Dag and in the
Airflow configuration. The following article will describe how you can
create your own module so that Airflow can load it correctly, as well as
diagnose problems when modules are not loaded properly.
Often you want to use your own python code in your Airflow deployment,
-for example common code, libraries, you might want to generate dags using
-shared python code and have several DAG python files.
+for example common code, libraries, you might want to generate Dags using
+shared python code and have several Dag python files.
You can do it in one of those ways:
@@ -118,13 +118,13 @@ In the case above, these are the ways you could import the python files:
You can see the ``.airflowignore`` file at the root of your folder. This is a file that you can put in your
``dags`` folder to tell Airflow which files from the folder should be ignored when the Airflow
-scheduler looks for dags. It should contain either regular expressions (the default) or glob expressions
+scheduler looks for Dags. It should contain either regular expressions (the default) or glob expressions
for the paths that should be ignored. You do not need to have that file in any other folder in
-``PYTHONPATH`` (and also you can only keep shared code in the other folders, not the actual dags).
+``PYTHONPATH`` (and also you can only keep shared code in the other folders, not the actual Dags).
-In the example above the dags are only in ``my_custom_dags`` folder, the ``common_package`` should not be
-scanned by scheduler when searching for DAGS, so we should ignore ``common_package`` folder. You also
-want to ignore the ``base_dag.py`` if you keep a base DAG there that ``my_dag1.py`` and ``my_dag2.py`` derives
+In the example above the Dags are only in ``my_custom_dags`` folder, the ``common_package`` should not be
+scanned by scheduler when searching for Dags, so we should ignore ``common_package`` folder. You also
+want to ignore the ``base_dag.py`` if you keep a base Dag there that ``my_dag1.py`` and ``my_dag2.py`` derives
from. Your ``.airflowignore`` should look then like this (using the default ``glob`` syntax):
.. code-block:: none
@@ -142,14 +142,14 @@ Airflow, when running dynamically adds three directories to the ``sys.path``:
- The ``plugins`` Folder: It is configured with option ``plugins_folder`` in section ``[core]``.
.. note::
- The DAGS folder in Airflow 2 should not be shared with the webserver. While you can do it, unlike in Airflow 1.10,
- Airflow has no expectations that the DAGS folder is present in the webserver. In fact it's a bit of
- security risk to share the ``dags`` folder with the webserver, because it means that people who write DAGS
+ The Dags folder in Airflow 2 and 3 should not be shared with the webserver. While you can do it, unlike in Airflow 1.10,
+ Airflow has no expectations that the Dags folder is present in the webserver. In fact it's a bit of
+ security risk to share the ``dags`` folder with the webserver, because it means that people who write Dags
can write code that the webserver will be able to execute (ideally the webserver should
- never run code which can be modified by users who write dags). Therefore if you need to share some code
+ never run code which can be modified by users who write Dags). Therefore if you need to share some code
with the webserver, it is highly recommended that you share it via ``config`` or ``plugins`` folder or
via installed Airflow packages (see below). Those folders are usually managed and accessible by different
- users (Admins/DevOps) than DAG folders (those are usually data-scientists), so they are considered
+ users (Admins/DevOps) than Dag folders (those are usually data-scientists), so they are considered
as safe because they are part of configuration of the Airflow installation and controlled by the
people managing the installation.
@@ -180,7 +180,7 @@ as packages (i.e. folders with ``__init__.py``) or as modules (i.e. ``.py`` file
The same applies to ``config`` and ``plugins`` folders which are also at the ``PYTHONPATH`` and anything
you add to your ``PYTHONPATH`` manually (see details in the following chapters).
-It is recommended that you always put your dags / common files in a subpackage which is unique to your
+It is recommended that you always put your Dags / common files in a subpackage which is unique to your
deployment (``my_company`` in the example below). It is far too easy to use generic names for the
folders that will clash with other packages already present in the system. For example if you
create ``airflow/operators`` subfolder it will not be accessible because Airflow already has a package
@@ -197,7 +197,7 @@ This is tempting to do something like that it in ``my_dag1.py``:
from .base_dag import BaseDag # NEVER DO THAT!!!!
-You should import such shared DAG using full path (starting from the directory which is added to
+You should import such shared Dags using full path (starting from the directory which is added to
``PYTHONPATH``):
.. code-block:: python
@@ -205,9 +205,9 @@ You should import such shared DAG using full path (starting from the directory w
from my_company.my_custom_dags.base_dag import BaseDag # This is cool
The relative imports are counter-intuitive, and depending on how you start your python code, they can behave
-differently. In Airflow the same DAG file might be parsed in different contexts (by schedulers, by workers
+differently. In Airflow the same Dag file might be parsed in different contexts (by schedulers, by workers
or during tests) and in those cases, relative imports might behave differently. Always use full
-python package paths when you import anything in Airflow dags, this will save you a lot of troubles.
+python package paths when you import anything in Airflow Dags, this will save you a lot of troubles.
You can read more about relative import caveats in
`this Stack Overflow thread <https://stackoverflow.com/q/16981921/516701>`_.
@@ -337,7 +337,7 @@ Creating a package in Python
This is most organized way of adding your custom code. Thanks to using packages,
you might organize your versioning approach, control which versions of the shared code are installed
and deploy the code to all your instances and containers in controlled way - all by system admins/DevOps
-rather than by the DAG writers. It is usually suitable when you have a separate team that manages this
+rather than by the Dag authors. It is usually suitable when you have a separate team that manages this
shared code, but if you know your python ways you can also distribute your code this way in smaller
deployments. You can also install your :doc:`../administration-and-deployment/plugins` and :doc:`apache-airflow-providers:index` as python
packages, so learning how to build your package is handy.
diff --git a/airflow-core/docs/administration-and-deployment/plugins.rst b/airflow-core/docs/administration-and-deployment/plugins.rst
index 2a74ca54d20..a93eb48ffdb 100644
--- a/airflow-core/docs/administration-and-deployment/plugins.rst
+++ b/airflow-core/docs/administration-and-deployment/plugins.rst
@@ -83,7 +83,7 @@ need to restart the worker (if using CeleryExecutor) or scheduler (LocalExecutor
option is you can accept the speed hit at start up set the ``core.execute_tasks_new_python_interpreter``
config setting to True, resulting in launching a whole new python interpreter for tasks.
-(Modules only imported by DAG files on the other hand do not suffer this problem, as DAG files are not
+(Modules only imported by Dag files on the other hand do not suffer this problem, as Dag files are not
loaded/parsed in any long-running Airflow process.)
.. _plugins-interface:
@@ -135,7 +135,7 @@ looks like:
# buttons.
operator_extra_links = []
- # A list of timetable classes to register so they can be used in dags.
+ # A list of timetable classes to register so they can be used in Dags.
timetables = []
# A list of Listeners that plugin provides. Listeners can register to
diff --git a/airflow-core/docs/administration-and-deployment/priority-weight.rst b/airflow-core/docs/administration-and-deployment/priority-weight.rst
index e0a2d43baec..3e8ed11cd4c 100644
--- a/airflow-core/docs/administration-and-deployment/priority-weight.rst
+++ b/airflow-core/docs/administration-and-deployment/priority-weight.rst
@@ -37,8 +37,8 @@ Below are the weighting methods. By default, Airflow's weighting method is ``dow
downstream descendants. As a result, upstream tasks will have
higher weight and will be scheduled more aggressively when
using positive weight values. This is useful when you have
- multiple DAG run instances and desire to have all upstream
- tasks to complete for all runs before each DAG can continue
+ multiple Dag run instances and desire to have all upstream
+ tasks to complete for all runs before each Dag can continue
processing downstream tasks.
.. grid-item-card:: ``upstream``
@@ -47,9 +47,9 @@ Below are the weighting methods. By default, Airflow's weighting method is ``dow
The effective weight is the aggregate sum of all upstream ancestors.
This is the opposite where downstream tasks have higher weight
and will be scheduled more aggressively when using positive weight
- values. This is useful when you have multiple DAG run instances
- and prefer to have each DAG complete before starting upstream
- tasks of other DAG runs.
+ values. This is useful when you have multiple Dag run instances
+ and prefer to have each Dag complete before starting upstream
+ tasks of other Dag runs.
.. grid-item-card:: ``absolute``
:shadow: none
@@ -59,7 +59,7 @@ Below are the weighting methods. By default, Airflow's weighting method is ``dow
know exactly what priority weight each task should have.
Additionally, when set to ``absolute``, there is bonus effect of
significantly speeding up the task creation process as for very
- large dags.
+ large Dags.
The ``priority_weight`` parameter can be used in conjunction with :ref:`concepts:pool`.
@@ -96,7 +96,7 @@ To check if the custom priority weight strategy is already available in Airflow,
:end-before: [END example_custom_weight_dag]
-After the DAG is running, you can check the ``priority_weight`` parameter on the task to verify that it is using
+After the Dag is running, you can check the ``priority_weight`` parameter on the task to verify that it is using
the custom priority strategy rule.
|experimental|
diff --git a/airflow-core/docs/administration-and-deployment/production-deployment.rst b/airflow-core/docs/administration-and-deployment/production-deployment.rst
index be16876d6cc..78d0e0966bc 100644
--- a/airflow-core/docs/administration-and-deployment/production-deployment.rst
+++ b/airflow-core/docs/administration-and-deployment/production-deployment.rst
@@ -18,7 +18,7 @@
Production Deployment
^^^^^^^^^^^^^^^^^^^^^
-It is time to deploy your DAG in production. To do this, first, you need to make sure that the Airflow
+It is time to deploy your Dag in production. To do this, first, you need to make sure that the Airflow
is itself production-ready. Let's see what precautions you need to take.
Database backend
@@ -62,9 +62,9 @@ the :doc:`Celery executor <apache-airflow-providers-celery:celery_executor>`.
Once you have configured the executor, it is necessary to make sure that every node in the cluster contains
-the same configuration and dags. Airflow sends simple instructions such as "execute task X of DAG Y", but
-does not send any DAG files or configuration. You can use a simple cronjob or any other mechanism to sync
-dags and configs across your nodes, e.g., checkout dags from git repo every 5 minutes on all nodes.
+the same configuration and Dags. Airflow sends simple instructions such as "execute task X of Dag Y", but
+does not send any Dag files or configuration. You can use a simple cronjob or any other mechanism to sync
+Dags and configs across your nodes, e.g., checkout Dags from git repo every 5 minutes on all nodes.
Logging
@@ -152,18 +152,18 @@ When it comes to live-upgrading the ``Webserver``, ``Triggerer`` components, if
environments and have more than one instances for each of them, you can rolling-restart them one by one,
without any downtime. This should usually be done as the first step in your upgrade procedure.
-When you are running a deployment with separate ``DAG processor``, in a
-:ref:`Separate DAG processing deployment <overview-separate-dag-processing-airflow-architecture>`
-the ``DAG processor`` is not horizontally scaled - even if you have more of them there is usually one
-``DAG processor`` running at a time per specific folder, so you can just stop it and start the new one -
-but since the ``DAG processor`` is not a critical component, it's ok for it to experience a short downtime.
+When you are running a deployment with separate ``Dag processor``, in a
+:ref:`Separate Dag processing deployment <overview-separate-dag-processing-airflow-architecture>`
+the ``Dag processor`` is not horizontally scaled - even if you have more of them there is usually one
+``Dag processor`` running at a time per specific folder, so you can just stop it and start the new one -
+but since the ``Dag processor`` is not a critical component, it's ok for it to experience a short downtime.
When it comes to upgrading the schedulers and workers, you can use the live upgrade capabilities
of the executor you use:
* For the :doc:`Local executor <../core-concepts/executor/local>` your tasks are running as subprocesses of
scheduler and you cannot upgrade the Scheduler without killing the tasks run by it. You can either
- pause all your dags and wait for the running tasks to complete or just stop the scheduler and kill all
+ pause all your Dags and wait for the running tasks to complete or just stop the scheduler and kill all
the tasks it runs - then you will need to clear and restart those tasks manually after the upgrade
is completed (or rely on ``retry`` being set for stopped tasks).
diff --git a/airflow-core/docs/administration-and-deployment/scheduler.rst b/airflow-core/docs/administration-and-deployment/scheduler.rst
index b5384b60fac..582a4ab36e8 100644
--- a/airflow-core/docs/administration-and-deployment/scheduler.rst
+++ b/airflow-core/docs/administration-and-deployment/scheduler.rst
@@ -20,11 +20,11 @@
Scheduler
==========
-The Airflow scheduler monitors all tasks and dags, then triggers the
+The Airflow scheduler monitors all tasks and Dags, then triggers the
task instances once their dependencies are complete. Behind the scenes,
the scheduler spins up a subprocess, which monitors and stays in sync with all
-dags in the specified DAG directory. Once per minute, by default, the scheduler
-collects DAG parsing results and checks whether any active tasks can be triggered.
+Dags in the specified Dag directory. Once per minute, by default, the scheduler
+collects Dag parsing results and checks whether any active tasks can be triggered.
The Airflow scheduler is designed to run as a persistent service in an
Airflow production environment. To kick it off, all you need to do is
@@ -39,25 +39,25 @@ To start a scheduler, simply run the command:
airflow scheduler
-Your dags will start executing once the scheduler is running successfully.
+Your Dags will start executing once the scheduler is running successfully.
.. note::
- The first DAG Run is created based on the minimum ``start_date`` for the tasks in your DAG.
- Subsequent DAG Runs are created according to your DAG's :doc:`timetable <../authoring-and-scheduling/timetable>`.
+ The first Dag Run is created based on the minimum ``start_date`` for the tasks in your Dag.
+ Subsequent Dag Runs are created according to your Dag's :doc:`timetable <../authoring-and-scheduling/timetable>`.
-For dags with a cron or timedelta schedule, scheduler won't trigger your tasks until the period it covers has ended e.g., A job with ``schedule`` set as ``@daily`` runs after the day
-has ended. This technique makes sure that whatever data is required for that period is fully available before the DAG is executed.
+For Dags with a cron or timedelta schedule, scheduler won't trigger your tasks until the period it covers has ended e.g., A job with ``schedule`` set as ``@daily`` runs after the day
+has ended. This technique makes sure that whatever data is required for that period is fully available before the Dag is executed.
In the UI, it appears as if Airflow is running your tasks a day **late**
.. note::
- If you run a DAG on a ``schedule`` of one day, the run with data interval starting on ``2019-11-21`` triggers after ``2019-11-21T23:59``.
+ If you run a Dag on a ``schedule`` of one day, the run with data interval starting on ``2019-11-21`` triggers after ``2019-11-21T23:59``.
**Let's Repeat That**, the scheduler runs your job one ``schedule`` AFTER the start date, at the END of the interval.
- You should refer to :doc:`../core-concepts/dag-run` for details on scheduling a DAG.
+ You should refer to :doc:`../core-concepts/dag-run` for details on scheduling a Dag.
.. note::
The scheduler is designed for high throughput. This is an informed design decision to achieve scheduling
@@ -85,10 +85,10 @@ not using direct communication or consensus algorithm between schedulers (Raft,
consensus tool (Apache Zookeeper, or Consul for instance) we have kept the "operational surface area" to a
minimum.
-The scheduler now uses the serialized DAG representation to make its scheduling decisions and the rough
+The scheduler now uses the serialized Dag representation to make its scheduling decisions and the rough
outline of the scheduling loop is:
-- Check for any dags needing a new DagRun, and create them
+- Check for any Dags needing a new DagRun, and create them
- Examine a batch of DagRuns for schedulable TaskInstances or complete DagRuns
- Select schedulable TaskInstances, and whilst respecting Pool limits and other concurrency limits, enqueue
them for execution
@@ -146,14 +146,14 @@ In order to fine-tune your scheduler, you need to include a number of factors:
* how much CPU you have available
* how much networking throughput you have available
-* The logic and definition of your DAG structure:
- * how many dags you have
+* The logic and definition of your Dag structure:
+ * how many Dags you have
* how complex they are (i.e. how many tasks and dependencies they have)
* The scheduler configuration
* How many schedulers you have
* How many task instances scheduler processes in one loop
- * How many new DAG runs should be created/scheduled per loop
+ * How many new Dag runs should be created/scheduled per loop
* How often the scheduler should perform cleanup and check for orphaned tasks/adopting them
In order to perform fine-tuning, it's good to understand how Scheduler works under-the-hood.
@@ -164,7 +164,7 @@ How to approach Scheduler's fine-tuning
"""""""""""""""""""""""""""""""""""""""
Airflow gives you a lot of "knobs" to turn to fine tune the performance but it's a separate task,
-depending on your particular deployment, your DAG structure, hardware availability and expectations,
+depending on your particular deployment, your Dag structure, hardware availability and expectations,
to decide which knobs to turn to get best effect for you. Part of the job when managing the
deployment is to decide what you are going to optimize for.
@@ -190,7 +190,7 @@ What resources might limit Scheduler's performance
There are several areas of resource usage that you should pay attention to:
* Database connections and Database usage might become a problem as you want to increase performance and
- process more things in parallel. Airflow is known for being "database-connection hungry" - the more dags
+ process more things in parallel. Airflow is known for being "database-connection hungry" - the more Dags
you have and the more you want to process in parallel, the more database connections will be opened.
This is generally not a problem for MySQL as its model of handling connections is thread-based, but this
might be a problem for Postgres, where connection handling is process-based. It is a general consensus
@@ -233,18 +233,18 @@ However, you can also look at other non-performance-related scheduler configurat
- :ref:`config:scheduler__max_dagruns_to_create_per_loop`
- This changes the number of dags that are locked by each scheduler when
- creating DAG runs. One possible reason for setting this lower is if you
- have huge dags (in the order of 10k+ tasks per DAG) and are running multiple schedulers, you won't want one
+ This changes the number of Dags that are locked by each scheduler when
+ creating Dag runs. One possible reason for setting this lower is if you
+ have huge Dags (in the order of 10k+ tasks per Dag) and are running multiple schedulers, you won't want one
scheduler to do all the work.
- :ref:`config:scheduler__max_dagruns_per_loop_to_schedule`
How many DagRuns should a scheduler examine (and lock) when scheduling
and queuing tasks. Increasing this limit will allow more throughput for
- smaller dags but will likely slow down throughput for larger (>500
- tasks for example) dags. Setting this too high when using multiple
- schedulers could also lead to one scheduler taking all the DAG runs
+ smaller Dags but will likely slow down throughput for larger (>500
+ tasks for example) Dags. Setting this too high when using multiple
+ schedulers could also lead to one scheduler taking all the Dag runs
leaving no work for the others.
- :ref:`config:scheduler__use_row_level_locking`
diff --git a/airflow-core/docs/authoring-and-scheduling/asset-scheduling.rst b/airflow-core/docs/authoring-and-scheduling/asset-scheduling.rst
index 6042b46ac0d..51ae64c62c7 100644
--- a/airflow-core/docs/authoring-and-scheduling/asset-scheduling.rst
+++ b/airflow-core/docs/authoring-and-scheduling/asset-scheduling.rst
@@ -23,7 +23,7 @@ Asset-Aware Scheduling
Quickstart
----------
-In addition to scheduling dags based on time, you can also schedule dags to run based on when a task updates an asset.
+In addition to scheduling Dags based on time, you can also schedule Dags to run based on when a task updates an asset.
.. code-block:: python
@@ -38,7 +38,7 @@ In addition to scheduling dags based on time, you can also schedule dags to run
with DAG(
- # this DAG should be run when example.csv is updated (by dag1)
+ # this Dag should be run when example.csv is updated (by dag1)
schedule=[Asset("s3://asset-bucket/example.csv")],
...,
):
@@ -51,10 +51,13 @@ In addition to scheduling dags based on time, you can also schedule dags to run
:ref:`asset_definitions` for how to declare assets.
-Schedule dags with assets
+Schedule Dags with assets
-------------------------
-You can use assets to specify data dependencies in your dags. The following example shows how after the ``producer`` task in the ``producer`` DAG successfully completes, Airflow schedules the ``consumer`` DAG. Airflow marks an asset as ``updated`` only if the task completes successfully. If the task fails or if it is skipped, no update occurs, and Airflow doesn't schedule the ``consumer`` DAG.
+You can use assets to specify data dependencies in your Dags. The following example shows how after the ``producer`` task
+in the ``producer`` Dag successfully completes, Airflow schedules the ``consumer`` Dag. Airflow marks an asset as ``updated``
+only if the task completes successfully. If the task fails or if it is skipped, no update occurs, and Airflow doesn't
+schedule the ``consumer`` Dag.
.. code-block:: python
@@ -67,12 +70,13 @@ You can use assets to specify data dependencies in your dags. The following exam
...
-You can find a listing of the relationships between assets and dags in the :ref:`Asset Views <ui-asset-views>`.
+You can find a listing of the relationships between assets and Dags in the :ref:`Asset Views <ui-asset-views>`.
Multiple assets
-----------------
-Because the ``schedule`` parameter is a list, dags can require multiple assets. Airflow schedules a DAG after **all** assets the DAG consumes have been updated at least once since the last time the DAG ran:
+Because the ``schedule`` parameter is a list, Dags can require multiple assets. Airflow schedules a Dag after **all** assets
+the Dag consumes have been updated at least once since the last time the Dag ran:
.. code-block:: python
@@ -88,7 +92,7 @@ Because the ``schedule`` parameter is a list, dags can require multiple assets.
...
-If one asset is updated multiple times before all consumed assets update, the downstream DAG still only runs once, as shown in this illustration:
+If one asset is updated multiple times before all consumed assets update, the downstream Dag still only runs once, as shown in this illustration:
.. ::
ASCII art representation of this diagram
@@ -96,7 +100,7 @@ If one asset is updated multiple times before all consumed assets update, the do
example_asset_1 x----x---x---x----------------------x-
example_asset_2 -------x---x-------x------x----x------
example_asset_3 ---------------x-----x------x---------
- DAG runs created * *
+ Dag runs created * *
.. graphviz::
@@ -146,7 +150,7 @@ If one asset is updated multiple times before all consumed assets update, the do
Fetching information from a triggering asset event
----------------------------------------------------
-A triggered DAG can fetch information from the asset that triggered it using the ``triggering_asset_events`` template or parameter. See more at :ref:`templates-ref`.
+A triggered Dag can fetch information from the asset that triggered it using the ``triggering_asset_events`` template or parameter. See more at :ref:`templates-ref`.
Example:
@@ -179,14 +183,16 @@ Example:
print_triggering_asset_events()
-Note that this example is using `(.values() | first | first) <https://jinja.palletsprojects.com/en/3.1.x/templates/#jinja-filters.first>`_ to fetch the first of one asset given to the DAG, and the first of one AssetEvent for that asset. An implementation can be quite complex if you have multiple assets, potentially with multiple AssetEvents.
+Note that this example is using `(.values() | first | first) <https://jinja.palletsprojects.com/en/3.1.x/templates/#jinja-filters.first>`_ to
+fetch the first of one asset given to the Dag, and the first of one AssetEvent for that asset. An implementation can be quite complex if you
+have multiple assets, potentially with multiple AssetEvents.
Event-driven scheduling
-------------------------
-Asset-based scheduling triggered by other DAGs (internal event-driven scheduling) does not cover all use cases.
-Often you also need to trigger DAGs from **external events** such as system signals, messages, or real-time data changes.
+Asset-based scheduling triggered by other Dags (internal event-driven scheduling) does not cover all use cases.
+Often you also need to trigger Dags from **external events** such as system signals, messages, or real-time data changes.
Airflow supports two main approaches for external event-driven scheduling:
Push-based scheduling (REST API)
@@ -195,7 +201,9 @@ Push-based scheduling (REST API)
.. versionadded:: 2.9
External systems can **push asset events into Airflow** using the REST API.
-For example, the DAG ``waiting_for_asset_1_and_2`` will be triggered when tasks update both assets "asset-1" and "asset-2". Once "asset-1" is updated, Airflow creates a record. This ensures that Airflow knows to trigger the DAG when "asset-2" is updated. These records are called *queued asset events*.
+For example, the Dag ``waiting_for_asset_1_and_2`` will be triggered when tasks update both assets "asset-1" and "asset-2".
+Once "asset-1" is updated, Airflow creates a record. This ensures that Airflow knows to trigger the Dag when "asset-2" is updated.
+These records are called *queued asset events*.
.. code-block:: python
@@ -209,10 +217,10 @@ For example, the DAG ``waiting_for_asset_1_and_2`` will be triggered when tasks
``queuedEvent`` API endpoints are available to manipulate these records:
-* Get a queued asset event for a DAG: ``/assets/queuedEvent/{uri}``
-* Get queued asset events for a DAG: ``/dags/{dag_id}/assets/queuedEvent``
-* Delete a queued asset event for a DAG: ``/assets/queuedEvent/{uri}``
-* Delete queued asset events for a DAG: ``/dags/{dag_id}/assets/queuedEvent``
+* Get a queued asset event for a Dag: ``/assets/queuedEvent/{uri}``
+* Get queued asset events for a Dag: ``/dags/{dag_id}/assets/queuedEvent``
+* Delete a queued asset event for a Dag: ``/assets/queuedEvent/{uri}``
+* Delete queued asset events for a Dag: ``/dags/{dag_id}/assets/queuedEvent``
* Get queued asset events for an asset: ``/dags/{dag_id}/assets/queuedEvent/{uri}``
* Delete queued asset events for an asset: ``DELETE /dags/{dag_id}/assets/queuedEvent/{uri}``
@@ -225,7 +233,7 @@ Instead of relying on external systems to push events, Airflow can **pull** from
This is done using ``AssetWatcher`` classes and triggers compatible with event-driven scheduling.
* An ``AssetWatcher`` monitors an external source such as a queue or storage system.
-* When a relevant event occurs, it updates the corresponding asset and triggers DAG execution.
+* When a relevant event occurs, it updates the corresponding asset and triggers Dag execution.
* Only triggers that inherit from ``BaseEventTrigger`` are compatible, to avoid infinite rescheduling scenarios.
For more details and examples, see :doc:`event-scheduling`.
@@ -233,15 +241,17 @@ For more details and examples, see :doc:`event-scheduling`.
Advanced asset scheduling with conditional expressions
--------------------------------------------------------
-Apache Airflow includes advanced scheduling capabilities that use conditional expressions with assets. This feature allows you to define complex dependencies for DAG executions based on asset updates, using logical operators for more control on workflow triggers.
+Apache Airflow includes advanced scheduling capabilities that use conditional expressions with assets.
+This feature allows you to define complex dependencies for Dag executions based on asset updates, using logical
+operators for more control on workflow triggers.
Logical operators for assets
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Airflow supports two logical operators for combining asset conditions:
-- **AND (``&``)**: Specifies that the DAG should be triggered only after all of the specified assets have been updated.
-- **OR (``|``)**: Specifies that the DAG should be triggered when any of the specified assets is updated.
+- **AND (``&``)**: Specifies that the Dag should be triggered only after all of the specified assets have been updated.
+- **OR (``|``)**: Specifies that the Dag should be triggered when any of the specified assets is updated.
These operators enable you to configure your Airflow workflows to use more complex asset update conditions, making them more dynamic and flexible.
@@ -250,7 +260,7 @@ Example Use
**Scheduling based on multiple asset updates**
-To schedule a DAG to run only when two specific assets have both been updated, use the AND operator (``&``):
+To schedule a Dag to run only when two specific assets have both been updated, use the AND operator (``&``):
.. code-block:: python
@@ -266,7 +276,7 @@ To schedule a DAG to run only when two specific assets have both been updated, u
**Scheduling based on any asset update**
-To trigger a DAG execution when either one of two assets is updated, apply the OR operator (``|``):
+To trigger a Dag execution when either one of two assets is updated, apply the OR operator (``|``):
.. code-block:: python
@@ -279,7 +289,7 @@ To trigger a DAG execution when either one of two assets is updated, apply the O
**Complex Conditional Logic**
-For scenarios requiring more intricate conditions, such as triggering a DAG when one asset is updated or when both of two other assets are updated, combine the OR and AND operators:
+For scenarios requiring more intricate conditions, such as triggering a Dag when one asset is updated or when both of two other assets are updated, combine the OR and AND operators:
.. code-block:: python
@@ -295,9 +305,9 @@ For scenarios requiring more intricate conditions, such as triggering a DAG when
Scheduling based on asset aliases
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Since asset events added to an alias are just simple asset events, a downstream DAG depending on the actual asset can read asset events of it normally, without considering the associated aliases. A downstream DAG can also depend on an asset alias. The authoring syntax is referencing the ``AssetAlias`` by name, and the associated asset events are picked up for scheduling. Note that a DAG can be triggered by a task with ``outlets=AssetAlias("xxx")`` if and only if the alias is resolved int [...]
+Since asset events added to an alias are just simple asset events, a downstream Dag depending on the actual asset can read asset events of it normally, without considering the associated aliases. A downstream Dag can also depend on an asset alias. The authoring syntax is referencing the ``AssetAlias`` by name, and the associated asset events are picked up for scheduling. Note that a Dag can be triggered by a task with ``outlets=AssetAlias("xxx")`` if and only if the alias is resolved int [...]
-The asset alias is resolved to the assets during DAG parsing. Thus, if the "min_file_process_interval" configuration is set to a high value, there is a possibility that the asset alias may not be resolved. To resolve this issue, you can trigger DAG parsing.
+The asset alias is resolved to the assets during Dag parsing. Thus, if the "min_file_process_interval" configuration is set to a high value, there is a possibility that the asset alias may not be resolved. To resolve this issue, you can trigger Dag parsing.
.. code-block:: python
@@ -322,7 +332,11 @@ The asset alias is resolved to the assets during DAG parsing. Thus, if the "min_
...
-In the example provided, once the DAG ``asset-alias-producer`` is executed, the asset alias ``AssetAlias("example-alias")`` will be resolved to ``Asset("s3://bucket/my-task")``. However, the DAG ``asset-alias-consumer`` will have to wait for the next DAG re-parsing to update its schedule. To address this, Airflow will re-parse the dags relying on the asset alias ``AssetAlias("example-alias")`` when it's resolved into assets that these dags did not previously depend on. As a result, both [...]
+In the example provided, once the Dag ``asset-alias-producer`` is executed, the asset alias ``AssetAlias("example-alias")`` will be
+resolved to ``Asset("s3://bucket/my-task")``. However, the Dag ``asset-alias-consumer`` will have to wait for the next Dag re-parsing
+to update its schedule. To address this, Airflow will re-parse the Dags relying on the asset alias ``AssetAlias("example-alias")``
+when it's resolved into assets that these Dags did not previously depend on. As a result, both the "asset-consumer" and
+"asset-alias-consumer" Dags will be triggered after the execution of Dag ``asset-alias-producer``.
Combining asset and time-based schedules
@@ -330,6 +344,6 @@ Combining asset and time-based schedules
AssetTimetable Integration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-You can schedule dags based on both asset events and time-based schedules using ``AssetOrTimeSchedule``. This allows you to create workflows when a DAG needs both to be triggered by data updates and run periodically according to a fixed timetable.
+You can schedule Dags based on both asset events and time-based schedules using ``AssetOrTimeSchedule``. This allows you to create workflows when a Dag needs both to be triggered by data updates and run periodically according to a fixed timetable.
For more detailed information on ``AssetOrTimeSchedule``, refer to the corresponding section in :ref:`AssetOrTimeSchedule <asset-timetable-section>`.
diff --git a/airflow-core/docs/authoring-and-scheduling/assets.rst b/airflow-core/docs/authoring-and-scheduling/assets.rst
index 394991e2029..e6980589997 100644
--- a/airflow-core/docs/authoring-and-scheduling/assets.rst
+++ b/airflow-core/docs/authoring-and-scheduling/assets.rst
@@ -29,7 +29,7 @@ Asset Definitions
What is an "Asset"?
--------------------
-An Airflow asset is a logical grouping of data. Upstream producer tasks can update assets, and asset updates contribute to scheduling downstream consumer dags.
+An Airflow asset is a logical grouping of data. Upstream producer tasks can update assets, and asset updates contribute to scheduling downstream consumer Dags.
`Uniform Resource Identifier (URI) <https://en.wikipedia.org/wiki/Uniform_Resource_Identifier>`_ define assets:
@@ -147,7 +147,7 @@ Once an asset is defined, tasks can be created to emit events against it by spec
with DAG(dag_id="example_asset", schedule="@daily"):
PythonOperator(task_id="example_asset", outlets=[example_asset], python_callable=_write_example_asset)
-This is quite a lot of boilerplate. Airflow provides a shorthand for this simple but most common case of *creating a DAG with one single task that emits events of one asset*. The code block below is exactly equivalent to the one above:
+This is quite a lot of boilerplate. Airflow provides a shorthand for this simple but most common case of *creating a Dag with one single task that emits events of one asset*. The code block below is exactly equivalent to the one above:
.. code-block:: python
@@ -222,7 +222,7 @@ Each value in the ``inlet_events`` mapping is a sequence-like object that orders
Dependency between ``@asset``, ``@task``, and classic operators
---------------------------------------------------------------
-Since an ``@asset`` is simply a wrapper around a dag with a task and an asset, it is quite easy to read and ``@asset`` in a ``@task`` or a classic operator. For example, the above ``post_process_s3_file`` can also be written as a task (inside a dag, omitted here for brevity):
+Since an ``@asset`` is simply a wrapper around a Dag with a task and an asset, it is quite easy to read and ``@asset`` in a ``@task`` or a classic operator. For example, the above ``post_process_s3_file`` can also be written as a task (inside a Dag, omitted here for brevity):
.. code-block:: python
@@ -300,7 +300,7 @@ The shorthand for this is ``@asset.multi``:
Dynamic data events emitting and asset creation through AssetAlias
-----------------------------------------------------------------------
-An asset alias can be used to emit asset events of assets with association to the aliases. Downstreams can depend on resolved asset. This feature allows you to define complex dependencies for DAG executions based on asset updates.
+An asset alias can be used to emit asset events of assets with association to the aliases. Downstreams can depend on resolved asset. This feature allows you to define complex dependencies for Dag executions based on asset updates.
How to use AssetAlias
~~~~~~~~~~~~~~~~~~~~~~~
diff --git a/airflow-core/docs/authoring-and-scheduling/cron.rst b/airflow-core/docs/authoring-and-scheduling/cron.rst
index 4f594bfd256..5683daeedf9 100644
--- a/airflow-core/docs/authoring-and-scheduling/cron.rst
+++ b/airflow-core/docs/authoring-and-scheduling/cron.rst
@@ -18,7 +18,7 @@
Cron & Time Intervals
======================
-You may set your DAG to run on a simple schedule by setting its ``schedule`` argument to either a
+You may set your Dag to run on a simple schedule by setting its ``schedule`` argument to either a
`cron expression <https://en.wikipedia.org/wiki/Cron#CRON_expression>`_, a ``datetime.timedelta`` object,
or one of the :ref:`cron-presets`.
@@ -39,7 +39,7 @@ Cron Presets
''''''''''''
For more elaborate scheduling requirements, you can implement a :doc:`custom timetable <../authoring-and-scheduling/timetable>`.
Note that Airflow parses cron expressions with the croniter library which supports an extended syntax for cron strings. See their documentation `in github <https://github.com/kiorky/croniter>`_.
-For example, you can create a DAG schedule to run at 12AM on the first Monday of the month with their extended cron syntax: ``0 0 * * MON#1``.
+For example, you can create a Dag schedule to run at 12AM on the first Monday of the month with their extended cron syntax: ``0 0 * * MON#1``.
.. tip::
You can use an online editor for CRON expressions such as `Crontab guru <https://crontab.guru/>`_
@@ -47,7 +47,7 @@ For example, you can create a DAG schedule to run at 12AM on the first Monday of
+----------------+--------------------------------------------------------------------+-----------------+
| preset | meaning | cron |
+================+====================================================================+=================+
-| ``None`` | Don't schedule, use for exclusively "externally triggered" dags | |
+| ``None`` | Don't schedule, use for exclusively "externally triggered" Dags | |
+----------------+--------------------------------------------------------------------+-----------------+
| ``@once`` | Schedule once and only once | |
+----------------+--------------------------------------------------------------------+-----------------+
@@ -66,5 +66,5 @@ For example, you can create a DAG schedule to run at 12AM on the first Monday of
| ``@yearly`` | Run once a year at midnight (24:00) of January 1 | ``0 0 1 1 *`` |
+----------------+--------------------------------------------------------------------+-----------------+
-Your DAG will be instantiated for each schedule along with a corresponding
-DAG Run entry in the database backend.
+Your Dag will be instantiated for each schedule along with a corresponding
+Dag Run entry in the database backend.
diff --git a/airflow-core/docs/authoring-and-scheduling/deferring.rst b/airflow-core/docs/authoring-and-scheduling/deferring.rst
index c2922a1ca4c..68e57a3c4d2 100644
--- a/airflow-core/docs/authoring-and-scheduling/deferring.rst
+++ b/airflow-core/docs/authoring-and-scheduling/deferring.rst
@@ -18,7 +18,7 @@
Deferrable Operators & Triggers
===============================
-Standard :doc:`Operators </core-concepts/operators>` and :doc:`Sensors <../core-concepts/sensors>` take up a full *worker slot* for the entire time they are running, even if they are idle. For example, if you only have 100 worker slots available to run tasks, and you have 100 dags waiting on a sensor that's currently running but idle, then you *cannot run anything else* - even though your entire Airflow cluster is essentially idle. ``reschedule`` mode for sensors solves some of this, by [...]
+Standard :doc:`Operators </core-concepts/operators>` and :doc:`Sensors <../core-concepts/sensors>` take up a full *worker slot* for the entire time they are running, even if they are idle. For example, if you only have 100 worker slots available to run tasks, and you have 100 Dags waiting on a sensor that's currently running but idle, then you *cannot run anything else* - even though your entire Airflow cluster is essentially idle. ``reschedule`` mode for sensors solves some of this, by [...]
This is where *Deferrable Operators* can be used. When it has nothing to do but wait, an operator can suspend itself and free up the worker for other processes by *deferring*. When an operator defers, execution moves to the triggerer, where the trigger specified by the operator will run. The trigger can do the polling or waiting required by the operator. Then, when the trigger finishes polling or waiting, it sends a signal for the operator to resume its execution. During the deferred ph [...]
@@ -39,11 +39,11 @@ Using Deferrable Operators
If you want to use pre-written deferrable operators that come with Airflow, such as ``TimeSensor``, then you only need to complete two steps:
* Ensure your Airflow installation runs at least one ``triggerer`` process, as well as the normal ``scheduler``
-* Use deferrable operators/sensors in your dags
+* Use deferrable operators/sensors in your Dags
Airflow automatically handles and implements the deferral processes for you.
-If you're upgrading existing dags to use deferrable operators, Airflow contains API-compatible sensor variants. Add these variants into your dag to use deferrable operators with no other changes required.
+If you're upgrading existing Dags to use deferrable operators, Airflow contains API-compatible sensor variants. Add these variants into your Dag to use deferrable operators with no other changes required.
Note that you can't use the deferral ability from inside custom PythonOperator or TaskFlow Python functions. Deferral is only available to traditional, class-based operators.
@@ -146,7 +146,7 @@ There's some design constraints to be aware of when writing your own trigger:
* If your trigger is designed to emit more than one event (not currently supported), then each emitted event *must* contain a payload that can be used to deduplicate events if the trigger is running in multiple places. If you only fire one event and don't need to pass information back to the operator, you can just set the payload to ``None``.
* A trigger can suddenly be removed from one triggerer service and started on a new one. For example, if subnets are changed and a network partition results or if there is a deployment. If desired, you can implement the ``cleanup`` method, which is always called after ``run``, whether the trigger exits cleanly or otherwise.
* In order for any changes to a trigger to be reflected, the *triggerer* needs to be restarted whenever the trigger is modified.
-* Your trigger must not come from a dag bundle - anywhere else on ``sys.path`` is fine. The triggerer does not initialize any bundles when running a trigger.
+* Your trigger must not come from a Dag bundle - anywhere else on ``sys.path`` is fine. The triggerer does not initialize any bundles when running a trigger.
.. note::
@@ -441,7 +441,7 @@ Triggers can have two options: they can either send execution back to the worker
In the above example, the trigger will end the task instance directly if ``end_from_trigger`` is set to ``True`` by yielding ``TaskSuccessEvent``. Otherwise, it will resume the task instance with the method specified in the operator.
.. note::
- Exiting from the trigger works only when listeners are not integrated for the deferrable operator. Currently, when deferrable operator has the ``end_from_trigger`` attribute set to ``True`` and listeners are integrated it raises an exception during parsing to indicate this limitation. While writing the custom trigger, ensure that the trigger is not set to end the task instance directly if the listeners are added from plugins. If the ``end_from_trigger`` attribute is changed to differ [...]
+ Exiting from the trigger works only when listeners are not integrated for the deferrable operator. Currently, when deferrable operator has the ``end_from_trigger`` attribute set to ``True`` and listeners are integrated it raises an exception during parsing to indicate this limitation. While writing the custom trigger, ensure that the trigger is not set to end the task instance directly if the listeners are added from plugins. If the ``end_from_trigger`` attribute is changed to differ [...]
High Availability
diff --git a/airflow-core/docs/authoring-and-scheduling/dynamic-task-mapping.rst b/airflow-core/docs/authoring-and-scheduling/dynamic-task-mapping.rst
index e254bf52071..c6bcc894696 100644
--- a/airflow-core/docs/authoring-and-scheduling/dynamic-task-mapping.rst
+++ b/airflow-core/docs/authoring-and-scheduling/dynamic-task-mapping.rst
@@ -23,14 +23,14 @@ Dynamic Task Mapping
Dynamic Task Mapping allows a way for a workflow to create a number of tasks at runtime based upon current data, rather than the Dag author having to know in advance how many tasks would be needed.
-This is similar to defining your tasks in a for loop, but instead of having the DAG file fetch the data and do that itself, the scheduler can do this based on the output of a previous task. Right before a mapped task is executed the scheduler will create *n* copies of the task, one for each input.
+This is similar to defining your tasks in a for loop, but instead of having the Dag file fetch the data and do that itself, the scheduler can do this based on the output of a previous task. Right before a mapped task is executed the scheduler will create *n* copies of the task, one for each input.
It is also possible to have a task operate on the collected output of a mapped task, commonly known as map and reduce.
Simple mapping
==============
-In its simplest form you can map over a list defined directly in your DAG file using the ``expand()`` function instead of calling your task directly.
+In its simplest form you can map over a list defined directly in your Dag file using the ``expand()`` function instead of calling your task directly.
If you want to see a simple usage of Dynamic Task Mapping, you can look below:
@@ -39,7 +39,7 @@ If you want to see a simple usage of Dynamic Task Mapping, you can look below:
This will show ``Total was 9`` in the task logs when executed.
-This is the resulting DAG structure:
+This is the resulting Dag structure:
.. image:: /img/ui-light/mapping_simple_graph.png
@@ -88,7 +88,7 @@ The grid view also provides visibility into your mapped tasks in the details pan
Task-generated Mapping
----------------------
-The above examples we've shown could all be achieved with a ``for`` loop in the DAG file, but the real power of dynamic task mapping comes from being able to have a task generate the list to iterate over.
+The above examples we've shown could all be achieved with a ``for`` loop in the Dag file, but the real power of dynamic task mapping comes from being able to have a task generate the list to iterate over.
.. code-block:: python
@@ -112,7 +112,7 @@ The ``make_list`` task runs as a normal task and must return a list or dict (see
.. warning:: Task-generated mapping cannot be utilized with ``TriggerRule.ALWAYS``
Assigning ``trigger_rule=TriggerRule.ALWAYS`` in task-generated mapping is not allowed, as expanded parameters are undefined with the task's immediate execution.
- This is enforced at the time of the DAG parsing, for both tasks and mapped tasks groups, and will raise an error if you try to use it.
+ This is enforced at the time of the Dag parsing, for both tasks and mapped tasks groups, and will raise an error if you try to use it.
In the recent example, setting ``trigger_rule=TriggerRule.ALWAYS`` in the ``consumer`` task will raise an error since ``make_list`` is a task-generated mapping.
Repeated mapping
@@ -528,7 +528,7 @@ Since it is common to want to transform the output data format for task mapping,
There are a couple of things to note:
-#. The callable argument of :func:`map()` (``create_copy_kwargs`` in the example) **must not** be a task, but a plain Python function. The transformation is as a part of the "pre-processing" of the downstream task (i.e. ``copy_files``), not a standalone task in the DAG.
+#. The callable argument of :func:`map()` (``create_copy_kwargs`` in the example) **must not** be a task, but a plain Python function. The transformation is as a part of the "pre-processing" of the downstream task (i.e. ``copy_files``), not a standalone task in the Dag.
#. The callable always take exactly one positional argument. This function is called for each item in the iterable used for task-mapping, similar to how Python's built-in :func:`map()` works.
#. Since the callable is executed as a part of the downstream task, you can use any existing techniques to write the task function. To mark a component as skipped, for example, you should raise ``AirflowSkipException``. Note that returning ``None`` **does not** work here.
@@ -589,7 +589,7 @@ Another common pattern to combine input sources is to run the same task against
download_file.override(task_id="download_file_a").expand(filename=list_filenames_a.output)
download_file.override(task_id="download_file_b").expand(filename=list_filenames_b.output)
-The DAG, however, would be both more scalable and easier to inspect if the tasks can be combined into one. This can done with ``concat``:
+The Dag, however, would be both more scalable and easier to inspect if the tasks can be combined into one. This can done with ``concat``:
.. code-block:: python
diff --git a/airflow-core/docs/authoring-and-scheduling/event-scheduling.rst b/airflow-core/docs/authoring-and-scheduling/event-scheduling.rst
index 8a81f01461c..4cd72edc63b 100644
--- a/airflow-core/docs/authoring-and-scheduling/event-scheduling.rst
+++ b/airflow-core/docs/authoring-and-scheduling/event-scheduling.rst
@@ -20,13 +20,13 @@ Event-driven scheduling
.. versionadded:: 3.0
-Apache Airflow allows for event-driven scheduling, enabling dags to be triggered based on external events rather than
+Apache Airflow allows for event-driven scheduling, enabling Dags to be triggered based on external events rather than
predefined time-based schedules.
This is particularly useful in modern data architectures where workflows need to react to real-time data changes,
messages, or system signals.
-By using assets, as described in :doc:`asset-scheduling`, you can configure dags to start execution when specific external events
-occur. Assets provide a mechanism to establish dependencies between external events and DAG execution, ensuring that
+By using assets, as described in :doc:`asset-scheduling`, you can configure Dags to start execution when specific external events
+occur. Assets provide a mechanism to establish dependencies between external events and Dag execution, ensuring that
workflows react dynamically to changes in the external environment.
The ``AssetWatcher`` class plays a crucial role in this mechanism. It monitors an external event source, such as a
@@ -41,10 +41,10 @@ Supported triggers for event-driven scheduling
Not all :doc:`triggers <deferring>` in Airflow can be used for event-driven scheduling. As opposed to all triggers that
inherit from ``BaseTrigger``, only a subset that inherit from ``BaseEventTrigger`` are compatible.
The reason for this restriction is that some triggers are not designed for event-driven scheduling, and using them to
-schedule dags could lead to unintended results.
+schedule Dags could lead to unintended results.
``BaseEventTrigger`` ensures that triggers used for scheduling adhere to an event-driven paradigm, reacting appropriately
-to external event changes without causing unexpected DAG behavior.
+to external event changes without causing unexpected Dag behavior.
Writing event-driven compatible triggers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -77,19 +77,19 @@ for an external resource to reach a given state. Examples:
Scheduling under such conditions can lead to infinite rescheduling. This is because once the condition becomes true,
it is likely to remain true for an extended period.
-For example, consider a DAG scheduled to run when a specific job reaches a "success" state.
-Once the job succeeds, it will typically remain in that state. As a result, the DAG will be triggered repeatedly every
+For example, consider a Dag scheduled to run when a specific job reaches a "success" state.
+Once the job succeeds, it will typically remain in that state. As a result, the Dag will be triggered repeatedly every
time the triggerer checks the condition.
Another example is the ``S3KeyTrigger``, which checks for the presence of a specific file in an S3 bucket.
Once the file is created, the trigger will continue to succeed on every check, since the condition
"is file X present in bucket Y" remains true.
-This leads to the DAG being triggered indefinitely every time the trigger mechanism runs.
+This leads to the Dag being triggered indefinitely every time the trigger mechanism runs.
When creating custom triggers, be cautious about using conditions that remain permanently true once met.
-This can unintentionally result in infinite DAG executions and overwhelm your system.
+This can unintentionally result in infinite Dag executions and overwhelm your system.
-Use cases for event-driven dags
+Use cases for event-driven Dags
-------------------------------
* **Data ingestion pipelines**: Trigger ETL workflows when new data arrives in a storage system.
diff --git a/airflow-core/docs/authoring-and-scheduling/index.rst b/airflow-core/docs/authoring-and-scheduling/index.rst
index beefa12e5cc..cbcac076063 100644
--- a/airflow-core/docs/authoring-and-scheduling/index.rst
+++ b/airflow-core/docs/authoring-and-scheduling/index.rst
@@ -18,7 +18,7 @@
Authoring and Scheduling
=========================
-Here you can find detailed documentation about advanced authoring and scheduling Airflow dags.
+Here you can find detailed documentation about advanced authoring and scheduling Airflow Dags.
It's recommended that you first review the pages in :doc:`core concepts </core-concepts/index>`
.. _authoring-section:
diff --git a/airflow-core/docs/authoring-and-scheduling/serializers.rst b/airflow-core/docs/authoring-and-scheduling/serializers.rst
index a8447671f60..433485bb3b8 100644
--- a/airflow-core/docs/authoring-and-scheduling/serializers.rst
+++ b/airflow-core/docs/authoring-and-scheduling/serializers.rst
@@ -20,7 +20,7 @@ Serialization
To support data exchange, like arguments, between tasks, Airflow needs to serialize the data to be exchanged and
deserialize it again when required in a downstream task. Serialization also happens so that the webserver and
-the scheduler (as opposed to the DAG processor) do not need to read the DAG file. This is done for security purposes
+the scheduler (as opposed to the Dag processor) do not need to read the Dag file. This is done for security purposes
and efficiency.
Serialization is a surprisingly hard job. Python out of the box only has support for serialization of primitives,
diff --git a/airflow-core/docs/authoring-and-scheduling/timetable.rst b/airflow-core/docs/authoring-and-scheduling/timetable.rst
index 163366b3bab..e92fc66130e 100644
--- a/airflow-core/docs/authoring-and-scheduling/timetable.rst
+++ b/airflow-core/docs/authoring-and-scheduling/timetable.rst
@@ -19,9 +19,9 @@
Timetables
==========
-For a DAG with a time-based schedule (as opposed to event-driven), the DAG's internal "timetable"
+For a Dag with a time-based schedule (as opposed to event-driven), the Dag's internal "timetable"
drives scheduling. The timetable also determines the data interval and the logical date of
-each run created for the DAG.
+each run created for the Dag.
Dags scheduled with a cron expression or ``timedelta`` object are
internally converted to always use a timetable.
@@ -29,7 +29,7 @@ internally converted to always use a timetable.
If a cron expression or ``timedelta`` is sufficient for your use case, you don't need
to worry about writing a custom timetable because Airflow has default timetables that handle those cases.
But for more complicated scheduling requirements,
-you can create your own timetable class and pass that to the DAG's ``schedule`` argument.
+you can create your own timetable class and pass that to the Dags ``schedule`` argument.
Some examples of when custom timetable implementations are useful:
@@ -49,7 +49,7 @@ Some examples of when custom timetable implementations are useful:
.. _`Traditional Chinese Calendar`: https://en.wikipedia.org/wiki/Chinese_calendar
Airflow allows you to write custom timetables in plugins and used by
-dags. You can find an example demonstrating a custom timetable in the
+Dags. You can find an example demonstrating a custom timetable in the
:doc:`/howto/timetable` how-to guide.
.. note::
@@ -70,7 +70,7 @@ DeltaTriggerTimetable
^^^^^^^^^^^^^^^^^^^^^
A timetable that accepts a :class:`datetime.timedelta` or ``dateutil.relativedelta.relativedelta``, and runs
-the DAG once a delta passes.
+the Dag once a delta passes.
.. seealso:: `Differences between "trigger" and "data interval" timetables`_
@@ -87,7 +87,7 @@ the DAG once a delta passes.
You can also provide a static data interval to the timetable. The optional ``interval`` argument also
should be a :class:`datetime.timedelta` or ``dateutil.relativedelta.relativedelta``. When using these
-arguments, a triggered DAG run's data interval spans the specified duration, and *ends* with the trigger time.
+arguments, a triggered Dag run's data interval spans the specified duration, and *ends* with the trigger time.
.. code-block:: python
@@ -116,7 +116,7 @@ arguments, a triggered DAG run's data interval spans the specified duration, and
CronTriggerTimetable
^^^^^^^^^^^^^^^^^^^^
-A timetable that accepts a cron expression, and triggers DAG runs according to it.
+A timetable that accepts a cron expression, and triggers Dag runs according to it.
.. seealso:: `Differences between "trigger" and "data interval" timetables`_
@@ -130,7 +130,7 @@ A timetable that accepts a cron expression, and triggers DAG runs according to i
pass
You can also provide a static data interval to the timetable. The optional ``interval`` argument
-must be a :class:`datetime.timedelta` or ``dateutil.relativedelta.relativedelta``. When using these arguments, a triggered DAG run's data interval spans the specified duration, and *ends* with the trigger time.
+must be a :class:`datetime.timedelta` or ``dateutil.relativedelta.relativedelta``. When using these arguments, a triggered Dag run's data interval spans the specified duration, and *ends* with the trigger time.
.. code-block:: python
@@ -157,7 +157,7 @@ must be a :class:`datetime.timedelta` or ``dateutil.relativedelta.relativedelta`
MultipleCronTriggerTimetable
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-This is similar to CronTriggerTimetable_ except it takes multiple cron expressions. A DAG run is scheduled whenever any of the expressions matches the time. It is particularly useful when the desired schedule cannot be expressed by one single cron expression.
+This is similar to CronTriggerTimetable_ except it takes multiple cron expressions. A Dag run is scheduled whenever any of the expressions matches the time. It is particularly useful when the desired schedule cannot be expressed by one single cron expression.
.. code-block:: python
@@ -197,7 +197,7 @@ DeltaDataIntervalTimetable
^^^^^^^^^^^^^^^^^^^^^^^^^^
A timetable that schedules data intervals with a time delta. You can select it by providing a
-:class:`datetime.timedelta` or ``dateutil.relativedelta.relativedelta`` to the ``schedule`` parameter of a DAG.
+:class:`datetime.timedelta` or ``dateutil.relativedelta.relativedelta`` to the ``schedule`` parameter of a Dag.
This timetable focuses on the data interval value and does not necessarily align execution dates with
arbitrary bounds, such as the start of day or of hour.
@@ -216,13 +216,13 @@ CronDataIntervalTimetable
^^^^^^^^^^^^^^^^^^^^^^^^^
A timetable that accepts a cron expression, creates data intervals according to the interval between each cron
-trigger points, and triggers a DAG run at the end of each data interval.
+trigger points, and triggers a Dag run at the end of each data interval.
.. seealso:: `Differences between "trigger" and "data interval" timetables`_
.. seealso:: `Differences between the cron and delta data interval timetables`_
Select this timetable by providing a valid cron expression as a string to the ``schedule``
-parameter of a DAG, as described in the :doc:`../core-concepts/dags` documentation.
+parameter of a Dag, as described in the :doc:`../core-concepts/dags` documentation.
.. code-block:: python
@@ -233,11 +233,11 @@ parameter of a DAG, as described in the :doc:`../core-concepts/dags` documentati
EventsTimetable
^^^^^^^^^^^^^^^
-Pass a list of ``datetime``\s for the DAG to run after. This can be useful for timing based on sporting
+Pass a list of ``datetime``\s for the Dag to run after. This can be useful for timing based on sporting
events, planned communication campaigns, and other schedules that are arbitrary and irregular, but predictable.
-The list of events must be finite and of reasonable size as it must be loaded every time the DAG is parsed. Optionally, use
-the ``restrict_to_events`` flag to force manual runs of the DAG that use the time of the most recent, or very
+The list of events must be finite and of reasonable size as it must be loaded every time the Dag is parsed. Optionally, use
+the ``restrict_to_events`` flag to force manual runs of the Dag that use the time of the most recent, or very
first, event for the data interval. Otherwise, manual runs begin with a ``data_interval_start`` and
``data_interval_end`` equal to the time at which the manual run started. You can also name the set of events using the
``description`` parameter, which will be displayed in the Airflow UI.
@@ -268,11 +268,11 @@ Asset event based scheduling with time based scheduling
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Combining conditional asset expressions with time-based schedules enhances scheduling flexibility.
-The ``AssetOrTimeSchedule`` is a specialized timetable that allows for the scheduling of dags based on both time-based schedules and asset events. It also facilitates the creation of both scheduled runs, as per traditional timetables, and asset-triggered runs, which operate independently.
+The ``AssetOrTimeSchedule`` is a specialized timetable that allows for the scheduling of Dags based on both time-based schedules and asset events. It also facilitates the creation of both scheduled runs, as per traditional timetables, and asset-triggered runs, which operate independently.
-This feature is particularly useful in scenarios where a DAG needs to run on asset updates and also at periodic intervals. It ensures that the workflow remains responsive to data changes and consistently runs regular checks or updates.
+This feature is particularly useful in scenarios where a Dag needs to run on asset updates and also at periodic intervals. It ensures that the workflow remains responsive to data changes and consistently runs regular checks or updates.
-Here's an example of a DAG using ``AssetOrTimeSchedule``:
+Here's an example of a Dag using ``AssetOrTimeSchedule``:
.. code-block:: python
@@ -287,7 +287,7 @@ Here's an example of a DAG using ``AssetOrTimeSchedule``:
# Additional arguments here, replace this comment with actual arguments
)
def example_dag():
- # DAG tasks go here
+ # Dag tasks go here
pass
@@ -312,48 +312,48 @@ Whether taking care of *Data Interval*
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
A trigger timetable *does not* include *data interval*. This means that the value of ``data_interval_start``
-and ``data_interval_end`` are the same; the time when a DAG run is triggered.
+and ``data_interval_end`` are the same; the time when a Dag run is triggered.
For a data interval timetable, the value of ``data_interval_start`` and ``data_interval_end`` are different.
-``data_interval_end`` is the time when a DAG run is triggered, while ``data_interval_start`` is the start of the interval.
+``data_interval_end`` is the time when a Dag run is triggered, while ``data_interval_start`` is the start of the interval.
*Catchup* behavior
^^^^^^^^^^^^^^^^^^
-By default, ``catchup`` is set to ``False``. This prevents running unnecessary dags in the following scenarios:
+By default, ``catchup`` is set to ``False``. This prevents running unnecessary Dags in the following scenarios:
-- If you create a new DAG with a start date in the past, and don't want to run dags for the past. If ``catchup`` is ``True``, Airflow runs all dags that would have run in that time interval.
-- If you pause an existing DAG, and then restart it at a later date, ``catchup`` being ``False`` means that Airflow does not run the dags that would have run during the paused period.
+- If you create a new Dag with a start date in the past, and don't want to run Dags for the past. If ``catchup`` is ``True``, Airflow runs all Dags that would have run in that time interval.
+- If you pause an existing Dag, and then restart it at a later date, ``catchup`` being ``False`` means that Airflow does not run the Dags that would have run during the paused period.
In these scenarios, the ``logical_date`` in the ``run_id`` are based on how how the timetable handles the data
interval.
You can change the default ``catchup`` behavior using the Airflow config ``[scheduler] catchup_by_default``.
-See :ref:`dag-catchup` for more information about how DAG runs are triggered when using ``catchup``.
+See :ref:`dag-catchup` for more information about how Dag runs are triggered when using ``catchup``.
.. _timetables_run_id_logical_date:
-The time when a DAG run is triggered
+The time when a Dag run is triggered
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Both trigger and data interval timetables trigger DAG runs at the same time. However, the timestamp for the
+Both trigger and data interval timetables trigger Dag runs at the same time. However, the timestamp for the
``run_id`` is different for each. This is because ``run_id`` is based on ``logical_date``.
-For example, suppose there is a cron expression ``@daily`` or ``0 0 * * *``, which is scheduled to run at 12AM every day. If you enable dags using the two timetables at 3PM on January
+For example, suppose there is a cron expression ``@daily`` or ``0 0 * * *``, which is scheduled to run at 12AM every day. If you enable Dags using the two timetables at 3PM on January
31st,
-- `CronTriggerTimetable`_ creates a new DAG run at 12AM on February 1st. The ``run_id`` timestamp is midnight, on February 1st.
-- `CronDataIntervalTimetable`_ immediately creates a new DAG run, because a DAG run for the daily time interval beginning at 12AM on January 31st did not occur yet. The ``run_id`` timestamp is midnight, on January 31st, since that is the beginning of the data interval.
+- `CronTriggerTimetable`_ creates a new Dag run at 12AM on February 1st. The ``run_id`` timestamp is midnight, on February 1st.
+- `CronDataIntervalTimetable`_ immediately creates a new Dag run, because a Dag run for the daily time interval beginning at 12AM on January 31st did not occur yet. The ``run_id`` timestamp is midnight, on January 31st, since that is the beginning of the data interval.
-The following is another example showing the difference in the case of skipping DAG runs:
+The following is another example showing the difference in the case of skipping Dag runs:
-Suppose there are two running dags with a cron expression ``@daily`` or ``0 0 * * *`` that use the two different timetables. If you pause the dags at 3PM on January 31st and re-enable them at 3PM on February 2nd,
+Suppose there are two running Dags with a cron expression ``@daily`` or ``0 0 * * *`` that use the two different timetables. If you pause the Dags at 3PM on January 31st and re-enable them at 3PM on February 2nd,
-- `CronTriggerTimetable`_ skips the DAG runs that were supposed to trigger on February 1st and 2nd. The next DAG run will be triggered at 12AM on February 3rd.
-- `CronDataIntervalTimetable`_ skips the DAG runs that were supposed to trigger on February 1st only. A DAG run for February 2nd is immediately triggered after you re-enable the DAG.
+- `CronTriggerTimetable`_ skips the Dag runs that were supposed to trigger on February 1st and 2nd. The next Dag run will be triggered at 12AM on February 3rd.
+- `CronDataIntervalTimetable`_ skips the Dag runs that were supposed to trigger on February 1st only. A Dag run for February 2nd is immediately triggered after you re-enable the Dag.
-In these examples, you see how a trigger timetable creates DAG runs more intuitively and similar to what
+In these examples, you see how a trigger timetable creates Dag runs more intuitively and similar to what
people expect a workflow to behave, while a data interval timetable is designed heavily around the data
interval it processes, and does not reflect a workflow's own properties.
@@ -364,7 +364,7 @@ Differences between the cron and delta data interval timetables
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Choosing between `DeltaDataIntervalTimetable`_ and `CronDataIntervalTimetable`_ depends on your use case.
-If you enable a DAG at 01:05 on February 1st, the following table summarizes the DAG runs created and the
+If you enable a Dag at 01:05 on February 1st, the following table summarizes the Dag runs created and the
data interval that they cover, depending on 3 arguments: ``schedule``, ``start_date`` and ``catchup``.
.. list-table::
diff --git a/airflow-core/docs/authoring-and-scheduling/timezone.rst b/airflow-core/docs/authoring-and-scheduling/timezone.rst
index 20b3219472e..ef887731626 100644
--- a/airflow-core/docs/authoring-and-scheduling/timezone.rst
+++ b/airflow-core/docs/authoring-and-scheduling/timezone.rst
@@ -21,9 +21,9 @@ Time Zones
==========
Support for time zones is enabled by default. Airflow stores datetime information in UTC internally and in the database.
-It allows you to run your dags with time zone dependent schedules. At the moment, Airflow does not convert them to the
+It allows you to run your Dags with time zone dependent schedules. At the moment, Airflow does not convert them to the
end user's time zone in the user interface. It will always be displayed in UTC there. Also, templates used in Operators
-are not converted. Time zone information is exposed and it is up to the writer of DAG to decide what do with it.
+are not converted. Time zone information is exposed and it is up to the writer of Dag to decide what do with it.
This is handy if your users live in more than one time zone and you want to display datetime information according to
each user's wall clock.
@@ -33,7 +33,7 @@ Even if you are running Airflow in only one time zone, it is still good practice
that many countries use Daylight Saving Time (DST), where clocks are moved forward in spring and backward
in autumn. If you're working in local time, you're likely to encounter errors twice a year, when the transitions
happen. (The pendulum and pytz documentation discuss these issues in greater detail.) This probably doesn't matter
-for a simple DAG, but it's a problem if you are in, for example, financial services where you have end of day
+for a simple Dag, but it's a problem if you are in, for example, financial services where you have end of day
deadlines to meet.
The time zone is set in ``airflow.cfg``. By default it is set to UTC, but you change it to use the system's settings or
@@ -85,7 +85,7 @@ Interpretation of naive datetime objects
''''''''''''''''''''''''''''''''''''''''
Although Airflow operates fully time zone aware, it still accepts naive date time objects for ``start_dates``
-and ``end_dates`` in your DAG definitions. This is mostly in order to preserve backwards compatibility. In
+and ``end_dates`` in your Dag definitions. This is mostly in order to preserve backwards compatibility. In
case a naive ``start_date`` or ``end_date`` is encountered the default time zone is applied. It is applied
in such a way that it is assumed that the naive date time is already in the default time zone. In other
words if you have a default time zone setting of ``Europe/Amsterdam`` and create a naive datetime ``start_date`` of
@@ -105,7 +105,7 @@ Unfortunately, during DST transitions, some datetimes don't exist or are ambiguo
In such situations, pendulum raises an exception. That's why you should always create aware
datetime objects when time zone support is enabled.
-In practice, this is rarely an issue. Airflow gives you time zone aware datetime objects in the models and dags, and most often,
+In practice, this is rarely an issue. Airflow gives you time zone aware datetime objects in the models and Dags, and most often,
new datetime objects are created from existing ones through timedelta arithmetic. The only datetime that's often
created in application code is the current time, and ``timezone.utcnow()`` automatically does the right thing.
@@ -129,13 +129,13 @@ it is therefore important to make sure this setting is equal on all Airflow node
.. _timezone_aware_dags:
-Time zone aware dags
+Time zone aware Dags
--------------------
-Creating a time zone aware DAG is quite simple. Just make sure to supply a time zone aware ``start_date``
+Creating a time zone aware Dag is quite simple. Just make sure to supply a time zone aware ``start_date``
using ``pendulum``. Don't try to use standard library
`timezone <https://docs.python.org/3/library/datetime.html#timezone-objects>`_ as they are known to
-have limitations and we deliberately disallow using them in dags.
+have limitations and we deliberately disallow using them in Dags.
.. code-block:: python
@@ -146,24 +146,24 @@ have limitations and we deliberately disallow using them in dags.
print(dag.timezone) # <Timezone [Europe/Amsterdam]>
Please note that while it is possible to set a ``start_date`` and ``end_date``
-for Tasks, the DAG timezone or global timezone (in that order) will always be
+for Tasks, the Dag timezone or global timezone (in that order) will always be
used to calculate data intervals. Upon first encounter, the start date or end
date will be converted to UTC using the timezone associated with ``start_date``
or ``end_date``, then for calculations this timezone information will be
disregarded.
.. note::
- When authoring a Timezone aware DAG you must make sure that the underlying timezone library (for example: pendulum)
+ When authoring a Timezone aware Dag you must make sure that the underlying timezone library (for example: pendulum)
is updated with recent changes to regulations (daylight saving changes etc...). When a change in time
is expected you should verify with the underlying timezone library that the switch will happen as expected.
- There might be a need to update the library version. As a general recommendation if you can author dags in UTC
+ There might be a need to update the library version. As a general recommendation if you can author Dags in UTC
that is preferred.
Templates
'''''''''
Airflow returns time zone aware datetimes in templates, but does not convert them to local time so they remain in UTC.
-It is left up to the DAG to handle this.
+It is left up to the Dag to handle this.
.. code-block:: python
@@ -175,17 +175,17 @@ It is left up to the DAG to handle this.
Cron schedules
''''''''''''''
-Time zone aware dags that use cron schedules respect daylight savings
-time. For example, a DAG with a start date in the ``US/Eastern`` time zone
+Time zone aware Dags that use cron schedules respect daylight savings
+time. For example, a Dag with a start date in the ``US/Eastern`` time zone
with a schedule of ``0 0 * * *`` will run daily at 04:00 UTC during
daylight savings time and at 05:00 otherwise.
Time deltas
'''''''''''
-Time zone aware dags that use ``timedelta`` or ``relativedelta`` schedules
+Time zone aware Dags that use ``timedelta`` or ``relativedelta`` schedules
respect daylight savings time for the start date but do not adjust for
daylight savings time when scheduling subsequent runs. For example, a
-DAG with a start date of ``pendulum.datetime(2020, 1, 1, tz="UTC")``
+Dag with a start date of ``pendulum.datetime(2020, 1, 1, tz="UTC")``
and a schedule interval of ``timedelta(days=1)`` will run daily at 05:00
UTC regardless of daylight savings time.
diff --git a/airflow-core/docs/best-practices.rst b/airflow-core/docs/best-practices.rst
index 839c95dbfe9..be79bd7e465 100644
--- a/airflow-core/docs/best-practices.rst
+++ b/airflow-core/docs/best-practices.rst
@@ -20,21 +20,21 @@
Best Practices
==============
-Creating a new DAG is a three-step process:
+Creating a new Dag is a three-step process:
-- writing Python code to create a DAG object,
+- writing Python code to create a Dag object,
- testing if the code meets your expectations,
-- configuring environment dependencies to run your DAG
+- configuring environment dependencies to run your Dag
This tutorial will introduce you to the best practices for these three steps.
.. _best_practice:writing_a_dag:
-Writing a DAG
+Writing a Dag
^^^^^^^^^^^^^^
-Creating a new DAG in Airflow is quite simple. However, there are many things that you need to take care of
-to ensure the DAG run or failure does not produce unexpected results.
+Creating a new Dag in Airflow is quite simple. However, there are many things that you need to take care of
+to ensure the Dag run or failure does not produce unexpected results.
Creating a Custom Operator/Hook
-------------------------------
@@ -75,17 +75,17 @@ result -
Deleting a task
----------------
-Be careful when deleting a task from a DAG. You would not be able to see the Task in Graph View, Grid View, etc making
-it difficult to check the logs of that Task from the Webserver. If that is not desired, please create a new DAG.
+Be careful when deleting a task from a Dag. You would not be able to see the Task in Graph View, Grid View, etc making
+it difficult to check the logs of that Task from the Webserver. If that is not desired, please create a new Dag.
Communication
--------------
-Airflow executes tasks of a DAG on different servers in case you are using :doc:`Kubernetes executor <apache-airflow-providers-cncf-kubernetes:kubernetes_executor>` or :doc:`Celery executor <apache-airflow-providers-celery:celery_executor>`.
+Airflow executes tasks of a Dag on different servers in case you are using :doc:`Kubernetes executor <apache-airflow-providers-cncf-kubernetes:kubernetes_executor>` or :doc:`Celery executor <apache-airflow-providers-celery:celery_executor>`.
Therefore, you should not store any file or config in the local filesystem as the next task is likely to run on a different server without access to it — for example, a task that downloads the data file that the next task processes.
In the case of :class:`Local executor <airflow.executors.local_executor.LocalExecutor>`,
-storing a file on disk can make retries harder e.g., your task requires a config file that is deleted by another task in DAG.
+storing a file on disk can make retries harder e.g., your task requires a config file that is deleted by another task in Dag.
If possible, use ``XCom`` to communicate small messages between tasks and a good way of passing larger data between tasks is to use a remote storage such as S3/HDFS.
For example, if we have a task that stores processed data in S3 that task can push the S3 path for the output data in ``Xcom``,
@@ -100,25 +100,25 @@ Top level Python Code
---------------------
You should avoid writing the top level code which is not necessary to create Operators
-and build DAG relations between them. This is because of the design decision for the scheduler of Airflow
+and build Dag relations between them. This is because of the design decision for the scheduler of Airflow
and the impact the top-level code parsing speed on both performance and scalability of Airflow.
Airflow scheduler executes the code outside the Operator's ``execute`` methods with the minimum interval of
:ref:`min_file_process_interval<config:dag_processor__min_file_process_interval>` seconds. This is done in order
-to allow dynamic scheduling of the dags - where scheduling and dependencies might change over time and
-impact the next schedule of the DAG. Airflow scheduler tries to continuously make sure that what you have
-in dags is correctly reflected in scheduled tasks.
+to allow dynamic scheduling of the Dags - where scheduling and dependencies might change over time and
+impact the next schedule of the Dag. Airflow scheduler tries to continuously make sure that what you have
+in Dags is correctly reflected in scheduled tasks.
Specifically you should not run any database access, heavy computations and networking operations.
-One of the important factors impacting DAG loading time, that might be overlooked by Python developers is
+One of the important factors impacting Dag loading time, that might be overlooked by Python developers is
that top-level imports might take surprisingly a lot of time and they can generate a lot of overhead
and this can be easily avoided by converting them to local imports inside Python callables for example.
-Consider the two examples below. In the first example, DAG will take an additional 1000 seconds to parse
-than the functionally equivalent DAG in the second example where the ``expensive_api_call`` is executed from the context of its task.
+Consider the two examples below. In the first example, Dag will take an additional 1000 seconds to parse
+than the functionally equivalent Dag in the second example where the ``expensive_api_call`` is executed from the context of its task.
-Not avoiding top-level DAG code:
+Not avoiding top-level Dag code:
.. code-block:: python
@@ -147,7 +147,7 @@ Not avoiding top-level DAG code:
def print_expensive_api_call():
print(my_expensive_response)
-Avoiding top-level DAG code:
+Avoiding top-level Dag code:
.. code-block:: python
@@ -175,17 +175,17 @@ Avoiding top-level DAG code:
my_expensive_response = expensive_api_call()
print(my_expensive_response)
-In the first example, ``expensive_api_call`` is executed each time the DAG file is parsed, which will result in suboptimal performance in the DAG file processing. In the second example, ``expensive_api_call`` is only called when the task is running and thus is able to be parsed without suffering any performance hits. To test it out yourself, implement the first DAG and see "Hello from Airflow!" printed in the scheduler logs!
+In the first example, ``expensive_api_call`` is executed each time the Dag file is parsed, which will result in suboptimal performance in the Dag file processing. In the second example, ``expensive_api_call`` is only called when the task is running and thus is able to be parsed without suffering any performance hits. To test it out yourself, implement the first Dag and see "Hello from Airflow!" printed in the scheduler logs!
Note that import statements also count as top-level code. So, if you have an import statement that takes a long time or the imported module itself executes code at the top-level, that can also impact the performance of the scheduler. The following example illustrates how to handle expensive imports.
.. code-block:: python
- # It's ok to import modules that are not expensive to load at top-level of a DAG file
+ # It's ok to import modules that are not expensive to load at top-level of a Dag file
import random
import pendulum
- # Expensive imports should be avoided as top level imports, because DAG files are parsed frequently, resulting in top-level code being executed.
+ # Expensive imports should be avoided as top level imports, because Dag files are parsed frequently, resulting in top-level code being executed.
#
# import pandas
# import torch
@@ -312,18 +312,18 @@ Installing and Using ruff
pip install "ruff>=0.11.6"
-2. **Running ruff**: Execute ``ruff`` to check your dags for potential issues:
+2. **Running ruff**: Execute ``ruff`` to check your Dags for potential issues:
.. code-block:: bash
ruff check dags/ --select AIR3 --preview
- This command will analyze your dags located in the ``dags/`` directory and report any issues related to the specified rules.
+ This command will analyze your Dags located in the ``dags/`` directory and report any issues related to the specified rules.
Example
-------
-Given a legacy DAG defined as:
+Given a legacy Dag defined as:
.. code-block:: python
@@ -340,7 +340,7 @@ Running ``ruff`` will produce:
.. code-block:: none
- dags/legacy_dag.py:7:2: AIR301 DAG should have an explicit schedule argument
+ dags/legacy_dag.py:7:2: AIR301 Dag should have an explicit schedule argument
dags/legacy_dag.py:12:6: AIR302 schedule_interval is removed in Airflow 3.0
dags/legacy_dag.py:17:15: AIR302 airflow.datasets.Dataset is removed in Airflow 3.0
dags/legacy_dag.py:19:5: AIR303 airflow.sensors.filesystem.FileSensor is moved into ``standard`` provider in Airflow 3.0
@@ -351,31 +351,31 @@ For more information on ``ruff`` and its integration with Airflow, refer to the
.. _best_practices/dynamic_dag_generation:
-Dynamic DAG Generation
+Dynamic Dag Generation
----------------------
-Sometimes writing dags manually isn't practical.
-Maybe you have a lot of dags that do similar things with just a parameter changing between them.
-Or maybe you need a set of dags to load tables, but don't want to manually update dags every time those tables change.
-In these and other cases, it can be more useful to dynamically generate dags.
+Sometimes writing Dags manually isn't practical.
+Maybe you have a lot of Dags that do similar things with just a parameter changing between them.
+Or maybe you need a set of Dags to load tables, but don't want to manually update Dags every time those tables change.
+In these and other cases, it can be more useful to dynamically generate Dags.
Avoiding excessive processing at the top level code described in the previous chapter is especially important
-in case of dynamic DAG configuration, which can be configured essentially in one of those ways:
+in case of dynamic Dag configuration, which can be configured essentially in one of those ways:
* via `environment variables <https://wiki.archlinux.org/title/environment_variables>`_ (not to be mistaken
with the :doc:`Airflow Variables </core-concepts/variables>`)
-* via externally provided, generated Python code, containing meta-data in the DAG folder
-* via externally provided, generated configuration meta-data file in the DAG folder
+* via externally provided, generated Python code, containing meta-data in the Dag folder
+* via externally provided, generated configuration meta-data file in the Dag folder
-Some cases of dynamic DAG generation are described in the :doc:`howto/dynamic-dag-generation` section.
+Some cases of dynamic Dag generation are described in the :doc:`howto/dynamic-dag-generation` section.
.. _best_practices/airflow_variables:
Airflow Variables
-----------------
-Using Airflow Variables yields network calls and database access, so their usage in top-level Python code for dags
+Using Airflow Variables yields network calls and database access, so their usage in top-level Python code for Dags
should be avoided as much as possible, as mentioned in the previous chapter, :ref:`best_practices/top_level_code`.
-If Airflow Variables must be used in top-level DAG code, then their impact on DAG parsing can be mitigated by
+If Airflow Variables must be used in top-level Dag code, then their impact on Dag parsing can be mitigated by
:ref:`enabling the experimental cache<config:secrets__use_cache>`, configured with a sensible :ref:`ttl<config:secrets__cache_ttl_seconds>`.
You can use the Airflow Variables freely inside the ``execute()`` methods of the operators, but you can also pass the
@@ -394,10 +394,10 @@ or if you need to deserialize a json object from the variable :
{{ var.json.<variable_name> }}
In top-level code, variables using jinja templates do not produce a request until a task is running, whereas,
-``Variable.get()`` produces a request every time the dag file is parsed by the scheduler if caching is not enabled.
+``Variable.get()`` produces a request every time the Dag file is parsed by the scheduler if caching is not enabled.
Using ``Variable.get()`` without :ref:`enabling caching<config:secrets__use_cache>` will lead to suboptimal
-performance in the dag file processing.
-In some cases this can cause the dag file to timeout before it is fully parsed.
+performance in the Dag file processing.
+In some cases this can cause the Dag file to timeout before it is fully parsed.
Bad example:
@@ -436,7 +436,7 @@ Good example:
@task
def my_task():
- var = Variable.get("foo") # This is ok since my_task is called only during task run, not during DAG scan.
+ var = Variable.get("foo") # This is ok since my_task is called only during task run, not during Dag scan.
print(var)
For security purpose, you're recommended to use the :ref:`Secrets Backend<secrets_backend_configuration>`
@@ -447,7 +447,7 @@ for any variable that contains sensitive data.
Timetables
----------
Avoid using Airflow Variables/Connections or accessing Airflow database at the top level of your timetable code.
-Database access should be delayed until the execution time of the DAG. This means that you should not have variables/connections retrieval
+Database access should be delayed until the execution time of the Dag. This means that you should not have variables/connections retrieval
as argument to your timetable class initialization or have Variable/connection at the top level of your custom timetable module.
Bad example:
@@ -477,18 +477,18 @@ Good example:
super().__init__(*args, **kwargs)
-Triggering dags after changes
+Triggering Dags after changes
-----------------------------
-Avoid triggering dags immediately after changing them or any other accompanying files that you change in the
-DAG folder.
+Avoid triggering Dags immediately after changing them or any other accompanying files that you change in the
+Dag folder.
You should give the system sufficient time to process the changed files. This takes several steps.
First the files have to be distributed to scheduler - usually via distributed filesystem or Git-Sync, then
scheduler has to parse the Python files and store them in the database. Depending on your configuration,
-speed of your distributed filesystem, number of files, number of dags, number of changes in the files,
+speed of your distributed filesystem, number of files, number of Dags, number of changes in the files,
sizes of the files, number of schedulers, speed of CPUS, this can take from seconds to minutes, in extreme
-cases many minutes. You should wait for your DAG to appear in the UI to be able to trigger it.
+cases many minutes. You should wait for your Dag to appear in the UI to be able to trigger it.
In case you see long delays between updating it and the time it is ready to be triggered, you can look
at the following configuration parameters and fine tune them according your needs (see details of
@@ -503,28 +503,28 @@ each parameter by following the links):
Example of watcher pattern with trigger rules
---------------------------------------------
-The watcher pattern is how we call a DAG with a task that is "watching" the states of the other tasks.
-Its primary purpose is to fail a DAG Run when any other task fail.
-The need came from the Airflow system tests that are dags with different tasks (similarly like a test containing steps).
+The watcher pattern is how we call a Dag with a task that is "watching" the states of the other tasks.
+Its primary purpose is to fail a Dag Run when any other task fail.
+The need came from the Airflow system tests that are Dags with different tasks (similarly like a test containing steps).
-Normally, when any task fails, all other tasks are not executed and the whole DAG Run gets failed status too. But
-when we use trigger rules, we can disrupt the normal flow of running tasks and the whole DAG may represent different
+Normally, when any task fails, all other tasks are not executed and the whole Dag Run gets failed status too. But
+when we use trigger rules, we can disrupt the normal flow of running tasks and the whole Dag may represent different
status that we expect. For example, we can have a teardown task (with trigger rule set to ``TriggerRule.ALL_DONE``)
that will be executed regardless of the state of the other tasks (e.g. to clean up the resources). In such
-situation, the DAG would always run this task and the DAG Run will get the status of this particular task, so we can
-potentially lose the information about failing tasks. If we want to ensure that the DAG with teardown task would fail
+situation, the Dag would always run this task and the Dag Run will get the status of this particular task, so we can
+potentially lose the information about failing tasks. If we want to ensure that the Dag with teardown task would fail
if any task fails, we need to use the watcher pattern. The watcher task is a task that will always fail if
triggered, but it needs to be triggered only if any other task fails. It needs to have a trigger rule set to
-``TriggerRule.ONE_FAILED`` and it needs also to be a downstream task for all other tasks in the DAG. Thanks to
+``TriggerRule.ONE_FAILED`` and it needs also to be a downstream task for all other tasks in the Dag. Thanks to
this, if every other task will pass, the watcher will be skipped, but when something fails, the watcher task will be
-executed and fail making the DAG Run fail too.
+executed and fail making the Dag Run fail too.
.. note::
Be aware that trigger rules only rely on the direct upstream (parent) tasks, e.g. ``TriggerRule.ONE_FAILED``
will ignore any failed (or ``upstream_failed``) tasks that are not a direct parent of the parameterized task.
-It's easier to grab the concept with an example. Let's say that we have the following DAG:
+It's easier to grab the concept with an example. Let's say that we have the following Dag:
.. code-block:: python
@@ -559,7 +559,7 @@ It's easier to grab the concept with an example. Let's say that we have the foll
failing_task >> passing_task >> teardown
list(dag.tasks) >> watcher()
-The visual representation of this DAG after execution looks like this:
+The visual representation of this Dag after execution looks like this:
.. image:: /img/watcher.png
@@ -568,19 +568,19 @@ We have several tasks that serve different purposes:
- ``failing_task`` always fails,
- ``passing_task`` always succeeds (if executed),
- ``teardown`` is always triggered (regardless the states of the other tasks) and it should always succeed,
-- ``watcher`` is a downstream task for each other task, i.e. it will be triggered when any task fails and thus fail the whole DAG Run, since it's a leaf task.
+- ``watcher`` is a downstream task for each other task, i.e. it will be triggered when any task fails and thus fail the whole Dag Run, since it's a leaf task.
-It's important to note, that without ``watcher`` task, the whole DAG Run will get the ``success`` state, since the only failing task is not the leaf task, and the ``teardown`` task will finish with ``success``.
-If we want the ``watcher`` to monitor the state of all tasks, we need to make it dependent on all of them separately. Thanks to this, we can fail the DAG Run if any of the tasks fail. Note that the watcher task has a trigger rule set to ``"one_failed"``.
-On the other hand, without the ``teardown`` task, the ``watcher`` task will not be needed, because ``failing_task`` will propagate its ``failed`` state to downstream task ``passed_task`` and the whole DAG Run will also get the ``failed`` status.
+It's important to note, that without ``watcher`` task, the whole Dag Run will get the ``success`` state, since the only failing task is not the leaf task, and the ``teardown`` task will finish with ``success``.
+If we want the ``watcher`` to monitor the state of all tasks, we need to make it dependent on all of them separately. Thanks to this, we can fail the Dag Run if any of the tasks fail. Note that the watcher task has a trigger rule set to ``"one_failed"``.
+On the other hand, without the ``teardown`` task, the ``watcher`` task will not be needed, because ``failing_task`` will propagate its ``failed`` state to downstream task ``passed_task`` and the whole Dag Run will also get the ``failed`` status.
-Using AirflowClusterPolicySkipDag exception in cluster policies to skip specific dags
+Using AirflowClusterPolicySkipDag exception in cluster policies to skip specific Dags
-------------------------------------------------------------------------------------
.. versionadded:: 2.7
-Airflow dags can usually be deployed and updated with the specific branch of Git repository via ``git-sync``.
+Airflow Dags can usually be deployed and updated with the specific branch of Git repository via ``git-sync``.
But, when you have to run multiple Airflow clusters for some operational reasons, it's very cumbersome to maintain multiple Git branches.
Especially, you have some difficulties when you need to synchronize two separate branches(like ``prod`` and ``beta``) periodically with proper branching strategy.
@@ -588,94 +588,94 @@ Especially, you have some difficulties when you need to synchronize two separate
- hard-reset is not recommended way for GitOps
So, you can consider connecting multiple Airflow clusters with same Git branch (like ``main``), and maintaining those with different environment variables and different connection configurations with same ``connection_id``.
-you can also raise :class:`~airflow.exceptions.AirflowClusterPolicySkipDag` exception on the cluster policy, to load specific dags to :class:`~airflow.models.dagbag.DagBag` on the specific Airflow deployment only, if needed.
+you can also raise :class:`~airflow.exceptions.AirflowClusterPolicySkipDag` exception on the cluster policy, to load specific Dags to :class:`~airflow.models.dagbag.DagBag` on the specific Airflow deployment only, if needed.
.. code-block:: python
def dag_policy(dag: DAG):
- """Skipping the DAG with `only_for_beta` tag."""
+ """Skipping the Dag with `only_for_beta` tag."""
if "only_for_beta" in dag.tags:
raise AirflowClusterPolicySkipDag(
- f"DAG {dag.dag_id} is not loaded on the production cluster, due to `only_for_beta` tag."
+ f"Dag {dag.dag_id} is not loaded on the production cluster, due to `only_for_beta` tag."
)
-The example above, shows the ``dag_policy`` code snippet to skip the DAG depending on the tags it has.
+The example above, shows the ``dag_policy`` code snippet to skip the Dag depending on the tags it has.
.. _best_practices/reducing_dag_complexity:
-Reducing DAG complexity
+Reducing Dag complexity
^^^^^^^^^^^^^^^^^^^^^^^
-While Airflow is good in handling a lot of dags with a lot of task and dependencies between them, when you
-have many complex dags, their complexity might impact performance of scheduling. One of the ways to keep
-your Airflow instance performant and well utilized, you should strive to simplify and optimize your dags
-whenever possible - you have to remember that DAG parsing process and creation is just executing
+While Airflow is good in handling a lot of Dags with a lot of task and dependencies between them, when you
+have many complex Dags, their complexity might impact performance of scheduling. One of the ways to keep
+your Airflow instance performant and well utilized, you should strive to simplify and optimize your Dags
+whenever possible - you have to remember that Dag parsing process and creation is just executing
Python code and it's up to you to make it as performant as possible. There are no magic recipes for making
-your DAG "less complex" - since this is a Python code, it's the DAG writer who controls the complexity of
+your Dag "less complex" - since this is a Python code, it's the Dag writer who controls the complexity of
their code.
-There are no "metrics" for DAG complexity, especially, there are no metrics that can tell you
-whether your DAG is "simple enough". However, as with any Python code, you can definitely tell that
-your DAG code is "simpler" or "faster" when it is optimized. If you
-want to optimize your dags there are the following actions you can take:
+There are no "metrics" for Dag complexity, especially, there are no metrics that can tell you
+whether your Dag is "simple enough". However, as with any Python code, you can definitely tell that
+your Dag code is "simpler" or "faster" when it is optimized. If you
+want to optimize your Dags there are the following actions you can take:
-* Make your DAG load faster. This is a single improvement advice that might be implemented in various ways
+* Make your Dag load faster. This is a single improvement advice that might be implemented in various ways
but this is the one that has biggest impact on scheduler's performance. Whenever you have a chance to make
- your DAG load faster - go for it, if your goal is to improve performance. Look at the
+ your Dag load faster - go for it, if your goal is to improve performance. Look at the
:ref:`best_practices/top_level_code` to get some tips of how you can do it. Also see at
- :ref:`best_practices/dag_loader_test` on how to asses your DAG loading time.
+ :ref:`best_practices/dag_loader_test` on how to asses your Dag loading time.
-* Make your DAG generate simpler structure. Every task dependency adds additional processing overhead for
- scheduling and execution. The DAG that has simple linear structure ``A -> B -> C`` will experience
- less delays in task scheduling than DAG that has a deeply nested tree structure with exponentially growing
- number of depending tasks for example. If you can make your dags more linear - where at single point in
+* Make your Dag generate simpler structure. Every task dependency adds additional processing overhead for
+ scheduling and execution. The Dag that has simple linear structure ``A -> B -> C`` will experience
+ less delays in task scheduling than Dag that has a deeply nested tree structure with exponentially growing
+ number of depending tasks for example. If you can make your Dags more linear - where at single point in
execution there are as few potential candidates to run among the tasks, this will likely improve overall
scheduling performance.
-* Make smaller number of dags per file. While Airflow 2 is optimized for the case of having multiple dags
+* Make smaller number of Dags per file. While Airflow 2 is optimized for the case of having multiple Dags
in one file, there are some parts of the system that make it sometimes less performant, or introduce more
- delays than having those dags split among many files. Just the fact that one file can only be parsed by one
- FileProcessor, makes it less scalable for example. If you have many dags generated from one file,
- consider splitting them if you observe it takes a long time to reflect changes in your DAG files in the
+ delays than having those Dags split among many files. Just the fact that one file can only be parsed by one
+ FileProcessor, makes it less scalable for example. If you have many Dags generated from one file,
+ consider splitting them if you observe it takes a long time to reflect changes in your Dag files in the
UI of Airflow.
-* Write efficient Python code. A balance must be struck between fewer dags per file, as stated above, and
- writing less code overall. Creating the Python files that describe dags should follow best programming
- practices and not be treated like configurations. If your dags share similar code you should not copy
+* Write efficient Python code. A balance must be struck between fewer Dags per file, as stated above, and
+ writing less code overall. Creating the Python files that describe Dags should follow best programming
+ practices and not be treated like configurations. If your Dags share similar code you should not copy
them over and over again to a large number of nearly identical source files, as this will cause a
number of unnecessary repeated imports of the same resources. Rather, you should aim to minimize
- repeated code across all of your dags so that the application can run efficiently and can be easily
- debugged. See :ref:`best_practices/dynamic_dag_generation` on how to create multiple dags with similar
+ repeated code across all of your Dags so that the application can run efficiently and can be easily
+ debugged. See :ref:`best_practices/dynamic_dag_generation` on how to create multiple Dags with similar
code.
-Testing a DAG
+Testing a Dag
^^^^^^^^^^^^^
-Airflow users should treat dags as production level code, and dags should have various associated tests to
-ensure that they produce expected results. You can write a wide variety of tests for a DAG.
+Airflow users should treat Dags as production level code, and Dags should have various associated tests to
+ensure that they produce expected results. You can write a wide variety of tests for a Dag.
Let's take a look at some of them.
.. _best_practices/dag_loader_test:
-DAG Loader Test
+Dag Loader Test
---------------
-This test should ensure that your DAG does not contain a piece of code that raises error while loading.
+This test should ensure that your Dag does not contain a piece of code that raises error while loading.
No additional code needs to be written by the user to run this test.
.. code-block:: bash
python your-dag-file.py
-Running the above command without any error ensures your DAG does not contain any uninstalled dependency,
-syntax errors, etc. Make sure that you load your DAG in an environment that corresponds to your
+Running the above command without any error ensures your Dag does not contain any uninstalled dependency,
+syntax errors, etc. Make sure that you load your Dag in an environment that corresponds to your
scheduler environment - with the same dependencies, environment variables, common code referred from the
-DAG.
+Dag.
-This is also a great way to check if your DAG loads faster after an optimization, if you want to attempt
-to optimize DAG loading time. Simply run the DAG and measure the time it takes, but again you have to
-make sure your DAG runs with the same dependencies, environment variables, common code.
+This is also a great way to check if your Dag loads faster after an optimization, if you want to attempt
+to optimize Dag loading time. Simply run the Dag and measure the time it takes, but again you have to
+make sure your Dag runs with the same dependencies, environment variables, common code.
There are many ways to measure the time of processing, one of them in Linux environment is to
use built-in ``time`` command. Make sure to run it several times in succession to account for
@@ -695,8 +695,8 @@ Result:
sys 0m0.108s
The important metrics is the "real time" - which tells you how long time it took
-to process the DAG. Note that when loading the file this way, you are starting a new interpreter so there is
-an initial loading time that is not present when Airflow parses the DAG. You can assess the
+to process the Dag. Note that when loading the file this way, you are starting a new interpreter so there is
+an initial loading time that is not present when Airflow parses the Dag. You can assess the
time of initialization by running:
.. code-block:: bash
@@ -712,16 +712,16 @@ Result:
sys 0m0.039s
In this case the initial interpreter startup time is ~ 0.07s which is about 10% of time needed to parse
-the example_python_operator.py above so the actual parsing time is about ~ 0.62 s for the example DAG.
+the example_python_operator.py above so the actual parsing time is about ~ 0.62 s for the example Dag.
-You can look into :ref:`Testing a DAG <testing>` for details on how to test individual operators.
+You can look into :ref:`Testing a Dag <testing>` for details on how to test individual operators.
Unit tests
-----------
-Unit tests ensure that there is no incorrect code in your DAG. You can write unit tests for both your tasks and your DAG.
+Unit tests ensure that there is no incorrect code in your Dag. You can write unit tests for both your tasks and your Dag.
-**Unit test for loading a DAG:**
+**Unit test for loading a Dag:**
.. code-block:: python
@@ -742,8 +742,8 @@ Unit tests ensure that there is no incorrect code in your DAG. You can write uni
assert len(dag.tasks) == 1
-**Unit test a DAG structure:**
-This is an example test want to verify the structure of a code-generated DAG against a dict object
+**Unit test a Dag structure:**
+This is an example test want to verify the structure of a code-generated Dag against a dict object
.. code-block:: python
@@ -798,7 +798,7 @@ This is an example test want to verify the structure of a code-generated DAG aga
Self-Checks
------------
-You can also implement checks in a DAG to make sure the tasks are producing the results as expected.
+You can also implement checks in a Dag to make sure the tasks are producing the results as expected.
As an example, if you have a task that pushes data to S3, you can implement a check in the next task. For example, the check could
make sure that the partition is created in S3 and perform some simple checks to determine if the data is correct.
@@ -821,11 +821,11 @@ Similarly, if you have a task that starts a microservice in Kubernetes or Mesos,
Staging environment
--------------------
-If possible, keep a staging environment to test the complete DAG run before deploying in the production.
-Make sure your DAG is parameterized to change the variables, e.g., the output path of S3 operation or the database used to read the configuration.
-Do not hard code values inside the DAG and then change them manually according to the environment.
+If possible, keep a staging environment to test the complete Dag run before deploying in the production.
+Make sure your Dag is parameterized to change the variables, e.g., the output path of S3 operation or the database used to read the configuration.
+Do not hard code values inside the Dag and then change them manually according to the environment.
-You can use environment variables to parameterize the DAG.
+You can use environment variables to parameterize the Dag.
.. code-block:: python
@@ -861,7 +861,7 @@ For connection, use :envvar:`AIRFLOW_CONN_{CONN_ID}`.
Metadata DB maintenance
^^^^^^^^^^^^^^^^^^^^^^^
-Over time, the metadata database will increase its storage footprint as more DAG and task runs and event logs accumulate.
+Over time, the metadata database will increase its storage footprint as more Dag and task runs and event logs accumulate.
You can use the Airflow CLI to purge old data with the command ``airflow db clean``.
@@ -880,16 +880,16 @@ Disable the scheduler
You might consider disabling the Airflow cluster while you perform such maintenance.
-One way to do so would be to set the param ``[scheduler] > use_job_schedule`` to ``False`` and wait for any running dags to complete; after this no new DAG runs will be created unless externally triggered.
+One way to do so would be to set the param ``[scheduler] > use_job_schedule`` to ``False`` and wait for any running Dags to complete; after this no new Dag runs will be created unless externally triggered.
-A *better* way (though it's a bit more manual) is to use the ``dags pause`` command. You'll need to keep track of the dags that are paused before you begin this operation so that you know which ones to unpause after maintenance is complete. First run ``airflow dags list`` and store the list of unpaused dags. Then use this same list to run both ``dags pause`` for each DAG prior to maintenance, and ``dags unpause`` after. A benefit of this is you can try un-pausing just one or two dags [...]
+A *better* way (though it's a bit more manual) is to use the ``dags pause`` command. You'll need to keep track of the Dags that are paused before you begin this operation so that you know which ones to unpause after maintenance is complete. First run ``airflow dags list`` and store the list of unpaused Dags. Then use this same list to run both ``dags pause`` for each Dag prior to maintenance, and ``dags unpause`` after. A benefit of this is you can try un-pausing just one or two Dags [...]
.. _integration-test-dags:
-Add "integration test" dags
+Add "integration test" Dags
---------------------------
-It can be helpful to add a couple "integration test" dags that use all the common services in your ecosystem (e.g. S3, Snowflake, Vault) but with test resources or "dev" accounts. These test dags can be the ones you turn on *first* after an upgrade, because if they fail, it doesn't matter and you can revert to your backup without negative consequences. However, if they succeed, they should prove that your cluster is able to run tasks with the libraries and services that you need to use.
+It can be helpful to add a couple "integration test" Dags that use all the common services in your ecosystem (e.g. S3, Snowflake, Vault) but with test resources or "dev" accounts. These test Dags can be the ones you turn on *first* after an upgrade, because if they fail, it doesn't matter and you can revert to your backup without negative consequences. However, if they succeed, they should prove that your cluster is able to run tasks with the libraries and services that you need to use.
For example, if you use an external secrets backend, make sure you have a task that retrieves a connection. If you use KubernetesPodOperator, add a task that runs ``sleep 30; echo "hello"``. If you need to write to s3, do so in a test task. And if you need to access a database, add a task that does ``select 1`` from the server.
@@ -960,7 +960,7 @@ The benefits of the operator are:
* No changes in deployment requirements - whether you use Local virtualenv, or Docker, or Kubernetes,
the tasks will work without adding anything to your deployment.
* No need to learn more about containers, Kubernetes as a Dag author. Only knowledge of Python requirements
- is required to author dags this way.
+ is required to author Dags this way.
There are certain limitations and overhead introduced by this operator:
@@ -968,7 +968,7 @@ There are certain limitations and overhead introduced by this operator:
using standard ``pickle`` library. You can mitigate some of those limitations by using ``dill`` library
but even that library does not solve all the serialization limitations.
* All dependencies that are not available in the Airflow environment must be locally imported in the callable you
- use and the top-level Python code of your DAG should not import/use those libraries.
+ use and the top-level Python code of your Dag should not import/use those libraries.
* The virtual environments are run in the same operating system, so they cannot have conflicting system-level
dependencies (``apt`` or ``yum`` installable packages). Only Python dependencies can be independently
installed in those environments.
@@ -1025,7 +1025,7 @@ The benefits of the operator are:
* Limited impact on your deployment - you do not need to switch to Docker containers or Kubernetes to
make a good use of the operator.
* No need to learn more about containers, Kubernetes as a Dag author. Only knowledge of Python, requirements
- is required to author dags this way.
+ is required to author Dags this way.
The drawbacks:
@@ -1036,7 +1036,7 @@ The drawbacks:
using standard ``pickle`` library. You can mitigate some of those limitations by using ``dill`` library
but even that library does not solve all the serialization limitations.
* All dependencies that are not available in Airflow environment must be locally imported in the callable you
- use and the top-level Python code of your DAG should not import/use those libraries.
+ use and the top-level Python code of your Dag should not import/use those libraries.
* The virtual environments are run in the same operating system, so they cannot have conflicting system-level
dependencies (``apt`` or ``yum`` installable packages). Only Python dependencies can be independently
installed in those environments
@@ -1046,7 +1046,7 @@ The drawbacks:
You can think about the ``PythonVirtualenvOperator`` and ``ExternalPythonOperator`` as counterparts -
that make it smoother to move from development phase to production phase. As a Dag author you'd normally
-iterate with dependencies and develop your DAG using ``PythonVirtualenvOperator`` (thus decorating
+iterate with dependencies and develop your Dag using ``PythonVirtualenvOperator`` (thus decorating
your tasks with ``@task.virtualenv`` decorators) while after the iteration and changes you would likely
want to change it for production to switch to the ``ExternalPythonOperator`` (and ``@task.external_python``)
after your DevOps/System Admin teams deploy your new dependencies in pre-existing virtualenv in production.
@@ -1112,10 +1112,10 @@ The drawbacks:
An appropriate deployment pipeline here is essential to be able to reliably maintain your deployment.
* Your python callable has to be serializable if you want to run it via decorators, also in this case
all dependencies that are not available in Airflow environment must be locally imported in the callable you
- use and the top-level Python code of your DAG should not import/use those libraries.
+ use and the top-level Python code of your Dag should not import/use those libraries.
* You need to understand more details about how Docker Containers or Kubernetes work. The abstraction
provided by those two are "leaky", so you need to understand a bit more about resources, networking,
- containers etc. in order to author a DAG that uses those operators.
+ containers etc. in order to author a Dag that uses those operators.
You can see detailed examples of using :class:`airflow.operators.providers.Docker` in
:ref:`TaskFlow Docker example <taskflow-docker_environment>`
@@ -1134,10 +1134,10 @@ are less chances for resource reuse and it's much more difficult to fine-tune su
cost of resources without impacting the performance and stability.
One of the possible ways to make it more useful is
-`AIP-46 Runtime isolation for Airflow tasks and DAG parsing <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-46+Runtime+isolation+for+airflow+tasks+and+dag+parsing>`_.
-and completion of `AIP-43 DAG Processor Separation <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-43+DAG+Processor+separation>`_
+`AIP-46 Runtime isolation for Airflow tasks and Dag parsing <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-46+Runtime+isolation+for+airflow+tasks+and+dag+parsing>`_.
+and completion of `AIP-43 Dag Processor Separation <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-43+DAG+Processor+separation>`_
Until those are implemented, there are very few benefits of using this approach and it is not recommended.
When those AIPs are implemented, however, this will open up the possibility of a more multi-tenant approach,
where multiple teams will be able to have completely isolated sets of dependencies that will be used across
-the full lifecycle of a DAG - from parsing to execution.
+the full lifecycle of a Dag - from parsing to execution.
diff --git a/airflow-core/docs/cli-and-env-variables-ref.rst b/airflow-core/docs/cli-and-env-variables-ref.rst
index 6336396911b..9fe8bdb80d0 100644
--- a/airflow-core/docs/cli-and-env-variables-ref.rst
+++ b/airflow-core/docs/cli-and-env-variables-ref.rst
@@ -24,7 +24,7 @@ Command Line Interface
''''''''''''''''''''''
Airflow has a very rich command line interface that allows for
-many types of operation on a DAG, starting services, and supporting
+many types of operation on a Dag, starting services, and supporting
development and testing.
.. note::
@@ -104,7 +104,7 @@ Environment Variables
.. envvar:: AIRFLOW_HOME
The root directory for the Airflow content.
- This is the default parent directory for Airflow assets such as dags and logs.
+ This is the default parent directory for Airflow assets such as Dags and logs.
.. envvar:: AIRFLOW_VAR_{KEY}
diff --git a/airflow-core/docs/core-concepts/auth-manager/index.rst b/airflow-core/docs/core-concepts/auth-manager/index.rst
index 845d03b03d0..d86083b66bc 100644
--- a/airflow-core/docs/core-concepts/auth-manager/index.rst
+++ b/airflow-core/docs/core-concepts/auth-manager/index.rst
@@ -132,10 +132,10 @@ These authorization methods are:
* ``is_authorized_configuration``: Return whether the user is authorized to access Airflow configuration. Some details about the configuration can be provided (e.g. the config section).
* ``is_authorized_connection``: Return whether the user is authorized to access Airflow connections. Some details about the connection can be provided (e.g. the connection ID).
-* ``is_authorized_dag``: Return whether the user is authorized to access a DAG. Some details about the DAG can be provided (e.g. the DAG ID).
- Also, ``is_authorized_dag`` is called for any entity related to dags (e.g. task instances, dag runs, ...). This information is passed in ``access_entity``.
+* ``is_authorized_dag``: Return whether the user is authorized to access a Dag. Some details about the Dag can be provided (e.g. the Dag ID).
+ Also, ``is_authorized_dag`` is called for any entity related to Dags (e.g. task instances, Dag runs, ...). This information is passed in ``access_entity``.
Example: ``auth_manager.is_authorized_dag(method="GET", access_entity=DagAccessEntity.Run, details=DagDetails(id="dag-1"))`` asks
- whether the user has permission to read the Dag runs of the dag "dag-1".
+ whether the user has permission to read the Dag runs of the Dag "dag-1".
* ``is_authorized_backfill``: Return whether the user is authorized to access Airflow backfills. Some details about the backfill can be provided (e.g. the backfill ID).
* ``is_authorized_asset``: Return whether the user is authorized to access Airflow assets. Some details about the asset can be provided (e.g. the asset ID).
* ``is_authorized_asset_alias``: Return whether the user is authorized to access Airflow asset aliases. Some details about the asset alias can be provided (e.g. the asset alias ID).
@@ -179,9 +179,9 @@ Optional methods recommended to override for optimization
The following methods aren't required to override to have a functional Airflow auth manager. However, it is recommended to override these to make your auth manager faster (and potentially less costly):
* ``batch_is_authorized_dag``: Batch version of ``is_authorized_dag``. If not overridden, it will call ``is_authorized_dag`` for every single item.
-* ``get_authorized_dag_ids``: Return the list of DAG IDs the user has access to. If not overridden, it will call ``is_authorized_dag`` for every single DAG available in the environment.
+* ``get_authorized_dag_ids``: Return the list of Dag IDs the user has access to. If not overridden, it will call ``is_authorized_dag`` for every single Dag available in the environment.
- * Note: To filter the results of ``get_authorized_dag_ids``, it is recommended that you define the filtering logic in your ``filter_authorized_dag_ids`` method. For example, this may be useful if you rely on per-DAG access controls derived from one or more fields on a given DAG (e.g. DAG tags).
+ * Note: To filter the results of ``get_authorized_dag_ids``, it is recommended that you define the filtering logic in your ``filter_authorized_dag_ids`` method. For example, this may be useful if you rely on per-Dag access controls derived from one or more fields on a given Dag (e.g. Dag tags).
* This method requires an active session with the Airflow metadata database. As such, overriding the ``get_authorized_dag_ids`` method is an advanced use case, which should be considered carefully -- it is recommended you refer to the :doc:`../../database-erd-ref`.
CLI
@@ -240,24 +240,24 @@ Additional Caveats
* Your auth manager should not reference anything from the ``airflow.security.permissions`` module, as that module is in the process of being deprecated.
Instead, your code should use the definitions in ``airflow.api_fastapi.auth.managers.models.resource_details``. For more details on the ``airflow.security.permissions`` deprecation, see :doc:`/security/deprecated_permissions`
-* The ``access_control`` attribute of a DAG instance is only compatible with the FAB auth manager. Custom auth manager implementations should leverage ``get_authorized_dag_ids`` for DAG instance attribute-based access controls in more customizable ways (e.g. authorization based on DAG tags, DAG bundles, etc.).
+* The ``access_control`` attribute of a Dag instance is only compatible with the FAB auth manager. Custom auth manager implementations should leverage ``get_authorized_dag_ids`` for Dag instance attribute-based access controls in more customizable ways (e.g. authorization based on Dag tags, Dag bundles, etc.).
* You may find it useful to define a private, generalized ``_is_authorized`` method which acts as the standardized authorization mechanism, and which each
public ``is_authorized_*`` method calls with the appropriate parameters.
For concrete examples of this, refer to the ``SimpleAuthManager._is_authorized_method``. Further, it may be useful to optionally use the ``airflow.api_fastapi.auth.managers.base_auth_manager.ExtendedResourceMethod`` reference within your private method.
-DAG and DAG Sub-Component Authorization
+Dag and Dag Sub-Component Authorization
---------------------------------------
-Given the hierarchical structure of DAGs and their composite resources, the auth manager's ``is_authorized_dag`` method should also handle the authorization logic for DAG runs, tasks, and task instances.
-The ``access_entity`` parameter passed to ``is_authorized_dag`` indicates which (if any) DAG sub-component the user is attempting to access. This leads to a few important points:
+Given the hierarchical structure of Dags and their composite resources, the auth manager's ``is_authorized_dag`` method should also handle the authorization logic for Dag runs, tasks, and task instances.
+The ``access_entity`` parameter passed to ``is_authorized_dag`` indicates which (if any) Dag sub-component the user is attempting to access. This leads to a few important points:
-* If the ``access_entity`` parameter is ``None``, then the user is attempting to interact directly with the DAG, not any of its sub-components.
-* When the ``access_entity`` parameter is not ``None``, it means the user is attempting to access some sub-component of the DAG. This is noteworthy, as in some cases the ``method`` parameter may be valid
- for the DAG's sub-entity, but not a valid action directly on the DAG itself. For example, the ``POST`` method is valid for DAG runs, but **not** for DAGs.
-* One potential way to model the example request mentioned above -- where the ``method`` only has meaning for the DAG sub-component -- is to authorize the user if **both** statements are true:
+* If the ``access_entity`` parameter is ``None``, then the user is attempting to interact directly with the Dag, not any of its sub-components.
+* When the ``access_entity`` parameter is not ``None``, it means the user is attempting to access some sub-component of the Dag. This is noteworthy, as in some cases the ``method`` parameter may be valid
+ for the Dag's sub-entity, but not a valid action directly on the Dag itself. For example, the ``POST`` method is valid for Dag runs, but **not** for Dags.
+* One potential way to model the example request mentioned above -- where the ``method`` only has meaning for the Dag sub-component -- is to authorize the user if **both** statements are true:
- * The user has ``PUT`` ("edit") permissions for the given DAG.
- * The user has ``POST`` ("create") permissions for DAG runs.
+ * The user has ``PUT`` ("edit") permissions for the given Dag.
+ * The user has ``POST`` ("create") permissions for Dag runs.
Next Steps
----------
diff --git a/airflow-core/docs/core-concepts/auth-manager/simple/index.rst b/airflow-core/docs/core-concepts/auth-manager/simple/index.rst
index 2a51b4821fa..38dbc9795a9 100644
--- a/airflow-core/docs/core-concepts/auth-manager/simple/index.rst
+++ b/airflow-core/docs/core-concepts/auth-manager/simple/index.rst
@@ -70,8 +70,8 @@ Manage roles and permissions
There is no option to manage roles and permissions in simple auth manager. They are defined as part of the simple auth manager implementation and cannot be modified.
Here is the list of roles defined in simple auth manager. These roles can be associated to users.
-* **viewer**. Read-only permissions on dags, assets and pools
-* **user**. **viewer** permissions plus all permissions (edit, create, delete) on dags
+* **viewer**. Read-only permissions on Dags, assets and pools
+* **user**. **viewer** permissions plus all permissions (edit, create, delete) on Dags
* **op**. **user** permissions plus all permissions on pools, assets, config, connections and variables
* **admin**. All permissions
diff --git a/airflow-core/docs/core-concepts/backfill.rst b/airflow-core/docs/core-concepts/backfill.rst
index 8262633174a..e194fe65a15 100644
--- a/airflow-core/docs/core-concepts/backfill.rst
+++ b/airflow-core/docs/core-concepts/backfill.rst
@@ -18,11 +18,11 @@
Backfill
========
-Backfill is when you create runs for past dates of a dag. Airflow provides a mechanism
-to do this through the CLI and REST API. You provide a dag, a start date, and an end date,
-and Airflow will create runs in the range according to the dag's schedule.
+Backfill is when you create runs for past dates of a Dag. Airflow provides a mechanism
+to do this through the CLI and REST API. You provide a Dag, a start date, and an end date,
+and Airflow will create runs in the range according to the Dag's schedule.
-Backfill does not make sense for dags that don't have a time-based schedule.
+Backfill does not make sense for Dags that don't have a time-based schedule.
Control over data reprocessing
------------------------------
@@ -38,9 +38,9 @@ If the latest run is still running or is queued, we do not create another run, n
Concurrency control
-------------------
-You can set ``max_active_runs`` on a backfill and it will control how many dag runs in
+You can set ``max_active_runs`` on a backfill and it will control how many Dag runs in
the backfill can run concurrently. Backfill ``max_active_runs`` is applied independently
-the DAG ``max_active_runs`` setting.
+the Dag ``max_active_runs`` setting.
Run ordering
------------
@@ -74,7 +74,7 @@ For CLI, below is an example command:
For UI, follow the following steps:
-1. Navigate to a DAG's Details page and click **Trigger**.
+1. Navigate to a Dag's Details page and click **Trigger**.
2. In the pop-up window, select **Backfill**.
3. Fill in the form:
@@ -83,7 +83,7 @@ For UI, follow the following steps:
- **Max active runs**: limit concurrent backfill runs for this backfill.
- **Run backwards**: execute most recent intervals first.
- **Advanced Config**: optionally provide JSON ``dag_run.conf``.
- - If the DAG is paused, you can **Unpause** it in the same window.
+ - If the Dag is paused, you can **Unpause** it in the same window.
.. image:: ../img/ui-light/backfill.png
:alt: Backfill pop-up window (Light Mode)
diff --git a/airflow-core/docs/core-concepts/dag-run.rst b/airflow-core/docs/core-concepts/dag-run.rst
index 4722ba15c0b..f524515e3b1 100644
--- a/airflow-core/docs/core-concepts/dag-run.rst
+++ b/airflow-core/docs/core-concepts/dag-run.rst
@@ -15,84 +15,84 @@
specific language governing permissions and limitations
under the License.
-DAG Runs
+Dag Runs
=========
-A DAG Run is an object representing an instantiation of the DAG in time.
-Any time the DAG is executed, a DAG Run is created and all tasks inside it are executed. The status of the DAG Run depends on the tasks states.
-Each DAG Run is run separately from one another, meaning that you can have many runs of a DAG at the same time.
+A Dag Run is an object representing an instantiation of the Dag in time.
+Any time the Dag is executed, a Dag Run is created and all tasks inside it are executed. The status of the Dag Run depends on the tasks states.
+Each Dag Run is run separately from one another, meaning that you can have many runs of a Dag at the same time.
.. _dag-run:dag-run-status:
-DAG Run Status
+Dag Run Status
''''''''''''''
-A DAG Run status is determined when the execution of the DAG is finished.
-The execution of the DAG depends on its containing tasks and their dependencies.
-The status is assigned to the DAG Run when all of the tasks are in the one of the terminal states (i.e. if there is no possible transition to another state) like ``success``, ``failed`` or ``skipped``.
-The DAG Run is having the status assigned based on the so-called "leaf nodes" or simply "leaves". Leaf nodes are the tasks with no children.
+A Dag Run status is determined when the execution of the Dag is finished.
+The execution of the Dag depends on its containing tasks and their dependencies.
+The status is assigned to the Dag Run when all of the tasks are in the one of the terminal states (i.e. if there is no possible transition to another state) like ``success``, ``failed`` or ``skipped``.
+The Dag Run is having the status assigned based on the so-called "leaf nodes" or simply "leaves". Leaf nodes are the tasks with no children.
-There are two possible terminal states for the DAG Run:
+There are two possible terminal states for the Dag Run:
- ``success`` if all of the leaf nodes states are either ``success`` or ``skipped``,
- ``failed`` if any of the leaf nodes state is either ``failed`` or ``upstream_failed``.
.. note::
Be careful if some of your tasks have defined some specific :ref:`trigger rule <concepts:trigger-rules>`.
- These can lead to some unexpected behavior, e.g. if you have a leaf task with trigger rule `"all_done"`, it will be executed regardless of the states of the rest of the tasks and if it will succeed, then the whole DAG Run will also be marked as ``success``, even if something failed in the middle.
+ These can lead to some unexpected behavior, e.g. if you have a leaf task with trigger rule `"all_done"`, it will be executed regardless of the states of the rest of the tasks and if it will succeed, then the whole Dag Run will also be marked as ``success``, even if something failed in the middle.
*Added in Airflow 2.7*
-Dags that have a currently running DAG run can be shown on the UI dashboard in the "Running" tab. Similarly, dags whose latest DAG run is marked as failed can be found on the "Failed" tab.
+Dags that have a currently running Dag run can be shown on the UI dashboard in the "Running" tab. Similarly, Dags whose latest Dag run is marked as failed can be found on the "Failed" tab.
.. _data-interval:
Data Interval
-------------
-Each DAG run in Airflow has an assigned "data interval" that represents the time
-range it operates in. For a DAG scheduled with ``@daily``, for example, each of
+Each Dag run in Airflow has an assigned "data interval" that represents the time
+range it operates in. For a Dag scheduled with ``@daily``, for example, each of
its data interval would start each day at midnight (00:00) and end at midnight
(24:00).
-A DAG run is usually scheduled *after* its associated data interval has ended,
+A Dag run is usually scheduled *after* its associated data interval has ended,
to ensure the run is able to collect all the data within the time period. In
other words, a run covering the data period of 2020-01-01 generally does not
start to run until 2020-01-01 has ended, i.e. after 2020-01-02 00:00:00.
All dates in Airflow are tied to the data interval concept in some way. The
"logical date" (also called ``execution_date`` in Airflow versions prior to 2.2)
-of a DAG run, for example, denotes the start of the data interval, not when the
-DAG is actually executed.
+of a Dag run, for example, denotes the start of the data interval, not when the
+Dag is actually executed.
-Similarly, since the ``start_date`` argument for the DAG and its tasks points to
-the same logical date, it marks the start of *the DAG's first data interval*, not
-when tasks in the DAG will start running. In other words, a DAG run will only be
+Similarly, since the ``start_date`` argument for the Dag and its tasks points to
+the same logical date, it marks the start of *the Dag's first data interval*, not
+when tasks in the Dag will start running. In other words, a Dag run will only be
scheduled one interval after ``start_date``.
.. tip::
- If a cron expression or timedelta object is not enough to express your DAG's schedule,
+ If a cron expression or timedelta object is not enough to express your Dag's schedule,
logical date, or data interval, see :doc:`../authoring-and-scheduling/timetable`.
For more information on ``logical date``, see :ref:`concepts-dag-run` and
:ref:`faq:what-does-execution-date-mean`
-Re-run DAG
+Re-run Dag
''''''''''
-There can be cases where you will want to execute your DAG again. One such case is when the scheduled
-DAG run fails.
+There can be cases where you will want to execute your Dag again. One such case is when the scheduled
+Dag run fails.
.. _dag-catchup:
Catchup
-------
-An Airflow DAG defined with a ``start_date``, possibly an ``end_date``, and a non-asset schedule, defines a series of intervals which the scheduler turns into individual DAG runs and executes.
-By default, DAG runs that have not been run since the last data interval are not created by the scheduler upon activation of a DAG ( Airflow config ``scheduler.catchup_by_default=False``). The scheduler creates a DAG run only for the latest interval.
+An Airflow Dag defined with a ``start_date``, possibly an ``end_date``, and a non-asset schedule, defines a series of intervals which the scheduler turns into individual Dag runs and executes.
+By default, Dag runs that have not been run since the last data interval are not created by the scheduler upon activation of a Dag ( Airflow config ``scheduler.catchup_by_default=False``). The scheduler creates a Dag run only for the latest interval.
-If you set ``catchup=True`` in the DAG, the scheduler will kick off a DAG Run for any data interval that has not been run since the last data interval (or has been cleared). This concept is called Catchup.
+If you set ``catchup=True`` in the Dag, the scheduler will kick off a Dag Run for any data interval that has not been run since the last data interval (or has been cleared). This concept is called Catchup.
-If your DAG is not written to handle its catchup (i.e., not limited to the interval, but instead to ``Now`` for instance.),
-then you will want to turn catchup off, which is the default setting or can be done explicitly by setting ``catchup=False`` in the DAG definition, if the default config has been changed for your Airflow environment.
+If your Dag is not written to handle its catchup (i.e., not limited to the interval, but instead to ``Now`` for instance.),
+then you will want to turn catchup off, which is the default setting or can be done explicitly by setting ``catchup=False`` in the Dag definition, if the default config has been changed for your Airflow environment.
.. code-block:: python
@@ -115,36 +115,36 @@ then you will want to turn catchup off, which is the default setting or can be d
"retry_delay": datetime.timedelta(minutes=3),
},
start_date=pendulum.datetime(2015, 12, 1, tz="UTC"),
- description="A simple tutorial DAG",
+ description="A simple tutorial Dag",
schedule="@daily",
)
-In the example above, if the DAG is picked up by the scheduler daemon on
-2016-01-02 at 6 AM, (or from the command line), a single DAG Run will be created
+In the example above, if the Dag is picked up by the scheduler daemon on
+2016-01-02 at 6 AM, (or from the command line), a single Dag Run will be created
with a data between 2016-01-01 and 2016-01-02, and the next one will be created
just after midnight on the morning of 2016-01-03 with a data interval between
2016-01-02 and 2016-01-03.
Be aware that using a ``datetime.timedelta`` object as schedule can lead to a different behavior.
-In such a case, the single DAG Run created will cover data between 2016-01-01 06:00 and
+In such a case, the single Dag Run created will cover data between 2016-01-01 06:00 and
2016-01-02 06:00 (one schedule interval ending now). For a more detailed description of the
differences between a cron and a delta based schedule, take a look at the
:ref:`timetables comparison <Differences between the cron and delta data interval timetables>`
-If the ``dag.catchup`` value had been ``True`` instead, the scheduler would have created a DAG Run
+If the ``dag.catchup`` value had been ``True`` instead, the scheduler would have created a Dag Run
for each completed interval between 2015-12-01 and 2016-01-02 (but not yet one for 2016-01-02,
as that interval hasn't completed) and the scheduler will execute them sequentially.
-Catchup is also triggered when you turn off a DAG for a specified period and then re-enable it.
+Catchup is also triggered when you turn off a Dag for a specified period and then re-enable it.
This behavior is great for atomic assets that can easily be split into periods. Leaving catchup off is great
-if your DAG performs catchup internally.
+if your Dag performs catchup internally.
Backfill
--------
-You may want to run the DAG for a specified historical period. For example,
-a DAG is created with ``start_date`` **2024-11-21**, but another user requires
+You may want to run the Dag for a specified historical period. For example,
+a Dag is created with ``start_date`` **2024-11-21**, but another user requires
the output data from a month prior, i.e. **2024-10-21**.
This process is known as Backfill.
@@ -152,7 +152,7 @@ This can be done through either the UI or CLI.
UI
''
-From the DAG Details page, click **Trigger** and select **Backfill** to open the backfill form. Set the date range, reprocess
+From the Dag Details page, click **Trigger** and select **Backfill** to open the backfill form. Set the date range, reprocess
behavior, max active runs, optional backwards ordering, and Advanced Config.
.. image:: ../img/ui-light/backfill.png
@@ -190,12 +190,12 @@ The executor will re-run it.
There are multiple options you can select to re-run -
-* **Past** - All the instances of the task in the runs before the DAG's most recent data interval
-* **Future** - All the instances of the task in the runs after the DAG's most recent data interval
-* **Upstream** - The upstream tasks in the current DAG
-* **Downstream** - The downstream tasks in the current DAG
-* **Recursive** - All the tasks in the child dags and parent dags
-* **Failed** - Only the failed tasks in the DAG's most recent run
+* **Past** - All the instances of the task in the runs before the Dag's most recent data interval
+* **Future** - All the instances of the task in the runs after the Dag's most recent data interval
+* **Upstream** - The upstream tasks in the current Dag
+* **Downstream** - The downstream tasks in the current Dag
+* **Recursive** - All the tasks in the child Dags and parent Dags
+* **Failed** - Only the failed tasks in the Dag's most recent run
You can also clear the task through CLI using the command:
@@ -233,27 +233,27 @@ This can be useful for debugging.
External Triggers
'''''''''''''''''
-Note that DAG Runs can also be created manually through the CLI. Just run the command -
+Note that Dag Runs can also be created manually through the CLI. Just run the command -
.. code-block:: bash
airflow dags trigger --logical-date logical_date run_id
-The DAG Runs created externally to the scheduler get associated with the trigger's timestamp and are displayed
-in the UI alongside scheduled DAG runs. The logical date passed inside the DAG can be specified using the ``-e`` argument.
+The Dag Runs created externally to the scheduler get associated with the trigger's timestamp and are displayed
+in the UI alongside scheduled Dag runs. The logical date passed inside the Dag can be specified using the ``-e`` argument.
The default is the current date in the UTC timezone.
-In addition, you can also manually trigger a DAG Run using the web UI (tab **Dags** -> column **Links** -> button **Trigger Dag**)
+In addition, you can also manually trigger a Dag Run using the web UI (tab **Dags** -> column **Links** -> button **Trigger Dag**)
.. _dagrun:parameters:
-Passing Parameters when triggering dags
-------------------------------------------
+Passing Parameters when triggering Dags
+---------------------------------------
-When triggering a DAG from the CLI, the REST API or the UI, it is possible to pass configuration for a DAG Run as
+When triggering a Dag from the CLI, the REST API or the UI, it is possible to pass configuration for a Dag Run as
a JSON blob.
-Example of a parameterized DAG:
+Example of a parameterized Dag:
.. code-block:: python
diff --git a/airflow-core/docs/core-concepts/dags.rst b/airflow-core/docs/core-concepts/dags.rst
index f2010bc558e..04551838bed 100644
--- a/airflow-core/docs/core-concepts/dags.rst
+++ b/airflow-core/docs/core-concepts/dags.rst
@@ -144,14 +144,14 @@ Chain can also do *pairwise* dependencies for lists the same size (this is diffe
.. _concepts-dag-loading:
-Loading dags
+Loading Dags
------------
-Airflow loads dags from Python source files in dag bundles. It will take each file, execute it, and then load any Dag objects from that file.
+Airflow loads Dags from Python source files in Dag bundles. It will take each file, execute it, and then load any Dag objects from that file.
-This means you can define multiple dags per Python file, or even spread one very complex Dag across multiple Python files using imports.
+This means you can define multiple Dags per Python file, or even spread one very complex Dag across multiple Python files using imports.
-Note, though, that when Airflow comes to load dags from a Python file, it will only pull any objects at the *top level* that are a Dag instance. For example, take this Dag file::
+Note, though, that when Airflow comes to load Dags from a Python file, it will only pull any objects at the *top level* that are a Dag instance. For example, take this Dag file::
dag_1 = DAG('this_dag_will_be_discovered')
@@ -164,11 +164,11 @@ While both Dag constructors get called when the file is accessed, only ``dag_1``
.. note::
- When searching for dags inside the dag bundle, Airflow only considers Python files that contain the strings ``airflow`` and ``dag`` (case-insensitively) as an optimization.
+ When searching for Dags inside the Dag bundle, Airflow only considers Python files that contain the strings ``airflow`` and ``dag`` (case-insensitively) as an optimization.
To consider all Python files instead, disable the ``DAG_DISCOVERY_SAFE_MODE`` configuration flag.
-You can also provide an ``.airflowignore`` file inside your dag bundle, or any of its subfolders, which describes patterns of files for the loader to ignore. It covers the directory it's in plus all subfolders underneath it. See :ref:`.airflowignore <concepts:airflowignore>` below for details of the file syntax.
+You can also provide an ``.airflowignore`` file inside your Dag bundle, or any of its subfolders, which describes patterns of files for the loader to ignore. It covers the directory it's in plus all subfolders underneath it. See :ref:`.airflowignore <concepts:airflowignore>` below for details of the file syntax.
In the case where the ``.airflowignore`` does not meet your needs and you want a more flexible way to control if a python file needs to be parsed by airflow, you can plug your callable by setting ``might_contain_dag_callable`` in the config file.
Note, this callable will replace the default Airflow heuristic, i.e. checking if the strings ``airflow`` and ``dag`` (case-insensitively) are present in the python file.
@@ -176,13 +176,13 @@ Note, this callable will replace the default Airflow heuristic, i.e. checking if
.. code-block::
def might_contain_dag(file_path: str, zip_file: zipfile.ZipFile | None = None) -> bool:
- # Your logic to check if there are dags defined in the file_path
+ # Your logic to check if there are Dags defined in the file_path
# Return True if the file_path needs to be parsed, otherwise False
.. _concepts-dag-run:
-Running dags
+Running Dags
------------
Dags will run in one of two ways:
@@ -290,18 +290,18 @@ The Dag decorator
.. versionadded:: 2.0
-As well as the more traditional ways of declaring a single Dag using a context manager or the ``DAG()`` constructor, you can also decorate a function with ``@dag`` to turn it into a Dag generator function:
+As well as the more traditional ways of declaring a single Dag using a context manager or the ``Dag()`` constructor, you can also decorate a function with ``@dag`` to turn it into a Dag generator function:
.. exampleinclude:: /../src/airflow/example_dags/example_dag_decorator.py
:language: python
:start-after: [START dag_decorator_usage]
:end-before: [END dag_decorator_usage]
-As well as being a new way of making dags cleanly, the decorator also sets up any parameters you have in your function as Dag parameters, letting you :ref:`set those parameters when triggering the Dag <dagrun:parameters>`. You can then access the parameters from Python code, or from ``{{ context.params }}`` inside a :ref:`Jinja template <concepts:jinja-templating>`.
+As well as being a new way of making Dags cleanly, the decorator also sets up any parameters you have in your function as Dag parameters, letting you :ref:`set those parameters when triggering the Dag <dagrun:parameters>`. You can then access the parameters from Python code, or from ``{{ context.params }}`` inside a :ref:`Jinja template <concepts:jinja-templating>`.
.. note::
- Airflow will only load dags that :ref:`appear in the top level <concepts-dag-loading>` of a Dag file. This means you cannot just declare a function with ``@dag`` - you must also call it at least once in your Dag file and assign it to a top-level object, as you can see in the example above.
+ Airflow will only load Dags that :ref:`appear in the top level <concepts-dag-loading>` of a Dag file. This means you cannot just declare a function with ``@dag`` - you must also call it at least once in your Dag file and assign it to a top-level object, as you can see in the example above.
.. _concepts-control-flow:
@@ -314,7 +314,7 @@ By default, a Dag will only run a Task when all the Tasks it depends on are succ
* :ref:`concepts:branching` - select which Task to move onto based on a condition
* :ref:`concepts:trigger-rules` - set the conditions under which a Dag will run a task
* :doc:`/howto/setup-and-teardown` - define setup and teardown relationships
-* :ref:`concepts:latest-only` - a special form of branching that only runs on dags running against the present
+* :ref:`concepts:latest-only` - a special form of branching that only runs on Dags running against the present
* :ref:`concepts:depends-on-past` - tasks can depend on themselves *from a previous run*
.. _concepts:branching:
@@ -507,7 +507,7 @@ In data workflows it's common to create a resource (such as a compute resource),
Please see main article :doc:`/howto/setup-and-teardown` for details on how to use this feature.
-Dynamic dags
+Dynamic Dags
------------
Since a Dag is defined by Python code, there is no need for it to be purely declarative; you are free to use loops, functions, and more to define your Dag.
@@ -526,7 +526,7 @@ For example, here is a Dag that uses a ``for`` loop to define some tasks:
t = EmptyOperator(task_id=option)
first >> t >> last
-In general, we advise you to try and keep the *topology* (the layout) of your Dag tasks relatively stable; dynamic dags are usually better used for dynamically loading configuration options or changing operator options.
+In general, we advise you to try and keep the *topology* (the layout) of your Dag tasks relatively stable; dynamic Dags are usually better used for dynamically loading configuration options or changing operator options.
Dag Visualization
@@ -539,7 +539,7 @@ If you want to see a visual representation of a Dag, you have two options:
We generally recommend you use the Graph view, as it will also show you the state of all the :ref:`Task Instances <concepts:task-instances>` within any Dag Run you select.
-Of course, as you develop out your dags they are going to get increasingly complex, so we provide a few ways to modify these Dag views to make them easier to understand.
+Of course, as you develop out your Dags they are going to get increasingly complex, so we provide a few ways to modify these Dag views to make them easier to understand.
.. _concepts:taskgroups:
@@ -645,7 +645,7 @@ Here's an example Dag which illustrates labeling different branches:
Dag & Task Documentation
------------------------
-It's possible to add documentation or notes to your dags & task objects that are visible in the web interface ("Graph" & "Tree" for dags, "Task Instance Details" for tasks).
+It's possible to add documentation or notes to your Dags & task objects that are visible in the web interface ("Graph" & "Tree" for Dags, "Task Instance Details" for tasks).
There are a set of special task attributes that get rendered as rich content if defined:
@@ -659,7 +659,7 @@ doc_md markdown
doc_rst reStructuredText
========== ================
-Please note that for dags, ``doc_md`` is the only attribute interpreted. For dags it can contain a string or the reference to a markdown file. Markdown files are recognized by str ending in ``.md``.
+Please note that for Dags, ``doc_md`` is the only attribute interpreted. For Dags it can contain a string or the reference to a markdown file. Markdown files are recognized by str ending in ``.md``.
If a relative path is supplied it will be loaded from the path relative to which the Airflow Scheduler or Dag parser was started. If the markdown file does not exist, the passed filename will be used as text, no exception will be displayed. Note that the markdown file is loaded during Dag parsing, changes to the markdown content take one Dag parsing cycle to have changes be displayed.
This is especially useful if your tasks are built dynamically from configuration files, as it allows you to expose the configuration that led to the related tasks in Airflow:
@@ -687,19 +687,19 @@ This is especially useful if your tasks are built dynamically from configuration
"""
-Packaging dags
+Packaging Dags
--------------
-While simpler dags are usually only in a single Python file, it is not uncommon that more complex dags might be spread across multiple files and have dependencies that should be shipped with them ("vendored").
+While simpler Dags are usually only in a single Python file, it is not uncommon that more complex Dags might be spread across multiple files and have dependencies that should be shipped with them ("vendored").
-You can either do this all inside of the dag bundle, with a standard filesystem layout, or you can package the Dag and all of its Python files up as a single zip file. For instance, you could ship two dags along with a dependency they need as a zip file with the following contents::
+You can either do this all inside of the Dag bundle, with a standard filesystem layout, or you can package the Dag and all of its Python files up as a single zip file. For instance, you could ship two Dags along with a dependency they need as a zip file with the following contents::
my_dag1.py
my_dag2.py
package1/__init__.py
package1/functions.py
-Note that packaged dags come with some caveats:
+Note that packaged Dags come with some caveats:
* They cannot be used if you have pickling enabled for serialization
* They cannot contain compiled libraries (e.g. ``libz.so``), only pure Python
@@ -712,7 +712,7 @@ In general, if you have a complex set of compiled dependencies and modules, you
``.airflowignore``
------------------
-An ``.airflowignore`` file specifies the directories or files in the dag bundle
+An ``.airflowignore`` file specifies the directories or files in the Dag bundle
or ``PLUGINS_FOLDER`` that Airflow should intentionally ignore. Airflow supports
two syntax flavors for patterns in the file, as specified by the ``DAG_IGNORE_FILE_SYNTAX``
configuration parameter (*added in Airflow 2.3*): ``regexp`` and ``glob``.
@@ -741,7 +741,7 @@ match any of the patterns would be ignored (under the hood, ``Pattern.search()``
to match the pattern). Use the ``#`` character to indicate a comment; all characters
on lines starting with ``#`` will be ignored.
-The ``.airflowignore`` file should be put in your dag bundle. For example, you can prepare
+The ``.airflowignore`` file should be put in your Dag bundle. For example, you can prepare
a ``.airflowignore`` file with the ``glob`` syntax
.. code-block::
@@ -750,12 +750,12 @@ a ``.airflowignore`` file with the ``glob`` syntax
tenant_[0-9]*
Then files like ``project_a_dag_1.py``, ``TESTING_project_a.py``, ``tenant_1.py``,
-``project_a/dag_1.py``, and ``tenant_1/dag_1.py`` in your dag bundle would be ignored
+``project_a/dag_1.py``, and ``tenant_1/dag_1.py`` in your Dag bundle would be ignored
(If a directory's name matches any of the patterns, this directory and all its subfolders
would not be scanned by Airflow at all. This improves efficiency of Dag finding).
The scope of a ``.airflowignore`` file is the directory it is in plus all its subfolders.
-You can also prepare ``.airflowignore`` file for a subfolder in your dag bundle and it
+You can also prepare ``.airflowignore`` file for a subfolder in your Dag bundle and it
would only be applicable for that subfolder.
Dag Dependencies
@@ -764,7 +764,7 @@ Dag Dependencies
*Added in Airflow 2.1*
While dependencies between tasks in a Dag are explicitly defined through upstream and downstream
-relationships, dependencies between dags are a bit more complex. In general, there are two ways
+relationships, dependencies between Dags are a bit more complex. In general, there are two ways
in which one Dag can depend on another:
- triggering - :class:`~airflow.providers.standard.operators.trigger_dagrun.TriggerDagRunOperator`
@@ -772,7 +772,7 @@ in which one Dag can depend on another:
Additional difficulty is that one Dag could wait for or trigger several runs of the other Dag
with different data intervals. The **Dag Dependencies** view
-``Menu -> Browse -> Dag Dependencies`` helps visualize dependencies between dags. The dependencies
+``Menu -> Browse -> Dag Dependencies`` helps visualize dependencies between Dags. The dependencies
are calculated by the scheduler during Dag serialization and the webserver uses them to build
the dependency graph.
@@ -782,13 +782,13 @@ The dependency detector is configurable, so you can implement your own logic dif
Dag pausing, deactivation and deletion
--------------------------------------
-The dags have several states when it comes to being "not running". Dags can be paused, deactivated
+The Dags have several states when it comes to being "not running". Dags can be paused, deactivated
and finally all metadata for the Dag can be deleted.
Dag can be paused via UI when it is present in the ``DAGS_FOLDER``, and scheduler stored it in
the database, but the user chose to disable it via the UI. The "pause" and "unpause" actions are available
-via UI and API. Paused Dag is not scheduled by the Scheduler, but you can trigger them via UI for
-manual runs. In the UI, you can see paused dags (in ``Paused`` tab). The dags that are un-paused
+via UI and API. Paused Dags are not scheduled by the Scheduler, but you can trigger them via UI for
+manual runs. In the UI, you can see paused Dags (in ``Paused`` tab). The Dags that are un-paused
can be found in the ``Active`` tab. When a Dag is paused, any running tasks are allowed to complete and all
downstream tasks are put in to a state of "Scheduled". When the Dag is unpaused, any "scheduled" tasks will
begin running according to the Dag logic. Dags with no "scheduled" tasks will begin running according to
@@ -797,14 +797,14 @@ their schedule.
Dags can be deactivated (do not confuse it with ``Active`` tag in the UI) by removing them from the
``DAGS_FOLDER``. When scheduler parses the ``DAGS_FOLDER`` and misses the Dag that it had seen
before and stored in the database it will set is as deactivated. The metadata and history of the
-Dag is kept for deactivated dags and when the dag is re-added to the ``DAGS_FOLDER`` it will be again
+Dag is kept for deactivated Dags and when the Dag is re-added to the ``DAGS_FOLDER`` it will be again
activated and history will be visible. You cannot activate/deactivate Dag via UI or API, this
can only be done by removing files from the ``DAGS_FOLDER``. Once again - no data for historical runs of the
Dag are lost when it is deactivated by the scheduler. Note that the ``Active`` tab in Airflow UI
-refers to dags that are not both ``Activated`` and ``Not paused`` so this might initially be a
+refers to Dags that are not both ``Activated`` and ``Not paused`` so this might initially be a
little confusing.
-You can't see the deactivated dags in the UI - you can sometimes see the historical runs, but when you try to
+You can't see the deactivated Dags in the UI - you can sometimes see the historical runs, but when you try to
see the information about those you will see the error that the Dag is missing.
You can also delete the Dag metadata from the metadata database using UI or API, but it does not
@@ -822,7 +822,7 @@ it in three steps:
Dag Auto-pausing (Experimental)
-------------------------------
Dags can be configured to be auto-paused as well.
-There is a Airflow configuration which allows for automatically disabling of a dag
+There is a Airflow configuration which allows for automatically disabling of a Dag
if it fails for ``N`` number of times consecutively.
- :ref:`config:core__max_consecutive_failed_dag_runs_per_dag`
diff --git a/airflow-core/docs/core-concepts/debug.rst b/airflow-core/docs/core-concepts/debug.rst
index 097754b6fd1..dd627bb3558 100644
--- a/airflow-core/docs/core-concepts/debug.rst
+++ b/airflow-core/docs/core-concepts/debug.rst
@@ -17,32 +17,32 @@
.. _concepts:debugging:
-Debugging Airflow dags
+Debugging Airflow Dags
======================
-Testing dags with dag.test()
-*****************************
+Testing Dags with dag.test()
+-----------------------------
-To debug dags in an IDE, you can set up the ``dag.test`` command in your dag file and run through your DAG in a single
+To debug Dags in an IDE, you can set up the ``dag.test`` command in your Dag file and run through your Dag in a single
serialized python process.
This approach can be used with any supported database (including a local SQLite database) and will
*fail fast* as all tasks run in a single process.
-To set up ``dag.test``, add these two lines to the bottom of your dag file:
+To set up ``dag.test``, add these two lines to the bottom of your Dag file:
.. code-block:: python
if __name__ == "__main__":
dag.test()
-and that's it! You can add optional arguments to fine tune the testing but otherwise you can run or debug dags as
+and that's it! You can add optional arguments to fine tune the testing but otherwise you can run or debug Dags as
needed. Here are some examples of arguments:
-* ``execution_date`` if you want to test argument-specific DAG runs
-* ``use_executor`` if you want to test the DAG using an executor. By default ``dag.test`` runs the DAG without an
+* ``execution_date`` if you want to test argument-specific Dag runs
+* ``use_executor`` if you want to test the Dag using an executor. By default ``dag.test`` runs the Dag without an
executor, it just runs all the tasks locally.
- By providing this argument, the DAG is executed using the executor configured in the Airflow environment.
+ By providing this argument, the Dag is executed using the executor configured in the Airflow environment.
Conditionally skipping tasks
----------------------------
@@ -50,7 +50,7 @@ Conditionally skipping tasks
If you don't wish to execute some subset of tasks in your local environment (e.g. dependency check sensors or cleanup steps),
you can automatically mark them successful supplying a pattern matching their ``task_id`` in the ``mark_success_pattern`` argument.
-In the following example, testing the dag won't wait for either of the upstream dags to complete. Instead, testing data
+In the following example, testing the Dag won't wait for either of the upstream Dags to complete. Instead, testing data
is manually ingested. The cleanup step is also skipped, making the intermediate csv is available for inspection.
.. code-block:: python
@@ -70,17 +70,17 @@ is manually ingested. The cleanup step is also skipped, making the intermediate
print(f"Intermediate csv: {run.get_task_instance('collect_stats').xcom_pull(task_id='collect_stats')}")
-Debugging Airflow dags on the command line
-******************************************
+Debugging Airflow Dags on the command line
+==========================================
-With the same two line addition as mentioned in the above section, you can now easily debug a DAG using pdb as well.
-Run ``python -m pdb <path to dag file>.py`` for an interactive debugging experience on the command line.
+With the same two line addition as mentioned in the above section, you can now easily debug a Dag using pdb as well.
+Run ``python -m pdb <path to Dag file>.py`` for an interactive debugging experience on the command line.
.. code-block:: bash
root@ef2c84ad4856:/opt/airflow# python -m pdb providers/standard/src/airflow/providers/standard/example_dags/example_bash_operator.py
> /opt/airflow/providers/standard/src/airflow/providers/standard/example_dags/example_bash_operator.py(18)<module>()
- -> """Example DAG demonstrating the usage of the BashOperator."""
+ -> """Example Dag demonstrating the usage of the BashOperator."""
(Pdb) b 45
Breakpoint 1 at /opt/airflow/providers/standard/src/airflow/providers/standard/example_dags/example_bash_operator.py:45
(Pdb) c
@@ -91,11 +91,11 @@ Run ``python -m pdb <path to dag file>.py`` for an interactive debugging experie
**IDE setup steps:**
-1. Add ``main`` block at the end of your DAG file to make it runnable.
+1. Add ``main`` block at the end of your Dag file to make it runnable.
.. code-block:: python
if __name__ == "__main__":
dag.test()
-2. Run / debug the DAG file.
+2. Run / debug the Dag file.
diff --git a/airflow-core/docs/core-concepts/executor/index.rst b/airflow-core/docs/core-concepts/executor/index.rst
index ef488f4d9f8..7a4b5e88fe3 100644
--- a/airflow-core/docs/core-concepts/executor/index.rst
+++ b/airflow-core/docs/core-concepts/executor/index.rst
@@ -125,7 +125,7 @@ Configuration
Configuring multiple executors uses the same configuration option (as described :ref:`here <executor-basic-configuration>`) as single executor use cases, leveraging a comma separated list notation to specify multiple executors.
.. note::
- The first executor in the list (either on its own or along with other executors) will behave the same as it did in pre-2.10.0 releases. In other words, this will be the default executor for the environment. Any Airflow Task or DAG that does not specify a specific executor will use this environment level executor. All other executors in the list will be initialized and ready to run tasks if specified on an Airflow Task or DAG. If you do not specify an executor in this configuration li [...]
+ The first executor in the list (either on its own or along with other executors) will behave the same as it did in pre-2.10.0 releases. In other words, this will be the default executor for the environment. Any Airflow Task or Dag that does not specify a specific executor will use this environment level executor. All other executors in the list will be initialized and ready to run tasks if specified on an Airflow Task or Dag. If you do not specify an executor in this configuration li [...]
Some examples of valid multiple executor configuration:
@@ -148,7 +148,7 @@ Some examples of valid multiple executor configuration:
.. note::
Using two instances of the _same_ executor class is not currently supported.
-To make it easier to specify executors on tasks and dags, executor configuration now supports aliases. You may then use this alias to refer to the executor in your dags (see below).
+To make it easier to specify executors on tasks and Dags, executor configuration now supports aliases. You may then use this alias to refer to the executor in your Dags (see below).
.. code-block:: ini
@@ -156,9 +156,9 @@ To make it easier to specify executors on tasks and dags, executor configuration
executor = LocalExecutor,ShortName:my.custom.module.ExecutorClass
.. note::
- If a DAG specifies a task to use an executor that is not configured, the DAG will fail to parse and a warning dialog will be shown in the Airflow UI. Please ensure that all executors you wish to use are specified in Airflow configuration on *any* host/container that is running an Airflow component (scheduler, workers, etc).
+ If a Dag specifies a task to use an executor that is not configured, the Dag will fail to parse and a warning dialog will be shown in the Airflow UI. Please ensure that all executors you wish to use are specified in Airflow configuration on *any* host/container that is running an Airflow component (scheduler, workers, etc).
-Writing dags and tasks
+Writing Dags and Tasks
^^^^^^^^^^^^^^^^^^^^^^
To specify an executor for a task, make use of the executor parameter on Airflow Operators:
@@ -177,7 +177,7 @@ To specify an executor for a task, make use of the executor parameter on Airflow
def hello_world():
print("hello world!")
-To specify an executor for an entire DAG, make use of the existing Airflow mechanism of default arguments. All tasks in the DAG will then use the specified executor (unless explicitly overridden by a specific task):
+To specify an executor for an entire Dag, make use of the existing Airflow mechanism of default arguments. All tasks in the Dag will then use the specified executor (unless explicitly overridden by a specific task):
.. code-block:: python
@@ -191,14 +191,14 @@ To specify an executor for an entire DAG, make use of the existing Airflow mecha
with DAG(
dag_id="hello_worlds",
- default_args={"executor": "LocalExecutor"}, # Applies to all tasks in the DAG
+ default_args={"executor": "LocalExecutor"}, # Applies to all tasks in the Dag
) as dag:
# All tasks will use the executor from default args automatically
hw = hello_world()
hw_again = hello_world_again()
.. note::
- Tasks store the executor they were configured to run on in the Airflow database. Changes are reflected after each parsing of a DAG.
+ Tasks store the executor they were configured to run on in the Airflow database. Changes are reflected after each parsing of a Dag.
Monitoring
^^^^^^^^^^
@@ -307,7 +307,7 @@ Compatibility Attributes
The ``BaseExecutor`` class interface contains a set of attributes that Airflow core code uses to check the features that your executor is compatible with. When writing your own Airflow executor be sure to set these correctly for your use case. Each attribute is simply a boolean to enable/disable a feature or indicate that a feature is supported/unsupported by the executor:
-* ``supports_pickling``: Whether or not the executor supports reading pickled dags from the Database before execution (rather than reading the DAG definition from the file system).
+* ``supports_pickling``: Whether or not the executor supports reading pickled Dags from the Database before execution (rather than reading the Dag definition from the file system).
* ``supports_sentry``: Whether or not the executor supports `Sentry <https://sentry.io>`_.
* ``is_local``: Whether or not the executor is remote or local. See the `Executor Types`_ section above.
diff --git a/airflow-core/docs/core-concepts/objectstorage.rst b/airflow-core/docs/core-concepts/objectstorage.rst
index 8c79dab2b22..4478980aeda 100644
--- a/airflow-core/docs/core-concepts/objectstorage.rst
+++ b/airflow-core/docs/core-concepts/objectstorage.rst
@@ -31,7 +31,7 @@ of object-name => data. To enable remote access, operations on objects are usual
(slow) HTTP REST operations.
Airflow provides a generic abstraction on top of object stores, like s3, gcs, and azure blob storage.
-This abstraction allows you to use a variety of object storage systems in your dags without having to
+This abstraction allows you to use a variety of object storage systems in your Dags without having to
change your code to deal with every different object storage system. In addition, it allows you to use
most of the standard Python modules, like ``shutil``, that can work with file-like objects.
@@ -58,7 +58,7 @@ operations that a real file system does. Key differences are:
Airflow relies on `fsspec <https://filesystem-spec.readthedocs.io/en/latest/>`_ to provide a consistent
experience across different object storage systems. It implements local file caching to speed up access.
-However, you should be aware of the limitations of object storage when designing your dags.
+However, you should be aware of the limitations of object storage when designing your Dags.
.. _concepts:basic-use:
@@ -163,7 +163,7 @@ would do the following:
.. note::
- To reuse the registration across tasks, make sure to attach the backend at the top-level of your DAG.
+ To reuse the registration across tasks, make sure to attach the backend at the top-level of your Dag.
Otherwise, the backend will not be available across multiple tasks.
diff --git a/airflow-core/docs/core-concepts/operators.rst b/airflow-core/docs/core-concepts/operators.rst
index 596b10ae169..256613537b4 100644
--- a/airflow-core/docs/core-concepts/operators.rst
+++ b/airflow-core/docs/core-concepts/operators.rst
@@ -18,7 +18,7 @@
Operators
=========
-An Operator is conceptually a template for a predefined :doc:`Task <tasks>`, that you can just define declaratively inside your DAG::
+An Operator is conceptually a template for a predefined :doc:`Task <tasks>`, that you can just define declaratively inside your Dag::
with DAG("my-dag") as dag:
ping = HttpOperator(endpoint="http://example.com/update/")
@@ -57,7 +57,7 @@ and transfers in our
Inside Airflow's code, we often mix the concepts of :doc:`tasks` and Operators, and they are mostly
interchangeable. However, when we talk about a *Task*, we mean the generic "unit of execution" of a
- DAG; when we talk about an *Operator*, we mean a reusable, pre-made Task template whose logic
+ Dag; when we talk about an *Operator*, we mean a reusable, pre-made Task template whose logic
is all done for you and that just needs some arguments.
@@ -162,7 +162,7 @@ Deep nested fields can also be substituted, as long as all intermediate fields a
)
-You can pass custom options to the Jinja ``Environment`` when creating your DAG. One common usage is to avoid Jinja from dropping a trailing newline from a template string:
+You can pass custom options to the Jinja ``Environment`` when creating your Dag. One common usage is to avoid Jinja from dropping a trailing newline from a template string:
.. code-block:: python
@@ -176,7 +176,7 @@ You can pass custom options to the Jinja ``Environment`` when creating your DAG.
See the `Jinja documentation <https://jinja.palletsprojects.com/en/2.11.x/api/#jinja2.Environment>`_ to find all available options.
-Some operators will also consider strings ending in specific suffixes (defined in ``template_ext``) to be references to files when rendering fields. This can be useful for loading scripts or queries directly from files rather than including them into DAG code.
+Some operators will also consider strings ending in specific suffixes (defined in ``template_ext``) to be references to files when rendering fields. This can be useful for loading scripts or queries directly from files rather than including them into Dag code.
For example, consider a BashOperator which runs a multi-line bash script, this will load the file at ``script.sh`` and use its contents as the value for ``bash_command``:
@@ -187,7 +187,7 @@ For example, consider a BashOperator which runs a multi-line bash script, this w
bash_command="script.sh",
)
-By default, paths provided in this way should be provided relative to the DAG's folder (as this is the default Jinja template search path), but additional paths can be added by setting the ``template_searchpath`` arg on the DAG.
+By default, paths provided in this way should be provided relative to the Dag's folder (as this is the default Jinja template search path), but additional paths can be added by setting the ``template_searchpath`` arg on the Dag.
In some cases, you may want to exclude a string from templating and use it directly. Consider the following task:
@@ -273,7 +273,7 @@ There are two solutions if we want to get the actual dict instead. The first is
python_callable=transform,
)
-Alternatively, Jinja can also be instructed to render a native Python object. This is done by passing ``render_template_as_native_obj=True`` to the DAG. This makes Airflow use `NativeEnvironment <https://jinja.palletsprojects.com/en/2.11.x/nativetypes/>`_ instead of the default ``SandboxedEnvironment``:
+Alternatively, Jinja can also be instructed to render a native Python object. This is done by passing ``render_template_as_native_obj=True`` to the Dag. This makes Airflow use `NativeEnvironment <https://jinja.palletsprojects.com/en/2.11.x/nativetypes/>`_ instead of the default ``SandboxedEnvironment``:
.. code-block:: python
@@ -296,7 +296,7 @@ Alternatively, Jinja can also be instructed to render a native Python object. Th
Reserved params keyword
-----------------------
-In Apache Airflow 2.2.0 ``params`` variable is used during DAG serialization. Please do not use that name in third party operators.
+In Apache Airflow 2.2.0 ``params`` variable is used during Dag serialization. Please do not use that name in third party operators.
If you upgrade your environment and get the following error:
.. code-block::
@@ -330,4 +330,4 @@ If you need to include a Jinja template expression (e.g., ``{{ ds }}``) literall
dag=dag,
)
-This ensures the f-string processing results in a string containing the literal double braces required by Jinja, which Airflow can then template correctly before execution. Failure to do this is a common issue for beginners and can lead to errors during DAG parsing or unexpected behavior at runtime when the templating does not occur as expected.
+This ensures the f-string processing results in a string containing the literal double braces required by Jinja, which Airflow can then template correctly before execution. Failure to do this is a common issue for beginners and can lead to errors during Dag parsing or unexpected behavior at runtime when the templating does not occur as expected.
diff --git a/airflow-core/docs/core-concepts/overview.rst b/airflow-core/docs/core-concepts/overview.rst
index 96eda6bdff1..9b80ff331b4 100644
--- a/airflow-core/docs/core-concepts/overview.rst
+++ b/airflow-core/docs/core-concepts/overview.rst
@@ -19,13 +19,13 @@ Architecture Overview
=====================
Airflow is a platform that lets you build and run *workflows*. A workflow is represented as a
-:doc:`DAG <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called
+:doc:`Dag <dags>` (a Directed Acyclic Graph), and contains individual pieces of work called
:doc:`tasks`, arranged with dependencies and data flows taken into account.
.. image:: ../img/ui-light/edge_label_example.png
- :alt: An example Airflow DAG, rendered in Graph
+ :alt: An example Airflow Dag, rendered in Graph
-A DAG specifies the dependencies between tasks, which defines the order in which to execute the tasks.
+A Dag specifies the dependencies between tasks, which defines the order in which to execute the tasks.
Tasks describe what to do, be it fetching data, running analysis, triggering other systems, or more.
Airflow itself is agnostic to what you're running - it will happily orchestrate and run anything,
@@ -49,17 +49,18 @@ A minimal Airflow installation consists of the following components:
a configuration property of the *scheduler*, not a separate component and runs within the scheduler
process. There are several executors available out of the box, and you can also write your own.
-* A *dag processor*, which parses DAG files and serializes them into the
- *metadata database*. More about processing DAG files can be found in
+* A *Dag processor*, which parses Dag files and serializes them into the
+ *metadata database*. More about processing Dag files can be found in
:doc:`/administration-and-deployment/dagfile-processing`
* A *webserver*, which presents a handy user interface to inspect, trigger and debug the behaviour of
- dags and tasks.
+ Dags and tasks.
-* A folder of *DAG files*, which is read by the *scheduler* to figure out what tasks to run and when to
+* A folder of *Dag files*, which is read by the *scheduler* to figure out what tasks to run and when to
run them.
-* A *metadata database*, which Airflow components use to store state of workflows and tasks.
+* A *metadata database*, usually PostgreSQL or MySQL, which stores the state of tasks, Dags, variables,
+ Dags and tasks.
Setting up a metadata database is described in :doc:`/howto/set-up-database` and is required for
Airflow to work.
@@ -79,7 +80,7 @@ performance in your Airflow:
found in :doc:`/authoring-and-scheduling/deferring`.
* Optional folder of *plugins*. Plugins are a way to extend Airflow's functionality (similar to installed
- packages). Plugins are read by the *scheduler*, *dag processor*, *triggerer* and *webserver*. More about
+ packages). Plugins are read by the *scheduler*, *Dag processor*, *triggerer* and *webserver*. More about
plugins can be found in :doc:`/administration-and-deployment/plugins`.
Deploying Airflow components
@@ -96,8 +97,8 @@ environment - where various components can run on different machines, with diffe
and can be scaled by running multiple instances of the components above.
The separation of components also allow for increased security, by isolating the components from each other
-and by allowing to perform different tasks. For example separating *dag processor* from *scheduler*
-allows to make sure that the *scheduler* does not have access to the *DAG files* and cannot execute
+and by allowing to perform different tasks. For example separating *Dag processor* from *scheduler*
+allows to make sure that the *scheduler* does not have access to the *Dag files* and cannot execute
code provided by *Dag author*.
Also while single person can run and manage Airflow installation, Airflow Deployment in more complex
@@ -106,8 +107,8 @@ an important aspect of secure Airflow deployment. The roles are described in det
:doc:`/security/security_model` and generally speaking include:
* Deployment Manager - a person that installs and configures Airflow and manages the deployment
-* Dag author - a person that writes dags and submits them to Airflow
-* Operations User - a person that triggers dags and tasks and monitors their execution
+* Dag author - a person that writes Dags and submits them to Airflow
+* Operations User - a person that triggers Dags and tasks and monitors their execution
Architecture Diagrams
---------------------
@@ -118,7 +119,7 @@ finally with more isolated security perimeters.
The meaning of the different connection types in the diagrams below is as follows:
-* **brown solid lines** represent *DAG files* submission and synchronization
+* **brown solid lines** represent *Dag files* submission and synchronization
* **blue solid lines** represent deploying and accessing *installed packages* and *plugins*
* **black dashed lines** represent control flow of workers by the *scheduler* (via executor)
* **black solid lines** represent accessing the UI to manage execution of the workflows
@@ -128,14 +129,14 @@ The meaning of the different connection types in the diagrams below is as follow
..
TODO AIP-66 / AIP-72: These example architectures and diagrams need to be updated to reflect AF3 changes
- like bundles, required DAG processor, execution api, etc.
+ like bundles, required Dag processor, execution api, etc.
Basic Airflow deployment
........................
This is the simplest deployment of Airflow, usually operated and managed on a single
machine. Such a deployment usually uses the LocalExecutor, where the *scheduler* and the *workers* are in
-the same Python process and the *DAG files* are read directly from the local filesystem by the *scheduler*.
+the same Python process and the *Dag files* are read directly from the local filesystem by the *scheduler*.
The *webserver* runs on the same machine as the *scheduler*. There is no *triggerer* component, which
means that task deferral is not possible.
@@ -157,26 +158,26 @@ and where various roles of users are introduced - *Deployment Manager*, **Dag au
**Operations User**. You can read more about those various roles in the :doc:`/security/security_model`.
In the case of a distributed deployment, it is important to consider the security aspects of the components.
-The *webserver* does not have access to the *DAG files* directly. The code in the ``Code`` tab of the
+The *webserver* does not have access to the *Dag files* directly. The code in the ``Code`` tab of the
UI is read from the *metadata database*. The *webserver* cannot execute any code submitted by the
**Dag author**. It can only execute code that is installed as an *installed package* or *plugin* by
the **Deployment Manager**. The **Operations User** only has access to the UI and can only trigger
-dags and tasks, but cannot author dags.
+Dags and tasks, but cannot author Dags.
-The *DAG files* need to be synchronized between all the components that use them - *scheduler*,
-*triggerer* and *workers*. The *DAG files* can be synchronized by various mechanisms - typical
-ways how dags can be synchronized are described in :doc:`helm-chart:manage-dag-files` of our
+The *Dag files* need to be synchronized between all the components that use them - *scheduler*,
+*triggerer* and *workers*. The *Dag files* can be synchronized by various mechanisms - typical
+ways how Dags can be synchronized are described in :doc:`helm-chart:manage-dag-files` of our
Helm Chart documentation. Helm chart is one of the ways how to deploy Airflow in K8S cluster.
.. image:: ../img/diagram_distributed_airflow_architecture.png
.. _overview-separate-dag-processing-airflow-architecture:
-Separate DAG processing architecture
+Separate Dag processing architecture
....................................
In a more complex installation where security and isolation are important, you'll also see the
-standalone *dag processor* component that allows to separate *scheduler* from accessing *DAG files*.
+standalone *Dag processor* component that allows to separate *scheduler* from accessing *Dag files*.
This is suitable if the deployment focus is on isolation between parsed tasks. While Airflow does not yet
support full multi-tenant features, it can be used to make sure that **Dag author** provided code is never
executed in the context of the scheduler.
@@ -185,10 +186,10 @@ executed in the context of the scheduler.
.. note::
- When DAG file is changed there can be cases where the scheduler and the worker will see different
- versions of the DAG until both components catch up. You can avoid the issue by making sure dag is
+ When Dag file is changed there can be cases where the scheduler and the worker will see different
+ versions of the Dag until both components catch up. You can avoid the issue by making sure Dag is
deactivated during deployment and reactivate once finished. If needed, the cadence of sync and scan
- of DAG folder can be configured. Please make sure you really know what you are doing if you change
+ of Dag folder can be configured. Please make sure you really know what you are doing if you change
the configurations.
.. _overview:workloads:
@@ -196,15 +197,15 @@ executed in the context of the scheduler.
Workloads
---------
-A DAG runs through a series of :doc:`tasks`, and there are three common types of task you will see:
+A Dag runs through a series of :doc:`tasks`, and there are three common types of task you will see:
-* :doc:`operators`, predefined tasks that you can string together quickly to build most parts of your dags.
+* :doc:`operators`, predefined tasks that you can string together quickly to build most parts of your Dags.
* :doc:`sensors`, a special subclass of Operators which are entirely about waiting for an external event to happen.
* A :doc:`taskflow`-decorated ``@task``, which is a custom Python function packaged up as a Task.
-Internally, these are all actually subclasses of Airflow's ``BaseOperator``, and the concepts of Task and Operator are somewhat interchangeable, but it's useful to think of them as separate concepts - essentially, Operators and Sensors are *templates*, and when you call one in a DAG file, you're making a Task.
+Internally, these are all actually subclasses of Airflow's ``BaseOperator``, and the concepts of Task and Operator are somewhat interchangeable, but it's useful to think of them as separate concepts - essentially, Operators and Sensors are *templates*, and when you call one in a Dag file, you're making a Task.
Control Flow
@@ -212,7 +213,7 @@ Control Flow
:doc:`dags` are designed to be run many times, and multiple runs of them can happen in parallel. Dags are parameterized, always including an interval they are "running for" (the :ref:`data interval <data-interval>`), but with other optional parameters as well.
-:doc:`tasks` have dependencies declared on each other. You'll see this in a DAG either using the ``>>`` and ``<<`` operators::
+:doc:`tasks` have dependencies declared on each other. You'll see this in a Dag either using the ``>>`` and ``<<`` operators::
first_task >> [second_task, third_task]
fourth_task << third_task
@@ -232,17 +233,17 @@ To pass data between tasks you have three options:
* TaskFlow API automatically passes data between tasks via implicit :doc:`xcoms`
-Airflow sends out Tasks to run on Workers as space becomes available, so there's no guarantee all the tasks in your DAG will run on the same worker or the same machine.
+Airflow sends out Tasks to run on Workers as space becomes available, so there's no guarantee all the tasks in your Dag will run on the same worker or the same machine.
-As you build out your dags, they are likely to get very complex, so Airflow provides several mechanisms for making this more sustainable, example :ref:`concepts:taskgroups` let you visually group tasks in the UI.
+As you build out your Dags, they are likely to get very complex, so Airflow provides several mechanisms for making this more sustainable, example :ref:`concepts:taskgroups` let you visually group tasks in the UI.
There are also features for letting you easily pre-configure access to a central resource, like a datastore, in the form of :doc:`../authoring-and-scheduling/connections`, and for limiting concurrency, via :doc:`../administration-and-deployment/pools`.
User interface
--------------
-Airflow comes with a user interface that lets you see what dags and their tasks are doing, trigger runs of dags, view logs, and do some limited debugging and resolution of problems with your dags.
+Airflow comes with a user interface that lets you see what Dags and their tasks are doing, trigger runs of Dags, view logs, and do some limited debugging and resolution of problems with your Dags.
.. image:: ../img/ui-dark/dags.png
-It's generally the best way to see the status of your Airflow installation as a whole, as well as diving into individual dags to see their layout, the status of each task, and the logs from each task.
+It's generally the best way to see the status of your Airflow installation as a whole, as well as diving into individual Dags to see their layout, the status of each task, and the logs from each task.
diff --git a/airflow-core/docs/core-concepts/params.rst b/airflow-core/docs/core-concepts/params.rst
index ac42a527c9c..cdcd150a73b 100644
--- a/airflow-core/docs/core-concepts/params.rst
+++ b/airflow-core/docs/core-concepts/params.rst
@@ -20,16 +20,16 @@
Params
======
-Params enable you to provide runtime configuration to tasks. You can configure default Params in your DAG
-code and supply additional Params, or overwrite Param values, at runtime when you trigger a DAG.
-:class:`~airflow.sdk.definitions.param.Param` values are validated with JSON Schema. For scheduled DAG runs,
+Params enable you to provide runtime configuration to tasks. You can configure default Params in your Dag
+code and supply additional Params, or overwrite Param values, at runtime when you trigger a Dag.
+:class:`~airflow.sdk.definitions.param.Param` values are validated with JSON Schema. For scheduled Dag runs,
default :class:`~airflow.sdk.definitions.param.Param` values are used.
Also defined Params are used to render a nice UI when triggering manually.
-When you trigger a DAG manually, you can modify its Params before the dagrun starts.
-If the user-supplied values don't pass validation, Airflow shows a warning instead of creating the dagrun.
+When you trigger a Dag manually, you can modify its Params before the Dag run starts.
+If the user-supplied values don't pass validation, Airflow shows a warning instead of creating the Dag run.
-DAG-level Params
+Dag-level Params
----------------
To add Params to a :class:`~airflow.sdk.DAG`, initialize it with the ``params`` kwarg.
@@ -72,9 +72,9 @@ Use a dictionary that maps Param names to either a :class:`~airflow.sdk.definiti
.. note::
- DAG-level parameters are the default values passed on to tasks. These should not be confused with values manually
+ Dag-level parameters are the default values passed on to tasks. These should not be confused with values manually
provided through the UI form or CLI, which exist solely within the context of a :class:`~airflow.models.dagrun.DagRun`
- and a :class:`~airflow.models.taskinstance.TaskInstance`. This distinction is crucial for TaskFlow dags, which may
+ and a :class:`~airflow.models.taskinstance.TaskInstance`. This distinction is crucial for TaskFlow Dags, which may
include logic within the ``with DAG(...) as dag:`` block. In such cases, users might try to access the manually-provided
parameter values using the ``dag`` object, but this will only ever contain the default values. To ensure that the
manually-provided values are accessed, use a template variable such as ``params`` or ``ti`` within your task.
@@ -95,7 +95,7 @@ You can also add Params to individual tasks.
python_callable=print_my_int_param,
)
-Task-level params take precedence over DAG-level params, and user-supplied params (when triggering the DAG)
+Task-level params take precedence over Dag-level params, and user-supplied params (when triggering the Dag)
take precedence over task-level params.
Referencing Params in a Task
@@ -191,8 +191,8 @@ JSON Schema Validation
):
.. note::
- If ``schedule`` is defined for a DAG, params with defaults must be valid. This is validated during DAG parsing.
- If ``schedule=None`` then params are not validated during DAG parsing but before triggering a DAG.
+ If ``schedule`` is defined for a Dag, params with defaults must be valid. This is validated during Dag parsing.
+ If ``schedule=None`` then params are not validated during Dag parsing but before triggering a Dag.
This is useful in cases where the Dag author does not want to provide defaults but wants to force users provide valid parameters
at time of trigger.
@@ -206,14 +206,14 @@ Use Params to Provide a Trigger UI Form
.. versionadded:: 2.6.0
:class:`~airflow.models.dag.DAG` level params are used to render a user friendly trigger form.
-This form is provided when a user clicks on the "Trigger DAG" button.
+This form is provided when a user clicks on the "Trigger Dag" button.
-The Trigger UI Form is rendered based on the pre-defined DAG Params. If the DAG has no params defined, the trigger form is skipped.
+The Trigger UI Form is rendered based on the pre-defined Dag Params. If the Dag has no params defined, the trigger form is skipped.
The form elements can be defined with the :class:`~airflow.sdk.definitions.param.Param` class and attributes define how a form field is displayed.
The following features are supported in the Trigger UI Form:
-- Direct scalar values (boolean, int, string, lists, dicts) from top-level DAG params are auto-boxed into :class:`~airflow.sdk.definitions.param.Param` objects.
+- Direct scalar values (boolean, int, string, lists, dicts) from top-level Dag params are auto-boxed into :class:`~airflow.sdk.definitions.param.Param` objects.
From the native Python data type the ``type`` attribute is auto detected. So these simple types render to a corresponding field type.
The name of the parameter is used as label and no further validation is made, all values are treated as optional.
- If you use the :class:`~airflow.sdk.definitions.param.Param` class as definition of the parameter value, the following attributes can be added:
@@ -222,7 +222,7 @@ The following features are supported in the Trigger UI Form:
If no ``title`` is defined the parameter name/key is used instead.
- The :class:`~airflow.sdk.definitions.param.Param` attribute ``description`` is rendered below an entry field as help text in gray color.
If you want to provide special formatting or links you need to use the Param attribute
- ``description_md``. See tutorial DAG :ref:`Params UI example DAG <params-ui-tutorial>` for an example.
+ ``description_md``. See tutorial Dag :ref:`Params UI example Dag <params-ui-tutorial>` for an example.
- The :class:`~airflow.sdk.definitions.param.Param` attribute ``type`` influences how a field is rendered. The following types are supported:
.. list-table::
@@ -262,7 +262,7 @@ The following features are supported in the Trigger UI Form:
| Also see
| `further JSON Schema string type validation options <https://json-schema.org/understanding-json-schema/reference/string.html>`_
- | which are checked before DAG trigger in the backend.
+ | which are checked before Dag trigger in the backend.
- ``Param("default", type="string", maxLength=10)``
``Param(f"{datetime.date.today()}", type="string", format="date")``
@@ -282,7 +282,7 @@ The following features are supported in the Trigger UI Form:
| Also see
| `further JSON Schema numeric type validation options <https://json-schema.org/understanding-json-schema/reference/numeric.html>`_
- | which are checked before DAG trigger in the backend.
+ | which are checked before Dag trigger in the backend.
- ``Param(42, type="integer", minimum=14, multipleOf=7)``
* - ``boolean``
@@ -342,7 +342,7 @@ The following features are supported in the Trigger UI Form:
- ``Param(None, type=["null", "string"])``
- If a form field is left empty, it is passed as ``None`` value to the params dict.
-- Form fields are rendered in the order of definition of ``params`` in the DAG.
+- Form fields are rendered in the order of definition of ``params`` in the Dag.
- If you want to add sections to the Form, add the attribute ``section`` to each field. The text will be used as section label.
Fields w/o ``section`` will be rendered in the default area.
Additional sections will be collapsed per default.
@@ -354,10 +354,10 @@ The following features are supported in the Trigger UI Form:
- Fields without a "section" will be rendered in the default area. Additional sections will be collapsed by default.
.. note::
- If the field is required the default value must be valid according to the schema as well. If the DAG is defined with
+ If the field is required the default value must be valid according to the schema as well. If the Dag is defined with
``schedule=None`` the parameter value validation is made at time of trigger.
-For examples, please take a look at the two example dags provided: :ref:`Params trigger example DAG <params-trigger-ui>` and :ref:`Params UI example DAG <params-ui-tutorial>`.
+For examples, please take a look at the two example dags provided: :ref:`Params trigger example Dag <params-trigger-ui>` and :ref:`Params UI example Dag <params-ui-tutorial>`.
.. _params-trigger-ui:
.. exampleinclude:: /../src/airflow/example_dags/example_params_trigger_ui.py
@@ -407,5 +407,5 @@ Finally the fourth section shows advanced form elements.
Disabling Runtime Param Modification
------------------------------------
-The ability to update params while triggering a DAG depends on the flag ``core.dag_run_conf_overrides_params``.
+The ability to update params while triggering a Dag depends on the flag ``core.dag_run_conf_overrides_params``.
Setting this config to ``False`` will effectively turn your default params into constants.
diff --git a/airflow-core/docs/core-concepts/taskflow.rst b/airflow-core/docs/core-concepts/taskflow.rst
index 29e91eb4413..2e91649a381 100644
--- a/airflow-core/docs/core-concepts/taskflow.rst
+++ b/airflow-core/docs/core-concepts/taskflow.rst
@@ -20,9 +20,9 @@ TaskFlow
.. versionadded:: 2.0
-If you write most of your dags using plain Python code rather than Operators, then the TaskFlow API will make it much easier to author clean dags without extra boilerplate, all using the ``@task`` decorator.
+If you write most of your Dags using plain Python code rather than Operators, then the TaskFlow API will make it much easier to author clean Dags without extra boilerplate, all using the ``@task`` decorator.
-TaskFlow takes care of moving inputs and outputs between your Tasks using XComs for you, as well as automatically calculating dependencies - when you call a TaskFlow function in your DAG file, rather than executing it, you will get an object representing the XCom for the result (an ``XComArg``), that you can then use as inputs to downstream tasks or operators. For example::
+TaskFlow takes care of moving inputs and outputs between your Tasks using XComs for you, as well as automatically calculating dependencies - when you call a TaskFlow function in your Dag file, rather than executing it, you will get an object representing the XCom for the result (an ``XComArg``), that you can then use as inputs to downstream tasks or operators. For example::
from airflow.sdk import task
from airflow.providers.smtp.operators.smtp import EmailOperator
@@ -53,7 +53,7 @@ The first two are declared using TaskFlow, and automatically pass the return val
``send_email_notification`` is a more traditional Operator, but even it can use the return value of ``compose_email`` to set its parameters, and again, automatically work out that it must be *downstream* of ``compose_email``.
-You can also use a plain value or variable to call a TaskFlow function - for example, this will work as you expect (but, of course, won't run the code inside the task until the DAG is executed - the ``name`` value is persisted as a task parameter until that time)::
+You can also use a plain value or variable to call a TaskFlow function - for example, this will work as you expect (but, of course, won't run the code inside the task until the Dag is executed - the ``name`` value is persisted as a task parameter until that time)::
@task
def hello_name(name: str):
@@ -204,7 +204,7 @@ For an example of writing a Sensor using the TaskFlow API, see
History
-------
-The TaskFlow API is new as of Airflow 2.0, and you are likely to encounter dags written for previous versions of Airflow that instead use ``PythonOperator`` to achieve similar goals, albeit with a lot more code.
+The TaskFlow API is new as of Airflow 2.0, and you are likely to encounter Dags written for previous versions of Airflow that instead use ``PythonOperator`` to achieve similar goals, albeit with a lot more code.
More context around the addition and design of the TaskFlow API can be found as part of its Airflow Improvement Proposal
-`AIP-31: "TaskFlow API" for clearer/simpler DAG definition <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=148638736>`_
+`AIP-31: "TaskFlow API" for clearer/simpler Dag definition <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=148638736>`_
diff --git a/airflow-core/docs/core-concepts/tasks.rst b/airflow-core/docs/core-concepts/tasks.rst
index 507c2696a75..8d5d94d1686 100644
--- a/airflow-core/docs/core-concepts/tasks.rst
+++ b/airflow-core/docs/core-concepts/tasks.rst
@@ -22,13 +22,13 @@ A Task is the basic unit of execution in Airflow. Tasks are arranged into :doc:`
There are three basic kinds of Task:
-* :doc:`operators`, predefined task templates that you can string together quickly to build most parts of your dags.
+* :doc:`operators`, predefined task templates that you can string together quickly to build most parts of your Dags.
* :doc:`sensors`, a special subclass of Operators which are entirely about waiting for an external event to happen.
* A :doc:`taskflow`-decorated ``@task``, which is a custom Python function packaged up as a Task.
-Internally, these are all actually subclasses of Airflow's ``BaseOperator``, and the concepts of Task and Operator are somewhat interchangeable, but it's useful to think of them as separate concepts - essentially, Operators and Sensors are *templates*, and when you call one in a DAG file, you're making a Task.
+Internally, these are all actually subclasses of Airflow's ``BaseOperator``, and the concepts of Task and Operator are somewhat interchangeable, but it's useful to think of them as separate concepts - essentially, Operators and Sensors are *templates*, and when you call one in a Dag file, you're making a Task.
Relationships
@@ -63,9 +63,9 @@ Tasks don't pass information to each other by default, and run entirely independ
Task Instances
--------------
-Much in the same way that a DAG is instantiated into a :ref:`DAG Run <concepts-dag-run>` each time it runs, the tasks under a DAG are instantiated into *Task Instances*.
+Much in the same way that a Dag is instantiated into a :ref:`Dag Run <concepts-dag-run>` each time it runs, the tasks under a Dag are instantiated into *Task Instances*.
-An instance of a Task is a specific run of that task for a given DAG (and thus for a given data interval). They are also the representation of a Task that has *state*, representing what stage of the lifecycle it is in.
+An instance of a Task is a specific run of that task for a given Dag (and thus for a given data interval). They are also the representation of a Task that has *state*, representing what stage of the lifecycle it is in.
.. _concepts:task-states:
@@ -83,7 +83,7 @@ The possible states for a Task Instance are:
* ``up_for_retry``: The task failed, but has retry attempts left and will be rescheduled.
* ``up_for_reschedule``: The task is a :doc:`Sensor <sensors>` that is in ``reschedule`` mode
* ``deferred``: The task has been :doc:`deferred to a trigger <../authoring-and-scheduling/deferring>`
-* ``removed``: The task has vanished from the DAG since the run started
+* ``removed``: The task has vanished from the Dag since the run started
.. image:: /img/diagram_task_lifecycle.png
@@ -101,9 +101,9 @@ Firstly, it can have *upstream* and *downstream* tasks::
task1 >> task2 >> task3
-When a DAG runs, it will create instances for each of these tasks that are upstream/downstream of each other, but which all have the same data interval.
+When a Dag runs, it will create instances for each of these tasks that are upstream/downstream of each other, but which all have the same data interval.
-There may also be instances of the *same task*, but for different data intervals - from other runs of the same DAG. We call these *previous* and *next* - it is a different relationship to *upstream* and *downstream*!
+There may also be instances of the *same task*, but for different data intervals - from other runs of the same Dag. We call these *previous* and *next* - it is a different relationship to *upstream* and *downstream*!
.. note::
@@ -196,7 +196,7 @@ If you'd like to reproduce task instance heartbeat timeouts for development/test
export AIRFLOW__SCHEDULER__TASK_INSTANCE_HEARTBEAT_TIMEOUT_DETECTION_INTERVAL=5
-2. Have a DAG with a task that takes about 10 minutes to complete(i.e. a long-running task). For example, you could use the below DAG:
+2. Have a Dag with a task that takes about 10 minutes to complete(i.e. a long-running task). For example, you could use the below Dag:
.. code-block:: python
@@ -216,7 +216,7 @@ If you'd like to reproduce task instance heartbeat timeouts for development/test
sleep_dag()
-Run the above DAG and wait for a while. The ``TaskInstance`` will be marked failed after <task_instance_heartbeat_timeout> seconds.
+Run the above Dag and wait for a while. The ``TaskInstance`` will be marked failed after <task_instance_heartbeat_timeout> seconds.
diff --git a/airflow-core/docs/core-concepts/variables.rst b/airflow-core/docs/core-concepts/variables.rst
index 6487fd0c131..33b20aaac5a 100644
--- a/airflow-core/docs/core-concepts/variables.rst
+++ b/airflow-core/docs/core-concepts/variables.rst
@@ -57,6 +57,6 @@ You can also use them from :ref:`templates <concepts:jinja-templating>`::
Variables are **global**, and should only be used for overall configuration that covers the entire installation; to pass data from one Task/Operator to another, you should use :doc:`xcoms` instead.
-We also recommend that you try to keep most of your settings and configuration in your DAG files, so it can be versioned using source control; Variables are really only for values that are truly runtime-dependent.
+We also recommend that you try to keep most of your settings and configuration in your Dag files, so it can be versioned using source control; Variables are really only for values that are truly runtime-dependent.
For more information on setting and managing variables, see :doc:`/howto/variable`.
diff --git a/airflow-core/docs/core-concepts/xcoms.rst b/airflow-core/docs/core-concepts/xcoms.rst
index 93463a75276..aa89050261e 100644
--- a/airflow-core/docs/core-concepts/xcoms.rst
+++ b/airflow-core/docs/core-concepts/xcoms.rst
@@ -53,7 +53,7 @@ You can also use XComs in :ref:`templates <concepts:jinja-templating>`::
SELECT * FROM {{ task_instance.xcom_pull(task_ids='foo', key='table_name') }}
-XComs are a relative of :doc:`variables`, with the main difference being that XComs are per-task-instance and designed for communication within a DAG run, while Variables are global and designed for overall configuration and value sharing.
+XComs are a relative of :doc:`variables`, with the main difference being that XComs are per-task-instance and designed for communication within a Dag run, while Variables are global and designed for overall configuration and value sharing.
If you want to push multiple XComs at once you can set ``do_xcom_push`` and ``multiple_outputs`` arguments to ``True``, and then return a dictionary of values.
diff --git a/airflow-core/docs/extra-packages-ref.rst b/airflow-core/docs/extra-packages-ref.rst
index 47106fab2ed..5e225bff65a 100644
--- a/airflow-core/docs/extra-packages-ref.rst
+++ b/airflow-core/docs/extra-packages-ref.rst
@@ -88,7 +88,7 @@ python dependencies for the provided package. The same extras are available as `
+=====================+=====================================================+============================================================================+
| async | ``pip install 'apache-airflow[async]'`` | Async worker classes for Gunicorn |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
-| graphviz | ``pip install 'apache-airflow[graphviz]'`` | Graphviz renderer for converting DAG to graphical output |
+| graphviz | ``pip install 'apache-airflow[graphviz]'`` | Graphviz renderer for converting Dag to graphical output |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
| kerberos | ``pip install 'apache-airflow[kerberos]'`` | Kerberos integration for Kerberized services (Hadoop, Presto, Trino) |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
@@ -122,7 +122,7 @@ other packages that can be used by airflow or some of its providers.
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
| google-auth | ``pip install 'apache-airflow[google-auth]'`` | Google auth backend |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
-| graphviz | ``pip install 'apache-airflow[graphviz]'`` | Graphviz renderer for converting DAG to graphical output |
+| graphviz | ``pip install 'apache-airflow[graphviz]'`` | Graphviz renderer for converting Dag to graphical output |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
| ldap | ``pip install 'apache-airflow[ldap]'`` | LDAP authentication for users |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
diff --git a/airflow-core/docs/faq.rst b/airflow-core/docs/faq.rst
index f74c4194928..00673a8e5f7 100644
--- a/airflow-core/docs/faq.rst
+++ b/airflow-core/docs/faq.rst
@@ -20,7 +20,7 @@
FAQ
========
-Scheduling / DAG file parsing
+Scheduling / Dag file parsing
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Why is task not getting scheduled?
@@ -29,25 +29,25 @@ Why is task not getting scheduled?
There are very many reasons why your task might not be getting scheduled. Here are some of the common causes:
- Does your script "compile", can the Airflow engine parse it and find your
- DAG object? To test this, you can run ``airflow dags list`` and
- confirm that your DAG shows up in the list. You can also run
+ Dag object? To test this, you can run ``airflow dags list`` and
+ confirm that your Dag shows up in the list. You can also run
``airflow dags show foo_dag_id`` and confirm that your task
shows up in the graphviz format as expected. If you use the CeleryExecutor, you
may want to confirm that this works both where the scheduler runs as well
as where the worker runs.
-- Does the file containing your DAG contain the string "airflow" and "DAG" somewhere
- in the contents? When searching the DAG directory, Airflow ignores files not containing
- "airflow" and "DAG" in order to prevent the DagBag parsing from importing all python
- files collocated with user's dags.
+- Does the file containing your Dag contain the string ``airflow`` and ``DAG`` somewhere
+ in the contents? When searching the Dag directory, Airflow ignores files not containing
+ ``airflow`` and ``DAG`` in order to prevent the DagBag parsing from importing all python
+ files collocated with user's Dags.
-- Is your ``start_date`` set properly? For time-based dags, the task won't be triggered until the
+- Is your ``start_date`` set properly? For time-based Dags, the task won't be triggered until the
the first schedule interval following the start date has passed.
- Is your ``schedule`` argument set properly? The default
is one day (``datetime.timedelta(1)``). You must specify a different ``schedule``
- directly to the DAG object you instantiate, not as a ``default_param``, as task instances
- do not override their parent DAG's ``schedule``.
+ directly to the Dag object you instantiate, not as a ``default_param``, as task instances
+ do not override their parent Dag's ``schedule``.
- Is your ``start_date`` beyond where you can see it in the UI? If you
set your ``start_date`` to some time say 3 months ago, you won't be able to see
@@ -65,26 +65,26 @@ There are very many reasons why your task might not be getting scheduled. Here a
page for your task.
- Are the DagRuns you need created and active? A DagRun represents a specific
- execution of an entire DAG and has a state (running, success, failed, ...).
+ execution of an entire Dag and has a state (running, success, failed, ...).
The scheduler creates new DagRun as it moves forward, but never goes back
in time to create new ones. The scheduler only evaluates ``running`` DagRuns
to see what task instances it can trigger. Note that clearing tasks
instances (from the UI or CLI) does set the state of a DagRun back to
running. You can bulk view the list of DagRuns and alter states by clicking
- on the schedule tag for a DAG.
+ on the schedule tag for a Dag.
-- Is the ``concurrency`` parameter of your DAG reached? ``concurrency`` defines
- how many ``running`` task instances a DAG is allowed to have, beyond which
+- Is the ``concurrency`` parameter of your Dag reached? ``concurrency`` defines
+ how many ``running`` task instances a Dag is allowed to have, beyond which
point things get queued.
-- Is the ``max_active_runs`` parameter of your DAG reached? ``max_active_runs`` defines
- how many ``running`` concurrent instances of a DAG there are allowed to be.
+- Is the ``max_active_runs`` parameter of your Dag reached? ``max_active_runs`` defines
+ how many ``running`` concurrent instances of a Dag there are allowed to be.
You may also want to read about the :ref:`scheduler` and make
sure you fully understand how the scheduler cycle.
-How to improve DAG performance?
+How to improve Dag performance?
-------------------------------
There are some Airflow configuration to allow for a larger scheduling capacity and frequency:
@@ -107,10 +107,10 @@ Operators or tasks also have configurations that improves efficiency and schedul
- ``queue``: See :ref:`apache-airflow-providers-celery:celery_executor:queue` for CeleryExecutor deployments only.
-How to reduce DAG scheduling latency / task delay?
+How to reduce Dag scheduling latency / task delay?
--------------------------------------------------
-Airflow 2.0 has low DAG scheduling latency out of the box (particularly when compared with Airflow 1.10.x),
+Airflow 2.0 has low Dag scheduling latency out of the box (particularly when compared with Airflow 1.10.x),
however, if you need more throughput you can :ref:`start multiple schedulers<scheduler:ha>`.
@@ -121,14 +121,14 @@ You can achieve this with :ref:`concepts:trigger-rules`.
.. _faq:how-to-control-dag-file-parsing-timeout:
-How to control DAG file parsing timeout for different DAG files?
+How to control Dag file parsing timeout for different Dag files?
----------------------------------------------------------------
(only valid for Airflow >= 2.3.0)
You can add a ``get_dagbag_import_timeout`` function in your ``airflow_local_settings.py`` which gets
-called right before a DAG file is parsed. You can return different timeout value based on the DAG file.
-When the return value is less than or equal to 0, it means no timeout during the DAG parsing.
+called right before a Dag file is parsed. You can return different timeout value based on the Dag file.
+When the return value is less than or equal to 0, it means no timeout during the Dag parsing.
.. code-block:: python
:caption: airflow_local_settings.py
@@ -136,12 +136,12 @@ When the return value is less than or equal to 0, it means no timeout during the
def get_dagbag_import_timeout(dag_file_path: str) -> Union[int, float]:
"""
- This setting allows to dynamically control the DAG file parsing timeout.
+ This setting allows to dynamically control the Dag file parsing timeout.
- It is useful when there are a few DAG files requiring longer parsing times, while others do not.
- You can control them separately instead of having one value for all DAG files.
+ It is useful when there are a few Dag files requiring longer parsing times, while others do not.
+ You can control them separately instead of having one value for all Dag files.
- If the return value is less than or equal to 0, it means no timeout during the DAG parsing.
+ If the return value is less than or equal to 0, it means no timeout during the Dag parsing.
"""
if "slow" in dag_file_path:
return 90
@@ -155,19 +155,19 @@ configure local settings.
-When there are a lot (>1000) of DAG files, how to speed up parsing of new files?
+When there are a lot (>1000) of Dag files, how to speed up parsing of new files?
---------------------------------------------------------------------------------
Change the :ref:`config:dag_processor__file_parsing_sort_mode` to ``modified_time``, raise
the :ref:`config:dag_processor__min_file_process_interval` to ``600`` (10 minutes), ``6000`` (100 minutes)
or a higher value.
-The DAG parser will skip the ``min_file_process_interval`` check if a file is recently modified.
+The Dag parser will skip the ``min_file_process_interval`` check if a file is recently modified.
-This might not work for case where the DAG is imported/created from a separate file. Example:
-``dag_file.py`` that imports ``dag_loader.py`` where the actual logic of the DAG file is as shown below.
+This might not work for case where the Dag is imported/created from a separate file. Example:
+``dag_file.py`` that imports ``dag_loader.py`` where the actual logic of the Dag file is as shown below.
In this case if ``dag_loader.py`` is updated but ``dag_file.py`` is not updated, the changes won't be reflected
-until ``min_file_process_interval`` is reached since DAG Parser will look for modified time for ``dag_file.py`` file.
+until ``min_file_process_interval`` is reached since Dag Parser will look for modified time for ``dag_file.py`` file.
.. code-block:: python
:caption: dag_file.py
@@ -200,34 +200,34 @@ until ``min_file_process_interval`` is reached since DAG Parser will look for mo
@task()
def hello_world():
print("Hello World")
- print(f"This is DAG: {dag_number}")
+ print(f"This is Dag: {dag_number}")
hello_world()
return dag
-What to do if you see disappearing DAGs on UI?
+What to do if you see disappearing Dags on UI?
----------------------------------------------
-There are several reasons why DAGs might disappear from the UI. Common causes include:
+There are several reasons why Dags might disappear from the UI. Common causes include:
-* **Total parsing of all DAGs is too long** - If parsing takes longer than :ref:`config:core__dagbag_import_timeout`,
- files may not be processed completely. This often occurs when DAGs don't follow
- :ref:`DAG writing best practices<best_practice:writing_a_dag>` like:
+* **Total parsing of all Dags is too long** - If parsing takes longer than :ref:`config:core__dagbag_import_timeout`,
+ files may not be processed completely. This often occurs when Dags don't follow
+ :ref:`Dag writing best practices<best_practice:writing_a_dag>` like:
* Excessive top-level code execution
* External system calls during parsing
- * Complex dynamic DAG generation
+ * Complex dynamic Dag generation
-* **Inconsistent dynamic DAG generation** - DAGs created through
- :doc:`dynamic generation </howto/dynamic-dag-generation>` must produce stable DAG IDs across parses.
+* **Inconsistent dynamic Dag generation** - Dags created through
+ :doc:`dynamic generation </howto/dynamic-dag-generation>` must produce stable Dag IDs across parses.
Verify consistency by running ``python your_dag_file.py`` repeatedly.
-* **File processing configuration issues** - A certain combination of parameters may lead to scenarios which certain DAGs are less likely to be processed at each loop. Check these parameters:
+* **File processing configuration issues** - A certain combination of parameters may lead to scenarios which certain Dags are less likely to be processed at each loop. Check these parameters:
* :ref:`config:dag_processor__file_parsing_sort_mode` - Ensure sorting method matches your sync strategy
* :ref:`config:dag_processor__parsing_processes` - Number of parallel parsers
- * :ref:`config:scheduler__parsing_cleanup_interval` - Controls stale DAG cleanup frequency
- * :ref:`config:scheduler__dag_stale_not_seen_duration` - Time threshold for marking DAGs as stale
+ * :ref:`config:scheduler__parsing_cleanup_interval` - Controls stale Dag cleanup frequency
+ * :ref:`config:scheduler__dag_stale_not_seen_duration` - Time threshold for marking Dags as stale
* **File synchronization problems** - Common with git-sync setups:
@@ -238,35 +238,35 @@ There are several reasons why DAGs might disappear from the UI. Common causes in
* **Time synchronization issues** - Ensure all nodes (database, schedulers, workers) use NTP with <1s clock drift.
-DAG construction
+Dag construction
^^^^^^^^^^^^^^^^
What's the deal with ``start_date``?
------------------------------------
``start_date`` is partly legacy from the pre-DagRun era, but it is still
-relevant in many ways. When creating a new DAG, you probably want to set
+relevant in many ways. When creating a new Dag, you probably want to set
a global ``start_date`` for your tasks. This can be done by declaring your
-``start_date`` directly in the ``DAG()`` object. A DAG's first
+``start_date`` directly in the ``DAG()`` object. A Dag's first
DagRun will be created based on the first complete ``data_interval``
-after ``start_date``. For example, for a DAG with
+after ``start_date``. For example, for a Dag with
``start_date=datetime(2024, 1, 1)`` and ``schedule="0 0 3 * *"``, the
-first DAG run will be triggered at midnight on 2024-02-03 with
+first Dag run will be triggered at midnight on 2024-02-03 with
``data_interval_start=datetime(2024, 1, 3)`` and
``data_interval_end=datetime(2024, 2, 3)``. From that point on, the scheduler
creates new DagRuns based on your ``schedule`` and the corresponding task
instances run as your dependencies are met. When introducing new tasks to
-your DAG, you need to pay special attention to ``start_date``, and may want
+your Dag, you need to pay special attention to ``start_date``, and may want
to reactivate inactive DagRuns to get the new task onboarded properly.
We recommend against using dynamic values as ``start_date``, especially
``datetime.now()`` as it can be quite confusing. The task is triggered
-once the period closes, and in theory an ``@hourly`` DAG would never get to
+once the period closes, and in theory an ``@hourly`` Dag would never get to
an hour after now as ``now()`` moves along.
Previously, we also recommended using rounded ``start_date`` in relation to your
-DAG's ``schedule``. This meant an ``@hourly`` would be at ``00:00``
+Dag's ``schedule``. This meant an ``@hourly`` would be at ``00:00``
minutes:seconds, a ``@daily`` job at midnight, a ``@monthly`` job on the
first of the month. This is no longer required. Airflow will now auto align
the ``start_date`` and the ``schedule``, by using the ``start_date``
@@ -290,10 +290,10 @@ It is also important to note that the task's ``start_date`` is ignored in backfi
Using time zones
----------------
-Creating a time zone aware datetime (e.g. DAG's ``start_date``) is quite simple. Just make sure to supply
+Creating a time zone aware datetime (e.g. Dag's ``start_date``) is quite simple. Just make sure to supply
a time zone aware dates using ``pendulum``. Don't try to use standard library
`timezone <https://docs.python.org/3/library/datetime.html#timezone-objects>`_ as they are known to
-have limitations and we deliberately disallow using them in dags.
+have limitations and we deliberately disallow using them in Dags.
.. _faq:what-does-execution-date-mean:
@@ -303,7 +303,7 @@ What does ``execution_date`` mean?
*Execution date* or ``execution_date`` is a historical name for what is called a
*logical date*, and also usually the start of the data interval represented by a
-DAG run.
+Dag run.
Airflow was developed as a solution for ETL needs. In the ETL world, you
typically summarize data. So, if you want to summarize data for ``2016-02-19``,
@@ -311,7 +311,7 @@ you would do it at ``2016-02-20`` midnight UTC, which would be right after all
data for ``2016-02-19`` becomes available. This interval between midnights of
``2016-02-19`` and ``2016-02-20`` is called the *data interval*, and since it
represents data in the date of ``2016-02-19``, this date is also called the
-run's *logical date*, or the date that this DAG run is executed for, thus
+run's *logical date*, or the date that this Dag run is executed for, thus
*execution date*.
For backward compatibility, a datetime value ``execution_date`` is still
@@ -338,7 +338,7 @@ Note that ``ds`` (the YYYY-MM-DD form of ``data_interval_start``) refers to
:ref:`concepts-dag-run`.
-How to create dags dynamically?
+How to create Dags dynamically?
-------------------------------
Airflow looks in your ``DAGS_FOLDER`` for modules that contain ``DAG`` objects
@@ -366,24 +366,24 @@ simple dictionary.
other_dag_id = f"bar_{i}"
globals()[other_dag_id] = create_dag(other_dag_id)
-Even though Airflow supports multiple DAG definition per python file, dynamically generated or otherwise, it is not
-recommended as Airflow would like better isolation between dags from a fault and deployment perspective and multiple
-dags in the same file goes against that.
+Even though Airflow supports multiple Dag definition per python file, dynamically generated or otherwise, it is not
+recommended as Airflow would like better isolation between Dags from a fault and deployment perspective and multiple
+Dags in the same file goes against that.
Are top level Python code allowed?
----------------------------------
While it is not recommended to write any code outside of defining Airflow constructs, Airflow does support any
-arbitrary python code as long as it does not break the DAG file processor or prolong file processing time past the
+arbitrary python code as long as it does not break the Dag file processor or prolong file processing time past the
:ref:`config:core__dagbag_import_timeout` value.
-A common example is the violation of the time limit when building a dynamic DAG which usually requires querying data
-from another service like a database. At the same time, the requested service is being swamped with DAG file
+A common example is the violation of the time limit when building a dynamic Dag which usually requires querying data
+from another service like a database. At the same time, the requested service is being swamped with Dag file
processors requests for data to process the file. These unintended interactions may cause the service to deteriorate
-and eventually cause DAG file processing to fail.
+and eventually cause Dag file processing to fail.
-Refer to :ref:`DAG writing best practices<best_practice:writing_a_dag>` for more information.
+Refer to :ref:`Dag writing best practices<best_practice:writing_a_dag>` for more information.
Do Macros resolves in another Jinja template?
@@ -401,7 +401,7 @@ commonly attempted in ``user_defined_macros``.
bo = BashOperator(task_id="my_task", bash_command="echo {{ my_custom_macro }}", dag=dag)
-This will echo "day={{ ds }}" instead of "day=2020-01-01" for a DAG run with a
+This will echo "day={{ ds }}" instead of "day=2020-01-01" for a Dag run with a
``data_interval_start`` of 2020-01-01 00:00:00.
.. code-block:: python
@@ -414,10 +414,10 @@ By using the ds macros directly in the template_field, the rendered value result
Why ``next_ds`` or ``prev_ds`` might not contain expected values?
------------------------------------------------------------------
-- When scheduling DAG, the ``next_ds`` ``next_ds_nodash`` ``prev_ds`` ``prev_ds_nodash`` are calculated using
- ``logical_date`` and the DAG's schedule (if applicable). If you set ``schedule`` as ``None`` or ``@once``,
+- When scheduling Dag, the ``next_ds`` ``next_ds_nodash`` ``prev_ds`` ``prev_ds_nodash`` are calculated using
+ ``logical_date`` and the Dag's schedule (if applicable). If you set ``schedule`` as ``None`` or ``@once``,
the ``next_ds``, ``next_ds_nodash``, ``prev_ds``, ``prev_ds_nodash`` values will be set to ``None``.
-- When manually triggering DAG, the schedule will be ignored, and ``prev_ds == next_ds == ds``.
+- When manually triggering Dag, the schedule will be ignored, and ``prev_ds == next_ds == ds``.
Task execution interactions
@@ -430,7 +430,7 @@ What does ``TemplateNotFound`` mean?
that trigger Jinja templating. A common occurrence is with :class:`~airflow.providers.standard.operators.BashOperator`.
Another commonly missed fact is that the files are resolved relative to where the pipeline file lives. You can add
-other directories to the ``template_searchpath`` of the DAG object to allow for other non-relative location.
+other directories to the ``template_searchpath`` of the Dag object to allow for other non-relative location.
How to trigger tasks based on another task's failure?
@@ -495,12 +495,12 @@ How do I stop the sync perms happening multiple times per webserver?
Set the value of ``[fab] update_fab_perms`` configuration in ``airflow.cfg`` to ``False``.
-Why did the pause DAG toggle turn red?
+Why did the pause Dag toggle turn red?
--------------------------------------
-If pausing or unpausing a DAG fails for any reason, the DAG toggle will
+If pausing or unpausing a Dag fails for any reason, the Dag toggle will
revert to its previous state and turn red. If you observe this behavior,
-try pausing the DAG again, or check the console or server logs if the
+try pausing the Dag again, or check the console or server logs if the
issue recurs.
@@ -559,7 +559,7 @@ such as the following Airflow 3 example:
from airflow import DAG
from airflow.sdk import task
- with DAG(dag_id="canary", schedule="@daily", doc_md="Canary DAG to regularly test connections to systems."):
+ with DAG(dag_id="canary", schedule="@daily", doc_md="Canary Dag to regularly test connections to systems."):
@task(doc_md="Test a connection by its Connection ID.")
def test_connection(conn_id):
diff --git a/airflow-core/docs/howto/add-dag-tags.rst b/airflow-core/docs/howto/add-dag-tags.rst
index 95418114c97..bf053d80755 100644
--- a/airflow-core/docs/howto/add-dag-tags.rst
+++ b/airflow-core/docs/howto/add-dag-tags.rst
@@ -18,17 +18,17 @@
-Add tags to dags and use it for filtering in the UI
+Add tags to Dags and use it for filtering in the UI
===================================================
.. versionadded:: 1.10.8
-In order to filter dags (e.g by team), you can add tags in each DAG.
+In order to filter Dags (e.g by team), you can add tags in each Dag.
The filter is saved in a cookie and can be reset by the reset button.
For example:
-In your DAG file, pass a list of tags you want to add to the DAG object:
+In your Dag file, pass a list of tags you want to add to the Dag object:
.. code-block:: python
@@ -39,6 +39,6 @@ In your DAG file, pass a list of tags you want to add to the DAG object:
.. image:: ../img/ui-dark/add-dag-tags.png
-Tags are registered as part of DAG parsing.
+Tags are registered as part of Dag parsing.
In case of stale tags, you can purge old data with the Airflow CLI command ``airflow db clean``.
See :ref:`db clean usage<cli-db-clean>` for more details.
diff --git a/airflow-core/docs/howto/add-owner-links.rst b/airflow-core/docs/howto/add-owner-links.rst
index fbc8445209d..437f840511c 100644
--- a/airflow-core/docs/howto/add-owner-links.rst
+++ b/airflow-core/docs/howto/add-owner-links.rst
@@ -18,21 +18,21 @@
-Add Owner Links to DAG
+Add Owner Links to Dag
=======================
.. versionadded:: 2.4.0
-You can set the ``owner_links`` argument on your DAG object, which will make the owner a clickable link in the
-main dags view page instead of a search filter.
+You can set the ``owner_links`` argument on your Dag object, which will make the owner a clickable link in the
+main Dags view page instead of a search filter.
Two options are supported:
* An HTTP link (e.g. ``https://www.example.com``) which opens the webpage in your default internet client
* A `mailto <https://en.wikipedia.org/wiki/Mailto>`_ link (e.g. ``mailto:example@airflow.com``) which opens your default email client to send an email to the specified address
-In your DAG, set the ``owner_links`` argument specifying a dictionary of an owner (key) and its link (value).
-Next define a task using this owner, and the owner in the dags view will link to the specified address.
+In your Dag, set the ``owner_links`` argument specifying a dictionary of an owner (key) and its link (value).
+Next define a task using this owner, and the owner in the Dags view will link to the specified address.
.. code-block:: python
:emphasize-lines: 5
diff --git a/airflow-core/docs/howto/custom-operator.rst b/airflow-core/docs/howto/custom-operator.rst
index ad09760c254..f250aa7447b 100644
--- a/airflow-core/docs/howto/custom-operator.rst
+++ b/airflow-core/docs/howto/custom-operator.rst
@@ -29,7 +29,7 @@ You can create any operator you want by extending the public SDK base class :cla
There are two methods that you need to override in a derived class:
* Constructor - Define the parameters required for the operator. You only need to specify the arguments specific to your operator.
- You can specify the ``default_args`` in the DAG file. See :ref:`Default args <concepts-default-arguments>` for more details.
+ You can specify the ``default_args`` in the Dag file. See :ref:`Default args <concepts-default-arguments>` for more details.
* Execute - The code to execute when the runner calls the operator. The method contains the
Airflow context as a parameter that can be used to read config values.
@@ -95,10 +95,10 @@ logic of an operation is in one place - in the operator.
Hooks
-----
-Hooks act as an interface to communicate with the external shared resources in a DAG.
-For example, multiple tasks in a DAG can require access to a MySQL database. Instead of
+Hooks act as an interface to communicate with the external shared resources in a Dag.
+For example, multiple tasks in a Dag can require access to a MySQL database. Instead of
creating a connection per task, you can retrieve a connection from the hook and utilize it.
-Hook also helps to avoid storing connection auth parameters in a DAG.
+Hook also helps to avoid storing connection auth parameters in a Dag.
See :doc:`connection` for how to create and manage connections and :doc:`apache-airflow-providers:index` for
details of how to add your custom connection types via providers.
@@ -125,14 +125,14 @@ When the operator invokes the query on the hook object, a new connection gets cr
The hook retrieves the auth parameters such as username and password from Airflow
backend and passes the params to the :py:func:`airflow.hooks.base.BaseHook.get_connection`.
You should create hook only in the ``execute`` method or any method which is called from ``execute``.
-The constructor gets called whenever Airflow parses a DAG which happens frequently. And instantiating a hook
+The constructor gets called whenever Airflow parses a Dag which happens frequently. And instantiating a hook
there will result in many unnecessary database connections.
-The ``execute`` gets called only during a DAG run.
+The ``execute`` gets called only during a Dag run.
User interface
--------------
-Airflow also allows the developer to control how the operator shows up in the DAG UI.
+Airflow also allows the developer to control how the operator shows up in the Dag UI.
Override ``ui_color`` to change the background color of the operator in UI.
Override ``ui_fgcolor`` to change the color of the label.
Override ``custom_operator_name`` to change the displayed name to something other than the classname.
diff --git a/airflow-core/docs/howto/customize-ui.rst b/airflow-core/docs/howto/customize-ui.rst
index 6d1c444f9a8..1eb39309610 100644
--- a/airflow-core/docs/howto/customize-ui.rst
+++ b/airflow-core/docs/howto/customize-ui.rst
@@ -20,10 +20,10 @@ Customizing the UI
.. _customizing-the-ui:
-Customizing DAG UI Header and Airflow Page Titles
--------------------------------------------------
+Customizing Dag UI Header and Airflow Page Titles
+==================================================
-Airflow now allows you to customize the DAG home page header and page title. This will help
+Airflow now allows you to customize the Dag home page header and page title. This will help
distinguish between various installations of Airflow or simply amend the page text.
.. note::
diff --git a/airflow-core/docs/howto/docker-compose/index.rst b/airflow-core/docs/howto/docker-compose/index.rst
index 62382469df7..7e6d15a9e67 100644
--- a/airflow-core/docs/howto/docker-compose/index.rst
+++ b/airflow-core/docs/howto/docker-compose/index.rst
@@ -86,9 +86,9 @@ Fetching ``docker-compose.yaml``
This file contains several service definitions:
-- ``airflow-scheduler`` - The :doc:`scheduler </administration-and-deployment/scheduler>` monitors all tasks and dags, then triggers the
+- ``airflow-scheduler`` - The :doc:`scheduler </administration-and-deployment/scheduler>` monitors all tasks and Dags, then triggers the
task instances once their dependencies are complete.
-- ``airflow-dag-processor`` - The DAG processor parses DAG files.
+- ``airflow-dag-processor`` - The Dag processor parses Dag files.
- ``airflow-api-server`` - The api server is available at ``http://localhost:8080``.
- ``airflow-worker`` - The worker that executes the tasks given by the scheduler.
- ``airflow-triggerer`` - The triggerer runs an event loop for deferrable tasks.
@@ -104,7 +104,7 @@ All these services allow you to run Airflow with :doc:`CeleryExecutor <apache-ai
Some directories in the container are mounted, which means that their contents are synchronized between your computer and the container.
-- ``./dags`` - you can put your DAG files here.
+- ``./dags`` - you can put your Dag files here.
- ``./logs`` - contains logs from task execution and scheduler.
- ``./config`` - you can add custom log parser or add ``airflow_local_settings.py`` to configure cluster policy.
- ``./plugins`` - you can put your :doc:`custom plugins </administration-and-deployment/plugins>` here.
@@ -261,7 +261,7 @@ python container.
Accessing the web interface
---------------------------
-Once the cluster has started up, you can log in to the web interface and begin experimenting with dags.
+Once the cluster has started up, you can log in to the web interface and begin experimenting with Dags.
The webserver is available at: ``http://localhost:8080``.
The default account has the login ``airflow`` and the password ``airflow``.
@@ -367,7 +367,7 @@ If you have a custom config file and wish to use it in your Airflow instance, yo
Networking
==========
-In general, if you want to use Airflow locally, your dags may try to connect to servers which are running on the host. In order to achieve that, an extra configuration must be added in ``docker-compose.yaml``. For example, on Linux the configuration must be in the section ``services: airflow-worker`` adding ``extra_hosts: - "host.docker.internal:host-gateway"``; and use ``host.docker.internal`` instead of ``localhost``. This configuration vary in different platforms. Please check the Doc [...]
+In general, if you want to use Airflow locally, your Dags may try to connect to servers which are running on the host. In order to achieve that, an extra configuration must be added in ``docker-compose.yaml``. For example, on Linux the configuration must be in the section ``services: airflow-worker`` adding ``extra_hosts: - "host.docker.internal:host-gateway"``; and use ``host.docker.internal`` instead of ``localhost``. This configuration vary in different platforms. Please check the Doc [...]
Debug Airflow inside docker container using PyCharm
===================================================
diff --git a/airflow-core/docs/howto/dynamic-dag-generation.rst b/airflow-core/docs/howto/dynamic-dag-generation.rst
index dcd07df8139..407ea8e05b6 100644
--- a/airflow-core/docs/howto/dynamic-dag-generation.rst
+++ b/airflow-core/docs/howto/dynamic-dag-generation.rst
@@ -17,23 +17,23 @@
-Dynamic DAG Generation
+Dynamic Dag Generation
======================
-This document describes creation of dags that have a structure generated dynamically, but where the number of
-tasks in the DAG does not change between DAG Runs. If you want to implement a DAG where number of Tasks (or
+This document describes creation of Dags that have a structure generated dynamically, but where the number of
+tasks in the Dag does not change between Dag Runs. If you want to implement a Dag where number of Tasks (or
Task Groups as of Airflow 2.6) can change based on the output/result of previous tasks, see
:doc:`/authoring-and-scheduling/dynamic-task-mapping`.
.. note:: Consistent sequence of generating tasks and task groups
- In all cases where you generate dags dynamically, you should make sure that Tasks and Task Groups
- are generated with consistent sequence every time the DAG is generated, otherwise you might end up with
+ In all cases where you generate Dags dynamically, you should make sure that Tasks and Task Groups
+ are generated with consistent sequence every time the Dag is generated, otherwise you might end up with
Tasks and Task Groups changing their sequence in the Grid View every time you refresh the page.
This can be achieved for example by using a stable sorting mechanism in your Database queries or by using
``sorted()`` function in Python.
-Dynamic dags with environment variables
+Dynamic Dags with environment variables
.......................................
If you want to use variables to configure your code, you should always use
@@ -42,11 +42,11 @@ top-level code rather than :doc:`Airflow Variables </core-concepts/variables>`.
in top-level code creates a connection to the metadata DB of Airflow to fetch the value, which can slow
down parsing and place extra load on the DB. See
:ref:`best practices on Airflow Variables <best_practices/airflow_variables>`
-to make the best use of Airflow Variables in your dags using Jinja templates.
+to make the best use of Airflow Variables in your Dags using Jinja templates.
For example you could set ``DEPLOYMENT`` variable differently for your production and development
environments. The variable ``DEPLOYMENT`` could be set to ``PROD`` in your production environment and to
-``DEV`` in your development environment. Then you could build your DAG differently in production and
+``DEV`` in your development environment. Then you could build your Dag differently in production and
development environment, depending on the value of the environment variable.
.. code-block:: python
@@ -62,20 +62,20 @@ Generating Python code with embedded meta-data
..............................................
You can externally generate Python code containing the meta-data as importable constants.
-Such constant can then be imported directly by your DAG and used to construct the object and build
-the dependencies. This makes it easy to import such code from multiple dags without the need to find,
+Such constant can then be imported directly by your Dag and used to construct the object and build
+the dependencies. This makes it easy to import such code from multiple Dags without the need to find,
load and parse the meta-data stored in the constant - this is done automatically by Python interpreter
when it processes the "import" statement. This sounds strange at first, but it is surprisingly easy
-to generate such code and make sure this is a valid Python code that you can import from your dags.
+to generate such code and make sure this is a valid Python code that you can import from your Dags.
-For example assume you dynamically generate (in your DAG folder), the ``my_company_utils/common.py`` file:
+For example assume you dynamically generate (in your Dag folder), the ``my_company_utils/common.py`` file:
.. code-block:: python
# This file is generated automatically !
ALL_TASKS = ["task1", "task2", "task3"]
-Then you can import and use the ``ALL_TASKS`` constant in all your dags like that:
+Then you can import and use the ``ALL_TASKS`` constant in all your Dags like that:
.. code-block:: python
@@ -93,22 +93,22 @@ Then you can import and use the ``ALL_TASKS`` constant in all your dags like tha
Don't forget that in this case you need to add empty ``__init__.py`` file in the ``my_company_utils`` folder
and you should add the ``my_company_utils/*`` line to ``.airflowignore`` file (using the default glob
-syntax), so that the whole folder is ignored by the scheduler when it looks for dags.
+syntax), so that the whole folder is ignored by the scheduler when it looks for Dags.
-Dynamic dags with external configuration from a structured data file
+Dynamic Dags with external configuration from a structured data file
....................................................................
-If you need to use a more complex meta-data to prepare your DAG structure and you would prefer to keep the
-data in a structured non-python format, you should export the data to the DAG folder in a file and push
-it to the DAG folder, rather than try to pull the data by the DAG's top-level code - for the reasons
+If you need to use a more complex meta-data to prepare your Dag structure and you would prefer to keep the
+data in a structured non-python format, you should export the data to the Dag folder in a file and push
+it to the Dag folder, rather than try to pull the data by the Dag's top-level code - for the reasons
explained in the parent :ref:`best_practices/top_level_code`.
-The meta-data should be exported and stored together with the dags in a convenient file format (JSON, YAML
-formats are good candidates) in DAG folder. Ideally, the meta-data should be published in the same
-package/folder as the module of the DAG file you load it from, because then you can find location of
-the meta-data file in your DAG easily. The location of the file to read can be found using the
-``__file__`` attribute of the module containing the DAG:
+The meta-data should be exported and stored together with the Dags in a convenient file format (JSON, YAML
+formats are good candidates) in Dag folder. Ideally, the meta-data should be published in the same
+package/folder as the module of the Dag file you load it from, because then you can find location of
+the meta-data file in your Dag easily. The location of the file to read can be found using the
+``__file__`` attribute of the module containing the Dag:
.. code-block:: python
@@ -119,10 +119,10 @@ the meta-data file in your DAG easily. The location of the file to read can be f
# Configuration dict is available here
-Registering dynamic dags
+Registering dynamic Dags
........................
-You can dynamically generate dags when using the ``@dag`` decorator or the ``with DAG(..)`` context manager
+You can dynamically generate Dags when using the ``@dag`` decorator or the ``with DAG(..)`` context manager
and Airflow will automatically register them.
.. code-block:: python
@@ -131,8 +131,8 @@ and Airflow will automatically register them.
from airflow.sdk import dag, task
configs = {
- "config1": {"message": "first DAG will receive this message"},
- "config2": {"message": "second DAG will receive this message"},
+ "config1": {"message": "first Dag will receive this message"},
+ "config2": {"message": "second Dag will receive this message"},
}
for config_name, config in configs.items():
@@ -148,36 +148,36 @@ and Airflow will automatically register them.
dynamic_generated_dag()
-The code below will generate a DAG for each config: ``dynamic_generated_dag_config1`` and ``dynamic_generated_dag_config2``.
+The code below will generate a Dag for each config: ``dynamic_generated_dag_config1`` and ``dynamic_generated_dag_config2``.
Each of them can run separately with related configuration.
-If you do not wish to have dags auto-registered, you can disable the behavior by setting ``auto_register=False`` on your DAG.
+If you do not wish to have Dags auto-registered, you can disable the behavior by setting ``auto_register=False`` on your Dag.
.. versionchanged:: 2.4
- As of version 2.4 dags that are created by calling a ``@dag`` decorated function (or that are used in the
+ As of version 2.4 Dags that are created by calling a ``@dag`` decorated function (or that are used in the
``with DAG(...)`` context manager are automatically registered, and no longer need to be stored in a
global variable.
-Optimizing DAG parsing delays during execution
+Optimizing Dag parsing delays during execution
----------------------------------------------
.. versionadded:: 2.4
-Sometimes when you generate a lot of Dynamic dags from a single DAG file, it might cause unnecessary delays
-when the DAG file is parsed during task execution. The impact is a delay before a task starts.
+Sometimes when you generate a lot of Dynamic Dags from a single Dag file, it might cause unnecessary delays
+when the Dag file is parsed during task execution. The impact is a delay before a task starts.
Why is this happening? You might not be aware but just before your task is executed,
-Airflow parses the Python file the DAG comes from.
+Airflow parses the Python file the Dag comes from.
-The Airflow DAG File Processor requires loading of a complete DAG file to process
-all metadata. However, task execution requires only a single DAG object to execute a task. Knowing this,
-we can skip the generation of unnecessary DAG objects when a task is executed, shortening the parsing time.
-This optimization is most effective when the number of generated dags is high.
+The Airflow Dag File Processor requires loading of a complete Dag file to process
+all metadata. However, task execution requires only a single Dag object to execute a task. Knowing this,
+we can skip the generation of unnecessary Dag objects when a task is executed, shortening the parsing time.
+This optimization is most effective when the number of generated Dags is high.
-Note that it is not always possible to use (for example when generation of subsequent dags depends on the previous dags) or when
-there are some side-effects of your dags generation. Use this solution with care and test it thoroughly.
+Note that it is not always possible to use (for example when generation of subsequent Dags depends on the previous Dags) or when
+there are some side-effects of your Dags generation. Use this solution with care and test it thoroughly.
A nice example of performance improvements you can gain is shown in the
`Airflow's Magic Loop <https://medium.com/apache-airflow/airflows-magic-loop-ec424b05b629>`_ blog post
@@ -187,14 +187,14 @@ written before Airflow 2.4 so it uses undocumented behaviour of Airflow, not :py
In Airflow 2.4+ instead you can use :py:meth:`~airflow.utils.dag_parsing_context.get_parsing_context` method
to retrieve the current context in a documented and predictable way.
-Upon iterating over the collection of things to generate dags for, you can use the context to determine
-whether you need to generate all DAG objects (when parsing in the DAG File processor), or to generate only
-a single DAG object (when executing the task).
+Upon iterating over the collection of things to generate Dags for, you can use the context to determine
+whether you need to generate all Dag objects (when parsing in the Dag File processor), or to generate only
+a single Dag object (when executing the task).
:py:meth:`~airflow.utils.dag_parsing_context.get_parsing_context` returns the current parsing
context. The context is an :py:class:`~airflow.utils.dag_parsing_context.AirflowParsingContext` and
-when only a single DAG/task is needed, it contains ``dag_id`` and ``task_id`` fields set.
-When "full" parsing is needed (for example in DAG File Processor), ``dag_id`` and ``task_id``
+when only a single Dag/task is needed, it contains ``dag_id`` and ``task_id`` fields set.
+When "full" parsing is needed (for example in Dag File Processor), ``dag_id`` and ``task_id``
of the context are set to ``None``.
@@ -209,7 +209,7 @@ of the context are set to ``None``.
for thing in list_of_things:
dag_id = f"generated_dag_{thing}"
if current_dag_id is not None and current_dag_id != dag_id:
- continue # skip generation of non-selected DAG
+ continue # skip generation of non-selected Dag
with DAG(dag_id=dag_id, ...):
...
diff --git a/airflow-core/docs/howto/listener-plugin.rst b/airflow-core/docs/howto/listener-plugin.rst
index 9d139093548..c5a0acd4a66 100644
--- a/airflow-core/docs/howto/listener-plugin.rst
+++ b/airflow-core/docs/howto/listener-plugin.rst
@@ -23,12 +23,12 @@ Airflow has feature that allows to add listener for monitoring and tracking
the task state using Plugins.
This is a simple example listener plugin of Airflow that helps to track the task
-state and collect useful metadata information about the task, dag run and dag.
+state and collect useful metadata information about the task, Dag run and dag.
This is an example plugin for Airflow that allows to create listener plugin of Airflow.
This plugin works by using SQLAlchemy's event mechanism. It watches
the task instance state change in the table level and triggers event.
-This will be notified for all the tasks across all the dags.
+This will be notified for all the tasks across all the Dags.
In this plugin, an object reference is derived from the base class
``airflow.plugins_manager.AirflowPlugin``.
@@ -41,9 +41,9 @@ Using this plugin, following events can be listened:
* task instance is in running state.
* task instance is in success state.
* task instance is in failure state.
- * dag run is in running state.
- * dag run is in success state.
- * dag run is in failure state.
+ * Dag run is in running state.
+ * Dag run is in success state.
+ * Dag run is in failure state.
* on start before event like Airflow job, scheduler
* before stop for event like Airflow job, scheduler
@@ -69,7 +69,7 @@ skeleton for us to implement a new listener:
Next, we can check code added into ``listener`` and see implementation
methods for each of those listeners. After the implementation, the listener part
-gets executed during all the task execution across all the dags
+gets executed during all the task execution across all the Dags
For reference, here's the plugin code within ``listener.py`` class that shows list of tables in the database:
@@ -82,7 +82,7 @@ This example listens when the task instance is in running state
Similarly, code to listen after task_instance success and failure can be implemented.
-This example listens when the dag run is change to failed state
+This example listens when the Dag run is change to failed state
.. exampleinclude:: /../src/airflow/example_dags/plugins/event_listener.py
:language: python
diff --git a/airflow-core/docs/howto/notifications.rst b/airflow-core/docs/howto/notifications.rst
index 1705ccf25dc..14a35d962a6 100644
--- a/airflow-core/docs/howto/notifications.rst
+++ b/airflow-core/docs/howto/notifications.rst
@@ -58,4 +58,4 @@ For a list of community-managed notifiers, see :doc:`apache-airflow-providers:co
Using Notifiers
===============
-For using Notifiers in event-based DAG callbacks, see :doc:`../administration-and-deployment/logging-monitoring/callbacks`.
+For using Notifiers in event-based Dag callbacks, see :doc:`../administration-and-deployment/logging-monitoring/callbacks`.
diff --git a/airflow-core/docs/howto/operator/index.rst b/airflow-core/docs/howto/operator/index.rst
index 45b53887436..a3af0d6b300 100644
--- a/airflow-core/docs/howto/operator/index.rst
+++ b/airflow-core/docs/howto/operator/index.rst
@@ -21,7 +21,7 @@ Using Operators
===============
An operator represents a single, ideally idempotent, task. Operators
-determine what actually executes when your DAG runs.
+determine what actually executes when your Dag runs.
.. note::
See the :doc:`Operators Concepts </core-concepts/operators>` documentation.
diff --git a/airflow-core/docs/howto/set-config.rst b/airflow-core/docs/howto/set-config.rst
index 13a46e15f57..30d29c924c6 100644
--- a/airflow-core/docs/howto/set-config.rst
+++ b/airflow-core/docs/howto/set-config.rst
@@ -192,7 +192,7 @@ Example settings you can configure this way:
* :ref:`Advanced logging configuration <write-logs-advanced>`
* :ref:`Dag serialization <dag-serialization>`
* :ref:`Pod mutation hook in Kubernetes Executor<kubernetes:pod_mutation_hook>`
-* :ref:`Control DAG parsing time <faq:how-to-control-dag-file-parsing-timeout>`
+* :ref:`Control Dag parsing time <faq:how-to-control-dag-file-parsing-timeout>`
* :ref:`Customize your UI <customizing-the-ui>`
* :ref:`Configure more variables to export <export_dynamic_environment_variables>`
* :ref:`Customize your DB configuration <set-up-database-backend>`
diff --git a/airflow-core/docs/howto/set-up-database.rst b/airflow-core/docs/howto/set-up-database.rst
index 245a8c912c8..6846efc303a 100644
--- a/airflow-core/docs/howto/set-up-database.rst
+++ b/airflow-core/docs/howto/set-up-database.rst
@@ -330,7 +330,7 @@ MsSQL Database
the Airflow's PMC members and Committers have reached a resolution to no longer maintain MsSQL as a supported Database Backend.
As of Airflow 2.9.0 support of MsSQL has been removed for Airflow Database Backend.
- This does not affect the existing providers (operators and hooks), dags can still access and process data from MsSQL.
+ This does not affect the existing providers (operators and hooks), Dags can still access and process data from MsSQL.
However, further usage may throw errors making Airflow's core functionality unusable.
diff --git a/airflow-core/docs/howto/setup-and-teardown.rst b/airflow-core/docs/howto/setup-and-teardown.rst
index b559447fa4a..cc7ec1c9c95 100644
--- a/airflow-core/docs/howto/setup-and-teardown.rst
+++ b/airflow-core/docs/howto/setup-and-teardown.rst
@@ -23,10 +23,10 @@ In data workflows it's common to create a resource (such as a compute resource),
Key features of setup and teardown tasks:
* If you clear a task, its setups and teardowns will be cleared.
- * By default, teardown tasks are ignored for the purpose of evaluating dag run state.
+ * By default, teardown tasks are ignored for the purpose of evaluating Dag run state.
* A teardown task will run if its setup was successful, even if its work tasks failed. But it will skip if the setup was skipped.
* Teardown tasks are ignored when setting dependencies against task groups.
- * Teardown will also be carried out if the DAG run is manually set to "failed" or "success" to ensure resources will be cleaned-up.
+ * Teardown will also be carried out if the Dag run is manually set to "failed" or "success" to ensure resources will be cleaned-up.
How setup and teardown works
""""""""""""""""""""""""""""
@@ -34,7 +34,7 @@ How setup and teardown works
Basic usage
"""""""""""
-Suppose you have a dag that creates a cluster, runs a query, and deletes the cluster. Without using setup and teardown tasks you might set these relationships:
+Suppose you have a Dag that creates a cluster, runs a query, and deletes the cluster. Without using setup and teardown tasks you might set these relationships:
.. code-block:: python
@@ -53,7 +53,7 @@ For convenience we can do this in one line by passing ``create_cluster`` to the
create_cluster >> run_query >> delete_cluster.as_teardown(setups=create_cluster)
-Here's the graph for this dag:
+Here's the graph for this Dag:
.. image:: ../img/ui-dark/setup-teardown-simple.png
@@ -61,7 +61,7 @@ Observations:
* If you clear ``run_query`` to run it again, then both ``create_cluster`` and ``delete_cluster`` will be cleared.
* If ``run_query`` fails, then ``delete_cluster`` will still run.
- * The success of the dag run will depend *only* on the success of ``run_query``.
+ * The success of the Dag run will depend *only* on the success of ``run_query``.
Additionally, if we have multiple tasks to wrap, we can use the teardown as a context manager:
@@ -135,10 +135,10 @@ wait for them to complete. If a setup fails or is skipped, the work tasks which
them will be marked ask failures or skips. We also require that any non-teardown directly
downstream of a setup must have trigger rule ALL_SUCCESS.
-Controlling dag run state
+Controlling Dag run state
"""""""""""""""""""""""""
-Another feature of setup / teardown tasks is you can choose whether or not the teardown task should have an impact on dag run state. Perhaps you don't care if the "cleanup" work performed by your teardown task fails, and you only consider the dag run a failure if the "work" tasks fail. By default, teardown tasks are not considered for dag run state.
+Another feature of setup / teardown tasks is you can choose whether or not the teardown task should have an impact on Dag run state. Perhaps you don't care if the "cleanup" work performed by your teardown task fails, and you only consider the Dag run a failure if the "work" tasks fail. By default, teardown tasks are not considered for Dag run state.
Continuing with the example above, if you want the run's success to depend on ``delete_cluster``, then set ``on_failure_fail_dagrun=True`` when setting ``delete_cluster`` as teardown. For example:
@@ -149,7 +149,7 @@ Continuing with the example above, if you want the run's success to depend on ``
Authoring with task groups
""""""""""""""""""""""""""
-When adding dependency from task group to task group, or from task group to *task*, we ignore teardowns. This allows teardowns to run in parallel, and allows dag execution to proceed even if teardown tasks fail.
+When adding dependency from task group to task group, or from task group to *task*, we ignore teardowns. This allows teardowns to run in parallel, and allows Dag execution to proceed even if teardown tasks fail.
Consider this example:
@@ -167,7 +167,7 @@ Graph:
.. image:: ../img/ui-dark/setup-teardown-group.png
-If ``t1`` were not a teardown task, then this dag would effectively be ``s1 >> w1 >> t1 >> w2``. But since we have marked ``t1`` as a teardown, it's ignored in ``tg >> w2``. So the dag is equivalent to the following:
+If ``t1`` were not a teardown task, then this Dag would effectively be ``s1 >> w1 >> t1 >> w2``. But since we have marked ``t1`` as a teardown, it's ignored in ``tg >> w2``. So the Dag is equivalent to the following:
.. code-block:: python
@@ -233,7 +233,7 @@ Trigger rule behavior for teardowns
Teardowns use a (non-configurable) trigger rule called ALL_DONE_SETUP_SUCCESS. With this rule, as long as all upstreams are done and at least one directly connected setup is successful, the teardown will run. If all of a teardown's setups were skipped or failed, those states will propagate to the teardown.
-Side-effect on manual DAG state changes
+Side-effect on manual Dag state changes
"""""""""""""""""""""""""""""""""""""""
-As teardown tasks are often used to clean-up resources they need to run also if the DAG is manually terminated. For the purpose of early termination a user can manually mark the DAG run as "success" or "failed" which kills all tasks before completion. If the DAG contains teardown tasks, they will still be executed. Therefore as a side effect allowing teardown tasks to be scheduled, a DAG will not be immediately set to a terminal state if the user requests so.
+As teardown tasks are often used to clean-up resources they need to run also if the Dag is manually terminated. For the purpose of early termination a user can manually mark the Dag run as "success" or "failed" which kills all tasks before completion. If the Dag contains teardown tasks, they will still be executed. Therefore as a side effect allowing teardown tasks to be scheduled, a Dag will not be immediately set to a terminal state if the user requests so.
diff --git a/airflow-core/docs/howto/timetable.rst b/airflow-core/docs/howto/timetable.rst
index 930409345f6..35ab7e3b3d3 100644
--- a/airflow-core/docs/howto/timetable.rst
+++ b/airflow-core/docs/howto/timetable.rst
@@ -16,7 +16,7 @@
under the License.
-Customizing DAG Scheduling with Timetables
+Customizing Dag Scheduling with Timetables
==========================================
For our example, let's say a company wants to run a job after each weekday to
@@ -66,7 +66,7 @@ The following is a skeleton for us to implement a new timetable:
timetables = [AfterWorkdayTimetable]
Next, we'll start putting code into ``AfterWorkdayTimetable``. After the
-implementation is finished, we should be able to use the timetable in our DAG
+implementation is finished, we should be able to use the timetable in our Dag
file:
.. code-block:: python
@@ -89,13 +89,13 @@ file:
Define Scheduling Logic
-----------------------
-When Airflow's scheduler encounters a DAG, it calls one of the two methods to
-know when to schedule the DAG's next run.
+When Airflow's scheduler encounters a Dag, it calls one of the two methods to
+know when to schedule the Dag's next run.
* ``next_dagrun_info``: The scheduler uses this to learn the timetable's regular
schedule, i.e. the "one for every workday, run at the end of it" part in our
example.
-* ``infer_manual_data_interval``: When a DAG run is manually triggered (from the web
+* ``infer_manual_data_interval``: When a Dag run is manually triggered (from the web
UI, for example), the scheduler uses this method to learn about how to
reverse-infer the out-of-schedule run's data interval.
@@ -108,7 +108,7 @@ We'll start with ``infer_manual_data_interval`` since it's the easier of the two
:end-before: [END howto_timetable_infer_manual_data_interval]
The method accepts one argument ``run_after``, a ``pendulum.DateTime`` object
-that indicates when the DAG is externally triggered. Since our timetable creates
+that indicates when the Dag is externally triggered. Since our timetable creates
a data interval for each complete work day, the data interval inferred here
should usually start at the midnight one day prior to ``run_after``, but if
``run_after`` falls on a Sunday or Monday (i.e. the prior day is Saturday or
@@ -127,21 +127,21 @@ Next is the implementation of ``next_dagrun_info``:
This method accepts two arguments. ``last_automated_data_interval`` is a
:class:`~airflow.timetables.base.DataInterval` instance indicating the data
-interval of this DAG's previous non-manually-triggered run, or ``None`` if this
-is the first time ever the DAG is being scheduled. ``restriction`` encapsulates
-how the DAG and its tasks specify the schedule, and contains three attributes:
+interval of this Dag's previous non-manually-triggered run, or ``None`` if this
+is the first time ever the Dag is being scheduled. ``restriction`` encapsulates
+how the Dag and its tasks specify the schedule, and contains three attributes:
-* ``earliest``: The earliest time the DAG may be scheduled. This is a
+* ``earliest``: The earliest time the Dag may be scheduled. This is a
``pendulum.DateTime`` calculated from all the ``start_date`` arguments from
- the DAG and its tasks, or ``None`` if there are no ``start_date`` arguments
+ the Dag and its tasks, or ``None`` if there are no ``start_date`` arguments
found at all.
-* ``latest``: Similar to ``earliest``, this is the latest time the DAG may be
+* ``latest``: Similar to ``earliest``, this is the latest time the Dag may be
scheduled, calculated from ``end_date`` arguments.
-* ``catchup``: A boolean reflecting the DAG's ``catchup`` argument. Defaults to ``False``.
+* ``catchup``: A boolean reflecting the Dag's ``catchup`` argument. Defaults to ``False``.
.. note::
- Both ``earliest`` and ``latest`` apply to the DAG run's logical date
+ Both ``earliest`` and ``latest`` apply to the Dag run's logical date
(the *start* of the data interval), not when the run will be scheduled
(usually after the end of the data interval).
@@ -163,7 +163,7 @@ attributes:
* ``data_interval``: A :class:`~airflow.timetables.base.DataInterval` instance
describing the next run's data interval.
* ``run_after``: A ``pendulum.DateTime`` instance that tells the scheduler when
- the DAG run can be scheduled.
+ the Dag run can be scheduled.
A ``DagRunInfo`` can be created like this:
@@ -183,7 +183,7 @@ provides a shortcut for this:
info = DagRunInfo.interval(start=start, end=end)
assert info.data_interval.end == info.run_after # Always True.
-For reference, here's our plugin and DAG files in their entirety:
+For reference, here's our plugin and Dag files in their entirety:
.. exampleinclude:: /../src/airflow/example_dags/plugins/workday.py
:language: python
@@ -212,8 +212,8 @@ Parameterized Timetables
------------------------
Sometimes we need to pass some run-time arguments to the timetable. Continuing
-with our ``AfterWorkdayTimetable`` example, maybe we have dags running on
-different timezones, and we want to schedule some dags at 8am the next day,
+with our ``AfterWorkdayTimetable`` example, maybe we have Dags running on
+different timezones, and we want to schedule some Dags at 8am the next day,
instead of on midnight. Instead of creating a separate timetable for each
purpose, we'd want to do something like:
@@ -231,7 +231,7 @@ purpose, we'd want to do something like:
run_after=DateTime.combine(end.date(), self._schedule_at).replace(tzinfo=UTC),
)
-However, since the timetable is a part of the DAG, we need to tell Airflow how
+However, since the timetable is a part of the Dag, we need to tell Airflow how
to serialize it with the context we provide in ``__init__``. This is done by
implementing two additional methods on our timetable class:
@@ -247,9 +247,9 @@ implementing two additional methods on our timetable class:
def deserialize(cls, value: dict[str, Any]) -> Timetable:
return cls(Time.fromisoformat(value["schedule_at"]))
-When the DAG is being serialized, ``serialize`` is called to obtain a
+When the Dag is being serialized, ``serialize`` is called to obtain a
JSON-serializable value. That value is passed to ``deserialize`` when the
-serialized DAG is accessed by the scheduler to reconstruct the timetable.
+serialized Dag is accessed by the scheduler to reconstruct the timetable.
Timetable Display in UI
@@ -267,7 +267,7 @@ our ``SometimeAfterWorkdayTimetable`` class, for example, we could have:
def summary(self) -> str:
return f"after each workday, at {self._schedule_at}"
-So for a DAG declared like this:
+So for a Dag declared like this:
.. code-block:: python
@@ -309,7 +309,7 @@ You can also wrap this inside ``__init__``, if you want to derive description.
This is specially useful when you want to provide comprehensive description which is different from ``summary`` property.
-So for a DAG declared like this:
+So for a Dag declared like this:
.. code-block:: python
@@ -352,4 +352,4 @@ For example to have the Run ID show a "human friendly" date of when the run star
)
-Remember that the RunID is limited to 250 characters, and must be unique within a DAG.
+Remember that the RunID is limited to 250 characters, and must be unique within a Dag.
diff --git a/airflow-core/docs/howto/usage-cli.rst b/airflow-core/docs/howto/usage-cli.rst
index ce9869b0d41..e954947d502 100644
--- a/airflow-core/docs/howto/usage-cli.rst
+++ b/airflow-core/docs/howto/usage-cli.rst
@@ -65,22 +65,22 @@ Creating a Connection
For information on creating connections using the CLI, see :ref:`connection/cli`
-Exporting DAG structure as an image
+Exporting Dag structure as an image
-----------------------------------
-Airflow lets you print or save your DAG structure as an image. This is useful for documenting or sharing your DAG structure. You'll need to have `Graphviz <https://graphviz.gitlab.io/download/>`_ installed.
+Airflow lets you print or save your Dag structure as an image. This is useful for documenting or sharing your Dag structure. You'll need to have `Graphviz <https://graphviz.gitlab.io/download/>`_ installed.
-For example, to print the ``example_complex`` DAG to the terminal:
+For example, to print the ``example_complex`` Dag to the terminal:
.. code-block:: bash
airflow dags show example_complex
-This will print the rendered DAG structure to the screen in DOT format.
+This will print the rendered Dag structure to the screen in DOT format.
Multiple file formats are supported. To use them, add the argument ``--save [filename].[format]``.
-To save the ``example_complex`` DAG as a PNG file:
+To save the ``example_complex`` Dag as a PNG file:
.. code-block:: bash
@@ -91,7 +91,7 @@ This will save the following image as a file:
.. figure:: /img/usage_cli_export.png
:width: 100%
- Example DAG representation
+ Example Dag representation
The following file formats are supported:
@@ -133,24 +133,24 @@ The following file formats are supported:
* ``xlib``
* ``x11``
-By default, Airflow looks for dags in the directory specified by the ``dags_folder`` option in the
+By default, Airflow looks for Dags in the directory specified by the ``dags_folder`` option in the
``[core]`` section of the ``airflow.cfg`` file. You can select a new directory with the ``--subdir`` argument.
-Display dag structure
+Display Dag structure
---------------------
-Sometimes you will work on dags that contain complex dependencies. It is helpful then to preview
-the DAG to see if it is correct.
+Sometimes you will work on Dags that contain complex dependencies. It is helpful then to preview
+the Dag to see if it is correct.
If you have macOS, you can use `iTerm2 <https://iterm2.com/>`__ together with
-the `imgcat <https://www.iterm2.com/documentation-images.html>`__ script to display the DAG structure in the
+the `imgcat <https://www.iterm2.com/documentation-images.html>`__ script to display the Dag structure in the
console. You also need to have `Graphviz <https://graphviz.gitlab.io/download/>`_ installed.
Other terminals do not support the display of high-quality graphics. You can convert the image to a text
form, but its resolution will prevent you from reading it.
To do this, you should use the ``--imgcat`` switch in the ``airflow dags show`` command. For example, if you
-want to display ``example_bash_operator`` DAG then you can use the following command:
+want to display ``example_bash_operator`` Dag then you can use the following command:
.. code-block:: bash
@@ -160,7 +160,7 @@ You will see a similar result as in the screenshot below.
.. figure:: /img/usage_cli_imgcat.png
- Preview of DAG in iTerm2
+ Preview of Dag in iTerm2
Formatting commands output
--------------------------
@@ -249,15 +249,15 @@ Beware cascading deletes
Keep in mind that some tables have foreign key relationships defined with ``ON DELETE CASCADE`` so deletes in one table may trigger deletes in others. For example, the ``task_instance`` table keys to the ``dag_run`` table, so if a DagRun record is deleted, all of its associated task instances will also be deleted.
-Special handling for DAG runs
+Special handling for Dag runs
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Commonly, Airflow determines which DagRun to run next by looking up the latest DagRun. If you delete all DAG runs, Airflow may schedule an old DAG run that was already completed, e.g. if you have set ``catchup=True``. So the ``db clean`` command will preserve the latest non-manually-triggered DAG run to preserve continuity in scheduling.
+Commonly, Airflow determines which DagRun to run next by looking up the latest DagRun. If you delete all Dag runs, Airflow may schedule an old Dag run that was already completed, e.g. if you have set ``catchup=True``. So the ``db clean`` command will preserve the latest non-manually-triggered Dag run to preserve continuity in scheduling.
-Considerations for backfillable dags
+Considerations for backfillable Dags
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Not all dags are designed for use with Airflow's backfill command. But for those which are, special care is warranted. If you delete DAG runs, and if you run backfill over a range of dates that includes the deleted DAG runs, those runs will be recreated and run again. For this reason, if you have dags that fall into this category you may want to refrain from deleting DAG runs and only clean other large tables such as task instance and log etc.
+Not all Dags are designed for use with Airflow's backfill command. But for those which are, special care is warranted. If you delete Dag runs, and if you run backfill over a range of dates that includes the deleted Dag runs, those runs will be recreated and run again. For this reason, if you have Dags that fall into this category you may want to refrain from deleting Dag runs and only clean other large tables such as task instance and log etc.
.. _cli-db-migrate:
@@ -291,8 +291,8 @@ For a mapping between Airflow version and Alembic revision see :doc:`/migrations
.. note::
- It's highly recommended that you reserialize your dags with ``dags reserialize`` after you finish downgrading your Airflow environment (meaning, after you've downgraded the Airflow version installed in your Python environment, not immediately after you've downgraded the database).
- This is to ensure that the serialized dags are compatible with the downgraded version of Airflow.
+ It's highly recommended that you reserialize your Dags with ``dags reserialize`` after you finish downgrading your Airflow environment (meaning, after you've downgraded the Airflow version installed in your Python environment, not immediately after you've downgraded the database).
+ This is to ensure that the serialized Dags are compatible with the downgraded version of Airflow.
.. _cli-export-connections:
@@ -381,9 +381,9 @@ JSON example output:
airflow_db={"conn_type": "mysql", "login": "root", "password": "plainpassword", "host": "mysql", "schema": "airflow"}
druid_broker_default={"conn_type": "druid", "host": "druid-broker", "port": 8082, "extra": "{\"endpoint\": \"druid/v2/sql\"}"}
-Testing for DAG Import Errors
+Testing for Dag Import Errors
-----------------------------
-The CLI can be used to check whether any discovered dags have import errors via the ``list-import-errors`` subcommand. It is possible to create an automation step which fails if any dags cannot be imported by checking the command output, particularly when used with ``--output`` to generate a standard file format.
+The CLI can be used to check whether any discovered Dags have import errors via the ``list-import-errors`` subcommand. It is possible to create an automation step which fails if any Dags cannot be imported by checking the command output, particularly when used with ``--output`` to generate a standard file format.
For example, the default output when there are no errors is ``No data found``, and the json output is ``[]``. The check can then be run in CI or as prek hook to speed up the review process and testing.
Example command that fails if there are any errors, using `jq <https://jqlang.github.io/jq/>`__ to parse the output:
@@ -402,7 +402,7 @@ Example in a Jenkins pipeline:
.. code-block:: groovy
- stage('All dags are loadable') {
+ stage('All Dags are loadable') {
steps {
sh 'airflow dags list-import-errors | tee import_errors.txt && jq -e \'select(type=="array" and length == 0)\' import_errors.txt'
}
diff --git a/airflow-core/docs/howto/variable.rst b/airflow-core/docs/howto/variable.rst
index a07a04a1545..21eff7a6b3e 100644
--- a/airflow-core/docs/howto/variable.rst
+++ b/airflow-core/docs/howto/variable.rst
@@ -48,7 +48,7 @@ For example,
# To use JSON, store them as JSON strings
export AIRFLOW_VAR_FOO_BAZ='{"hello":"world"}'
-You can use them in your dags as:
+You can use them in your Dags as:
.. code-block:: python
diff --git a/airflow-core/docs/index.rst b/airflow-core/docs/index.rst
index 1b48a4a3b44..d3788b024c4 100644
--- a/airflow-core/docs/index.rst
+++ b/airflow-core/docs/index.rst
@@ -28,7 +28,7 @@ Workflows as code
=========================================
Airflow workflows are defined entirely in Python. This "workflows as code" approach brings several advantages:
-- **Dynamic**: Pipelines are defined in code, enabling dynamic dag generation and parameterization.
+- **Dynamic**: Pipelines are defined in code, enabling dynamic Dag generation and parameterization.
- **Extensible**: The Airflow framework includes a wide range of built-in operators and can be extended to fit your needs.
- **Flexible**: Airflow leverages the `Jinja <https://jinja.palletsprojects.com>`_ templating engine, allowing rich customizations.
@@ -48,7 +48,7 @@ Dags
:start-after: .. dag-definition-start
:end-before: .. dag-definition-end
-Let's look at a code snippet that defines a simple dag:
+Let's look at a code snippet that defines a simple Dag:
.. code-block:: python
@@ -57,7 +57,7 @@ Let's look at a code snippet that defines a simple dag:
from airflow.sdk import DAG, task
from airflow.providers.standard.operators.bash import BashOperator
- # A DAG represents a workflow, a collection of tasks
+ # A Dag represents a workflow, a collection of tasks
with DAG(dag_id="demo", start_date=datetime(2022, 1, 1), schedule="0 0 * * *") as dag:
# Tasks are represented as operators
hello = BashOperator(task_id="hello", bash_command="echo hello")
@@ -72,32 +72,32 @@ Let's look at a code snippet that defines a simple dag:
Here you see:
-- A dag named ``"demo"``, scheduled to run daily starting on January 1st, 2022. A dag is how Airflow represents a workflow.
+- A Dag named ``"demo"``, scheduled to run daily starting on January 1st, 2022. A Dag is how Airflow represents a workflow.
- Two tasks: One using a ``BashOperator`` to run a shell script, and another using the ``@task`` decorator to define a Python function.
- The ``>>`` operator defines a dependency between the two tasks and controls execution order.
-Airflow parses the script, schedules the tasks, and executes them in the defined order. The status of the ``"demo"`` dag
+Airflow parses the script, schedules the tasks, and executes them in the defined order. The status of the ``"demo"`` Dag
is displayed in the web interface:
.. image:: /img/ui-dark/demo_graph_and_code_view.png
- :alt: Demo dag in the Graph View, showing the status of one dag run along with dag code.
+ :alt: Demo Dag in the Graph View, showing the status of one Dag run along with Dag code.
|
This example uses a simple Bash command and Python function, but Airflow tasks can run virtually any code. You might use
-tasks to run a Spark job, move files between storage buckets, or send a notification email. Here's what that same dag looks
+tasks to run a Spark job, move files between storage buckets, or send a notification email. Here's what that same Dag looks
like over time, with multiple runs:
.. image:: /img/ui-dark/demo_grid_view_with_task_logs.png
- :alt: Demo dag in the Grid View, showing the status of all dag runs, as well as logs for a task instance
+ :alt: Demo Dag in the Grid View, showing the status of all Dag runs, as well as logs for a task instance
|
-Each column in the grid represents a single dag run. While the graph and grid views are most commonly used, Airflow provides
-several other views to help you monitor and troubleshoot workflows — such as the ``DAG Overview`` view:
+Each column in the grid represents a single Dag run. While the graph and grid views are most commonly used, Airflow provides
+several other views to help you monitor and troubleshoot workflows — such as the ``Dag Overview`` view:
.. image:: /img/ui-dark/demo_dag_overview_with_failed_tasks.png
- :alt: Overview of a complex dag in the Grid View, showing the status of all dag runs, as well as quick links to recently failed task logs
+ :alt: Overview of a complex Dag in the Grid View, showing the status of all Dag runs, as well as quick links to recently failed task logs
|
@@ -110,7 +110,7 @@ Why Airflow®?
Airflow is a platform for orchestrating batch workflows. It offers a flexible framework with a wide range of built-in operators
and makes it easy to integrate with new technologies.
-If your workflows have a clear start and end and run on a schedule, they're a great fit for Airflow dags.
+If your workflows have a clear start and end and run on a schedule, they're a great fit for Airflow Dags.
If you prefer coding over clicking, Airflow is built for you. Defining workflows as Python code provides several key benefits:
@@ -120,7 +120,7 @@ If you prefer coding over clicking, Airflow is built for you. Defining workflows
- **Extensibility**: Customize workflows using a large ecosystem of existing components — or build your own.
Airflow's rich scheduling and execution semantics make it easy to define complex, recurring pipelines. From the web interface,
-you can manually trigger dags, inspect logs, and monitor task status. You can also backfill dag runs to process historical
+you can manually trigger Dags, inspect logs, and monitor task status. You can also backfill Dag runs to process historical
data, or rerun only failed tasks to minimize cost and time.
The Airflow platform is highly customizable. With the :doc:`public-airflow-interface` you can extend and adapt nearly
@@ -133,7 +133,7 @@ others via the `community <https://airflow.apache.org/community>`_, `Slack <http
Why not Airflow®?
=================
-Airflow® is designed for finite, batch-oriented workflows. While you can trigger dags using the CLI or REST API, Airflow is not
+Airflow® is designed for finite, batch-oriented workflows. While you can trigger Dags using the CLI or REST API, Airflow is not
intended for continuously running, event-driven, or streaming workloads. That said, Airflow often complements streaming systems like Apache Kafka.
Kafka handles real-time ingestion, writing data to storage. Airflow can then periodically pick up that data and process it in batch.
diff --git a/airflow-core/docs/installation/index.rst b/airflow-core/docs/installation/index.rst
index f6a327444a2..b033a091070 100644
--- a/airflow-core/docs/installation/index.rst
+++ b/airflow-core/docs/installation/index.rst
@@ -346,7 +346,7 @@ The requirements that Airflow might need depend on many factors, including (but
with more (or less) pods and containers that are needed that might depend on particular choice of
the technology/cloud/integration of monitoring etc.
* Technical details of database, hardware, network, etc. that your deployment is running on
- * The complexity of the code you add to your DAGS, configuration, plugins, settings etc. (note, that
+ * The complexity of the code you add to your Dags, configuration, plugins, settings etc. (note, that
Airflow runs the code that Dag author and Deployment Manager provide)
* The number and choice of providers you install and use (Airflow has more than 80 providers) that can
be installed by choice of the Deployment Manager and using them might require more resources.
@@ -355,7 +355,7 @@ The requirements that Airflow might need depend on many factors, including (but
* The number of DagRuns and tasks instances you run with parallel instances of each in consideration
* How complex are the tasks you run
-The above "DAG" characteristics will change over time and even will change depending on the time of the day
+The above "Dag" characteristics will change over time and even will change depending on the time of the day
or week, so you have to be prepared to continuously monitor the system and adjust the parameters to make
it works smoothly.
diff --git a/airflow-core/docs/installation/upgrading_to_airflow3.rst b/airflow-core/docs/installation/upgrading_to_airflow3.rst
index 64d5a02ed9a..c0aa1b8e33f 100644
--- a/airflow-core/docs/installation/upgrading_to_airflow3.rst
+++ b/airflow-core/docs/installation/upgrading_to_airflow3.rst
@@ -46,7 +46,7 @@ Airflow 3.x Architecture
- The API server is currently the sole access point for the metadata DB for tasks and workers.
- It supports several applications: the Airflow REST API, an internal API for the Airflow UI that hosts static JS, and an API for workers to interact with when executing TIs via the task execution interface.
- Workers communicate with the API server instead of directly with the database.
-- DAG processor and Triggerer utilize the task execution mechanism for their tasks, especially when they require variables or connections.
+- Dag processor and Triggerer utilize the task execution mechanism for their tasks, especially when they require variables or connections.
Database Access Restrictions
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -78,19 +78,19 @@ Step 2: Clean and back up your existing Airflow Instance
upgrade process. These schema changes can take a long time if the database is large. For a faster, safer migration, we recommend that you clean up your Airflow meta-database before the upgrade.
You can use the ``airflow db clean`` :ref:`Airflow CLI command<cli-db-clean>` to trim your Airflow database.
-- Ensure that there are no errors related to dag processing, such as ``AirflowDagDuplicatedIdException``. You should
- be able to run ``airflow dags reserialize`` with no errors. If you have to resolve errors from dag processing,
- ensure you deploy your changes to your old instance prior to upgrade, and wait until your dags have all been reprocessed
+- Ensure that there are no errors related to Dag processing, such as ``AirflowDagDuplicatedIdException``. You should
+ be able to run ``airflow dags reserialize`` with no errors. If you have to resolve errors from Dag processing,
+ ensure you deploy your changes to your old instance prior to upgrade, and wait until your Dags have all been reprocessed
(and all errors gone) before you proceed with upgrade.
-Step 3: Dag authors - Check your Airflow dags for compatibility
+Step 3: Dag authors - Check your Airflow Dags for compatibility
----------------------------------------------------------------
-To minimize friction for users upgrading from prior versions of Airflow, we have created a dag upgrade check utility using `Ruff <https://docs.astral.sh/ruff/>`_ combined with `AIR <https://docs.astral.sh/ruff/rules/#airflow-air>`_ rules.
+To minimize friction for users upgrading from prior versions of Airflow, we have created a Dag upgrade check utility using `Ruff <https://docs.astral.sh/ruff/>`_ combined with `AIR <https://docs.astral.sh/ruff/rules/#airflow-air>`_ rules.
The rules AIR301 and AIR302 indicate breaking changes in Airflow 3, while AIR311 and AIR312 highlight changes that are not currently breaking but are strongly recommended for updates.
The latest available ``ruff`` version will have the most up-to-date rules, but be sure to use at least version ``0.11.13``. The below example demonstrates how to check
-for dag incompatibilities that will need to be fixed before they will work as expected on Airflow 3.
+for Dag incompatibilities that will need to be fixed before they will work as expected on Airflow 3.
.. code-block:: bash
@@ -109,7 +109,7 @@ Some changes can be automatically fixed. To do so, run the following command:
ruff check dags/ --select AIR301 --fix --preview
-Some of the fixes are marked as unsafe. Unsafe fixes usually do not break dag code. They're marked as unsafe as they may change some runtime behavior. For more information, see `Fix Safety <https://docs.astral.sh/ruff/linter/#fix-safety>`_.
+Some of the fixes are marked as unsafe. Unsafe fixes usually do not break Dag code. They're marked as unsafe as they may change some runtime behavior. For more information, see `Fix Safety <https://docs.astral.sh/ruff/linter/#fix-safety>`_.
To trigger these fixes, run the following command:
.. code-block:: bash
@@ -132,7 +132,7 @@ Step 4: Install the Standard Provider
- Some of the commonly used Operators which were bundled as part of the ``airflow-core`` package (for example ``BashOperator`` and ``PythonOperator``)
have now been split out into a separate package: ``apache-airflow-providers-standard``.
-- For convenience, this package can also be installed on Airflow 2.x versions, so that DAGs can be modified to reference these Operators from the standard provider
+- For convenience, this package can also be installed on Airflow 2.x versions, so that Dags can be modified to reference these Operators from the standard provider
package instead of Airflow Core.
Step 5: Review custom operators for direct db access
@@ -183,7 +183,7 @@ In Airflow 3, the Webserver has become a generic API server. The API server can
airflow api-server
-The dag processor must now be started independently, even for local or development setups:
+The Dag processor must now be started independently, even for local or development setups:
.. code-block:: bash
@@ -203,9 +203,9 @@ These include:
- **Sequential Executor**: Replaced by LocalExecutor, which can be used with SQLite for local development use cases.
- **CeleryKubernetesExecutor and LocalKubernetesExecutor**: Replaced by `Multiple Executor Configuration <https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/index.html#using-multiple-executors-concurrently>`_
- **SLAs**: Deprecated and removed; Will be replaced by forthcoming `Deadline Alerts <https://cwiki.apache.org/confluence/x/tglIEw>`_.
-- **Subdir**: Used as an argument on many CLI commands, ``--subdir`` or ``-S`` has been superseded by :doc:`DAG bundles </administration-and-deployment/dag-bundles>`.
+- **Subdir**: Used as an argument on many CLI commands, ``--subdir`` or ``-S`` has been superseded by :doc:`Dag bundles </administration-and-deployment/dag-bundles>`.
- **REST API** (``/api/v1``) replaced: Use the modern FastAPI-based stable ``/api/v2`` instead; see :doc:`Airflow API v2 </stable-rest-api-ref>` for details.
-- **Some Airflow context variables**: The following keys are no longer available in a :ref:`task instance's context <templates:variables>`. If not replaced, will cause dag errors:
+- **Some Airflow context variables**: The following keys are no longer available in a :ref:`task instance's context <templates:variables>`. If not replaced, will cause Dag errors:
- ``tomorrow_ds``
- ``tomorrow_ds_nodash``
- ``yesterday_ds``
@@ -218,7 +218,7 @@ These include:
- ``next_ds_nodash``
- ``next_ds``
- ``execution_date``
-- The ``catchup_by_default`` dag parameter is now ``False`` by default.
+- The ``catchup_by_default`` Dag parameter is now ``False`` by default.
- The ``create_cron_data_intervals`` configuration is now ``False`` by default. This means that the ``CronTriggerTimetable`` will be used by default instead of the ``CronDataIntervalTimetable``
- **Simple Auth** is now default ``auth_manager``. To continue using FAB as the Auth Manager, please install the FAB provider and set ``auth_manager`` to ``FabAuthManager``:
diff --git a/airflow-core/docs/public-airflow-interface.rst b/airflow-core/docs/public-airflow-interface.rst
index 32fc89a4173..615508754c6 100644
--- a/airflow-core/docs/public-airflow-interface.rst
+++ b/airflow-core/docs/public-airflow-interface.rst
@@ -32,7 +32,7 @@ Public Interface of Airflow
The Public Interface of Apache Airflow is the collection of interfaces and behaviors in Apache Airflow
whose changes are governed by semantic versioning. A user interacts with Airflow's public interface
-by creating and managing dags, managing tasks and dependencies,
+by creating and managing Dags, managing tasks and dependencies,
and extending Airflow capabilities by writing new executors, plugins, operators and providers. The
Public Interface can be useful for building custom tools and integrations with other systems,
and for automating certain aspects of the Airflow workflow.
@@ -67,7 +67,7 @@ The following are some examples of the public interface of Airflow:
* When you are writing your own operators or hooks. This is commonly done when no hook or operator exists for your use case, or when perhaps when one exists but you need to customize the behavior.
* When writing new :doc:`Plugins <administration-and-deployment/plugins>` that extend Airflow's functionality beyond
- DAG building blocks. Secrets, Timetables, Triggers, Listeners are all examples of such functionality. This
+ Dag building blocks. Secrets, Timetables, Triggers, Listeners are all examples of such functionality. This
is usually done by users who manage Airflow instances.
* Bundling custom Operators, Hooks, Plugins and releasing them together via
:doc:`providers <apache-airflow-providers:index>` - this is usually done by those who intend to
@@ -92,9 +92,9 @@ Using the Public Interface for Dag authors
==========================================
The primary interface for Dag authors is the :doc:`airflow.sdk namespace <core-concepts/taskflow>`.
-This provides a stable, well-defined interface for creating DAGs and tasks that is not subject to internal
+This provides a stable, well-defined interface for creating Dags and tasks that is not subject to internal
implementation changes. The goal of this change is to decouple Dag authoring from Airflow internals (Scheduler,
-API Server, etc.), providing a version-agnostic, stable interface for writing and maintaining DAGs across Airflow versions.
+API Server, etc.), providing a version-agnostic, stable interface for writing and maintaining Dags across Airflow versions.
**Key Imports from airflow.sdk:**
@@ -137,21 +137,21 @@ see the :doc:`Migration Guide <installation/upgrading_to_airflow3>`.
For an exhaustive list of available classes, decorators, and functions, check ``airflow.sdk.__all__``.
-All DAGs should update imports to use ``airflow.sdk`` instead of referencing internal Airflow modules directly.
+All Dags should update imports to use ``airflow.sdk`` instead of referencing internal Airflow modules directly.
Legacy import paths (e.g., ``airflow.models.dag.DAG``, ``airflow.decorator.task``) are deprecated and will be
removed in a future Airflow version.
Dags
====
-The DAG is Airflow's core entity that represents a recurring workflow. You can create a DAG by
-instantiating the :class:`~airflow.sdk.DAG` class in your DAG file. Dags can also have parameters
+The Dag is Airflow's core entity that represents a recurring workflow. You can create a Dag by
+instantiating the :class:`~airflow.sdk.DAG` class in your Dag file. Dags can also have parameters
specified via :class:`~airflow.sdk.Param` class.
-The recommended way to create DAGs is using the :func:`~airflow.sdk.dag` decorator
+The recommended way to create Dags is using the :func:`~airflow.sdk.dag` decorator
from the airflow.sdk namespace.
-Airflow has a set of example dags that you can use to learn how to write dags
+Airflow has a set of example Dags that you can use to learn how to write Dags
.. toctree::
:includehidden:
@@ -161,22 +161,22 @@ Airflow has a set of example dags that you can use to learn how to write dags
_api/airflow/example_dags/index
-You can read more about dags in :doc:`Dags <core-concepts/dags>`.
+You can read more about Dags in :doc:`Dags <core-concepts/dags>`.
-References for the modules used in dags are here:
+References for the modules used in Dags are here:
.. note::
The airflow.sdk namespace provides the primary interface for Dag authors.
For detailed API documentation, see the `Task SDK Reference <https://airflow.apache.org/docs/task-sdk/stable/>`_.
.. note::
- The :class:`~airflow.models.dagbag.DagBag` class is used internally by Airflow for loading DAGs
+ The :class:`~airflow.models.dagbag.DagBag` class is used internally by Airflow for loading Dags
from files and folders. Dag authors should use the :class:`~airflow.sdk.DAG` class from the
airflow.sdk namespace instead.
.. note::
- The :class:`~airflow.models.dagrun.DagRun` class is used internally by Airflow for DAG run
- management. Dag authors should access DAG run information through the Task Context via
+ The :class:`~airflow.models.dagrun.DagRun` class is used internally by Airflow for Dag run
+ management. Dag authors should access Dag run information through the Task Context via
:func:`~airflow.sdk.get_current_context` or use the :class:`~airflow.sdk.types.DagRunProtocol`
interface.
@@ -195,7 +195,7 @@ Subclasses of BaseOperator which are published in Apache Airflow are public in *
Task Instances
==============
-Task instances are the individual runs of a single task in a DAG (in a DAG Run). Task instances are accessed through
+Task instances are the individual runs of a single task in a Dag (in a Dag Run). Task instances are accessed through
the Task Context via :func:`~airflow.sdk.get_current_context`. Direct database access is not possible.
.. note::
@@ -205,7 +205,7 @@ the Task Context via :func:`~airflow.sdk.get_current_context`. Direct database a
Task Instance Keys
==================
-Task instance keys are unique identifiers of task instances in a DAG (in a DAG Run). A key is a tuple that consists of
+Task instance keys are unique identifiers of task instances in a Dag (in a Dag Run). A key is a tuple that consists of
``dag_id``, ``task_id``, ``run_id``, ``try_number``, and ``map_index``.
Direct access to task instance keys via the :class:`~airflow.models.taskinstance.TaskInstance`
@@ -370,7 +370,7 @@ Timetables
==========
Custom timetable implementations provide Airflow's scheduler additional logic to
-schedule DAG runs in ways not possible with built-in schedule expressions.
+schedule Dag runs in ways not possible with built-in schedule expressions.
All Timetables derive from :class:`~airflow.timetables.base.Timetable`.
Airflow has a set of Timetables that are considered public. You are free to extend their functionality
@@ -388,10 +388,10 @@ You can read more about Timetables in :doc:`howto/timetable`.
Listeners
=========
-Listeners enable you to respond to DAG/Task lifecycle events.
+Listeners enable you to respond to Dag/Task lifecycle events.
This is implemented via :class:`~airflow.listeners.listener.ListenerManager` class that provides hooks that
-can be implemented to respond to DAG/Task lifecycle events.
+can be implemented to respond to Dag/Task lifecycle events.
.. versionadded:: 2.5
@@ -493,7 +493,7 @@ implemented in the community providers.
Decorators
==========
-Dag authors can use decorators to author dags using the :doc:`TaskFlow <core-concepts/taskflow>` concept.
+Dag authors can use decorators to author Dags using the :doc:`TaskFlow <core-concepts/taskflow>` concept.
All Decorators derive from :class:`~airflow.sdk.bases.decorator.TaskDecorator`.
The primary decorators for Dag authors are now in the airflow.sdk namespace:
@@ -525,7 +525,7 @@ about notifications in :doc:`howto/notifications`.
Cluster Policies
================
-Cluster Policies are the way to dynamically apply cluster-wide policies to the dags being parsed or tasks
+Cluster Policies are the way to dynamically apply cluster-wide policies to the Dags being parsed or tasks
being executed. You can read more about Cluster Policies in :doc:`administration-and-deployment/cluster-policies`.
Lineage
@@ -557,8 +557,8 @@ but in Airflow they are not parts of the Public Interface and might change any t
in a backwards-compatible way.
**Direct metadata database access from task code is no longer allowed**.
-Task code cannot directly access the metadata database to query DAG state, task history,
-or DAG runs. Instead, use one of the following alternatives:
+Task code cannot directly access the metadata database to query Dag state, task history,
+or Dag runs. Instead, use one of the following alternatives:
* **Task Context**: Use :func:`~airflow.sdk.get_current_context` to access task instance
information and methods like :meth:`~airflow.sdk.types.RuntimeTaskInstanceProtocol.get_dr_count`,
@@ -595,7 +595,7 @@ Example of using Task Context instead of direct database access:
dr_count = ti.get_dr_count(dag_id="example_dag")
dagrun_state = ti.get_dagrun_state(dag_id="example_dag", run_id=dag_run.run_id)
- return f"DAG run count: {dr_count}, current state: {dagrun_state}"
+ return f"Dag run count: {dr_count}, current state: {dagrun_state}"
check_state()
diff --git a/airflow-core/docs/security/audit_logs.rst b/airflow-core/docs/security/audit_logs.rst
index 6b28b637a8c..8c86a224736 100644
--- a/airflow-core/docs/security/audit_logs.rst
+++ b/airflow-core/docs/security/audit_logs.rst
@@ -25,10 +25,10 @@ Overview
Audit logs are a critical component of any system that needs to maintain a high level of security and compliance.
They provide a way to track user actions and system events, which can be used to troubleshoot issues, detect security breaches, and ensure regulatory compliance.
-In Airflow, audit logs are used to track user actions and system events that occur during the execution of dags and tasks.
+In Airflow, audit logs are used to track user actions and system events that occur during the execution of Dags and tasks.
They are stored in a database and can be accessed through the Airflow UI.
-To be able to see audit logs, a user needs to have the ``Audit Logs.can_read`` permission. Such user will be able to see all audit logs, independently of the dags permissions applied.
+To be able to see audit logs, a user needs to have the ``Audit Logs.can_read`` permission. Such user will be able to see all audit logs, independently of the Dags permissions applied.
Level of Audit Logs
@@ -38,7 +38,7 @@ Audit logs exist at the task level and the user level.
- Task Level: At the task level, audit logs capture information related to the execution of a task, such as the start time, end time, and status of the task.
-- User Level: At the user level, audit logs capture information related to user actions, such as creating, modifying, or deleting a DAG or task.
+- User Level: At the user level, audit logs capture information related to user actions, such as creating, modifying, or deleting a Dag or task.
Location of Audit Logs
@@ -52,11 +52,11 @@ Types of Events
Airflow provides a set of predefined events that can be tracked in audit logs. These events include, but aren't limited to:
-- ``trigger``: Triggering a DAG
+- ``trigger``: Triggering a Dag
- ``[variable,connection].create``: A user created a Connection or Variable
- ``[variable,connection].edit``: A user modified a Connection or Variable
- ``[variable,connection].delete``: A user deleted a Connection or Variable
-- ``delete``: A user deleted a DAG or task
+- ``delete``: A user deleted a Dag or task
- ``failed``: Airflow or a user set a task as failed
- ``success``: Airflow or a user set a task as success
- ``retry``: Airflow or a user retried a task instance
diff --git a/airflow-core/docs/security/deprecated_permissions.rst b/airflow-core/docs/security/deprecated_permissions.rst
index 69deb91e58e..35334c9c430 100644
--- a/airflow-core/docs/security/deprecated_permissions.rst
+++ b/airflow-core/docs/security/deprecated_permissions.rst
@@ -51,7 +51,7 @@ replacement available from Airflow core:
* ``airflow.security.permissions.ACTION_*`` --> ``airflow.api_fastapi.auth.managers.base_auth_manager.ResourceMethod``
* ``airflow.security.permissions.RESOURCE_*`` --> ``airflow.api_fastapi.auth.managers.models.resource_details``
-* ``DAG.access_control`` --> DAG-level permissions should be handled by the chosen Auth Manager's ``filter_authorized_dag_ids`` method.
+* ``DAG.access_control`` --> Dag-level permissions should be handled by the chosen Auth Manager's ``filter_authorized_dag_ids`` method.
If you maintain a custom :doc:`/core-concepts/auth-manager/index` which relies on the deprecated module, it is
recommended you refer to the ``SimpleAuthManager``'s `source code <https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/api_fastapi/auth/managers/simple/simple_auth_manager.py>`_
diff --git a/airflow-core/docs/security/kerberos.rst b/airflow-core/docs/security/kerberos.rst
index 0dea38c60bb..38c77d5a66c 100644
--- a/airflow-core/docs/security/kerberos.rst
+++ b/airflow-core/docs/security/kerberos.rst
@@ -19,7 +19,7 @@ Kerberos
--------
Airflow has initial support for Kerberos. This means that Airflow can renew Kerberos
-tickets for itself and store it in the ticket cache. The hooks and dags can make use of ticket
+tickets for itself and store it in the ticket cache. The hooks and Dags can make use of ticket
to authenticate against kerberized services.
Limitations
@@ -150,7 +150,7 @@ Of course if you need to tighten your security replace the asterisk with somethi
Using Kerberos authentication
'''''''''''''''''''''''''''''
-The Hive hook has been updated to take advantage of Kerberos authentication. To allow your dags to
+The Hive hook has been updated to take advantage of Kerberos authentication. To allow your Dags to
use it, simply update the connection details with, for example:
.. code-block:: json
@@ -160,20 +160,20 @@ use it, simply update the connection details with, for example:
Adjust the principal to your settings. The ``_HOST`` part will be replaced by the fully qualified domain name of
the server.
-You can specify if you would like to use the DAG owner as the user for the connection or the user specified in the login
+You can specify if you would like to use the Dag owner as the user for the connection or the user specified in the login
section of the connection. For the login user, specify the following as extra:
.. code-block:: json
{ "use_beeline": true, "principal": "hive/_HOST@EXAMPLE.COM", "proxy_user": "login"}
-For the DAG owner use:
+For the Dag owner use:
.. code-block:: json
{ "use_beeline": true, "principal": "hive/_HOST@EXAMPLE.COM", "proxy_user": "owner"}
-and in your DAG, when initializing the HiveOperator, specify:
+and in your Dag, when initializing the HiveOperator, specify:
.. code-block:: bash
diff --git a/airflow-core/docs/security/secrets/mask-sensitive-values.rst b/airflow-core/docs/security/secrets/mask-sensitive-values.rst
index 60c7de44b37..1663883b2ca 100644
--- a/airflow-core/docs/security/secrets/mask-sensitive-values.rst
+++ b/airflow-core/docs/security/secrets/mask-sensitive-values.rst
@@ -52,7 +52,7 @@ Adding your own masks
"""""""""""""""""""""
If you want to mask an additional secret that is not already masked by one of the above methods, you can do it in
-your DAG file or operator's ``execute`` function using the ``mask_secret`` function. For example:
+your Dag file or operator's ``execute`` function using the ``mask_secret`` function. For example:
.. code-block:: python
diff --git a/airflow-core/docs/security/security_model.rst b/airflow-core/docs/security/security_model.rst
index 1915b3117ab..10b2ad3fb3e 100644
--- a/airflow-core/docs/security/security_model.rst
+++ b/airflow-core/docs/security/security_model.rst
@@ -39,7 +39,7 @@ This is why Airflow has the following user types:
* Deployment Managers - overall responsible for the Airflow installation, security and configuration
* Authenticated UI users - users that can access Airflow UI and API and interact with it
-* Dag authors - responsible for creating dags and submitting them to Airflow
+* Dag authors - responsible for creating Dags and submitting them to Airflow
You can see more on how the user types influence Airflow's architecture in :doc:`/core-concepts/overview`,
including, seeing the diagrams of less and more complex deployments.
@@ -61,10 +61,10 @@ model.
Dag authors
...........
-They can create, modify, and delete DAG files. The
-code in DAG files is executed on workers and in the DAG Processor.
+They can create, modify, and delete Dag files. The
+code in Dag files is executed on workers and in the Dag Processor.
Therefore, Dag authors can create and change code executed on workers
-and the DAG Processor and potentially access the credentials that the DAG
+and the Dag Processor and potentially access the credentials that the Dag
code uses to access external systems. Dag authors have full access
to the metadata database.
@@ -86,7 +86,7 @@ Capabilities of authenticated UI users
The capabilities of **Authenticated UI users** can vary depending on
what roles have been configured by the Deployment Manager or Admin users
as well as what permissions those roles have. Permissions on roles can be
-scoped as tightly as a single DAG, for example, or as broad as Admin.
+scoped as tightly as a single Dag, for example, or as broad as Admin.
Below are four general categories to help conceptualize some of the
capabilities authenticated users may have:
@@ -114,13 +114,13 @@ to other users, and access audit logs - only admins are able to do this. Otherwi
Connection configuration users
..............................
-They configure connections and potentially execute code on workers during DAG execution. Trust is
+They configure connections and potentially execute code on workers during Dag execution. Trust is
required to prevent misuse of these privileges. They have full write-only access
to sensitive credentials stored in connections and can modify them, but cannot view them.
Access to write sensitive information through connection configuration
should be trusted not to be abused. They also have the ability to configure connections wrongly
that might create a API Server Denial of Service situations and specify insecure connection options
-which might create situations where executing dags will lead to arbitrary Remote Code Execution
+which might create situations where executing Dags will lead to arbitrary Remote Code Execution
for some providers - either community released or custom ones.
Those users should be highly trusted not to misuse this capability.
@@ -142,14 +142,14 @@ They can view audit events for the whole Airflow installation.
Regular users
.............
-They can view and interact with the UI and API. They are able to view and edit dags,
-task instances, and DAG runs, and view task logs.
+They can view and interact with the UI and API. They are able to view and edit Dags,
+task instances, and Dag runs, and view task logs.
Viewer users
............
-They can view information related to dags, in a read only fashion, task logs, and other relevant details.
-This role is suitable for users who require read-only access without the ability to trigger or modify dags.
+They can view information related to Dags, in a read only fashion, task logs, and other relevant details.
+This role is suitable for users who require read-only access without the ability to trigger or modify Dags.
Viewers also do not have permission to access audit logs.
@@ -158,11 +158,11 @@ For more information on the capabilities of authenticated UI users, see :doc:`ap
Capabilities of Dag authors
---------------------------
-Dag authors are able to create or edit code - via Python files placed in a dag bundle - that will be executed
+Dag authors are able to create or edit code - via Python files placed in a Dag bundle - that will be executed
in a number of circumstances. The code to execute is neither verified, checked nor sand-boxed by Airflow
(that would be very difficult if not impossible to do), so effectively Dag authors can execute arbitrary
code on the workers (part of Celery Workers for Celery Executor, local processes run by scheduler in case
-of Local Executor, Task Kubernetes POD in case of Kubernetes Executor), in the DAG Processor
+of Local Executor, Task Kubernetes POD in case of Kubernetes Executor), in the Dag Processor
and in the Triggerer.
There are several consequences of this model chosen by Airflow, that deployment managers need to be aware of:
@@ -199,14 +199,14 @@ enforcement mechanisms that would allow to isolate tasks that are using deferrab
each other and arbitrary code from various tasks can be executed in the same process/machine. Deployment
Manager must trust that Dag authors will not abuse this capability.
-DAG files not needed for Scheduler and API Server
+Dag files not needed for Scheduler and API Server
.................................................
The Deployment Manager might isolate the code execution provided by Dag authors - particularly in
Scheduler and API Server by making sure that the Scheduler and API Server don't even
-have access to the DAG Files. Generally speaking - no Dag author provided code should ever be
+have access to the Dag Files. Generally speaking - no Dag author provided code should ever be
executed in the Scheduler or API Server process. This means the deployment manager can exclude credentials
-needed for dag bundles on the Scheduler and API Server - but the bundles must still be configured on those
+needed for Dag bundles on the Scheduler and API Server - but the bundles must still be configured on those
components.
Allowing Dag authors to execute selected code in Scheduler and API Server
@@ -216,9 +216,9 @@ There are a number of functionalities that allow the Dag author to use pre-regis
executed in the Scheduler or API Server process - for example they can choose custom Timetables, UI plugins,
Connection UI Fields, Operator extra links, macros, listeners - all of those functionalities allow the
Dag author to choose the code that will be executed in the Scheduler or API Server process. However this
-should not be arbitrary code that Dag author can add dag bundles. All those functionalities are
+should not be arbitrary code that Dag author can add Dag bundles. All those functionalities are
only available via ``plugins`` and ``providers`` mechanisms where the code that is executed can only be
-provided by installed packages (or in case of plugins it can also be added to PLUGINS folder where DAG
+provided by installed packages (or in case of plugins it can also be added to PLUGINS folder where Dag
authors should not have write access to). PLUGINS_FOLDER is a legacy mechanism coming from Airflow 1.10
- but we recommend using entrypoint mechanism that allows the Deployment Manager to - effectively -
choose and register the code that will be executed in those contexts. Dag author has no access to
@@ -232,11 +232,11 @@ The Deployment Manager might decide to introduce additional control mechanisms t
executing arbitrary code. This is all fully in hands of the Deployment Manager and it is discussed in the
following chapter.
-Access to all dags
+Access to all Dags
........................................................................
-All Dag authors have access to all dags in the Airflow deployment. This means that they can view, modify,
-and update any dag without restrictions at any time.
+All Dag authors have access to all Dags in the Airflow deployment. This means that they can view, modify,
+and update any Dag without restrictions at any time.
Responsibilities of Deployment Managers
---------------------------------------
@@ -265,11 +265,11 @@ Limiting Dag author capabilities
.................................
The Deployment Manager might also use additional mechanisms to prevent Dag authors from executing
-arbitrary code - for example they might introduce tooling around DAG submission that would allow
+arbitrary code - for example they might introduce tooling around Dag submission that would allow
to review the code before it is deployed, statically-check it and add other ways to prevent malicious
-code to be submitted. The way submitting code to a DAG bundle is done and protected is completely
+code to be submitted. The way submitting code to a Dag bundle is done and protected is completely
up to the Deployment Manager - Airflow does not provide any tooling or mechanisms around it and it
-expects that the Deployment Manager will provide the tooling to protect access to DAG bundles and
+expects that the Deployment Manager will provide the tooling to protect access to Dag bundles and
make sure that only trusted code is submitted there.
Airflow does not implement any of those feature natively, and delegates it to the deployment managers
@@ -288,7 +288,7 @@ Examples of fine-grained access control include (but are not limited to):
* Limiting login permissions: Restricting the accounts that users can log in with, allowing only specific
accounts or roles belonging to access the Airflow system.
-* Access restrictions to views or dags: Controlling user access to certain views or specific dags,
+* Access restrictions to views or Dags: Controlling user access to certain views or specific Dags,
ensuring that users can only view or interact with authorized components.
Future: multi-tenancy isolation
diff --git a/airflow-core/docs/security/sql.rst b/airflow-core/docs/security/sql.rst
index fe33f2450ae..b6b81d2180c 100644
--- a/airflow-core/docs/security/sql.rst
+++ b/airflow-core/docs/security/sql.rst
@@ -21,6 +21,6 @@ SQL Injection
Previously, Airflow issued CVE like `CVE-2025-27018 SQL injection in MySQL provider core function <https://www.cve.org/CVERecord?id=CVE-2025-27018/>`_.
The CVE were about the ability to inject SQL without considering the actor performing it.
Airflow will no longer issue CVE for cases of SQL Injection unless the reporter can demonstrate a scenario of exploitation.
-For example, if in a security report the only actor that can operate the injection is Actor who has access to DAGs folder the report will be rejected.
+For example, if in a security report the only actor that can operate the injection is Actor who has access to Dags folder the report will be rejected.
When submitting a security report of SQL injection the reporter must explain who is the user that can utilize the injection and how the user gained access to be able to perform it.
-In simple words, if a user has legit access to write and access the specific DAG, there is no risk that this user will do SQL injection.
+In simple words, if a user has legit access to write and access the specific Dag, there is no risk that this user will do SQL injection.
diff --git a/airflow-core/docs/start.rst b/airflow-core/docs/start.rst
index eadc39469ef..3ad574bdeef 100644
--- a/airflow-core/docs/start.rst
+++ b/airflow-core/docs/start.rst
@@ -89,7 +89,7 @@ This quick start guide will help you bootstrap an Airflow standalone instance on
5. Access the Airflow UI:
- Visit ``localhost:8080`` in your browser and log in with the admin account details shown in the terminal. Enable the ``example_bash_operator`` DAG in the home page.
+ Visit ``localhost:8080`` in your browser and log in with the admin account details shown in the terminal. Enable the ``example_bash_operator`` Dag in the home page.
Upon running these commands, Airflow will create the ``$AIRFLOW_HOME`` folder
and create the "airflow.cfg" file with defaults that will get you going fast.
@@ -104,7 +104,7 @@ from the ``standalone`` command we use here to running the components
separately. You can read more in :doc:`/administration-and-deployment/production-deployment`.
Here are a few commands that will trigger a few task instances. You should
-be able to see the status of the jobs change in the ``example_bash_operator`` DAG as you
+be able to see the status of the jobs change in the ``example_bash_operator`` Dag as you
run the commands below.
.. code-block:: bash
diff --git a/airflow-core/docs/templates-ref.rst b/airflow-core/docs/templates-ref.rst
index 1d226b1ec12..1ce0b1852e7 100644
--- a/airflow-core/docs/templates-ref.rst
+++ b/airflow-core/docs/templates-ref.rst
@@ -23,7 +23,7 @@ Templates reference
Variables, macros and filters can be used in templates (see the :ref:`concepts:jinja-templating` section)
The following come for free out of the box with Airflow.
-Additional custom macros can be added globally through :doc:`administration-and-deployment/plugins`, or at a DAG level through the
+Additional custom macros can be added globally through :doc:`administration-and-deployment/plugins`, or at a Dag level through the
``DAG.user_defined_macros`` argument.
.. _templates:variables:
@@ -38,7 +38,7 @@ Variable Type Description
=========================================== ===================== ===================================================================
``{{ data_interval_start }}`` `pendulum.DateTime`_ Start of the data interval. Added in version 2.2.
``{{ data_interval_end }}`` `pendulum.DateTime`_ End of the data interval. Added in version 2.2.
-``{{ logical_date }}`` `pendulum.DateTime`_ | A date-time that logically identifies the current DAG run. This value does not contain any semantics, but is simply a value for identification.
+``{{ logical_date }}`` `pendulum.DateTime`_ | A date-time that logically identifies the current Dag run. This value does not contain any semantics, but is simply a value for identification.
| Use ``data_interval_start`` and ``data_interval_end`` instead if you want a value that has real-world semantics,
| such as to get a slice of rows from the database based on timestamps.
``{{ exception }}`` None | str | | Error occurred while running task instance.
@@ -58,7 +58,7 @@ Variable Type Description
``{{ outlets }}`` list List of outlets declared on the task.
``{{ outlet_events }}`` dict[str, ...] | Accessors to attach information to asset events that will be emitted by the current task.
| See :doc:`Assets <authoring-and-scheduling/asset-scheduling>`. Added in version 2.10.
-``{{ dag }}`` DAG The currently running :class:`~airflow.models.dag.DAG`. You can read more about dags in :doc:`Dags <core-concepts/dags>`.
+``{{ dag }}`` DAG The currently running :class:`~airflow.models.dag.DAG`. You can read more about Dags in :doc:`Dags <core-concepts/dags>`.
``{{ task }}`` BaseOperator | The currently running :class:`~airflow.models.baseoperator.BaseOperator`. You can read more about Tasks in :doc:`core-concepts/operators`
``{{ task_reschedule_count }}`` int How many times current task has been rescheduled. Relevant to ``mode="reschedule"`` sensors.
``{{ macros }}`` | A reference to the macros package. See Macros_ below.
@@ -79,7 +79,7 @@ Variable Type Description
``{{ expanded_ti_count }}`` int | ``None`` | Number of task instances that a mapped task was expanded into. If
| the current task is not mapped, this should be ``None``.
| Added in version 2.5.
-``{{ triggering_asset_events }}`` dict[str, | If in an Asset Scheduled DAG, a map of Asset URI to a list of triggering :class:`~airflow.models.asset.AssetEvent`
+``{{ triggering_asset_events }}`` dict[str, | If in an Asset Scheduled Dag, a map of Asset URI to a list of triggering :class:`~airflow.models.asset.AssetEvent`
list[AssetEvent]] | (there may be more than one, if there are multiple Assets with different frequencies).
| Read more here :doc:`Assets <authoring-and-scheduling/asset-scheduling>`.
| Added in version 2.4.
@@ -90,7 +90,7 @@ The following are only available when the DagRun has a ``logical_date``
=========================================== ===================== ===================================================================
Variable Type Description
=========================================== ===================== ===================================================================
-``{{ ds }}`` str | The DAG run's logical date as ``YYYY-MM-DD``.
+``{{ ds }}`` str | The Dag run's logical date as ``YYYY-MM-DD``.
| Same as ``{{ logical_date | ds }}``.
``{{ ds_nodash }}`` str Same as ``{{ logical_date | ds_nodash }}``.
``{{ ts }}`` str | Same as ``{{ logical_date | ts }}``.
@@ -103,8 +103,8 @@ Variable Type Description
.. note::
- The DAG run's logical date, and values derived from it, such as ``ds`` and
- ``ts``, **should not** be considered unique in a DAG. Use ``run_id`` instead.
+ The Dag run's logical date, and values derived from it, such as ``ds`` and
+ ``ts``, **should not** be considered unique in a Dag. Use ``run_id`` instead.
Accessing Airflow context variables from TaskFlow tasks
-------------------------------------------------------
diff --git a/airflow-core/docs/troubleshooting.rst b/airflow-core/docs/troubleshooting.rst
index f354ea1a2ff..6756ab87e10 100644
--- a/airflow-core/docs/troubleshooting.rst
+++ b/airflow-core/docs/troubleshooting.rst
@@ -30,7 +30,7 @@ There are many potential causes for a task's state to be changed by a component
Below are some example scenarios that could cause a task's state to change by a component other than the executor:
-- If a task's DAG failed to parse on the worker, the scheduler may mark the task as failed. If confirmed, consider increasing :ref:`core.dagbag_import_timeout <config:core__dagbag_import_timeout>` and :ref:`dag_processor.dag_file_processor_timeout <config:dag_processor__dag_file_processor_timeout>`.
+- If a task's Dag failed to parse on the worker, the scheduler may mark the task as failed. If confirmed, consider increasing :ref:`core.dagbag_import_timeout <config:core__dagbag_import_timeout>` and :ref:`dag_processor.dag_file_processor_timeout <config:dag_processor__dag_file_processor_timeout>`.
- The scheduler will mark a task as failed if the task has been queued for longer than :ref:`scheduler.task_queued_timeout <config:scheduler__task_queued_timeout>`.
- If a :ref:`task instance's heartbeat times out <concepts:task-instance-heartbeat-timeout>`, it will be marked failed by the scheduler.
- A user marked the task as successful or failed in the Airflow UI.
@@ -43,7 +43,7 @@ Sometimes, Airflow or some adjacent system will kill a task instance's ``TaskRun
Here are some examples that could cause such an event:
-- A DAG run timeout, specified by ``dagrun_timeout`` in the DAG's definition.
+- A Dag run timeout, specified by ``dagrun_timeout`` in the Dag's definition.
- An Airflow worker running out of memory
- Usually, Airflow workers that run out of memory receive a SIGKILL, and the scheduler will fail the corresponding task instance for not having a heartbeat. However, in some scenarios, Airflow kills the task before that happens.
diff --git a/airflow-core/docs/tutorial/fundamentals.rst b/airflow-core/docs/tutorial/fundamentals.rst
index 2f4cee031dc..0017860e343 100644
--- a/airflow-core/docs/tutorial/fundamentals.rst
+++ b/airflow-core/docs/tutorial/fundamentals.rst
@@ -21,12 +21,12 @@
Airflow 101: Building Your First Workflow
=========================================
Welcome to world of Apache Airflow! In this tutorial, we'll guide you through the essential concepts of Airflow, helping
-you understand how to write your first DAG. Whether you're familiar with Python or just starting out, we'll make the
+you understand how to write your first Dag. Whether you're familiar with Python or just starting out, we'll make the
journey enjoyable and straightforward.
-What is a DAG?
+What is a Dag?
--------------
-At its core, a DAG is a collection of tasks organized in a way that reflects their relationships and dependencies. It's
+At its core, a Dag is a collection of tasks organized in a way that reflects their relationships and dependencies. It's
like a roadmap for your workflow, showing how each task connects to the others. Don't worry if this sounds a bit complex;
we'll break it down step by step.
@@ -40,11 +40,11 @@ each line in detail.
:start-after: [START tutorial]
:end-before: [END tutorial]
-Understanding the DAG Definition File
+Understanding the Dag Definition File
-------------------------------------
-Think of the Airflow Python script as a configuration file that lays out the structure of your DAG in code. The actual
+Think of the Airflow Python script as a configuration file that lays out the structure of your Dag in code. The actual
tasks you define here run in a different environment, which means this script isn't meant for data processing. Its main
-job is to define the DAG object, and it needs to evaluate quickly since the DAG File Processor checks it regularly for
+job is to define the Dag object, and it needs to evaluate quickly since the Dag File Processor checks it regularly for
any changes.
Importing Modules
@@ -63,7 +63,7 @@ For more details on how Python and Airflow handle modules, check out
Setting Default Arguments
-------------------------
-When creating a DAG and its tasks, you can either pass arguments directly to each task or define a set of default
+When creating a Dag and its tasks, you can either pass arguments directly to each task or define a set of default
parameters in a dictionary. The latter approach is usually more efficient and cleaner.
.. exampleinclude:: /../src/airflow/example_dags/tutorial.py
@@ -77,10 +77,10 @@ parameters in a dictionary. The latter approach is usually more efficient and cl
If you want to dive deeper into the parameters of the BaseOperator, take a look at the documentation for
:py:class:`airflow.sdk.BaseOperator` documentation.
-Creating a DAG
+Creating a Dag
--------------
-Next, we'll need to create a DAG object to house our tasks. We'll provide a unique identifier for the DAG, known as the
-``dag_id``, and specify the default arguments we just defined. We'll also set a schedule for our DAG to run every day.
+Next, we'll need to create a Dag object to house our tasks. We'll provide a unique identifier for the Dag, known as the
+``dag_id``, and specify the default arguments we just defined. We'll also set a schedule for our Dag to run every day.
.. exampleinclude:: /../src/airflow/example_dags/tutorial.py
:language: python
@@ -100,7 +100,7 @@ focus on the ``BashOperator`` to execute some simple bash commands.
Defining Tasks
--------------
To use an operator, you must instantiate it as a task. Tasks dictate how the operator will perform its work within the
-DAG's context. In the example below, we instantiate the BashOperator twice to run two different bash scripts. The
+Dag's context. In the example below, we instantiate the BashOperator twice to run two different bash scripts. The
``task_id`` serves as a unique identifier for each task.
.. exampleinclude:: /../src/airflow/example_dags/tutorial.py
@@ -151,11 +151,11 @@ your own variables and filters for use in templates. For more on custom filters,
For more information on the variables and macros that can be referenced in templates, please read through the
:ref:`templates-ref`.
-Adding DAG and Tasks documentation
+Adding Dag and Tasks documentation
----------------------------------
-You can add documentation to your DAG or individual tasks. While DAG documentation currently supports markdown, task
+You can add documentation to your Dag or individual tasks. While Dag documentation currently supports markdown, task
documentation can be in plain text, markdown reStructuredText, JSON, or YAML. It's a good practice to include
-documentation at the start of your DAG file.
+documentation at the start of your Dag file.
.. exampleinclude:: /../src/airflow/example_dags/tutorial.py
:language: python
@@ -204,19 +204,19 @@ their dependencies in several ways:
t1 >> [t2, t3]
[t2, t3] << t1
-Be mindful that Airflow will raise errors if it detects cycles in your DAG or if a dependency is referenced multiple
+Be mindful that Airflow will raise errors if it detects cycles in your Dag or if a dependency is referenced multiple
times.
Working with Time Zones
-----------------------
-Creating a time zone aware DAG is straightforward. Just ensure you use time zone aware dates
+Creating a time zone aware Dag is straightforward. Just ensure you use time zone aware dates
with `pendulum <https://github.com/python-pendulum/pendulum>`_. Avoid using the standard library
`timezone <https://docs.python.org/3/library/datetime.html#timezone-objects>`_ as they have known limitations.
Recap
-----
-Congratulations! By now, you should have a basic understanding of how to create a DAG, define tasks and their
+Congratulations! By now, you should have a basic understanding of how to create a Dag, define tasks and their
dependencies, and use templating in Airflow. Your code should resemble the following:
.. exampleinclude:: /../src/airflow/example_dags/tutorial.py
@@ -229,13 +229,13 @@ dependencies, and use templating in Airflow. Your code should resemble the follo
Testing Your Pipeline
---------------------
Now it's time to test your pipeline! First, ensure that your script parses successfully. If you saved your code in
-``tutorial.py`` within the dags folder specified in your ``airflow.cfg``, you can run:
+``tutorial.py`` within the Dags folder specified in your ``airflow.cfg``, you can run:
.. code-block:: bash
python ~/airflow/dags/tutorial.py
-If the script runs without errors, congratulations! Your DAG is set up correctly.
+If the script runs without errors, congratulations! Your Dag is set up correctly.
Command Line Metadata Validation
'''''''''''''''''''''''''''''''''
@@ -246,28 +246,28 @@ Let's validate your script further by running a few commands:
# initialize the database tables
airflow db migrate
- # print the list of active dags
+ # print the list of active Dags
airflow dags list
- # prints the list of tasks in the "tutorial" DAG
+ # prints the list of tasks in the "tutorial" Dag
airflow tasks list tutorial
- # prints the graphviz representation of "tutorial" DAG
+ # prints the graphviz representation of "tutorial" Dag
airflow dags show tutorial
-Testing Task Instances and DAG Runs
+Testing Task Instances and Dag Runs
'''''''''''''''''''''''''''''''''''
You can test specific task instances for a designated *logical date*. This simulates the scheduler running your task for
a particular date and time.
.. note::
Notice that the scheduler runs your task *for* a specific date and time, not necessarily *at* that date or time.
- The *logical date* is the timestamp that a DAG run is **named after**, and it typically corresponds to the **end**
- of the time period your workflow is operating on — or the time at which the DAG run was manually triggered.
+ The *logical date* is the timestamp that a Dag run is **named after**, and it typically corresponds to the **end**
+ of the time period your workflow is operating on — or the time at which the Dag run was manually triggered.
Airflow uses this logical date to organize and track each run; it's how you refer to a specific execution in the UI,
- logs, and code. When triggering a DAG via the UI or API, you can supply your own logical date to run the workflow
+ logs, and code. When triggering a Dag via the UI or API, you can supply your own logical date to run the workflow
*as of* a specific point in time.
.. code-block:: bash
@@ -292,17 +292,17 @@ This command will provide detailed logs and execute your bash command.
Keep in mind that the ``airflow tasks test`` command runs task instances locally, outputs their logs to stdout, and
doesn't track state in the database. This is a handy way to test individual task instances.
-Similarly, ``airflow dags test`` runs a single DAG run without registering any state in the database, which is useful
-for testing your entire DAG locally.
+Similarly, ``airflow dags test`` runs a single Dag run without registering any state in the database, which is useful
+for testing your entire Dag locally.
What's Next?
-------------
That's a wrap! You've successfully written and tested your first Airflow pipeline. As you continue your journey,
-consider merging your code into a repository with a Scheduler running against it, which will allow your DAG to be
+consider merging your code into a repository with a Scheduler running against it, which will allow your Dag to be
triggered and executed daily.
Here are a few suggestions for your next steps:
.. seealso::
- Continue to the next step of the tutorial: :doc:`/tutorial/taskflow`
- - Explore the :doc:`/core-concepts/index` section for detailed explanation of Airflow concepts such as DAGs, Tasks, Operators, and more.
+ - Explore the :doc:`/core-concepts/index` section for detailed explanation of Airflow concepts such as Dags, Tasks, Operators, and more.
diff --git a/airflow-core/docs/tutorial/objectstorage.rst b/airflow-core/docs/tutorial/objectstorage.rst
index de80e8c973c..0b8a890138f 100644
--- a/airflow-core/docs/tutorial/objectstorage.rst
+++ b/airflow-core/docs/tutorial/objectstorage.rst
@@ -23,7 +23,7 @@ Cloud-Native Workflows with Object Storage
.. versionadded:: 2.8
-Welcome to the final tutorial in our Airflow series! By now, you've built DAGs with Python and the TaskFlow API, passed
+Welcome to the final tutorial in our Airflow series! By now, you've built Dags with Python and the TaskFlow API, passed
data with XComs, and chained tasks together into clear, reusable workflows.
In this tutorial we'll take it a step further by introducing the **Object Storage API**. This API makes it easier to
@@ -86,7 +86,7 @@ You can also provide the ``conn_id`` as keyword argument for clarity:
This is especially handy when reusing a path defined elsewhere (like in an Asset), or when the connection isn't baked
into the URL. The keyword argument always takes precedence.
-.. tip:: You can safely create an ``ObjectStoragePath`` in your global DAG scope. Connections are resolved only when the
+.. tip:: You can safely create an ``ObjectStoragePath`` in your global Dag scope. Connections are resolved only when the
path is used, not when it's created.
Saving Data to Object Storage
@@ -139,7 +139,7 @@ This makes the task portable and decoupled from earlier logic.
Bringing It All Together
------------------------
-Here's the full DAG that ties everything together:
+Here's the full Dag that ties everything together:
.. exampleinclude:: /../src/airflow/example_dags/tutorial_objectstorage.py
:language: python
@@ -148,7 +148,7 @@ Here's the full DAG that ties everything together:
|
-You can trigger this DAG and view it in the Graph View in the Airflow UI. Each task logs its inputs and outputs clearly,
+You can trigger this Dag and view it in the Graph View in the Airflow UI. Each task logs its inputs and outputs clearly,
and you can inspect returned paths in the Xcom tab.
What to Explore Next
diff --git a/airflow-core/docs/tutorial/pipeline.rst b/airflow-core/docs/tutorial/pipeline.rst
index 4928064401f..c1b956ae299 100644
--- a/airflow-core/docs/tutorial/pipeline.rst
+++ b/airflow-core/docs/tutorial/pipeline.rst
@@ -21,7 +21,7 @@
Building a Simple Data Pipeline
===============================
-Welcome to the third tutorial in our series! At this point, you've already written your first DAG and used some basic
+Welcome to the third tutorial in our series! At this point, you've already written your first Dag and used some basic
operators. Now it's time to build a small but meaningful data pipeline -- one that retrieves data from an external
source, loads it into a database, and cleans it up along the way.
@@ -76,7 +76,7 @@ Log in with:
- **Username:** ``airflow``
- **Password:** ``airflow``
-You'll land in the Airflow dashboard, where you can trigger DAGs, explore logs, and manage your environment.
+You'll land in the Airflow dashboard, where you can trigger Dags, explore logs, and manage your environment.
Create a Postgres Connection
----------------------------
@@ -146,7 +146,7 @@ We'll use the ``SQLExecuteQueryOperator`` to run the SQL statements needed to cr
)
You can optionally place these SQL statements in ``.sql`` files inside your ``dags/`` folder and pass the file path to
-the ``sql=`` argument. This can be a great way to keep your DAG code clean.
+the ``sql=`` argument. This can be a great way to keep your Dag code clean.
Load data into the staging table
--------------------------------
@@ -226,10 +226,10 @@ CONFLICT DO UPDATE`.
-Defining the DAG
+Defining the Dag
----------------
-Now that we've defined all our tasks, it's time to put them together into a DAG.
+Now that we've defined all our tasks, it's time to put them together into a Dag.
.. code-block:: python
@@ -331,23 +331,23 @@ Now that we've defined all our tasks, it's time to put them together into a DAG.
dag = ProcessEmployees()
-Save this DAG as ``dags/process_employees.py``. After a short delay, it will show up in the UI.
+Save this Dag as ``dags/process_employees.py``. After a short delay, it will show up in the UI.
-Trigger and explore your DAG
+Trigger and explore your Dag
----------------------------
-Open the Airflow UI and find the ``process_employees`` DAG in the list. Toggle it "on" using the slider, then trigger a
+Open the Airflow UI and find the ``process_employees`` Dag in the list. Toggle it "on" using the slider, then trigger a
run using the play button.
You can watch each task as it runs in the **Grid** view, and explore logs for each step.
.. image:: ../img/ui-dark/tutorial_pipeline_dag_list.png
- :alt: DAG List view showing the ``process_employees`` DAG
+ :alt: Dag List view showing the ``process_employees`` Dag
|
.. image:: ../img/ui-dark/tutorial_pipeline_dag_overview_processed.png
- :alt: DAG Overview page for ``process_employees`` DAG showing the DAG run
+ :alt: Dag Overview page for ``process_employees`` Dag showing the Dag run
|
@@ -361,7 +361,7 @@ Nice work! You've now built a real pipeline using Airflow's core patterns and to
go next:
- Try swapping in a different SQL provider, like MySQL or SQLite.
-- Split your DAG into TaskGroups or refactor into a more usable pattern.
+- Split your Dag into TaskGroups or refactor into a more usable pattern.
- Add an alerting step or send a notification when data is processed.
.. seealso::
diff --git a/airflow-core/docs/tutorial/taskflow.rst b/airflow-core/docs/tutorial/taskflow.rst
index 8cef080ab6e..44187d7f4b8 100644
--- a/airflow-core/docs/tutorial/taskflow.rst
+++ b/airflow-core/docs/tutorial/taskflow.rst
@@ -16,10 +16,10 @@
under the License.
-Pythonic DAGs with the TaskFlow API
+Pythonic Dags with the TaskFlow API
===================================
-In the first tutorial, you built your first Airflow DAG using traditional Operators like ``BashOperator``.
+In the first tutorial, you built your first Airflow Dag using traditional Operators like ``BashOperator``.
Now let's look at a more modern and Pythonic way to write workflows using the **TaskFlow API** — introduced in Airflow
2.0.
@@ -41,10 +41,10 @@ down step-by-step.
:start-after: [START tutorial]
:end-before: [END tutorial]
-Step 1: Define the DAG
+Step 1: Define the Dag
----------------------
-Just like before, your DAG is a Python script that Airflow loads and parses. But this time, we're using the ``@dag``
+Just like before, your Dag is a Python script that Airflow loads and parses. But this time, we're using the ``@dag``
decorator to define it.
.. exampleinclude:: /../src/airflow/example_dags/tutorial_taskflow_api.py
@@ -54,7 +54,7 @@ decorator to define it.
|
-To make this DAG discoverable by Airflow, we can call the Python function that was decorated with ``@dag``:
+To make this Dag discoverable by Airflow, we can call the Python function that was decorated with ``@dag``:
.. exampleinclude:: /../src/airflow/example_dags/tutorial_taskflow_api.py
:language: python
@@ -64,10 +64,10 @@ To make this DAG discoverable by Airflow, we can call the Python function that w
|
.. versionchanged:: 2.4
- If you're using the ``@dag`` decorator or defining your DAG in a ``with`` block, you no longer need to assign it to a
+ If you're using the ``@dag`` decorator or defining your Dag in a ``with`` block, you no longer need to assign it to a
global variable. Airflow will find it automatically.
-You can visualize your DAG in the Airflow UI! Once your DAG is loaded, navigate to the Graph View to see how tasks are
+You can visualize your Dag in the Airflow UI! Once your Dag is loaded, navigate to the Graph View to see how tasks are
connected.
Step 2: Write Your Tasks with ``@task``
@@ -109,14 +109,14 @@ functional invocation to set task dependencies and manage data passing.
That's it! Airflow knows how to schedule and orchestrate your pipeline from this code alone.
-Running Your DAG
+Running Your Dag
----------------
-To enable and trigger your DAG:
+To enable and trigger your Dag:
1. Navigate to the Airflow UI.
-2. Find your DAG in the list and click the toggle to enable it.
-3. You can trigger it manually by clicking the "Trigger DAG" button, or wait for it to run on its schedule.
+2. Find your Dag in the list and click the toggle to enable it.
+3. You can trigger it manually by clicking the "Trigger Dag" button, or wait for it to run on its schedule.
What's Happening Behind the Scenes?
-----------------------------------
@@ -129,7 +129,7 @@ The "Old Way": Manual Wiring and XComs
Before the TaskFlow API, you had to use Operators like ``PythonOperator`` and pass data manually between tasks using
``XComs``.
-Here's what the same DAG might have looked like using the traditional approach:
+Here's what the same Dag might have looked like using the traditional approach:
.. code-block:: python
@@ -212,7 +212,7 @@ This helps ensure that transient failures do not lead to task failure.
Task Parameterization
---------------------
-You can reuse decorated tasks in multiple DAGs and override parameters like ``task_id`` or ``retries``.
+You can reuse decorated tasks in multiple Dags and override parameters like ``task_id`` or ``retries``.
.. code-block:: python
@@ -227,7 +227,7 @@ What to Explore Next
Nice work! You've now written your first pipeline using the TaskFlow API. Curious where to go from here?
-- Add a new task to the DAG -- maybe a filter or validation step
+- Add a new task to the Dag -- maybe a filter or validation step
- Modify return values and pass multiple outputs
- Explore retries and overrides with ``.override(task_id="...")``
- Open the Airflow UI and inspect how the data flows between tasks, including task logs and dependencies
@@ -248,7 +248,7 @@ Once you're comfortable with the basics, here are a few powerful techniques you
Reusing Decorated Tasks
'''''''''''''''''''''''
-You can reuse decorated tasks across multiple DAGs or DAG runs. This is especially useful for common logic like reusable
+You can reuse decorated tasks across multiple Dags or Dag runs. This is especially useful for common logic like reusable
utilities or shared business rules. Use ``.override()`` to customize task metadata like ``task_id`` or ``retries``.
.. code-block:: python
@@ -260,7 +260,7 @@ You can even import decorated tasks from a shared module.
Handling Conflicting Dependencies
'''''''''''''''''''''''''''''''''
-Sometimes tasks require different Python dependencies than the rest of your DAG — for example, specialized libraries or
+Sometimes tasks require different Python dependencies than the rest of your Dag — for example, specialized libraries or
system-level packages. TaskFlow supports multiple execution environments to isolate those dependencies.
.. _taskflow-dynamically-created-virtualenv:
@@ -410,7 +410,7 @@ Conditional Execution
'''''''''''''''''''''
Use ``@task.run_if()`` or ``@task.skip_if()`` to control whether a task runs based on dynamic conditions at runtime —
-without altering your DAG structure.
+without altering your Dag structure.
.. code-block:: python
@@ -422,7 +422,7 @@ without altering your DAG structure.
What's Next
-----------
-Now that you've seen how to build clean, maintainable DAGs using the TaskFlow API, here are some good next steps:
+Now that you've seen how to build clean, maintainable Dags using the TaskFlow API, here are some good next steps:
- Explore asset-aware workflows in :doc:`/authoring-and-scheduling/asset-scheduling`
- Dive into scheduling patterns in :ref:`Scheduling Options <scheduling-section>`
diff --git a/airflow-core/docs/ui.rst b/airflow-core/docs/ui.rst
index 119f5182961..602880500b3 100644
--- a/airflow-core/docs/ui.rst
+++ b/airflow-core/docs/ui.rst
@@ -37,123 +37,123 @@ The Home Page provides a high-level overview of the system state and recent acti
Airflow 3 and includes:
- **Health indicators** for system components such as the MetaDatabase, Scheduler, Triggerer, and Dag Processor
-- **Quick links** to DAGs filtered by status (e.g., Failed DAGs, Running DAGs, Active DAGs)
-- **DAG and Task Instance history**, showing counts and success/failure rates over a selectable time range
-- **Recent asset events**, including materializations and triggered DAGs
+- **Quick links** to Dags filtered by status (e.g., Failed Dags, Running Dags, Active Dags)
+- **Dag and Task Instance history**, showing counts and success/failure rates over a selectable time range
+- **Recent asset events**, including materializations and triggered Dags
This page is useful for quickly assessing the health of your environment and identifying recent issues or
asset-triggered events.
.. image:: img/ui-dark/home_dark.png
- :alt: Airflow Home Page showing system health, DAG/task stats, and asset events (Dark Mode)
+ :alt: Airflow Home Page showing system health, Dag/task stats, and asset events (Dark Mode)
|
.. image:: img/ui-light/home_light.png
- :alt: Airflow Home Page showing system health, DAG/task stats, and asset events (Dark Mode)
+ :alt: Airflow Home Page showing system health, Dag/task stats, and asset events (Light Mode)
.. _ui-dag-list:
-DAG List View
+Dag List View
-------------
-The DAG List View appears when you click the **DAGs** tab in the main navigation bar. It displays all DAGs available in
+The Dag List View appears when you click the **Dags** tab in the main navigation bar. It displays all Dags available in
your environment, with a clear summary of their status, recent runs, and configuration.
Each row includes:
-- **DAG ID**
+- **Dag ID**
- **Schedule** and next run time
-- **Status of the latest DAG run**
+- **Status of the latest Dag run**
- **Bar chart of recent runs**
-- **Tags**, which can be used for grouping or filtering DAGs (e.g., ``example``, ``produces``)
+- **Tags**, which can be used for grouping or filtering Dags (e.g., ``example``, ``produces``)
- **Pause/resume toggle**
-- Links to access DAG-level views
+- Links to access Dag-level views
At the top of the view, you can:
-- Use **filters** for DAG status, schedule state, and tags
-- Use **search** or **advanced search (⌘+K)** to find specific DAGs
+- Use **filters** for Dag status, schedule state, and tags
+- Use **search** or **advanced search (⌘+K)** to find specific Dags
- Sort the list using the dropdown (e.g., Latest Run Start Date)
.. image:: img/ui-dark/dag_list.png
- :alt: DAG List View in dark mode showing search, filters, and DAG-level controls
+ :alt: Dag List View in dark mode showing search, filters, and Dag-level controls
|
.. image:: img/ui-light/dag_list.png
- :alt: DAG List View in light mode showing the same DAGs and actions for comparison
+ :alt: Dag List View in light mode showing the same Dags and actions for comparison
|
-Some DAGs in this list may interact with data assets. For example, DAGs that are triggered by asset conditions may
+Some Dags in this list may interact with data assets. For example, Dags that are triggered by asset conditions may
display popups showing upstream asset inputs.
.. image:: img/ui-dark/dag_list_asset_condition_popup.png
- :alt: DAG List View showing asset condition popup (Dark Mode)
+ :alt: Dag List View showing asset condition popup (Dark Mode)
|
.. image:: img/ui-light/dag_list_asset_condition_popup.png
- :alt: DAG List View showing asset condition popup (Light Mode)
+ :alt: Dag List View showing asset condition popup (Light Mode)
.. _ui-dag-details:
-DAG Details Page
+Dag Details Page
----------------
-Clicking a DAG from the list opens the DAG Details Page. This view offers centralized access to a DAG's metadata, recent
+Clicking a Dag from the list opens the Dag Details Page. This view offers centralized access to a Dag's metadata, recent
activity, and task-level diagnostics.
Key elements include:
-- **DAG metadata**, including ID, owner, tags, schedule, and latest DAG version
-- **Action buttons** to trigger the DAG, reparse it, or pause/resume
-- **Tabbed interface**: Overview (recent failures, run counts, task logs); Grid View (status heatmap); Graph View (task dependencies); Runs (full run history); Tasks (aggregated stats); Events (system- or asset-triggered); Code (DAG source); and Details (extended metadata)
+- **Dag metadata**, including ID, owner, tags, schedule, and latest Dag version
+- **Action buttons** to trigger the Dag, reparse it, or pause/resume
+- **Tabbed interface**: Overview (recent failures, run counts, task logs); Grid View (status heatmap); Graph View (task dependencies); Runs (full run history); Tasks (aggregated stats); Events (system- or asset-triggered); Code (Dag source); and Details (extended metadata)
-This page also includes a visual **timeline of recent DAG runs** and a **log preview for failures**, helping users quickly identify issues across runs.
+This page also includes a visual **timeline of recent Dag runs** and a **log preview for failures**, helping users quickly identify issues across runs.
.. image:: img/ui-dark/dag_overview_dashboard.png
- :alt: DAG Details Page in dark mode showing overview dashboard and failure diagnostics
+ :alt: Dag Details Page in dark mode showing overview dashboard and failure diagnostics
|
.. image:: img/ui-light/dag_overview_dashboard.png
- :alt: DAG Details Page in light mode showing overview dashboard and failure diagnostics
+ :alt: Dag Details Page in light mode showing overview dashboard and failure diagnostics
.. _ui-grid-view:
Grid View
'''''''''
-The Grid View is the primary interface for inspecting DAG runs and task states. It offers an interactive way to debug,
+The Grid View is the primary interface for inspecting Dag runs and task states. It offers an interactive way to debug,
retry, or monitor workflows over time.
Use Grid View to:
-- **Understand the status of recent DAG runs** at a glance
+- **Understand the status of recent Dag runs** at a glance
- **Identify failed or retried tasks** by color and tooltip
- **Take action** by clicking a task cell to view logs or mark task instances as successful, failed, or cleared
- **Filter tasks** by name or partial ID
- **Select a run range**, like "last 25 runs" using the dropdown above the grid
-Each row represents a task, and each column represents a DAG run. You can hover over any task instance for more detail,
+Each row represents a task, and each column represents a Dag run. You can hover over any task instance for more detail,
or click to drill down into logs and metadata.
.. image:: img/ui-dark/dag_overview_grid.png
- :alt: Grid View showing DAG run status matrix with varied task states (Dark Mode)
+ :alt: Grid View showing Dag run status matrix with varied task states (Dark Mode)
|
.. image:: img/ui-light/dag_overview_grid.png
- :alt: Grid View showing DAG run status matrix with varied task states (Light Mode)
+ :alt: Grid View showing Dag run status matrix with varied task states (Light Mode)
.. _ui-graph-view:
Graph View
''''''''''
-The Graph View shows the logical structure of your DAG—how tasks are connected, what order they run in, and how
+The Graph View shows the logical structure of your Dag - how tasks are connected, what order they run in, and how
branching or retries are configured.
This view is helpful when:
@@ -165,119 +165,119 @@ This view is helpful when:
Each node represents a task. Edges show the dependencies between them. You can click any task to view its metadata and
recent run history.
-Use the dropdown at the top to switch between DAG runs and see how task states changed across executions.
+Use the dropdown at the top to switch between Dag runs and see how task states changed across executions.
.. image:: img/ui-dark/dag_overview_graph.png
- :alt: Graph View showing DAG structure with no DAG run selected (Dark Mode)
+ :alt: Graph View showing Dag structure with no Dag run selected (Dark Mode)
|
.. image:: img/ui-light/dag_overview_graph.png
- :alt: Graph View showing DAG structure with no DAG run selected (Light Mode)
+ :alt: Graph View showing Dag structure with no Dag run selected (Light Mode)
.. _ui-dag-tabs:
-DAG Tabs
+Dag Tabs
--------
-In addition to the interactive views like Grid and Graph, the DAG Details page includes several other tabs that provide
+In addition to the interactive views like Grid and Graph, the Dag Details page includes several other tabs that provide
deeper insights and metadata:
Runs Tab
''''''''
-The **Runs** tab displays a sortable table of all DAG runs, along with their status, execution duration, run type, and DAG version.
+The **Runs** tab displays a sortable table of all Dag runs, along with their status, execution duration, run type, and Dag version.
.. image:: img/ui-dark/dag_overview_runs.png
- :alt: DAG Runs Tab (Dark Mode)
+ :alt: Dag Runs Tab (Dark Mode)
|
.. image:: img/ui-light/dag_overview_runs.png
- :alt: DAG Runs Tab (Light Mode)
+ :alt: Dag Runs Tab (Light Mode)
|
Tasks Tab
'''''''''
-The **Tasks** tab shows metadata for each task in the DAG, including operator type, trigger rule, most recent run status, and run history.
+The **Tasks** tab shows metadata for each task in the Dag, including operator type, trigger rule, most recent run status, and run history.
.. image:: img/ui-dark/dag_overview_tasks.png
- :alt: DAG Tasks Tab (Dark Mode)
+ :alt: Dag Tasks Tab (Dark Mode)
|
.. image:: img/ui-light/dag_overview_tasks.png
- :alt: DAG Tasks Tab (Light Mode)
+ :alt: Dag Tasks Tab (Light Mode)
|
Events Tab
''''''''''
-The **Events** tab surfaces structured events related to the DAG, such as DAG triggers and version patches. This tab is especially useful for DAG versioning and troubleshooting changes.
+The **Events** tab surfaces structured events related to the Dag, such as Dag triggers and version patches. This tab is especially useful for Dag versioning and troubleshooting changes.
.. image:: img/ui-dark/dag_overview_events.png
- :alt: DAG Events Tab (Dark Mode)
+ :alt: Dag Events Tab (Dark Mode)
|
.. image:: img/ui-light/dag_overview_events.png
- :alt: DAG Events Tab (Light Mode)
+ :alt: Dag Events Tab (Light Mode)
Code Tab
''''''''
-The **Code** tab displays the current version of the DAG definition, including the timestamp of the last parse. Users can view the code for any specific DAG version.
+The **Code** tab displays the current version of the Dag definition, including the timestamp of the last parse. Users can view the code for any specific Dag version.
.. image:: img/ui-dark/dag_overview_code.png
- :alt: DAG Code Tab (Dark Mode)
+ :alt: Dag Code Tab (Dark Mode)
|
.. image:: img/ui-light/dag_overview_code.png
- :alt: DAG Code Tab (Light Mode)
+ :alt: Dag Code Tab (Light Mode)
|
Details Tab
'''''''''''
-The **Details** tab provides configuration details and metadata for the DAG, including schedule, file location, concurrency limits, and version identifiers.
+The **Details** tab provides configuration details and metadata for the Dag, including schedule, file location, concurrency limits, and version identifiers.
.. image:: img/ui-dark/dag_overview_details.png
- :alt: DAG Details Tab (Dark Mode)
+ :alt: Dag Details Tab (Dark Mode)
|
.. image:: img/ui-light/dag_overview_details.png
- :alt: DAG Details Tab (Light Mode)
+ :alt: Dag Details Tab (Light Mode)
.. _ui-dag-runs:
-DAG Run View
+Dag Run View
------------
-Each DAG Run has its own view, accessible by selecting a specific row in the DAG's **Runs** tab. The DAG Run view
+Each Dag Run has its own view, accessible by selecting a specific row in the Dag's **Runs** tab. The Dag Run view
displays metadata about the selected run, as well as task-level details, rendered code, and more.
.. image:: img/ui-dark/dag_run_task_instances.png
- :alt: DAG Run - Task Instances tab (dark mode)
+ :alt: Dag Run - Task Instances tab (dark mode)
|
Key elements include:
-- **DAG Run metadata**, including logical date, run type, duration, DAG version, and parsed time
+- **Dag Run metadata**, including logical date, run type, duration, Dag version, and parsed time
- **Action buttons** to clear or mark the run, or add a note
-- A persistent **Grid View sidebar**, which shows task durations and states across recent DAG runs. This helps spot recurring issues or performance trends at a glance.
+- A persistent **Grid View sidebar**, which shows task durations and states across recent Dag runs. This helps spot recurring issues or performance trends at a glance.
-DAG Run Tabs
+Dag Run Tabs
------------
Task Instances
''''''''''''''
-Displays the status and metadata for each task instance within the DAG Run. Columns include:
+Displays the status and metadata for each task instance within the Dag Run. Columns include:
- Task ID
- State
@@ -285,115 +285,115 @@ Displays the status and metadata for each task instance within the DAG Run. Colu
- Try Number
- Operator Type
- Duration
-- DAG Version
+- Dag Version
Each row also includes a mini Gantt-style timeline that visually represents the task's duration.
.. image:: img/ui-light/dag_run_task_instances.png
- :alt: DAG Run - Task Instances (light mode)
+ :alt: Dag Run - Task Instances (light mode)
Events
''''''
-If available, this tab lists system-level or asset-triggered events that contributed to this DAG Run's execution.
+If available, this tab lists system-level or asset-triggered events that contributed to this Dag Run's execution.
Code
''''
-Displays the DAG source code as it was at the time this run was parsed. This view is helpful for debugging version drift
-or comparing behavior across DAG Runs that used different code.
+Displays the Dag source code as it was at the time this run was parsed. This view is helpful for debugging version drift
+or comparing behavior across Dag Runs that used different code.
-DAG Run code for ``hello >> airflow()``:
+Dag Run code for ``hello >> airflow()``:
.. image:: img/ui-dark/dag_run_code_hello_airflow.png
- :alt: DAG Run Code Snapshot - airflow() (dark mode)
+ :alt: Dag Run Code Snapshot - airflow() (dark mode)
|
.. image:: img/ui-light/dag_run_code_hello_airflow.png
- :alt: DAG Run Code Snapshot - airflow() (light mode)
+ :alt: Dag Run Code Snapshot - airflow() (light mode)
|
-DAG Run code for ``hello >> world()``:
+Dag Run code for ``hello >> world()``:
.. image:: img/ui-dark/dag_run_code_hello_world.png
- :alt: DAG Run Code Snapshot - world() (dark mode)
+ :alt: Dag Run Code Snapshot - world() (dark mode)
|
.. image:: img/ui-light/dag_run_code_hello_world.png
- :alt: DAG Run Code Snapshot - world() (light mode)
+ :alt: Dag Run Code Snapshot - world() (light mode)
Details
'''''''
-Provides extended metadata for the DAG Run, including:
+Provides extended metadata for the Dag Run, including:
- Run ID and Trigger Type
- Queued At, Start and End Time, and Duration
- Data Interval boundaries
- Trigger Source and Run Config
-- DAG Version ID and Bundle Name
+- Dag Version ID and Bundle Name
.. image:: img/ui-dark/dag_run_details.png
- :alt: DAG Run - Details tab (dark mode)
+ :alt: Dag Run - Details tab (dark mode)
|
.. image:: img/ui-light/dag_run_details.png
- :alt: DAG Run - Details tab (light mode)
+ :alt: Dag Run - Details tab (light mode)
Graph View
''''''''''
-Shows the DAG's task dependency structure overlaid with the status of each task in this specific run. This is useful for visual debugging of task failure paths or identifying downstream blockers.
+Shows the Dag's task dependency structure overlaid with the status of each task in this specific run. This is useful for visual debugging of task failure paths or identifying downstream blockers.
Each node includes a visual indicator of task duration.
.. image:: img/ui-dark/dag_run_graph.png
- :alt: DAG Run - Graph View (dark mode)
+ :alt: Dag Run - Graph View (dark mode)
|
.. image:: img/ui-light/dag_run_graph.png
- :alt: DAG Run - Graph View (light mode)
+ :alt: Dag Run - Graph View (light mode)
.. _ui-ti-view:
-DAG Trigger Window
+Dag Trigger Window
------------------
Single Run
''''''''''
-The Single Run window allows you to trigger a DAG run.
+The Single Run window allows you to trigger a Dag run.
.. image:: img/ui-light/dag_trigger_window_single_run.png
- :alt: DAG Trigger Window - Single Run (Light Mode)
+ :alt: Dag Trigger Window - Single Run (Light Mode)
|
.. image:: img/ui-dark/dag_trigger_window_single_run.png
- :alt: DAG Trigger Window - Single Run (Dark Mode)
+ :alt: Dag Trigger Window - Single Run (Dark Mode)
Backfill
''''''''
-The Backfill window allows you to trigger a DAG run for past dates.
+The Backfill window allows you to trigger a Dag run for past dates.
.. image:: img/ui-light/backfill.png
- :alt: DAG Trigger Window - Backfill (Light Mode)
+ :alt: Dag Trigger Window - Backfill (Light Mode)
|
.. image:: img/ui-dark/backfill.png
- :alt: DAG Trigger Window - Backfill (Dark Mode)
+ :alt: Dag Trigger Window - Backfill (Dark Mode)
Task Instance View
------------------
-When you click on a specific task from the DAG Run view, you're brought to the **Task Instance View**, which shows
+When you click on a specific task from the Dag Run view, you're brought to the **Task Instance View**, which shows
detailed logs and metadata for that individual task execution.
.. image:: img/ui-dark/dag_task_instance_logs.png
@@ -436,14 +436,14 @@ If present, displays relevant events related to this specific task instance exec
Code
''''
-Shows the DAG source code parsed at the time of execution. This helps verify what version of the DAG the task ran with.
+Shows the Dag source code parsed at the time of execution. This helps verify what version of the Dag the task ran with.
Details
'''''''
Displays runtime metadata about the task instance, including:
- Task ID and State
-- DAG Run ID, DAG Version, and Bundle Name
+- Dag Run ID, Dag Version, and Bundle Name
- Operator used and runtime duration
- Pool and slot usage
- Executor and configuration
@@ -471,10 +471,10 @@ Asset List
The Asset List shows all known assets, grouped by name. For each asset, you can see:
- The group the asset belongs to (if any)
-- The DAGs that consume the asset
+- The Dags that consume the asset
- The tasks that produce the asset
-Hovering over a count of DAGs or tasks shows a tooltip with the full list of producers or consumers.
+Hovering over a count of Dags or tasks shows a tooltip with the full list of producers or consumers.
.. image:: img/ui-dark/asset_list_consuming_dags.png
:alt: Asset Graph View (dark mode)
@@ -491,9 +491,9 @@ Asset Graph View
The Asset Graph View shows the asset in context, including upstream producers and downstream consumers. You can use this view to:
-- Understand asset lineage and the DAGs involved
+- Understand asset lineage and the Dags involved
- Trigger asset events manually
-- View recent asset events and the DAG runs they triggered
+- View recent asset events and the Dag runs they triggered
.. image:: img/ui-dark/asset_view.png
:alt: Asset Graph View (dark mode)
@@ -504,37 +504,37 @@ The Asset Graph View shows the asset in context, including upstream producers an
:alt: Asset Graph View (light mode)
-Graph Overlays in DAG View
+Graph Overlays in Dag View
''''''''''''''''''''''''''
-When a DAG contains asset-producing or asset-consuming tasks, you can enable asset overlays on the DAG Graph view. Toggle the switches next to each asset to:
+When a Dag contains asset-producing or asset-consuming tasks, you can enable asset overlays on the Dag Graph view. Toggle the switches next to each asset to:
-- See how assets flow between DAGs
+- See how assets flow between Dags
- Inspect asset-triggered dependencies
Two graph modes are available:
-- **All DAG Dependencies**: Shows all DAG-to-DAG and task-level connections
+- **All Dag Dependencies**: Shows all Dag-to-Dag and task-level connections
.. image:: img/ui-dark/dag_graph_all_dependencies.png
- :alt: DAG Graph View - All Dependencies (dark mode)
+ :alt: Dag Graph View - All Dependencies (dark mode)
|
.. image:: img/ui-light/dag_graph_all_dependencies.png
- :alt: DAG Graph View - All Dependencies (light mode)
+ :alt: Dag Graph View - All Dependencies (light mode)
|
-- **External Conditions**: Shows only DAGs triggered via asset events
+- **External Conditions**: Shows only Dags triggered via asset events
.. image:: img/ui-dark/dag_graph_external_conditions.png
- :alt: DAG Graph View - External Conditions Only (dark mode)
+ :alt: Dag Graph View - External Conditions Only (dark mode)
|
.. image:: img/ui-light/dag_graph_external_conditions.png
- :alt: DAG Graph View - External Conditions Only (light mode)
+ :alt: Dag Graph View - External Conditions Only (light mode)
.. _ui-admin-views:
@@ -545,7 +545,7 @@ The **Admin** tab provides system-level tools for configuring and extending Airf
Key pages include:
-- **Variables** – Store key-value pairs accessible from DAGs. Variables can be used to manage environment-specific parameters or secrets.
+- **Variables** – Store key-value pairs accessible from Dags. Variables can be used to manage environment-specific parameters or secrets.
- **Connections** – Define connection URIs to external systems such as databases, cloud services, or APIs. These are consumed by Airflow operators and hooks.
- **Pools** – Control resource allocation by limiting the number of concurrently running tasks assigned to a named pool. Useful for managing contention or quota-constrained systems.
- **Providers** – View installed provider packages (e.g., ``apache-airflow-providers-google``), including available hooks, sensors, and operators. This is helpful for verifying provider versions or troubleshooting import errors.
diff --git a/airflow-core/newsfragments/50374.feature.rst b/airflow-core/newsfragments/50374.feature.rst
index f6a241d3819..85e4dc165e9 100644
--- a/airflow-core/newsfragments/50374.feature.rst
+++ b/airflow-core/newsfragments/50374.feature.rst
@@ -1 +1 @@
-When a dag specifies ``schedule="@once"`` without an explicit ``start_date``, run it as soon as convenient. (Previously, the dag would never run.)
+When a Dag specifies ``schedule="@once"`` without an explicit ``start_date``, run it as soon as convenient. (Previously, the Dag would never run.)
diff --git a/airflow-core/newsfragments/51424.significant.rst b/airflow-core/newsfragments/51424.significant.rst
index a04140ba74b..a455a28eb1c 100644
--- a/airflow-core/newsfragments/51424.significant.rst
+++ b/airflow-core/newsfragments/51424.significant.rst
@@ -1,7 +1,7 @@
The ``consuming_dags`` key in asset API has been renamed to ``scheduled_dags``.
The previous name caused confusion to users since the list does not contain all
-dags that technically *use* the asset, but only those that use it in their
+Dags that technically *use* the asset, but only those that use it in their
``schedule`` argument. As a bug fix, the key has been renamed to clarify its
intention.
diff --git a/airflow-core/newsfragments/53796.misc.rst b/airflow-core/newsfragments/53796.misc.rst
index 07c127679c5..a582154dc3a 100644
--- a/airflow-core/newsfragments/53796.misc.rst
+++ b/airflow-core/newsfragments/53796.misc.rst
@@ -1 +1 @@
-Add ``RemovedInAirflow4Warning`` warnings for ``airflow.security.permissions`` imports and ``access_control`` DAG attribute usage.
+Add ``RemovedInAirflow4Warning`` warnings for ``airflow.security.permissions`` imports and ``access_control`` Dag attribute usage.
diff --git a/airflow-core/newsfragments/template.significant.rst b/airflow-core/newsfragments/template.significant.rst
index 702670ebec0..464486489c6 100644
--- a/airflow-core/newsfragments/template.significant.rst
+++ b/airflow-core/newsfragments/template.significant.rst
@@ -3,7 +3,7 @@
.. Provide additional contextual information
.. Check the type of change that applies to this change
-.. Dag changes: requires users to change their dag code
+.. Dag changes: requires users to change their Dag code
.. Config changes: requires users to change their Airflow config
.. API changes: requires users to change their Airflow REST API calls
.. CLI changes: requires users to change their Airflow CLI usage