You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by ka...@apache.org on 2025/04/19 10:20:21 UTC
(airflow) branch v3-0-test updated (15ae741f235 -> 40cb51d1350)
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a change to branch v3-0-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
from 15ae741f235 Fix issue with setproctitle on macos (#49451)
new 326c65cd82d Reset error state after successfully adding a connection (#49364)
new 23505f24614 Remove common.messaging from chicken-egg-providers (#49369)
new abb46b347f8 Update warning message to note in simple auth manager doc (#49367)
new 6b335f386bb Floor version suffix when referring to other airflow pre-releases (#49373)
new 25c5d62baa7 Set rc0 as suffix used for chicken-egg-providers (#49387)
new dad2192f9dc Fix dockerfile generation for PMC installation check (#49435)
new db565d28367 Use EmailOperator from smtp provider (#49386)
new ead8990c2ef Fix broken link for CeleryKubernetesExecutor upgrade (#49391)
new 6bf14444bf2 Use Label class from task sdk (#49385)
new 16cc912a229 Fix the taskflow context example in docs (#49399)
new d8219acfb06 Rewrite custom view plugin docs (#49408)
new fad509d822a Fix doc alignment in group-extras section (#49425)
new 10113f00c13 docs: remove paragraph related to Bazel in quickstart (#49430)
new fa72d055b80 Add note in FAB migration doc (#49423)
new 35e46224b23 docs: grammar fixes (#49432)
new b71c0f57930 Edit airflow 3 migration guide (#49414)
new 40cb51d1350 Use sync calls for list backfills (#49454)
The 17 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/release_dockerhub_image.yml | 12 +---
PROVIDERS.rst | 4 +-
.../cluster-policies.rst | 2 +-
.../docs/administration-and-deployment/index.rst | 2 +-
.../advanced-logging-configuration.rst | 2 +-
.../modules_management.rst | 2 +-
.../docs/administration-and-deployment/plugins.rst | 4 +-
.../production-deployment.rst | 2 +-
.../docs/authoring-and-scheduling/connections.rst | 2 +-
.../docs/authoring-and-scheduling/index.rst | 2 +-
airflow-core/docs/best-practices.rst | 4 +-
airflow-core/docs/configurations-ref.rst | 2 +-
.../core-concepts/auth-manager/simple/index.rst | 2 +-
airflow-core/docs/core-concepts/backfill.rst | 2 +-
airflow-core/docs/core-concepts/dags.rst | 6 +-
airflow-core/docs/core-concepts/operators.rst | 2 +-
airflow-core/docs/core-concepts/overview.rst | 2 +-
airflow-core/docs/core-concepts/taskflow.rst | 2 +-
airflow-core/docs/extra-packages-ref.rst | 20 +++---
airflow-core/docs/howto/custom-view-plugin.rst | 72 +++++++++-------------
airflow-core/docs/howto/email-config.rst | 2 +-
airflow-core/docs/howto/export-more-env-vars.rst | 2 +-
airflow-core/docs/howto/listener-plugin.rst | 6 +-
airflow-core/docs/howto/set-config.rst | 2 +-
airflow-core/docs/howto/usage-cli.rst | 2 +-
airflow-core/docs/installation/dependencies.rst | 4 +-
airflow-core/docs/installation/index.rst | 50 +++++++--------
.../docs/installation/installing-from-pypi.rst | 40 ++++++------
.../docs/installation/installing-from-sources.rst | 12 ++--
airflow-core/docs/installation/prerequisites.rst | 11 ++--
.../docs/installation/supported-versions.rst | 10 +--
.../docs/installation/upgrading_to_airflow3.rst | 69 ++++++++++-----------
airflow-core/docs/start.rst | 13 ++--
airflow-core/docs/templates-ref.rst | 2 +-
.../core_api/routes/public/backfills.py | 11 ++--
.../src/airflow/config_templates/config.yml | 3 +-
.../airflow/example_dags/example_branch_labels.py | 3 +-
.../example_dags/example_branch_operator.py | 3 +-
.../example_branch_operator_decorator.py | 3 +-
.../src/airflow/ui/src/queries/useAddConnection.ts | 2 +
.../unit/serialization/test_dag_serialization.py | 2 +-
airflow-core/tests/unit/utils/test_dag_cycle.py | 2 +-
airflow-core/tests/unit/utils/test_edgemodifier.py | 2 +-
airflow-core/tests/unit/utils/test_task_group.py | 3 +-
.../commands/release_management_commands.py | 6 +-
dev/breeze/src/airflow_breeze/utils/packages.py | 10 +--
dev/check_files.py | 9 +--
.../src/docs/shared/template-examples/taskflow.rst | 2 +-
docs/spelling_wordlist.txt | 2 +
providers/fab/docs/upgrading.rst | 4 ++
50 files changed, 207 insertions(+), 233 deletions(-)
(airflow) 10/17: Fix the taskflow context example in docs (#49399)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v3-0-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 16cc912a2298641e497ed4767be1a0f2321499a0
Author: Amogh Desai <am...@gmail.com>
AuthorDate: Thu Apr 17 19:14:00 2025 +0530
Fix the taskflow context example in docs (#49399)
(cherry picked from commit 1c59cedd8aecd443ee9a4bd10e219d96f6679623)
---
devel-common/src/docs/shared/template-examples/taskflow.rst | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/devel-common/src/docs/shared/template-examples/taskflow.rst b/devel-common/src/docs/shared/template-examples/taskflow.rst
index 5ea69554518..ee5251ba256 100644
--- a/devel-common/src/docs/shared/template-examples/taskflow.rst
+++ b/devel-common/src/docs/shared/template-examples/taskflow.rst
@@ -22,7 +22,7 @@
@task
- def print_ti_info(task_instance: TaskInstance | None = None, dag_run: DagRun | None = None):
+ def print_ti_info(task_instance: TaskInstance, dag_run: DagRun):
print(f"Run ID: {task_instance.run_id}") # Run ID: scheduled__2023-08-09T00:00:00+00:00
print(f"Duration: {task_instance.duration}") # Duration: 0.972019
print(f"DAG Run queued at: {dag_run.queued_at}") # 2023-08-10 00:00:01+02:20
(airflow) 11/17: Rewrite custom view plugin docs (#49408)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v3-0-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit d8219acfb065924fc9567b7426c4285a57b3a007
Author: Pierre Jeambrun <pi...@gmail.com>
AuthorDate: Thu Apr 17 23:27:48 2025 +0200
Rewrite custom view plugin docs (#49408)
(cherry picked from commit 37bd0f0dad3078247c2db8c95a79614f98950dbd)
---
airflow-core/docs/howto/custom-view-plugin.rst | 72 ++++++++++----------------
docs/spelling_wordlist.txt | 2 +
2 files changed, 30 insertions(+), 44 deletions(-)
diff --git a/airflow-core/docs/howto/custom-view-plugin.rst b/airflow-core/docs/howto/custom-view-plugin.rst
index e9a5b795e47..9f9dbf15f49 100644
--- a/airflow-core/docs/howto/custom-view-plugin.rst
+++ b/airflow-core/docs/howto/custom-view-plugin.rst
@@ -20,57 +20,41 @@ Customize view of Apache from Airflow web UI
============================================
Airflow has feature that allows to integrate a custom UI along with its
-core UI using the Plugin manager
+core UI using the Plugin manager.
-This is an example plugin for Airflow that displays absolutely nothing.
+Plugins integrate with the Airflow core RestAPI. In this plugin,
+two object references are derived from the base class ``airflow.plugins_manager.AirflowPlugin``.
+They are fastapi_apps and fastapi_root_middlewares.
-In this plugin, two object references are derived from the base class
-``airflow.plugins_manager.AirflowPlugin``. They are flask_blueprints and
-appbuilder_views
+Using fastapi_apps in Airflow plugin, the core RestAPI can be extended
+to support extra endpoints to serve custom static file or any other json/application responses.
+In this object reference, the list of dictionaries with FastAPI application and metadata information
+like the name and the url prefix are passed on.
-Using flask_blueprints in Airflow plugin, the core application can be extended
-to support the customized application to view Empty Plugin.
-In this object reference, the list of Blueprint object with the static template for
-rendering the information.
+Using fastapi_root_middlewares in Airflow plugin, allows to register custom middleware at the root of
+the FastAPI application. This middleware can be used to add custom headers, logging, or any other
+functionality to the entire FastAPI application, including core endpoints.
+In this object reference, the list of dictionaries with Middleware factories object,
+initialization parameters and some metadata information like the name are passed on.
-Using appbuilder_views in Airflow plugin, a class that represents a concept is
-added and presented with views and methods to implement it.
-In this object reference, the list of dictionaries with FlaskAppBuilder BaseView object
-and metadata information like name and category is passed on.
+Information and code samples to register ``fastapi_apps`` and ``fastapi_root_middlewares`` are
+available in :doc:`plugin </administration-and-deployment/plugins>`.
+Support for Airflow 2 plugins
+=============================
-Custom view Registration
-------------------------
+Airflow 2 plugins are still supported with some limitations. More information on such
+plugins can be found in the Airflow 2 documentation.
-A custom view with object reference to flask_appbuilder and Blueprint from flask
-and be registered as a part of a :doc:`plugin </administration-and-deployment/plugins>`.
+Adding Rest endpoints through the blueprints is still supported, those endpoints will
+be integrated in the FastAPI application via the WSGI Middleware and accessible
+under ``/pluginsv2``.
-The following is a skeleton for us to implement a new custom view:
+It is not possible to extend the core UI, for instance by extending the base template, nonetheless extra menu items
+of the auth managers are added to the core UI security tab and their ``href`` are rendered in iframes.
+This is how the fab provider integrates users, roles, actions, resources and permissions custom views in the Airflow 3 UI.
-.. exampleinclude:: /empty_plugin/empty_plugin.py
- :language: python
-
-``Plugins`` specified in the ``category`` key of ``appbuilder_views`` dictionary is
-the name of the tab in the navigation bar of the Airflow UI. ``Empty Plugin``
-is the name of the link under the tab ``Plugins``, which will launch the plugin
-
-We need to add Blueprint for generating the part of the application
-that needs to be rendered in Airflow web UI. We can define templates, static files
-and this blueprint will be registered as part of the Airflow application when the
-plugin gets loaded.
-
-The ``$AIRFLOW_HOME/plugins`` folder with custom view UI have the following folder structure.
-
-::
-
- plugins
- ├── empty_plugin.py
- ├── templates
- | └── empty_plugin
- | ├── index.html
- └── README.md
-
-The HTML files required to render the views built are added as part of the
-Airflow plugin into ``$AIRFLOW_HOME/plugins/templates`` folder and defined in the
-blueprint.
+Airflow 3 plugins will be improved to allow UI customization for the entire react app, it is recommended
+to upgrade your plugins to Airflow 3 plugins when possible. Until then for a temporary or custom needs
+it is possible to use a Middleware to inject custom javascript or css to the core UI index request.
diff --git a/docs/spelling_wordlist.txt b/docs/spelling_wordlist.txt
index b2688e6f299..60a8d318992 100644
--- a/docs/spelling_wordlist.txt
+++ b/docs/spelling_wordlist.txt
@@ -845,6 +845,7 @@ idempotency
IdP
ie
iframe
+iframes
IGM
igm
ignorable
@@ -1108,6 +1109,7 @@ microservice
microservices
microsoft
middleware
+middlewares
Midnights
midnights
milli
(airflow) 09/17: Use Label class from task sdk (#49385)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v3-0-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 6bf14444bf257ba59579fe71b21c689e1b21dacd
Author: Amogh Desai <am...@gmail.com>
AuthorDate: Thu Apr 17 18:46:31 2025 +0530
Use Label class from task sdk (#49385)
(cherry picked from commit f830f9f4d2c4b5b0ba43bc10bdaf9d6ddcbe6815)
---
airflow-core/docs/core-concepts/dags.rst | 6 +++---
airflow-core/src/airflow/example_dags/example_branch_labels.py | 3 +--
airflow-core/src/airflow/example_dags/example_branch_operator.py | 3 +--
.../src/airflow/example_dags/example_branch_operator_decorator.py | 3 +--
airflow-core/tests/unit/serialization/test_dag_serialization.py | 2 +-
airflow-core/tests/unit/utils/test_dag_cycle.py | 2 +-
airflow-core/tests/unit/utils/test_edgemodifier.py | 2 +-
airflow-core/tests/unit/utils/test_task_group.py | 3 +--
8 files changed, 10 insertions(+), 14 deletions(-)
diff --git a/airflow-core/docs/core-concepts/dags.rst b/airflow-core/docs/core-concepts/dags.rst
index a06ee6a9569..5537ea12d94 100644
--- a/airflow-core/docs/core-concepts/dags.rst
+++ b/airflow-core/docs/core-concepts/dags.rst
@@ -620,7 +620,7 @@ To add labels, you can use them directly inline with the ``>>`` and ``<<`` opera
.. code-block:: python
- from airflow.utils.edgemodifier import Label
+ from airflow.sdk import Label
my_task >> Label("When empty") >> other_task
@@ -628,7 +628,7 @@ Or, you can pass a Label object to ``set_upstream``/``set_downstream``:
.. code-block:: python
- from airflow.utils.edgemodifier import Label
+ from airflow.sdk import Label
my_task.set_downstream(other_task, Label("When empty"))
@@ -638,7 +638,7 @@ Here's an example DAG which illustrates labeling different branches:
.. exampleinclude:: /../src/airflow/example_dags/example_branch_labels.py
:language: python
- :start-after: from airflow.utils.edgemodifier import Label
+ :start-after: from airflow.sdk import DAG, Label
DAG & Task Documentation
diff --git a/airflow-core/src/airflow/example_dags/example_branch_labels.py b/airflow-core/src/airflow/example_dags/example_branch_labels.py
index edc1b059738..fcff3d5a3a1 100644
--- a/airflow-core/src/airflow/example_dags/example_branch_labels.py
+++ b/airflow-core/src/airflow/example_dags/example_branch_labels.py
@@ -24,8 +24,7 @@ from __future__ import annotations
import pendulum
from airflow.providers.standard.operators.empty import EmptyOperator
-from airflow.sdk import DAG
-from airflow.utils.edgemodifier import Label
+from airflow.sdk import DAG, Label
with DAG(
"example_branch_labels",
diff --git a/airflow-core/src/airflow/example_dags/example_branch_operator.py b/airflow-core/src/airflow/example_dags/example_branch_operator.py
index d824de4aa08..0bb429a6f8f 100644
--- a/airflow-core/src/airflow/example_dags/example_branch_operator.py
+++ b/airflow-core/src/airflow/example_dags/example_branch_operator.py
@@ -38,8 +38,7 @@ from airflow.providers.standard.operators.python import (
PythonOperator,
PythonVirtualenvOperator,
)
-from airflow.sdk import DAG
-from airflow.utils.edgemodifier import Label
+from airflow.sdk import DAG, Label
from airflow.utils.trigger_rule import TriggerRule
PATH_TO_PYTHON_BINARY = sys.executable
diff --git a/airflow-core/src/airflow/example_dags/example_branch_operator_decorator.py b/airflow-core/src/airflow/example_dags/example_branch_operator_decorator.py
index 7bc1d94ee95..d3634acb998 100644
--- a/airflow-core/src/airflow/example_dags/example_branch_operator_decorator.py
+++ b/airflow-core/src/airflow/example_dags/example_branch_operator_decorator.py
@@ -31,8 +31,7 @@ import tempfile
import pendulum
from airflow.providers.standard.operators.empty import EmptyOperator
-from airflow.sdk import DAG, task
-from airflow.utils.edgemodifier import Label
+from airflow.sdk import DAG, Label, task
from airflow.utils.trigger_rule import TriggerRule
PATH_TO_PYTHON_BINARY = sys.executable
diff --git a/airflow-core/tests/unit/serialization/test_dag_serialization.py b/airflow-core/tests/unit/serialization/test_dag_serialization.py
index a6609454700..69570392158 100644
--- a/airflow-core/tests/unit/serialization/test_dag_serialization.py
+++ b/airflow-core/tests/unit/serialization/test_dag_serialization.py
@@ -2025,7 +2025,7 @@ class TestStringifiedDAGs:
Tests edge_info serialization/deserialization.
"""
from airflow.providers.standard.operators.empty import EmptyOperator
- from airflow.utils.edgemodifier import Label
+ from airflow.sdk import Label
with DAG(
"test_edge_info_serialization",
diff --git a/airflow-core/tests/unit/utils/test_dag_cycle.py b/airflow-core/tests/unit/utils/test_dag_cycle.py
index ff36fddd20a..c436af01c7d 100644
--- a/airflow-core/tests/unit/utils/test_dag_cycle.py
+++ b/airflow-core/tests/unit/utils/test_dag_cycle.py
@@ -21,8 +21,8 @@ import pytest
from airflow.exceptions import AirflowDagCycleException
from airflow.models.dag import DAG
from airflow.providers.standard.operators.empty import EmptyOperator
+from airflow.sdk import Label
from airflow.utils.dag_cycle_tester import check_cycle
-from airflow.utils.edgemodifier import Label
from airflow.utils.task_group import TaskGroup
from unit.models import DEFAULT_DATE
diff --git a/airflow-core/tests/unit/utils/test_edgemodifier.py b/airflow-core/tests/unit/utils/test_edgemodifier.py
index 79749af5cc0..0885230e113 100644
--- a/airflow-core/tests/unit/utils/test_edgemodifier.py
+++ b/airflow-core/tests/unit/utils/test_edgemodifier.py
@@ -24,8 +24,8 @@ from airflow.models.dag import DAG
from airflow.models.xcom_arg import XComArg
from airflow.providers.standard.operators.empty import EmptyOperator
from airflow.providers.standard.operators.python import PythonOperator
+from airflow.sdk import Label
from airflow.utils.dag_edges import dag_edges
-from airflow.utils.edgemodifier import Label
from airflow.utils.task_group import TaskGroup
DEFAULT_ARGS = {
diff --git a/airflow-core/tests/unit/utils/test_task_group.py b/airflow-core/tests/unit/utils/test_task_group.py
index 38db9cd9dd2..60acb78fbb4 100644
--- a/airflow-core/tests/unit/utils/test_task_group.py
+++ b/airflow-core/tests/unit/utils/test_task_group.py
@@ -1433,8 +1433,7 @@ def test_add_to_another_group():
def test_task_group_edge_modifier_chain():
- from airflow.sdk import chain
- from airflow.utils.edgemodifier import Label
+ from airflow.sdk import Label, chain
with DAG(dag_id="test", schedule=None, start_date=pendulum.DateTime(2022, 5, 20)) as dag:
start = EmptyOperator(task_id="sleep_3_seconds")
(airflow) 12/17: Fix doc alignment in group-extras section (#49425)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v3-0-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit fad509d822aa992149374f91f79c53befee2f2cf
Author: GPK <go...@gmail.com>
AuthorDate: Fri Apr 18 11:41:52 2025 +0100
Fix doc alignment in group-extras section (#49425)
(cherry picked from commit e4eadac8726fd5e27dce275f373278d508a8c1d1)
---
airflow-core/docs/extra-packages-ref.rst | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/airflow-core/docs/extra-packages-ref.rst b/airflow-core/docs/extra-packages-ref.rst
index 46166ee46aa..5ff1fd6d0d8 100644
--- a/airflow-core/docs/extra-packages-ref.rst
+++ b/airflow-core/docs/extra-packages-ref.rst
@@ -402,9 +402,9 @@ Group extras
The group extras are convenience extras. Such extra installs many optional dependencies together.
It is not recommended to use it in production, but it is useful for CI, development and testing purposes.
-+---------+--------------------------------------------+---------------------------------------------------+
-| extra | install command | enables |
-+=========+============================================+===================================================+
++-----------+------------------------------------------+---------------------------------------------------+
+| extra | install command | enables |
++===========+==========================================+===================================================+
| all | ``pip install apache-airflow[all]`` | All optional dependencies including all providers |
+-----------+------------------------------------------+---------------------------------------------------+
| all-core | ``pip install apache-airflow[all-core]`` | All optional core dependencies |
(airflow) 03/17: Update warning message to note in simple auth manager doc (#49367)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v3-0-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit abb46b347f8681f0bd651098f4d7dc3132f2b231
Author: Vincent <97...@users.noreply.github.com>
AuthorDate: Wed Apr 16 14:12:44 2025 -0400
Update warning message to note in simple auth manager doc (#49367)
(cherry picked from commit 84cd1d6eb25fa2c9484c4310fdbab037e6f19924)
---
airflow-core/docs/core-concepts/auth-manager/simple/index.rst | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/airflow-core/docs/core-concepts/auth-manager/simple/index.rst b/airflow-core/docs/core-concepts/auth-manager/simple/index.rst
index edc8fdd1a56..2a51b4821fa 100644
--- a/airflow-core/docs/core-concepts/auth-manager/simple/index.rst
+++ b/airflow-core/docs/core-concepts/auth-manager/simple/index.rst
@@ -18,7 +18,7 @@
Simple auth manager
===================
-.. warning::
+.. note::
The Simple auth manager is intended for development and testing. If you're using it in production, ensure that
access is controlled through other means.
(airflow) 16/17: Edit airflow 3 migration guide (#49414)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v3-0-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit b71c0f57930e579d21a9e46336993f8c4bd2bef9
Author: Ryan Hatter <25...@users.noreply.github.com>
AuthorDate: Sat Apr 19 06:16:44 2025 -0400
Edit airflow 3 migration guide (#49414)
(cherry picked from commit 1b44508b342c4c94284d854b981316cf2022c6d9)
---
.../docs/installation/upgrading_to_airflow3.rst | 69 +++++++++++-----------
1 file changed, 33 insertions(+), 36 deletions(-)
diff --git a/airflow-core/docs/installation/upgrading_to_airflow3.rst b/airflow-core/docs/installation/upgrading_to_airflow3.rst
index 208f4ba5cd4..e30a0a41039 100644
--- a/airflow-core/docs/installation/upgrading_to_airflow3.rst
+++ b/airflow-core/docs/installation/upgrading_to_airflow3.rst
@@ -18,52 +18,49 @@
Upgrading to Airflow 3
=======================
-Apache Airflow 3 is a major release. This guide walks you through the steps required to upgrade from Airflow 2.x to Airflow 3.0.
+Apache Airflow 3 is a major release and contains :ref:`breaking changes<breaking-changes>`. This guide walks you through the steps required to upgrade from Airflow 2.x to Airflow 3.0.
Step 1: Take care of prerequisites
----------------------------------
- Make sure that you are on Airflow 2.7 or later.
- Make sure that your Python version is in the supported list. Airflow 3.0.0 supports the following Python versions: Python 3.9, 3.10, 3.11 and 3.12.
-- Ensure that you are not using SubDAGs. These were deprecated in Airflow 2.0 and removed in Airflow 3.
-- For a complete list of breaking changes, which you should note before the upgrade, please check the breaking changes section below.
+- Ensure that you are not using any features or functionality that have been :ref:`removed in Airflow 3<breaking-changes>`.
+
Step 2: Clean and back up your existing Airflow Instance
---------------------------------------------------------
-- It is highly recommended to make a backup of your Airflow instance specifically including your Airflow metadata DB before starting the migration process.
-- If you do not have a "hot backup" capability for your DB, you should do it after shutting down your Airflow instances, so that the backup of your database will be consistent.
-- If you did not make a backup and your migration fails, you might end up in a half-migrated state and restoring DB from backup and repeating the migration
- might be the only easy way out. This can for example be caused by a broken network connection between your CLI and the database while the migration happens, so taking a
- backup is an important precaution to avoid problems like this.
-- A long running Airflow instance can accumulate a certain amount of silt, in the form of old database entries, which are no longer
- required. This is typically in the form of old XCom data which is no longer required, and so on. As part of the Airflow 3 upgrade
- process, there will be schema changes. Based on the size of the Airflow meta-database this can be somewhat time
- consuming. For a faster, safer migration, we recommend that you clean up your Airflow meta-database before the upgrade.
- You can use ``airflow db clean`` command for that.
+- It is highly recommended that you make a backup of your Airflow instance, specifically your Airflow metadata database before starting the migration process.
+
+ - If you do not have a "hot backup" capability for your database, you should do it after shutting down your Airflow instances, so that the backup of your database will be consistent. For example, if you don't turn off your Airflow instance, the backup of the database will not include all TaskInstances or DagRuns.
+
+ - If you did not make a backup and your migration fails, you might end up in a half-migrated state. This can be caused by, for example, a broken network connection between your Airflow CLI and the database during the migration. Having a backup is an important precaution to avoid problems like this.
+
+- A long running Airflow instance can accumulate a substantial amount of data that are no longer required (for example, old XCom data). Schema changes will be a part of the Airflow 3
+ upgrade process. These schema changes can take a long time if the database is large. For a faster, safer migration, we recommend that you clean up your Airflow meta-database before the upgrade.
+ You can use the ``airflow db clean`` :ref:`Airflow CLI command<cli-db-clean>` to trim your Airflow database.
+
Step 3: DAG Authors - Check your Airflow DAGs for compatibility
----------------------------------------------------------------
-To minimize friction for users upgrading from prior versions of Airflow, we have created a DAG upgrade check utility using `Ruff <https://docs.astral.sh/ruff/>`_.
+To minimize friction for users upgrading from prior versions of Airflow, we have created a dag upgrade check utility using `Ruff <https://docs.astral.sh/ruff/>`_.
-Use the latest available ``ruff`` version to get updates to the rules but at the very least use ``0.11.6``:
+The latest available ``ruff`` version will have the most up-to-date rules, but be sure to use at least version ``0.11.6``. The below example demonstrates how to check
+for dag incompatibilities that will need to be fixed before they will work as expected on Airflow 3.
.. code-block:: bash
ruff check dag/ --select AIR301
-This command above shows you all the errors which need to be fixed before these DAGs can be used on Airflow 3.
-
-Some of these changes are automatically fixable and you can also rerun the command above with the auto-fix option as shown below.
-
-To preview the changes:
+To preview the recommended fixes, run the following command:
.. code-block:: bash
ruff check dag/ --select AIR301 --show-fixes
-To auto-fix:
+Some changes can be automatically fixed. To do so, run the following command:
.. code-block:: bash
@@ -72,15 +69,16 @@ To auto-fix:
Step 4: Install the Standard Providers
--------------------------------------
-- Some of the commonly used Operators which were bundled as part of the Core Airflow OSS package such as the
- Bash and Python Operators have now been split out into a separate package: ``apache-airflow-providers-standard``.
-- For user convenience, this package can also be installed on Airflow 2.x versions, so that DAGs can be modified to reference these Operators from the Standard Provider package instead of Airflow Core.
+- Some of the commonly used Operators which were bundled as part of the ``airflow-core`` package (for example ``BashOperator`` and ``PythonOperator``)
+ have now been split out into a separate package: ``apache-airflow-providers-standard``.
+- For convenience, this package can also be installed on Airflow 2.x versions, so that DAGs can be modified to reference these Operators from the standard provider
+ package instead of Airflow Core.
Step 5: Deployment Managers - Upgrade your Airflow Instance
------------------------------------------------------------
-For an easier and safer upgrade process, we have also created a utility to upgrade your Airflow instance configuration as a deployment manager.
+For an easier and safer upgrade process, we have also created a utility to upgrade your Airflow instance configuration.
The first step is to run this configuration check utility as shown below:
@@ -97,8 +95,7 @@ This configuration utility can also update your configuration to automatically b
airflow config update --fix
-The biggest part of an Airflow upgrade is the database upgrade. The database upgrade process for Airflow 3 is the same as for Airflow 2.7 or later.
-
+The biggest part of an Airflow upgrade is the database upgrade. The database upgrade process for Airflow 3 is the same as for Airflow 2.7 or later:
.. code-block:: bash
@@ -111,19 +108,21 @@ You should now be able to start up your Airflow 3 instance.
Step 6: Changes to your startup scripts
---------------------------------------
-- In Airflow 3, the Webserver has now become a generic API-server. The api-server can be started up using the following command:
+In Airflow 3, the Webserver has become a generic API server. The API server can be started up using the following command:
.. code-block:: bash
airflow api-server
-- The DAG processor must now be started independently, even for local or development setups.
+The dag processor must now be started independently, even for local or development setups:
.. code-block:: bash
airflow dag-processor
+.. _breaking-changes:
+
Breaking Changes
================
@@ -133,9 +132,8 @@ These include:
- **SubDAGs**: Replaced by TaskGroups, Datasets, and Data Aware Scheduling.
- **Sequential Executor**: Replaced by LocalExecutor, which can be used with SQLite for local development use cases.
- **SLAs**: Deprecated and removed; Will be replaced by forthcoming `Deadline Alerts <https://cwiki.apache.org/confluence/x/tglIEw>`_.
-- **Subdir**: Used as an argument on many CLI commands (``--subdir`` or ``-S`` has been superseded by DAG bundles.
-- **Following keys are no longer available in task context. If not replaced, will cause DAG errors**:
-
+- **Subdir**: Used as an argument on many CLI commands, ``--subdir`` or ``-S`` has been superseded by :doc:`DAG bundles </administration-and-deployment/dag-bundles>`.
+- **Some Airflow context variables**: The following keys are no longer available in a :ref:`task instance's context <templates:variables>`. If not replaced, will cause dag errors:
- ``tomorrow_ds``
- ``tomorrow_ds_nodash``
- ``yesterday_ds``
@@ -148,10 +146,9 @@ These include:
- ``next_ds_nodash``
- ``next_ds``
- ``execution_date``
-
-- ``catchup_by_default`` is now ``False`` by default.
-- ``create_cron_data_intervals`` is now ``False``. This means that the ``CronTriggerTimetable`` will be used by default instead of the ``CronDataIntervalTimetable``
-- **Simple Auth** is now default ``auth_manager``. To continue using FAB as the Auth Manager, please install the FAB provider and set ``auth_manager`` to
+- The ``catchup_by_default`` dag parameter is now ``False`` by default.
+- The ``create_cron_data_intervals`` configuration is now ``False`` by default. This means that the ``CronTriggerTimetable`` will be used by default instead of the ``CronDataIntervalTimetable``
+- **Simple Auth** is now default ``auth_manager``. To continue using FAB as the Auth Manager, please install the FAB provider and set ``auth_manager`` to ``FabAuthManager``:
.. code-block:: ini
(airflow) 17/17: Use sync calls for list backfills (#49454)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v3-0-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 40cb51d135094c6eca7c69e75fb192f9d1113bde
Author: Daniel Standish <15...@users.noreply.github.com>
AuthorDate: Sat Apr 19 03:19:13 2025 -0700
Use sync calls for list backfills (#49454)
We found issues when running this with pg bouncer. We can make it sync until we sort that stuff out.
(cherry picked from commit 01ed94b6372db7c8531080097acedbba12db0a0b)
---
.../airflow/api_fastapi/core_api/routes/public/backfills.py | 11 +++++------
1 file changed, 5 insertions(+), 6 deletions(-)
diff --git a/airflow-core/src/airflow/api_fastapi/core_api/routes/public/backfills.py b/airflow-core/src/airflow/api_fastapi/core_api/routes/public/backfills.py
index e4995cb36c3..7b2e30ba2ab 100644
--- a/airflow-core/src/airflow/api_fastapi/core_api/routes/public/backfills.py
+++ b/airflow-core/src/airflow/api_fastapi/core_api/routes/public/backfills.py
@@ -24,9 +24,8 @@ from sqlalchemy import select, update
from airflow.api_fastapi.auth.managers.models.resource_details import DagAccessEntity
from airflow.api_fastapi.common.db.common import (
- AsyncSessionDep,
SessionDep,
- paginated_select_async,
+ paginated_select,
)
from airflow.api_fastapi.common.parameters import QueryLimit, QueryOffset, SortParam
from airflow.api_fastapi.common.router import AirflowRouter
@@ -67,7 +66,7 @@ backfills_router = AirflowRouter(tags=["Backfill"], prefix="/backfills")
Depends(requires_access_backfill(method="GET")),
],
)
-async def list_backfills(
+def list_backfills(
dag_id: str,
limit: QueryLimit,
offset: QueryOffset,
@@ -75,16 +74,16 @@ async def list_backfills(
SortParam,
Depends(SortParam(["id"], Backfill).dynamic_depends()),
],
- session: AsyncSessionDep,
+ session: SessionDep,
) -> BackfillCollectionResponse:
- select_stmt, total_entries = await paginated_select_async(
+ select_stmt, total_entries = paginated_select(
statement=select(Backfill).where(Backfill.dag_id == dag_id),
order_by=order_by,
offset=offset,
limit=limit,
session=session,
)
- backfills = await session.scalars(select_stmt)
+ backfills = session.scalars(select_stmt)
return BackfillCollectionResponse(
backfills=backfills,
total_entries=total_entries,
(airflow) 15/17: docs: grammar fixes (#49432)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v3-0-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 35e46224b2398afc3a788b7652a193a941ad7151
Author: Kalyan R <ka...@live.com>
AuthorDate: Fri Apr 18 16:41:28 2025 +0530
docs: grammar fixes (#49432)
* proof read installation's index page
* proof read installation's index page and dependencies
* pf supported-versions.rst
* pf install from source
* pf install from pypi
* remove extra colon in upgrading to airflow3
* Capitalize the Airflow
(cherry picked from commit 2532f727b1501297325b718095c4765d55572de8)
---
PROVIDERS.rst | 4 +-
.../cluster-policies.rst | 2 +-
.../docs/administration-and-deployment/index.rst | 2 +-
.../advanced-logging-configuration.rst | 2 +-
.../modules_management.rst | 2 +-
.../docs/administration-and-deployment/plugins.rst | 4 +-
.../docs/authoring-and-scheduling/connections.rst | 2 +-
.../docs/authoring-and-scheduling/index.rst | 2 +-
airflow-core/docs/best-practices.rst | 4 +-
airflow-core/docs/configurations-ref.rst | 2 +-
airflow-core/docs/core-concepts/backfill.rst | 2 +-
airflow-core/docs/core-concepts/operators.rst | 2 +-
airflow-core/docs/core-concepts/overview.rst | 2 +-
airflow-core/docs/extra-packages-ref.rst | 14 +++---
airflow-core/docs/howto/email-config.rst | 2 +-
airflow-core/docs/howto/export-more-env-vars.rst | 2 +-
airflow-core/docs/howto/listener-plugin.rst | 6 +--
airflow-core/docs/howto/set-config.rst | 2 +-
airflow-core/docs/howto/usage-cli.rst | 2 +-
airflow-core/docs/installation/dependencies.rst | 4 +-
airflow-core/docs/installation/index.rst | 50 +++++++++++-----------
.../docs/installation/installing-from-pypi.rst | 40 ++++++++---------
.../docs/installation/installing-from-sources.rst | 12 +++---
airflow-core/docs/installation/prerequisites.rst | 11 +++--
.../docs/installation/supported-versions.rst | 10 ++---
.../docs/installation/upgrading_to_airflow3.rst | 4 +-
airflow-core/docs/templates-ref.rst | 2 +-
27 files changed, 94 insertions(+), 99 deletions(-)
diff --git a/PROVIDERS.rst b/PROVIDERS.rst
index 1d1386cceb4..ced8f5fb1b4 100644
--- a/PROVIDERS.rst
+++ b/PROVIDERS.rst
@@ -57,7 +57,7 @@ releasing new versions of the providers. This means that the code changes in the
reviewed by Airflow committers and merged when they are accepted by them. Also we must have sufficient
test coverage and documentation that allow us to maintain the providers, and our users to use them.
-The providers - their latest version in "main" branch of airflow repository - are installed and tested together
+The providers - their latest version in "main" branch of Airflow repository - are installed and tested together
with other community providers and one of the key properties of the community providers is that the latest
version of providers contribute their dependencies to constraints of Airflow, published when Airflow Core is
released. This means that when users are using constraints published by Airflow, they can install all
@@ -92,7 +92,7 @@ Accepting new community providers
---------------------------------
Accepting new community providers should be a deliberate process that requires ``[DISCUSSION]``
-followed by ``[VOTE]`` thread at the airflow `devlist <https://airflow.apache.org/community/#mailing-list>`_.
+followed by ``[VOTE]`` thread at the Airflow `devlist <https://airflow.apache.org/community/#mailing-list>`_.
In case the provider is integration with an open-source software rather than service we can relax the vote
procedure a bit. Particularly if the open-source software is an Apache Software Foundation,
diff --git a/airflow-core/docs/administration-and-deployment/cluster-policies.rst b/airflow-core/docs/administration-and-deployment/cluster-policies.rst
index e0b71eef7a3..e43f13e7450 100644
--- a/airflow-core/docs/administration-and-deployment/cluster-policies.rst
+++ b/airflow-core/docs/administration-and-deployment/cluster-policies.rst
@@ -152,7 +152,7 @@ Here's an example of enforcing a maximum timeout policy on every task:
:start-after: [START example_task_cluster_policy]
:end-before: [END example_task_cluster_policy]
-You could also implement to protect against common errors, rather than as technical security controls. For example, don't run tasks without airflow owners:
+You could also implement to protect against common errors, rather than as technical security controls. For example, don't run tasks without Airflow owners:
.. literalinclude:: /../tests/unit/cluster_policies/__init__.py
:language: python
diff --git a/airflow-core/docs/administration-and-deployment/index.rst b/airflow-core/docs/administration-and-deployment/index.rst
index ec39a526a77..720f5f695eb 100644
--- a/airflow-core/docs/administration-and-deployment/index.rst
+++ b/airflow-core/docs/administration-and-deployment/index.rst
@@ -18,7 +18,7 @@
Administration and Deployment
=====================================
-This section contains information about deploying dags into production and the administration of airflow deployments.
+This section contains information about deploying dags into production and the administration of Airflow deployments.
.. toctree::
:maxdepth: 2
diff --git a/airflow-core/docs/administration-and-deployment/logging-monitoring/advanced-logging-configuration.rst b/airflow-core/docs/administration-and-deployment/logging-monitoring/advanced-logging-configuration.rst
index 2cfc44e7255..342de5e016a 100644
--- a/airflow-core/docs/administration-and-deployment/logging-monitoring/advanced-logging-configuration.rst
+++ b/airflow-core/docs/administration-and-deployment/logging-monitoring/advanced-logging-configuration.rst
@@ -25,7 +25,7 @@ Not all configuration options are available from the ``airflow.cfg`` file. The c
how to configure logging for tasks, because the logs generated by tasks are not only logged in separate
files by default but has to be also accessible via the webserver.
-By default standard airflow component logs are written to the ``$AIRFLOW_HOME/logs`` directory, but you
+By default standard Airflow component logs are written to the ``$AIRFLOW_HOME/logs`` directory, but you
can also customize it and configure it as you want by overriding Python logger configuration that can
be configured by providing custom logging configuration object. You can also create and use logging configuration
for specific operators and tasks.
diff --git a/airflow-core/docs/administration-and-deployment/modules_management.rst b/airflow-core/docs/administration-and-deployment/modules_management.rst
index 865dab06867..e6b8b7d14fc 100644
--- a/airflow-core/docs/administration-and-deployment/modules_management.rst
+++ b/airflow-core/docs/administration-and-deployment/modules_management.rst
@@ -318,7 +318,7 @@ try to import the package now:
>>>
We can also use :envvar:`PYTHONPATH` variable with the airflow commands.
-For example, if we run the following airflow command:
+For example, if we run the following Airflow command:
.. code-block:: bash
diff --git a/airflow-core/docs/administration-and-deployment/plugins.rst b/airflow-core/docs/administration-and-deployment/plugins.rst
index 47c0f2fecdf..6d143c09609 100644
--- a/airflow-core/docs/administration-and-deployment/plugins.rst
+++ b/airflow-core/docs/administration-and-deployment/plugins.rst
@@ -115,7 +115,7 @@ looks like:
# A list of dictionaries containing kwargs for FlaskAppBuilder add_link. See example below
appbuilder_menu_items = []
- # A callback to perform actions when airflow starts and the plugin is loaded.
+ # A callback to perform actions when Airflow starts and the plugin is loaded.
# NOTE: Ensure your plugin has *args, and **kwargs in the method definition
# to protect against extra parameters injected into the on_load(...)
# function in future changes
@@ -192,7 +192,7 @@ definitions in Airflow.
static_url_path="/static/test_plugin",
)
- # Creating a FastAPI application to integrate in airflow Rest API.
+ # Creating a FastAPI application to integrate in Airflow Rest API.
app = FastAPI()
diff --git a/airflow-core/docs/authoring-and-scheduling/connections.rst b/airflow-core/docs/authoring-and-scheduling/connections.rst
index 2f24e3cc83b..ba74826519e 100644
--- a/airflow-core/docs/authoring-and-scheduling/connections.rst
+++ b/airflow-core/docs/authoring-and-scheduling/connections.rst
@@ -38,7 +38,7 @@ A Hook is a high-level interface to an external platform that lets you quickly a
They integrate with Connections to gather credentials, and many have a default ``conn_id``; for example, the :class:`~airflow.providers.postgres.hooks.postgres.PostgresHook` automatically looks for the Connection with a ``conn_id`` of ``postgres_default`` if you don't pass one in.
-You can view a :ref:`full list of airflow hooks <pythonapi:hooks>` in our API documentation.
+You can view a :ref:`full list of Airflow hooks <pythonapi:hooks>` in our API documentation.
Custom connections
------------------
diff --git a/airflow-core/docs/authoring-and-scheduling/index.rst b/airflow-core/docs/authoring-and-scheduling/index.rst
index fab495298c4..1b78d588bab 100644
--- a/airflow-core/docs/authoring-and-scheduling/index.rst
+++ b/airflow-core/docs/authoring-and-scheduling/index.rst
@@ -18,7 +18,7 @@
Authoring and Scheduling
=========================
-Here you can find detailed documentation about advanced authoring and scheduling airflow dags.
+Here you can find detailed documentation about advanced authoring and scheduling Airflow dags.
It's recommended that you first review the pages in :doc:`core concepts </core-concepts/index>`
.. _authoring-section:
diff --git a/airflow-core/docs/best-practices.rst b/airflow-core/docs/best-practices.rst
index 9f596a9d257..bc4889b5f20 100644
--- a/airflow-core/docs/best-practices.rst
+++ b/airflow-core/docs/best-practices.rst
@@ -451,7 +451,7 @@ for any variable that contains sensitive data.
Timetables
----------
-Avoid using Airflow Variables/Connections or accessing airflow database at the top level of your timetable code.
+Avoid using Airflow Variables/Connections or accessing Airflow database at the top level of your timetable code.
Database access should be delayed until the execution time of the DAG. This means that you should not have variables/connections retrieval
as argument to your timetable class initialization or have Variable/connection at the top level of your custom timetable module.
@@ -980,7 +980,7 @@ The benefits of the operator are:
* There is no need to prepare the venv upfront. It will be dynamically created before task is run, and
removed after it is finished, so there is nothing special (except having virtualenv package in your
- airflow dependencies) to make use of multiple virtual environments
+ Airflow dependencies) to make use of multiple virtual environments
* You can run tasks with different sets of dependencies on the same workers - thus Memory resources are
reused (though see below about the CPU overhead involved in creating the venvs).
* In bigger installations, DAG Authors do not need to ask anyone to create the venvs for you.
diff --git a/airflow-core/docs/configurations-ref.rst b/airflow-core/docs/configurations-ref.rst
index 2d6b9cc3cc8..e67b8e39ce2 100644
--- a/airflow-core/docs/configurations-ref.rst
+++ b/airflow-core/docs/configurations-ref.rst
@@ -29,7 +29,7 @@ should be same on the Webserver and Worker to allow Webserver to fetch logs from
The webserver key is also used to authorize requests to Celery workers when logs are retrieved. The token
generated using the secret key has a short expiry time though - make sure that time on ALL the machines
-that you run airflow components on is synchronized (for example using ntpd) otherwise you might get
+that you run Airflow components on is synchronized (for example using ntpd) otherwise you might get
"forbidden" errors when the logs are accessed.
.. note::
diff --git a/airflow-core/docs/core-concepts/backfill.rst b/airflow-core/docs/core-concepts/backfill.rst
index ac6729b23d0..955a3661557 100644
--- a/airflow-core/docs/core-concepts/backfill.rst
+++ b/airflow-core/docs/core-concepts/backfill.rst
@@ -20,7 +20,7 @@ Backfill
Backfill is when you create runs for past dates of a dag. Airflow provides a mechanism
to do this through the CLI and REST API. You provide a dag, a start date, and an end date,
-and airflow will create runs in the range according to the dag's schedule.
+and Airflow will create runs in the range according to the dag's schedule.
Backfill does not make sense for dags that don't have a time-based schedule.
diff --git a/airflow-core/docs/core-concepts/operators.rst b/airflow-core/docs/core-concepts/operators.rst
index 0d66ba3f61d..596b10ae169 100644
--- a/airflow-core/docs/core-concepts/operators.rst
+++ b/airflow-core/docs/core-concepts/operators.rst
@@ -199,7 +199,7 @@ In some cases, you may want to exclude a string from templating and use it direc
)
This will fail with ``TemplateNotFound: cat script.sh`` since Airflow would treat the string as a path to a file, not a command.
-We can prevent airflow from treating this value as a reference to a file by wrapping it in :func:`~airflow.util.template.literal`.
+We can prevent Airflow from treating this value as a reference to a file by wrapping it in :func:`~airflow.util.template.literal`.
This approach disables the rendering of both macros and files and can be applied to selected nested fields while retaining the default templating rules for the remainder of the content.
.. code-block:: python
diff --git a/airflow-core/docs/core-concepts/overview.rst b/airflow-core/docs/core-concepts/overview.rst
index 027d14da12e..a0111b59183 100644
--- a/airflow-core/docs/core-concepts/overview.rst
+++ b/airflow-core/docs/core-concepts/overview.rst
@@ -59,7 +59,7 @@ A minimal Airflow installation consists of the following components:
* A folder of *DAG files*, which is read by the *scheduler* to figure out what tasks to run and when to
run them.
-* A *metadata database*, which airflow components use to store state of workflows and tasks.
+* A *metadata database*, which Airflow components use to store state of workflows and tasks.
Setting up a metadata database is described in :doc:`/howto/set-up-database` and is required for
Airflow to work.
diff --git a/airflow-core/docs/extra-packages-ref.rst b/airflow-core/docs/extra-packages-ref.rst
index 5ff1fd6d0d8..9c360209ac2 100644
--- a/airflow-core/docs/extra-packages-ref.rst
+++ b/airflow-core/docs/extra-packages-ref.rst
@@ -26,18 +26,18 @@ already existing ``providers`` and the dependencies are isolated and simplified
packages.
While the original installation methods via ``apache-airflow`` distribution package and extras still
-work as previously and it installs complete airflow installation ready to serve as scheduler, webserver, triggerer
+work as previously and it installs complete Airflow installation ready to serve as scheduler, webserver, triggerer
and worker, the ``apache-airflow`` package is now a meta-package that installs all the other distribution
packages, it's also possible to install only the distribution packages that are needed for a specific
-component you want to run airflow with.
+component you want to run Airflow with.
The following distribution packages are available:
+----------------------------+------------------------------------------------------------------+----------------------------------------------------------+
| Distribution package | Purpose | Optional extras |
+----------------------------+------------------------------------------------------------------+----------------------------------------------------------+
-| apache-airflow-core | This is the core distribution package that contains | * Core extras that add optional functionality to airflow |
-| | the airflow scheduler, webserver, triggerer code. | core system - enhancing its functionality across |
+| apache-airflow-core | This is the core distribution package that contains | * Core extras that add optional functionality to Airflow |
+| | the Airflow scheduler, webserver, triggerer code. | core system - enhancing its functionality across |
| | | multiple providers. |
| | | |
| | | * Group ``all`` extra that installs all optional |
@@ -71,7 +71,7 @@ The following distribution packages are available:
As mentioned above, Airflow has a number of optional "extras" that you can use to add features to your
installation when you are installing Airflow. Those extras are a good way for the users to manage their
-installation, but also they are useful for contributors to airflow when they want to contribute some of
+installation, but also they are useful for contributors to Airflow when they want to contribute some of
the features - including optional integrations of Airflow - via providers.
Here's the list of all the extra dependencies of Apache Airflow.
@@ -79,7 +79,7 @@ Here's the list of all the extra dependencies of Apache Airflow.
Core Airflow extras
-------------------
-These are core airflow extras that extend capabilities of core Airflow. They usually do not install provider
+These are core Airflow extras that extend capabilities of core Airflow. They usually do not install provider
packages (with the exception of ``celery`` and ``cncf.kubernetes`` extras), they just install necessary
python dependencies for the provided package.
@@ -129,7 +129,7 @@ Providers extras
These providers extras are simply convenience extras to install providers so that you can install the providers with simple command - including
provider package and necessary dependencies in single command, which allows PIP to resolve any conflicting dependencies. This is extremely useful
-for first time installation where you want to repeatably install version of dependencies which are 'valid' for both airflow and providers installed.
+for first time installation where you want to repeatably install version of dependencies which are 'valid' for both Airflow and providers installed.
For example the below command will install:
diff --git a/airflow-core/docs/howto/email-config.rst b/airflow-core/docs/howto/email-config.rst
index 82a3e745c1b..532f987aca7 100644
--- a/airflow-core/docs/howto/email-config.rst
+++ b/airflow-core/docs/howto/email-config.rst
@@ -88,7 +88,7 @@ Send email using SendGrid
Using Default SMTP
^^^^^^^^^^^^^^^^^^
-You can use the default airflow SMTP backend to send email with SendGrid
+You can use the default Airflow SMTP backend to send email with SendGrid
.. code-block:: ini
diff --git a/airflow-core/docs/howto/export-more-env-vars.rst b/airflow-core/docs/howto/export-more-env-vars.rst
index e393f479302..a35b3be6fb5 100644
--- a/airflow-core/docs/howto/export-more-env-vars.rst
+++ b/airflow-core/docs/howto/export-more-env-vars.rst
@@ -23,7 +23,7 @@ Export dynamic environment variables available for operators to use
The key value pairs returned in ``get_airflow_context_vars`` defined in
-``airflow_local_settings.py`` are injected to default airflow context environment variables,
+``airflow_local_settings.py`` are injected to default Airflow context environment variables,
which are available as environment variables when running tasks. Note, both key and
value are must be string.
diff --git a/airflow-core/docs/howto/listener-plugin.rst b/airflow-core/docs/howto/listener-plugin.rst
index 20569a3fc6f..9d139093548 100644
--- a/airflow-core/docs/howto/listener-plugin.rst
+++ b/airflow-core/docs/howto/listener-plugin.rst
@@ -44,14 +44,14 @@ Using this plugin, following events can be listened:
* dag run is in running state.
* dag run is in success state.
* dag run is in failure state.
- * on start before event like airflow job, scheduler
- * before stop for event like airflow job, scheduler
+ * on start before event like Airflow job, scheduler
+ * before stop for event like Airflow job, scheduler
Listener Registration
---------------------
A listener plugin with object reference to listener object is registered
-as part of airflow plugin. The following is a
+as part of Airflow plugin. The following is a
skeleton for us to implement a new listener:
.. code-block:: python
diff --git a/airflow-core/docs/howto/set-config.rst b/airflow-core/docs/howto/set-config.rst
index b25bf3470a8..2ff69415964 100644
--- a/airflow-core/docs/howto/set-config.rst
+++ b/airflow-core/docs/howto/set-config.rst
@@ -164,7 +164,7 @@ the example below.
The webserver key is also used to authorize requests to Celery workers when logs are retrieved. The token
generated using the secret key has a short expiry time though - make sure that time on ALL the machines
- that you run airflow components on is synchronized (for example using ntpd) otherwise you might get
+ that you run Airflow components on is synchronized (for example using ntpd) otherwise you might get
"forbidden" errors when the logs are accessed.
.. _set-config:configuring-local-settings:
diff --git a/airflow-core/docs/howto/usage-cli.rst b/airflow-core/docs/howto/usage-cli.rst
index a3bb13e0759..82005804b10 100644
--- a/airflow-core/docs/howto/usage-cli.rst
+++ b/airflow-core/docs/howto/usage-cli.rst
@@ -44,7 +44,7 @@ For permanent (but not global) airflow activation, use:
register-python-argcomplete airflow >> ~/.bashrc
-For one-time activation of argcomplete for airflow only, use:
+For one-time activation of argcomplete for Airflow only, use:
.. code-block:: bash
diff --git a/airflow-core/docs/installation/dependencies.rst b/airflow-core/docs/installation/dependencies.rst
index 515b2a9d3c8..e41e03ab46a 100644
--- a/airflow-core/docs/installation/dependencies.rst
+++ b/airflow-core/docs/installation/dependencies.rst
@@ -23,7 +23,7 @@ Airflow extra dependencies
The ``apache-airflow`` PyPI basic package only installs what's needed to get started.
Additional packages can be installed depending on what will be useful in your
-environment. For instance, if you don't need connectivity with Postgres,
+environment. For instance, if you don't need connectivity with PostgreSQL,
you won't have to go through the trouble of installing the ``postgres-devel``
yum package, or whatever equivalent applies on the distribution you are using.
@@ -55,7 +55,7 @@ Just to prevent confusion of extras versus providers: Extras and providers are d
though many extras are leading to installing providers.
Extras are standard Python setuptools feature that allows to add additional set of dependencies as
-optional features to "core" Apache Airflow. One of the type of such optional features are providers
+optional features to "core" Apache Airflow. One type of such optional features is providers
packages, but not all optional features of Apache Airflow have corresponding providers.
We are using the ``extras`` setuptools features to also install providers.
diff --git a/airflow-core/docs/installation/index.rst b/airflow-core/docs/installation/index.rst
index 492dd76c68c..a72f44fb40f 100644
--- a/airflow-core/docs/installation/index.rst
+++ b/airflow-core/docs/installation/index.rst
@@ -35,12 +35,12 @@ Installation of Airflow®
Upgrading <upgrading>
Upgrading to Airflow 3 <upgrading_to_airflow3>
-This page describes installations options that you might use when considering how to install Airflow®.
+This page describes installation options that you might use when considering how to install Airflow®.
Airflow consists of many components, often distributed among many physical or virtual machines, therefore
installation of Airflow might be quite complex, depending on the options you choose.
-You should also check-out the :doc:`Prerequisites <prerequisites>` that must be fulfilled when installing Airflow
-as well as :doc:`Supported versions <supported-versions>` to know what are the policies for supporting
+You should also check out the :doc:`Prerequisites <prerequisites>` that must be fulfilled when installing Airflow
+as well as :doc:`Supported versions <supported-versions>` to know what are the policies for the supporting
Airflow, Python and Kubernetes.
Airflow requires additional :doc:`Dependencies <dependencies>` to be installed - which can be done
@@ -68,9 +68,9 @@ More details: :doc:`installing-from-sources`
**What are you expected to handle**
-* You are expected to build and install airflow and its components on your own.
+* You are expected to build and install Airflow and its components on your own.
* You should develop and handle the deployment for all components of Airflow.
-* You are responsible for setting up database, creating and managing database schema with ``airflow db`` commands,
+* You are responsible for setting up the database, creating and managing database schema with ``airflow db`` commands,
automated startup and recovery, maintenance, cleanup and upgrades of Airflow and the Airflow Providers.
* You need to setup monitoring of your system allowing you to observe resources and react to problems.
* You are expected to configure and manage appropriate resources for the installation (memory, CPU, etc) based
@@ -84,14 +84,14 @@ More details: :doc:`installing-from-sources`
**Where to ask for help**
-* The ``#user-troubleshooting`` channel on slack can be used for quick general troubleshooting questions. The
+* The ``#user-troubleshooting`` channel on Slack can be used for quick general troubleshooting questions. The
`GitHub discussions <https://github.com/apache/airflow/discussions>`__ if you look for longer discussion and have more information to share.
-* The ``#user-best-practices`` channel on slack can be used to ask for and share best practices on using and deploying airflow.
+* The ``#user-best-practices`` channel on Slack can be used to ask for and share best practices on using and deploying Airflow.
* If you can provide description of a reproducible problem with Airflow software, you can open issue at `GitHub issues <https://github.com/apache/airflow/issues>`_
-* If you want to contribute back to Airflow, the ``#contributors`` slack channel for building the Airflow itself
+* If you want to contribute back to Airflow, the ``#contributors`` Slack channel for building the Airflow itself
Using PyPI
@@ -103,7 +103,7 @@ More details: :doc:`/installation/installing-from-pypi`
* This installation method is useful when you are not familiar with Containers and Docker and want to install
Apache Airflow on physical or virtual machines and you are used to installing and running software using custom
- deployment mechanism.
+ deployment mechanisms.
* The only officially supported mechanism of installation is via ``pip`` using constraint mechanisms. The constraint
files are managed by Apache Airflow release managers to make sure that you can repeatably install Airflow from PyPI with all Providers and
@@ -122,7 +122,7 @@ More details: :doc:`/installation/installing-from-pypi`
* You are expected to install Airflow - all components of it - on your own.
* You should develop and handle the deployment for all components of Airflow.
-* You are responsible for setting up database, creating and managing database schema with ``airflow db`` commands,
+* You are responsible for setting up the database, creating and managing database schema with ``airflow db`` commands,
automated startup and recovery, maintenance, cleanup and upgrades of Airflow and Airflow Providers.
* You need to setup monitoring of your system allowing you to observe resources and react to problems.
* You are expected to configure and manage appropriate resources for the installation (memory, CPU, etc) based
@@ -144,8 +144,8 @@ More details: :doc:`/installation/installing-from-pypi`
* The ``#user-troubleshooting`` channel on Airflow Slack for quick general
troubleshooting questions. The `GitHub discussions <https://github.com/apache/airflow/discussions>`__
if you look for longer discussion and have more information to share.
-* The ``#user-best-practices`` channel on slack can be used to ask for and share best
- practices on using and deploying airflow.
+* The ``#user-best-practices`` channel on Slack can be used to ask for and share best
+ practices on using and deploying Airflow.
* If you can provide description of a reproducible problem with Airflow software, you can open
issue at `GitHub issues <https://github.com/apache/airflow/issues>`__
@@ -162,7 +162,7 @@ running Airflow components in isolation from other software running on the same
maintenance of dependencies.
The images are built by Apache Airflow release managers and they use officially released packages from PyPI
-and official constraint files- same that are used for installing Airflow from PyPI.
+and official constraint files - same that are used for installing Airflow from PyPI.
**Intended users**
@@ -174,16 +174,16 @@ and official constraint files- same that are used for installing Airflow from Py
* You are expected to be able to customize or extend Container/Docker images if you want to
add extra dependencies. You are expected to put together a deployment built of several containers
- (for example using docker-compose) and to make sure that they are linked together.
-* You are responsible for setting up database, creating and managing database schema with ``airflow db`` commands,
+ (for example using ``docker-compose``) and to make sure that they are linked together.
+* You are responsible for setting up the database, creating and managing database schema with ``airflow db`` commands,
automated startup and recovery, maintenance, cleanup and upgrades of Airflow and the Airflow Providers.
* You are responsible to manage your own customizations and extensions for your custom dependencies.
With the Official Airflow Docker Images, upgrades of Airflow and Airflow Providers which
are part of the reference image are handled by the community - you need to make sure to pick up
- those changes when released by upgrading the base image. However, you are responsible in creating a
+ those changes when released by upgrading the base image. However, you are responsible for creating a
pipeline of building your own custom images with your own added dependencies and Providers and need to
repeat the customization step and building your own image when new version of Airflow image is released.
-* You should choose the right deployment mechanism. There a number of available options of
+* You should choose the right deployment mechanism. There are a number of available options of
deployments of containers. You can use your own custom mechanism, custom Kubernetes deployments,
custom Docker Compose, custom Helm charts etc., and you should choose it based on your experience
and expectations.
@@ -208,8 +208,8 @@ and official constraint files- same that are used for installing Airflow from Py
* The ``#user-troubleshooting`` channel on Airflow Slack for quick general
troubleshooting questions. The `GitHub discussions <https://github.com/apache/airflow/discussions>`__
if you look for longer discussion and have more information to share.
-* The ``#user-best-practices`` channel on slack can be used to ask for and share best
- practices on using and deploying airflow.
+* The ``#user-best-practices`` channel on Slack can be used to ask for and share best
+ practices on using and deploying Airflow.
* If you can provide description of a reproducible problem with Airflow software, you can open
issue at `GitHub issues <https://github.com/apache/airflow/issues>`__
@@ -246,7 +246,7 @@ More details: :doc:`helm-chart:index`
* You are responsible to manage your own customizations and extensions for your custom dependencies.
With the Official Airflow Docker Images, upgrades of Airflow and Airflow Providers which
are part of the reference image are handled by the community - you need to make sure to pick up
- those changes when released by upgrading the base image. However, you are responsible in creating a
+ those changes when released by upgrading the base image. However, you are responsible for creating a
pipeline of building your own custom images with your own added dependencies and Providers and need to
repeat the customization step and building your own image when new version of Airflow image is released.
* You need to setup monitoring of your system allowing you to observe resources and react to problems.
@@ -267,8 +267,8 @@ More details: :doc:`helm-chart:index`
* The ``#user-troubleshooting`` channel on Airflow Slack for quick general
troubleshooting questions. The `GitHub discussions <https://github.com/apache/airflow/discussions>`__
if you look for longer discussion and have more information to share.
-* The ``#user-best-practices`` channel on slack can be used to ask for and share best
- practices on using and deploying airflow.
+* The ``#user-best-practices`` channel on Slack can be used to ask for and share best
+ practices on using and deploying Airflow.
* If you can provide description of a reproducible problem with Airflow software, you can open
issue at `GitHub issues <https://github.com/apache/airflow/issues>`__
@@ -372,15 +372,15 @@ control theory - where there are two types of systems:
2. Complex systems with multiple variables, that are hard to predict and where you need to monitor
the system and adjust the knobs continuously to make sure the system is running smoothly.
-Airflow (and generally any modern system running usually on cloud services, with multiple layers responsible
-for resources as well multiple parameters to control their behaviour) is a complex system and they fall
+Airflow (and generally any modern systems running usually on cloud services, with multiple layers responsible
+for resources as well multiple parameters to control their behaviour) is a complex system and it fall
much more in the second category. If you decide to run Airflow in production on your own, you should be
prepared for the monitor/observe/adjust feedback loop to make sure the system is running smoothly.
Having a good monitoring system that will allow you to monitor the system and adjust the parameters
is a must to put that in practice.
-There are few guidelines that you can use for optimizing your resource usage as well. The
+There are a few guidelines that you can use for optimizing your resource usage as well. The
:ref:`fine-tuning-scheduler` is a good starting point to fine-tune your scheduler, you can also follow
the :ref:`best_practice` guide to make sure you are using Airflow in the most efficient way.
diff --git a/airflow-core/docs/installation/installing-from-pypi.rst b/airflow-core/docs/installation/installing-from-pypi.rst
index 542c7463fde..1a70723842a 100644
--- a/airflow-core/docs/installation/installing-from-pypi.rst
+++ b/airflow-core/docs/installation/installing-from-pypi.rst
@@ -31,16 +31,12 @@ Only ``pip`` installation is currently officially supported.
While there are some successes with using other tools like `poetry <https://python-poetry.org/>`_ or
`pip-tools <https://pypi.org/project/pip-tools/>`_, they do not share the same workflow as
``pip`` - especially when it comes to constraint vs. requirements management.
- Installing via ``Poetry`` or ``pip-tools`` is not currently supported. If you wish to install airflow
+ Installing via ``Poetry`` or ``pip-tools`` is not currently supported. If you wish to install Airflow
using those tools you should use the constraints and convert them to appropriate
format and workflow that your tool requires.
- There are known issues with ``bazel`` that might lead to circular dependencies when using it to install
- Airflow. Please switch to ``pip`` if you encounter such problems. ``Bazel`` community works on fixing
- the problem in `this PR <https://github.com/bazelbuild/rules_python/pull/1166>`_ so it might be that
- newer versions of ``bazel`` will handle it.
-Typical command to install airflow from scratch in a reproducible way from PyPI looks like below:
+Typical command to install Airflow from scratch in a reproducible way from PyPI looks like below:
.. code-block:: bash
@@ -63,12 +59,12 @@ Those are just examples, see further for more explanation why those are the best
.. note::
Generally speaking, Python community established practice is to perform application installation in a
- virtualenv created with ``virtualenv`` or ``venv`` tools. You can also use ``uv`` or ``pipx`` to install
+ virtual environment created with ``virtualenv`` or ``venv`` tools. You can also use ``uv`` or ``pipx`` to install
Airflow in application dedicated virtual environment created for you. There are also other tools that can be used
- to manage your virtualenv installation and you are free to choose how you are managing the environments.
- Airflow has no limitation regarding to the tool of your choice when it comes to virtual environment.
+ to manage your virtual environment installation and you are free to choose how you are managing the environments.
+ Airflow has no limitation regarding the tool of your choice when it comes to virtual environment.
- The only exception where you might consider not using virtualenv is when you are building a container
+ The only exception where you might consider not using virtual environment is when you are building a container
image with only Airflow installed - this is for example how Airflow is installed in the official Container
image.
@@ -84,7 +80,7 @@ Airflow® installation can be tricky because Airflow is both a library and an ap
Libraries usually keep their dependencies open and applications usually pin them, but we should do neither
and both at the same time. We decided to keep our dependencies as open as possible
-(in ``pyproject.toml``) so users can install different version of libraries if needed. This means that
+(in ``pyproject.toml``) so users can install different versions of libraries if needed. This means that
from time to time plain ``pip install apache-airflow`` will not work or will produce an unusable
Airflow installation.
@@ -96,7 +92,7 @@ In order to have a reproducible installation, we also keep a set of constraint f
for each released version e.g. :subst-code:`constraints-|version|`.
This way, we keep a tested set of dependencies at the moment of release. This provides you with the ability
-of having the exact same installation of airflow + providers + dependencies as was known to be working
+of having the exact same installation of Airflow + providers + dependencies as was known to be working
at the moment of release - frozen set of dependencies for that version of Airflow. There is a separate
constraints file for each version of Python that Airflow supports.
@@ -111,7 +107,7 @@ where:
- ``AIRFLOW_VERSION`` - Airflow version (e.g. :subst-code:`|version|`) or ``main``, ``2-0``, for latest development version
- ``PYTHON_VERSION`` Python version e.g. ``3.9``, ``3.10``
-The examples below assume that you want to use install airflow in a reproducible way with the ``celery`` extra,
+The examples below assume that you want to use install Airflow in a reproducible way with the ``celery`` extra,
but you can pick your own set of extras and providers to install.
.. code-block:: bash
@@ -135,7 +131,7 @@ Upgrading and installing dependencies (including providers)
providers and other dependencies to other versions**
You can, for example, install new versions of providers and dependencies after the release to use the latest
-version and up-to-date with latest security fixes - even if you do not want upgrade airflow core version.
+version and up-to-date with latest security fixes - even if you do not want upgrade Airflow core version.
Or you can downgrade some dependencies or providers if you want to keep previous versions for compatibility
reasons. Installing such dependencies should be done without constraints as a separate pip command.
@@ -184,7 +180,7 @@ consistent and not conflicting.
No broken requirements found.
-When you see such message and the exit code from ``pip check`` is 0, you can be sure, that there are no
+When you see such message and the exit code from ``pip check`` is 0, you can be sure that there are no
conflicting dependencies in your environment.
@@ -211,7 +207,7 @@ a local constraints file:
pip install "apache-airflow[celery]==|version|" --constraint "my-constraints.txt"
-Similarly as in case of Airflow original constraints, you can also host your constraints at your own
+Similarly as in the case of Airflow original constraints, you can also host your constraints at your own
repository or server and use it remotely from there.
Fixing Constraints at release time
@@ -219,7 +215,7 @@ Fixing Constraints at release time
The released "versioned" constraints are mostly ``fixed`` when we release Airflow version and we only
update them in exceptional circumstances. For example when we find out that the released constraints might prevent
-Airflow from being installed consistently from the scratch.
+Airflow from being installed consistently from scratch.
In normal circumstances, the constraint files are not going to change if new version of Airflow
dependencies are released - not even when those versions contain critical security fixes.
@@ -260,7 +256,7 @@ providers in case they were released afterwards.
Upgrading Airflow together with providers
=========================================
-You can upgrade airflow together with extras (providers available at the time of the release of Airflow
+You can upgrade Airflow together with extras (providers available at the time of the release of Airflow
being installed. This will bring ``apache-airflow`` and all providers to the versions that were
released and tested together when the version of Airflow you are installing was released.
@@ -291,7 +287,7 @@ Constraints are only effective during the ``pip install`` command they were used
It is the best practice to install apache-airflow in the same version as the one that comes from the
original image. This way you can be sure that ``pip`` will not try to downgrade or upgrade apache
-airflow while installing other requirements, which might happen in case you try to add a dependency
+Airflow while installing other requirements, which might happen in case you try to add a dependency
that conflicts with the version of apache-airflow that you are using:
.. code-block:: bash
@@ -314,11 +310,11 @@ Managing just Airflow core without providers
============================================
If you don't want to install any providers you have, just install or upgrade Apache Airflow, you can simply
-install airflow in the version you need. You can use the special ``constraints-no-providers`` constraints
+install Airflow in the version you need. You can use the special ``constraints-no-providers`` constraints
file, which is smaller and limits the dependencies to the core of Airflow only, however this can lead to
conflicts if your environment already has some of the dependencies installed in different versions and
in case you have other providers installed. This command, however, gives you the latest versions of
-dependencies compatible with just airflow core at the moment Airflow was released.
+dependencies compatible with just Airflow core at the moment Airflow was released.
.. code-block:: bash
:substitutions:
@@ -345,7 +341,7 @@ ensure that ``~/.local/bin`` is in your ``PATH`` environment variable, and add i
PATH=$PATH:~/.local/bin
-You can also start airflow with ``python -m airflow``
+You can also start Airflow with ``python -m airflow``
Symbol not found: ``_Py_GetArgcArgv``
=====================================
diff --git a/airflow-core/docs/installation/installing-from-sources.rst b/airflow-core/docs/installation/installing-from-sources.rst
index 949d949df2a..bafd3df9b33 100644
--- a/airflow-core/docs/installation/installing-from-sources.rst
+++ b/airflow-core/docs/installation/installing-from-sources.rst
@@ -26,18 +26,18 @@ Released packages
This page describes downloading and verifying Airflow® version
``{{ airflow_version }}`` using officially released packages.
You can also install ``Apache Airflow`` - as most Python packages - via :doc:`PyPI <installing-from-pypi>`.
- You can choose different version of Airflow by selecting different version from the drop-down at
+ You can choose different version of Airflow by selecting a different version from the drop-down at
the top-left of the page.
The ``source``, ``sdist`` and ``whl`` packages released are the "official" sources of installation that you
can use if you want to verify the origin of the packages and want to verify checksums and signatures of
the packages. The packages are available via the
-`Official Apache Software Foundations Downloads <https://dlcdn.apache.org/>`_
+`Official Apache Software Foundation Downloads <https://dlcdn.apache.org/>`_
As of version 2.8 Airflow follows PEP 517/518 and uses ``pyproject.toml`` file to define build dependencies
-and build process and it requires relatively modern versions of packaging tools to get airflow built from
+and build process and it requires relatively modern versions of packaging tools to get Airflow built from
local sources or ``sdist`` packages, as PEP 517 compliant build hooks are used to determine dynamic build
-dependencies. In case of ``pip`` it means that at least version 22.1.0 is needed (released at the beginning of
+dependencies. In case of ``pip``, it means that at least version 22.1.0 is needed (released at the beginning of
2022) to build or install Airflow from sources. This does not affect the ability of installing Airflow from
released wheel packages.
@@ -116,7 +116,7 @@ Example:
The "Good signature from ..." is indication that the signatures are correct.
Do not worry about the "not certified with a trusted signature" warning. Most of the certificates used
-by release managers are self signed, that's why you get this warning. By importing the server in the
+by release managers are self-signed, that's why you get this warning. By importing the server in the
previous step and importing it via ID from ``KEYS`` page, you know that this is a valid Key already.
For SHA512 sum check, download the relevant ``sha512`` and run the following:
@@ -159,5 +159,5 @@ and SHA sum files with the script below:
ls -la "${airflow_download_dir}"
echo
-Once you verify the files following the instructions from previous chapter you can remove the temporary
+Once you verify the files following the instructions from previous section, you can remove the temporary
folder created.
diff --git a/airflow-core/docs/installation/prerequisites.rst b/airflow-core/docs/installation/prerequisites.rst
index 3342b63bf15..323af74f8e5 100644
--- a/airflow-core/docs/installation/prerequisites.rst
+++ b/airflow-core/docs/installation/prerequisites.rst
@@ -30,8 +30,7 @@ Airflow® is tested with:
* Kubernetes: 1.26, 1.27, 1.28, 1.29, 1.30
-The minimum memory required we recommend Airflow to run with is 4GB, but the actual requirements depend
-wildly on the deployment options you have
+While we recommend a minimum of 4GB of memory for Airflow, the actual requirements heavily depend on your chosen deployment.
.. warning::
@@ -43,17 +42,17 @@ wildly on the deployment options you have
because the number of users who tried to use MariaDB for Airflow is very small.
.. warning::
- SQLite is used in Airflow tests. Do not use it in production. We recommend
+ SQLite is used in Airflow tests. DO NOT use it in production. We recommend
using the latest stable version of SQLite for local development.
.. warning::
Airflow® currently can be run on POSIX-compliant Operating Systems. For development it is regularly
- tested on fairly modern Linux Distros that our contributors use and recent versions of MacOS.
+ tested on fairly modern Linux distributions that our contributors use and recent versions of MacOS.
On Windows you can run it via WSL2 (Windows Subsystem for Linux 2) or via Linux Containers.
The work to add Windows support is tracked via `#10388 <https://github.com/apache/airflow/issues/10388>`__
- but it is not a high priority. You should only use Linux-based distros as "Production" execution environment
- as this is the only environment that is supported. The only distro that is used in our CI tests and that
+ but it is not a high priority. You should only use Linux-based distributions as "Production environment"
+ as this is the only environment that is supported. The only distribution that is used in our CI tests and that
is used in the `Community managed DockerHub image <https://hub.docker.com/p/apache/airflow>`__ is
``Debian Bookworm``.
diff --git a/airflow-core/docs/installation/supported-versions.rst b/airflow-core/docs/installation/supported-versions.rst
index 8bc8da06032..b1063453cc2 100644
--- a/airflow-core/docs/installation/supported-versions.rst
+++ b/airflow-core/docs/installation/supported-versions.rst
@@ -40,28 +40,28 @@ Version Current Patch/Minor State First Release Limited Support
.. End of auto-generated table
-Limited support versions will be supported with security and critical bug fix only.
-EOL versions will not get any fixes nor support.
+Limited support versions will be supported with security and critical bug fixes only.
+EOL versions will not get any fixes or support.
We **highly** recommend installing the latest Airflow release which has richer features.
Support for Python and Kubernetes versions
``````````````````````````````````````````
-As of Airflow 2.0 we agreed to certain rules we follow for Python and Kubernetes support.
+For Airflow 2.0+ versions, we agreed to certain rules we follow for Python and Kubernetes support.
They are based on the official release schedule of Python and Kubernetes, nicely summarized in the
`Python Developer's Guide <https://devguide.python.org/#status-of-python-branches>`_ and
`Kubernetes version skew policy <https://kubernetes.io/docs/setup/release/version-skew-policy>`_.
1. We drop support for Python and Kubernetes versions when they reach EOL. We drop support for those
- EOL versions in main right after EOL date, and it is effectively removed when we release the
+ EOL versions in main right after the EOL date, and it is effectively removed when we release the
first new MINOR (Or MAJOR if there is no new MINOR version) of Airflow
For example for Python 3.6 it means that we drop support in main right after 23.12.2021, and the first
MAJOR or MINOR version of Airflow released after will not have it.
2. The "oldest" supported version of Python/Kubernetes is the default one. "Default" is only meaningful
in terms of "smoke tests" in CI PRs which are run using this default version and default reference
- image available in DockerHub. Currently the ``apache/airflow:latest`` and ``apache/airflow:2.10.2`` images
+ image available in Docker Hub. Currently the ``apache/airflow:latest`` and ``apache/airflow:2.10.2`` images
are Python 3.8 images, however, in the first MINOR/MAJOR release of Airflow released after 2024-10-14,
they will become Python 3.9 images.
diff --git a/airflow-core/docs/installation/upgrading_to_airflow3.rst b/airflow-core/docs/installation/upgrading_to_airflow3.rst
index 355ae58a9d3..208f4ba5cd4 100644
--- a/airflow-core/docs/installation/upgrading_to_airflow3.rst
+++ b/airflow-core/docs/installation/upgrading_to_airflow3.rst
@@ -33,7 +33,7 @@ Step 2: Clean and back up your existing Airflow Instance
- It is highly recommended to make a backup of your Airflow instance specifically including your Airflow metadata DB before starting the migration process.
- If you do not have a "hot backup" capability for your DB, you should do it after shutting down your Airflow instances, so that the backup of your database will be consistent.
-- If you did not make a backup and your migration fails, you might end-up in a half-migrated state and restoring DB from backup and repeating the migration
+- If you did not make a backup and your migration fails, you might end up in a half-migrated state and restoring DB from backup and repeating the migration
might be the only easy way out. This can for example be caused by a broken network connection between your CLI and the database while the migration happens, so taking a
backup is an important precaution to avoid problems like this.
- A long running Airflow instance can accumulate a certain amount of silt, in the form of old database entries, which are no longer
@@ -134,7 +134,7 @@ These include:
- **Sequential Executor**: Replaced by LocalExecutor, which can be used with SQLite for local development use cases.
- **SLAs**: Deprecated and removed; Will be replaced by forthcoming `Deadline Alerts <https://cwiki.apache.org/confluence/x/tglIEw>`_.
- **Subdir**: Used as an argument on many CLI commands (``--subdir`` or ``-S`` has been superseded by DAG bundles.
-- **Following keys are no longer available in task context. If not replaced, will cause DAG errors:**:
+- **Following keys are no longer available in task context. If not replaced, will cause DAG errors**:
- ``tomorrow_ds``
- ``tomorrow_ds_nodash``
diff --git a/airflow-core/docs/templates-ref.rst b/airflow-core/docs/templates-ref.rst
index ccd2d08c889..fe292501a58 100644
--- a/airflow-core/docs/templates-ref.rst
+++ b/airflow-core/docs/templates-ref.rst
@@ -186,7 +186,7 @@ Variable Description
``macros.random`` The standard lib's :class:`random.random`
================================= ==============================================
-Some airflow specific macros are also defined:
+Some Airflow specific macros are also defined:
.. automodule:: airflow.macros
:members:
(airflow) 08/17: Fix broken link for CeleryKubernetesExecutor upgrade (#49391)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v3-0-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit ead8990c2ef0546be0346bdea02c354ba53cd477
Author: Amogh Desai <am...@gmail.com>
AuthorDate: Thu Apr 17 15:21:11 2025 +0530
Fix broken link for CeleryKubernetesExecutor upgrade (#49391)
(cherry picked from commit 8761bf895d3e6896a018ba4f2ff740c9fd43e07c)
---
.../docs/administration-and-deployment/production-deployment.rst | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/airflow-core/docs/administration-and-deployment/production-deployment.rst b/airflow-core/docs/administration-and-deployment/production-deployment.rst
index 86558091ed6..be16876d6cc 100644
--- a/airflow-core/docs/administration-and-deployment/production-deployment.rst
+++ b/airflow-core/docs/administration-and-deployment/production-deployment.rst
@@ -179,7 +179,7 @@ of the executor you use:
are managed by the Kubernetes cluster and will be automatically adopted by ``Schedulers`` when they are
upgraded and restarted.
-* For the :doc:``CeleryKubernetesExecutor <apache-airflow-providers-celery:celery_kubernetes_executor>``, you follow the
+* For the :doc:`CeleryKubernetesExecutor <apache-airflow-providers-celery:celery_kubernetes_executor>`, you follow the
same procedure as for the ``CeleryExecutor`` - you put the workers in offline mode, wait for the running
tasks to complete, upgrade the workers, and then upgrade the scheduler, triggerer and webserver in a
rolling restart mode - which should also adopt tasks run via the ``KubernetesExecutor`` part of the
(airflow) 07/17: Use EmailOperator from smtp provider (#49386)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v3-0-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit db565d28367d1acb216cba222fcb9ea156aba61f
Author: Amogh Desai <am...@gmail.com>
AuthorDate: Thu Apr 17 15:20:25 2025 +0530
Use EmailOperator from smtp provider (#49386)
(cherry picked from commit 071bf3f4f3bf1ec88b3d0b5140fe6e81e5edc79a)
---
airflow-core/docs/core-concepts/taskflow.rst | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/airflow-core/docs/core-concepts/taskflow.rst b/airflow-core/docs/core-concepts/taskflow.rst
index 73150735bd4..29e91eb4413 100644
--- a/airflow-core/docs/core-concepts/taskflow.rst
+++ b/airflow-core/docs/core-concepts/taskflow.rst
@@ -25,7 +25,7 @@ If you write most of your dags using plain Python code rather than Operators, th
TaskFlow takes care of moving inputs and outputs between your Tasks using XComs for you, as well as automatically calculating dependencies - when you call a TaskFlow function in your DAG file, rather than executing it, you will get an object representing the XCom for the result (an ``XComArg``), that you can then use as inputs to downstream tasks or operators. For example::
from airflow.sdk import task
- from airflow.providers.email import EmailOperator
+ from airflow.providers.smtp.operators.smtp import EmailOperator
@task
def get_ip():
(airflow) 02/17: Remove common.messaging from chicken-egg-providers (#49369)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v3-0-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 23505f246140e507845f92d98a5ace20dcb60e4b
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Wed Apr 16 20:11:47 2025 +0200
Remove common.messaging from chicken-egg-providers (#49369)
(cherry picked from commit d92fc06d21ef66958d3c7041fd33b48b3963c60c)
---
dev/breeze/src/airflow_breeze/global_constants.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/dev/breeze/src/airflow_breeze/global_constants.py b/dev/breeze/src/airflow_breeze/global_constants.py
index ccb56e1a11b..18084867964 100644
--- a/dev/breeze/src/airflow_breeze/global_constants.py
+++ b/dev/breeze/src/airflow_breeze/global_constants.py
@@ -635,7 +635,7 @@ DEFAULT_EXTRAS = [
# END OF EXTRAS LIST UPDATED BY PRE COMMIT
]
-CHICKEN_EGG_PROVIDERS = " ".join(["git", "common.messaging", "fab", "standard", "openlineage"])
+CHICKEN_EGG_PROVIDERS = " ".join(["git", "fab", "standard", "openlineage"])
PROVIDERS_COMPATIBILITY_TESTS_MATRIX: list[dict[str, str | list[str]]] = [
(airflow) 01/17: Reset error state after successfully adding a connection (#49364)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v3-0-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 326c65cd82db82c0e8292836055a44d263b851c7
Author: Jason <46...@users.noreply.github.com>
AuthorDate: Wed Apr 16 23:48:52 2025 +0700
Reset error state after successfully adding a connection (#49364)
(cherry picked from commit 3c063b157076e24e184f63205d07e7032f883ffb)
---
airflow-core/src/airflow/ui/src/queries/useAddConnection.ts | 2 ++
1 file changed, 2 insertions(+)
diff --git a/airflow-core/src/airflow/ui/src/queries/useAddConnection.ts b/airflow-core/src/airflow/ui/src/queries/useAddConnection.ts
index 626dd10afd3..24aa2715883 100644
--- a/airflow-core/src/airflow/ui/src/queries/useAddConnection.ts
+++ b/airflow-core/src/airflow/ui/src/queries/useAddConnection.ts
@@ -37,6 +37,8 @@ export const useAddConnection = ({ onSuccessConfirm }: { onSuccessConfirm: () =>
title: "Connection Add Request Submitted",
type: "success",
});
+
+ setError(undefined);
onSuccessConfirm();
};
(airflow) 05/17: Set rc0 as suffix used for chicken-egg-providers (#49387)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v3-0-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 25c5d62baa724acd0fc7572f8a3630755f8b7b21
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Thu Apr 17 10:51:15 2025 +0200
Set rc0 as suffix used for chicken-egg-providers (#49387)
Learning from yesterday frantic attempts to relase Airflow 3.0.0rc3
with some providers that were changed mid-flight, this is the last
step to make both regular release in main (with dev0) and the
dockerhub release (with rc0) works.
The way how it works:
* after #49373 we are flooring >= dependencies for all airflow
distributions to >= XX0 (where XX might be dev or rc)
* we are building all chicken-egg-providers with rc0 suffix now.
for example fab 2.0.1 will be locally built as 2.0.1rc0, git
0.0.2 will be built as 0.0.2rc0. Both will have apache-airflow
>= 3.0.0rc0 as dependency. Similarly common.messaging that is
already released as 1.0.0 will be locally build as 1.0.0rc0 as
it is still chicken-egg-provider until we release Airflow 3.0.0.
if it is already released as 1.0.0rc0, we will skip building it
locally and use PyPI version.
This should work in all cases:
* when we already released final version (common.messaging), we
are going to have locally built common.messaging 1.0.0rc0 which
will have apache-airflow>=3.0.0rc0. The 1.0.0 of common.messaging
from PyPI cannot be used because it has apache-airflow>=3.0.0.
Airlfow 3.0.0rc0 wil be built with common.messaging >= 1.0.0rc0
so all requirements will be satisfied.
* when we have not yet released chicken-egg-provider such as git 0.0.2)
we will locally build 0.0.2rc0 (unless it is already released) and
use this one locally. Airflow will use git >= 0.0.2rc0 if min version
is set to 0.0.2.
* In case such provider (git 0.0.2) has been released already as rc0, it
will be skipped automatically when building it and we will use `PyPI`
version instead
(cherry picked from commit f0ec450c8eb6adbe0124ad4e8fa17dcaea81018d)
---
.github/workflows/release_dockerhub_image.yml | 12 +++---------
1 file changed, 3 insertions(+), 9 deletions(-)
diff --git a/.github/workflows/release_dockerhub_image.yml b/.github/workflows/release_dockerhub_image.yml
index 7f83b7274c6..0afeb8e143e 100644
--- a/.github/workflows/release_dockerhub_image.yml
+++ b/.github/workflows/release_dockerhub_image.yml
@@ -27,11 +27,6 @@ on: # yamllint disable-line rule:truthy
description: 'Skip Latest: Set to true if not latest.'
default: ''
required: false
- includePreRelease:
- description: 'Include pre-release: Set to true to include pre-releases.'
- default: ''
- type: boolean
- required: false
permissions:
contents: read
packages: read
@@ -52,7 +47,6 @@ jobs:
defaultPythonVersion: ${{ steps.selective-checks.outputs.default-python-version }}
chicken-egg-providers: ${{ steps.selective-checks.outputs.chicken-egg-providers }}
skipLatest: ${{ github.event.inputs.skipLatest == '' && ' ' || '--skip-latest' }}
- includePreRelease: ${{ github.event.inputs.includePreRelease == '' && ' ' || '--include-pre-release' }}
limitPlatform: ${{ github.repository == 'apache/airflow' && ' ' || '--limit-platform linux/amd64' }}
env:
GITHUB_CONTEXT: ${{ toJson(github) }}
@@ -151,16 +145,16 @@ jobs:
- name: "Create airflow_cache builder"
run: docker buildx create --name airflow_cache
- name: "Prepare chicken-eggs provider distributions"
- # In case of provider distributions which use latest dev0 version of providers, we should prepare them
+ # In case of provider distributions which use latest r00 version of providers, we should prepare them
# from the source code, not from the PyPI because they have apache-airflow>=X.Y.Z dependency
- # And when we prepare them from sources they will have apache-airflow>=X.Y.Z.dev0
+ # And when we prepare them from sources they will have apache-airflow>=X.Y.Z.rc0
shell: bash
env:
CHICKEN_EGG_PROVIDERS: ${{ needs.build-info.outputs.chicken-egg-providers }}
run: >
breeze release-management prepare-provider-distributions
--distribution-format wheel
- --version-suffix-for-pypi dev0 ${CHICKEN_EGG_PROVIDERS}
+ --version-suffix-for-pypi rc0 ${CHICKEN_EGG_PROVIDERS}
if: needs.build-info.outputs.chicken-egg-providers != ''
- name: "Copy dist packages to docker-context files"
shell: bash
(airflow) 04/17: Floor version suffix when referring to other airflow pre-releases (#49373)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v3-0-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 6b335f386bb655e65484e1314a4416440707339b
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Wed Apr 16 23:27:39 2025 +0200
Floor version suffix when referring to other airflow pre-releases (#49373)
When we are referring to other airflow distribution packages with
pre-release suffix, when we have >= comparision we should always
floor the suffix version (i.e. if we are preparing rc3, we should
have >= ...rc0 ) - this should cover the edge case (like in case of
common.messaging) where we have already released final version, and
we are not releasing a new RC. This allows to install the latest
RC candidate of such already released provider.
We were already doing it when preparing providers, in order to allow
providers to depend on each other when released together, but we
also add the rc suffix also in airflow packages and there we
did not floor the pre-release version.
(cherry picked from commit bdf412412398837fd41562459a79726225b48960)
---
.../src/airflow_breeze/commands/release_management_commands.py | 6 ++++--
dev/breeze/src/airflow_breeze/global_constants.py | 2 +-
dev/breeze/src/airflow_breeze/utils/packages.py | 10 ++++++----
3 files changed, 11 insertions(+), 7 deletions(-)
diff --git a/dev/breeze/src/airflow_breeze/commands/release_management_commands.py b/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
index 2c638f03a0e..a91ceae0f65 100644
--- a/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
@@ -127,6 +127,7 @@ from airflow_breeze.utils.packages import (
PackageSuspendedException,
expand_all_provider_distributions,
find_matching_long_package_names,
+ floor_version_suffix,
get_available_distributions,
get_provider_details,
get_provider_distributions_metadata,
@@ -421,8 +422,9 @@ def update_version_suffix_in_pyproject_toml(version_suffix: str, pyproject_toml_
if not line.endswith(
f'.{version_suffix}"',
):
- get_console().print(f"[info]Updating version suffix to {version_suffix} for {line}.")
- line = line.rstrip('",') + f'.{version_suffix}",'
+ floored_version_suffix = floor_version_suffix(version_suffix)
+ get_console().print(f"[info]Updating version suffix to {floored_version_suffix} for {line}.")
+ line = line.rstrip('",') + f'.{floored_version_suffix}",'
else:
get_console().print(
f"[info]Not updating version suffix to {version_suffix} for {line} as it already has the "
diff --git a/dev/breeze/src/airflow_breeze/global_constants.py b/dev/breeze/src/airflow_breeze/global_constants.py
index 18084867964..ccb56e1a11b 100644
--- a/dev/breeze/src/airflow_breeze/global_constants.py
+++ b/dev/breeze/src/airflow_breeze/global_constants.py
@@ -635,7 +635,7 @@ DEFAULT_EXTRAS = [
# END OF EXTRAS LIST UPDATED BY PRE COMMIT
]
-CHICKEN_EGG_PROVIDERS = " ".join(["git", "fab", "standard", "openlineage"])
+CHICKEN_EGG_PROVIDERS = " ".join(["git", "common.messaging", "fab", "standard", "openlineage"])
PROVIDERS_COMPATIBILITY_TESTS_MATRIX: list[dict[str, str | list[str]]] = [
diff --git a/dev/breeze/src/airflow_breeze/utils/packages.py b/dev/breeze/src/airflow_breeze/utils/packages.py
index 0a34be721f8..296bd5e4591 100644
--- a/dev/breeze/src/airflow_breeze/utils/packages.py
+++ b/dev/breeze/src/airflow_breeze/utils/packages.py
@@ -471,6 +471,11 @@ def get_dist_package_name_prefix(provider_id: str) -> str:
return "apache_airflow_providers_" + provider_id.replace(".", "_")
+def floor_version_suffix(version_suffix: str) -> str:
+ # always use `pre-release`+ `0` as the version suffix
+ return version_suffix.rstrip("0123456789") + "0"
+
+
def apply_version_suffix(install_clause: str, version_suffix: str) -> str:
# Need to resolve a version suffix based on PyPi versions, but can ignore local version suffix.
pypi_version_suffix = remove_local_version_suffix(version_suffix)
@@ -492,10 +497,7 @@ def apply_version_suffix(install_clause: str, version_suffix: str) -> str:
from packaging.version import Version
base_version = Version(version).base_version
- # always use `pre-release`+ `0` as the version suffix
- pypi_version_suffix = pypi_version_suffix.rstrip("0123456789") + "0"
-
- target_version = Version(str(base_version) + "." + pypi_version_suffix)
+ target_version = Version(str(base_version) + "." + floor_version_suffix(pypi_version_suffix))
return prefix + ">=" + str(target_version)
return install_clause
(airflow) 14/17: Add note in FAB migration doc (#49423)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v3-0-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit fa72d055b80a0ad77ceea0f96e2c1789e48117e1
Author: Vincent <97...@users.noreply.github.com>
AuthorDate: Fri Apr 18 08:03:23 2025 -0400
Add note in FAB migration doc (#49423)
(cherry picked from commit f65eeb05f48d9ad67cdbbeb81a5db085c519aaba)
---
airflow-core/src/airflow/config_templates/config.yml | 3 ++-
providers/fab/docs/upgrading.rst | 4 ++++
2 files changed, 6 insertions(+), 1 deletion(-)
diff --git a/airflow-core/src/airflow/config_templates/config.yml b/airflow-core/src/airflow/config_templates/config.yml
index 1b71e5ae019..61641618393 100644
--- a/airflow-core/src/airflow/config_templates/config.yml
+++ b/airflow-core/src/airflow/config_templates/config.yml
@@ -697,7 +697,8 @@ database:
external_db_managers:
description: |
List of DB managers to use to migrate external tables in airflow database. The managers must inherit
- from BaseDBManager
+ from BaseDBManager. If ``FabAuthManager`` is configured in the environment,
+ ``airflow.providers.fab.auth_manager.models.db.FABDBManager`` is automatically added.
version_added: 3.0.0
type: string
example: "airflow.providers.fab.auth_manager.models.db.FABDBManager"
diff --git a/providers/fab/docs/upgrading.rst b/providers/fab/docs/upgrading.rst
index 28c046d9c03..3e861fb1a7e 100644
--- a/providers/fab/docs/upgrading.rst
+++ b/providers/fab/docs/upgrading.rst
@@ -28,6 +28,10 @@ must run ``airflow fab-db migrate`` to migrate your database with the schema cha
upgrading to. If ``FABDBManager`` is included in the ``[core] external_db_managers`` configuration, the migrations will
be run automatically as part of the ``airflow db migrate`` command.
+.. note::
+ If FAB auth manager is configured as auth manager in your environment, ``FABDBManager`` is automatically added in the
+ ``[core] external_db_managers`` configuration.
+
How to upgrade
==============
To upgrade the FAB provider, you need to install the new version of the package. You can do this using ``pip``.
(airflow) 13/17: docs: remove paragraph related to Bazel in quickstart (#49430)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v3-0-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 10113f00c134de557bdeee2facc00a54ff027f45
Author: Kalyan R <ka...@live.com>
AuthorDate: Fri Apr 18 16:12:38 2025 +0530
docs: remove paragraph related to Bazel in quickstart (#49430)
* minor docs updates in quickstart
* Update airflow-core/docs/start.rst
Co-authored-by: LIU ZHE YOU <68...@users.noreply.github.com>
---------
Co-authored-by: LIU ZHE YOU <68...@users.noreply.github.com>
(cherry picked from commit e524041b8011956ebfafe660c8db8068c4d94349)
---
airflow-core/docs/start.rst | 13 ++++---------
1 file changed, 4 insertions(+), 9 deletions(-)
diff --git a/airflow-core/docs/start.rst b/airflow-core/docs/start.rst
index 9986b575a27..81b57ea3b38 100644
--- a/airflow-core/docs/start.rst
+++ b/airflow-core/docs/start.rst
@@ -35,11 +35,6 @@ This quick start guide will help you bootstrap an Airflow standalone instance on
``pip`` or ``uv`` - especially when it comes to constraint vs. requirements management.
Installing via ``Poetry`` or ``pip-tools`` is not currently supported.
- There are known issues with ``bazel`` that might lead to circular dependencies when using it to install
- Airflow. Please switch to ``pip`` or ``uv`` if you encounter such problems. ``Bazel`` community works on fixing
- the problem in `this PR <https://github.com/bazelbuild/rules_python/pull/1166>`_ so it might be that
- newer versions of ``bazel`` will handle it.
-
If you wish to install Airflow using those tools you should use the constraint files and convert
them to appropriate format and workflow that your tool requires.
@@ -61,8 +56,8 @@ This quick start guide will help you bootstrap an Airflow standalone instance on
Install uv: `uv Installation Guide <https://docs.astral.sh/uv/getting-started/installation/>`_
- For creating Virtualenv with uv, refer to the documentation here:
- `Creating and Maintaining Local Virtualenv with uv <https://github.com/apache/airflow/blob/main/contributing-docs/07_local_virtualenv.rst#creating-and-maintaining-local-virtualenv-with-uv>`_
+ For creating virtual environment with ``uv``, refer to the documentation here:
+ `Creating and Maintaining Local virtual environment with uv <https://github.com/apache/airflow/blob/main/contributing-docs/07_local_virtualenv.rst#creating-and-maintaining-local-virtualenv-with-uv-recommended>`_
3. Install Airflow using the constraints file, which is determined based on the URL we pass:
@@ -71,14 +66,14 @@ This quick start guide will help you bootstrap an Airflow standalone instance on
:substitutions:
- AIRFLOW_VERSION=2.10.5
+ AIRFLOW_VERSION=3.0.0
# Extract the version of Python you have installed. If you're currently using a Python version that is not supported by Airflow, you may want to set this manually.
# See above for supported versions.
PYTHON_VERSION="$(python -c 'import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")')"
CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"
- # For example this would install 2.10.5 with python 3.9: https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.9.txt
+ # For example this would install 3.0.0 with python 3.9: https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.9.txt
uv pip install "apache-airflow==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"
(airflow) 06/17: Fix dockerfile generation for PMC installation check (#49435)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v3-0-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit dad2192f9dc95caf07ab29150e2d87e82ab580e9
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Fri Apr 18 14:00:27 2025 +0200
Fix dockerfile generation for PMC installation check (#49435)
The dockerfile generated by check_files.py did not work with the
new `uv` tooling approach - this one fixes it.
(cherry picked from commit 973410752317c71e74c2001bff0c8846625f8e81)
---
dev/check_files.py | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/dev/check_files.py b/dev/check_files.py
index ebca9d6ff8c..042546a19eb 100644
--- a/dev/check_files.py
+++ b/dev/check_files.py
@@ -24,9 +24,8 @@ import rich_click as click
from rich import print
PROVIDERS_DOCKER = """\
-FROM ghcr.io/apache/airflow/main/ci/python3.10
-RUN rm -rf /opt/airflow/airflow/providers
-
+FROM ghcr.io/apache/airflow/main/ci/python3.9
+RUN cd airflow-core; uv sync --no-sources
# Install providers
{}
@@ -206,7 +205,9 @@ def providers(ctx, path: str):
files = os.listdir(os.path.join(path, "providers"))
pips = [f"{name}=={version}" for name, version in get_packages()]
missing_files = check_providers(files)
- create_docker(PROVIDERS_DOCKER.format("RUN uv pip install --system " + " ".join(f"'{p}'" for p in pips)))
+ create_docker(
+ PROVIDERS_DOCKER.format("RUN uv pip install --pre --system " + " ".join(f"'{p}'" for p in pips))
+ )
if missing_files:
warn_of_missing_files(missing_files)