You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by ka...@apache.org on 2025/07/07 11:06:12 UTC
(airflow) branch v3-0-test updated (35952960033 -> 6062f68055b)
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a change to branch v3-0-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
discard 35952960033 [v3-0-test] Make the update_installer_and_pre_commit script more resilient to re matches (#52960) (#52961)
discard 25a75791bd8 [v3-0-test] Workaround the need to use latest version on Lucas pre-commit's (#52931) (#52942)
discard 0d5d530153d [v3-0-test] Advanced auto classification for provider documentation (#52902) (#52925)
discard a3898075060 [Backport v3-0-test] Add back dag parsing pre-import optimization (#50371) (#52698)
new 989295aa07e Update release notes for 3.0.3rc4
new b929b4e036b [Backport v3-0-test] Add back dag parsing pre-import optimization (#50371) (#52698)
new a6d1a5b04bd [v3-0-test] Advanced auto classification for provider documentation (#52902) (#52925)
new 05f11fff986 [v3-0-test] Workaround the need to use latest version on Lucas pre-commit's (#52931) (#52942)
new 6062f68055b [v3-0-test] Make the update_installer_and_pre_commit script more resilient to re matches (#52960) (#52961)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (35952960033)
\
N -- N -- N refs/heads/v3-0-test (6062f68055b)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 5 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
RELEASE_NOTES.rst | 3 ++-
reproducible_build.yaml | 4 ++--
2 files changed, 4 insertions(+), 3 deletions(-)
(airflow) 04/05: [v3-0-test] Workaround the need to use latest version on Lucas pre-commit's (#52931) (#52942)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v3-0-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 05f11fff986831b9d861287b478a404cfb2996d0
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Sun Jul 6 21:13:35 2025 +0200
[v3-0-test] Workaround the need to use latest version on Lucas pre-commit's (#52931) (#52942)
We are still waiting for release of the Lucas's pre-commit after
the https://github.com/Lucas-C/pre-commit-hooks/pull/103 has
been merged - and for now we need to use `bleeding-edge` for
the repo.
Also upgrades zizmor and solves using "env." in shell commands
that might lead to security issues.
(cherry picked from commit 535d71d91f36d3d9859d32e4695f0e81a2696a4b)
---
.github/actions/migration_tests/action.yml | 16 ++++++++--------
.github/workflows/basic-tests.yml | 2 ++
.pre-commit-config.yaml | 5 +++--
3 files changed, 13 insertions(+), 10 deletions(-)
diff --git a/.github/actions/migration_tests/action.yml b/.github/actions/migration_tests/action.yml
index 0198cbf472b..34951bac5ff 100644
--- a/.github/actions/migration_tests/action.yml
+++ b/.github/actions/migration_tests/action.yml
@@ -24,9 +24,9 @@ runs:
- name: "Test migration file 2 to 3 migration: ${{env.BACKEND}}"
shell: bash
run: |
- breeze shell "${{ env.AIRFLOW_2_CMD }}" --use-airflow-version 2.11.0 --answer y &&
- breeze shell "export AIRFLOW__DATABASE__EXTERNAL_DB_MANAGERS=${{env.DB_MANGERS}}
- ${{ env.AIRFLOW_3_CMD }}" --no-db-cleanup
+ breeze shell "${AIRFLOW_2_CMD}" --use-airflow-version 2.11.0 --answer y &&
+ breeze shell "export AIRFLOW__DATABASE__EXTERNAL_DB_MANAGERS=${DB_MANGERS}
+ ${AIRFLOW_3_CMD}" --no-db-cleanup
env:
COMPOSE_PROJECT_NAME: "docker-compose"
DB_RESET: "false"
@@ -47,9 +47,9 @@ runs:
- name: "Test ORM migration 2 to 3: ${{env.BACKEND}}"
shell: bash
run: >
- breeze shell "${{ env.AIRFLOW_2_CMD }}" --use-airflow-version 2.11.0 --answer y &&
- breeze shell "export AIRFLOW__DATABASE__EXTERNAL_DB_MANAGERS=${{env.DB_MANGERS}}
- ${{ env.AIRFLOW_3_CMD }}" --no-db-cleanup
+ breeze shell "${AIRFLOW_2_CMD}" --use-airflow-version 2.11.0 --answer y &&
+ breeze shell "export AIRFLOW__DATABASE__EXTERNAL_DB_MANAGERS=${DB_MANGERS}
+ ${AIRFLOW_3_CMD}" --no-db-cleanup
env:
COMPOSE_PROJECT_NAME: "docker-compose"
DB_RESET: "false"
@@ -69,7 +69,7 @@ runs:
- name: "Test ORM migration ${{env.BACKEND}}"
shell: bash
run: >
- breeze shell "export AIRFLOW__DATABASE__EXTERNAL_DB_MANAGERS=${{env.DB_MANAGERS}} &&
+ breeze shell "export AIRFLOW__DATABASE__EXTERNAL_DB_MANAGERS=${DB_MANAGERS} &&
airflow db reset -y &&
airflow db migrate --to-revision heads &&
airflow db downgrade -n 2.7.0 -y &&
@@ -86,7 +86,7 @@ runs:
shell: bash
run: >
breeze shell
- "export AIRFLOW__DATABASE__EXTERNAL_DB_MANAGERS=${{env.DB_MANAGERS}} &&
+ "export AIRFLOW__DATABASE__EXTERNAL_DB_MANAGERS=${DB_MANAGERS} &&
airflow db reset -y &&
airflow db downgrade -n 2.7.0 -y &&
airflow db migrate -s"
diff --git a/.github/workflows/basic-tests.yml b/.github/workflows/basic-tests.yml
index 59a326d49b1..bc23674f442 100644
--- a/.github/workflows/basic-tests.yml
+++ b/.github/workflows/basic-tests.yml
@@ -267,6 +267,8 @@ jobs:
skip-pre-commits: ${{ inputs.skip-pre-commits }}
- name: "Autoupdate all pre-commits"
run: pre-commit autoupdate
+ - name: "Autoupdate Lucas-C/pre-commit-hooks to bleeding edge"
+ run: pre-commit autoupdate --bleeding-edge --freeze --repo https://github.com/Lucas-C/pre-commit-hooks
- name: "Run automated upgrade for black"
run: >
pre-commit run
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index 3dc226a64df..3adeefd79b6 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -49,7 +49,8 @@ repos:
- "--maxlevel"
- "2"
- repo: https://github.com/Lucas-C/pre-commit-hooks
- rev: v1.5.5
+ # replace hash with version once PR #103 merged comes in a release
+ rev: abdd8b62891099da34162217ecb3872d22184a51
hooks:
- id: insert-license
name: Add license for all SQL files
@@ -356,7 +357,7 @@ repos:
- --skip=providers/.*/src/airflow/providers/*/*.rst,providers/*/docs/changelog.rst,docs/*/commits.rst,providers/*/docs/commits.rst,providers/*/*/docs/commits.rst,docs/apache-airflow/tutorial/pipeline_example.csv,*.min.js,*.lock,INTHEWILD.md,*.svg
- --exclude-file=.codespellignorelines
- repo: https://github.com/woodruffw/zizmor-pre-commit
- rev: v1.7.0
+ rev: v1.11.0
hooks:
- id: zizmor
name: Run zizmor to check for github workflow syntax errors
(airflow) 01/05: Update release notes for 3.0.3rc4
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v3-0-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 989295aa07ecc90b63cab3191a9148d796f6ade8
Author: Kaxil Naik <ka...@apache.org>
AuthorDate: Mon Jul 7 16:35:31 2025 +0530
Update release notes for 3.0.3rc4
---
RELEASE_NOTES.rst | 3 ++-
reproducible_build.yaml | 4 ++--
2 files changed, 4 insertions(+), 3 deletions(-)
diff --git a/RELEASE_NOTES.rst b/RELEASE_NOTES.rst
index 006a3861350..6554068974c 100644
--- a/RELEASE_NOTES.rst
+++ b/RELEASE_NOTES.rst
@@ -67,7 +67,7 @@ Bug Fixes
- UI: Enable iframe script execution (#52568)
- UI: Fix DAG tags filter not showing all tags in UI when tags are greater than 50 (#52714)
- UI: Add real-time clock updates to timezone selector (#52414)
-- Improve Grid view performance and responsiveness with optimized data loading (#52718)(#52822))
+- Improve Grid view performance and responsiveness with optimized data loading (#52718,#52822,#52919)
- Fix editing connection with sensitive extra field (#52445)
- Fix archival for cascading deletes by archiving dependent tables first (#51952)
- Fix whitespace handling in DAG owners parsing for multiple owners (#52221)
@@ -79,6 +79,7 @@ Bug Fixes
- Ensure trigger kwargs are properly deserialized during trigger execution (#52721)
- Fixing bad cadwyn migration for upstream map indexes (#52797)
- Run trigger expansion logic only when ``start_from_trigger`` is True (#52873)
+- Fix example dag ``example_external_task_parent_deferrable.py`` imports (#52957)
Miscellaneous
"""""""""""""
diff --git a/reproducible_build.yaml b/reproducible_build.yaml
index 5487e72fdff..5da4c005137 100644
--- a/reproducible_build.yaml
+++ b/reproducible_build.yaml
@@ -1,2 +1,2 @@
-release-notes-hash: 719ad24b6054f87b65affa20ce1cd198
-source-date-epoch: 1751632100
+release-notes-hash: ddd077b4b5c548694222c69b5f4f239f
+source-date-epoch: 1751886321
(airflow) 05/05: [v3-0-test] Make the update_installer_and_pre_commit script more resilient to re matches (#52960) (#52961)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v3-0-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 6062f68055b61e5218742dca11ec2e3c97b3e4d5
Author: github-actions[bot] <41...@users.noreply.github.com>
AuthorDate: Mon Jul 7 08:33:09 2025 +0200
[v3-0-test] Make the update_installer_and_pre_commit script more resilient to re matches (#52960) (#52961)
(cherry picked from commit 8e5c284913f550b5b583e06f400f4e761c87fbd5)
Co-authored-by: Amogh Desai <am...@gmail.com>
---
scripts/ci/pre_commit/update_installers_and_pre_commit.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/scripts/ci/pre_commit/update_installers_and_pre_commit.py b/scripts/ci/pre_commit/update_installers_and_pre_commit.py
index 34b6a0b85ff..612acdaaa2f 100755
--- a/scripts/ci/pre_commit/update_installers_and_pre_commit.py
+++ b/scripts/ci/pre_commit/update_installers_and_pre_commit.py
@@ -96,8 +96,8 @@ UV_PATTERNS: list[tuple[re.Pattern, Quoting]] = [
(re.compile(r"(AIRFLOW_UV_VERSION=)([0-9.]+)"), Quoting.UNQUOTED),
(re.compile(r"(uv>=)([0-9.]+)"), Quoting.UNQUOTED),
(re.compile(r"(AIRFLOW_UV_VERSION = )(\"[0-9.]+\")"), Quoting.DOUBLE_QUOTED),
- (re.compile(r"(UV_VERSION = )(\"[0-9.]+\")"), Quoting.DOUBLE_QUOTED),
- (re.compile(r"(UV_VERSION=)(\"[0-9.]+\")"), Quoting.DOUBLE_QUOTED),
+ (re.compile(r"^(\s*UV_VERSION = )(\"[0-9.]+\")", re.MULTILINE), Quoting.DOUBLE_QUOTED),
+ (re.compile(r"^(\s*UV_VERSION=)(\"[0-9.]+\")", re.MULTILINE), Quoting.DOUBLE_QUOTED),
(re.compile(r"(\| *`AIRFLOW_UV_VERSION` *\| *)(`[0-9.]+`)( *\|)"), Quoting.REVERSE_SINGLE_QUOTED),
(
re.compile(
(airflow) 02/05: [Backport v3-0-test] Add back dag parsing pre-import optimization (#50371) (#52698)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v3-0-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit b929b4e036b867fa5b6f36613178b9e3766d3c2c
Author: Kevin Liu <58...@users.noreply.github.com>
AuthorDate: Fri Jul 4 16:25:27 2025 +0800
[Backport v3-0-test] Add back dag parsing pre-import optimization (#50371) (#52698)
* Add back dag parsing pre-import optimization (#50371)
Co-authored-by: Tzu-ping Chung <ur...@gmail.com>
* Apply suggestions from code review
---------
Co-authored-by: Tzu-ping Chung <ur...@gmail.com>
Co-authored-by: Kaxil Naik <ka...@gmail.com>
---
.../src/airflow/config_templates/config.yml | 20 ++---
airflow-core/src/airflow/configuration.py | 1 +
.../src/airflow/dag_processing/processor.py | 28 +++++++
.../tests/unit/dag_processing/test_processor.py | 85 +++++++++++++++++++++-
4 files changed, 122 insertions(+), 12 deletions(-)
diff --git a/airflow-core/src/airflow/config_templates/config.yml b/airflow-core/src/airflow/config_templates/config.yml
index 92134fa4bf1..503084e13a9 100644
--- a/airflow-core/src/airflow/config_templates/config.yml
+++ b/airflow-core/src/airflow/config_templates/config.yml
@@ -2199,16 +2199,6 @@ scheduler:
type: integer
default: "20"
see_also: ":ref:`scheduler:ha:tunables`"
- parsing_pre_import_modules:
- description: |
- The scheduler reads dag files to extract the airflow modules that are going to be used,
- and imports them ahead of time to avoid having to re-do it for each parsing process.
- This flag can be set to ``False`` to disable this behavior in case an airflow module needs
- to be freshly imported each time (at the cost of increased DAG parsing time).
- version_added: 2.6.0
- type: boolean
- example: ~
- default: "True"
dag_stale_not_seen_duration:
description: |
Time in seconds after which dags, which were not updated by Dag Processor are deactivated.
@@ -2563,3 +2553,13 @@ dag_processor:
type: integer
example: ~
default: "10"
+ parsing_pre_import_modules:
+ description: |
+ The dag_processor reads dag files to extract the airflow modules that are going to be used,
+ and imports them ahead of time to avoid having to re-do it for each parsing process.
+ This flag can be set to ``False`` to disable this behavior in case an airflow module needs
+ to be freshly imported each time (at the cost of increased DAG parsing time).
+ version_added: 2.6.0
+ type: boolean
+ example: ~
+ default: "True"
diff --git a/airflow-core/src/airflow/configuration.py b/airflow-core/src/airflow/configuration.py
index 2bf3df98163..5fd67f53343 100644
--- a/airflow-core/src/airflow/configuration.py
+++ b/airflow-core/src/airflow/configuration.py
@@ -364,6 +364,7 @@ class AirflowConfigParser(ConfigParser):
("fab", "navbar_text_hover_color"): ("webserver", "navbar_text_hover_color", "3.0.2"),
("api", "secret_key"): ("webserver", "secret_key", "3.0.2"),
("api", "enable_swagger_ui"): ("webserver", "enable_swagger_ui", "3.0.2"),
+ ("dag_processor", "parsing_pre_import_modules"): ("scheduler", "parsing_pre_import_modules", "3.0.3"),
}
# A mapping of new section -> (old section, since_version).
diff --git a/airflow-core/src/airflow/dag_processing/processor.py b/airflow-core/src/airflow/dag_processing/processor.py
index 011393f22c8..a54c56ddbe2 100644
--- a/airflow-core/src/airflow/dag_processing/processor.py
+++ b/airflow-core/src/airflow/dag_processing/processor.py
@@ -16,6 +16,7 @@
# under the License.
from __future__ import annotations
+import importlib
import os
import sys
import traceback
@@ -45,6 +46,7 @@ from airflow.sdk.execution_time.comms import (
from airflow.sdk.execution_time.supervisor import WatchedSubprocess
from airflow.serialization.serialized_objects import LazyDeserializedDAG, SerializedDAG
from airflow.stats import Stats
+from airflow.utils.file import iter_airflow_imports
if TYPE_CHECKING:
from structlog.typing import FilteringBoundLogger
@@ -98,6 +100,27 @@ ToDagProcessor = Annotated[
]
+def _pre_import_airflow_modules(file_path: str, log: FilteringBoundLogger) -> None:
+ """
+ Pre-import Airflow modules found in the given file.
+
+ This prevents modules from being re-imported in each processing process,
+ saving CPU time and memory.
+ (The default value of "parsing_pre_import_modules" is set to True)
+
+ :param file_path: Path to the file to scan for imports
+ :param log: Logger instance to use for warnings
+ """
+ if not conf.getboolean("dag_processor", "parsing_pre_import_modules", fallback=True):
+ return
+
+ for module in iter_airflow_imports(file_path):
+ try:
+ importlib.import_module(module)
+ except ModuleNotFoundError as e:
+ log.warning("Error when trying to pre-import module '%s' found in %s: %s", module, file_path, e)
+
+
def _parse_file_entrypoint():
import structlog
@@ -127,6 +150,7 @@ def _parse_file_entrypoint():
def _parse_file(msg: DagFileParseRequest, log: FilteringBoundLogger) -> DagFileParsingResult | None:
# TODO: Set known_pool names on DagBag!
+
bag = DagBag(
dag_folder=msg.file,
bundle_path=msg.bundle_path,
@@ -250,6 +274,10 @@ class DagFileProcessorProcess(WatchedSubprocess):
client: Client,
**kwargs,
) -> Self:
+ logger = kwargs["logger"]
+
+ _pre_import_airflow_modules(os.fspath(path), logger)
+
proc: Self = super().start(target=target, client=client, **kwargs)
proc._on_child_started(callbacks, path, bundle_path)
return proc
diff --git a/airflow-core/tests/unit/dag_processing/test_processor.py b/airflow-core/tests/unit/dag_processing/test_processor.py
index 8d77da61caf..e8ab15a0222 100644
--- a/airflow-core/tests/unit/dag_processing/test_processor.py
+++ b/airflow-core/tests/unit/dag_processing/test_processor.py
@@ -37,6 +37,7 @@ from airflow.dag_processing.processor import (
DagFileParsingResult,
DagFileProcessorProcess,
_parse_file,
+ _pre_import_airflow_modules,
)
from airflow.models import DagBag, TaskInstance
from airflow.models.baseoperator import BaseOperator
@@ -140,8 +141,13 @@ class TestDagFileProcessor:
assert "a.py" in resp.import_errors
def test_top_level_variable_access(
- self, spy_agency: SpyAgency, tmp_path: pathlib.Path, monkeypatch: pytest.MonkeyPatch, inprocess_client
+ self,
+ spy_agency: SpyAgency,
+ tmp_path: pathlib.Path,
+ monkeypatch: pytest.MonkeyPatch,
+ inprocess_client,
):
+ logger = MagicMock()
logger_filehandle = MagicMock()
def dag_in_a_fn():
@@ -158,6 +164,7 @@ class TestDagFileProcessor:
path=path,
bundle_path=tmp_path,
callbacks=[],
+ logger=logger,
logger_filehandle=logger_filehandle,
client=inprocess_client,
)
@@ -171,8 +178,13 @@ class TestDagFileProcessor:
assert result.serialized_dags[0].dag_id == "test_abc"
def test_top_level_variable_access_not_found(
- self, spy_agency: SpyAgency, tmp_path: pathlib.Path, monkeypatch: pytest.MonkeyPatch, inprocess_client
+ self,
+ spy_agency: SpyAgency,
+ tmp_path: pathlib.Path,
+ monkeypatch: pytest.MonkeyPatch,
+ inprocess_client,
):
+ logger = MagicMock()
logger_filehandle = MagicMock()
def dag_in_a_fn():
@@ -187,6 +199,7 @@ class TestDagFileProcessor:
path=path,
bundle_path=tmp_path,
callbacks=[],
+ logger=logger,
logger_filehandle=logger_filehandle,
client=inprocess_client,
)
@@ -203,6 +216,7 @@ class TestDagFileProcessor:
def test_top_level_variable_set(self, tmp_path: pathlib.Path, inprocess_client):
from airflow.models.variable import Variable as VariableORM
+ logger = MagicMock()
logger_filehandle = MagicMock()
def dag_in_a_fn():
@@ -218,6 +232,7 @@ class TestDagFileProcessor:
path=path,
bundle_path=tmp_path,
callbacks=[],
+ logger=logger,
logger_filehandle=logger_filehandle,
client=inprocess_client,
)
@@ -238,6 +253,7 @@ class TestDagFileProcessor:
def test_top_level_variable_delete(self, tmp_path: pathlib.Path, inprocess_client):
from airflow.models.variable import Variable as VariableORM
+ logger = MagicMock()
logger_filehandle = MagicMock()
def dag_in_a_fn():
@@ -259,6 +275,7 @@ class TestDagFileProcessor:
path=path,
bundle_path=tmp_path,
callbacks=[],
+ logger=logger,
logger_filehandle=logger_filehandle,
client=inprocess_client,
)
@@ -278,6 +295,7 @@ class TestDagFileProcessor:
def test_top_level_connection_access(
self, tmp_path: pathlib.Path, monkeypatch: pytest.MonkeyPatch, inprocess_client
):
+ logger = MagicMock()
logger_filehandle = MagicMock()
def dag_in_a_fn():
@@ -295,6 +313,7 @@ class TestDagFileProcessor:
path=path,
bundle_path=tmp_path,
callbacks=[],
+ logger=logger,
logger_filehandle=logger_filehandle,
client=inprocess_client,
)
@@ -308,6 +327,7 @@ class TestDagFileProcessor:
assert result.serialized_dags[0].dag_id == "test_my_conn"
def test_top_level_connection_access_not_found(self, tmp_path: pathlib.Path, inprocess_client):
+ logger = MagicMock()
logger_filehandle = MagicMock()
def dag_in_a_fn():
@@ -323,6 +343,7 @@ class TestDagFileProcessor:
path=path,
bundle_path=tmp_path,
callbacks=[],
+ logger=logger,
logger_filehandle=logger_filehandle,
client=inprocess_client,
)
@@ -355,6 +376,7 @@ class TestDagFileProcessor:
path=dag1_path,
bundle_path=tmp_path,
callbacks=[],
+ logger=MagicMock(),
logger_filehandle=MagicMock(),
client=inprocess_client,
)
@@ -366,6 +388,65 @@ class TestDagFileProcessor:
assert result.import_errors == {}
assert result.serialized_dags[0].dag_id == "dag_name"
+ def test__pre_import_airflow_modules_when_disabled(self):
+ logger = MagicMock()
+ with (
+ env_vars({"AIRFLOW__DAG_PROCESSOR__PARSING_PRE_IMPORT_MODULES": "false"}),
+ patch("airflow.dag_processing.processor.iter_airflow_imports") as mock_iter,
+ ):
+ _pre_import_airflow_modules("test.py", logger)
+
+ mock_iter.assert_not_called()
+ logger.warning.assert_not_called()
+
+ def test__pre_import_airflow_modules_when_enabled(self):
+ logger = MagicMock()
+ with (
+ env_vars({"AIRFLOW__DAG_PROCESSOR__PARSING_PRE_IMPORT_MODULES": "true"}),
+ patch("airflow.dag_processing.processor.iter_airflow_imports", return_value=["airflow.models"]),
+ patch("airflow.dag_processing.processor.importlib.import_module") as mock_import,
+ ):
+ _pre_import_airflow_modules("test.py", logger)
+
+ mock_import.assert_called_once_with("airflow.models")
+ logger.warning.assert_not_called()
+
+ def test__pre_import_airflow_modules_warns_on_missing_module(self):
+ logger = MagicMock()
+ with (
+ env_vars({"AIRFLOW__DAG_PROCESSOR__PARSING_PRE_IMPORT_MODULES": "true"}),
+ patch(
+ "airflow.dag_processing.processor.iter_airflow_imports", return_value=["non_existent_module"]
+ ),
+ patch(
+ "airflow.dag_processing.processor.importlib.import_module", side_effect=ModuleNotFoundError()
+ ),
+ ):
+ _pre_import_airflow_modules("test.py", logger)
+
+ logger.warning.assert_called_once()
+ warning_args = logger.warning.call_args[0]
+ assert "Error when trying to pre-import module" in warning_args[0]
+ assert "non_existent_module" in warning_args[1]
+ assert "test.py" in warning_args[2]
+
+ def test__pre_import_airflow_modules_partial_success_and_warning(self):
+ logger = MagicMock()
+ with (
+ env_vars({"AIRFLOW__DAG_PROCESSOR__PARSING_PRE_IMPORT_MODULES": "true"}),
+ patch(
+ "airflow.dag_processing.processor.iter_airflow_imports",
+ return_value=["airflow.models", "non_existent_module"],
+ ),
+ patch(
+ "airflow.dag_processing.processor.importlib.import_module",
+ side_effect=[None, ModuleNotFoundError()],
+ ),
+ ):
+ _pre_import_airflow_modules("test.py", logger)
+
+ assert logger.warning.call_count == 1
+
def write_dag_in_a_fn_to_file(fn: Callable[[], None], folder: pathlib.Path) -> pathlib.Path:
# Create the dag in a fn, and use inspect.getsource to write it to a file so that
(airflow) 03/05: [v3-0-test] Advanced auto classification for provider documentation (#52902) (#52925)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v3-0-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit a6d1a5b04bd81ba732d85b4c47c6b1085722cc3e
Author: github-actions[bot] <41...@users.noreply.github.com>
AuthorDate: Sun Jul 6 01:05:07 2025 +0200
[v3-0-test] Advanced auto classification for provider documentation (#52902) (#52925)
(cherry picked from commit 19961713421885c8306d3a51a4c3eaf5f27f6cb7)
Co-authored-by: Amogh Desai <am...@gmail.com>
---
.../prepare_providers/provider_documentation.py | 11 +++---
dev/breeze/tests/test_provider_documentation.py | 39 ++++++++++++++++++----
2 files changed, 39 insertions(+), 11 deletions(-)
diff --git a/dev/breeze/src/airflow_breeze/prepare_providers/provider_documentation.py b/dev/breeze/src/airflow_breeze/prepare_providers/provider_documentation.py
index c330047b7cd..b5f6a339cbd 100644
--- a/dev/breeze/src/airflow_breeze/prepare_providers/provider_documentation.py
+++ b/dev/breeze/src/airflow_breeze/prepare_providers/provider_documentation.py
@@ -195,7 +195,10 @@ TYPE_OF_CHANGE_DESCRIPTION = {
}
-def classification_result(changed_files):
+def classification_result(provider_id, changed_files):
+ provider_path = f"providers/{provider_id}/"
+ changed_files = list(filter(lambda f: f.startswith(provider_path), changed_files))
+
if not changed_files:
return "other"
@@ -214,7 +217,7 @@ def classification_result(changed_files):
return "other"
-def classify_provider_pr_files(commit_hash: str) -> str:
+def classify_provider_pr_files(provider_id: str, commit_hash: str) -> str:
"""
Classify a provider commit based on changed files.
@@ -235,7 +238,7 @@ def classify_provider_pr_files(commit_hash: str) -> str:
# safe to return other here
return "other"
- return classification_result(changed_files)
+ return classification_result(provider_id, changed_files)
def _get_git_log_command(
@@ -827,7 +830,7 @@ def update_release_notes(
)
change = list_of_list_of_changes[0][table_iter]
- classification = classify_provider_pr_files(change.full_hash)
+ classification = classify_provider_pr_files(provider_id, change.full_hash)
if classification == "documentation":
get_console().print(
f"[green]Automatically classifying change as DOCUMENTATION since it contains only doc changes:[/]\n"
diff --git a/dev/breeze/tests/test_provider_documentation.py b/dev/breeze/tests/test_provider_documentation.py
index b55088987c6..6ff67e267d3 100644
--- a/dev/breeze/tests/test_provider_documentation.py
+++ b/dev/breeze/tests/test_provider_documentation.py
@@ -395,16 +395,20 @@ def test_get_most_impactful_change(changes, expected):
@pytest.mark.parametrize(
- "changed_files, expected",
+ "provider_id, changed_files, expected",
[
- pytest.param(["providers/slack/docs/slack.rst"], "documentation", id="only_docs"),
- pytest.param(["providers/slack/tests/test_slack.py"], "test_or_example_only", id="only_tests"),
+ pytest.param("slack", ["providers/slack/docs/slack.rst"], "documentation", id="only_docs"),
pytest.param(
+ "slack", ["providers/slack/tests/test_slack.py"], "test_or_example_only", id="only_tests"
+ ),
+ pytest.param(
+ "slack",
["providers/slack/src/airflow/providers/slack/example_dags/example_notify.py"],
"test_or_example_only",
id="only_example_dags",
),
pytest.param(
+ "slack",
[
"providers/slack/tests/test_slack.py",
"providers/slack/src/airflow/providers/slack/example_dags/example_notify.py",
@@ -413,6 +417,7 @@ def test_get_most_impactful_change(changes, expected):
id="tests_and_example_dags",
),
pytest.param(
+ "slack",
[
"providers/slack/tests/test_slack.py",
"providers/slack/docs/slack.rst",
@@ -421,6 +426,7 @@ def test_get_most_impactful_change(changes, expected):
id="docs_and_tests",
),
pytest.param(
+ "slack",
[
"providers/slack/src/airflow/providers/slack/hooks/slack.py",
"providers/slack/docs/slack.rst",
@@ -429,6 +435,7 @@ def test_get_most_impactful_change(changes, expected):
id="docs_and_real_code",
),
pytest.param(
+ "slack",
[
"providers/slack/src/airflow/providers/slack/hooks/slack.py",
"providers/slack/tests/test_slack.py",
@@ -436,10 +443,28 @@ def test_get_most_impactful_change(changes, expected):
"other",
id="real_code_and_tests",
),
- pytest.param(["airflow/utils/db.py"], "other", id="non_provider_file"),
- pytest.param([], "other", id="empty_commit"),
+ pytest.param(
+ "google",
+ [
+ "providers/google/tests/some_test.py",
+ "providers/amazon/tests/test_something.py",
+ ],
+ "test_or_example_only",
+ id="tests_in_multiple_providers",
+ ),
+ pytest.param(
+ "amazon",
+ [
+ "providers/google/tests/some_test.py",
+ "providers/amazon/tests/test_something.py",
+ ],
+ "test_or_example_only",
+ id="tests_in_multiple_providers",
+ ),
+ pytest.param("slack", ["airflow/utils/db.py"], "other", id="non_provider_file"),
+ pytest.param("slack", [], "other", id="empty_commit"),
],
)
-def test_classify_provider_pr_files_logic(changed_files, expected):
- result = classification_result(changed_files)
+def test_classify_provider_pr_files_logic(provider_id, changed_files, expected):
+ result = classification_result(provider_id, changed_files)
assert result == expected