You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by po...@apache.org on 2026/04/20 11:04:40 UTC
(airflow) branch main updated: Isolate non-provider mypy hooks per distribution with dedicated .build/ venvs (#65492)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 4f3b228be1a Isolate non-provider mypy hooks per distribution with dedicated .build/ venvs (#65492)
4f3b228be1a is described below
commit 4f3b228be1a785ea5df7b89d8d3291ded78fea4d
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Mon Apr 20 13:04:32 2026 +0200
Isolate non-provider mypy hooks per distribution with dedicated .build/ venvs (#65492)
* Isolate mypy prek hooks, cover all non-provider dirs, and clean up type errors
Each non-provider mypy prek hook now builds and caches its own virtualenv
at .build/mypy-venvs/<hook>/ and its own mypy cache at
.build/mypy-caches/<hook>/. UV_PROJECT_ENVIRONMENT redirects uv away from
the project's .venv, so running the hook never mutates a contributor's
regular development environment while still matching CI's frozen
dependency set. Mypy runs with --follow-imports=silent so each hook only
reports errors for files it owns; transitive code is covered by its own
hook and different venvs no longer produce divergent results on shared
code.
Adds mypy hooks for the non-provider directories that were previously
uncovered: airflow-ctl-tests, helm-tests, airflow-e2e-tests,
task-sdk-integration-tests, docker-tests, kubernetes-tests, and shared.
The mypy-shared hook iterates every shared/<dist> workspace distribution
and builds a separate venv + cache per distribution so each shared
library is type-checked against its own dependency set.
breeze down --cleanup-mypy-cache additionally removes
.build/mypy-venvs/ and .build/mypy-caches/ so all per-hook state is
wiped alongside the existing .mypy_cache and mypy-cache-volume.
Also fixes pre-existing type errors surfaced by the newly added and
cleaned-up checks: platform-specific ignores for Linux-only
os.posix_fadvise in the shared logging helper, narrower types and
type: ignore where appropriate in shared configuration/observability/
timezones/secrets_backend/secrets_masker, Liskov override markers on
the AirflowConfigParser subclass methods, and small correctness fixes
in dev/breeze and the docker-tests / kubernetes-tests helpers so the
full non-provider mypy suite runs clean on macOS and in CI.
* Move mypy prek hooks to their respective distribution configs
The new mypy hooks for airflow-ctl-tests, helm-tests, airflow-e2e-tests,
task-sdk-integration-tests, docker-tests, and kubernetes-tests now live
in each distribution's own .pre-commit-config.yaml, matching the pattern
already used by airflow-core, task-sdk, and airflow-ctl. New .pre-commit-
config.yaml files are added to distributions that didn't have one. prek
auto-discovers nested configs, so the hooks remain part of the default
check set.
mypy-dev (covers dev + scripts), mypy-devel-common, and mypy-shared stay
at the repo root: dev/scripts/devel-common don't have their own configs,
and mypy-shared iterates every shared/<dist> distribution so has no
single home.
* Split mypy-dev and mypy-scripts, each with its own pyproject.toml config
Previously the mypy-dev prek hook ran mypy against dev/ and scripts/ in
a single invocation under the dev project's virtualenv. The two now get
independent hooks — mypy-dev in dev/.pre-commit-config.yaml and
mypy-scripts in scripts/.pre-commit-config.yaml — so each can evolve its
own dependency set and check its own folder.
Copy the full [tool.mypy] section from the root pyproject.toml into both
dev/pyproject.toml and scripts/pyproject.toml so each sub-project owns its
mypy configuration. Paths inside mypy_path are rewritten from
$MYPY_CONFIG_FILE_DIR/ to $MYPY_CONFIG_FILE_DIR/../ so they still resolve
to the repo-root siblings from the sub-project location. The decorator/
outputs plugins are scoped to dev only (scripts does not author DAG code).
mypy_local_folder.py now passes --config-file <project>/pyproject.toml
when the folder maps to one of these sub-project configs, so mypy uses
the sub-project's configuration rather than the root one.
* Teach selective-checks about the new non-provider mypy hooks
Add FileGroupForCi entries and regex patterns for helm-tests,
airflow-e2e-tests, docker-tests, kubernetes-tests, scripts, and shared
Python files, then wire them into skip_prek_hooks so the corresponding
mypy-* prek hook is only kept when its folder changed:
- mypy-scripts (split off from the old combined mypy-dev)
- mypy-airflow-ctl-tests, mypy-helm-tests, mypy-airflow-e2e-tests,
mypy-task-sdk-integration-tests, mypy-docker-tests, mypy-kubernetes-tests
- mypy-shared
Update test_selective_checks.py skip-list constants and per-case inline
skip lists to include the new hooks. Targeted test cases for files under
the new-hook directories override skip-prek-hooks to leave the matching
hook out of the skip set, confirming it will run when its folder changes.
* Trim dev/scripts pyproject mypy_path to just relevant distributions
Drop the 200+ provider path entries that were blindly copied from the root
pyproject.toml. dev and scripts only import from other non-provider
workspace members, so listing every provider src/tests directory under
mypy_path just adds noise. The remaining non-provider entries cover
everything dev or scripts plausibly import from.
* Install mypy into per-hook venvs from uv.lock via a `mypy` dep group
Each non-provider distribution with a mypy prek hook now declares a
`mypy` dependency group in its pyproject.toml resolving to
`apache-airflow-devel-common[mypy]`. mypy_local_folder.py syncs each
dedicated virtualenv with `uv sync --frozen --project <X> --group mypy`
and runs mypy with `uv run --frozen --project <X> --group mypy` — so
mypy and its type stubs come from the workspace uv.lock, not from an
ephemeral `--with` overlay whose resolution is independent of the main
lockfile. uv.lock is refreshed to include the new group.
Covers airflow-core, task-sdk, airflow-ctl, devel-common, dev, scripts,
airflow-ctl-tests, helm-tests, airflow-e2e-tests, task-sdk-integration-
tests, docker-tests, kubernetes-tests, and every shared/<dist>
workspace member.
* Drop mypy_path from dev/scripts pyprojects — venv site-packages is enough
After the switch to installing mypy (and every transitive workspace
dependency) directly into each hook's virtualenv via the `mypy` dep
group, workspace packages like airflow, airflow.sdk, airflowctl,
airflow_breeze, tests_common are all available via the venv's
site-packages. mypy resolves them without needing mypy_path entries,
so drop the copied list and leave a short comment explaining why.
* Split mypy-shared into per-distribution hooks and enforce the pattern
Each shared/<dist> workspace member now owns a mypy-shared-<dist> prek hook
backed by its own shared/<dist>/.pre-commit-config.yaml. The single
mypy-shared iterator is gone — mypy_local_folder.py accepts shared/<dist>
as a first-class folder and the per-hook virtualenv now lives at
.build/mypy-venvs/shared-<dist>/ (slash in the folder name is replaced
with a dash in the venv/cache path).
Adds a new check-shared-mypy-hooks prek hook that fails when a
shared/<dist> workspace member is missing its dedicated .pre-commit-
config.yaml, printing the exact YAML to add. Selective-checks emits one
skip entry per dist, enumerated from shared/ at run time. Contributing
docs cover the two-step process for adding a new shared library.
* Pin minimum_prek_version to 0.3.4 consistently across all configs
All .pre-commit-config.yaml files now require prek >= 0.3.4 (the version
already declared by the root config). Previously the nested configs
pinned a mix of 0.2.0, 0.3.2, and 0.3.4, so a contributor could pass the
root's version check and still trip on stale subproject pins as they
moved between directories.
* Refresh uv.lock after rebase to reflect the `mypy` dep groups
The rebase onto main resolved the uv.lock conflict by taking main's
version, so `uv sync --group mypy` would fail against uv.lock until
the groups added to the per-distribution pyprojects were re-resolved.
Regenerates the lockfile to include them.
* Add explicit selective-checks test for per-shared-dist mypy hook skipping
Verifies that when a file under shared/logging/ changes, only
mypy-shared-logging is kept among the thirteen mypy-shared-* hooks;
all other shared distributions' hooks land in the skip list. Pins the
contract that the runtime enumeration over shared/*/pyproject.toml
works as intended.
* Refresh mypy docs to match the per-hook venv + --group mypy workflow
Fills in the docs that still referenced the pre-split workflow:
- AGENTS.md: mentions `mypy-shared-<dist>` per shared workspace member and
the `uv sync --group mypy` install path for mypy itself.
- scripts/ci/prek/AGENTS.md: clarifies that non-provider mypy hooks run
locally through mypy_local_folder.py (Breeze image only needed for the
providers hook).
- dev/breeze/doc/03_developer_tasks.rst: renames stale `mypy-airflow` to
`mypy-airflow-core`, and expands the cache note to cover the per-hook
virtualenvs and caches under .build/.
- dev/breeze/doc/ci/04_selective_checks.md: expands the file-group and
skip-reason lists so every new mypy hook (scripts, task-sdk, airflow-ctl,
the six test-dir hooks, and mypy-shared-<dist> enumerated at runtime)
is documented.
* Rename mypy_local_folder.py to run_mypy_full_dist_local_venv_or_breeze_in_ci.py
Updates every .pre-commit-config.yaml entry and prose references so they
point at the new script name. Two shared configs use YAML folded-scalar
entries to stay under the 110-char yamllint limit; updates the validation
script's expected template to match.
---
.pre-commit-config.yaml | 14 +-
AGENTS.md | 2 +-
airflow-core/.pre-commit-config.yaml | 4 +-
airflow-core/pyproject.toml | 4 +
airflow-ctl-tests/.pre-commit-config.yaml | 9 +-
airflow-ctl-tests/pyproject.toml | 5 +
airflow-ctl/.pre-commit-config.yaml | 4 +-
airflow-ctl/pyproject.toml | 4 +
.../.pre-commit-config.yaml | 13 +-
airflow-e2e-tests/pyproject.toml | 5 +
chart/.pre-commit-config.yaml | 2 +-
contributing-docs/08_static_code_checks.rst | 46 +--
{airflow-ctl-tests => dev}/.pre-commit-config.yaml | 13 +-
dev/breeze/doc/03_developer_tasks.rst | 22 +-
dev/breeze/doc/ci/04_selective_checks.md | 24 +-
.../airflow_breeze/commands/developer_commands.py | 5 +
.../src/airflow_breeze/commands/pr_commands.py | 8 +-
.../commands/release_candidate_command.py | 3 +
.../airflow_breeze/commands/testing_commands.py | 2 +-
.../src/airflow_breeze/utils/reproducible.py | 2 +-
.../src/airflow_breeze/utils/selective_checks.py | 50 ++++
dev/breeze/tests/test_selective_checks.py | 181 ++++++++++--
dev/pyproject.toml | 70 +++++
devel-common/pyproject.toml | 4 +
.../.pre-commit-config.yaml | 13 +-
docker-tests/pyproject.toml | 5 +
.../test_docker_compose_quick_start.py | 2 +-
docker-tests/tests/docker_tests/test_prod_image.py | 2 +-
go-sdk/.pre-commit-config.yaml | 2 +-
.../.pre-commit-config.yaml | 13 +-
helm-tests/pyproject.toml | 5 +
.../.pre-commit-config.yaml | 13 +-
kubernetes-tests/pyproject.toml | 5 +
.../tests/kubernetes_tests/test_base.py | 4 +-
.../test_kubernetes_pod_operator.py | 2 +-
providers/.pre-commit-config.yaml | 2 +-
providers/common/ai/.pre-commit-config.yaml | 2 +-
providers/common/compat/.pre-commit-config.yaml | 2 +-
providers/edge3/.pre-commit-config.yaml | 2 +-
providers/fab/.pre-commit-config.yaml | 2 +-
providers/keycloak/.pre-commit-config.yaml | 2 +-
.../.pre-commit-config.yaml | 13 +-
scripts/ci/prek/AGENTS.md | 9 +-
scripts/ci/prek/check_shared_mypy_hooks.py | 92 ++++++
scripts/ci/prek/mypy_local_folder.py | 250 ----------------
...un_mypy_full_dist_local_venv_or_breeze_in_ci.py | 325 +++++++++++++++++++++
scripts/pyproject.toml | 65 +++++
.../configuration}/.pre-commit-config.yaml | 13 +-
shared/configuration/pyproject.toml | 4 +
.../src/airflow_shared/configuration/parser.py | 12 +-
.../dagnode}/.pre-commit-config.yaml | 13 +-
shared/dagnode/pyproject.toml | 4 +
shared/dagnode/tests/dagnode/test_node.py | 6 +-
.../listeners}/.pre-commit-config.yaml | 13 +-
shared/listeners/pyproject.toml | 4 +
.../logging}/.pre-commit-config.yaml | 13 +-
shared/logging/pyproject.toml | 4 +
.../src/airflow_shared/logging/_noncaching.py | 3 +-
.../module_loading}/.pre-commit-config.yaml | 13 +-
shared/module_loading/pyproject.toml | 4 +
.../observability}/.pre-commit-config.yaml | 13 +-
shared/observability/pyproject.toml | 4 +
.../src/airflow_shared/observability/common.py | 22 +-
.../observability/metrics/otel_logger.py | 2 +-
.../airflow_shared/observability/metrics/stats.py | 1 +
.../plugins_manager}/.pre-commit-config.yaml | 13 +-
shared/plugins_manager/pyproject.toml | 4 +
.../providers_discovery}/.pre-commit-config.yaml | 15 +-
shared/providers_discovery/pyproject.toml | 4 +
.../secrets_backend}/.pre-commit-config.yaml | 13 +-
shared/secrets_backend/pyproject.toml | 4 +
.../src/airflow_shared/secrets_backend/base.py | 2 +-
.../secrets_masker}/.pre-commit-config.yaml | 13 +-
shared/secrets_masker/pyproject.toml | 4 +
.../tests/secrets_masker/test_secrets_masker.py | 2 +-
.../serialization}/.pre-commit-config.yaml | 13 +-
shared/serialization/pyproject.toml | 4 +
.../template_rendering}/.pre-commit-config.yaml | 15 +-
shared/template_rendering/pyproject.toml | 4 +
.../timezones}/.pre-commit-config.yaml | 13 +-
shared/timezones/pyproject.toml | 4 +
.../src/airflow_shared/timezones/timezone.py | 8 +-
.../.pre-commit-config.yaml | 13 +-
task-sdk-integration-tests/pyproject.toml | 5 +
task-sdk/.pre-commit-config.yaml | 4 +-
task-sdk/pyproject.toml | 4 +
task-sdk/src/airflow/sdk/execution_time/comms.py | 4 +-
uv.lock | 130 ++++++++-
88 files changed, 1278 insertions(+), 493 deletions(-)
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index 1a3faed61ab..7cce2b9e56e 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -1031,19 +1031,19 @@ repos:
^uv\.lock$
pass_filenames: false
require_serial: true
- - id: mypy-dev
- name: Run mypy for dev
+ - id: mypy-devel-common
+ name: Run mypy for devel-common
language: python
- entry: ./scripts/ci/prek/mypy_local_folder.py dev scripts
+ entry: ./scripts/ci/prek/run_mypy_full_dist_local_venv_or_breeze_in_ci.py devel-common
pass_filenames: false
files: ^.*\.py$
require_serial: true
- - id: mypy-devel-common
- name: Run mypy for devel-common
+ - id: check-shared-mypy-hooks
+ name: Every shared/<dist> has a mypy-shared-<dist> prek hook
language: python
- entry: ./scripts/ci/prek/mypy_local_folder.py devel-common
+ entry: ./scripts/ci/prek/check_shared_mypy_hooks.py
pass_filenames: false
- files: ^.*\.py$
+ files: ^shared/.*|^scripts/ci/prek/check_shared_mypy_hooks\.py$
require_serial: true
## ADD MOST PREK HOOK ABOVE THAT LINE
# The below prek hooks are those requiring CI image to be built
diff --git a/AGENTS.md b/AGENTS.md
index 5ad16864d84..11cc4dcb8fa 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -30,7 +30,7 @@
- **Run other suites of tests** `breeze testing <test_group>` (test groups: `airflow-ctl-tests`, `docker-compose-tests`, `task-sdk-tests`)
- **Run scripts tests:** `uv run --project scripts pytest scripts/tests/ -xvs`
- **Run Airflow CLI:** `breeze run airflow dags list`
-- **Type-check (non-providers):** first run `uv sync --frozen --project <PROJECT>` to align the local virtualenv with `uv.lock` (the dependency set CI uses), then `uv run --frozen --project <PROJECT> --with "apache-airflow-devel-common[mypy]" mypy path/to/code`
+- **Type-check (non-providers):** run the prek hook — `prek run mypy-<project> --all-files` (e.g. `mypy-airflow-core`, `mypy-task-sdk`, `mypy-shared-logging`; each `shared/<dist>` workspace member has its own `mypy-shared-<dist>` hook). The hook uses a dedicated virtualenv and mypy cache under `.build/mypy-venvs/<hook>/` and `.build/mypy-caches/<hook>/`; mypy itself is installed from `uv.lock` via the `mypy` dependency group (`uv sync --group mypy`), so it never mutates your project `.ve [...]
- **Type-check (providers):** `breeze run mypy path/to/code`
- **Lint with ruff only:** `prek run ruff --from-ref <target_branch>`
- **Format with ruff only:** `prek run ruff-format --from-ref <target_branch>`
diff --git a/airflow-core/.pre-commit-config.yaml b/airflow-core/.pre-commit-config.yaml
index 779f0bb2da5..de0a566c333 100644
--- a/airflow-core/.pre-commit-config.yaml
+++ b/airflow-core/.pre-commit-config.yaml
@@ -16,7 +16,7 @@
# under the License.
---
default_stages: [pre-commit, pre-push]
-minimum_prek_version: '0.2.0'
+minimum_prek_version: '0.3.4'
default_language_version:
python: python3
node: 22.19.0
@@ -224,7 +224,7 @@ repos:
- id: mypy-airflow-core
name: Run mypy for airflow-core
language: python
- entry: ../scripts/ci/prek/mypy_local_folder.py airflow-core
+ entry: ../scripts/ci/prek/run_mypy_full_dist_local_venv_or_breeze_in_ci.py airflow-core
pass_filenames: false
files: ^.*\.py$
require_serial: true
diff --git a/airflow-core/pyproject.toml b/airflow-core/pyproject.toml
index f383dbb37cc..8e9bf4b214b 100644
--- a/airflow-core/pyproject.toml
+++ b/airflow-core/pyproject.toml
@@ -310,6 +310,10 @@ dev = [
docs = [
"apache-airflow-devel-common[docs]"
]
+mypy = [
+ "apache-airflow-devel-common[mypy]",
+]
+
[tool.uv]
diff --git a/airflow-ctl-tests/.pre-commit-config.yaml b/airflow-ctl-tests/.pre-commit-config.yaml
index f6213b0f9a8..1016fa939eb 100644
--- a/airflow-ctl-tests/.pre-commit-config.yaml
+++ b/airflow-ctl-tests/.pre-commit-config.yaml
@@ -16,7 +16,7 @@
# under the License.
---
default_stages: [pre-commit, pre-push]
-minimum_prek_version: '0.2.0'
+minimum_prek_version: '0.3.4'
default_language_version:
python: python3
repos:
@@ -30,3 +30,10 @@ repos:
files:
(?x)
^tests/airflowctl_tests/.*\.py$
+ - id: mypy-airflow-ctl-tests
+ name: Run mypy for airflow-ctl-tests
+ language: python
+ entry: ../scripts/ci/prek/run_mypy_full_dist_local_venv_or_breeze_in_ci.py airflow-ctl-tests
+ pass_filenames: false
+ files: ^.*\.py$
+ require_serial: true
diff --git a/airflow-ctl-tests/pyproject.toml b/airflow-ctl-tests/pyproject.toml
index 230c5378bfd..d7857e0cb81 100644
--- a/airflow-ctl-tests/pyproject.toml
+++ b/airflow-ctl-tests/pyproject.toml
@@ -76,3 +76,8 @@ exclude = ["*"]
[tool.hatch.build.targets.wheel]
bypass-selection = true
+
+[dependency-groups]
+mypy = [
+ "apache-airflow-devel-common[mypy]",
+]
diff --git a/airflow-ctl/.pre-commit-config.yaml b/airflow-ctl/.pre-commit-config.yaml
index a5773e94aab..4561533c574 100644
--- a/airflow-ctl/.pre-commit-config.yaml
+++ b/airflow-ctl/.pre-commit-config.yaml
@@ -16,7 +16,7 @@
# under the License.
---
default_stages: [pre-commit, pre-push]
-minimum_prek_version: '0.2.0'
+minimum_prek_version: '0.3.4'
default_language_version:
python: python3
node: 22.19.0
@@ -27,7 +27,7 @@ repos:
- id: mypy-airflow-ctl
name: Run mypy for airflow-ctl
language: python
- entry: ../scripts/ci/prek/mypy_local_folder.py airflow-ctl
+ entry: ../scripts/ci/prek/run_mypy_full_dist_local_venv_or_breeze_in_ci.py airflow-ctl
pass_filenames: false
files: ^.*\.py$
require_serial: true
diff --git a/airflow-ctl/pyproject.toml b/airflow-ctl/pyproject.toml
index 1ab36e367e5..1e3ef8f0181 100644
--- a/airflow-ctl/pyproject.toml
+++ b/airflow-ctl/pyproject.toml
@@ -152,6 +152,10 @@ codegen = [
]
# uv run --verbose --group codegen --project apache-airflow-ctl --directory airflow-ctl/ datamodel-codegen --url="http://0.0.0.0:28080/auth/openapi.json" --output=src/airflowctl/api/datamodels/auth_generated.py
+mypy = [
+ "apache-airflow-devel-common[mypy]",
+]
+
[tool.datamodel-codegen]
capitalise-enum-members=true # `State.RUNNING` not `State.running`
disable-timestamp=true
diff --git a/airflow-ctl-tests/.pre-commit-config.yaml b/airflow-e2e-tests/.pre-commit-config.yaml
similarity index 77%
copy from airflow-ctl-tests/.pre-commit-config.yaml
copy to airflow-e2e-tests/.pre-commit-config.yaml
index f6213b0f9a8..e3d6413706d 100644
--- a/airflow-ctl-tests/.pre-commit-config.yaml
+++ b/airflow-e2e-tests/.pre-commit-config.yaml
@@ -16,17 +16,16 @@
# under the License.
---
default_stages: [pre-commit, pre-push]
-minimum_prek_version: '0.2.0'
+minimum_prek_version: '0.3.4'
default_language_version:
python: python3
repos:
- repo: local
hooks:
- - id: check-airflowctl-command-coverage
- name: Check airflowctl CLI command test coverage
- entry: ../scripts/ci/prek/check_airflowctl_command_coverage.py
+ - id: mypy-airflow-e2e-tests
+ name: Run mypy for airflow-e2e-tests
language: python
+ entry: ../scripts/ci/prek/run_mypy_full_dist_local_venv_or_breeze_in_ci.py airflow-e2e-tests
pass_filenames: false
- files:
- (?x)
- ^tests/airflowctl_tests/.*\.py$
+ files: ^.*\.py$
+ require_serial: true
diff --git a/airflow-e2e-tests/pyproject.toml b/airflow-e2e-tests/pyproject.toml
index 3b4f61e550e..8eacba1565c 100644
--- a/airflow-e2e-tests/pyproject.toml
+++ b/airflow-e2e-tests/pyproject.toml
@@ -77,3 +77,8 @@ exclude = ["*"]
[tool.hatch.build.targets.wheel]
bypass-selection = true
+
+[dependency-groups]
+mypy = [
+ "apache-airflow-devel-common[mypy]",
+]
diff --git a/chart/.pre-commit-config.yaml b/chart/.pre-commit-config.yaml
index f95650c5324..16634adb4e1 100644
--- a/chart/.pre-commit-config.yaml
+++ b/chart/.pre-commit-config.yaml
@@ -16,7 +16,7 @@
# under the License.
---
default_stages: [pre-commit, pre-push]
-minimum_prek_version: '0.2.0'
+minimum_prek_version: '0.3.4'
default_language_version:
python: python3
node: 22.19.0
diff --git a/contributing-docs/08_static_code_checks.rst b/contributing-docs/08_static_code_checks.rst
index 4eed1a98a13..01c5843677c 100644
--- a/contributing-docs/08_static_code_checks.rst
+++ b/contributing-docs/08_static_code_checks.rst
@@ -281,26 +281,39 @@ Mypy checks
-----------
When we run mypy checks locally, the ``mypy-*`` checks run depending on the files you are changing:
-``mypy-airflow-core``, ``mypy-dev``, ``mypy-providers``, ``mypy-task-sdk``, ``mypy-airflow-ctl``, etc.
+``mypy-airflow-core``, ``mypy-dev``, ``mypy-providers``, ``mypy-scripts``, ``mypy-task-sdk``,
+``mypy-airflow-ctl``, ``mypy-devel-common``, ``mypy-airflow-ctl-tests``, ``mypy-helm-tests``,
+``mypy-airflow-e2e-tests``, ``mypy-task-sdk-integration-tests``, ``mypy-docker-tests``,
+``mypy-kubernetes-tests``, and one ``mypy-shared-<dist>`` hook per ``shared/<dist>`` workspace
+distribution (e.g. ``mypy-shared-configuration``, ``mypy-shared-logging``).
-For **non-provider projects** (airflow-core, task-sdk, airflow-ctl, dev, scripts, devel-common), mypy
-runs locally using the ``uv`` virtualenv — no breeze CI image is needed. These checks run as regular
-prek hooks in the ``pre-commit`` stage, checking whole directories at once. This means they run both
-as part of local commits and as part of regular static checks in CI (not as separate mypy CI jobs).
+For **non-provider projects**, mypy runs locally using ``uv`` — no breeze CI image is needed. These
+checks run as regular prek hooks in the ``pre-commit`` stage, checking whole directories at once. This
+means they run both as part of local commits and as part of regular static checks in CI (not as
+separate mypy CI jobs).
-Before running mypy directly (or via the ``mypy-*`` prek hooks), synchronize your local virtualenv
-with ``uv.lock`` so it matches the dependency set CI uses — otherwise mypy may pick up a different
-set of installed packages than CI and produce results that diverge from CI:
+Each non-provider ``mypy-*`` hook uses a **dedicated virtualenv and mypy cache** under ``.build/`` so
+running mypy never mutates your regular project ``.venv`` and each hook keeps a stable, CI-aligned
+dependency set:
-.. code-block:: bash
+- virtualenvs: ``.build/mypy-venvs/<hook-name>/``
+- mypy caches: ``.build/mypy-caches/<hook-name>/``
- uv sync --frozen --project <PROJECT>
+Adding a new shared library
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Then run mypy directly. Use ``--frozen`` so ``uv`` does not update ``uv.lock``:
+Every ``shared/<dist>`` workspace member has its own ``mypy-shared-<dist>`` prek hook so it is
+type-checked in isolation against its own dependency set. When you add a new shared library under
+``shared/<new-dist>/``, you also need to:
-.. code-block:: bash
+1. Add a ``[dependency-groups]`` section with ``mypy = ["apache-airflow-devel-common[mypy]"]`` in
+ ``shared/<new-dist>/pyproject.toml`` (so ``uv sync --group mypy`` installs mypy into the hook's
+ dedicated virtualenv).
+2. Create ``shared/<new-dist>/.pre-commit-config.yaml`` with a ``mypy-shared-<new-dist>`` hook
+ entry that calls ``../../scripts/ci/prek/run_mypy_full_dist_local_venv_or_breeze_in_ci.py shared/<new-dist>``.
- uv run --frozen --project <PROJECT> --with "apache-airflow-devel-common[mypy]" mypy path/to/code
+The ``check-shared-mypy-hooks`` prek hook enforces step 2 — it fails and prints the exact config
+contents to add when any ``shared/<dist>`` is missing its dedicated mypy hook.
To run the prek hook for a specific project (example for ``airflow-core`` files):
@@ -308,20 +321,19 @@ To run the prek hook for a specific project (example for ``airflow-core`` files)
prek mypy-airflow-core --all-files
-To show unused mypy ignores for any providers/airflow etc, eg: run below command:
+To show unused mypy ignores, run:
.. code-block:: bash
export SHOW_UNUSED_MYPY_WARNINGS=true
prek mypy-airflow-core --all-files
-For non-provider projects, the local mypy cache is stored in ``.mypy_cache`` at the repo root.
-
For **providers**, mypy still runs via breeze (``breeze run mypy``) as a separate CI job and requires
``breeze ci-image build --python 3.10`` to be built locally. Providers use a separate docker-volume
(called ``mypy-cache-volume``) that keeps the cache of last MyPy execution.
-To clear all mypy caches (both local ``.mypy_cache`` and the Docker volume), run
+To clear all mypy caches (the Docker volume used by providers, any legacy repo-root ``.mypy_cache``,
+and the per-hook venvs + caches under ``.build/mypy-venvs/`` and ``.build/mypy-caches/``), run
``breeze down --cleanup-mypy-cache``.
-----------
diff --git a/airflow-ctl-tests/.pre-commit-config.yaml b/dev/.pre-commit-config.yaml
similarity index 77%
copy from airflow-ctl-tests/.pre-commit-config.yaml
copy to dev/.pre-commit-config.yaml
index f6213b0f9a8..f401bd97545 100644
--- a/airflow-ctl-tests/.pre-commit-config.yaml
+++ b/dev/.pre-commit-config.yaml
@@ -16,17 +16,16 @@
# under the License.
---
default_stages: [pre-commit, pre-push]
-minimum_prek_version: '0.2.0'
+minimum_prek_version: '0.3.4'
default_language_version:
python: python3
repos:
- repo: local
hooks:
- - id: check-airflowctl-command-coverage
- name: Check airflowctl CLI command test coverage
- entry: ../scripts/ci/prek/check_airflowctl_command_coverage.py
+ - id: mypy-dev
+ name: Run mypy for dev
language: python
+ entry: ../scripts/ci/prek/run_mypy_full_dist_local_venv_or_breeze_in_ci.py dev
pass_filenames: false
- files:
- (?x)
- ^tests/airflowctl_tests/.*\.py$
+ files: ^.*\.py$
+ require_serial: true
diff --git a/dev/breeze/doc/03_developer_tasks.rst b/dev/breeze/doc/03_developer_tasks.rst
index 0db3d77fed2..37dff9045ad 100644
--- a/dev/breeze/doc/03_developer_tasks.rst
+++ b/dev/breeze/doc/03_developer_tasks.rst
@@ -321,7 +321,7 @@ For example, this following command:
.. code-block:: bash
- prek mypy-airflow
+ prek mypy-airflow-core
will run mypy check for currently staged files inside ``airflow/`` excluding providers.
.. _breeze-dev:running-prek-in-breeze:
@@ -349,7 +349,7 @@ re-run latest prek hooks on your changes, but it can take a long time (few minut
.. code-block:: bash
- prek mypy-airflow --all-files
+ prek mypy-airflow-core --all-files
The above will run mypy check for all files.
@@ -358,7 +358,7 @@ specifying (can be multiple times) ``--file`` flag.
.. code-block:: bash
- prek mypy-airflow --file airflow/utils/code_utils.py --file airflow/utils/timeout.py
+ prek mypy-airflow-core --file airflow/utils/code_utils.py --file airflow/utils/timeout.py
The above will run mypy check for those to files (note: autocomplete should work for the file selection).
@@ -370,7 +370,7 @@ of commits you choose.
.. code-block:: bash
- prek mypy-airflow --last-commit
+ prek mypy-airflow-core --last-commit
The above will run mypy check for all files in the last commit in your branch.
@@ -383,9 +383,17 @@ in ``--from-ref`` and ``--to-ref`` flags.
.. note::
- When you run static checks, some of the artifacts (mypy_cache) is stored in docker-compose volume
- so that it can speed up static checks execution significantly. However, sometimes, the cache might
- get broken, in which case you should run ``breeze down`` to clean up the cache.
+ When you run static checks, some of the artifacts (mypy_cache) is stored to speed up static
+ checks execution significantly:
+
+ - The providers ``mypy-providers`` hook runs via Breeze and stores its cache in the
+ ``mypy-cache-volume`` docker-compose volume.
+ - Each non-provider ``mypy-*`` hook uses its own dedicated virtualenv and mypy cache under
+ ``.build/mypy-venvs/<hook>/`` and ``.build/mypy-caches/<hook>/``; mypy itself is installed
+ from the workspace ``uv.lock`` via the ``mypy`` dependency group (``uv sync --group mypy``).
+
+ If the cache gets broken, run ``breeze down --cleanup-mypy-cache`` which wipes the docker
+ volume and every per-hook ``.build/mypy-venvs/`` and ``.build/mypy-caches/`` directory.
.. note::
diff --git a/dev/breeze/doc/ci/04_selective_checks.md b/dev/breeze/doc/ci/04_selective_checks.md
index ab20ef96add..0290bd9ee53 100644
--- a/dev/breeze/doc/ci/04_selective_checks.md
+++ b/dev/breeze/doc/ci/04_selective_checks.md
@@ -68,9 +68,20 @@ We have the following Groups of files for CI that determine which tests are run:
* `All Python files` - if none of the Python file changed, that indicates that we should not run unit tests
* `All source files` - if none of the sources change, that indicates that we should probably not build
an image and run any image-based static checks
-* `All Airflow Python files` - files that are checked by `mypy-airflow` static checks
+* `All Airflow Python files` - files that are checked by `mypy-airflow-core` static checks
* `All Providers Python files` - files that are checked by `mypy-providers` static checks
* `All Dev Python files` - files that are checked by `mypy-dev` static checks
+* `All Scripts Python files` - files that are checked by `mypy-scripts` static checks
+* `Task SDK files` - files that are checked by `mypy-task-sdk` static checks
+* `All Airflow CTL Python files` - files that are checked by `mypy-airflow-ctl` static checks
+* `All Devel Common Python files` - files that are checked by `mypy-devel-common` static checks
+* `All Helm Tests Python files` / `All Docker Tests Python files` /
+ `All Kubernetes Tests Python files` / `All Airflow E2E Tests Python files` /
+ `Airflow CTL Integration Test files` / `Task SDK Integration Test files` - files that are
+ checked by the respective `mypy-helm-tests` / `mypy-docker-tests` / `mypy-kubernetes-tests` /
+ `mypy-airflow-e2e-tests` / `mypy-airflow-ctl-tests` / `mypy-task-sdk-integration-tests` hooks
+* `shared/<dist>/**/*.py` - each `shared/<dist>` workspace member has its own `mypy-shared-<dist>`
+ hook (selective-checks enumerates them at runtime)
* `All Provider Yaml files` - all provider yaml files
We have a number of `TEST_TYPES` that can be selectively disabled/enabled based on the
@@ -140,8 +151,17 @@ when some files are not changed. Those are the rules implemented:
* If "full tests" mode is detected, no more prek hooks are skipped - we run all of them
* The following checks are skipped if those files are not changed:
* if no `All Providers Python files` changed - `mypy-providers` check is skipped
- * if no `All Airflow Python files` changed - `mypy-airflow` check is skipped
+ * if no `All Airflow Python files` changed - `mypy-airflow-core` check is skipped
* if no `All Dev Python files` changed - `mypy-dev` check is skipped
+ * if no `All Scripts Python files` changed - `mypy-scripts` check is skipped
+ * if no `Task SDK files` changed - `mypy-task-sdk` check is skipped
+ * if no `All Airflow CTL Python files` changed - `mypy-airflow-ctl` check is skipped
+ * if no `All Devel Common Python files` changed - `mypy-devel-common` check is skipped
+ * if no files under the matching folder changed, the corresponding per-folder mypy hook
+ is skipped (`mypy-airflow-ctl-tests`, `mypy-helm-tests`, `mypy-airflow-e2e-tests`,
+ `mypy-task-sdk-integration-tests`, `mypy-docker-tests`, `mypy-kubernetes-tests`)
+ * for each `shared/<dist>` workspace member, `mypy-shared-<dist>` is skipped when no
+ file under `shared/<dist>/` changed (enumerated at runtime)
* if no `UI files` changed - `ts-compile-format-lint-ui` check is skipped
* if no `WWW files` changed - `ts-compile-format-lint-www` check is skipped
* if no `All Python files` changed - `flynt` check is skipped
diff --git a/dev/breeze/src/airflow_breeze/commands/developer_commands.py b/dev/breeze/src/airflow_breeze/commands/developer_commands.py
index f10ed0ec0bd..afb22da6355 100644
--- a/dev/breeze/src/airflow_breeze/commands/developer_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/developer_commands.py
@@ -936,6 +936,11 @@ def down(preserve_volumes: bool, cleanup_mypy_cache: bool, cleanup_build_cache:
if local_mypy_cache.exists():
console_print(f"\n[info]Removing local mypy cache: {local_mypy_cache}\n")
shutil.rmtree(local_mypy_cache)
+ for subdir in ("mypy-venvs", "mypy-caches"):
+ hook_dir = AIRFLOW_ROOT_PATH / ".build" / subdir
+ if hook_dir.exists():
+ console_print(f"\n[info]Removing dedicated mypy {subdir}: {hook_dir}\n")
+ shutil.rmtree(hook_dir)
if cleanup_build_cache:
command_to_execute = ["docker", "volume", "rm", "--force", "airflow-cache-volume"]
run_command(command_to_execute)
diff --git a/dev/breeze/src/airflow_breeze/commands/pr_commands.py b/dev/breeze/src/airflow_breeze/commands/pr_commands.py
index 5eddc266a7d..b59043febb4 100644
--- a/dev/breeze/src/airflow_breeze/commands/pr_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/pr_commands.py
@@ -3597,7 +3597,9 @@ def _prompt_and_execute_flagged_pr(
if action == TriageAction.CLOSE:
console_print(Panel(close_comment, title="Comment to be posted", border_style="red"))
elif action == TriageAction.COMMENT:
- console_print(Panel(comment_only_text, title="Comment to be posted", border_style="green"))
+ console_print(
+ Panel(comment_only_text or "", title="Comment to be posted", border_style="green")
+ )
else:
console_print(Panel(draft_comment, title="Comment to be posted", border_style="green"))
@@ -4759,7 +4761,7 @@ def _run_tui_triage(
except Exception:
result = None
if result:
- diff_cache[num] = result
+ diff_cache[num] = result # type: ignore[assignment]
paths = extract_file_paths_from_diff(result)
if paths:
_pr_file_paths[num] = paths
@@ -6382,7 +6384,7 @@ def _review_deterministic_flagged_prs(
if pr.author_login != current_author:
current_author = pr.author_login
- count = ctx.author_flagged_count[current_author]
+ count = ctx.author_flagged_count[current_author] if current_author else 0
console_print()
get_console().rule(
f"[bold]Author: {current_author}[/] ({count} PR{'s' if count != 1 else ''} with issues)",
diff --git a/dev/breeze/src/airflow_breeze/commands/release_candidate_command.py b/dev/breeze/src/airflow_breeze/commands/release_candidate_command.py
index 651aa803dd6..f3401b6074f 100644
--- a/dev/breeze/src/airflow_breeze/commands/release_candidate_command.py
+++ b/dev/breeze/src/airflow_breeze/commands/release_candidate_command.py
@@ -349,6 +349,9 @@ def create_tarball_release(
console_print(f"[error]Unsupported tarball type: {tarball_type}")
exit(1)
source_date_epoch = get_source_date_epoch(AIRFLOW_ROOT_PATH)
+ if version is None:
+ console_print("[error]Version must be set before creating the tarball")
+ exit(1)
# Create the tarball
tarball_release(
version=version,
diff --git a/dev/breeze/src/airflow_breeze/commands/testing_commands.py b/dev/breeze/src/airflow_breeze/commands/testing_commands.py
index eabef38ce52..a29b1471ecf 100644
--- a/dev/breeze/src/airflow_breeze/commands/testing_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/testing_commands.py
@@ -1454,7 +1454,7 @@ def airflow_e2e_tests(
console_print(f"[info]Running Airflow E2E tests with PROD image: {image_name}[/]")
# If the image is used from docker hub, test container will pull that part of test.
- skip_image_check = True if image_name.startswith("apache/airflow") else False
+ skip_image_check = bool(image_name and image_name.startswith("apache/airflow"))
return_code, info = run_docker_compose_tests(
image_name=image_name,
python_version=python,
diff --git a/dev/breeze/src/airflow_breeze/utils/reproducible.py b/dev/breeze/src/airflow_breeze/utils/reproducible.py
index a48f6b048b8..218598dd660 100644
--- a/dev/breeze/src/airflow_breeze/utils/reproducible.py
+++ b/dev/breeze/src/airflow_breeze/utils/reproducible.py
@@ -123,7 +123,7 @@ def repack_deterministically(
temp_file = f"{dest_archive}.temp~"
with os.fdopen(os.open(temp_file, os.O_WRONLY | os.O_CREAT, 0o644), "wb") as out_file:
with gzip.GzipFile(fileobj=out_file, mtime=0, mode="wb") as gzip_file:
- with tarfile.open(fileobj=gzip_file, mode="w:") as tar_file:
+ with tarfile.open(fileobj=gzip_file, mode="w:") as tar_file: # type: ignore[arg-type]
for entry in file_list:
entry_path = Path(entry)
if not entry_path.is_symlink():
diff --git a/dev/breeze/src/airflow_breeze/utils/selective_checks.py b/dev/breeze/src/airflow_breeze/utils/selective_checks.py
index 5b399ed3d8c..44896190c55 100644
--- a/dev/breeze/src/airflow_breeze/utils/selective_checks.py
+++ b/dev/breeze/src/airflow_breeze/utils/selective_checks.py
@@ -136,6 +136,11 @@ class FileGroupForCi(Enum):
ALL_PROVIDERS_DISTRIBUTION_CONFIG_FILES = auto()
ALL_DEV_PYTHON_FILES = auto()
ALL_DEVEL_COMMON_PYTHON_FILES = auto()
+ ALL_SCRIPTS_PYTHON_FILES = auto()
+ ALL_HELM_TESTS_PYTHON_FILES = auto()
+ ALL_AIRFLOW_E2E_TESTS_PYTHON_FILES = auto()
+ ALL_DOCKER_TESTS_PYTHON_FILES = auto()
+ ALL_KUBERNETES_TESTS_PYTHON_FILES = auto()
ALL_PROVIDER_YAML_FILES = auto()
TESTS_UTILS_FILES = auto()
ASSET_FILES = auto()
@@ -298,6 +303,21 @@ CI_FILE_GROUP_MATCHES: HashableDict[FileGroupForCi] = HashableDict(
FileGroupForCi.ALL_DEVEL_COMMON_PYTHON_FILES: [
r"^devel-common/.*\.py$",
],
+ FileGroupForCi.ALL_SCRIPTS_PYTHON_FILES: [
+ r"^scripts/.*\.py$",
+ ],
+ FileGroupForCi.ALL_HELM_TESTS_PYTHON_FILES: [
+ r"^helm-tests/.*\.py$",
+ ],
+ FileGroupForCi.ALL_AIRFLOW_E2E_TESTS_PYTHON_FILES: [
+ r"^airflow-e2e-tests/.*\.py$",
+ ],
+ FileGroupForCi.ALL_DOCKER_TESTS_PYTHON_FILES: [
+ r"^docker-tests/.*\.py$",
+ ],
+ FileGroupForCi.ALL_KUBERNETES_TESTS_PYTHON_FILES: [
+ r"^kubernetes-tests/.*\.py$",
+ ],
FileGroupForCi.ALL_SOURCE_FILES: [
r"^.pre-commit-config.yaml$",
r"^airflow-core/src/.*",
@@ -1486,12 +1506,42 @@ class SelectiveChecks:
prek_hooks_to_skip.add("mypy-airflow-core")
if not self._matching_files(FileGroupForCi.ALL_DEV_PYTHON_FILES, CI_FILE_GROUP_MATCHES):
prek_hooks_to_skip.add("mypy-dev")
+ if not self._matching_files(FileGroupForCi.ALL_SCRIPTS_PYTHON_FILES, CI_FILE_GROUP_MATCHES):
+ prek_hooks_to_skip.add("mypy-scripts")
if not self._matching_files(FileGroupForCi.TASK_SDK_FILES, CI_FILE_GROUP_MATCHES):
prek_hooks_to_skip.add("mypy-task-sdk")
if not self._matching_files(FileGroupForCi.ALL_DEVEL_COMMON_PYTHON_FILES, CI_FILE_GROUP_MATCHES):
prek_hooks_to_skip.add("mypy-devel-common")
if not self._matching_files(FileGroupForCi.ALL_AIRFLOW_CTL_PYTHON_FILES, CI_FILE_GROUP_MATCHES):
prek_hooks_to_skip.add("mypy-airflow-ctl")
+ if not self._matching_files(
+ FileGroupForCi.AIRFLOW_CTL_INTEGRATION_TEST_FILES, CI_FILE_GROUP_MATCHES
+ ):
+ prek_hooks_to_skip.add("mypy-airflow-ctl-tests")
+ if not self._matching_files(FileGroupForCi.ALL_HELM_TESTS_PYTHON_FILES, CI_FILE_GROUP_MATCHES):
+ prek_hooks_to_skip.add("mypy-helm-tests")
+ if not self._matching_files(
+ FileGroupForCi.ALL_AIRFLOW_E2E_TESTS_PYTHON_FILES, CI_FILE_GROUP_MATCHES
+ ):
+ prek_hooks_to_skip.add("mypy-airflow-e2e-tests")
+ if not self._matching_files(
+ FileGroupForCi.TASK_SDK_INTEGRATION_TEST_FILES, CI_FILE_GROUP_MATCHES
+ ):
+ prek_hooks_to_skip.add("mypy-task-sdk-integration-tests")
+ if not self._matching_files(FileGroupForCi.ALL_DOCKER_TESTS_PYTHON_FILES, CI_FILE_GROUP_MATCHES):
+ prek_hooks_to_skip.add("mypy-docker-tests")
+ if not self._matching_files(
+ FileGroupForCi.ALL_KUBERNETES_TESTS_PYTHON_FILES, CI_FILE_GROUP_MATCHES
+ ):
+ prek_hooks_to_skip.add("mypy-kubernetes-tests")
+ # One mypy-shared-<dist> hook per shared/<dist> workspace member; each is only
+ # kept when that distribution's own files changed.
+ for dist in sorted(
+ p.name for p in (AIRFLOW_ROOT_PATH / "shared").iterdir() if (p / "pyproject.toml").exists()
+ ):
+ pattern = re.compile(rf"^shared/{re.escape(dist)}/.*\.py$")
+ if not any(pattern.match(f) for f in self._files):
+ prek_hooks_to_skip.add(f"mypy-shared-{dist}")
return ",".join(sorted(prek_hooks_to_skip))
@cached_property
diff --git a/dev/breeze/tests/test_selective_checks.py b/dev/breeze/tests/test_selective_checks.py
index 24c2f9befa6..cf402d8818f 100644
--- a/dev/breeze/tests/test_selective_checks.py
+++ b/dev/breeze/tests/test_selective_checks.py
@@ -101,54 +101,108 @@ LIST_OF_ALL_PROVIDER_TESTS_AS_JSON = json.dumps(
ALL_SKIPPED_COMMITS_ON_NO_CI_IMAGE = (
"check-provider-yaml-valid,flynt,identity,lint-helm-chart,"
- "mypy-airflow-core,mypy-airflow-ctl,mypy-dev,mypy-devel-common,mypy-task-sdk,"
+ "mypy-airflow-core,mypy-airflow-ctl,mypy-airflow-ctl-tests,mypy-airflow-e2e-tests,"
+ "mypy-dev,mypy-devel-common,mypy-docker-tests,mypy-helm-tests,mypy-kubernetes-tests,"
+ "mypy-scripts,"
+ "mypy-shared-configuration,mypy-shared-dagnode,mypy-shared-listeners,mypy-shared-logging,"
+ "mypy-shared-module_loading,mypy-shared-observability,mypy-shared-plugins_manager,"
+ "mypy-shared-providers_discovery,mypy-shared-secrets_backend,mypy-shared-secrets_masker,"
+ "mypy-shared-serialization,mypy-shared-template_rendering,mypy-shared-timezones,mypy-task-sdk,mypy-task-sdk-integration-tests,"
"ts-compile-lint-simple-auth-manager-ui,ts-compile-lint-ui,update-uv-lock"
)
ALL_SKIPPED_COMMITS_BY_DEFAULT_ON_ALL_TESTS_NEEDED = "identity,update-uv-lock"
ALL_SKIPPED_COMMITS_IF_NO_UI = (
- "identity,mypy-airflow-core,mypy-airflow-ctl,mypy-dev,mypy-devel-common,mypy-task-sdk,"
+ "identity,mypy-airflow-core,mypy-airflow-ctl,mypy-airflow-ctl-tests,mypy-airflow-e2e-tests,"
+ "mypy-dev,mypy-devel-common,mypy-docker-tests,mypy-helm-tests,mypy-kubernetes-tests,"
+ "mypy-scripts,"
+ "mypy-shared-configuration,mypy-shared-dagnode,mypy-shared-listeners,mypy-shared-logging,"
+ "mypy-shared-module_loading,mypy-shared-observability,mypy-shared-plugins_manager,"
+ "mypy-shared-providers_discovery,mypy-shared-secrets_backend,mypy-shared-secrets_masker,"
+ "mypy-shared-serialization,mypy-shared-template_rendering,mypy-shared-timezones,mypy-task-sdk,mypy-task-sdk-integration-tests,"
"ts-compile-lint-simple-auth-manager-ui,ts-compile-lint-ui,update-uv-lock"
)
ALL_SKIPPED_COMMITS_IF_NO_HELM_TESTS = (
"identity,lint-helm-chart,"
- "mypy-airflow-core,mypy-airflow-ctl,mypy-dev,mypy-devel-common,mypy-task-sdk,update-uv-lock"
+ "mypy-airflow-core,mypy-airflow-ctl,mypy-airflow-ctl-tests,mypy-airflow-e2e-tests,"
+ "mypy-dev,mypy-devel-common,mypy-docker-tests,mypy-helm-tests,mypy-kubernetes-tests,"
+ "mypy-scripts,"
+ "mypy-shared-configuration,mypy-shared-dagnode,mypy-shared-listeners,mypy-shared-logging,"
+ "mypy-shared-module_loading,mypy-shared-observability,mypy-shared-plugins_manager,"
+ "mypy-shared-providers_discovery,mypy-shared-secrets_backend,mypy-shared-secrets_masker,"
+ "mypy-shared-serialization,mypy-shared-template_rendering,mypy-shared-timezones,mypy-task-sdk,mypy-task-sdk-integration-tests,update-uv-lock"
)
ALL_SKIPPED_COMMITS_IF_NO_UI_AND_HELM_TESTS = (
"identity,lint-helm-chart,"
- "mypy-airflow-core,mypy-airflow-ctl,mypy-dev,mypy-devel-common,mypy-task-sdk,"
+ "mypy-airflow-core,mypy-airflow-ctl,mypy-airflow-ctl-tests,mypy-airflow-e2e-tests,"
+ "mypy-dev,mypy-devel-common,mypy-docker-tests,mypy-helm-tests,mypy-kubernetes-tests,"
+ "mypy-scripts,"
+ "mypy-shared-configuration,mypy-shared-dagnode,mypy-shared-listeners,mypy-shared-logging,"
+ "mypy-shared-module_loading,mypy-shared-observability,mypy-shared-plugins_manager,"
+ "mypy-shared-providers_discovery,mypy-shared-secrets_backend,mypy-shared-secrets_masker,"
+ "mypy-shared-serialization,mypy-shared-template_rendering,mypy-shared-timezones,mypy-task-sdk,mypy-task-sdk-integration-tests,"
"ts-compile-lint-simple-auth-manager-ui,ts-compile-lint-ui,update-uv-lock"
)
ALL_SKIPPED_COMMITS_IF_NO_PROVIDERS_AND_UI = (
"check-provider-yaml-valid,identity,"
- "mypy-airflow-core,mypy-airflow-ctl,mypy-dev,mypy-devel-common,mypy-task-sdk,"
+ "mypy-airflow-core,mypy-airflow-ctl,mypy-airflow-ctl-tests,mypy-airflow-e2e-tests,"
+ "mypy-dev,mypy-devel-common,mypy-docker-tests,mypy-helm-tests,mypy-kubernetes-tests,"
+ "mypy-scripts,"
+ "mypy-shared-configuration,mypy-shared-dagnode,mypy-shared-listeners,mypy-shared-logging,"
+ "mypy-shared-module_loading,mypy-shared-observability,mypy-shared-plugins_manager,"
+ "mypy-shared-providers_discovery,mypy-shared-secrets_backend,mypy-shared-secrets_masker,"
+ "mypy-shared-serialization,mypy-shared-template_rendering,mypy-shared-timezones,mypy-task-sdk,mypy-task-sdk-integration-tests,"
"ts-compile-lint-simple-auth-manager-ui,ts-compile-lint-ui,update-uv-lock"
)
ALL_SKIPPED_COMMITS_IF_NO_PROVIDERS = (
"check-provider-yaml-valid,identity,lint-helm-chart,"
- "mypy-airflow-core,mypy-airflow-ctl,mypy-dev,mypy-devel-common,mypy-task-sdk,"
+ "mypy-airflow-core,mypy-airflow-ctl,mypy-airflow-ctl-tests,mypy-airflow-e2e-tests,"
+ "mypy-dev,mypy-devel-common,mypy-docker-tests,mypy-helm-tests,mypy-kubernetes-tests,"
+ "mypy-scripts,"
+ "mypy-shared-configuration,mypy-shared-dagnode,mypy-shared-listeners,mypy-shared-logging,"
+ "mypy-shared-module_loading,mypy-shared-observability,mypy-shared-plugins_manager,"
+ "mypy-shared-providers_discovery,mypy-shared-secrets_backend,mypy-shared-secrets_masker,"
+ "mypy-shared-serialization,mypy-shared-template_rendering,mypy-shared-timezones,mypy-task-sdk,mypy-task-sdk-integration-tests,"
"ts-compile-lint-simple-auth-manager-ui,ts-compile-lint-ui,update-uv-lock"
)
ALL_SKIPPED_COMMITS_IF_NO_PROVIDERS_UI_AND_HELM_TESTS = (
"check-provider-yaml-valid,identity,lint-helm-chart,"
- "mypy-airflow-core,mypy-airflow-ctl,mypy-dev,mypy-devel-common,mypy-task-sdk,"
+ "mypy-airflow-core,mypy-airflow-ctl,mypy-airflow-ctl-tests,mypy-airflow-e2e-tests,"
+ "mypy-dev,mypy-devel-common,mypy-docker-tests,mypy-helm-tests,mypy-kubernetes-tests,"
+ "mypy-scripts,"
+ "mypy-shared-configuration,mypy-shared-dagnode,mypy-shared-listeners,mypy-shared-logging,"
+ "mypy-shared-module_loading,mypy-shared-observability,mypy-shared-plugins_manager,"
+ "mypy-shared-providers_discovery,mypy-shared-secrets_backend,mypy-shared-secrets_masker,"
+ "mypy-shared-serialization,mypy-shared-template_rendering,mypy-shared-timezones,mypy-task-sdk,mypy-task-sdk-integration-tests,"
"ts-compile-lint-simple-auth-manager-ui,ts-compile-lint-ui,update-uv-lock"
)
ALL_SKIPPED_COMMITS_IF_NO_CODE_PROVIDERS_AND_HELM_TESTS = (
"check-provider-yaml-valid,flynt,identity,lint-helm-chart,"
- "mypy-airflow-core,mypy-airflow-ctl,mypy-dev,mypy-devel-common,mypy-task-sdk,update-uv-lock"
+ "mypy-airflow-core,mypy-airflow-ctl,mypy-airflow-ctl-tests,mypy-airflow-e2e-tests,"
+ "mypy-dev,mypy-devel-common,mypy-docker-tests,mypy-helm-tests,mypy-kubernetes-tests,"
+ "mypy-scripts,"
+ "mypy-shared-configuration,mypy-shared-dagnode,mypy-shared-listeners,mypy-shared-logging,"
+ "mypy-shared-module_loading,mypy-shared-observability,mypy-shared-plugins_manager,"
+ "mypy-shared-providers_discovery,mypy-shared-secrets_backend,mypy-shared-secrets_masker,"
+ "mypy-shared-serialization,mypy-shared-template_rendering,mypy-shared-timezones,mypy-task-sdk,mypy-task-sdk-integration-tests,update-uv-lock"
)
ALL_SKIPPED_COMMITS_IF_NOT_IMPORTANT_FILES_CHANGED = (
"check-provider-yaml-valid,flynt,identity,lint-helm-chart,"
- "mypy-airflow-core,mypy-airflow-ctl,mypy-dev,mypy-devel-common,mypy-task-sdk,"
+ "mypy-airflow-core,mypy-airflow-ctl,mypy-airflow-ctl-tests,mypy-airflow-e2e-tests,"
+ "mypy-dev,mypy-devel-common,mypy-docker-tests,mypy-helm-tests,mypy-kubernetes-tests,"
+ "mypy-scripts,"
+ "mypy-shared-configuration,mypy-shared-dagnode,mypy-shared-listeners,mypy-shared-logging,"
+ "mypy-shared-module_loading,mypy-shared-observability,mypy-shared-plugins_manager,"
+ "mypy-shared-providers_discovery,mypy-shared-secrets_backend,mypy-shared-secrets_masker,"
+ "mypy-shared-serialization,mypy-shared-template_rendering,mypy-shared-timezones,mypy-task-sdk,mypy-task-sdk-integration-tests,"
"ts-compile-lint-simple-auth-manager-ui,ts-compile-lint-ui,update-uv-lock"
)
@@ -345,7 +399,13 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
"docs-build": "true",
"skip-prek-hooks": (
"check-provider-yaml-valid,identity,lint-helm-chart,"
- "mypy-airflow-ctl,mypy-dev,mypy-devel-common,mypy-task-sdk,"
+ "mypy-airflow-ctl,mypy-airflow-ctl-tests,mypy-airflow-e2e-tests,"
+ "mypy-dev,mypy-devel-common,mypy-docker-tests,mypy-helm-tests,mypy-kubernetes-tests,"
+ "mypy-scripts,"
+ "mypy-shared-configuration,mypy-shared-dagnode,mypy-shared-listeners,mypy-shared-logging,"
+ "mypy-shared-module_loading,mypy-shared-observability,mypy-shared-plugins_manager,"
+ "mypy-shared-providers_discovery,mypy-shared-secrets_backend,mypy-shared-secrets_masker,"
+ "mypy-shared-serialization,mypy-shared-template_rendering,mypy-shared-timezones,mypy-task-sdk,mypy-task-sdk-integration-tests,"
"ts-compile-lint-simple-auth-manager-ui,ts-compile-lint-ui,update-uv-lock"
),
"upgrade-to-newer-dependencies": "false",
@@ -612,7 +672,13 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
"full-tests-needed": "false",
"skip-prek-hooks": (
"check-provider-yaml-valid,identity,lint-helm-chart,"
- "mypy-airflow-core,mypy-airflow-ctl,mypy-dev,mypy-devel-common,"
+ "mypy-airflow-core,mypy-airflow-ctl,mypy-airflow-ctl-tests,mypy-airflow-e2e-tests,"
+ "mypy-dev,mypy-devel-common,mypy-docker-tests,mypy-helm-tests,mypy-kubernetes-tests,"
+ "mypy-scripts,"
+ "mypy-shared-configuration,mypy-shared-dagnode,mypy-shared-listeners,mypy-shared-logging,"
+ "mypy-shared-module_loading,mypy-shared-observability,mypy-shared-plugins_manager,"
+ "mypy-shared-providers_discovery,mypy-shared-secrets_backend,mypy-shared-secrets_masker,"
+ "mypy-shared-serialization,mypy-shared-template_rendering,mypy-shared-timezones,mypy-task-sdk-integration-tests,"
"ts-compile-lint-simple-auth-manager-ui,ts-compile-lint-ui,update-uv-lock"
),
"skip-providers-tests": "false",
@@ -642,7 +708,17 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
"run-task-sdk-integration-tests": "true",
"docs-build": "false",
"full-tests-needed": "false",
- "skip-prek-hooks": ALL_SKIPPED_COMMITS_IF_NO_PROVIDERS_UI_AND_HELM_TESTS,
+ "skip-prek-hooks": (
+ "check-provider-yaml-valid,identity,lint-helm-chart,"
+ "mypy-airflow-core,mypy-airflow-ctl,mypy-airflow-ctl-tests,mypy-airflow-e2e-tests,"
+ "mypy-dev,mypy-devel-common,mypy-docker-tests,mypy-helm-tests,mypy-kubernetes-tests,"
+ "mypy-scripts,"
+ "mypy-shared-configuration,mypy-shared-dagnode,mypy-shared-listeners,mypy-shared-logging,"
+ "mypy-shared-module_loading,mypy-shared-observability,mypy-shared-plugins_manager,"
+ "mypy-shared-providers_discovery,mypy-shared-secrets_backend,mypy-shared-secrets_masker,"
+ "mypy-shared-serialization,mypy-shared-template_rendering,mypy-shared-timezones,mypy-task-sdk,"
+ "ts-compile-lint-simple-auth-manager-ui,ts-compile-lint-ui,update-uv-lock"
+ ),
"skip-providers-tests": "true",
"upgrade-to-newer-dependencies": "false",
"run-mypy-providers": "false",
@@ -671,7 +747,13 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
"full-tests-needed": "false",
"skip-prek-hooks": (
"check-provider-yaml-valid,identity,lint-helm-chart,"
- "mypy-airflow-core,mypy-dev,mypy-devel-common,mypy-task-sdk,"
+ "mypy-airflow-core,mypy-airflow-ctl-tests,mypy-airflow-e2e-tests,"
+ "mypy-dev,mypy-devel-common,mypy-docker-tests,mypy-helm-tests,mypy-kubernetes-tests,"
+ "mypy-scripts,"
+ "mypy-shared-configuration,mypy-shared-dagnode,mypy-shared-listeners,mypy-shared-logging,"
+ "mypy-shared-module_loading,mypy-shared-observability,mypy-shared-plugins_manager,"
+ "mypy-shared-providers_discovery,mypy-shared-secrets_backend,mypy-shared-secrets_masker,"
+ "mypy-shared-serialization,mypy-shared-template_rendering,mypy-shared-timezones,mypy-task-sdk,mypy-task-sdk-integration-tests,"
"ts-compile-lint-simple-auth-manager-ui,ts-compile-lint-ui,update-uv-lock"
),
"skip-providers-tests": "true",
@@ -699,7 +781,17 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
"run-airflow-ctl-integration-tests": "true",
"docs-build": "false",
"full-tests-needed": "false",
- "skip-prek-hooks": ALL_SKIPPED_COMMITS_IF_NO_PROVIDERS_UI_AND_HELM_TESTS,
+ "skip-prek-hooks": (
+ "check-provider-yaml-valid,identity,lint-helm-chart,"
+ "mypy-airflow-core,mypy-airflow-ctl,mypy-airflow-e2e-tests,"
+ "mypy-dev,mypy-devel-common,mypy-docker-tests,mypy-helm-tests,mypy-kubernetes-tests,"
+ "mypy-scripts,"
+ "mypy-shared-configuration,mypy-shared-dagnode,mypy-shared-listeners,mypy-shared-logging,"
+ "mypy-shared-module_loading,mypy-shared-observability,mypy-shared-plugins_manager,"
+ "mypy-shared-providers_discovery,mypy-shared-secrets_backend,mypy-shared-secrets_masker,"
+ "mypy-shared-serialization,mypy-shared-template_rendering,mypy-shared-timezones,mypy-task-sdk,mypy-task-sdk-integration-tests,"
+ "ts-compile-lint-simple-auth-manager-ui,ts-compile-lint-ui,update-uv-lock"
+ ),
"skip-providers-tests": "true",
"upgrade-to-newer-dependencies": "false",
"run-mypy-providers": "false",
@@ -1032,7 +1124,13 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
"run-kubernetes-tests": "false",
"skip-prek-hooks": (
"identity,lint-helm-chart,"
- "mypy-airflow-ctl,mypy-dev,mypy-devel-common,mypy-task-sdk,"
+ "mypy-airflow-ctl,mypy-airflow-ctl-tests,mypy-airflow-e2e-tests,"
+ "mypy-dev,mypy-devel-common,mypy-docker-tests,mypy-helm-tests,mypy-kubernetes-tests,"
+ "mypy-scripts,"
+ "mypy-shared-configuration,mypy-shared-dagnode,mypy-shared-listeners,mypy-shared-logging,"
+ "mypy-shared-module_loading,mypy-shared-observability,mypy-shared-plugins_manager,"
+ "mypy-shared-providers_discovery,mypy-shared-secrets_backend,mypy-shared-secrets_masker,"
+ "mypy-shared-serialization,mypy-shared-template_rendering,mypy-shared-timezones,mypy-task-sdk,mypy-task-sdk-integration-tests,"
"ts-compile-lint-simple-auth-manager-ui,ts-compile-lint-ui,update-uv-lock"
),
"upgrade-to-newer-dependencies": "false",
@@ -1190,7 +1288,13 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
"docs-build": "true",
"skip-prek-hooks": (
"check-provider-yaml-valid,flynt,identity,"
- "mypy-airflow-core,mypy-airflow-ctl,mypy-dev,mypy-devel-common,mypy-task-sdk,"
+ "mypy-airflow-core,mypy-airflow-ctl,mypy-airflow-ctl-tests,mypy-airflow-e2e-tests,"
+ "mypy-dev,mypy-devel-common,mypy-docker-tests,mypy-helm-tests,mypy-kubernetes-tests,"
+ "mypy-scripts,"
+ "mypy-shared-configuration,mypy-shared-dagnode,mypy-shared-listeners,mypy-shared-logging,"
+ "mypy-shared-module_loading,mypy-shared-observability,mypy-shared-plugins_manager,"
+ "mypy-shared-providers_discovery,mypy-shared-secrets_backend,mypy-shared-secrets_masker,"
+ "mypy-shared-serialization,mypy-shared-template_rendering,mypy-shared-timezones,mypy-task-sdk,mypy-task-sdk-integration-tests,"
"ts-compile-lint-simple-auth-manager-ui,ts-compile-lint-ui,update-uv-lock"
),
"upgrade-to-newer-dependencies": "false",
@@ -1337,6 +1441,27 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
id="Shared library python changes trigger unit tests",
)
),
+ (
+ pytest.param(
+ ("shared/logging/src/airflow_shared/logging/remote.py",),
+ {
+ "skip-prek-hooks": (
+ "check-provider-yaml-valid,identity,lint-helm-chart,"
+ "mypy-airflow-core,mypy-airflow-ctl,mypy-airflow-ctl-tests,"
+ "mypy-airflow-e2e-tests,mypy-dev,mypy-devel-common,mypy-docker-tests,"
+ "mypy-helm-tests,mypy-kubernetes-tests,mypy-scripts,"
+ "mypy-shared-configuration,mypy-shared-dagnode,mypy-shared-listeners,"
+ "mypy-shared-module_loading,mypy-shared-observability,"
+ "mypy-shared-plugins_manager,mypy-shared-providers_discovery,"
+ "mypy-shared-secrets_backend,mypy-shared-secrets_masker,"
+ "mypy-shared-serialization,mypy-shared-template_rendering,"
+ "mypy-shared-timezones,mypy-task-sdk,mypy-task-sdk-integration-tests,"
+ "ts-compile-lint-simple-auth-manager-ui,ts-compile-lint-ui,update-uv-lock"
+ ),
+ },
+ id=("Shared logging change keeps only mypy-shared-logging among the mypy-shared-* hooks"),
+ )
+ ),
],
)
def test_expected_output_pull_request_main(
@@ -2060,7 +2185,13 @@ def test_expected_output_push(
"docs-list-as-string": ALL_DOCS_SELECTED_FOR_BUILD,
"skip-prek-hooks": (
"check-provider-yaml-valid,identity,lint-helm-chart,"
- "mypy-airflow-ctl,mypy-dev,mypy-devel-common,mypy-task-sdk,"
+ "mypy-airflow-ctl,mypy-airflow-ctl-tests,mypy-airflow-e2e-tests,"
+ "mypy-dev,mypy-devel-common,mypy-docker-tests,mypy-helm-tests,mypy-kubernetes-tests,"
+ "mypy-scripts,"
+ "mypy-shared-configuration,mypy-shared-dagnode,mypy-shared-listeners,mypy-shared-logging,"
+ "mypy-shared-module_loading,mypy-shared-observability,mypy-shared-plugins_manager,"
+ "mypy-shared-providers_discovery,mypy-shared-secrets_backend,mypy-shared-secrets_masker,"
+ "mypy-shared-serialization,mypy-shared-template_rendering,mypy-shared-timezones,mypy-task-sdk,mypy-task-sdk-integration-tests,"
"ts-compile-lint-simple-auth-manager-ui,ts-compile-lint-ui,update-uv-lock"
),
"upgrade-to-newer-dependencies": "false",
@@ -2095,7 +2226,13 @@ def test_expected_output_push(
"microsoft.mssql mysql openlineage oracle postgres "
"presto salesforce samba sftp ssh standard trino",
"skip-prek-hooks": (
- "identity,mypy-airflow-ctl,mypy-dev,mypy-devel-common,mypy-task-sdk,"
+ "identity,mypy-airflow-ctl,mypy-airflow-ctl-tests,mypy-airflow-e2e-tests,"
+ "mypy-dev,mypy-devel-common,mypy-docker-tests,mypy-helm-tests,mypy-kubernetes-tests,"
+ "mypy-scripts,"
+ "mypy-shared-configuration,mypy-shared-dagnode,mypy-shared-listeners,mypy-shared-logging,"
+ "mypy-shared-module_loading,mypy-shared-observability,mypy-shared-plugins_manager,"
+ "mypy-shared-providers_discovery,mypy-shared-secrets_backend,mypy-shared-secrets_masker,"
+ "mypy-shared-serialization,mypy-shared-template_rendering,mypy-shared-timezones,mypy-task-sdk,mypy-task-sdk-integration-tests,"
"ts-compile-lint-simple-auth-manager-ui,ts-compile-lint-ui,update-uv-lock"
),
"run-kubernetes-tests": "true",
@@ -2135,7 +2272,13 @@ def test_expected_output_push(
"docs-list-as-string": "apache-airflow",
"skip-prek-hooks": (
"check-provider-yaml-valid,identity,lint-helm-chart,"
- "mypy-airflow-ctl,mypy-dev,mypy-devel-common,mypy-task-sdk,"
+ "mypy-airflow-ctl,mypy-airflow-ctl-tests,mypy-airflow-e2e-tests,"
+ "mypy-dev,mypy-devel-common,mypy-docker-tests,mypy-helm-tests,mypy-kubernetes-tests,"
+ "mypy-scripts,"
+ "mypy-shared-configuration,mypy-shared-dagnode,mypy-shared-listeners,mypy-shared-logging,"
+ "mypy-shared-module_loading,mypy-shared-observability,mypy-shared-plugins_manager,"
+ "mypy-shared-providers_discovery,mypy-shared-secrets_backend,mypy-shared-secrets_masker,"
+ "mypy-shared-serialization,mypy-shared-template_rendering,mypy-shared-timezones,mypy-task-sdk,mypy-task-sdk-integration-tests,"
"ts-compile-lint-simple-auth-manager-ui,ts-compile-lint-ui,update-uv-lock"
),
"run-kubernetes-tests": "false",
diff --git a/dev/pyproject.toml b/dev/pyproject.toml
index a29fd7687c6..d42a6d42d83 100644
--- a/dev/pyproject.toml
+++ b/dev/pyproject.toml
@@ -66,3 +66,73 @@ exclude = ["*"]
[tool.hatch.build.targets.wheel]
bypass-selection = true
+
+[tool.mypy]
+ignore_missing_imports = true
+no_implicit_optional = true
+warn_redundant_casts = true
+warn_unused_ignores = false
+plugins = [
+ "airflow_mypy/plugin/decorators.py",
+ "airflow_mypy/plugin/outputs.py",
+]
+pretty = true
+show_error_codes = true
+disable_error_code = [
+ "annotation-unchecked",
+]
+namespace_packages = true
+explicit_package_bases = true
+exclude = [
+ ".*/node_modules/.*",
+ # Exclude hidden files and directories
+ ".*/\\..*"
+]
+# Workspace packages (airflow_breeze, airflow_perf, airflow_mypy, etc.) are installed
+# editable into the hook's virtualenv by `uv sync`, so mypy resolves them via
+# site-packages — no mypy_path entries needed.
+
+[[tool.mypy.overrides]]
+module="airflow.config_templates.default_webserver_config"
+disable_error_code = [
+ "var-annotated",
+]
+
+[[tool.mypy.overrides]]
+module="airflow.migrations.*"
+ignore_errors = true
+
+[[tool.mypy.overrides]]
+module="airflow.*._vendor.*"
+ignore_errors = true
+
+[[tool.mypy.overrides]]
+module= [
+ "google.cloud.*",
+ "azure.*",
+]
+no_implicit_optional = false
+
+[[tool.mypy.overrides]]
+module = "google.api_core.gapic_v1"
+ignore_errors = true
+
+[[tool.mypy.overrides]]
+module=[
+ "referencing.*",
+ # Beam has some old type annotations, and they introduced an error recently with bad signature of
+ # a function. This is captured in https://github.com/apache/beam/issues/29927
+ # and we should remove this exclusion when it is fixed.
+ "apache_beam.*"
+]
+ignore_errors = true
+
+# airflowctl autogenered datamodels
+[[tool.mypy.overrides]]
+module="airflowctl.api.datamodels.*"
+ignore_errors = true
+
+[dependency-groups]
+mypy = [
+ "apache-airflow-devel-common[mypy]",
+]
diff --git a/devel-common/pyproject.toml b/devel-common/pyproject.toml
index a1e37d3c54a..09ee3b00866 100644
--- a/devel-common/pyproject.toml
+++ b/devel-common/pyproject.toml
@@ -171,6 +171,10 @@ docs = [
docs-gen = [
"apache-airflow-devel-common[docs-gen]",
]
+mypy = [
+ "apache-airflow-devel-common[mypy]",
+]
+
[project.scripts]
build-docs = "docs.build_docs:build_docs"
diff --git a/airflow-ctl-tests/.pre-commit-config.yaml b/docker-tests/.pre-commit-config.yaml
similarity index 77%
copy from airflow-ctl-tests/.pre-commit-config.yaml
copy to docker-tests/.pre-commit-config.yaml
index f6213b0f9a8..27e73ecc067 100644
--- a/airflow-ctl-tests/.pre-commit-config.yaml
+++ b/docker-tests/.pre-commit-config.yaml
@@ -16,17 +16,16 @@
# under the License.
---
default_stages: [pre-commit, pre-push]
-minimum_prek_version: '0.2.0'
+minimum_prek_version: '0.3.4'
default_language_version:
python: python3
repos:
- repo: local
hooks:
- - id: check-airflowctl-command-coverage
- name: Check airflowctl CLI command test coverage
- entry: ../scripts/ci/prek/check_airflowctl_command_coverage.py
+ - id: mypy-docker-tests
+ name: Run mypy for docker-tests
language: python
+ entry: ../scripts/ci/prek/run_mypy_full_dist_local_venv_or_breeze_in_ci.py docker-tests
pass_filenames: false
- files:
- (?x)
- ^tests/airflowctl_tests/.*\.py$
+ files: ^.*\.py$
+ require_serial: true
diff --git a/docker-tests/pyproject.toml b/docker-tests/pyproject.toml
index 486e00ea280..3514e78e4f1 100644
--- a/docker-tests/pyproject.toml
+++ b/docker-tests/pyproject.toml
@@ -74,3 +74,8 @@ exclude = ["*"]
[tool.hatch.build.targets.wheel]
bypass-selection = true
+
+[dependency-groups]
+mypy = [
+ "apache-airflow-devel-common[mypy]",
+]
diff --git a/docker-tests/tests/docker_tests/test_docker_compose_quick_start.py b/docker-tests/tests/docker_tests/test_docker_compose_quick_start.py
index bfefd46fc4c..3cd93078431 100644
--- a/docker-tests/tests/docker_tests/test_docker_compose_quick_start.py
+++ b/docker-tests/tests/docker_tests/test_docker_compose_quick_start.py
@@ -169,7 +169,7 @@ def print_diagnostics(compose: DockerClient, compose_version: str, docker_versio
console.print(" Docker Compose Version ".center(72, "="))
console.print(compose_version)
console.print(" Compose Config ".center(72, "="))
- console.print(json.dumps(compose.config(return_json=True), indent=4))
+ console.print(json.dumps(compose.config(return_json=True), indent=4)) # type: ignore[operator]
for service in compose.ps(all=True):
console.print(f"Service: {service.name} ".center(72, "="))
console.print(f" Service State {service.name}".center(50, "."))
diff --git a/docker-tests/tests/docker_tests/test_prod_image.py b/docker-tests/tests/docker_tests/test_prod_image.py
index 808013538fe..43b5f3cea96 100644
--- a/docker-tests/tests/docker_tests/test_prod_image.py
+++ b/docker-tests/tests/docker_tests/test_prod_image.py
@@ -39,7 +39,7 @@ AIRFLOW_CORE_PYPROJECT_TOML = AIRFLOW_ROOT_PATH / "airflow-core" / "pyproject.to
try:
from tomllib import loads as load_tomllib
except ImportError:
- from tomli import loads as load_tomllib
+ from tomli import loads as load_tomllib # type: ignore[no-redef]
airflow_core_pyproject_toml = load_tomllib(AIRFLOW_CORE_PYPROJECT_TOML.read_text())
diff --git a/go-sdk/.pre-commit-config.yaml b/go-sdk/.pre-commit-config.yaml
index 74278d28bee..72dd3a6cbeb 100644
--- a/go-sdk/.pre-commit-config.yaml
+++ b/go-sdk/.pre-commit-config.yaml
@@ -16,7 +16,7 @@
# under the License.
---
default_stages: [pre-commit, pre-push]
-minimum_prek_version: '0.3.2'
+minimum_prek_version: '0.3.4'
default_language_version:
python: python3
node: 22.19.0
diff --git a/airflow-ctl-tests/.pre-commit-config.yaml b/helm-tests/.pre-commit-config.yaml
similarity index 77%
copy from airflow-ctl-tests/.pre-commit-config.yaml
copy to helm-tests/.pre-commit-config.yaml
index f6213b0f9a8..b654d6994ce 100644
--- a/airflow-ctl-tests/.pre-commit-config.yaml
+++ b/helm-tests/.pre-commit-config.yaml
@@ -16,17 +16,16 @@
# under the License.
---
default_stages: [pre-commit, pre-push]
-minimum_prek_version: '0.2.0'
+minimum_prek_version: '0.3.4'
default_language_version:
python: python3
repos:
- repo: local
hooks:
- - id: check-airflowctl-command-coverage
- name: Check airflowctl CLI command test coverage
- entry: ../scripts/ci/prek/check_airflowctl_command_coverage.py
+ - id: mypy-helm-tests
+ name: Run mypy for helm-tests
language: python
+ entry: ../scripts/ci/prek/run_mypy_full_dist_local_venv_or_breeze_in_ci.py helm-tests
pass_filenames: false
- files:
- (?x)
- ^tests/airflowctl_tests/.*\.py$
+ files: ^.*\.py$
+ require_serial: true
diff --git a/helm-tests/pyproject.toml b/helm-tests/pyproject.toml
index 3b4568b3215..1811abebf7d 100644
--- a/helm-tests/pyproject.toml
+++ b/helm-tests/pyproject.toml
@@ -74,3 +74,8 @@ exclude = ["*"]
[tool.hatch.build.targets.wheel]
bypass-selection = true
+
+[dependency-groups]
+mypy = [
+ "apache-airflow-devel-common[mypy]",
+]
diff --git a/airflow-ctl-tests/.pre-commit-config.yaml b/kubernetes-tests/.pre-commit-config.yaml
similarity index 77%
copy from airflow-ctl-tests/.pre-commit-config.yaml
copy to kubernetes-tests/.pre-commit-config.yaml
index f6213b0f9a8..4bfb8077b9c 100644
--- a/airflow-ctl-tests/.pre-commit-config.yaml
+++ b/kubernetes-tests/.pre-commit-config.yaml
@@ -16,17 +16,16 @@
# under the License.
---
default_stages: [pre-commit, pre-push]
-minimum_prek_version: '0.2.0'
+minimum_prek_version: '0.3.4'
default_language_version:
python: python3
repos:
- repo: local
hooks:
- - id: check-airflowctl-command-coverage
- name: Check airflowctl CLI command test coverage
- entry: ../scripts/ci/prek/check_airflowctl_command_coverage.py
+ - id: mypy-kubernetes-tests
+ name: Run mypy for kubernetes-tests
language: python
+ entry: ../scripts/ci/prek/run_mypy_full_dist_local_venv_or_breeze_in_ci.py kubernetes-tests
pass_filenames: false
- files:
- (?x)
- ^tests/airflowctl_tests/.*\.py$
+ files: ^.*\.py$
+ require_serial: true
diff --git a/kubernetes-tests/pyproject.toml b/kubernetes-tests/pyproject.toml
index 3081a732624..2734651e8d2 100644
--- a/kubernetes-tests/pyproject.toml
+++ b/kubernetes-tests/pyproject.toml
@@ -81,3 +81,8 @@ exclude = ["*"]
[tool.hatch.build.targets.wheel]
bypass-selection = true
+
+[dependency-groups]
+mypy = [
+ "apache-airflow-devel-common[mypy]",
+]
diff --git a/kubernetes-tests/tests/kubernetes_tests/test_base.py b/kubernetes-tests/tests/kubernetes_tests/test_base.py
index 326c1191d9c..bffc6574153 100644
--- a/kubernetes-tests/tests/kubernetes_tests/test_base.py
+++ b/kubernetes-tests/tests/kubernetes_tests/test_base.py
@@ -130,8 +130,8 @@ class BaseK8STest:
@staticmethod
def _num_pods_in_namespace(namespace: str):
air_pod = check_output(["kubectl", "get", "pods", "-n", namespace]).decode()
- air_pod = air_pod.splitlines()
- names = [re.compile(r"\s+").split(x)[0] for x in air_pod if "airflow" in x]
+ pod_lines = air_pod.splitlines()
+ names = [re.compile(r"\s+").split(x)[0] for x in pod_lines if "airflow" in x]
return len(names)
@staticmethod
diff --git a/kubernetes-tests/tests/kubernetes_tests/test_kubernetes_pod_operator.py b/kubernetes-tests/tests/kubernetes_tests/test_kubernetes_pod_operator.py
index b9c4ae9e653..65a82d44566 100644
--- a/kubernetes-tests/tests/kubernetes_tests/test_kubernetes_pod_operator.py
+++ b/kubernetes-tests/tests/kubernetes_tests/test_kubernetes_pod_operator.py
@@ -41,7 +41,7 @@ from airflow.providers.cncf.kubernetes.hooks.kubernetes import KubernetesHook
from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator
from airflow.providers.cncf.kubernetes.utils.pod_manager import OnFinishAction, PodManager
from airflow.sdk.definitions.context import Context
-from airflow.utils import timezone
+from airflow.utils import timezone # type: ignore[attr-defined]
from airflow.utils.types import DagRunType
from airflow.version import version as airflow_version
from kubernetes_tests.test_base import BaseK8STest, StringContainingId
diff --git a/providers/.pre-commit-config.yaml b/providers/.pre-commit-config.yaml
index 2cf29a4957e..4dd1b892de0 100644
--- a/providers/.pre-commit-config.yaml
+++ b/providers/.pre-commit-config.yaml
@@ -16,7 +16,7 @@
# under the License.
---
default_stages: [pre-commit, pre-push]
-minimum_prek_version: '0.2.0'
+minimum_prek_version: '0.3.4'
default_language_version:
python: python3
node: 22.19.0
diff --git a/providers/common/ai/.pre-commit-config.yaml b/providers/common/ai/.pre-commit-config.yaml
index 89ffddc5608..1a7bc73596f 100644
--- a/providers/common/ai/.pre-commit-config.yaml
+++ b/providers/common/ai/.pre-commit-config.yaml
@@ -16,7 +16,7 @@
# under the License.
---
default_stages: [pre-commit, pre-push]
-minimum_prek_version: '0.3.2'
+minimum_prek_version: '0.3.4'
default_language_version:
python: python3
node: 22.19.0
diff --git a/providers/common/compat/.pre-commit-config.yaml b/providers/common/compat/.pre-commit-config.yaml
index 32a7f0fd76b..71cfbd6dad2 100644
--- a/providers/common/compat/.pre-commit-config.yaml
+++ b/providers/common/compat/.pre-commit-config.yaml
@@ -16,7 +16,7 @@
# under the License.
---
default_stages: [pre-commit, pre-push]
-minimum_prek_version: '0.2.0'
+minimum_prek_version: '0.3.4'
default_language_version:
python: python3
node: 22.19.0
diff --git a/providers/edge3/.pre-commit-config.yaml b/providers/edge3/.pre-commit-config.yaml
index 133e2c3c4d2..90f5a0344da 100644
--- a/providers/edge3/.pre-commit-config.yaml
+++ b/providers/edge3/.pre-commit-config.yaml
@@ -16,7 +16,7 @@
# under the License.
---
default_stages: [pre-commit, pre-push]
-minimum_prek_version: '0.3.2'
+minimum_prek_version: '0.3.4'
default_language_version:
python: python3
node: 22.19.0
diff --git a/providers/fab/.pre-commit-config.yaml b/providers/fab/.pre-commit-config.yaml
index a77a705551c..5d59ede01f8 100644
--- a/providers/fab/.pre-commit-config.yaml
+++ b/providers/fab/.pre-commit-config.yaml
@@ -16,7 +16,7 @@
# under the License.
---
default_stages: [pre-commit, pre-push]
-minimum_prek_version: '0.2.0'
+minimum_prek_version: '0.3.4'
default_language_version:
python: python3
node: 22.19.0
diff --git a/providers/keycloak/.pre-commit-config.yaml b/providers/keycloak/.pre-commit-config.yaml
index ff86d492be7..68cdb82835c 100644
--- a/providers/keycloak/.pre-commit-config.yaml
+++ b/providers/keycloak/.pre-commit-config.yaml
@@ -16,7 +16,7 @@
# under the License.
---
default_stages: [pre-commit, pre-push]
-minimum_prek_version: '0.2.0'
+minimum_prek_version: '0.3.4'
default_language_version:
python: python3
node: 22.19.0
diff --git a/airflow-ctl-tests/.pre-commit-config.yaml b/scripts/.pre-commit-config.yaml
similarity index 77%
copy from airflow-ctl-tests/.pre-commit-config.yaml
copy to scripts/.pre-commit-config.yaml
index f6213b0f9a8..2b55cb7cfc9 100644
--- a/airflow-ctl-tests/.pre-commit-config.yaml
+++ b/scripts/.pre-commit-config.yaml
@@ -16,17 +16,16 @@
# under the License.
---
default_stages: [pre-commit, pre-push]
-minimum_prek_version: '0.2.0'
+minimum_prek_version: '0.3.4'
default_language_version:
python: python3
repos:
- repo: local
hooks:
- - id: check-airflowctl-command-coverage
- name: Check airflowctl CLI command test coverage
- entry: ../scripts/ci/prek/check_airflowctl_command_coverage.py
+ - id: mypy-scripts
+ name: Run mypy for scripts
language: python
+ entry: ./ci/prek/run_mypy_full_dist_local_venv_or_breeze_in_ci.py scripts
pass_filenames: false
- files:
- (?x)
- ^tests/airflowctl_tests/.*\.py$
+ files: ^.*\.py$
+ require_serial: true
diff --git a/scripts/ci/prek/AGENTS.md b/scripts/ci/prek/AGENTS.md
index 2b36b334f73..e07c53a256c 100644
--- a/scripts/ci/prek/AGENTS.md
+++ b/scripts/ci/prek/AGENTS.md
@@ -10,9 +10,12 @@ This directory contains prek (pre-commit) hook scripts. Shared utilities live in
## Breeze CI image scripts
-Some prek scripts require the Breeze CI Docker image to run (e.g. mypy checks, OpenAPI spec
-generation, provider validation). These scripts use the `run_command_via_breeze_shell` helper
-from `common_prek_utils.py` to execute commands inside the container.
+Some prek scripts require the Breeze CI Docker image to run (e.g. `mypy-providers`, OpenAPI
+spec generation, provider validation). These scripts use the `run_command_via_breeze_shell`
+helper from `common_prek_utils.py` to execute commands inside the container. Non-provider
+mypy hooks (`mypy-airflow-core`, `mypy-task-sdk`, `mypy-shared-<dist>`, etc.) run locally via
+`run_mypy_full_dist_local_venv_or_breeze_in_ci.py`, which builds a dedicated virtualenv per hook under `.build/mypy-venvs/`
+using `uv sync --frozen --project <X> --group mypy` — no Breeze image needed.
When adding a new breeze-dependent hook:
diff --git a/scripts/ci/prek/check_shared_mypy_hooks.py b/scripts/ci/prek/check_shared_mypy_hooks.py
new file mode 100755
index 00000000000..819d81a09f3
--- /dev/null
+++ b/scripts/ci/prek/check_shared_mypy_hooks.py
@@ -0,0 +1,92 @@
+#!/usr/bin/env python
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# /// script
+# requires-python = ">=3.10"
+# dependencies = []
+# ///
+"""Fail when any shared/<dist> workspace member is missing its mypy prek config.
+
+Every shared library has its own `mypy-shared-<dist>` prek hook backed by a
+dedicated `.pre-commit-config.yaml`. When a new shared distribution is added
+under `shared/`, the contributor must also add the matching prek config so the
+mypy hook runs for that library. This check enforces that rule.
+"""
+
+from __future__ import annotations
+
+import sys
+
+from common_prek_utils import AIRFLOW_ROOT_PATH
+
+SHARED_DIR = AIRFLOW_ROOT_PATH / "shared"
+
+EXPECTED_TEMPLATE = """\
+# <license header>
+---
+default_stages: [pre-commit, pre-push]
+minimum_prek_version: '0.3.4'
+default_language_version:
+ python: python3
+repos:
+ - repo: local
+ hooks:
+ - id: mypy-shared-{dist}
+ name: Run mypy for shared-{dist}
+ language: python
+ entry: >-
+ ../../scripts/ci/prek/run_mypy_full_dist_local_venv_or_breeze_in_ci.py
+ shared/{dist}
+ pass_filenames: false
+ files: ^.*\\.py$
+ require_serial: true
+"""
+
+
+def main() -> int:
+ missing: list[str] = []
+ for dist_dir in sorted(SHARED_DIR.iterdir()):
+ if not (dist_dir / "pyproject.toml").exists():
+ continue
+ dist = dist_dir.name
+ hook_id = f"mypy-shared-{dist}"
+ config = dist_dir / ".pre-commit-config.yaml"
+ if not config.exists() or hook_id not in config.read_text():
+ missing.append(dist)
+
+ if missing:
+ print(
+ "ERROR: The following shared/<dist> workspace members are missing their "
+ f"dedicated mypy prek hook: {', '.join(missing)}\n"
+ )
+ print(
+ "Every shared library must ship its own mypy-shared-<dist> hook so it is\n"
+ "type-checked in isolation (dedicated virtualenv + mypy cache under .build/).\n"
+ )
+ print(
+ "Create shared/<dist>/.pre-commit-config.yaml with the following contents\n"
+ "(add the ASF license header at the top) for each missing distribution:\n"
+ )
+ for dist in missing:
+ print(f"--- shared/{dist}/.pre-commit-config.yaml ---")
+ print(EXPECTED_TEMPLATE.format(dist=dist))
+ return 1
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/scripts/ci/prek/mypy_local_folder.py b/scripts/ci/prek/mypy_local_folder.py
deleted file mode 100755
index 568fe1c06e4..00000000000
--- a/scripts/ci/prek/mypy_local_folder.py
+++ /dev/null
@@ -1,250 +0,0 @@
-#!/usr/bin/env python
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-# /// script
-# requires-python = ">=3.10,<3.11"
-# dependencies = [
-# "rich>=13.6.0",
-# ]
-# ///
-"""Run mypy on entire folders using local virtualenv (uv) instead of breeze.
-
-Used for non-provider projects: airflow-core, task-sdk, airflow-ctl, dev, scripts, devel-common.
-"""
-
-from __future__ import annotations
-
-import os
-import re
-import shlex
-import subprocess
-import sys
-
-from common_prek_utils import (
- AIRFLOW_ROOT_PATH,
-)
-
-CI = os.environ.get("CI")
-
-try:
- from rich.console import Console
-
- console = Console(width=400, color_system="standard")
-except ImportError:
- console = None # type: ignore[assignment]
-
-if __name__ not in ("__main__", "__mp_main__"):
- raise SystemExit(
- "This file is intended to be executed as an executable program. You cannot use it as a module."
- f"To run this script, run the ./{__file__} command"
- )
-
-ALLOWED_FOLDERS = [
- "airflow-core",
- "dev",
- "scripts",
- "devel-common",
- "task-sdk",
- "airflow-ctl",
-]
-
-# Map folder(s) to the uv project to use for running mypy.
-# When multiple folders are checked together (e.g. dev + scripts), the first folder's project is used.
-FOLDER_TO_PROJECT = {
- "airflow-core": "airflow-core",
- "task-sdk": "task-sdk",
- "airflow-ctl": "airflow-ctl",
- "devel-common": "devel-common",
- "dev": "dev",
- "scripts": "scripts",
-}
-
-if len(sys.argv) < 2:
- if console:
- console.print(f"[yellow]You need to specify the folder to test as parameter: {ALLOWED_FOLDERS}\n")
- else:
- print(f"You need to specify the folder to test as parameter: {ALLOWED_FOLDERS}")
- sys.exit(1)
-
-mypy_folders = sys.argv[1:]
-
-show_unused_warnings = os.environ.get("SHOW_UNUSED_MYPY_WARNINGS", "false")
-show_unreachable_warnings = os.environ.get("SHOW_UNREACHABLE_MYPY_WARNINGS", "false")
-
-for mypy_folder in mypy_folders:
- if mypy_folder not in ALLOWED_FOLDERS:
- if console:
- console.print(
- f"\n[red]ERROR: Folder `{mypy_folder}` is wrong.[/]\n\n"
- f"All folders passed should be one of those: {ALLOWED_FOLDERS}\n"
- )
- else:
- print(
- f"\nERROR: Folder `{mypy_folder}` is wrong.\n\n"
- f"All folders passed should be one of those: {ALLOWED_FOLDERS}\n"
- )
- sys.exit(1)
-
-exclude_regexps = [
- re.compile(x)
- for x in [
- r"^.*/node_modules/.*",
- r"^.*\\..*",
- r"^.*/src/airflow/__init__.py$",
- ]
-]
-
-
-def get_all_files(folder: str) -> list[str]:
- files_to_check = []
- python_file_paths = (AIRFLOW_ROOT_PATH / folder).resolve().rglob("*.py")
- for file in python_file_paths:
- if (
- file.name not in ("conftest.py",)
- and not any(x.match(file.as_posix()) for x in exclude_regexps)
- and not any(part.startswith(".") for part in file.parts)
- ):
- files_to_check.append(file.relative_to(AIRFLOW_ROOT_PATH).as_posix())
- return files_to_check
-
-
-all_files_to_check: list[str] = []
-for mypy_folder in mypy_folders:
- all_files_to_check.extend(get_all_files(mypy_folder))
-
-if not all_files_to_check:
- print("No files to test. Quitting")
- sys.exit(0)
-
-# Write file list
-mypy_file_list = AIRFLOW_ROOT_PATH / "files" / "mypy_files.txt"
-mypy_file_list.parent.mkdir(parents=True, exist_ok=True)
-mypy_file_list.write_text("\n".join(all_files_to_check))
-
-if console:
- console.print(f"[info]You can check the list of files in:[/] {mypy_file_list}")
-else:
- print(f"You can check the list of files in: {mypy_file_list}")
-
-file_argument_local = f"@{mypy_file_list}"
-file_argument_ci = "@/files/mypy_files.txt"
-
-project = FOLDER_TO_PROJECT.get(mypy_folders[0], "devel-common")
-
-mypy_extra_args: list[str] = []
-
-if show_unused_warnings == "true":
- if console:
- console.print("[info]Running mypy with --warn-unused-ignores")
- else:
- print("Running mypy with --warn-unused-ignores")
- mypy_extra_args.append("--warn-unused-ignores")
-
-if show_unreachable_warnings == "true":
- if console:
- console.print("[info]Running mypy with --warn-unreachable")
- else:
- print("Running mypy with --warn-unreachable")
- mypy_extra_args.append("--warn-unreachable")
-
-if console:
- console.print(f"[magenta]Running mypy for folders: {mypy_folders}[/]")
-else:
- print(f"Running mypy for folders: {mypy_folders}")
-
-if CI:
- # In CI, run inside the breeze Docker image to avoid needing a local environment
- # and to not change uv.lock or synchronize dependencies.
- from common_prek_utils import (
- initialize_breeze_prek,
- run_command_via_breeze_shell,
- )
-
- initialize_breeze_prek(__name__, __file__)
-
- mypy_cmd = f"TERM=ansi mypy {shlex.quote(file_argument_ci)} {' '.join(mypy_extra_args)}"
- result = run_command_via_breeze_shell(
- cmd=["bash", "-c", mypy_cmd],
- warn_image_upgrade_needed=True,
- extra_env={
- "INCLUDE_MYPY_VOLUME": "false",
- "MOUNT_SOURCES": "selected",
- },
- )
-else:
- # Locally, first synchronize the project's virtualenv with uv.lock so that mypy runs
- # against the same dependency set CI uses. Without this, the local .venv can drift from
- # uv.lock (e.g. after switching branches or installing extras) and mypy results would
- # diverge from CI. --frozen ensures uv.lock itself is not updated.
- sync_cmd = ["uv", "sync", "--frozen", "--project", project]
- if console:
- console.print(f"[magenta]Syncing virtualenv for project {project}: {' '.join(sync_cmd)}[/]")
- else:
- print(f"Syncing virtualenv for project {project}: {' '.join(sync_cmd)}")
- sync_result = subprocess.run(
- sync_cmd,
- cwd=str(AIRFLOW_ROOT_PATH),
- check=False,
- env={**os.environ, "TERM": "ansi"},
- )
- if sync_result.returncode != 0:
- msg = (
- f"`uv sync --frozen --project {project}` failed. Fix the sync error before running mypy — "
- "otherwise the local virtualenv may not match uv.lock and mypy results will diverge from CI.\n"
- )
- if console:
- console.print(f"[red]{msg}")
- else:
- print(msg)
- sys.exit(sync_result.returncode)
-
- # Then run mypy via uv with --frozen to not update the lock file.
- cmd = [
- "uv",
- "run",
- "--frozen",
- "--project",
- project,
- "--with",
- "apache-airflow-devel-common[mypy]",
- "mypy",
- file_argument_local,
- *mypy_extra_args,
- ]
-
- result = subprocess.run(
- cmd,
- cwd=str(AIRFLOW_ROOT_PATH),
- check=False,
- env={**os.environ, "TERM": "ansi"},
- )
-
-if result.returncode != 0:
- msg = (
- "Mypy check failed. You can run mypy locally with:\n"
- f" prek run mypy-{mypy_folders[0]} --all-files\n"
- "Or directly (first sync the virtualenv to match CI's dependency set):\n"
- f" uv sync --frozen --project {project}\n"
- f' uv run --frozen --project {project} --with "apache-airflow-devel-common[mypy]" mypy <files>\n'
- "You can also clear the mypy cache with:\n"
- " breeze down --cleanup-mypy-cache\n"
- )
- if console:
- console.print(f"[yellow]{msg}")
- else:
- print(msg)
-sys.exit(result.returncode)
diff --git a/scripts/ci/prek/run_mypy_full_dist_local_venv_or_breeze_in_ci.py b/scripts/ci/prek/run_mypy_full_dist_local_venv_or_breeze_in_ci.py
new file mode 100755
index 00000000000..44664d37c27
--- /dev/null
+++ b/scripts/ci/prek/run_mypy_full_dist_local_venv_or_breeze_in_ci.py
@@ -0,0 +1,325 @@
+#!/usr/bin/env python
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# /// script
+# requires-python = ">=3.10,<3.11"
+# dependencies = [
+# "rich>=13.6.0",
+# ]
+# ///
+"""Run mypy on entire folders using local virtualenv (uv) instead of breeze.
+
+Used for non-provider projects: airflow-core, task-sdk, airflow-ctl, dev, scripts,
+devel-common, airflow-ctl-tests, helm-tests, airflow-e2e-tests,
+task-sdk-integration-tests, docker-tests, kubernetes-tests, shared.
+"""
+
+from __future__ import annotations
+
+import os
+import re
+import shlex
+import subprocess
+import sys
+from pathlib import Path
+
+from common_prek_utils import (
+ AIRFLOW_ROOT_PATH,
+)
+
+CI = os.environ.get("CI")
+
+try:
+ from rich.console import Console
+
+ console = Console(width=400, color_system="standard")
+except ImportError:
+ console = None # type: ignore[assignment]
+
+if __name__ not in ("__main__", "__mp_main__"):
+ raise SystemExit(
+ "This file is intended to be executed as an executable program. You cannot use it as a module."
+ f"To run this script, run the ./{__file__} command"
+ )
+
+_TOP_LEVEL_ALLOWED_FOLDERS = [
+ "airflow-core",
+ "dev",
+ "scripts",
+ "devel-common",
+ "task-sdk",
+ "airflow-ctl",
+ "airflow-ctl-tests",
+ "helm-tests",
+ "airflow-e2e-tests",
+ "task-sdk-integration-tests",
+ "docker-tests",
+ "kubernetes-tests",
+]
+
+# Each shared/<dist> workspace member is an allowed folder in its own right, giving
+# every shared library its own dedicated mypy hook / virtualenv / cache.
+_SHARED_DISTS = sorted(
+ f"shared/{p.name}" for p in (AIRFLOW_ROOT_PATH / "shared").iterdir() if (p / "pyproject.toml").exists()
+)
+
+ALLOWED_FOLDERS = _TOP_LEVEL_ALLOWED_FOLDERS + _SHARED_DISTS
+
+# Map folder(s) to the uv project to use for running mypy.
+# When multiple folders are checked together, the first folder's project is used.
+FOLDER_TO_PROJECT = {f: f for f in ALLOWED_FOLDERS}
+
+# Projects that ship their own [tool.mypy] configuration in their pyproject.toml;
+# mypy must be invoked with --config-file pointing at that file so those sections
+# take precedence over the root pyproject.toml.
+FOLDER_TO_MYPY_CONFIG = {
+ "dev": "dev/pyproject.toml",
+ "scripts": "scripts/pyproject.toml",
+}
+
+if len(sys.argv) < 2:
+ if console:
+ console.print(f"[yellow]You need to specify the folder to test as parameter: {ALLOWED_FOLDERS}\n")
+ else:
+ print(f"You need to specify the folder to test as parameter: {ALLOWED_FOLDERS}")
+ sys.exit(1)
+
+mypy_folders = sys.argv[1:]
+
+show_unused_warnings = os.environ.get("SHOW_UNUSED_MYPY_WARNINGS", "false")
+show_unreachable_warnings = os.environ.get("SHOW_UNREACHABLE_MYPY_WARNINGS", "false")
+
+for mypy_folder in mypy_folders:
+ if mypy_folder not in ALLOWED_FOLDERS:
+ if console:
+ console.print(
+ f"\n[red]ERROR: Folder `{mypy_folder}` is wrong.[/]\n\n"
+ f"All folders passed should be one of those: {ALLOWED_FOLDERS}\n"
+ )
+ else:
+ print(
+ f"\nERROR: Folder `{mypy_folder}` is wrong.\n\n"
+ f"All folders passed should be one of those: {ALLOWED_FOLDERS}\n"
+ )
+ sys.exit(1)
+
+exclude_regexps = [
+ re.compile(x)
+ for x in [
+ r"^.*/node_modules/.*",
+ r"^.*\\..*",
+ r"^.*/src/airflow/__init__.py$",
+ ]
+]
+
+
+def get_all_files(folder: str) -> list[str]:
+ files_to_check = []
+ python_file_paths = (AIRFLOW_ROOT_PATH / folder).resolve().rglob("*.py")
+ for file in python_file_paths:
+ if (
+ file.name not in ("conftest.py",)
+ and not any(x.match(file.as_posix()) for x in exclude_regexps)
+ and not any(part.startswith(".") for part in file.parts)
+ ):
+ files_to_check.append(file.relative_to(AIRFLOW_ROOT_PATH).as_posix())
+ return files_to_check
+
+
+mypy_extra_args: list[str] = []
+
+if show_unused_warnings == "true":
+ if console:
+ console.print("[info]Running mypy with --warn-unused-ignores")
+ else:
+ print("Running mypy with --warn-unused-ignores")
+ mypy_extra_args.append("--warn-unused-ignores")
+
+if show_unreachable_warnings == "true":
+ if console:
+ console.print("[info]Running mypy with --warn-unreachable")
+ else:
+ print("Running mypy with --warn-unreachable")
+ mypy_extra_args.append("--warn-unreachable")
+
+
+def write_file_list(files: list[str], suffix: str = "") -> Path:
+ name = "mypy_files.txt" if not suffix else f"mypy_files_{suffix}.txt"
+ mypy_file_list = AIRFLOW_ROOT_PATH / "files" / name
+ mypy_file_list.parent.mkdir(parents=True, exist_ok=True)
+ mypy_file_list.write_text("\n".join(files))
+ if console:
+ console.print(f"[info]You can check the list of files in:[/] {mypy_file_list}")
+ else:
+ print(f"You can check the list of files in: {mypy_file_list}")
+ return mypy_file_list
+
+
+def run_local_mypy(project: str, hook_name: str, files: list[str], config_file: str | None = None) -> int:
+ """Sync a dedicated mypy venv under .build/ and run mypy on the given files.
+
+ Each hook gets its own virtualenv and mypy cache so running mypy never mutates the
+ contributor's regular project .venv and each hook keeps a stable, CI-aligned
+ dependency set. UV_PROJECT_ENVIRONMENT redirects uv away from the default
+ <project>/.venv to our cached location.
+ """
+ mypy_venv_dir = AIRFLOW_ROOT_PATH / ".build" / "mypy-venvs" / hook_name
+ mypy_cache_dir = AIRFLOW_ROOT_PATH / ".build" / "mypy-caches" / hook_name
+ mypy_venv_dir.parent.mkdir(parents=True, exist_ok=True)
+ mypy_cache_dir.parent.mkdir(parents=True, exist_ok=True)
+
+ run_env = {
+ **os.environ,
+ "TERM": "ansi",
+ "UV_PROJECT_ENVIRONMENT": str(mypy_venv_dir),
+ }
+
+ sync_cmd = ["uv", "sync", "--frozen", "--project", project, "--group", "mypy"]
+ if console:
+ console.print(
+ f"[magenta]Syncing dedicated mypy virtualenv ({mypy_venv_dir}) "
+ f"for project {project}: {' '.join(sync_cmd)}[/]"
+ )
+ else:
+ print(
+ f"Syncing dedicated mypy virtualenv ({mypy_venv_dir}) for project {project}: {' '.join(sync_cmd)}"
+ )
+ sync_result = subprocess.run(
+ sync_cmd,
+ cwd=str(AIRFLOW_ROOT_PATH),
+ check=False,
+ env=run_env,
+ )
+ if sync_result.returncode != 0:
+ msg = (
+ f"`uv sync --frozen --project {project}` failed for the mypy virtualenv at "
+ f"{mypy_venv_dir}. Fix the sync error before running mypy — otherwise the "
+ "dedicated mypy virtualenv will not match uv.lock and results will diverge "
+ "from CI. You can remove the cached virtualenv with:\n"
+ " breeze down --cleanup-mypy-cache\n"
+ )
+ if console:
+ console.print(f"[red]{msg}")
+ else:
+ print(msg)
+ return sync_result.returncode
+
+ mypy_file_list = write_file_list(files, suffix=hook_name.replace("/", "_"))
+
+ # --follow-imports=silent: each hook only reports errors for files it owns. Transitive
+ # errors from imports into other workspace projects are not reported here — those files
+ # are covered by their own hook. Without this, mypy can produce different results for the
+ # same file across hooks because each hook's virtualenv installs a different dependency
+ # set that influences type inference.
+ cmd = [
+ "uv",
+ "run",
+ "--frozen",
+ "--project",
+ project,
+ "--group",
+ "mypy",
+ "mypy",
+ "--cache-dir",
+ str(mypy_cache_dir),
+ "--follow-imports=silent",
+ ]
+ if config_file is not None:
+ cmd += ["--config-file", config_file]
+ cmd += [f"@{mypy_file_list}", *mypy_extra_args]
+
+ result = subprocess.run(
+ cmd,
+ cwd=str(AIRFLOW_ROOT_PATH),
+ check=False,
+ env=run_env,
+ )
+ return result.returncode
+
+
+if console:
+ console.print(f"[magenta]Running mypy for folders: {mypy_folders}[/]")
+else:
+ print(f"Running mypy for folders: {mypy_folders}")
+
+if CI:
+ # In CI, run inside the breeze Docker image to avoid needing a local environment
+ # and to not change uv.lock or synchronize dependencies.
+ from common_prek_utils import (
+ initialize_breeze_prek,
+ run_command_via_breeze_shell,
+ )
+
+ initialize_breeze_prek(__name__, __file__)
+
+ all_files_to_check = []
+ for mypy_folder in mypy_folders:
+ all_files_to_check.extend(get_all_files(mypy_folder))
+
+ if not all_files_to_check:
+ print("No files to test. Quitting")
+ sys.exit(0)
+
+ write_file_list(all_files_to_check)
+ file_argument_ci = "@/files/mypy_files.txt"
+
+ mypy_cmd = (
+ f"TERM=ansi mypy --follow-imports=silent {shlex.quote(file_argument_ci)} {' '.join(mypy_extra_args)}"
+ )
+ result = run_command_via_breeze_shell(
+ cmd=["bash", "-c", mypy_cmd],
+ warn_image_upgrade_needed=True,
+ extra_env={
+ "INCLUDE_MYPY_VOLUME": "false",
+ "MOUNT_SOURCES": "selected",
+ },
+ )
+ returncode = result.returncode
+else:
+ all_files_to_check = []
+ for mypy_folder in mypy_folders:
+ all_files_to_check.extend(get_all_files(mypy_folder))
+
+ if not all_files_to_check:
+ print("No files to test. Quitting")
+ sys.exit(0)
+
+ project = FOLDER_TO_PROJECT.get(mypy_folders[0], "devel-common")
+ # hook_name derives venv/cache paths under .build/; replace "/" so "shared/dist"
+ # becomes ".build/mypy-venvs/shared-dist" rather than a nested path.
+ hook_name = mypy_folders[0].replace("/", "-")
+ returncode = run_local_mypy(
+ project=project,
+ hook_name=hook_name,
+ files=all_files_to_check,
+ config_file=FOLDER_TO_MYPY_CONFIG.get(mypy_folders[0]),
+ )
+
+if returncode != 0:
+ hook_label = mypy_folders[0].replace("/", "-")
+ msg = (
+ "Mypy check failed. You can run mypy locally with:\n"
+ f" prek run mypy-{hook_label} --all-files\n"
+ "The hook uses dedicated virtualenv(s) and mypy cache(s) under .build/ so it does\n"
+ "not touch your regular project .venv. You can clear both with:\n"
+ " breeze down --cleanup-mypy-cache\n"
+ )
+ if console:
+ console.print(f"[yellow]{msg}")
+ else:
+ print(msg)
+sys.exit(returncode)
diff --git a/scripts/pyproject.toml b/scripts/pyproject.toml
index 3282d4e88c2..72bdf13bf28 100644
--- a/scripts/pyproject.toml
+++ b/scripts/pyproject.toml
@@ -61,6 +61,10 @@ dependencies = [
dev = [
"apache-airflow-devel-common",
]
+mypy = [
+ "apache-airflow-devel-common[mypy]",
+]
+
[tool.uv.sources]
apache-airflow-devel-common = {workspace = true}
@@ -76,3 +80,64 @@ packages = ["ci", "cov", "docker", "in_container", "tools"]
# "from common_prek_utils import ..." inside those modules resolve correctly
# (mirroring what Python does automatically when scripts are run directly).
pythonpath = [".", "ci/prek", "in_container"]
+
+[tool.mypy]
+ignore_missing_imports = true
+no_implicit_optional = true
+warn_redundant_casts = true
+warn_unused_ignores = false
+pretty = true
+show_error_codes = true
+disable_error_code = [
+ "annotation-unchecked",
+]
+namespace_packages = true
+explicit_package_bases = true
+exclude = [
+ ".*/node_modules/.*",
+ # Exclude hidden files and directories
+ ".*/\\..*"
+]
+# Workspace packages (airflow, airflow.sdk, airflowctl, airflow_breeze, tests_common)
+# are installed editable into the hook's virtualenv by `uv sync`, so mypy resolves them
+# via site-packages — no mypy_path entries needed.
+
+[[tool.mypy.overrides]]
+module="airflow.config_templates.default_webserver_config"
+disable_error_code = [
+ "var-annotated",
+]
+
+[[tool.mypy.overrides]]
+module="airflow.migrations.*"
+ignore_errors = true
+
+[[tool.mypy.overrides]]
+module="airflow.*._vendor.*"
+ignore_errors = true
+
+[[tool.mypy.overrides]]
+module= [
+ "google.cloud.*",
+ "azure.*",
+]
+no_implicit_optional = false
+
+[[tool.mypy.overrides]]
+module = "google.api_core.gapic_v1"
+ignore_errors = true
+
+[[tool.mypy.overrides]]
+module=[
+ "referencing.*",
+ # Beam has some old type annotations, and they introduced an error recently with bad signature of
+ # a function. This is captured in https://github.com/apache/beam/issues/29927
+ # and we should remove this exclusion when it is fixed.
+ "apache_beam.*"
+]
+ignore_errors = true
+
+# airflowctl autogenered datamodels
+[[tool.mypy.overrides]]
+module="airflowctl.api.datamodels.*"
+ignore_errors = true
diff --git a/airflow-ctl-tests/.pre-commit-config.yaml b/shared/configuration/.pre-commit-config.yaml
similarity index 77%
copy from airflow-ctl-tests/.pre-commit-config.yaml
copy to shared/configuration/.pre-commit-config.yaml
index f6213b0f9a8..9706d8602d6 100644
--- a/airflow-ctl-tests/.pre-commit-config.yaml
+++ b/shared/configuration/.pre-commit-config.yaml
@@ -16,17 +16,16 @@
# under the License.
---
default_stages: [pre-commit, pre-push]
-minimum_prek_version: '0.2.0'
+minimum_prek_version: '0.3.4'
default_language_version:
python: python3
repos:
- repo: local
hooks:
- - id: check-airflowctl-command-coverage
- name: Check airflowctl CLI command test coverage
- entry: ../scripts/ci/prek/check_airflowctl_command_coverage.py
+ - id: mypy-shared-configuration
+ name: Run mypy for shared-configuration
language: python
+ entry: ../../scripts/ci/prek/run_mypy_full_dist_local_venv_or_breeze_in_ci.py shared/configuration
pass_filenames: false
- files:
- (?x)
- ^tests/airflowctl_tests/.*\.py$
+ files: ^.*\.py$
+ require_serial: true
diff --git a/shared/configuration/pyproject.toml b/shared/configuration/pyproject.toml
index e1429c0699b..3c561223afa 100644
--- a/shared/configuration/pyproject.toml
+++ b/shared/configuration/pyproject.toml
@@ -35,6 +35,10 @@ dev = [
"apache-airflow-devel-common",
"apache-airflow-shared-module-loading",
]
+mypy = [
+ "apache-airflow-devel-common[mypy]",
+]
+
[build-system]
requires = [
diff --git a/shared/configuration/src/airflow_shared/configuration/parser.py b/shared/configuration/src/airflow_shared/configuration/parser.py
index b47d773405c..04cbe6f4ade 100644
--- a/shared/configuration/src/airflow_shared/configuration/parser.py
+++ b/shared/configuration/src/airflow_shared/configuration/parser.py
@@ -623,7 +623,7 @@ class AirflowConfigParser(ConfigParser):
)
if value is VALUE_NOT_FOUND_SENTINEL:
value = fallback
- if raw and value is not None:
+ if raw and isinstance(value, str):
return value.replace("%", "%%")
return value
@@ -845,7 +845,7 @@ class AirflowConfigParser(ConfigParser):
# when display_source = true, we know that the config_sources contains tuple
opt, source = config_sources[section][key] # type: ignore
else:
- opt = config_sources[section][key]
+ opt = config_sources[section][key] # type: ignore[assignment]
if opt == self.get_default_value(section, key):
del config_sources[section][key]
@@ -1632,7 +1632,9 @@ class AirflowConfigParser(ConfigParser):
)
return list(dict.fromkeys(itertools.chain(all_options_from_defaults, my_own_options)))
- def has_option(self, section: str, option: str, lookup_from_deprecated: bool = True, **kwargs) -> bool:
+ def has_option( # type: ignore[override]
+ self, section: str, option: str, lookup_from_deprecated: bool = True, **kwargs
+ ) -> bool:
"""
Check if option is defined.
@@ -1661,7 +1663,7 @@ class AirflowConfigParser(ConfigParser):
except (NoOptionError, NoSectionError, AirflowConfigException):
return False
- def set(self, section: str, option: str, value: str | None = None) -> None:
+ def set(self, section: str, option: str, value: str | None = None) -> None: # type: ignore[override]
"""
Set an option to the given value.
@@ -1676,7 +1678,7 @@ class AirflowConfigParser(ConfigParser):
self.add_section(section)
super().set(section, option, value)
- def remove_option(self, section: str, option: str, remove_default: bool = True):
+ def remove_option(self, section: str, option: str, remove_default: bool = True): # type: ignore[override]
"""
Remove an option if it exists in config from a file or default config.
diff --git a/airflow-ctl-tests/.pre-commit-config.yaml b/shared/dagnode/.pre-commit-config.yaml
similarity index 77%
copy from airflow-ctl-tests/.pre-commit-config.yaml
copy to shared/dagnode/.pre-commit-config.yaml
index f6213b0f9a8..34f7fe42212 100644
--- a/airflow-ctl-tests/.pre-commit-config.yaml
+++ b/shared/dagnode/.pre-commit-config.yaml
@@ -16,17 +16,16 @@
# under the License.
---
default_stages: [pre-commit, pre-push]
-minimum_prek_version: '0.2.0'
+minimum_prek_version: '0.3.4'
default_language_version:
python: python3
repos:
- repo: local
hooks:
- - id: check-airflowctl-command-coverage
- name: Check airflowctl CLI command test coverage
- entry: ../scripts/ci/prek/check_airflowctl_command_coverage.py
+ - id: mypy-shared-dagnode
+ name: Run mypy for shared-dagnode
language: python
+ entry: ../../scripts/ci/prek/run_mypy_full_dist_local_venv_or_breeze_in_ci.py shared/dagnode
pass_filenames: false
- files:
- (?x)
- ^tests/airflowctl_tests/.*\.py$
+ files: ^.*\.py$
+ require_serial: true
diff --git a/shared/dagnode/pyproject.toml b/shared/dagnode/pyproject.toml
index 6c208a2fddd..a12432deb29 100644
--- a/shared/dagnode/pyproject.toml
+++ b/shared/dagnode/pyproject.toml
@@ -31,6 +31,10 @@ dependencies = [
dev = [
"apache-airflow-devel-common",
]
+mypy = [
+ "apache-airflow-devel-common[mypy]",
+]
+
[build-system]
requires = [
diff --git a/shared/dagnode/tests/dagnode/test_node.py b/shared/dagnode/tests/dagnode/test_node.py
index 4259ca7555f..fa6600b8804 100644
--- a/shared/dagnode/tests/dagnode/test_node.py
+++ b/shared/dagnode/tests/dagnode/test_node.py
@@ -34,7 +34,7 @@ class TaskGroup:
"""Task group type for tests."""
node_id: str = attrs.field(init=False, default="test_group_id")
- prefix_group_id: str
+ prefix_group_id: bool
class Dag:
@@ -46,8 +46,8 @@ class Dag:
class ConcreteDAGNode(GenericDAGNode[Dag, Task, TaskGroup]):
"""Concrete DAGNode variant for tests."""
- dag = None
- task_group = None
+ dag: Dag | None = None
+ task_group: TaskGroup | None = None
@property
def node_id(self) -> str:
diff --git a/airflow-ctl-tests/.pre-commit-config.yaml b/shared/listeners/.pre-commit-config.yaml
similarity index 77%
copy from airflow-ctl-tests/.pre-commit-config.yaml
copy to shared/listeners/.pre-commit-config.yaml
index f6213b0f9a8..62f226fab74 100644
--- a/airflow-ctl-tests/.pre-commit-config.yaml
+++ b/shared/listeners/.pre-commit-config.yaml
@@ -16,17 +16,16 @@
# under the License.
---
default_stages: [pre-commit, pre-push]
-minimum_prek_version: '0.2.0'
+minimum_prek_version: '0.3.4'
default_language_version:
python: python3
repos:
- repo: local
hooks:
- - id: check-airflowctl-command-coverage
- name: Check airflowctl CLI command test coverage
- entry: ../scripts/ci/prek/check_airflowctl_command_coverage.py
+ - id: mypy-shared-listeners
+ name: Run mypy for shared-listeners
language: python
+ entry: ../../scripts/ci/prek/run_mypy_full_dist_local_venv_or_breeze_in_ci.py shared/listeners
pass_filenames: false
- files:
- (?x)
- ^tests/airflowctl_tests/.*\.py$
+ files: ^.*\.py$
+ require_serial: true
diff --git a/shared/listeners/pyproject.toml b/shared/listeners/pyproject.toml
index 0d20e678d56..f3b9b75170d 100644
--- a/shared/listeners/pyproject.toml
+++ b/shared/listeners/pyproject.toml
@@ -32,6 +32,10 @@ dependencies = [
dev = [
"apache-airflow-devel-common",
]
+mypy = [
+ "apache-airflow-devel-common[mypy]",
+]
+
[build-system]
requires = [
diff --git a/airflow-ctl-tests/.pre-commit-config.yaml b/shared/logging/.pre-commit-config.yaml
similarity index 77%
copy from airflow-ctl-tests/.pre-commit-config.yaml
copy to shared/logging/.pre-commit-config.yaml
index f6213b0f9a8..7440bc4add9 100644
--- a/airflow-ctl-tests/.pre-commit-config.yaml
+++ b/shared/logging/.pre-commit-config.yaml
@@ -16,17 +16,16 @@
# under the License.
---
default_stages: [pre-commit, pre-push]
-minimum_prek_version: '0.2.0'
+minimum_prek_version: '0.3.4'
default_language_version:
python: python3
repos:
- repo: local
hooks:
- - id: check-airflowctl-command-coverage
- name: Check airflowctl CLI command test coverage
- entry: ../scripts/ci/prek/check_airflowctl_command_coverage.py
+ - id: mypy-shared-logging
+ name: Run mypy for shared-logging
language: python
+ entry: ../../scripts/ci/prek/run_mypy_full_dist_local_venv_or_breeze_in_ci.py shared/logging
pass_filenames: false
- files:
- (?x)
- ^tests/airflowctl_tests/.*\.py$
+ files: ^.*\.py$
+ require_serial: true
diff --git a/shared/logging/pyproject.toml b/shared/logging/pyproject.toml
index 7640191ecba..ac5607647b5 100644
--- a/shared/logging/pyproject.toml
+++ b/shared/logging/pyproject.toml
@@ -33,6 +33,10 @@ dependencies = [
dev = [
"apache-airflow-devel-common",
]
+mypy = [
+ "apache-airflow-devel-common[mypy]",
+]
+
[build-system]
requires = [
diff --git a/shared/logging/src/airflow_shared/logging/_noncaching.py b/shared/logging/src/airflow_shared/logging/_noncaching.py
index 7a8e21fbe38..e6509e14a40 100644
--- a/shared/logging/src/airflow_shared/logging/_noncaching.py
+++ b/shared/logging/src/airflow_shared/logging/_noncaching.py
@@ -29,7 +29,8 @@ _IO = TypeVar("_IO", TextIO, BinaryIO)
def make_file_io_non_caching(io: _IO) -> _IO:
try:
fd = io.fileno()
- os.posix_fadvise(fd, 0, 0, os.POSIX_FADV_DONTNEED)
+ # posix_fadvise / POSIX_FADV_DONTNEED are Linux-only; ignored on other platforms at runtime.
+ os.posix_fadvise(fd, 0, 0, os.POSIX_FADV_DONTNEED) # type: ignore[attr-defined,unused-ignore]
except Exception:
# in case either file descriptor cannot be retrieved or fadvise is not available
# we should simply return the wrapper retrieved by FileHandler's open method
diff --git a/airflow-ctl-tests/.pre-commit-config.yaml b/shared/module_loading/.pre-commit-config.yaml
similarity index 77%
copy from airflow-ctl-tests/.pre-commit-config.yaml
copy to shared/module_loading/.pre-commit-config.yaml
index f6213b0f9a8..fd00e315b0e 100644
--- a/airflow-ctl-tests/.pre-commit-config.yaml
+++ b/shared/module_loading/.pre-commit-config.yaml
@@ -16,17 +16,16 @@
# under the License.
---
default_stages: [pre-commit, pre-push]
-minimum_prek_version: '0.2.0'
+minimum_prek_version: '0.3.4'
default_language_version:
python: python3
repos:
- repo: local
hooks:
- - id: check-airflowctl-command-coverage
- name: Check airflowctl CLI command test coverage
- entry: ../scripts/ci/prek/check_airflowctl_command_coverage.py
+ - id: mypy-shared-module_loading
+ name: Run mypy for shared-module_loading
language: python
+ entry: ../../scripts/ci/prek/run_mypy_full_dist_local_venv_or_breeze_in_ci.py shared/module_loading
pass_filenames: false
- files:
- (?x)
- ^tests/airflowctl_tests/.*\.py$
+ files: ^.*\.py$
+ require_serial: true
diff --git a/shared/module_loading/pyproject.toml b/shared/module_loading/pyproject.toml
index b91ad43d333..868b2df92af 100644
--- a/shared/module_loading/pyproject.toml
+++ b/shared/module_loading/pyproject.toml
@@ -32,6 +32,10 @@ dependencies = [
dev = [
"apache-airflow-devel-common",
]
+mypy = [
+ "apache-airflow-devel-common[mypy]",
+]
+
[build-system]
requires = [
diff --git a/airflow-ctl-tests/.pre-commit-config.yaml b/shared/observability/.pre-commit-config.yaml
similarity index 77%
copy from airflow-ctl-tests/.pre-commit-config.yaml
copy to shared/observability/.pre-commit-config.yaml
index f6213b0f9a8..eab5206f133 100644
--- a/airflow-ctl-tests/.pre-commit-config.yaml
+++ b/shared/observability/.pre-commit-config.yaml
@@ -16,17 +16,16 @@
# under the License.
---
default_stages: [pre-commit, pre-push]
-minimum_prek_version: '0.2.0'
+minimum_prek_version: '0.3.4'
default_language_version:
python: python3
repos:
- repo: local
hooks:
- - id: check-airflowctl-command-coverage
- name: Check airflowctl CLI command test coverage
- entry: ../scripts/ci/prek/check_airflowctl_command_coverage.py
+ - id: mypy-shared-observability
+ name: Run mypy for shared-observability
language: python
+ entry: ../../scripts/ci/prek/run_mypy_full_dist_local_venv_or_breeze_in_ci.py shared/observability
pass_filenames: false
- files:
- (?x)
- ^tests/airflowctl_tests/.*\.py$
+ files: ^.*\.py$
+ require_serial: true
diff --git a/shared/observability/pyproject.toml b/shared/observability/pyproject.toml
index dd493f68e96..ced06e5eab2 100644
--- a/shared/observability/pyproject.toml
+++ b/shared/observability/pyproject.toml
@@ -49,6 +49,10 @@ dev = [
"apache-airflow-devel-common",
"apache-airflow-shared-observability[all]",
]
+mypy = [
+ "apache-airflow-devel-common[mypy]",
+]
+
[build-system]
requires = [
diff --git a/shared/observability/src/airflow_shared/observability/common.py b/shared/observability/src/airflow_shared/observability/common.py
index a92a26543d3..c611782daa0 100644
--- a/shared/observability/src/airflow_shared/observability/common.py
+++ b/shared/observability/src/airflow_shared/observability/common.py
@@ -48,13 +48,25 @@ def get_otel_data_exporter(
# If the protocol env var isn't set, then it will be None,
# and it will default to an http/protobuf exporter.
+ # The grpc and http variants are incompatible types to mypy but functionally interchangeable here.
+ OTLPMetricExporter: type
+ OTLPSpanExporter: type
if env_endpoint and env_exporter_protocol == "grpc":
- from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
- from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
+ from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import (
+ OTLPMetricExporter, # type: ignore[no-redef]
+ )
+ from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import (
+ OTLPSpanExporter, # type: ignore[no-redef]
+ )
else:
- from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
- from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
-
+ from opentelemetry.exporter.otlp.proto.http.metric_exporter import (
+ OTLPMetricExporter, # type: ignore[no-redef]
+ )
+ from opentelemetry.exporter.otlp.proto.http.trace_exporter import (
+ OTLPSpanExporter, # type: ignore[no-redef]
+ )
+
+ exporter: SpanExporter | MetricExporter
if env_endpoint:
if host is not None and port is not None:
log.warning(
diff --git a/shared/observability/src/airflow_shared/observability/metrics/otel_logger.py b/shared/observability/src/airflow_shared/observability/metrics/otel_logger.py
index 14726e3ecc0..c8db3ee02f9 100644
--- a/shared/observability/src/airflow_shared/observability/metrics/otel_logger.py
+++ b/shared/observability/src/airflow_shared/observability/metrics/otel_logger.py
@@ -418,7 +418,7 @@ def get_otel_logger(
readers = [
PeriodicExportingMetricReader(
- exporter=metric_exporter,
+ exporter=metric_exporter, # type: ignore[arg-type]
export_interval_millis=interval, # type: ignore[arg-type]
)
]
diff --git a/shared/observability/src/airflow_shared/observability/metrics/stats.py b/shared/observability/src/airflow_shared/observability/metrics/stats.py
index be153c298d6..5cd3c098d81 100644
--- a/shared/observability/src/airflow_shared/observability/metrics/stats.py
+++ b/shared/observability/src/airflow_shared/observability/metrics/stats.py
@@ -65,6 +65,7 @@ def normalize_name_for_stats(name: str, log_warning: bool = True) -> str:
class _Stats(type):
factory: Callable[[], StatsLogger | NoStatsLogger] | None = None
instance: StatsLogger | NoStatsLogger | None = None
+ _instance_pid: int | None = None
def __getattr__(cls, name: str) -> str:
factory = type.__getattribute__(cls, "factory")
diff --git a/airflow-ctl-tests/.pre-commit-config.yaml b/shared/plugins_manager/.pre-commit-config.yaml
similarity index 77%
copy from airflow-ctl-tests/.pre-commit-config.yaml
copy to shared/plugins_manager/.pre-commit-config.yaml
index f6213b0f9a8..9180e2b1035 100644
--- a/airflow-ctl-tests/.pre-commit-config.yaml
+++ b/shared/plugins_manager/.pre-commit-config.yaml
@@ -16,17 +16,16 @@
# under the License.
---
default_stages: [pre-commit, pre-push]
-minimum_prek_version: '0.2.0'
+minimum_prek_version: '0.3.4'
default_language_version:
python: python3
repos:
- repo: local
hooks:
- - id: check-airflowctl-command-coverage
- name: Check airflowctl CLI command test coverage
- entry: ../scripts/ci/prek/check_airflowctl_command_coverage.py
+ - id: mypy-shared-plugins_manager
+ name: Run mypy for shared-plugins_manager
language: python
+ entry: ../../scripts/ci/prek/run_mypy_full_dist_local_venv_or_breeze_in_ci.py shared/plugins_manager
pass_filenames: false
- files:
- (?x)
- ^tests/airflowctl_tests/.*\.py$
+ files: ^.*\.py$
+ require_serial: true
diff --git a/shared/plugins_manager/pyproject.toml b/shared/plugins_manager/pyproject.toml
index 50afca76987..b70872cf145 100644
--- a/shared/plugins_manager/pyproject.toml
+++ b/shared/plugins_manager/pyproject.toml
@@ -36,6 +36,10 @@ dev = [
"apache-airflow-shared-module-loading",
"apache-airflow-shared-listeners",
]
+mypy = [
+ "apache-airflow-devel-common[mypy]",
+]
+
[build-system]
requires = [
diff --git a/providers/keycloak/.pre-commit-config.yaml b/shared/providers_discovery/.pre-commit-config.yaml
similarity index 75%
copy from providers/keycloak/.pre-commit-config.yaml
copy to shared/providers_discovery/.pre-commit-config.yaml
index ff86d492be7..2c26b25a954 100644
--- a/providers/keycloak/.pre-commit-config.yaml
+++ b/shared/providers_discovery/.pre-commit-config.yaml
@@ -16,17 +16,18 @@
# under the License.
---
default_stages: [pre-commit, pre-push]
-minimum_prek_version: '0.2.0'
+minimum_prek_version: '0.3.4'
default_language_version:
python: python3
- node: 22.19.0
- golang: 1.24.0
repos:
- repo: local
hooks:
- - id: generate-openapi-spec-keycloak
- name: Generate the FastAPI API spec for Keycloak
+ - id: mypy-shared-providers_discovery
+ name: Run mypy for shared-providers_discovery
language: python
- entry: ../../scripts/ci/prek/generate_openapi_spec_providers.py keycloak
+ entry: >-
+ ../../scripts/ci/prek/run_mypy_full_dist_local_venv_or_breeze_in_ci.py
+ shared/providers_discovery
pass_filenames: false
- files: ^src/airflow/providers/keycloak/auth_manager/.*\.py$
+ files: ^.*\.py$
+ require_serial: true
diff --git a/shared/providers_discovery/pyproject.toml b/shared/providers_discovery/pyproject.toml
index ec643093d09..f659d280405 100644
--- a/shared/providers_discovery/pyproject.toml
+++ b/shared/providers_discovery/pyproject.toml
@@ -39,6 +39,10 @@ dev = [
"apache-airflow-devel-common",
"apache-airflow-shared-module-loading",
]
+mypy = [
+ "apache-airflow-devel-common[mypy]",
+]
+
[build-system]
requires = [
diff --git a/airflow-ctl-tests/.pre-commit-config.yaml b/shared/secrets_backend/.pre-commit-config.yaml
similarity index 77%
copy from airflow-ctl-tests/.pre-commit-config.yaml
copy to shared/secrets_backend/.pre-commit-config.yaml
index f6213b0f9a8..93175830f9f 100644
--- a/airflow-ctl-tests/.pre-commit-config.yaml
+++ b/shared/secrets_backend/.pre-commit-config.yaml
@@ -16,17 +16,16 @@
# under the License.
---
default_stages: [pre-commit, pre-push]
-minimum_prek_version: '0.2.0'
+minimum_prek_version: '0.3.4'
default_language_version:
python: python3
repos:
- repo: local
hooks:
- - id: check-airflowctl-command-coverage
- name: Check airflowctl CLI command test coverage
- entry: ../scripts/ci/prek/check_airflowctl_command_coverage.py
+ - id: mypy-shared-secrets_backend
+ name: Run mypy for shared-secrets_backend
language: python
+ entry: ../../scripts/ci/prek/run_mypy_full_dist_local_venv_or_breeze_in_ci.py shared/secrets_backend
pass_filenames: false
- files:
- (?x)
- ^tests/airflowctl_tests/.*\.py$
+ files: ^.*\.py$
+ require_serial: true
diff --git a/shared/secrets_backend/pyproject.toml b/shared/secrets_backend/pyproject.toml
index 03bc67c9528..f94f7c1163f 100644
--- a/shared/secrets_backend/pyproject.toml
+++ b/shared/secrets_backend/pyproject.toml
@@ -29,6 +29,10 @@ dependencies = []
dev = [
"apache-airflow-devel-common",
]
+mypy = [
+ "apache-airflow-devel-common[mypy]",
+]
+
[build-system]
requires = [
diff --git a/shared/secrets_backend/src/airflow_shared/secrets_backend/base.py b/shared/secrets_backend/src/airflow_shared/secrets_backend/base.py
index 566ed379c85..a2715b93ca9 100644
--- a/shared/secrets_backend/src/airflow_shared/secrets_backend/base.py
+++ b/shared/secrets_backend/src/airflow_shared/secrets_backend/base.py
@@ -84,7 +84,7 @@ class BaseSecretsBackend(ABC):
def _deserialize_connection_value(conn_class: type, conn_id: str, value: str):
value = value.strip()
if value[0] == "{":
- return conn_class.from_json(value=value, conn_id=conn_id)
+ return conn_class.from_json(value=value, conn_id=conn_id) # type: ignore[attr-defined]
# TODO: Only sdk has from_uri defined on it. Is it worthwhile developing the core path or not?
if hasattr(conn_class, "from_uri"):
diff --git a/airflow-ctl-tests/.pre-commit-config.yaml b/shared/secrets_masker/.pre-commit-config.yaml
similarity index 77%
copy from airflow-ctl-tests/.pre-commit-config.yaml
copy to shared/secrets_masker/.pre-commit-config.yaml
index f6213b0f9a8..8025b4429c2 100644
--- a/airflow-ctl-tests/.pre-commit-config.yaml
+++ b/shared/secrets_masker/.pre-commit-config.yaml
@@ -16,17 +16,16 @@
# under the License.
---
default_stages: [pre-commit, pre-push]
-minimum_prek_version: '0.2.0'
+minimum_prek_version: '0.3.4'
default_language_version:
python: python3
repos:
- repo: local
hooks:
- - id: check-airflowctl-command-coverage
- name: Check airflowctl CLI command test coverage
- entry: ../scripts/ci/prek/check_airflowctl_command_coverage.py
+ - id: mypy-shared-secrets_masker
+ name: Run mypy for shared-secrets_masker
language: python
+ entry: ../../scripts/ci/prek/run_mypy_full_dist_local_venv_or_breeze_in_ci.py shared/secrets_masker
pass_filenames: false
- files:
- (?x)
- ^tests/airflowctl_tests/.*\.py$
+ files: ^.*\.py$
+ require_serial: true
diff --git a/shared/secrets_masker/pyproject.toml b/shared/secrets_masker/pyproject.toml
index 479b5cb8168..610516fd3eb 100644
--- a/shared/secrets_masker/pyproject.toml
+++ b/shared/secrets_masker/pyproject.toml
@@ -34,6 +34,10 @@ dependencies = [
dev = [
"apache-airflow-devel-common",
]
+mypy = [
+ "apache-airflow-devel-common[mypy]",
+]
+
[build-system]
requires = [
diff --git a/shared/secrets_masker/tests/secrets_masker/test_secrets_masker.py b/shared/secrets_masker/tests/secrets_masker/test_secrets_masker.py
index 274e611d57c..e23cb1c2744 100644
--- a/shared/secrets_masker/tests/secrets_masker/test_secrets_masker.py
+++ b/shared/secrets_masker/tests/secrets_masker/test_secrets_masker.py
@@ -46,7 +46,7 @@ PASSWORD = "password"
def configure_secrets_masker_for_test(
- masker: SecretsMasker, min_length: int = 5, sensitive_fields: list[str] = None
+ masker: SecretsMasker, min_length: int = 5, sensitive_fields: list[str] | None = None
):
"""Helper function to configure a SecretsMasker instance for testing."""
masker.min_length_to_mask = min_length
diff --git a/airflow-ctl-tests/.pre-commit-config.yaml b/shared/serialization/.pre-commit-config.yaml
similarity index 77%
copy from airflow-ctl-tests/.pre-commit-config.yaml
copy to shared/serialization/.pre-commit-config.yaml
index f6213b0f9a8..e0619c11dfd 100644
--- a/airflow-ctl-tests/.pre-commit-config.yaml
+++ b/shared/serialization/.pre-commit-config.yaml
@@ -16,17 +16,16 @@
# under the License.
---
default_stages: [pre-commit, pre-push]
-minimum_prek_version: '0.2.0'
+minimum_prek_version: '0.3.4'
default_language_version:
python: python3
repos:
- repo: local
hooks:
- - id: check-airflowctl-command-coverage
- name: Check airflowctl CLI command test coverage
- entry: ../scripts/ci/prek/check_airflowctl_command_coverage.py
+ - id: mypy-shared-serialization
+ name: Run mypy for shared-serialization
language: python
+ entry: ../../scripts/ci/prek/run_mypy_full_dist_local_venv_or_breeze_in_ci.py shared/serialization
pass_filenames: false
- files:
- (?x)
- ^tests/airflowctl_tests/.*\.py$
+ files: ^.*\.py$
+ require_serial: true
diff --git a/shared/serialization/pyproject.toml b/shared/serialization/pyproject.toml
index 45450194143..91bd5412f67 100644
--- a/shared/serialization/pyproject.toml
+++ b/shared/serialization/pyproject.toml
@@ -29,6 +29,10 @@ dependencies = []
dev = [
"apache-airflow-devel-common",
]
+mypy = [
+ "apache-airflow-devel-common[mypy]",
+]
+
[build-system]
requires = [
diff --git a/providers/keycloak/.pre-commit-config.yaml b/shared/template_rendering/.pre-commit-config.yaml
similarity index 75%
copy from providers/keycloak/.pre-commit-config.yaml
copy to shared/template_rendering/.pre-commit-config.yaml
index ff86d492be7..cbe27c38eba 100644
--- a/providers/keycloak/.pre-commit-config.yaml
+++ b/shared/template_rendering/.pre-commit-config.yaml
@@ -16,17 +16,18 @@
# under the License.
---
default_stages: [pre-commit, pre-push]
-minimum_prek_version: '0.2.0'
+minimum_prek_version: '0.3.4'
default_language_version:
python: python3
- node: 22.19.0
- golang: 1.24.0
repos:
- repo: local
hooks:
- - id: generate-openapi-spec-keycloak
- name: Generate the FastAPI API spec for Keycloak
+ - id: mypy-shared-template_rendering
+ name: Run mypy for shared-template_rendering
language: python
- entry: ../../scripts/ci/prek/generate_openapi_spec_providers.py keycloak
+ entry: >-
+ ../../scripts/ci/prek/run_mypy_full_dist_local_venv_or_breeze_in_ci.py
+ shared/template_rendering
pass_filenames: false
- files: ^src/airflow/providers/keycloak/auth_manager/.*\.py$
+ files: ^.*\.py$
+ require_serial: true
diff --git a/shared/template_rendering/pyproject.toml b/shared/template_rendering/pyproject.toml
index 574b4f713c5..4905c6771f2 100644
--- a/shared/template_rendering/pyproject.toml
+++ b/shared/template_rendering/pyproject.toml
@@ -29,6 +29,10 @@ dependencies = []
dev = [
"apache-airflow-devel-common",
]
+mypy = [
+ "apache-airflow-devel-common[mypy]",
+]
+
[build-system]
requires = [
diff --git a/airflow-ctl-tests/.pre-commit-config.yaml b/shared/timezones/.pre-commit-config.yaml
similarity index 77%
copy from airflow-ctl-tests/.pre-commit-config.yaml
copy to shared/timezones/.pre-commit-config.yaml
index f6213b0f9a8..25c6cf6d8a5 100644
--- a/airflow-ctl-tests/.pre-commit-config.yaml
+++ b/shared/timezones/.pre-commit-config.yaml
@@ -16,17 +16,16 @@
# under the License.
---
default_stages: [pre-commit, pre-push]
-minimum_prek_version: '0.2.0'
+minimum_prek_version: '0.3.4'
default_language_version:
python: python3
repos:
- repo: local
hooks:
- - id: check-airflowctl-command-coverage
- name: Check airflowctl CLI command test coverage
- entry: ../scripts/ci/prek/check_airflowctl_command_coverage.py
+ - id: mypy-shared-timezones
+ name: Run mypy for shared-timezones
language: python
+ entry: ../../scripts/ci/prek/run_mypy_full_dist_local_venv_or_breeze_in_ci.py shared/timezones
pass_filenames: false
- files:
- (?x)
- ^tests/airflowctl_tests/.*\.py$
+ files: ^.*\.py$
+ require_serial: true
diff --git a/shared/timezones/pyproject.toml b/shared/timezones/pyproject.toml
index 6315e30648d..ec71c5648cf 100644
--- a/shared/timezones/pyproject.toml
+++ b/shared/timezones/pyproject.toml
@@ -32,6 +32,10 @@ dependencies = [
dev = [
"apache-airflow-devel-common",
]
+mypy = [
+ "apache-airflow-devel-common[mypy]",
+]
+
[build-system]
requires = [
diff --git a/shared/timezones/src/airflow_shared/timezones/timezone.py b/shared/timezones/src/airflow_shared/timezones/timezone.py
index 69898b24d0b..45b538a52a8 100644
--- a/shared/timezones/src/airflow_shared/timezones/timezone.py
+++ b/shared/timezones/src/airflow_shared/timezones/timezone.py
@@ -107,7 +107,9 @@ def make_aware(value: DateTime, timezone: dt.tzinfo | None = None) -> DateTime:
@overload
-def make_aware(value: dt.datetime, timezone: dt.tzinfo | None = None) -> dt.datetime: ...
+def make_aware( # type: ignore[overload-cannot-match]
+ value: dt.datetime, timezone: dt.tzinfo | None = None
+) -> dt.datetime: ...
def make_aware(value: dt.datetime | None, timezone: dt.tzinfo | None = None) -> dt.datetime | None:
@@ -200,7 +202,9 @@ def coerce_datetime(v: DateTime, tz: dt.tzinfo | None = None) -> DateTime: ...
@overload
-def coerce_datetime(v: dt.datetime, tz: dt.tzinfo | None = None) -> DateTime: ...
+def coerce_datetime( # type: ignore[overload-cannot-match]
+ v: dt.datetime, tz: dt.tzinfo | None = None
+) -> DateTime: ...
def coerce_datetime(v: dt.datetime | None, tz: dt.tzinfo | None = None) -> DateTime | None:
diff --git a/airflow-ctl-tests/.pre-commit-config.yaml b/task-sdk-integration-tests/.pre-commit-config.yaml
similarity index 76%
copy from airflow-ctl-tests/.pre-commit-config.yaml
copy to task-sdk-integration-tests/.pre-commit-config.yaml
index f6213b0f9a8..f09e0e1f6fe 100644
--- a/airflow-ctl-tests/.pre-commit-config.yaml
+++ b/task-sdk-integration-tests/.pre-commit-config.yaml
@@ -16,17 +16,16 @@
# under the License.
---
default_stages: [pre-commit, pre-push]
-minimum_prek_version: '0.2.0'
+minimum_prek_version: '0.3.4'
default_language_version:
python: python3
repos:
- repo: local
hooks:
- - id: check-airflowctl-command-coverage
- name: Check airflowctl CLI command test coverage
- entry: ../scripts/ci/prek/check_airflowctl_command_coverage.py
+ - id: mypy-task-sdk-integration-tests
+ name: Run mypy for task-sdk-integration-tests
language: python
+ entry: ../scripts/ci/prek/run_mypy_full_dist_local_venv_or_breeze_in_ci.py task-sdk-integration-tests
pass_filenames: false
- files:
- (?x)
- ^tests/airflowctl_tests/.*\.py$
+ files: ^.*\.py$
+ require_serial: true
diff --git a/task-sdk-integration-tests/pyproject.toml b/task-sdk-integration-tests/pyproject.toml
index 15d9819f5ca..50f58f80e09 100644
--- a/task-sdk-integration-tests/pyproject.toml
+++ b/task-sdk-integration-tests/pyproject.toml
@@ -75,3 +75,8 @@ exclude = ["*"]
[tool.hatch.build.targets.wheel]
bypass-selection = true
+
+[dependency-groups]
+mypy = [
+ "apache-airflow-devel-common[mypy]",
+]
diff --git a/task-sdk/.pre-commit-config.yaml b/task-sdk/.pre-commit-config.yaml
index 8966b402044..100a6e64908 100644
--- a/task-sdk/.pre-commit-config.yaml
+++ b/task-sdk/.pre-commit-config.yaml
@@ -16,7 +16,7 @@
# under the License.
---
default_stages: [pre-commit, pre-push]
-minimum_prek_version: '0.2.0'
+minimum_prek_version: '0.3.4'
default_language_version:
python: python3
node: 22.19.0
@@ -59,7 +59,7 @@ repos:
- id: mypy-task-sdk
name: Run mypy for task-sdk
language: python
- entry: ../scripts/ci/prek/mypy_local_folder.py task-sdk
+ entry: ../scripts/ci/prek/run_mypy_full_dist_local_venv_or_breeze_in_ci.py task-sdk
pass_filenames: false
files: ^.*\.py$
require_serial: true
diff --git a/task-sdk/pyproject.toml b/task-sdk/pyproject.toml
index 230a3dba2d2..3f92d73417f 100644
--- a/task-sdk/pyproject.toml
+++ b/task-sdk/pyproject.toml
@@ -226,6 +226,10 @@ dev = [
docs = [
"apache-airflow-devel-common[docs]",
]
+mypy = [
+ "apache-airflow-devel-common[mypy]",
+]
+
[tool.uv.sources]
# These names must match the names as defined in the pyproject.toml of the workspace items,
# *not* the workspace folder paths
diff --git a/task-sdk/src/airflow/sdk/execution_time/comms.py b/task-sdk/src/airflow/sdk/execution_time/comms.py
index f4d83baef6d..87c7881333a 100644
--- a/task-sdk/src/airflow/sdk/execution_time/comms.py
+++ b/task-sdk/src/airflow/sdk/execution_time/comms.py
@@ -133,7 +133,7 @@ def _new_encoder() -> msgspec.msgpack.Encoder:
return msgspec.msgpack.Encoder(enc_hook=_msgpack_enc_hook)
-class _RequestFrame(msgspec.Struct, array_like=True, frozen=True, omit_defaults=True):
+class _RequestFrame(msgspec.Struct, array_like=True, frozen=True, omit_defaults=True): # type: ignore[call-arg]
id: int
"""
The request id, set by the sender.
@@ -159,7 +159,7 @@ class _RequestFrame(msgspec.Struct, array_like=True, frozen=True, omit_defaults=
return buffer
-class _ResponseFrame(_RequestFrame, frozen=True):
+class _ResponseFrame(_RequestFrame, frozen=True): # type: ignore[call-arg]
id: int
"""
The id of the request this is a response to
diff --git a/uv.lock b/uv.lock
index 78a053c6cfd..b2a7512072e 100644
--- a/uv.lock
+++ b/uv.lock
@@ -12,7 +12,7 @@ resolution-markers = [
]
[options]
-exclude-newer = "2026-04-15T13:08:58.648422Z"
+exclude-newer = "2026-04-15T13:52:22.676684Z"
exclude-newer-span = "P4D"
[options.exclude-newer-package]
@@ -1840,6 +1840,9 @@ dev = [
docs = [
{ name = "apache-airflow-devel-common", extra = ["docs"] },
]
+mypy = [
+ { name = "apache-airflow-devel-common", extra = ["mypy"] },
+]
[package.metadata]
requires-dist = [
@@ -1942,6 +1945,7 @@ dev = [
{ name = "apache-airflow-task-sdk", editable = "task-sdk" },
]
docs = [{ name = "apache-airflow-devel-common", extras = ["docs"], editable = "devel-common" }]
+mypy = [{ name = "apache-airflow-devel-common", extras = ["mypy"], editable = "devel-common" }]
[[package]]
name = "apache-airflow-ctl"
@@ -1979,6 +1983,9 @@ dev = [
docs = [
{ name = "apache-airflow-devel-common", extra = ["docs"] },
]
+mypy = [
+ { name = "apache-airflow-devel-common", extra = ["mypy"] },
+]
[package.metadata]
requires-dist = [
@@ -2009,6 +2016,7 @@ dev = [
{ name = "apache-airflow-devel-common", editable = "devel-common" },
]
docs = [{ name = "apache-airflow-devel-common", extras = ["docs"], editable = "devel-common" }]
+mypy = [{ name = "apache-airflow-devel-common", extras = ["mypy"], editable = "devel-common" }]
[[package]]
name = "apache-airflow-ctl-tests"
@@ -2019,12 +2027,20 @@ dependencies = [
{ name = "apache-airflow-devel-common" },
]
+[package.dev-dependencies]
+mypy = [
+ { name = "apache-airflow-devel-common", extra = ["mypy"] },
+]
+
[package.metadata]
requires-dist = [
{ name = "apache-airflow-ctl", editable = "airflow-ctl" },
{ name = "apache-airflow-devel-common", editable = "devel-common" },
]
+[package.metadata.requires-dev]
+mypy = [{ name = "apache-airflow-devel-common", extras = ["mypy"], editable = "devel-common" }]
+
[[package]]
name = "apache-airflow-dev"
version = "0.0.1"
@@ -2048,6 +2064,11 @@ dependencies = [
{ name = "tabulate" },
]
+[package.dev-dependencies]
+mypy = [
+ { name = "apache-airflow-devel-common", extra = ["mypy"] },
+]
+
[package.metadata]
requires-dist = [
{ name = "beautifulsoup4", specifier = ">=4.12.0" },
@@ -2068,6 +2089,9 @@ requires-dist = [
{ name = "tabulate", specifier = ">=0.9.0" },
]
+[package.metadata.requires-dev]
+mypy = [{ name = "apache-airflow-devel-common", extras = ["mypy"], editable = "devel-common" }]
+
[[package]]
name = "apache-airflow-devel-common"
version = "0.1.0"
@@ -2379,6 +2403,9 @@ docs = [
docs-gen = [
{ name = "apache-airflow-devel-common", extra = ["docs-gen"] },
]
+mypy = [
+ { name = "apache-airflow-devel-common", extra = ["mypy"] },
+]
[package.metadata]
requires-dist = [
@@ -2484,6 +2511,7 @@ provides-extras = ["all", "basic", "coverage", "duckdb", "debuggers", "devscript
dev = [{ name = "apache-airflow-core", editable = "airflow-core" }]
docs = [{ name = "apache-airflow-devel-common", extras = ["docs"], editable = "devel-common" }]
docs-gen = [{ name = "apache-airflow-devel-common", extras = ["docs-gen"], editable = "devel-common" }]
+mypy = [{ name = "apache-airflow-devel-common", extras = ["mypy"], editable = "devel-common" }]
[[package]]
name = "apache-airflow-docker-tests"
@@ -2493,9 +2521,17 @@ dependencies = [
{ name = "apache-airflow-devel-common" },
]
+[package.dev-dependencies]
+mypy = [
+ { name = "apache-airflow-devel-common", extra = ["mypy"] },
+]
+
[package.metadata]
requires-dist = [{ name = "apache-airflow-devel-common", editable = "devel-common" }]
+[package.metadata.requires-dev]
+mypy = [{ name = "apache-airflow-devel-common", extras = ["mypy"], editable = "devel-common" }]
+
[[package]]
name = "apache-airflow-e2e-tests"
version = "0.0.1"
@@ -2507,6 +2543,11 @@ dependencies = [
{ name = "testcontainers" },
]
+[package.dev-dependencies]
+mypy = [
+ { name = "apache-airflow-devel-common", extra = ["mypy"] },
+]
+
[package.metadata]
requires-dist = [
{ name = "apache-airflow-core", editable = "airflow-core" },
@@ -2515,6 +2556,9 @@ requires-dist = [
{ name = "testcontainers", specifier = ">=4.12.0" },
]
+[package.metadata.requires-dev]
+mypy = [{ name = "apache-airflow-devel-common", extras = ["mypy"], editable = "devel-common" }]
+
[[package]]
name = "apache-airflow-helm-tests"
version = "0.0.1"
@@ -2524,12 +2568,20 @@ dependencies = [
{ name = "apache-airflow-providers-cncf-kubernetes" },
]
+[package.dev-dependencies]
+mypy = [
+ { name = "apache-airflow-devel-common", extra = ["mypy"] },
+]
+
[package.metadata]
requires-dist = [
{ name = "apache-airflow-devel-common", editable = "devel-common" },
{ name = "apache-airflow-providers-cncf-kubernetes", editable = "providers/cncf/kubernetes" },
]
+[package.metadata.requires-dev]
+mypy = [{ name = "apache-airflow-devel-common", extras = ["mypy"], editable = "devel-common" }]
+
[[package]]
name = "apache-airflow-kubernetes-tests"
version = "0.0.1"
@@ -2544,6 +2596,11 @@ dependencies = [
{ name = "urllib3" },
]
+[package.dev-dependencies]
+mypy = [
+ { name = "apache-airflow-devel-common", extra = ["mypy"] },
+]
+
[package.metadata]
requires-dist = [
{ name = "apache-airflow-core", editable = "airflow-core" },
@@ -2555,6 +2612,9 @@ requires-dist = [
{ name = "urllib3", specifier = ">=2.1.0,!=2.6.0" },
]
+[package.metadata.requires-dev]
+mypy = [{ name = "apache-airflow-devel-common", extras = ["mypy"], editable = "devel-common" }]
+
[[package]]
name = "apache-airflow-providers"
version = "0.0.1"
@@ -7777,6 +7837,9 @@ dependencies = [
dev = [
{ name = "apache-airflow-devel-common" },
]
+mypy = [
+ { name = "apache-airflow-devel-common", extra = ["mypy"] },
+]
[package.metadata]
requires-dist = [
@@ -7796,6 +7859,7 @@ requires-dist = [
[package.metadata.requires-dev]
dev = [{ name = "apache-airflow-devel-common", editable = "devel-common" }]
+mypy = [{ name = "apache-airflow-devel-common", extras = ["mypy"], editable = "devel-common" }]
[[package]]
name = "apache-airflow-shared-configuration"
@@ -7813,6 +7877,9 @@ dev = [
{ name = "apache-airflow-devel-common" },
{ name = "apache-airflow-shared-module-loading" },
]
+mypy = [
+ { name = "apache-airflow-devel-common", extra = ["mypy"] },
+]
[package.metadata]
requires-dist = [
@@ -7827,6 +7894,7 @@ dev = [
{ name = "apache-airflow-devel-common", editable = "devel-common" },
{ name = "apache-airflow-shared-module-loading", editable = "shared/module_loading" },
]
+mypy = [{ name = "apache-airflow-devel-common", extras = ["mypy"], editable = "devel-common" }]
[[package]]
name = "apache-airflow-shared-dagnode"
@@ -7840,12 +7908,16 @@ dependencies = [
dev = [
{ name = "apache-airflow-devel-common" },
]
+mypy = [
+ { name = "apache-airflow-devel-common", extra = ["mypy"] },
+]
[package.metadata]
requires-dist = [{ name = "structlog", specifier = ">=25.4.0" }]
[package.metadata.requires-dev]
dev = [{ name = "apache-airflow-devel-common", editable = "devel-common" }]
+mypy = [{ name = "apache-airflow-devel-common", extras = ["mypy"], editable = "devel-common" }]
[[package]]
name = "apache-airflow-shared-listeners"
@@ -7860,6 +7932,9 @@ dependencies = [
dev = [
{ name = "apache-airflow-devel-common" },
]
+mypy = [
+ { name = "apache-airflow-devel-common", extra = ["mypy"] },
+]
[package.metadata]
requires-dist = [
@@ -7869,6 +7944,7 @@ requires-dist = [
[package.metadata.requires-dev]
dev = [{ name = "apache-airflow-devel-common", editable = "devel-common" }]
+mypy = [{ name = "apache-airflow-devel-common", extras = ["mypy"], editable = "devel-common" }]
[[package]]
name = "apache-airflow-shared-logging"
@@ -7884,6 +7960,9 @@ dependencies = [
dev = [
{ name = "apache-airflow-devel-common" },
]
+mypy = [
+ { name = "apache-airflow-devel-common", extra = ["mypy"] },
+]
[package.metadata]
requires-dist = [
@@ -7894,6 +7973,7 @@ requires-dist = [
[package.metadata.requires-dev]
dev = [{ name = "apache-airflow-devel-common", editable = "devel-common" }]
+mypy = [{ name = "apache-airflow-devel-common", extras = ["mypy"], editable = "devel-common" }]
[[package]]
name = "apache-airflow-shared-module-loading"
@@ -7908,6 +7988,9 @@ dependencies = [
dev = [
{ name = "apache-airflow-devel-common" },
]
+mypy = [
+ { name = "apache-airflow-devel-common", extra = ["mypy"] },
+]
[package.metadata]
requires-dist = [
@@ -7917,6 +8000,7 @@ requires-dist = [
[package.metadata.requires-dev]
dev = [{ name = "apache-airflow-devel-common", editable = "devel-common" }]
+mypy = [{ name = "apache-airflow-devel-common", extras = ["mypy"], editable = "devel-common" }]
[[package]]
name = "apache-airflow-shared-observability"
@@ -7954,6 +8038,9 @@ dev = [
{ name = "apache-airflow-devel-common" },
{ name = "apache-airflow-shared-observability", extra = ["all"] },
]
+mypy = [
+ { name = "apache-airflow-devel-common", extra = ["mypy"] },
+]
[package.metadata]
requires-dist = [
@@ -7975,6 +8062,7 @@ dev = [
{ name = "apache-airflow-devel-common", editable = "devel-common" },
{ name = "apache-airflow-shared-observability", extras = ["all"], editable = "shared/observability" },
]
+mypy = [{ name = "apache-airflow-devel-common", extras = ["mypy"], editable = "devel-common" }]
[[package]]
name = "apache-airflow-shared-plugins-manager"
@@ -7992,6 +8080,9 @@ dev = [
{ name = "apache-airflow-shared-listeners" },
{ name = "apache-airflow-shared-module-loading" },
]
+mypy = [
+ { name = "apache-airflow-devel-common", extra = ["mypy"] },
+]
[package.metadata]
requires-dist = [
@@ -8007,6 +8098,7 @@ dev = [
{ name = "apache-airflow-shared-listeners", editable = "shared/listeners" },
{ name = "apache-airflow-shared-module-loading", editable = "shared/module_loading" },
]
+mypy = [{ name = "apache-airflow-devel-common", extras = ["mypy"], editable = "devel-common" }]
[[package]]
name = "apache-airflow-shared-providers-discovery"
@@ -8027,6 +8119,9 @@ dev = [
{ name = "apache-airflow-devel-common" },
{ name = "apache-airflow-shared-module-loading" },
]
+mypy = [
+ { name = "apache-airflow-devel-common", extra = ["mypy"] },
+]
[package.metadata]
requires-dist = [
@@ -8045,6 +8140,7 @@ dev = [
{ name = "apache-airflow-devel-common", editable = "devel-common" },
{ name = "apache-airflow-shared-module-loading", editable = "shared/module_loading" },
]
+mypy = [{ name = "apache-airflow-devel-common", extras = ["mypy"], editable = "devel-common" }]
[[package]]
name = "apache-airflow-shared-secrets-backend"
@@ -8055,11 +8151,15 @@ source = { editable = "shared/secrets_backend" }
dev = [
{ name = "apache-airflow-devel-common" },
]
+mypy = [
+ { name = "apache-airflow-devel-common", extra = ["mypy"] },
+]
[package.metadata]
[package.metadata.requires-dev]
dev = [{ name = "apache-airflow-devel-common", editable = "devel-common" }]
+mypy = [{ name = "apache-airflow-devel-common", extras = ["mypy"], editable = "devel-common" }]
[[package]]
name = "apache-airflow-shared-secrets-masker"
@@ -8076,6 +8176,9 @@ dependencies = [
dev = [
{ name = "apache-airflow-devel-common" },
]
+mypy = [
+ { name = "apache-airflow-devel-common", extra = ["mypy"] },
+]
[package.metadata]
requires-dist = [
@@ -8087,6 +8190,7 @@ requires-dist = [
[package.metadata.requires-dev]
dev = [{ name = "apache-airflow-devel-common", editable = "devel-common" }]
+mypy = [{ name = "apache-airflow-devel-common", extras = ["mypy"], editable = "devel-common" }]
[[package]]
name = "apache-airflow-shared-serialization"
@@ -8097,11 +8201,15 @@ source = { editable = "shared/serialization" }
dev = [
{ name = "apache-airflow-devel-common" },
]
+mypy = [
+ { name = "apache-airflow-devel-common", extra = ["mypy"] },
+]
[package.metadata]
[package.metadata.requires-dev]
dev = [{ name = "apache-airflow-devel-common", editable = "devel-common" }]
+mypy = [{ name = "apache-airflow-devel-common", extras = ["mypy"], editable = "devel-common" }]
[[package]]
name = "apache-airflow-shared-template-rendering"
@@ -8112,11 +8220,15 @@ source = { editable = "shared/template_rendering" }
dev = [
{ name = "apache-airflow-devel-common" },
]
+mypy = [
+ { name = "apache-airflow-devel-common", extra = ["mypy"] },
+]
[package.metadata]
[package.metadata.requires-dev]
dev = [{ name = "apache-airflow-devel-common", editable = "devel-common" }]
+mypy = [{ name = "apache-airflow-devel-common", extras = ["mypy"], editable = "devel-common" }]
[[package]]
name = "apache-airflow-shared-timezones"
@@ -8131,6 +8243,9 @@ dependencies = [
dev = [
{ name = "apache-airflow-devel-common" },
]
+mypy = [
+ { name = "apache-airflow-devel-common", extra = ["mypy"] },
+]
[package.metadata]
requires-dist = [
@@ -8140,6 +8255,7 @@ requires-dist = [
[package.metadata.requires-dev]
dev = [{ name = "apache-airflow-devel-common", editable = "devel-common" }]
+mypy = [{ name = "apache-airflow-devel-common", extras = ["mypy"], editable = "devel-common" }]
[[package]]
name = "apache-airflow-task-sdk"
@@ -8211,6 +8327,9 @@ dev = [
docs = [
{ name = "apache-airflow-devel-common", extra = ["docs"] },
]
+mypy = [
+ { name = "apache-airflow-devel-common", extra = ["mypy"] },
+]
[package.metadata]
requires-dist = [
@@ -8270,6 +8389,7 @@ dev = [
{ name = "pandas", marker = "python_full_version >= '3.14'", specifier = ">=2.3.3" },
]
docs = [{ name = "apache-airflow-devel-common", extras = ["docs"], editable = "devel-common" }]
+mypy = [{ name = "apache-airflow-devel-common", extras = ["mypy"], editable = "devel-common" }]
[[package]]
name = "apache-airflow-task-sdk-integration-tests"
@@ -8280,12 +8400,20 @@ dependencies = [
{ name = "apache-airflow-devel-common" },
]
+[package.dev-dependencies]
+mypy = [
+ { name = "apache-airflow-devel-common", extra = ["mypy"] },
+]
+
[package.metadata]
requires-dist = [
{ name = "apache-airflow-core", editable = "airflow-core" },
{ name = "apache-airflow-devel-common", editable = "devel-common" },
]
+[package.metadata.requires-dev]
+mypy = [{ name = "apache-airflow-devel-common", extras = ["mypy"], editable = "devel-common" }]
+
[[package]]
name = "apispec"
version = "6.10.0"