You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2026/04/19 13:54:27 UTC
[PR] Isolate non-provider mypy hooks per distribution with dedicated .build/ venvs [airflow]
potiuk opened a new pull request, #65492:
URL: https://github.com/apache/airflow/pull/65492
## Summary
Every non-provider mypy prek hook now lives in its own distribution's `.pre-commit-config.yaml`, runs in a dedicated virtualenv at `.build/mypy-venvs/<hook>/` with a dedicated mypy cache at `.build/mypy-caches/<hook>/`, and installs everything (including mypy itself) from the workspace `uv.lock` via a per-distribution `mypy` dep group — no more shared `.venv`, no more ephemeral `--with` overlays.
## What's new
- **Coverage** — adds mypy hooks for the non-provider directories that were previously uncovered: `airflow-ctl-tests`, `helm-tests`, `airflow-e2e-tests`, `task-sdk-integration-tests`, `docker-tests`, `kubernetes-tests`, plus one `mypy-shared-<dist>` hook per `shared/<dist>` workspace member (13 total). Each hook lives in its distribution's own `.pre-commit-config.yaml` — prek auto-discovers nested configs.
- **Split `mypy-dev` and `mypy-scripts`** — each has its own independent hook, its own dedicated venv, and its own `pyproject.toml`-level `[tool.mypy]` configuration. The script passes `--config-file <project>/pyproject.toml` for these two so their config takes precedence over the root one.
- **Mypy installed from the frozen lock** — every distribution with a mypy hook declares `mypy = ["apache-airflow-devel-common[mypy]"]` in its `[dependency-groups]`. The script syncs and runs mypy with `--frozen --group mypy`, so mypy itself, its plugins, and its stubs all resolve against the main workspace `uv.lock`.
- **`--follow-imports=silent`** — each hook only reports errors for files it owns; transitive code is covered by its own hook, avoiding divergent results across venvs on shared code.
- **`breeze down --cleanup-mypy-cache`** — also wipes `.build/mypy-venvs/` and `.build/mypy-caches/` alongside the existing `.mypy_cache` and `mypy-cache-volume`.
- **Selective-checks** — emits the right `skip-prek-hooks` set for every new hook, enumerating `shared/<dist>` dists at runtime so no hand-maintenance is needed.
- **Enforcement** — a new `check-shared-mypy-hooks` prek hook fails when a `shared/<dist>` workspace member is missing its dedicated `.pre-commit-config.yaml`, printing the exact YAML to paste. Contributing docs describe the two-step process for adding a new shared library.
## Type-error fixes
Pre-existing type errors surfaced by the newly added and cleaned-up checks are addressed: platform-specific ignores for Linux-only `os.posix_fadvise`, narrower types and `# type: ignore` markers in shared `configuration` / `observability` / `timezones` / `secrets_backend` / `secrets_masker`, Liskov override markers on the `AirflowConfigParser` subclass methods, and small correctness fixes in `dev/breeze` and the `docker-tests` / `kubernetes-tests` helpers. The full non-provider mypy suite now runs clean.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code (claude-opus-4-7)
Generated-by: Claude Code (claude-opus-4-7) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Isolate non-provider mypy hooks per distribution with dedicated .build/ venvs [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #65492:
URL: https://github.com/apache/airflow/pull/65492
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Isolate non-provider mypy hooks per distribution with dedicated .build/ venvs [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #65492:
URL: https://github.com/apache/airflow/pull/65492#issuecomment-4279789310
> Same with Python 3.10 but nothing that I see is by this PR, so it is improving here... LGTM! (Will need another PR to fix the other strange things showing on my side...)
You need to update `uv` to later version. The problem is that "min_uv_version" is there, but `4 days` cooldown causes it to ... well ... fail even parsing the main version 😱
However. I think I can improve that - and use the uv that is already installed in the local .venv - this one should always be good one - no matter which `uv` version you have on your path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Isolate non-provider mypy hooks per distribution with dedicated .build/ venvs [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #65492:
URL: https://github.com/apache/airflow/pull/65492#issuecomment-4280091139
### Backport failed to create: v3-2-test. View the failure log <a href='https://github.com/apache/airflow/actions/runs/24663033330'> Run details </a>
Note: As of [Merging PRs targeted for Airflow 3.X](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#merging-prs-targeted-for-airflow-3x)
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
In matter of doubt please ask in [#release-management](https://apache-airflow.slack.com/archives/C03G9H97MM2) Slack channel.
<table>
<tr>
<th>Status</th>
<th>Branch</th>
<th>Result</th>
</tr>
<tr>
<td>❌</td>
<td>v3-2-test</td>
<td><a href="https://github.com/apache/airflow/commit/4f3b228be1a785ea5df7b89d8d3291ded78fea4d"><img src='https://img.shields.io/badge/Commit-4f3b228-red' alt='Commit Link'></a></td>
</tr>
</table>
You can attempt to backport this manually by running:
```bash
cherry_picker 4f3b228 v3-2-test
```
This should apply the commit to the v3-2-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.
After you have resolved the conflicts, you can continue the backport process by running:
```bash
cherry_picker --continue
```
If you don't have cherry-picker installed, see the [installation guide](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#how-to-backport-pr-with-cherry-picker-cli).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Isolate non-provider mypy hooks per distribution with dedicated .build/ venvs [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #65492:
URL: https://github.com/apache/airflow/pull/65492#issuecomment-4280083712
Better way of handling `uv` versions is coming.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Isolate non-provider mypy hooks per distribution with dedicated .build/ venvs [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #65492:
URL: https://github.com/apache/airflow/pull/65492#issuecomment-4279774048
> Wow, almost 1.4 GB temp disk space is needed for all mypy caches :-O
>
> ```
> (apache-airflow-breeze) jscheffl@hp860g9:~/Workspace/airflow_review/.build$ du -sm mypy*
> 615 mypy-caches
> 781 mypy-venvs
> ```
Yeah .. that's a bit of a price to pay for speed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Isolate non-provider mypy hooks per distribution with dedicated .build/ venvs [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on PR #65492:
URL: https://github.com/apache/airflow/pull/65492#issuecomment-4276344558
Note, local re-run is still running... previously/currently on main I have problems with mypy, same @AutomationDev85 had on his PC on Friday... hope this is fixing mypy errors implicitly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Isolate non-provider mypy hooks per distribution with dedicated .build/ venvs [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on PR #65492:
URL: https://github.com/apache/airflow/pull/65492#issuecomment-4276380786
Wow, almost 1.4 GB temp disk space is needed for all mypy caches :-O
```
(apache-airflow-breeze) jscheffl@hp860g9:~/Workspace/airflow_review/.build$ du -sm mypy*
615 mypy-caches
781 mypy-venvs
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Isolate non-provider mypy hooks per distribution with dedicated .build/ venvs [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on PR #65492:
URL: https://github.com/apache/airflow/pull/65492#issuecomment-4276448972
Interesting, running in Linux and `prek run -a` git two errors, Python 3.13:
1)
```
Running hooks for `task-sdk-integration-tests`:
Run mypy for task-sdk-integration-tests...................................................................Failed
- hook id: mypy-task-sdk-integration-tests
- exit code: 2
Running mypy for folders: ['task-sdk-integration-tests']
Syncing dedicated mypy virtualenv (/home/jscheffl/Workspace/airflow_review/.build/mypy-venvs/task-sdk-integration-tests) for project task-sdk-integration-tests: uv sync --frozen --project task-sdk-integration-tests --group mypy
warning: Failed to parse `pyproject.toml` during settings discovery:
TOML parse error at line 1355, column 17
|
1355 | exclude-newer = "4 days"
| ^^^^^^^^
failed to parse year in date "4 days": failed to parse "4 da" as year (a four digit integer): invalid digit, expected 0-9 but got
warning: `VIRTUAL_ENV=/home/jscheffl/.cache/prek/hooks/python-NFPi8u10rFeZ5V6tnNNQ` does not match the project environment path `.build/mypy-venvs/task-sdk-integration-tests` and will be ignored; use `--active` to target the active environment instead
error: Group `mypy` is not defined in the project's `dependency-groups` table
`uv sync --frozen --project task-sdk-integration-tests` failed for the mypy virtualenv at /home/jscheffl/Workspace/airflow_review/.build/mypy-venvs/task-sdk-integration-tests. Fix the sync error before running mypy — otherwise the dedicated mypy virtualenv will not match uv.lock and results will diverge from CI. You can remove the cached virtualenv with:
breeze down --cleanup-mypy-cache
Mypy check failed. You can run mypy locally with:
prek run mypy-task-sdk-integration-tests --all-files
The hook uses dedicated virtualenv(s) and mypy cache(s) under .build/ so it does
not touch your regular project .venv. You can clear both with:
breeze down --cleanup-mypy-cache
```
...but running individually via `prek run -a mypy-task-sdk-integration-tests` is fine. Could reproduce.
2)
```
Generate Datamodels for TaskSDK client....................................................................Failed
- hook id: generate-tasksdk-datamodels
- files were modified by this hook
warning: Failed to parse `/home/jscheffl/Workspace/airflow_review/pyproject.toml` during settings discovery:
TOML parse error at line 1355, column 17
|
1355 | exclude-newer = "4 days"
| ^^^^^^^^
failed to parse year in date "4 days": failed to parse "4 da" as year (a four digit integer): invalid digit, expected 0-9 but got
Using CPython 3.12.3 interpreter at: /usr/bin/python3.12
Removed virtual environment at: /home/jscheffl/.cache/prek/hooks/python-YDYfQRMBSDtQiUlNIGnr
Creating virtual environment at: /home/jscheffl/.cache/prek/hooks/python-YDYfQRMBSDtQiUlNIGnr
Ignoring existing lockfile due to removal of timestamp cutoff: `2026-04-15T17:09:07.955557547Z`
Installed 220 packages in 289ms
```
maybe the same but unrelated to the change.
Willr epeat testing with Python 3.10...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Isolate non-provider mypy hooks per distribution with dedicated .build/ venvs [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #65492:
URL: https://github.com/apache/airflow/pull/65492#issuecomment-4282059836
> Same with Python 3.10 but nothing that I see is by this PR, so it is improving here... LGTM! (Will need another PR to fix the other strange things showing on my side...)
https://github.com/apache/airflow/pull/65531
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org