You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Fury0508 (via GitHub)" <gi...@apache.org> on 2026/01/17 19:33:57 UTC
[PR] Add uv sync --no-dev before provider YAML checks [airflow]
Fury0508 opened a new pull request, #60728:
URL: https://github.com/apache/airflow/pull/60728
**What change does this PR introduce?**
Strips dev dependencies before running provider YAML validation checks by running `uv sync --no-dev --all-packages`.
**Why is this change needed?**
Currently, the provider YAML check runs with all dependencies installed, including dev dependencies. This can mask issues where provider code imports cross-provider dependencies without proper optional handling (try/except blocks). By stripping dev dependencies first, I create an environment closer to production and can catch these issues during CI.
**Related issue(s)**
issue #60662
**Changes**
- Added `sync_dependencies_without_dev()` function that runs `uv sync --no-dev --all-packages` before provider checks
- Moved `jsonpath_ng` import from top-level to inside `check_providers_are_mentioned_in_issue_template()` as a lazy import (since `jsonpath_ng` is a dev dependency, the top-level import would fail after stripping dev dependencies)
**Testing**
- Function runs `uv sync --no-dev --all-packages --no-python-downloads --no-managed-python`
- Prints status messages for success/failure
- Script continues to work after dev dependencies are removed due to lazy import fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add uv sync --no-dev before provider YAML checks [airflow]
Posted by "Fury0508 (via GitHub)" <gi...@apache.org>.
Fury0508 commented on code in PR #60728:
URL: https://github.com/apache/airflow/pull/60728#discussion_r2701904920
##########
scripts/in_container/run_provider_yaml_files_check.py:
##########
@@ -101,6 +102,30 @@
suspended_integrations: set[str] = set()
+def sync_dependencies_without_dev() -> None:
Review Comment:
I've added the function call now ,it's invoked at the start of the main execution block.
After investigation, `jsonpath-ng` is a dependency of the amazon provider (not a dev-only dependency), so it remains available after `uv sync --no-dev` runs. The top-level import works correctly since provider dependencies stay installed during validation.
Thanks for the review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add uv sync --no-dev before provider YAML checks [airflow]
Posted by "Fury0508 (via GitHub)" <gi...@apache.org>.
Fury0508 commented on PR #60728:
URL: https://github.com/apache/airflow/pull/60728#issuecomment-3764273938
Hi! I've implemented a fix for this in PR #60728.
## The Problem
The script imports `jsonpath_ng` at the top level, but it's a dev dependency. Simply adding `uv sync --no-dev` would cause an `ImportError` when the script tries to use it.
## My Solution
1. **Added a `sync_dependencies_without_dev()` function** that runs:
```bash
uv sync --no-dev --all-packages --no-python-downloads --no-managed-python
```
This strips dev dependencies to create an environment closer to production, helping detect unhandled optional cross-provider dependencies.
2. **Converted the `jsonpath_ng` import to a lazy import** inside `check_providers_are_mentioned_in_issue_template()` to avoid the `ImportError`.
This approach ensures the validation runs without dev dependencies while still allowing the script to function when it needs `jsonpath_ng` for that specific check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add uv sync --no-dev before provider YAML checks [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on PR #60728:
URL: https://github.com/apache/airflow/pull/60728#issuecomment-3877072557
Somehow I missed this PR and created another to solve the same issue (#61713), which fails the pre-commit even before running the tests.
I think that it would be nice to have both to ensure that we're covered E2E.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add uv sync --no-dev before provider YAML checks [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on code in PR #60728:
URL: https://github.com/apache/airflow/pull/60728#discussion_r2701694662
##########
scripts/in_container/run_provider_yaml_files_check.py:
##########
@@ -101,6 +102,30 @@
suspended_integrations: set[str] = set()
+def sync_dependencies_without_dev() -> None:
Review Comment:
This function is not called anywhere? Or do I miss it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add uv sync --no-dev before provider YAML checks [airflow]
Posted by "Fury0508 (via GitHub)" <gi...@apache.org>.
Fury0508 commented on PR #60728:
URL: https://github.com/apache/airflow/pull/60728#issuecomment-3765216435
@bugraoz93 Thanks for pointing that out! I've updated the PR description with the Gen-AI disclosure section.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add uv sync --no-dev before provider YAML checks [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #60728:
URL: https://github.com/apache/airflow/pull/60728#issuecomment-3902308471
Let me rebase it with all the tests enabled and let's see if it passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add uv sync --no-dev before provider YAML checks [airflow]
Posted by "Fury0508 (via GitHub)" <gi...@apache.org>.
Fury0508 commented on PR #60728:
URL: https://github.com/apache/airflow/pull/60728#issuecomment-3813683360
@jason810496 I'm working on implementing the isolated temp venv but running into an issue I can't figure out.
Here's what I've done so far:
1. Create temp venv with `python -m venv`
2. Export non-dev deps: `uv export --no-dev --no-hashes -o requirements.txt`
3. Install them: `pip install -r requirements.txt` in the temp venv
4. Install Airflow itself: `pip install -e /opt/airflow` in the temp venv
5. Run validation function as subprocess using temp venv's python
But when the subprocess tries to import the validation function, it fails with:
```
ModuleNotFoundError: No module named 'jsonpath_ng'
```
Even though I can verify the packages are installed in the temp venv. The issue seems to be that the script has module-level imports like `from jsonpath_ng.ext import parse` at the top, and these fail before the function even runs.
I tried explicitly installing the script's direct dependencies (`jsonpath_ng`, `rich`, `tabulate`, etc.) but still hitting the same issue.
Am I approaching the temp venv creation wrong? Should I be using `uv sync` differently, or is there a specific way to install Airflow + its dependencies in the temp venv that I'm missing?
Any guidance would be helpful - I feel like I'm close but missing something about how to properly set up the isolated environment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add uv sync --no-dev before provider YAML checks [airflow]
Posted by "Fury0508 (via GitHub)" <gi...@apache.org>.
Fury0508 commented on PR #60728:
URL: https://github.com/apache/airflow/pull/60728#issuecomment-3809099804
> > alwas need to re-sync my UV venv... is it intended by pre-commit checks to change / alter venvs? Or shall a temporary secondary be created for this check then?
>
> Yeah, I also feel a bit strange at the first glance, which might also introduces some dependencies environment race condition issue from my perspective (e.g. other in-container pre-commit checks depend on the dev dependencies while current script is running).
>
> I think the better way might be
>
> 1. Creating a new `venv` under a tmp dir and install the dependencies without dev
> 2. Extract the current `__main__` logic as another function (excluding the `sync_dependencies_without_dev` call) into a separate function named `provider_yaml_files_check`.
> 3. Run `sync_dependencies_without_dev` for the `venv` under tmp dir.
> 4. Run `provider_yaml_files_check` as a subprocess using the virtual environment we just created under the temporary path, ensuring that it runs without development dependencies (for example, by overriding `PYTHONPATH`, etc.).
>
> to avoid the race condition and environment side effect.
Thanks for the detailed feedback @jason810496! You're absolutely right about the race condition and environment side effects.
I'll refactor to use an isolated temporary venv as you suggested:
1. Create temp venv with non-dev dependencies
2. Extract validation logic to `provider_yaml_files_check()`
3. Run as subprocess using the temp environment
Does this sound like the right direction? Happy to implement if this approach works for the team.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add uv sync --no-dev before provider YAML checks [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #60728:
URL: https://github.com/apache/airflow/pull/60728#issuecomment-3843094262
Actually - ideally the whole check should be split into runnnig a check "per-provider" - and before every check you should do:
```
cd <PROVIDER_FOLDER>
uv sync --no-dev
# HERE import for all files in this provider should happen
```
We can use the feature of `uv sync` and workspace where we can automatically sync all the provider and all it's dependencies (including transitive) by simply entering the distribution folder and runnin `uv sync`
Alternativelly the same can be done by specifying `--project` explicitly in `uv sync` in the root folder, but going to the distribution and running `uv sync` is way better.
You can find out more about it from my recent FOSDEM presentation: https://fosdem.org/2026/schedule/event/WE7NHM-modern-python-monorepo-apache-airflow/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add uv sync --no-dev before provider YAML checks [airflow]
Posted by "Fury0508 (via GitHub)" <gi...@apache.org>.
Fury0508 commented on PR #60728:
URL: https://github.com/apache/airflow/pull/60728#issuecomment-3804480758
Hi @jscheffl, thanks for the approval! This PR has been approved and all checks pass. Just wondering if there's anything else needed before merge, or if it's waiting in the merge queue. Thanks for your time!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add uv sync --no-dev before provider YAML checks [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on PR #60728:
URL: https://github.com/apache/airflow/pull/60728#issuecomment-3807483672
Yes as written ^^^- I am okay merging and thus approved, would liek another pair of eyes as reviewer before merge as being not 100% sure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add uv sync --no-dev before provider YAML checks [airflow]
Posted by "Fury0508 (via GitHub)" <gi...@apache.org>.
Fury0508 commented on PR #60728:
URL: https://github.com/apache/airflow/pull/60728#issuecomment-3765223435
> Looks OK for me.
>
> Just a bit unsure - so would request another pair of eyes - I saw it also today (before this PR) that my venv is "wiped" after running soe prek commands... alwas need to re-sync my UV venv... is it intended by pre-commit checks to change / alter venvs? Or shall a temporary secondary be created for this check then?
>
> Otherwise LGTM
Good point! I didn't think about how this would affect local development - only focused on catching the dependency issues in CI.
So yeah, wiping the venv every time someone runs pre-commit hooks is definitely annoying. Should this maybe only run in CI and not locally? Or would creating a temp venv for just this check work better?
Not sure what's the best fix here - happy to adjust based on what you think makes sense!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add uv sync --no-dev before provider YAML checks [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #60728:
URL: https://github.com/apache/airflow/pull/60728
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add uv sync --no-dev before provider YAML checks [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on PR #60728:
URL: https://github.com/apache/airflow/pull/60728#issuecomment-3809740927
> Does this sound like the right direction? Happy to implement if this approach works for the team.
Yes, thanks for the follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add uv sync --no-dev before provider YAML checks [airflow]
Posted by "bugraoz93 (via GitHub)" <gi...@apache.org>.
bugraoz93 commented on PR #60728:
URL: https://github.com/apache/airflow/pull/60728#issuecomment-3765189734
https://github.com/apache/airflow/blob/main/contributing-docs%2F05_pull_requests.rst#gen-ai-assisted-contributions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add uv sync --no-dev before provider YAML checks [airflow]
Posted by "bugraoz93 (via GitHub)" <gi...@apache.org>.
bugraoz93 commented on PR #60728:
URL: https://github.com/apache/airflow/pull/60728#issuecomment-3765276665
> @bugraoz93 Thanks for pointing that out! I've updated the PR description with the Gen-AI disclosure section.
Thanks @Fury0508!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add uv sync --no-dev before provider YAML checks [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on PR #60728:
URL: https://github.com/apache/airflow/pull/60728#issuecomment-3813049367
> > Does this sound like the right direction? Happy to implement if this approach works for the team.
>
> Yes, thanks for the follow-up.
Thanks for raising the point of race conditions of multiple checks using the same venv. Did not consider this but will be important!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add uv sync --no-dev before provider YAML checks [airflow]
Posted by "Fury0508 (via GitHub)" <gi...@apache.org>.
Fury0508 commented on PR #60728:
URL: https://github.com/apache/airflow/pull/60728#issuecomment-3877405998
@potiuk - You're right that running uv sync at the distribution folder level would be a cleaner approach. I can see how using the workspace feature would automatically sync all provider dependencies. I'll look into the FOSDEM presentation you linked to better understand the implementation.
@shahar1 - I see you created #61713 which handles the pre-commit validation side. That makes sense to have both layers:
Your PR catches issues at pre-commit stage (faster feedback)
This PR ensures we have the full E2E validation when running the actual YAML checks
Current status:
This PR is approved and all checks are passing
However, based on potiuk's suggestion, I think we should refactor this to use the uv sync approach at the provider folder level rather than the current --no-dev flag implementation
Some work remains to implement this better approach
Sorry for the delay in my work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org