You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "namanjain24-sudo (via GitHub)" <gi...@apache.org> on 2026/09/14 14:06:49 UTC
[PR] Lower cross-provider dependencies in provider lowest-deps tests [airflow]
namanjain24-sudo opened a new pull request, #73129:
URL: https://github.com/apache/airflow/pull/73129
The lowest-deps provider tests sync with `--resolution lowest-direct`, but every other provider is a workspace member, so cross-provider dependencies always came from the checkout. A provider declaring `common-compat>=1.8.0` was really tested against the current `common-compat`.
uv can't be told to skip that for a few packages here: `--no-sources-package` fails in this workspace ("references a workspace in tool.uv.sources but is not a workspace member"), and dropping providers from the workspace breaks the lock while some providers pin versions that only exist as release candidates. So the lowering now happens right after the sync:
- `scripts/in_container/lower_cross_provider_dependencies.py` reads the tested provider's `[project].dependencies`. For each provider dependency with a lower bound it picks the lowest final (non-yanked) release that every installed distribution accepts, looking up releases on pypi.org, then repeats with the pinned releases' own requirements until the pins stop changing. Lines with `# use next version`, dependencies without a lower bound, and ones where only an rc satisfies the bounds keep the workspace version. It logs why each dependency was pinned or kept.
- `entrypoint_ci.sh` installs those pins and runs `uv pip check`. They are installed with their dependencies on purpose: an old `cncf-kubernetes` needs an old `kubernetes` client, and `--no-deps` would leave a broken environment.
Running this over all providers found two lower bounds that don't work, fixed here so the job stays green:
- `apache.flink` required `cncf-kubernetes>=5.1.0`, but 36 of the 39 releases from 5.1.0 to 8.4.2, including every one from 7.4.0 on, import `airflow.compat`, which Airflow 3 no longer has. flink's tests fail to import on those I tried (5.1.0, 7.4.0, 8.0.0–8.1.1, 8.3.0–8.4.2). 9.0.0 is the first release without that import, and the tests pass on 9.0.0, 10.0.0, 10.5.0, 10.6.0 and 10.12.2. Bumped to `>=9.0.0`.
- `airflow-core` requires `common-sql>=1.28.1`, but common-sql releases before 1.30.2 import `airflow.utils.module_loading`, removed in 3.2.0 by #59139. Bumped to `>=1.30.2`, the same way the pre-installed minimums were bumped in #56208. This also covers `apache.arrow` and `slack`, whose own bounds are lower.
What I ran:
| | |
|---|---|
| Every provider, locally (lowest-direct sync, lowering, `uv pip check`, tests; failing ones compared with a run without lowering) | 107 providers, 99 of them got pins (127 in total), `uv pip check` clean for every provider that synced. `apache.arrow` was the only failure caused by the lowering, fixed by the core bump above. Every other test that failed with the lowering also failed in a run without it. Providers that could not sync or run their tests on my Mac (native libraries, an SSH tunnel test) are in the Breeze row. |
| Breeze `--force-lowest-dependencies` | amazon, google, ibm.db2, odbc, sftp, ssh, vespa on this branch; apache.drill, apache.flink, ftp, postgres on an earlier revision of it (before the common-sql bump, which doesn't change their pins). All pass, `uv pip check` clean |
| `scripts/tests/in_container/test_lower_cross_provider_dependencies.py` | 22 passed |
closes: #60888
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: a Gen-AI coding assistant, following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions). I reviewed the change and ran the checks above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Lower cross-provider dependencies in provider lowest-deps tests [airflow]
Posted by "namanjain24-sudo (via GitHub)" <gi...@apache.org>.
namanjain24-sudo commented on PR #73129:
URL: https://github.com/apache/airflow/pull/73129#issuecomment-5722409927
Small ping on this one. It has been ready since Monday and the only red check is the Build info label gate rather than anything in the code.
The gate fires on the two provider `>=` bumps (`common-sql` 1.28.1 -> 1.30.2 in airflow-core, `cncf-kubernetes` 5.1.0 -> 9.0.0 in apache.flink), which selective checks reserve for release managers. They are in the PR because without them the lowest-deps jobs for apache.flink and apache.arrow fail once the cross-provider dependencies are actually lowered.
Two ways forward, whichever is easier to review: set the `allow provider dependency bump` label if the bumps look right, or I move those two lines into a separate PR and keep the lowering script here.
I rechecked it on current main today. The branch still merges cleanly and the 22 tests in `scripts/tests/in_container/test_lower_cross_provider_dependencies.py` pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Lower cross-provider dependencies in provider lowest-deps tests [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed pull request #73129: Lower cross-provider dependencies in provider lowest-deps tests
URL: https://github.com/apache/airflow/pull/73129
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Lower cross-provider dependencies in provider lowest-deps tests [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #73129:
URL: https://github.com/apache/airflow/pull/73129#issuecomment-5839391443
Hello @namanjain24-sudo - thank you for your contributions to Apache Airflow!
The Airflow community has introduced a limit of **5 open pull requests at a time** for contributors without write access to the repository. You currently have 7 open pull requests, so - as a one-time step of introducing the limit - we closed the ones where maintainers have not engaged yet:
* #73129 - Lower cross-provider dependencies in provider lowest-deps tests
* #73172 - Skip executor validation when a worker parses a Dag
* #73213 - Pass the scheduler's session to set_state when processing executor events
* #73356 - Allow DataflowJobStatusSensor to wait on a job name
* #73375 - Reset a mapped task's state store when its expansion is recomputed
* #73395 - Fail a task whose execute is interrupted by SIGTERM
These pull requests stay open because maintainers are already engaged in them - they count towards your limit:
* #73104 - Stop rebuilding the FAB app on every users and roles API call
This is **not** a judgement of you or of your changes. We never told contributors before that opening many pull requests at once was a problem, so there is nothing to feel bad about - and nothing is lost: your branches, commits and the review history stay where they are.
What we ask you to do is to make your **first prioritization decision**: choose which of the pull requests above matter most to you, and reopen them (up to 5 open at a time, including the ones still open) with the "Reopen pull request" button or `gh pr reopen <PR_NUMBER> --repo apache/airflow`. Reopen the ones you are ready to follow through - keep them rebased, respond to review comments and fix failing checks.
While your pull requests are waiting for review, the most valuable thing you can do is help in other ways - reviewing other contributors' pull requests, helping with issues, and taking part in the discussions on the devlist and Slack.
Why we introduced the limit, what it means for you and how to reopen or restore a pull request is explained in https://github.com/apache/airflow/blob/main/contributing-docs/32_open_pull_request_limit.rst.
---
Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org