You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2025/07/22 12:39:32 UTC
[PR] Remove dev dependencies from PyPI constraints. [airflow]
potiuk opened a new pull request, #53631:
URL: https://github.com/apache/airflow/pull/53631
Historically, `PyPI` constraints of airflow contained also the development dependencies of Airflow - because it was not very easy to separate and remove them from the "core" constraints.
This however might not be really needed and those dev tools could be removed from PyPI constraints.
There are however certain consequences:
a) currently when we generate PyPi constraints we generally *know* that the dependencies of ours have been tested at some point of time at least when we released the provider - because potentially some of our dev dependencies might limit some 3rd-party deps
b) if we remove devel deps when we generate the PyPI constraints this **might** change - i.e. removal of development dependencies might cause some of the transient dependencies to be upgraded to newer - potentially incompatible versions. It's not very likely but possible.
This means that if we remove devel deps, there is a higher risk that constraints published for users might contain dependency versions that has never been tested in our CI.
Another option considered is to use a) and b) and remove the dependencies missing in b) from a) - leaving the versions that were matching the versions used during CI. This however has limited impact on the users (just removes the "devel noise" from PyPI constraints
- and the noise that should not matter when installing airflow). And it has anothe risk that some edge cases might kick-in. For example there might be a case where our dependencies add or remove their transient dependencies, which might change status of those from devel to prod or the other way round. And it's quite a bit more complex.
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove dev dependencies from PyPI constraints. [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #53631:
URL: https://github.com/apache/airflow/pull/53631#issuecomment-3102551068
note - this is experimental.. I want to run it to see the impact it will bring, to be able to probably start discussion on devlist whether we can accept the risks involved (but I would like to have some real examples of consequences).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove dev dependencies from PyPI constraints. [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #53631:
URL: https://github.com/apache/airflow/pull/53631
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove dev dependencies from PyPI constraints. [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on PR #53631:
URL: https://github.com/apache/airflow/pull/53631#issuecomment-3104932971
> If there is a risk that we hat dev-dependencies not contained in constraints... would it help to split these out in a separate constraint file and keep this just for CI purposes? Don't know otherwise if the diff is easy to be calculated...
OMG, sorry, now reading the email thread... okay sorry - understand this is a bit more complex...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove dev dependencies from PyPI constraints. [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on PR #53631:
URL: https://github.com/apache/airflow/pull/53631#issuecomment-3104890459
If there is a risk that we hat dev-dependencies not contained in constraints... would it help to split these out in a separate constraint file and keep this just for CI purposes? Don't know otherwise if the diff is easy to be calculated...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove dev dependencies from PyPI constraints. [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #53631:
URL: https://github.com/apache/airflow/pull/53631#issuecomment-3108258204
> OMG, sorry, now reading the email thread... okay sorry - understand this is a bit more complex...
Yeah. It's not separate file - we **really** want to keep dev deps in pyproject.toml - and `dev` dependency group is a "standard" feature of `uv` and the nice thing about it is that it gets automatically installed when you run `uv sync` (for your current project that you are in working directory of) but `dev` dependnecies of any dependent projects are not. So we want to keep that workflow - it's nice and useful (for example when you `cd task-sdk` and run `uv sync` there - you will only get dev dependencies of task-sdk but no dev dependencies of `task-sdk` which is exactly what you expect - you want use "dev" dependencies of `task-sdk` for tests, but you want to just "use" the dependent libraries as if they were in production - so you do not want to pull their development dependencies.
And in this case it's not even that - the problem is tha those development dependencies might have their own dependnecies - and they might (but do not have to be) be production dependencies of other dependencies. Yes ... the word "dependency" is used far too often in the previous sentence.
In short. Yes. It's complex.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org