You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2025/11/18 14:37:05 UTC
[PR] Manages cross-distribution dependencies better [airflow]
potiuk opened a new pull request, #58430:
URL: https://github.com/apache/airflow/pull/58430
This PR addresses the case where we have to manage dependencies between Airfow distributions and we have to manage the right lower bind versions for them.
Since we missed a comprehensive documentation on how our dependencies are managed in general, and we had no good place to describe it, this PR not only adds code to manage it, but also extends the documentation of the "12_airflw_dependencies_and_extras.rst" chapter of the contributing documentation to include comprehensive (but also hopefully helpful) documentation explaining to contributors how our dependencies are managed.
The changes in the doc:
* add missing documentation explaining dependency management in airflow distributions in the monorepo
* adds selective check to verify if cross-airflow-distribution dependencies have not been modified in the PR, and fails if they were, unless appropriate label is set on the PR
* adds breeze tests covering the test cases of selective check
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Manages cross-distribution dependencies better [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #58430:
URL: https://github.com/apache/airflow/pull/58430#discussion_r2539218451
##########
dev/breeze/tests/test_selective_checks.py:
##########
@@ -2779,3 +2779,169 @@ def test_testable_providers_integrations_excludes_arm_disabled_on_arm():
assert "postgres" in result
assert "trino" not in result
assert "ydb" not in result
+
+
+@patch("airflow_breeze.utils.selective_checks.run_command")
+def test_provider_dependency_bump_check_no_changes(mock_run_command):
+ """Test that provider dependency bump check passes when no pyproject.toml files are changed."""
+ selective_checks = SelectiveChecks(
+ files=("some_other_file.py",),
+ commit_ref=NEUTRAL_COMMIT,
+ pr_labels=(),
+ github_event=GithubEvents.PULL_REQUEST,
+ default_branch="main",
+ )
+ result = selective_checks.provider_dependency_bump
+ assert result is False
+
+
+@patch("airflow_breeze.utils.selective_checks.run_command")
+def test_provider_dependency_bump_check_fails_on_provider_version_bump(mock_run_command):
+ """Test that provider dependency bump check fails when provider version is bumped without label."""
+ old_toml = """
+[project]
+dependencies = [
+ "apache-airflow-providers-common-sql>=1.0.0",
+]
+"""
+ new_toml = """
+[project]
+dependencies = [
+ "apache-airflow-providers-common-sql>=1.1.0",
+]
+"""
+
+ def side_effect(*args, **kwargs):
+ result = Mock()
+ result.returncode = 0
+ if "^:" in args[0][2]:
+ result.stdout = old_toml
+ else:
+ result.stdout = new_toml
+ return result
+
+ mock_run_command.side_effect = side_effect
+
+ with pytest.raises(SystemExit):
+ SelectiveChecks(
+ files=("providers/amazon/pyproject.toml",),
+ commit_ref=NEUTRAL_COMMIT,
+ pr_labels=(),
+ github_event=GithubEvents.PULL_REQUEST,
+ default_branch="main",
+ ).provider_dependency_bump
+
+
+@patch("airflow_breeze.utils.selective_checks.run_command")
+def test_provider_dependency_bump_check_passes_with_label(mock_run_command):
+ """Test that provider dependency bump check passes when label is set."""
+ old_toml = """
+[project]
+dependencies = [
+ "apache-airflow-providers-common-sql>=1.0.0",
+]
+"""
+ new_toml = """
+[project]
+dependencies = [
+ "apache-airflow-providers-common-sql>=1.1.0",
+]
+"""
+
+ def side_effect(*args, **kwargs):
+ result = Mock()
+ result.returncode = 0
+ if "^:" in args[0][2]:
+ result.stdout = old_toml
+ else:
+ result.stdout = new_toml
+ return result
+
+ mock_run_command.side_effect = side_effect
+
+ selective_checks = SelectiveChecks(
+ files=("providers/amazon/pyproject.toml",),
+ commit_ref=NEUTRAL_COMMIT,
+ pr_labels=("allow provider dependency bump",),
+ github_event=GithubEvents.PULL_REQUEST,
+ default_branch="main",
+ )
+ result = selective_checks.provider_dependency_bump
+ assert result is True
+
+
+@patch("airflow_breeze.utils.selective_checks.run_command")
+def test_provider_dependency_bump_check_passes_on_non_provider_dependency_changes(mock_run_command):
+ """Test that provider dependency bump check passes when non-provider dependencies change."""
+ old_toml = """
+[project]
+dependencies = [
+ "apache-airflow>=2.10.0",
+ "boto3>=1.37.0",
+]
+"""
+ new_toml = """
+[project]
+dependencies = [
+ "apache-airflow>=2.10.0",
+ "boto3>=1.38.0",
+]
+"""
+
+ def side_effect(*args, **kwargs):
+ result = Mock()
+ result.returncode = 0
+ if "^:" in args[0][2]:
+ result.stdout = old_toml
+ else:
+ result.stdout = new_toml
+ return result
+
+ mock_run_command.side_effect = side_effect
+
+ selective_checks = SelectiveChecks(
+ files=("providers/amazon/pyproject.toml",),
+ commit_ref=NEUTRAL_COMMIT,
+ pr_labels=(),
+ github_event=GithubEvents.PULL_REQUEST,
+ default_branch="main",
+ )
+ result = selective_checks.provider_dependency_bump
+ assert result is False
+
+
+@patch("airflow_breeze.utils.selective_checks.run_command")
+def test_provider_dependency_bump_check_in_optional_dependencies(mock_run_command):
Review Comment:
Nope. They are not. The test actually works as expected here - selective check will raise SystemExist when you modify also optional dependencies.
Or am I missing something?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Manages cross-distribution dependencies better [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #58430:
URL: https://github.com/apache/airflow/pull/58430#discussion_r2538486434
##########
contributing-docs/13_airflow_dependencies_and_extras.rst:
##########
@@ -18,6 +18,226 @@
Airflow dependencies
====================
+This document describes how we manage Apache Airflow dependencies, as well as how we make sure
+that users can use Airflow both as an application and as a library when they are deploying
+their own Airflow - using our constraints mechanism.
+
+Airflow ``pyproject.toml`` files and ``uv`` workspace
+.....................................................
+
+Managing dependencies is an important part of maintaining Apache Airflow, we have more than 700
Review Comment:
```suggestion
Managing dependencies is an important part of developing Apache Airflow, we have more than 700
```
To me, maintaining Airflow is things you do with a running Airflow deployment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Manages cross-distribution dependencies better [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #58430:
URL: https://github.com/apache/airflow/pull/58430#discussion_r2539187453
##########
contributing-docs/13_airflow_dependencies_and_extras.rst:
##########
@@ -18,6 +18,226 @@
Airflow dependencies
====================
+This document describes how we manage Apache Airflow dependencies, as well as how we make sure
+that users can use Airflow both as an application and as a library when they are deploying
+their own Airflow - using our constraints mechanism.
+
+Airflow ``pyproject.toml`` files and ``uv`` workspace
+.....................................................
+
+Managing dependencies is an important part of maintaining Apache Airflow, we have more than 700
+dependencies, and often when you add a new feature that requires a new dependency, you should
+also update the dependency list. This also happens when you want to add a new tool that is used
+for development, testing, or building the documentation and this document describes how to do it.
+
+When it comes to defining dependencies for any of the Apache Airflow distributions, we are following the
+standard ``pyproject.toml`` format that has been defined in `PEP 518 <https://peps.python.org/pep-0518/>`_
+and `PEP 621 <https://peps.python.org/pep-0621/>`_, we are also using dependency groups defined in
+`PEP 735 <https://peps.python.org/pep-0735/>`_ - particularly ``dev`` dependency group which we use in
+all our ``pyproject.toml`` files to define development dependencies.
+
+We have more than 100 python distributions in Apache Airflow repository - including the main
+``apache-airflow`` package, ``apache-airflow-providers-*`` packages, ``apache-airflow-core`` package,
+``apache-airflow-task-sdk`` package, ``apache-airflow-ctl`` package, and several other packages.
+
+They are all connected together via ``uv`` workspace feature (workspace is defined in the root ``pyproject.toml``
+file of the repository in the ``apache-airflow`` distribution definition. The workspace feature allows us to
+run ``uv sync`` at the top of the repository to install all packages in editable mode in the development
+environment from all the 100+ distributions (!) and resolve the dependencies together, so that we know that
+the dependencies have no conflicting requirements. Also the distributions are referring to each other via
+name - which means that when you run locally ``uv sync``, the local version of the packages are used, not the
+ones released on PyPI, which means that you can develop and test changes that span multiple packages at the
+same time. This is a very powerful feature that allows us to maintain the complex ecosystem of Apache Airflow
+distributions in a single monorepo, and allows us - for example to add new feature to common distributions used
+by multiple providers and test them all together before releasing new versions of either of those pacckages
+
+Managing dependencies in ``pyproject.toml`` files
+.................................................
+
+Each of the ``pyproject.toml`` files in Apache Airflow repository defines dependencies in one of the
+following sections:
+
+* ``[project.dependencies]`` - this section defines the required dependencies of the package. These
+ dependencies are installed when you install the package without any extras.
+* ``[project.optional-dependencies]`` - this section defines optional dependencies (extras) of the package.
+ These dependencies are installed when you install the package with extras - for example
+ ``pip install apache-airflow[ssh]`` will install the ``ssh`` extra dependencies defined in this section.
+* ``[dependency-group.dev]`` - this section defines development dependencies of the package.
+ These dependencies are installed when you run ``uv sync`` by default. when ``uv`` syncs sources with
+ local pyproject.toml it adds ``dev`` dependency group and package is installed in editable mode with
+ development dependencies.
+
+
+Adding and modifying dependencies
+.................................
+
+Adding and modifying dependencies in Apache Airflow is done by modifying the appropriate
+``pyproject.toml`` file in the appropriate distribution.
+
+When you add a new dependency, you should make sure that:
+
+* The dependency is added to the right section (main dependencies, optional dependencies, or
+ development dependencies)
+
+* Some parts of those dependencies might be automatically generated (and overwritten) by our ``prek``
+ hooks. Those are the necessary dependencies that ``prek`` hoks can figure out automatically by
+ analyzing the imports in the sources and structure of the project. We also have special case of
+ shared dependencies (described in `shared dependencies document <../shared/README.md>`__) where we
+ effectively "static-link" some libraries into multiple distributions, to avoid unnecessary coupling and
+ circular dependencies, and those dependencies contribute some dependencies automatically as well.
+ Pay attention to comments such us example start and end of such generated block below.
+ All the dependencies between those comments are automatically generated and you should not modify
+ them manually. Tou might instead modify the root source of those dependencies - depending on the automation,
+ you can usually also add extra dependencies manually outside of those comment blocks
+
+ .. code-block:: python
+
+ # Automatically generated airflow optional dependencies (update_airflow_pyproject_toml.py)
+ # ....
+ # LOTS OF GENERATED DEPENDENCIES HERE
+ # ....
+ # End of automatically generated airflow optional dependencies
+
+* The version specifier is as open as possible (upper) while still allowing the package to install
+ and pass all tests. We very rarely upper-bind dependencies - only when there is a known
+ conflict with a new or upcoming version of a dependency that breaks the installation or tests
+ (and we always make a comment why we are upper-bounding a dependency).
+
+* Make sure to lower-bind any dependency you add. Usually we lower-bind dependencies to the
+ minimum version that is required for the package to work but in order to simplify the work of
+ resolvers such as ``pip``, we often lower-bind to higher (and newer) version than the absolute minimum
+ especially when the minimum version is very old. This is for example good practice in ``boto`` and related
+ packages, where new version of those packages are released frequently (almost daily) and there are many
+ versions that need to be considered by the resolver if the version is not new enough.
+
+* Make sure to run ``uv sync`` after modifying dependencies to make sure that there are no
+ conflicts between dependencies of different packages in the workspace. You can run it in multiple
+ ways - either from the root of the repository (which will sync all packages) or from the package
+ you modified (which will sync only that package and its dependencies). Also good idea might be to
+ run ``uv sync --all-packages --all-extras`` at the root of the repository to make sure that
+ all packages with all extras can be installed together without conflicts, but this might be sometimes
+ difficult and slow as some of the extras require some additional system level dependencies to be installed
+ (for example ``mysql`` or ``postgres`` extras require client libraries to be installed on the system).
+
+* Make sure to run all tests after modifying dependencies to make sure that nothing is broken
+
+
+Referring to other Apache Airflow distributions in dependencies
+...............................................................
+
+With having more than 100 distributions in the repository, it is often necessary to refer to
+other distributions in the dependencies in order to use some common features or simply to use the
+features that the other distribution provides. There are two ways of doing it:
Review Comment:
Again - repetitions are kind of expected and built in.
I 100% do **NOT** expect people to read that as a book. Or even actually read it from the beginning to an end even once. Not even close.
I do expect people to look for particular issue they have and find the decription about it without them going back to the beginning of the doc and reading it from the very beginning (but I do expect AI agents reading all of it and making it part of their context) - so sometimss repetitions are very deliberate in this docs.
But yeah - especially that we might have 200+ soon, maybe repeating the number itself makes no sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Manages cross-distribution dependencies better [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #58430:
URL: https://github.com/apache/airflow/pull/58430#issuecomment-3549041236
> > Docs are longer than I'd like, but make sense.
>
> The thing is that AI actually likes detailed context. This documentation is also for AI agents to understand what they should do by reading the documentation. Again - i do not expect people to read them like a book, more like trying to find what they are looking for - but I do 100% expect it will be read by Gen-AI tools in it's entirety to build the right context for whatever they will be doing.
Speaking of which... When I realized and checked how Agents work, I kind of started to see why I added such a detailed documentation for all things in our env. Thanks to this I do not even have to ask Cloude how to run the tests or build breeze image and run things sometimes - it just **knows** by reading our docs and I do not have to explain it in my prompts.
It happened multiple times when I was using agents, that they added tests and iterated on them and created a number of them following the structure and description of our test environment without me even asking for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Manages cross-distribution dependencies better [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #58430:
URL: https://github.com/apache/airflow/pull/58430#discussion_r2539192644
##########
contributing-docs/13_airflow_dependencies_and_extras.rst:
##########
@@ -18,6 +18,226 @@
Airflow dependencies
====================
+This document describes how we manage Apache Airflow dependencies, as well as how we make sure
+that users can use Airflow both as an application and as a library when they are deploying
+their own Airflow - using our constraints mechanism.
+
+Airflow ``pyproject.toml`` files and ``uv`` workspace
+.....................................................
+
+Managing dependencies is an important part of maintaining Apache Airflow, we have more than 700
+dependencies, and often when you add a new feature that requires a new dependency, you should
+also update the dependency list. This also happens when you want to add a new tool that is used
+for development, testing, or building the documentation and this document describes how to do it.
+
+When it comes to defining dependencies for any of the Apache Airflow distributions, we are following the
+standard ``pyproject.toml`` format that has been defined in `PEP 518 <https://peps.python.org/pep-0518/>`_
+and `PEP 621 <https://peps.python.org/pep-0621/>`_, we are also using dependency groups defined in
+`PEP 735 <https://peps.python.org/pep-0735/>`_ - particularly ``dev`` dependency group which we use in
+all our ``pyproject.toml`` files to define development dependencies.
+
+We have more than 100 python distributions in Apache Airflow repository - including the main
+``apache-airflow`` package, ``apache-airflow-providers-*`` packages, ``apache-airflow-core`` package,
+``apache-airflow-task-sdk`` package, ``apache-airflow-ctl`` package, and several other packages.
+
+They are all connected together via ``uv`` workspace feature (workspace is defined in the root ``pyproject.toml``
+file of the repository in the ``apache-airflow`` distribution definition. The workspace feature allows us to
+run ``uv sync`` at the top of the repository to install all packages in editable mode in the development
+environment from all the 100+ distributions (!) and resolve the dependencies together, so that we know that
+the dependencies have no conflicting requirements. Also the distributions are referring to each other via
+name - which means that when you run locally ``uv sync``, the local version of the packages are used, not the
+ones released on PyPI, which means that you can develop and test changes that span multiple packages at the
+same time. This is a very powerful feature that allows us to maintain the complex ecosystem of Apache Airflow
+distributions in a single monorepo, and allows us - for example to add new feature to common distributions used
+by multiple providers and test them all together before releasing new versions of either of those pacckages
+
+Managing dependencies in ``pyproject.toml`` files
+.................................................
+
+Each of the ``pyproject.toml`` files in Apache Airflow repository defines dependencies in one of the
+following sections:
+
+* ``[project.dependencies]`` - this section defines the required dependencies of the package. These
+ dependencies are installed when you install the package without any extras.
+* ``[project.optional-dependencies]`` - this section defines optional dependencies (extras) of the package.
+ These dependencies are installed when you install the package with extras - for example
+ ``pip install apache-airflow[ssh]`` will install the ``ssh`` extra dependencies defined in this section.
+* ``[dependency-group.dev]`` - this section defines development dependencies of the package.
+ These dependencies are installed when you run ``uv sync`` by default. when ``uv`` syncs sources with
+ local pyproject.toml it adds ``dev`` dependency group and package is installed in editable mode with
+ development dependencies.
+
+
+Adding and modifying dependencies
+.................................
+
+Adding and modifying dependencies in Apache Airflow is done by modifying the appropriate
+``pyproject.toml`` file in the appropriate distribution.
+
+When you add a new dependency, you should make sure that:
+
+* The dependency is added to the right section (main dependencies, optional dependencies, or
+ development dependencies)
+
+* Some parts of those dependencies might be automatically generated (and overwritten) by our ``prek``
+ hooks. Those are the necessary dependencies that ``prek`` hoks can figure out automatically by
+ analyzing the imports in the sources and structure of the project. We also have special case of
+ shared dependencies (described in `shared dependencies document <../shared/README.md>`__) where we
+ effectively "static-link" some libraries into multiple distributions, to avoid unnecessary coupling and
+ circular dependencies, and those dependencies contribute some dependencies automatically as well.
+ Pay attention to comments such us example start and end of such generated block below.
+ All the dependencies between those comments are automatically generated and you should not modify
+ them manually. Tou might instead modify the root source of those dependencies - depending on the automation,
+ you can usually also add extra dependencies manually outside of those comment blocks
+
+ .. code-block:: python
+
+ # Automatically generated airflow optional dependencies (update_airflow_pyproject_toml.py)
+ # ....
+ # LOTS OF GENERATED DEPENDENCIES HERE
+ # ....
+ # End of automatically generated airflow optional dependencies
+
+* The version specifier is as open as possible (upper) while still allowing the package to install
+ and pass all tests. We very rarely upper-bind dependencies - only when there is a known
+ conflict with a new or upcoming version of a dependency that breaks the installation or tests
+ (and we always make a comment why we are upper-bounding a dependency).
+
+* Make sure to lower-bind any dependency you add. Usually we lower-bind dependencies to the
+ minimum version that is required for the package to work but in order to simplify the work of
+ resolvers such as ``pip``, we often lower-bind to higher (and newer) version than the absolute minimum
+ especially when the minimum version is very old. This is for example good practice in ``boto`` and related
+ packages, where new version of those packages are released frequently (almost daily) and there are many
+ versions that need to be considered by the resolver if the version is not new enough.
+
+* Make sure to run ``uv sync`` after modifying dependencies to make sure that there are no
+ conflicts between dependencies of different packages in the workspace. You can run it in multiple
+ ways - either from the root of the repository (which will sync all packages) or from the package
+ you modified (which will sync only that package and its dependencies). Also good idea might be to
+ run ``uv sync --all-packages --all-extras`` at the root of the repository to make sure that
+ all packages with all extras can be installed together without conflicts, but this might be sometimes
+ difficult and slow as some of the extras require some additional system level dependencies to be installed
+ (for example ``mysql`` or ``postgres`` extras require client libraries to be installed on the system).
+
+* Make sure to run all tests after modifying dependencies to make sure that nothing is broken
+
+
+Referring to other Apache Airflow distributions in dependencies
+...............................................................
+
+With having more than 100 distributions in the repository, it is often necessary to refer to
+other distributions in the dependencies in order to use some common features or simply to use the
+features that the other distribution provides. There are two ways of doing it:
+
+* Regular package linking with ``apache-airflow-*`` dependency
+* Airflow "shared dependencies" mechanism - which is a bit of a custom hack for Airflow monorepo
+ that allows us to "static-link" some common dependencies into multiple distributions without
+ creating circular dependencies.
+
+We are not going to describe the shared dependencies mechanism here, please refer to the
+`shared dependencies document <../shared/README.md>`__ for details, but there are certain rules
+when it comes to referring to other Airflow distributions in dependencies - here are the important
+rules to remember:
+
+* You can refer to other distributions in your dependencies - as usual using distribution name. For example,
+ if you are adding a dependency to ``apache-airflow-providers-common-compat`` package from
+ ``apache-airflow-providers-google``, you can just add ``apache-airflow-providers-common>=x.y.z`` to the
+ dependencies and when you run ``uv sync``, the local version of the package will be used automatically
+ (this is thanks to the workspace feature of ``uv`` that does great job of binding our monorepo together).
+ This is very useful when you are developing changes to multiple packages at the same time. Some of those
+ are added automatically by prek hooks - when it can detect such dependencies by analyzing imports in the
+ sources - then they are added automatically between the special comments mentioned above, but sometimes
+ (especially when such dependencies are not at the top-level imports) you might need to add them manually.
Review Comment:
Yeah. Taht was too much.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Manages cross-distribution dependencies better [airflow]
Posted by "bugraoz93 (via GitHub)" <gi...@apache.org>.
bugraoz93 commented on PR #58430:
URL: https://github.com/apache/airflow/pull/58430#issuecomment-3549827481
> > Look good to me! I totally agree that not many contributors will read this. Since the documents are public, someone can summarise them easily. Selective checks for the providers are definitely a better place for this check rather than `prek`. We don't need selective checks for other distributions since they are easy to follow as long as RMs are CodeOwners on those repositories. It would be beneficial to have a list of who can be tagged for which distribution as an RM list, perhaps. It would be easier to check and ping the correct people for Commiters, PMCs, but mostly contributors. It would also be informative in general and serve as an Onboarding piece for new members
>
> I think just #releae-management is good enough (I mentioned it in the description) - I guess keeping list of RMs separately is less-than-needed if everyone is in #release-management channel. Those are rather "rare" events to handle and I guess there is little issue with whole #release-management being notified (especially also for redundancy reasons, vacations and so on).
True, make sense!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Manages cross-distribution dependencies better [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #58430:
URL: https://github.com/apache/airflow/pull/58430#discussion_r2539355597
##########
dev/breeze/tests/test_selective_checks.py:
##########
@@ -2779,3 +2779,169 @@ def test_testable_providers_integrations_excludes_arm_disabled_on_arm():
assert "postgres" in result
assert "trino" not in result
assert "ydb" not in result
+
+
+@patch("airflow_breeze.utils.selective_checks.run_command")
+def test_provider_dependency_bump_check_no_changes(mock_run_command):
+ """Test that provider dependency bump check passes when no pyproject.toml files are changed."""
+ selective_checks = SelectiveChecks(
+ files=("some_other_file.py",),
+ commit_ref=NEUTRAL_COMMIT,
+ pr_labels=(),
+ github_event=GithubEvents.PULL_REQUEST,
+ default_branch="main",
+ )
+ result = selective_checks.provider_dependency_bump
+ assert result is False
+
+
+@patch("airflow_breeze.utils.selective_checks.run_command")
+def test_provider_dependency_bump_check_fails_on_provider_version_bump(mock_run_command):
+ """Test that provider dependency bump check fails when provider version is bumped without label."""
+ old_toml = """
+[project]
+dependencies = [
+ "apache-airflow-providers-common-sql>=1.0.0",
+]
+"""
+ new_toml = """
+[project]
+dependencies = [
+ "apache-airflow-providers-common-sql>=1.1.0",
+]
+"""
+
+ def side_effect(*args, **kwargs):
+ result = Mock()
+ result.returncode = 0
+ if "^:" in args[0][2]:
+ result.stdout = old_toml
+ else:
+ result.stdout = new_toml
+ return result
+
+ mock_run_command.side_effect = side_effect
+
+ with pytest.raises(SystemExit):
+ SelectiveChecks(
+ files=("providers/amazon/pyproject.toml",),
+ commit_ref=NEUTRAL_COMMIT,
+ pr_labels=(),
+ github_event=GithubEvents.PULL_REQUEST,
+ default_branch="main",
+ ).provider_dependency_bump
+
+
+@patch("airflow_breeze.utils.selective_checks.run_command")
+def test_provider_dependency_bump_check_passes_with_label(mock_run_command):
+ """Test that provider dependency bump check passes when label is set."""
+ old_toml = """
+[project]
+dependencies = [
+ "apache-airflow-providers-common-sql>=1.0.0",
+]
+"""
+ new_toml = """
+[project]
+dependencies = [
+ "apache-airflow-providers-common-sql>=1.1.0",
+]
+"""
+
+ def side_effect(*args, **kwargs):
+ result = Mock()
+ result.returncode = 0
+ if "^:" in args[0][2]:
+ result.stdout = old_toml
+ else:
+ result.stdout = new_toml
+ return result
+
+ mock_run_command.side_effect = side_effect
+
+ selective_checks = SelectiveChecks(
+ files=("providers/amazon/pyproject.toml",),
+ commit_ref=NEUTRAL_COMMIT,
+ pr_labels=("allow provider dependency bump",),
+ github_event=GithubEvents.PULL_REQUEST,
+ default_branch="main",
+ )
+ result = selective_checks.provider_dependency_bump
+ assert result is True
+
+
+@patch("airflow_breeze.utils.selective_checks.run_command")
+def test_provider_dependency_bump_check_passes_on_non_provider_dependency_changes(mock_run_command):
+ """Test that provider dependency bump check passes when non-provider dependencies change."""
+ old_toml = """
+[project]
+dependencies = [
+ "apache-airflow>=2.10.0",
+ "boto3>=1.37.0",
+]
+"""
+ new_toml = """
+[project]
+dependencies = [
+ "apache-airflow>=2.10.0",
+ "boto3>=1.38.0",
+]
+"""
+
+ def side_effect(*args, **kwargs):
+ result = Mock()
+ result.returncode = 0
+ if "^:" in args[0][2]:
+ result.stdout = old_toml
+ else:
+ result.stdout = new_toml
+ return result
+
+ mock_run_command.side_effect = side_effect
+
+ selective_checks = SelectiveChecks(
+ files=("providers/amazon/pyproject.toml",),
+ commit_ref=NEUTRAL_COMMIT,
+ pr_labels=(),
+ github_event=GithubEvents.PULL_REQUEST,
+ default_branch="main",
+ )
+ result = selective_checks.provider_dependency_bump
+ assert result is False
+
+
+@patch("airflow_breeze.utils.selective_checks.run_command")
+def test_provider_dependency_bump_check_in_optional_dependencies(mock_run_command):
Review Comment:
Nope, I am. I clearly can't read.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Manages cross-distribution dependencies better [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #58430:
URL: https://github.com/apache/airflow/pull/58430#discussion_r2539210138
##########
contributing-docs/13_airflow_dependencies_and_extras.rst:
##########
@@ -18,6 +18,226 @@
Airflow dependencies
====================
+This document describes how we manage Apache Airflow dependencies, as well as how we make sure
+that users can use Airflow both as an application and as a library when they are deploying
+their own Airflow - using our constraints mechanism.
+
+Airflow ``pyproject.toml`` files and ``uv`` workspace
+.....................................................
+
+Managing dependencies is an important part of maintaining Apache Airflow, we have more than 700
+dependencies, and often when you add a new feature that requires a new dependency, you should
+also update the dependency list. This also happens when you want to add a new tool that is used
+for development, testing, or building the documentation and this document describes how to do it.
+
+When it comes to defining dependencies for any of the Apache Airflow distributions, we are following the
+standard ``pyproject.toml`` format that has been defined in `PEP 518 <https://peps.python.org/pep-0518/>`_
+and `PEP 621 <https://peps.python.org/pep-0621/>`_, we are also using dependency groups defined in
+`PEP 735 <https://peps.python.org/pep-0735/>`_ - particularly ``dev`` dependency group which we use in
+all our ``pyproject.toml`` files to define development dependencies.
+
+We have more than 100 python distributions in Apache Airflow repository - including the main
+``apache-airflow`` package, ``apache-airflow-providers-*`` packages, ``apache-airflow-core`` package,
+``apache-airflow-task-sdk`` package, ``apache-airflow-ctl`` package, and several other packages.
+
+They are all connected together via ``uv`` workspace feature (workspace is defined in the root ``pyproject.toml``
+file of the repository in the ``apache-airflow`` distribution definition. The workspace feature allows us to
+run ``uv sync`` at the top of the repository to install all packages in editable mode in the development
+environment from all the 100+ distributions (!) and resolve the dependencies together, so that we know that
+the dependencies have no conflicting requirements. Also the distributions are referring to each other via
+name - which means that when you run locally ``uv sync``, the local version of the packages are used, not the
+ones released on PyPI, which means that you can develop and test changes that span multiple packages at the
+same time. This is a very powerful feature that allows us to maintain the complex ecosystem of Apache Airflow
+distributions in a single monorepo, and allows us - for example to add new feature to common distributions used
+by multiple providers and test them all together before releasing new versions of either of those pacckages
+
+Managing dependencies in ``pyproject.toml`` files
+.................................................
+
+Each of the ``pyproject.toml`` files in Apache Airflow repository defines dependencies in one of the
+following sections:
+
+* ``[project.dependencies]`` - this section defines the required dependencies of the package. These
+ dependencies are installed when you install the package without any extras.
+* ``[project.optional-dependencies]`` - this section defines optional dependencies (extras) of the package.
+ These dependencies are installed when you install the package with extras - for example
+ ``pip install apache-airflow[ssh]`` will install the ``ssh`` extra dependencies defined in this section.
+* ``[dependency-group.dev]`` - this section defines development dependencies of the package.
+ These dependencies are installed when you run ``uv sync`` by default. when ``uv`` syncs sources with
+ local pyproject.toml it adds ``dev`` dependency group and package is installed in editable mode with
+ development dependencies.
+
+
+Adding and modifying dependencies
+.................................
+
+Adding and modifying dependencies in Apache Airflow is done by modifying the appropriate
+``pyproject.toml`` file in the appropriate distribution.
+
+When you add a new dependency, you should make sure that:
+
+* The dependency is added to the right section (main dependencies, optional dependencies, or
+ development dependencies)
+
+* Some parts of those dependencies might be automatically generated (and overwritten) by our ``prek``
+ hooks. Those are the necessary dependencies that ``prek`` hoks can figure out automatically by
+ analyzing the imports in the sources and structure of the project. We also have special case of
+ shared dependencies (described in `shared dependencies document <../shared/README.md>`__) where we
+ effectively "static-link" some libraries into multiple distributions, to avoid unnecessary coupling and
+ circular dependencies, and those dependencies contribute some dependencies automatically as well.
+ Pay attention to comments such us example start and end of such generated block below.
+ All the dependencies between those comments are automatically generated and you should not modify
+ them manually. Tou might instead modify the root source of those dependencies - depending on the automation,
+ you can usually also add extra dependencies manually outside of those comment blocks
+
+ .. code-block:: python
+
+ # Automatically generated airflow optional dependencies (update_airflow_pyproject_toml.py)
+ # ....
+ # LOTS OF GENERATED DEPENDENCIES HERE
+ # ....
+ # End of automatically generated airflow optional dependencies
+
+* The version specifier is as open as possible (upper) while still allowing the package to install
+ and pass all tests. We very rarely upper-bind dependencies - only when there is a known
+ conflict with a new or upcoming version of a dependency that breaks the installation or tests
+ (and we always make a comment why we are upper-bounding a dependency).
+
+* Make sure to lower-bind any dependency you add. Usually we lower-bind dependencies to the
+ minimum version that is required for the package to work but in order to simplify the work of
+ resolvers such as ``pip``, we often lower-bind to higher (and newer) version than the absolute minimum
+ especially when the minimum version is very old. This is for example good practice in ``boto`` and related
+ packages, where new version of those packages are released frequently (almost daily) and there are many
+ versions that need to be considered by the resolver if the version is not new enough.
+
+* Make sure to run ``uv sync`` after modifying dependencies to make sure that there are no
+ conflicts between dependencies of different packages in the workspace. You can run it in multiple
+ ways - either from the root of the repository (which will sync all packages) or from the package
+ you modified (which will sync only that package and its dependencies). Also good idea might be to
+ run ``uv sync --all-packages --all-extras`` at the root of the repository to make sure that
+ all packages with all extras can be installed together without conflicts, but this might be sometimes
+ difficult and slow as some of the extras require some additional system level dependencies to be installed
+ (for example ``mysql`` or ``postgres`` extras require client libraries to be installed on the system).
+
+* Make sure to run all tests after modifying dependencies to make sure that nothing is broken
+
+
+Referring to other Apache Airflow distributions in dependencies
+...............................................................
+
+With having more than 100 distributions in the repository, it is often necessary to refer to
+other distributions in the dependencies in order to use some common features or simply to use the
+features that the other distribution provides. There are two ways of doing it:
+
+* Regular package linking with ``apache-airflow-*`` dependency
+* Airflow "shared dependencies" mechanism - which is a bit of a custom hack for Airflow monorepo
+ that allows us to "static-link" some common dependencies into multiple distributions without
+ creating circular dependencies.
+
+We are not going to describe the shared dependencies mechanism here, please refer to the
+`shared dependencies document <../shared/README.md>`__ for details, but there are certain rules
+when it comes to referring to other Airflow distributions in dependencies - here are the important
+rules to remember:
+
+* You can refer to other distributions in your dependencies - as usual using distribution name. For example,
+ if you are adding a dependency to ``apache-airflow-providers-common-compat`` package from
+ ``apache-airflow-providers-google``, you can just add ``apache-airflow-providers-common>=x.y.z`` to the
+ dependencies and when you run ``uv sync``, the local version of the package will be used automatically
+ (this is thanks to the workspace feature of ``uv`` that does great job of binding our monorepo together).
+ This is very useful when you are developing changes to multiple packages at the same time. Some of those
+ are added automatically by prek hooks - when it can detect such dependencies by analyzing imports in the
+ sources - then they are added automatically between the special comments mentioned above, but sometimes
+ (especially when such dependencies are not at the top-level imports) you might need to add them manually.
+
+* Never update distribution versions on your own. It is **entirely** up to the Release Manager to bump the
+ version of distributions that are defined as ``project.version``. This goes almost without an exception
+ and any diversions from this rule should be discussed at ``#release-management`` channel in Airflow Slack
+ beforehand. The only exception to this rule is when you are adding a new distribution to the repository -
+ in that case you should set the initial version of the distribution - usually ``0.0.1``, ``0.1.0``
+ or ``1.0.0`` depending on the maturity of the package. But still you should discuss it in the channel.
+
+* Sometimes, when you add a new feature to a common distribution, you might add a feature to it or
+ change the API in the way that other packages can use it. This is especially true for common packages such as
+ ``apache-airflow-providers-common-compat``, but can happen for other packages (for example
+ ``apache-airflow-providers-apache-beam`` is used by ``apache-airflow-providers-google`` to use ``Apache Beam``
+ hooks to communicate with Google Dataflow). In such case, when you are adding a feature to a common package
+ remember that the feature you just add. will only be released in the **FUTURE** release of such common
+ package and you cannot add ``>==x.y.z`` dependency to it where ``x.y.z`` is the version you are
+ going to release in the future. Ultimately, this should happen (and happens) when the Release Manager prepares
+ both packages together. Let us repeat - such changes in versions between different airflow package should
+ **NOT** be added to the dependencies manually by the contributor. They should **exclusively** be added by
+ the Release Manager. when preparing the release of **both** packages together.
+ We have a custom mechanism to support such additions, where it is contributor's responsibility to mark
+ dependency with a special comment - simply communicating with the Release Manager that such dependency
+ should be updated to the next version when the release is prepared. If you see such a need to use newly
+ added feature and using it at the same time in a different distribution - make sure to add this comment
+ in the line where dependency you want to use the new feature from is defined:
+
+ .. code-block:: python
+
+ # some deps
+ "apache-airflow-SOMETHING>1.0.0", # use next version
Review Comment:
Good point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Manages cross-distribution dependencies better [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #58430:
URL: https://github.com/apache/airflow/pull/58430#discussion_r2539205079
##########
contributing-docs/13_airflow_dependencies_and_extras.rst:
##########
@@ -18,6 +18,226 @@
Airflow dependencies
====================
+This document describes how we manage Apache Airflow dependencies, as well as how we make sure
+that users can use Airflow both as an application and as a library when they are deploying
+their own Airflow - using our constraints mechanism.
+
+Airflow ``pyproject.toml`` files and ``uv`` workspace
+.....................................................
+
+Managing dependencies is an important part of maintaining Apache Airflow, we have more than 700
+dependencies, and often when you add a new feature that requires a new dependency, you should
+also update the dependency list. This also happens when you want to add a new tool that is used
+for development, testing, or building the documentation and this document describes how to do it.
+
+When it comes to defining dependencies for any of the Apache Airflow distributions, we are following the
+standard ``pyproject.toml`` format that has been defined in `PEP 518 <https://peps.python.org/pep-0518/>`_
+and `PEP 621 <https://peps.python.org/pep-0621/>`_, we are also using dependency groups defined in
+`PEP 735 <https://peps.python.org/pep-0735/>`_ - particularly ``dev`` dependency group which we use in
+all our ``pyproject.toml`` files to define development dependencies.
+
+We have more than 100 python distributions in Apache Airflow repository - including the main
+``apache-airflow`` package, ``apache-airflow-providers-*`` packages, ``apache-airflow-core`` package,
+``apache-airflow-task-sdk`` package, ``apache-airflow-ctl`` package, and several other packages.
+
+They are all connected together via ``uv`` workspace feature (workspace is defined in the root ``pyproject.toml``
+file of the repository in the ``apache-airflow`` distribution definition. The workspace feature allows us to
+run ``uv sync`` at the top of the repository to install all packages in editable mode in the development
+environment from all the 100+ distributions (!) and resolve the dependencies together, so that we know that
+the dependencies have no conflicting requirements. Also the distributions are referring to each other via
+name - which means that when you run locally ``uv sync``, the local version of the packages are used, not the
+ones released on PyPI, which means that you can develop and test changes that span multiple packages at the
+same time. This is a very powerful feature that allows us to maintain the complex ecosystem of Apache Airflow
+distributions in a single monorepo, and allows us - for example to add new feature to common distributions used
+by multiple providers and test them all together before releasing new versions of either of those pacckages
+
+Managing dependencies in ``pyproject.toml`` files
+.................................................
+
+Each of the ``pyproject.toml`` files in Apache Airflow repository defines dependencies in one of the
+following sections:
+
+* ``[project.dependencies]`` - this section defines the required dependencies of the package. These
+ dependencies are installed when you install the package without any extras.
+* ``[project.optional-dependencies]`` - this section defines optional dependencies (extras) of the package.
+ These dependencies are installed when you install the package with extras - for example
+ ``pip install apache-airflow[ssh]`` will install the ``ssh`` extra dependencies defined in this section.
+* ``[dependency-group.dev]`` - this section defines development dependencies of the package.
+ These dependencies are installed when you run ``uv sync`` by default. when ``uv`` syncs sources with
+ local pyproject.toml it adds ``dev`` dependency group and package is installed in editable mode with
+ development dependencies.
+
+
+Adding and modifying dependencies
+.................................
+
+Adding and modifying dependencies in Apache Airflow is done by modifying the appropriate
+``pyproject.toml`` file in the appropriate distribution.
+
+When you add a new dependency, you should make sure that:
+
+* The dependency is added to the right section (main dependencies, optional dependencies, or
+ development dependencies)
+
+* Some parts of those dependencies might be automatically generated (and overwritten) by our ``prek``
+ hooks. Those are the necessary dependencies that ``prek`` hoks can figure out automatically by
+ analyzing the imports in the sources and structure of the project. We also have special case of
+ shared dependencies (described in `shared dependencies document <../shared/README.md>`__) where we
+ effectively "static-link" some libraries into multiple distributions, to avoid unnecessary coupling and
+ circular dependencies, and those dependencies contribute some dependencies automatically as well.
+ Pay attention to comments such us example start and end of such generated block below.
+ All the dependencies between those comments are automatically generated and you should not modify
+ them manually. Tou might instead modify the root source of those dependencies - depending on the automation,
+ you can usually also add extra dependencies manually outside of those comment blocks
+
+ .. code-block:: python
+
+ # Automatically generated airflow optional dependencies (update_airflow_pyproject_toml.py)
+ # ....
+ # LOTS OF GENERATED DEPENDENCIES HERE
+ # ....
+ # End of automatically generated airflow optional dependencies
+
+* The version specifier is as open as possible (upper) while still allowing the package to install
+ and pass all tests. We very rarely upper-bind dependencies - only when there is a known
+ conflict with a new or upcoming version of a dependency that breaks the installation or tests
+ (and we always make a comment why we are upper-bounding a dependency).
+
+* Make sure to lower-bind any dependency you add. Usually we lower-bind dependencies to the
+ minimum version that is required for the package to work but in order to simplify the work of
+ resolvers such as ``pip``, we often lower-bind to higher (and newer) version than the absolute minimum
+ especially when the minimum version is very old. This is for example good practice in ``boto`` and related
+ packages, where new version of those packages are released frequently (almost daily) and there are many
+ versions that need to be considered by the resolver if the version is not new enough.
+
+* Make sure to run ``uv sync`` after modifying dependencies to make sure that there are no
+ conflicts between dependencies of different packages in the workspace. You can run it in multiple
+ ways - either from the root of the repository (which will sync all packages) or from the package
+ you modified (which will sync only that package and its dependencies). Also good idea might be to
+ run ``uv sync --all-packages --all-extras`` at the root of the repository to make sure that
+ all packages with all extras can be installed together without conflicts, but this might be sometimes
+ difficult and slow as some of the extras require some additional system level dependencies to be installed
+ (for example ``mysql`` or ``postgres`` extras require client libraries to be installed on the system).
+
+* Make sure to run all tests after modifying dependencies to make sure that nothing is broken
+
+
+Referring to other Apache Airflow distributions in dependencies
+...............................................................
+
+With having more than 100 distributions in the repository, it is often necessary to refer to
+other distributions in the dependencies in order to use some common features or simply to use the
+features that the other distribution provides. There are two ways of doing it:
+
+* Regular package linking with ``apache-airflow-*`` dependency
+* Airflow "shared dependencies" mechanism - which is a bit of a custom hack for Airflow monorepo
+ that allows us to "static-link" some common dependencies into multiple distributions without
+ creating circular dependencies.
+
+We are not going to describe the shared dependencies mechanism here, please refer to the
+`shared dependencies document <../shared/README.md>`__ for details, but there are certain rules
+when it comes to referring to other Airflow distributions in dependencies - here are the important
+rules to remember:
+
+* You can refer to other distributions in your dependencies - as usual using distribution name. For example,
+ if you are adding a dependency to ``apache-airflow-providers-common-compat`` package from
+ ``apache-airflow-providers-google``, you can just add ``apache-airflow-providers-common>=x.y.z`` to the
+ dependencies and when you run ``uv sync``, the local version of the package will be used automatically
+ (this is thanks to the workspace feature of ``uv`` that does great job of binding our monorepo together).
+ This is very useful when you are developing changes to multiple packages at the same time. Some of those
+ are added automatically by prek hooks - when it can detect such dependencies by analyzing imports in the
+ sources - then they are added automatically between the special comments mentioned above, but sometimes
+ (especially when such dependencies are not at the top-level imports) you might need to add them manually.
+
+* Never update distribution versions on your own. It is **entirely** up to the Release Manager to bump the
+ version of distributions that are defined as ``project.version``. This goes almost without an exception
+ and any diversions from this rule should be discussed at ``#release-management`` channel in Airflow Slack
+ beforehand. The only exception to this rule is when you are adding a new distribution to the repository -
+ in that case you should set the initial version of the distribution - usually ``0.0.1``, ``0.1.0``
+ or ``1.0.0`` depending on the maturity of the package. But still you should discuss it in the channel.
+
Review Comment:
This is actually part of it and it makes sense in terms of internal cross-dependencies - and came up in maaaany conversations:
"I added this feature, should I update the version of this distribution or will RM do it for me". People were absolutely not clear about it.
But I added this sentence before to make it clearer:
> In case you make a feature change in a distribution and would like to update its version, you should never update the distribution version on your own
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Manages cross-distribution dependencies better [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #58430:
URL: https://github.com/apache/airflow/pull/58430#issuecomment-3548882330
> Docs are longer than I'd like, but make sense.
The thing is that AI actually likes detailed context. This documentation is also for AI agents to understand what they should do with documentation. Again - i do not expect people to read them like a book, more like trying to find what they are looking for - but I do 100% expect it will be read by Gen-AI tools in it's entiety to build the right context for whatever they will be doing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Manages cross-distribution dependencies better [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #58430:
URL: https://github.com/apache/airflow/pull/58430#issuecomment-3549005651
cc: @kacpermuda -> I have not tagged you on this one -> but it basically implements 150% of what we discussed. in https://github.com/apache/airflow/pull/58393#issuecomment-3542040747
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Manages cross-distribution dependencies better [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #58430:
URL: https://github.com/apache/airflow/pull/58430#issuecomment-3549674414
> Look good to me! I totally agree that not many contributors will read this. Since the documents are public, someone can summarise them easily. Selective checks for the providers are definitely a better place for this check rather than `prek`. We don't need selective checks for other distributions since they are easy to follow as long as RMs are CodeOwners on those repositories. It would be beneficial to have a list of who can be tagged for which distribution as an RM list, perhaps. It would be easier to check and ping the correct people for Commiters, PMCs, but mostly contributors. It would also be informative in general and serve as an Onboarding piece for new members
I think just #releae-management is good enough (I mentioned it in the description) - I guess keeping list of RMs separately is less-than-needed if everyone is in #release-management channel. Those are rather "rare" events to handle and I guess there is little issue with whole #release-management being notified (especially also for redundancy reasons, vacations and so on).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Manages cross-distribution dependencies better [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #58430:
URL: https://github.com/apache/airflow/pull/58430#discussion_r2539218451
##########
dev/breeze/tests/test_selective_checks.py:
##########
@@ -2779,3 +2779,169 @@ def test_testable_providers_integrations_excludes_arm_disabled_on_arm():
assert "postgres" in result
assert "trino" not in result
assert "ydb" not in result
+
+
+@patch("airflow_breeze.utils.selective_checks.run_command")
+def test_provider_dependency_bump_check_no_changes(mock_run_command):
+ """Test that provider dependency bump check passes when no pyproject.toml files are changed."""
+ selective_checks = SelectiveChecks(
+ files=("some_other_file.py",),
+ commit_ref=NEUTRAL_COMMIT,
+ pr_labels=(),
+ github_event=GithubEvents.PULL_REQUEST,
+ default_branch="main",
+ )
+ result = selective_checks.provider_dependency_bump
+ assert result is False
+
+
+@patch("airflow_breeze.utils.selective_checks.run_command")
+def test_provider_dependency_bump_check_fails_on_provider_version_bump(mock_run_command):
+ """Test that provider dependency bump check fails when provider version is bumped without label."""
+ old_toml = """
+[project]
+dependencies = [
+ "apache-airflow-providers-common-sql>=1.0.0",
+]
+"""
+ new_toml = """
+[project]
+dependencies = [
+ "apache-airflow-providers-common-sql>=1.1.0",
+]
+"""
+
+ def side_effect(*args, **kwargs):
+ result = Mock()
+ result.returncode = 0
+ if "^:" in args[0][2]:
+ result.stdout = old_toml
+ else:
+ result.stdout = new_toml
+ return result
+
+ mock_run_command.side_effect = side_effect
+
+ with pytest.raises(SystemExit):
+ SelectiveChecks(
+ files=("providers/amazon/pyproject.toml",),
+ commit_ref=NEUTRAL_COMMIT,
+ pr_labels=(),
+ github_event=GithubEvents.PULL_REQUEST,
+ default_branch="main",
+ ).provider_dependency_bump
+
+
+@patch("airflow_breeze.utils.selective_checks.run_command")
+def test_provider_dependency_bump_check_passes_with_label(mock_run_command):
+ """Test that provider dependency bump check passes when label is set."""
+ old_toml = """
+[project]
+dependencies = [
+ "apache-airflow-providers-common-sql>=1.0.0",
+]
+"""
+ new_toml = """
+[project]
+dependencies = [
+ "apache-airflow-providers-common-sql>=1.1.0",
+]
+"""
+
+ def side_effect(*args, **kwargs):
+ result = Mock()
+ result.returncode = 0
+ if "^:" in args[0][2]:
+ result.stdout = old_toml
+ else:
+ result.stdout = new_toml
+ return result
+
+ mock_run_command.side_effect = side_effect
+
+ selective_checks = SelectiveChecks(
+ files=("providers/amazon/pyproject.toml",),
+ commit_ref=NEUTRAL_COMMIT,
+ pr_labels=("allow provider dependency bump",),
+ github_event=GithubEvents.PULL_REQUEST,
+ default_branch="main",
+ )
+ result = selective_checks.provider_dependency_bump
+ assert result is True
+
+
+@patch("airflow_breeze.utils.selective_checks.run_command")
+def test_provider_dependency_bump_check_passes_on_non_provider_dependency_changes(mock_run_command):
+ """Test that provider dependency bump check passes when non-provider dependencies change."""
+ old_toml = """
+[project]
+dependencies = [
+ "apache-airflow>=2.10.0",
+ "boto3>=1.37.0",
+]
+"""
+ new_toml = """
+[project]
+dependencies = [
+ "apache-airflow>=2.10.0",
+ "boto3>=1.38.0",
+]
+"""
+
+ def side_effect(*args, **kwargs):
+ result = Mock()
+ result.returncode = 0
+ if "^:" in args[0][2]:
+ result.stdout = old_toml
+ else:
+ result.stdout = new_toml
+ return result
+
+ mock_run_command.side_effect = side_effect
+
+ selective_checks = SelectiveChecks(
+ files=("providers/amazon/pyproject.toml",),
+ commit_ref=NEUTRAL_COMMIT,
+ pr_labels=(),
+ github_event=GithubEvents.PULL_REQUEST,
+ default_branch="main",
+ )
+ result = selective_checks.provider_dependency_bump
+ assert result is False
+
+
+@patch("airflow_breeze.utils.selective_checks.run_command")
+def test_provider_dependency_bump_check_in_optional_dependencies(mock_run_command):
Review Comment:
Nope. They are not exempt. The test actually works as expected here - selective check will raise SystemExist when you modify also optional dependencies.
Or am I missing something?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Manages cross-distribution dependencies better [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #58430:
URL: https://github.com/apache/airflow/pull/58430#discussion_r2539241139
##########
dev/breeze/tests/test_selective_checks.py:
##########
@@ -2779,3 +2779,169 @@ def test_testable_providers_integrations_excludes_arm_disabled_on_arm():
assert "postgres" in result
assert "trino" not in result
assert "ydb" not in result
+
+
+@patch("airflow_breeze.utils.selective_checks.run_command")
+def test_provider_dependency_bump_check_no_changes(mock_run_command):
+ """Test that provider dependency bump check passes when no pyproject.toml files are changed."""
+ selective_checks = SelectiveChecks(
+ files=("some_other_file.py",),
+ commit_ref=NEUTRAL_COMMIT,
+ pr_labels=(),
+ github_event=GithubEvents.PULL_REQUEST,
+ default_branch="main",
+ )
+ result = selective_checks.provider_dependency_bump
+ assert result is False
+
+
+@patch("airflow_breeze.utils.selective_checks.run_command")
+def test_provider_dependency_bump_check_fails_on_provider_version_bump(mock_run_command):
Review Comment:
Yes. We do it very often - for example when Amazon provider expects a new feature from google provider in "transfer" operator (happened a few times) or when google provider expected new features from beam provider (happened more than once) - or when about 20 providers expected new feature from openlineage provider or when we had new feature in common.compat we wanted to add.
Last time it happend was few days ago when new compat feature was added and apprise and slack provider used new feature from common.compat -resulting in removing both of them from rc1 and creating rc2.
In this case we **really** want people to try to bump the version of providers (if they have never read the documentation and learn from the failure what they should do.
So **yeah** absolutely we want to test it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Manages cross-distribution dependencies better [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #58430:
URL: https://github.com/apache/airflow/pull/58430#issuecomment-3550110564
### Backport successfully created: v3-1-test
<table>
<tr>
<th>Status</th>
<th>Branch</th>
<th>Result</th>
</tr>
<tr>
<td>✅</td>
<td>v3-1-test</td>
<td><a href="https://github.com/apache/airflow/pull/58472"><img src="https://img.shields.io/badge/PR-58472-blue" alt="PR Link"></a></td>
</tr>
</table>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Manages cross-distribution dependencies better [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #58430:
URL: https://github.com/apache/airflow/pull/58430#discussion_r2539164498
##########
contributing-docs/13_airflow_dependencies_and_extras.rst:
##########
@@ -18,6 +18,226 @@
Airflow dependencies
====================
+This document describes how we manage Apache Airflow dependencies, as well as how we make sure
+that users can use Airflow both as an application and as a library when they are deploying
+their own Airflow - using our constraints mechanism.
+
+Airflow ``pyproject.toml`` files and ``uv`` workspace
+.....................................................
+
+Managing dependencies is an important part of maintaining Apache Airflow, we have more than 700
Review Comment:
Fine for me :). Indeed maintainers might do a lot more with dependencies than developers and this guideline is mostly for developers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Manages cross-distribution dependencies better [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #58430:
URL: https://github.com/apache/airflow/pull/58430
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Manages cross-distribution dependencies better [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #58430:
URL: https://github.com/apache/airflow/pull/58430#discussion_r2539208076
##########
contributing-docs/13_airflow_dependencies_and_extras.rst:
##########
@@ -18,6 +18,226 @@
Airflow dependencies
====================
+This document describes how we manage Apache Airflow dependencies, as well as how we make sure
+that users can use Airflow both as an application and as a library when they are deploying
+their own Airflow - using our constraints mechanism.
+
+Airflow ``pyproject.toml`` files and ``uv`` workspace
+.....................................................
+
+Managing dependencies is an important part of maintaining Apache Airflow, we have more than 700
+dependencies, and often when you add a new feature that requires a new dependency, you should
+also update the dependency list. This also happens when you want to add a new tool that is used
+for development, testing, or building the documentation and this document describes how to do it.
+
+When it comes to defining dependencies for any of the Apache Airflow distributions, we are following the
+standard ``pyproject.toml`` format that has been defined in `PEP 518 <https://peps.python.org/pep-0518/>`_
+and `PEP 621 <https://peps.python.org/pep-0621/>`_, we are also using dependency groups defined in
+`PEP 735 <https://peps.python.org/pep-0735/>`_ - particularly ``dev`` dependency group which we use in
+all our ``pyproject.toml`` files to define development dependencies.
+
+We have more than 100 python distributions in Apache Airflow repository - including the main
+``apache-airflow`` package, ``apache-airflow-providers-*`` packages, ``apache-airflow-core`` package,
+``apache-airflow-task-sdk`` package, ``apache-airflow-ctl`` package, and several other packages.
+
+They are all connected together via ``uv`` workspace feature (workspace is defined in the root ``pyproject.toml``
+file of the repository in the ``apache-airflow`` distribution definition. The workspace feature allows us to
+run ``uv sync`` at the top of the repository to install all packages in editable mode in the development
+environment from all the 100+ distributions (!) and resolve the dependencies together, so that we know that
+the dependencies have no conflicting requirements. Also the distributions are referring to each other via
+name - which means that when you run locally ``uv sync``, the local version of the packages are used, not the
+ones released on PyPI, which means that you can develop and test changes that span multiple packages at the
+same time. This is a very powerful feature that allows us to maintain the complex ecosystem of Apache Airflow
+distributions in a single monorepo, and allows us - for example to add new feature to common distributions used
+by multiple providers and test them all together before releasing new versions of either of those pacckages
+
+Managing dependencies in ``pyproject.toml`` files
+.................................................
+
+Each of the ``pyproject.toml`` files in Apache Airflow repository defines dependencies in one of the
+following sections:
+
+* ``[project.dependencies]`` - this section defines the required dependencies of the package. These
+ dependencies are installed when you install the package without any extras.
+* ``[project.optional-dependencies]`` - this section defines optional dependencies (extras) of the package.
+ These dependencies are installed when you install the package with extras - for example
+ ``pip install apache-airflow[ssh]`` will install the ``ssh`` extra dependencies defined in this section.
+* ``[dependency-group.dev]`` - this section defines development dependencies of the package.
+ These dependencies are installed when you run ``uv sync`` by default. when ``uv`` syncs sources with
+ local pyproject.toml it adds ``dev`` dependency group and package is installed in editable mode with
+ development dependencies.
+
+
+Adding and modifying dependencies
+.................................
+
+Adding and modifying dependencies in Apache Airflow is done by modifying the appropriate
+``pyproject.toml`` file in the appropriate distribution.
+
+When you add a new dependency, you should make sure that:
+
+* The dependency is added to the right section (main dependencies, optional dependencies, or
+ development dependencies)
+
+* Some parts of those dependencies might be automatically generated (and overwritten) by our ``prek``
+ hooks. Those are the necessary dependencies that ``prek`` hoks can figure out automatically by
+ analyzing the imports in the sources and structure of the project. We also have special case of
+ shared dependencies (described in `shared dependencies document <../shared/README.md>`__) where we
+ effectively "static-link" some libraries into multiple distributions, to avoid unnecessary coupling and
+ circular dependencies, and those dependencies contribute some dependencies automatically as well.
+ Pay attention to comments such us example start and end of such generated block below.
+ All the dependencies between those comments are automatically generated and you should not modify
+ them manually. Tou might instead modify the root source of those dependencies - depending on the automation,
+ you can usually also add extra dependencies manually outside of those comment blocks
+
+ .. code-block:: python
+
+ # Automatically generated airflow optional dependencies (update_airflow_pyproject_toml.py)
+ # ....
+ # LOTS OF GENERATED DEPENDENCIES HERE
+ # ....
+ # End of automatically generated airflow optional dependencies
+
+* The version specifier is as open as possible (upper) while still allowing the package to install
+ and pass all tests. We very rarely upper-bind dependencies - only when there is a known
+ conflict with a new or upcoming version of a dependency that breaks the installation or tests
+ (and we always make a comment why we are upper-bounding a dependency).
+
+* Make sure to lower-bind any dependency you add. Usually we lower-bind dependencies to the
+ minimum version that is required for the package to work but in order to simplify the work of
+ resolvers such as ``pip``, we often lower-bind to higher (and newer) version than the absolute minimum
+ especially when the minimum version is very old. This is for example good practice in ``boto`` and related
+ packages, where new version of those packages are released frequently (almost daily) and there are many
+ versions that need to be considered by the resolver if the version is not new enough.
+
+* Make sure to run ``uv sync`` after modifying dependencies to make sure that there are no
+ conflicts between dependencies of different packages in the workspace. You can run it in multiple
+ ways - either from the root of the repository (which will sync all packages) or from the package
+ you modified (which will sync only that package and its dependencies). Also good idea might be to
+ run ``uv sync --all-packages --all-extras`` at the root of the repository to make sure that
+ all packages with all extras can be installed together without conflicts, but this might be sometimes
+ difficult and slow as some of the extras require some additional system level dependencies to be installed
+ (for example ``mysql`` or ``postgres`` extras require client libraries to be installed on the system).
+
+* Make sure to run all tests after modifying dependencies to make sure that nothing is broken
+
+
+Referring to other Apache Airflow distributions in dependencies
+...............................................................
+
+With having more than 100 distributions in the repository, it is often necessary to refer to
+other distributions in the dependencies in order to use some common features or simply to use the
+features that the other distribution provides. There are two ways of doing it:
+
+* Regular package linking with ``apache-airflow-*`` dependency
+* Airflow "shared dependencies" mechanism - which is a bit of a custom hack for Airflow monorepo
+ that allows us to "static-link" some common dependencies into multiple distributions without
+ creating circular dependencies.
+
+We are not going to describe the shared dependencies mechanism here, please refer to the
+`shared dependencies document <../shared/README.md>`__ for details, but there are certain rules
+when it comes to referring to other Airflow distributions in dependencies - here are the important
+rules to remember:
+
+* You can refer to other distributions in your dependencies - as usual using distribution name. For example,
+ if you are adding a dependency to ``apache-airflow-providers-common-compat`` package from
+ ``apache-airflow-providers-google``, you can just add ``apache-airflow-providers-common>=x.y.z`` to the
+ dependencies and when you run ``uv sync``, the local version of the package will be used automatically
+ (this is thanks to the workspace feature of ``uv`` that does great job of binding our monorepo together).
+ This is very useful when you are developing changes to multiple packages at the same time. Some of those
+ are added automatically by prek hooks - when it can detect such dependencies by analyzing imports in the
+ sources - then they are added automatically between the special comments mentioned above, but sometimes
+ (especially when such dependencies are not at the top-level imports) you might need to add them manually.
+
+* Never update distribution versions on your own. It is **entirely** up to the Release Manager to bump the
+ version of distributions that are defined as ``project.version``. This goes almost without an exception
+ and any diversions from this rule should be discussed at ``#release-management`` channel in Airflow Slack
+ beforehand. The only exception to this rule is when you are adding a new distribution to the repository -
+ in that case you should set the initial version of the distribution - usually ``0.0.1``, ``0.1.0``
+ or ``1.0.0`` depending on the maturity of the package. But still you should discuss it in the channel.
+
+* Sometimes, when you add a new feature to a common distribution, you might add a feature to it or
+ change the API in the way that other packages can use it. This is especially true for common packages such as
+ ``apache-airflow-providers-common-compat``, but can happen for other packages (for example
+ ``apache-airflow-providers-apache-beam`` is used by ``apache-airflow-providers-google`` to use ``Apache Beam``
+ hooks to communicate with Google Dataflow). In such case, when you are adding a feature to a common package
+ remember that the feature you just add. will only be released in the **FUTURE** release of such common
+ package and you cannot add ``>==x.y.z`` dependency to it where ``x.y.z`` is the version you are
+ going to release in the future. Ultimately, this should happen (and happens) when the Release Manager prepares
+ both packages together. Let us repeat - such changes in versions between different airflow package should
+ **NOT** be added to the dependencies manually by the contributor. They should **exclusively** be added by
+ the Release Manager. when preparing the release of **both** packages together.
+ We have a custom mechanism to support such additions, where it is contributor's responsibility to mark
+ dependency with a special comment - simply communicating with the Release Manager that such dependency
+ should be updated to the next version when the release is prepared. If you see such a need to use newly
+ added feature and using it at the same time in a different distribution - make sure to add this comment
+ in the line where dependency you want to use the new feature from is defined:
+
+ .. code-block:: python
+
+ # some deps
+ "apache-airflow-SOMETHING>1.0.0", # use next version
Review Comment:
It is precise string. I will make it clearer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Manages cross-distribution dependencies better [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #58430:
URL: https://github.com/apache/airflow/pull/58430#discussion_r2539206762
##########
contributing-docs/13_airflow_dependencies_and_extras.rst:
##########
@@ -18,6 +18,226 @@
Airflow dependencies
====================
+This document describes how we manage Apache Airflow dependencies, as well as how we make sure
+that users can use Airflow both as an application and as a library when they are deploying
+their own Airflow - using our constraints mechanism.
+
+Airflow ``pyproject.toml`` files and ``uv`` workspace
+.....................................................
+
+Managing dependencies is an important part of maintaining Apache Airflow, we have more than 700
+dependencies, and often when you add a new feature that requires a new dependency, you should
+also update the dependency list. This also happens when you want to add a new tool that is used
+for development, testing, or building the documentation and this document describes how to do it.
+
+When it comes to defining dependencies for any of the Apache Airflow distributions, we are following the
+standard ``pyproject.toml`` format that has been defined in `PEP 518 <https://peps.python.org/pep-0518/>`_
+and `PEP 621 <https://peps.python.org/pep-0621/>`_, we are also using dependency groups defined in
+`PEP 735 <https://peps.python.org/pep-0735/>`_ - particularly ``dev`` dependency group which we use in
+all our ``pyproject.toml`` files to define development dependencies.
+
+We have more than 100 python distributions in Apache Airflow repository - including the main
+``apache-airflow`` package, ``apache-airflow-providers-*`` packages, ``apache-airflow-core`` package,
+``apache-airflow-task-sdk`` package, ``apache-airflow-ctl`` package, and several other packages.
+
+They are all connected together via ``uv`` workspace feature (workspace is defined in the root ``pyproject.toml``
+file of the repository in the ``apache-airflow`` distribution definition. The workspace feature allows us to
+run ``uv sync`` at the top of the repository to install all packages in editable mode in the development
+environment from all the 100+ distributions (!) and resolve the dependencies together, so that we know that
+the dependencies have no conflicting requirements. Also the distributions are referring to each other via
+name - which means that when you run locally ``uv sync``, the local version of the packages are used, not the
+ones released on PyPI, which means that you can develop and test changes that span multiple packages at the
+same time. This is a very powerful feature that allows us to maintain the complex ecosystem of Apache Airflow
+distributions in a single monorepo, and allows us - for example to add new feature to common distributions used
+by multiple providers and test them all together before releasing new versions of either of those pacckages
+
+Managing dependencies in ``pyproject.toml`` files
+.................................................
+
+Each of the ``pyproject.toml`` files in Apache Airflow repository defines dependencies in one of the
+following sections:
+
+* ``[project.dependencies]`` - this section defines the required dependencies of the package. These
+ dependencies are installed when you install the package without any extras.
+* ``[project.optional-dependencies]`` - this section defines optional dependencies (extras) of the package.
+ These dependencies are installed when you install the package with extras - for example
+ ``pip install apache-airflow[ssh]`` will install the ``ssh`` extra dependencies defined in this section.
+* ``[dependency-group.dev]`` - this section defines development dependencies of the package.
+ These dependencies are installed when you run ``uv sync`` by default. when ``uv`` syncs sources with
+ local pyproject.toml it adds ``dev`` dependency group and package is installed in editable mode with
+ development dependencies.
+
+
+Adding and modifying dependencies
+.................................
+
+Adding and modifying dependencies in Apache Airflow is done by modifying the appropriate
+``pyproject.toml`` file in the appropriate distribution.
+
+When you add a new dependency, you should make sure that:
+
+* The dependency is added to the right section (main dependencies, optional dependencies, or
+ development dependencies)
+
+* Some parts of those dependencies might be automatically generated (and overwritten) by our ``prek``
+ hooks. Those are the necessary dependencies that ``prek`` hoks can figure out automatically by
+ analyzing the imports in the sources and structure of the project. We also have special case of
+ shared dependencies (described in `shared dependencies document <../shared/README.md>`__) where we
+ effectively "static-link" some libraries into multiple distributions, to avoid unnecessary coupling and
+ circular dependencies, and those dependencies contribute some dependencies automatically as well.
+ Pay attention to comments such us example start and end of such generated block below.
+ All the dependencies between those comments are automatically generated and you should not modify
+ them manually. Tou might instead modify the root source of those dependencies - depending on the automation,
+ you can usually also add extra dependencies manually outside of those comment blocks
+
+ .. code-block:: python
+
+ # Automatically generated airflow optional dependencies (update_airflow_pyproject_toml.py)
+ # ....
+ # LOTS OF GENERATED DEPENDENCIES HERE
+ # ....
+ # End of automatically generated airflow optional dependencies
+
+* The version specifier is as open as possible (upper) while still allowing the package to install
+ and pass all tests. We very rarely upper-bind dependencies - only when there is a known
+ conflict with a new or upcoming version of a dependency that breaks the installation or tests
+ (and we always make a comment why we are upper-bounding a dependency).
+
+* Make sure to lower-bind any dependency you add. Usually we lower-bind dependencies to the
+ minimum version that is required for the package to work but in order to simplify the work of
+ resolvers such as ``pip``, we often lower-bind to higher (and newer) version than the absolute minimum
+ especially when the minimum version is very old. This is for example good practice in ``boto`` and related
+ packages, where new version of those packages are released frequently (almost daily) and there are many
+ versions that need to be considered by the resolver if the version is not new enough.
+
+* Make sure to run ``uv sync`` after modifying dependencies to make sure that there are no
+ conflicts between dependencies of different packages in the workspace. You can run it in multiple
+ ways - either from the root of the repository (which will sync all packages) or from the package
+ you modified (which will sync only that package and its dependencies). Also good idea might be to
+ run ``uv sync --all-packages --all-extras`` at the root of the repository to make sure that
+ all packages with all extras can be installed together without conflicts, but this might be sometimes
+ difficult and slow as some of the extras require some additional system level dependencies to be installed
+ (for example ``mysql`` or ``postgres`` extras require client libraries to be installed on the system).
+
+* Make sure to run all tests after modifying dependencies to make sure that nothing is broken
+
+
+Referring to other Apache Airflow distributions in dependencies
+...............................................................
+
+With having more than 100 distributions in the repository, it is often necessary to refer to
+other distributions in the dependencies in order to use some common features or simply to use the
+features that the other distribution provides. There are two ways of doing it:
+
+* Regular package linking with ``apache-airflow-*`` dependency
+* Airflow "shared dependencies" mechanism - which is a bit of a custom hack for Airflow monorepo
+ that allows us to "static-link" some common dependencies into multiple distributions without
+ creating circular dependencies.
+
+We are not going to describe the shared dependencies mechanism here, please refer to the
+`shared dependencies document <../shared/README.md>`__ for details, but there are certain rules
+when it comes to referring to other Airflow distributions in dependencies - here are the important
+rules to remember:
+
+* You can refer to other distributions in your dependencies - as usual using distribution name. For example,
+ if you are adding a dependency to ``apache-airflow-providers-common-compat`` package from
+ ``apache-airflow-providers-google``, you can just add ``apache-airflow-providers-common>=x.y.z`` to the
+ dependencies and when you run ``uv sync``, the local version of the package will be used automatically
+ (this is thanks to the workspace feature of ``uv`` that does great job of binding our monorepo together).
+ This is very useful when you are developing changes to multiple packages at the same time. Some of those
+ are added automatically by prek hooks - when it can detect such dependencies by analyzing imports in the
+ sources - then they are added automatically between the special comments mentioned above, but sometimes
+ (especially when such dependencies are not at the top-level imports) you might need to add them manually.
+
+* Never update distribution versions on your own. It is **entirely** up to the Release Manager to bump the
+ version of distributions that are defined as ``project.version``. This goes almost without an exception
+ and any diversions from this rule should be discussed at ``#release-management`` channel in Airflow Slack
+ beforehand. The only exception to this rule is when you are adding a new distribution to the repository -
+ in that case you should set the initial version of the distribution - usually ``0.0.1``, ``0.1.0``
+ or ``1.0.0`` depending on the maturity of the package. But still you should discuss it in the channel.
+
+* Sometimes, when you add a new feature to a common distribution, you might add a feature to it or
+ change the API in the way that other packages can use it. This is especially true for common packages such as
+ ``apache-airflow-providers-common-compat``, but can happen for other packages (for example
+ ``apache-airflow-providers-apache-beam`` is used by ``apache-airflow-providers-google`` to use ``Apache Beam``
+ hooks to communicate with Google Dataflow). In such case, when you are adding a feature to a common package
+ remember that the feature you just add. will only be released in the **FUTURE** release of such common
+ package and you cannot add ``>==x.y.z`` dependency to it where ``x.y.z`` is the version you are
+ going to release in the future. Ultimately, this should happen (and happens) when the Release Manager prepares
+ both packages together. Let us repeat - such changes in versions between different airflow package should
+ **NOT** be added to the dependencies manually by the contributor. They should **exclusively** be added by
+ the Release Manager. when preparing the release of **both** packages together.
+ We have a custom mechanism to support such additions, where it is contributor's responsibility to mark
+ dependency with a special comment - simply communicating with the Release Manager that such dependency
+ should be updated to the next version when the release is prepared. If you see such a need to use newly
+ added feature and using it at the same time in a different distribution - make sure to add this comment
+ in the line where dependency you want to use the new feature from is defined:
+
+ .. code-block:: python
+
+ # some deps
Review Comment:
Ah yeah. Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Manages cross-distribution dependencies better [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #58430:
URL: https://github.com/apache/airflow/pull/58430#issuecomment-3548970126
Applied @ashb comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Manages cross-distribution dependencies better [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #58430:
URL: https://github.com/apache/airflow/pull/58430#discussion_r2538486434
##########
contributing-docs/13_airflow_dependencies_and_extras.rst:
##########
@@ -18,6 +18,226 @@
Airflow dependencies
====================
+This document describes how we manage Apache Airflow dependencies, as well as how we make sure
+that users can use Airflow both as an application and as a library when they are deploying
+their own Airflow - using our constraints mechanism.
+
+Airflow ``pyproject.toml`` files and ``uv`` workspace
+.....................................................
+
+Managing dependencies is an important part of maintaining Apache Airflow, we have more than 700
Review Comment:
```suggestion
Managing dependencies is an important part of developing Apache Airflow, we have more than 700
```
Maintaining Airflow is things you do with a running Airflow deployment.
##########
contributing-docs/13_airflow_dependencies_and_extras.rst:
##########
@@ -18,6 +18,226 @@
Airflow dependencies
====================
+This document describes how we manage Apache Airflow dependencies, as well as how we make sure
+that users can use Airflow both as an application and as a library when they are deploying
+their own Airflow - using our constraints mechanism.
+
+Airflow ``pyproject.toml`` files and ``uv`` workspace
+.....................................................
+
+Managing dependencies is an important part of maintaining Apache Airflow, we have more than 700
+dependencies, and often when you add a new feature that requires a new dependency, you should
+also update the dependency list. This also happens when you want to add a new tool that is used
+for development, testing, or building the documentation and this document describes how to do it.
+
+When it comes to defining dependencies for any of the Apache Airflow distributions, we are following the
+standard ``pyproject.toml`` format that has been defined in `PEP 518 <https://peps.python.org/pep-0518/>`_
+and `PEP 621 <https://peps.python.org/pep-0621/>`_, we are also using dependency groups defined in
+`PEP 735 <https://peps.python.org/pep-0735/>`_ - particularly ``dev`` dependency group which we use in
+all our ``pyproject.toml`` files to define development dependencies.
+
+We have more than 100 python distributions in Apache Airflow repository - including the main
+``apache-airflow`` package, ``apache-airflow-providers-*`` packages, ``apache-airflow-core`` package,
+``apache-airflow-task-sdk`` package, ``apache-airflow-ctl`` package, and several other packages.
+
+They are all connected together via ``uv`` workspace feature (workspace is defined in the root ``pyproject.toml``
+file of the repository in the ``apache-airflow`` distribution definition. The workspace feature allows us to
+run ``uv sync`` at the top of the repository to install all packages in editable mode in the development
+environment from all the 100+ distributions (!) and resolve the dependencies together, so that we know that
+the dependencies have no conflicting requirements. Also the distributions are referring to each other via
+name - which means that when you run locally ``uv sync``, the local version of the packages are used, not the
+ones released on PyPI, which means that you can develop and test changes that span multiple packages at the
+same time. This is a very powerful feature that allows us to maintain the complex ecosystem of Apache Airflow
+distributions in a single monorepo, and allows us - for example to add new feature to common distributions used
+by multiple providers and test them all together before releasing new versions of either of those pacckages
+
+Managing dependencies in ``pyproject.toml`` files
+.................................................
+
+Each of the ``pyproject.toml`` files in Apache Airflow repository defines dependencies in one of the
+following sections:
+
+* ``[project.dependencies]`` - this section defines the required dependencies of the package. These
+ dependencies are installed when you install the package without any extras.
+* ``[project.optional-dependencies]`` - this section defines optional dependencies (extras) of the package.
+ These dependencies are installed when you install the package with extras - for example
+ ``pip install apache-airflow[ssh]`` will install the ``ssh`` extra dependencies defined in this section.
+* ``[dependency-group.dev]`` - this section defines development dependencies of the package.
+ These dependencies are installed when you run ``uv sync`` by default. when ``uv`` syncs sources with
+ local pyproject.toml it adds ``dev`` dependency group and package is installed in editable mode with
+ development dependencies.
+
+
+Adding and modifying dependencies
+.................................
+
+Adding and modifying dependencies in Apache Airflow is done by modifying the appropriate
+``pyproject.toml`` file in the appropriate distribution.
+
+When you add a new dependency, you should make sure that:
+
+* The dependency is added to the right section (main dependencies, optional dependencies, or
+ development dependencies)
+
+* Some parts of those dependencies might be automatically generated (and overwritten) by our ``prek``
+ hooks. Those are the necessary dependencies that ``prek`` hoks can figure out automatically by
+ analyzing the imports in the sources and structure of the project. We also have special case of
+ shared dependencies (described in `shared dependencies document <../shared/README.md>`__) where we
+ effectively "static-link" some libraries into multiple distributions, to avoid unnecessary coupling and
+ circular dependencies, and those dependencies contribute some dependencies automatically as well.
+ Pay attention to comments such us example start and end of such generated block below.
+ All the dependencies between those comments are automatically generated and you should not modify
+ them manually. Tou might instead modify the root source of those dependencies - depending on the automation,
+ you can usually also add extra dependencies manually outside of those comment blocks
+
+ .. code-block:: python
+
+ # Automatically generated airflow optional dependencies (update_airflow_pyproject_toml.py)
+ # ....
+ # LOTS OF GENERATED DEPENDENCIES HERE
+ # ....
+ # End of automatically generated airflow optional dependencies
+
+* The version specifier is as open as possible (upper) while still allowing the package to install
+ and pass all tests. We very rarely upper-bind dependencies - only when there is a known
+ conflict with a new or upcoming version of a dependency that breaks the installation or tests
+ (and we always make a comment why we are upper-bounding a dependency).
+
+* Make sure to lower-bind any dependency you add. Usually we lower-bind dependencies to the
+ minimum version that is required for the package to work but in order to simplify the work of
+ resolvers such as ``pip``, we often lower-bind to higher (and newer) version than the absolute minimum
+ especially when the minimum version is very old. This is for example good practice in ``boto`` and related
+ packages, where new version of those packages are released frequently (almost daily) and there are many
+ versions that need to be considered by the resolver if the version is not new enough.
+
+* Make sure to run ``uv sync`` after modifying dependencies to make sure that there are no
+ conflicts between dependencies of different packages in the workspace. You can run it in multiple
+ ways - either from the root of the repository (which will sync all packages) or from the package
+ you modified (which will sync only that package and its dependencies). Also good idea might be to
+ run ``uv sync --all-packages --all-extras`` at the root of the repository to make sure that
+ all packages with all extras can be installed together without conflicts, but this might be sometimes
+ difficult and slow as some of the extras require some additional system level dependencies to be installed
+ (for example ``mysql`` or ``postgres`` extras require client libraries to be installed on the system).
+
+* Make sure to run all tests after modifying dependencies to make sure that nothing is broken
+
+
+Referring to other Apache Airflow distributions in dependencies
+...............................................................
+
+With having more than 100 distributions in the repository, it is often necessary to refer to
+other distributions in the dependencies in order to use some common features or simply to use the
+features that the other distribution provides. There are two ways of doing it:
+
+* Regular package linking with ``apache-airflow-*`` dependency
+* Airflow "shared dependencies" mechanism - which is a bit of a custom hack for Airflow monorepo
+ that allows us to "static-link" some common dependencies into multiple distributions without
+ creating circular dependencies.
+
+We are not going to describe the shared dependencies mechanism here, please refer to the
+`shared dependencies document <../shared/README.md>`__ for details, but there are certain rules
+when it comes to referring to other Airflow distributions in dependencies - here are the important
+rules to remember:
+
+* You can refer to other distributions in your dependencies - as usual using distribution name. For example,
+ if you are adding a dependency to ``apache-airflow-providers-common-compat`` package from
+ ``apache-airflow-providers-google``, you can just add ``apache-airflow-providers-common>=x.y.z`` to the
+ dependencies and when you run ``uv sync``, the local version of the package will be used automatically
+ (this is thanks to the workspace feature of ``uv`` that does great job of binding our monorepo together).
+ This is very useful when you are developing changes to multiple packages at the same time. Some of those
+ are added automatically by prek hooks - when it can detect such dependencies by analyzing imports in the
+ sources - then they are added automatically between the special comments mentioned above, but sometimes
+ (especially when such dependencies are not at the top-level imports) you might need to add them manually.
Review Comment:
You have already mentioned the developing changes locally above. Adding it just makes the point harder to follow.
##########
contributing-docs/13_airflow_dependencies_and_extras.rst:
##########
@@ -18,6 +18,226 @@
Airflow dependencies
====================
+This document describes how we manage Apache Airflow dependencies, as well as how we make sure
+that users can use Airflow both as an application and as a library when they are deploying
+their own Airflow - using our constraints mechanism.
+
+Airflow ``pyproject.toml`` files and ``uv`` workspace
+.....................................................
+
+Managing dependencies is an important part of maintaining Apache Airflow, we have more than 700
+dependencies, and often when you add a new feature that requires a new dependency, you should
+also update the dependency list. This also happens when you want to add a new tool that is used
+for development, testing, or building the documentation and this document describes how to do it.
+
+When it comes to defining dependencies for any of the Apache Airflow distributions, we are following the
+standard ``pyproject.toml`` format that has been defined in `PEP 518 <https://peps.python.org/pep-0518/>`_
+and `PEP 621 <https://peps.python.org/pep-0621/>`_, we are also using dependency groups defined in
+`PEP 735 <https://peps.python.org/pep-0735/>`_ - particularly ``dev`` dependency group which we use in
+all our ``pyproject.toml`` files to define development dependencies.
+
+We have more than 100 python distributions in Apache Airflow repository - including the main
+``apache-airflow`` package, ``apache-airflow-providers-*`` packages, ``apache-airflow-core`` package,
+``apache-airflow-task-sdk`` package, ``apache-airflow-ctl`` package, and several other packages.
+
+They are all connected together via ``uv`` workspace feature (workspace is defined in the root ``pyproject.toml``
+file of the repository in the ``apache-airflow`` distribution definition. The workspace feature allows us to
+run ``uv sync`` at the top of the repository to install all packages in editable mode in the development
+environment from all the 100+ distributions (!) and resolve the dependencies together, so that we know that
+the dependencies have no conflicting requirements. Also the distributions are referring to each other via
+name - which means that when you run locally ``uv sync``, the local version of the packages are used, not the
+ones released on PyPI, which means that you can develop and test changes that span multiple packages at the
+same time. This is a very powerful feature that allows us to maintain the complex ecosystem of Apache Airflow
+distributions in a single monorepo, and allows us - for example to add new feature to common distributions used
+by multiple providers and test them all together before releasing new versions of either of those pacckages
+
+Managing dependencies in ``pyproject.toml`` files
+.................................................
+
+Each of the ``pyproject.toml`` files in Apache Airflow repository defines dependencies in one of the
+following sections:
+
+* ``[project.dependencies]`` - this section defines the required dependencies of the package. These
+ dependencies are installed when you install the package without any extras.
+* ``[project.optional-dependencies]`` - this section defines optional dependencies (extras) of the package.
+ These dependencies are installed when you install the package with extras - for example
+ ``pip install apache-airflow[ssh]`` will install the ``ssh`` extra dependencies defined in this section.
+* ``[dependency-group.dev]`` - this section defines development dependencies of the package.
+ These dependencies are installed when you run ``uv sync`` by default. when ``uv`` syncs sources with
+ local pyproject.toml it adds ``dev`` dependency group and package is installed in editable mode with
+ development dependencies.
+
+
+Adding and modifying dependencies
+.................................
+
+Adding and modifying dependencies in Apache Airflow is done by modifying the appropriate
+``pyproject.toml`` file in the appropriate distribution.
+
+When you add a new dependency, you should make sure that:
+
+* The dependency is added to the right section (main dependencies, optional dependencies, or
+ development dependencies)
+
+* Some parts of those dependencies might be automatically generated (and overwritten) by our ``prek``
+ hooks. Those are the necessary dependencies that ``prek`` hoks can figure out automatically by
+ analyzing the imports in the sources and structure of the project. We also have special case of
+ shared dependencies (described in `shared dependencies document <../shared/README.md>`__) where we
+ effectively "static-link" some libraries into multiple distributions, to avoid unnecessary coupling and
+ circular dependencies, and those dependencies contribute some dependencies automatically as well.
+ Pay attention to comments such us example start and end of such generated block below.
+ All the dependencies between those comments are automatically generated and you should not modify
+ them manually. Tou might instead modify the root source of those dependencies - depending on the automation,
Review Comment:
```suggestion
them manually. You might instead modify the root source of those dependencies - depending on the automation,
```
##########
contributing-docs/13_airflow_dependencies_and_extras.rst:
##########
@@ -18,6 +18,226 @@
Airflow dependencies
====================
+This document describes how we manage Apache Airflow dependencies, as well as how we make sure
+that users can use Airflow both as an application and as a library when they are deploying
+their own Airflow - using our constraints mechanism.
+
+Airflow ``pyproject.toml`` files and ``uv`` workspace
+.....................................................
+
+Managing dependencies is an important part of maintaining Apache Airflow, we have more than 700
+dependencies, and often when you add a new feature that requires a new dependency, you should
+also update the dependency list. This also happens when you want to add a new tool that is used
+for development, testing, or building the documentation and this document describes how to do it.
+
+When it comes to defining dependencies for any of the Apache Airflow distributions, we are following the
+standard ``pyproject.toml`` format that has been defined in `PEP 518 <https://peps.python.org/pep-0518/>`_
+and `PEP 621 <https://peps.python.org/pep-0621/>`_, we are also using dependency groups defined in
+`PEP 735 <https://peps.python.org/pep-0735/>`_ - particularly ``dev`` dependency group which we use in
+all our ``pyproject.toml`` files to define development dependencies.
+
+We have more than 100 python distributions in Apache Airflow repository - including the main
+``apache-airflow`` package, ``apache-airflow-providers-*`` packages, ``apache-airflow-core`` package,
+``apache-airflow-task-sdk`` package, ``apache-airflow-ctl`` package, and several other packages.
+
+They are all connected together via ``uv`` workspace feature (workspace is defined in the root ``pyproject.toml``
+file of the repository in the ``apache-airflow`` distribution definition. The workspace feature allows us to
+run ``uv sync`` at the top of the repository to install all packages in editable mode in the development
+environment from all the 100+ distributions (!) and resolve the dependencies together, so that we know that
+the dependencies have no conflicting requirements. Also the distributions are referring to each other via
+name - which means that when you run locally ``uv sync``, the local version of the packages are used, not the
+ones released on PyPI, which means that you can develop and test changes that span multiple packages at the
+same time. This is a very powerful feature that allows us to maintain the complex ecosystem of Apache Airflow
+distributions in a single monorepo, and allows us - for example to add new feature to common distributions used
+by multiple providers and test them all together before releasing new versions of either of those pacckages
+
+Managing dependencies in ``pyproject.toml`` files
+.................................................
+
+Each of the ``pyproject.toml`` files in Apache Airflow repository defines dependencies in one of the
+following sections:
+
+* ``[project.dependencies]`` - this section defines the required dependencies of the package. These
+ dependencies are installed when you install the package without any extras.
+* ``[project.optional-dependencies]`` - this section defines optional dependencies (extras) of the package.
+ These dependencies are installed when you install the package with extras - for example
+ ``pip install apache-airflow[ssh]`` will install the ``ssh`` extra dependencies defined in this section.
+* ``[dependency-group.dev]`` - this section defines development dependencies of the package.
+ These dependencies are installed when you run ``uv sync`` by default. when ``uv`` syncs sources with
+ local pyproject.toml it adds ``dev`` dependency group and package is installed in editable mode with
+ development dependencies.
+
+
+Adding and modifying dependencies
+.................................
+
+Adding and modifying dependencies in Apache Airflow is done by modifying the appropriate
+``pyproject.toml`` file in the appropriate distribution.
+
+When you add a new dependency, you should make sure that:
+
+* The dependency is added to the right section (main dependencies, optional dependencies, or
+ development dependencies)
+
+* Some parts of those dependencies might be automatically generated (and overwritten) by our ``prek``
+ hooks. Those are the necessary dependencies that ``prek`` hoks can figure out automatically by
+ analyzing the imports in the sources and structure of the project. We also have special case of
+ shared dependencies (described in `shared dependencies document <../shared/README.md>`__) where we
+ effectively "static-link" some libraries into multiple distributions, to avoid unnecessary coupling and
+ circular dependencies, and those dependencies contribute some dependencies automatically as well.
+ Pay attention to comments such us example start and end of such generated block below.
+ All the dependencies between those comments are automatically generated and you should not modify
+ them manually. Tou might instead modify the root source of those dependencies - depending on the automation,
+ you can usually also add extra dependencies manually outside of those comment blocks
+
+ .. code-block:: python
+
+ # Automatically generated airflow optional dependencies (update_airflow_pyproject_toml.py)
+ # ....
+ # LOTS OF GENERATED DEPENDENCIES HERE
+ # ....
+ # End of automatically generated airflow optional dependencies
+
+* The version specifier is as open as possible (upper) while still allowing the package to install
+ and pass all tests. We very rarely upper-bind dependencies - only when there is a known
+ conflict with a new or upcoming version of a dependency that breaks the installation or tests
+ (and we always make a comment why we are upper-bounding a dependency).
+
+* Make sure to lower-bind any dependency you add. Usually we lower-bind dependencies to the
+ minimum version that is required for the package to work but in order to simplify the work of
+ resolvers such as ``pip``, we often lower-bind to higher (and newer) version than the absolute minimum
+ especially when the minimum version is very old. This is for example good practice in ``boto`` and related
+ packages, where new version of those packages are released frequently (almost daily) and there are many
+ versions that need to be considered by the resolver if the version is not new enough.
+
+* Make sure to run ``uv sync`` after modifying dependencies to make sure that there are no
+ conflicts between dependencies of different packages in the workspace. You can run it in multiple
+ ways - either from the root of the repository (which will sync all packages) or from the package
+ you modified (which will sync only that package and its dependencies). Also good idea might be to
+ run ``uv sync --all-packages --all-extras`` at the root of the repository to make sure that
+ all packages with all extras can be installed together without conflicts, but this might be sometimes
+ difficult and slow as some of the extras require some additional system level dependencies to be installed
+ (for example ``mysql`` or ``postgres`` extras require client libraries to be installed on the system).
+
+* Make sure to run all tests after modifying dependencies to make sure that nothing is broken
+
+
+Referring to other Apache Airflow distributions in dependencies
+...............................................................
+
+With having more than 100 distributions in the repository, it is often necessary to refer to
+other distributions in the dependencies in order to use some common features or simply to use the
+features that the other distribution provides. There are two ways of doing it:
+
+* Regular package linking with ``apache-airflow-*`` dependency
+* Airflow "shared dependencies" mechanism - which is a bit of a custom hack for Airflow monorepo
+ that allows us to "static-link" some common dependencies into multiple distributions without
+ creating circular dependencies.
+
+We are not going to describe the shared dependencies mechanism here, please refer to the
+`shared dependencies document <../shared/README.md>`__ for details, but there are certain rules
+when it comes to referring to other Airflow distributions in dependencies - here are the important
+rules to remember:
+
+* You can refer to other distributions in your dependencies - as usual using distribution name. For example,
+ if you are adding a dependency to ``apache-airflow-providers-common-compat`` package from
+ ``apache-airflow-providers-google``, you can just add ``apache-airflow-providers-common>=x.y.z`` to the
+ dependencies and when you run ``uv sync``, the local version of the package will be used automatically
+ (this is thanks to the workspace feature of ``uv`` that does great job of binding our monorepo together).
+ This is very useful when you are developing changes to multiple packages at the same time. Some of those
+ are added automatically by prek hooks - when it can detect such dependencies by analyzing imports in the
+ sources - then they are added automatically between the special comments mentioned above, but sometimes
+ (especially when such dependencies are not at the top-level imports) you might need to add them manually.
+
+* Never update distribution versions on your own. It is **entirely** up to the Release Manager to bump the
+ version of distributions that are defined as ``project.version``. This goes almost without an exception
+ and any diversions from this rule should be discussed at ``#release-management`` channel in Airflow Slack
+ beforehand. The only exception to this rule is when you are adding a new distribution to the repository -
+ in that case you should set the initial version of the distribution - usually ``0.0.1``, ``0.1.0``
+ or ``1.0.0`` depending on the maturity of the package. But still you should discuss it in the channel.
+
+* Sometimes, when you add a new feature to a common distribution, you might add a feature to it or
+ change the API in the way that other packages can use it. This is especially true for common packages such as
+ ``apache-airflow-providers-common-compat``, but can happen for other packages (for example
+ ``apache-airflow-providers-apache-beam`` is used by ``apache-airflow-providers-google`` to use ``Apache Beam``
+ hooks to communicate with Google Dataflow). In such case, when you are adding a feature to a common package
+ remember that the feature you just add. will only be released in the **FUTURE** release of such common
+ package and you cannot add ``>==x.y.z`` dependency to it where ``x.y.z`` is the version you are
+ going to release in the future. Ultimately, this should happen (and happens) when the Release Manager prepares
+ both packages together. Let us repeat - such changes in versions between different airflow package should
+ **NOT** be added to the dependencies manually by the contributor. They should **exclusively** be added by
+ the Release Manager. when preparing the release of **both** packages together.
+ We have a custom mechanism to support such additions, where it is contributor's responsibility to mark
+ dependency with a special comment - simply communicating with the Release Manager that such dependency
+ should be updated to the next version when the release is prepared. If you see such a need to use newly
+ added feature and using it at the same time in a different distribution - make sure to add this comment
+ in the line where dependency you want to use the new feature from is defined:
+
+ .. code-block:: python
+
+ # some deps
+ "apache-airflow-SOMETHING>1.0.0", # use next version
Review Comment:
Is `use next version` an instruction to a human, or a precise string that CI picks up and must match?
##########
contributing-docs/13_airflow_dependencies_and_extras.rst:
##########
@@ -18,6 +18,226 @@
Airflow dependencies
====================
+This document describes how we manage Apache Airflow dependencies, as well as how we make sure
+that users can use Airflow both as an application and as a library when they are deploying
+their own Airflow - using our constraints mechanism.
+
+Airflow ``pyproject.toml`` files and ``uv`` workspace
+.....................................................
+
+Managing dependencies is an important part of maintaining Apache Airflow, we have more than 700
+dependencies, and often when you add a new feature that requires a new dependency, you should
+also update the dependency list. This also happens when you want to add a new tool that is used
+for development, testing, or building the documentation and this document describes how to do it.
+
+When it comes to defining dependencies for any of the Apache Airflow distributions, we are following the
+standard ``pyproject.toml`` format that has been defined in `PEP 518 <https://peps.python.org/pep-0518/>`_
+and `PEP 621 <https://peps.python.org/pep-0621/>`_, we are also using dependency groups defined in
+`PEP 735 <https://peps.python.org/pep-0735/>`_ - particularly ``dev`` dependency group which we use in
+all our ``pyproject.toml`` files to define development dependencies.
+
+We have more than 100 python distributions in Apache Airflow repository - including the main
+``apache-airflow`` package, ``apache-airflow-providers-*`` packages, ``apache-airflow-core`` package,
+``apache-airflow-task-sdk`` package, ``apache-airflow-ctl`` package, and several other packages.
+
+They are all connected together via ``uv`` workspace feature (workspace is defined in the root ``pyproject.toml``
+file of the repository in the ``apache-airflow`` distribution definition. The workspace feature allows us to
+run ``uv sync`` at the top of the repository to install all packages in editable mode in the development
+environment from all the 100+ distributions (!) and resolve the dependencies together, so that we know that
+the dependencies have no conflicting requirements. Also the distributions are referring to each other via
+name - which means that when you run locally ``uv sync``, the local version of the packages are used, not the
+ones released on PyPI, which means that you can develop and test changes that span multiple packages at the
+same time. This is a very powerful feature that allows us to maintain the complex ecosystem of Apache Airflow
+distributions in a single monorepo, and allows us - for example to add new feature to common distributions used
+by multiple providers and test them all together before releasing new versions of either of those pacckages
+
+Managing dependencies in ``pyproject.toml`` files
+.................................................
+
+Each of the ``pyproject.toml`` files in Apache Airflow repository defines dependencies in one of the
+following sections:
+
+* ``[project.dependencies]`` - this section defines the required dependencies of the package. These
+ dependencies are installed when you install the package without any extras.
+* ``[project.optional-dependencies]`` - this section defines optional dependencies (extras) of the package.
+ These dependencies are installed when you install the package with extras - for example
+ ``pip install apache-airflow[ssh]`` will install the ``ssh`` extra dependencies defined in this section.
+* ``[dependency-group.dev]`` - this section defines development dependencies of the package.
+ These dependencies are installed when you run ``uv sync`` by default. when ``uv`` syncs sources with
+ local pyproject.toml it adds ``dev`` dependency group and package is installed in editable mode with
+ development dependencies.
+
+
+Adding and modifying dependencies
+.................................
+
+Adding and modifying dependencies in Apache Airflow is done by modifying the appropriate
+``pyproject.toml`` file in the appropriate distribution.
+
+When you add a new dependency, you should make sure that:
+
+* The dependency is added to the right section (main dependencies, optional dependencies, or
+ development dependencies)
+
+* Some parts of those dependencies might be automatically generated (and overwritten) by our ``prek``
+ hooks. Those are the necessary dependencies that ``prek`` hoks can figure out automatically by
+ analyzing the imports in the sources and structure of the project. We also have special case of
+ shared dependencies (described in `shared dependencies document <../shared/README.md>`__) where we
+ effectively "static-link" some libraries into multiple distributions, to avoid unnecessary coupling and
+ circular dependencies, and those dependencies contribute some dependencies automatically as well.
+ Pay attention to comments such us example start and end of such generated block below.
+ All the dependencies between those comments are automatically generated and you should not modify
+ them manually. Tou might instead modify the root source of those dependencies - depending on the automation,
+ you can usually also add extra dependencies manually outside of those comment blocks
+
+ .. code-block:: python
+
+ # Automatically generated airflow optional dependencies (update_airflow_pyproject_toml.py)
+ # ....
+ # LOTS OF GENERATED DEPENDENCIES HERE
+ # ....
+ # End of automatically generated airflow optional dependencies
+
+* The version specifier is as open as possible (upper) while still allowing the package to install
+ and pass all tests. We very rarely upper-bind dependencies - only when there is a known
+ conflict with a new or upcoming version of a dependency that breaks the installation or tests
+ (and we always make a comment why we are upper-bounding a dependency).
+
+* Make sure to lower-bind any dependency you add. Usually we lower-bind dependencies to the
+ minimum version that is required for the package to work but in order to simplify the work of
+ resolvers such as ``pip``, we often lower-bind to higher (and newer) version than the absolute minimum
+ especially when the minimum version is very old. This is for example good practice in ``boto`` and related
+ packages, where new version of those packages are released frequently (almost daily) and there are many
+ versions that need to be considered by the resolver if the version is not new enough.
+
+* Make sure to run ``uv sync`` after modifying dependencies to make sure that there are no
+ conflicts between dependencies of different packages in the workspace. You can run it in multiple
+ ways - either from the root of the repository (which will sync all packages) or from the package
+ you modified (which will sync only that package and its dependencies). Also good idea might be to
+ run ``uv sync --all-packages --all-extras`` at the root of the repository to make sure that
+ all packages with all extras can be installed together without conflicts, but this might be sometimes
+ difficult and slow as some of the extras require some additional system level dependencies to be installed
+ (for example ``mysql`` or ``postgres`` extras require client libraries to be installed on the system).
+
+* Make sure to run all tests after modifying dependencies to make sure that nothing is broken
+
+
+Referring to other Apache Airflow distributions in dependencies
+...............................................................
+
+With having more than 100 distributions in the repository, it is often necessary to refer to
+other distributions in the dependencies in order to use some common features or simply to use the
+features that the other distribution provides. There are two ways of doing it:
+
+* Regular package linking with ``apache-airflow-*`` dependency
+* Airflow "shared dependencies" mechanism - which is a bit of a custom hack for Airflow monorepo
+ that allows us to "static-link" some common dependencies into multiple distributions without
+ creating circular dependencies.
+
+We are not going to describe the shared dependencies mechanism here, please refer to the
+`shared dependencies document <../shared/README.md>`__ for details, but there are certain rules
+when it comes to referring to other Airflow distributions in dependencies - here are the important
+rules to remember:
+
+* You can refer to other distributions in your dependencies - as usual using distribution name. For example,
+ if you are adding a dependency to ``apache-airflow-providers-common-compat`` package from
+ ``apache-airflow-providers-google``, you can just add ``apache-airflow-providers-common>=x.y.z`` to the
+ dependencies and when you run ``uv sync``, the local version of the package will be used automatically
+ (this is thanks to the workspace feature of ``uv`` that does great job of binding our monorepo together).
+ This is very useful when you are developing changes to multiple packages at the same time. Some of those
+ are added automatically by prek hooks - when it can detect such dependencies by analyzing imports in the
+ sources - then they are added automatically between the special comments mentioned above, but sometimes
+ (especially when such dependencies are not at the top-level imports) you might need to add them manually.
+
+* Never update distribution versions on your own. It is **entirely** up to the Release Manager to bump the
+ version of distributions that are defined as ``project.version``. This goes almost without an exception
+ and any diversions from this rule should be discussed at ``#release-management`` channel in Airflow Slack
+ beforehand. The only exception to this rule is when you are adding a new distribution to the repository -
+ in that case you should set the initial version of the distribution - usually ``0.0.1``, ``0.1.0``
+ or ``1.0.0`` depending on the maturity of the package. But still you should discuss it in the channel.
+
+* Sometimes, when you add a new feature to a common distribution, you might add a feature to it or
+ change the API in the way that other packages can use it. This is especially true for common packages such as
+ ``apache-airflow-providers-common-compat``, but can happen for other packages (for example
+ ``apache-airflow-providers-apache-beam`` is used by ``apache-airflow-providers-google`` to use ``Apache Beam``
+ hooks to communicate with Google Dataflow). In such case, when you are adding a feature to a common package
+ remember that the feature you just add. will only be released in the **FUTURE** release of such common
+ package and you cannot add ``>==x.y.z`` dependency to it where ``x.y.z`` is the version you are
+ going to release in the future. Ultimately, this should happen (and happens) when the Release Manager prepares
+ both packages together. Let us repeat - such changes in versions between different airflow package should
+ **NOT** be added to the dependencies manually by the contributor. They should **exclusively** be added by
+ the Release Manager. when preparing the release of **both** packages together.
+ We have a custom mechanism to support such additions, where it is contributor's responsibility to mark
+ dependency with a special comment - simply communicating with the Release Manager that such dependency
+ should be updated to the next version when the release is prepared. If you see such a need to use newly
+ added feature and using it at the same time in a different distribution - make sure to add this comment
+ in the line where dependency you want to use the new feature from is defined:
+
+ .. code-block:: python
+
+ # some deps
Review Comment:
At first I thought "some deps" was an important comment.
```suggestion
# ...
# other deps here
# ...
```
##########
dev/breeze/tests/test_selective_checks.py:
##########
@@ -2779,3 +2779,169 @@ def test_testable_providers_integrations_excludes_arm_disabled_on_arm():
assert "postgres" in result
assert "trino" not in result
assert "ydb" not in result
+
+
+@patch("airflow_breeze.utils.selective_checks.run_command")
+def test_provider_dependency_bump_check_no_changes(mock_run_command):
+ """Test that provider dependency bump check passes when no pyproject.toml files are changed."""
+ selective_checks = SelectiveChecks(
+ files=("some_other_file.py",),
+ commit_ref=NEUTRAL_COMMIT,
+ pr_labels=(),
+ github_event=GithubEvents.PULL_REQUEST,
+ default_branch="main",
+ )
+ result = selective_checks.provider_dependency_bump
+ assert result is False
+
+
+@patch("airflow_breeze.utils.selective_checks.run_command")
+def test_provider_dependency_bump_check_fails_on_provider_version_bump(mock_run_command):
+ """Test that provider dependency bump check fails when provider version is bumped without label."""
+ old_toml = """
+[project]
+dependencies = [
+ "apache-airflow-providers-common-sql>=1.0.0",
+]
+"""
+ new_toml = """
+[project]
+dependencies = [
+ "apache-airflow-providers-common-sql>=1.1.0",
+]
+"""
+
+ def side_effect(*args, **kwargs):
+ result = Mock()
+ result.returncode = 0
+ if "^:" in args[0][2]:
+ result.stdout = old_toml
+ else:
+ result.stdout = new_toml
+ return result
+
+ mock_run_command.side_effect = side_effect
+
+ with pytest.raises(SystemExit):
+ SelectiveChecks(
+ files=("providers/amazon/pyproject.toml",),
+ commit_ref=NEUTRAL_COMMIT,
+ pr_labels=(),
+ github_event=GithubEvents.PULL_REQUEST,
+ default_branch="main",
+ ).provider_dependency_bump
+
+
+@patch("airflow_breeze.utils.selective_checks.run_command")
+def test_provider_dependency_bump_check_passes_with_label(mock_run_command):
+ """Test that provider dependency bump check passes when label is set."""
+ old_toml = """
+[project]
+dependencies = [
+ "apache-airflow-providers-common-sql>=1.0.0",
+]
+"""
+ new_toml = """
+[project]
+dependencies = [
+ "apache-airflow-providers-common-sql>=1.1.0",
+]
+"""
+
+ def side_effect(*args, **kwargs):
+ result = Mock()
+ result.returncode = 0
+ if "^:" in args[0][2]:
+ result.stdout = old_toml
+ else:
+ result.stdout = new_toml
+ return result
+
+ mock_run_command.side_effect = side_effect
+
+ selective_checks = SelectiveChecks(
+ files=("providers/amazon/pyproject.toml",),
+ commit_ref=NEUTRAL_COMMIT,
+ pr_labels=("allow provider dependency bump",),
+ github_event=GithubEvents.PULL_REQUEST,
+ default_branch="main",
+ )
+ result = selective_checks.provider_dependency_bump
+ assert result is True
+
+
+@patch("airflow_breeze.utils.selective_checks.run_command")
+def test_provider_dependency_bump_check_passes_on_non_provider_dependency_changes(mock_run_command):
+ """Test that provider dependency bump check passes when non-provider dependencies change."""
+ old_toml = """
+[project]
+dependencies = [
+ "apache-airflow>=2.10.0",
+ "boto3>=1.37.0",
+]
+"""
+ new_toml = """
+[project]
+dependencies = [
+ "apache-airflow>=2.10.0",
+ "boto3>=1.38.0",
+]
+"""
+
+ def side_effect(*args, **kwargs):
+ result = Mock()
+ result.returncode = 0
+ if "^:" in args[0][2]:
+ result.stdout = old_toml
+ else:
+ result.stdout = new_toml
+ return result
+
+ mock_run_command.side_effect = side_effect
+
+ selective_checks = SelectiveChecks(
+ files=("providers/amazon/pyproject.toml",),
+ commit_ref=NEUTRAL_COMMIT,
+ pr_labels=(),
+ github_event=GithubEvents.PULL_REQUEST,
+ default_branch="main",
+ )
+ result = selective_checks.provider_dependency_bump
+ assert result is False
+
+
+@patch("airflow_breeze.utils.selective_checks.run_command")
+def test_provider_dependency_bump_check_in_optional_dependencies(mock_run_command):
Review Comment:
Why are optional deps exempt from the check? If the goal is to stop things being used before they are released, then wouldn't these be equally as prone to version issues?
##########
contributing-docs/13_airflow_dependencies_and_extras.rst:
##########
@@ -18,6 +18,226 @@
Airflow dependencies
====================
+This document describes how we manage Apache Airflow dependencies, as well as how we make sure
+that users can use Airflow both as an application and as a library when they are deploying
+their own Airflow - using our constraints mechanism.
+
+Airflow ``pyproject.toml`` files and ``uv`` workspace
+.....................................................
+
+Managing dependencies is an important part of maintaining Apache Airflow, we have more than 700
+dependencies, and often when you add a new feature that requires a new dependency, you should
+also update the dependency list. This also happens when you want to add a new tool that is used
+for development, testing, or building the documentation and this document describes how to do it.
+
+When it comes to defining dependencies for any of the Apache Airflow distributions, we are following the
+standard ``pyproject.toml`` format that has been defined in `PEP 518 <https://peps.python.org/pep-0518/>`_
+and `PEP 621 <https://peps.python.org/pep-0621/>`_, we are also using dependency groups defined in
+`PEP 735 <https://peps.python.org/pep-0735/>`_ - particularly ``dev`` dependency group which we use in
+all our ``pyproject.toml`` files to define development dependencies.
+
+We have more than 100 python distributions in Apache Airflow repository - including the main
+``apache-airflow`` package, ``apache-airflow-providers-*`` packages, ``apache-airflow-core`` package,
+``apache-airflow-task-sdk`` package, ``apache-airflow-ctl`` package, and several other packages.
+
+They are all connected together via ``uv`` workspace feature (workspace is defined in the root ``pyproject.toml``
+file of the repository in the ``apache-airflow`` distribution definition. The workspace feature allows us to
+run ``uv sync`` at the top of the repository to install all packages in editable mode in the development
+environment from all the 100+ distributions (!) and resolve the dependencies together, so that we know that
+the dependencies have no conflicting requirements. Also the distributions are referring to each other via
+name - which means that when you run locally ``uv sync``, the local version of the packages are used, not the
+ones released on PyPI, which means that you can develop and test changes that span multiple packages at the
+same time. This is a very powerful feature that allows us to maintain the complex ecosystem of Apache Airflow
+distributions in a single monorepo, and allows us - for example to add new feature to common distributions used
+by multiple providers and test them all together before releasing new versions of either of those pacckages
+
+Managing dependencies in ``pyproject.toml`` files
+.................................................
+
+Each of the ``pyproject.toml`` files in Apache Airflow repository defines dependencies in one of the
+following sections:
+
+* ``[project.dependencies]`` - this section defines the required dependencies of the package. These
+ dependencies are installed when you install the package without any extras.
+* ``[project.optional-dependencies]`` - this section defines optional dependencies (extras) of the package.
+ These dependencies are installed when you install the package with extras - for example
+ ``pip install apache-airflow[ssh]`` will install the ``ssh`` extra dependencies defined in this section.
+* ``[dependency-group.dev]`` - this section defines development dependencies of the package.
+ These dependencies are installed when you run ``uv sync`` by default. when ``uv`` syncs sources with
+ local pyproject.toml it adds ``dev`` dependency group and package is installed in editable mode with
+ development dependencies.
+
+
+Adding and modifying dependencies
+.................................
+
+Adding and modifying dependencies in Apache Airflow is done by modifying the appropriate
+``pyproject.toml`` file in the appropriate distribution.
+
+When you add a new dependency, you should make sure that:
+
+* The dependency is added to the right section (main dependencies, optional dependencies, or
+ development dependencies)
+
+* Some parts of those dependencies might be automatically generated (and overwritten) by our ``prek``
+ hooks. Those are the necessary dependencies that ``prek`` hoks can figure out automatically by
+ analyzing the imports in the sources and structure of the project. We also have special case of
+ shared dependencies (described in `shared dependencies document <../shared/README.md>`__) where we
+ effectively "static-link" some libraries into multiple distributions, to avoid unnecessary coupling and
+ circular dependencies, and those dependencies contribute some dependencies automatically as well.
+ Pay attention to comments such us example start and end of such generated block below.
+ All the dependencies between those comments are automatically generated and you should not modify
+ them manually. Tou might instead modify the root source of those dependencies - depending on the automation,
+ you can usually also add extra dependencies manually outside of those comment blocks
+
+ .. code-block:: python
+
+ # Automatically generated airflow optional dependencies (update_airflow_pyproject_toml.py)
+ # ....
+ # LOTS OF GENERATED DEPENDENCIES HERE
+ # ....
+ # End of automatically generated airflow optional dependencies
+
+* The version specifier is as open as possible (upper) while still allowing the package to install
+ and pass all tests. We very rarely upper-bind dependencies - only when there is a known
+ conflict with a new or upcoming version of a dependency that breaks the installation or tests
+ (and we always make a comment why we are upper-bounding a dependency).
+
+* Make sure to lower-bind any dependency you add. Usually we lower-bind dependencies to the
+ minimum version that is required for the package to work but in order to simplify the work of
+ resolvers such as ``pip``, we often lower-bind to higher (and newer) version than the absolute minimum
+ especially when the minimum version is very old. This is for example good practice in ``boto`` and related
+ packages, where new version of those packages are released frequently (almost daily) and there are many
+ versions that need to be considered by the resolver if the version is not new enough.
+
+* Make sure to run ``uv sync`` after modifying dependencies to make sure that there are no
+ conflicts between dependencies of different packages in the workspace. You can run it in multiple
+ ways - either from the root of the repository (which will sync all packages) or from the package
+ you modified (which will sync only that package and its dependencies). Also good idea might be to
+ run ``uv sync --all-packages --all-extras`` at the root of the repository to make sure that
+ all packages with all extras can be installed together without conflicts, but this might be sometimes
+ difficult and slow as some of the extras require some additional system level dependencies to be installed
+ (for example ``mysql`` or ``postgres`` extras require client libraries to be installed on the system).
+
+* Make sure to run all tests after modifying dependencies to make sure that nothing is broken
+
+
+Referring to other Apache Airflow distributions in dependencies
+...............................................................
+
+With having more than 100 distributions in the repository, it is often necessary to refer to
+other distributions in the dependencies in order to use some common features or simply to use the
+features that the other distribution provides. There are two ways of doing it:
Review Comment:
This is the third time we mention how many dists we have.
```suggestion
It is often necessary to refer to other distributions in the dependencies. There are two
ways of doing it:
```
##########
dev/breeze/tests/test_selective_checks.py:
##########
@@ -2779,3 +2779,169 @@ def test_testable_providers_integrations_excludes_arm_disabled_on_arm():
assert "postgres" in result
assert "trino" not in result
assert "ydb" not in result
+
+
+@patch("airflow_breeze.utils.selective_checks.run_command")
+def test_provider_dependency_bump_check_no_changes(mock_run_command):
+ """Test that provider dependency bump check passes when no pyproject.toml files are changed."""
+ selective_checks = SelectiveChecks(
+ files=("some_other_file.py",),
+ commit_ref=NEUTRAL_COMMIT,
+ pr_labels=(),
+ github_event=GithubEvents.PULL_REQUEST,
+ default_branch="main",
+ )
+ result = selective_checks.provider_dependency_bump
+ assert result is False
+
+
+@patch("airflow_breeze.utils.selective_checks.run_command")
+def test_provider_dependency_bump_check_fails_on_provider_version_bump(mock_run_command):
Review Comment:
Is this a common enough occurance that we need to bake it in to our CI?
##########
contributing-docs/13_airflow_dependencies_and_extras.rst:
##########
@@ -18,6 +18,226 @@
Airflow dependencies
====================
+This document describes how we manage Apache Airflow dependencies, as well as how we make sure
+that users can use Airflow both as an application and as a library when they are deploying
+their own Airflow - using our constraints mechanism.
+
+Airflow ``pyproject.toml`` files and ``uv`` workspace
+.....................................................
+
+Managing dependencies is an important part of maintaining Apache Airflow, we have more than 700
+dependencies, and often when you add a new feature that requires a new dependency, you should
+also update the dependency list. This also happens when you want to add a new tool that is used
+for development, testing, or building the documentation and this document describes how to do it.
+
+When it comes to defining dependencies for any of the Apache Airflow distributions, we are following the
+standard ``pyproject.toml`` format that has been defined in `PEP 518 <https://peps.python.org/pep-0518/>`_
+and `PEP 621 <https://peps.python.org/pep-0621/>`_, we are also using dependency groups defined in
+`PEP 735 <https://peps.python.org/pep-0735/>`_ - particularly ``dev`` dependency group which we use in
+all our ``pyproject.toml`` files to define development dependencies.
+
+We have more than 100 python distributions in Apache Airflow repository - including the main
+``apache-airflow`` package, ``apache-airflow-providers-*`` packages, ``apache-airflow-core`` package,
+``apache-airflow-task-sdk`` package, ``apache-airflow-ctl`` package, and several other packages.
+
+They are all connected together via ``uv`` workspace feature (workspace is defined in the root ``pyproject.toml``
+file of the repository in the ``apache-airflow`` distribution definition. The workspace feature allows us to
+run ``uv sync`` at the top of the repository to install all packages in editable mode in the development
+environment from all the 100+ distributions (!) and resolve the dependencies together, so that we know that
+the dependencies have no conflicting requirements. Also the distributions are referring to each other via
+name - which means that when you run locally ``uv sync``, the local version of the packages are used, not the
+ones released on PyPI, which means that you can develop and test changes that span multiple packages at the
+same time. This is a very powerful feature that allows us to maintain the complex ecosystem of Apache Airflow
+distributions in a single monorepo, and allows us - for example to add new feature to common distributions used
+by multiple providers and test them all together before releasing new versions of either of those pacckages
+
+Managing dependencies in ``pyproject.toml`` files
+.................................................
+
+Each of the ``pyproject.toml`` files in Apache Airflow repository defines dependencies in one of the
+following sections:
+
+* ``[project.dependencies]`` - this section defines the required dependencies of the package. These
+ dependencies are installed when you install the package without any extras.
+* ``[project.optional-dependencies]`` - this section defines optional dependencies (extras) of the package.
+ These dependencies are installed when you install the package with extras - for example
+ ``pip install apache-airflow[ssh]`` will install the ``ssh`` extra dependencies defined in this section.
+* ``[dependency-group.dev]`` - this section defines development dependencies of the package.
+ These dependencies are installed when you run ``uv sync`` by default. when ``uv`` syncs sources with
+ local pyproject.toml it adds ``dev`` dependency group and package is installed in editable mode with
+ development dependencies.
+
+
+Adding and modifying dependencies
+.................................
+
+Adding and modifying dependencies in Apache Airflow is done by modifying the appropriate
+``pyproject.toml`` file in the appropriate distribution.
+
+When you add a new dependency, you should make sure that:
+
+* The dependency is added to the right section (main dependencies, optional dependencies, or
+ development dependencies)
+
+* Some parts of those dependencies might be automatically generated (and overwritten) by our ``prek``
+ hooks. Those are the necessary dependencies that ``prek`` hoks can figure out automatically by
+ analyzing the imports in the sources and structure of the project. We also have special case of
+ shared dependencies (described in `shared dependencies document <../shared/README.md>`__) where we
+ effectively "static-link" some libraries into multiple distributions, to avoid unnecessary coupling and
+ circular dependencies, and those dependencies contribute some dependencies automatically as well.
+ Pay attention to comments such us example start and end of such generated block below.
+ All the dependencies between those comments are automatically generated and you should not modify
+ them manually. Tou might instead modify the root source of those dependencies - depending on the automation,
+ you can usually also add extra dependencies manually outside of those comment blocks
+
+ .. code-block:: python
+
+ # Automatically generated airflow optional dependencies (update_airflow_pyproject_toml.py)
+ # ....
+ # LOTS OF GENERATED DEPENDENCIES HERE
+ # ....
+ # End of automatically generated airflow optional dependencies
+
+* The version specifier is as open as possible (upper) while still allowing the package to install
+ and pass all tests. We very rarely upper-bind dependencies - only when there is a known
+ conflict with a new or upcoming version of a dependency that breaks the installation or tests
+ (and we always make a comment why we are upper-bounding a dependency).
+
+* Make sure to lower-bind any dependency you add. Usually we lower-bind dependencies to the
+ minimum version that is required for the package to work but in order to simplify the work of
+ resolvers such as ``pip``, we often lower-bind to higher (and newer) version than the absolute minimum
+ especially when the minimum version is very old. This is for example good practice in ``boto`` and related
+ packages, where new version of those packages are released frequently (almost daily) and there are many
+ versions that need to be considered by the resolver if the version is not new enough.
+
+* Make sure to run ``uv sync`` after modifying dependencies to make sure that there are no
+ conflicts between dependencies of different packages in the workspace. You can run it in multiple
+ ways - either from the root of the repository (which will sync all packages) or from the package
+ you modified (which will sync only that package and its dependencies). Also good idea might be to
+ run ``uv sync --all-packages --all-extras`` at the root of the repository to make sure that
+ all packages with all extras can be installed together without conflicts, but this might be sometimes
+ difficult and slow as some of the extras require some additional system level dependencies to be installed
+ (for example ``mysql`` or ``postgres`` extras require client libraries to be installed on the system).
+
+* Make sure to run all tests after modifying dependencies to make sure that nothing is broken
+
+
+Referring to other Apache Airflow distributions in dependencies
+...............................................................
+
+With having more than 100 distributions in the repository, it is often necessary to refer to
+other distributions in the dependencies in order to use some common features or simply to use the
+features that the other distribution provides. There are two ways of doing it:
+
+* Regular package linking with ``apache-airflow-*`` dependency
+* Airflow "shared dependencies" mechanism - which is a bit of a custom hack for Airflow monorepo
+ that allows us to "static-link" some common dependencies into multiple distributions without
+ creating circular dependencies.
+
+We are not going to describe the shared dependencies mechanism here, please refer to the
+`shared dependencies document <../shared/README.md>`__ for details, but there are certain rules
+when it comes to referring to other Airflow distributions in dependencies - here are the important
+rules to remember:
+
+* You can refer to other distributions in your dependencies - as usual using distribution name. For example,
+ if you are adding a dependency to ``apache-airflow-providers-common-compat`` package from
+ ``apache-airflow-providers-google``, you can just add ``apache-airflow-providers-common>=x.y.z`` to the
+ dependencies and when you run ``uv sync``, the local version of the package will be used automatically
+ (this is thanks to the workspace feature of ``uv`` that does great job of binding our monorepo together).
+ This is very useful when you are developing changes to multiple packages at the same time. Some of those
+ are added automatically by prek hooks - when it can detect such dependencies by analyzing imports in the
+ sources - then they are added automatically between the special comments mentioned above, but sometimes
+ (especially when such dependencies are not at the top-level imports) you might need to add them manually.
+
+* Never update distribution versions on your own. It is **entirely** up to the Release Manager to bump the
+ version of distributions that are defined as ``project.version``. This goes almost without an exception
+ and any diversions from this rule should be discussed at ``#release-management`` channel in Airflow Slack
+ beforehand. The only exception to this rule is when you are adding a new distribution to the repository -
+ in that case you should set the initial version of the distribution - usually ``0.0.1``, ``0.1.0``
+ or ``1.0.0`` depending on the maturity of the package. But still you should discuss it in the channel.
+
+* Sometimes, when you add a new feature to a common distribution, you might add a feature to it or
+ change the API in the way that other packages can use it. This is especially true for common packages such as
+ ``apache-airflow-providers-common-compat``, but can happen for other packages (for example
+ ``apache-airflow-providers-apache-beam`` is used by ``apache-airflow-providers-google`` to use ``Apache Beam``
+ hooks to communicate with Google Dataflow). In such case, when you are adding a feature to a common package
+ remember that the feature you just add. will only be released in the **FUTURE** release of such common
Review Comment:
```suggestion
remember that the feature you just add will only be released in the **FUTURE** release of such common
```
##########
contributing-docs/13_airflow_dependencies_and_extras.rst:
##########
@@ -18,6 +18,226 @@
Airflow dependencies
====================
+This document describes how we manage Apache Airflow dependencies, as well as how we make sure
+that users can use Airflow both as an application and as a library when they are deploying
+their own Airflow - using our constraints mechanism.
+
+Airflow ``pyproject.toml`` files and ``uv`` workspace
+.....................................................
+
+Managing dependencies is an important part of maintaining Apache Airflow, we have more than 700
+dependencies, and often when you add a new feature that requires a new dependency, you should
+also update the dependency list. This also happens when you want to add a new tool that is used
+for development, testing, or building the documentation and this document describes how to do it.
+
+When it comes to defining dependencies for any of the Apache Airflow distributions, we are following the
+standard ``pyproject.toml`` format that has been defined in `PEP 518 <https://peps.python.org/pep-0518/>`_
+and `PEP 621 <https://peps.python.org/pep-0621/>`_, we are also using dependency groups defined in
+`PEP 735 <https://peps.python.org/pep-0735/>`_ - particularly ``dev`` dependency group which we use in
+all our ``pyproject.toml`` files to define development dependencies.
+
+We have more than 100 python distributions in Apache Airflow repository - including the main
+``apache-airflow`` package, ``apache-airflow-providers-*`` packages, ``apache-airflow-core`` package,
+``apache-airflow-task-sdk`` package, ``apache-airflow-ctl`` package, and several other packages.
+
+They are all connected together via ``uv`` workspace feature (workspace is defined in the root ``pyproject.toml``
+file of the repository in the ``apache-airflow`` distribution definition. The workspace feature allows us to
+run ``uv sync`` at the top of the repository to install all packages in editable mode in the development
+environment from all the 100+ distributions (!) and resolve the dependencies together, so that we know that
+the dependencies have no conflicting requirements. Also the distributions are referring to each other via
+name - which means that when you run locally ``uv sync``, the local version of the packages are used, not the
+ones released on PyPI, which means that you can develop and test changes that span multiple packages at the
+same time. This is a very powerful feature that allows us to maintain the complex ecosystem of Apache Airflow
+distributions in a single monorepo, and allows us - for example to add new feature to common distributions used
+by multiple providers and test them all together before releasing new versions of either of those pacckages
+
+Managing dependencies in ``pyproject.toml`` files
+.................................................
+
+Each of the ``pyproject.toml`` files in Apache Airflow repository defines dependencies in one of the
+following sections:
+
+* ``[project.dependencies]`` - this section defines the required dependencies of the package. These
+ dependencies are installed when you install the package without any extras.
+* ``[project.optional-dependencies]`` - this section defines optional dependencies (extras) of the package.
+ These dependencies are installed when you install the package with extras - for example
+ ``pip install apache-airflow[ssh]`` will install the ``ssh`` extra dependencies defined in this section.
+* ``[dependency-group.dev]`` - this section defines development dependencies of the package.
+ These dependencies are installed when you run ``uv sync`` by default. when ``uv`` syncs sources with
+ local pyproject.toml it adds ``dev`` dependency group and package is installed in editable mode with
+ development dependencies.
+
+
+Adding and modifying dependencies
+.................................
+
+Adding and modifying dependencies in Apache Airflow is done by modifying the appropriate
+``pyproject.toml`` file in the appropriate distribution.
+
+When you add a new dependency, you should make sure that:
+
+* The dependency is added to the right section (main dependencies, optional dependencies, or
+ development dependencies)
+
+* Some parts of those dependencies might be automatically generated (and overwritten) by our ``prek``
+ hooks. Those are the necessary dependencies that ``prek`` hoks can figure out automatically by
+ analyzing the imports in the sources and structure of the project. We also have special case of
+ shared dependencies (described in `shared dependencies document <../shared/README.md>`__) where we
+ effectively "static-link" some libraries into multiple distributions, to avoid unnecessary coupling and
+ circular dependencies, and those dependencies contribute some dependencies automatically as well.
+ Pay attention to comments such us example start and end of such generated block below.
+ All the dependencies between those comments are automatically generated and you should not modify
+ them manually. Tou might instead modify the root source of those dependencies - depending on the automation,
+ you can usually also add extra dependencies manually outside of those comment blocks
+
+ .. code-block:: python
+
+ # Automatically generated airflow optional dependencies (update_airflow_pyproject_toml.py)
+ # ....
+ # LOTS OF GENERATED DEPENDENCIES HERE
+ # ....
+ # End of automatically generated airflow optional dependencies
+
+* The version specifier is as open as possible (upper) while still allowing the package to install
+ and pass all tests. We very rarely upper-bind dependencies - only when there is a known
+ conflict with a new or upcoming version of a dependency that breaks the installation or tests
+ (and we always make a comment why we are upper-bounding a dependency).
+
+* Make sure to lower-bind any dependency you add. Usually we lower-bind dependencies to the
+ minimum version that is required for the package to work but in order to simplify the work of
+ resolvers such as ``pip``, we often lower-bind to higher (and newer) version than the absolute minimum
+ especially when the minimum version is very old. This is for example good practice in ``boto`` and related
+ packages, where new version of those packages are released frequently (almost daily) and there are many
+ versions that need to be considered by the resolver if the version is not new enough.
+
+* Make sure to run ``uv sync`` after modifying dependencies to make sure that there are no
+ conflicts between dependencies of different packages in the workspace. You can run it in multiple
+ ways - either from the root of the repository (which will sync all packages) or from the package
+ you modified (which will sync only that package and its dependencies). Also good idea might be to
+ run ``uv sync --all-packages --all-extras`` at the root of the repository to make sure that
+ all packages with all extras can be installed together without conflicts, but this might be sometimes
+ difficult and slow as some of the extras require some additional system level dependencies to be installed
+ (for example ``mysql`` or ``postgres`` extras require client libraries to be installed on the system).
+
+* Make sure to run all tests after modifying dependencies to make sure that nothing is broken
+
+
+Referring to other Apache Airflow distributions in dependencies
+...............................................................
+
+With having more than 100 distributions in the repository, it is often necessary to refer to
+other distributions in the dependencies in order to use some common features or simply to use the
+features that the other distribution provides. There are two ways of doing it:
+
+* Regular package linking with ``apache-airflow-*`` dependency
+* Airflow "shared dependencies" mechanism - which is a bit of a custom hack for Airflow monorepo
+ that allows us to "static-link" some common dependencies into multiple distributions without
+ creating circular dependencies.
+
+We are not going to describe the shared dependencies mechanism here, please refer to the
+`shared dependencies document <../shared/README.md>`__ for details, but there are certain rules
+when it comes to referring to other Airflow distributions in dependencies - here are the important
+rules to remember:
+
+* You can refer to other distributions in your dependencies - as usual using distribution name. For example,
+ if you are adding a dependency to ``apache-airflow-providers-common-compat`` package from
+ ``apache-airflow-providers-google``, you can just add ``apache-airflow-providers-common>=x.y.z`` to the
+ dependencies and when you run ``uv sync``, the local version of the package will be used automatically
+ (this is thanks to the workspace feature of ``uv`` that does great job of binding our monorepo together).
+ This is very useful when you are developing changes to multiple packages at the same time. Some of those
+ are added automatically by prek hooks - when it can detect such dependencies by analyzing imports in the
+ sources - then they are added automatically between the special comments mentioned above, but sometimes
+ (especially when such dependencies are not at the top-level imports) you might need to add them manually.
+
+* Never update distribution versions on your own. It is **entirely** up to the Release Manager to bump the
+ version of distributions that are defined as ``project.version``. This goes almost without an exception
+ and any diversions from this rule should be discussed at ``#release-management`` channel in Airflow Slack
+ beforehand. The only exception to this rule is when you are adding a new distribution to the repository -
+ in that case you should set the initial version of the distribution - usually ``0.0.1``, ``0.1.0``
+ or ``1.0.0`` depending on the maturity of the package. But still you should discuss it in the channel.
+
Review Comment:
This feels out of place in a document talking about dependencies.
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Manages cross-distribution dependencies better [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #58430:
URL: https://github.com/apache/airflow/pull/58430#issuecomment-3547943565
Hey @amoghrajesh @xBis7 @eladkal @bugraoz93 @ashb -> You have recently been involved in modifying such cross-distribution dependencies - including shared distributions. I think we missed a bit comprehensive documentation on how our dependencies (especially cross-airflow-distribution) dependencies are managed in Airflow and it caused a number of back-forth. So I decided - in this one - not only to add protection against accidentally modifying the min-versions for cros-airflow-distributions by contributors, but also - in orded to properly explain it - to add more complete explanation on how those dependencies are managed - with all the nuances and caveats.
This is close to main brain-dump on all the work on dependencies and providers, done over the last year or so, down to the level that I think might be useful to explain some basic assumptions and how it works - including explaining the parts that are automated, those that shoudl be done manually and some more specific edge cases and reasoning for them.
I - of course - do not expect every contributor to read them, but this might be a good reference to send someone to when they are asking how things work.
Have fun revieing, any comments on how to make it better are most welcome.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org