[airflow] branch main updated: Clarify that versioned constraints are fixed at release time (#28762)

[po...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
     new 8290ade26d Clarify that versioned constraints are fixed at release time (#28762)
8290ade26d is described below

commit 8290ade26deba02ca6cf3d8254981b31cf89ee5b
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Fri Jan 6 10:29:28 2023 +0100

    Clarify that versioned constraints are fixed at release time (#28762)
    
    We received a number of requests to upgrade individual dependencies in
    the constraint files (mostly due to those dependencies releasing version
    with vulnerabilities fixed). This is not how our constraint works, their
    main purpose is to provide "consistent installation" mechanism for
    anyone who installs airflow from the scratch, we are not going to keep
    such relased versions up-to-date with versions of dependencies released
    after the release.
    
    This PR provides additional explanation about that in both constraint
    files as well as in reference container images which follow similar
    patterns.
---
 .../installation/installing-from-pypi.rst          | 34 ++++++++++++++++++++-
 docs/docker-stack/index.rst                        | 35 ++++++++++++++++++++++
 2 files changed, 68 insertions(+), 1 deletion(-)

diff --git a/docs/apache-airflow/installation/installing-from-pypi.rst b/docs/apache-airflow/installation/installing-from-pypi.rst
index 3ec492b0ed..e6cf162bc3 100644
--- a/docs/apache-airflow/installation/installing-from-pypi.rst
+++ b/docs/apache-airflow/installation/installing-from-pypi.rst
@@ -53,7 +53,7 @@ and both at the same time. We decided to keep our dependencies as open as possib
 version of libraries if needed. This means that from time to time plain ``pip install apache-airflow`` will
 not work or will produce an unusable Airflow installation.
 
-In order to have a repeatable installation, we also keep a set of "known-to-be-working" constraint files in the
+In order to have a repeatable installation (and only for that reason), we also keep a set of "known-to-be-working" constraint files in the
 ``constraints-main``, ``constraints-2-0``, ``constraints-2-1`` etc. orphan branches and then we create a tag
 for each released version e.g. :subst-code:`constraints-|version|`. This way, we keep a tested and working set of dependencies.
 
@@ -88,6 +88,38 @@ constraints always points to the "latest" released Airflow version constraints:
 
   https://raw.githubusercontent.com/apache/airflow/constraints-latest/constraints-3.7.txt
 
+
+Fixing Constraint files at release time
+'''''''''''''''''''''''''''''''''''''''
+
+The released "versioned" constraints are mostly ``fixed`` when we release Airflow version and we only
+update them in exceptional circumstances. For example when we find out that the released constraints might prevent
+Airflow from being installed consistently from the scratch. In normal circumstances, the constraint files
+are not going to change if new version of Airflow dependencies are released - not even when those
+versions contain critical security fixes. The process of Airflow releases is designed around upgrading
+dependencies automatically where applicable but only when we release a new version of Airflow,
+not for already released versions.
+
+If you want to make sure that Airflow dependencies are upgraded to the latest released versions containing
+latest security fixes, you should implement your own process to upgrade those yourself when
+you detect the need for that. Airflow usually does not upper-bound versions of its dependencies via
+requirements, so you should be able to upgrade them to the latest versions - usually without any problems.
+
+Obviously - since we have no control over what gets released in new versions of the dependencies, we
+cannot give any guarantees that tests and functionality of those dependencies will be compatible with
+Airflow after you upgrade them - testing if Airflow still works with those is in your hands,
+and in case of any problems, you should raise issue with the authors of the dependencies that are problematic.
+You can also - in such cases - look at the `Airflow issues <https://github.com/apache/airflow/issues>`_
+`Airflow Pull Requests <https://github.com/apache/airflow/pulls>`_ and
+`Airflow Discussions <https://github.com/apache/airflow/discussions>`_, searching for similar
+problems to see if there are any fixes or workarounds found in the ``main`` version of Airflow and apply them
+to your deployment.
+
+The easiest way to keep-up with the latest released dependencies is however, to upgrade to the latest released
+Airflow version. Whenever we release a new version of Airflow, we upgrade all dependencies to the latest
+applicable versions and test them together, so if you want to keep up with those tests - staying up-to-date
+with latest version of Airflow is the easiest way to update those dependencies.
+
 Installation and upgrade scenarios
 ''''''''''''''''''''''''''''''''''
 
diff --git a/docs/docker-stack/index.rst b/docs/docker-stack/index.rst
index be8877be06..d54c374b70 100644
--- a/docs/docker-stack/index.rst
+++ b/docs/docker-stack/index.rst
@@ -83,6 +83,41 @@ are also images published from branches but they are used mainly for development
 See `Airflow Git Branching <https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#airflow-git-branches>`_
 for details.
 
+Fixing images at release time
+=============================
+
+The released "versioned" reference images are mostly ``fixed`` when we release Airflow version and we only
+update them in exceptional circumstances. For example when we find out that there are dependency errors
+that might prevent important Airflow or embedded provider's functionalities working. In normal circumstances,
+the images are not going to change after release, even if new version of Airflow dependencies are released -
+not even when those versions contain critical security fixes. The process of Airflow releases is designed
+around upgrading dependencies automatically where applicable but only when we release a new version of Airflow,
+not for already released versions.
+
+If you want to make sure that Airflow dependencies are upgraded to the latest released versions containing
+latest security fixes in the image you use, you should implement your own process to upgrade
+those yourself when you build custom image based on the Airflow reference one. Airflow usually does not
+upper-bound versions of its dependencies via requirements, so you should be able to upgrade them to the
+latest versions - usually without any problems. And you can follow the process described in
+:ref:`Building the image <build:build_image>` to do it (even in automated way).
+
+Obviously - since we have no control over what gets released in new versions of the dependencies, we
+cannot give any guarantees that tests and functionality of those dependencies will be compatible with
+Airflow after you upgrade them - testing if Airflow still works with those is in your hands,
+and in case of any problems, you should raise issue with the authors of the dependencies that are problematic.
+You can also - in such cases - look at the `Airflow issues <https://github.com/apache/airflow/issues>`_
+`Airflow Pull Requests <https://github.com/apache/airflow/pulls>`_ and
+`Airflow Discussions <https://github.com/apache/airflow/discussions>`_, searching for similar
+problems to see if there are any fixes or workarounds found in the ``main`` version of Airflow and apply them
+to your custom image.
+
+The easiest way to keep-up with the latest released dependencies is however, to upgrade to the latest released
+Airflow version via switching to newly released images as base for your images, when a new version of
+Airflow is released. Whenever we release a new version of Airflow, we upgrade all dependencies to the latest
+applicable versions and test them together, so if you want to keep up with those tests - staying up-to-date
+with latest version of Airflow is the easiest way to update those dependencies.
+
+
 Support
 =======