You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2023/06/13 22:31:08 UTC

[GitHub] [airflow] potiuk opened a new pull request, #31882: Update documentation for constraints installation

potiuk opened a new pull request, #31882:
URL: https://github.com/apache/airflow/pull/31882

   Airlow constraints and custom depenencies are a mystery for our users. The reason why we are using constraints is not clear and user are confused when they should and when they shoudl not use constraints. Recent discussions with users and "Mastering Custom Dependencies" presentation in London has proven that we need to do better and simplify the documentation while giving more, better examples. That includes trimming down of boilerplate description in the constraint information, explaining a little more on what reproducibke (not repeatable!) build is, adding expectation that users will not use constraints for all pip commands and explaining how to produce your own constraints if you need a reproducible, custom build.
   
   This PR aims to fulfill those goals.
   
   <!--
   Thank you for contributing! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   In case of an existing issue, reference it using one of the following:
   
   closes: #ISSUE
   related: #ISSUE
   
   How to write a good git commit message:
   http://chris.beams.io/posts/git-commit/
   -->
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on a diff in pull request #31882: Update documentation for constraints installation

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #31882:
URL: https://github.com/apache/airflow/pull/31882#discussion_r1228822923


##########
docs/apache-airflow/installation/installing-from-pypi.rst:
##########
@@ -79,53 +87,64 @@ where:
 - ``AIRFLOW_VERSION`` - Airflow version (e.g. :subst-code:`|version|`) or ``main``, ``2-0``, for latest development version
 - ``PYTHON_VERSION`` Python version e.g. ``3.8``, ``3.9``
 
-There is also a ``constraints-no-providers`` constraint file, which contains just constraints required to
-install Airflow core. This allows to install and upgrade airflow separately and independently from providers.
+However, that should not prevent you from being able to install providers and dependencies that were upgraded
+after the release to keep your installation up-to-dated, even if you do not upgrade airflow core version.
+Installing such dependencies should be done without constraints as a separate pip command.

Review Comment:
   Yes. I also rephrased it a little.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk merged pull request #31882: Update documentation for constraints installation

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #31882:
URL: https://github.com/apache/airflow/pull/31882


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] o-nikolas commented on a diff in pull request #31882: Update documentation for constraints installation

Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on code in PR #31882:
URL: https://github.com/apache/airflow/pull/31882#discussion_r1228802464


##########
docs/apache-airflow/installation/installing-from-pypi.rst:
##########
@@ -53,20 +53,28 @@ This is an example, see further for more explanation.
 Constraints files
 '''''''''''''''''
 
-Airflow installation can be tricky sometimes because Airflow is both a library and an application.
+Why do we need constraints

Review Comment:
   The below suggestion is more of a statement so it doesn't need a `?` at the end.
   ```suggestion
   Why we need constraints
   ```



##########
docs/apache-airflow/installation/installing-from-pypi.rst:
##########
@@ -79,53 +87,64 @@ where:
 - ``AIRFLOW_VERSION`` - Airflow version (e.g. :subst-code:`|version|`) or ``main``, ``2-0``, for latest development version
 - ``PYTHON_VERSION`` Python version e.g. ``3.8``, ``3.9``
 
-There is also a ``constraints-no-providers`` constraint file, which contains just constraints required to
-install Airflow core. This allows to install and upgrade airflow separately and independently from providers.
+However, that should not prevent you from being able to install providers and dependencies that were upgraded
+after the release to keep your installation up-to-dated, even if you do not upgrade airflow core version.
+Installing such dependencies should be done without constraints as a separate pip command.
 
-You can create the URL to the file substituting the variables in the template below.
+You should make sure to also add the ``apache-airflow`` package to the list of packages to install and pin
+it to the version that you have, otherwise you might end up with a different version of Airflow than you
+expect as ``pip`` can upgrade/downgrade it automatically when performing dependency resolution.
 
-.. code-block::
+.. code-block:: bash
 
-  https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-no-providers-${PYTHON_VERSION}.txt
+    pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
+    pip install "apache-airflow[celery]==|version|" apache-airflow-providers-google==10.1.1
 
-You can also use "latest" as version when you install "latest" stable version of Airflow. The "latest"
-constraints always points to the "latest" released Airflow version constraints:
+You can also downgrade providers or other dependencies this way if you need to (for example if you do not
+want to upgrade to the latest version of a provider).
 
-.. code-block::
+.. code-block:: bash
+
+    pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
+    pip install "apache-airflow[celery]==|version|" dbt-core==0.20.0
+
+
+Not always such upgrade is possible - you might have conflicting dependencies, but by not using constraints,
+you give ``pip`` a chance to resolve the conflicts for you, while keeping within the limits of dependencies
+that Apache Airflow and other providers require. The combination of those dependencies might not be as well
+tested as the "golden" set of dependencies, but it should work in most cases. You can also always run
+the ``pip check`` command to test if the set of your Python packages is consistent and not conflicting.
+
+
+Using your own constraints

Review Comment:
   Love this section, it should help lots of folks :+1: 



##########
docs/apache-airflow/installation/installing-from-pypi.rst:
##########
@@ -53,20 +53,28 @@ This is an example, see further for more explanation.
 Constraints files
 '''''''''''''''''
 
-Airflow installation can be tricky sometimes because Airflow is both a library and an application.
+Why do we need constraints
+==========================
+
+Airflow installation can be tricky because Airflow is both a library and an application.
+
 Libraries usually keep their dependencies open and applications usually pin them, but we should do neither
 and both at the same time. We decided to keep our dependencies as open as possible
 (in ``setup.cfg`` and ``setup.py``) so users can install different
 version of libraries if needed. This means that from time to time plain ``pip install apache-airflow`` will
 not work or will produce an unusable Airflow installation.
 
-In order to have a repeatable installation (and only for that reason), we also keep a set of "known-to-be-working" constraint files in the
+Reproducible Airflow installation
+=================================
+
+In order to have a reproducible installation, we also keep a set of constraint files in the
 ``constraints-main``, ``constraints-2-0``, ``constraints-2-1`` etc. orphan branches and then we create a tag
-for each released version e.g. :subst-code:`constraints-|version|`. This way, we keep a tested and working set of dependencies.
+for each released version e.g. :subst-code:`constraints-|version|`.
 
-Those "known-to-be-working" constraints are per major/minor Python version. You can use them as constraint
-files when installing Airflow from PyPI. Note that you have to specify the correct Airflow
-and Python versions in the URL.
+This way, we keep a tested set of dependencies at the moment of release. This provides you with the ability
+of having the exact same installation of airflow + providers + dependencies as was known to be working
+at the moment of release - "golden" set of dependencies for that version of Airflow. There is a separate

Review Comment:
   I'd drop the golden piece. It makes it sound as overly desirable and useful in all cases, which may lead to some of the confusion we're trying to avoid.
   ```suggestion
   This way, we keep a tested set of dependencies at the moment of release. This provides you with the ability
   of having the exact same installation of airflow + providers + dependencies as was known to be working
   at the moment of release. There is a separate
   ```



##########
docs/apache-airflow/installation/installing-from-pypi.rst:
##########
@@ -79,53 +87,64 @@ where:
 - ``AIRFLOW_VERSION`` - Airflow version (e.g. :subst-code:`|version|`) or ``main``, ``2-0``, for latest development version
 - ``PYTHON_VERSION`` Python version e.g. ``3.8``, ``3.9``
 
-There is also a ``constraints-no-providers`` constraint file, which contains just constraints required to
-install Airflow core. This allows to install and upgrade airflow separately and independently from providers.
+However, that should not prevent you from being able to install providers and dependencies that were upgraded
+after the release to keep your installation up-to-dated, even if you do not upgrade airflow core version.
+Installing such dependencies should be done without constraints as a separate pip command.

Review Comment:
   What do you think about bolding this line to really make it stand out and hit the point home?



##########
docs/apache-airflow/installation/installing-from-pypi.rst:
##########
@@ -79,53 +87,64 @@ where:
 - ``AIRFLOW_VERSION`` - Airflow version (e.g. :subst-code:`|version|`) or ``main``, ``2-0``, for latest development version
 - ``PYTHON_VERSION`` Python version e.g. ``3.8``, ``3.9``
 
-There is also a ``constraints-no-providers`` constraint file, which contains just constraints required to
-install Airflow core. This allows to install and upgrade airflow separately and independently from providers.
+However, that should not prevent you from being able to install providers and dependencies that were upgraded
+after the release to keep your installation up-to-dated, even if you do not upgrade airflow core version.
+Installing such dependencies should be done without constraints as a separate pip command.
 
-You can create the URL to the file substituting the variables in the template below.
+You should make sure to also add the ``apache-airflow`` package to the list of packages to install and pin
+it to the version that you have, otherwise you might end up with a different version of Airflow than you
+expect as ``pip`` can upgrade/downgrade it automatically when performing dependency resolution.
 
-.. code-block::
+.. code-block:: bash
 
-  https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-no-providers-${PYTHON_VERSION}.txt
+    pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
+    pip install "apache-airflow[celery]==|version|" apache-airflow-providers-google==10.1.1
 
-You can also use "latest" as version when you install "latest" stable version of Airflow. The "latest"
-constraints always points to the "latest" released Airflow version constraints:
+You can also downgrade providers or other dependencies this way if you need to (for example if you do not
+want to upgrade to the latest version of a provider).
 
-.. code-block::
+.. code-block:: bash
+
+    pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
+    pip install "apache-airflow[celery]==|version|" dbt-core==0.20.0
+
+
+Not always such upgrade is possible - you might have conflicting dependencies, but by not using constraints,
+you give ``pip`` a chance to resolve the conflicts for you, while keeping within the limits of dependencies
+that Apache Airflow and other providers require. The combination of those dependencies might not be as well
+tested as the "golden" set of dependencies, but it should work in most cases. You can also always run
+the ``pip check`` command to test if the set of your Python packages is consistent and not conflicting.
+
+
+Using your own constraints
+==========================
+
+When you decide to install your own dependencies, or want to upgrade or downgrade providers, you might want
+to continue being able to run reproducible installation of Airflow and those dependencies. In order to do
+that you can produce your own constraints file and use it to install Airflow instead of the one provided
+by the community.
+
+.. code-block:: bash
+
+    pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
+    pip install "apache-airflow[celery]==|version|" dbt-core==0.20.0
+    pip freeze > my-constraints.txt
+
+
+Then you can use it to run reproducible installation of your environment in a single operation via local file:

Review Comment:
   ```suggestion
   Then you can use it to create reproducible installations of your environment in a single operation via a local constraints file:
   ```



##########
docs/apache-airflow/installation/installing-from-pypi.rst:
##########
@@ -79,53 +87,64 @@ where:
 - ``AIRFLOW_VERSION`` - Airflow version (e.g. :subst-code:`|version|`) or ``main``, ``2-0``, for latest development version
 - ``PYTHON_VERSION`` Python version e.g. ``3.8``, ``3.9``
 
-There is also a ``constraints-no-providers`` constraint file, which contains just constraints required to
-install Airflow core. This allows to install and upgrade airflow separately and independently from providers.
+However, that should not prevent you from being able to install providers and dependencies that were upgraded
+after the release to keep your installation up-to-dated, even if you do not upgrade airflow core version.
+Installing such dependencies should be done without constraints as a separate pip command.
 
-You can create the URL to the file substituting the variables in the template below.
+You should make sure to also add the ``apache-airflow`` package to the list of packages to install and pin
+it to the version that you have, otherwise you might end up with a different version of Airflow than you
+expect as ``pip`` can upgrade/downgrade it automatically when performing dependency resolution.
 
-.. code-block::
+.. code-block:: bash
 
-  https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-no-providers-${PYTHON_VERSION}.txt
+    pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
+    pip install "apache-airflow[celery]==|version|" apache-airflow-providers-google==10.1.1
 
-You can also use "latest" as version when you install "latest" stable version of Airflow. The "latest"
-constraints always points to the "latest" released Airflow version constraints:
+You can also downgrade providers or other dependencies this way if you need to (for example if you do not
+want to upgrade to the latest version of a provider).
 
-.. code-block::
+.. code-block:: bash
+
+    pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
+    pip install "apache-airflow[celery]==|version|" dbt-core==0.20.0
+
+
+Not always such upgrade is possible - you might have conflicting dependencies, but by not using constraints,
+you give ``pip`` a chance to resolve the conflicts for you, while keeping within the limits of dependencies
+that Apache Airflow and other providers require. The combination of those dependencies might not be as well
+tested as the "golden" set of dependencies, but it should work in most cases. You can also always run
+the ``pip check`` command to test if the set of your Python packages is consistent and not conflicting.
+
+
+Using your own constraints
+==========================
+
+When you decide to install your own dependencies, or want to upgrade or downgrade providers, you might want
+to continue being able to run reproducible installation of Airflow and those dependencies. In order to do
+that you can produce your own constraints file and use it to install Airflow instead of the one provided
+by the community.
+
+.. code-block:: bash
+
+    pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
+    pip install "apache-airflow[celery]==|version|" dbt-core==0.20.0
+    pip freeze > my-constraints.txt
+
+
+Then you can use it to run reproducible installation of your environment in a single operation via local file:
+
+.. code-block:: bash
+
+    pip install "apache-airflow[celery]==|version|" --constraint "my-constraints.txt"
+
+
+The constraint file might also be hosted via a webserver of your choice and made available for remote use:
+
+
+.. code-block:: bash
+
+    pip install "apache-airflow[celery]==|version|" --constraint "https://my-company.org/my-constraints.txt"
 
-  https://raw.githubusercontent.com/apache/airflow/constraints-latest/constraints-3.8.txt
-
-
-Fixing Constraint files at release time

Review Comment:
   I think this header and the first paragraph are worth keeping to set expectations about the constraints files.



##########
docs/apache-airflow/installation/installing-from-pypi.rst:
##########
@@ -79,53 +87,64 @@ where:
 - ``AIRFLOW_VERSION`` - Airflow version (e.g. :subst-code:`|version|`) or ``main``, ``2-0``, for latest development version
 - ``PYTHON_VERSION`` Python version e.g. ``3.8``, ``3.9``
 
-There is also a ``constraints-no-providers`` constraint file, which contains just constraints required to
-install Airflow core. This allows to install and upgrade airflow separately and independently from providers.
+However, that should not prevent you from being able to install providers and dependencies that were upgraded
+after the release to keep your installation up-to-dated, even if you do not upgrade airflow core version.
+Installing such dependencies should be done without constraints as a separate pip command.
 
-You can create the URL to the file substituting the variables in the template below.
+You should make sure to also add the ``apache-airflow`` package to the list of packages to install and pin
+it to the version that you have, otherwise you might end up with a different version of Airflow than you
+expect as ``pip`` can upgrade/downgrade it automatically when performing dependency resolution.
 
-.. code-block::
+.. code-block:: bash
 
-  https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-no-providers-${PYTHON_VERSION}.txt
+    pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
+    pip install "apache-airflow[celery]==|version|" apache-airflow-providers-google==10.1.1
 
-You can also use "latest" as version when you install "latest" stable version of Airflow. The "latest"
-constraints always points to the "latest" released Airflow version constraints:
+You can also downgrade providers or other dependencies this way if you need to (for example if you do not
+want to upgrade to the latest version of a provider).
 
-.. code-block::
+.. code-block:: bash
+
+    pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
+    pip install "apache-airflow[celery]==|version|" dbt-core==0.20.0
+
+
+Not always such upgrade is possible - you might have conflicting dependencies, but by not using constraints,

Review Comment:
   Such an upgrade is not always possible - you might have conflicting dependencies, but by not using constraints,
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] pankajkoti commented on a diff in pull request #31882: Update documentation for constraints installation

Posted by "pankajkoti (via GitHub)" <gi...@apache.org>.
pankajkoti commented on code in PR #31882:
URL: https://github.com/apache/airflow/pull/31882#discussion_r1229356103


##########
docs/apache-airflow/installation/installing-from-pypi.rst:
##########
@@ -40,33 +40,65 @@ Only ``pip`` installation is currently officially supported.
   the problem in `this PR <https://github.com/bazelbuild/rules_python/pull/1166>`_ so it might be that
   newer versions of ``bazel`` will handle it.
 
-Typical command to install airflow from PyPI looks like below:
+Typical command to install airflow from scratch in a reproducible way from PyPI looks like below:
 
-.. code-block::
+.. code-block:: bash
 
     pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
 
-This is an example, see further for more explanation.
+
+Typically, you can add other dependencies and providers as separate command after the reproducible
+installation - this way you can upgrade or downgrade the dependencies as you see fit, without limiting them to
+constraints. Good practice for those is to extend such ``pip install`` command with the ``apache-airflow``
+pinned to the version you have already installed to make sure it is not accidentally
+upgraded or downgraded by ``pip``.
+
+.. code-block:: bash
+
+    pip install "apache-airflow==|version|" apache-airflow-providers-google=10.1.0

Review Comment:
   ```suggestion
       pip install "apache-airflow==|version|" apache-airflow-providers-google==10.1.0
   ```



##########
docs/apache-airflow/installation/installing-from-pypi.rst:
##########
@@ -40,33 +40,65 @@ Only ``pip`` installation is currently officially supported.
   the problem in `this PR <https://github.com/bazelbuild/rules_python/pull/1166>`_ so it might be that
   newer versions of ``bazel`` will handle it.
 
-Typical command to install airflow from PyPI looks like below:
+Typical command to install airflow from scratch in a reproducible way from PyPI looks like below:
 
-.. code-block::
+.. code-block:: bash
 
     pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
 
-This is an example, see further for more explanation.
+
+Typically, you can add other dependencies and providers as separate command after the reproducible
+installation - this way you can upgrade or downgrade the dependencies as you see fit, without limiting them to
+constraints. Good practice for those is to extend such ``pip install`` command with the ``apache-airflow``
+pinned to the version you have already installed to make sure it is not accidentally
+upgraded or downgraded by ``pip``.
+
+.. code-block:: bash
+
+    pip install "apache-airflow==|version|" apache-airflow-providers-google=10.1.0
+
+
+Those are just examples, see further for more explanation why those are the best practices.
+
+.. note::
+
+   Generally speaking, Python community established practice is to perform application installation in a
+   virtualenv created with ``virtualenv`` or ``venv`` tools. You can also use ``pipx`` to install Airflow in a
+   application dedicated virtual environment created for you. There are also other tools that can be used
+   to manage your virtualenv installation and you are free to choose how you are managing the environments,

Review Comment:
   ```suggestion
      to manage your virtualenv installation and you are free to choose how you are managing the environments.
   ```



##########
docs/apache-airflow/installation/installing-from-pypi.rst:
##########
@@ -79,50 +111,124 @@ where:
 - ``AIRFLOW_VERSION`` - Airflow version (e.g. :subst-code:`|version|`) or ``main``, ``2-0``, for latest development version
 - ``PYTHON_VERSION`` Python version e.g. ``3.8``, ``3.9``
 
-There is also a ``constraints-no-providers`` constraint file, which contains just constraints required to
-install Airflow core. This allows to install and upgrade airflow separately and independently from providers.
+The examples below assume that you want to use install airflow in a reproducible way with the ``celery`` extra,
+but you can pick your own set of extras and providers to install.
 
-You can create the URL to the file substituting the variables in the template below.
+.. code-block:: bash
 
-.. code-block::
+    pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
 
-  https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-no-providers-${PYTHON_VERSION}.txt
 
-You can also use "latest" as version when you install "latest" stable version of Airflow. The "latest"
-constraints always points to the "latest" released Airflow version constraints:
+.. note::
 
-.. code-block::
+    The reproducible installation guarantees that this initial installation steps will always work for you -
+    providing that you use the right Python version and that you have appropriate Operating System dependencies
+    installed for the providers to be installed. Some of the providers require additional OS dependencies to
+    be installed such us ``build-essential`` in order to compile libraries, or for example database client
+    libraries in case you install a database provider, etc.. You need to figure out which system dependencies
+    you need when your installation fails and install them before retrying the installation.
+
+Upgrading and installing dependencies (including providers)
+===========================================================
+
+**The reproducible installation above should not prevent you from being able to upgrade or downgrade
+providers and other dependencies to other versions**
 
-  https://raw.githubusercontent.com/apache/airflow/constraints-latest/constraints-3.8.txt
+You can, for example, install new versions of providers and dependencies after the release to use the latest
+version and up-to-date with latest security fixes - even if you do not want upgrade airflow core version.
+Or you can downgrade some dependencies or providers if you want to keep previous versions for compatibility
+reasons. Installing such dependencies should be done without constraints as a separate pip command.
 
+When you do such an upgrade, you should make sure to also add the ``apache-airflow`` package to the list of
+packages to install and pin it to the version that you have, otherwise you might end up with a
+different version of Airflow than you expect as ``pip`` can upgrade/downgrade it automatically when

Review Comment:
   ```suggestion
   different version of Airflow than you expect because ``pip`` can upgrade/downgrade it automatically when
   ```



##########
docs/apache-airflow/installation/installing-from-pypi.rst:
##########
@@ -79,50 +111,124 @@ where:
 - ``AIRFLOW_VERSION`` - Airflow version (e.g. :subst-code:`|version|`) or ``main``, ``2-0``, for latest development version
 - ``PYTHON_VERSION`` Python version e.g. ``3.8``, ``3.9``
 
-There is also a ``constraints-no-providers`` constraint file, which contains just constraints required to
-install Airflow core. This allows to install and upgrade airflow separately and independently from providers.
+The examples below assume that you want to use install airflow in a reproducible way with the ``celery`` extra,
+but you can pick your own set of extras and providers to install.
 
-You can create the URL to the file substituting the variables in the template below.
+.. code-block:: bash
 
-.. code-block::
+    pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
 
-  https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-no-providers-${PYTHON_VERSION}.txt
 
-You can also use "latest" as version when you install "latest" stable version of Airflow. The "latest"
-constraints always points to the "latest" released Airflow version constraints:
+.. note::
 
-.. code-block::
+    The reproducible installation guarantees that this initial installation steps will always work for you -
+    providing that you use the right Python version and that you have appropriate Operating System dependencies
+    installed for the providers to be installed. Some of the providers require additional OS dependencies to
+    be installed such us ``build-essential`` in order to compile libraries, or for example database client
+    libraries in case you install a database provider, etc.. You need to figure out which system dependencies
+    you need when your installation fails and install them before retrying the installation.
+
+Upgrading and installing dependencies (including providers)
+===========================================================
+
+**The reproducible installation above should not prevent you from being able to upgrade or downgrade
+providers and other dependencies to other versions**
 
-  https://raw.githubusercontent.com/apache/airflow/constraints-latest/constraints-3.8.txt
+You can, for example, install new versions of providers and dependencies after the release to use the latest
+version and up-to-date with latest security fixes - even if you do not want upgrade airflow core version.
+Or you can downgrade some dependencies or providers if you want to keep previous versions for compatibility
+reasons. Installing such dependencies should be done without constraints as a separate pip command.
 
+When you do such an upgrade, you should make sure to also add the ``apache-airflow`` package to the list of
+packages to install and pin it to the version that you have, otherwise you might end up with a
+different version of Airflow than you expect as ``pip`` can upgrade/downgrade it automatically when
+performing dependency resolution.
+
+
+.. code-block:: bash
+
+    pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
+    pip install "apache-airflow==|version|" apache-airflow-providers-google==10.1.1
 
-Fixing Constraint files at release time
-'''''''''''''''''''''''''''''''''''''''
+You can also downgrade or upgrade other dependencies this way - even if they are not compatible with
+those dependencies that are stored in the original constraints file:
+
+.. code-block:: bash
+
+    pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
+    pip install "apache-airflow[celery]==|version|" dbt-core==0.20.0
+
+.. warning::
+
+    Not all dependencies can be installed this way - you might have dependencies conflicting with basic
+    requirements of Airflow or other dependencies installed in your system. However, by skipping constraints
+    when you install or upgrade dependencies, you give ``pip`` a chance to resolve the conflicts for you,
+    while keeping dependencies within the limits that Apache Airflow, providers and other dependencies require.
+    The resulting combination of those dependencies might not be tested together before as well as the set
+    of dependencies that come with constraints, but it should work in most cases as we usually add
+    requirements, when Airflow depends on particular versions of some dependencies. In cases you cannot
+    install some dependencies in the same environment as Airflow - you can attempt to use other approaches.
+    See :ref:`best practices for handling conflicting/complex Python dependencies <best_practices/handling_conflicting_complex_python_dependencies>`
+
+
+Verifying installed dependencies
+================================
+
+You can also always run the ``pip check`` command to test if the set of your Python packages is
+consistent and not conflicting.
+
+
+.. code-block:: bash
+
+    > pip check
+    No broken requirements found.
+
+
+When you see such message and the exit code from ``pip check`` is 0, you can be sure, that there are no
+conflicting dependencies in your environment.
+
+
+Using your own constraints
+==========================
+
+When you decide to install your own dependencies, or want to upgrade or downgrade providers, you might want
+to continue being able to keep reproducible installation of Airflow and those dependencies via a single command. In order to do that, you can produce your own constraints file and use it
+to install Airflow instead of the one provided by the community.
+
+.. code-block:: bash
+
+    pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
+    pip install "apache-airflow==|version|" dbt-core==0.20.0
+    pip freeze > my-constraints.txt

Review Comment:
   I like this section a lot 👍🏽 



##########
docs/apache-airflow/installation/installing-from-pypi.rst:
##########
@@ -197,11 +296,25 @@ that conflicts with the version of apache-airflow that you are using:
 
     pip install "apache-airflow==|version|" "apache-airflow-providers-google==8.0.0"
 
+.. note::
+
+    Installing, upgrading, downgrading providers separately is not guaranteed to work with all
+    Airflow versions or other providers. Some providers have minimum-required version of Airflow and some
+    versions of providers might have conflicting requirements with Airflow or other dependencies you

Review Comment:
   could we please improvise this statement "Some versions of providers might have conflicting requirements with Airflow" a bit? 
   
   Do we mean some versions of Airflow here or we meant conflicting requirements with other Airflow providers?
   Otherwise, this might be questioned if it is an Airflow provider why or in which cases it would have conflicting requirements with Airflow?



##########
docs/apache-airflow/installation/installing-from-pypi.rst:
##########
@@ -79,50 +111,124 @@ where:
 - ``AIRFLOW_VERSION`` - Airflow version (e.g. :subst-code:`|version|`) or ``main``, ``2-0``, for latest development version
 - ``PYTHON_VERSION`` Python version e.g. ``3.8``, ``3.9``
 
-There is also a ``constraints-no-providers`` constraint file, which contains just constraints required to
-install Airflow core. This allows to install and upgrade airflow separately and independently from providers.
+The examples below assume that you want to use install airflow in a reproducible way with the ``celery`` extra,
+but you can pick your own set of extras and providers to install.
 
-You can create the URL to the file substituting the variables in the template below.
+.. code-block:: bash
 
-.. code-block::
+    pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
 
-  https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-no-providers-${PYTHON_VERSION}.txt
 
-You can also use "latest" as version when you install "latest" stable version of Airflow. The "latest"
-constraints always points to the "latest" released Airflow version constraints:
+.. note::
 
-.. code-block::
+    The reproducible installation guarantees that this initial installation steps will always work for you -
+    providing that you use the right Python version and that you have appropriate Operating System dependencies
+    installed for the providers to be installed. Some of the providers require additional OS dependencies to
+    be installed such us ``build-essential`` in order to compile libraries, or for example database client
+    libraries in case you install a database provider, etc.. You need to figure out which system dependencies
+    you need when your installation fails and install them before retrying the installation.
+
+Upgrading and installing dependencies (including providers)
+===========================================================
+
+**The reproducible installation above should not prevent you from being able to upgrade or downgrade
+providers and other dependencies to other versions**
 
-  https://raw.githubusercontent.com/apache/airflow/constraints-latest/constraints-3.8.txt
+You can, for example, install new versions of providers and dependencies after the release to use the latest
+version and up-to-date with latest security fixes - even if you do not want upgrade airflow core version.
+Or you can downgrade some dependencies or providers if you want to keep previous versions for compatibility
+reasons. Installing such dependencies should be done without constraints as a separate pip command.
 
+When you do such an upgrade, you should make sure to also add the ``apache-airflow`` package to the list of
+packages to install and pin it to the version that you have, otherwise you might end up with a
+different version of Airflow than you expect as ``pip`` can upgrade/downgrade it automatically when
+performing dependency resolution.
+
+
+.. code-block:: bash
+
+    pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
+    pip install "apache-airflow==|version|" apache-airflow-providers-google==10.1.1
 
-Fixing Constraint files at release time
-'''''''''''''''''''''''''''''''''''''''
+You can also downgrade or upgrade other dependencies this way - even if they are not compatible with
+those dependencies that are stored in the original constraints file:
+
+.. code-block:: bash
+
+    pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
+    pip install "apache-airflow[celery]==|version|" dbt-core==0.20.0
+
+.. warning::
+
+    Not all dependencies can be installed this way - you might have dependencies conflicting with basic
+    requirements of Airflow or other dependencies installed in your system. However, by skipping constraints
+    when you install or upgrade dependencies, you give ``pip`` a chance to resolve the conflicts for you,
+    while keeping dependencies within the limits that Apache Airflow, providers and other dependencies require.
+    The resulting combination of those dependencies might not be tested together before as well as the set
+    of dependencies that come with constraints, but it should work in most cases as we usually add

Review Comment:
   ```suggestion
       The resulting combination of those dependencies and the set of dependencies that come with the 
       constraints might not be tested before, but it should work in most cases as we usually add
   ```



##########
docs/apache-airflow/installation/installing-from-pypi.rst:
##########
@@ -79,50 +111,124 @@ where:
 - ``AIRFLOW_VERSION`` - Airflow version (e.g. :subst-code:`|version|`) or ``main``, ``2-0``, for latest development version
 - ``PYTHON_VERSION`` Python version e.g. ``3.8``, ``3.9``
 
-There is also a ``constraints-no-providers`` constraint file, which contains just constraints required to
-install Airflow core. This allows to install and upgrade airflow separately and independently from providers.
+The examples below assume that you want to use install airflow in a reproducible way with the ``celery`` extra,
+but you can pick your own set of extras and providers to install.
 
-You can create the URL to the file substituting the variables in the template below.
+.. code-block:: bash
 
-.. code-block::
+    pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
 
-  https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-no-providers-${PYTHON_VERSION}.txt
 
-You can also use "latest" as version when you install "latest" stable version of Airflow. The "latest"
-constraints always points to the "latest" released Airflow version constraints:
+.. note::
 
-.. code-block::
+    The reproducible installation guarantees that this initial installation steps will always work for you -
+    providing that you use the right Python version and that you have appropriate Operating System dependencies
+    installed for the providers to be installed. Some of the providers require additional OS dependencies to
+    be installed such us ``build-essential`` in order to compile libraries, or for example database client

Review Comment:
   ```suggestion
       be installed such as ``build-essential`` in order to compile libraries, or for example database client
   ```



##########
docs/apache-airflow/installation/installing-from-pypi.rst:
##########
@@ -79,50 +111,124 @@ where:
 - ``AIRFLOW_VERSION`` - Airflow version (e.g. :subst-code:`|version|`) or ``main``, ``2-0``, for latest development version
 - ``PYTHON_VERSION`` Python version e.g. ``3.8``, ``3.9``
 
-There is also a ``constraints-no-providers`` constraint file, which contains just constraints required to
-install Airflow core. This allows to install and upgrade airflow separately and independently from providers.
+The examples below assume that you want to use install airflow in a reproducible way with the ``celery`` extra,
+but you can pick your own set of extras and providers to install.
 
-You can create the URL to the file substituting the variables in the template below.
+.. code-block:: bash
 
-.. code-block::
+    pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
 
-  https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-no-providers-${PYTHON_VERSION}.txt
 
-You can also use "latest" as version when you install "latest" stable version of Airflow. The "latest"
-constraints always points to the "latest" released Airflow version constraints:
+.. note::
 
-.. code-block::
+    The reproducible installation guarantees that this initial installation steps will always work for you -
+    providing that you use the right Python version and that you have appropriate Operating System dependencies
+    installed for the providers to be installed. Some of the providers require additional OS dependencies to
+    be installed such us ``build-essential`` in order to compile libraries, or for example database client
+    libraries in case you install a database provider, etc.. You need to figure out which system dependencies
+    you need when your installation fails and install them before retrying the installation.
+
+Upgrading and installing dependencies (including providers)
+===========================================================
+
+**The reproducible installation above should not prevent you from being able to upgrade or downgrade
+providers and other dependencies to other versions**
 
-  https://raw.githubusercontent.com/apache/airflow/constraints-latest/constraints-3.8.txt
+You can, for example, install new versions of providers and dependencies after the release to use the latest
+version and up-to-date with latest security fixes - even if you do not want upgrade airflow core version.
+Or you can downgrade some dependencies or providers if you want to keep previous versions for compatibility
+reasons. Installing such dependencies should be done without constraints as a separate pip command.
 
+When you do such an upgrade, you should make sure to also add the ``apache-airflow`` package to the list of
+packages to install and pin it to the version that you have, otherwise you might end up with a
+different version of Airflow than you expect as ``pip`` can upgrade/downgrade it automatically when
+performing dependency resolution.
+
+
+.. code-block:: bash
+
+    pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
+    pip install "apache-airflow==|version|" apache-airflow-providers-google==10.1.1
 
-Fixing Constraint files at release time
-'''''''''''''''''''''''''''''''''''''''
+You can also downgrade or upgrade other dependencies this way - even if they are not compatible with
+those dependencies that are stored in the original constraints file:
+
+.. code-block:: bash
+
+    pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
+    pip install "apache-airflow[celery]==|version|" dbt-core==0.20.0
+
+.. warning::
+
+    Not all dependencies can be installed this way - you might have dependencies conflicting with basic
+    requirements of Airflow or other dependencies installed in your system. However, by skipping constraints
+    when you install or upgrade dependencies, you give ``pip`` a chance to resolve the conflicts for you,
+    while keeping dependencies within the limits that Apache Airflow, providers and other dependencies require.
+    The resulting combination of those dependencies might not be tested together before as well as the set
+    of dependencies that come with constraints, but it should work in most cases as we usually add
+    requirements, when Airflow depends on particular versions of some dependencies. In cases you cannot
+    install some dependencies in the same environment as Airflow - you can attempt to use other approaches.
+    See :ref:`best practices for handling conflicting/complex Python dependencies <best_practices/handling_conflicting_complex_python_dependencies>`
+
+
+Verifying installed dependencies
+================================
+
+You can also always run the ``pip check`` command to test if the set of your Python packages is
+consistent and not conflicting.
+
+
+.. code-block:: bash
+
+    > pip check
+    No broken requirements found.
+
+
+When you see such message and the exit code from ``pip check`` is 0, you can be sure, that there are no
+conflicting dependencies in your environment.
+
+
+Using your own constraints
+==========================
+
+When you decide to install your own dependencies, or want to upgrade or downgrade providers, you might want
+to continue being able to keep reproducible installation of Airflow and those dependencies via a single command. In order to do that, you can produce your own constraints file and use it
+to install Airflow instead of the one provided by the community.

Review Comment:
   ```suggestion
   to continue being able to keep reproducible installation of Airflow and those dependencies via a single command.
   In order to do that, you can produce your own constraints file and use it to install Airflow instead of the 
   one provided by the community.
   ```
   
   Just suggesting a new line char as I see this line is relatively longer than all other lines :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on a diff in pull request #31882: Update documentation for constraints installation

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #31882:
URL: https://github.com/apache/airflow/pull/31882#discussion_r1229111800


##########
docs/apache-airflow/installation/installing-from-pypi.rst:
##########
@@ -79,50 +87,86 @@ where:
 - ``AIRFLOW_VERSION`` - Airflow version (e.g. :subst-code:`|version|`) or ``main``, ``2-0``, for latest development version
 - ``PYTHON_VERSION`` Python version e.g. ``3.8``, ``3.9``
 
-There is also a ``constraints-no-providers`` constraint file, which contains just constraints required to
-install Airflow core. This allows to install and upgrade airflow separately and independently from providers.
+**However, that should not prevent you from being able to upgrade or downgrade providers and dependencies to other versions**
 
-You can create the URL to the file substituting the variables in the template below.
+You can, for example, install new versions of providers and dependencies after the release to use the latest
+version and up-to-date with latest security fixes - even if you do not want upgrade airflow core version.
+Or you can downgrade some dependencies or providers if you want to keep previous versions for compatibility
+reasons. Installing such dependencies should be done without constraints as a separate pip command.
 
-.. code-block::
+When you do such an upgrade, you should make sure to also add the ``apache-airflow`` package to the list of
+packages to install and pin it to the version that you have, otherwise you might end up with a
+different version of Airflow than you expect as ``pip`` can upgrade/downgrade it automatically when
+performing dependency resolution.
 
-  https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-no-providers-${PYTHON_VERSION}.txt
+.. code-block:: bash
 
-You can also use "latest" as version when you install "latest" stable version of Airflow. The "latest"
-constraints always points to the "latest" released Airflow version constraints:
+    pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
+    pip install "apache-airflow[celery]==|version|" apache-airflow-providers-google==10.1.1
 
-.. code-block::
+You can also downgrade or upgrade other dependencies this way - even if they are not compatible with
+those dependencies that are stored in the original constraints file:
+
+.. code-block:: bash
+
+    pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
+    pip install "apache-airflow[celery]==|version|" dbt-core==0.20.0
 
-  https://raw.githubusercontent.com/apache/airflow/constraints-latest/constraints-3.8.txt
+
+Such an upgrade is not always possible - you might have conflicting dependencies. However, by not using constraints,
+you give ``pip`` a chance to resolve the conflicts for you, while keeping within the limits of dependencies
+that Apache Airflow and other providers require. The combination of those dependencies might not be as well
+tested as the "golden" set of dependencies, but it should work in most cases. You can also always run
+the ``pip check`` command to test if the set of your Python packages is consistent and not conflicting.
+
+
+Using your own constraints
+==========================
+
+When you decide to install your own dependencies, or want to upgrade or downgrade providers, you might want
+to continue being able to keep reproducible installation of Airflow and those dependencies via single command
+line . In order to do that, you can produce your own constraints file and use it
+to install Airflow instead of the one provided by the community.
+
+.. code-block:: bash
+
+    pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
+    pip install "apache-airflow[celery]==|version|" dbt-core==0.20.0
+    pip freeze > my-constraints.txt
+
+
+Then you can use it to create reproducible installations of your environment in a single operation via
+a local constraints file:
+
+.. code-block:: bash
+
+    pip install "apache-airflow[celery]==|version|" --constraint "my-constraints.txt"
+
+
+The constraint file might also be hosted via a webserver of your choice and made available for remote use:

Review Comment:
   I will just make a general statement for it - no example is needed indeed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on a diff in pull request #31882: Update documentation for constraints installation

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #31882:
URL: https://github.com/apache/airflow/pull/31882#discussion_r1229109930


##########
docs/apache-airflow/installation/installing-from-pypi.rst:
##########
@@ -79,50 +87,85 @@ where:
 - ``AIRFLOW_VERSION`` - Airflow version (e.g. :subst-code:`|version|`) or ``main``, ``2-0``, for latest development version
 - ``PYTHON_VERSION`` Python version e.g. ``3.8``, ``3.9``
 
-There is also a ``constraints-no-providers`` constraint file, which contains just constraints required to
-install Airflow core. This allows to install and upgrade airflow separately and independently from providers.
+**However, that should not prevent you from being able to upgrade or downgrade providers and dependencies to other versions**
 
-You can create the URL to the file substituting the variables in the template below.
+You can, for example, install new versions of providers and dependencies after the release to use the latest
+version and up-to-date with latest security fixes - even if you do not want upgrade airflow core version.
+Or you can downgrade some dependencies or providers if you want to keep previous versions for compatibility
+reasons. Installing such dependencies should be done without constraints as a separate pip command.
 
-.. code-block::
+When you do such an upgrade, you should make sure to also add the ``apache-airflow`` package to the list of
+packages to install and pin it to the version that you have, otherwise you might end up with a
+different version of Airflow than you expect as ``pip`` can upgrade/downgrade it automatically when
+performing dependency resolution.
 
-  https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-no-providers-${PYTHON_VERSION}.txt
+.. code-block:: bash
 
-You can also use "latest" as version when you install "latest" stable version of Airflow. The "latest"
-constraints always points to the "latest" released Airflow version constraints:
+    pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
+    pip install "apache-airflow==|version|" apache-airflow-providers-google==10.1.1
 
-.. code-block::
+You can also downgrade or upgrade other dependencies this way - even if they are not compatible with
+those dependencies that are stored in the original constraints file:
+
+.. code-block:: bash
+
+    pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
+    pip install "apache-airflow[celery]==|version|" dbt-core==0.20.0
 
-  https://raw.githubusercontent.com/apache/airflow/constraints-latest/constraints-3.8.txt
+
+Such an upgrade is not always possible - you might have conflicting dependencies. However, by not using constraints,
+you give ``pip`` a chance to resolve the conflicts for you, while keeping within the limits of dependencies
+that Apache Airflow and other providers require. The combination of those dependencies might not be as well
+tested as the "golden" set of dependencies, but it should work in most cases. You can also always run
+the ``pip check`` command to test if the set of your Python packages is consistent and not conflicting.
+
+
+Using your own constraints
+==========================
+
+When you decide to install your own dependencies, or want to upgrade or downgrade providers, you might want
+to continue being able to keep reproducible installation of Airflow and those dependencies via a single command. In order to do that, you can produce your own constraints file and use it
+to install Airflow instead of the one provided by the community.
+
+.. code-block:: bash
+
+    pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
+    pip install "apache-airflow[celery]==|version|" dbt-core==0.20.0

Review Comment:
   ```suggestion
       pip install "apache-airflow==|version|" dbt-core==0.20.0
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] pankajkoti commented on a diff in pull request #31882: Update documentation for constraints installation

Posted by "pankajkoti (via GitHub)" <gi...@apache.org>.
pankajkoti commented on code in PR #31882:
URL: https://github.com/apache/airflow/pull/31882#discussion_r1229677331


##########
docs/apache-airflow/installation/installing-from-pypi.rst:
##########
@@ -197,11 +296,25 @@ that conflicts with the version of apache-airflow that you are using:
 
     pip install "apache-airflow==|version|" "apache-airflow-providers-google==8.0.0"
 
+.. note::
+
+    Installing, upgrading, downgrading providers separately is not guaranteed to work with all
+    Airflow versions or other providers. Some providers have minimum-required version of Airflow and some
+    versions of providers might have conflicting requirements with Airflow or other dependencies you

Review Comment:
   yes, pretty clear and detailed now. Thank you!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on a diff in pull request #31882: Update documentation for constraints installation

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #31882:
URL: https://github.com/apache/airflow/pull/31882#discussion_r1228825582


##########
docs/apache-airflow/installation/installing-from-pypi.rst:
##########
@@ -79,53 +87,64 @@ where:
 - ``AIRFLOW_VERSION`` - Airflow version (e.g. :subst-code:`|version|`) or ``main``, ``2-0``, for latest development version
 - ``PYTHON_VERSION`` Python version e.g. ``3.8``, ``3.9``
 
-There is also a ``constraints-no-providers`` constraint file, which contains just constraints required to
-install Airflow core. This allows to install and upgrade airflow separately and independently from providers.
+However, that should not prevent you from being able to install providers and dependencies that were upgraded
+after the release to keep your installation up-to-dated, even if you do not upgrade airflow core version.
+Installing such dependencies should be done without constraints as a separate pip command.
 
-You can create the URL to the file substituting the variables in the template below.
+You should make sure to also add the ``apache-airflow`` package to the list of packages to install and pin
+it to the version that you have, otherwise you might end up with a different version of Airflow than you
+expect as ``pip`` can upgrade/downgrade it automatically when performing dependency resolution.
 
-.. code-block::
+.. code-block:: bash
 
-  https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-no-providers-${PYTHON_VERSION}.txt
+    pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
+    pip install "apache-airflow[celery]==|version|" apache-airflow-providers-google==10.1.1
 
-You can also use "latest" as version when you install "latest" stable version of Airflow. The "latest"
-constraints always points to the "latest" released Airflow version constraints:
+You can also downgrade providers or other dependencies this way if you need to (for example if you do not
+want to upgrade to the latest version of a provider).
 
-.. code-block::
+.. code-block:: bash
+
+    pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
+    pip install "apache-airflow[celery]==|version|" dbt-core==0.20.0
+
+
+Not always such upgrade is possible - you might have conflicting dependencies, but by not using constraints,
+you give ``pip`` a chance to resolve the conflicts for you, while keeping within the limits of dependencies
+that Apache Airflow and other providers require. The combination of those dependencies might not be as well
+tested as the "golden" set of dependencies, but it should work in most cases. You can also always run
+the ``pip check`` command to test if the set of your Python packages is consistent and not conflicting.
+
+
+Using your own constraints
+==========================
+
+When you decide to install your own dependencies, or want to upgrade or downgrade providers, you might want
+to continue being able to run reproducible installation of Airflow and those dependencies. In order to do
+that you can produce your own constraints file and use it to install Airflow instead of the one provided
+by the community.
+
+.. code-block:: bash
+
+    pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
+    pip install "apache-airflow[celery]==|version|" dbt-core==0.20.0
+    pip freeze > my-constraints.txt
+
+
+Then you can use it to run reproducible installation of your environment in a single operation via local file:
+
+.. code-block:: bash
+
+    pip install "apache-airflow[celery]==|version|" --constraint "my-constraints.txt"
+
+
+The constraint file might also be hosted via a webserver of your choice and made available for remote use:
+
+
+.. code-block:: bash
+
+    pip install "apache-airflow[celery]==|version|" --constraint "https://my-company.org/my-constraints.txt"
 
-  https://raw.githubusercontent.com/apache/airflow/constraints-latest/constraints-3.8.txt
-
-
-Fixing Constraint files at release time

Review Comment:
   Yes. That was a bit too much, but I simplified it, removed middles section (which was a bit bullshit), and referred to "use your own constraints" in case the users want to keep updated between the released.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #31882: Update documentation for constraints installation

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #31882:
URL: https://github.com/apache/airflow/pull/31882#issuecomment-1590231730

   First pass of review applied.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #31882: Update documentation for constraints installation

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #31882:
URL: https://github.com/apache/airflow/pull/31882#issuecomment-1590668599

   I added few more touches in the latest iteration (see fixup):
   
   * explicitly showing the "no constraints" example in the intro section to make users aware this is recommended to upgrade things
   * added notes about recommending virtualenv and the need by the users to have appropriate OS-level dependencies installed
   * extracted note telling that in some cases it is impossible to solve conflict - and redirect users to our "best practices" of dealing with conflicting requirements (Python Venv/ExternalPython/ Docker/K8S etc.)
   * Extracted `pip check` to a separate chapter
   * Clarified and modernized a bit the "scenarios" - referring to the examples above.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on a diff in pull request #31882: Update documentation for constraints installation

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #31882:
URL: https://github.com/apache/airflow/pull/31882#discussion_r1228819317


##########
docs/apache-airflow/installation/installing-from-pypi.rst:
##########
@@ -53,20 +53,28 @@ This is an example, see further for more explanation.
 Constraints files
 '''''''''''''''''
 
-Airflow installation can be tricky sometimes because Airflow is both a library and an application.
+Why do we need constraints

Review Comment:
   yep



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on a diff in pull request #31882: Update documentation for constraints installation

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #31882:
URL: https://github.com/apache/airflow/pull/31882#discussion_r1229107629


##########
docs/apache-airflow/installation/installing-from-pypi.rst:
##########
@@ -79,50 +87,86 @@ where:
 - ``AIRFLOW_VERSION`` - Airflow version (e.g. :subst-code:`|version|`) or ``main``, ``2-0``, for latest development version
 - ``PYTHON_VERSION`` Python version e.g. ``3.8``, ``3.9``
 
-There is also a ``constraints-no-providers`` constraint file, which contains just constraints required to
-install Airflow core. This allows to install and upgrade airflow separately and independently from providers.
+**However, that should not prevent you from being able to upgrade or downgrade providers and dependencies to other versions**
 
-You can create the URL to the file substituting the variables in the template below.
+You can, for example, install new versions of providers and dependencies after the release to use the latest
+version and up-to-date with latest security fixes - even if you do not want upgrade airflow core version.
+Or you can downgrade some dependencies or providers if you want to keep previous versions for compatibility
+reasons. Installing such dependencies should be done without constraints as a separate pip command.
 
-.. code-block::
+When you do such an upgrade, you should make sure to also add the ``apache-airflow`` package to the list of
+packages to install and pin it to the version that you have, otherwise you might end up with a
+different version of Airflow than you expect as ``pip`` can upgrade/downgrade it automatically when
+performing dependency resolution.
 
-  https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-no-providers-${PYTHON_VERSION}.txt
+.. code-block:: bash
 
-You can also use "latest" as version when you install "latest" stable version of Airflow. The "latest"
-constraints always points to the "latest" released Airflow version constraints:
+    pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
+    pip install "apache-airflow[celery]==|version|" apache-airflow-providers-google==10.1.1

Review Comment:
   Ah... NO :) Copy&paste -> great spot !



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on a diff in pull request #31882: Update documentation for constraints installation

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #31882:
URL: https://github.com/apache/airflow/pull/31882#discussion_r1229636516


##########
docs/apache-airflow/installation/installing-from-pypi.rst:
##########
@@ -79,50 +111,124 @@ where:
 - ``AIRFLOW_VERSION`` - Airflow version (e.g. :subst-code:`|version|`) or ``main``, ``2-0``, for latest development version
 - ``PYTHON_VERSION`` Python version e.g. ``3.8``, ``3.9``
 
-There is also a ``constraints-no-providers`` constraint file, which contains just constraints required to
-install Airflow core. This allows to install and upgrade airflow separately and independently from providers.
+The examples below assume that you want to use install airflow in a reproducible way with the ``celery`` extra,
+but you can pick your own set of extras and providers to install.
 
-You can create the URL to the file substituting the variables in the template below.
+.. code-block:: bash
 
-.. code-block::
+    pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
 
-  https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-no-providers-${PYTHON_VERSION}.txt
 
-You can also use "latest" as version when you install "latest" stable version of Airflow. The "latest"
-constraints always points to the "latest" released Airflow version constraints:
+.. note::
 
-.. code-block::
+    The reproducible installation guarantees that this initial installation steps will always work for you -
+    providing that you use the right Python version and that you have appropriate Operating System dependencies
+    installed for the providers to be installed. Some of the providers require additional OS dependencies to
+    be installed such us ``build-essential`` in order to compile libraries, or for example database client
+    libraries in case you install a database provider, etc.. You need to figure out which system dependencies
+    you need when your installation fails and install them before retrying the installation.
+
+Upgrading and installing dependencies (including providers)
+===========================================================
+
+**The reproducible installation above should not prevent you from being able to upgrade or downgrade
+providers and other dependencies to other versions**
 
-  https://raw.githubusercontent.com/apache/airflow/constraints-latest/constraints-3.8.txt
+You can, for example, install new versions of providers and dependencies after the release to use the latest
+version and up-to-date with latest security fixes - even if you do not want upgrade airflow core version.
+Or you can downgrade some dependencies or providers if you want to keep previous versions for compatibility
+reasons. Installing such dependencies should be done without constraints as a separate pip command.
 
+When you do such an upgrade, you should make sure to also add the ``apache-airflow`` package to the list of
+packages to install and pin it to the version that you have, otherwise you might end up with a
+different version of Airflow than you expect as ``pip`` can upgrade/downgrade it automatically when
+performing dependency resolution.
+
+
+.. code-block:: bash
+
+    pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
+    pip install "apache-airflow==|version|" apache-airflow-providers-google==10.1.1
 
-Fixing Constraint files at release time
-'''''''''''''''''''''''''''''''''''''''
+You can also downgrade or upgrade other dependencies this way - even if they are not compatible with
+those dependencies that are stored in the original constraints file:
+
+.. code-block:: bash
+
+    pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
+    pip install "apache-airflow[celery]==|version|" dbt-core==0.20.0
+
+.. warning::
+
+    Not all dependencies can be installed this way - you might have dependencies conflicting with basic
+    requirements of Airflow or other dependencies installed in your system. However, by skipping constraints
+    when you install or upgrade dependencies, you give ``pip`` a chance to resolve the conflicts for you,
+    while keeping dependencies within the limits that Apache Airflow, providers and other dependencies require.
+    The resulting combination of those dependencies might not be tested together before as well as the set
+    of dependencies that come with constraints, but it should work in most cases as we usually add
+    requirements, when Airflow depends on particular versions of some dependencies. In cases you cannot
+    install some dependencies in the same environment as Airflow - you can attempt to use other approaches.
+    See :ref:`best practices for handling conflicting/complex Python dependencies <best_practices/handling_conflicting_complex_python_dependencies>`
+
+
+Verifying installed dependencies
+================================
+
+You can also always run the ``pip check`` command to test if the set of your Python packages is
+consistent and not conflicting.
+
+
+.. code-block:: bash
+
+    > pip check
+    No broken requirements found.
+
+
+When you see such message and the exit code from ``pip check`` is 0, you can be sure, that there are no
+conflicting dependencies in your environment.
+
+
+Using your own constraints
+==========================
+
+When you decide to install your own dependencies, or want to upgrade or downgrade providers, you might want
+to continue being able to keep reproducible installation of Airflow and those dependencies via a single command. In order to do that, you can produce your own constraints file and use it
+to install Airflow instead of the one provided by the community.
+
+.. code-block:: bash
+
+    pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
+    pip install "apache-airflow==|version|" dbt-core==0.20.0
+    pip freeze > my-constraints.txt

Review Comment:
   This is what a few people at the London meetup told as well. For me that was pretty obvious (It's how constraints were generated from the very beginning) but with the recent discussion and the talk I gave I realized that It was only "my bubble". People were - apparently so far- manually modifying the constraints and complained for example that recent google provider is not compatible with their (it turned out) custom constraints. 
   
   Funny how some things obvious to you might be completely alien to others :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on a diff in pull request #31882: Update documentation for constraints installation

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #31882:
URL: https://github.com/apache/airflow/pull/31882#discussion_r1229108971


##########
docs/apache-airflow/installation/installing-from-pypi.rst:
##########
@@ -79,50 +87,85 @@ where:
 - ``AIRFLOW_VERSION`` - Airflow version (e.g. :subst-code:`|version|`) or ``main``, ``2-0``, for latest development version
 - ``PYTHON_VERSION`` Python version e.g. ``3.8``, ``3.9``
 
-There is also a ``constraints-no-providers`` constraint file, which contains just constraints required to
-install Airflow core. This allows to install and upgrade airflow separately and independently from providers.
+**However, that should not prevent you from being able to upgrade or downgrade providers and dependencies to other versions**
 
-You can create the URL to the file substituting the variables in the template below.
+You can, for example, install new versions of providers and dependencies after the release to use the latest
+version and up-to-date with latest security fixes - even if you do not want upgrade airflow core version.
+Or you can downgrade some dependencies or providers if you want to keep previous versions for compatibility
+reasons. Installing such dependencies should be done without constraints as a separate pip command.
 
-.. code-block::
+When you do such an upgrade, you should make sure to also add the ``apache-airflow`` package to the list of
+packages to install and pin it to the version that you have, otherwise you might end up with a
+different version of Airflow than you expect as ``pip`` can upgrade/downgrade it automatically when
+performing dependency resolution.
 
-  https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-no-providers-${PYTHON_VERSION}.txt
+.. code-block:: bash
 
-You can also use "latest" as version when you install "latest" stable version of Airflow. The "latest"
-constraints always points to the "latest" released Airflow version constraints:
+    pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
+    pip install "apache-airflow[celery]==|version|" apache-airflow-providers-google==10.1.1

Review Comment:
   ```suggestion
       pip install "apache-airflow==|version|" apache-airflow-providers-google==10.1.1
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on a diff in pull request #31882: Update documentation for constraints installation

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #31882:
URL: https://github.com/apache/airflow/pull/31882#discussion_r1229654391


##########
docs/apache-airflow/installation/installing-from-pypi.rst:
##########
@@ -197,11 +296,25 @@ that conflicts with the version of apache-airflow that you are using:
 
     pip install "apache-airflow==|version|" "apache-airflow-providers-google==8.0.0"
 
+.. note::
+
+    Installing, upgrading, downgrading providers separately is not guaranteed to work with all
+    Airflow versions or other providers. Some providers have minimum-required version of Airflow and some
+    versions of providers might have conflicting requirements with Airflow or other dependencies you

Review Comment:
   Good point. I added a very specific recent example that illustrates it (google provider + google ads).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] jedcunningham commented on a diff in pull request #31882: Update documentation for constraints installation

Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on code in PR #31882:
URL: https://github.com/apache/airflow/pull/31882#discussion_r1228939074


##########
docs/apache-airflow/installation/installing-from-pypi.rst:
##########
@@ -79,50 +87,86 @@ where:
 - ``AIRFLOW_VERSION`` - Airflow version (e.g. :subst-code:`|version|`) or ``main``, ``2-0``, for latest development version
 - ``PYTHON_VERSION`` Python version e.g. ``3.8``, ``3.9``
 
-There is also a ``constraints-no-providers`` constraint file, which contains just constraints required to
-install Airflow core. This allows to install and upgrade airflow separately and independently from providers.
+**However, that should not prevent you from being able to upgrade or downgrade providers and dependencies to other versions**
 
-You can create the URL to the file substituting the variables in the template below.
+You can, for example, install new versions of providers and dependencies after the release to use the latest
+version and up-to-date with latest security fixes - even if you do not want upgrade airflow core version.
+Or you can downgrade some dependencies or providers if you want to keep previous versions for compatibility
+reasons. Installing such dependencies should be done without constraints as a separate pip command.
 
-.. code-block::
+When you do such an upgrade, you should make sure to also add the ``apache-airflow`` package to the list of
+packages to install and pin it to the version that you have, otherwise you might end up with a
+different version of Airflow than you expect as ``pip`` can upgrade/downgrade it automatically when
+performing dependency resolution.
 
-  https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-no-providers-${PYTHON_VERSION}.txt
+.. code-block:: bash
 
-You can also use "latest" as version when you install "latest" stable version of Airflow. The "latest"
-constraints always points to the "latest" released Airflow version constraints:
+    pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
+    pip install "apache-airflow[celery]==|version|" apache-airflow-providers-google==10.1.1

Review Comment:
   Hmm, is it necessary to specify extras again?



##########
docs/apache-airflow/installation/installing-from-pypi.rst:
##########
@@ -79,50 +87,86 @@ where:
 - ``AIRFLOW_VERSION`` - Airflow version (e.g. :subst-code:`|version|`) or ``main``, ``2-0``, for latest development version
 - ``PYTHON_VERSION`` Python version e.g. ``3.8``, ``3.9``
 
-There is also a ``constraints-no-providers`` constraint file, which contains just constraints required to
-install Airflow core. This allows to install and upgrade airflow separately and independently from providers.
+**However, that should not prevent you from being able to upgrade or downgrade providers and dependencies to other versions**
 
-You can create the URL to the file substituting the variables in the template below.
+You can, for example, install new versions of providers and dependencies after the release to use the latest
+version and up-to-date with latest security fixes - even if you do not want upgrade airflow core version.
+Or you can downgrade some dependencies or providers if you want to keep previous versions for compatibility
+reasons. Installing such dependencies should be done without constraints as a separate pip command.
 
-.. code-block::
+When you do such an upgrade, you should make sure to also add the ``apache-airflow`` package to the list of
+packages to install and pin it to the version that you have, otherwise you might end up with a
+different version of Airflow than you expect as ``pip`` can upgrade/downgrade it automatically when
+performing dependency resolution.
 
-  https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-no-providers-${PYTHON_VERSION}.txt
+.. code-block:: bash
 
-You can also use "latest" as version when you install "latest" stable version of Airflow. The "latest"
-constraints always points to the "latest" released Airflow version constraints:
+    pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
+    pip install "apache-airflow[celery]==|version|" apache-airflow-providers-google==10.1.1
 
-.. code-block::
+You can also downgrade or upgrade other dependencies this way - even if they are not compatible with
+those dependencies that are stored in the original constraints file:
+
+.. code-block:: bash
+
+    pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
+    pip install "apache-airflow[celery]==|version|" dbt-core==0.20.0
 
-  https://raw.githubusercontent.com/apache/airflow/constraints-latest/constraints-3.8.txt
+
+Such an upgrade is not always possible - you might have conflicting dependencies. However, by not using constraints,
+you give ``pip`` a chance to resolve the conflicts for you, while keeping within the limits of dependencies
+that Apache Airflow and other providers require. The combination of those dependencies might not be as well
+tested as the "golden" set of dependencies, but it should work in most cases. You can also always run
+the ``pip check`` command to test if the set of your Python packages is consistent and not conflicting.
+
+
+Using your own constraints
+==========================
+
+When you decide to install your own dependencies, or want to upgrade or downgrade providers, you might want
+to continue being able to keep reproducible installation of Airflow and those dependencies via single command
+line . In order to do that, you can produce your own constraints file and use it

Review Comment:
   ```suggestion
   to continue being able to keep reproducible installation of Airflow and those dependencies via a single command. In order to do that, you can produce your own constraints file and use it
   ```



##########
docs/apache-airflow/installation/installing-from-pypi.rst:
##########
@@ -79,50 +87,86 @@ where:
 - ``AIRFLOW_VERSION`` - Airflow version (e.g. :subst-code:`|version|`) or ``main``, ``2-0``, for latest development version
 - ``PYTHON_VERSION`` Python version e.g. ``3.8``, ``3.9``
 
-There is also a ``constraints-no-providers`` constraint file, which contains just constraints required to
-install Airflow core. This allows to install and upgrade airflow separately and independently from providers.
+**However, that should not prevent you from being able to upgrade or downgrade providers and dependencies to other versions**
 
-You can create the URL to the file substituting the variables in the template below.
+You can, for example, install new versions of providers and dependencies after the release to use the latest
+version and up-to-date with latest security fixes - even if you do not want upgrade airflow core version.
+Or you can downgrade some dependencies or providers if you want to keep previous versions for compatibility
+reasons. Installing such dependencies should be done without constraints as a separate pip command.
 
-.. code-block::
+When you do such an upgrade, you should make sure to also add the ``apache-airflow`` package to the list of
+packages to install and pin it to the version that you have, otherwise you might end up with a
+different version of Airflow than you expect as ``pip`` can upgrade/downgrade it automatically when
+performing dependency resolution.
 
-  https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-no-providers-${PYTHON_VERSION}.txt
+.. code-block:: bash
 
-You can also use "latest" as version when you install "latest" stable version of Airflow. The "latest"
-constraints always points to the "latest" released Airflow version constraints:
+    pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
+    pip install "apache-airflow[celery]==|version|" apache-airflow-providers-google==10.1.1
 
-.. code-block::
+You can also downgrade or upgrade other dependencies this way - even if they are not compatible with
+those dependencies that are stored in the original constraints file:
+
+.. code-block:: bash
+
+    pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
+    pip install "apache-airflow[celery]==|version|" dbt-core==0.20.0
 
-  https://raw.githubusercontent.com/apache/airflow/constraints-latest/constraints-3.8.txt
+
+Such an upgrade is not always possible - you might have conflicting dependencies. However, by not using constraints,
+you give ``pip`` a chance to resolve the conflicts for you, while keeping within the limits of dependencies
+that Apache Airflow and other providers require. The combination of those dependencies might not be as well
+tested as the "golden" set of dependencies, but it should work in most cases. You can also always run
+the ``pip check`` command to test if the set of your Python packages is consistent and not conflicting.
+
+
+Using your own constraints
+==========================
+
+When you decide to install your own dependencies, or want to upgrade or downgrade providers, you might want
+to continue being able to keep reproducible installation of Airflow and those dependencies via single command
+line . In order to do that, you can produce your own constraints file and use it
+to install Airflow instead of the one provided by the community.
+
+.. code-block:: bash
+
+    pip install "apache-airflow[celery]==|version|" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-|version|/constraints-3.8.txt"
+    pip install "apache-airflow[celery]==|version|" dbt-core==0.20.0
+    pip freeze > my-constraints.txt
+
+
+Then you can use it to create reproducible installations of your environment in a single operation via
+a local constraints file:
+
+.. code-block:: bash
+
+    pip install "apache-airflow[celery]==|version|" --constraint "my-constraints.txt"
+
+
+The constraint file might also be hosted via a webserver of your choice and made available for remote use:

Review Comment:
   I'm not sure we need this example, we have plenty of examples here of using it over http.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on a diff in pull request #31882: Update documentation for constraints installation

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #31882:
URL: https://github.com/apache/airflow/pull/31882#discussion_r1228819098


##########
docs/apache-airflow/installation/installing-from-pypi.rst:
##########
@@ -53,20 +53,28 @@ This is an example, see further for more explanation.
 Constraints files
 '''''''''''''''''
 
-Airflow installation can be tricky sometimes because Airflow is both a library and an application.
+Why do we need constraints
+==========================
+
+Airflow installation can be tricky because Airflow is both a library and an application.
+
 Libraries usually keep their dependencies open and applications usually pin them, but we should do neither
 and both at the same time. We decided to keep our dependencies as open as possible
 (in ``setup.cfg`` and ``setup.py``) so users can install different
 version of libraries if needed. This means that from time to time plain ``pip install apache-airflow`` will
 not work or will produce an unusable Airflow installation.
 
-In order to have a repeatable installation (and only for that reason), we also keep a set of "known-to-be-working" constraint files in the
+Reproducible Airflow installation
+=================================
+
+In order to have a reproducible installation, we also keep a set of constraint files in the
 ``constraints-main``, ``constraints-2-0``, ``constraints-2-1`` etc. orphan branches and then we create a tag
-for each released version e.g. :subst-code:`constraints-|version|`. This way, we keep a tested and working set of dependencies.
+for each released version e.g. :subst-code:`constraints-|version|`.
 
-Those "known-to-be-working" constraints are per major/minor Python version. You can use them as constraint
-files when installing Airflow from PyPI. Note that you have to specify the correct Airflow
-and Python versions in the URL.
+This way, we keep a tested set of dependencies at the moment of release. This provides you with the ability
+of having the exact same installation of airflow + providers + dependencies as was known to be working
+at the moment of release - "golden" set of dependencies for that version of Airflow. There is a separate

Review Comment:
   Good idea. I think this could be why people are "clinging" to it. I changed it to "frozen" to get it a bit "chilled".



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org