[GitHub] [airflow] potiuk commented on issue #13336: argcomplete is not compatible with importlib-metadata

[GitBox <gi...@apache.org>]

potiuk commented on issue #13336:
URL: https://github.com/apache/airflow/issues/13336#issuecomment-751638597


   Unfortunately we do not support `pip-compile` installation. Airflow has fairly complex set of dependencies and the official way of installation is via PIP using constraint files:
   
   http://airflow.apache.org/docs/apache-airflow/stable/installation.html#getting-airflow
   
   For example for python 3.6 this is the right way of installing airflow (and it is stable to work):
   ```
   pip install "apache-airflow==1.10.14" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.14/constraints-3.6.txt"
   ```
   
   In the constraint file for 1.10.14 you will find that we have (for python 3.6 - you can check other versions by changing python version in the URL):
   
   * argcomplete==1.12.2
   * importlib-metadata==2.1.0
   
   We have a whole CI system implemented that verifies and automatically updates the constrainfts for upcoming Airflow relases and makes sure that those constraints are consistent (they pass `pip check`) and that all the 4000+ unit tests works with those constraints. Additionally those are constraints not requirements, which makes them stable to install, but at the same time they do not limit you from individually upgrading dependencies if needed (within the boundaries limited by Airflow. We try to limit as lilttle as possible because Airflow is both - application and library  (there are people who need to upgrade their dependencies for the custom DAGs/Operators they write.
   
   You can read more about Airflow's dependency management here and the reasoning why we are doing it: https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#dependency-management
   
   Unfortunately neither `poetry` (we disscussed it recently) nor (as far as I know) `pip-compile` is flexible enough to give us what we want. They are opinionated in the way they treat dependencies and the models they introduce are not flexible enough to handle the case where you want to keep stable installation and leave the freedom to upgrade particular dependencies, that's why we have the 'pip' + CI generated constraints solution which handle all the cases for us.
   
   If you know `pip-compile` well enough and want to continue using it and replicate what we have done with constraints, you are most welcome to describe and add alternative installation method also for others to use. However the per-python-version constratint files automatically generated in our repo should be the starting point for that, because we have quite sophisticated engineering behind to make it both consistent and automatically upgraded when new versions of packages are released.
   
   
   
   
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org