[GitHub] [airflow] potiuk edited a comment on issue #17584: Replace constraints file requirement with `pyproject.toml` version rules

[GitBox <gi...@apache.org>]

potiuk edited a comment on issue #17584:
URL: https://github.com/apache/airflow/issues/17584#issuecomment-898045025


   Yeah. Constraints are mechanism which is applied at install time only (and they are not persisted after airflow is installed), there are no mechanism in pyproject.toml that can provide anything like that. The reasons why we are using constraints is explained here https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#dependency-management . Let me repeat it here:
   
   > Airflow is not a standard python project. Most of the python projects fall into one of two types - application or library. As described in this StackOverflow question, the decision whether to pin (freeze) dependency versions for a python project depends on the type. For applications, dependencies should be pinned, but for libraries, they should be open.
   
   > For application, pinning the dependencies makes it more stable to install in the future - because new (even transitive) dependencies might cause installation to fail. For libraries - the dependencies should be open to allow several different libraries with the same requirements to be installed at the same time.
   
   > The problem is that Apache Airflow is a bit of both - application to install and library to be used when you are developing your own operators and DAGs.
   
   > This - seemingly unsolvable - puzzle is solved by having pinned constraints files.
   
   We've searched  and discussed how to apply a a good solution for ~1 year after I joined the project. And  we spent another year on perfecting and improving the constraint solution. I have very good reasons to believe that there is no simpler way to achieve it. But maybe @aaronsters you can have some proposals.
   
   The solution must fullfill two basic requirements:
   
   1) When released, users should be able to install airflow independently if any of dependencies releases a new version. Same command to install airflow 2.1.2 on August 1 should work on December 31. Note that we cannot depend on semver versioning for external dependencies - we have ~ 500 dependencies in total, not all of them follow semver, and every few weeks one of those dependencies will introduce a backwards-incompatible change even in patchlevel release. This has happened multiple times before we introduced the constraints - for example 1.10.8 was immediately broken as next day after release Werkzeug released a breaking change that stopped Airflow not only from working but also from installing - we had to release 1.10.9 same day (breaking ASF rules of release) - see announcement here: https://airflow.apache.org/announcements/#feb-7-2020
   
   2) However, once airflow is installed, users should be able to upgrade any of the dependencies if new versions are released - so we cannot limit the versions by == in install_requires.
   
   Unless someone points out on how to define those rules in pyproject.toml I consider that as invalid. Please @aaronsteers  - if you know how to do it, have an example or documentation explaining it - please post it here, I will close it until we have more explanations but I am all ears and glad to hear to any proposal that might improve and simplify it. 
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org