[GitHub] [airflow] potiuk commented on issue #10038: Using >= or =~ in dependency management is a very bad practice

[GitBox <gi...@apache.org>]

potiuk commented on issue #10038:
URL: https://github.com/apache/airflow/issues/10038#issuecomment-665131906


   Hello Noel,
   
   Indeed the problem you describe is very true for Applications but it is
   quite the opposite for libraries. Unfortunately Airflow is a bit of both,
   that's why we had  to find a bit different solution - keeping the setup.py
   dependencies as open as possible but allowing Airflow as application to be
   installed with frozen set of dependencies. Those are frozen both - for
   master (we automatically test and re-freeze them continuously after testing
   during our CI process) but also we freeze the fixed set of dependencies
   whenever we release a stable version of Airflow (and you can very easily
   use either the latest or the 1.10.x frozen version at any time by providing
   the right `--constraint` URL.
   
   You can take a look at our 'Installimg from PyPI' chapter of README.md -
   https://github.com/apache/airflow#installing-from-pypi where it is
   explained how to install Airflow in repeatable way (using constraint files
   we maintain automatically with all the dependencies frozen). Also
   INSTALL.txt contains the right instructions (note that they changed a bit
   in the upcoming version but you can find the corresponding instructions in
   1.10.10 and 1.10.11 tags of the same files - for 1.10.10 and 1.10.12. the
   current set of instructions will be valid as of 1.10.12)
   
   For detailed instructions and explanation why we had gone that route, you
   can see
   https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#airflow-dependencies
   (also the following chapter 'Pinned constraint files' where we explained
   how we addressed it). As mentioned - it's the second iteration of approach
   to fix it. The first one in 1.10.10 and 1.10.11 has proven to be too
   'heavy' for maintenance by contributors, so we improved it, simplified it,
   and we fully automated the process of dependency freezing last week (and it
   will be released in 1.10.12).
   
   I hope this addresses your concerns.
   
   Please read those links I sent and let us know if it does or does not
   address your concerns (and let
   Is know please why - if not). Also if you have any concrete proposals how
   we can improve the current setup without breaking our 'both library and
   app' requirement - we are happy to hear it. Whether it is the documentation
   or the process improvement proposals.
   
   I am particularly interested in how we can surface information about it
   better. It seems that despite all our efforts to explain it in INSTALL.txt,
   README.md and CONTRIBUTING.rst - it's not enough as people sometimes do not
   find it easily, so I would love to hear some suggestions and proposals how
   we could improve it - especially from person not familiar with the process
   and whole reasoning.
   
   The most welcome is a PR improving it that we can review and merge :).
   
   J
   
   wt., 28 lip 2020, 16:57 użytkownik Noël Bardelot <no...@github.com>
   napisał:
   
   > From an ops point of view, packaging Airflow with about 150 dependencies
   > not strictly tied to a version is a bad practice. In setup.py there are
   > about 20 dependencies using =~ versions and about 130 dependencies using
   > >=  versions. This means there is no guarantee that two instances of
   > Airflow run the same verions of python libraries.
   >
   > Being forced to manually compute dozens of dependency versions and force
   > them in requirements.txt is not a viable solution.
   >
   > Airflow should freeze its dependency tree when a new stable version comes
   > out.
   >
   > —
   > You are receiving this because you are subscribed to this thread.
   > Reply to this email directly, view it on GitHub
   > <https://github.com/apache/airflow/issues/10038>, or unsubscribe
   > <https://github.com/notifications/unsubscribe-auth/AAERMI3VCH6PZ5ASZEYDYV3R53RMNANCNFSM4PKTPFRQ>
   > f
   >
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org