[GitHub] [beam] kennknowles opened a new issue, #18542: Consider not pinning all python dependencies, or moving them to requirements.txt

[GitBox <gi...@apache.org>]

kennknowles opened a new issue, #18542:
URL: https://github.com/apache/beam/issues/18542

   Currently all python dependencies are [pinned or capped](https://github.com/apache/beam/blob/master/sdks/python/setup.py#L97)
   
   While there's a good argument for supplying a `requirements.txt` with well tested dependencies, having them specified in `setup.py` forces them to an exact state on each install of Beam. This makes using Beam in any environment with other libraries nigh on impossible. 
   
   This is particularly severe for the `gcp` dependencies, where we have libraries that won't work with an older version (but Beam _does_ work with an newer version). We have to do a bunch of gymnastics to get the correct versions installed because of this. Unfortunately, airflow repeats this practice and conflicts on a number of dependencies, adding further complication (but, again there is no real conflict).
   
   I haven't seen this practice outside of the Apache & Google ecosystem - for example no libraries in numerical python do this. Here's a [discussion on SO](https://stackoverflow.com/questions/28509481/should-i-pin-my-python-dependencies-versions)
   
   Imported from Jira [BEAM-3106](https://issues.apache.org/jira/browse/BEAM-3106). Original Jira may contain additional context.
   Reported by: maximilianr.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org