[jira] [Created] (BEAM-3106) Consider not pinning all python dependencies, or moving them to requirements.txt

["Maximilian Roos (JIRA)" <ji...@apache.org>]

Maximilian Roos created BEAM-3106:
-------------------------------------

             Summary: Consider not pinning all python dependencies, or moving them to requirements.txt
                 Key: BEAM-3106
                 URL: https://issues.apache.org/jira/browse/BEAM-3106
             Project: Beam
          Issue Type: Wish
          Components: build-system
    Affects Versions: 2.1.0
         Environment: python
            Reporter: Maximilian Roos
            Assignee: Davor Bonaci


Currently all python dependencies are [pinned or capped|https://github.com/apache/beam/blob/master/sdks/python/setup.py#L97]

While there's a good argument for supplying a `requirements.txt` with well tested dependencies, having them specified in `setup.py` forces them to an exact state on each install of Beam. This makes using Beam in any environment with other libraries nigh on impossible. 

This is particularly severe for the `gcp` dependencies, where we have libraries that won't work with an older version (but Beam _does_ work with an newer version). We have to do a bunch of gymnastics to get the correct versions installed because of this. Unfortunately, airflow repeats this practice and conflicts on a number of dependencies, adding further complication (but, again there is no real conflict).

I haven't seen this practice outside of the Apache & Google ecosystem - for example no libraries in numerical python do this. Here's a [discussion on SO|https://stackoverflow.com/questions/28509481/should-i-pin-my-python-dependencies-versions]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)