[jira] [Commented] (BEAM-3106) Consider not pinning all python dependencies, or moving them to requirements.txt

["Ahmet Altay (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-3106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16222645#comment-16222645 ] 

Ahmet Altay commented on BEAM-3106:
-----------------------------------

The reason behind pinning or capping dependencies is to prevent broken releases after time passes. In the past there were many occasions of a dependency releasing a backward-incompatible change and breaking and already release Beam version. This is bad for existing Beam users, because at some point in the future their currently working Beam release may stop and force them to do upgrades.

We follow the semantic versioning rules (capping) in general with dependencies. (For example: 'avro>=1.8.1,<2.0.0'). However, in the past, some dependencies, also released breaking changes without incrementing the major version. For those dependencies only, we pin their version to prevent any future breakages.

We can consider an alternative policy to what we are doing today, but it is important for us to ensure (as much as possible) that already released Beam versions will continue to work even after breaking change in a dependency. 


> Consider not pinning all python dependencies, or moving them to requirements.txt
> --------------------------------------------------------------------------------
>
>                 Key: BEAM-3106
>                 URL: https://issues.apache.org/jira/browse/BEAM-3106
>             Project: Beam
>          Issue Type: Wish
>          Components: build-system
>    Affects Versions: 2.1.0
>         Environment: python
>            Reporter: Maximilian Roos
>            Assignee: Ahmet Altay
>
> Currently all python dependencies are [pinned or capped|https://github.com/apache/beam/blob/master/sdks/python/setup.py#L97]
> While there's a good argument for supplying a `requirements.txt` with well tested dependencies, having them specified in `setup.py` forces them to an exact state on each install of Beam. This makes using Beam in any environment with other libraries nigh on impossible. 
> This is particularly severe for the `gcp` dependencies, where we have libraries that won't work with an older version (but Beam _does_ work with an newer version). We have to do a bunch of gymnastics to get the correct versions installed because of this. Unfortunately, airflow repeats this practice and conflicts on a number of dependencies, adding further complication (but, again there is no real conflict).
> I haven't seen this practice outside of the Apache & Google ecosystem - for example no libraries in numerical python do this. Here's a [discussion on SO|https://stackoverflow.com/questions/28509481/should-i-pin-my-python-dependencies-versions]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)