You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Robert Bradshaw <ro...@google.com> on 2017/12/15 21:38:15 UTC
Re: [jira] [Commented] (BEAM-3357) Python SDK head fails to run tests
due to Requirement.parse('protobuf<=3.4.0,>=3.2.0')
I am also in favor of pinning as an immediate fix, bumping the bound otherwise.
Regarding putting an upper bound to avoid being broken, the last two
breaks have been due to just having an (unneeded) upper bound (which
held us back to broken/incompatible releases in relationship to other
dependencies). We should try to trust semantic versioning when
possible, and when not we must regularly audit.
On Fri, Dec 15, 2017 at 1:33 PM, Chamikara Jayalath (JIRA)
<ji...@apache.org> wrote:
>
> [ https://issues.apache.org/jira/browse/BEAM-3357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16293276#comment-16293276 ]
>
> Chamikara Jayalath commented on BEAM-3357:
> ------------------------------------------
>
> You mean we could bump up the upper bound ? I think we should keep some upper bound in case we get badly broken by a future protobuf release.
>
>> Python SDK head fails to run tests due to Requirement.parse('protobuf<=3.4.0,>=3.2.0')
>> --------------------------------------------------------------------------------------
>>
>> Key: BEAM-3357
>> URL: https://issues.apache.org/jira/browse/BEAM-3357
>> Project: Beam
>> Issue Type: Bug
>> Components: sdk-py-core
>> Reporter: Chamikara Jayalath
>> Priority: Critical
>>
>> Error is:
>> running build_ext
>> Traceback (most recent call last):
>> File "setup.py", line 202, in <module>
>> 'test': generate_protos_first(test),
>> File "/Users/chamikara/testing/test_py_12_14_2017_2/env_proto_3.4/lib/python2.7/site-packages/setuptools/__init__.py", line 129, in setup
>> return distutils.core.setup(**attrs)
>> File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 151, in setup
>> dist.run_commands()
>> File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 953, in run_commands
>> self.run_command(cmd)
>> File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
>> cmd_obj.run()
>> File "setup.py", line 142, in run
>> super(cmd, self).run()
>> File "/Users/chamikara/testing/test_py_12_14_2017_2/env_proto_3.4/lib/python2.7/site-packages/setuptools/command/test.py", line 225, in run
>> with self.project_on_sys_path():
>> File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py", line 17, in __enter__
>> return self.gen.next()
>> File "/Users/chamikara/testing/test_py_12_14_2017_2/env_proto_3.4/lib/python2.7/site-packages/setuptools/command/test.py", line 164, in project_on_sys_path
>> require('%s==%s' % (ei_cmd.egg_name, ei_cmd.egg_version))
>> File "/Users/chamikara/testing/test_py_12_14_2017_2/env_proto_3.4/lib/python2.7/site-packages/pkg_resources/__init__.py", line 984, in require
>> needed = self.resolve(parse_requirements(requirements))
>> File "/Users/chamikara/testing/test_py_12_14_2017_2/env_proto_3.4/lib/python2.7/site-packages/pkg_resources/__init__.py", line 875, in resolve
>> raise VersionConflict(dist, req).with_context(dependent_req)
>> pkg_resources.ContextualVersionConflict: (protobuf 3.5.0.post1 (/Users/chamikara/testing/test_py_12_14_2017_2/beam/sdks/python/.eggs/protobuf-3.5.0.post1-py2.7.egg), Requirement.parse('protobuf<=3.4.0,>=3.2.0'), set(['apache-beam']))
>> Seems like grpcio did a release today which is breaking us: https://pypi.python.org/pypi/grpcio/1.8.1
>> We have to either bump our protobuf dependency or reduce the upper bound of grpcio dependency to previous release (1.7.3).
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.4.14#64029)
Re: [jira] [Commented] (BEAM-3357) Python SDK head fails to run tests
due to Requirement.parse('protobuf<=3.4.0,>=3.2.0')
Posted by Chamikara Jayalath <ch...@google.com>.
Agree that with the workaround for specifying specific versions, pinning
exact versions becomes less useful. +1 for trusting semantic versioning and
allowing dependencies to be updated till next major version whenever
possible. If we run into issues we can ask users to pin specific versions
when installing.
- Cham
On Fri, Dec 15, 2017 at 3:09 PM Udi Meiri <eh...@google.com> wrote:
> Yeah, I guess that's the difference between releasing source vs running
> your own service.
>
> On Fri, Dec 15, 2017 at 2:58 PM Ahmet Altay <al...@google.com> wrote:
>
>> I agree with Robert. Also there is usually a workaround to unbrake
>> previous version by installing a specific version of an offending
>> dependency.
>>
>>
>> On Fri, Dec 15, 2017 at 2:53 PM, Robert Bradshaw <ro...@google.com>
>> wrote:
>>
>>> This has the downside of pinning the dependencies for all downstream
>>> projects, making it impossible for them to use different versions than
>>> the ones we happened to choose. (Imagine the pain of two or more of
>>> our dependencies pinned all their dependencies...)
>>>
>>> On Fri, Dec 15, 2017 at 2:48 PM, Udi Meiri <eh...@google.com> wrote:
>>> > +1 to pinning to exact versions, to be sure that our releases do not
>>> break
>>> > when newer versions of dependencies are released.
>>> >
>>> > On Fri, Dec 15, 2017 at 2:44 PM Ahmet Altay <al...@google.com> wrote:
>>> >>
>>> >> On Fri, Dec 15, 2017 at 2:42 PM, Chamikara Jayalath <
>>> chamikara@google.com>
>>> >> wrote:
>>> >>>
>>> >>> +1 for automating the process of checking for possible version bumps.
>>> >>>
>>> >>> Also, what do you think about pinning dependencies to exact versions
>>> >>> (instead of ranges) after cutting a release branch ? This should
>>> improve the
>>> >>> stability of released SDKs (but not a prefect solution since
>>> transitive
>>> >>> dependencies can still change).
>>> >>
>>> >>
>>> >> This is a reasonable suggestion. The issue with that is, by being less
>>> >> flexible we will prevent users from using latest versions of
>>> dependencies.
>>> >> On the other hand it will prevent breaking of already released
>>> versions.
>>> >>
>>> >>>
>>> >>>
>>> >>> Thanks,
>>> >>> Cham
>>> >>>
>>> >>> On Fri, Dec 15, 2017 at 2:19 PM Ahmet Altay <al...@google.com>
>>> wrote:
>>> >>>>
>>> >>>> On Fri, Dec 15, 2017 at 2:02 PM, Robert Bradshaw <
>>> robertwb@google.com>
>>> >>>> wrote:
>>> >>>>>
>>> >>>>> On Fri, Dec 15, 2017 at 1:51 PM, Ahmet Altay <al...@google.com>
>>> wrote:
>>> >>>>> >
>>> >>>>> > On Fri, Dec 15, 2017 at 1:38 PM, Robert Bradshaw
>>> >>>>> > <ro...@google.com>
>>> >>>>> > wrote:
>>> >>>>> >>
>>> >>>>> >> I am also in favor of pinning as an immediate fix, bumping the
>>> bound
>>> >>>>> >> otherwise.
>>> >>>>> >>
>>> >>>>> >> Regarding putting an upper bound to avoid being broken, the
>>> last two
>>> >>>>> >> breaks have been due to just having an (unneeded) upper bound
>>> (which
>>> >>>>> >> held us back to broken/incompatible releases in relationship to
>>> >>>>> >> other
>>> >>>>> >> dependencies). We should try to trust semantic versioning when
>>> >>>>> >> possible, and when not we must regularly audit.
>>> >>>>> >
>>> >>>>> > +1 to this, especially the auditing part. We also had breaks
>>> because
>>> >>>>> > we
>>> >>>>> > trusted semantic versioning. So far our semi-official policy was
>>> to
>>> >>>>> > trust a
>>> >>>>> > package until they prove it otherwise. I will argue that grpc
>>> here is
>>> >>>>> > making
>>> >>>>> > a breaking change in a minor version increment by changing the
>>> way
>>> >>>>> > they are
>>> >>>>> > depending on a major package.
>>> >>>>>
>>> >>>>> A minor version bump should be allowed to require a minor version
>>> bump
>>> >>>>> in its dependencies.
>>> >>>>>
>>> >>>>> > We have done a good job of auditing and updating those pinned (or
>>> >>>>> > upper
>>> >>>>> > bounded) dependencies, and probably we are behind in some of
>>> those.
>>> >>>>> >
>>> >>>>> > I wonder if we can automate some of this? If we can get a report,
>>> >>>>> > that
>>> >>>>> > audits our dependencies, warns us about new releases and
>>> potential
>>> >>>>> > conflicts
>>> >>>>> > it would be much easier to keep things up to date.
>>> >>>>>
>>> >>>>> Big +1, it should be easy to set up a nightly that relaxes some of
>>> the
>>> >>>>> requirements and sees what (if anything) breaks. Not breaking is
>>> >>>>> likely a signal that we should relax ours.
>>> >>>>
>>> >>>>
>>> >>>> Filed https://issues.apache.org/jira/browse/BEAM-3363 to track
>>> this. I
>>> >>>> think it would be awesome if we can tackle this as part of a better
>>> >>>> infrastructure for testing work.
>>> >>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> >> On Fri, Dec 15, 2017 at 1:33 PM, Chamikara Jayalath (JIRA)
>>> >>>>> >> <ji...@apache.org> wrote:
>>> >>>>> >> >
>>> >>>>> >> > [
>>> >>>>> >> >
>>> >>>>> >> >
>>> https://issues.apache.org/jira/browse/BEAM-3357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16293276#comment-16293276
>>> >>>>> >> > ]
>>> >>>>> >> >
>>> >>>>> >> > Chamikara Jayalath commented on BEAM-3357:
>>> >>>>> >> > ------------------------------------------
>>> >>>>> >> >
>>> >>>>> >> > You mean we could bump up the upper bound ? I think we should
>>> keep
>>> >>>>> >> > some
>>> >>>>> >> > upper bound in case we get badly broken by a future protobuf
>>> >>>>> >> > release.
>>> >>>>> >> >
>>> >>>>> >> >> Python SDK head fails to run tests due to
>>> >>>>> >> >> Requirement.parse('protobuf<=3.4.0,>=3.2.0')
>>> >>>>> >> >>
>>> >>>>> >> >>
>>> >>>>> >> >>
>>> --------------------------------------------------------------------------------------
>>> >>>>> >> >>
>>> >>>>> >> >> Key: BEAM-3357
>>> >>>>> >> >> URL:
>>> >>>>> >> >> https://issues.apache.org/jira/browse/BEAM-3357
>>> >>>>> >> >> Project: Beam
>>> >>>>> >> >> Issue Type: Bug
>>> >>>>> >> >> Components: sdk-py-core
>>> >>>>> >> >> Reporter: Chamikara Jayalath
>>> >>>>> >> >> Priority: Critical
>>> >>>>> >> >>
>>> >>>>> >> >> Error is:
>>> >>>>> >> >> running build_ext
>>> >>>>> >> >> Traceback (most recent call last):
>>> >>>>> >> >> File "setup.py", line 202, in <module>
>>> >>>>> >> >> 'test': generate_protos_first(test),
>>> >>>>> >> >> File
>>> >>>>> >> >>
>>> >>>>> >> >>
>>> "/Users/chamikara/testing/test_py_12_14_2017_2/env_proto_3.4/lib/python2.7/site-packages/setuptools/__init__.py",
>>> >>>>> >> >> line 129, in setup
>>> >>>>> >> >> return distutils.core.setup(**attrs)
>>> >>>>> >> >> File
>>> >>>>> >> >>
>>> >>>>> >> >>
>>> "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py",
>>> >>>>> >> >> line 151, in setup
>>> >>>>> >> >> dist.run_commands()
>>> >>>>> >> >> File
>>> >>>>> >> >>
>>> >>>>> >> >>
>>> "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py",
>>> >>>>> >> >> line 953, in run_commands
>>> >>>>> >> >> self.run_command(cmd)
>>> >>>>> >> >> File
>>> >>>>> >> >>
>>> >>>>> >> >>
>>> "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py",
>>> >>>>> >> >> line 972, in run_command
>>> >>>>> >> >> cmd_obj.run()
>>> >>>>> >> >> File "setup.py", line 142, in run
>>> >>>>> >> >> super(cmd, self).run()
>>> >>>>> >> >> File
>>> >>>>> >> >>
>>> >>>>> >> >>
>>> "/Users/chamikara/testing/test_py_12_14_2017_2/env_proto_3.4/lib/python2.7/site-packages/setuptools/command/test.py",
>>> >>>>> >> >> line 225, in run
>>> >>>>> >> >> with self.project_on_sys_path():
>>> >>>>> >> >> File
>>> >>>>> >> >>
>>> >>>>> >> >>
>>> "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py",
>>> >>>>> >> >> line 17, in __enter__
>>> >>>>> >> >> return self.gen.next()
>>> >>>>> >> >> File
>>> >>>>> >> >>
>>> >>>>> >> >>
>>> "/Users/chamikara/testing/test_py_12_14_2017_2/env_proto_3.4/lib/python2.7/site-packages/setuptools/command/test.py",
>>> >>>>> >> >> line 164, in project_on_sys_path
>>> >>>>> >> >> require('%s==%s' % (ei_cmd.egg_name, ei_cmd.egg_version))
>>> >>>>> >> >> File
>>> >>>>> >> >>
>>> >>>>> >> >>
>>> "/Users/chamikara/testing/test_py_12_14_2017_2/env_proto_3.4/lib/python2.7/site-packages/pkg_resources/__init__.py",
>>> >>>>> >> >> line 984, in require
>>> >>>>> >> >> needed = self.resolve(parse_requirements(requirements))
>>> >>>>> >> >> File
>>> >>>>> >> >>
>>> >>>>> >> >>
>>> "/Users/chamikara/testing/test_py_12_14_2017_2/env_proto_3.4/lib/python2.7/site-packages/pkg_resources/__init__.py",
>>> >>>>> >> >> line 875, in resolve
>>> >>>>> >> >> raise VersionConflict(dist,
>>> req).with_context(dependent_req)
>>> >>>>> >> >> pkg_resources.ContextualVersionConflict: (protobuf
>>> 3.5.0.post1
>>> >>>>> >> >>
>>> >>>>> >> >>
>>> (/Users/chamikara/testing/test_py_12_14_2017_2/beam/sdks/python/.eggs/protobuf-3.5.0.post1-py2.7.egg),
>>> >>>>> >> >> Requirement.parse('protobuf<=3.4.0,>=3.2.0'),
>>> >>>>> >> >> set(['apache-beam']))
>>> >>>>> >> >> Seems like grpcio did a release today which is breaking us:
>>> >>>>> >> >> https://pypi.python.org/pypi/grpcio/1.8.1
>>> >>>>> >> >> We have to either bump our protobuf dependency or reduce the
>>> >>>>> >> >> upper
>>> >>>>> >> >> bound of grpcio dependency to previous release (1.7.3).
>>> >>>>> >> >
>>> >>>>> >> >
>>> >>>>> >> >
>>> >>>>> >> > --
>>> >>>>> >> > This message was sent by Atlassian JIRA
>>> >>>>> >> > (v6.4.14#64029)
>>> >>>>> >
>>> >>>>> >
>>>
>>
>>
Re: [jira] [Commented] (BEAM-3357) Python SDK head fails to run tests
due to Requirement.parse('protobuf<=3.4.0,>=3.2.0')
Posted by Udi Meiri <eh...@google.com>.
Yeah, I guess that's the difference between releasing source vs running
your own service.
On Fri, Dec 15, 2017 at 2:58 PM Ahmet Altay <al...@google.com> wrote:
> I agree with Robert. Also there is usually a workaround to unbrake
> previous version by installing a specific version of an offending
> dependency.
>
>
> On Fri, Dec 15, 2017 at 2:53 PM, Robert Bradshaw <ro...@google.com>
> wrote:
>
>> This has the downside of pinning the dependencies for all downstream
>> projects, making it impossible for them to use different versions than
>> the ones we happened to choose. (Imagine the pain of two or more of
>> our dependencies pinned all their dependencies...)
>>
>> On Fri, Dec 15, 2017 at 2:48 PM, Udi Meiri <eh...@google.com> wrote:
>> > +1 to pinning to exact versions, to be sure that our releases do not
>> break
>> > when newer versions of dependencies are released.
>> >
>> > On Fri, Dec 15, 2017 at 2:44 PM Ahmet Altay <al...@google.com> wrote:
>> >>
>> >> On Fri, Dec 15, 2017 at 2:42 PM, Chamikara Jayalath <
>> chamikara@google.com>
>> >> wrote:
>> >>>
>> >>> +1 for automating the process of checking for possible version bumps.
>> >>>
>> >>> Also, what do you think about pinning dependencies to exact versions
>> >>> (instead of ranges) after cutting a release branch ? This should
>> improve the
>> >>> stability of released SDKs (but not a prefect solution since
>> transitive
>> >>> dependencies can still change).
>> >>
>> >>
>> >> This is a reasonable suggestion. The issue with that is, by being less
>> >> flexible we will prevent users from using latest versions of
>> dependencies.
>> >> On the other hand it will prevent breaking of already released
>> versions.
>> >>
>> >>>
>> >>>
>> >>> Thanks,
>> >>> Cham
>> >>>
>> >>> On Fri, Dec 15, 2017 at 2:19 PM Ahmet Altay <al...@google.com> wrote:
>> >>>>
>> >>>> On Fri, Dec 15, 2017 at 2:02 PM, Robert Bradshaw <
>> robertwb@google.com>
>> >>>> wrote:
>> >>>>>
>> >>>>> On Fri, Dec 15, 2017 at 1:51 PM, Ahmet Altay <al...@google.com>
>> wrote:
>> >>>>> >
>> >>>>> > On Fri, Dec 15, 2017 at 1:38 PM, Robert Bradshaw
>> >>>>> > <ro...@google.com>
>> >>>>> > wrote:
>> >>>>> >>
>> >>>>> >> I am also in favor of pinning as an immediate fix, bumping the
>> bound
>> >>>>> >> otherwise.
>> >>>>> >>
>> >>>>> >> Regarding putting an upper bound to avoid being broken, the last
>> two
>> >>>>> >> breaks have been due to just having an (unneeded) upper bound
>> (which
>> >>>>> >> held us back to broken/incompatible releases in relationship to
>> >>>>> >> other
>> >>>>> >> dependencies). We should try to trust semantic versioning when
>> >>>>> >> possible, and when not we must regularly audit.
>> >>>>> >
>> >>>>> > +1 to this, especially the auditing part. We also had breaks
>> because
>> >>>>> > we
>> >>>>> > trusted semantic versioning. So far our semi-official policy was
>> to
>> >>>>> > trust a
>> >>>>> > package until they prove it otherwise. I will argue that grpc
>> here is
>> >>>>> > making
>> >>>>> > a breaking change in a minor version increment by changing the way
>> >>>>> > they are
>> >>>>> > depending on a major package.
>> >>>>>
>> >>>>> A minor version bump should be allowed to require a minor version
>> bump
>> >>>>> in its dependencies.
>> >>>>>
>> >>>>> > We have done a good job of auditing and updating those pinned (or
>> >>>>> > upper
>> >>>>> > bounded) dependencies, and probably we are behind in some of
>> those.
>> >>>>> >
>> >>>>> > I wonder if we can automate some of this? If we can get a report,
>> >>>>> > that
>> >>>>> > audits our dependencies, warns us about new releases and potential
>> >>>>> > conflicts
>> >>>>> > it would be much easier to keep things up to date.
>> >>>>>
>> >>>>> Big +1, it should be easy to set up a nightly that relaxes some of
>> the
>> >>>>> requirements and sees what (if anything) breaks. Not breaking is
>> >>>>> likely a signal that we should relax ours.
>> >>>>
>> >>>>
>> >>>> Filed https://issues.apache.org/jira/browse/BEAM-3363 to track
>> this. I
>> >>>> think it would be awesome if we can tackle this as part of a better
>> >>>> infrastructure for testing work.
>> >>>>
>> >>>>>
>> >>>>>
>> >>>>> >> On Fri, Dec 15, 2017 at 1:33 PM, Chamikara Jayalath (JIRA)
>> >>>>> >> <ji...@apache.org> wrote:
>> >>>>> >> >
>> >>>>> >> > [
>> >>>>> >> >
>> >>>>> >> >
>> https://issues.apache.org/jira/browse/BEAM-3357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16293276#comment-16293276
>> >>>>> >> > ]
>> >>>>> >> >
>> >>>>> >> > Chamikara Jayalath commented on BEAM-3357:
>> >>>>> >> > ------------------------------------------
>> >>>>> >> >
>> >>>>> >> > You mean we could bump up the upper bound ? I think we should
>> keep
>> >>>>> >> > some
>> >>>>> >> > upper bound in case we get badly broken by a future protobuf
>> >>>>> >> > release.
>> >>>>> >> >
>> >>>>> >> >> Python SDK head fails to run tests due to
>> >>>>> >> >> Requirement.parse('protobuf<=3.4.0,>=3.2.0')
>> >>>>> >> >>
>> >>>>> >> >>
>> >>>>> >> >>
>> --------------------------------------------------------------------------------------
>> >>>>> >> >>
>> >>>>> >> >> Key: BEAM-3357
>> >>>>> >> >> URL:
>> >>>>> >> >> https://issues.apache.org/jira/browse/BEAM-3357
>> >>>>> >> >> Project: Beam
>> >>>>> >> >> Issue Type: Bug
>> >>>>> >> >> Components: sdk-py-core
>> >>>>> >> >> Reporter: Chamikara Jayalath
>> >>>>> >> >> Priority: Critical
>> >>>>> >> >>
>> >>>>> >> >> Error is:
>> >>>>> >> >> running build_ext
>> >>>>> >> >> Traceback (most recent call last):
>> >>>>> >> >> File "setup.py", line 202, in <module>
>> >>>>> >> >> 'test': generate_protos_first(test),
>> >>>>> >> >> File
>> >>>>> >> >>
>> >>>>> >> >>
>> "/Users/chamikara/testing/test_py_12_14_2017_2/env_proto_3.4/lib/python2.7/site-packages/setuptools/__init__.py",
>> >>>>> >> >> line 129, in setup
>> >>>>> >> >> return distutils.core.setup(**attrs)
>> >>>>> >> >> File
>> >>>>> >> >>
>> >>>>> >> >>
>> "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py",
>> >>>>> >> >> line 151, in setup
>> >>>>> >> >> dist.run_commands()
>> >>>>> >> >> File
>> >>>>> >> >>
>> >>>>> >> >>
>> "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py",
>> >>>>> >> >> line 953, in run_commands
>> >>>>> >> >> self.run_command(cmd)
>> >>>>> >> >> File
>> >>>>> >> >>
>> >>>>> >> >>
>> "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py",
>> >>>>> >> >> line 972, in run_command
>> >>>>> >> >> cmd_obj.run()
>> >>>>> >> >> File "setup.py", line 142, in run
>> >>>>> >> >> super(cmd, self).run()
>> >>>>> >> >> File
>> >>>>> >> >>
>> >>>>> >> >>
>> "/Users/chamikara/testing/test_py_12_14_2017_2/env_proto_3.4/lib/python2.7/site-packages/setuptools/command/test.py",
>> >>>>> >> >> line 225, in run
>> >>>>> >> >> with self.project_on_sys_path():
>> >>>>> >> >> File
>> >>>>> >> >>
>> >>>>> >> >>
>> "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py",
>> >>>>> >> >> line 17, in __enter__
>> >>>>> >> >> return self.gen.next()
>> >>>>> >> >> File
>> >>>>> >> >>
>> >>>>> >> >>
>> "/Users/chamikara/testing/test_py_12_14_2017_2/env_proto_3.4/lib/python2.7/site-packages/setuptools/command/test.py",
>> >>>>> >> >> line 164, in project_on_sys_path
>> >>>>> >> >> require('%s==%s' % (ei_cmd.egg_name, ei_cmd.egg_version))
>> >>>>> >> >> File
>> >>>>> >> >>
>> >>>>> >> >>
>> "/Users/chamikara/testing/test_py_12_14_2017_2/env_proto_3.4/lib/python2.7/site-packages/pkg_resources/__init__.py",
>> >>>>> >> >> line 984, in require
>> >>>>> >> >> needed = self.resolve(parse_requirements(requirements))
>> >>>>> >> >> File
>> >>>>> >> >>
>> >>>>> >> >>
>> "/Users/chamikara/testing/test_py_12_14_2017_2/env_proto_3.4/lib/python2.7/site-packages/pkg_resources/__init__.py",
>> >>>>> >> >> line 875, in resolve
>> >>>>> >> >> raise VersionConflict(dist,
>> req).with_context(dependent_req)
>> >>>>> >> >> pkg_resources.ContextualVersionConflict: (protobuf 3.5.0.post1
>> >>>>> >> >>
>> >>>>> >> >>
>> (/Users/chamikara/testing/test_py_12_14_2017_2/beam/sdks/python/.eggs/protobuf-3.5.0.post1-py2.7.egg),
>> >>>>> >> >> Requirement.parse('protobuf<=3.4.0,>=3.2.0'),
>> >>>>> >> >> set(['apache-beam']))
>> >>>>> >> >> Seems like grpcio did a release today which is breaking us:
>> >>>>> >> >> https://pypi.python.org/pypi/grpcio/1.8.1
>> >>>>> >> >> We have to either bump our protobuf dependency or reduce the
>> >>>>> >> >> upper
>> >>>>> >> >> bound of grpcio dependency to previous release (1.7.3).
>> >>>>> >> >
>> >>>>> >> >
>> >>>>> >> >
>> >>>>> >> > --
>> >>>>> >> > This message was sent by Atlassian JIRA
>> >>>>> >> > (v6.4.14#64029)
>> >>>>> >
>> >>>>> >
>>
>
>
Re: [jira] [Commented] (BEAM-3357) Python SDK head fails to run tests
due to Requirement.parse('protobuf<=3.4.0,>=3.2.0')
Posted by Ahmet Altay <al...@google.com>.
I agree with Robert. Also there is usually a workaround to unbrake previous
version by installing a specific version of an offending dependency.
On Fri, Dec 15, 2017 at 2:53 PM, Robert Bradshaw <ro...@google.com>
wrote:
> This has the downside of pinning the dependencies for all downstream
> projects, making it impossible for them to use different versions than
> the ones we happened to choose. (Imagine the pain of two or more of
> our dependencies pinned all their dependencies...)
>
> On Fri, Dec 15, 2017 at 2:48 PM, Udi Meiri <eh...@google.com> wrote:
> > +1 to pinning to exact versions, to be sure that our releases do not
> break
> > when newer versions of dependencies are released.
> >
> > On Fri, Dec 15, 2017 at 2:44 PM Ahmet Altay <al...@google.com> wrote:
> >>
> >> On Fri, Dec 15, 2017 at 2:42 PM, Chamikara Jayalath <
> chamikara@google.com>
> >> wrote:
> >>>
> >>> +1 for automating the process of checking for possible version bumps.
> >>>
> >>> Also, what do you think about pinning dependencies to exact versions
> >>> (instead of ranges) after cutting a release branch ? This should
> improve the
> >>> stability of released SDKs (but not a prefect solution since transitive
> >>> dependencies can still change).
> >>
> >>
> >> This is a reasonable suggestion. The issue with that is, by being less
> >> flexible we will prevent users from using latest versions of
> dependencies.
> >> On the other hand it will prevent breaking of already released versions.
> >>
> >>>
> >>>
> >>> Thanks,
> >>> Cham
> >>>
> >>> On Fri, Dec 15, 2017 at 2:19 PM Ahmet Altay <al...@google.com> wrote:
> >>>>
> >>>> On Fri, Dec 15, 2017 at 2:02 PM, Robert Bradshaw <robertwb@google.com
> >
> >>>> wrote:
> >>>>>
> >>>>> On Fri, Dec 15, 2017 at 1:51 PM, Ahmet Altay <al...@google.com>
> wrote:
> >>>>> >
> >>>>> > On Fri, Dec 15, 2017 at 1:38 PM, Robert Bradshaw
> >>>>> > <ro...@google.com>
> >>>>> > wrote:
> >>>>> >>
> >>>>> >> I am also in favor of pinning as an immediate fix, bumping the
> bound
> >>>>> >> otherwise.
> >>>>> >>
> >>>>> >> Regarding putting an upper bound to avoid being broken, the last
> two
> >>>>> >> breaks have been due to just having an (unneeded) upper bound
> (which
> >>>>> >> held us back to broken/incompatible releases in relationship to
> >>>>> >> other
> >>>>> >> dependencies). We should try to trust semantic versioning when
> >>>>> >> possible, and when not we must regularly audit.
> >>>>> >
> >>>>> > +1 to this, especially the auditing part. We also had breaks
> because
> >>>>> > we
> >>>>> > trusted semantic versioning. So far our semi-official policy was to
> >>>>> > trust a
> >>>>> > package until they prove it otherwise. I will argue that grpc here
> is
> >>>>> > making
> >>>>> > a breaking change in a minor version increment by changing the way
> >>>>> > they are
> >>>>> > depending on a major package.
> >>>>>
> >>>>> A minor version bump should be allowed to require a minor version
> bump
> >>>>> in its dependencies.
> >>>>>
> >>>>> > We have done a good job of auditing and updating those pinned (or
> >>>>> > upper
> >>>>> > bounded) dependencies, and probably we are behind in some of those.
> >>>>> >
> >>>>> > I wonder if we can automate some of this? If we can get a report,
> >>>>> > that
> >>>>> > audits our dependencies, warns us about new releases and potential
> >>>>> > conflicts
> >>>>> > it would be much easier to keep things up to date.
> >>>>>
> >>>>> Big +1, it should be easy to set up a nightly that relaxes some of
> the
> >>>>> requirements and sees what (if anything) breaks. Not breaking is
> >>>>> likely a signal that we should relax ours.
> >>>>
> >>>>
> >>>> Filed https://issues.apache.org/jira/browse/BEAM-3363 to track this.
> I
> >>>> think it would be awesome if we can tackle this as part of a better
> >>>> infrastructure for testing work.
> >>>>
> >>>>>
> >>>>>
> >>>>> >> On Fri, Dec 15, 2017 at 1:33 PM, Chamikara Jayalath (JIRA)
> >>>>> >> <ji...@apache.org> wrote:
> >>>>> >> >
> >>>>> >> > [
> >>>>> >> >
> >>>>> >> > https://issues.apache.org/jira/browse/BEAM-3357?page=
> com.atlassian.jira.plugin.system.issuetabpanels:comment-
> tabpanel&focusedCommentId=16293276#comment-16293276
> >>>>> >> > ]
> >>>>> >> >
> >>>>> >> > Chamikara Jayalath commented on BEAM-3357:
> >>>>> >> > ------------------------------------------
> >>>>> >> >
> >>>>> >> > You mean we could bump up the upper bound ? I think we should
> keep
> >>>>> >> > some
> >>>>> >> > upper bound in case we get badly broken by a future protobuf
> >>>>> >> > release.
> >>>>> >> >
> >>>>> >> >> Python SDK head fails to run tests due to
> >>>>> >> >> Requirement.parse('protobuf<=3.4.0,>=3.2.0')
> >>>>> >> >>
> >>>>> >> >>
> >>>>> >> >> ------------------------------------------------------------
> --------------------------
> >>>>> >> >>
> >>>>> >> >> Key: BEAM-3357
> >>>>> >> >> URL:
> >>>>> >> >> https://issues.apache.org/jira/browse/BEAM-3357
> >>>>> >> >> Project: Beam
> >>>>> >> >> Issue Type: Bug
> >>>>> >> >> Components: sdk-py-core
> >>>>> >> >> Reporter: Chamikara Jayalath
> >>>>> >> >> Priority: Critical
> >>>>> >> >>
> >>>>> >> >> Error is:
> >>>>> >> >> running build_ext
> >>>>> >> >> Traceback (most recent call last):
> >>>>> >> >> File "setup.py", line 202, in <module>
> >>>>> >> >> 'test': generate_protos_first(test),
> >>>>> >> >> File
> >>>>> >> >>
> >>>>> >> >> "/Users/chamikara/testing/test_py_12_14_2017_2/env_
> proto_3.4/lib/python2.7/site-packages/setuptools/__init__.py",
> >>>>> >> >> line 129, in setup
> >>>>> >> >> return distutils.core.setup(**attrs)
> >>>>> >> >> File
> >>>>> >> >>
> >>>>> >> >> "/System/Library/Frameworks/Python.framework/Versions/2.7/
> lib/python2.7/distutils/core.py",
> >>>>> >> >> line 151, in setup
> >>>>> >> >> dist.run_commands()
> >>>>> >> >> File
> >>>>> >> >>
> >>>>> >> >> "/System/Library/Frameworks/Python.framework/Versions/2.7/
> lib/python2.7/distutils/dist.py",
> >>>>> >> >> line 953, in run_commands
> >>>>> >> >> self.run_command(cmd)
> >>>>> >> >> File
> >>>>> >> >>
> >>>>> >> >> "/System/Library/Frameworks/Python.framework/Versions/2.7/
> lib/python2.7/distutils/dist.py",
> >>>>> >> >> line 972, in run_command
> >>>>> >> >> cmd_obj.run()
> >>>>> >> >> File "setup.py", line 142, in run
> >>>>> >> >> super(cmd, self).run()
> >>>>> >> >> File
> >>>>> >> >>
> >>>>> >> >> "/Users/chamikara/testing/test_py_12_14_2017_2/env_
> proto_3.4/lib/python2.7/site-packages/setuptools/command/test.py",
> >>>>> >> >> line 225, in run
> >>>>> >> >> with self.project_on_sys_path():
> >>>>> >> >> File
> >>>>> >> >>
> >>>>> >> >> "/System/Library/Frameworks/Python.framework/Versions/2.7/
> lib/python2.7/contextlib.py",
> >>>>> >> >> line 17, in __enter__
> >>>>> >> >> return self.gen.next()
> >>>>> >> >> File
> >>>>> >> >>
> >>>>> >> >> "/Users/chamikara/testing/test_py_12_14_2017_2/env_
> proto_3.4/lib/python2.7/site-packages/setuptools/command/test.py",
> >>>>> >> >> line 164, in project_on_sys_path
> >>>>> >> >> require('%s==%s' % (ei_cmd.egg_name, ei_cmd.egg_version))
> >>>>> >> >> File
> >>>>> >> >>
> >>>>> >> >> "/Users/chamikara/testing/test_py_12_14_2017_2/env_
> proto_3.4/lib/python2.7/site-packages/pkg_resources/__init__.py",
> >>>>> >> >> line 984, in require
> >>>>> >> >> needed = self.resolve(parse_requirements(requirements))
> >>>>> >> >> File
> >>>>> >> >>
> >>>>> >> >> "/Users/chamikara/testing/test_py_12_14_2017_2/env_
> proto_3.4/lib/python2.7/site-packages/pkg_resources/__init__.py",
> >>>>> >> >> line 875, in resolve
> >>>>> >> >> raise VersionConflict(dist, req).with_context(dependent_
> req)
> >>>>> >> >> pkg_resources.ContextualVersionConflict: (protobuf 3.5.0.post1
> >>>>> >> >>
> >>>>> >> >> (/Users/chamikara/testing/test_py_12_14_2017_2/beam/
> sdks/python/.eggs/protobuf-3.5.0.post1-py2.7.egg),
> >>>>> >> >> Requirement.parse('protobuf<=3.4.0,>=3.2.0'),
> >>>>> >> >> set(['apache-beam']))
> >>>>> >> >> Seems like grpcio did a release today which is breaking us:
> >>>>> >> >> https://pypi.python.org/pypi/grpcio/1.8.1
> >>>>> >> >> We have to either bump our protobuf dependency or reduce the
> >>>>> >> >> upper
> >>>>> >> >> bound of grpcio dependency to previous release (1.7.3).
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > --
> >>>>> >> > This message was sent by Atlassian JIRA
> >>>>> >> > (v6.4.14#64029)
> >>>>> >
> >>>>> >
>
Re: [jira] [Commented] (BEAM-3357) Python SDK head fails to run tests
due to Requirement.parse('protobuf<=3.4.0,>=3.2.0')
Posted by Robert Bradshaw <ro...@google.com>.
This has the downside of pinning the dependencies for all downstream
projects, making it impossible for them to use different versions than
the ones we happened to choose. (Imagine the pain of two or more of
our dependencies pinned all their dependencies...)
On Fri, Dec 15, 2017 at 2:48 PM, Udi Meiri <eh...@google.com> wrote:
> +1 to pinning to exact versions, to be sure that our releases do not break
> when newer versions of dependencies are released.
>
> On Fri, Dec 15, 2017 at 2:44 PM Ahmet Altay <al...@google.com> wrote:
>>
>> On Fri, Dec 15, 2017 at 2:42 PM, Chamikara Jayalath <ch...@google.com>
>> wrote:
>>>
>>> +1 for automating the process of checking for possible version bumps.
>>>
>>> Also, what do you think about pinning dependencies to exact versions
>>> (instead of ranges) after cutting a release branch ? This should improve the
>>> stability of released SDKs (but not a prefect solution since transitive
>>> dependencies can still change).
>>
>>
>> This is a reasonable suggestion. The issue with that is, by being less
>> flexible we will prevent users from using latest versions of dependencies.
>> On the other hand it will prevent breaking of already released versions.
>>
>>>
>>>
>>> Thanks,
>>> Cham
>>>
>>> On Fri, Dec 15, 2017 at 2:19 PM Ahmet Altay <al...@google.com> wrote:
>>>>
>>>> On Fri, Dec 15, 2017 at 2:02 PM, Robert Bradshaw <ro...@google.com>
>>>> wrote:
>>>>>
>>>>> On Fri, Dec 15, 2017 at 1:51 PM, Ahmet Altay <al...@google.com> wrote:
>>>>> >
>>>>> > On Fri, Dec 15, 2017 at 1:38 PM, Robert Bradshaw
>>>>> > <ro...@google.com>
>>>>> > wrote:
>>>>> >>
>>>>> >> I am also in favor of pinning as an immediate fix, bumping the bound
>>>>> >> otherwise.
>>>>> >>
>>>>> >> Regarding putting an upper bound to avoid being broken, the last two
>>>>> >> breaks have been due to just having an (unneeded) upper bound (which
>>>>> >> held us back to broken/incompatible releases in relationship to
>>>>> >> other
>>>>> >> dependencies). We should try to trust semantic versioning when
>>>>> >> possible, and when not we must regularly audit.
>>>>> >
>>>>> > +1 to this, especially the auditing part. We also had breaks because
>>>>> > we
>>>>> > trusted semantic versioning. So far our semi-official policy was to
>>>>> > trust a
>>>>> > package until they prove it otherwise. I will argue that grpc here is
>>>>> > making
>>>>> > a breaking change in a minor version increment by changing the way
>>>>> > they are
>>>>> > depending on a major package.
>>>>>
>>>>> A minor version bump should be allowed to require a minor version bump
>>>>> in its dependencies.
>>>>>
>>>>> > We have done a good job of auditing and updating those pinned (or
>>>>> > upper
>>>>> > bounded) dependencies, and probably we are behind in some of those.
>>>>> >
>>>>> > I wonder if we can automate some of this? If we can get a report,
>>>>> > that
>>>>> > audits our dependencies, warns us about new releases and potential
>>>>> > conflicts
>>>>> > it would be much easier to keep things up to date.
>>>>>
>>>>> Big +1, it should be easy to set up a nightly that relaxes some of the
>>>>> requirements and sees what (if anything) breaks. Not breaking is
>>>>> likely a signal that we should relax ours.
>>>>
>>>>
>>>> Filed https://issues.apache.org/jira/browse/BEAM-3363 to track this. I
>>>> think it would be awesome if we can tackle this as part of a better
>>>> infrastructure for testing work.
>>>>
>>>>>
>>>>>
>>>>> >> On Fri, Dec 15, 2017 at 1:33 PM, Chamikara Jayalath (JIRA)
>>>>> >> <ji...@apache.org> wrote:
>>>>> >> >
>>>>> >> > [
>>>>> >> >
>>>>> >> > https://issues.apache.org/jira/browse/BEAM-3357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16293276#comment-16293276
>>>>> >> > ]
>>>>> >> >
>>>>> >> > Chamikara Jayalath commented on BEAM-3357:
>>>>> >> > ------------------------------------------
>>>>> >> >
>>>>> >> > You mean we could bump up the upper bound ? I think we should keep
>>>>> >> > some
>>>>> >> > upper bound in case we get badly broken by a future protobuf
>>>>> >> > release.
>>>>> >> >
>>>>> >> >> Python SDK head fails to run tests due to
>>>>> >> >> Requirement.parse('protobuf<=3.4.0,>=3.2.0')
>>>>> >> >>
>>>>> >> >>
>>>>> >> >> --------------------------------------------------------------------------------------
>>>>> >> >>
>>>>> >> >> Key: BEAM-3357
>>>>> >> >> URL:
>>>>> >> >> https://issues.apache.org/jira/browse/BEAM-3357
>>>>> >> >> Project: Beam
>>>>> >> >> Issue Type: Bug
>>>>> >> >> Components: sdk-py-core
>>>>> >> >> Reporter: Chamikara Jayalath
>>>>> >> >> Priority: Critical
>>>>> >> >>
>>>>> >> >> Error is:
>>>>> >> >> running build_ext
>>>>> >> >> Traceback (most recent call last):
>>>>> >> >> File "setup.py", line 202, in <module>
>>>>> >> >> 'test': generate_protos_first(test),
>>>>> >> >> File
>>>>> >> >>
>>>>> >> >> "/Users/chamikara/testing/test_py_12_14_2017_2/env_proto_3.4/lib/python2.7/site-packages/setuptools/__init__.py",
>>>>> >> >> line 129, in setup
>>>>> >> >> return distutils.core.setup(**attrs)
>>>>> >> >> File
>>>>> >> >>
>>>>> >> >> "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py",
>>>>> >> >> line 151, in setup
>>>>> >> >> dist.run_commands()
>>>>> >> >> File
>>>>> >> >>
>>>>> >> >> "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py",
>>>>> >> >> line 953, in run_commands
>>>>> >> >> self.run_command(cmd)
>>>>> >> >> File
>>>>> >> >>
>>>>> >> >> "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py",
>>>>> >> >> line 972, in run_command
>>>>> >> >> cmd_obj.run()
>>>>> >> >> File "setup.py", line 142, in run
>>>>> >> >> super(cmd, self).run()
>>>>> >> >> File
>>>>> >> >>
>>>>> >> >> "/Users/chamikara/testing/test_py_12_14_2017_2/env_proto_3.4/lib/python2.7/site-packages/setuptools/command/test.py",
>>>>> >> >> line 225, in run
>>>>> >> >> with self.project_on_sys_path():
>>>>> >> >> File
>>>>> >> >>
>>>>> >> >> "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py",
>>>>> >> >> line 17, in __enter__
>>>>> >> >> return self.gen.next()
>>>>> >> >> File
>>>>> >> >>
>>>>> >> >> "/Users/chamikara/testing/test_py_12_14_2017_2/env_proto_3.4/lib/python2.7/site-packages/setuptools/command/test.py",
>>>>> >> >> line 164, in project_on_sys_path
>>>>> >> >> require('%s==%s' % (ei_cmd.egg_name, ei_cmd.egg_version))
>>>>> >> >> File
>>>>> >> >>
>>>>> >> >> "/Users/chamikara/testing/test_py_12_14_2017_2/env_proto_3.4/lib/python2.7/site-packages/pkg_resources/__init__.py",
>>>>> >> >> line 984, in require
>>>>> >> >> needed = self.resolve(parse_requirements(requirements))
>>>>> >> >> File
>>>>> >> >>
>>>>> >> >> "/Users/chamikara/testing/test_py_12_14_2017_2/env_proto_3.4/lib/python2.7/site-packages/pkg_resources/__init__.py",
>>>>> >> >> line 875, in resolve
>>>>> >> >> raise VersionConflict(dist, req).with_context(dependent_req)
>>>>> >> >> pkg_resources.ContextualVersionConflict: (protobuf 3.5.0.post1
>>>>> >> >>
>>>>> >> >> (/Users/chamikara/testing/test_py_12_14_2017_2/beam/sdks/python/.eggs/protobuf-3.5.0.post1-py2.7.egg),
>>>>> >> >> Requirement.parse('protobuf<=3.4.0,>=3.2.0'),
>>>>> >> >> set(['apache-beam']))
>>>>> >> >> Seems like grpcio did a release today which is breaking us:
>>>>> >> >> https://pypi.python.org/pypi/grpcio/1.8.1
>>>>> >> >> We have to either bump our protobuf dependency or reduce the
>>>>> >> >> upper
>>>>> >> >> bound of grpcio dependency to previous release (1.7.3).
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> > --
>>>>> >> > This message was sent by Atlassian JIRA
>>>>> >> > (v6.4.14#64029)
>>>>> >
>>>>> >
Re: [jira] [Commented] (BEAM-3357) Python SDK head fails to run tests
due to Requirement.parse('protobuf<=3.4.0,>=3.2.0')
Posted by Udi Meiri <eh...@google.com>.
+1 to pinning to exact versions, to be sure that our releases do not break
when newer versions of dependencies are released.
On Fri, Dec 15, 2017 at 2:44 PM Ahmet Altay <al...@google.com> wrote:
> On Fri, Dec 15, 2017 at 2:42 PM, Chamikara Jayalath <ch...@google.com>
> wrote:
>
>> +1 for automating the process of checking for possible version bumps.
>>
>> Also, what do you think about pinning dependencies to exact versions
>> (instead of ranges) after cutting a release branch ? This should improve
>> the stability of released SDKs (but not a prefect solution since transitive
>> dependencies can still change).
>>
>
> This is a reasonable suggestion. The issue with that is, by being less
> flexible we will prevent users from using latest versions of dependencies.
> On the other hand it will prevent breaking of already released versions.
>
>
>>
>> Thanks,
>> Cham
>>
>> On Fri, Dec 15, 2017 at 2:19 PM Ahmet Altay <al...@google.com> wrote:
>>
>>> On Fri, Dec 15, 2017 at 2:02 PM, Robert Bradshaw <ro...@google.com>
>>> wrote:
>>>
>>>> On Fri, Dec 15, 2017 at 1:51 PM, Ahmet Altay <al...@google.com> wrote:
>>>> >
>>>> > On Fri, Dec 15, 2017 at 1:38 PM, Robert Bradshaw <robertwb@google.com
>>>> >
>>>> > wrote:
>>>> >>
>>>> >> I am also in favor of pinning as an immediate fix, bumping the bound
>>>> >> otherwise.
>>>> >>
>>>> >> Regarding putting an upper bound to avoid being broken, the last two
>>>> >> breaks have been due to just having an (unneeded) upper bound (which
>>>> >> held us back to broken/incompatible releases in relationship to other
>>>> >> dependencies). We should try to trust semantic versioning when
>>>> >> possible, and when not we must regularly audit.
>>>> >
>>>> > +1 to this, especially the auditing part. We also had breaks because
>>>> we
>>>> > trusted semantic versioning. So far our semi-official policy was to
>>>> trust a
>>>> > package until they prove it otherwise. I will argue that grpc here is
>>>> making
>>>> > a breaking change in a minor version increment by changing the way
>>>> they are
>>>> > depending on a major package.
>>>>
>>>> A minor version bump should be allowed to require a minor version bump
>>>> in its dependencies.
>>>>
>>>> > We have done a good job of auditing and updating those pinned (or
>>>> upper
>>>> > bounded) dependencies, and probably we are behind in some of those.
>>>> >
>>>> > I wonder if we can automate some of this? If we can get a report, that
>>>> > audits our dependencies, warns us about new releases and potential
>>>> conflicts
>>>> > it would be much easier to keep things up to date.
>>>>
>>>> Big +1, it should be easy to set up a nightly that relaxes some of the
>>>> requirements and sees what (if anything) breaks. Not breaking is
>>>> likely a signal that we should relax ours.
>>>>
>>>
>>> Filed https://issues.apache.org/jira/browse/BEAM-3363 to track this. I
>>> think it would be awesome if we can tackle this as part of a better
>>> infrastructure for testing work.
>>>
>>>
>>>>
>>>> >> On Fri, Dec 15, 2017 at 1:33 PM, Chamikara Jayalath (JIRA)
>>>> >> <ji...@apache.org> wrote:
>>>> >> >
>>>> >> > [
>>>> >> >
>>>> https://issues.apache.org/jira/browse/BEAM-3357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16293276#comment-16293276
>>>> >> > ]
>>>> >> >
>>>> >> > Chamikara Jayalath commented on BEAM-3357:
>>>> >> > ------------------------------------------
>>>> >> >
>>>> >> > You mean we could bump up the upper bound ? I think we should keep
>>>> some
>>>> >> > upper bound in case we get badly broken by a future protobuf
>>>> release.
>>>> >> >
>>>> >> >> Python SDK head fails to run tests due to
>>>> >> >> Requirement.parse('protobuf<=3.4.0,>=3.2.0')
>>>> >> >>
>>>> >> >>
>>>> --------------------------------------------------------------------------------------
>>>> >> >>
>>>> >> >> Key: BEAM-3357
>>>> >> >> URL:
>>>> https://issues.apache.org/jira/browse/BEAM-3357
>>>> >> >> Project: Beam
>>>> >> >> Issue Type: Bug
>>>> >> >> Components: sdk-py-core
>>>> >> >> Reporter: Chamikara Jayalath
>>>> >> >> Priority: Critical
>>>> >> >>
>>>> >> >> Error is:
>>>> >> >> running build_ext
>>>> >> >> Traceback (most recent call last):
>>>> >> >> File "setup.py", line 202, in <module>
>>>> >> >> 'test': generate_protos_first(test),
>>>> >> >> File
>>>> >> >>
>>>> "/Users/chamikara/testing/test_py_12_14_2017_2/env_proto_3.4/lib/python2.7/site-packages/setuptools/__init__.py",
>>>> >> >> line 129, in setup
>>>> >> >> return distutils.core.setup(**attrs)
>>>> >> >> File
>>>> >> >>
>>>> "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py",
>>>> >> >> line 151, in setup
>>>> >> >> dist.run_commands()
>>>> >> >> File
>>>> >> >>
>>>> "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py",
>>>> >> >> line 953, in run_commands
>>>> >> >> self.run_command(cmd)
>>>> >> >> File
>>>> >> >>
>>>> "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py",
>>>> >> >> line 972, in run_command
>>>> >> >> cmd_obj.run()
>>>> >> >> File "setup.py", line 142, in run
>>>> >> >> super(cmd, self).run()
>>>> >> >> File
>>>> >> >>
>>>> "/Users/chamikara/testing/test_py_12_14_2017_2/env_proto_3.4/lib/python2.7/site-packages/setuptools/command/test.py",
>>>> >> >> line 225, in run
>>>> >> >> with self.project_on_sys_path():
>>>> >> >> File
>>>> >> >>
>>>> "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py",
>>>> >> >> line 17, in __enter__
>>>> >> >> return self.gen.next()
>>>> >> >> File
>>>> >> >>
>>>> "/Users/chamikara/testing/test_py_12_14_2017_2/env_proto_3.4/lib/python2.7/site-packages/setuptools/command/test.py",
>>>> >> >> line 164, in project_on_sys_path
>>>> >> >> require('%s==%s' % (ei_cmd.egg_name, ei_cmd.egg_version))
>>>> >> >> File
>>>> >> >>
>>>> "/Users/chamikara/testing/test_py_12_14_2017_2/env_proto_3.4/lib/python2.7/site-packages/pkg_resources/__init__.py",
>>>> >> >> line 984, in require
>>>> >> >> needed = self.resolve(parse_requirements(requirements))
>>>> >> >> File
>>>> >> >>
>>>> "/Users/chamikara/testing/test_py_12_14_2017_2/env_proto_3.4/lib/python2.7/site-packages/pkg_resources/__init__.py",
>>>> >> >> line 875, in resolve
>>>> >> >> raise VersionConflict(dist, req).with_context(dependent_req)
>>>> >> >> pkg_resources.ContextualVersionConflict: (protobuf 3.5.0.post1
>>>> >> >>
>>>> (/Users/chamikara/testing/test_py_12_14_2017_2/beam/sdks/python/.eggs/protobuf-3.5.0.post1-py2.7.egg),
>>>> >> >> Requirement.parse('protobuf<=3.4.0,>=3.2.0'),
>>>> set(['apache-beam']))
>>>> >> >> Seems like grpcio did a release today which is breaking us:
>>>> >> >> https://pypi.python.org/pypi/grpcio/1.8.1
>>>> >> >> We have to either bump our protobuf dependency or reduce the upper
>>>> >> >> bound of grpcio dependency to previous release (1.7.3).
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > --
>>>> >> > This message was sent by Atlassian JIRA
>>>> >> > (v6.4.14#64029)
>>>> >
>>>> >
>>>>
>>>
Re: [jira] [Commented] (BEAM-3357) Python SDK head fails to run tests
due to Requirement.parse('protobuf<=3.4.0,>=3.2.0')
Posted by Ahmet Altay <al...@google.com>.
On Fri, Dec 15, 2017 at 2:42 PM, Chamikara Jayalath <ch...@google.com>
wrote:
> +1 for automating the process of checking for possible version bumps.
>
> Also, what do you think about pinning dependencies to exact versions
> (instead of ranges) after cutting a release branch ? This should improve
> the stability of released SDKs (but not a prefect solution since transitive
> dependencies can still change).
>
This is a reasonable suggestion. The issue with that is, by being less
flexible we will prevent users from using latest versions of dependencies.
On the other hand it will prevent breaking of already released versions.
>
> Thanks,
> Cham
>
> On Fri, Dec 15, 2017 at 2:19 PM Ahmet Altay <al...@google.com> wrote:
>
>> On Fri, Dec 15, 2017 at 2:02 PM, Robert Bradshaw <ro...@google.com>
>> wrote:
>>
>>> On Fri, Dec 15, 2017 at 1:51 PM, Ahmet Altay <al...@google.com> wrote:
>>> >
>>> > On Fri, Dec 15, 2017 at 1:38 PM, Robert Bradshaw <ro...@google.com>
>>> > wrote:
>>> >>
>>> >> I am also in favor of pinning as an immediate fix, bumping the bound
>>> >> otherwise.
>>> >>
>>> >> Regarding putting an upper bound to avoid being broken, the last two
>>> >> breaks have been due to just having an (unneeded) upper bound (which
>>> >> held us back to broken/incompatible releases in relationship to other
>>> >> dependencies). We should try to trust semantic versioning when
>>> >> possible, and when not we must regularly audit.
>>> >
>>> > +1 to this, especially the auditing part. We also had breaks because we
>>> > trusted semantic versioning. So far our semi-official policy was to
>>> trust a
>>> > package until they prove it otherwise. I will argue that grpc here is
>>> making
>>> > a breaking change in a minor version increment by changing the way
>>> they are
>>> > depending on a major package.
>>>
>>> A minor version bump should be allowed to require a minor version bump
>>> in its dependencies.
>>>
>>> > We have done a good job of auditing and updating those pinned (or upper
>>> > bounded) dependencies, and probably we are behind in some of those.
>>> >
>>> > I wonder if we can automate some of this? If we can get a report, that
>>> > audits our dependencies, warns us about new releases and potential
>>> conflicts
>>> > it would be much easier to keep things up to date.
>>>
>>> Big +1, it should be easy to set up a nightly that relaxes some of the
>>> requirements and sees what (if anything) breaks. Not breaking is
>>> likely a signal that we should relax ours.
>>>
>>
>> Filed https://issues.apache.org/jira/browse/BEAM-3363 to track this. I
>> think it would be awesome if we can tackle this as part of a better
>> infrastructure for testing work.
>>
>>
>>>
>>> >> On Fri, Dec 15, 2017 at 1:33 PM, Chamikara Jayalath (JIRA)
>>> >> <ji...@apache.org> wrote:
>>> >> >
>>> >> > [
>>> >> > https://issues.apache.org/jira/browse/BEAM-3357?page=
>>> com.atlassian.jira.plugin.system.issuetabpanels:comment-
>>> tabpanel&focusedCommentId=16293276#comment-16293276
>>> >> > ]
>>> >> >
>>> >> > Chamikara Jayalath commented on BEAM-3357:
>>> >> > ------------------------------------------
>>> >> >
>>> >> > You mean we could bump up the upper bound ? I think we should keep
>>> some
>>> >> > upper bound in case we get badly broken by a future protobuf
>>> release.
>>> >> >
>>> >> >> Python SDK head fails to run tests due to
>>> >> >> Requirement.parse('protobuf<=3.4.0,>=3.2.0')
>>> >> >>
>>> >> >> ------------------------------------------------------------
>>> --------------------------
>>> >> >>
>>> >> >> Key: BEAM-3357
>>> >> >> URL: https://issues.apache.org/
>>> jira/browse/BEAM-3357
>>> >> >> Project: Beam
>>> >> >> Issue Type: Bug
>>> >> >> Components: sdk-py-core
>>> >> >> Reporter: Chamikara Jayalath
>>> >> >> Priority: Critical
>>> >> >>
>>> >> >> Error is:
>>> >> >> running build_ext
>>> >> >> Traceback (most recent call last):
>>> >> >> File "setup.py", line 202, in <module>
>>> >> >> 'test': generate_protos_first(test),
>>> >> >> File
>>> >> >> "/Users/chamikara/testing/test_py_12_14_2017_2/env_
>>> proto_3.4/lib/python2.7/site-packages/setuptools/__init__.py",
>>> >> >> line 129, in setup
>>> >> >> return distutils.core.setup(**attrs)
>>> >> >> File
>>> >> >> "/System/Library/Frameworks/Python.framework/Versions/2.7/
>>> lib/python2.7/distutils/core.py",
>>> >> >> line 151, in setup
>>> >> >> dist.run_commands()
>>> >> >> File
>>> >> >> "/System/Library/Frameworks/Python.framework/Versions/2.7/
>>> lib/python2.7/distutils/dist.py",
>>> >> >> line 953, in run_commands
>>> >> >> self.run_command(cmd)
>>> >> >> File
>>> >> >> "/System/Library/Frameworks/Python.framework/Versions/2.7/
>>> lib/python2.7/distutils/dist.py",
>>> >> >> line 972, in run_command
>>> >> >> cmd_obj.run()
>>> >> >> File "setup.py", line 142, in run
>>> >> >> super(cmd, self).run()
>>> >> >> File
>>> >> >> "/Users/chamikara/testing/test_py_12_14_2017_2/env_
>>> proto_3.4/lib/python2.7/site-packages/setuptools/command/test.py",
>>> >> >> line 225, in run
>>> >> >> with self.project_on_sys_path():
>>> >> >> File
>>> >> >> "/System/Library/Frameworks/Python.framework/Versions/2.7/
>>> lib/python2.7/contextlib.py",
>>> >> >> line 17, in __enter__
>>> >> >> return self.gen.next()
>>> >> >> File
>>> >> >> "/Users/chamikara/testing/test_py_12_14_2017_2/env_
>>> proto_3.4/lib/python2.7/site-packages/setuptools/command/test.py",
>>> >> >> line 164, in project_on_sys_path
>>> >> >> require('%s==%s' % (ei_cmd.egg_name, ei_cmd.egg_version))
>>> >> >> File
>>> >> >> "/Users/chamikara/testing/test_py_12_14_2017_2/env_
>>> proto_3.4/lib/python2.7/site-packages/pkg_resources/__init__.py",
>>> >> >> line 984, in require
>>> >> >> needed = self.resolve(parse_requirements(requirements))
>>> >> >> File
>>> >> >> "/Users/chamikara/testing/test_py_12_14_2017_2/env_
>>> proto_3.4/lib/python2.7/site-packages/pkg_resources/__init__.py",
>>> >> >> line 875, in resolve
>>> >> >> raise VersionConflict(dist, req).with_context(dependent_req)
>>> >> >> pkg_resources.ContextualVersionConflict: (protobuf 3.5.0.post1
>>> >> >> (/Users/chamikara/testing/test_py_12_14_2017_2/beam/
>>> sdks/python/.eggs/protobuf-3.5.0.post1-py2.7.egg),
>>> >> >> Requirement.parse('protobuf<=3.4.0,>=3.2.0'),
>>> set(['apache-beam']))
>>> >> >> Seems like grpcio did a release today which is breaking us:
>>> >> >> https://pypi.python.org/pypi/grpcio/1.8.1
>>> >> >> We have to either bump our protobuf dependency or reduce the upper
>>> >> >> bound of grpcio dependency to previous release (1.7.3).
>>> >> >
>>> >> >
>>> >> >
>>> >> > --
>>> >> > This message was sent by Atlassian JIRA
>>> >> > (v6.4.14#64029)
>>> >
>>> >
>>>
>>
Re: [jira] [Commented] (BEAM-3357) Python SDK head fails to run tests
due to Requirement.parse('protobuf<=3.4.0,>=3.2.0')
Posted by Chamikara Jayalath <ch...@google.com>.
+1 for automating the process of checking for possible version bumps.
Also, what do you think about pinning dependencies to exact versions
(instead of ranges) after cutting a release branch ? This should improve
the stability of released SDKs (but not a prefect solution since transitive
dependencies can still change).
Thanks,
Cham
On Fri, Dec 15, 2017 at 2:19 PM Ahmet Altay <al...@google.com> wrote:
> On Fri, Dec 15, 2017 at 2:02 PM, Robert Bradshaw <ro...@google.com>
> wrote:
>
>> On Fri, Dec 15, 2017 at 1:51 PM, Ahmet Altay <al...@google.com> wrote:
>> >
>> > On Fri, Dec 15, 2017 at 1:38 PM, Robert Bradshaw <ro...@google.com>
>> > wrote:
>> >>
>> >> I am also in favor of pinning as an immediate fix, bumping the bound
>> >> otherwise.
>> >>
>> >> Regarding putting an upper bound to avoid being broken, the last two
>> >> breaks have been due to just having an (unneeded) upper bound (which
>> >> held us back to broken/incompatible releases in relationship to other
>> >> dependencies). We should try to trust semantic versioning when
>> >> possible, and when not we must regularly audit.
>> >
>> > +1 to this, especially the auditing part. We also had breaks because we
>> > trusted semantic versioning. So far our semi-official policy was to
>> trust a
>> > package until they prove it otherwise. I will argue that grpc here is
>> making
>> > a breaking change in a minor version increment by changing the way they
>> are
>> > depending on a major package.
>>
>> A minor version bump should be allowed to require a minor version bump
>> in its dependencies.
>>
>> > We have done a good job of auditing and updating those pinned (or upper
>> > bounded) dependencies, and probably we are behind in some of those.
>> >
>> > I wonder if we can automate some of this? If we can get a report, that
>> > audits our dependencies, warns us about new releases and potential
>> conflicts
>> > it would be much easier to keep things up to date.
>>
>> Big +1, it should be easy to set up a nightly that relaxes some of the
>> requirements and sees what (if anything) breaks. Not breaking is
>> likely a signal that we should relax ours.
>>
>
> Filed https://issues.apache.org/jira/browse/BEAM-3363 to track this. I
> think it would be awesome if we can tackle this as part of a better
> infrastructure for testing work.
>
>
>>
>> >> On Fri, Dec 15, 2017 at 1:33 PM, Chamikara Jayalath (JIRA)
>> >> <ji...@apache.org> wrote:
>> >> >
>> >> > [
>> >> >
>> https://issues.apache.org/jira/browse/BEAM-3357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16293276#comment-16293276
>> >> > ]
>> >> >
>> >> > Chamikara Jayalath commented on BEAM-3357:
>> >> > ------------------------------------------
>> >> >
>> >> > You mean we could bump up the upper bound ? I think we should keep
>> some
>> >> > upper bound in case we get badly broken by a future protobuf release.
>> >> >
>> >> >> Python SDK head fails to run tests due to
>> >> >> Requirement.parse('protobuf<=3.4.0,>=3.2.0')
>> >> >>
>> >> >>
>> --------------------------------------------------------------------------------------
>> >> >>
>> >> >> Key: BEAM-3357
>> >> >> URL:
>> https://issues.apache.org/jira/browse/BEAM-3357
>> >> >> Project: Beam
>> >> >> Issue Type: Bug
>> >> >> Components: sdk-py-core
>> >> >> Reporter: Chamikara Jayalath
>> >> >> Priority: Critical
>> >> >>
>> >> >> Error is:
>> >> >> running build_ext
>> >> >> Traceback (most recent call last):
>> >> >> File "setup.py", line 202, in <module>
>> >> >> 'test': generate_protos_first(test),
>> >> >> File
>> >> >>
>> "/Users/chamikara/testing/test_py_12_14_2017_2/env_proto_3.4/lib/python2.7/site-packages/setuptools/__init__.py",
>> >> >> line 129, in setup
>> >> >> return distutils.core.setup(**attrs)
>> >> >> File
>> >> >>
>> "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py",
>> >> >> line 151, in setup
>> >> >> dist.run_commands()
>> >> >> File
>> >> >>
>> "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py",
>> >> >> line 953, in run_commands
>> >> >> self.run_command(cmd)
>> >> >> File
>> >> >>
>> "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py",
>> >> >> line 972, in run_command
>> >> >> cmd_obj.run()
>> >> >> File "setup.py", line 142, in run
>> >> >> super(cmd, self).run()
>> >> >> File
>> >> >>
>> "/Users/chamikara/testing/test_py_12_14_2017_2/env_proto_3.4/lib/python2.7/site-packages/setuptools/command/test.py",
>> >> >> line 225, in run
>> >> >> with self.project_on_sys_path():
>> >> >> File
>> >> >>
>> "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py",
>> >> >> line 17, in __enter__
>> >> >> return self.gen.next()
>> >> >> File
>> >> >>
>> "/Users/chamikara/testing/test_py_12_14_2017_2/env_proto_3.4/lib/python2.7/site-packages/setuptools/command/test.py",
>> >> >> line 164, in project_on_sys_path
>> >> >> require('%s==%s' % (ei_cmd.egg_name, ei_cmd.egg_version))
>> >> >> File
>> >> >>
>> "/Users/chamikara/testing/test_py_12_14_2017_2/env_proto_3.4/lib/python2.7/site-packages/pkg_resources/__init__.py",
>> >> >> line 984, in require
>> >> >> needed = self.resolve(parse_requirements(requirements))
>> >> >> File
>> >> >>
>> "/Users/chamikara/testing/test_py_12_14_2017_2/env_proto_3.4/lib/python2.7/site-packages/pkg_resources/__init__.py",
>> >> >> line 875, in resolve
>> >> >> raise VersionConflict(dist, req).with_context(dependent_req)
>> >> >> pkg_resources.ContextualVersionConflict: (protobuf 3.5.0.post1
>> >> >>
>> (/Users/chamikara/testing/test_py_12_14_2017_2/beam/sdks/python/.eggs/protobuf-3.5.0.post1-py2.7.egg),
>> >> >> Requirement.parse('protobuf<=3.4.0,>=3.2.0'), set(['apache-beam']))
>> >> >> Seems like grpcio did a release today which is breaking us:
>> >> >> https://pypi.python.org/pypi/grpcio/1.8.1
>> >> >> We have to either bump our protobuf dependency or reduce the upper
>> >> >> bound of grpcio dependency to previous release (1.7.3).
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > This message was sent by Atlassian JIRA
>> >> > (v6.4.14#64029)
>> >
>> >
>>
>
Re: [jira] [Commented] (BEAM-3357) Python SDK head fails to run tests
due to Requirement.parse('protobuf<=3.4.0,>=3.2.0')
Posted by Ahmet Altay <al...@google.com>.
On Fri, Dec 15, 2017 at 2:02 PM, Robert Bradshaw <ro...@google.com>
wrote:
> On Fri, Dec 15, 2017 at 1:51 PM, Ahmet Altay <al...@google.com> wrote:
> >
> > On Fri, Dec 15, 2017 at 1:38 PM, Robert Bradshaw <ro...@google.com>
> > wrote:
> >>
> >> I am also in favor of pinning as an immediate fix, bumping the bound
> >> otherwise.
> >>
> >> Regarding putting an upper bound to avoid being broken, the last two
> >> breaks have been due to just having an (unneeded) upper bound (which
> >> held us back to broken/incompatible releases in relationship to other
> >> dependencies). We should try to trust semantic versioning when
> >> possible, and when not we must regularly audit.
> >
> > +1 to this, especially the auditing part. We also had breaks because we
> > trusted semantic versioning. So far our semi-official policy was to
> trust a
> > package until they prove it otherwise. I will argue that grpc here is
> making
> > a breaking change in a minor version increment by changing the way they
> are
> > depending on a major package.
>
> A minor version bump should be allowed to require a minor version bump
> in its dependencies.
>
> > We have done a good job of auditing and updating those pinned (or upper
> > bounded) dependencies, and probably we are behind in some of those.
> >
> > I wonder if we can automate some of this? If we can get a report, that
> > audits our dependencies, warns us about new releases and potential
> conflicts
> > it would be much easier to keep things up to date.
>
> Big +1, it should be easy to set up a nightly that relaxes some of the
> requirements and sees what (if anything) breaks. Not breaking is
> likely a signal that we should relax ours.
>
Filed https://issues.apache.org/jira/browse/BEAM-3363 to track this. I
think it would be awesome if we can tackle this as part of a better
infrastructure for testing work.
>
> >> On Fri, Dec 15, 2017 at 1:33 PM, Chamikara Jayalath (JIRA)
> >> <ji...@apache.org> wrote:
> >> >
> >> > [
> >> > https://issues.apache.org/jira/browse/BEAM-3357?page=
> com.atlassian.jira.plugin.system.issuetabpanels:comment-
> tabpanel&focusedCommentId=16293276#comment-16293276
> >> > ]
> >> >
> >> > Chamikara Jayalath commented on BEAM-3357:
> >> > ------------------------------------------
> >> >
> >> > You mean we could bump up the upper bound ? I think we should keep
> some
> >> > upper bound in case we get badly broken by a future protobuf release.
> >> >
> >> >> Python SDK head fails to run tests due to
> >> >> Requirement.parse('protobuf<=3.4.0,>=3.2.0')
> >> >>
> >> >> ------------------------------------------------------------
> --------------------------
> >> >>
> >> >> Key: BEAM-3357
> >> >> URL: https://issues.apache.org/jira/browse/BEAM-3357
> >> >> Project: Beam
> >> >> Issue Type: Bug
> >> >> Components: sdk-py-core
> >> >> Reporter: Chamikara Jayalath
> >> >> Priority: Critical
> >> >>
> >> >> Error is:
> >> >> running build_ext
> >> >> Traceback (most recent call last):
> >> >> File "setup.py", line 202, in <module>
> >> >> 'test': generate_protos_first(test),
> >> >> File
> >> >> "/Users/chamikara/testing/test_py_12_14_2017_2/env_
> proto_3.4/lib/python2.7/site-packages/setuptools/__init__.py",
> >> >> line 129, in setup
> >> >> return distutils.core.setup(**attrs)
> >> >> File
> >> >> "/System/Library/Frameworks/Python.framework/Versions/2.7/
> lib/python2.7/distutils/core.py",
> >> >> line 151, in setup
> >> >> dist.run_commands()
> >> >> File
> >> >> "/System/Library/Frameworks/Python.framework/Versions/2.7/
> lib/python2.7/distutils/dist.py",
> >> >> line 953, in run_commands
> >> >> self.run_command(cmd)
> >> >> File
> >> >> "/System/Library/Frameworks/Python.framework/Versions/2.7/
> lib/python2.7/distutils/dist.py",
> >> >> line 972, in run_command
> >> >> cmd_obj.run()
> >> >> File "setup.py", line 142, in run
> >> >> super(cmd, self).run()
> >> >> File
> >> >> "/Users/chamikara/testing/test_py_12_14_2017_2/env_
> proto_3.4/lib/python2.7/site-packages/setuptools/command/test.py",
> >> >> line 225, in run
> >> >> with self.project_on_sys_path():
> >> >> File
> >> >> "/System/Library/Frameworks/Python.framework/Versions/2.7/
> lib/python2.7/contextlib.py",
> >> >> line 17, in __enter__
> >> >> return self.gen.next()
> >> >> File
> >> >> "/Users/chamikara/testing/test_py_12_14_2017_2/env_
> proto_3.4/lib/python2.7/site-packages/setuptools/command/test.py",
> >> >> line 164, in project_on_sys_path
> >> >> require('%s==%s' % (ei_cmd.egg_name, ei_cmd.egg_version))
> >> >> File
> >> >> "/Users/chamikara/testing/test_py_12_14_2017_2/env_
> proto_3.4/lib/python2.7/site-packages/pkg_resources/__init__.py",
> >> >> line 984, in require
> >> >> needed = self.resolve(parse_requirements(requirements))
> >> >> File
> >> >> "/Users/chamikara/testing/test_py_12_14_2017_2/env_
> proto_3.4/lib/python2.7/site-packages/pkg_resources/__init__.py",
> >> >> line 875, in resolve
> >> >> raise VersionConflict(dist, req).with_context(dependent_req)
> >> >> pkg_resources.ContextualVersionConflict: (protobuf 3.5.0.post1
> >> >> (/Users/chamikara/testing/test_py_12_14_2017_2/beam/
> sdks/python/.eggs/protobuf-3.5.0.post1-py2.7.egg),
> >> >> Requirement.parse('protobuf<=3.4.0,>=3.2.0'), set(['apache-beam']))
> >> >> Seems like grpcio did a release today which is breaking us:
> >> >> https://pypi.python.org/pypi/grpcio/1.8.1
> >> >> We have to either bump our protobuf dependency or reduce the upper
> >> >> bound of grpcio dependency to previous release (1.7.3).
> >> >
> >> >
> >> >
> >> > --
> >> > This message was sent by Atlassian JIRA
> >> > (v6.4.14#64029)
> >
> >
>
Re: [jira] [Commented] (BEAM-3357) Python SDK head fails to run tests
due to Requirement.parse('protobuf<=3.4.0,>=3.2.0')
Posted by Robert Bradshaw <ro...@google.com>.
On Fri, Dec 15, 2017 at 1:51 PM, Ahmet Altay <al...@google.com> wrote:
>
> On Fri, Dec 15, 2017 at 1:38 PM, Robert Bradshaw <ro...@google.com>
> wrote:
>>
>> I am also in favor of pinning as an immediate fix, bumping the bound
>> otherwise.
>>
>> Regarding putting an upper bound to avoid being broken, the last two
>> breaks have been due to just having an (unneeded) upper bound (which
>> held us back to broken/incompatible releases in relationship to other
>> dependencies). We should try to trust semantic versioning when
>> possible, and when not we must regularly audit.
>
> +1 to this, especially the auditing part. We also had breaks because we
> trusted semantic versioning. So far our semi-official policy was to trust a
> package until they prove it otherwise. I will argue that grpc here is making
> a breaking change in a minor version increment by changing the way they are
> depending on a major package.
A minor version bump should be allowed to require a minor version bump
in its dependencies.
> We have done a good job of auditing and updating those pinned (or upper
> bounded) dependencies, and probably we are behind in some of those.
>
> I wonder if we can automate some of this? If we can get a report, that
> audits our dependencies, warns us about new releases and potential conflicts
> it would be much easier to keep things up to date.
Big +1, it should be easy to set up a nightly that relaxes some of the
requirements and sees what (if anything) breaks. Not breaking is
likely a signal that we should relax ours.
>> On Fri, Dec 15, 2017 at 1:33 PM, Chamikara Jayalath (JIRA)
>> <ji...@apache.org> wrote:
>> >
>> > [
>> > https://issues.apache.org/jira/browse/BEAM-3357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16293276#comment-16293276
>> > ]
>> >
>> > Chamikara Jayalath commented on BEAM-3357:
>> > ------------------------------------------
>> >
>> > You mean we could bump up the upper bound ? I think we should keep some
>> > upper bound in case we get badly broken by a future protobuf release.
>> >
>> >> Python SDK head fails to run tests due to
>> >> Requirement.parse('protobuf<=3.4.0,>=3.2.0')
>> >>
>> >> --------------------------------------------------------------------------------------
>> >>
>> >> Key: BEAM-3357
>> >> URL: https://issues.apache.org/jira/browse/BEAM-3357
>> >> Project: Beam
>> >> Issue Type: Bug
>> >> Components: sdk-py-core
>> >> Reporter: Chamikara Jayalath
>> >> Priority: Critical
>> >>
>> >> Error is:
>> >> running build_ext
>> >> Traceback (most recent call last):
>> >> File "setup.py", line 202, in <module>
>> >> 'test': generate_protos_first(test),
>> >> File
>> >> "/Users/chamikara/testing/test_py_12_14_2017_2/env_proto_3.4/lib/python2.7/site-packages/setuptools/__init__.py",
>> >> line 129, in setup
>> >> return distutils.core.setup(**attrs)
>> >> File
>> >> "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py",
>> >> line 151, in setup
>> >> dist.run_commands()
>> >> File
>> >> "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py",
>> >> line 953, in run_commands
>> >> self.run_command(cmd)
>> >> File
>> >> "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py",
>> >> line 972, in run_command
>> >> cmd_obj.run()
>> >> File "setup.py", line 142, in run
>> >> super(cmd, self).run()
>> >> File
>> >> "/Users/chamikara/testing/test_py_12_14_2017_2/env_proto_3.4/lib/python2.7/site-packages/setuptools/command/test.py",
>> >> line 225, in run
>> >> with self.project_on_sys_path():
>> >> File
>> >> "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py",
>> >> line 17, in __enter__
>> >> return self.gen.next()
>> >> File
>> >> "/Users/chamikara/testing/test_py_12_14_2017_2/env_proto_3.4/lib/python2.7/site-packages/setuptools/command/test.py",
>> >> line 164, in project_on_sys_path
>> >> require('%s==%s' % (ei_cmd.egg_name, ei_cmd.egg_version))
>> >> File
>> >> "/Users/chamikara/testing/test_py_12_14_2017_2/env_proto_3.4/lib/python2.7/site-packages/pkg_resources/__init__.py",
>> >> line 984, in require
>> >> needed = self.resolve(parse_requirements(requirements))
>> >> File
>> >> "/Users/chamikara/testing/test_py_12_14_2017_2/env_proto_3.4/lib/python2.7/site-packages/pkg_resources/__init__.py",
>> >> line 875, in resolve
>> >> raise VersionConflict(dist, req).with_context(dependent_req)
>> >> pkg_resources.ContextualVersionConflict: (protobuf 3.5.0.post1
>> >> (/Users/chamikara/testing/test_py_12_14_2017_2/beam/sdks/python/.eggs/protobuf-3.5.0.post1-py2.7.egg),
>> >> Requirement.parse('protobuf<=3.4.0,>=3.2.0'), set(['apache-beam']))
>> >> Seems like grpcio did a release today which is breaking us:
>> >> https://pypi.python.org/pypi/grpcio/1.8.1
>> >> We have to either bump our protobuf dependency or reduce the upper
>> >> bound of grpcio dependency to previous release (1.7.3).
>> >
>> >
>> >
>> > --
>> > This message was sent by Atlassian JIRA
>> > (v6.4.14#64029)
>
>
Re: [jira] [Commented] (BEAM-3357) Python SDK head fails to run tests
due to Requirement.parse('protobuf<=3.4.0,>=3.2.0')
Posted by Ahmet Altay <al...@google.com>.
On Fri, Dec 15, 2017 at 1:38 PM, Robert Bradshaw <ro...@google.com>
wrote:
> I am also in favor of pinning as an immediate fix, bumping the bound
> otherwise.
>
> Regarding putting an upper bound to avoid being broken, the last two
> breaks have been due to just having an (unneeded) upper bound (which
> held us back to broken/incompatible releases in relationship to other
> dependencies). We should try to trust semantic versioning when
> possible, and when not we must regularly audit.
>
+1 to this, especially the auditing part. We also had breaks because we
trusted semantic versioning. So far our semi-official policy was to trust a
package until they prove it otherwise. I will argue that grpc here is
making a breaking change in a minor version increment by changing the way
they are depending on a major package.
We have done a good job of auditing and updating those pinned (or upper
bounded) dependencies, and probably we are behind in some of those.
I wonder if we can automate some of this? If we can get a report, that
audits our dependencies, warns us about new releases and potential
conflicts it would be much easier to keep things up to date.
>
> On Fri, Dec 15, 2017 at 1:33 PM, Chamikara Jayalath (JIRA)
> <ji...@apache.org> wrote:
> >
> > [ https://issues.apache.org/jira/browse/BEAM-3357?page=
> com.atlassian.jira.plugin.system.issuetabpanels:comment-
> tabpanel&focusedCommentId=16293276#comment-16293276 ]
> >
> > Chamikara Jayalath commented on BEAM-3357:
> > ------------------------------------------
> >
> > You mean we could bump up the upper bound ? I think we should keep some
> upper bound in case we get badly broken by a future protobuf release.
> >
> >> Python SDK head fails to run tests due to Requirement.parse('protobuf<=
> 3.4.0,>=3.2.0')
> >> ------------------------------------------------------------
> --------------------------
> >>
> >> Key: BEAM-3357
> >> URL: https://issues.apache.org/jira/browse/BEAM-3357
> >> Project: Beam
> >> Issue Type: Bug
> >> Components: sdk-py-core
> >> Reporter: Chamikara Jayalath
> >> Priority: Critical
> >>
> >> Error is:
> >> running build_ext
> >> Traceback (most recent call last):
> >> File "setup.py", line 202, in <module>
> >> 'test': generate_protos_first(test),
> >> File "/Users/chamikara/testing/test_py_12_14_2017_2/env_
> proto_3.4/lib/python2.7/site-packages/setuptools/__init__.py", line 129,
> in setup
> >> return distutils.core.setup(**attrs)
> >> File "/System/Library/Frameworks/Python.framework/Versions/2.7/
> lib/python2.7/distutils/core.py", line 151, in setup
> >> dist.run_commands()
> >> File "/System/Library/Frameworks/Python.framework/Versions/2.7/
> lib/python2.7/distutils/dist.py", line 953, in run_commands
> >> self.run_command(cmd)
> >> File "/System/Library/Frameworks/Python.framework/Versions/2.7/
> lib/python2.7/distutils/dist.py", line 972, in run_command
> >> cmd_obj.run()
> >> File "setup.py", line 142, in run
> >> super(cmd, self).run()
> >> File "/Users/chamikara/testing/test_py_12_14_2017_2/env_
> proto_3.4/lib/python2.7/site-packages/setuptools/command/test.py", line
> 225, in run
> >> with self.project_on_sys_path():
> >> File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py",
> line 17, in __enter__
> >> return self.gen.next()
> >> File "/Users/chamikara/testing/test_py_12_14_2017_2/env_
> proto_3.4/lib/python2.7/site-packages/setuptools/command/test.py", line
> 164, in project_on_sys_path
> >> require('%s==%s' % (ei_cmd.egg_name, ei_cmd.egg_version))
> >> File "/Users/chamikara/testing/test_py_12_14_2017_2/env_
> proto_3.4/lib/python2.7/site-packages/pkg_resources/__init__.py", line
> 984, in require
> >> needed = self.resolve(parse_requirements(requirements))
> >> File "/Users/chamikara/testing/test_py_12_14_2017_2/env_
> proto_3.4/lib/python2.7/site-packages/pkg_resources/__init__.py", line
> 875, in resolve
> >> raise VersionConflict(dist, req).with_context(dependent_req)
> >> pkg_resources.ContextualVersionConflict: (protobuf 3.5.0.post1
> (/Users/chamikara/testing/test_py_12_14_2017_2/beam/
> sdks/python/.eggs/protobuf-3.5.0.post1-py2.7.egg),
> Requirement.parse('protobuf<=3.4.0,>=3.2.0'), set(['apache-beam']))
> >> Seems like grpcio did a release today which is breaking us:
> https://pypi.python.org/pypi/grpcio/1.8.1
> >> We have to either bump our protobuf dependency or reduce the upper
> bound of grpcio dependency to previous release (1.7.3).
> >
> >
> >
> > --
> > This message was sent by Atlassian JIRA
> > (v6.4.14#64029)
>