You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Robert Bradshaw <ro...@google.com> on 2020/05/08 16:19:03 UTC
Re: Python2.7 Beam End-of-Life Date
It hasn't been 3 months yet, but I wanted to call out a milestone that
Python 3 downloads crossed the 50% threshold on pypi, if just briefly.
On Thu, Feb 13, 2020 at 12:40 AM Ismaël Mejía <ie...@gmail.com> wrote:
>
> > I would suggest re-evaluating this within the next 3 months again. We need to balance between user pain/contributor pain/our ability to continuously test with python 2 in a shifting environment.
>
> Good idea for the in 3 months evaluation, at that point also distributions will probably be phasing out python2 by default which definitely help in this direction.
> Thanks for updating the roadmap Ahmet
>
>
> On Thu, Feb 13, 2020 at 2:49 AM Ahmet Altay <al...@google.com> wrote:
>>
>>
>>
>> On Wed, Feb 12, 2020 at 1:29 AM Ismaël Mejía <ie...@gmail.com> wrote:
>>>
>>> I am with Chad on this, we should probably extend it a bit more, even if it
>>> makes us struggle a bit at least we have some workarounds as Robert suggests,
>>> and as Chad said there are still many people playing the python 3 catchup game,
>>> so worth to support those users.
>>>
>>>
>>> But maybe it is worth to evaluate the current state later in the year.
>>
>>
>> I would suggest re-evaluating this within the next 3 months again. We need to balance between user pain/contributor pain/our ability to continuously test with python 2 in a shifting environment.
>>
>>>
>>> In the
>>> meantime can someone please update our Roadmap in the website with this info and
>>> where we are with Python 3 support (it looks not up to date).
>>> https://beam.apache.org/roadmap/
>>
>>
>> I made a minor change to update that page (https://github.com/apache/beam/pull/10848). A more comprehensive update to that page and linked (https://beam.apache.org/roadmap/python-sdk/#python-3-support) would still be welcome.
>>
>>>
>>>
>>> - Ismaël
>>>
>>>
>>> On Tue, Feb 4, 2020 at 10:49 PM Robert Bradshaw <ro...@google.com> wrote:
>>>>
>>>> On Tue, Feb 4, 2020 at 12:12 PM Chad Dombrova <ch...@gmail.com> wrote:
>>>> >>
>>>> >> Not to mention that all the nice work for the type hints will have to be redone in the for 3.x.
>>>> >
>>>> > Note that there's a tool for automatically converting type comments to annotations: https://github.com/ilevkivskyi/com2ann
>>>> >
>>>> > So don't let that part bother you.
>>>>
>>>> +1, I wouldn't worry about what can be easily automated.
>>>>
>>>> > I'm curious what other features you'd like to be using in the Beam source that you cannot now.
>>>>
>>>> I hit things occasionally, e.g. I just ran into wanting keyword-only
>>>> arguments the other day.
>>>>
>>>> >> It seems the faster we drop support the better.
>>>> >
>>>> >
>>>> > I've already gone over my position on this, but a refresher for those who care: some of the key vendors that support my industry will not offer python3-compatible versions of their software until the 4th quarter of 2020. If Beam switches to python3-only before that point we may be forced to stop contributing features (note: I'm the guy who added the type hints :). Every month you can give us would be greatly appreciated.
>>>>
>>>> As another data point, we're still 80/20 on Py2/Py3 for downloads at
>>>> PyPi [1] (which I've heard should be taken with a grain of salt, but
>>>> likely isn't totally off). IMHO that ratio needs to be way higher for
>>>> Python 3 to consider dropping Python 2. It's pretty noisy, but say it
>>>> doubles every 3 months that would put us at least mid-year before we
>>>> hit a cross-over point. On the other hand Q4 2020 is probably a
>>>> stretch.
>>>>
>>>> We could consider whether it needs to be an all-or-nothing thing as
>>>> well. E.g. perhaps some features could be Python 3 only sooner than
>>>> the whole codebase. (This would have to be well justified.) Another
>>>> mitigation is that it is possible to mix Python 2 and Python 3 in the
>>>> same pipeline with portability, so if there's a library that you need
>>>> for one DoFn it doesn't mean you have to hold back your whole
>>>> pipeline.
>>>>
>>>> - Robert
>>>>
>>>> [1] https://pypistats.org/packages/apache-beam , and that 20% may just
>>>> be a spike.
Re: Python2.7 Beam End-of-Life Date
Posted by Valentyn Tymofieiev <va...@google.com>.
Thank you! Shared on Slack as well.
On Thu, Jun 18, 2020 at 4:33 PM Ahmet Altay <al...@google.com> wrote:
> OK, tweeted the message. Could you share on Slack?
>
> On Thu, Jun 18, 2020 at 4:28 PM Valentyn Tymofieiev <va...@google.com>
> wrote:
>
>> Alright, let's publish the message on Twitter and echo on Slack once
>> that's done.
>> Thank you!
>>
>> On Tue, Jun 16, 2020 at 5:31 PM Ahmet Altay <al...@google.com> wrote:
>>
>>> That sounds reasonable to me. I agree, it is better to get specific
>>> reasons rather than a yes/no answer.
>>>
>>> On Tue, Jun 16, 2020 at 1:50 PM Valentyn Tymofieiev <va...@google.com>
>>> wrote:
>>>
>>>> After thinking about it for a bit, I am not sure whether a poll would
>>>> be actionable. For example, if 1000 users provide a positive response and 5
>>>> users provide a negative response, how do we interpret that and where do
>>>> we draw a line? How about instead we encourage users to reach out, and tell
>>>> what is not working for them? For example:
>>>>
>>>> Beam is considering making 2.23.0 a final release supporting Py2. If
>>>> you are not able to switch to Python 3, please let us know why: [short link
>>>> to user@ thread] [1].
>>>>
>>>> Thoughts?
>>>>
>>>> [1]
>>>> https://lists.apache.org/thread.html/r0de71d98d98b213dd1d0c45c1f5642135116f25def5637a5f41c8d29%40%3Cuser.beam.apache.org%3E
>>>>
>>>> On Tue, Jun 16, 2020 at 11:50 AM Ahmet Altay <al...@google.com> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Tue, Jun 16, 2020 at 11:38 AM Valentyn Tymofieiev <
>>>>> valentyn@google.com> wrote:
>>>>>
>>>>>> I have reached out to user@ for feedback on Python 3 migration[1].
>>>>>>
>>>>>
>>>>> Maybe also ask on slack? There are quite a bit of users there as well.
>>>>>
>>>>>
>>>>>>
>>>>>> Could somebody from PMC please help with Twitter poll?
>>>>>>
>>>>>
>>>>> I can try to do this. What question would you like to ask? I do not
>>>>> know much about twitter polls but I assume they have character limits
>>>>> similar to regular tweets.
>>>>>
>>>>>
>>>>>>
>>>>>> Technically, we can proceed with the change between 2.23.0 and
>>>>>> 2.24.0, so that's after 2.23.0 is cut and we give sufficient time for users
>>>>>> to respond.
>>>>>>
>>>>>> [1]
>>>>>> https://lists.apache.org/thread.html/r0de71d98d98b213dd1d0c45c1f5642135116f25def5637a5f41c8d29%40%3Cuser.beam.apache.org%3E
>>>>>> On Tue, Jun 16, 2020 at 9:22 AM Ismaël Mejía <ie...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Yes we need to poll this outside as Robert proposed.
>>>>>>>
>>>>>>> The proposed last support version corresponds with the next release
>>>>>>> that will be
>>>>>>> cut in two weeks. Sounds a bit short if we count the time to poll
>>>>>>> people on this
>>>>>>> subject but still could be.
>>>>>>
>>>>>> Technically, we can proceed with the change between 2.23.0 and
>>>>>> 2.24.0, so that's after 2.23.0 is cut and we give sufficient time for users
>>>>>> to respond.
>>>>>>
>>>>>>
>>>>>>> I remember Chad mentioned in this thread the impossibility to get
>>>>>>> support for
>>>>>>> python 2 in his industry until the end of the year, Maybe things
>>>>>>> have improved.
>>>>>>> Have they?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Jun 16, 2020 at 6:10 PM Robert Bradshaw <ro...@google.com>
>>>>>>> wrote:
>>>>>>> >
>>>>>>> > I like that option as a concrete proposal. I think we should
>>>>>>> circulate it more widely (the users list, twitter poll, at least a new
>>>>>>> thread...), maybe phrasing it as "is there any reason you couldn't migrate
>>>>>>> to Python 3 (or stick with an older version of Beam) after 2.23 (due to be
>>>>>>> cut here in a couple of weeks)?" If there is strong concern/pushback, we
>>>>>>> could consider holding on for one more release.
>>>>>>> >
>>>>>>> > On Tue, Jun 16, 2020 at 8:54 AM David Cavazos <dc...@google.com>
>>>>>>> wrote:
>>>>>>> >>
>>>>>>> >> +1
>>>>>>> >>
>>>>>>> >> On Mon, Jun 15, 2020 at 6:52 PM Udi Meiri <eh...@google.com>
>>>>>>> wrote:
>>>>>>> >>>
>>>>>>> >>> +1
>>>>>>> >>>
>>>>>>> >>> On Mon, Jun 15, 2020 at 4:27 PM Ahmet Altay <al...@google.com>
>>>>>>> wrote:
>>>>>>> >>>>
>>>>>>> >>>> As a concrete proposal, could we commit to removing python 2
>>>>>>> support by 2.24? In other words, mark the next release 2.23 as the last
>>>>>>> python 2 compatible Beam version.
>>>>>>> >>>>
>>>>>>> >>>> On Mon, Jun 15, 2020 at 2:09 PM Valentyn Tymofieiev <
>>>>>>> valentyn@google.com> wrote:
>>>>>>> >>>>>
>>>>>>> >>>>> Another input here:
>>>>>>> >>>>>
>>>>>>> >>>>> If you opened a Python PR in the last few days, you probably
>>>>>>> noticed that our test suites were broken by a transitive dependency of Beam
>>>>>>> that dropped python 2 support, but did not declare python_requires>=3 in
>>>>>>> its setup.py [1]. This temporarily broke a subset of Beam Py2 users (who
>>>>>>> did not explicitly pin the 'rsa' dependency), and still affects Beam
>>>>>>> development[2].
>>>>>>> >>>>>
>>>>>>> >>>>> This is the second time[3] Beam is affected with an issue of
>>>>>>> this kind, so support of Python 2 starts to slow down our development, and
>>>>>>> add toil for maintainers of packages we depend on (both directly and
>>>>>>> transitively).
>>>>>>> >>>>>
>>>>>>> >>>>> [1] https://github.com/sybrenstuvel/python-rsa/issues/152
>>>>>>> >>>>> [2]
>>>>>>> https://lists.apache.org/thread.html/r9993b40b0c1cb8682ce56013165d4b80fdde0ee469a73bcb9466ddfb%40%3Cdev.beam.apache.org%3E
>>>>>>> >>>>> [3] https://github.com/hamcrest/PyHamcrest/issues/131
>>>>>>> >>>>>
>>>>>>> >>>>> On Tue, Jun 9, 2020 at 4:06 PM Ahmet Altay <al...@google.com>
>>>>>>> wrote:
>>>>>>> >>>>>>
>>>>>>> >>>>>> Thank you for re-opening this Valentyn. I am in favor of
>>>>>>> EOLing py2 support sooner than later. The reality is that we will not be
>>>>>>> effectively supporting beam python 2 for a long time while the ecosystem
>>>>>>> already EOLed python 2. That said, a significant chunk (but no longer a
>>>>>>> majority) of our users are still using python 2. Upgrades are painful, it
>>>>>>> might be especially painful nowadays. It would be good to hear counter view
>>>>>>> points, user voices related to this.
>>>>>>> >>>>>>
>>>>>>> >>>>>> On Thu, Jun 4, 2020 at 4:53 PM Valentyn Tymofieiev <
>>>>>>> valentyn@google.com> wrote:
>>>>>>> >>>>>>>
>>>>>>> >>>>>>> Back at the end of February we decided to revisit this
>>>>>>> conversation in 3 months. Do folks on this thread have any new input or
>>>>>>> perspective regarding us balancing "user pain/contributor pain/our ability
>>>>>>> to continuously test with python 2 in a shifting environment"?
>>>>>>> >>>>>>>
>>>>>>> >>>>>>> Some new information on my end is that we have been seeing
>>>>>>> steady adoption of Python 3 among Beam Python users in Dataflow,
>>>>>>> particularly strong adoption among streaming users, and Dataflow is
>>>>>>> sunsetting Python 2 support for all released Beam SDKs later this year [1].
>>>>>>> We will have to remove Python 2 Beam test suites that use Dataflow when
>>>>>>> Dataflow runner disables Py2 support if this happens before Beam Py2 EOL
>>>>>>> (when we have to remove all Py2 suites), including performance tests that
>>>>>>> still use Dataflow on Python 3.
>>>>>>> >>>>>>>
>>>>>>> >>>>>>> I am curious how much motivation there is in the community
>>>>>>> at this moment to continue Py2 support in Beam, whether any previous Py3
>>>>>>> migration blockers were resolved or any new blockers discovered among Beam
>>>>>>> users.
>>>>>>> >>>>>>>
>>>>>>> >>>>>>> [1]
>>>>>>> https://cloud.google.com/python/docs/python2-sunset/#dataflow
>>>>>>> >>>>>>>
>>>>>>> >>>>>>> On Fri, May 8, 2020 at 3:52 PM Valentyn Tymofieiev <
>>>>>>> valentyn@google.com> wrote:
>>>>>>> >>>>>>>>
>>>>>>> >>>>>>>> That's good news! Thanks for sharing.
>>>>>>> >>>>>>>>
>>>>>>> >>>>>>>> Another datapoint, here are a few of Beam's dependencies
>>>>>>> that no longer release new py2 artifacts (I looked at REQUIRED_PACKAGES +
>>>>>>> aws, gcp, and interactive extras):
>>>>>>> >>>>>>>>
>>>>>>> >>>>>>>> hdfs
>>>>>>> >>>>>>>> numpy
>>>>>>> >>>>>>>> pyarrow
>>>>>>> >>>>>>>> ipython
>>>>>>> >>>>>>>>
>>>>>>> >>>>>>>> There are more if we include transitive dependencies and
>>>>>>> test-only packages. I also remember encountering one issue last month that
>>>>>>> was broken only on Py2, which we had to go back and fix.
>>>>>>> >>>>>>>>
>>>>>>> >>>>>>>> If others have noticed frictions related to ongoing Py2
>>>>>>> support or have updates on previously mentioned Py3 migration blockers,
>>>>>>> feel free to post them.
>>>>>>> >>>>>>>>
>>>>>>> >>>>>>>> On Fri, May 8, 2020 at 9:19 AM Robert Bradshaw <
>>>>>>> robertwb@google.com> wrote:
>>>>>>> >>>>>>>>>
>>>>>>> >>>>>>>>> It hasn't been 3 months yet, but I wanted to call out a
>>>>>>> milestone that
>>>>>>> >>>>>>>>> Python 3 downloads crossed the 50% threshold on pypi, if
>>>>>>> just briefly.
>>>>>>> >>>>>>>>>
>>>>>>> >>>>>>>>> On Thu, Feb 13, 2020 at 12:40 AM Ismaël Mejía <
>>>>>>> iemejia@gmail.com> wrote:
>>>>>>> >>>>>>>>> >
>>>>>>> >>>>>>>>> > > I would suggest re-evaluating this within the next 3
>>>>>>> months again. We need to balance between user pain/contributor pain/our
>>>>>>> ability to continuously test with python 2 in a shifting environment.
>>>>>>> >>>>>>>>> >
>>>>>>> >>>>>>>>> > Good idea for the in 3 months evaluation, at that point
>>>>>>> also distributions will probably be phasing out python2 by default which
>>>>>>> definitely help in this direction.
>>>>>>> >>>>>>>>> > Thanks for updating the roadmap Ahmet
>>>>>>> >>>>>>>>> >
>>>>>>> >>>>>>>>> >
>>>>>>> >>>>>>>>> > On Thu, Feb 13, 2020 at 2:49 AM Ahmet Altay <
>>>>>>> altay@google.com> wrote:
>>>>>>> >>>>>>>>> >>
>>>>>>> >>>>>>>>> >>
>>>>>>> >>>>>>>>> >>
>>>>>>> >>>>>>>>> >> On Wed, Feb 12, 2020 at 1:29 AM Ismaël Mejía <
>>>>>>> iemejia@gmail.com> wrote:
>>>>>>> >>>>>>>>> >>>
>>>>>>> >>>>>>>>> >>> I am with Chad on this, we should probably extend it a
>>>>>>> bit more, even if it
>>>>>>> >>>>>>>>> >>> makes us struggle a bit at least we have some
>>>>>>> workarounds as Robert suggests,
>>>>>>> >>>>>>>>> >>> and as Chad said there are still many people playing
>>>>>>> the python 3 catchup game,
>>>>>>> >>>>>>>>> >>> so worth to support those users.
>>>>>>> >>>>>>>>> >>>
>>>>>>> >>>>>>>>> >>>
>>>>>>> >>>>>>>>> >>> But maybe it is worth to evaluate the current state
>>>>>>> later in the year.
>>>>>>> >>>>>>>>> >>
>>>>>>> >>>>>>>>> >>
>>>>>>> >>>>>>>>> >> I would suggest re-evaluating this within the next 3
>>>>>>> months again. We need to balance between user pain/contributor pain/our
>>>>>>> ability to continuously test with python 2 in a shifting environment.
>>>>>>> >>>>>>>>> >>
>>>>>>> >>>>>>>>> >>>
>>>>>>> >>>>>>>>> >>> In the
>>>>>>> >>>>>>>>> >>> meantime can someone please update our Roadmap in the
>>>>>>> website with this info and
>>>>>>> >>>>>>>>> >>> where we are with Python 3 support (it looks not up to
>>>>>>> date).
>>>>>>> >>>>>>>>> >>> https://beam.apache.org/roadmap/
>>>>>>> >>>>>>>>> >>
>>>>>>> >>>>>>>>> >>
>>>>>>> >>>>>>>>> >> I made a minor change to update that page (
>>>>>>> https://github.com/apache/beam/pull/10848). A more comprehensive
>>>>>>> update to that page and linked (
>>>>>>> https://beam.apache.org/roadmap/python-sdk/#python-3-support) would
>>>>>>> still be welcome.
>>>>>>> >>>>>>>>> >>
>>>>>>> >>>>>>>>> >>>
>>>>>>> >>>>>>>>> >>>
>>>>>>> >>>>>>>>> >>> - Ismaël
>>>>>>> >>>>>>>>> >>>
>>>>>>> >>>>>>>>> >>>
>>>>>>> >>>>>>>>> >>> On Tue, Feb 4, 2020 at 10:49 PM Robert Bradshaw <
>>>>>>> robertwb@google.com> wrote:
>>>>>>> >>>>>>>>> >>>>
>>>>>>> >>>>>>>>> >>>> On Tue, Feb 4, 2020 at 12:12 PM Chad Dombrova <
>>>>>>> chadrik@gmail.com> wrote:
>>>>>>> >>>>>>>>> >>>> >>
>>>>>>> >>>>>>>>> >>>> >> Not to mention that all the nice work for the
>>>>>>> type hints will have to be redone in the for 3.x.
>>>>>>> >>>>>>>>> >>>> >
>>>>>>> >>>>>>>>> >>>> > Note that there's a tool for automatically
>>>>>>> converting type comments to annotations:
>>>>>>> https://github.com/ilevkivskyi/com2ann
>>>>>>> >>>>>>>>> >>>> >
>>>>>>> >>>>>>>>> >>>> > So don't let that part bother you.
>>>>>>> >>>>>>>>> >>>>
>>>>>>> >>>>>>>>> >>>> +1, I wouldn't worry about what can be easily
>>>>>>> automated.
>>>>>>> >>>>>>>>> >>>>
>>>>>>> >>>>>>>>> >>>> > I'm curious what other features you'd like to be
>>>>>>> using in the Beam source that you cannot now.
>>>>>>> >>>>>>>>> >>>>
>>>>>>> >>>>>>>>> >>>> I hit things occasionally, e.g. I just ran into
>>>>>>> wanting keyword-only
>>>>>>> >>>>>>>>> >>>> arguments the other day.
>>>>>>> >>>>>>>>> >>>>
>>>>>>> >>>>>>>>> >>>> >> It seems the faster we drop support the better.
>>>>>>> >>>>>>>>> >>>> >
>>>>>>> >>>>>>>>> >>>> >
>>>>>>> >>>>>>>>> >>>> > I've already gone over my position on this, but a
>>>>>>> refresher for those who care: some of the key vendors that support my
>>>>>>> industry will not offer python3-compatible versions of their software until
>>>>>>> the 4th quarter of 2020. If Beam switches to python3-only before that
>>>>>>> point we may be forced to stop contributing features (note: I'm the guy who
>>>>>>> added the type hints :). Every month you can give us would be greatly
>>>>>>> appreciated.
>>>>>>> >>>>>>>>> >>>>
>>>>>>> >>>>>>>>> >>>> As another data point, we're still 80/20 on Py2/Py3
>>>>>>> for downloads at
>>>>>>> >>>>>>>>> >>>> PyPi [1] (which I've heard should be taken with a
>>>>>>> grain of salt, but
>>>>>>> >>>>>>>>> >>>> likely isn't totally off). IMHO that ratio needs to
>>>>>>> be way higher for
>>>>>>> >>>>>>>>> >>>> Python 3 to consider dropping Python 2. It's pretty
>>>>>>> noisy, but say it
>>>>>>> >>>>>>>>> >>>> doubles every 3 months that would put us at least
>>>>>>> mid-year before we
>>>>>>> >>>>>>>>> >>>> hit a cross-over point. On the other hand Q4 2020 is
>>>>>>> probably a
>>>>>>> >>>>>>>>> >>>> stretch.
>>>>>>> >>>>>>>>> >>>>
>>>>>>> >>>>>>>>> >>>> We could consider whether it needs to be an
>>>>>>> all-or-nothing thing as
>>>>>>> >>>>>>>>> >>>> well. E.g. perhaps some features could be Python 3
>>>>>>> only sooner than
>>>>>>> >>>>>>>>> >>>> the whole codebase. (This would have to be well
>>>>>>> justified.) Another
>>>>>>> >>>>>>>>> >>>> mitigation is that it is possible to mix Python 2 and
>>>>>>> Python 3 in the
>>>>>>> >>>>>>>>> >>>> same pipeline with portability, so if there's a
>>>>>>> library that you need
>>>>>>> >>>>>>>>> >>>> for one DoFn it doesn't mean you have to hold back
>>>>>>> your whole
>>>>>>> >>>>>>>>> >>>> pipeline.
>>>>>>> >>>>>>>>> >>>>
>>>>>>> >>>>>>>>> >>>> - Robert
>>>>>>> >>>>>>>>> >>>>
>>>>>>> >>>>>>>>> >>>> [1] https://pypistats.org/packages/apache-beam , and
>>>>>>> that 20% may just
>>>>>>> >>>>>>>>> >>>> be a spike.
>>>>>>>
>>>>>>
Re: Python2.7 Beam End-of-Life Date
Posted by Valentyn Tymofieiev <va...@google.com>.
> Separately, I am curious what the precise symptoms of dropping Python 2.7
support will be. I admit I haven't been able to follow all the threads on
the topic. Is it moving on to newer dependencies, or also wanting to use
new language features, or something more subtle?
Once we remove Py2 support we will update python_requires stanza[1], so new
Apache Beam release artifacts will not be installable on Python 2. We will
remove Py2 test suites and will gradually cleanup parts in the codebase
where we branch to make sure the code can run both on Py2 and Py3 (random
example: [2]).
> Is it moving on to newer dependencies,
We will reduce the risk of not being able to upgrade to new dependencies
and chances of other dependencies breaking us[3] as they drop Py2 support.
> or also wanting to use new language features, or something more subtle?
Once we are no longer required to support Python 2, Beam developers will be
able to use Python 3 language features and common standard library features
without backports. With Python 3.5 reaching EOL in September we can expect
Python 3.6 to become our next lowest common denominator.
[1]
https://github.com/apache/beam/blob/956e4eb39a7fedbae05985c759284557dcc3d9ec/sdks/python/setup.py#L261
[2]
https://github.com/apache/beam/blob/956e4eb39a7fedbae05985c759284557dcc3d9ec/sdks/python/apache_beam/runners/worker/operations.py#L936
[3]
https://lists.apache.org/thread.html/rbac6e3b5c5629b756945640c5b0432a32dd76b3e5b261a1420bbc4a0%40%3Cdev.beam.apache.org%3E
On Fri, Jul 24, 2020 at 3:41 PM Kenneth Knowles <ke...@apache.org> wrote:
> Regardless of the outcome, it would be good to have some more details
> here. Can you share links for people to find out more about Python 3
> support in those products and their timeline? I did some quick searching
> out of curiosity but I do not believe I found the authoritative information.
>
> Separately, I am curious what the precise symptoms of dropping Python 2.7
> support will be. I admit I haven't been able to follow all the threads on
> the topic. Is it moving on to newer dependencies, or also wanting to use
> new language features, or something more subtle?
>
> Kenn
>
> On Thu, Jun 18, 2020 at 5:01 PM Chad Dombrova <ch...@gmail.com> wrote:
>
>> Hi all,
>> Sorry I've been AWOL. I've been pulled in a number of different
>> directions.
>>
>> We are still increasing our use of Beam, and I have it on my todo list to
>> reach out to Kenneth about how Beam could be expanded into the VFX /
>> Animation industries.
>>
>> Our industry uses a number of specialized applications with embedded
>> python interpreters. We run Beam inside these interpreters, so we're
>> waiting for them to switch to python3.
>>
>> Here's the status report for python3 adoption in our key applications:
>>
>> *Maya*: In Beta
>> *Houdini*: Released
>> *Nuke*: In Beta
>> *Katana*: Not started (Alpha?)
>>
>> I hate to be the one holding the project back, and I understand if you
>> all ultimately decide it's untenable to wait any longer. The good news is
>> 3 out of 4 applications should be ready in the next 2-3 months. I can do
>> some investigation into what workarounds might look like for Katana, or
>> maybe we can use the Beta version once python3 support arrives, which would
>> move our schedule forward.
>>
>> When would 2.24 release?
>>
>> -chad
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Thu, Jun 18, 2020 at 4:33 PM Ahmet Altay <al...@google.com> wrote:
>>
>>> OK, tweeted the message. Could you share on Slack?
>>>
>>> On Thu, Jun 18, 2020 at 4:28 PM Valentyn Tymofieiev <va...@google.com>
>>> wrote:
>>>
>>>> Alright, let's publish the message on Twitter and echo on Slack once
>>>> that's done.
>>>> Thank you!
>>>>
>>>> On Tue, Jun 16, 2020 at 5:31 PM Ahmet Altay <al...@google.com> wrote:
>>>>
>>>>> That sounds reasonable to me. I agree, it is better to get specific
>>>>> reasons rather than a yes/no answer.
>>>>>
>>>>> On Tue, Jun 16, 2020 at 1:50 PM Valentyn Tymofieiev <
>>>>> valentyn@google.com> wrote:
>>>>>
>>>>>> After thinking about it for a bit, I am not sure whether a poll would
>>>>>> be actionable. For example, if 1000 users provide a positive response and 5
>>>>>> users provide a negative response, how do we interpret that and where do
>>>>>> we draw a line? How about instead we encourage users to reach out, and tell
>>>>>> what is not working for them? For example:
>>>>>>
>>>>>> Beam is considering making 2.23.0 a final release supporting Py2. If
>>>>>> you are not able to switch to Python 3, please let us know why: [short link
>>>>>> to user@ thread] [1].
>>>>>>
>>>>>> Thoughts?
>>>>>>
>>>>>> [1]
>>>>>> https://lists.apache.org/thread.html/r0de71d98d98b213dd1d0c45c1f5642135116f25def5637a5f41c8d29%40%3Cuser.beam.apache.org%3E
>>>>>>
>>>>>> On Tue, Jun 16, 2020 at 11:50 AM Ahmet Altay <al...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Jun 16, 2020 at 11:38 AM Valentyn Tymofieiev <
>>>>>>> valentyn@google.com> wrote:
>>>>>>>
>>>>>>>> I have reached out to user@ for feedback on Python 3 migration[1].
>>>>>>>>
>>>>>>>
>>>>>>> Maybe also ask on slack? There are quite a bit of users there as
>>>>>>> well.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Could somebody from PMC please help with Twitter poll?
>>>>>>>>
>>>>>>>
>>>>>>> I can try to do this. What question would you like to ask? I do not
>>>>>>> know much about twitter polls but I assume they have character limits
>>>>>>> similar to regular tweets.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Technically, we can proceed with the change between 2.23.0 and
>>>>>>>> 2.24.0, so that's after 2.23.0 is cut and we give sufficient time for users
>>>>>>>> to respond.
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> https://lists.apache.org/thread.html/r0de71d98d98b213dd1d0c45c1f5642135116f25def5637a5f41c8d29%40%3Cuser.beam.apache.org%3E
>>>>>>>> On Tue, Jun 16, 2020 at 9:22 AM Ismaël Mejía <ie...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Yes we need to poll this outside as Robert proposed.
>>>>>>>>>
>>>>>>>>> The proposed last support version corresponds with the next
>>>>>>>>> release that will be
>>>>>>>>> cut in two weeks. Sounds a bit short if we count the time to poll
>>>>>>>>> people on this
>>>>>>>>> subject but still could be.
>>>>>>>>
>>>>>>>> Technically, we can proceed with the change between 2.23.0 and
>>>>>>>> 2.24.0, so that's after 2.23.0 is cut and we give sufficient time for users
>>>>>>>> to respond.
>>>>>>>>
>>>>>>>>
>>>>>>>>> I remember Chad mentioned in this thread the impossibility to get
>>>>>>>>> support for
>>>>>>>>> python 2 in his industry until the end of the year, Maybe things
>>>>>>>>> have improved.
>>>>>>>>> Have they?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Jun 16, 2020 at 6:10 PM Robert Bradshaw <
>>>>>>>>> robertwb@google.com> wrote:
>>>>>>>>> >
>>>>>>>>> > I like that option as a concrete proposal. I think we should
>>>>>>>>> circulate it more widely (the users list, twitter poll, at least a new
>>>>>>>>> thread...), maybe phrasing it as "is there any reason you couldn't migrate
>>>>>>>>> to Python 3 (or stick with an older version of Beam) after 2.23 (due to be
>>>>>>>>> cut here in a couple of weeks)?" If there is strong concern/pushback, we
>>>>>>>>> could consider holding on for one more release.
>>>>>>>>> >
>>>>>>>>> > On Tue, Jun 16, 2020 at 8:54 AM David Cavazos <
>>>>>>>>> dcavazos@google.com> wrote:
>>>>>>>>> >>
>>>>>>>>> >> +1
>>>>>>>>> >>
>>>>>>>>> >> On Mon, Jun 15, 2020 at 6:52 PM Udi Meiri <eh...@google.com>
>>>>>>>>> wrote:
>>>>>>>>> >>>
>>>>>>>>> >>> +1
>>>>>>>>> >>>
>>>>>>>>> >>> On Mon, Jun 15, 2020 at 4:27 PM Ahmet Altay <al...@google.com>
>>>>>>>>> wrote:
>>>>>>>>> >>>>
>>>>>>>>> >>>> As a concrete proposal, could we commit to removing python 2
>>>>>>>>> support by 2.24? In other words, mark the next release 2.23 as the last
>>>>>>>>> python 2 compatible Beam version.
>>>>>>>>> >>>>
>>>>>>>>> >>>> On Mon, Jun 15, 2020 at 2:09 PM Valentyn Tymofieiev <
>>>>>>>>> valentyn@google.com> wrote:
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> Another input here:
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> If you opened a Python PR in the last few days, you probably
>>>>>>>>> noticed that our test suites were broken by a transitive dependency of Beam
>>>>>>>>> that dropped python 2 support, but did not declare python_requires>=3 in
>>>>>>>>> its setup.py [1]. This temporarily broke a subset of Beam Py2 users (who
>>>>>>>>> did not explicitly pin the 'rsa' dependency), and still affects Beam
>>>>>>>>> development[2].
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> This is the second time[3] Beam is affected with an issue of
>>>>>>>>> this kind, so support of Python 2 starts to slow down our development, and
>>>>>>>>> add toil for maintainers of packages we depend on (both directly and
>>>>>>>>> transitively).
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> [1] https://github.com/sybrenstuvel/python-rsa/issues/152
>>>>>>>>> >>>>> [2]
>>>>>>>>> https://lists.apache.org/thread.html/r9993b40b0c1cb8682ce56013165d4b80fdde0ee469a73bcb9466ddfb%40%3Cdev.beam.apache.org%3E
>>>>>>>>> >>>>> [3] https://github.com/hamcrest/PyHamcrest/issues/131
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> On Tue, Jun 9, 2020 at 4:06 PM Ahmet Altay <al...@google.com>
>>>>>>>>> wrote:
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>> Thank you for re-opening this Valentyn. I am in favor of
>>>>>>>>> EOLing py2 support sooner than later. The reality is that we will not be
>>>>>>>>> effectively supporting beam python 2 for a long time while the ecosystem
>>>>>>>>> already EOLed python 2. That said, a significant chunk (but no longer a
>>>>>>>>> majority) of our users are still using python 2. Upgrades are painful, it
>>>>>>>>> might be especially painful nowadays. It would be good to hear counter view
>>>>>>>>> points, user voices related to this.
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>> On Thu, Jun 4, 2020 at 4:53 PM Valentyn Tymofieiev <
>>>>>>>>> valentyn@google.com> wrote:
>>>>>>>>> >>>>>>>
>>>>>>>>> >>>>>>> Back at the end of February we decided to revisit this
>>>>>>>>> conversation in 3 months. Do folks on this thread have any new input or
>>>>>>>>> perspective regarding us balancing "user pain/contributor pain/our ability
>>>>>>>>> to continuously test with python 2 in a shifting environment"?
>>>>>>>>> >>>>>>>
>>>>>>>>> >>>>>>> Some new information on my end is that we have been seeing
>>>>>>>>> steady adoption of Python 3 among Beam Python users in Dataflow,
>>>>>>>>> particularly strong adoption among streaming users, and Dataflow is
>>>>>>>>> sunsetting Python 2 support for all released Beam SDKs later this year [1].
>>>>>>>>> We will have to remove Python 2 Beam test suites that use Dataflow when
>>>>>>>>> Dataflow runner disables Py2 support if this happens before Beam Py2 EOL
>>>>>>>>> (when we have to remove all Py2 suites), including performance tests that
>>>>>>>>> still use Dataflow on Python 3.
>>>>>>>>> >>>>>>>
>>>>>>>>> >>>>>>> I am curious how much motivation there is in the community
>>>>>>>>> at this moment to continue Py2 support in Beam, whether any previous Py3
>>>>>>>>> migration blockers were resolved or any new blockers discovered among Beam
>>>>>>>>> users.
>>>>>>>>> >>>>>>>
>>>>>>>>> >>>>>>> [1]
>>>>>>>>> https://cloud.google.com/python/docs/python2-sunset/#dataflow
>>>>>>>>> >>>>>>>
>>>>>>>>> >>>>>>> On Fri, May 8, 2020 at 3:52 PM Valentyn Tymofieiev <
>>>>>>>>> valentyn@google.com> wrote:
>>>>>>>>> >>>>>>>>
>>>>>>>>> >>>>>>>> That's good news! Thanks for sharing.
>>>>>>>>> >>>>>>>>
>>>>>>>>> >>>>>>>> Another datapoint, here are a few of Beam's dependencies
>>>>>>>>> that no longer release new py2 artifacts (I looked at REQUIRED_PACKAGES +
>>>>>>>>> aws, gcp, and interactive extras):
>>>>>>>>> >>>>>>>>
>>>>>>>>> >>>>>>>> hdfs
>>>>>>>>> >>>>>>>> numpy
>>>>>>>>> >>>>>>>> pyarrow
>>>>>>>>> >>>>>>>> ipython
>>>>>>>>> >>>>>>>>
>>>>>>>>> >>>>>>>> There are more if we include transitive dependencies and
>>>>>>>>> test-only packages. I also remember encountering one issue last month that
>>>>>>>>> was broken only on Py2, which we had to go back and fix.
>>>>>>>>> >>>>>>>>
>>>>>>>>> >>>>>>>> If others have noticed frictions related to ongoing Py2
>>>>>>>>> support or have updates on previously mentioned Py3 migration blockers,
>>>>>>>>> feel free to post them.
>>>>>>>>> >>>>>>>>
>>>>>>>>> >>>>>>>> On Fri, May 8, 2020 at 9:19 AM Robert Bradshaw <
>>>>>>>>> robertwb@google.com> wrote:
>>>>>>>>> >>>>>>>>>
>>>>>>>>> >>>>>>>>> It hasn't been 3 months yet, but I wanted to call out a
>>>>>>>>> milestone that
>>>>>>>>> >>>>>>>>> Python 3 downloads crossed the 50% threshold on pypi, if
>>>>>>>>> just briefly.
>>>>>>>>> >>>>>>>>>
>>>>>>>>> >>>>>>>>> On Thu, Feb 13, 2020 at 12:40 AM Ismaël Mejía <
>>>>>>>>> iemejia@gmail.com> wrote:
>>>>>>>>> >>>>>>>>> >
>>>>>>>>> >>>>>>>>> > > I would suggest re-evaluating this within the next 3
>>>>>>>>> months again. We need to balance between user pain/contributor pain/our
>>>>>>>>> ability to continuously test with python 2 in a shifting environment.
>>>>>>>>> >>>>>>>>> >
>>>>>>>>> >>>>>>>>> > Good idea for the in 3 months evaluation, at that
>>>>>>>>> point also distributions will probably be phasing out python2 by default
>>>>>>>>> which definitely help in this direction.
>>>>>>>>> >>>>>>>>> > Thanks for updating the roadmap Ahmet
>>>>>>>>> >>>>>>>>> >
>>>>>>>>> >>>>>>>>> >
>>>>>>>>> >>>>>>>>> > On Thu, Feb 13, 2020 at 2:49 AM Ahmet Altay <
>>>>>>>>> altay@google.com> wrote:
>>>>>>>>> >>>>>>>>> >>
>>>>>>>>> >>>>>>>>> >>
>>>>>>>>> >>>>>>>>> >>
>>>>>>>>> >>>>>>>>> >> On Wed, Feb 12, 2020 at 1:29 AM Ismaël Mejía <
>>>>>>>>> iemejia@gmail.com> wrote:
>>>>>>>>> >>>>>>>>> >>>
>>>>>>>>> >>>>>>>>> >>> I am with Chad on this, we should probably extend it
>>>>>>>>> a bit more, even if it
>>>>>>>>> >>>>>>>>> >>> makes us struggle a bit at least we have some
>>>>>>>>> workarounds as Robert suggests,
>>>>>>>>> >>>>>>>>> >>> and as Chad said there are still many people playing
>>>>>>>>> the python 3 catchup game,
>>>>>>>>> >>>>>>>>> >>> so worth to support those users.
>>>>>>>>> >>>>>>>>> >>>
>>>>>>>>> >>>>>>>>> >>>
>>>>>>>>> >>>>>>>>> >>> But maybe it is worth to evaluate the current state
>>>>>>>>> later in the year.
>>>>>>>>> >>>>>>>>> >>
>>>>>>>>> >>>>>>>>> >>
>>>>>>>>> >>>>>>>>> >> I would suggest re-evaluating this within the next 3
>>>>>>>>> months again. We need to balance between user pain/contributor pain/our
>>>>>>>>> ability to continuously test with python 2 in a shifting environment.
>>>>>>>>> >>>>>>>>> >>
>>>>>>>>> >>>>>>>>> >>>
>>>>>>>>> >>>>>>>>> >>> In the
>>>>>>>>> >>>>>>>>> >>> meantime can someone please update our Roadmap in
>>>>>>>>> the website with this info and
>>>>>>>>> >>>>>>>>> >>> where we are with Python 3 support (it looks not up
>>>>>>>>> to date).
>>>>>>>>> >>>>>>>>> >>> https://beam.apache.org/roadmap/
>>>>>>>>> >>>>>>>>> >>
>>>>>>>>> >>>>>>>>> >>
>>>>>>>>> >>>>>>>>> >> I made a minor change to update that page (
>>>>>>>>> https://github.com/apache/beam/pull/10848). A more comprehensive
>>>>>>>>> update to that page and linked (
>>>>>>>>> https://beam.apache.org/roadmap/python-sdk/#python-3-support)
>>>>>>>>> would still be welcome.
>>>>>>>>> >>>>>>>>> >>
>>>>>>>>> >>>>>>>>> >>>
>>>>>>>>> >>>>>>>>> >>>
>>>>>>>>> >>>>>>>>> >>> - Ismaël
>>>>>>>>> >>>>>>>>> >>>
>>>>>>>>> >>>>>>>>> >>>
>>>>>>>>> >>>>>>>>> >>> On Tue, Feb 4, 2020 at 10:49 PM Robert Bradshaw <
>>>>>>>>> robertwb@google.com> wrote:
>>>>>>>>> >>>>>>>>> >>>>
>>>>>>>>> >>>>>>>>> >>>> On Tue, Feb 4, 2020 at 12:12 PM Chad Dombrova <
>>>>>>>>> chadrik@gmail.com> wrote:
>>>>>>>>> >>>>>>>>> >>>> >>
>>>>>>>>> >>>>>>>>> >>>> >> Not to mention that all the nice work for the
>>>>>>>>> type hints will have to be redone in the for 3.x.
>>>>>>>>> >>>>>>>>> >>>> >
>>>>>>>>> >>>>>>>>> >>>> > Note that there's a tool for automatically
>>>>>>>>> converting type comments to annotations:
>>>>>>>>> https://github.com/ilevkivskyi/com2ann
>>>>>>>>> >>>>>>>>> >>>> >
>>>>>>>>> >>>>>>>>> >>>> > So don't let that part bother you.
>>>>>>>>> >>>>>>>>> >>>>
>>>>>>>>> >>>>>>>>> >>>> +1, I wouldn't worry about what can be easily
>>>>>>>>> automated.
>>>>>>>>> >>>>>>>>> >>>>
>>>>>>>>> >>>>>>>>> >>>> > I'm curious what other features you'd like to be
>>>>>>>>> using in the Beam source that you cannot now.
>>>>>>>>> >>>>>>>>> >>>>
>>>>>>>>> >>>>>>>>> >>>> I hit things occasionally, e.g. I just ran into
>>>>>>>>> wanting keyword-only
>>>>>>>>> >>>>>>>>> >>>> arguments the other day.
>>>>>>>>> >>>>>>>>> >>>>
>>>>>>>>> >>>>>>>>> >>>> >> It seems the faster we drop support the better.
>>>>>>>>> >>>>>>>>> >>>> >
>>>>>>>>> >>>>>>>>> >>>> >
>>>>>>>>> >>>>>>>>> >>>> > I've already gone over my position on this, but a
>>>>>>>>> refresher for those who care: some of the key vendors that support my
>>>>>>>>> industry will not offer python3-compatible versions of their software until
>>>>>>>>> the 4th quarter of 2020. If Beam switches to python3-only before that
>>>>>>>>> point we may be forced to stop contributing features (note: I'm the guy who
>>>>>>>>> added the type hints :). Every month you can give us would be greatly
>>>>>>>>> appreciated.
>>>>>>>>> >>>>>>>>> >>>>
>>>>>>>>> >>>>>>>>> >>>> As another data point, we're still 80/20 on Py2/Py3
>>>>>>>>> for downloads at
>>>>>>>>> >>>>>>>>> >>>> PyPi [1] (which I've heard should be taken with a
>>>>>>>>> grain of salt, but
>>>>>>>>> >>>>>>>>> >>>> likely isn't totally off). IMHO that ratio needs to
>>>>>>>>> be way higher for
>>>>>>>>> >>>>>>>>> >>>> Python 3 to consider dropping Python 2. It's pretty
>>>>>>>>> noisy, but say it
>>>>>>>>> >>>>>>>>> >>>> doubles every 3 months that would put us at least
>>>>>>>>> mid-year before we
>>>>>>>>> >>>>>>>>> >>>> hit a cross-over point. On the other hand Q4 2020
>>>>>>>>> is probably a
>>>>>>>>> >>>>>>>>> >>>> stretch.
>>>>>>>>> >>>>>>>>> >>>>
>>>>>>>>> >>>>>>>>> >>>> We could consider whether it needs to be an
>>>>>>>>> all-or-nothing thing as
>>>>>>>>> >>>>>>>>> >>>> well. E.g. perhaps some features could be Python 3
>>>>>>>>> only sooner than
>>>>>>>>> >>>>>>>>> >>>> the whole codebase. (This would have to be well
>>>>>>>>> justified.) Another
>>>>>>>>> >>>>>>>>> >>>> mitigation is that it is possible to mix Python 2
>>>>>>>>> and Python 3 in the
>>>>>>>>> >>>>>>>>> >>>> same pipeline with portability, so if there's a
>>>>>>>>> library that you need
>>>>>>>>> >>>>>>>>> >>>> for one DoFn it doesn't mean you have to hold back
>>>>>>>>> your whole
>>>>>>>>> >>>>>>>>> >>>> pipeline.
>>>>>>>>> >>>>>>>>> >>>>
>>>>>>>>> >>>>>>>>> >>>> - Robert
>>>>>>>>> >>>>>>>>> >>>>
>>>>>>>>> >>>>>>>>> >>>> [1] https://pypistats.org/packages/apache-beam ,
>>>>>>>>> and that 20% may just
>>>>>>>>> >>>>>>>>> >>>> be a spike.
>>>>>>>>>
>>>>>>>>
Re: Python2.7 Beam End-of-Life Date
Posted by Kenneth Knowles <ke...@apache.org>.
Regardless of the outcome, it would be good to have some more details here.
Can you share links for people to find out more about Python 3 support in
those products and their timeline? I did some quick searching out of
curiosity but I do not believe I found the authoritative information.
Separately, I am curious what the precise symptoms of dropping Python 2.7
support will be. I admit I haven't been able to follow all the threads on
the topic. Is it moving on to newer dependencies, or also wanting to use
new language features, or something more subtle?
Kenn
On Thu, Jun 18, 2020 at 5:01 PM Chad Dombrova <ch...@gmail.com> wrote:
> Hi all,
> Sorry I've been AWOL. I've been pulled in a number of different
> directions.
>
> We are still increasing our use of Beam, and I have it on my todo list to
> reach out to Kenneth about how Beam could be expanded into the VFX /
> Animation industries.
>
> Our industry uses a number of specialized applications with embedded
> python interpreters. We run Beam inside these interpreters, so we're
> waiting for them to switch to python3.
>
> Here's the status report for python3 adoption in our key applications:
>
> *Maya*: In Beta
> *Houdini*: Released
> *Nuke*: In Beta
> *Katana*: Not started (Alpha?)
>
> I hate to be the one holding the project back, and I understand if you all
> ultimately decide it's untenable to wait any longer. The good news is 3
> out of 4 applications should be ready in the next 2-3 months. I can do
> some investigation into what workarounds might look like for Katana, or
> maybe we can use the Beta version once python3 support arrives, which would
> move our schedule forward.
>
> When would 2.24 release?
>
> -chad
>
>
>
>
>
>
>
>
>
> On Thu, Jun 18, 2020 at 4:33 PM Ahmet Altay <al...@google.com> wrote:
>
>> OK, tweeted the message. Could you share on Slack?
>>
>> On Thu, Jun 18, 2020 at 4:28 PM Valentyn Tymofieiev <va...@google.com>
>> wrote:
>>
>>> Alright, let's publish the message on Twitter and echo on Slack once
>>> that's done.
>>> Thank you!
>>>
>>> On Tue, Jun 16, 2020 at 5:31 PM Ahmet Altay <al...@google.com> wrote:
>>>
>>>> That sounds reasonable to me. I agree, it is better to get specific
>>>> reasons rather than a yes/no answer.
>>>>
>>>> On Tue, Jun 16, 2020 at 1:50 PM Valentyn Tymofieiev <
>>>> valentyn@google.com> wrote:
>>>>
>>>>> After thinking about it for a bit, I am not sure whether a poll would
>>>>> be actionable. For example, if 1000 users provide a positive response and 5
>>>>> users provide a negative response, how do we interpret that and where do
>>>>> we draw a line? How about instead we encourage users to reach out, and tell
>>>>> what is not working for them? For example:
>>>>>
>>>>> Beam is considering making 2.23.0 a final release supporting Py2. If
>>>>> you are not able to switch to Python 3, please let us know why: [short link
>>>>> to user@ thread] [1].
>>>>>
>>>>> Thoughts?
>>>>>
>>>>> [1]
>>>>> https://lists.apache.org/thread.html/r0de71d98d98b213dd1d0c45c1f5642135116f25def5637a5f41c8d29%40%3Cuser.beam.apache.org%3E
>>>>>
>>>>> On Tue, Jun 16, 2020 at 11:50 AM Ahmet Altay <al...@google.com> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Jun 16, 2020 at 11:38 AM Valentyn Tymofieiev <
>>>>>> valentyn@google.com> wrote:
>>>>>>
>>>>>>> I have reached out to user@ for feedback on Python 3 migration[1].
>>>>>>>
>>>>>>
>>>>>> Maybe also ask on slack? There are quite a bit of users there as well.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Could somebody from PMC please help with Twitter poll?
>>>>>>>
>>>>>>
>>>>>> I can try to do this. What question would you like to ask? I do not
>>>>>> know much about twitter polls but I assume they have character limits
>>>>>> similar to regular tweets.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Technically, we can proceed with the change between 2.23.0 and
>>>>>>> 2.24.0, so that's after 2.23.0 is cut and we give sufficient time for users
>>>>>>> to respond.
>>>>>>>
>>>>>>> [1]
>>>>>>> https://lists.apache.org/thread.html/r0de71d98d98b213dd1d0c45c1f5642135116f25def5637a5f41c8d29%40%3Cuser.beam.apache.org%3E
>>>>>>> On Tue, Jun 16, 2020 at 9:22 AM Ismaël Mejía <ie...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Yes we need to poll this outside as Robert proposed.
>>>>>>>>
>>>>>>>> The proposed last support version corresponds with the next release
>>>>>>>> that will be
>>>>>>>> cut in two weeks. Sounds a bit short if we count the time to poll
>>>>>>>> people on this
>>>>>>>> subject but still could be.
>>>>>>>
>>>>>>> Technically, we can proceed with the change between 2.23.0 and
>>>>>>> 2.24.0, so that's after 2.23.0 is cut and we give sufficient time for users
>>>>>>> to respond.
>>>>>>>
>>>>>>>
>>>>>>>> I remember Chad mentioned in this thread the impossibility to get
>>>>>>>> support for
>>>>>>>> python 2 in his industry until the end of the year, Maybe things
>>>>>>>> have improved.
>>>>>>>> Have they?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Jun 16, 2020 at 6:10 PM Robert Bradshaw <
>>>>>>>> robertwb@google.com> wrote:
>>>>>>>> >
>>>>>>>> > I like that option as a concrete proposal. I think we should
>>>>>>>> circulate it more widely (the users list, twitter poll, at least a new
>>>>>>>> thread...), maybe phrasing it as "is there any reason you couldn't migrate
>>>>>>>> to Python 3 (or stick with an older version of Beam) after 2.23 (due to be
>>>>>>>> cut here in a couple of weeks)?" If there is strong concern/pushback, we
>>>>>>>> could consider holding on for one more release.
>>>>>>>> >
>>>>>>>> > On Tue, Jun 16, 2020 at 8:54 AM David Cavazos <
>>>>>>>> dcavazos@google.com> wrote:
>>>>>>>> >>
>>>>>>>> >> +1
>>>>>>>> >>
>>>>>>>> >> On Mon, Jun 15, 2020 at 6:52 PM Udi Meiri <eh...@google.com>
>>>>>>>> wrote:
>>>>>>>> >>>
>>>>>>>> >>> +1
>>>>>>>> >>>
>>>>>>>> >>> On Mon, Jun 15, 2020 at 4:27 PM Ahmet Altay <al...@google.com>
>>>>>>>> wrote:
>>>>>>>> >>>>
>>>>>>>> >>>> As a concrete proposal, could we commit to removing python 2
>>>>>>>> support by 2.24? In other words, mark the next release 2.23 as the last
>>>>>>>> python 2 compatible Beam version.
>>>>>>>> >>>>
>>>>>>>> >>>> On Mon, Jun 15, 2020 at 2:09 PM Valentyn Tymofieiev <
>>>>>>>> valentyn@google.com> wrote:
>>>>>>>> >>>>>
>>>>>>>> >>>>> Another input here:
>>>>>>>> >>>>>
>>>>>>>> >>>>> If you opened a Python PR in the last few days, you probably
>>>>>>>> noticed that our test suites were broken by a transitive dependency of Beam
>>>>>>>> that dropped python 2 support, but did not declare python_requires>=3 in
>>>>>>>> its setup.py [1]. This temporarily broke a subset of Beam Py2 users (who
>>>>>>>> did not explicitly pin the 'rsa' dependency), and still affects Beam
>>>>>>>> development[2].
>>>>>>>> >>>>>
>>>>>>>> >>>>> This is the second time[3] Beam is affected with an issue of
>>>>>>>> this kind, so support of Python 2 starts to slow down our development, and
>>>>>>>> add toil for maintainers of packages we depend on (both directly and
>>>>>>>> transitively).
>>>>>>>> >>>>>
>>>>>>>> >>>>> [1] https://github.com/sybrenstuvel/python-rsa/issues/152
>>>>>>>> >>>>> [2]
>>>>>>>> https://lists.apache.org/thread.html/r9993b40b0c1cb8682ce56013165d4b80fdde0ee469a73bcb9466ddfb%40%3Cdev.beam.apache.org%3E
>>>>>>>> >>>>> [3] https://github.com/hamcrest/PyHamcrest/issues/131
>>>>>>>> >>>>>
>>>>>>>> >>>>> On Tue, Jun 9, 2020 at 4:06 PM Ahmet Altay <al...@google.com>
>>>>>>>> wrote:
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> Thank you for re-opening this Valentyn. I am in favor of
>>>>>>>> EOLing py2 support sooner than later. The reality is that we will not be
>>>>>>>> effectively supporting beam python 2 for a long time while the ecosystem
>>>>>>>> already EOLed python 2. That said, a significant chunk (but no longer a
>>>>>>>> majority) of our users are still using python 2. Upgrades are painful, it
>>>>>>>> might be especially painful nowadays. It would be good to hear counter view
>>>>>>>> points, user voices related to this.
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> On Thu, Jun 4, 2020 at 4:53 PM Valentyn Tymofieiev <
>>>>>>>> valentyn@google.com> wrote:
>>>>>>>> >>>>>>>
>>>>>>>> >>>>>>> Back at the end of February we decided to revisit this
>>>>>>>> conversation in 3 months. Do folks on this thread have any new input or
>>>>>>>> perspective regarding us balancing "user pain/contributor pain/our ability
>>>>>>>> to continuously test with python 2 in a shifting environment"?
>>>>>>>> >>>>>>>
>>>>>>>> >>>>>>> Some new information on my end is that we have been seeing
>>>>>>>> steady adoption of Python 3 among Beam Python users in Dataflow,
>>>>>>>> particularly strong adoption among streaming users, and Dataflow is
>>>>>>>> sunsetting Python 2 support for all released Beam SDKs later this year [1].
>>>>>>>> We will have to remove Python 2 Beam test suites that use Dataflow when
>>>>>>>> Dataflow runner disables Py2 support if this happens before Beam Py2 EOL
>>>>>>>> (when we have to remove all Py2 suites), including performance tests that
>>>>>>>> still use Dataflow on Python 3.
>>>>>>>> >>>>>>>
>>>>>>>> >>>>>>> I am curious how much motivation there is in the community
>>>>>>>> at this moment to continue Py2 support in Beam, whether any previous Py3
>>>>>>>> migration blockers were resolved or any new blockers discovered among Beam
>>>>>>>> users.
>>>>>>>> >>>>>>>
>>>>>>>> >>>>>>> [1]
>>>>>>>> https://cloud.google.com/python/docs/python2-sunset/#dataflow
>>>>>>>> >>>>>>>
>>>>>>>> >>>>>>> On Fri, May 8, 2020 at 3:52 PM Valentyn Tymofieiev <
>>>>>>>> valentyn@google.com> wrote:
>>>>>>>> >>>>>>>>
>>>>>>>> >>>>>>>> That's good news! Thanks for sharing.
>>>>>>>> >>>>>>>>
>>>>>>>> >>>>>>>> Another datapoint, here are a few of Beam's dependencies
>>>>>>>> that no longer release new py2 artifacts (I looked at REQUIRED_PACKAGES +
>>>>>>>> aws, gcp, and interactive extras):
>>>>>>>> >>>>>>>>
>>>>>>>> >>>>>>>> hdfs
>>>>>>>> >>>>>>>> numpy
>>>>>>>> >>>>>>>> pyarrow
>>>>>>>> >>>>>>>> ipython
>>>>>>>> >>>>>>>>
>>>>>>>> >>>>>>>> There are more if we include transitive dependencies and
>>>>>>>> test-only packages. I also remember encountering one issue last month that
>>>>>>>> was broken only on Py2, which we had to go back and fix.
>>>>>>>> >>>>>>>>
>>>>>>>> >>>>>>>> If others have noticed frictions related to ongoing Py2
>>>>>>>> support or have updates on previously mentioned Py3 migration blockers,
>>>>>>>> feel free to post them.
>>>>>>>> >>>>>>>>
>>>>>>>> >>>>>>>> On Fri, May 8, 2020 at 9:19 AM Robert Bradshaw <
>>>>>>>> robertwb@google.com> wrote:
>>>>>>>> >>>>>>>>>
>>>>>>>> >>>>>>>>> It hasn't been 3 months yet, but I wanted to call out a
>>>>>>>> milestone that
>>>>>>>> >>>>>>>>> Python 3 downloads crossed the 50% threshold on pypi, if
>>>>>>>> just briefly.
>>>>>>>> >>>>>>>>>
>>>>>>>> >>>>>>>>> On Thu, Feb 13, 2020 at 12:40 AM Ismaël Mejía <
>>>>>>>> iemejia@gmail.com> wrote:
>>>>>>>> >>>>>>>>> >
>>>>>>>> >>>>>>>>> > > I would suggest re-evaluating this within the next 3
>>>>>>>> months again. We need to balance between user pain/contributor pain/our
>>>>>>>> ability to continuously test with python 2 in a shifting environment.
>>>>>>>> >>>>>>>>> >
>>>>>>>> >>>>>>>>> > Good idea for the in 3 months evaluation, at that point
>>>>>>>> also distributions will probably be phasing out python2 by default which
>>>>>>>> definitely help in this direction.
>>>>>>>> >>>>>>>>> > Thanks for updating the roadmap Ahmet
>>>>>>>> >>>>>>>>> >
>>>>>>>> >>>>>>>>> >
>>>>>>>> >>>>>>>>> > On Thu, Feb 13, 2020 at 2:49 AM Ahmet Altay <
>>>>>>>> altay@google.com> wrote:
>>>>>>>> >>>>>>>>> >>
>>>>>>>> >>>>>>>>> >>
>>>>>>>> >>>>>>>>> >>
>>>>>>>> >>>>>>>>> >> On Wed, Feb 12, 2020 at 1:29 AM Ismaël Mejía <
>>>>>>>> iemejia@gmail.com> wrote:
>>>>>>>> >>>>>>>>> >>>
>>>>>>>> >>>>>>>>> >>> I am with Chad on this, we should probably extend it
>>>>>>>> a bit more, even if it
>>>>>>>> >>>>>>>>> >>> makes us struggle a bit at least we have some
>>>>>>>> workarounds as Robert suggests,
>>>>>>>> >>>>>>>>> >>> and as Chad said there are still many people playing
>>>>>>>> the python 3 catchup game,
>>>>>>>> >>>>>>>>> >>> so worth to support those users.
>>>>>>>> >>>>>>>>> >>>
>>>>>>>> >>>>>>>>> >>>
>>>>>>>> >>>>>>>>> >>> But maybe it is worth to evaluate the current state
>>>>>>>> later in the year.
>>>>>>>> >>>>>>>>> >>
>>>>>>>> >>>>>>>>> >>
>>>>>>>> >>>>>>>>> >> I would suggest re-evaluating this within the next 3
>>>>>>>> months again. We need to balance between user pain/contributor pain/our
>>>>>>>> ability to continuously test with python 2 in a shifting environment.
>>>>>>>> >>>>>>>>> >>
>>>>>>>> >>>>>>>>> >>>
>>>>>>>> >>>>>>>>> >>> In the
>>>>>>>> >>>>>>>>> >>> meantime can someone please update our Roadmap in the
>>>>>>>> website with this info and
>>>>>>>> >>>>>>>>> >>> where we are with Python 3 support (it looks not up
>>>>>>>> to date).
>>>>>>>> >>>>>>>>> >>> https://beam.apache.org/roadmap/
>>>>>>>> >>>>>>>>> >>
>>>>>>>> >>>>>>>>> >>
>>>>>>>> >>>>>>>>> >> I made a minor change to update that page (
>>>>>>>> https://github.com/apache/beam/pull/10848). A more comprehensive
>>>>>>>> update to that page and linked (
>>>>>>>> https://beam.apache.org/roadmap/python-sdk/#python-3-support)
>>>>>>>> would still be welcome.
>>>>>>>> >>>>>>>>> >>
>>>>>>>> >>>>>>>>> >>>
>>>>>>>> >>>>>>>>> >>>
>>>>>>>> >>>>>>>>> >>> - Ismaël
>>>>>>>> >>>>>>>>> >>>
>>>>>>>> >>>>>>>>> >>>
>>>>>>>> >>>>>>>>> >>> On Tue, Feb 4, 2020 at 10:49 PM Robert Bradshaw <
>>>>>>>> robertwb@google.com> wrote:
>>>>>>>> >>>>>>>>> >>>>
>>>>>>>> >>>>>>>>> >>>> On Tue, Feb 4, 2020 at 12:12 PM Chad Dombrova <
>>>>>>>> chadrik@gmail.com> wrote:
>>>>>>>> >>>>>>>>> >>>> >>
>>>>>>>> >>>>>>>>> >>>> >> Not to mention that all the nice work for the
>>>>>>>> type hints will have to be redone in the for 3.x.
>>>>>>>> >>>>>>>>> >>>> >
>>>>>>>> >>>>>>>>> >>>> > Note that there's a tool for automatically
>>>>>>>> converting type comments to annotations:
>>>>>>>> https://github.com/ilevkivskyi/com2ann
>>>>>>>> >>>>>>>>> >>>> >
>>>>>>>> >>>>>>>>> >>>> > So don't let that part bother you.
>>>>>>>> >>>>>>>>> >>>>
>>>>>>>> >>>>>>>>> >>>> +1, I wouldn't worry about what can be easily
>>>>>>>> automated.
>>>>>>>> >>>>>>>>> >>>>
>>>>>>>> >>>>>>>>> >>>> > I'm curious what other features you'd like to be
>>>>>>>> using in the Beam source that you cannot now.
>>>>>>>> >>>>>>>>> >>>>
>>>>>>>> >>>>>>>>> >>>> I hit things occasionally, e.g. I just ran into
>>>>>>>> wanting keyword-only
>>>>>>>> >>>>>>>>> >>>> arguments the other day.
>>>>>>>> >>>>>>>>> >>>>
>>>>>>>> >>>>>>>>> >>>> >> It seems the faster we drop support the better.
>>>>>>>> >>>>>>>>> >>>> >
>>>>>>>> >>>>>>>>> >>>> >
>>>>>>>> >>>>>>>>> >>>> > I've already gone over my position on this, but a
>>>>>>>> refresher for those who care: some of the key vendors that support my
>>>>>>>> industry will not offer python3-compatible versions of their software until
>>>>>>>> the 4th quarter of 2020. If Beam switches to python3-only before that
>>>>>>>> point we may be forced to stop contributing features (note: I'm the guy who
>>>>>>>> added the type hints :). Every month you can give us would be greatly
>>>>>>>> appreciated.
>>>>>>>> >>>>>>>>> >>>>
>>>>>>>> >>>>>>>>> >>>> As another data point, we're still 80/20 on Py2/Py3
>>>>>>>> for downloads at
>>>>>>>> >>>>>>>>> >>>> PyPi [1] (which I've heard should be taken with a
>>>>>>>> grain of salt, but
>>>>>>>> >>>>>>>>> >>>> likely isn't totally off). IMHO that ratio needs to
>>>>>>>> be way higher for
>>>>>>>> >>>>>>>>> >>>> Python 3 to consider dropping Python 2. It's pretty
>>>>>>>> noisy, but say it
>>>>>>>> >>>>>>>>> >>>> doubles every 3 months that would put us at least
>>>>>>>> mid-year before we
>>>>>>>> >>>>>>>>> >>>> hit a cross-over point. On the other hand Q4 2020 is
>>>>>>>> probably a
>>>>>>>> >>>>>>>>> >>>> stretch.
>>>>>>>> >>>>>>>>> >>>>
>>>>>>>> >>>>>>>>> >>>> We could consider whether it needs to be an
>>>>>>>> all-or-nothing thing as
>>>>>>>> >>>>>>>>> >>>> well. E.g. perhaps some features could be Python 3
>>>>>>>> only sooner than
>>>>>>>> >>>>>>>>> >>>> the whole codebase. (This would have to be well
>>>>>>>> justified.) Another
>>>>>>>> >>>>>>>>> >>>> mitigation is that it is possible to mix Python 2
>>>>>>>> and Python 3 in the
>>>>>>>> >>>>>>>>> >>>> same pipeline with portability, so if there's a
>>>>>>>> library that you need
>>>>>>>> >>>>>>>>> >>>> for one DoFn it doesn't mean you have to hold back
>>>>>>>> your whole
>>>>>>>> >>>>>>>>> >>>> pipeline.
>>>>>>>> >>>>>>>>> >>>>
>>>>>>>> >>>>>>>>> >>>> - Robert
>>>>>>>> >>>>>>>>> >>>>
>>>>>>>> >>>>>>>>> >>>> [1] https://pypistats.org/packages/apache-beam ,
>>>>>>>> and that 20% may just
>>>>>>>> >>>>>>>>> >>>> be a spike.
>>>>>>>>
>>>>>>>
Re: Python2.7 Beam End-of-Life Date
Posted by Valentyn Tymofieiev <va...@google.com>.
I have reviewed Slack, Twitter and User@ channels [1-3] we used to
communicate our consideration to make 2.23.0 a final release supporting
Py2.
I did not see any objections or negative feedback there.
I suggest we hold a VOTE to decide whether 2.24.0 should be the final
release supporting Py2. I can send the voting email.
[1]
https://app.slack.com/client/T4S1WH2J3/CBDNLQZM1/thread/C9H0YNP3P-1592524218.050700
[2] https://twitter.com/ApacheBeam/status/1273760375570194432
[3]
https://lists.apache.org/thread.html/r0de71d98d98b213dd1d0c45c1f5642135116f25def5637a5f41c8d29%40%3Cuser.beam.apache.org%3E
On Thu, Jul 23, 2020 at 10:57 AM Robert Bradshaw <ro...@google.com>
wrote:
> The release cut date for 2.24 is in a couple of weeks; if this is the last
> release supporting 2.7 we should make the call and announce it soon.
>
> On Mon, Jul 6, 2020 at 2:26 PM Robert Bradshaw <ro...@google.com>
> wrote:
>
>> Note that just because Beam drops 2.x support in new releases doesn't
>> mean that the old releases won't continue to work. One can even use an
>> expansion service to run 2.x transforms (on an older version of Beam)
>> within a Python 3 pipeline running the newest version of Beam until
>> such a time that (possibly incrementally) the dependent libraries
>> catch up, if it comes to that. Not that this will be ideal.
>>
>> Now that the 2.23 release branch has been cut, we *could* remove 2.7
>> support if we will actually plan to remove it in 2.24. (One concern,
>> however, is how tied the release testing infrastructure is to
>> mainline...) However, other than Chad's response (the VFX / Animation
>> isn't quite there yet) it seems we still don't have a very strong
>> signal either direction. Pypi downloads are still hovering around 50%.
>> Did we hear anything back from twitter/slack?
>>
>> My inclination would be to publish loudly that 2.24 would be the last
>> Beam to support Python 2.7 (this is pushing it back one release), so
>> if you're still stuck on it make sure it has everything you need by
>> the release cut date (which is still in the future).
>>
>>
>> On Mon, Jul 6, 2020 at 9:52 AM David Yan <da...@google.com> wrote:
>> >
>> > +1 for removing Python 2.7 support sooner than later.
>> >
>> > I recently added a small feature in Beam Python and I found that having
>> to write code that worked with Python2 was quite awkward and time consuming
>> (needing to make sure code works for both 2 and 3 and doubling the Jenkins
>> running time), and I can imagine that may hinder or even discourage new
>> contributions.
>> >
>> > On Thu, Jun 18, 2020 at 5:12 PM Ahmet Altay <al...@google.com> wrote:
>> >>
>> >> Good to hear from you and good to hear the news. Release branch cut
>> date for 2.24 is 8/12.
>> >>
>> >> On Thu, Jun 18, 2020 at 5:01 PM Chad Dombrova <ch...@gmail.com>
>> wrote:
>> >>>
>> >>> Hi all,
>> >>> Sorry I've been AWOL. I've been pulled in a number of different
>> directions.
>> >>>
>> >>> We are still increasing our use of Beam, and I have it on my todo
>> list to reach out to Kenneth about how Beam could be expanded into the VFX
>> / Animation industries.
>> >>>
>> >>> Our industry uses a number of specialized applications with embedded
>> python interpreters. We run Beam inside these interpreters, so we're
>> waiting for them to switch to python3.
>> >>>
>> >>> Here's the status report for python3 adoption in our key applications:
>> >>>
>> >>> Maya: In Beta
>> >>> Houdini: Released
>> >>> Nuke: In Beta
>> >>> Katana: Not started (Alpha?)
>> >>>
>> >>> I hate to be the one holding the project back, and I understand if
>> you all ultimately decide it's untenable to wait any longer. The good news
>> is 3 out of 4 applications should be ready in the next 2-3 months. I can
>> do some investigation into what workarounds might look like for Katana, or
>> maybe we can use the Beta version once python3 support arrives, which would
>> move our schedule forward.
>> >>>
>> >>> When would 2.24 release?
>> >>>
>> >>> -chad
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> On Thu, Jun 18, 2020 at 4:33 PM Ahmet Altay <al...@google.com> wrote:
>> >>>>
>> >>>> OK, tweeted the message. Could you share on Slack?
>> >>>>
>> >>>> On Thu, Jun 18, 2020 at 4:28 PM Valentyn Tymofieiev <
>> valentyn@google.com> wrote:
>> >>>>>
>> >>>>> Alright, let's publish the message on Twitter and echo on Slack
>> once that's done.
>> >>>>> Thank you!
>> >>>>>
>> >>>>> On Tue, Jun 16, 2020 at 5:31 PM Ahmet Altay <al...@google.com>
>> wrote:
>> >>>>>>
>> >>>>>> That sounds reasonable to me. I agree, it is better to get
>> specific reasons rather than a yes/no answer.
>> >>>>>>
>> >>>>>> On Tue, Jun 16, 2020 at 1:50 PM Valentyn Tymofieiev <
>> valentyn@google.com> wrote:
>> >>>>>>>
>> >>>>>>> After thinking about it for a bit, I am not sure whether a poll
>> would be actionable. For example, if 1000 users provide a positive response
>> and 5 users provide a negative response, how do we interpret that and
>> where do we draw a line? How about instead we encourage users to reach out,
>> and tell what is not working for them? For example:
>> >>>>>>>
>> >>>>>>> Beam is considering making 2.23.0 a final release supporting Py2.
>> If you are not able to switch to Python 3, please let us know why: [short
>> link to user@ thread] [1].
>> >>>>>>>
>> >>>>>>> Thoughts?
>> >>>>>>>
>> >>>>>>> [1]
>> https://lists.apache.org/thread.html/r0de71d98d98b213dd1d0c45c1f5642135116f25def5637a5f41c8d29%40%3Cuser.beam.apache.org%3E
>> >>>>>>>
>> >>>>>>> On Tue, Jun 16, 2020 at 11:50 AM Ahmet Altay <al...@google.com>
>> wrote:
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> On Tue, Jun 16, 2020 at 11:38 AM Valentyn Tymofieiev <
>> valentyn@google.com> wrote:
>> >>>>>>>>>
>> >>>>>>>>> I have reached out to user@ for feedback on Python 3
>> migration[1].
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> Maybe also ask on slack? There are quite a bit of users there as
>> well.
>> >>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> Could somebody from PMC please help with Twitter poll?
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> I can try to do this. What question would you like to ask? I do
>> not know much about twitter polls but I assume they have character limits
>> similar to regular tweets.
>> >>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> Technically, we can proceed with the change between 2.23.0 and
>> 2.24.0, so that's after 2.23.0 is cut and we give sufficient time for users
>> to respond.
>> >>>>>>>>>
>> >>>>>>>>> [1]
>> https://lists.apache.org/thread.html/r0de71d98d98b213dd1d0c45c1f5642135116f25def5637a5f41c8d29%40%3Cuser.beam.apache.org%3E
>> >>>>>>>>> On Tue, Jun 16, 2020 at 9:22 AM Ismaël Mejía <ie...@gmail.com>
>> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>> Yes we need to poll this outside as Robert proposed.
>> >>>>>>>>>>
>> >>>>>>>>>> The proposed last support version corresponds with the next
>> release that will be
>> >>>>>>>>>> cut in two weeks. Sounds a bit short if we count the time to
>> poll people on this
>> >>>>>>>>>> subject but still could be.
>> >>>>>>>>>
>> >>>>>>>>> Technically, we can proceed with the change between 2.23.0 and
>> 2.24.0, so that's after 2.23.0 is cut and we give sufficient time for users
>> to respond.
>> >>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> I remember Chad mentioned in this thread the impossibility to
>> get support for
>> >>>>>>>>>> python 2 in his industry until the end of the year, Maybe
>> things have improved.
>> >>>>>>>>>> Have they?
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> On Tue, Jun 16, 2020 at 6:10 PM Robert Bradshaw <
>> robertwb@google.com> wrote:
>> >>>>>>>>>> >
>> >>>>>>>>>> > I like that option as a concrete proposal. I think we should
>> circulate it more widely (the users list, twitter poll, at least a new
>> thread...), maybe phrasing it as "is there any reason you couldn't migrate
>> to Python 3 (or stick with an older version of Beam) after 2.23 (due to be
>> cut here in a couple of weeks)?" If there is strong concern/pushback, we
>> could consider holding on for one more release.
>> >>>>>>>>>> >
>> >>>>>>>>>> > On Tue, Jun 16, 2020 at 8:54 AM David Cavazos <
>> dcavazos@google.com> wrote:
>> >>>>>>>>>> >>
>> >>>>>>>>>> >> +1
>> >>>>>>>>>> >>
>> >>>>>>>>>> >> On Mon, Jun 15, 2020 at 6:52 PM Udi Meiri <eh...@google.com>
>> wrote:
>> >>>>>>>>>> >>>
>> >>>>>>>>>> >>> +1
>> >>>>>>>>>> >>>
>> >>>>>>>>>> >>> On Mon, Jun 15, 2020 at 4:27 PM Ahmet Altay <
>> altay@google.com> wrote:
>> >>>>>>>>>> >>>>
>> >>>>>>>>>> >>>> As a concrete proposal, could we commit to removing
>> python 2 support by 2.24? In other words, mark the next release 2.23 as the
>> last python 2 compatible Beam version.
>> >>>>>>>>>> >>>>
>> >>>>>>>>>> >>>> On Mon, Jun 15, 2020 at 2:09 PM Valentyn Tymofieiev <
>> valentyn@google.com> wrote:
>> >>>>>>>>>> >>>>>
>> >>>>>>>>>> >>>>> Another input here:
>> >>>>>>>>>> >>>>>
>> >>>>>>>>>> >>>>> If you opened a Python PR in the last few days, you
>> probably noticed that our test suites were broken by a transitive
>> dependency of Beam that dropped python 2 support, but did not declare
>> python_requires>=3 in its setup.py [1]. This temporarily broke a subset of
>> Beam Py2 users (who did not explicitly pin the 'rsa' dependency), and still
>> affects Beam development[2].
>> >>>>>>>>>> >>>>>
>> >>>>>>>>>> >>>>> This is the second time[3] Beam is affected with an
>> issue of this kind, so support of Python 2 starts to slow down our
>> development, and add toil for maintainers of packages we depend on (both
>> directly and transitively).
>> >>>>>>>>>> >>>>>
>> >>>>>>>>>> >>>>> [1]
>> https://github.com/sybrenstuvel/python-rsa/issues/152
>> >>>>>>>>>> >>>>> [2]
>> https://lists.apache.org/thread.html/r9993b40b0c1cb8682ce56013165d4b80fdde0ee469a73bcb9466ddfb%40%3Cdev.beam.apache.org%3E
>> >>>>>>>>>> >>>>> [3] https://github.com/hamcrest/PyHamcrest/issues/131
>> >>>>>>>>>> >>>>>
>> >>>>>>>>>> >>>>> On Tue, Jun 9, 2020 at 4:06 PM Ahmet Altay <
>> altay@google.com> wrote:
>> >>>>>>>>>> >>>>>>
>> >>>>>>>>>> >>>>>> Thank you for re-opening this Valentyn. I am in favor
>> of EOLing py2 support sooner than later. The reality is that we will not be
>> effectively supporting beam python 2 for a long time while the ecosystem
>> already EOLed python 2. That said, a significant chunk (but no longer a
>> majority) of our users are still using python 2. Upgrades are painful, it
>> might be especially painful nowadays. It would be good to hear counter view
>> points, user voices related to this.
>> >>>>>>>>>> >>>>>>
>> >>>>>>>>>> >>>>>> On Thu, Jun 4, 2020 at 4:53 PM Valentyn Tymofieiev <
>> valentyn@google.com> wrote:
>> >>>>>>>>>> >>>>>>>
>> >>>>>>>>>> >>>>>>> Back at the end of February we decided to revisit this
>> conversation in 3 months. Do folks on this thread have any new input or
>> perspective regarding us balancing "user pain/contributor pain/our ability
>> to continuously test with python 2 in a shifting environment"?
>> >>>>>>>>>> >>>>>>>
>> >>>>>>>>>> >>>>>>> Some new information on my end is that we have been
>> seeing steady adoption of Python 3 among Beam Python users in Dataflow,
>> particularly strong adoption among streaming users, and Dataflow is
>> sunsetting Python 2 support for all released Beam SDKs later this year [1].
>> We will have to remove Python 2 Beam test suites that use Dataflow when
>> Dataflow runner disables Py2 support if this happens before Beam Py2 EOL
>> (when we have to remove all Py2 suites), including performance tests that
>> still use Dataflow on Python 3.
>> >>>>>>>>>> >>>>>>>
>> >>>>>>>>>> >>>>>>> I am curious how much motivation there is in the
>> community at this moment to continue Py2 support in Beam, whether any
>> previous Py3 migration blockers were resolved or any new blockers
>> discovered among Beam users.
>> >>>>>>>>>> >>>>>>>
>> >>>>>>>>>> >>>>>>> [1]
>> https://cloud.google.com/python/docs/python2-sunset/#dataflow
>> >>>>>>>>>> >>>>>>>
>> >>>>>>>>>> >>>>>>> On Fri, May 8, 2020 at 3:52 PM Valentyn Tymofieiev <
>> valentyn@google.com> wrote:
>> >>>>>>>>>> >>>>>>>>
>> >>>>>>>>>> >>>>>>>> That's good news! Thanks for sharing.
>> >>>>>>>>>> >>>>>>>>
>> >>>>>>>>>> >>>>>>>> Another datapoint, here are a few of Beam's
>> dependencies that no longer release new py2 artifacts (I looked at
>> REQUIRED_PACKAGES + aws, gcp, and interactive extras):
>> >>>>>>>>>> >>>>>>>>
>> >>>>>>>>>> >>>>>>>> hdfs
>> >>>>>>>>>> >>>>>>>> numpy
>> >>>>>>>>>> >>>>>>>> pyarrow
>> >>>>>>>>>> >>>>>>>> ipython
>> >>>>>>>>>> >>>>>>>>
>> >>>>>>>>>> >>>>>>>> There are more if we include transitive dependencies
>> and test-only packages. I also remember encountering one issue last month
>> that was broken only on Py2, which we had to go back and fix.
>> >>>>>>>>>> >>>>>>>>
>> >>>>>>>>>> >>>>>>>> If others have noticed frictions related to ongoing
>> Py2 support or have updates on previously mentioned Py3 migration blockers,
>> feel free to post them.
>> >>>>>>>>>> >>>>>>>>
>> >>>>>>>>>> >>>>>>>> On Fri, May 8, 2020 at 9:19 AM Robert Bradshaw <
>> robertwb@google.com> wrote:
>> >>>>>>>>>> >>>>>>>>>
>> >>>>>>>>>> >>>>>>>>> It hasn't been 3 months yet, but I wanted to call
>> out a milestone that
>> >>>>>>>>>> >>>>>>>>> Python 3 downloads crossed the 50% threshold on
>> pypi, if just briefly.
>> >>>>>>>>>> >>>>>>>>>
>> >>>>>>>>>> >>>>>>>>> On Thu, Feb 13, 2020 at 12:40 AM Ismaël Mejía <
>> iemejia@gmail.com> wrote:
>> >>>>>>>>>> >>>>>>>>> >
>> >>>>>>>>>> >>>>>>>>> > > I would suggest re-evaluating this within the
>> next 3 months again. We need to balance between user pain/contributor
>> pain/our ability to continuously test with python 2 in a shifting
>> environment.
>> >>>>>>>>>> >>>>>>>>> >
>> >>>>>>>>>> >>>>>>>>> > Good idea for the in 3 months evaluation, at that
>> point also distributions will probably be phasing out python2 by default
>> which definitely help in this direction.
>> >>>>>>>>>> >>>>>>>>> > Thanks for updating the roadmap Ahmet
>> >>>>>>>>>> >>>>>>>>> >
>> >>>>>>>>>> >>>>>>>>> >
>> >>>>>>>>>> >>>>>>>>> > On Thu, Feb 13, 2020 at 2:49 AM Ahmet Altay <
>> altay@google.com> wrote:
>> >>>>>>>>>> >>>>>>>>> >>
>> >>>>>>>>>> >>>>>>>>> >>
>> >>>>>>>>>> >>>>>>>>> >>
>> >>>>>>>>>> >>>>>>>>> >> On Wed, Feb 12, 2020 at 1:29 AM Ismaël Mejía <
>> iemejia@gmail.com> wrote:
>> >>>>>>>>>> >>>>>>>>> >>>
>> >>>>>>>>>> >>>>>>>>> >>> I am with Chad on this, we should probably
>> extend it a bit more, even if it
>> >>>>>>>>>> >>>>>>>>> >>> makes us struggle a bit at least we have some
>> workarounds as Robert suggests,
>> >>>>>>>>>> >>>>>>>>> >>> and as Chad said there are still many people
>> playing the python 3 catchup game,
>> >>>>>>>>>> >>>>>>>>> >>> so worth to support those users.
>> >>>>>>>>>> >>>>>>>>> >>>
>> >>>>>>>>>> >>>>>>>>> >>>
>> >>>>>>>>>> >>>>>>>>> >>> But maybe it is worth to evaluate the current
>> state later in the year.
>> >>>>>>>>>> >>>>>>>>> >>
>> >>>>>>>>>> >>>>>>>>> >>
>> >>>>>>>>>> >>>>>>>>> >> I would suggest re-evaluating this within the
>> next 3 months again. We need to balance between user pain/contributor
>> pain/our ability to continuously test with python 2 in a shifting
>> environment.
>> >>>>>>>>>> >>>>>>>>> >>
>> >>>>>>>>>> >>>>>>>>> >>>
>> >>>>>>>>>> >>>>>>>>> >>> In the
>> >>>>>>>>>> >>>>>>>>> >>> meantime can someone please update our Roadmap
>> in the website with this info and
>> >>>>>>>>>> >>>>>>>>> >>> where we are with Python 3 support (it looks not
>> up to date).
>> >>>>>>>>>> >>>>>>>>> >>> https://beam.apache.org/roadmap/
>> >>>>>>>>>> >>>>>>>>> >>
>> >>>>>>>>>> >>>>>>>>> >>
>> >>>>>>>>>> >>>>>>>>> >> I made a minor change to update that page (
>> https://github.com/apache/beam/pull/10848). A more comprehensive update
>> to that page and linked (
>> https://beam.apache.org/roadmap/python-sdk/#python-3-support) would
>> still be welcome.
>> >>>>>>>>>> >>>>>>>>> >>
>> >>>>>>>>>> >>>>>>>>> >>>
>> >>>>>>>>>> >>>>>>>>> >>>
>> >>>>>>>>>> >>>>>>>>> >>> - Ismaël
>> >>>>>>>>>> >>>>>>>>> >>>
>> >>>>>>>>>> >>>>>>>>> >>>
>> >>>>>>>>>> >>>>>>>>> >>> On Tue, Feb 4, 2020 at 10:49 PM Robert Bradshaw <
>> robertwb@google.com> wrote:
>> >>>>>>>>>> >>>>>>>>> >>>>
>> >>>>>>>>>> >>>>>>>>> >>>> On Tue, Feb 4, 2020 at 12:12 PM Chad Dombrova <
>> chadrik@gmail.com> wrote:
>> >>>>>>>>>> >>>>>>>>> >>>> >>
>> >>>>>>>>>> >>>>>>>>> >>>> >> Not to mention that all the nice work for
>> the type hints will have to be redone in the for 3.x.
>> >>>>>>>>>> >>>>>>>>> >>>> >
>> >>>>>>>>>> >>>>>>>>> >>>> > Note that there's a tool for automatically
>> converting type comments to annotations:
>> https://github.com/ilevkivskyi/com2ann
>> >>>>>>>>>> >>>>>>>>> >>>> >
>> >>>>>>>>>> >>>>>>>>> >>>> > So don't let that part bother you.
>> >>>>>>>>>> >>>>>>>>> >>>>
>> >>>>>>>>>> >>>>>>>>> >>>> +1, I wouldn't worry about what can be easily
>> automated.
>> >>>>>>>>>> >>>>>>>>> >>>>
>> >>>>>>>>>> >>>>>>>>> >>>> > I'm curious what other features you'd like to
>> be using in the Beam source that you cannot now.
>> >>>>>>>>>> >>>>>>>>> >>>>
>> >>>>>>>>>> >>>>>>>>> >>>> I hit things occasionally, e.g. I just ran into
>> wanting keyword-only
>> >>>>>>>>>> >>>>>>>>> >>>> arguments the other day.
>> >>>>>>>>>> >>>>>>>>> >>>>
>> >>>>>>>>>> >>>>>>>>> >>>> >> It seems the faster we drop support the
>> better.
>> >>>>>>>>>> >>>>>>>>> >>>> >
>> >>>>>>>>>> >>>>>>>>> >>>> >
>> >>>>>>>>>> >>>>>>>>> >>>> > I've already gone over my position on this,
>> but a refresher for those who care: some of the key vendors that support
>> my industry will not offer python3-compatible versions of their software
>> until the 4th quarter of 2020. If Beam switches to python3-only before
>> that point we may be forced to stop contributing features (note: I'm the
>> guy who added the type hints :). Every month you can give us would be
>> greatly appreciated.
>> >>>>>>>>>> >>>>>>>>> >>>>
>> >>>>>>>>>> >>>>>>>>> >>>> As another data point, we're still 80/20 on
>> Py2/Py3 for downloads at
>> >>>>>>>>>> >>>>>>>>> >>>> PyPi [1] (which I've heard should be taken with
>> a grain of salt, but
>> >>>>>>>>>> >>>>>>>>> >>>> likely isn't totally off). IMHO that ratio
>> needs to be way higher for
>> >>>>>>>>>> >>>>>>>>> >>>> Python 3 to consider dropping Python 2. It's
>> pretty noisy, but say it
>> >>>>>>>>>> >>>>>>>>> >>>> doubles every 3 months that would put us at
>> least mid-year before we
>> >>>>>>>>>> >>>>>>>>> >>>> hit a cross-over point. On the other hand Q4
>> 2020 is probably a
>> >>>>>>>>>> >>>>>>>>> >>>> stretch.
>> >>>>>>>>>> >>>>>>>>> >>>>
>> >>>>>>>>>> >>>>>>>>> >>>> We could consider whether it needs to be an
>> all-or-nothing thing as
>> >>>>>>>>>> >>>>>>>>> >>>> well. E.g. perhaps some features could be
>> Python 3 only sooner than
>> >>>>>>>>>> >>>>>>>>> >>>> the whole codebase. (This would have to be well
>> justified.) Another
>> >>>>>>>>>> >>>>>>>>> >>>> mitigation is that it is possible to mix Python
>> 2 and Python 3 in the
>> >>>>>>>>>> >>>>>>>>> >>>> same pipeline with portability, so if there's a
>> library that you need
>> >>>>>>>>>> >>>>>>>>> >>>> for one DoFn it doesn't mean you have to hold
>> back your whole
>> >>>>>>>>>> >>>>>>>>> >>>> pipeline.
>> >>>>>>>>>> >>>>>>>>> >>>>
>> >>>>>>>>>> >>>>>>>>> >>>> - Robert
>> >>>>>>>>>> >>>>>>>>> >>>>
>> >>>>>>>>>> >>>>>>>>> >>>> [1] https://pypistats.org/packages/apache-beam
>> , and that 20% may just
>> >>>>>>>>>> >>>>>>>>> >>>> be a spike.
>>
>
Re: Python2.7 Beam End-of-Life Date
Posted by Robert Bradshaw <ro...@google.com>.
The release cut date for 2.24 is in a couple of weeks; if this is the last
release supporting 2.7 we should make the call and announce it soon.
On Mon, Jul 6, 2020 at 2:26 PM Robert Bradshaw <ro...@google.com> wrote:
> Note that just because Beam drops 2.x support in new releases doesn't
> mean that the old releases won't continue to work. One can even use an
> expansion service to run 2.x transforms (on an older version of Beam)
> within a Python 3 pipeline running the newest version of Beam until
> such a time that (possibly incrementally) the dependent libraries
> catch up, if it comes to that. Not that this will be ideal.
>
> Now that the 2.23 release branch has been cut, we *could* remove 2.7
> support if we will actually plan to remove it in 2.24. (One concern,
> however, is how tied the release testing infrastructure is to
> mainline...) However, other than Chad's response (the VFX / Animation
> isn't quite there yet) it seems we still don't have a very strong
> signal either direction. Pypi downloads are still hovering around 50%.
> Did we hear anything back from twitter/slack?
>
> My inclination would be to publish loudly that 2.24 would be the last
> Beam to support Python 2.7 (this is pushing it back one release), so
> if you're still stuck on it make sure it has everything you need by
> the release cut date (which is still in the future).
>
>
> On Mon, Jul 6, 2020 at 9:52 AM David Yan <da...@google.com> wrote:
> >
> > +1 for removing Python 2.7 support sooner than later.
> >
> > I recently added a small feature in Beam Python and I found that having
> to write code that worked with Python2 was quite awkward and time consuming
> (needing to make sure code works for both 2 and 3 and doubling the Jenkins
> running time), and I can imagine that may hinder or even discourage new
> contributions.
> >
> > On Thu, Jun 18, 2020 at 5:12 PM Ahmet Altay <al...@google.com> wrote:
> >>
> >> Good to hear from you and good to hear the news. Release branch cut
> date for 2.24 is 8/12.
> >>
> >> On Thu, Jun 18, 2020 at 5:01 PM Chad Dombrova <ch...@gmail.com>
> wrote:
> >>>
> >>> Hi all,
> >>> Sorry I've been AWOL. I've been pulled in a number of different
> directions.
> >>>
> >>> We are still increasing our use of Beam, and I have it on my todo list
> to reach out to Kenneth about how Beam could be expanded into the VFX /
> Animation industries.
> >>>
> >>> Our industry uses a number of specialized applications with embedded
> python interpreters. We run Beam inside these interpreters, so we're
> waiting for them to switch to python3.
> >>>
> >>> Here's the status report for python3 adoption in our key applications:
> >>>
> >>> Maya: In Beta
> >>> Houdini: Released
> >>> Nuke: In Beta
> >>> Katana: Not started (Alpha?)
> >>>
> >>> I hate to be the one holding the project back, and I understand if you
> all ultimately decide it's untenable to wait any longer. The good news is
> 3 out of 4 applications should be ready in the next 2-3 months. I can do
> some investigation into what workarounds might look like for Katana, or
> maybe we can use the Beta version once python3 support arrives, which would
> move our schedule forward.
> >>>
> >>> When would 2.24 release?
> >>>
> >>> -chad
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Thu, Jun 18, 2020 at 4:33 PM Ahmet Altay <al...@google.com> wrote:
> >>>>
> >>>> OK, tweeted the message. Could you share on Slack?
> >>>>
> >>>> On Thu, Jun 18, 2020 at 4:28 PM Valentyn Tymofieiev <
> valentyn@google.com> wrote:
> >>>>>
> >>>>> Alright, let's publish the message on Twitter and echo on Slack once
> that's done.
> >>>>> Thank you!
> >>>>>
> >>>>> On Tue, Jun 16, 2020 at 5:31 PM Ahmet Altay <al...@google.com>
> wrote:
> >>>>>>
> >>>>>> That sounds reasonable to me. I agree, it is better to get specific
> reasons rather than a yes/no answer.
> >>>>>>
> >>>>>> On Tue, Jun 16, 2020 at 1:50 PM Valentyn Tymofieiev <
> valentyn@google.com> wrote:
> >>>>>>>
> >>>>>>> After thinking about it for a bit, I am not sure whether a poll
> would be actionable. For example, if 1000 users provide a positive response
> and 5 users provide a negative response, how do we interpret that and
> where do we draw a line? How about instead we encourage users to reach out,
> and tell what is not working for them? For example:
> >>>>>>>
> >>>>>>> Beam is considering making 2.23.0 a final release supporting Py2.
> If you are not able to switch to Python 3, please let us know why: [short
> link to user@ thread] [1].
> >>>>>>>
> >>>>>>> Thoughts?
> >>>>>>>
> >>>>>>> [1]
> https://lists.apache.org/thread.html/r0de71d98d98b213dd1d0c45c1f5642135116f25def5637a5f41c8d29%40%3Cuser.beam.apache.org%3E
> >>>>>>>
> >>>>>>> On Tue, Jun 16, 2020 at 11:50 AM Ahmet Altay <al...@google.com>
> wrote:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Tue, Jun 16, 2020 at 11:38 AM Valentyn Tymofieiev <
> valentyn@google.com> wrote:
> >>>>>>>>>
> >>>>>>>>> I have reached out to user@ for feedback on Python 3
> migration[1].
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Maybe also ask on slack? There are quite a bit of users there as
> well.
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Could somebody from PMC please help with Twitter poll?
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> I can try to do this. What question would you like to ask? I do
> not know much about twitter polls but I assume they have character limits
> similar to regular tweets.
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Technically, we can proceed with the change between 2.23.0 and
> 2.24.0, so that's after 2.23.0 is cut and we give sufficient time for users
> to respond.
> >>>>>>>>>
> >>>>>>>>> [1]
> https://lists.apache.org/thread.html/r0de71d98d98b213dd1d0c45c1f5642135116f25def5637a5f41c8d29%40%3Cuser.beam.apache.org%3E
> >>>>>>>>> On Tue, Jun 16, 2020 at 9:22 AM Ismaël Mejía <ie...@gmail.com>
> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Yes we need to poll this outside as Robert proposed.
> >>>>>>>>>>
> >>>>>>>>>> The proposed last support version corresponds with the next
> release that will be
> >>>>>>>>>> cut in two weeks. Sounds a bit short if we count the time to
> poll people on this
> >>>>>>>>>> subject but still could be.
> >>>>>>>>>
> >>>>>>>>> Technically, we can proceed with the change between 2.23.0 and
> 2.24.0, so that's after 2.23.0 is cut and we give sufficient time for users
> to respond.
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> I remember Chad mentioned in this thread the impossibility to
> get support for
> >>>>>>>>>> python 2 in his industry until the end of the year, Maybe
> things have improved.
> >>>>>>>>>> Have they?
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Tue, Jun 16, 2020 at 6:10 PM Robert Bradshaw <
> robertwb@google.com> wrote:
> >>>>>>>>>> >
> >>>>>>>>>> > I like that option as a concrete proposal. I think we should
> circulate it more widely (the users list, twitter poll, at least a new
> thread...), maybe phrasing it as "is there any reason you couldn't migrate
> to Python 3 (or stick with an older version of Beam) after 2.23 (due to be
> cut here in a couple of weeks)?" If there is strong concern/pushback, we
> could consider holding on for one more release.
> >>>>>>>>>> >
> >>>>>>>>>> > On Tue, Jun 16, 2020 at 8:54 AM David Cavazos <
> dcavazos@google.com> wrote:
> >>>>>>>>>> >>
> >>>>>>>>>> >> +1
> >>>>>>>>>> >>
> >>>>>>>>>> >> On Mon, Jun 15, 2020 at 6:52 PM Udi Meiri <eh...@google.com>
> wrote:
> >>>>>>>>>> >>>
> >>>>>>>>>> >>> +1
> >>>>>>>>>> >>>
> >>>>>>>>>> >>> On Mon, Jun 15, 2020 at 4:27 PM Ahmet Altay <
> altay@google.com> wrote:
> >>>>>>>>>> >>>>
> >>>>>>>>>> >>>> As a concrete proposal, could we commit to removing python
> 2 support by 2.24? In other words, mark the next release 2.23 as the last
> python 2 compatible Beam version.
> >>>>>>>>>> >>>>
> >>>>>>>>>> >>>> On Mon, Jun 15, 2020 at 2:09 PM Valentyn Tymofieiev <
> valentyn@google.com> wrote:
> >>>>>>>>>> >>>>>
> >>>>>>>>>> >>>>> Another input here:
> >>>>>>>>>> >>>>>
> >>>>>>>>>> >>>>> If you opened a Python PR in the last few days, you
> probably noticed that our test suites were broken by a transitive
> dependency of Beam that dropped python 2 support, but did not declare
> python_requires>=3 in its setup.py [1]. This temporarily broke a subset of
> Beam Py2 users (who did not explicitly pin the 'rsa' dependency), and still
> affects Beam development[2].
> >>>>>>>>>> >>>>>
> >>>>>>>>>> >>>>> This is the second time[3] Beam is affected with an issue
> of this kind, so support of Python 2 starts to slow down our development,
> and add toil for maintainers of packages we depend on (both directly and
> transitively).
> >>>>>>>>>> >>>>>
> >>>>>>>>>> >>>>> [1] https://github.com/sybrenstuvel/python-rsa/issues/152
> >>>>>>>>>> >>>>> [2]
> https://lists.apache.org/thread.html/r9993b40b0c1cb8682ce56013165d4b80fdde0ee469a73bcb9466ddfb%40%3Cdev.beam.apache.org%3E
> >>>>>>>>>> >>>>> [3] https://github.com/hamcrest/PyHamcrest/issues/131
> >>>>>>>>>> >>>>>
> >>>>>>>>>> >>>>> On Tue, Jun 9, 2020 at 4:06 PM Ahmet Altay <
> altay@google.com> wrote:
> >>>>>>>>>> >>>>>>
> >>>>>>>>>> >>>>>> Thank you for re-opening this Valentyn. I am in favor of
> EOLing py2 support sooner than later. The reality is that we will not be
> effectively supporting beam python 2 for a long time while the ecosystem
> already EOLed python 2. That said, a significant chunk (but no longer a
> majority) of our users are still using python 2. Upgrades are painful, it
> might be especially painful nowadays. It would be good to hear counter view
> points, user voices related to this.
> >>>>>>>>>> >>>>>>
> >>>>>>>>>> >>>>>> On Thu, Jun 4, 2020 at 4:53 PM Valentyn Tymofieiev <
> valentyn@google.com> wrote:
> >>>>>>>>>> >>>>>>>
> >>>>>>>>>> >>>>>>> Back at the end of February we decided to revisit this
> conversation in 3 months. Do folks on this thread have any new input or
> perspective regarding us balancing "user pain/contributor pain/our ability
> to continuously test with python 2 in a shifting environment"?
> >>>>>>>>>> >>>>>>>
> >>>>>>>>>> >>>>>>> Some new information on my end is that we have been
> seeing steady adoption of Python 3 among Beam Python users in Dataflow,
> particularly strong adoption among streaming users, and Dataflow is
> sunsetting Python 2 support for all released Beam SDKs later this year [1].
> We will have to remove Python 2 Beam test suites that use Dataflow when
> Dataflow runner disables Py2 support if this happens before Beam Py2 EOL
> (when we have to remove all Py2 suites), including performance tests that
> still use Dataflow on Python 3.
> >>>>>>>>>> >>>>>>>
> >>>>>>>>>> >>>>>>> I am curious how much motivation there is in the
> community at this moment to continue Py2 support in Beam, whether any
> previous Py3 migration blockers were resolved or any new blockers
> discovered among Beam users.
> >>>>>>>>>> >>>>>>>
> >>>>>>>>>> >>>>>>> [1]
> https://cloud.google.com/python/docs/python2-sunset/#dataflow
> >>>>>>>>>> >>>>>>>
> >>>>>>>>>> >>>>>>> On Fri, May 8, 2020 at 3:52 PM Valentyn Tymofieiev <
> valentyn@google.com> wrote:
> >>>>>>>>>> >>>>>>>>
> >>>>>>>>>> >>>>>>>> That's good news! Thanks for sharing.
> >>>>>>>>>> >>>>>>>>
> >>>>>>>>>> >>>>>>>> Another datapoint, here are a few of Beam's
> dependencies that no longer release new py2 artifacts (I looked at
> REQUIRED_PACKAGES + aws, gcp, and interactive extras):
> >>>>>>>>>> >>>>>>>>
> >>>>>>>>>> >>>>>>>> hdfs
> >>>>>>>>>> >>>>>>>> numpy
> >>>>>>>>>> >>>>>>>> pyarrow
> >>>>>>>>>> >>>>>>>> ipython
> >>>>>>>>>> >>>>>>>>
> >>>>>>>>>> >>>>>>>> There are more if we include transitive dependencies
> and test-only packages. I also remember encountering one issue last month
> that was broken only on Py2, which we had to go back and fix.
> >>>>>>>>>> >>>>>>>>
> >>>>>>>>>> >>>>>>>> If others have noticed frictions related to ongoing
> Py2 support or have updates on previously mentioned Py3 migration blockers,
> feel free to post them.
> >>>>>>>>>> >>>>>>>>
> >>>>>>>>>> >>>>>>>> On Fri, May 8, 2020 at 9:19 AM Robert Bradshaw <
> robertwb@google.com> wrote:
> >>>>>>>>>> >>>>>>>>>
> >>>>>>>>>> >>>>>>>>> It hasn't been 3 months yet, but I wanted to call out
> a milestone that
> >>>>>>>>>> >>>>>>>>> Python 3 downloads crossed the 50% threshold on pypi,
> if just briefly.
> >>>>>>>>>> >>>>>>>>>
> >>>>>>>>>> >>>>>>>>> On Thu, Feb 13, 2020 at 12:40 AM Ismaël Mejía <
> iemejia@gmail.com> wrote:
> >>>>>>>>>> >>>>>>>>> >
> >>>>>>>>>> >>>>>>>>> > > I would suggest re-evaluating this within the
> next 3 months again. We need to balance between user pain/contributor
> pain/our ability to continuously test with python 2 in a shifting
> environment.
> >>>>>>>>>> >>>>>>>>> >
> >>>>>>>>>> >>>>>>>>> > Good idea for the in 3 months evaluation, at that
> point also distributions will probably be phasing out python2 by default
> which definitely help in this direction.
> >>>>>>>>>> >>>>>>>>> > Thanks for updating the roadmap Ahmet
> >>>>>>>>>> >>>>>>>>> >
> >>>>>>>>>> >>>>>>>>> >
> >>>>>>>>>> >>>>>>>>> > On Thu, Feb 13, 2020 at 2:49 AM Ahmet Altay <
> altay@google.com> wrote:
> >>>>>>>>>> >>>>>>>>> >>
> >>>>>>>>>> >>>>>>>>> >>
> >>>>>>>>>> >>>>>>>>> >>
> >>>>>>>>>> >>>>>>>>> >> On Wed, Feb 12, 2020 at 1:29 AM Ismaël Mejía <
> iemejia@gmail.com> wrote:
> >>>>>>>>>> >>>>>>>>> >>>
> >>>>>>>>>> >>>>>>>>> >>> I am with Chad on this, we should probably extend
> it a bit more, even if it
> >>>>>>>>>> >>>>>>>>> >>> makes us struggle a bit at least we have some
> workarounds as Robert suggests,
> >>>>>>>>>> >>>>>>>>> >>> and as Chad said there are still many people
> playing the python 3 catchup game,
> >>>>>>>>>> >>>>>>>>> >>> so worth to support those users.
> >>>>>>>>>> >>>>>>>>> >>>
> >>>>>>>>>> >>>>>>>>> >>>
> >>>>>>>>>> >>>>>>>>> >>> But maybe it is worth to evaluate the current
> state later in the year.
> >>>>>>>>>> >>>>>>>>> >>
> >>>>>>>>>> >>>>>>>>> >>
> >>>>>>>>>> >>>>>>>>> >> I would suggest re-evaluating this within the next
> 3 months again. We need to balance between user pain/contributor pain/our
> ability to continuously test with python 2 in a shifting environment.
> >>>>>>>>>> >>>>>>>>> >>
> >>>>>>>>>> >>>>>>>>> >>>
> >>>>>>>>>> >>>>>>>>> >>> In the
> >>>>>>>>>> >>>>>>>>> >>> meantime can someone please update our Roadmap in
> the website with this info and
> >>>>>>>>>> >>>>>>>>> >>> where we are with Python 3 support (it looks not
> up to date).
> >>>>>>>>>> >>>>>>>>> >>> https://beam.apache.org/roadmap/
> >>>>>>>>>> >>>>>>>>> >>
> >>>>>>>>>> >>>>>>>>> >>
> >>>>>>>>>> >>>>>>>>> >> I made a minor change to update that page (
> https://github.com/apache/beam/pull/10848). A more comprehensive update
> to that page and linked (
> https://beam.apache.org/roadmap/python-sdk/#python-3-support) would still
> be welcome.
> >>>>>>>>>> >>>>>>>>> >>
> >>>>>>>>>> >>>>>>>>> >>>
> >>>>>>>>>> >>>>>>>>> >>>
> >>>>>>>>>> >>>>>>>>> >>> - Ismaël
> >>>>>>>>>> >>>>>>>>> >>>
> >>>>>>>>>> >>>>>>>>> >>>
> >>>>>>>>>> >>>>>>>>> >>> On Tue, Feb 4, 2020 at 10:49 PM Robert Bradshaw <
> robertwb@google.com> wrote:
> >>>>>>>>>> >>>>>>>>> >>>>
> >>>>>>>>>> >>>>>>>>> >>>> On Tue, Feb 4, 2020 at 12:12 PM Chad Dombrova <
> chadrik@gmail.com> wrote:
> >>>>>>>>>> >>>>>>>>> >>>> >>
> >>>>>>>>>> >>>>>>>>> >>>> >> Not to mention that all the nice work for
> the type hints will have to be redone in the for 3.x.
> >>>>>>>>>> >>>>>>>>> >>>> >
> >>>>>>>>>> >>>>>>>>> >>>> > Note that there's a tool for automatically
> converting type comments to annotations:
> https://github.com/ilevkivskyi/com2ann
> >>>>>>>>>> >>>>>>>>> >>>> >
> >>>>>>>>>> >>>>>>>>> >>>> > So don't let that part bother you.
> >>>>>>>>>> >>>>>>>>> >>>>
> >>>>>>>>>> >>>>>>>>> >>>> +1, I wouldn't worry about what can be easily
> automated.
> >>>>>>>>>> >>>>>>>>> >>>>
> >>>>>>>>>> >>>>>>>>> >>>> > I'm curious what other features you'd like to
> be using in the Beam source that you cannot now.
> >>>>>>>>>> >>>>>>>>> >>>>
> >>>>>>>>>> >>>>>>>>> >>>> I hit things occasionally, e.g. I just ran into
> wanting keyword-only
> >>>>>>>>>> >>>>>>>>> >>>> arguments the other day.
> >>>>>>>>>> >>>>>>>>> >>>>
> >>>>>>>>>> >>>>>>>>> >>>> >> It seems the faster we drop support the
> better.
> >>>>>>>>>> >>>>>>>>> >>>> >
> >>>>>>>>>> >>>>>>>>> >>>> >
> >>>>>>>>>> >>>>>>>>> >>>> > I've already gone over my position on this,
> but a refresher for those who care: some of the key vendors that support
> my industry will not offer python3-compatible versions of their software
> until the 4th quarter of 2020. If Beam switches to python3-only before
> that point we may be forced to stop contributing features (note: I'm the
> guy who added the type hints :). Every month you can give us would be
> greatly appreciated.
> >>>>>>>>>> >>>>>>>>> >>>>
> >>>>>>>>>> >>>>>>>>> >>>> As another data point, we're still 80/20 on
> Py2/Py3 for downloads at
> >>>>>>>>>> >>>>>>>>> >>>> PyPi [1] (which I've heard should be taken with
> a grain of salt, but
> >>>>>>>>>> >>>>>>>>> >>>> likely isn't totally off). IMHO that ratio needs
> to be way higher for
> >>>>>>>>>> >>>>>>>>> >>>> Python 3 to consider dropping Python 2. It's
> pretty noisy, but say it
> >>>>>>>>>> >>>>>>>>> >>>> doubles every 3 months that would put us at
> least mid-year before we
> >>>>>>>>>> >>>>>>>>> >>>> hit a cross-over point. On the other hand Q4
> 2020 is probably a
> >>>>>>>>>> >>>>>>>>> >>>> stretch.
> >>>>>>>>>> >>>>>>>>> >>>>
> >>>>>>>>>> >>>>>>>>> >>>> We could consider whether it needs to be an
> all-or-nothing thing as
> >>>>>>>>>> >>>>>>>>> >>>> well. E.g. perhaps some features could be Python
> 3 only sooner than
> >>>>>>>>>> >>>>>>>>> >>>> the whole codebase. (This would have to be well
> justified.) Another
> >>>>>>>>>> >>>>>>>>> >>>> mitigation is that it is possible to mix Python
> 2 and Python 3 in the
> >>>>>>>>>> >>>>>>>>> >>>> same pipeline with portability, so if there's a
> library that you need
> >>>>>>>>>> >>>>>>>>> >>>> for one DoFn it doesn't mean you have to hold
> back your whole
> >>>>>>>>>> >>>>>>>>> >>>> pipeline.
> >>>>>>>>>> >>>>>>>>> >>>>
> >>>>>>>>>> >>>>>>>>> >>>> - Robert
> >>>>>>>>>> >>>>>>>>> >>>>
> >>>>>>>>>> >>>>>>>>> >>>> [1] https://pypistats.org/packages/apache-beam
> , and that 20% may just
> >>>>>>>>>> >>>>>>>>> >>>> be a spike.
>
Re: Python2.7 Beam End-of-Life Date
Posted by Robert Bradshaw <ro...@google.com>.
Note that just because Beam drops 2.x support in new releases doesn't
mean that the old releases won't continue to work. One can even use an
expansion service to run 2.x transforms (on an older version of Beam)
within a Python 3 pipeline running the newest version of Beam until
such a time that (possibly incrementally) the dependent libraries
catch up, if it comes to that. Not that this will be ideal.
Now that the 2.23 release branch has been cut, we *could* remove 2.7
support if we will actually plan to remove it in 2.24. (One concern,
however, is how tied the release testing infrastructure is to
mainline...) However, other than Chad's response (the VFX / Animation
isn't quite there yet) it seems we still don't have a very strong
signal either direction. Pypi downloads are still hovering around 50%.
Did we hear anything back from twitter/slack?
My inclination would be to publish loudly that 2.24 would be the last
Beam to support Python 2.7 (this is pushing it back one release), so
if you're still stuck on it make sure it has everything you need by
the release cut date (which is still in the future).
On Mon, Jul 6, 2020 at 9:52 AM David Yan <da...@google.com> wrote:
>
> +1 for removing Python 2.7 support sooner than later.
>
> I recently added a small feature in Beam Python and I found that having to write code that worked with Python2 was quite awkward and time consuming (needing to make sure code works for both 2 and 3 and doubling the Jenkins running time), and I can imagine that may hinder or even discourage new contributions.
>
> On Thu, Jun 18, 2020 at 5:12 PM Ahmet Altay <al...@google.com> wrote:
>>
>> Good to hear from you and good to hear the news. Release branch cut date for 2.24 is 8/12.
>>
>> On Thu, Jun 18, 2020 at 5:01 PM Chad Dombrova <ch...@gmail.com> wrote:
>>>
>>> Hi all,
>>> Sorry I've been AWOL. I've been pulled in a number of different directions.
>>>
>>> We are still increasing our use of Beam, and I have it on my todo list to reach out to Kenneth about how Beam could be expanded into the VFX / Animation industries.
>>>
>>> Our industry uses a number of specialized applications with embedded python interpreters. We run Beam inside these interpreters, so we're waiting for them to switch to python3.
>>>
>>> Here's the status report for python3 adoption in our key applications:
>>>
>>> Maya: In Beta
>>> Houdini: Released
>>> Nuke: In Beta
>>> Katana: Not started (Alpha?)
>>>
>>> I hate to be the one holding the project back, and I understand if you all ultimately decide it's untenable to wait any longer. The good news is 3 out of 4 applications should be ready in the next 2-3 months. I can do some investigation into what workarounds might look like for Katana, or maybe we can use the Beta version once python3 support arrives, which would move our schedule forward.
>>>
>>> When would 2.24 release?
>>>
>>> -chad
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Thu, Jun 18, 2020 at 4:33 PM Ahmet Altay <al...@google.com> wrote:
>>>>
>>>> OK, tweeted the message. Could you share on Slack?
>>>>
>>>> On Thu, Jun 18, 2020 at 4:28 PM Valentyn Tymofieiev <va...@google.com> wrote:
>>>>>
>>>>> Alright, let's publish the message on Twitter and echo on Slack once that's done.
>>>>> Thank you!
>>>>>
>>>>> On Tue, Jun 16, 2020 at 5:31 PM Ahmet Altay <al...@google.com> wrote:
>>>>>>
>>>>>> That sounds reasonable to me. I agree, it is better to get specific reasons rather than a yes/no answer.
>>>>>>
>>>>>> On Tue, Jun 16, 2020 at 1:50 PM Valentyn Tymofieiev <va...@google.com> wrote:
>>>>>>>
>>>>>>> After thinking about it for a bit, I am not sure whether a poll would be actionable. For example, if 1000 users provide a positive response and 5 users provide a negative response, how do we interpret that and where do we draw a line? How about instead we encourage users to reach out, and tell what is not working for them? For example:
>>>>>>>
>>>>>>> Beam is considering making 2.23.0 a final release supporting Py2. If you are not able to switch to Python 3, please let us know why: [short link to user@ thread] [1].
>>>>>>>
>>>>>>> Thoughts?
>>>>>>>
>>>>>>> [1] https://lists.apache.org/thread.html/r0de71d98d98b213dd1d0c45c1f5642135116f25def5637a5f41c8d29%40%3Cuser.beam.apache.org%3E
>>>>>>>
>>>>>>> On Tue, Jun 16, 2020 at 11:50 AM Ahmet Altay <al...@google.com> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Jun 16, 2020 at 11:38 AM Valentyn Tymofieiev <va...@google.com> wrote:
>>>>>>>>>
>>>>>>>>> I have reached out to user@ for feedback on Python 3 migration[1].
>>>>>>>>
>>>>>>>>
>>>>>>>> Maybe also ask on slack? There are quite a bit of users there as well.
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Could somebody from PMC please help with Twitter poll?
>>>>>>>>
>>>>>>>>
>>>>>>>> I can try to do this. What question would you like to ask? I do not know much about twitter polls but I assume they have character limits similar to regular tweets.
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Technically, we can proceed with the change between 2.23.0 and 2.24.0, so that's after 2.23.0 is cut and we give sufficient time for users to respond.
>>>>>>>>>
>>>>>>>>> [1] https://lists.apache.org/thread.html/r0de71d98d98b213dd1d0c45c1f5642135116f25def5637a5f41c8d29%40%3Cuser.beam.apache.org%3E
>>>>>>>>> On Tue, Jun 16, 2020 at 9:22 AM Ismaël Mejía <ie...@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Yes we need to poll this outside as Robert proposed.
>>>>>>>>>>
>>>>>>>>>> The proposed last support version corresponds with the next release that will be
>>>>>>>>>> cut in two weeks. Sounds a bit short if we count the time to poll people on this
>>>>>>>>>> subject but still could be.
>>>>>>>>>
>>>>>>>>> Technically, we can proceed with the change between 2.23.0 and 2.24.0, so that's after 2.23.0 is cut and we give sufficient time for users to respond.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I remember Chad mentioned in this thread the impossibility to get support for
>>>>>>>>>> python 2 in his industry until the end of the year, Maybe things have improved.
>>>>>>>>>> Have they?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Jun 16, 2020 at 6:10 PM Robert Bradshaw <ro...@google.com> wrote:
>>>>>>>>>> >
>>>>>>>>>> > I like that option as a concrete proposal. I think we should circulate it more widely (the users list, twitter poll, at least a new thread...), maybe phrasing it as "is there any reason you couldn't migrate to Python 3 (or stick with an older version of Beam) after 2.23 (due to be cut here in a couple of weeks)?" If there is strong concern/pushback, we could consider holding on for one more release.
>>>>>>>>>> >
>>>>>>>>>> > On Tue, Jun 16, 2020 at 8:54 AM David Cavazos <dc...@google.com> wrote:
>>>>>>>>>> >>
>>>>>>>>>> >> +1
>>>>>>>>>> >>
>>>>>>>>>> >> On Mon, Jun 15, 2020 at 6:52 PM Udi Meiri <eh...@google.com> wrote:
>>>>>>>>>> >>>
>>>>>>>>>> >>> +1
>>>>>>>>>> >>>
>>>>>>>>>> >>> On Mon, Jun 15, 2020 at 4:27 PM Ahmet Altay <al...@google.com> wrote:
>>>>>>>>>> >>>>
>>>>>>>>>> >>>> As a concrete proposal, could we commit to removing python 2 support by 2.24? In other words, mark the next release 2.23 as the last python 2 compatible Beam version.
>>>>>>>>>> >>>>
>>>>>>>>>> >>>> On Mon, Jun 15, 2020 at 2:09 PM Valentyn Tymofieiev <va...@google.com> wrote:
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>> Another input here:
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>> If you opened a Python PR in the last few days, you probably noticed that our test suites were broken by a transitive dependency of Beam that dropped python 2 support, but did not declare python_requires>=3 in its setup.py [1]. This temporarily broke a subset of Beam Py2 users (who did not explicitly pin the 'rsa' dependency), and still affects Beam development[2].
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>> This is the second time[3] Beam is affected with an issue of this kind, so support of Python 2 starts to slow down our development, and add toil for maintainers of packages we depend on (both directly and transitively).
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>> [1] https://github.com/sybrenstuvel/python-rsa/issues/152
>>>>>>>>>> >>>>> [2] https://lists.apache.org/thread.html/r9993b40b0c1cb8682ce56013165d4b80fdde0ee469a73bcb9466ddfb%40%3Cdev.beam.apache.org%3E
>>>>>>>>>> >>>>> [3] https://github.com/hamcrest/PyHamcrest/issues/131
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>> On Tue, Jun 9, 2020 at 4:06 PM Ahmet Altay <al...@google.com> wrote:
>>>>>>>>>> >>>>>>
>>>>>>>>>> >>>>>> Thank you for re-opening this Valentyn. I am in favor of EOLing py2 support sooner than later. The reality is that we will not be effectively supporting beam python 2 for a long time while the ecosystem already EOLed python 2. That said, a significant chunk (but no longer a majority) of our users are still using python 2. Upgrades are painful, it might be especially painful nowadays. It would be good to hear counter view points, user voices related to this.
>>>>>>>>>> >>>>>>
>>>>>>>>>> >>>>>> On Thu, Jun 4, 2020 at 4:53 PM Valentyn Tymofieiev <va...@google.com> wrote:
>>>>>>>>>> >>>>>>>
>>>>>>>>>> >>>>>>> Back at the end of February we decided to revisit this conversation in 3 months. Do folks on this thread have any new input or perspective regarding us balancing "user pain/contributor pain/our ability to continuously test with python 2 in a shifting environment"?
>>>>>>>>>> >>>>>>>
>>>>>>>>>> >>>>>>> Some new information on my end is that we have been seeing steady adoption of Python 3 among Beam Python users in Dataflow, particularly strong adoption among streaming users, and Dataflow is sunsetting Python 2 support for all released Beam SDKs later this year [1]. We will have to remove Python 2 Beam test suites that use Dataflow when Dataflow runner disables Py2 support if this happens before Beam Py2 EOL (when we have to remove all Py2 suites), including performance tests that still use Dataflow on Python 3.
>>>>>>>>>> >>>>>>>
>>>>>>>>>> >>>>>>> I am curious how much motivation there is in the community at this moment to continue Py2 support in Beam, whether any previous Py3 migration blockers were resolved or any new blockers discovered among Beam users.
>>>>>>>>>> >>>>>>>
>>>>>>>>>> >>>>>>> [1] https://cloud.google.com/python/docs/python2-sunset/#dataflow
>>>>>>>>>> >>>>>>>
>>>>>>>>>> >>>>>>> On Fri, May 8, 2020 at 3:52 PM Valentyn Tymofieiev <va...@google.com> wrote:
>>>>>>>>>> >>>>>>>>
>>>>>>>>>> >>>>>>>> That's good news! Thanks for sharing.
>>>>>>>>>> >>>>>>>>
>>>>>>>>>> >>>>>>>> Another datapoint, here are a few of Beam's dependencies that no longer release new py2 artifacts (I looked at REQUIRED_PACKAGES + aws, gcp, and interactive extras):
>>>>>>>>>> >>>>>>>>
>>>>>>>>>> >>>>>>>> hdfs
>>>>>>>>>> >>>>>>>> numpy
>>>>>>>>>> >>>>>>>> pyarrow
>>>>>>>>>> >>>>>>>> ipython
>>>>>>>>>> >>>>>>>>
>>>>>>>>>> >>>>>>>> There are more if we include transitive dependencies and test-only packages. I also remember encountering one issue last month that was broken only on Py2, which we had to go back and fix.
>>>>>>>>>> >>>>>>>>
>>>>>>>>>> >>>>>>>> If others have noticed frictions related to ongoing Py2 support or have updates on previously mentioned Py3 migration blockers, feel free to post them.
>>>>>>>>>> >>>>>>>>
>>>>>>>>>> >>>>>>>> On Fri, May 8, 2020 at 9:19 AM Robert Bradshaw <ro...@google.com> wrote:
>>>>>>>>>> >>>>>>>>>
>>>>>>>>>> >>>>>>>>> It hasn't been 3 months yet, but I wanted to call out a milestone that
>>>>>>>>>> >>>>>>>>> Python 3 downloads crossed the 50% threshold on pypi, if just briefly.
>>>>>>>>>> >>>>>>>>>
>>>>>>>>>> >>>>>>>>> On Thu, Feb 13, 2020 at 12:40 AM Ismaël Mejía <ie...@gmail.com> wrote:
>>>>>>>>>> >>>>>>>>> >
>>>>>>>>>> >>>>>>>>> > > I would suggest re-evaluating this within the next 3 months again. We need to balance between user pain/contributor pain/our ability to continuously test with python 2 in a shifting environment.
>>>>>>>>>> >>>>>>>>> >
>>>>>>>>>> >>>>>>>>> > Good idea for the in 3 months evaluation, at that point also distributions will probably be phasing out python2 by default which definitely help in this direction.
>>>>>>>>>> >>>>>>>>> > Thanks for updating the roadmap Ahmet
>>>>>>>>>> >>>>>>>>> >
>>>>>>>>>> >>>>>>>>> >
>>>>>>>>>> >>>>>>>>> > On Thu, Feb 13, 2020 at 2:49 AM Ahmet Altay <al...@google.com> wrote:
>>>>>>>>>> >>>>>>>>> >>
>>>>>>>>>> >>>>>>>>> >>
>>>>>>>>>> >>>>>>>>> >>
>>>>>>>>>> >>>>>>>>> >> On Wed, Feb 12, 2020 at 1:29 AM Ismaël Mejía <ie...@gmail.com> wrote:
>>>>>>>>>> >>>>>>>>> >>>
>>>>>>>>>> >>>>>>>>> >>> I am with Chad on this, we should probably extend it a bit more, even if it
>>>>>>>>>> >>>>>>>>> >>> makes us struggle a bit at least we have some workarounds as Robert suggests,
>>>>>>>>>> >>>>>>>>> >>> and as Chad said there are still many people playing the python 3 catchup game,
>>>>>>>>>> >>>>>>>>> >>> so worth to support those users.
>>>>>>>>>> >>>>>>>>> >>>
>>>>>>>>>> >>>>>>>>> >>>
>>>>>>>>>> >>>>>>>>> >>> But maybe it is worth to evaluate the current state later in the year.
>>>>>>>>>> >>>>>>>>> >>
>>>>>>>>>> >>>>>>>>> >>
>>>>>>>>>> >>>>>>>>> >> I would suggest re-evaluating this within the next 3 months again. We need to balance between user pain/contributor pain/our ability to continuously test with python 2 in a shifting environment.
>>>>>>>>>> >>>>>>>>> >>
>>>>>>>>>> >>>>>>>>> >>>
>>>>>>>>>> >>>>>>>>> >>> In the
>>>>>>>>>> >>>>>>>>> >>> meantime can someone please update our Roadmap in the website with this info and
>>>>>>>>>> >>>>>>>>> >>> where we are with Python 3 support (it looks not up to date).
>>>>>>>>>> >>>>>>>>> >>> https://beam.apache.org/roadmap/
>>>>>>>>>> >>>>>>>>> >>
>>>>>>>>>> >>>>>>>>> >>
>>>>>>>>>> >>>>>>>>> >> I made a minor change to update that page (https://github.com/apache/beam/pull/10848). A more comprehensive update to that page and linked (https://beam.apache.org/roadmap/python-sdk/#python-3-support) would still be welcome.
>>>>>>>>>> >>>>>>>>> >>
>>>>>>>>>> >>>>>>>>> >>>
>>>>>>>>>> >>>>>>>>> >>>
>>>>>>>>>> >>>>>>>>> >>> - Ismaël
>>>>>>>>>> >>>>>>>>> >>>
>>>>>>>>>> >>>>>>>>> >>>
>>>>>>>>>> >>>>>>>>> >>> On Tue, Feb 4, 2020 at 10:49 PM Robert Bradshaw <ro...@google.com> wrote:
>>>>>>>>>> >>>>>>>>> >>>>
>>>>>>>>>> >>>>>>>>> >>>> On Tue, Feb 4, 2020 at 12:12 PM Chad Dombrova <ch...@gmail.com> wrote:
>>>>>>>>>> >>>>>>>>> >>>> >>
>>>>>>>>>> >>>>>>>>> >>>> >> Not to mention that all the nice work for the type hints will have to be redone in the for 3.x.
>>>>>>>>>> >>>>>>>>> >>>> >
>>>>>>>>>> >>>>>>>>> >>>> > Note that there's a tool for automatically converting type comments to annotations: https://github.com/ilevkivskyi/com2ann
>>>>>>>>>> >>>>>>>>> >>>> >
>>>>>>>>>> >>>>>>>>> >>>> > So don't let that part bother you.
>>>>>>>>>> >>>>>>>>> >>>>
>>>>>>>>>> >>>>>>>>> >>>> +1, I wouldn't worry about what can be easily automated.
>>>>>>>>>> >>>>>>>>> >>>>
>>>>>>>>>> >>>>>>>>> >>>> > I'm curious what other features you'd like to be using in the Beam source that you cannot now.
>>>>>>>>>> >>>>>>>>> >>>>
>>>>>>>>>> >>>>>>>>> >>>> I hit things occasionally, e.g. I just ran into wanting keyword-only
>>>>>>>>>> >>>>>>>>> >>>> arguments the other day.
>>>>>>>>>> >>>>>>>>> >>>>
>>>>>>>>>> >>>>>>>>> >>>> >> It seems the faster we drop support the better.
>>>>>>>>>> >>>>>>>>> >>>> >
>>>>>>>>>> >>>>>>>>> >>>> >
>>>>>>>>>> >>>>>>>>> >>>> > I've already gone over my position on this, but a refresher for those who care: some of the key vendors that support my industry will not offer python3-compatible versions of their software until the 4th quarter of 2020. If Beam switches to python3-only before that point we may be forced to stop contributing features (note: I'm the guy who added the type hints :). Every month you can give us would be greatly appreciated.
>>>>>>>>>> >>>>>>>>> >>>>
>>>>>>>>>> >>>>>>>>> >>>> As another data point, we're still 80/20 on Py2/Py3 for downloads at
>>>>>>>>>> >>>>>>>>> >>>> PyPi [1] (which I've heard should be taken with a grain of salt, but
>>>>>>>>>> >>>>>>>>> >>>> likely isn't totally off). IMHO that ratio needs to be way higher for
>>>>>>>>>> >>>>>>>>> >>>> Python 3 to consider dropping Python 2. It's pretty noisy, but say it
>>>>>>>>>> >>>>>>>>> >>>> doubles every 3 months that would put us at least mid-year before we
>>>>>>>>>> >>>>>>>>> >>>> hit a cross-over point. On the other hand Q4 2020 is probably a
>>>>>>>>>> >>>>>>>>> >>>> stretch.
>>>>>>>>>> >>>>>>>>> >>>>
>>>>>>>>>> >>>>>>>>> >>>> We could consider whether it needs to be an all-or-nothing thing as
>>>>>>>>>> >>>>>>>>> >>>> well. E.g. perhaps some features could be Python 3 only sooner than
>>>>>>>>>> >>>>>>>>> >>>> the whole codebase. (This would have to be well justified.) Another
>>>>>>>>>> >>>>>>>>> >>>> mitigation is that it is possible to mix Python 2 and Python 3 in the
>>>>>>>>>> >>>>>>>>> >>>> same pipeline with portability, so if there's a library that you need
>>>>>>>>>> >>>>>>>>> >>>> for one DoFn it doesn't mean you have to hold back your whole
>>>>>>>>>> >>>>>>>>> >>>> pipeline.
>>>>>>>>>> >>>>>>>>> >>>>
>>>>>>>>>> >>>>>>>>> >>>> - Robert
>>>>>>>>>> >>>>>>>>> >>>>
>>>>>>>>>> >>>>>>>>> >>>> [1] https://pypistats.org/packages/apache-beam , and that 20% may just
>>>>>>>>>> >>>>>>>>> >>>> be a spike.
Re: Python2.7 Beam End-of-Life Date
Posted by David Yan <da...@google.com>.
+1 for removing Python 2.7 support sooner than later.
I recently added a small feature in Beam Python and I found that having to
write code that worked with Python2 was quite awkward and time consuming
(needing to make sure code works for both 2 and 3 and doubling the Jenkins
running time), and I can imagine that may hinder or even discourage new
contributions.
On Thu, Jun 18, 2020 at 5:12 PM Ahmet Altay <al...@google.com> wrote:
> Good to hear from you and good to hear the news. Release branch cut date
> for 2.24 is 8/12.
>
> On Thu, Jun 18, 2020 at 5:01 PM Chad Dombrova <ch...@gmail.com> wrote:
>
>> Hi all,
>> Sorry I've been AWOL. I've been pulled in a number of different
>> directions.
>>
>> We are still increasing our use of Beam, and I have it on my todo list to
>> reach out to Kenneth about how Beam could be expanded into the VFX /
>> Animation industries.
>>
>> Our industry uses a number of specialized applications with embedded
>> python interpreters. We run Beam inside these interpreters, so we're
>> waiting for them to switch to python3.
>>
>> Here's the status report for python3 adoption in our key applications:
>>
>> *Maya*: In Beta
>> *Houdini*: Released
>> *Nuke*: In Beta
>> *Katana*: Not started (Alpha?)
>>
>> I hate to be the one holding the project back, and I understand if you
>> all ultimately decide it's untenable to wait any longer. The good news is
>> 3 out of 4 applications should be ready in the next 2-3 months. I can do
>> some investigation into what workarounds might look like for Katana, or
>> maybe we can use the Beta version once python3 support arrives, which would
>> move our schedule forward.
>>
>> When would 2.24 release?
>>
>> -chad
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Thu, Jun 18, 2020 at 4:33 PM Ahmet Altay <al...@google.com> wrote:
>>
>>> OK, tweeted the message. Could you share on Slack?
>>>
>>> On Thu, Jun 18, 2020 at 4:28 PM Valentyn Tymofieiev <va...@google.com>
>>> wrote:
>>>
>>>> Alright, let's publish the message on Twitter and echo on Slack once
>>>> that's done.
>>>> Thank you!
>>>>
>>>> On Tue, Jun 16, 2020 at 5:31 PM Ahmet Altay <al...@google.com> wrote:
>>>>
>>>>> That sounds reasonable to me. I agree, it is better to get specific
>>>>> reasons rather than a yes/no answer.
>>>>>
>>>>> On Tue, Jun 16, 2020 at 1:50 PM Valentyn Tymofieiev <
>>>>> valentyn@google.com> wrote:
>>>>>
>>>>>> After thinking about it for a bit, I am not sure whether a poll would
>>>>>> be actionable. For example, if 1000 users provide a positive response and 5
>>>>>> users provide a negative response, how do we interpret that and where do
>>>>>> we draw a line? How about instead we encourage users to reach out, and tell
>>>>>> what is not working for them? For example:
>>>>>>
>>>>>> Beam is considering making 2.23.0 a final release supporting Py2. If
>>>>>> you are not able to switch to Python 3, please let us know why: [short link
>>>>>> to user@ thread] [1].
>>>>>>
>>>>>> Thoughts?
>>>>>>
>>>>>> [1]
>>>>>> https://lists.apache.org/thread.html/r0de71d98d98b213dd1d0c45c1f5642135116f25def5637a5f41c8d29%40%3Cuser.beam.apache.org%3E
>>>>>>
>>>>>> On Tue, Jun 16, 2020 at 11:50 AM Ahmet Altay <al...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Jun 16, 2020 at 11:38 AM Valentyn Tymofieiev <
>>>>>>> valentyn@google.com> wrote:
>>>>>>>
>>>>>>>> I have reached out to user@ for feedback on Python 3 migration[1].
>>>>>>>>
>>>>>>>
>>>>>>> Maybe also ask on slack? There are quite a bit of users there as
>>>>>>> well.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Could somebody from PMC please help with Twitter poll?
>>>>>>>>
>>>>>>>
>>>>>>> I can try to do this. What question would you like to ask? I do not
>>>>>>> know much about twitter polls but I assume they have character limits
>>>>>>> similar to regular tweets.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Technically, we can proceed with the change between 2.23.0 and
>>>>>>>> 2.24.0, so that's after 2.23.0 is cut and we give sufficient time for users
>>>>>>>> to respond.
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> https://lists.apache.org/thread.html/r0de71d98d98b213dd1d0c45c1f5642135116f25def5637a5f41c8d29%40%3Cuser.beam.apache.org%3E
>>>>>>>> On Tue, Jun 16, 2020 at 9:22 AM Ismaël Mejía <ie...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Yes we need to poll this outside as Robert proposed.
>>>>>>>>>
>>>>>>>>> The proposed last support version corresponds with the next
>>>>>>>>> release that will be
>>>>>>>>> cut in two weeks. Sounds a bit short if we count the time to poll
>>>>>>>>> people on this
>>>>>>>>> subject but still could be.
>>>>>>>>
>>>>>>>> Technically, we can proceed with the change between 2.23.0 and
>>>>>>>> 2.24.0, so that's after 2.23.0 is cut and we give sufficient time for users
>>>>>>>> to respond.
>>>>>>>>
>>>>>>>>
>>>>>>>>> I remember Chad mentioned in this thread the impossibility to get
>>>>>>>>> support for
>>>>>>>>> python 2 in his industry until the end of the year, Maybe things
>>>>>>>>> have improved.
>>>>>>>>> Have they?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Jun 16, 2020 at 6:10 PM Robert Bradshaw <
>>>>>>>>> robertwb@google.com> wrote:
>>>>>>>>> >
>>>>>>>>> > I like that option as a concrete proposal. I think we should
>>>>>>>>> circulate it more widely (the users list, twitter poll, at least a new
>>>>>>>>> thread...), maybe phrasing it as "is there any reason you couldn't migrate
>>>>>>>>> to Python 3 (or stick with an older version of Beam) after 2.23 (due to be
>>>>>>>>> cut here in a couple of weeks)?" If there is strong concern/pushback, we
>>>>>>>>> could consider holding on for one more release.
>>>>>>>>> >
>>>>>>>>> > On Tue, Jun 16, 2020 at 8:54 AM David Cavazos <
>>>>>>>>> dcavazos@google.com> wrote:
>>>>>>>>> >>
>>>>>>>>> >> +1
>>>>>>>>> >>
>>>>>>>>> >> On Mon, Jun 15, 2020 at 6:52 PM Udi Meiri <eh...@google.com>
>>>>>>>>> wrote:
>>>>>>>>> >>>
>>>>>>>>> >>> +1
>>>>>>>>> >>>
>>>>>>>>> >>> On Mon, Jun 15, 2020 at 4:27 PM Ahmet Altay <al...@google.com>
>>>>>>>>> wrote:
>>>>>>>>> >>>>
>>>>>>>>> >>>> As a concrete proposal, could we commit to removing python 2
>>>>>>>>> support by 2.24? In other words, mark the next release 2.23 as the last
>>>>>>>>> python 2 compatible Beam version.
>>>>>>>>> >>>>
>>>>>>>>> >>>> On Mon, Jun 15, 2020 at 2:09 PM Valentyn Tymofieiev <
>>>>>>>>> valentyn@google.com> wrote:
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> Another input here:
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> If you opened a Python PR in the last few days, you probably
>>>>>>>>> noticed that our test suites were broken by a transitive dependency of Beam
>>>>>>>>> that dropped python 2 support, but did not declare python_requires>=3 in
>>>>>>>>> its setup.py [1]. This temporarily broke a subset of Beam Py2 users (who
>>>>>>>>> did not explicitly pin the 'rsa' dependency), and still affects Beam
>>>>>>>>> development[2].
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> This is the second time[3] Beam is affected with an issue of
>>>>>>>>> this kind, so support of Python 2 starts to slow down our development, and
>>>>>>>>> add toil for maintainers of packages we depend on (both directly and
>>>>>>>>> transitively).
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> [1] https://github.com/sybrenstuvel/python-rsa/issues/152
>>>>>>>>> >>>>> [2]
>>>>>>>>> https://lists.apache.org/thread.html/r9993b40b0c1cb8682ce56013165d4b80fdde0ee469a73bcb9466ddfb%40%3Cdev.beam.apache.org%3E
>>>>>>>>> >>>>> [3] https://github.com/hamcrest/PyHamcrest/issues/131
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> On Tue, Jun 9, 2020 at 4:06 PM Ahmet Altay <al...@google.com>
>>>>>>>>> wrote:
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>> Thank you for re-opening this Valentyn. I am in favor of
>>>>>>>>> EOLing py2 support sooner than later. The reality is that we will not be
>>>>>>>>> effectively supporting beam python 2 for a long time while the ecosystem
>>>>>>>>> already EOLed python 2. That said, a significant chunk (but no longer a
>>>>>>>>> majority) of our users are still using python 2. Upgrades are painful, it
>>>>>>>>> might be especially painful nowadays. It would be good to hear counter view
>>>>>>>>> points, user voices related to this.
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>> On Thu, Jun 4, 2020 at 4:53 PM Valentyn Tymofieiev <
>>>>>>>>> valentyn@google.com> wrote:
>>>>>>>>> >>>>>>>
>>>>>>>>> >>>>>>> Back at the end of February we decided to revisit this
>>>>>>>>> conversation in 3 months. Do folks on this thread have any new input or
>>>>>>>>> perspective regarding us balancing "user pain/contributor pain/our ability
>>>>>>>>> to continuously test with python 2 in a shifting environment"?
>>>>>>>>> >>>>>>>
>>>>>>>>> >>>>>>> Some new information on my end is that we have been seeing
>>>>>>>>> steady adoption of Python 3 among Beam Python users in Dataflow,
>>>>>>>>> particularly strong adoption among streaming users, and Dataflow is
>>>>>>>>> sunsetting Python 2 support for all released Beam SDKs later this year [1].
>>>>>>>>> We will have to remove Python 2 Beam test suites that use Dataflow when
>>>>>>>>> Dataflow runner disables Py2 support if this happens before Beam Py2 EOL
>>>>>>>>> (when we have to remove all Py2 suites), including performance tests that
>>>>>>>>> still use Dataflow on Python 3.
>>>>>>>>> >>>>>>>
>>>>>>>>> >>>>>>> I am curious how much motivation there is in the community
>>>>>>>>> at this moment to continue Py2 support in Beam, whether any previous Py3
>>>>>>>>> migration blockers were resolved or any new blockers discovered among Beam
>>>>>>>>> users.
>>>>>>>>> >>>>>>>
>>>>>>>>> >>>>>>> [1]
>>>>>>>>> https://cloud.google.com/python/docs/python2-sunset/#dataflow
>>>>>>>>> >>>>>>>
>>>>>>>>> >>>>>>> On Fri, May 8, 2020 at 3:52 PM Valentyn Tymofieiev <
>>>>>>>>> valentyn@google.com> wrote:
>>>>>>>>> >>>>>>>>
>>>>>>>>> >>>>>>>> That's good news! Thanks for sharing.
>>>>>>>>> >>>>>>>>
>>>>>>>>> >>>>>>>> Another datapoint, here are a few of Beam's dependencies
>>>>>>>>> that no longer release new py2 artifacts (I looked at REQUIRED_PACKAGES +
>>>>>>>>> aws, gcp, and interactive extras):
>>>>>>>>> >>>>>>>>
>>>>>>>>> >>>>>>>> hdfs
>>>>>>>>> >>>>>>>> numpy
>>>>>>>>> >>>>>>>> pyarrow
>>>>>>>>> >>>>>>>> ipython
>>>>>>>>> >>>>>>>>
>>>>>>>>> >>>>>>>> There are more if we include transitive dependencies and
>>>>>>>>> test-only packages. I also remember encountering one issue last month that
>>>>>>>>> was broken only on Py2, which we had to go back and fix.
>>>>>>>>> >>>>>>>>
>>>>>>>>> >>>>>>>> If others have noticed frictions related to ongoing Py2
>>>>>>>>> support or have updates on previously mentioned Py3 migration blockers,
>>>>>>>>> feel free to post them.
>>>>>>>>> >>>>>>>>
>>>>>>>>> >>>>>>>> On Fri, May 8, 2020 at 9:19 AM Robert Bradshaw <
>>>>>>>>> robertwb@google.com> wrote:
>>>>>>>>> >>>>>>>>>
>>>>>>>>> >>>>>>>>> It hasn't been 3 months yet, but I wanted to call out a
>>>>>>>>> milestone that
>>>>>>>>> >>>>>>>>> Python 3 downloads crossed the 50% threshold on pypi, if
>>>>>>>>> just briefly.
>>>>>>>>> >>>>>>>>>
>>>>>>>>> >>>>>>>>> On Thu, Feb 13, 2020 at 12:40 AM Ismaël Mejía <
>>>>>>>>> iemejia@gmail.com> wrote:
>>>>>>>>> >>>>>>>>> >
>>>>>>>>> >>>>>>>>> > > I would suggest re-evaluating this within the next 3
>>>>>>>>> months again. We need to balance between user pain/contributor pain/our
>>>>>>>>> ability to continuously test with python 2 in a shifting environment.
>>>>>>>>> >>>>>>>>> >
>>>>>>>>> >>>>>>>>> > Good idea for the in 3 months evaluation, at that
>>>>>>>>> point also distributions will probably be phasing out python2 by default
>>>>>>>>> which definitely help in this direction.
>>>>>>>>> >>>>>>>>> > Thanks for updating the roadmap Ahmet
>>>>>>>>> >>>>>>>>> >
>>>>>>>>> >>>>>>>>> >
>>>>>>>>> >>>>>>>>> > On Thu, Feb 13, 2020 at 2:49 AM Ahmet Altay <
>>>>>>>>> altay@google.com> wrote:
>>>>>>>>> >>>>>>>>> >>
>>>>>>>>> >>>>>>>>> >>
>>>>>>>>> >>>>>>>>> >>
>>>>>>>>> >>>>>>>>> >> On Wed, Feb 12, 2020 at 1:29 AM Ismaël Mejía <
>>>>>>>>> iemejia@gmail.com> wrote:
>>>>>>>>> >>>>>>>>> >>>
>>>>>>>>> >>>>>>>>> >>> I am with Chad on this, we should probably extend it
>>>>>>>>> a bit more, even if it
>>>>>>>>> >>>>>>>>> >>> makes us struggle a bit at least we have some
>>>>>>>>> workarounds as Robert suggests,
>>>>>>>>> >>>>>>>>> >>> and as Chad said there are still many people playing
>>>>>>>>> the python 3 catchup game,
>>>>>>>>> >>>>>>>>> >>> so worth to support those users.
>>>>>>>>> >>>>>>>>> >>>
>>>>>>>>> >>>>>>>>> >>>
>>>>>>>>> >>>>>>>>> >>> But maybe it is worth to evaluate the current state
>>>>>>>>> later in the year.
>>>>>>>>> >>>>>>>>> >>
>>>>>>>>> >>>>>>>>> >>
>>>>>>>>> >>>>>>>>> >> I would suggest re-evaluating this within the next 3
>>>>>>>>> months again. We need to balance between user pain/contributor pain/our
>>>>>>>>> ability to continuously test with python 2 in a shifting environment.
>>>>>>>>> >>>>>>>>> >>
>>>>>>>>> >>>>>>>>> >>>
>>>>>>>>> >>>>>>>>> >>> In the
>>>>>>>>> >>>>>>>>> >>> meantime can someone please update our Roadmap in
>>>>>>>>> the website with this info and
>>>>>>>>> >>>>>>>>> >>> where we are with Python 3 support (it looks not up
>>>>>>>>> to date).
>>>>>>>>> >>>>>>>>> >>> https://beam.apache.org/roadmap/
>>>>>>>>> >>>>>>>>> >>
>>>>>>>>> >>>>>>>>> >>
>>>>>>>>> >>>>>>>>> >> I made a minor change to update that page (
>>>>>>>>> https://github.com/apache/beam/pull/10848). A more comprehensive
>>>>>>>>> update to that page and linked (
>>>>>>>>> https://beam.apache.org/roadmap/python-sdk/#python-3-support)
>>>>>>>>> would still be welcome.
>>>>>>>>> >>>>>>>>> >>
>>>>>>>>> >>>>>>>>> >>>
>>>>>>>>> >>>>>>>>> >>>
>>>>>>>>> >>>>>>>>> >>> - Ismaël
>>>>>>>>> >>>>>>>>> >>>
>>>>>>>>> >>>>>>>>> >>>
>>>>>>>>> >>>>>>>>> >>> On Tue, Feb 4, 2020 at 10:49 PM Robert Bradshaw <
>>>>>>>>> robertwb@google.com> wrote:
>>>>>>>>> >>>>>>>>> >>>>
>>>>>>>>> >>>>>>>>> >>>> On Tue, Feb 4, 2020 at 12:12 PM Chad Dombrova <
>>>>>>>>> chadrik@gmail.com> wrote:
>>>>>>>>> >>>>>>>>> >>>> >>
>>>>>>>>> >>>>>>>>> >>>> >> Not to mention that all the nice work for the
>>>>>>>>> type hints will have to be redone in the for 3.x.
>>>>>>>>> >>>>>>>>> >>>> >
>>>>>>>>> >>>>>>>>> >>>> > Note that there's a tool for automatically
>>>>>>>>> converting type comments to annotations:
>>>>>>>>> https://github.com/ilevkivskyi/com2ann
>>>>>>>>> >>>>>>>>> >>>> >
>>>>>>>>> >>>>>>>>> >>>> > So don't let that part bother you.
>>>>>>>>> >>>>>>>>> >>>>
>>>>>>>>> >>>>>>>>> >>>> +1, I wouldn't worry about what can be easily
>>>>>>>>> automated.
>>>>>>>>> >>>>>>>>> >>>>
>>>>>>>>> >>>>>>>>> >>>> > I'm curious what other features you'd like to be
>>>>>>>>> using in the Beam source that you cannot now.
>>>>>>>>> >>>>>>>>> >>>>
>>>>>>>>> >>>>>>>>> >>>> I hit things occasionally, e.g. I just ran into
>>>>>>>>> wanting keyword-only
>>>>>>>>> >>>>>>>>> >>>> arguments the other day.
>>>>>>>>> >>>>>>>>> >>>>
>>>>>>>>> >>>>>>>>> >>>> >> It seems the faster we drop support the better.
>>>>>>>>> >>>>>>>>> >>>> >
>>>>>>>>> >>>>>>>>> >>>> >
>>>>>>>>> >>>>>>>>> >>>> > I've already gone over my position on this, but a
>>>>>>>>> refresher for those who care: some of the key vendors that support my
>>>>>>>>> industry will not offer python3-compatible versions of their software until
>>>>>>>>> the 4th quarter of 2020. If Beam switches to python3-only before that
>>>>>>>>> point we may be forced to stop contributing features (note: I'm the guy who
>>>>>>>>> added the type hints :). Every month you can give us would be greatly
>>>>>>>>> appreciated.
>>>>>>>>> >>>>>>>>> >>>>
>>>>>>>>> >>>>>>>>> >>>> As another data point, we're still 80/20 on Py2/Py3
>>>>>>>>> for downloads at
>>>>>>>>> >>>>>>>>> >>>> PyPi [1] (which I've heard should be taken with a
>>>>>>>>> grain of salt, but
>>>>>>>>> >>>>>>>>> >>>> likely isn't totally off). IMHO that ratio needs to
>>>>>>>>> be way higher for
>>>>>>>>> >>>>>>>>> >>>> Python 3 to consider dropping Python 2. It's pretty
>>>>>>>>> noisy, but say it
>>>>>>>>> >>>>>>>>> >>>> doubles every 3 months that would put us at least
>>>>>>>>> mid-year before we
>>>>>>>>> >>>>>>>>> >>>> hit a cross-over point. On the other hand Q4 2020
>>>>>>>>> is probably a
>>>>>>>>> >>>>>>>>> >>>> stretch.
>>>>>>>>> >>>>>>>>> >>>>
>>>>>>>>> >>>>>>>>> >>>> We could consider whether it needs to be an
>>>>>>>>> all-or-nothing thing as
>>>>>>>>> >>>>>>>>> >>>> well. E.g. perhaps some features could be Python 3
>>>>>>>>> only sooner than
>>>>>>>>> >>>>>>>>> >>>> the whole codebase. (This would have to be well
>>>>>>>>> justified.) Another
>>>>>>>>> >>>>>>>>> >>>> mitigation is that it is possible to mix Python 2
>>>>>>>>> and Python 3 in the
>>>>>>>>> >>>>>>>>> >>>> same pipeline with portability, so if there's a
>>>>>>>>> library that you need
>>>>>>>>> >>>>>>>>> >>>> for one DoFn it doesn't mean you have to hold back
>>>>>>>>> your whole
>>>>>>>>> >>>>>>>>> >>>> pipeline.
>>>>>>>>> >>>>>>>>> >>>>
>>>>>>>>> >>>>>>>>> >>>> - Robert
>>>>>>>>> >>>>>>>>> >>>>
>>>>>>>>> >>>>>>>>> >>>> [1] https://pypistats.org/packages/apache-beam ,
>>>>>>>>> and that 20% may just
>>>>>>>>> >>>>>>>>> >>>> be a spike.
>>>>>>>>>
>>>>>>>>
Re: Python2.7 Beam End-of-Life Date
Posted by Ahmet Altay <al...@google.com>.
Good to hear from you and good to hear the news. Release branch cut date
for 2.24 is 8/12.
On Thu, Jun 18, 2020 at 5:01 PM Chad Dombrova <ch...@gmail.com> wrote:
> Hi all,
> Sorry I've been AWOL. I've been pulled in a number of different
> directions.
>
> We are still increasing our use of Beam, and I have it on my todo list to
> reach out to Kenneth about how Beam could be expanded into the VFX /
> Animation industries.
>
> Our industry uses a number of specialized applications with embedded
> python interpreters. We run Beam inside these interpreters, so we're
> waiting for them to switch to python3.
>
> Here's the status report for python3 adoption in our key applications:
>
> *Maya*: In Beta
> *Houdini*: Released
> *Nuke*: In Beta
> *Katana*: Not started (Alpha?)
>
> I hate to be the one holding the project back, and I understand if you all
> ultimately decide it's untenable to wait any longer. The good news is 3
> out of 4 applications should be ready in the next 2-3 months. I can do
> some investigation into what workarounds might look like for Katana, or
> maybe we can use the Beta version once python3 support arrives, which would
> move our schedule forward.
>
> When would 2.24 release?
>
> -chad
>
>
>
>
>
>
>
>
>
> On Thu, Jun 18, 2020 at 4:33 PM Ahmet Altay <al...@google.com> wrote:
>
>> OK, tweeted the message. Could you share on Slack?
>>
>> On Thu, Jun 18, 2020 at 4:28 PM Valentyn Tymofieiev <va...@google.com>
>> wrote:
>>
>>> Alright, let's publish the message on Twitter and echo on Slack once
>>> that's done.
>>> Thank you!
>>>
>>> On Tue, Jun 16, 2020 at 5:31 PM Ahmet Altay <al...@google.com> wrote:
>>>
>>>> That sounds reasonable to me. I agree, it is better to get specific
>>>> reasons rather than a yes/no answer.
>>>>
>>>> On Tue, Jun 16, 2020 at 1:50 PM Valentyn Tymofieiev <
>>>> valentyn@google.com> wrote:
>>>>
>>>>> After thinking about it for a bit, I am not sure whether a poll would
>>>>> be actionable. For example, if 1000 users provide a positive response and 5
>>>>> users provide a negative response, how do we interpret that and where do
>>>>> we draw a line? How about instead we encourage users to reach out, and tell
>>>>> what is not working for them? For example:
>>>>>
>>>>> Beam is considering making 2.23.0 a final release supporting Py2. If
>>>>> you are not able to switch to Python 3, please let us know why: [short link
>>>>> to user@ thread] [1].
>>>>>
>>>>> Thoughts?
>>>>>
>>>>> [1]
>>>>> https://lists.apache.org/thread.html/r0de71d98d98b213dd1d0c45c1f5642135116f25def5637a5f41c8d29%40%3Cuser.beam.apache.org%3E
>>>>>
>>>>> On Tue, Jun 16, 2020 at 11:50 AM Ahmet Altay <al...@google.com> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Jun 16, 2020 at 11:38 AM Valentyn Tymofieiev <
>>>>>> valentyn@google.com> wrote:
>>>>>>
>>>>>>> I have reached out to user@ for feedback on Python 3 migration[1].
>>>>>>>
>>>>>>
>>>>>> Maybe also ask on slack? There are quite a bit of users there as well.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Could somebody from PMC please help with Twitter poll?
>>>>>>>
>>>>>>
>>>>>> I can try to do this. What question would you like to ask? I do not
>>>>>> know much about twitter polls but I assume they have character limits
>>>>>> similar to regular tweets.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Technically, we can proceed with the change between 2.23.0 and
>>>>>>> 2.24.0, so that's after 2.23.0 is cut and we give sufficient time for users
>>>>>>> to respond.
>>>>>>>
>>>>>>> [1]
>>>>>>> https://lists.apache.org/thread.html/r0de71d98d98b213dd1d0c45c1f5642135116f25def5637a5f41c8d29%40%3Cuser.beam.apache.org%3E
>>>>>>> On Tue, Jun 16, 2020 at 9:22 AM Ismaël Mejía <ie...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Yes we need to poll this outside as Robert proposed.
>>>>>>>>
>>>>>>>> The proposed last support version corresponds with the next release
>>>>>>>> that will be
>>>>>>>> cut in two weeks. Sounds a bit short if we count the time to poll
>>>>>>>> people on this
>>>>>>>> subject but still could be.
>>>>>>>
>>>>>>> Technically, we can proceed with the change between 2.23.0 and
>>>>>>> 2.24.0, so that's after 2.23.0 is cut and we give sufficient time for users
>>>>>>> to respond.
>>>>>>>
>>>>>>>
>>>>>>>> I remember Chad mentioned in this thread the impossibility to get
>>>>>>>> support for
>>>>>>>> python 2 in his industry until the end of the year, Maybe things
>>>>>>>> have improved.
>>>>>>>> Have they?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Jun 16, 2020 at 6:10 PM Robert Bradshaw <
>>>>>>>> robertwb@google.com> wrote:
>>>>>>>> >
>>>>>>>> > I like that option as a concrete proposal. I think we should
>>>>>>>> circulate it more widely (the users list, twitter poll, at least a new
>>>>>>>> thread...), maybe phrasing it as "is there any reason you couldn't migrate
>>>>>>>> to Python 3 (or stick with an older version of Beam) after 2.23 (due to be
>>>>>>>> cut here in a couple of weeks)?" If there is strong concern/pushback, we
>>>>>>>> could consider holding on for one more release.
>>>>>>>> >
>>>>>>>> > On Tue, Jun 16, 2020 at 8:54 AM David Cavazos <
>>>>>>>> dcavazos@google.com> wrote:
>>>>>>>> >>
>>>>>>>> >> +1
>>>>>>>> >>
>>>>>>>> >> On Mon, Jun 15, 2020 at 6:52 PM Udi Meiri <eh...@google.com>
>>>>>>>> wrote:
>>>>>>>> >>>
>>>>>>>> >>> +1
>>>>>>>> >>>
>>>>>>>> >>> On Mon, Jun 15, 2020 at 4:27 PM Ahmet Altay <al...@google.com>
>>>>>>>> wrote:
>>>>>>>> >>>>
>>>>>>>> >>>> As a concrete proposal, could we commit to removing python 2
>>>>>>>> support by 2.24? In other words, mark the next release 2.23 as the last
>>>>>>>> python 2 compatible Beam version.
>>>>>>>> >>>>
>>>>>>>> >>>> On Mon, Jun 15, 2020 at 2:09 PM Valentyn Tymofieiev <
>>>>>>>> valentyn@google.com> wrote:
>>>>>>>> >>>>>
>>>>>>>> >>>>> Another input here:
>>>>>>>> >>>>>
>>>>>>>> >>>>> If you opened a Python PR in the last few days, you probably
>>>>>>>> noticed that our test suites were broken by a transitive dependency of Beam
>>>>>>>> that dropped python 2 support, but did not declare python_requires>=3 in
>>>>>>>> its setup.py [1]. This temporarily broke a subset of Beam Py2 users (who
>>>>>>>> did not explicitly pin the 'rsa' dependency), and still affects Beam
>>>>>>>> development[2].
>>>>>>>> >>>>>
>>>>>>>> >>>>> This is the second time[3] Beam is affected with an issue of
>>>>>>>> this kind, so support of Python 2 starts to slow down our development, and
>>>>>>>> add toil for maintainers of packages we depend on (both directly and
>>>>>>>> transitively).
>>>>>>>> >>>>>
>>>>>>>> >>>>> [1] https://github.com/sybrenstuvel/python-rsa/issues/152
>>>>>>>> >>>>> [2]
>>>>>>>> https://lists.apache.org/thread.html/r9993b40b0c1cb8682ce56013165d4b80fdde0ee469a73bcb9466ddfb%40%3Cdev.beam.apache.org%3E
>>>>>>>> >>>>> [3] https://github.com/hamcrest/PyHamcrest/issues/131
>>>>>>>> >>>>>
>>>>>>>> >>>>> On Tue, Jun 9, 2020 at 4:06 PM Ahmet Altay <al...@google.com>
>>>>>>>> wrote:
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> Thank you for re-opening this Valentyn. I am in favor of
>>>>>>>> EOLing py2 support sooner than later. The reality is that we will not be
>>>>>>>> effectively supporting beam python 2 for a long time while the ecosystem
>>>>>>>> already EOLed python 2. That said, a significant chunk (but no longer a
>>>>>>>> majority) of our users are still using python 2. Upgrades are painful, it
>>>>>>>> might be especially painful nowadays. It would be good to hear counter view
>>>>>>>> points, user voices related to this.
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> On Thu, Jun 4, 2020 at 4:53 PM Valentyn Tymofieiev <
>>>>>>>> valentyn@google.com> wrote:
>>>>>>>> >>>>>>>
>>>>>>>> >>>>>>> Back at the end of February we decided to revisit this
>>>>>>>> conversation in 3 months. Do folks on this thread have any new input or
>>>>>>>> perspective regarding us balancing "user pain/contributor pain/our ability
>>>>>>>> to continuously test with python 2 in a shifting environment"?
>>>>>>>> >>>>>>>
>>>>>>>> >>>>>>> Some new information on my end is that we have been seeing
>>>>>>>> steady adoption of Python 3 among Beam Python users in Dataflow,
>>>>>>>> particularly strong adoption among streaming users, and Dataflow is
>>>>>>>> sunsetting Python 2 support for all released Beam SDKs later this year [1].
>>>>>>>> We will have to remove Python 2 Beam test suites that use Dataflow when
>>>>>>>> Dataflow runner disables Py2 support if this happens before Beam Py2 EOL
>>>>>>>> (when we have to remove all Py2 suites), including performance tests that
>>>>>>>> still use Dataflow on Python 3.
>>>>>>>> >>>>>>>
>>>>>>>> >>>>>>> I am curious how much motivation there is in the community
>>>>>>>> at this moment to continue Py2 support in Beam, whether any previous Py3
>>>>>>>> migration blockers were resolved or any new blockers discovered among Beam
>>>>>>>> users.
>>>>>>>> >>>>>>>
>>>>>>>> >>>>>>> [1]
>>>>>>>> https://cloud.google.com/python/docs/python2-sunset/#dataflow
>>>>>>>> >>>>>>>
>>>>>>>> >>>>>>> On Fri, May 8, 2020 at 3:52 PM Valentyn Tymofieiev <
>>>>>>>> valentyn@google.com> wrote:
>>>>>>>> >>>>>>>>
>>>>>>>> >>>>>>>> That's good news! Thanks for sharing.
>>>>>>>> >>>>>>>>
>>>>>>>> >>>>>>>> Another datapoint, here are a few of Beam's dependencies
>>>>>>>> that no longer release new py2 artifacts (I looked at REQUIRED_PACKAGES +
>>>>>>>> aws, gcp, and interactive extras):
>>>>>>>> >>>>>>>>
>>>>>>>> >>>>>>>> hdfs
>>>>>>>> >>>>>>>> numpy
>>>>>>>> >>>>>>>> pyarrow
>>>>>>>> >>>>>>>> ipython
>>>>>>>> >>>>>>>>
>>>>>>>> >>>>>>>> There are more if we include transitive dependencies and
>>>>>>>> test-only packages. I also remember encountering one issue last month that
>>>>>>>> was broken only on Py2, which we had to go back and fix.
>>>>>>>> >>>>>>>>
>>>>>>>> >>>>>>>> If others have noticed frictions related to ongoing Py2
>>>>>>>> support or have updates on previously mentioned Py3 migration blockers,
>>>>>>>> feel free to post them.
>>>>>>>> >>>>>>>>
>>>>>>>> >>>>>>>> On Fri, May 8, 2020 at 9:19 AM Robert Bradshaw <
>>>>>>>> robertwb@google.com> wrote:
>>>>>>>> >>>>>>>>>
>>>>>>>> >>>>>>>>> It hasn't been 3 months yet, but I wanted to call out a
>>>>>>>> milestone that
>>>>>>>> >>>>>>>>> Python 3 downloads crossed the 50% threshold on pypi, if
>>>>>>>> just briefly.
>>>>>>>> >>>>>>>>>
>>>>>>>> >>>>>>>>> On Thu, Feb 13, 2020 at 12:40 AM Ismaël Mejía <
>>>>>>>> iemejia@gmail.com> wrote:
>>>>>>>> >>>>>>>>> >
>>>>>>>> >>>>>>>>> > > I would suggest re-evaluating this within the next 3
>>>>>>>> months again. We need to balance between user pain/contributor pain/our
>>>>>>>> ability to continuously test with python 2 in a shifting environment.
>>>>>>>> >>>>>>>>> >
>>>>>>>> >>>>>>>>> > Good idea for the in 3 months evaluation, at that point
>>>>>>>> also distributions will probably be phasing out python2 by default which
>>>>>>>> definitely help in this direction.
>>>>>>>> >>>>>>>>> > Thanks for updating the roadmap Ahmet
>>>>>>>> >>>>>>>>> >
>>>>>>>> >>>>>>>>> >
>>>>>>>> >>>>>>>>> > On Thu, Feb 13, 2020 at 2:49 AM Ahmet Altay <
>>>>>>>> altay@google.com> wrote:
>>>>>>>> >>>>>>>>> >>
>>>>>>>> >>>>>>>>> >>
>>>>>>>> >>>>>>>>> >>
>>>>>>>> >>>>>>>>> >> On Wed, Feb 12, 2020 at 1:29 AM Ismaël Mejía <
>>>>>>>> iemejia@gmail.com> wrote:
>>>>>>>> >>>>>>>>> >>>
>>>>>>>> >>>>>>>>> >>> I am with Chad on this, we should probably extend it
>>>>>>>> a bit more, even if it
>>>>>>>> >>>>>>>>> >>> makes us struggle a bit at least we have some
>>>>>>>> workarounds as Robert suggests,
>>>>>>>> >>>>>>>>> >>> and as Chad said there are still many people playing
>>>>>>>> the python 3 catchup game,
>>>>>>>> >>>>>>>>> >>> so worth to support those users.
>>>>>>>> >>>>>>>>> >>>
>>>>>>>> >>>>>>>>> >>>
>>>>>>>> >>>>>>>>> >>> But maybe it is worth to evaluate the current state
>>>>>>>> later in the year.
>>>>>>>> >>>>>>>>> >>
>>>>>>>> >>>>>>>>> >>
>>>>>>>> >>>>>>>>> >> I would suggest re-evaluating this within the next 3
>>>>>>>> months again. We need to balance between user pain/contributor pain/our
>>>>>>>> ability to continuously test with python 2 in a shifting environment.
>>>>>>>> >>>>>>>>> >>
>>>>>>>> >>>>>>>>> >>>
>>>>>>>> >>>>>>>>> >>> In the
>>>>>>>> >>>>>>>>> >>> meantime can someone please update our Roadmap in the
>>>>>>>> website with this info and
>>>>>>>> >>>>>>>>> >>> where we are with Python 3 support (it looks not up
>>>>>>>> to date).
>>>>>>>> >>>>>>>>> >>> https://beam.apache.org/roadmap/
>>>>>>>> >>>>>>>>> >>
>>>>>>>> >>>>>>>>> >>
>>>>>>>> >>>>>>>>> >> I made a minor change to update that page (
>>>>>>>> https://github.com/apache/beam/pull/10848). A more comprehensive
>>>>>>>> update to that page and linked (
>>>>>>>> https://beam.apache.org/roadmap/python-sdk/#python-3-support)
>>>>>>>> would still be welcome.
>>>>>>>> >>>>>>>>> >>
>>>>>>>> >>>>>>>>> >>>
>>>>>>>> >>>>>>>>> >>>
>>>>>>>> >>>>>>>>> >>> - Ismaël
>>>>>>>> >>>>>>>>> >>>
>>>>>>>> >>>>>>>>> >>>
>>>>>>>> >>>>>>>>> >>> On Tue, Feb 4, 2020 at 10:49 PM Robert Bradshaw <
>>>>>>>> robertwb@google.com> wrote:
>>>>>>>> >>>>>>>>> >>>>
>>>>>>>> >>>>>>>>> >>>> On Tue, Feb 4, 2020 at 12:12 PM Chad Dombrova <
>>>>>>>> chadrik@gmail.com> wrote:
>>>>>>>> >>>>>>>>> >>>> >>
>>>>>>>> >>>>>>>>> >>>> >> Not to mention that all the nice work for the
>>>>>>>> type hints will have to be redone in the for 3.x.
>>>>>>>> >>>>>>>>> >>>> >
>>>>>>>> >>>>>>>>> >>>> > Note that there's a tool for automatically
>>>>>>>> converting type comments to annotations:
>>>>>>>> https://github.com/ilevkivskyi/com2ann
>>>>>>>> >>>>>>>>> >>>> >
>>>>>>>> >>>>>>>>> >>>> > So don't let that part bother you.
>>>>>>>> >>>>>>>>> >>>>
>>>>>>>> >>>>>>>>> >>>> +1, I wouldn't worry about what can be easily
>>>>>>>> automated.
>>>>>>>> >>>>>>>>> >>>>
>>>>>>>> >>>>>>>>> >>>> > I'm curious what other features you'd like to be
>>>>>>>> using in the Beam source that you cannot now.
>>>>>>>> >>>>>>>>> >>>>
>>>>>>>> >>>>>>>>> >>>> I hit things occasionally, e.g. I just ran into
>>>>>>>> wanting keyword-only
>>>>>>>> >>>>>>>>> >>>> arguments the other day.
>>>>>>>> >>>>>>>>> >>>>
>>>>>>>> >>>>>>>>> >>>> >> It seems the faster we drop support the better.
>>>>>>>> >>>>>>>>> >>>> >
>>>>>>>> >>>>>>>>> >>>> >
>>>>>>>> >>>>>>>>> >>>> > I've already gone over my position on this, but a
>>>>>>>> refresher for those who care: some of the key vendors that support my
>>>>>>>> industry will not offer python3-compatible versions of their software until
>>>>>>>> the 4th quarter of 2020. If Beam switches to python3-only before that
>>>>>>>> point we may be forced to stop contributing features (note: I'm the guy who
>>>>>>>> added the type hints :). Every month you can give us would be greatly
>>>>>>>> appreciated.
>>>>>>>> >>>>>>>>> >>>>
>>>>>>>> >>>>>>>>> >>>> As another data point, we're still 80/20 on Py2/Py3
>>>>>>>> for downloads at
>>>>>>>> >>>>>>>>> >>>> PyPi [1] (which I've heard should be taken with a
>>>>>>>> grain of salt, but
>>>>>>>> >>>>>>>>> >>>> likely isn't totally off). IMHO that ratio needs to
>>>>>>>> be way higher for
>>>>>>>> >>>>>>>>> >>>> Python 3 to consider dropping Python 2. It's pretty
>>>>>>>> noisy, but say it
>>>>>>>> >>>>>>>>> >>>> doubles every 3 months that would put us at least
>>>>>>>> mid-year before we
>>>>>>>> >>>>>>>>> >>>> hit a cross-over point. On the other hand Q4 2020 is
>>>>>>>> probably a
>>>>>>>> >>>>>>>>> >>>> stretch.
>>>>>>>> >>>>>>>>> >>>>
>>>>>>>> >>>>>>>>> >>>> We could consider whether it needs to be an
>>>>>>>> all-or-nothing thing as
>>>>>>>> >>>>>>>>> >>>> well. E.g. perhaps some features could be Python 3
>>>>>>>> only sooner than
>>>>>>>> >>>>>>>>> >>>> the whole codebase. (This would have to be well
>>>>>>>> justified.) Another
>>>>>>>> >>>>>>>>> >>>> mitigation is that it is possible to mix Python 2
>>>>>>>> and Python 3 in the
>>>>>>>> >>>>>>>>> >>>> same pipeline with portability, so if there's a
>>>>>>>> library that you need
>>>>>>>> >>>>>>>>> >>>> for one DoFn it doesn't mean you have to hold back
>>>>>>>> your whole
>>>>>>>> >>>>>>>>> >>>> pipeline.
>>>>>>>> >>>>>>>>> >>>>
>>>>>>>> >>>>>>>>> >>>> - Robert
>>>>>>>> >>>>>>>>> >>>>
>>>>>>>> >>>>>>>>> >>>> [1] https://pypistats.org/packages/apache-beam ,
>>>>>>>> and that 20% may just
>>>>>>>> >>>>>>>>> >>>> be a spike.
>>>>>>>>
>>>>>>>
Re: Python2.7 Beam End-of-Life Date
Posted by Chad Dombrova <ch...@gmail.com>.
Hi all,
Sorry I've been AWOL. I've been pulled in a number of different directions.
We are still increasing our use of Beam, and I have it on my todo list to
reach out to Kenneth about how Beam could be expanded into the VFX /
Animation industries.
Our industry uses a number of specialized applications with embedded python
interpreters. We run Beam inside these interpreters, so we're waiting for
them to switch to python3.
Here's the status report for python3 adoption in our key applications:
*Maya*: In Beta
*Houdini*: Released
*Nuke*: In Beta
*Katana*: Not started (Alpha?)
I hate to be the one holding the project back, and I understand if you all
ultimately decide it's untenable to wait any longer. The good news is 3
out of 4 applications should be ready in the next 2-3 months. I can do
some investigation into what workarounds might look like for Katana, or
maybe we can use the Beta version once python3 support arrives, which would
move our schedule forward.
When would 2.24 release?
-chad
On Thu, Jun 18, 2020 at 4:33 PM Ahmet Altay <al...@google.com> wrote:
> OK, tweeted the message. Could you share on Slack?
>
> On Thu, Jun 18, 2020 at 4:28 PM Valentyn Tymofieiev <va...@google.com>
> wrote:
>
>> Alright, let's publish the message on Twitter and echo on Slack once
>> that's done.
>> Thank you!
>>
>> On Tue, Jun 16, 2020 at 5:31 PM Ahmet Altay <al...@google.com> wrote:
>>
>>> That sounds reasonable to me. I agree, it is better to get specific
>>> reasons rather than a yes/no answer.
>>>
>>> On Tue, Jun 16, 2020 at 1:50 PM Valentyn Tymofieiev <va...@google.com>
>>> wrote:
>>>
>>>> After thinking about it for a bit, I am not sure whether a poll would
>>>> be actionable. For example, if 1000 users provide a positive response and 5
>>>> users provide a negative response, how do we interpret that and where do
>>>> we draw a line? How about instead we encourage users to reach out, and tell
>>>> what is not working for them? For example:
>>>>
>>>> Beam is considering making 2.23.0 a final release supporting Py2. If
>>>> you are not able to switch to Python 3, please let us know why: [short link
>>>> to user@ thread] [1].
>>>>
>>>> Thoughts?
>>>>
>>>> [1]
>>>> https://lists.apache.org/thread.html/r0de71d98d98b213dd1d0c45c1f5642135116f25def5637a5f41c8d29%40%3Cuser.beam.apache.org%3E
>>>>
>>>> On Tue, Jun 16, 2020 at 11:50 AM Ahmet Altay <al...@google.com> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Tue, Jun 16, 2020 at 11:38 AM Valentyn Tymofieiev <
>>>>> valentyn@google.com> wrote:
>>>>>
>>>>>> I have reached out to user@ for feedback on Python 3 migration[1].
>>>>>>
>>>>>
>>>>> Maybe also ask on slack? There are quite a bit of users there as well.
>>>>>
>>>>>
>>>>>>
>>>>>> Could somebody from PMC please help with Twitter poll?
>>>>>>
>>>>>
>>>>> I can try to do this. What question would you like to ask? I do not
>>>>> know much about twitter polls but I assume they have character limits
>>>>> similar to regular tweets.
>>>>>
>>>>>
>>>>>>
>>>>>> Technically, we can proceed with the change between 2.23.0 and
>>>>>> 2.24.0, so that's after 2.23.0 is cut and we give sufficient time for users
>>>>>> to respond.
>>>>>>
>>>>>> [1]
>>>>>> https://lists.apache.org/thread.html/r0de71d98d98b213dd1d0c45c1f5642135116f25def5637a5f41c8d29%40%3Cuser.beam.apache.org%3E
>>>>>> On Tue, Jun 16, 2020 at 9:22 AM Ismaël Mejía <ie...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Yes we need to poll this outside as Robert proposed.
>>>>>>>
>>>>>>> The proposed last support version corresponds with the next release
>>>>>>> that will be
>>>>>>> cut in two weeks. Sounds a bit short if we count the time to poll
>>>>>>> people on this
>>>>>>> subject but still could be.
>>>>>>
>>>>>> Technically, we can proceed with the change between 2.23.0 and
>>>>>> 2.24.0, so that's after 2.23.0 is cut and we give sufficient time for users
>>>>>> to respond.
>>>>>>
>>>>>>
>>>>>>> I remember Chad mentioned in this thread the impossibility to get
>>>>>>> support for
>>>>>>> python 2 in his industry until the end of the year, Maybe things
>>>>>>> have improved.
>>>>>>> Have they?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Jun 16, 2020 at 6:10 PM Robert Bradshaw <ro...@google.com>
>>>>>>> wrote:
>>>>>>> >
>>>>>>> > I like that option as a concrete proposal. I think we should
>>>>>>> circulate it more widely (the users list, twitter poll, at least a new
>>>>>>> thread...), maybe phrasing it as "is there any reason you couldn't migrate
>>>>>>> to Python 3 (or stick with an older version of Beam) after 2.23 (due to be
>>>>>>> cut here in a couple of weeks)?" If there is strong concern/pushback, we
>>>>>>> could consider holding on for one more release.
>>>>>>> >
>>>>>>> > On Tue, Jun 16, 2020 at 8:54 AM David Cavazos <dc...@google.com>
>>>>>>> wrote:
>>>>>>> >>
>>>>>>> >> +1
>>>>>>> >>
>>>>>>> >> On Mon, Jun 15, 2020 at 6:52 PM Udi Meiri <eh...@google.com>
>>>>>>> wrote:
>>>>>>> >>>
>>>>>>> >>> +1
>>>>>>> >>>
>>>>>>> >>> On Mon, Jun 15, 2020 at 4:27 PM Ahmet Altay <al...@google.com>
>>>>>>> wrote:
>>>>>>> >>>>
>>>>>>> >>>> As a concrete proposal, could we commit to removing python 2
>>>>>>> support by 2.24? In other words, mark the next release 2.23 as the last
>>>>>>> python 2 compatible Beam version.
>>>>>>> >>>>
>>>>>>> >>>> On Mon, Jun 15, 2020 at 2:09 PM Valentyn Tymofieiev <
>>>>>>> valentyn@google.com> wrote:
>>>>>>> >>>>>
>>>>>>> >>>>> Another input here:
>>>>>>> >>>>>
>>>>>>> >>>>> If you opened a Python PR in the last few days, you probably
>>>>>>> noticed that our test suites were broken by a transitive dependency of Beam
>>>>>>> that dropped python 2 support, but did not declare python_requires>=3 in
>>>>>>> its setup.py [1]. This temporarily broke a subset of Beam Py2 users (who
>>>>>>> did not explicitly pin the 'rsa' dependency), and still affects Beam
>>>>>>> development[2].
>>>>>>> >>>>>
>>>>>>> >>>>> This is the second time[3] Beam is affected with an issue of
>>>>>>> this kind, so support of Python 2 starts to slow down our development, and
>>>>>>> add toil for maintainers of packages we depend on (both directly and
>>>>>>> transitively).
>>>>>>> >>>>>
>>>>>>> >>>>> [1] https://github.com/sybrenstuvel/python-rsa/issues/152
>>>>>>> >>>>> [2]
>>>>>>> https://lists.apache.org/thread.html/r9993b40b0c1cb8682ce56013165d4b80fdde0ee469a73bcb9466ddfb%40%3Cdev.beam.apache.org%3E
>>>>>>> >>>>> [3] https://github.com/hamcrest/PyHamcrest/issues/131
>>>>>>> >>>>>
>>>>>>> >>>>> On Tue, Jun 9, 2020 at 4:06 PM Ahmet Altay <al...@google.com>
>>>>>>> wrote:
>>>>>>> >>>>>>
>>>>>>> >>>>>> Thank you for re-opening this Valentyn. I am in favor of
>>>>>>> EOLing py2 support sooner than later. The reality is that we will not be
>>>>>>> effectively supporting beam python 2 for a long time while the ecosystem
>>>>>>> already EOLed python 2. That said, a significant chunk (but no longer a
>>>>>>> majority) of our users are still using python 2. Upgrades are painful, it
>>>>>>> might be especially painful nowadays. It would be good to hear counter view
>>>>>>> points, user voices related to this.
>>>>>>> >>>>>>
>>>>>>> >>>>>> On Thu, Jun 4, 2020 at 4:53 PM Valentyn Tymofieiev <
>>>>>>> valentyn@google.com> wrote:
>>>>>>> >>>>>>>
>>>>>>> >>>>>>> Back at the end of February we decided to revisit this
>>>>>>> conversation in 3 months. Do folks on this thread have any new input or
>>>>>>> perspective regarding us balancing "user pain/contributor pain/our ability
>>>>>>> to continuously test with python 2 in a shifting environment"?
>>>>>>> >>>>>>>
>>>>>>> >>>>>>> Some new information on my end is that we have been seeing
>>>>>>> steady adoption of Python 3 among Beam Python users in Dataflow,
>>>>>>> particularly strong adoption among streaming users, and Dataflow is
>>>>>>> sunsetting Python 2 support for all released Beam SDKs later this year [1].
>>>>>>> We will have to remove Python 2 Beam test suites that use Dataflow when
>>>>>>> Dataflow runner disables Py2 support if this happens before Beam Py2 EOL
>>>>>>> (when we have to remove all Py2 suites), including performance tests that
>>>>>>> still use Dataflow on Python 3.
>>>>>>> >>>>>>>
>>>>>>> >>>>>>> I am curious how much motivation there is in the community
>>>>>>> at this moment to continue Py2 support in Beam, whether any previous Py3
>>>>>>> migration blockers were resolved or any new blockers discovered among Beam
>>>>>>> users.
>>>>>>> >>>>>>>
>>>>>>> >>>>>>> [1]
>>>>>>> https://cloud.google.com/python/docs/python2-sunset/#dataflow
>>>>>>> >>>>>>>
>>>>>>> >>>>>>> On Fri, May 8, 2020 at 3:52 PM Valentyn Tymofieiev <
>>>>>>> valentyn@google.com> wrote:
>>>>>>> >>>>>>>>
>>>>>>> >>>>>>>> That's good news! Thanks for sharing.
>>>>>>> >>>>>>>>
>>>>>>> >>>>>>>> Another datapoint, here are a few of Beam's dependencies
>>>>>>> that no longer release new py2 artifacts (I looked at REQUIRED_PACKAGES +
>>>>>>> aws, gcp, and interactive extras):
>>>>>>> >>>>>>>>
>>>>>>> >>>>>>>> hdfs
>>>>>>> >>>>>>>> numpy
>>>>>>> >>>>>>>> pyarrow
>>>>>>> >>>>>>>> ipython
>>>>>>> >>>>>>>>
>>>>>>> >>>>>>>> There are more if we include transitive dependencies and
>>>>>>> test-only packages. I also remember encountering one issue last month that
>>>>>>> was broken only on Py2, which we had to go back and fix.
>>>>>>> >>>>>>>>
>>>>>>> >>>>>>>> If others have noticed frictions related to ongoing Py2
>>>>>>> support or have updates on previously mentioned Py3 migration blockers,
>>>>>>> feel free to post them.
>>>>>>> >>>>>>>>
>>>>>>> >>>>>>>> On Fri, May 8, 2020 at 9:19 AM Robert Bradshaw <
>>>>>>> robertwb@google.com> wrote:
>>>>>>> >>>>>>>>>
>>>>>>> >>>>>>>>> It hasn't been 3 months yet, but I wanted to call out a
>>>>>>> milestone that
>>>>>>> >>>>>>>>> Python 3 downloads crossed the 50% threshold on pypi, if
>>>>>>> just briefly.
>>>>>>> >>>>>>>>>
>>>>>>> >>>>>>>>> On Thu, Feb 13, 2020 at 12:40 AM Ismaël Mejía <
>>>>>>> iemejia@gmail.com> wrote:
>>>>>>> >>>>>>>>> >
>>>>>>> >>>>>>>>> > > I would suggest re-evaluating this within the next 3
>>>>>>> months again. We need to balance between user pain/contributor pain/our
>>>>>>> ability to continuously test with python 2 in a shifting environment.
>>>>>>> >>>>>>>>> >
>>>>>>> >>>>>>>>> > Good idea for the in 3 months evaluation, at that point
>>>>>>> also distributions will probably be phasing out python2 by default which
>>>>>>> definitely help in this direction.
>>>>>>> >>>>>>>>> > Thanks for updating the roadmap Ahmet
>>>>>>> >>>>>>>>> >
>>>>>>> >>>>>>>>> >
>>>>>>> >>>>>>>>> > On Thu, Feb 13, 2020 at 2:49 AM Ahmet Altay <
>>>>>>> altay@google.com> wrote:
>>>>>>> >>>>>>>>> >>
>>>>>>> >>>>>>>>> >>
>>>>>>> >>>>>>>>> >>
>>>>>>> >>>>>>>>> >> On Wed, Feb 12, 2020 at 1:29 AM Ismaël Mejía <
>>>>>>> iemejia@gmail.com> wrote:
>>>>>>> >>>>>>>>> >>>
>>>>>>> >>>>>>>>> >>> I am with Chad on this, we should probably extend it a
>>>>>>> bit more, even if it
>>>>>>> >>>>>>>>> >>> makes us struggle a bit at least we have some
>>>>>>> workarounds as Robert suggests,
>>>>>>> >>>>>>>>> >>> and as Chad said there are still many people playing
>>>>>>> the python 3 catchup game,
>>>>>>> >>>>>>>>> >>> so worth to support those users.
>>>>>>> >>>>>>>>> >>>
>>>>>>> >>>>>>>>> >>>
>>>>>>> >>>>>>>>> >>> But maybe it is worth to evaluate the current state
>>>>>>> later in the year.
>>>>>>> >>>>>>>>> >>
>>>>>>> >>>>>>>>> >>
>>>>>>> >>>>>>>>> >> I would suggest re-evaluating this within the next 3
>>>>>>> months again. We need to balance between user pain/contributor pain/our
>>>>>>> ability to continuously test with python 2 in a shifting environment.
>>>>>>> >>>>>>>>> >>
>>>>>>> >>>>>>>>> >>>
>>>>>>> >>>>>>>>> >>> In the
>>>>>>> >>>>>>>>> >>> meantime can someone please update our Roadmap in the
>>>>>>> website with this info and
>>>>>>> >>>>>>>>> >>> where we are with Python 3 support (it looks not up to
>>>>>>> date).
>>>>>>> >>>>>>>>> >>> https://beam.apache.org/roadmap/
>>>>>>> >>>>>>>>> >>
>>>>>>> >>>>>>>>> >>
>>>>>>> >>>>>>>>> >> I made a minor change to update that page (
>>>>>>> https://github.com/apache/beam/pull/10848). A more comprehensive
>>>>>>> update to that page and linked (
>>>>>>> https://beam.apache.org/roadmap/python-sdk/#python-3-support) would
>>>>>>> still be welcome.
>>>>>>> >>>>>>>>> >>
>>>>>>> >>>>>>>>> >>>
>>>>>>> >>>>>>>>> >>>
>>>>>>> >>>>>>>>> >>> - Ismaël
>>>>>>> >>>>>>>>> >>>
>>>>>>> >>>>>>>>> >>>
>>>>>>> >>>>>>>>> >>> On Tue, Feb 4, 2020 at 10:49 PM Robert Bradshaw <
>>>>>>> robertwb@google.com> wrote:
>>>>>>> >>>>>>>>> >>>>
>>>>>>> >>>>>>>>> >>>> On Tue, Feb 4, 2020 at 12:12 PM Chad Dombrova <
>>>>>>> chadrik@gmail.com> wrote:
>>>>>>> >>>>>>>>> >>>> >>
>>>>>>> >>>>>>>>> >>>> >> Not to mention that all the nice work for the
>>>>>>> type hints will have to be redone in the for 3.x.
>>>>>>> >>>>>>>>> >>>> >
>>>>>>> >>>>>>>>> >>>> > Note that there's a tool for automatically
>>>>>>> converting type comments to annotations:
>>>>>>> https://github.com/ilevkivskyi/com2ann
>>>>>>> >>>>>>>>> >>>> >
>>>>>>> >>>>>>>>> >>>> > So don't let that part bother you.
>>>>>>> >>>>>>>>> >>>>
>>>>>>> >>>>>>>>> >>>> +1, I wouldn't worry about what can be easily
>>>>>>> automated.
>>>>>>> >>>>>>>>> >>>>
>>>>>>> >>>>>>>>> >>>> > I'm curious what other features you'd like to be
>>>>>>> using in the Beam source that you cannot now.
>>>>>>> >>>>>>>>> >>>>
>>>>>>> >>>>>>>>> >>>> I hit things occasionally, e.g. I just ran into
>>>>>>> wanting keyword-only
>>>>>>> >>>>>>>>> >>>> arguments the other day.
>>>>>>> >>>>>>>>> >>>>
>>>>>>> >>>>>>>>> >>>> >> It seems the faster we drop support the better.
>>>>>>> >>>>>>>>> >>>> >
>>>>>>> >>>>>>>>> >>>> >
>>>>>>> >>>>>>>>> >>>> > I've already gone over my position on this, but a
>>>>>>> refresher for those who care: some of the key vendors that support my
>>>>>>> industry will not offer python3-compatible versions of their software until
>>>>>>> the 4th quarter of 2020. If Beam switches to python3-only before that
>>>>>>> point we may be forced to stop contributing features (note: I'm the guy who
>>>>>>> added the type hints :). Every month you can give us would be greatly
>>>>>>> appreciated.
>>>>>>> >>>>>>>>> >>>>
>>>>>>> >>>>>>>>> >>>> As another data point, we're still 80/20 on Py2/Py3
>>>>>>> for downloads at
>>>>>>> >>>>>>>>> >>>> PyPi [1] (which I've heard should be taken with a
>>>>>>> grain of salt, but
>>>>>>> >>>>>>>>> >>>> likely isn't totally off). IMHO that ratio needs to
>>>>>>> be way higher for
>>>>>>> >>>>>>>>> >>>> Python 3 to consider dropping Python 2. It's pretty
>>>>>>> noisy, but say it
>>>>>>> >>>>>>>>> >>>> doubles every 3 months that would put us at least
>>>>>>> mid-year before we
>>>>>>> >>>>>>>>> >>>> hit a cross-over point. On the other hand Q4 2020 is
>>>>>>> probably a
>>>>>>> >>>>>>>>> >>>> stretch.
>>>>>>> >>>>>>>>> >>>>
>>>>>>> >>>>>>>>> >>>> We could consider whether it needs to be an
>>>>>>> all-or-nothing thing as
>>>>>>> >>>>>>>>> >>>> well. E.g. perhaps some features could be Python 3
>>>>>>> only sooner than
>>>>>>> >>>>>>>>> >>>> the whole codebase. (This would have to be well
>>>>>>> justified.) Another
>>>>>>> >>>>>>>>> >>>> mitigation is that it is possible to mix Python 2 and
>>>>>>> Python 3 in the
>>>>>>> >>>>>>>>> >>>> same pipeline with portability, so if there's a
>>>>>>> library that you need
>>>>>>> >>>>>>>>> >>>> for one DoFn it doesn't mean you have to hold back
>>>>>>> your whole
>>>>>>> >>>>>>>>> >>>> pipeline.
>>>>>>> >>>>>>>>> >>>>
>>>>>>> >>>>>>>>> >>>> - Robert
>>>>>>> >>>>>>>>> >>>>
>>>>>>> >>>>>>>>> >>>> [1] https://pypistats.org/packages/apache-beam , and
>>>>>>> that 20% may just
>>>>>>> >>>>>>>>> >>>> be a spike.
>>>>>>>
>>>>>>
Re: Python2.7 Beam End-of-Life Date
Posted by Ahmet Altay <al...@google.com>.
OK, tweeted the message. Could you share on Slack?
On Thu, Jun 18, 2020 at 4:28 PM Valentyn Tymofieiev <va...@google.com>
wrote:
> Alright, let's publish the message on Twitter and echo on Slack once
> that's done.
> Thank you!
>
> On Tue, Jun 16, 2020 at 5:31 PM Ahmet Altay <al...@google.com> wrote:
>
>> That sounds reasonable to me. I agree, it is better to get specific
>> reasons rather than a yes/no answer.
>>
>> On Tue, Jun 16, 2020 at 1:50 PM Valentyn Tymofieiev <va...@google.com>
>> wrote:
>>
>>> After thinking about it for a bit, I am not sure whether a poll would be
>>> actionable. For example, if 1000 users provide a positive response and 5
>>> users provide a negative response, how do we interpret that and where do
>>> we draw a line? How about instead we encourage users to reach out, and tell
>>> what is not working for them? For example:
>>>
>>> Beam is considering making 2.23.0 a final release supporting Py2. If you
>>> are not able to switch to Python 3, please let us know why: [short link to
>>> user@ thread] [1].
>>>
>>> Thoughts?
>>>
>>> [1]
>>> https://lists.apache.org/thread.html/r0de71d98d98b213dd1d0c45c1f5642135116f25def5637a5f41c8d29%40%3Cuser.beam.apache.org%3E
>>>
>>> On Tue, Jun 16, 2020 at 11:50 AM Ahmet Altay <al...@google.com> wrote:
>>>
>>>>
>>>>
>>>> On Tue, Jun 16, 2020 at 11:38 AM Valentyn Tymofieiev <
>>>> valentyn@google.com> wrote:
>>>>
>>>>> I have reached out to user@ for feedback on Python 3 migration[1].
>>>>>
>>>>
>>>> Maybe also ask on slack? There are quite a bit of users there as well.
>>>>
>>>>
>>>>>
>>>>> Could somebody from PMC please help with Twitter poll?
>>>>>
>>>>
>>>> I can try to do this. What question would you like to ask? I do not
>>>> know much about twitter polls but I assume they have character limits
>>>> similar to regular tweets.
>>>>
>>>>
>>>>>
>>>>> Technically, we can proceed with the change between 2.23.0 and 2.24.0,
>>>>> so that's after 2.23.0 is cut and we give sufficient time for users to
>>>>> respond.
>>>>>
>>>>> [1]
>>>>> https://lists.apache.org/thread.html/r0de71d98d98b213dd1d0c45c1f5642135116f25def5637a5f41c8d29%40%3Cuser.beam.apache.org%3E
>>>>> On Tue, Jun 16, 2020 at 9:22 AM Ismaël Mejía <ie...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Yes we need to poll this outside as Robert proposed.
>>>>>>
>>>>>> The proposed last support version corresponds with the next release
>>>>>> that will be
>>>>>> cut in two weeks. Sounds a bit short if we count the time to poll
>>>>>> people on this
>>>>>> subject but still could be.
>>>>>
>>>>> Technically, we can proceed with the change between 2.23.0 and 2.24.0,
>>>>> so that's after 2.23.0 is cut and we give sufficient time for users to
>>>>> respond.
>>>>>
>>>>>
>>>>>> I remember Chad mentioned in this thread the impossibility to get
>>>>>> support for
>>>>>> python 2 in his industry until the end of the year, Maybe things have
>>>>>> improved.
>>>>>> Have they?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Jun 16, 2020 at 6:10 PM Robert Bradshaw <ro...@google.com>
>>>>>> wrote:
>>>>>> >
>>>>>> > I like that option as a concrete proposal. I think we should
>>>>>> circulate it more widely (the users list, twitter poll, at least a new
>>>>>> thread...), maybe phrasing it as "is there any reason you couldn't migrate
>>>>>> to Python 3 (or stick with an older version of Beam) after 2.23 (due to be
>>>>>> cut here in a couple of weeks)?" If there is strong concern/pushback, we
>>>>>> could consider holding on for one more release.
>>>>>> >
>>>>>> > On Tue, Jun 16, 2020 at 8:54 AM David Cavazos <dc...@google.com>
>>>>>> wrote:
>>>>>> >>
>>>>>> >> +1
>>>>>> >>
>>>>>> >> On Mon, Jun 15, 2020 at 6:52 PM Udi Meiri <eh...@google.com>
>>>>>> wrote:
>>>>>> >>>
>>>>>> >>> +1
>>>>>> >>>
>>>>>> >>> On Mon, Jun 15, 2020 at 4:27 PM Ahmet Altay <al...@google.com>
>>>>>> wrote:
>>>>>> >>>>
>>>>>> >>>> As a concrete proposal, could we commit to removing python 2
>>>>>> support by 2.24? In other words, mark the next release 2.23 as the last
>>>>>> python 2 compatible Beam version.
>>>>>> >>>>
>>>>>> >>>> On Mon, Jun 15, 2020 at 2:09 PM Valentyn Tymofieiev <
>>>>>> valentyn@google.com> wrote:
>>>>>> >>>>>
>>>>>> >>>>> Another input here:
>>>>>> >>>>>
>>>>>> >>>>> If you opened a Python PR in the last few days, you probably
>>>>>> noticed that our test suites were broken by a transitive dependency of Beam
>>>>>> that dropped python 2 support, but did not declare python_requires>=3 in
>>>>>> its setup.py [1]. This temporarily broke a subset of Beam Py2 users (who
>>>>>> did not explicitly pin the 'rsa' dependency), and still affects Beam
>>>>>> development[2].
>>>>>> >>>>>
>>>>>> >>>>> This is the second time[3] Beam is affected with an issue of
>>>>>> this kind, so support of Python 2 starts to slow down our development, and
>>>>>> add toil for maintainers of packages we depend on (both directly and
>>>>>> transitively).
>>>>>> >>>>>
>>>>>> >>>>> [1] https://github.com/sybrenstuvel/python-rsa/issues/152
>>>>>> >>>>> [2]
>>>>>> https://lists.apache.org/thread.html/r9993b40b0c1cb8682ce56013165d4b80fdde0ee469a73bcb9466ddfb%40%3Cdev.beam.apache.org%3E
>>>>>> >>>>> [3] https://github.com/hamcrest/PyHamcrest/issues/131
>>>>>> >>>>>
>>>>>> >>>>> On Tue, Jun 9, 2020 at 4:06 PM Ahmet Altay <al...@google.com>
>>>>>> wrote:
>>>>>> >>>>>>
>>>>>> >>>>>> Thank you for re-opening this Valentyn. I am in favor of
>>>>>> EOLing py2 support sooner than later. The reality is that we will not be
>>>>>> effectively supporting beam python 2 for a long time while the ecosystem
>>>>>> already EOLed python 2. That said, a significant chunk (but no longer a
>>>>>> majority) of our users are still using python 2. Upgrades are painful, it
>>>>>> might be especially painful nowadays. It would be good to hear counter view
>>>>>> points, user voices related to this.
>>>>>> >>>>>>
>>>>>> >>>>>> On Thu, Jun 4, 2020 at 4:53 PM Valentyn Tymofieiev <
>>>>>> valentyn@google.com> wrote:
>>>>>> >>>>>>>
>>>>>> >>>>>>> Back at the end of February we decided to revisit this
>>>>>> conversation in 3 months. Do folks on this thread have any new input or
>>>>>> perspective regarding us balancing "user pain/contributor pain/our ability
>>>>>> to continuously test with python 2 in a shifting environment"?
>>>>>> >>>>>>>
>>>>>> >>>>>>> Some new information on my end is that we have been seeing
>>>>>> steady adoption of Python 3 among Beam Python users in Dataflow,
>>>>>> particularly strong adoption among streaming users, and Dataflow is
>>>>>> sunsetting Python 2 support for all released Beam SDKs later this year [1].
>>>>>> We will have to remove Python 2 Beam test suites that use Dataflow when
>>>>>> Dataflow runner disables Py2 support if this happens before Beam Py2 EOL
>>>>>> (when we have to remove all Py2 suites), including performance tests that
>>>>>> still use Dataflow on Python 3.
>>>>>> >>>>>>>
>>>>>> >>>>>>> I am curious how much motivation there is in the community at
>>>>>> this moment to continue Py2 support in Beam, whether any previous Py3
>>>>>> migration blockers were resolved or any new blockers discovered among Beam
>>>>>> users.
>>>>>> >>>>>>>
>>>>>> >>>>>>> [1]
>>>>>> https://cloud.google.com/python/docs/python2-sunset/#dataflow
>>>>>> >>>>>>>
>>>>>> >>>>>>> On Fri, May 8, 2020 at 3:52 PM Valentyn Tymofieiev <
>>>>>> valentyn@google.com> wrote:
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> That's good news! Thanks for sharing.
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> Another datapoint, here are a few of Beam's dependencies
>>>>>> that no longer release new py2 artifacts (I looked at REQUIRED_PACKAGES +
>>>>>> aws, gcp, and interactive extras):
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> hdfs
>>>>>> >>>>>>>> numpy
>>>>>> >>>>>>>> pyarrow
>>>>>> >>>>>>>> ipython
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> There are more if we include transitive dependencies and
>>>>>> test-only packages. I also remember encountering one issue last month that
>>>>>> was broken only on Py2, which we had to go back and fix.
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> If others have noticed frictions related to ongoing Py2
>>>>>> support or have updates on previously mentioned Py3 migration blockers,
>>>>>> feel free to post them.
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> On Fri, May 8, 2020 at 9:19 AM Robert Bradshaw <
>>>>>> robertwb@google.com> wrote:
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>> It hasn't been 3 months yet, but I wanted to call out a
>>>>>> milestone that
>>>>>> >>>>>>>>> Python 3 downloads crossed the 50% threshold on pypi, if
>>>>>> just briefly.
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>> On Thu, Feb 13, 2020 at 12:40 AM Ismaël Mejía <
>>>>>> iemejia@gmail.com> wrote:
>>>>>> >>>>>>>>> >
>>>>>> >>>>>>>>> > > I would suggest re-evaluating this within the next 3
>>>>>> months again. We need to balance between user pain/contributor pain/our
>>>>>> ability to continuously test with python 2 in a shifting environment.
>>>>>> >>>>>>>>> >
>>>>>> >>>>>>>>> > Good idea for the in 3 months evaluation, at that point
>>>>>> also distributions will probably be phasing out python2 by default which
>>>>>> definitely help in this direction.
>>>>>> >>>>>>>>> > Thanks for updating the roadmap Ahmet
>>>>>> >>>>>>>>> >
>>>>>> >>>>>>>>> >
>>>>>> >>>>>>>>> > On Thu, Feb 13, 2020 at 2:49 AM Ahmet Altay <
>>>>>> altay@google.com> wrote:
>>>>>> >>>>>>>>> >>
>>>>>> >>>>>>>>> >>
>>>>>> >>>>>>>>> >>
>>>>>> >>>>>>>>> >> On Wed, Feb 12, 2020 at 1:29 AM Ismaël Mejía <
>>>>>> iemejia@gmail.com> wrote:
>>>>>> >>>>>>>>> >>>
>>>>>> >>>>>>>>> >>> I am with Chad on this, we should probably extend it a
>>>>>> bit more, even if it
>>>>>> >>>>>>>>> >>> makes us struggle a bit at least we have some
>>>>>> workarounds as Robert suggests,
>>>>>> >>>>>>>>> >>> and as Chad said there are still many people playing
>>>>>> the python 3 catchup game,
>>>>>> >>>>>>>>> >>> so worth to support those users.
>>>>>> >>>>>>>>> >>>
>>>>>> >>>>>>>>> >>>
>>>>>> >>>>>>>>> >>> But maybe it is worth to evaluate the current state
>>>>>> later in the year.
>>>>>> >>>>>>>>> >>
>>>>>> >>>>>>>>> >>
>>>>>> >>>>>>>>> >> I would suggest re-evaluating this within the next 3
>>>>>> months again. We need to balance between user pain/contributor pain/our
>>>>>> ability to continuously test with python 2 in a shifting environment.
>>>>>> >>>>>>>>> >>
>>>>>> >>>>>>>>> >>>
>>>>>> >>>>>>>>> >>> In the
>>>>>> >>>>>>>>> >>> meantime can someone please update our Roadmap in the
>>>>>> website with this info and
>>>>>> >>>>>>>>> >>> where we are with Python 3 support (it looks not up to
>>>>>> date).
>>>>>> >>>>>>>>> >>> https://beam.apache.org/roadmap/
>>>>>> >>>>>>>>> >>
>>>>>> >>>>>>>>> >>
>>>>>> >>>>>>>>> >> I made a minor change to update that page (
>>>>>> https://github.com/apache/beam/pull/10848). A more comprehensive
>>>>>> update to that page and linked (
>>>>>> https://beam.apache.org/roadmap/python-sdk/#python-3-support) would
>>>>>> still be welcome.
>>>>>> >>>>>>>>> >>
>>>>>> >>>>>>>>> >>>
>>>>>> >>>>>>>>> >>>
>>>>>> >>>>>>>>> >>> - Ismaël
>>>>>> >>>>>>>>> >>>
>>>>>> >>>>>>>>> >>>
>>>>>> >>>>>>>>> >>> On Tue, Feb 4, 2020 at 10:49 PM Robert Bradshaw <
>>>>>> robertwb@google.com> wrote:
>>>>>> >>>>>>>>> >>>>
>>>>>> >>>>>>>>> >>>> On Tue, Feb 4, 2020 at 12:12 PM Chad Dombrova <
>>>>>> chadrik@gmail.com> wrote:
>>>>>> >>>>>>>>> >>>> >>
>>>>>> >>>>>>>>> >>>> >> Not to mention that all the nice work for the type
>>>>>> hints will have to be redone in the for 3.x.
>>>>>> >>>>>>>>> >>>> >
>>>>>> >>>>>>>>> >>>> > Note that there's a tool for automatically
>>>>>> converting type comments to annotations:
>>>>>> https://github.com/ilevkivskyi/com2ann
>>>>>> >>>>>>>>> >>>> >
>>>>>> >>>>>>>>> >>>> > So don't let that part bother you.
>>>>>> >>>>>>>>> >>>>
>>>>>> >>>>>>>>> >>>> +1, I wouldn't worry about what can be easily
>>>>>> automated.
>>>>>> >>>>>>>>> >>>>
>>>>>> >>>>>>>>> >>>> > I'm curious what other features you'd like to be
>>>>>> using in the Beam source that you cannot now.
>>>>>> >>>>>>>>> >>>>
>>>>>> >>>>>>>>> >>>> I hit things occasionally, e.g. I just ran into
>>>>>> wanting keyword-only
>>>>>> >>>>>>>>> >>>> arguments the other day.
>>>>>> >>>>>>>>> >>>>
>>>>>> >>>>>>>>> >>>> >> It seems the faster we drop support the better.
>>>>>> >>>>>>>>> >>>> >
>>>>>> >>>>>>>>> >>>> >
>>>>>> >>>>>>>>> >>>> > I've already gone over my position on this, but a
>>>>>> refresher for those who care: some of the key vendors that support my
>>>>>> industry will not offer python3-compatible versions of their software until
>>>>>> the 4th quarter of 2020. If Beam switches to python3-only before that
>>>>>> point we may be forced to stop contributing features (note: I'm the guy who
>>>>>> added the type hints :). Every month you can give us would be greatly
>>>>>> appreciated.
>>>>>> >>>>>>>>> >>>>
>>>>>> >>>>>>>>> >>>> As another data point, we're still 80/20 on Py2/Py3
>>>>>> for downloads at
>>>>>> >>>>>>>>> >>>> PyPi [1] (which I've heard should be taken with a
>>>>>> grain of salt, but
>>>>>> >>>>>>>>> >>>> likely isn't totally off). IMHO that ratio needs to be
>>>>>> way higher for
>>>>>> >>>>>>>>> >>>> Python 3 to consider dropping Python 2. It's pretty
>>>>>> noisy, but say it
>>>>>> >>>>>>>>> >>>> doubles every 3 months that would put us at least
>>>>>> mid-year before we
>>>>>> >>>>>>>>> >>>> hit a cross-over point. On the other hand Q4 2020 is
>>>>>> probably a
>>>>>> >>>>>>>>> >>>> stretch.
>>>>>> >>>>>>>>> >>>>
>>>>>> >>>>>>>>> >>>> We could consider whether it needs to be an
>>>>>> all-or-nothing thing as
>>>>>> >>>>>>>>> >>>> well. E.g. perhaps some features could be Python 3
>>>>>> only sooner than
>>>>>> >>>>>>>>> >>>> the whole codebase. (This would have to be well
>>>>>> justified.) Another
>>>>>> >>>>>>>>> >>>> mitigation is that it is possible to mix Python 2 and
>>>>>> Python 3 in the
>>>>>> >>>>>>>>> >>>> same pipeline with portability, so if there's a
>>>>>> library that you need
>>>>>> >>>>>>>>> >>>> for one DoFn it doesn't mean you have to hold back
>>>>>> your whole
>>>>>> >>>>>>>>> >>>> pipeline.
>>>>>> >>>>>>>>> >>>>
>>>>>> >>>>>>>>> >>>> - Robert
>>>>>> >>>>>>>>> >>>>
>>>>>> >>>>>>>>> >>>> [1] https://pypistats.org/packages/apache-beam , and
>>>>>> that 20% may just
>>>>>> >>>>>>>>> >>>> be a spike.
>>>>>>
>>>>>
Re: Python2.7 Beam End-of-Life Date
Posted by Valentyn Tymofieiev <va...@google.com>.
Alright, let's publish the message on Twitter and echo on Slack once that's
done.
Thank you!
On Tue, Jun 16, 2020 at 5:31 PM Ahmet Altay <al...@google.com> wrote:
> That sounds reasonable to me. I agree, it is better to get specific
> reasons rather than a yes/no answer.
>
> On Tue, Jun 16, 2020 at 1:50 PM Valentyn Tymofieiev <va...@google.com>
> wrote:
>
>> After thinking about it for a bit, I am not sure whether a poll would be
>> actionable. For example, if 1000 users provide a positive response and 5
>> users provide a negative response, how do we interpret that and where do
>> we draw a line? How about instead we encourage users to reach out, and tell
>> what is not working for them? For example:
>>
>> Beam is considering making 2.23.0 a final release supporting Py2. If you
>> are not able to switch to Python 3, please let us know why: [short link to
>> user@ thread] [1].
>>
>> Thoughts?
>>
>> [1]
>> https://lists.apache.org/thread.html/r0de71d98d98b213dd1d0c45c1f5642135116f25def5637a5f41c8d29%40%3Cuser.beam.apache.org%3E
>>
>> On Tue, Jun 16, 2020 at 11:50 AM Ahmet Altay <al...@google.com> wrote:
>>
>>>
>>>
>>> On Tue, Jun 16, 2020 at 11:38 AM Valentyn Tymofieiev <
>>> valentyn@google.com> wrote:
>>>
>>>> I have reached out to user@ for feedback on Python 3 migration[1].
>>>>
>>>
>>> Maybe also ask on slack? There are quite a bit of users there as well.
>>>
>>>
>>>>
>>>> Could somebody from PMC please help with Twitter poll?
>>>>
>>>
>>> I can try to do this. What question would you like to ask? I do not know
>>> much about twitter polls but I assume they have character limits similar to
>>> regular tweets.
>>>
>>>
>>>>
>>>> Technically, we can proceed with the change between 2.23.0 and 2.24.0,
>>>> so that's after 2.23.0 is cut and we give sufficient time for users to
>>>> respond.
>>>>
>>>> [1]
>>>> https://lists.apache.org/thread.html/r0de71d98d98b213dd1d0c45c1f5642135116f25def5637a5f41c8d29%40%3Cuser.beam.apache.org%3E
>>>> On Tue, Jun 16, 2020 at 9:22 AM Ismaël Mejía <ie...@gmail.com> wrote:
>>>>
>>>>> Yes we need to poll this outside as Robert proposed.
>>>>>
>>>>> The proposed last support version corresponds with the next release
>>>>> that will be
>>>>> cut in two weeks. Sounds a bit short if we count the time to poll
>>>>> people on this
>>>>> subject but still could be.
>>>>
>>>> Technically, we can proceed with the change between 2.23.0 and 2.24.0,
>>>> so that's after 2.23.0 is cut and we give sufficient time for users to
>>>> respond.
>>>>
>>>>
>>>>> I remember Chad mentioned in this thread the impossibility to get
>>>>> support for
>>>>> python 2 in his industry until the end of the year, Maybe things have
>>>>> improved.
>>>>> Have they?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Jun 16, 2020 at 6:10 PM Robert Bradshaw <ro...@google.com>
>>>>> wrote:
>>>>> >
>>>>> > I like that option as a concrete proposal. I think we should
>>>>> circulate it more widely (the users list, twitter poll, at least a new
>>>>> thread...), maybe phrasing it as "is there any reason you couldn't migrate
>>>>> to Python 3 (or stick with an older version of Beam) after 2.23 (due to be
>>>>> cut here in a couple of weeks)?" If there is strong concern/pushback, we
>>>>> could consider holding on for one more release.
>>>>> >
>>>>> > On Tue, Jun 16, 2020 at 8:54 AM David Cavazos <dc...@google.com>
>>>>> wrote:
>>>>> >>
>>>>> >> +1
>>>>> >>
>>>>> >> On Mon, Jun 15, 2020 at 6:52 PM Udi Meiri <eh...@google.com> wrote:
>>>>> >>>
>>>>> >>> +1
>>>>> >>>
>>>>> >>> On Mon, Jun 15, 2020 at 4:27 PM Ahmet Altay <al...@google.com>
>>>>> wrote:
>>>>> >>>>
>>>>> >>>> As a concrete proposal, could we commit to removing python 2
>>>>> support by 2.24? In other words, mark the next release 2.23 as the last
>>>>> python 2 compatible Beam version.
>>>>> >>>>
>>>>> >>>> On Mon, Jun 15, 2020 at 2:09 PM Valentyn Tymofieiev <
>>>>> valentyn@google.com> wrote:
>>>>> >>>>>
>>>>> >>>>> Another input here:
>>>>> >>>>>
>>>>> >>>>> If you opened a Python PR in the last few days, you probably
>>>>> noticed that our test suites were broken by a transitive dependency of Beam
>>>>> that dropped python 2 support, but did not declare python_requires>=3 in
>>>>> its setup.py [1]. This temporarily broke a subset of Beam Py2 users (who
>>>>> did not explicitly pin the 'rsa' dependency), and still affects Beam
>>>>> development[2].
>>>>> >>>>>
>>>>> >>>>> This is the second time[3] Beam is affected with an issue of
>>>>> this kind, so support of Python 2 starts to slow down our development, and
>>>>> add toil for maintainers of packages we depend on (both directly and
>>>>> transitively).
>>>>> >>>>>
>>>>> >>>>> [1] https://github.com/sybrenstuvel/python-rsa/issues/152
>>>>> >>>>> [2]
>>>>> https://lists.apache.org/thread.html/r9993b40b0c1cb8682ce56013165d4b80fdde0ee469a73bcb9466ddfb%40%3Cdev.beam.apache.org%3E
>>>>> >>>>> [3] https://github.com/hamcrest/PyHamcrest/issues/131
>>>>> >>>>>
>>>>> >>>>> On Tue, Jun 9, 2020 at 4:06 PM Ahmet Altay <al...@google.com>
>>>>> wrote:
>>>>> >>>>>>
>>>>> >>>>>> Thank you for re-opening this Valentyn. I am in favor of EOLing
>>>>> py2 support sooner than later. The reality is that we will not be
>>>>> effectively supporting beam python 2 for a long time while the ecosystem
>>>>> already EOLed python 2. That said, a significant chunk (but no longer a
>>>>> majority) of our users are still using python 2. Upgrades are painful, it
>>>>> might be especially painful nowadays. It would be good to hear counter view
>>>>> points, user voices related to this.
>>>>> >>>>>>
>>>>> >>>>>> On Thu, Jun 4, 2020 at 4:53 PM Valentyn Tymofieiev <
>>>>> valentyn@google.com> wrote:
>>>>> >>>>>>>
>>>>> >>>>>>> Back at the end of February we decided to revisit this
>>>>> conversation in 3 months. Do folks on this thread have any new input or
>>>>> perspective regarding us balancing "user pain/contributor pain/our ability
>>>>> to continuously test with python 2 in a shifting environment"?
>>>>> >>>>>>>
>>>>> >>>>>>> Some new information on my end is that we have been seeing
>>>>> steady adoption of Python 3 among Beam Python users in Dataflow,
>>>>> particularly strong adoption among streaming users, and Dataflow is
>>>>> sunsetting Python 2 support for all released Beam SDKs later this year [1].
>>>>> We will have to remove Python 2 Beam test suites that use Dataflow when
>>>>> Dataflow runner disables Py2 support if this happens before Beam Py2 EOL
>>>>> (when we have to remove all Py2 suites), including performance tests that
>>>>> still use Dataflow on Python 3.
>>>>> >>>>>>>
>>>>> >>>>>>> I am curious how much motivation there is in the community at
>>>>> this moment to continue Py2 support in Beam, whether any previous Py3
>>>>> migration blockers were resolved or any new blockers discovered among Beam
>>>>> users.
>>>>> >>>>>>>
>>>>> >>>>>>> [1]
>>>>> https://cloud.google.com/python/docs/python2-sunset/#dataflow
>>>>> >>>>>>>
>>>>> >>>>>>> On Fri, May 8, 2020 at 3:52 PM Valentyn Tymofieiev <
>>>>> valentyn@google.com> wrote:
>>>>> >>>>>>>>
>>>>> >>>>>>>> That's good news! Thanks for sharing.
>>>>> >>>>>>>>
>>>>> >>>>>>>> Another datapoint, here are a few of Beam's dependencies that
>>>>> no longer release new py2 artifacts (I looked at REQUIRED_PACKAGES + aws,
>>>>> gcp, and interactive extras):
>>>>> >>>>>>>>
>>>>> >>>>>>>> hdfs
>>>>> >>>>>>>> numpy
>>>>> >>>>>>>> pyarrow
>>>>> >>>>>>>> ipython
>>>>> >>>>>>>>
>>>>> >>>>>>>> There are more if we include transitive dependencies and
>>>>> test-only packages. I also remember encountering one issue last month that
>>>>> was broken only on Py2, which we had to go back and fix.
>>>>> >>>>>>>>
>>>>> >>>>>>>> If others have noticed frictions related to ongoing Py2
>>>>> support or have updates on previously mentioned Py3 migration blockers,
>>>>> feel free to post them.
>>>>> >>>>>>>>
>>>>> >>>>>>>> On Fri, May 8, 2020 at 9:19 AM Robert Bradshaw <
>>>>> robertwb@google.com> wrote:
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> It hasn't been 3 months yet, but I wanted to call out a
>>>>> milestone that
>>>>> >>>>>>>>> Python 3 downloads crossed the 50% threshold on pypi, if
>>>>> just briefly.
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> On Thu, Feb 13, 2020 at 12:40 AM Ismaël Mejía <
>>>>> iemejia@gmail.com> wrote:
>>>>> >>>>>>>>> >
>>>>> >>>>>>>>> > > I would suggest re-evaluating this within the next 3
>>>>> months again. We need to balance between user pain/contributor pain/our
>>>>> ability to continuously test with python 2 in a shifting environment.
>>>>> >>>>>>>>> >
>>>>> >>>>>>>>> > Good idea for the in 3 months evaluation, at that point
>>>>> also distributions will probably be phasing out python2 by default which
>>>>> definitely help in this direction.
>>>>> >>>>>>>>> > Thanks for updating the roadmap Ahmet
>>>>> >>>>>>>>> >
>>>>> >>>>>>>>> >
>>>>> >>>>>>>>> > On Thu, Feb 13, 2020 at 2:49 AM Ahmet Altay <
>>>>> altay@google.com> wrote:
>>>>> >>>>>>>>> >>
>>>>> >>>>>>>>> >>
>>>>> >>>>>>>>> >>
>>>>> >>>>>>>>> >> On Wed, Feb 12, 2020 at 1:29 AM Ismaël Mejía <
>>>>> iemejia@gmail.com> wrote:
>>>>> >>>>>>>>> >>>
>>>>> >>>>>>>>> >>> I am with Chad on this, we should probably extend it a
>>>>> bit more, even if it
>>>>> >>>>>>>>> >>> makes us struggle a bit at least we have some
>>>>> workarounds as Robert suggests,
>>>>> >>>>>>>>> >>> and as Chad said there are still many people playing the
>>>>> python 3 catchup game,
>>>>> >>>>>>>>> >>> so worth to support those users.
>>>>> >>>>>>>>> >>>
>>>>> >>>>>>>>> >>>
>>>>> >>>>>>>>> >>> But maybe it is worth to evaluate the current state
>>>>> later in the year.
>>>>> >>>>>>>>> >>
>>>>> >>>>>>>>> >>
>>>>> >>>>>>>>> >> I would suggest re-evaluating this within the next 3
>>>>> months again. We need to balance between user pain/contributor pain/our
>>>>> ability to continuously test with python 2 in a shifting environment.
>>>>> >>>>>>>>> >>
>>>>> >>>>>>>>> >>>
>>>>> >>>>>>>>> >>> In the
>>>>> >>>>>>>>> >>> meantime can someone please update our Roadmap in the
>>>>> website with this info and
>>>>> >>>>>>>>> >>> where we are with Python 3 support (it looks not up to
>>>>> date).
>>>>> >>>>>>>>> >>> https://beam.apache.org/roadmap/
>>>>> >>>>>>>>> >>
>>>>> >>>>>>>>> >>
>>>>> >>>>>>>>> >> I made a minor change to update that page (
>>>>> https://github.com/apache/beam/pull/10848). A more comprehensive
>>>>> update to that page and linked (
>>>>> https://beam.apache.org/roadmap/python-sdk/#python-3-support) would
>>>>> still be welcome.
>>>>> >>>>>>>>> >>
>>>>> >>>>>>>>> >>>
>>>>> >>>>>>>>> >>>
>>>>> >>>>>>>>> >>> - Ismaël
>>>>> >>>>>>>>> >>>
>>>>> >>>>>>>>> >>>
>>>>> >>>>>>>>> >>> On Tue, Feb 4, 2020 at 10:49 PM Robert Bradshaw <
>>>>> robertwb@google.com> wrote:
>>>>> >>>>>>>>> >>>>
>>>>> >>>>>>>>> >>>> On Tue, Feb 4, 2020 at 12:12 PM Chad Dombrova <
>>>>> chadrik@gmail.com> wrote:
>>>>> >>>>>>>>> >>>> >>
>>>>> >>>>>>>>> >>>> >> Not to mention that all the nice work for the type
>>>>> hints will have to be redone in the for 3.x.
>>>>> >>>>>>>>> >>>> >
>>>>> >>>>>>>>> >>>> > Note that there's a tool for automatically converting
>>>>> type comments to annotations: https://github.com/ilevkivskyi/com2ann
>>>>> >>>>>>>>> >>>> >
>>>>> >>>>>>>>> >>>> > So don't let that part bother you.
>>>>> >>>>>>>>> >>>>
>>>>> >>>>>>>>> >>>> +1, I wouldn't worry about what can be easily automated.
>>>>> >>>>>>>>> >>>>
>>>>> >>>>>>>>> >>>> > I'm curious what other features you'd like to be
>>>>> using in the Beam source that you cannot now.
>>>>> >>>>>>>>> >>>>
>>>>> >>>>>>>>> >>>> I hit things occasionally, e.g. I just ran into wanting
>>>>> keyword-only
>>>>> >>>>>>>>> >>>> arguments the other day.
>>>>> >>>>>>>>> >>>>
>>>>> >>>>>>>>> >>>> >> It seems the faster we drop support the better.
>>>>> >>>>>>>>> >>>> >
>>>>> >>>>>>>>> >>>> >
>>>>> >>>>>>>>> >>>> > I've already gone over my position on this, but a
>>>>> refresher for those who care: some of the key vendors that support my
>>>>> industry will not offer python3-compatible versions of their software until
>>>>> the 4th quarter of 2020. If Beam switches to python3-only before that
>>>>> point we may be forced to stop contributing features (note: I'm the guy who
>>>>> added the type hints :). Every month you can give us would be greatly
>>>>> appreciated.
>>>>> >>>>>>>>> >>>>
>>>>> >>>>>>>>> >>>> As another data point, we're still 80/20 on Py2/Py3 for
>>>>> downloads at
>>>>> >>>>>>>>> >>>> PyPi [1] (which I've heard should be taken with a grain
>>>>> of salt, but
>>>>> >>>>>>>>> >>>> likely isn't totally off). IMHO that ratio needs to be
>>>>> way higher for
>>>>> >>>>>>>>> >>>> Python 3 to consider dropping Python 2. It's pretty
>>>>> noisy, but say it
>>>>> >>>>>>>>> >>>> doubles every 3 months that would put us at least
>>>>> mid-year before we
>>>>> >>>>>>>>> >>>> hit a cross-over point. On the other hand Q4 2020 is
>>>>> probably a
>>>>> >>>>>>>>> >>>> stretch.
>>>>> >>>>>>>>> >>>>
>>>>> >>>>>>>>> >>>> We could consider whether it needs to be an
>>>>> all-or-nothing thing as
>>>>> >>>>>>>>> >>>> well. E.g. perhaps some features could be Python 3 only
>>>>> sooner than
>>>>> >>>>>>>>> >>>> the whole codebase. (This would have to be well
>>>>> justified.) Another
>>>>> >>>>>>>>> >>>> mitigation is that it is possible to mix Python 2 and
>>>>> Python 3 in the
>>>>> >>>>>>>>> >>>> same pipeline with portability, so if there's a library
>>>>> that you need
>>>>> >>>>>>>>> >>>> for one DoFn it doesn't mean you have to hold back your
>>>>> whole
>>>>> >>>>>>>>> >>>> pipeline.
>>>>> >>>>>>>>> >>>>
>>>>> >>>>>>>>> >>>> - Robert
>>>>> >>>>>>>>> >>>>
>>>>> >>>>>>>>> >>>> [1] https://pypistats.org/packages/apache-beam , and
>>>>> that 20% may just
>>>>> >>>>>>>>> >>>> be a spike.
>>>>>
>>>>
Re: Python2.7 Beam End-of-Life Date
Posted by Ahmet Altay <al...@google.com>.
That sounds reasonable to me. I agree, it is better to get specific reasons
rather than a yes/no answer.
On Tue, Jun 16, 2020 at 1:50 PM Valentyn Tymofieiev <va...@google.com>
wrote:
> After thinking about it for a bit, I am not sure whether a poll would be
> actionable. For example, if 1000 users provide a positive response and 5
> users provide a negative response, how do we interpret that and where do
> we draw a line? How about instead we encourage users to reach out, and tell
> what is not working for them? For example:
>
> Beam is considering making 2.23.0 a final release supporting Py2. If you
> are not able to switch to Python 3, please let us know why: [short link to
> user@ thread] [1].
>
> Thoughts?
>
> [1]
> https://lists.apache.org/thread.html/r0de71d98d98b213dd1d0c45c1f5642135116f25def5637a5f41c8d29%40%3Cuser.beam.apache.org%3E
>
> On Tue, Jun 16, 2020 at 11:50 AM Ahmet Altay <al...@google.com> wrote:
>
>>
>>
>> On Tue, Jun 16, 2020 at 11:38 AM Valentyn Tymofieiev <va...@google.com>
>> wrote:
>>
>>> I have reached out to user@ for feedback on Python 3 migration[1].
>>>
>>
>> Maybe also ask on slack? There are quite a bit of users there as well.
>>
>>
>>>
>>> Could somebody from PMC please help with Twitter poll?
>>>
>>
>> I can try to do this. What question would you like to ask? I do not know
>> much about twitter polls but I assume they have character limits similar to
>> regular tweets.
>>
>>
>>>
>>> Technically, we can proceed with the change between 2.23.0 and 2.24.0,
>>> so that's after 2.23.0 is cut and we give sufficient time for users to
>>> respond.
>>>
>>> [1]
>>> https://lists.apache.org/thread.html/r0de71d98d98b213dd1d0c45c1f5642135116f25def5637a5f41c8d29%40%3Cuser.beam.apache.org%3E
>>> On Tue, Jun 16, 2020 at 9:22 AM Ismaël Mejía <ie...@gmail.com> wrote:
>>>
>>>> Yes we need to poll this outside as Robert proposed.
>>>>
>>>> The proposed last support version corresponds with the next release
>>>> that will be
>>>> cut in two weeks. Sounds a bit short if we count the time to poll
>>>> people on this
>>>> subject but still could be.
>>>
>>> Technically, we can proceed with the change between 2.23.0 and 2.24.0,
>>> so that's after 2.23.0 is cut and we give sufficient time for users to
>>> respond.
>>>
>>>
>>>> I remember Chad mentioned in this thread the impossibility to get
>>>> support for
>>>> python 2 in his industry until the end of the year, Maybe things have
>>>> improved.
>>>> Have they?
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, Jun 16, 2020 at 6:10 PM Robert Bradshaw <ro...@google.com>
>>>> wrote:
>>>> >
>>>> > I like that option as a concrete proposal. I think we should
>>>> circulate it more widely (the users list, twitter poll, at least a new
>>>> thread...), maybe phrasing it as "is there any reason you couldn't migrate
>>>> to Python 3 (or stick with an older version of Beam) after 2.23 (due to be
>>>> cut here in a couple of weeks)?" If there is strong concern/pushback, we
>>>> could consider holding on for one more release.
>>>> >
>>>> > On Tue, Jun 16, 2020 at 8:54 AM David Cavazos <dc...@google.com>
>>>> wrote:
>>>> >>
>>>> >> +1
>>>> >>
>>>> >> On Mon, Jun 15, 2020 at 6:52 PM Udi Meiri <eh...@google.com> wrote:
>>>> >>>
>>>> >>> +1
>>>> >>>
>>>> >>> On Mon, Jun 15, 2020 at 4:27 PM Ahmet Altay <al...@google.com>
>>>> wrote:
>>>> >>>>
>>>> >>>> As a concrete proposal, could we commit to removing python 2
>>>> support by 2.24? In other words, mark the next release 2.23 as the last
>>>> python 2 compatible Beam version.
>>>> >>>>
>>>> >>>> On Mon, Jun 15, 2020 at 2:09 PM Valentyn Tymofieiev <
>>>> valentyn@google.com> wrote:
>>>> >>>>>
>>>> >>>>> Another input here:
>>>> >>>>>
>>>> >>>>> If you opened a Python PR in the last few days, you probably
>>>> noticed that our test suites were broken by a transitive dependency of Beam
>>>> that dropped python 2 support, but did not declare python_requires>=3 in
>>>> its setup.py [1]. This temporarily broke a subset of Beam Py2 users (who
>>>> did not explicitly pin the 'rsa' dependency), and still affects Beam
>>>> development[2].
>>>> >>>>>
>>>> >>>>> This is the second time[3] Beam is affected with an issue of this
>>>> kind, so support of Python 2 starts to slow down our development, and add
>>>> toil for maintainers of packages we depend on (both directly and
>>>> transitively).
>>>> >>>>>
>>>> >>>>> [1] https://github.com/sybrenstuvel/python-rsa/issues/152
>>>> >>>>> [2]
>>>> https://lists.apache.org/thread.html/r9993b40b0c1cb8682ce56013165d4b80fdde0ee469a73bcb9466ddfb%40%3Cdev.beam.apache.org%3E
>>>> >>>>> [3] https://github.com/hamcrest/PyHamcrest/issues/131
>>>> >>>>>
>>>> >>>>> On Tue, Jun 9, 2020 at 4:06 PM Ahmet Altay <al...@google.com>
>>>> wrote:
>>>> >>>>>>
>>>> >>>>>> Thank you for re-opening this Valentyn. I am in favor of EOLing
>>>> py2 support sooner than later. The reality is that we will not be
>>>> effectively supporting beam python 2 for a long time while the ecosystem
>>>> already EOLed python 2. That said, a significant chunk (but no longer a
>>>> majority) of our users are still using python 2. Upgrades are painful, it
>>>> might be especially painful nowadays. It would be good to hear counter view
>>>> points, user voices related to this.
>>>> >>>>>>
>>>> >>>>>> On Thu, Jun 4, 2020 at 4:53 PM Valentyn Tymofieiev <
>>>> valentyn@google.com> wrote:
>>>> >>>>>>>
>>>> >>>>>>> Back at the end of February we decided to revisit this
>>>> conversation in 3 months. Do folks on this thread have any new input or
>>>> perspective regarding us balancing "user pain/contributor pain/our ability
>>>> to continuously test with python 2 in a shifting environment"?
>>>> >>>>>>>
>>>> >>>>>>> Some new information on my end is that we have been seeing
>>>> steady adoption of Python 3 among Beam Python users in Dataflow,
>>>> particularly strong adoption among streaming users, and Dataflow is
>>>> sunsetting Python 2 support for all released Beam SDKs later this year [1].
>>>> We will have to remove Python 2 Beam test suites that use Dataflow when
>>>> Dataflow runner disables Py2 support if this happens before Beam Py2 EOL
>>>> (when we have to remove all Py2 suites), including performance tests that
>>>> still use Dataflow on Python 3.
>>>> >>>>>>>
>>>> >>>>>>> I am curious how much motivation there is in the community at
>>>> this moment to continue Py2 support in Beam, whether any previous Py3
>>>> migration blockers were resolved or any new blockers discovered among Beam
>>>> users.
>>>> >>>>>>>
>>>> >>>>>>> [1]
>>>> https://cloud.google.com/python/docs/python2-sunset/#dataflow
>>>> >>>>>>>
>>>> >>>>>>> On Fri, May 8, 2020 at 3:52 PM Valentyn Tymofieiev <
>>>> valentyn@google.com> wrote:
>>>> >>>>>>>>
>>>> >>>>>>>> That's good news! Thanks for sharing.
>>>> >>>>>>>>
>>>> >>>>>>>> Another datapoint, here are a few of Beam's dependencies that
>>>> no longer release new py2 artifacts (I looked at REQUIRED_PACKAGES + aws,
>>>> gcp, and interactive extras):
>>>> >>>>>>>>
>>>> >>>>>>>> hdfs
>>>> >>>>>>>> numpy
>>>> >>>>>>>> pyarrow
>>>> >>>>>>>> ipython
>>>> >>>>>>>>
>>>> >>>>>>>> There are more if we include transitive dependencies and
>>>> test-only packages. I also remember encountering one issue last month that
>>>> was broken only on Py2, which we had to go back and fix.
>>>> >>>>>>>>
>>>> >>>>>>>> If others have noticed frictions related to ongoing Py2
>>>> support or have updates on previously mentioned Py3 migration blockers,
>>>> feel free to post them.
>>>> >>>>>>>>
>>>> >>>>>>>> On Fri, May 8, 2020 at 9:19 AM Robert Bradshaw <
>>>> robertwb@google.com> wrote:
>>>> >>>>>>>>>
>>>> >>>>>>>>> It hasn't been 3 months yet, but I wanted to call out a
>>>> milestone that
>>>> >>>>>>>>> Python 3 downloads crossed the 50% threshold on pypi, if just
>>>> briefly.
>>>> >>>>>>>>>
>>>> >>>>>>>>> On Thu, Feb 13, 2020 at 12:40 AM Ismaël Mejía <
>>>> iemejia@gmail.com> wrote:
>>>> >>>>>>>>> >
>>>> >>>>>>>>> > > I would suggest re-evaluating this within the next 3
>>>> months again. We need to balance between user pain/contributor pain/our
>>>> ability to continuously test with python 2 in a shifting environment.
>>>> >>>>>>>>> >
>>>> >>>>>>>>> > Good idea for the in 3 months evaluation, at that point
>>>> also distributions will probably be phasing out python2 by default which
>>>> definitely help in this direction.
>>>> >>>>>>>>> > Thanks for updating the roadmap Ahmet
>>>> >>>>>>>>> >
>>>> >>>>>>>>> >
>>>> >>>>>>>>> > On Thu, Feb 13, 2020 at 2:49 AM Ahmet Altay <
>>>> altay@google.com> wrote:
>>>> >>>>>>>>> >>
>>>> >>>>>>>>> >>
>>>> >>>>>>>>> >>
>>>> >>>>>>>>> >> On Wed, Feb 12, 2020 at 1:29 AM Ismaël Mejía <
>>>> iemejia@gmail.com> wrote:
>>>> >>>>>>>>> >>>
>>>> >>>>>>>>> >>> I am with Chad on this, we should probably extend it a
>>>> bit more, even if it
>>>> >>>>>>>>> >>> makes us struggle a bit at least we have some workarounds
>>>> as Robert suggests,
>>>> >>>>>>>>> >>> and as Chad said there are still many people playing the
>>>> python 3 catchup game,
>>>> >>>>>>>>> >>> so worth to support those users.
>>>> >>>>>>>>> >>>
>>>> >>>>>>>>> >>>
>>>> >>>>>>>>> >>> But maybe it is worth to evaluate the current state later
>>>> in the year.
>>>> >>>>>>>>> >>
>>>> >>>>>>>>> >>
>>>> >>>>>>>>> >> I would suggest re-evaluating this within the next 3
>>>> months again. We need to balance between user pain/contributor pain/our
>>>> ability to continuously test with python 2 in a shifting environment.
>>>> >>>>>>>>> >>
>>>> >>>>>>>>> >>>
>>>> >>>>>>>>> >>> In the
>>>> >>>>>>>>> >>> meantime can someone please update our Roadmap in the
>>>> website with this info and
>>>> >>>>>>>>> >>> where we are with Python 3 support (it looks not up to
>>>> date).
>>>> >>>>>>>>> >>> https://beam.apache.org/roadmap/
>>>> >>>>>>>>> >>
>>>> >>>>>>>>> >>
>>>> >>>>>>>>> >> I made a minor change to update that page (
>>>> https://github.com/apache/beam/pull/10848). A more comprehensive
>>>> update to that page and linked (
>>>> https://beam.apache.org/roadmap/python-sdk/#python-3-support) would
>>>> still be welcome.
>>>> >>>>>>>>> >>
>>>> >>>>>>>>> >>>
>>>> >>>>>>>>> >>>
>>>> >>>>>>>>> >>> - Ismaël
>>>> >>>>>>>>> >>>
>>>> >>>>>>>>> >>>
>>>> >>>>>>>>> >>> On Tue, Feb 4, 2020 at 10:49 PM Robert Bradshaw <
>>>> robertwb@google.com> wrote:
>>>> >>>>>>>>> >>>>
>>>> >>>>>>>>> >>>> On Tue, Feb 4, 2020 at 12:12 PM Chad Dombrova <
>>>> chadrik@gmail.com> wrote:
>>>> >>>>>>>>> >>>> >>
>>>> >>>>>>>>> >>>> >> Not to mention that all the nice work for the type
>>>> hints will have to be redone in the for 3.x.
>>>> >>>>>>>>> >>>> >
>>>> >>>>>>>>> >>>> > Note that there's a tool for automatically converting
>>>> type comments to annotations: https://github.com/ilevkivskyi/com2ann
>>>> >>>>>>>>> >>>> >
>>>> >>>>>>>>> >>>> > So don't let that part bother you.
>>>> >>>>>>>>> >>>>
>>>> >>>>>>>>> >>>> +1, I wouldn't worry about what can be easily automated.
>>>> >>>>>>>>> >>>>
>>>> >>>>>>>>> >>>> > I'm curious what other features you'd like to be using
>>>> in the Beam source that you cannot now.
>>>> >>>>>>>>> >>>>
>>>> >>>>>>>>> >>>> I hit things occasionally, e.g. I just ran into wanting
>>>> keyword-only
>>>> >>>>>>>>> >>>> arguments the other day.
>>>> >>>>>>>>> >>>>
>>>> >>>>>>>>> >>>> >> It seems the faster we drop support the better.
>>>> >>>>>>>>> >>>> >
>>>> >>>>>>>>> >>>> >
>>>> >>>>>>>>> >>>> > I've already gone over my position on this, but a
>>>> refresher for those who care: some of the key vendors that support my
>>>> industry will not offer python3-compatible versions of their software until
>>>> the 4th quarter of 2020. If Beam switches to python3-only before that
>>>> point we may be forced to stop contributing features (note: I'm the guy who
>>>> added the type hints :). Every month you can give us would be greatly
>>>> appreciated.
>>>> >>>>>>>>> >>>>
>>>> >>>>>>>>> >>>> As another data point, we're still 80/20 on Py2/Py3 for
>>>> downloads at
>>>> >>>>>>>>> >>>> PyPi [1] (which I've heard should be taken with a grain
>>>> of salt, but
>>>> >>>>>>>>> >>>> likely isn't totally off). IMHO that ratio needs to be
>>>> way higher for
>>>> >>>>>>>>> >>>> Python 3 to consider dropping Python 2. It's pretty
>>>> noisy, but say it
>>>> >>>>>>>>> >>>> doubles every 3 months that would put us at least
>>>> mid-year before we
>>>> >>>>>>>>> >>>> hit a cross-over point. On the other hand Q4 2020 is
>>>> probably a
>>>> >>>>>>>>> >>>> stretch.
>>>> >>>>>>>>> >>>>
>>>> >>>>>>>>> >>>> We could consider whether it needs to be an
>>>> all-or-nothing thing as
>>>> >>>>>>>>> >>>> well. E.g. perhaps some features could be Python 3 only
>>>> sooner than
>>>> >>>>>>>>> >>>> the whole codebase. (This would have to be well
>>>> justified.) Another
>>>> >>>>>>>>> >>>> mitigation is that it is possible to mix Python 2 and
>>>> Python 3 in the
>>>> >>>>>>>>> >>>> same pipeline with portability, so if there's a library
>>>> that you need
>>>> >>>>>>>>> >>>> for one DoFn it doesn't mean you have to hold back your
>>>> whole
>>>> >>>>>>>>> >>>> pipeline.
>>>> >>>>>>>>> >>>>
>>>> >>>>>>>>> >>>> - Robert
>>>> >>>>>>>>> >>>>
>>>> >>>>>>>>> >>>> [1] https://pypistats.org/packages/apache-beam , and
>>>> that 20% may just
>>>> >>>>>>>>> >>>> be a spike.
>>>>
>>>
Re: Python2.7 Beam End-of-Life Date
Posted by Valentyn Tymofieiev <va...@google.com>.
After thinking about it for a bit, I am not sure whether a poll would be
actionable. For example, if 1000 users provide a positive response and 5
users provide a negative response, how do we interpret that and where do
we draw a line? How about instead we encourage users to reach out, and tell
what is not working for them? For example:
Beam is considering making 2.23.0 a final release supporting Py2. If you
are not able to switch to Python 3, please let us know why: [short link to
user@ thread] [1].
Thoughts?
[1]
https://lists.apache.org/thread.html/r0de71d98d98b213dd1d0c45c1f5642135116f25def5637a5f41c8d29%40%3Cuser.beam.apache.org%3E
On Tue, Jun 16, 2020 at 11:50 AM Ahmet Altay <al...@google.com> wrote:
>
>
> On Tue, Jun 16, 2020 at 11:38 AM Valentyn Tymofieiev <va...@google.com>
> wrote:
>
>> I have reached out to user@ for feedback on Python 3 migration[1].
>>
>
> Maybe also ask on slack? There are quite a bit of users there as well.
>
>
>>
>> Could somebody from PMC please help with Twitter poll?
>>
>
> I can try to do this. What question would you like to ask? I do not know
> much about twitter polls but I assume they have character limits similar to
> regular tweets.
>
>
>>
>> Technically, we can proceed with the change between 2.23.0 and 2.24.0, so
>> that's after 2.23.0 is cut and we give sufficient time for users to
>> respond.
>>
>> [1]
>> https://lists.apache.org/thread.html/r0de71d98d98b213dd1d0c45c1f5642135116f25def5637a5f41c8d29%40%3Cuser.beam.apache.org%3E
>> On Tue, Jun 16, 2020 at 9:22 AM Ismaël Mejía <ie...@gmail.com> wrote:
>>
>>> Yes we need to poll this outside as Robert proposed.
>>>
>>> The proposed last support version corresponds with the next release that
>>> will be
>>> cut in two weeks. Sounds a bit short if we count the time to poll people
>>> on this
>>> subject but still could be.
>>
>> Technically, we can proceed with the change between 2.23.0 and 2.24.0, so
>> that's after 2.23.0 is cut and we give sufficient time for users to
>> respond.
>>
>>
>>> I remember Chad mentioned in this thread the impossibility to get
>>> support for
>>> python 2 in his industry until the end of the year, Maybe things have
>>> improved.
>>> Have they?
>>>
>>>
>>>
>>>
>>> On Tue, Jun 16, 2020 at 6:10 PM Robert Bradshaw <ro...@google.com>
>>> wrote:
>>> >
>>> > I like that option as a concrete proposal. I think we should circulate
>>> it more widely (the users list, twitter poll, at least a new thread...),
>>> maybe phrasing it as "is there any reason you couldn't migrate to Python 3
>>> (or stick with an older version of Beam) after 2.23 (due to be cut here in
>>> a couple of weeks)?" If there is strong concern/pushback, we could consider
>>> holding on for one more release.
>>> >
>>> > On Tue, Jun 16, 2020 at 8:54 AM David Cavazos <dc...@google.com>
>>> wrote:
>>> >>
>>> >> +1
>>> >>
>>> >> On Mon, Jun 15, 2020 at 6:52 PM Udi Meiri <eh...@google.com> wrote:
>>> >>>
>>> >>> +1
>>> >>>
>>> >>> On Mon, Jun 15, 2020 at 4:27 PM Ahmet Altay <al...@google.com>
>>> wrote:
>>> >>>>
>>> >>>> As a concrete proposal, could we commit to removing python 2
>>> support by 2.24? In other words, mark the next release 2.23 as the last
>>> python 2 compatible Beam version.
>>> >>>>
>>> >>>> On Mon, Jun 15, 2020 at 2:09 PM Valentyn Tymofieiev <
>>> valentyn@google.com> wrote:
>>> >>>>>
>>> >>>>> Another input here:
>>> >>>>>
>>> >>>>> If you opened a Python PR in the last few days, you probably
>>> noticed that our test suites were broken by a transitive dependency of Beam
>>> that dropped python 2 support, but did not declare python_requires>=3 in
>>> its setup.py [1]. This temporarily broke a subset of Beam Py2 users (who
>>> did not explicitly pin the 'rsa' dependency), and still affects Beam
>>> development[2].
>>> >>>>>
>>> >>>>> This is the second time[3] Beam is affected with an issue of this
>>> kind, so support of Python 2 starts to slow down our development, and add
>>> toil for maintainers of packages we depend on (both directly and
>>> transitively).
>>> >>>>>
>>> >>>>> [1] https://github.com/sybrenstuvel/python-rsa/issues/152
>>> >>>>> [2]
>>> https://lists.apache.org/thread.html/r9993b40b0c1cb8682ce56013165d4b80fdde0ee469a73bcb9466ddfb%40%3Cdev.beam.apache.org%3E
>>> >>>>> [3] https://github.com/hamcrest/PyHamcrest/issues/131
>>> >>>>>
>>> >>>>> On Tue, Jun 9, 2020 at 4:06 PM Ahmet Altay <al...@google.com>
>>> wrote:
>>> >>>>>>
>>> >>>>>> Thank you for re-opening this Valentyn. I am in favor of EOLing
>>> py2 support sooner than later. The reality is that we will not be
>>> effectively supporting beam python 2 for a long time while the ecosystem
>>> already EOLed python 2. That said, a significant chunk (but no longer a
>>> majority) of our users are still using python 2. Upgrades are painful, it
>>> might be especially painful nowadays. It would be good to hear counter view
>>> points, user voices related to this.
>>> >>>>>>
>>> >>>>>> On Thu, Jun 4, 2020 at 4:53 PM Valentyn Tymofieiev <
>>> valentyn@google.com> wrote:
>>> >>>>>>>
>>> >>>>>>> Back at the end of February we decided to revisit this
>>> conversation in 3 months. Do folks on this thread have any new input or
>>> perspective regarding us balancing "user pain/contributor pain/our ability
>>> to continuously test with python 2 in a shifting environment"?
>>> >>>>>>>
>>> >>>>>>> Some new information on my end is that we have been seeing
>>> steady adoption of Python 3 among Beam Python users in Dataflow,
>>> particularly strong adoption among streaming users, and Dataflow is
>>> sunsetting Python 2 support for all released Beam SDKs later this year [1].
>>> We will have to remove Python 2 Beam test suites that use Dataflow when
>>> Dataflow runner disables Py2 support if this happens before Beam Py2 EOL
>>> (when we have to remove all Py2 suites), including performance tests that
>>> still use Dataflow on Python 3.
>>> >>>>>>>
>>> >>>>>>> I am curious how much motivation there is in the community at
>>> this moment to continue Py2 support in Beam, whether any previous Py3
>>> migration blockers were resolved or any new blockers discovered among Beam
>>> users.
>>> >>>>>>>
>>> >>>>>>> [1]
>>> https://cloud.google.com/python/docs/python2-sunset/#dataflow
>>> >>>>>>>
>>> >>>>>>> On Fri, May 8, 2020 at 3:52 PM Valentyn Tymofieiev <
>>> valentyn@google.com> wrote:
>>> >>>>>>>>
>>> >>>>>>>> That's good news! Thanks for sharing.
>>> >>>>>>>>
>>> >>>>>>>> Another datapoint, here are a few of Beam's dependencies that
>>> no longer release new py2 artifacts (I looked at REQUIRED_PACKAGES + aws,
>>> gcp, and interactive extras):
>>> >>>>>>>>
>>> >>>>>>>> hdfs
>>> >>>>>>>> numpy
>>> >>>>>>>> pyarrow
>>> >>>>>>>> ipython
>>> >>>>>>>>
>>> >>>>>>>> There are more if we include transitive dependencies and
>>> test-only packages. I also remember encountering one issue last month that
>>> was broken only on Py2, which we had to go back and fix.
>>> >>>>>>>>
>>> >>>>>>>> If others have noticed frictions related to ongoing Py2 support
>>> or have updates on previously mentioned Py3 migration blockers, feel free
>>> to post them.
>>> >>>>>>>>
>>> >>>>>>>> On Fri, May 8, 2020 at 9:19 AM Robert Bradshaw <
>>> robertwb@google.com> wrote:
>>> >>>>>>>>>
>>> >>>>>>>>> It hasn't been 3 months yet, but I wanted to call out a
>>> milestone that
>>> >>>>>>>>> Python 3 downloads crossed the 50% threshold on pypi, if just
>>> briefly.
>>> >>>>>>>>>
>>> >>>>>>>>> On Thu, Feb 13, 2020 at 12:40 AM Ismaël Mejía <
>>> iemejia@gmail.com> wrote:
>>> >>>>>>>>> >
>>> >>>>>>>>> > > I would suggest re-evaluating this within the next 3
>>> months again. We need to balance between user pain/contributor pain/our
>>> ability to continuously test with python 2 in a shifting environment.
>>> >>>>>>>>> >
>>> >>>>>>>>> > Good idea for the in 3 months evaluation, at that point also
>>> distributions will probably be phasing out python2 by default which
>>> definitely help in this direction.
>>> >>>>>>>>> > Thanks for updating the roadmap Ahmet
>>> >>>>>>>>> >
>>> >>>>>>>>> >
>>> >>>>>>>>> > On Thu, Feb 13, 2020 at 2:49 AM Ahmet Altay <
>>> altay@google.com> wrote:
>>> >>>>>>>>> >>
>>> >>>>>>>>> >>
>>> >>>>>>>>> >>
>>> >>>>>>>>> >> On Wed, Feb 12, 2020 at 1:29 AM Ismaël Mejía <
>>> iemejia@gmail.com> wrote:
>>> >>>>>>>>> >>>
>>> >>>>>>>>> >>> I am with Chad on this, we should probably extend it a bit
>>> more, even if it
>>> >>>>>>>>> >>> makes us struggle a bit at least we have some workarounds
>>> as Robert suggests,
>>> >>>>>>>>> >>> and as Chad said there are still many people playing the
>>> python 3 catchup game,
>>> >>>>>>>>> >>> so worth to support those users.
>>> >>>>>>>>> >>>
>>> >>>>>>>>> >>>
>>> >>>>>>>>> >>> But maybe it is worth to evaluate the current state later
>>> in the year.
>>> >>>>>>>>> >>
>>> >>>>>>>>> >>
>>> >>>>>>>>> >> I would suggest re-evaluating this within the next 3 months
>>> again. We need to balance between user pain/contributor pain/our ability to
>>> continuously test with python 2 in a shifting environment.
>>> >>>>>>>>> >>
>>> >>>>>>>>> >>>
>>> >>>>>>>>> >>> In the
>>> >>>>>>>>> >>> meantime can someone please update our Roadmap in the
>>> website with this info and
>>> >>>>>>>>> >>> where we are with Python 3 support (it looks not up to
>>> date).
>>> >>>>>>>>> >>> https://beam.apache.org/roadmap/
>>> >>>>>>>>> >>
>>> >>>>>>>>> >>
>>> >>>>>>>>> >> I made a minor change to update that page (
>>> https://github.com/apache/beam/pull/10848). A more comprehensive update
>>> to that page and linked (
>>> https://beam.apache.org/roadmap/python-sdk/#python-3-support) would
>>> still be welcome.
>>> >>>>>>>>> >>
>>> >>>>>>>>> >>>
>>> >>>>>>>>> >>>
>>> >>>>>>>>> >>> - Ismaël
>>> >>>>>>>>> >>>
>>> >>>>>>>>> >>>
>>> >>>>>>>>> >>> On Tue, Feb 4, 2020 at 10:49 PM Robert Bradshaw <
>>> robertwb@google.com> wrote:
>>> >>>>>>>>> >>>>
>>> >>>>>>>>> >>>> On Tue, Feb 4, 2020 at 12:12 PM Chad Dombrova <
>>> chadrik@gmail.com> wrote:
>>> >>>>>>>>> >>>> >>
>>> >>>>>>>>> >>>> >> Not to mention that all the nice work for the type
>>> hints will have to be redone in the for 3.x.
>>> >>>>>>>>> >>>> >
>>> >>>>>>>>> >>>> > Note that there's a tool for automatically converting
>>> type comments to annotations: https://github.com/ilevkivskyi/com2ann
>>> >>>>>>>>> >>>> >
>>> >>>>>>>>> >>>> > So don't let that part bother you.
>>> >>>>>>>>> >>>>
>>> >>>>>>>>> >>>> +1, I wouldn't worry about what can be easily automated.
>>> >>>>>>>>> >>>>
>>> >>>>>>>>> >>>> > I'm curious what other features you'd like to be using
>>> in the Beam source that you cannot now.
>>> >>>>>>>>> >>>>
>>> >>>>>>>>> >>>> I hit things occasionally, e.g. I just ran into wanting
>>> keyword-only
>>> >>>>>>>>> >>>> arguments the other day.
>>> >>>>>>>>> >>>>
>>> >>>>>>>>> >>>> >> It seems the faster we drop support the better.
>>> >>>>>>>>> >>>> >
>>> >>>>>>>>> >>>> >
>>> >>>>>>>>> >>>> > I've already gone over my position on this, but a
>>> refresher for those who care: some of the key vendors that support my
>>> industry will not offer python3-compatible versions of their software until
>>> the 4th quarter of 2020. If Beam switches to python3-only before that
>>> point we may be forced to stop contributing features (note: I'm the guy who
>>> added the type hints :). Every month you can give us would be greatly
>>> appreciated.
>>> >>>>>>>>> >>>>
>>> >>>>>>>>> >>>> As another data point, we're still 80/20 on Py2/Py3 for
>>> downloads at
>>> >>>>>>>>> >>>> PyPi [1] (which I've heard should be taken with a grain
>>> of salt, but
>>> >>>>>>>>> >>>> likely isn't totally off). IMHO that ratio needs to be
>>> way higher for
>>> >>>>>>>>> >>>> Python 3 to consider dropping Python 2. It's pretty
>>> noisy, but say it
>>> >>>>>>>>> >>>> doubles every 3 months that would put us at least
>>> mid-year before we
>>> >>>>>>>>> >>>> hit a cross-over point. On the other hand Q4 2020 is
>>> probably a
>>> >>>>>>>>> >>>> stretch.
>>> >>>>>>>>> >>>>
>>> >>>>>>>>> >>>> We could consider whether it needs to be an
>>> all-or-nothing thing as
>>> >>>>>>>>> >>>> well. E.g. perhaps some features could be Python 3 only
>>> sooner than
>>> >>>>>>>>> >>>> the whole codebase. (This would have to be well
>>> justified.) Another
>>> >>>>>>>>> >>>> mitigation is that it is possible to mix Python 2 and
>>> Python 3 in the
>>> >>>>>>>>> >>>> same pipeline with portability, so if there's a library
>>> that you need
>>> >>>>>>>>> >>>> for one DoFn it doesn't mean you have to hold back your
>>> whole
>>> >>>>>>>>> >>>> pipeline.
>>> >>>>>>>>> >>>>
>>> >>>>>>>>> >>>> - Robert
>>> >>>>>>>>> >>>>
>>> >>>>>>>>> >>>> [1] https://pypistats.org/packages/apache-beam , and
>>> that 20% may just
>>> >>>>>>>>> >>>> be a spike.
>>>
>>
Re: Python2.7 Beam End-of-Life Date
Posted by Ahmet Altay <al...@google.com>.
On Tue, Jun 16, 2020 at 11:38 AM Valentyn Tymofieiev <va...@google.com>
wrote:
> I have reached out to user@ for feedback on Python 3 migration[1].
>
Maybe also ask on slack? There are quite a bit of users there as well.
>
> Could somebody from PMC please help with Twitter poll?
>
I can try to do this. What question would you like to ask? I do not know
much about twitter polls but I assume they have character limits similar to
regular tweets.
>
> Technically, we can proceed with the change between 2.23.0 and 2.24.0, so
> that's after 2.23.0 is cut and we give sufficient time for users to
> respond.
>
> [1]
> https://lists.apache.org/thread.html/r0de71d98d98b213dd1d0c45c1f5642135116f25def5637a5f41c8d29%40%3Cuser.beam.apache.org%3E
> On Tue, Jun 16, 2020 at 9:22 AM Ismaël Mejía <ie...@gmail.com> wrote:
>
>> Yes we need to poll this outside as Robert proposed.
>>
>> The proposed last support version corresponds with the next release that
>> will be
>> cut in two weeks. Sounds a bit short if we count the time to poll people
>> on this
>> subject but still could be.
>
> Technically, we can proceed with the change between 2.23.0 and 2.24.0, so
> that's after 2.23.0 is cut and we give sufficient time for users to
> respond.
>
>
>> I remember Chad mentioned in this thread the impossibility to get support
>> for
>> python 2 in his industry until the end of the year, Maybe things have
>> improved.
>> Have they?
>>
>>
>>
>>
>> On Tue, Jun 16, 2020 at 6:10 PM Robert Bradshaw <ro...@google.com>
>> wrote:
>> >
>> > I like that option as a concrete proposal. I think we should circulate
>> it more widely (the users list, twitter poll, at least a new thread...),
>> maybe phrasing it as "is there any reason you couldn't migrate to Python 3
>> (or stick with an older version of Beam) after 2.23 (due to be cut here in
>> a couple of weeks)?" If there is strong concern/pushback, we could consider
>> holding on for one more release.
>> >
>> > On Tue, Jun 16, 2020 at 8:54 AM David Cavazos <dc...@google.com>
>> wrote:
>> >>
>> >> +1
>> >>
>> >> On Mon, Jun 15, 2020 at 6:52 PM Udi Meiri <eh...@google.com> wrote:
>> >>>
>> >>> +1
>> >>>
>> >>> On Mon, Jun 15, 2020 at 4:27 PM Ahmet Altay <al...@google.com> wrote:
>> >>>>
>> >>>> As a concrete proposal, could we commit to removing python 2 support
>> by 2.24? In other words, mark the next release 2.23 as the last python 2
>> compatible Beam version.
>> >>>>
>> >>>> On Mon, Jun 15, 2020 at 2:09 PM Valentyn Tymofieiev <
>> valentyn@google.com> wrote:
>> >>>>>
>> >>>>> Another input here:
>> >>>>>
>> >>>>> If you opened a Python PR in the last few days, you probably
>> noticed that our test suites were broken by a transitive dependency of Beam
>> that dropped python 2 support, but did not declare python_requires>=3 in
>> its setup.py [1]. This temporarily broke a subset of Beam Py2 users (who
>> did not explicitly pin the 'rsa' dependency), and still affects Beam
>> development[2].
>> >>>>>
>> >>>>> This is the second time[3] Beam is affected with an issue of this
>> kind, so support of Python 2 starts to slow down our development, and add
>> toil for maintainers of packages we depend on (both directly and
>> transitively).
>> >>>>>
>> >>>>> [1] https://github.com/sybrenstuvel/python-rsa/issues/152
>> >>>>> [2]
>> https://lists.apache.org/thread.html/r9993b40b0c1cb8682ce56013165d4b80fdde0ee469a73bcb9466ddfb%40%3Cdev.beam.apache.org%3E
>> >>>>> [3] https://github.com/hamcrest/PyHamcrest/issues/131
>> >>>>>
>> >>>>> On Tue, Jun 9, 2020 at 4:06 PM Ahmet Altay <al...@google.com>
>> wrote:
>> >>>>>>
>> >>>>>> Thank you for re-opening this Valentyn. I am in favor of EOLing
>> py2 support sooner than later. The reality is that we will not be
>> effectively supporting beam python 2 for a long time while the ecosystem
>> already EOLed python 2. That said, a significant chunk (but no longer a
>> majority) of our users are still using python 2. Upgrades are painful, it
>> might be especially painful nowadays. It would be good to hear counter view
>> points, user voices related to this.
>> >>>>>>
>> >>>>>> On Thu, Jun 4, 2020 at 4:53 PM Valentyn Tymofieiev <
>> valentyn@google.com> wrote:
>> >>>>>>>
>> >>>>>>> Back at the end of February we decided to revisit this
>> conversation in 3 months. Do folks on this thread have any new input or
>> perspective regarding us balancing "user pain/contributor pain/our ability
>> to continuously test with python 2 in a shifting environment"?
>> >>>>>>>
>> >>>>>>> Some new information on my end is that we have been seeing steady
>> adoption of Python 3 among Beam Python users in Dataflow, particularly
>> strong adoption among streaming users, and Dataflow is sunsetting Python 2
>> support for all released Beam SDKs later this year [1]. We will have to
>> remove Python 2 Beam test suites that use Dataflow when Dataflow runner
>> disables Py2 support if this happens before Beam Py2 EOL (when we have to
>> remove all Py2 suites), including performance tests that still use Dataflow
>> on Python 3.
>> >>>>>>>
>> >>>>>>> I am curious how much motivation there is in the community at
>> this moment to continue Py2 support in Beam, whether any previous Py3
>> migration blockers were resolved or any new blockers discovered among Beam
>> users.
>> >>>>>>>
>> >>>>>>> [1] https://cloud.google.com/python/docs/python2-sunset/#dataflow
>> >>>>>>>
>> >>>>>>> On Fri, May 8, 2020 at 3:52 PM Valentyn Tymofieiev <
>> valentyn@google.com> wrote:
>> >>>>>>>>
>> >>>>>>>> That's good news! Thanks for sharing.
>> >>>>>>>>
>> >>>>>>>> Another datapoint, here are a few of Beam's dependencies that no
>> longer release new py2 artifacts (I looked at REQUIRED_PACKAGES + aws,
>> gcp, and interactive extras):
>> >>>>>>>>
>> >>>>>>>> hdfs
>> >>>>>>>> numpy
>> >>>>>>>> pyarrow
>> >>>>>>>> ipython
>> >>>>>>>>
>> >>>>>>>> There are more if we include transitive dependencies and
>> test-only packages. I also remember encountering one issue last month that
>> was broken only on Py2, which we had to go back and fix.
>> >>>>>>>>
>> >>>>>>>> If others have noticed frictions related to ongoing Py2 support
>> or have updates on previously mentioned Py3 migration blockers, feel free
>> to post them.
>> >>>>>>>>
>> >>>>>>>> On Fri, May 8, 2020 at 9:19 AM Robert Bradshaw <
>> robertwb@google.com> wrote:
>> >>>>>>>>>
>> >>>>>>>>> It hasn't been 3 months yet, but I wanted to call out a
>> milestone that
>> >>>>>>>>> Python 3 downloads crossed the 50% threshold on pypi, if just
>> briefly.
>> >>>>>>>>>
>> >>>>>>>>> On Thu, Feb 13, 2020 at 12:40 AM Ismaël Mejía <
>> iemejia@gmail.com> wrote:
>> >>>>>>>>> >
>> >>>>>>>>> > > I would suggest re-evaluating this within the next 3 months
>> again. We need to balance between user pain/contributor pain/our ability to
>> continuously test with python 2 in a shifting environment.
>> >>>>>>>>> >
>> >>>>>>>>> > Good idea for the in 3 months evaluation, at that point also
>> distributions will probably be phasing out python2 by default which
>> definitely help in this direction.
>> >>>>>>>>> > Thanks for updating the roadmap Ahmet
>> >>>>>>>>> >
>> >>>>>>>>> >
>> >>>>>>>>> > On Thu, Feb 13, 2020 at 2:49 AM Ahmet Altay <al...@google.com>
>> wrote:
>> >>>>>>>>> >>
>> >>>>>>>>> >>
>> >>>>>>>>> >>
>> >>>>>>>>> >> On Wed, Feb 12, 2020 at 1:29 AM Ismaël Mejía <
>> iemejia@gmail.com> wrote:
>> >>>>>>>>> >>>
>> >>>>>>>>> >>> I am with Chad on this, we should probably extend it a bit
>> more, even if it
>> >>>>>>>>> >>> makes us struggle a bit at least we have some workarounds
>> as Robert suggests,
>> >>>>>>>>> >>> and as Chad said there are still many people playing the
>> python 3 catchup game,
>> >>>>>>>>> >>> so worth to support those users.
>> >>>>>>>>> >>>
>> >>>>>>>>> >>>
>> >>>>>>>>> >>> But maybe it is worth to evaluate the current state later
>> in the year.
>> >>>>>>>>> >>
>> >>>>>>>>> >>
>> >>>>>>>>> >> I would suggest re-evaluating this within the next 3 months
>> again. We need to balance between user pain/contributor pain/our ability to
>> continuously test with python 2 in a shifting environment.
>> >>>>>>>>> >>
>> >>>>>>>>> >>>
>> >>>>>>>>> >>> In the
>> >>>>>>>>> >>> meantime can someone please update our Roadmap in the
>> website with this info and
>> >>>>>>>>> >>> where we are with Python 3 support (it looks not up to
>> date).
>> >>>>>>>>> >>> https://beam.apache.org/roadmap/
>> >>>>>>>>> >>
>> >>>>>>>>> >>
>> >>>>>>>>> >> I made a minor change to update that page (
>> https://github.com/apache/beam/pull/10848). A more comprehensive update
>> to that page and linked (
>> https://beam.apache.org/roadmap/python-sdk/#python-3-support) would
>> still be welcome.
>> >>>>>>>>> >>
>> >>>>>>>>> >>>
>> >>>>>>>>> >>>
>> >>>>>>>>> >>> - Ismaël
>> >>>>>>>>> >>>
>> >>>>>>>>> >>>
>> >>>>>>>>> >>> On Tue, Feb 4, 2020 at 10:49 PM Robert Bradshaw <
>> robertwb@google.com> wrote:
>> >>>>>>>>> >>>>
>> >>>>>>>>> >>>> On Tue, Feb 4, 2020 at 12:12 PM Chad Dombrova <
>> chadrik@gmail.com> wrote:
>> >>>>>>>>> >>>> >>
>> >>>>>>>>> >>>> >> Not to mention that all the nice work for the type
>> hints will have to be redone in the for 3.x.
>> >>>>>>>>> >>>> >
>> >>>>>>>>> >>>> > Note that there's a tool for automatically converting
>> type comments to annotations: https://github.com/ilevkivskyi/com2ann
>> >>>>>>>>> >>>> >
>> >>>>>>>>> >>>> > So don't let that part bother you.
>> >>>>>>>>> >>>>
>> >>>>>>>>> >>>> +1, I wouldn't worry about what can be easily automated.
>> >>>>>>>>> >>>>
>> >>>>>>>>> >>>> > I'm curious what other features you'd like to be using
>> in the Beam source that you cannot now.
>> >>>>>>>>> >>>>
>> >>>>>>>>> >>>> I hit things occasionally, e.g. I just ran into wanting
>> keyword-only
>> >>>>>>>>> >>>> arguments the other day.
>> >>>>>>>>> >>>>
>> >>>>>>>>> >>>> >> It seems the faster we drop support the better.
>> >>>>>>>>> >>>> >
>> >>>>>>>>> >>>> >
>> >>>>>>>>> >>>> > I've already gone over my position on this, but a
>> refresher for those who care: some of the key vendors that support my
>> industry will not offer python3-compatible versions of their software until
>> the 4th quarter of 2020. If Beam switches to python3-only before that
>> point we may be forced to stop contributing features (note: I'm the guy who
>> added the type hints :). Every month you can give us would be greatly
>> appreciated.
>> >>>>>>>>> >>>>
>> >>>>>>>>> >>>> As another data point, we're still 80/20 on Py2/Py3 for
>> downloads at
>> >>>>>>>>> >>>> PyPi [1] (which I've heard should be taken with a grain of
>> salt, but
>> >>>>>>>>> >>>> likely isn't totally off). IMHO that ratio needs to be way
>> higher for
>> >>>>>>>>> >>>> Python 3 to consider dropping Python 2. It's pretty noisy,
>> but say it
>> >>>>>>>>> >>>> doubles every 3 months that would put us at least mid-year
>> before we
>> >>>>>>>>> >>>> hit a cross-over point. On the other hand Q4 2020 is
>> probably a
>> >>>>>>>>> >>>> stretch.
>> >>>>>>>>> >>>>
>> >>>>>>>>> >>>> We could consider whether it needs to be an all-or-nothing
>> thing as
>> >>>>>>>>> >>>> well. E.g. perhaps some features could be Python 3 only
>> sooner than
>> >>>>>>>>> >>>> the whole codebase. (This would have to be well
>> justified.) Another
>> >>>>>>>>> >>>> mitigation is that it is possible to mix Python 2 and
>> Python 3 in the
>> >>>>>>>>> >>>> same pipeline with portability, so if there's a library
>> that you need
>> >>>>>>>>> >>>> for one DoFn it doesn't mean you have to hold back your
>> whole
>> >>>>>>>>> >>>> pipeline.
>> >>>>>>>>> >>>>
>> >>>>>>>>> >>>> - Robert
>> >>>>>>>>> >>>>
>> >>>>>>>>> >>>> [1] https://pypistats.org/packages/apache-beam , and that
>> 20% may just
>> >>>>>>>>> >>>> be a spike.
>>
>
Re: Python2.7 Beam End-of-Life Date
Posted by Valentyn Tymofieiev <va...@google.com>.
I have reached out to user@ for feedback on Python 3 migration[1].
Could somebody from PMC please help with Twitter poll?
Technically, we can proceed with the change between 2.23.0 and 2.24.0, so
that's after 2.23.0 is cut and we give sufficient time for users to
respond.
[1]
https://lists.apache.org/thread.html/r0de71d98d98b213dd1d0c45c1f5642135116f25def5637a5f41c8d29%40%3Cuser.beam.apache.org%3E
On Tue, Jun 16, 2020 at 9:22 AM Ismaël Mejía <ie...@gmail.com> wrote:
> Yes we need to poll this outside as Robert proposed.
>
> The proposed last support version corresponds with the next release that
> will be
> cut in two weeks. Sounds a bit short if we count the time to poll people
> on this
> subject but still could be.
Technically, we can proceed with the change between 2.23.0 and 2.24.0, so
that's after 2.23.0 is cut and we give sufficient time for users to
respond.
> I remember Chad mentioned in this thread the impossibility to get support
> for
> python 2 in his industry until the end of the year, Maybe things have
> improved.
> Have they?
>
>
>
>
> On Tue, Jun 16, 2020 at 6:10 PM Robert Bradshaw <ro...@google.com>
> wrote:
> >
> > I like that option as a concrete proposal. I think we should circulate
> it more widely (the users list, twitter poll, at least a new thread...),
> maybe phrasing it as "is there any reason you couldn't migrate to Python 3
> (or stick with an older version of Beam) after 2.23 (due to be cut here in
> a couple of weeks)?" If there is strong concern/pushback, we could consider
> holding on for one more release.
> >
> > On Tue, Jun 16, 2020 at 8:54 AM David Cavazos <dc...@google.com>
> wrote:
> >>
> >> +1
> >>
> >> On Mon, Jun 15, 2020 at 6:52 PM Udi Meiri <eh...@google.com> wrote:
> >>>
> >>> +1
> >>>
> >>> On Mon, Jun 15, 2020 at 4:27 PM Ahmet Altay <al...@google.com> wrote:
> >>>>
> >>>> As a concrete proposal, could we commit to removing python 2 support
> by 2.24? In other words, mark the next release 2.23 as the last python 2
> compatible Beam version.
> >>>>
> >>>> On Mon, Jun 15, 2020 at 2:09 PM Valentyn Tymofieiev <
> valentyn@google.com> wrote:
> >>>>>
> >>>>> Another input here:
> >>>>>
> >>>>> If you opened a Python PR in the last few days, you probably noticed
> that our test suites were broken by a transitive dependency of Beam that
> dropped python 2 support, but did not declare python_requires>=3 in its
> setup.py [1]. This temporarily broke a subset of Beam Py2 users (who did
> not explicitly pin the 'rsa' dependency), and still affects Beam
> development[2].
> >>>>>
> >>>>> This is the second time[3] Beam is affected with an issue of this
> kind, so support of Python 2 starts to slow down our development, and add
> toil for maintainers of packages we depend on (both directly and
> transitively).
> >>>>>
> >>>>> [1] https://github.com/sybrenstuvel/python-rsa/issues/152
> >>>>> [2]
> https://lists.apache.org/thread.html/r9993b40b0c1cb8682ce56013165d4b80fdde0ee469a73bcb9466ddfb%40%3Cdev.beam.apache.org%3E
> >>>>> [3] https://github.com/hamcrest/PyHamcrest/issues/131
> >>>>>
> >>>>> On Tue, Jun 9, 2020 at 4:06 PM Ahmet Altay <al...@google.com> wrote:
> >>>>>>
> >>>>>> Thank you for re-opening this Valentyn. I am in favor of EOLing py2
> support sooner than later. The reality is that we will not be effectively
> supporting beam python 2 for a long time while the ecosystem already EOLed
> python 2. That said, a significant chunk (but no longer a majority) of our
> users are still using python 2. Upgrades are painful, it might be
> especially painful nowadays. It would be good to hear counter view points,
> user voices related to this.
> >>>>>>
> >>>>>> On Thu, Jun 4, 2020 at 4:53 PM Valentyn Tymofieiev <
> valentyn@google.com> wrote:
> >>>>>>>
> >>>>>>> Back at the end of February we decided to revisit this
> conversation in 3 months. Do folks on this thread have any new input or
> perspective regarding us balancing "user pain/contributor pain/our ability
> to continuously test with python 2 in a shifting environment"?
> >>>>>>>
> >>>>>>> Some new information on my end is that we have been seeing steady
> adoption of Python 3 among Beam Python users in Dataflow, particularly
> strong adoption among streaming users, and Dataflow is sunsetting Python 2
> support for all released Beam SDKs later this year [1]. We will have to
> remove Python 2 Beam test suites that use Dataflow when Dataflow runner
> disables Py2 support if this happens before Beam Py2 EOL (when we have to
> remove all Py2 suites), including performance tests that still use Dataflow
> on Python 3.
> >>>>>>>
> >>>>>>> I am curious how much motivation there is in the community at this
> moment to continue Py2 support in Beam, whether any previous Py3 migration
> blockers were resolved or any new blockers discovered among Beam users.
> >>>>>>>
> >>>>>>> [1] https://cloud.google.com/python/docs/python2-sunset/#dataflow
> >>>>>>>
> >>>>>>> On Fri, May 8, 2020 at 3:52 PM Valentyn Tymofieiev <
> valentyn@google.com> wrote:
> >>>>>>>>
> >>>>>>>> That's good news! Thanks for sharing.
> >>>>>>>>
> >>>>>>>> Another datapoint, here are a few of Beam's dependencies that no
> longer release new py2 artifacts (I looked at REQUIRED_PACKAGES + aws,
> gcp, and interactive extras):
> >>>>>>>>
> >>>>>>>> hdfs
> >>>>>>>> numpy
> >>>>>>>> pyarrow
> >>>>>>>> ipython
> >>>>>>>>
> >>>>>>>> There are more if we include transitive dependencies and
> test-only packages. I also remember encountering one issue last month that
> was broken only on Py2, which we had to go back and fix.
> >>>>>>>>
> >>>>>>>> If others have noticed frictions related to ongoing Py2 support
> or have updates on previously mentioned Py3 migration blockers, feel free
> to post them.
> >>>>>>>>
> >>>>>>>> On Fri, May 8, 2020 at 9:19 AM Robert Bradshaw <
> robertwb@google.com> wrote:
> >>>>>>>>>
> >>>>>>>>> It hasn't been 3 months yet, but I wanted to call out a
> milestone that
> >>>>>>>>> Python 3 downloads crossed the 50% threshold on pypi, if just
> briefly.
> >>>>>>>>>
> >>>>>>>>> On Thu, Feb 13, 2020 at 12:40 AM Ismaël Mejía <ie...@gmail.com>
> wrote:
> >>>>>>>>> >
> >>>>>>>>> > > I would suggest re-evaluating this within the next 3 months
> again. We need to balance between user pain/contributor pain/our ability to
> continuously test with python 2 in a shifting environment.
> >>>>>>>>> >
> >>>>>>>>> > Good idea for the in 3 months evaluation, at that point also
> distributions will probably be phasing out python2 by default which
> definitely help in this direction.
> >>>>>>>>> > Thanks for updating the roadmap Ahmet
> >>>>>>>>> >
> >>>>>>>>> >
> >>>>>>>>> > On Thu, Feb 13, 2020 at 2:49 AM Ahmet Altay <al...@google.com>
> wrote:
> >>>>>>>>> >>
> >>>>>>>>> >>
> >>>>>>>>> >>
> >>>>>>>>> >> On Wed, Feb 12, 2020 at 1:29 AM Ismaël Mejía <
> iemejia@gmail.com> wrote:
> >>>>>>>>> >>>
> >>>>>>>>> >>> I am with Chad on this, we should probably extend it a bit
> more, even if it
> >>>>>>>>> >>> makes us struggle a bit at least we have some workarounds as
> Robert suggests,
> >>>>>>>>> >>> and as Chad said there are still many people playing the
> python 3 catchup game,
> >>>>>>>>> >>> so worth to support those users.
> >>>>>>>>> >>>
> >>>>>>>>> >>>
> >>>>>>>>> >>> But maybe it is worth to evaluate the current state later in
> the year.
> >>>>>>>>> >>
> >>>>>>>>> >>
> >>>>>>>>> >> I would suggest re-evaluating this within the next 3 months
> again. We need to balance between user pain/contributor pain/our ability to
> continuously test with python 2 in a shifting environment.
> >>>>>>>>> >>
> >>>>>>>>> >>>
> >>>>>>>>> >>> In the
> >>>>>>>>> >>> meantime can someone please update our Roadmap in the
> website with this info and
> >>>>>>>>> >>> where we are with Python 3 support (it looks not up to date).
> >>>>>>>>> >>> https://beam.apache.org/roadmap/
> >>>>>>>>> >>
> >>>>>>>>> >>
> >>>>>>>>> >> I made a minor change to update that page (
> https://github.com/apache/beam/pull/10848). A more comprehensive update
> to that page and linked (
> https://beam.apache.org/roadmap/python-sdk/#python-3-support) would still
> be welcome.
> >>>>>>>>> >>
> >>>>>>>>> >>>
> >>>>>>>>> >>>
> >>>>>>>>> >>> - Ismaël
> >>>>>>>>> >>>
> >>>>>>>>> >>>
> >>>>>>>>> >>> On Tue, Feb 4, 2020 at 10:49 PM Robert Bradshaw <
> robertwb@google.com> wrote:
> >>>>>>>>> >>>>
> >>>>>>>>> >>>> On Tue, Feb 4, 2020 at 12:12 PM Chad Dombrova <
> chadrik@gmail.com> wrote:
> >>>>>>>>> >>>> >>
> >>>>>>>>> >>>> >> Not to mention that all the nice work for the type
> hints will have to be redone in the for 3.x.
> >>>>>>>>> >>>> >
> >>>>>>>>> >>>> > Note that there's a tool for automatically converting
> type comments to annotations: https://github.com/ilevkivskyi/com2ann
> >>>>>>>>> >>>> >
> >>>>>>>>> >>>> > So don't let that part bother you.
> >>>>>>>>> >>>>
> >>>>>>>>> >>>> +1, I wouldn't worry about what can be easily automated.
> >>>>>>>>> >>>>
> >>>>>>>>> >>>> > I'm curious what other features you'd like to be using in
> the Beam source that you cannot now.
> >>>>>>>>> >>>>
> >>>>>>>>> >>>> I hit things occasionally, e.g. I just ran into wanting
> keyword-only
> >>>>>>>>> >>>> arguments the other day.
> >>>>>>>>> >>>>
> >>>>>>>>> >>>> >> It seems the faster we drop support the better.
> >>>>>>>>> >>>> >
> >>>>>>>>> >>>> >
> >>>>>>>>> >>>> > I've already gone over my position on this, but a
> refresher for those who care: some of the key vendors that support my
> industry will not offer python3-compatible versions of their software until
> the 4th quarter of 2020. If Beam switches to python3-only before that
> point we may be forced to stop contributing features (note: I'm the guy who
> added the type hints :). Every month you can give us would be greatly
> appreciated.
> >>>>>>>>> >>>>
> >>>>>>>>> >>>> As another data point, we're still 80/20 on Py2/Py3 for
> downloads at
> >>>>>>>>> >>>> PyPi [1] (which I've heard should be taken with a grain of
> salt, but
> >>>>>>>>> >>>> likely isn't totally off). IMHO that ratio needs to be way
> higher for
> >>>>>>>>> >>>> Python 3 to consider dropping Python 2. It's pretty noisy,
> but say it
> >>>>>>>>> >>>> doubles every 3 months that would put us at least mid-year
> before we
> >>>>>>>>> >>>> hit a cross-over point. On the other hand Q4 2020 is
> probably a
> >>>>>>>>> >>>> stretch.
> >>>>>>>>> >>>>
> >>>>>>>>> >>>> We could consider whether it needs to be an all-or-nothing
> thing as
> >>>>>>>>> >>>> well. E.g. perhaps some features could be Python 3 only
> sooner than
> >>>>>>>>> >>>> the whole codebase. (This would have to be well justified.)
> Another
> >>>>>>>>> >>>> mitigation is that it is possible to mix Python 2 and
> Python 3 in the
> >>>>>>>>> >>>> same pipeline with portability, so if there's a library
> that you need
> >>>>>>>>> >>>> for one DoFn it doesn't mean you have to hold back your
> whole
> >>>>>>>>> >>>> pipeline.
> >>>>>>>>> >>>>
> >>>>>>>>> >>>> - Robert
> >>>>>>>>> >>>>
> >>>>>>>>> >>>> [1] https://pypistats.org/packages/apache-beam , and that
> 20% may just
> >>>>>>>>> >>>> be a spike.
>
Re: Python2.7 Beam End-of-Life Date
Posted by Ismaël Mejía <ie...@gmail.com>.
Yes we need to poll this outside as Robert proposed.
The proposed last support version corresponds with the next release that will be
cut in two weeks. Sounds a bit short if we count the time to poll people on this
subject but still could be.
I remember Chad mentioned in this thread the impossibility to get support for
python 2 in his industry until the end of the year, Maybe things have improved.
Have they?
On Tue, Jun 16, 2020 at 6:10 PM Robert Bradshaw <ro...@google.com> wrote:
>
> I like that option as a concrete proposal. I think we should circulate it more widely (the users list, twitter poll, at least a new thread...), maybe phrasing it as "is there any reason you couldn't migrate to Python 3 (or stick with an older version of Beam) after 2.23 (due to be cut here in a couple of weeks)?" If there is strong concern/pushback, we could consider holding on for one more release.
>
> On Tue, Jun 16, 2020 at 8:54 AM David Cavazos <dc...@google.com> wrote:
>>
>> +1
>>
>> On Mon, Jun 15, 2020 at 6:52 PM Udi Meiri <eh...@google.com> wrote:
>>>
>>> +1
>>>
>>> On Mon, Jun 15, 2020 at 4:27 PM Ahmet Altay <al...@google.com> wrote:
>>>>
>>>> As a concrete proposal, could we commit to removing python 2 support by 2.24? In other words, mark the next release 2.23 as the last python 2 compatible Beam version.
>>>>
>>>> On Mon, Jun 15, 2020 at 2:09 PM Valentyn Tymofieiev <va...@google.com> wrote:
>>>>>
>>>>> Another input here:
>>>>>
>>>>> If you opened a Python PR in the last few days, you probably noticed that our test suites were broken by a transitive dependency of Beam that dropped python 2 support, but did not declare python_requires>=3 in its setup.py [1]. This temporarily broke a subset of Beam Py2 users (who did not explicitly pin the 'rsa' dependency), and still affects Beam development[2].
>>>>>
>>>>> This is the second time[3] Beam is affected with an issue of this kind, so support of Python 2 starts to slow down our development, and add toil for maintainers of packages we depend on (both directly and transitively).
>>>>>
>>>>> [1] https://github.com/sybrenstuvel/python-rsa/issues/152
>>>>> [2] https://lists.apache.org/thread.html/r9993b40b0c1cb8682ce56013165d4b80fdde0ee469a73bcb9466ddfb%40%3Cdev.beam.apache.org%3E
>>>>> [3] https://github.com/hamcrest/PyHamcrest/issues/131
>>>>>
>>>>> On Tue, Jun 9, 2020 at 4:06 PM Ahmet Altay <al...@google.com> wrote:
>>>>>>
>>>>>> Thank you for re-opening this Valentyn. I am in favor of EOLing py2 support sooner than later. The reality is that we will not be effectively supporting beam python 2 for a long time while the ecosystem already EOLed python 2. That said, a significant chunk (but no longer a majority) of our users are still using python 2. Upgrades are painful, it might be especially painful nowadays. It would be good to hear counter view points, user voices related to this.
>>>>>>
>>>>>> On Thu, Jun 4, 2020 at 4:53 PM Valentyn Tymofieiev <va...@google.com> wrote:
>>>>>>>
>>>>>>> Back at the end of February we decided to revisit this conversation in 3 months. Do folks on this thread have any new input or perspective regarding us balancing "user pain/contributor pain/our ability to continuously test with python 2 in a shifting environment"?
>>>>>>>
>>>>>>> Some new information on my end is that we have been seeing steady adoption of Python 3 among Beam Python users in Dataflow, particularly strong adoption among streaming users, and Dataflow is sunsetting Python 2 support for all released Beam SDKs later this year [1]. We will have to remove Python 2 Beam test suites that use Dataflow when Dataflow runner disables Py2 support if this happens before Beam Py2 EOL (when we have to remove all Py2 suites), including performance tests that still use Dataflow on Python 3.
>>>>>>>
>>>>>>> I am curious how much motivation there is in the community at this moment to continue Py2 support in Beam, whether any previous Py3 migration blockers were resolved or any new blockers discovered among Beam users.
>>>>>>>
>>>>>>> [1] https://cloud.google.com/python/docs/python2-sunset/#dataflow
>>>>>>>
>>>>>>> On Fri, May 8, 2020 at 3:52 PM Valentyn Tymofieiev <va...@google.com> wrote:
>>>>>>>>
>>>>>>>> That's good news! Thanks for sharing.
>>>>>>>>
>>>>>>>> Another datapoint, here are a few of Beam's dependencies that no longer release new py2 artifacts (I looked at REQUIRED_PACKAGES + aws, gcp, and interactive extras):
>>>>>>>>
>>>>>>>> hdfs
>>>>>>>> numpy
>>>>>>>> pyarrow
>>>>>>>> ipython
>>>>>>>>
>>>>>>>> There are more if we include transitive dependencies and test-only packages. I also remember encountering one issue last month that was broken only on Py2, which we had to go back and fix.
>>>>>>>>
>>>>>>>> If others have noticed frictions related to ongoing Py2 support or have updates on previously mentioned Py3 migration blockers, feel free to post them.
>>>>>>>>
>>>>>>>> On Fri, May 8, 2020 at 9:19 AM Robert Bradshaw <ro...@google.com> wrote:
>>>>>>>>>
>>>>>>>>> It hasn't been 3 months yet, but I wanted to call out a milestone that
>>>>>>>>> Python 3 downloads crossed the 50% threshold on pypi, if just briefly.
>>>>>>>>>
>>>>>>>>> On Thu, Feb 13, 2020 at 12:40 AM Ismaël Mejía <ie...@gmail.com> wrote:
>>>>>>>>> >
>>>>>>>>> > > I would suggest re-evaluating this within the next 3 months again. We need to balance between user pain/contributor pain/our ability to continuously test with python 2 in a shifting environment.
>>>>>>>>> >
>>>>>>>>> > Good idea for the in 3 months evaluation, at that point also distributions will probably be phasing out python2 by default which definitely help in this direction.
>>>>>>>>> > Thanks for updating the roadmap Ahmet
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > On Thu, Feb 13, 2020 at 2:49 AM Ahmet Altay <al...@google.com> wrote:
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >> On Wed, Feb 12, 2020 at 1:29 AM Ismaël Mejía <ie...@gmail.com> wrote:
>>>>>>>>> >>>
>>>>>>>>> >>> I am with Chad on this, we should probably extend it a bit more, even if it
>>>>>>>>> >>> makes us struggle a bit at least we have some workarounds as Robert suggests,
>>>>>>>>> >>> and as Chad said there are still many people playing the python 3 catchup game,
>>>>>>>>> >>> so worth to support those users.
>>>>>>>>> >>>
>>>>>>>>> >>>
>>>>>>>>> >>> But maybe it is worth to evaluate the current state later in the year.
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >> I would suggest re-evaluating this within the next 3 months again. We need to balance between user pain/contributor pain/our ability to continuously test with python 2 in a shifting environment.
>>>>>>>>> >>
>>>>>>>>> >>>
>>>>>>>>> >>> In the
>>>>>>>>> >>> meantime can someone please update our Roadmap in the website with this info and
>>>>>>>>> >>> where we are with Python 3 support (it looks not up to date).
>>>>>>>>> >>> https://beam.apache.org/roadmap/
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >> I made a minor change to update that page (https://github.com/apache/beam/pull/10848). A more comprehensive update to that page and linked (https://beam.apache.org/roadmap/python-sdk/#python-3-support) would still be welcome.
>>>>>>>>> >>
>>>>>>>>> >>>
>>>>>>>>> >>>
>>>>>>>>> >>> - Ismaël
>>>>>>>>> >>>
>>>>>>>>> >>>
>>>>>>>>> >>> On Tue, Feb 4, 2020 at 10:49 PM Robert Bradshaw <ro...@google.com> wrote:
>>>>>>>>> >>>>
>>>>>>>>> >>>> On Tue, Feb 4, 2020 at 12:12 PM Chad Dombrova <ch...@gmail.com> wrote:
>>>>>>>>> >>>> >>
>>>>>>>>> >>>> >> Not to mention that all the nice work for the type hints will have to be redone in the for 3.x.
>>>>>>>>> >>>> >
>>>>>>>>> >>>> > Note that there's a tool for automatically converting type comments to annotations: https://github.com/ilevkivskyi/com2ann
>>>>>>>>> >>>> >
>>>>>>>>> >>>> > So don't let that part bother you.
>>>>>>>>> >>>>
>>>>>>>>> >>>> +1, I wouldn't worry about what can be easily automated.
>>>>>>>>> >>>>
>>>>>>>>> >>>> > I'm curious what other features you'd like to be using in the Beam source that you cannot now.
>>>>>>>>> >>>>
>>>>>>>>> >>>> I hit things occasionally, e.g. I just ran into wanting keyword-only
>>>>>>>>> >>>> arguments the other day.
>>>>>>>>> >>>>
>>>>>>>>> >>>> >> It seems the faster we drop support the better.
>>>>>>>>> >>>> >
>>>>>>>>> >>>> >
>>>>>>>>> >>>> > I've already gone over my position on this, but a refresher for those who care: some of the key vendors that support my industry will not offer python3-compatible versions of their software until the 4th quarter of 2020. If Beam switches to python3-only before that point we may be forced to stop contributing features (note: I'm the guy who added the type hints :). Every month you can give us would be greatly appreciated.
>>>>>>>>> >>>>
>>>>>>>>> >>>> As another data point, we're still 80/20 on Py2/Py3 for downloads at
>>>>>>>>> >>>> PyPi [1] (which I've heard should be taken with a grain of salt, but
>>>>>>>>> >>>> likely isn't totally off). IMHO that ratio needs to be way higher for
>>>>>>>>> >>>> Python 3 to consider dropping Python 2. It's pretty noisy, but say it
>>>>>>>>> >>>> doubles every 3 months that would put us at least mid-year before we
>>>>>>>>> >>>> hit a cross-over point. On the other hand Q4 2020 is probably a
>>>>>>>>> >>>> stretch.
>>>>>>>>> >>>>
>>>>>>>>> >>>> We could consider whether it needs to be an all-or-nothing thing as
>>>>>>>>> >>>> well. E.g. perhaps some features could be Python 3 only sooner than
>>>>>>>>> >>>> the whole codebase. (This would have to be well justified.) Another
>>>>>>>>> >>>> mitigation is that it is possible to mix Python 2 and Python 3 in the
>>>>>>>>> >>>> same pipeline with portability, so if there's a library that you need
>>>>>>>>> >>>> for one DoFn it doesn't mean you have to hold back your whole
>>>>>>>>> >>>> pipeline.
>>>>>>>>> >>>>
>>>>>>>>> >>>> - Robert
>>>>>>>>> >>>>
>>>>>>>>> >>>> [1] https://pypistats.org/packages/apache-beam , and that 20% may just
>>>>>>>>> >>>> be a spike.
Re: Python2.7 Beam End-of-Life Date
Posted by Robert Bradshaw <ro...@google.com>.
I like that option as a concrete proposal. I think we should circulate it
more widely (the users list, twitter poll, at least a new thread...), maybe
phrasing it as "is there any reason you couldn't migrate to Python 3 (or
stick with an older version of Beam) after 2.23 (due to be cut here in a
couple of weeks)?" If there is strong concern/pushback, we could consider
holding on for one more release.
On Tue, Jun 16, 2020 at 8:54 AM David Cavazos <dc...@google.com> wrote:
> +1
>
> On Mon, Jun 15, 2020 at 6:52 PM Udi Meiri <eh...@google.com> wrote:
>
>> +1
>>
>> On Mon, Jun 15, 2020 at 4:27 PM Ahmet Altay <al...@google.com> wrote:
>>
>>> As a concrete proposal, could we commit to removing python 2 support by
>>> 2.24? In other words, mark the next release 2.23 as the last python 2
>>> compatible Beam version.
>>>
>>> On Mon, Jun 15, 2020 at 2:09 PM Valentyn Tymofieiev <va...@google.com>
>>> wrote:
>>>
>>>> Another input here:
>>>>
>>>> If you opened a Python PR in the last few days, you probably noticed
>>>> that our test suites were broken by a transitive dependency of Beam that
>>>> dropped python 2 support, but did not declare python_requires>=3 in its
>>>> setup.py [1]. This temporarily broke a subset of Beam Py2 users (who did
>>>> not explicitly pin the 'rsa' dependency), and still affects Beam
>>>> development[2].
>>>>
>>>> This is the second time[3] Beam is affected with an issue of this kind,
>>>> so support of Python 2 starts to slow down our development, and add toil
>>>> for maintainers of packages we depend on (both directly and transitively).
>>>>
>>>> [1] https://github.com/sybrenstuvel/python-rsa/issues/152
>>>> [2]
>>>> https://lists.apache.org/thread.html/r9993b40b0c1cb8682ce56013165d4b80fdde0ee469a73bcb9466ddfb%40%3Cdev.beam.apache.org%3E
>>>> [3] https://github.com/hamcrest/PyHamcrest/issues/131
>>>>
>>>> On Tue, Jun 9, 2020 at 4:06 PM Ahmet Altay <al...@google.com> wrote:
>>>>
>>>>> Thank you for re-opening this Valentyn. I am in favor of EOLing py2
>>>>> support sooner than later. The reality is that we will not be effectively
>>>>> supporting beam python 2 for a long time while the ecosystem already EOLed
>>>>> python 2. That said, a significant chunk (but no longer a majority) of our
>>>>> users are still using python 2. Upgrades are painful, it might be
>>>>> especially painful nowadays. It would be good to hear counter view points,
>>>>> user voices related to this.
>>>>>
>>>>> On Thu, Jun 4, 2020 at 4:53 PM Valentyn Tymofieiev <
>>>>> valentyn@google.com> wrote:
>>>>>
>>>>>> Back at the end of February we decided to revisit this conversation
>>>>>> in 3 months. Do folks on this thread have any new input or perspective
>>>>>> regarding us balancing "user pain/contributor pain/our ability to
>>>>>> continuously test with python 2 in a shifting environment"?
>>>>>>
>>>>>> Some new information on my end is that we have been seeing steady
>>>>>> adoption of Python 3 among Beam Python users in Dataflow, particularly
>>>>>> strong adoption among streaming users, and Dataflow is sunsetting Python 2
>>>>>> support for all released Beam SDKs later this year [1]. We will have to
>>>>>> remove Python 2 Beam test suites that use Dataflow when Dataflow runner
>>>>>> disables Py2 support if this happens before Beam Py2 EOL (when we have to
>>>>>> remove all Py2 suites), including performance tests that still use Dataflow
>>>>>> on Python 3.
>>>>>>
>>>>>> I am curious how much motivation there is in the community at this
>>>>>> moment to continue Py2 support in Beam, whether any previous Py3 migration
>>>>>> blockers were resolved or any new blockers discovered among Beam users.
>>>>>>
>>>>>> [1] https://cloud.google.com/python/docs/python2-sunset/#dataflow
>>>>>>
>>>>>> On Fri, May 8, 2020 at 3:52 PM Valentyn Tymofieiev <
>>>>>> valentyn@google.com> wrote:
>>>>>>
>>>>>>> That's good news! Thanks for sharing.
>>>>>>>
>>>>>>> Another datapoint, here are a few of Beam's dependencies that no
>>>>>>> longer release new py2 artifacts (I looked at REQUIRED_PACKAGES + aws,
>>>>>>> gcp, and interactive extras):
>>>>>>>
>>>>>>> hdfs
>>>>>>> numpy
>>>>>>> pyarrow
>>>>>>> ipython
>>>>>>>
>>>>>>> There are more if we include transitive dependencies and test-only
>>>>>>> packages. I also remember encountering one issue last month that was broken
>>>>>>> only on Py2, which we had to go back and fix.
>>>>>>>
>>>>>>> If others have noticed frictions related to ongoing Py2 support or
>>>>>>> have updates on previously mentioned Py3 migration blockers, feel free to
>>>>>>> post them.
>>>>>>>
>>>>>>> On Fri, May 8, 2020 at 9:19 AM Robert Bradshaw <ro...@google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> It hasn't been 3 months yet, but I wanted to call out a milestone
>>>>>>>> that
>>>>>>>> Python 3 downloads crossed the 50% threshold on pypi, if just
>>>>>>>> briefly.
>>>>>>>>
>>>>>>>> On Thu, Feb 13, 2020 at 12:40 AM Ismaël Mejía <ie...@gmail.com>
>>>>>>>> wrote:
>>>>>>>> >
>>>>>>>> > > I would suggest re-evaluating this within the next 3 months
>>>>>>>> again. We need to balance between user pain/contributor pain/our ability to
>>>>>>>> continuously test with python 2 in a shifting environment.
>>>>>>>> >
>>>>>>>> > Good idea for the in 3 months evaluation, at that point also
>>>>>>>> distributions will probably be phasing out python2 by default which
>>>>>>>> definitely help in this direction.
>>>>>>>> > Thanks for updating the roadmap Ahmet
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > On Thu, Feb 13, 2020 at 2:49 AM Ahmet Altay <al...@google.com>
>>>>>>>> wrote:
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >> On Wed, Feb 12, 2020 at 1:29 AM Ismaël Mejía <ie...@gmail.com>
>>>>>>>> wrote:
>>>>>>>> >>>
>>>>>>>> >>> I am with Chad on this, we should probably extend it a bit
>>>>>>>> more, even if it
>>>>>>>> >>> makes us struggle a bit at least we have some workarounds as
>>>>>>>> Robert suggests,
>>>>>>>> >>> and as Chad said there are still many people playing the python
>>>>>>>> 3 catchup game,
>>>>>>>> >>> so worth to support those users.
>>>>>>>> >>>
>>>>>>>> >>>
>>>>>>>> >>> But maybe it is worth to evaluate the current state later in
>>>>>>>> the year.
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >> I would suggest re-evaluating this within the next 3 months
>>>>>>>> again. We need to balance between user pain/contributor pain/our ability to
>>>>>>>> continuously test with python 2 in a shifting environment.
>>>>>>>> >>
>>>>>>>> >>>
>>>>>>>> >>> In the
>>>>>>>> >>> meantime can someone please update our Roadmap in the website
>>>>>>>> with this info and
>>>>>>>> >>> where we are with Python 3 support (it looks not up to date).
>>>>>>>> >>> https://beam.apache.org/roadmap/
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >> I made a minor change to update that page (
>>>>>>>> https://github.com/apache/beam/pull/10848). A more comprehensive
>>>>>>>> update to that page and linked (
>>>>>>>> https://beam.apache.org/roadmap/python-sdk/#python-3-support)
>>>>>>>> would still be welcome.
>>>>>>>> >>
>>>>>>>> >>>
>>>>>>>> >>>
>>>>>>>> >>> - Ismaël
>>>>>>>> >>>
>>>>>>>> >>>
>>>>>>>> >>> On Tue, Feb 4, 2020 at 10:49 PM Robert Bradshaw <
>>>>>>>> robertwb@google.com> wrote:
>>>>>>>> >>>>
>>>>>>>> >>>> On Tue, Feb 4, 2020 at 12:12 PM Chad Dombrova <
>>>>>>>> chadrik@gmail.com> wrote:
>>>>>>>> >>>> >>
>>>>>>>> >>>> >> Not to mention that all the nice work for the type hints
>>>>>>>> will have to be redone in the for 3.x.
>>>>>>>> >>>> >
>>>>>>>> >>>> > Note that there's a tool for automatically converting type
>>>>>>>> comments to annotations: https://github.com/ilevkivskyi/com2ann
>>>>>>>> >>>> >
>>>>>>>> >>>> > So don't let that part bother you.
>>>>>>>> >>>>
>>>>>>>> >>>> +1, I wouldn't worry about what can be easily automated.
>>>>>>>> >>>>
>>>>>>>> >>>> > I'm curious what other features you'd like to be using in
>>>>>>>> the Beam source that you cannot now.
>>>>>>>> >>>>
>>>>>>>> >>>> I hit things occasionally, e.g. I just ran into wanting
>>>>>>>> keyword-only
>>>>>>>> >>>> arguments the other day.
>>>>>>>> >>>>
>>>>>>>> >>>> >> It seems the faster we drop support the better.
>>>>>>>> >>>> >
>>>>>>>> >>>> >
>>>>>>>> >>>> > I've already gone over my position on this, but a refresher
>>>>>>>> for those who care: some of the key vendors that support my industry will
>>>>>>>> not offer python3-compatible versions of their software until the 4th
>>>>>>>> quarter of 2020. If Beam switches to python3-only before that point we may
>>>>>>>> be forced to stop contributing features (note: I'm the guy who added the
>>>>>>>> type hints :). Every month you can give us would be greatly appreciated.
>>>>>>>> >>>>
>>>>>>>> >>>> As another data point, we're still 80/20 on Py2/Py3 for
>>>>>>>> downloads at
>>>>>>>> >>>> PyPi [1] (which I've heard should be taken with a grain of
>>>>>>>> salt, but
>>>>>>>> >>>> likely isn't totally off). IMHO that ratio needs to be way
>>>>>>>> higher for
>>>>>>>> >>>> Python 3 to consider dropping Python 2. It's pretty noisy, but
>>>>>>>> say it
>>>>>>>> >>>> doubles every 3 months that would put us at least mid-year
>>>>>>>> before we
>>>>>>>> >>>> hit a cross-over point. On the other hand Q4 2020 is probably a
>>>>>>>> >>>> stretch.
>>>>>>>> >>>>
>>>>>>>> >>>> We could consider whether it needs to be an all-or-nothing
>>>>>>>> thing as
>>>>>>>> >>>> well. E.g. perhaps some features could be Python 3 only sooner
>>>>>>>> than
>>>>>>>> >>>> the whole codebase. (This would have to be well justified.)
>>>>>>>> Another
>>>>>>>> >>>> mitigation is that it is possible to mix Python 2 and Python 3
>>>>>>>> in the
>>>>>>>> >>>> same pipeline with portability, so if there's a library that
>>>>>>>> you need
>>>>>>>> >>>> for one DoFn it doesn't mean you have to hold back your whole
>>>>>>>> >>>> pipeline.
>>>>>>>> >>>>
>>>>>>>> >>>> - Robert
>>>>>>>> >>>>
>>>>>>>> >>>> [1] https://pypistats.org/packages/apache-beam , and that 20%
>>>>>>>> may just
>>>>>>>> >>>> be a spike.
>>>>>>>>
>>>>>>>
Re: Python2.7 Beam End-of-Life Date
Posted by David Cavazos <dc...@google.com>.
+1
On Mon, Jun 15, 2020 at 6:52 PM Udi Meiri <eh...@google.com> wrote:
> +1
>
> On Mon, Jun 15, 2020 at 4:27 PM Ahmet Altay <al...@google.com> wrote:
>
>> As a concrete proposal, could we commit to removing python 2 support by
>> 2.24? In other words, mark the next release 2.23 as the last python 2
>> compatible Beam version.
>>
>> On Mon, Jun 15, 2020 at 2:09 PM Valentyn Tymofieiev <va...@google.com>
>> wrote:
>>
>>> Another input here:
>>>
>>> If you opened a Python PR in the last few days, you probably noticed
>>> that our test suites were broken by a transitive dependency of Beam that
>>> dropped python 2 support, but did not declare python_requires>=3 in its
>>> setup.py [1]. This temporarily broke a subset of Beam Py2 users (who did
>>> not explicitly pin the 'rsa' dependency), and still affects Beam
>>> development[2].
>>>
>>> This is the second time[3] Beam is affected with an issue of this kind,
>>> so support of Python 2 starts to slow down our development, and add toil
>>> for maintainers of packages we depend on (both directly and transitively).
>>>
>>> [1] https://github.com/sybrenstuvel/python-rsa/issues/152
>>> [2]
>>> https://lists.apache.org/thread.html/r9993b40b0c1cb8682ce56013165d4b80fdde0ee469a73bcb9466ddfb%40%3Cdev.beam.apache.org%3E
>>> [3] https://github.com/hamcrest/PyHamcrest/issues/131
>>>
>>> On Tue, Jun 9, 2020 at 4:06 PM Ahmet Altay <al...@google.com> wrote:
>>>
>>>> Thank you for re-opening this Valentyn. I am in favor of EOLing py2
>>>> support sooner than later. The reality is that we will not be effectively
>>>> supporting beam python 2 for a long time while the ecosystem already EOLed
>>>> python 2. That said, a significant chunk (but no longer a majority) of our
>>>> users are still using python 2. Upgrades are painful, it might be
>>>> especially painful nowadays. It would be good to hear counter view points,
>>>> user voices related to this.
>>>>
>>>> On Thu, Jun 4, 2020 at 4:53 PM Valentyn Tymofieiev <va...@google.com>
>>>> wrote:
>>>>
>>>>> Back at the end of February we decided to revisit this conversation in
>>>>> 3 months. Do folks on this thread have any new input or perspective
>>>>> regarding us balancing "user pain/contributor pain/our ability to
>>>>> continuously test with python 2 in a shifting environment"?
>>>>>
>>>>> Some new information on my end is that we have been seeing steady
>>>>> adoption of Python 3 among Beam Python users in Dataflow, particularly
>>>>> strong adoption among streaming users, and Dataflow is sunsetting Python 2
>>>>> support for all released Beam SDKs later this year [1]. We will have to
>>>>> remove Python 2 Beam test suites that use Dataflow when Dataflow runner
>>>>> disables Py2 support if this happens before Beam Py2 EOL (when we have to
>>>>> remove all Py2 suites), including performance tests that still use Dataflow
>>>>> on Python 3.
>>>>>
>>>>> I am curious how much motivation there is in the community at this
>>>>> moment to continue Py2 support in Beam, whether any previous Py3 migration
>>>>> blockers were resolved or any new blockers discovered among Beam users.
>>>>>
>>>>> [1] https://cloud.google.com/python/docs/python2-sunset/#dataflow
>>>>>
>>>>> On Fri, May 8, 2020 at 3:52 PM Valentyn Tymofieiev <
>>>>> valentyn@google.com> wrote:
>>>>>
>>>>>> That's good news! Thanks for sharing.
>>>>>>
>>>>>> Another datapoint, here are a few of Beam's dependencies that no
>>>>>> longer release new py2 artifacts (I looked at REQUIRED_PACKAGES + aws,
>>>>>> gcp, and interactive extras):
>>>>>>
>>>>>> hdfs
>>>>>> numpy
>>>>>> pyarrow
>>>>>> ipython
>>>>>>
>>>>>> There are more if we include transitive dependencies and test-only
>>>>>> packages. I also remember encountering one issue last month that was broken
>>>>>> only on Py2, which we had to go back and fix.
>>>>>>
>>>>>> If others have noticed frictions related to ongoing Py2 support or
>>>>>> have updates on previously mentioned Py3 migration blockers, feel free to
>>>>>> post them.
>>>>>>
>>>>>> On Fri, May 8, 2020 at 9:19 AM Robert Bradshaw <ro...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> It hasn't been 3 months yet, but I wanted to call out a milestone
>>>>>>> that
>>>>>>> Python 3 downloads crossed the 50% threshold on pypi, if just
>>>>>>> briefly.
>>>>>>>
>>>>>>> On Thu, Feb 13, 2020 at 12:40 AM Ismaël Mejía <ie...@gmail.com>
>>>>>>> wrote:
>>>>>>> >
>>>>>>> > > I would suggest re-evaluating this within the next 3 months
>>>>>>> again. We need to balance between user pain/contributor pain/our ability to
>>>>>>> continuously test with python 2 in a shifting environment.
>>>>>>> >
>>>>>>> > Good idea for the in 3 months evaluation, at that point also
>>>>>>> distributions will probably be phasing out python2 by default which
>>>>>>> definitely help in this direction.
>>>>>>> > Thanks for updating the roadmap Ahmet
>>>>>>> >
>>>>>>> >
>>>>>>> > On Thu, Feb 13, 2020 at 2:49 AM Ahmet Altay <al...@google.com>
>>>>>>> wrote:
>>>>>>> >>
>>>>>>> >>
>>>>>>> >>
>>>>>>> >> On Wed, Feb 12, 2020 at 1:29 AM Ismaël Mejía <ie...@gmail.com>
>>>>>>> wrote:
>>>>>>> >>>
>>>>>>> >>> I am with Chad on this, we should probably extend it a bit more,
>>>>>>> even if it
>>>>>>> >>> makes us struggle a bit at least we have some workarounds as
>>>>>>> Robert suggests,
>>>>>>> >>> and as Chad said there are still many people playing the python
>>>>>>> 3 catchup game,
>>>>>>> >>> so worth to support those users.
>>>>>>> >>>
>>>>>>> >>>
>>>>>>> >>> But maybe it is worth to evaluate the current state later in the
>>>>>>> year.
>>>>>>> >>
>>>>>>> >>
>>>>>>> >> I would suggest re-evaluating this within the next 3 months
>>>>>>> again. We need to balance between user pain/contributor pain/our ability to
>>>>>>> continuously test with python 2 in a shifting environment.
>>>>>>> >>
>>>>>>> >>>
>>>>>>> >>> In the
>>>>>>> >>> meantime can someone please update our Roadmap in the website
>>>>>>> with this info and
>>>>>>> >>> where we are with Python 3 support (it looks not up to date).
>>>>>>> >>> https://beam.apache.org/roadmap/
>>>>>>> >>
>>>>>>> >>
>>>>>>> >> I made a minor change to update that page (
>>>>>>> https://github.com/apache/beam/pull/10848). A more comprehensive
>>>>>>> update to that page and linked (
>>>>>>> https://beam.apache.org/roadmap/python-sdk/#python-3-support) would
>>>>>>> still be welcome.
>>>>>>> >>
>>>>>>> >>>
>>>>>>> >>>
>>>>>>> >>> - Ismaël
>>>>>>> >>>
>>>>>>> >>>
>>>>>>> >>> On Tue, Feb 4, 2020 at 10:49 PM Robert Bradshaw <
>>>>>>> robertwb@google.com> wrote:
>>>>>>> >>>>
>>>>>>> >>>> On Tue, Feb 4, 2020 at 12:12 PM Chad Dombrova <
>>>>>>> chadrik@gmail.com> wrote:
>>>>>>> >>>> >>
>>>>>>> >>>> >> Not to mention that all the nice work for the type hints
>>>>>>> will have to be redone in the for 3.x.
>>>>>>> >>>> >
>>>>>>> >>>> > Note that there's a tool for automatically converting type
>>>>>>> comments to annotations: https://github.com/ilevkivskyi/com2ann
>>>>>>> >>>> >
>>>>>>> >>>> > So don't let that part bother you.
>>>>>>> >>>>
>>>>>>> >>>> +1, I wouldn't worry about what can be easily automated.
>>>>>>> >>>>
>>>>>>> >>>> > I'm curious what other features you'd like to be using in the
>>>>>>> Beam source that you cannot now.
>>>>>>> >>>>
>>>>>>> >>>> I hit things occasionally, e.g. I just ran into wanting
>>>>>>> keyword-only
>>>>>>> >>>> arguments the other day.
>>>>>>> >>>>
>>>>>>> >>>> >> It seems the faster we drop support the better.
>>>>>>> >>>> >
>>>>>>> >>>> >
>>>>>>> >>>> > I've already gone over my position on this, but a refresher
>>>>>>> for those who care: some of the key vendors that support my industry will
>>>>>>> not offer python3-compatible versions of their software until the 4th
>>>>>>> quarter of 2020. If Beam switches to python3-only before that point we may
>>>>>>> be forced to stop contributing features (note: I'm the guy who added the
>>>>>>> type hints :). Every month you can give us would be greatly appreciated.
>>>>>>> >>>>
>>>>>>> >>>> As another data point, we're still 80/20 on Py2/Py3 for
>>>>>>> downloads at
>>>>>>> >>>> PyPi [1] (which I've heard should be taken with a grain of
>>>>>>> salt, but
>>>>>>> >>>> likely isn't totally off). IMHO that ratio needs to be way
>>>>>>> higher for
>>>>>>> >>>> Python 3 to consider dropping Python 2. It's pretty noisy, but
>>>>>>> say it
>>>>>>> >>>> doubles every 3 months that would put us at least mid-year
>>>>>>> before we
>>>>>>> >>>> hit a cross-over point. On the other hand Q4 2020 is probably a
>>>>>>> >>>> stretch.
>>>>>>> >>>>
>>>>>>> >>>> We could consider whether it needs to be an all-or-nothing
>>>>>>> thing as
>>>>>>> >>>> well. E.g. perhaps some features could be Python 3 only sooner
>>>>>>> than
>>>>>>> >>>> the whole codebase. (This would have to be well justified.)
>>>>>>> Another
>>>>>>> >>>> mitigation is that it is possible to mix Python 2 and Python 3
>>>>>>> in the
>>>>>>> >>>> same pipeline with portability, so if there's a library that
>>>>>>> you need
>>>>>>> >>>> for one DoFn it doesn't mean you have to hold back your whole
>>>>>>> >>>> pipeline.
>>>>>>> >>>>
>>>>>>> >>>> - Robert
>>>>>>> >>>>
>>>>>>> >>>> [1] https://pypistats.org/packages/apache-beam , and that 20%
>>>>>>> may just
>>>>>>> >>>> be a spike.
>>>>>>>
>>>>>>
Re: Python2.7 Beam End-of-Life Date
Posted by Udi Meiri <eh...@google.com>.
+1
On Mon, Jun 15, 2020 at 4:27 PM Ahmet Altay <al...@google.com> wrote:
> As a concrete proposal, could we commit to removing python 2 support by
> 2.24? In other words, mark the next release 2.23 as the last python 2
> compatible Beam version.
>
> On Mon, Jun 15, 2020 at 2:09 PM Valentyn Tymofieiev <va...@google.com>
> wrote:
>
>> Another input here:
>>
>> If you opened a Python PR in the last few days, you probably noticed that
>> our test suites were broken by a transitive dependency of Beam that dropped
>> python 2 support, but did not declare python_requires>=3 in its setup.py
>> [1]. This temporarily broke a subset of Beam Py2 users (who did not
>> explicitly pin the 'rsa' dependency), and still affects Beam
>> development[2].
>>
>> This is the second time[3] Beam is affected with an issue of this kind,
>> so support of Python 2 starts to slow down our development, and add toil
>> for maintainers of packages we depend on (both directly and transitively).
>>
>> [1] https://github.com/sybrenstuvel/python-rsa/issues/152
>> [2]
>> https://lists.apache.org/thread.html/r9993b40b0c1cb8682ce56013165d4b80fdde0ee469a73bcb9466ddfb%40%3Cdev.beam.apache.org%3E
>> [3] https://github.com/hamcrest/PyHamcrest/issues/131
>>
>> On Tue, Jun 9, 2020 at 4:06 PM Ahmet Altay <al...@google.com> wrote:
>>
>>> Thank you for re-opening this Valentyn. I am in favor of EOLing py2
>>> support sooner than later. The reality is that we will not be effectively
>>> supporting beam python 2 for a long time while the ecosystem already EOLed
>>> python 2. That said, a significant chunk (but no longer a majority) of our
>>> users are still using python 2. Upgrades are painful, it might be
>>> especially painful nowadays. It would be good to hear counter view points,
>>> user voices related to this.
>>>
>>> On Thu, Jun 4, 2020 at 4:53 PM Valentyn Tymofieiev <va...@google.com>
>>> wrote:
>>>
>>>> Back at the end of February we decided to revisit this conversation in
>>>> 3 months. Do folks on this thread have any new input or perspective
>>>> regarding us balancing "user pain/contributor pain/our ability to
>>>> continuously test with python 2 in a shifting environment"?
>>>>
>>>> Some new information on my end is that we have been seeing steady
>>>> adoption of Python 3 among Beam Python users in Dataflow, particularly
>>>> strong adoption among streaming users, and Dataflow is sunsetting Python 2
>>>> support for all released Beam SDKs later this year [1]. We will have to
>>>> remove Python 2 Beam test suites that use Dataflow when Dataflow runner
>>>> disables Py2 support if this happens before Beam Py2 EOL (when we have to
>>>> remove all Py2 suites), including performance tests that still use Dataflow
>>>> on Python 3.
>>>>
>>>> I am curious how much motivation there is in the community at this
>>>> moment to continue Py2 support in Beam, whether any previous Py3 migration
>>>> blockers were resolved or any new blockers discovered among Beam users.
>>>>
>>>> [1] https://cloud.google.com/python/docs/python2-sunset/#dataflow
>>>>
>>>> On Fri, May 8, 2020 at 3:52 PM Valentyn Tymofieiev <va...@google.com>
>>>> wrote:
>>>>
>>>>> That's good news! Thanks for sharing.
>>>>>
>>>>> Another datapoint, here are a few of Beam's dependencies that no
>>>>> longer release new py2 artifacts (I looked at REQUIRED_PACKAGES + aws,
>>>>> gcp, and interactive extras):
>>>>>
>>>>> hdfs
>>>>> numpy
>>>>> pyarrow
>>>>> ipython
>>>>>
>>>>> There are more if we include transitive dependencies and test-only
>>>>> packages. I also remember encountering one issue last month that was broken
>>>>> only on Py2, which we had to go back and fix.
>>>>>
>>>>> If others have noticed frictions related to ongoing Py2 support or
>>>>> have updates on previously mentioned Py3 migration blockers, feel free to
>>>>> post them.
>>>>>
>>>>> On Fri, May 8, 2020 at 9:19 AM Robert Bradshaw <ro...@google.com>
>>>>> wrote:
>>>>>
>>>>>> It hasn't been 3 months yet, but I wanted to call out a milestone that
>>>>>> Python 3 downloads crossed the 50% threshold on pypi, if just briefly.
>>>>>>
>>>>>> On Thu, Feb 13, 2020 at 12:40 AM Ismaël Mejía <ie...@gmail.com>
>>>>>> wrote:
>>>>>> >
>>>>>> > > I would suggest re-evaluating this within the next 3 months
>>>>>> again. We need to balance between user pain/contributor pain/our ability to
>>>>>> continuously test with python 2 in a shifting environment.
>>>>>> >
>>>>>> > Good idea for the in 3 months evaluation, at that point also
>>>>>> distributions will probably be phasing out python2 by default which
>>>>>> definitely help in this direction.
>>>>>> > Thanks for updating the roadmap Ahmet
>>>>>> >
>>>>>> >
>>>>>> > On Thu, Feb 13, 2020 at 2:49 AM Ahmet Altay <al...@google.com>
>>>>>> wrote:
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> >> On Wed, Feb 12, 2020 at 1:29 AM Ismaël Mejía <ie...@gmail.com>
>>>>>> wrote:
>>>>>> >>>
>>>>>> >>> I am with Chad on this, we should probably extend it a bit more,
>>>>>> even if it
>>>>>> >>> makes us struggle a bit at least we have some workarounds as
>>>>>> Robert suggests,
>>>>>> >>> and as Chad said there are still many people playing the python 3
>>>>>> catchup game,
>>>>>> >>> so worth to support those users.
>>>>>> >>>
>>>>>> >>>
>>>>>> >>> But maybe it is worth to evaluate the current state later in the
>>>>>> year.
>>>>>> >>
>>>>>> >>
>>>>>> >> I would suggest re-evaluating this within the next 3 months again.
>>>>>> We need to balance between user pain/contributor pain/our ability to
>>>>>> continuously test with python 2 in a shifting environment.
>>>>>> >>
>>>>>> >>>
>>>>>> >>> In the
>>>>>> >>> meantime can someone please update our Roadmap in the website
>>>>>> with this info and
>>>>>> >>> where we are with Python 3 support (it looks not up to date).
>>>>>> >>> https://beam.apache.org/roadmap/
>>>>>> >>
>>>>>> >>
>>>>>> >> I made a minor change to update that page (
>>>>>> https://github.com/apache/beam/pull/10848). A more comprehensive
>>>>>> update to that page and linked (
>>>>>> https://beam.apache.org/roadmap/python-sdk/#python-3-support) would
>>>>>> still be welcome.
>>>>>> >>
>>>>>> >>>
>>>>>> >>>
>>>>>> >>> - Ismaël
>>>>>> >>>
>>>>>> >>>
>>>>>> >>> On Tue, Feb 4, 2020 at 10:49 PM Robert Bradshaw <
>>>>>> robertwb@google.com> wrote:
>>>>>> >>>>
>>>>>> >>>> On Tue, Feb 4, 2020 at 12:12 PM Chad Dombrova <
>>>>>> chadrik@gmail.com> wrote:
>>>>>> >>>> >>
>>>>>> >>>> >> Not to mention that all the nice work for the type hints
>>>>>> will have to be redone in the for 3.x.
>>>>>> >>>> >
>>>>>> >>>> > Note that there's a tool for automatically converting type
>>>>>> comments to annotations: https://github.com/ilevkivskyi/com2ann
>>>>>> >>>> >
>>>>>> >>>> > So don't let that part bother you.
>>>>>> >>>>
>>>>>> >>>> +1, I wouldn't worry about what can be easily automated.
>>>>>> >>>>
>>>>>> >>>> > I'm curious what other features you'd like to be using in the
>>>>>> Beam source that you cannot now.
>>>>>> >>>>
>>>>>> >>>> I hit things occasionally, e.g. I just ran into wanting
>>>>>> keyword-only
>>>>>> >>>> arguments the other day.
>>>>>> >>>>
>>>>>> >>>> >> It seems the faster we drop support the better.
>>>>>> >>>> >
>>>>>> >>>> >
>>>>>> >>>> > I've already gone over my position on this, but a refresher
>>>>>> for those who care: some of the key vendors that support my industry will
>>>>>> not offer python3-compatible versions of their software until the 4th
>>>>>> quarter of 2020. If Beam switches to python3-only before that point we may
>>>>>> be forced to stop contributing features (note: I'm the guy who added the
>>>>>> type hints :). Every month you can give us would be greatly appreciated.
>>>>>> >>>>
>>>>>> >>>> As another data point, we're still 80/20 on Py2/Py3 for
>>>>>> downloads at
>>>>>> >>>> PyPi [1] (which I've heard should be taken with a grain of salt,
>>>>>> but
>>>>>> >>>> likely isn't totally off). IMHO that ratio needs to be way
>>>>>> higher for
>>>>>> >>>> Python 3 to consider dropping Python 2. It's pretty noisy, but
>>>>>> say it
>>>>>> >>>> doubles every 3 months that would put us at least mid-year
>>>>>> before we
>>>>>> >>>> hit a cross-over point. On the other hand Q4 2020 is probably a
>>>>>> >>>> stretch.
>>>>>> >>>>
>>>>>> >>>> We could consider whether it needs to be an all-or-nothing thing
>>>>>> as
>>>>>> >>>> well. E.g. perhaps some features could be Python 3 only sooner
>>>>>> than
>>>>>> >>>> the whole codebase. (This would have to be well justified.)
>>>>>> Another
>>>>>> >>>> mitigation is that it is possible to mix Python 2 and Python 3
>>>>>> in the
>>>>>> >>>> same pipeline with portability, so if there's a library that you
>>>>>> need
>>>>>> >>>> for one DoFn it doesn't mean you have to hold back your whole
>>>>>> >>>> pipeline.
>>>>>> >>>>
>>>>>> >>>> - Robert
>>>>>> >>>>
>>>>>> >>>> [1] https://pypistats.org/packages/apache-beam , and that 20%
>>>>>> may just
>>>>>> >>>> be a spike.
>>>>>>
>>>>>
Re: Python2.7 Beam End-of-Life Date
Posted by Ahmet Altay <al...@google.com>.
As a concrete proposal, could we commit to removing python 2 support by
2.24? In other words, mark the next release 2.23 as the last python 2
compatible Beam version.
On Mon, Jun 15, 2020 at 2:09 PM Valentyn Tymofieiev <va...@google.com>
wrote:
> Another input here:
>
> If you opened a Python PR in the last few days, you probably noticed that
> our test suites were broken by a transitive dependency of Beam that dropped
> python 2 support, but did not declare python_requires>=3 in its setup.py
> [1]. This temporarily broke a subset of Beam Py2 users (who did not
> explicitly pin the 'rsa' dependency), and still affects Beam
> development[2].
>
> This is the second time[3] Beam is affected with an issue of this kind, so
> support of Python 2 starts to slow down our development, and add toil for
> maintainers of packages we depend on (both directly and transitively).
>
> [1] https://github.com/sybrenstuvel/python-rsa/issues/152
> [2]
> https://lists.apache.org/thread.html/r9993b40b0c1cb8682ce56013165d4b80fdde0ee469a73bcb9466ddfb%40%3Cdev.beam.apache.org%3E
> [3] https://github.com/hamcrest/PyHamcrest/issues/131
>
> On Tue, Jun 9, 2020 at 4:06 PM Ahmet Altay <al...@google.com> wrote:
>
>> Thank you for re-opening this Valentyn. I am in favor of EOLing py2
>> support sooner than later. The reality is that we will not be effectively
>> supporting beam python 2 for a long time while the ecosystem already EOLed
>> python 2. That said, a significant chunk (but no longer a majority) of our
>> users are still using python 2. Upgrades are painful, it might be
>> especially painful nowadays. It would be good to hear counter view points,
>> user voices related to this.
>>
>> On Thu, Jun 4, 2020 at 4:53 PM Valentyn Tymofieiev <va...@google.com>
>> wrote:
>>
>>> Back at the end of February we decided to revisit this conversation in 3
>>> months. Do folks on this thread have any new input or perspective regarding
>>> us balancing "user pain/contributor pain/our ability to continuously test
>>> with python 2 in a shifting environment"?
>>>
>>> Some new information on my end is that we have been seeing steady
>>> adoption of Python 3 among Beam Python users in Dataflow, particularly
>>> strong adoption among streaming users, and Dataflow is sunsetting Python 2
>>> support for all released Beam SDKs later this year [1]. We will have to
>>> remove Python 2 Beam test suites that use Dataflow when Dataflow runner
>>> disables Py2 support if this happens before Beam Py2 EOL (when we have to
>>> remove all Py2 suites), including performance tests that still use Dataflow
>>> on Python 3.
>>>
>>> I am curious how much motivation there is in the community at this
>>> moment to continue Py2 support in Beam, whether any previous Py3 migration
>>> blockers were resolved or any new blockers discovered among Beam users.
>>>
>>> [1] https://cloud.google.com/python/docs/python2-sunset/#dataflow
>>>
>>> On Fri, May 8, 2020 at 3:52 PM Valentyn Tymofieiev <va...@google.com>
>>> wrote:
>>>
>>>> That's good news! Thanks for sharing.
>>>>
>>>> Another datapoint, here are a few of Beam's dependencies that no longer
>>>> release new py2 artifacts (I looked at REQUIRED_PACKAGES + aws, gcp, and
>>>> interactive extras):
>>>>
>>>> hdfs
>>>> numpy
>>>> pyarrow
>>>> ipython
>>>>
>>>> There are more if we include transitive dependencies and test-only
>>>> packages. I also remember encountering one issue last month that was broken
>>>> only on Py2, which we had to go back and fix.
>>>>
>>>> If others have noticed frictions related to ongoing Py2 support or have
>>>> updates on previously mentioned Py3 migration blockers, feel free to post
>>>> them.
>>>>
>>>> On Fri, May 8, 2020 at 9:19 AM Robert Bradshaw <ro...@google.com>
>>>> wrote:
>>>>
>>>>> It hasn't been 3 months yet, but I wanted to call out a milestone that
>>>>> Python 3 downloads crossed the 50% threshold on pypi, if just briefly.
>>>>>
>>>>> On Thu, Feb 13, 2020 at 12:40 AM Ismaël Mejía <ie...@gmail.com>
>>>>> wrote:
>>>>> >
>>>>> > > I would suggest re-evaluating this within the next 3 months again.
>>>>> We need to balance between user pain/contributor pain/our ability to
>>>>> continuously test with python 2 in a shifting environment.
>>>>> >
>>>>> > Good idea for the in 3 months evaluation, at that point also
>>>>> distributions will probably be phasing out python2 by default which
>>>>> definitely help in this direction.
>>>>> > Thanks for updating the roadmap Ahmet
>>>>> >
>>>>> >
>>>>> > On Thu, Feb 13, 2020 at 2:49 AM Ahmet Altay <al...@google.com>
>>>>> wrote:
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> On Wed, Feb 12, 2020 at 1:29 AM Ismaël Mejía <ie...@gmail.com>
>>>>> wrote:
>>>>> >>>
>>>>> >>> I am with Chad on this, we should probably extend it a bit more,
>>>>> even if it
>>>>> >>> makes us struggle a bit at least we have some workarounds as
>>>>> Robert suggests,
>>>>> >>> and as Chad said there are still many people playing the python 3
>>>>> catchup game,
>>>>> >>> so worth to support those users.
>>>>> >>>
>>>>> >>>
>>>>> >>> But maybe it is worth to evaluate the current state later in the
>>>>> year.
>>>>> >>
>>>>> >>
>>>>> >> I would suggest re-evaluating this within the next 3 months again.
>>>>> We need to balance between user pain/contributor pain/our ability to
>>>>> continuously test with python 2 in a shifting environment.
>>>>> >>
>>>>> >>>
>>>>> >>> In the
>>>>> >>> meantime can someone please update our Roadmap in the website with
>>>>> this info and
>>>>> >>> where we are with Python 3 support (it looks not up to date).
>>>>> >>> https://beam.apache.org/roadmap/
>>>>> >>
>>>>> >>
>>>>> >> I made a minor change to update that page (
>>>>> https://github.com/apache/beam/pull/10848). A more comprehensive
>>>>> update to that page and linked (
>>>>> https://beam.apache.org/roadmap/python-sdk/#python-3-support) would
>>>>> still be welcome.
>>>>> >>
>>>>> >>>
>>>>> >>>
>>>>> >>> - Ismaël
>>>>> >>>
>>>>> >>>
>>>>> >>> On Tue, Feb 4, 2020 at 10:49 PM Robert Bradshaw <
>>>>> robertwb@google.com> wrote:
>>>>> >>>>
>>>>> >>>> On Tue, Feb 4, 2020 at 12:12 PM Chad Dombrova <ch...@gmail.com>
>>>>> wrote:
>>>>> >>>> >>
>>>>> >>>> >> Not to mention that all the nice work for the type hints will
>>>>> have to be redone in the for 3.x.
>>>>> >>>> >
>>>>> >>>> > Note that there's a tool for automatically converting type
>>>>> comments to annotations: https://github.com/ilevkivskyi/com2ann
>>>>> >>>> >
>>>>> >>>> > So don't let that part bother you.
>>>>> >>>>
>>>>> >>>> +1, I wouldn't worry about what can be easily automated.
>>>>> >>>>
>>>>> >>>> > I'm curious what other features you'd like to be using in the
>>>>> Beam source that you cannot now.
>>>>> >>>>
>>>>> >>>> I hit things occasionally, e.g. I just ran into wanting
>>>>> keyword-only
>>>>> >>>> arguments the other day.
>>>>> >>>>
>>>>> >>>> >> It seems the faster we drop support the better.
>>>>> >>>> >
>>>>> >>>> >
>>>>> >>>> > I've already gone over my position on this, but a refresher for
>>>>> those who care: some of the key vendors that support my industry will not
>>>>> offer python3-compatible versions of their software until the 4th quarter
>>>>> of 2020. If Beam switches to python3-only before that point we may be
>>>>> forced to stop contributing features (note: I'm the guy who added the type
>>>>> hints :). Every month you can give us would be greatly appreciated.
>>>>> >>>>
>>>>> >>>> As another data point, we're still 80/20 on Py2/Py3 for downloads
>>>>> at
>>>>> >>>> PyPi [1] (which I've heard should be taken with a grain of salt,
>>>>> but
>>>>> >>>> likely isn't totally off). IMHO that ratio needs to be way higher
>>>>> for
>>>>> >>>> Python 3 to consider dropping Python 2. It's pretty noisy, but
>>>>> say it
>>>>> >>>> doubles every 3 months that would put us at least mid-year before
>>>>> we
>>>>> >>>> hit a cross-over point. On the other hand Q4 2020 is probably a
>>>>> >>>> stretch.
>>>>> >>>>
>>>>> >>>> We could consider whether it needs to be an all-or-nothing thing
>>>>> as
>>>>> >>>> well. E.g. perhaps some features could be Python 3 only sooner
>>>>> than
>>>>> >>>> the whole codebase. (This would have to be well justified.)
>>>>> Another
>>>>> >>>> mitigation is that it is possible to mix Python 2 and Python 3 in
>>>>> the
>>>>> >>>> same pipeline with portability, so if there's a library that you
>>>>> need
>>>>> >>>> for one DoFn it doesn't mean you have to hold back your whole
>>>>> >>>> pipeline.
>>>>> >>>>
>>>>> >>>> - Robert
>>>>> >>>>
>>>>> >>>> [1] https://pypistats.org/packages/apache-beam , and that 20%
>>>>> may just
>>>>> >>>> be a spike.
>>>>>
>>>>
Re: Python2.7 Beam End-of-Life Date
Posted by Valentyn Tymofieiev <va...@google.com>.
Another input here:
If you opened a Python PR in the last few days, you probably noticed that
our test suites were broken by a transitive dependency of Beam that dropped
python 2 support, but did not declare python_requires>=3 in its setup.py
[1]. This temporarily broke a subset of Beam Py2 users (who did not
explicitly pin the 'rsa' dependency), and still affects Beam
development[2].
This is the second time[3] Beam is affected with an issue of this kind, so
support of Python 2 starts to slow down our development, and add toil for
maintainers of packages we depend on (both directly and transitively).
[1] https://github.com/sybrenstuvel/python-rsa/issues/152
[2]
https://lists.apache.org/thread.html/r9993b40b0c1cb8682ce56013165d4b80fdde0ee469a73bcb9466ddfb%40%3Cdev.beam.apache.org%3E
[3] https://github.com/hamcrest/PyHamcrest/issues/131
On Tue, Jun 9, 2020 at 4:06 PM Ahmet Altay <al...@google.com> wrote:
> Thank you for re-opening this Valentyn. I am in favor of EOLing py2
> support sooner than later. The reality is that we will not be effectively
> supporting beam python 2 for a long time while the ecosystem already EOLed
> python 2. That said, a significant chunk (but no longer a majority) of our
> users are still using python 2. Upgrades are painful, it might be
> especially painful nowadays. It would be good to hear counter view points,
> user voices related to this.
>
> On Thu, Jun 4, 2020 at 4:53 PM Valentyn Tymofieiev <va...@google.com>
> wrote:
>
>> Back at the end of February we decided to revisit this conversation in 3
>> months. Do folks on this thread have any new input or perspective regarding
>> us balancing "user pain/contributor pain/our ability to continuously test
>> with python 2 in a shifting environment"?
>>
>> Some new information on my end is that we have been seeing steady
>> adoption of Python 3 among Beam Python users in Dataflow, particularly
>> strong adoption among streaming users, and Dataflow is sunsetting Python 2
>> support for all released Beam SDKs later this year [1]. We will have to
>> remove Python 2 Beam test suites that use Dataflow when Dataflow runner
>> disables Py2 support if this happens before Beam Py2 EOL (when we have to
>> remove all Py2 suites), including performance tests that still use Dataflow
>> on Python 3.
>>
>> I am curious how much motivation there is in the community at this moment
>> to continue Py2 support in Beam, whether any previous Py3 migration
>> blockers were resolved or any new blockers discovered among Beam users.
>>
>> [1] https://cloud.google.com/python/docs/python2-sunset/#dataflow
>>
>> On Fri, May 8, 2020 at 3:52 PM Valentyn Tymofieiev <va...@google.com>
>> wrote:
>>
>>> That's good news! Thanks for sharing.
>>>
>>> Another datapoint, here are a few of Beam's dependencies that no longer
>>> release new py2 artifacts (I looked at REQUIRED_PACKAGES + aws, gcp, and
>>> interactive extras):
>>>
>>> hdfs
>>> numpy
>>> pyarrow
>>> ipython
>>>
>>> There are more if we include transitive dependencies and test-only
>>> packages. I also remember encountering one issue last month that was broken
>>> only on Py2, which we had to go back and fix.
>>>
>>> If others have noticed frictions related to ongoing Py2 support or have
>>> updates on previously mentioned Py3 migration blockers, feel free to post
>>> them.
>>>
>>> On Fri, May 8, 2020 at 9:19 AM Robert Bradshaw <ro...@google.com>
>>> wrote:
>>>
>>>> It hasn't been 3 months yet, but I wanted to call out a milestone that
>>>> Python 3 downloads crossed the 50% threshold on pypi, if just briefly.
>>>>
>>>> On Thu, Feb 13, 2020 at 12:40 AM Ismaël Mejía <ie...@gmail.com>
>>>> wrote:
>>>> >
>>>> > > I would suggest re-evaluating this within the next 3 months again.
>>>> We need to balance between user pain/contributor pain/our ability to
>>>> continuously test with python 2 in a shifting environment.
>>>> >
>>>> > Good idea for the in 3 months evaluation, at that point also
>>>> distributions will probably be phasing out python2 by default which
>>>> definitely help in this direction.
>>>> > Thanks for updating the roadmap Ahmet
>>>> >
>>>> >
>>>> > On Thu, Feb 13, 2020 at 2:49 AM Ahmet Altay <al...@google.com> wrote:
>>>> >>
>>>> >>
>>>> >>
>>>> >> On Wed, Feb 12, 2020 at 1:29 AM Ismaël Mejía <ie...@gmail.com>
>>>> wrote:
>>>> >>>
>>>> >>> I am with Chad on this, we should probably extend it a bit more,
>>>> even if it
>>>> >>> makes us struggle a bit at least we have some workarounds as Robert
>>>> suggests,
>>>> >>> and as Chad said there are still many people playing the python 3
>>>> catchup game,
>>>> >>> so worth to support those users.
>>>> >>>
>>>> >>>
>>>> >>> But maybe it is worth to evaluate the current state later in the
>>>> year.
>>>> >>
>>>> >>
>>>> >> I would suggest re-evaluating this within the next 3 months again.
>>>> We need to balance between user pain/contributor pain/our ability to
>>>> continuously test with python 2 in a shifting environment.
>>>> >>
>>>> >>>
>>>> >>> In the
>>>> >>> meantime can someone please update our Roadmap in the website with
>>>> this info and
>>>> >>> where we are with Python 3 support (it looks not up to date).
>>>> >>> https://beam.apache.org/roadmap/
>>>> >>
>>>> >>
>>>> >> I made a minor change to update that page (
>>>> https://github.com/apache/beam/pull/10848). A more comprehensive
>>>> update to that page and linked (
>>>> https://beam.apache.org/roadmap/python-sdk/#python-3-support) would
>>>> still be welcome.
>>>> >>
>>>> >>>
>>>> >>>
>>>> >>> - Ismaël
>>>> >>>
>>>> >>>
>>>> >>> On Tue, Feb 4, 2020 at 10:49 PM Robert Bradshaw <
>>>> robertwb@google.com> wrote:
>>>> >>>>
>>>> >>>> On Tue, Feb 4, 2020 at 12:12 PM Chad Dombrova <ch...@gmail.com>
>>>> wrote:
>>>> >>>> >>
>>>> >>>> >> Not to mention that all the nice work for the type hints will
>>>> have to be redone in the for 3.x.
>>>> >>>> >
>>>> >>>> > Note that there's a tool for automatically converting type
>>>> comments to annotations: https://github.com/ilevkivskyi/com2ann
>>>> >>>> >
>>>> >>>> > So don't let that part bother you.
>>>> >>>>
>>>> >>>> +1, I wouldn't worry about what can be easily automated.
>>>> >>>>
>>>> >>>> > I'm curious what other features you'd like to be using in the
>>>> Beam source that you cannot now.
>>>> >>>>
>>>> >>>> I hit things occasionally, e.g. I just ran into wanting
>>>> keyword-only
>>>> >>>> arguments the other day.
>>>> >>>>
>>>> >>>> >> It seems the faster we drop support the better.
>>>> >>>> >
>>>> >>>> >
>>>> >>>> > I've already gone over my position on this, but a refresher for
>>>> those who care: some of the key vendors that support my industry will not
>>>> offer python3-compatible versions of their software until the 4th quarter
>>>> of 2020. If Beam switches to python3-only before that point we may be
>>>> forced to stop contributing features (note: I'm the guy who added the type
>>>> hints :). Every month you can give us would be greatly appreciated.
>>>> >>>>
>>>> >>>> As another data point, we're still 80/20 on Py2/Py3 for downloads
>>>> at
>>>> >>>> PyPi [1] (which I've heard should be taken with a grain of salt,
>>>> but
>>>> >>>> likely isn't totally off). IMHO that ratio needs to be way higher
>>>> for
>>>> >>>> Python 3 to consider dropping Python 2. It's pretty noisy, but say
>>>> it
>>>> >>>> doubles every 3 months that would put us at least mid-year before
>>>> we
>>>> >>>> hit a cross-over point. On the other hand Q4 2020 is probably a
>>>> >>>> stretch.
>>>> >>>>
>>>> >>>> We could consider whether it needs to be an all-or-nothing thing as
>>>> >>>> well. E.g. perhaps some features could be Python 3 only sooner than
>>>> >>>> the whole codebase. (This would have to be well justified.) Another
>>>> >>>> mitigation is that it is possible to mix Python 2 and Python 3 in
>>>> the
>>>> >>>> same pipeline with portability, so if there's a library that you
>>>> need
>>>> >>>> for one DoFn it doesn't mean you have to hold back your whole
>>>> >>>> pipeline.
>>>> >>>>
>>>> >>>> - Robert
>>>> >>>>
>>>> >>>> [1] https://pypistats.org/packages/apache-beam , and that 20% may
>>>> just
>>>> >>>> be a spike.
>>>>
>>>
Re: Python2.7 Beam End-of-Life Date
Posted by Ahmet Altay <al...@google.com>.
Thank you for re-opening this Valentyn. I am in favor of EOLing py2 support
sooner than later. The reality is that we will not be effectively
supporting beam python 2 for a long time while the ecosystem already EOLed
python 2. That said, a significant chunk (but no longer a majority) of our
users are still using python 2. Upgrades are painful, it might be
especially painful nowadays. It would be good to hear counter view points,
user voices related to this.
On Thu, Jun 4, 2020 at 4:53 PM Valentyn Tymofieiev <va...@google.com>
wrote:
> Back at the end of February we decided to revisit this conversation in 3
> months. Do folks on this thread have any new input or perspective regarding
> us balancing "user pain/contributor pain/our ability to continuously test
> with python 2 in a shifting environment"?
>
> Some new information on my end is that we have been seeing steady adoption
> of Python 3 among Beam Python users in Dataflow, particularly strong
> adoption among streaming users, and Dataflow is sunsetting Python 2 support
> for all released Beam SDKs later this year [1]. We will have to remove
> Python 2 Beam test suites that use Dataflow when Dataflow runner disables
> Py2 support if this happens before Beam Py2 EOL (when we have to remove all
> Py2 suites), including performance tests that still use Dataflow on Python
> 3.
>
> I am curious how much motivation there is in the community at this moment
> to continue Py2 support in Beam, whether any previous Py3 migration
> blockers were resolved or any new blockers discovered among Beam users.
>
> [1] https://cloud.google.com/python/docs/python2-sunset/#dataflow
>
> On Fri, May 8, 2020 at 3:52 PM Valentyn Tymofieiev <va...@google.com>
> wrote:
>
>> That's good news! Thanks for sharing.
>>
>> Another datapoint, here are a few of Beam's dependencies that no longer
>> release new py2 artifacts (I looked at REQUIRED_PACKAGES + aws, gcp, and
>> interactive extras):
>>
>> hdfs
>> numpy
>> pyarrow
>> ipython
>>
>> There are more if we include transitive dependencies and test-only
>> packages. I also remember encountering one issue last month that was broken
>> only on Py2, which we had to go back and fix.
>>
>> If others have noticed frictions related to ongoing Py2 support or have
>> updates on previously mentioned Py3 migration blockers, feel free to post
>> them.
>>
>> On Fri, May 8, 2020 at 9:19 AM Robert Bradshaw <ro...@google.com>
>> wrote:
>>
>>> It hasn't been 3 months yet, but I wanted to call out a milestone that
>>> Python 3 downloads crossed the 50% threshold on pypi, if just briefly.
>>>
>>> On Thu, Feb 13, 2020 at 12:40 AM Ismaël Mejía <ie...@gmail.com> wrote:
>>> >
>>> > > I would suggest re-evaluating this within the next 3 months again.
>>> We need to balance between user pain/contributor pain/our ability to
>>> continuously test with python 2 in a shifting environment.
>>> >
>>> > Good idea for the in 3 months evaluation, at that point also
>>> distributions will probably be phasing out python2 by default which
>>> definitely help in this direction.
>>> > Thanks for updating the roadmap Ahmet
>>> >
>>> >
>>> > On Thu, Feb 13, 2020 at 2:49 AM Ahmet Altay <al...@google.com> wrote:
>>> >>
>>> >>
>>> >>
>>> >> On Wed, Feb 12, 2020 at 1:29 AM Ismaël Mejía <ie...@gmail.com>
>>> wrote:
>>> >>>
>>> >>> I am with Chad on this, we should probably extend it a bit more,
>>> even if it
>>> >>> makes us struggle a bit at least we have some workarounds as Robert
>>> suggests,
>>> >>> and as Chad said there are still many people playing the python 3
>>> catchup game,
>>> >>> so worth to support those users.
>>> >>>
>>> >>>
>>> >>> But maybe it is worth to evaluate the current state later in the
>>> year.
>>> >>
>>> >>
>>> >> I would suggest re-evaluating this within the next 3 months again. We
>>> need to balance between user pain/contributor pain/our ability to
>>> continuously test with python 2 in a shifting environment.
>>> >>
>>> >>>
>>> >>> In the
>>> >>> meantime can someone please update our Roadmap in the website with
>>> this info and
>>> >>> where we are with Python 3 support (it looks not up to date).
>>> >>> https://beam.apache.org/roadmap/
>>> >>
>>> >>
>>> >> I made a minor change to update that page (
>>> https://github.com/apache/beam/pull/10848). A more comprehensive update
>>> to that page and linked (
>>> https://beam.apache.org/roadmap/python-sdk/#python-3-support) would
>>> still be welcome.
>>> >>
>>> >>>
>>> >>>
>>> >>> - Ismaël
>>> >>>
>>> >>>
>>> >>> On Tue, Feb 4, 2020 at 10:49 PM Robert Bradshaw <ro...@google.com>
>>> wrote:
>>> >>>>
>>> >>>> On Tue, Feb 4, 2020 at 12:12 PM Chad Dombrova <ch...@gmail.com>
>>> wrote:
>>> >>>> >>
>>> >>>> >> Not to mention that all the nice work for the type hints will
>>> have to be redone in the for 3.x.
>>> >>>> >
>>> >>>> > Note that there's a tool for automatically converting type
>>> comments to annotations: https://github.com/ilevkivskyi/com2ann
>>> >>>> >
>>> >>>> > So don't let that part bother you.
>>> >>>>
>>> >>>> +1, I wouldn't worry about what can be easily automated.
>>> >>>>
>>> >>>> > I'm curious what other features you'd like to be using in the
>>> Beam source that you cannot now.
>>> >>>>
>>> >>>> I hit things occasionally, e.g. I just ran into wanting keyword-only
>>> >>>> arguments the other day.
>>> >>>>
>>> >>>> >> It seems the faster we drop support the better.
>>> >>>> >
>>> >>>> >
>>> >>>> > I've already gone over my position on this, but a refresher for
>>> those who care: some of the key vendors that support my industry will not
>>> offer python3-compatible versions of their software until the 4th quarter
>>> of 2020. If Beam switches to python3-only before that point we may be
>>> forced to stop contributing features (note: I'm the guy who added the type
>>> hints :). Every month you can give us would be greatly appreciated.
>>> >>>>
>>> >>>> As another data point, we're still 80/20 on Py2/Py3 for downloads at
>>> >>>> PyPi [1] (which I've heard should be taken with a grain of salt, but
>>> >>>> likely isn't totally off). IMHO that ratio needs to be way higher
>>> for
>>> >>>> Python 3 to consider dropping Python 2. It's pretty noisy, but say
>>> it
>>> >>>> doubles every 3 months that would put us at least mid-year before we
>>> >>>> hit a cross-over point. On the other hand Q4 2020 is probably a
>>> >>>> stretch.
>>> >>>>
>>> >>>> We could consider whether it needs to be an all-or-nothing thing as
>>> >>>> well. E.g. perhaps some features could be Python 3 only sooner than
>>> >>>> the whole codebase. (This would have to be well justified.) Another
>>> >>>> mitigation is that it is possible to mix Python 2 and Python 3 in
>>> the
>>> >>>> same pipeline with portability, so if there's a library that you
>>> need
>>> >>>> for one DoFn it doesn't mean you have to hold back your whole
>>> >>>> pipeline.
>>> >>>>
>>> >>>> - Robert
>>> >>>>
>>> >>>> [1] https://pypistats.org/packages/apache-beam , and that 20% may
>>> just
>>> >>>> be a spike.
>>>
>>
Re: Python2.7 Beam End-of-Life Date
Posted by Valentyn Tymofieiev <va...@google.com>.
Back at the end of February we decided to revisit this conversation in 3
months. Do folks on this thread have any new input or perspective regarding
us balancing "user pain/contributor pain/our ability to continuously test
with python 2 in a shifting environment"?
Some new information on my end is that we have been seeing steady adoption
of Python 3 among Beam Python users in Dataflow, particularly strong
adoption among streaming users, and Dataflow is sunsetting Python 2 support
for all released Beam SDKs later this year [1]. We will have to remove
Python 2 Beam test suites that use Dataflow when Dataflow runner disables
Py2 support if this happens before Beam Py2 EOL (when we have to remove all
Py2 suites), including performance tests that still use Dataflow on Python
3.
I am curious how much motivation there is in the community at this moment
to continue Py2 support in Beam, whether any previous Py3 migration
blockers were resolved or any new blockers discovered among Beam users.
[1] https://cloud.google.com/python/docs/python2-sunset/#dataflow
On Fri, May 8, 2020 at 3:52 PM Valentyn Tymofieiev <va...@google.com>
wrote:
> That's good news! Thanks for sharing.
>
> Another datapoint, here are a few of Beam's dependencies that no longer
> release new py2 artifacts (I looked at REQUIRED_PACKAGES + aws, gcp, and
> interactive extras):
>
> hdfs
> numpy
> pyarrow
> ipython
>
> There are more if we include transitive dependencies and test-only
> packages. I also remember encountering one issue last month that was broken
> only on Py2, which we had to go back and fix.
>
> If others have noticed frictions related to ongoing Py2 support or have
> updates on previously mentioned Py3 migration blockers, feel free to post
> them.
>
> On Fri, May 8, 2020 at 9:19 AM Robert Bradshaw <ro...@google.com>
> wrote:
>
>> It hasn't been 3 months yet, but I wanted to call out a milestone that
>> Python 3 downloads crossed the 50% threshold on pypi, if just briefly.
>>
>> On Thu, Feb 13, 2020 at 12:40 AM Ismaël Mejía <ie...@gmail.com> wrote:
>> >
>> > > I would suggest re-evaluating this within the next 3 months again. We
>> need to balance between user pain/contributor pain/our ability to
>> continuously test with python 2 in a shifting environment.
>> >
>> > Good idea for the in 3 months evaluation, at that point also
>> distributions will probably be phasing out python2 by default which
>> definitely help in this direction.
>> > Thanks for updating the roadmap Ahmet
>> >
>> >
>> > On Thu, Feb 13, 2020 at 2:49 AM Ahmet Altay <al...@google.com> wrote:
>> >>
>> >>
>> >>
>> >> On Wed, Feb 12, 2020 at 1:29 AM Ismaël Mejía <ie...@gmail.com>
>> wrote:
>> >>>
>> >>> I am with Chad on this, we should probably extend it a bit more, even
>> if it
>> >>> makes us struggle a bit at least we have some workarounds as Robert
>> suggests,
>> >>> and as Chad said there are still many people playing the python 3
>> catchup game,
>> >>> so worth to support those users.
>> >>>
>> >>>
>> >>> But maybe it is worth to evaluate the current state later in the year.
>> >>
>> >>
>> >> I would suggest re-evaluating this within the next 3 months again. We
>> need to balance between user pain/contributor pain/our ability to
>> continuously test with python 2 in a shifting environment.
>> >>
>> >>>
>> >>> In the
>> >>> meantime can someone please update our Roadmap in the website with
>> this info and
>> >>> where we are with Python 3 support (it looks not up to date).
>> >>> https://beam.apache.org/roadmap/
>> >>
>> >>
>> >> I made a minor change to update that page (
>> https://github.com/apache/beam/pull/10848). A more comprehensive update
>> to that page and linked (
>> https://beam.apache.org/roadmap/python-sdk/#python-3-support) would
>> still be welcome.
>> >>
>> >>>
>> >>>
>> >>> - Ismaël
>> >>>
>> >>>
>> >>> On Tue, Feb 4, 2020 at 10:49 PM Robert Bradshaw <ro...@google.com>
>> wrote:
>> >>>>
>> >>>> On Tue, Feb 4, 2020 at 12:12 PM Chad Dombrova <ch...@gmail.com>
>> wrote:
>> >>>> >>
>> >>>> >> Not to mention that all the nice work for the type hints will
>> have to be redone in the for 3.x.
>> >>>> >
>> >>>> > Note that there's a tool for automatically converting type
>> comments to annotations: https://github.com/ilevkivskyi/com2ann
>> >>>> >
>> >>>> > So don't let that part bother you.
>> >>>>
>> >>>> +1, I wouldn't worry about what can be easily automated.
>> >>>>
>> >>>> > I'm curious what other features you'd like to be using in the Beam
>> source that you cannot now.
>> >>>>
>> >>>> I hit things occasionally, e.g. I just ran into wanting keyword-only
>> >>>> arguments the other day.
>> >>>>
>> >>>> >> It seems the faster we drop support the better.
>> >>>> >
>> >>>> >
>> >>>> > I've already gone over my position on this, but a refresher for
>> those who care: some of the key vendors that support my industry will not
>> offer python3-compatible versions of their software until the 4th quarter
>> of 2020. If Beam switches to python3-only before that point we may be
>> forced to stop contributing features (note: I'm the guy who added the type
>> hints :). Every month you can give us would be greatly appreciated.
>> >>>>
>> >>>> As another data point, we're still 80/20 on Py2/Py3 for downloads at
>> >>>> PyPi [1] (which I've heard should be taken with a grain of salt, but
>> >>>> likely isn't totally off). IMHO that ratio needs to be way higher for
>> >>>> Python 3 to consider dropping Python 2. It's pretty noisy, but say it
>> >>>> doubles every 3 months that would put us at least mid-year before we
>> >>>> hit a cross-over point. On the other hand Q4 2020 is probably a
>> >>>> stretch.
>> >>>>
>> >>>> We could consider whether it needs to be an all-or-nothing thing as
>> >>>> well. E.g. perhaps some features could be Python 3 only sooner than
>> >>>> the whole codebase. (This would have to be well justified.) Another
>> >>>> mitigation is that it is possible to mix Python 2 and Python 3 in the
>> >>>> same pipeline with portability, so if there's a library that you need
>> >>>> for one DoFn it doesn't mean you have to hold back your whole
>> >>>> pipeline.
>> >>>>
>> >>>> - Robert
>> >>>>
>> >>>> [1] https://pypistats.org/packages/apache-beam , and that 20% may
>> just
>> >>>> be a spike.
>>
>
Re: Python2.7 Beam End-of-Life Date
Posted by Valentyn Tymofieiev <va...@google.com>.
That's good news! Thanks for sharing.
Another datapoint, here are a few of Beam's dependencies that no longer
release new py2 artifacts (I looked at REQUIRED_PACKAGES + aws, gcp, and
interactive extras):
hdfs
numpy
pyarrow
ipython
There are more if we include transitive dependencies and test-only
packages. I also remember encountering one issue last month that was broken
only on Py2, which we had to go back and fix.
If others have noticed frictions related to ongoing Py2 support or have
updates on previously mentioned Py3 migration blockers, feel free to post
them.
On Fri, May 8, 2020 at 9:19 AM Robert Bradshaw <ro...@google.com> wrote:
> It hasn't been 3 months yet, but I wanted to call out a milestone that
> Python 3 downloads crossed the 50% threshold on pypi, if just briefly.
>
> On Thu, Feb 13, 2020 at 12:40 AM Ismaël Mejía <ie...@gmail.com> wrote:
> >
> > > I would suggest re-evaluating this within the next 3 months again. We
> need to balance between user pain/contributor pain/our ability to
> continuously test with python 2 in a shifting environment.
> >
> > Good idea for the in 3 months evaluation, at that point also
> distributions will probably be phasing out python2 by default which
> definitely help in this direction.
> > Thanks for updating the roadmap Ahmet
> >
> >
> > On Thu, Feb 13, 2020 at 2:49 AM Ahmet Altay <al...@google.com> wrote:
> >>
> >>
> >>
> >> On Wed, Feb 12, 2020 at 1:29 AM Ismaël Mejía <ie...@gmail.com> wrote:
> >>>
> >>> I am with Chad on this, we should probably extend it a bit more, even
> if it
> >>> makes us struggle a bit at least we have some workarounds as Robert
> suggests,
> >>> and as Chad said there are still many people playing the python 3
> catchup game,
> >>> so worth to support those users.
> >>>
> >>>
> >>> But maybe it is worth to evaluate the current state later in the year.
> >>
> >>
> >> I would suggest re-evaluating this within the next 3 months again. We
> need to balance between user pain/contributor pain/our ability to
> continuously test with python 2 in a shifting environment.
> >>
> >>>
> >>> In the
> >>> meantime can someone please update our Roadmap in the website with
> this info and
> >>> where we are with Python 3 support (it looks not up to date).
> >>> https://beam.apache.org/roadmap/
> >>
> >>
> >> I made a minor change to update that page (
> https://github.com/apache/beam/pull/10848). A more comprehensive update
> to that page and linked (
> https://beam.apache.org/roadmap/python-sdk/#python-3-support) would still
> be welcome.
> >>
> >>>
> >>>
> >>> - Ismaël
> >>>
> >>>
> >>> On Tue, Feb 4, 2020 at 10:49 PM Robert Bradshaw <ro...@google.com>
> wrote:
> >>>>
> >>>> On Tue, Feb 4, 2020 at 12:12 PM Chad Dombrova <ch...@gmail.com>
> wrote:
> >>>> >>
> >>>> >> Not to mention that all the nice work for the type hints will
> have to be redone in the for 3.x.
> >>>> >
> >>>> > Note that there's a tool for automatically converting type comments
> to annotations: https://github.com/ilevkivskyi/com2ann
> >>>> >
> >>>> > So don't let that part bother you.
> >>>>
> >>>> +1, I wouldn't worry about what can be easily automated.
> >>>>
> >>>> > I'm curious what other features you'd like to be using in the Beam
> source that you cannot now.
> >>>>
> >>>> I hit things occasionally, e.g. I just ran into wanting keyword-only
> >>>> arguments the other day.
> >>>>
> >>>> >> It seems the faster we drop support the better.
> >>>> >
> >>>> >
> >>>> > I've already gone over my position on this, but a refresher for
> those who care: some of the key vendors that support my industry will not
> offer python3-compatible versions of their software until the 4th quarter
> of 2020. If Beam switches to python3-only before that point we may be
> forced to stop contributing features (note: I'm the guy who added the type
> hints :). Every month you can give us would be greatly appreciated.
> >>>>
> >>>> As another data point, we're still 80/20 on Py2/Py3 for downloads at
> >>>> PyPi [1] (which I've heard should be taken with a grain of salt, but
> >>>> likely isn't totally off). IMHO that ratio needs to be way higher for
> >>>> Python 3 to consider dropping Python 2. It's pretty noisy, but say it
> >>>> doubles every 3 months that would put us at least mid-year before we
> >>>> hit a cross-over point. On the other hand Q4 2020 is probably a
> >>>> stretch.
> >>>>
> >>>> We could consider whether it needs to be an all-or-nothing thing as
> >>>> well. E.g. perhaps some features could be Python 3 only sooner than
> >>>> the whole codebase. (This would have to be well justified.) Another
> >>>> mitigation is that it is possible to mix Python 2 and Python 3 in the
> >>>> same pipeline with portability, so if there's a library that you need
> >>>> for one DoFn it doesn't mean you have to hold back your whole
> >>>> pipeline.
> >>>>
> >>>> - Robert
> >>>>
> >>>> [1] https://pypistats.org/packages/apache-beam , and that 20% may
> just
> >>>> be a spike.
>