You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Manu Zhang <ow...@gmail.com> on 2019/01/05 05:03:01 UTC
Re: Python 3: final step
Guys,
Happy New Year !!!
I haven't got much time to contribute to Python 3 support. What is the
progress now ? It seems there are quite a few open issues under
https://issues.apache.org/jira/browse/BEAM-1251. People have kept asking
about Python 3 support in tf.transform (
https://github.com/tensorflow/transform/issues/1) which is blocked by
BEAM-1251.
Thanks,
Manu Zhang
On Fri, Oct 12, 2018 at 3:17 AM Valentyn Tymofieiev <va...@google.com>
wrote:
> I cc'ed a few folks who are familiar with Jenkins setup on
> https://issues.apache.org/jira/browse/BEAM-5663, I think we can continue
> the discussion there or start a separate thread.
>
> On Wed, Oct 10, 2018 at 8:54 PM Manu Zhang <ow...@gmail.com>
> wrote:
>
>> Does anyone know how to set up python version on Jenkins ? It’s Python
>> 3.5.2 now.
>>
>> Thanks,
>> Manu Zhang
>> On Oct 5, 2018, 9:24 AM +0800, Valentyn Tymofieiev <va...@google.com>,
>> wrote:
>>
>> I have put together a guide [1] to help get started with investigating
>> Python 3-related test failures that may be helpful for new folks joining
>> the effort.
>>
>> Comments and improvements welcome!
>>
>> Thanks,
>> Valentyn
>>
>> [1]
>> https://docs.google.com/document/d/1s1BJVCY65LB_SYK1SU1u7NbZiFANoq-nEYaEvzRbYlA
>>
>>
>> On Thu, Oct 4, 2018 at 11:26 AM Valentyn Tymofieiev <va...@google.com>
>> wrote:
>>
>>> I agree there is some overlap between JIRAs that track individual
>>> failures and module-level JIRAs. We originally wanted to do the conversion
>>> on a module-by-module basis, however we learned that test failures in some
>>> modules require changes in other modules, and it may be a little easier to
>>> slice the problem if we focus on classes of failures.
>>>
>>> Module-level JIRAs can still be useful for tracking the end result: tox
>>> suites cover all tests in the module in Py3 environment, and there are no
>>> disabled tests in the module that don't have individual JIRAs tracking them.
>>>
>>> I suggest that folks who are working on module-level JIRAs assign to
>>> themselves the JIRAs that track individual failures if/when they are
>>> actively addressing them. This way, unassigned problem-specific JIRAs can
>>> use help from the community.
>>>
>>> Thanks,
>>> Valentyn
>>>
>>>
>>> On Wed, Oct 3, 2018 at 8:14 PM Manu Zhang <ow...@gmail.com>
>>> wrote:
>>>
>>>> Thanks Valentyn. Note some test failing issues are covered by “Finish
>>>> Python 3 porting for *** module”, e.g.
>>>> https://issues.apache.org/jira/browse/BEAM-5315.
>>>>
>>>> Manu
>>>> 在 2018年10月3日 +0800 PM4:18，Valentyn Tymofieiev <va...@google.com>，写道：
>>>>
>>>> Hi Rakesh and Manu,
>>>>
>>>> Thanks to both of you for offering help (in different threads). It's
>>>> great to see that more and more people get involved with helping to make
>>>> Beam Python 3 compatible!
>>>>
>>>> There are a few PRs in flight, and several people in the community
>>>> actively work on Python 3 support now. I would be happy to coordinate the
>>>> work so that we don't step at each others toes and avoid duplication of
>>>> effort.
>>>>
>>>> I recently looked at unit tests that are still failing in Python 3
>>>> environment and filed a few issues (within range BEAM-5615 - BEAM-5629),
>>>> to track similar classes of errors. You can also find them on Kanban board
>>>> [1].
>>>> In particular, BEAM-5620 and BEAM-5627 should be easy issues to get
>>>> started.
>>>>
>>>> There are multiple ways you can help:
>>>> - Helping to rootcause errors. Even a comment why a test is failing and
>>>> a suggestion how to fix it, will be helpful for others when you don't have
>>>> time to do the fix.
>>>> - Helping with code reviews.
>>>> - Reporting new issues (as subtasks to BEAM-1251), deduplicating or
>>>> splitting the existing issues. We probably don't want to file a Jira for
>>>> each of 250+ currently failing tests at this point, but it may make sense
>>>> to track the errors that occur repeatedly share the root cause.
>>>> - Fixing the issues. Feel free to assign an issue to yourself if you
>>>> have a fix in mind and plan to actively work on it. Due to the nature of
>>>> the problem it may occasionally happen that two issues share the rootcause,
>>>> or fixing one issue is a prerequisite for fixing another issue, so sync to
>>>> master often to make sure the issue you are working on is not already
>>>> fixed.
>>>>
>>>> I'll also keep an eye on the PRs and will try to keep the list of open
>>>> issues up to date.
>>>>
>>>> Thanks,
>>>> Valentyn
>>>>
>>>> [1]:
>>>> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=245&view=detail
>>>>
>>>>
>>>> On Tue, Oct 2, 2018 at 9:38 AM Pablo Estrada <pa...@google.com>
>>>> wrote:
>>>>
>>>>> Very cool : ) I'm also available to review / merge if you need help
>>>>> from my side.
>>>>> Best
>>>>> -P.
>>>>>
>>>>> On Tue, Oct 2, 2018 at 7:45 AM Rakesh Kumar <ra...@lyft.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Rob,
>>>>>>
>>>>>> I am, Rakesh Kumar, using Beam SDK for one of my projects at Lyft. I
>>>>>> have been working closely with Thomas Weise. I have already met a couple of
>>>>>> Python SDK developers in person.
>>>>>> I am interested to help migrate to Python 3. You can assign me PRs
>>>>>> for review. I am also more than happy to take a simple ticket to begin
>>>>>> development work on Beam.
>>>>>>
>>>>>> Thank you,
>>>>>> Rakesh
>>>>>>
>>>>>> On Wed, Sep 5, 2018 at 9:12 AM Robbe Sneyders <ro...@ml6.eu>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi everyone,
>>>>>>>
>>>>>>> With the merging of [1], we now have Python 3 tests running on
>>>>>>> Jenkins, which allows us to move forward with the last step of the Python 3
>>>>>>> porting.
>>>>>>>
>>>>>>> You can follow the progress on the Jira Kanban Board [2]. If you're
>>>>>>> interested in helping by porting a module, you can assign one of the issues
>>>>>>> to yourself and start coding. You can find the different steps outlined in
>>>>>>> the design document [3].
>>>>>>>
>>>>>>> We could also use some extra reviewers. If you're interested, let us
>>>>>>> know, and we'll tag you in our PRs.
>>>>>>>
>>>>>>> [1] https://github.com/apache/beam/pull/6266
>>>>>>> [2]
>>>>>>> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=245
>>>>>>> [3] https://s.apache.org/beam-python-3
>>>>>>>
>>>>>>> kind regards,
>>>>>>> Robbe
>>>>>>> --
>>>>>>>
>>>>>>> [image: https://ml6.eu] <https://ml6.eu/>
>>>>>>>
>>>>>>> *Robbe Sneyders*
>>>>>>>
>>>>>>> ML6 Gent
>>>>>>> <https://www.google.be/maps/place/ML6/@51.037408,3.7044893,17z/data=!3m1!4b1!4m5!3m4!1s0x47c37161feeca14b:0xb8f72585fdd21c90!8m2!3d51.037408!4d3.706678?hl=nl>
>>>>>>>
>>>>>>> M: +32 474 71 31 08 <+32%20474%2071%2031%2008>
>>>>>>>
>>>>>> --
>>>>>> Rakesh Kumar
>>>>>> Software Engineer
>>>>>> 510-761-1364 <(510)%20761-1364> |
>>>>>>
>>>>>> <https://www.lyft.com/>
>>>>>>
>>>>>
Re: Python 3: final step
Posted by Robbe Sneyders <ro...@ml6.eu>.
Hi all,
We've been making quite some progress these last weeks. I'll give a short
status update on where we are right now.
Our current goal is to make all unittests succeed in Python 3. We are
currently at:
1842 tests: (SKIP=350, errors=100, failures=9)
All of these remaining errors and failures are in the io and examples
packages, except for some non-blocking failing typehints tests. All other
packages have been ported. We're currently working on porting the io
package, and have finished all core modules (iobase, filesystemio,
filebasedsource/sink, textio, ...). This allowed us to fix the wordcount
end to end examples, which now run successfully on Python 3! [1]
Next up are all other io sources and sinks. Most of the skipped tests are
due to missing GCP components in the Python 3 test suite. We will add a
separate GCP test suite for Python 3 when we start porting the first GCP io
module, which should be very soon.
Anyone who wants to help out, can start with porting one of these
sources/sinks.
Kind regards,
Robbe
[1] https://github.com/apache/beam/pull/7447
[image: https://ml6.eu] <https://ml6.eu/>
* Robbe Sneyders*
ML6 Gent
<https://www.google.be/maps/place/ML6/@51.037408,3.7044893,17z/data=!3m1!4b1!4m5!3m4!1s0x47c37161feeca14b:0xb8f72585fdd21c90!8m2!3d51.037408!4d3.706678?hl=nl>
M: +32 474 71 31 08
On Tue, 8 Jan 2019 at 04:58, Ahmet Altay <al...@google.com> wrote:
> +Matthias Feys <ma...@ml6.eu> +Valentyn Tymofieiev
> <va...@google.com> +Mark Liu <ma...@google.com> could add more
> details here since they are working on Python 3 for a while now.
>
> Hopeful state is that we might have python 3 working with DirectRunner in
> the release after this one (2.12). Mark is also working on getting python 3
> working on a cluster. He has been able to run WordCount on Dataflow service
> with some hacks but it was not yet ready to run out of the box. I would
> like to note that we are targeting python 3 support only for portable
> runners so running on Dataflow and Flink should happen at the same time.
>
> Matthias and Valentyn are still working on converting the SDK to be Python
> 3 compatible. They are now mostly dealing with harder to convert parts of
> the SDK. (e.g. Parts were python 2/3 differences results in performance
> regressions or subtle changes in the behavior).
>
> To the folks working on this, it would be really helpful if you could
> update BEAM-1251 regularly. We have shared this issue with many people and
> not all of them will read this thread.
>
> Ahmet
>
> On Mon, Jan 7, 2019 at 8:15 AM Maximilian Michels <mx...@apache.org> wrote:
>
>> Also curious because I see Python 3 requests quite often. I always say,
>> we're
>> close, but how close are we? :)
>>
>> Thanks,
>> Max
>>
>> On 05.01.19 00:03, Manu Zhang wrote:
>> > Guys,
>> >
>> > Happy New Year !!!
>> > I haven't got much time to contribute to Python 3 support. What is the
>> progress
>> > now ? It seems there are quite a few open issues under
>> > https://issues.apache.org/jira/browse/BEAM-1251. People have kept
>> asking about
>> > Python 3 support in tf.transform
>> > (https://github.com/tensorflow/transform/issues/1) which is blocked by
>> BEAM-1251.
>> >
>> > Thanks,
>> > Manu Zhang
>> >
>> >
>> > On Fri, Oct 12, 2018 at 3:17 AM Valentyn Tymofieiev <
>> valentyn@google.com
>> > <ma...@google.com>> wrote:
>> >
>> > I cc'ed a few folks who are familiar with Jenkins setup on
>> > https://issues.apache.org/jira/browse/BEAM-5663, I think we can
>> continue the
>> > discussion there or start a separate thread.
>> >
>> > On Wed, Oct 10, 2018 at 8:54 PM Manu Zhang <owenzhang1990@gmail.com
>> > <ma...@gmail.com>> wrote:
>> >
>> > Does anyone know how to set up python version on Jenkins ? It’s
>> Python
>> > 3.5.2 now.
>> >
>> > Thanks,
>> > Manu Zhang
>> > On Oct 5, 2018, 9:24 AM +0800, Valentyn Tymofieiev <
>> valentyn@google.com
>> > <ma...@google.com>>, wrote:
>> >> I have put together a guide [1] to help get started with
>> investigating
>> >> Python 3-related test failures that may be helpful for new
>> folks
>> >> joining the effort.
>> >>
>> >> Comments and improvements welcome!
>> >>
>> >> Thanks,
>> >> Valentyn
>> >> [1]
>> >>
>> https://docs.google.com/document/d/1s1BJVCY65LB_SYK1SU1u7NbZiFANoq-nEYaEvzRbYlA
>> >>
>> >>
>> >> On Thu, Oct 4, 2018 at 11:26 AM Valentyn Tymofieiev
>> >> <valentyn@google.com <ma...@google.com>> wrote:
>> >>
>> >> I agree there is some overlap between JIRAs that track
>> individual
>> >> failures and module-level JIRAs. We originally wanted to
>> do the
>> >> conversion on a module-by-module basis, however we learned
>> that
>> >> test failures in some modules require changes in other
>> modules,
>> >> and it may be a little easier to slice the problem if we
>> focus on
>> >> classes of failures.
>> >>
>> >> Module-level JIRAs can still be useful for tracking the end
>> >> result: tox suites cover all tests in the module in Py3
>> >> environment, and there are no disabled tests in the module
>> that
>> >> don't have individual JIRAs tracking them.
>> >>
>> >> I suggest that folks who are working on module-level JIRAs
>> assign
>> >> to themselves the JIRAs that track individual failures
>> if/when
>> >> they are actively addressing them. This way, unassigned
>> >> problem-specific JIRAs can use help from the community.
>> >>
>> >> Thanks,
>> >> Valentyn
>> >>
>> >>
>> >> On Wed, Oct 3, 2018 at 8:14 PM Manu Zhang <
>> owenzhang1990@gmail.com
>> >> <ma...@gmail.com>> wrote:
>> >>
>> >> Thanks Valentyn. Note some test failing issues are
>> covered by
>> >> “Finish Python 3 porting for *** module”, e.g.
>> >> https://issues.apache.org/jira/browse/BEAM-5315.
>> >>
>> >> Manu
>> >> 在 2018年10月3日 +0800 PM4:18，Valentyn Tymofieiev
>> >> <valentyn@google.com <ma...@google.com>>，写道：
>> >>> Hi Rakesh and Manu,
>> >>>
>> >>> Thanks to both of you for offering help (in different
>> >>> threads). It's great to see that more and more people
>> get
>> >>> involved with helping to make Beam Python 3
>> compatible!
>> >>>
>> >>> There are a few PRs in flight, and several people in
>> the
>> >>> community actively work on Python 3 support now. I
>> would be
>> >>> happy to coordinate the work so that we don't step at
>> each
>> >>> others toes and avoid duplication of effort.
>> >>>
>> >>> I recently looked at unit tests that are still
>> failing in
>> >>> Python 3 environment and filed a few issues (within
>> range
>> >>> BEAM-5615 - BEAM-5629), to track similar classes of
>> errors.
>> >>> You can also find them on Kanban board [1].
>> >>> In particular, BEAM-5620 and BEAM-5627 should be easy
>> issues
>> >>> to get started.
>> >>>
>> >>> There are multiple ways you can help:
>> >>> - Helping to rootcause errors. Even a comment why a
>> test is
>> >>> failing and a suggestion how to fix it, will be
>> helpful for
>> >>> others when you don't have time to do the fix.
>> >>> - Helping with code reviews.
>> >>> - Reporting new issues (as subtasks to BEAM-1251),
>> >>> deduplicating or splitting the existing issues. We
>> probably
>> >>> don't want to file a Jira for each of 250+ currently
>> failing
>> >>> tests at this point, but it may make sense to track
>> the
>> >>> errors that occur repeatedly share the root cause.
>> >>> - Fixing the issues. Feel free to assign an issue to
>> yourself
>> >>> if you have a fix in mind and plan to actively work
>> on it.
>> >>> Due to the nature of the problem it may occasionally
>> happen
>> >>> that two issues share the rootcause, or fixing one
>> issue is a
>> >>> prerequisite for fixing another issue, so sync to
>> master
>> >>> often to make sure the issue you are working on is not
>> >>> already fixed.
>> >>>
>> >>> I'll also keep an eye on the PRs and will try to keep
>> the
>> >>> list of open issues up to date.
>> >>>
>> >>> Thanks,
>> >>> Valentyn
>> >>>
>> >>> [1]:
>> >>>
>> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=245&view=detail
>> >>>
>> >>>
>> >>> On Tue, Oct 2, 2018 at 9:38 AM Pablo Estrada
>> >>> <pabloem@google.com <ma...@google.com>>
>> wrote:
>> >>>
>> >>> Very cool : ) I'm also available to review /
>> merge if you
>> >>> need help from my side.
>> >>> Best
>> >>> -P.
>> >>>
>> >>> On Tue, Oct 2, 2018 at 7:45 AM Rakesh Kumar
>> >>> <rakeshkumar@lyft.com <mailto:
>> rakeshkumar@lyft.com>> wrote:
>> >>>
>> >>> Hi Rob,
>> >>>
>> >>> I am, Rakesh Kumar, using Beam SDK for one of
>> my
>> >>> projects at Lyft. I have been working closely
>> with
>> >>> Thomas Weise. I have already met a couple of
>> Python
>> >>> SDK developers in person.
>> >>> I am interested to help migrate to Python 3.
>> You can
>> >>> assign me PRs for review. I am also more than
>> happy
>> >>> to take a simple ticket to begin development
>> work on
>> >>> Beam.
>> >>>
>> >>> Thank you,
>> >>> Rakesh
>> >>>
>> >>> On Wed, Sep 5, 2018 at 9:12 AM Robbe Sneyders
>> >>> <robbe.sneyders@ml6.eu
>> >>> <ma...@ml6.eu>> wrote:
>> >>>
>> >>> Hi everyone,
>> >>>
>> >>> With the merging of [1], we now have
>> Python 3
>> >>> tests running on Jenkins, which allows us
>> to move
>> >>> forward with the last step of the Python
>> 3 porting.
>> >>>
>> >>> You can follow the progress on the Jira
>> Kanban
>> >>> Board [2]. If you're interested in
>> helping by
>> >>> porting a module, you can assign one of
>> the
>> >>> issues to yourself and start coding. You
>> can find
>> >>> the different steps outlined in the design
>> >>> document [3].
>> >>>
>> >>> We could also use some extra reviewers.
>> If you're
>> >>> interested, let us know, and we'll tag
>> you in our
>> >>> PRs.
>> >>>
>> >>> [1]
>> https://github.com/apache/beam/pull/6266
>> >>> [2]
>> >>>
>> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=245
>> >>> [3] https://s.apache.org/beam-python-3
>> >>>
>> >>> kind regards,
>> >>> Robbe
>> >>> --
>> >>>
>> >>> https://ml6.eu <https://ml6.eu/>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> *Robbe Sneyders*
>> >>>
>> >>> ML6 Gent
>> >>> <
>> https://www.google.be/maps/place/ML6/@51.037408,3.7044893,17z/data=!3m1!4b1!4m5!3m4!1s0x47c37161feeca14b:0xb8f72585fdd21c90!8m2!3d51.037408!4d3.706678?hl=nl
>> >
>> >>>
>> >>> M: +32 474 71 31 08
>> <tel:+32%20474%2071%2031%2008>
>> >>>
>> >>> --
>> >>> Rakesh Kumar
>> >>> Software Engineer
>> >>> 510-761-1364 <tel:(510)%20761-1364> |
>> >>> <https://www.lyft.com/>
>> >>>
>>
>
Re: Python 3: final step
Posted by Ahmet Altay <al...@google.com>.
+Matthias Feys <ma...@ml6.eu> +Valentyn Tymofieiev
<va...@google.com> +Mark
Liu <ma...@google.com> could add more details here since they are working
on Python 3 for a while now.
Hopeful state is that we might have python 3 working with DirectRunner in
the release after this one (2.12). Mark is also working on getting python 3
working on a cluster. He has been able to run WordCount on Dataflow service
with some hacks but it was not yet ready to run out of the box. I would
like to note that we are targeting python 3 support only for portable
runners so running on Dataflow and Flink should happen at the same time.
Matthias and Valentyn are still working on converting the SDK to be Python
3 compatible. They are now mostly dealing with harder to convert parts of
the SDK. (e.g. Parts were python 2/3 differences results in performance
regressions or subtle changes in the behavior).
To the folks working on this, it would be really helpful if you could
update BEAM-1251 regularly. We have shared this issue with many people and
not all of them will read this thread.
Ahmet
On Mon, Jan 7, 2019 at 8:15 AM Maximilian Michels <mx...@apache.org> wrote:
> Also curious because I see Python 3 requests quite often. I always say,
> we're
> close, but how close are we? :)
>
> Thanks,
> Max
>
> On 05.01.19 00:03, Manu Zhang wrote:
> > Guys,
> >
> > Happy New Year !!!
> > I haven't got much time to contribute to Python 3 support. What is the
> progress
> > now ? It seems there are quite a few open issues under
> > https://issues.apache.org/jira/browse/BEAM-1251. People have kept
> asking about
> > Python 3 support in tf.transform
> > (https://github.com/tensorflow/transform/issues/1) which is blocked by
> BEAM-1251.
> >
> > Thanks,
> > Manu Zhang
> >
> >
> > On Fri, Oct 12, 2018 at 3:17 AM Valentyn Tymofieiev <valentyn@google.com
> > <ma...@google.com>> wrote:
> >
> > I cc'ed a few folks who are familiar with Jenkins setup on
> > https://issues.apache.org/jira/browse/BEAM-5663, I think we can
> continue the
> > discussion there or start a separate thread.
> >
> > On Wed, Oct 10, 2018 at 8:54 PM Manu Zhang <owenzhang1990@gmail.com
> > <ma...@gmail.com>> wrote:
> >
> > Does anyone know how to set up python version on Jenkins ? It’s
> Python
> > 3.5.2 now.
> >
> > Thanks,
> > Manu Zhang
> > On Oct 5, 2018, 9:24 AM +0800, Valentyn Tymofieiev <
> valentyn@google.com
> > <ma...@google.com>>, wrote:
> >> I have put together a guide [1] to help get started with
> investigating
> >> Python 3-related test failures that may be helpful for new folks
> >> joining the effort.
> >>
> >> Comments and improvements welcome!
> >>
> >> Thanks,
> >> Valentyn
> >> [1]
> >>
> https://docs.google.com/document/d/1s1BJVCY65LB_SYK1SU1u7NbZiFANoq-nEYaEvzRbYlA
> >>
> >>
> >> On Thu, Oct 4, 2018 at 11:26 AM Valentyn Tymofieiev
> >> <valentyn@google.com <ma...@google.com>> wrote:
> >>
> >> I agree there is some overlap between JIRAs that track
> individual
> >> failures and module-level JIRAs. We originally wanted to do
> the
> >> conversion on a module-by-module basis, however we learned
> that
> >> test failures in some modules require changes in other
> modules,
> >> and it may be a little easier to slice the problem if we
> focus on
> >> classes of failures.
> >>
> >> Module-level JIRAs can still be useful for tracking the end
> >> result: tox suites cover all tests in the module in Py3
> >> environment, and there are no disabled tests in the module
> that
> >> don't have individual JIRAs tracking them.
> >>
> >> I suggest that folks who are working on module-level JIRAs
> assign
> >> to themselves the JIRAs that track individual failures
> if/when
> >> they are actively addressing them. This way, unassigned
> >> problem-specific JIRAs can use help from the community.
> >>
> >> Thanks,
> >> Valentyn
> >>
> >>
> >> On Wed, Oct 3, 2018 at 8:14 PM Manu Zhang <
> owenzhang1990@gmail.com
> >> <ma...@gmail.com>> wrote:
> >>
> >> Thanks Valentyn. Note some test failing issues are
> covered by
> >> “Finish Python 3 porting for *** module”, e.g.
> >> https://issues.apache.org/jira/browse/BEAM-5315.
> >>
> >> Manu
> >> 在 2018年10月3日 +0800 PM4:18，Valentyn Tymofieiev
> >> <valentyn@google.com <ma...@google.com>>，写道：
> >>> Hi Rakesh and Manu,
> >>>
> >>> Thanks to both of you for offering help (in different
> >>> threads). It's great to see that more and more people
> get
> >>> involved with helping to make Beam Python 3 compatible!
> >>>
> >>> There are a few PRs in flight, and several people in
> the
> >>> community actively work on Python 3 support now. I
> would be
> >>> happy to coordinate the work so that we don't step at
> each
> >>> others toes and avoid duplication of effort.
> >>>
> >>> I recently looked at unit tests that are still failing
> in
> >>> Python 3 environment and filed a few issues (within
> range
> >>> BEAM-5615 - BEAM-5629), to track similar classes of
> errors.
> >>> You can also find them on Kanban board [1].
> >>> In particular, BEAM-5620 and BEAM-5627 should be easy
> issues
> >>> to get started.
> >>>
> >>> There are multiple ways you can help:
> >>> - Helping to rootcause errors. Even a comment why a
> test is
> >>> failing and a suggestion how to fix it, will be
> helpful for
> >>> others when you don't have time to do the fix.
> >>> - Helping with code reviews.
> >>> - Reporting new issues (as subtasks to BEAM-1251),
> >>> deduplicating or splitting the existing issues. We
> probably
> >>> don't want to file a Jira for each of 250+ currently
> failing
> >>> tests at this point, but it may make sense to track the
> >>> errors that occur repeatedly share the root cause.
> >>> - Fixing the issues. Feel free to assign an issue to
> yourself
> >>> if you have a fix in mind and plan to actively work on
> it.
> >>> Due to the nature of the problem it may occasionally
> happen
> >>> that two issues share the rootcause, or fixing one
> issue is a
> >>> prerequisite for fixing another issue, so sync to
> master
> >>> often to make sure the issue you are working on is not
> >>> already fixed.
> >>>
> >>> I'll also keep an eye on the PRs and will try to keep
> the
> >>> list of open issues up to date.
> >>>
> >>> Thanks,
> >>> Valentyn
> >>>
> >>> [1]:
> >>>
> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=245&view=detail
> >>>
> >>>
> >>> On Tue, Oct 2, 2018 at 9:38 AM Pablo Estrada
> >>> <pabloem@google.com <ma...@google.com>>
> wrote:
> >>>
> >>> Very cool : ) I'm also available to review / merge
> if you
> >>> need help from my side.
> >>> Best
> >>> -P.
> >>>
> >>> On Tue, Oct 2, 2018 at 7:45 AM Rakesh Kumar
> >>> <rakeshkumar@lyft.com <ma...@lyft.com>>
> wrote:
> >>>
> >>> Hi Rob,
> >>>
> >>> I am, Rakesh Kumar, using Beam SDK for one of
> my
> >>> projects at Lyft. I have been working closely
> with
> >>> Thomas Weise. I have already met a couple of
> Python
> >>> SDK developers in person.
> >>> I am interested to help migrate to Python 3.
> You can
> >>> assign me PRs for review. I am also more than
> happy
> >>> to take a simple ticket to begin development
> work on
> >>> Beam.
> >>>
> >>> Thank you,
> >>> Rakesh
> >>>
> >>> On Wed, Sep 5, 2018 at 9:12 AM Robbe Sneyders
> >>> <robbe.sneyders@ml6.eu
> >>> <ma...@ml6.eu>> wrote:
> >>>
> >>> Hi everyone,
> >>>
> >>> With the merging of [1], we now have
> Python 3
> >>> tests running on Jenkins, which allows us
> to move
> >>> forward with the last step of the Python 3
> porting.
> >>>
> >>> You can follow the progress on the Jira
> Kanban
> >>> Board [2]. If you're interested in helping
> by
> >>> porting a module, you can assign one of the
> >>> issues to yourself and start coding. You
> can find
> >>> the different steps outlined in the design
> >>> document [3].
> >>>
> >>> We could also use some extra reviewers. If
> you're
> >>> interested, let us know, and we'll tag you
> in our
> >>> PRs.
> >>>
> >>> [1]
> https://github.com/apache/beam/pull/6266
> >>> [2]
> >>>
> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=245
> >>> [3] https://s.apache.org/beam-python-3
> >>>
> >>> kind regards,
> >>> Robbe
> >>> --
> >>>
> >>> https://ml6.eu <https://ml6.eu/>
> >>>
> >>>
> >>>
> >>>
> >>> *Robbe Sneyders*
> >>>
> >>> ML6 Gent
> >>> <
> https://www.google.be/maps/place/ML6/@51.037408,3.7044893,17z/data=!3m1!4b1!4m5!3m4!1s0x47c37161feeca14b:0xb8f72585fdd21c90!8m2!3d51.037408!4d3.706678?hl=nl
> >
> >>>
> >>> M: +32 474 71 31 08
> <tel:+32%20474%2071%2031%2008>
> >>>
> >>> --
> >>> Rakesh Kumar
> >>> Software Engineer
> >>> 510-761-1364 <tel:(510)%20761-1364> |
> >>> <https://www.lyft.com/>
> >>>
>
Re: Python 3: final step
Posted by Maximilian Michels <mx...@apache.org>.
Also curious because I see Python 3 requests quite often. I always say, we're
close, but how close are we? :)
Thanks,
Max
On 05.01.19 00:03, Manu Zhang wrote:
> Guys,
>
> Happy New Year !!!
> I haven't got much time to contribute to Python 3 support. What is the progress
> now ? It seems there are quite a few open issues under
> https://issues.apache.org/jira/browse/BEAM-1251. People have kept asking about
> Python 3 support in tf.transform
> (https://github.com/tensorflow/transform/issues/1) which is blocked by BEAM-1251.
>
> Thanks,
> Manu Zhang
>
>
> On Fri, Oct 12, 2018 at 3:17 AM Valentyn Tymofieiev <valentyn@google.com
> <ma...@google.com>> wrote:
>
> I cc'ed a few folks who are familiar with Jenkins setup on
> https://issues.apache.org/jira/browse/BEAM-5663, I think we can continue the
> discussion there or start a separate thread.
>
> On Wed, Oct 10, 2018 at 8:54 PM Manu Zhang <owenzhang1990@gmail.com
> <ma...@gmail.com>> wrote:
>
> Does anyone know how to set up python version on Jenkins ? It’s Python
> 3.5.2 now.
>
> Thanks,
> Manu Zhang
> On Oct 5, 2018, 9:24 AM +0800, Valentyn Tymofieiev <valentyn@google.com
> <ma...@google.com>>, wrote:
>> I have put together a guide [1] to help get started with investigating
>> Python 3-related test failures that may be helpful for new folks
>> joining the effort.
>>
>> Comments and improvements welcome!
>>
>> Thanks,
>> Valentyn
>> [1]
>> https://docs.google.com/document/d/1s1BJVCY65LB_SYK1SU1u7NbZiFANoq-nEYaEvzRbYlA
>>
>>
>> On Thu, Oct 4, 2018 at 11:26 AM Valentyn Tymofieiev
>> <valentyn@google.com <ma...@google.com>> wrote:
>>
>> I agree there is some overlap between JIRAs that track individual
>> failures and module-level JIRAs. We originally wanted to do the
>> conversion on a module-by-module basis, however we learned that
>> test failures in some modules require changes in other modules,
>> and it may be a little easier to slice the problem if we focus on
>> classes of failures.
>>
>> Module-level JIRAs can still be useful for tracking the end
>> result: tox suites cover all tests in the module in Py3
>> environment, and there are no disabled tests in the module that
>> don't have individual JIRAs tracking them.
>>
>> I suggest that folks who are working on module-level JIRAs assign
>> to themselves the JIRAs that track individual failures if/when
>> they are actively addressing them. This way, unassigned
>> problem-specific JIRAs can use help from the community.
>>
>> Thanks,
>> Valentyn
>>
>>
>> On Wed, Oct 3, 2018 at 8:14 PM Manu Zhang <owenzhang1990@gmail.com
>> <ma...@gmail.com>> wrote:
>>
>> Thanks Valentyn. Note some test failing issues are covered by
>> “Finish Python 3 porting for *** module”, e.g.
>> https://issues.apache.org/jira/browse/BEAM-5315.
>>
>> Manu
>> 在 2018年10月3日 +0800 PM4:18，Valentyn Tymofieiev
>> <valentyn@google.com <ma...@google.com>>，写道：
>>> Hi Rakesh and Manu,
>>>
>>> Thanks to both of you for offering help (in different
>>> threads). It's great to see that more and more people get
>>> involved with helping to make Beam Python 3 compatible!
>>>
>>> There are a few PRs in flight, and several people in the
>>> community actively work on Python 3 support now. I would be
>>> happy to coordinate the work so that we don't step at each
>>> others toes and avoid duplication of effort.
>>>
>>> I recently looked at unit tests that are still failing in
>>> Python 3 environment and filed a few issues (within range
>>> BEAM-5615 - BEAM-5629), to track similar classes of errors.
>>> You can also find them on Kanban board [1].
>>> In particular, BEAM-5620 and BEAM-5627 should be easy issues
>>> to get started.
>>>
>>> There are multiple ways you can help:
>>> - Helping to rootcause errors. Even a comment why a test is
>>> failing and a suggestion how to fix it, will be helpful for
>>> others when you don't have time to do the fix.
>>> - Helping with code reviews.
>>> - Reporting new issues (as subtasks to BEAM-1251),
>>> deduplicating or splitting the existing issues. We probably
>>> don't want to file a Jira for each of 250+ currently failing
>>> tests at this point, but it may make sense to track the
>>> errors that occur repeatedly share the root cause.
>>> - Fixing the issues. Feel free to assign an issue to yourself
>>> if you have a fix in mind and plan to actively work on it.
>>> Due to the nature of the problem it may occasionally happen
>>> that two issues share the rootcause, or fixing one issue is a
>>> prerequisite for fixing another issue, so sync to master
>>> often to make sure the issue you are working on is not
>>> already fixed.
>>>
>>> I'll also keep an eye on the PRs and will try to keep the
>>> list of open issues up to date.
>>>
>>> Thanks,
>>> Valentyn
>>>
>>> [1]:
>>> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=245&view=detail
>>>
>>>
>>> On Tue, Oct 2, 2018 at 9:38 AM Pablo Estrada
>>> <pabloem@google.com <ma...@google.com>> wrote:
>>>
>>> Very cool : ) I'm also available to review / merge if you
>>> need help from my side.
>>> Best
>>> -P.
>>>
>>> On Tue, Oct 2, 2018 at 7:45 AM Rakesh Kumar
>>> <rakeshkumar@lyft.com <ma...@lyft.com>> wrote:
>>>
>>> Hi Rob,
>>>
>>> I am, Rakesh Kumar, using Beam SDK for one of my
>>> projects at Lyft. I have been working closely with
>>> Thomas Weise. I have already met a couple of Python
>>> SDK developers in person.
>>> I am interested to help migrate to Python 3. You can
>>> assign me PRs for review. I am also more than happy
>>> to take a simple ticket to begin development work on
>>> Beam.
>>>
>>> Thank you,
>>> Rakesh
>>>
>>> On Wed, Sep 5, 2018 at 9:12 AM Robbe Sneyders
>>> <robbe.sneyders@ml6.eu
>>> <ma...@ml6.eu>> wrote:
>>>
>>> Hi everyone,
>>>
>>> With the merging of [1], we now have Python 3
>>> tests running on Jenkins, which allows us to move
>>> forward with the last step of the Python 3 porting.
>>>
>>> You can follow the progress on the Jira Kanban
>>> Board [2]. If you're interested in helping by
>>> porting a module, you can assign one of the
>>> issues to yourself and start coding. You can find
>>> the different steps outlined in the design
>>> document [3].
>>>
>>> We could also use some extra reviewers. If you're
>>> interested, let us know, and we'll tag you in our
>>> PRs.
>>>
>>> [1] https://github.com/apache/beam/pull/6266
>>> [2]
>>> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=245
>>> [3] https://s.apache.org/beam-python-3
>>>
>>> kind regards,
>>> Robbe
>>> --
>>>
>>> https://ml6.eu <https://ml6.eu/>
>>>
>>>
>>>
>>>
>>> *Robbe Sneyders*
>>>
>>> ML6 Gent
>>> <https://www.google.be/maps/place/ML6/@51.037408,3.7044893,17z/data=!3m1!4b1!4m5!3m4!1s0x47c37161feeca14b:0xb8f72585fdd21c90!8m2!3d51.037408!4d3.706678?hl=nl>
>>>
>>> M: +32 474 71 31 08 <tel:+32%20474%2071%2031%2008>
>>>
>>> --
>>> Rakesh Kumar
>>> Software Engineer
>>> 510-761-1364 <tel:(510)%20761-1364> |
>>> <https://www.lyft.com/>
>>>