You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Anand Inguva via dev <de...@beam.apache.org> on 2023/04/05 20:12:20 UTC
Re: Python 3.11 support in Apache Beam
Python 3.11 support has been merged at
https://github.com/apache/beam/pull/26121 targeting Beam 2.47.0 release.
Please let me know if you have any questions.
Thanks,
Anand
On Tue, Feb 21, 2023 at 6:04 PM Valentyn Tymofieiev <va...@google.com>
wrote:
> Thanks a lot Anand. I'll take a look at the PRs.
>
> On Tue, Feb 21, 2023 at 1:56 PM Anand Inguva <an...@google.com>
> wrote:
>
>> I was able to spin up a PR: https://github.com/apache/beam/pull/24599
>> that updates the build dependencies of Apache Beam.
>>
>> Several GCP dependencies needed to be updated as well. I covered them in
>> the PR: https://github.com/apache/beam/pull/24599
>>
>> On Thu, Feb 9, 2023 at 3:29 PM Anand Inguva <an...@google.com>
>> wrote:
>>
>>> Yes, we may need to update all of them
>>> <https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt#L28-L33>.
>>> I can add more information once I dig into the issue(most likely next
>>> week). I will comment on my findings on the issue:
>>> https://github.com/apache/beam/issues/24569 and will periodically
>>> update this thread.
>>>
>>> On Tue, Feb 7, 2023 at 5:47 PM Valentyn Tymofieiev <va...@google.com>
>>> wrote:
>>>
>>>> On Tue, Feb 7, 2023 at 2:35 PM Anand Inguva <an...@google.com>
>>>> wrote:
>>>>
>>>>> Yes, it is related to protobuf only. But I think the update of these
>>>>> dependencies are required for Python 3.11 since the newer versions have
>>>>> support for Python 3.11 wheels.
>>>>>
>>>> Assuming you refer to protobuf. Yes, there are no wheels for 3.10 for
>>>> protobuf==3.x.x and that can cause friction.
>>>> https://pypi.org/project/protobuf/3.20.3/#files
>>>>
>>>> I would probably narrow the problem further to demonstrate which stubs
>>>> are not being generated, and if reason not obvious we can also ask for
>>>> feedback from protobuf maintainers. Also - do we by chance need to
>>>> update some other deps from
>>>> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt#L28-L33
>>>> for this to work?
>>>>
>>>> Also: tracking issue for protobuf4 support in Beam:
>>>> https://github.com/apache/beam/issues/24569.
>>>>
>>>> If we use older versions of these packages, then we have to depend on
>>>>> installing those packages on Python 3.11 from source distributions which is
>>>>> not desired.
>>>>>
>>>>> I am working parallely on that issue in a different PR
>>>>> https://github.com/apache/beam/pull/24599 but I think this issue
>>>>> should be a blocker for Python 3.11 update.
>>>>>
>>>>> On Tue, Feb 7, 2023 at 5:25 PM Valentyn Tymofieiev <
>>>>> valentyn@google.com> wrote:
>>>>>
>>>>>> Hi Anand,
>>>>>>
>>>>>> On Tue, Feb 7, 2023 at 1:35 PM Anand Inguva via dev <
>>>>>> dev@beam.apache.org> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> We are planning to work on adding support for Python 3.11[1] to
>>>>>>> Apache Beam Python SDK.
>>>>>>>
>>>>>>> As part of this effort, we are going to update the python build
>>>>>>> dependencies defined at [2].
>>>>>>>
>>>>>>> Right now, there is an error with the newer version of
>>>>>>> protobuf(4.21.11). It is not generating _urn files.
>>>>>>>
>>>>>>> It can be reproduced by
>>>>>>>
>>>>>>
>>>>>>> 1. python setup.py sdist
>>>>>>> 2. pip install dist/apache-beam-x.xx.x.dev0.tar.gz
>>>>>>> 3. switch to python interpreter and run import apache_beam as beam
>>>>>>>
>>>>>> I think the error you are describing is related to protobuf 4, so the
>>>>>> repro should focus on the portion where generation of stubs is happening.
>>>>>> Presumably some stubs are not generated on protobuf 4 + Python 3.11?
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> will lead to *ImportError: cannot import name
>>>>>>> 'beam_runner_api_pb2_urns' from 'apache_beam.portability.api'. *Running
>>>>>>> `python gen_protos.py` to forcefully generate files didn't help either.
>>>>>>>
>>>>>>> If you have encountered this error and found a resolution, please
>>>>>>> let me know(that would be super helpful).
>>>>>>>
>>>>>>> I am going to work on this soon. Please let me know if you want to
>>>>>>> collaborate.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Anand Inguva
>>>>>>>
>>>>>>> *[1] *https://github.com/apache/beam/pull/24721
>>>>>>> [2]
>>>>>>> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt
>>>>>>>
>>>>>>
Re: Python 3.11 support in Apache Beam
Posted by Anand Inguva via dev <de...@beam.apache.org>.
Thanks Ahmet. I will check them out soon.
On Thu, Apr 13, 2023 at 6:24 PM Ahmet Altay <al...@google.com> wrote:
> I forgot to add the link, [1] was meant to be :
> https://docs.python.org/3/whatsnew/3.11.html#faster-cpython
>
> On Thu, Apr 13, 2023 at 10:17 AM Anand Inguva <an...@google.com>
> wrote:
>
>> Yes Ahmet. That would be great.
>>
>> There are some load tests defined in the
>> https://github.com/apache/beam/blob/master/.test-infra which could be
>> useful for performance testing of Beam between 3.10 and 3.11. Do you
>> suggest any other tests?
>>
>
> I have not looked at the full list. I do not think we will see much in IO
> bound pipelines, or pipelines that do most of their work with a C extension
> library already. Maybe some of the load tests like pardo load tests?
>
> If feasible, we could convert benchmarks to run on 3.11 and see which ones
> will see a larger improvement.
>
> Also apparently there is a potential regression of using up to 20% more
> memory (
> https://docs.python.org/3/whatsnew/3.11.html#will-cpython-3-11-use-more-memory).
> I wonder if that will negatively impact us. If feasible, it would be useful
> to understand that as well.
>
>
>>
>> On Wed, Apr 12, 2023 at 8:04 PM Ahmet Altay <al...@google.com> wrote:
>>
>>> Thank you, this is great!
>>>
>>> Python 3.11 announcement had a claim about performance [1]:
>>>
>>> "CPython 3.11 is an average of 25% faster than CPython 3.10 as measured
>>> with the pyperformance benchmark suite, when compiled with GCC on Ubuntu
>>> Linux. Depending on your workload, the overall speedup could be 10-60%."
>>>
>>> Have we measured this in Beam? Are we seeing any benefits? If not, why?
>>> If yes, this would be a cool blog post as well.
>>>
>>> Ahmet
>>>
>>>
>>> On Wed, Apr 5, 2023 at 1:12 PM Anand Inguva via dev <de...@beam.apache.org>
>>> wrote:
>>>
>>>> Python 3.11 support has been merged at
>>>> https://github.com/apache/beam/pull/26121 targeting Beam 2.47.0
>>>> release.
>>>>
>>>> Please let me know if you have any questions.
>>>>
>>>> Thanks,
>>>> Anand
>>>>
>>>> On Tue, Feb 21, 2023 at 6:04 PM Valentyn Tymofieiev <
>>>> valentyn@google.com> wrote:
>>>>
>>>>> Thanks a lot Anand. I'll take a look at the PRs.
>>>>>
>>>>> On Tue, Feb 21, 2023 at 1:56 PM Anand Inguva <an...@google.com>
>>>>> wrote:
>>>>>
>>>>>> I was able to spin up a PR: https://github.com/apache/beam/pull/24599
>>>>>> that updates the build dependencies of Apache Beam.
>>>>>>
>>>>>> Several GCP dependencies needed to be updated as well. I covered them
>>>>>> in the PR: https://github.com/apache/beam/pull/24599
>>>>>>
>>>>>> On Thu, Feb 9, 2023 at 3:29 PM Anand Inguva <an...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Yes, we may need to update all of them
>>>>>>> <https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt#L28-L33>.
>>>>>>> I can add more information once I dig into the issue(most likely next
>>>>>>> week). I will comment on my findings on the issue:
>>>>>>> https://github.com/apache/beam/issues/24569 and will periodically
>>>>>>> update this thread.
>>>>>>>
>>>>>>> On Tue, Feb 7, 2023 at 5:47 PM Valentyn Tymofieiev <
>>>>>>> valentyn@google.com> wrote:
>>>>>>>
>>>>>>>> On Tue, Feb 7, 2023 at 2:35 PM Anand Inguva <an...@google.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Yes, it is related to protobuf only. But I think the update of
>>>>>>>>> these dependencies are required for Python 3.11 since the newer versions
>>>>>>>>> have support for Python 3.11 wheels.
>>>>>>>>>
>>>>>>>> Assuming you refer to protobuf. Yes, there are no wheels for 3.10
>>>>>>>> for protobuf==3.x.x and that can cause friction.
>>>>>>>> https://pypi.org/project/protobuf/3.20.3/#files
>>>>>>>>
>>>>>>>> I would probably narrow the problem further to demonstrate which
>>>>>>>> stubs are not being generated, and if reason not obvious we can also ask
>>>>>>>> for feedback from protobuf maintainers. Also - do we by chance need to
>>>>>>>> update some other deps from
>>>>>>>> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt#L28-L33
>>>>>>>> for this to work?
>>>>>>>>
>>>>>>>> Also: tracking issue for protobuf4 support in Beam:
>>>>>>>> https://github.com/apache/beam/issues/24569.
>>>>>>>>
>>>>>>>> If we use older versions of these packages, then we have to depend
>>>>>>>>> on installing those packages on Python 3.11 from source distributions which
>>>>>>>>> is not desired.
>>>>>>>>>
>>>>>>>>> I am working parallely on that issue in a different PR
>>>>>>>>> https://github.com/apache/beam/pull/24599 but I think this issue
>>>>>>>>> should be a blocker for Python 3.11 update.
>>>>>>>>>
>>>>>>>>> On Tue, Feb 7, 2023 at 5:25 PM Valentyn Tymofieiev <
>>>>>>>>> valentyn@google.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Anand,
>>>>>>>>>>
>>>>>>>>>> On Tue, Feb 7, 2023 at 1:35 PM Anand Inguva via dev <
>>>>>>>>>> dev@beam.apache.org> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi all,
>>>>>>>>>>>
>>>>>>>>>>> We are planning to work on adding support for Python 3.11[1] to
>>>>>>>>>>> Apache Beam Python SDK.
>>>>>>>>>>>
>>>>>>>>>>> As part of this effort, we are going to update the python build
>>>>>>>>>>> dependencies defined at [2].
>>>>>>>>>>>
>>>>>>>>>>> Right now, there is an error with the newer version of
>>>>>>>>>>> protobuf(4.21.11). It is not generating _urn files.
>>>>>>>>>>>
>>>>>>>>>>> It can be reproduced by
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> 1. python setup.py sdist
>>>>>>>>>>> 2. pip install dist/apache-beam-x.xx.x.dev0.tar.gz
>>>>>>>>>>> 3. switch to python interpreter and run import apache_beam as
>>>>>>>>>>> beam
>>>>>>>>>>>
>>>>>>>>>> I think the error you are describing is related to protobuf 4, so
>>>>>>>>>> the repro should focus on the portion where generation of stubs is
>>>>>>>>>> happening. Presumably some stubs are not generated on protobuf 4 + Python
>>>>>>>>>> 3.11?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> will lead to *ImportError: cannot import name
>>>>>>>>>>> 'beam_runner_api_pb2_urns' from 'apache_beam.portability.api'. *Running
>>>>>>>>>>> `python gen_protos.py` to forcefully generate files didn't help either.
>>>>>>>>>>>
>>>>>>>>>>> If you have encountered this error and found a resolution,
>>>>>>>>>>> please let me know(that would be super helpful).
>>>>>>>>>>>
>>>>>>>>>>> I am going to work on this soon. Please let me know if you want
>>>>>>>>>>> to collaborate.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Anand Inguva
>>>>>>>>>>>
>>>>>>>>>>> *[1] *https://github.com/apache/beam/pull/24721
>>>>>>>>>>> [2]
>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt
>>>>>>>>>>>
>>>>>>>>>>
Re: Python 3.11 support in Apache Beam
Posted by Ahmet Altay via dev <de...@beam.apache.org>.
I forgot to add the link, [1] was meant to be :
https://docs.python.org/3/whatsnew/3.11.html#faster-cpython
On Thu, Apr 13, 2023 at 10:17 AM Anand Inguva <an...@google.com>
wrote:
> Yes Ahmet. That would be great.
>
> There are some load tests defined in the
> https://github.com/apache/beam/blob/master/.test-infra which could be
> useful for performance testing of Beam between 3.10 and 3.11. Do you
> suggest any other tests?
>
I have not looked at the full list. I do not think we will see much in IO
bound pipelines, or pipelines that do most of their work with a C extension
library already. Maybe some of the load tests like pardo load tests?
If feasible, we could convert benchmarks to run on 3.11 and see which ones
will see a larger improvement.
Also apparently there is a potential regression of using up to 20% more
memory (
https://docs.python.org/3/whatsnew/3.11.html#will-cpython-3-11-use-more-memory).
I wonder if that will negatively impact us. If feasible, it would be useful
to understand that as well.
>
> On Wed, Apr 12, 2023 at 8:04 PM Ahmet Altay <al...@google.com> wrote:
>
>> Thank you, this is great!
>>
>> Python 3.11 announcement had a claim about performance [1]:
>>
>> "CPython 3.11 is an average of 25% faster than CPython 3.10 as measured
>> with the pyperformance benchmark suite, when compiled with GCC on Ubuntu
>> Linux. Depending on your workload, the overall speedup could be 10-60%."
>>
>> Have we measured this in Beam? Are we seeing any benefits? If not, why?
>> If yes, this would be a cool blog post as well.
>>
>> Ahmet
>>
>>
>> On Wed, Apr 5, 2023 at 1:12 PM Anand Inguva via dev <de...@beam.apache.org>
>> wrote:
>>
>>> Python 3.11 support has been merged at
>>> https://github.com/apache/beam/pull/26121 targeting Beam 2.47.0
>>> release.
>>>
>>> Please let me know if you have any questions.
>>>
>>> Thanks,
>>> Anand
>>>
>>> On Tue, Feb 21, 2023 at 6:04 PM Valentyn Tymofieiev <va...@google.com>
>>> wrote:
>>>
>>>> Thanks a lot Anand. I'll take a look at the PRs.
>>>>
>>>> On Tue, Feb 21, 2023 at 1:56 PM Anand Inguva <an...@google.com>
>>>> wrote:
>>>>
>>>>> I was able to spin up a PR: https://github.com/apache/beam/pull/24599
>>>>> that updates the build dependencies of Apache Beam.
>>>>>
>>>>> Several GCP dependencies needed to be updated as well. I covered them
>>>>> in the PR: https://github.com/apache/beam/pull/24599
>>>>>
>>>>> On Thu, Feb 9, 2023 at 3:29 PM Anand Inguva <an...@google.com>
>>>>> wrote:
>>>>>
>>>>>> Yes, we may need to update all of them
>>>>>> <https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt#L28-L33>.
>>>>>> I can add more information once I dig into the issue(most likely next
>>>>>> week). I will comment on my findings on the issue:
>>>>>> https://github.com/apache/beam/issues/24569 and will periodically
>>>>>> update this thread.
>>>>>>
>>>>>> On Tue, Feb 7, 2023 at 5:47 PM Valentyn Tymofieiev <
>>>>>> valentyn@google.com> wrote:
>>>>>>
>>>>>>> On Tue, Feb 7, 2023 at 2:35 PM Anand Inguva <an...@google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Yes, it is related to protobuf only. But I think the update of
>>>>>>>> these dependencies are required for Python 3.11 since the newer versions
>>>>>>>> have support for Python 3.11 wheels.
>>>>>>>>
>>>>>>> Assuming you refer to protobuf. Yes, there are no wheels for 3.10
>>>>>>> for protobuf==3.x.x and that can cause friction.
>>>>>>> https://pypi.org/project/protobuf/3.20.3/#files
>>>>>>>
>>>>>>> I would probably narrow the problem further to demonstrate which
>>>>>>> stubs are not being generated, and if reason not obvious we can also ask
>>>>>>> for feedback from protobuf maintainers. Also - do we by chance need to
>>>>>>> update some other deps from
>>>>>>> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt#L28-L33
>>>>>>> for this to work?
>>>>>>>
>>>>>>> Also: tracking issue for protobuf4 support in Beam:
>>>>>>> https://github.com/apache/beam/issues/24569.
>>>>>>>
>>>>>>> If we use older versions of these packages, then we have to depend
>>>>>>>> on installing those packages on Python 3.11 from source distributions which
>>>>>>>> is not desired.
>>>>>>>>
>>>>>>>> I am working parallely on that issue in a different PR
>>>>>>>> https://github.com/apache/beam/pull/24599 but I think this issue
>>>>>>>> should be a blocker for Python 3.11 update.
>>>>>>>>
>>>>>>>> On Tue, Feb 7, 2023 at 5:25 PM Valentyn Tymofieiev <
>>>>>>>> valentyn@google.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Anand,
>>>>>>>>>
>>>>>>>>> On Tue, Feb 7, 2023 at 1:35 PM Anand Inguva via dev <
>>>>>>>>> dev@beam.apache.org> wrote:
>>>>>>>>>
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>> We are planning to work on adding support for Python 3.11[1] to
>>>>>>>>>> Apache Beam Python SDK.
>>>>>>>>>>
>>>>>>>>>> As part of this effort, we are going to update the python build
>>>>>>>>>> dependencies defined at [2].
>>>>>>>>>>
>>>>>>>>>> Right now, there is an error with the newer version of
>>>>>>>>>> protobuf(4.21.11). It is not generating _urn files.
>>>>>>>>>>
>>>>>>>>>> It can be reproduced by
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> 1. python setup.py sdist
>>>>>>>>>> 2. pip install dist/apache-beam-x.xx.x.dev0.tar.gz
>>>>>>>>>> 3. switch to python interpreter and run import apache_beam as beam
>>>>>>>>>>
>>>>>>>>> I think the error you are describing is related to protobuf 4, so
>>>>>>>>> the repro should focus on the portion where generation of stubs is
>>>>>>>>> happening. Presumably some stubs are not generated on protobuf 4 + Python
>>>>>>>>> 3.11?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> will lead to *ImportError: cannot import name
>>>>>>>>>> 'beam_runner_api_pb2_urns' from 'apache_beam.portability.api'. *Running
>>>>>>>>>> `python gen_protos.py` to forcefully generate files didn't help either.
>>>>>>>>>>
>>>>>>>>>> If you have encountered this error and found a resolution, please
>>>>>>>>>> let me know(that would be super helpful).
>>>>>>>>>>
>>>>>>>>>> I am going to work on this soon. Please let me know if you want
>>>>>>>>>> to collaborate.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Anand Inguva
>>>>>>>>>>
>>>>>>>>>> *[1] *https://github.com/apache/beam/pull/24721
>>>>>>>>>> [2]
>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt
>>>>>>>>>>
>>>>>>>>>
Re: Python 3.11 support in Apache Beam
Posted by Anand Inguva via dev <de...@beam.apache.org>.
Yes Ahmet. That would be great.
There are some load tests defined in the
https://github.com/apache/beam/blob/master/.test-infra which could be
useful for performance testing of Beam between 3.10 and 3.11. Do you
suggest any other tests?
On Wed, Apr 12, 2023 at 8:04 PM Ahmet Altay <al...@google.com> wrote:
> Thank you, this is great!
>
> Python 3.11 announcement had a claim about performance [1]:
>
> "CPython 3.11 is an average of 25% faster than CPython 3.10 as measured
> with the pyperformance benchmark suite, when compiled with GCC on Ubuntu
> Linux. Depending on your workload, the overall speedup could be 10-60%."
>
> Have we measured this in Beam? Are we seeing any benefits? If not, why? If
> yes, this would be a cool blog post as well.
>
> Ahmet
>
>
> On Wed, Apr 5, 2023 at 1:12 PM Anand Inguva via dev <de...@beam.apache.org>
> wrote:
>
>> Python 3.11 support has been merged at
>> https://github.com/apache/beam/pull/26121 targeting Beam 2.47.0 release.
>>
>> Please let me know if you have any questions.
>>
>> Thanks,
>> Anand
>>
>> On Tue, Feb 21, 2023 at 6:04 PM Valentyn Tymofieiev <va...@google.com>
>> wrote:
>>
>>> Thanks a lot Anand. I'll take a look at the PRs.
>>>
>>> On Tue, Feb 21, 2023 at 1:56 PM Anand Inguva <an...@google.com>
>>> wrote:
>>>
>>>> I was able to spin up a PR: https://github.com/apache/beam/pull/24599
>>>> that updates the build dependencies of Apache Beam.
>>>>
>>>> Several GCP dependencies needed to be updated as well. I covered them
>>>> in the PR: https://github.com/apache/beam/pull/24599
>>>>
>>>> On Thu, Feb 9, 2023 at 3:29 PM Anand Inguva <an...@google.com>
>>>> wrote:
>>>>
>>>>> Yes, we may need to update all of them
>>>>> <https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt#L28-L33>.
>>>>> I can add more information once I dig into the issue(most likely next
>>>>> week). I will comment on my findings on the issue:
>>>>> https://github.com/apache/beam/issues/24569 and will periodically
>>>>> update this thread.
>>>>>
>>>>> On Tue, Feb 7, 2023 at 5:47 PM Valentyn Tymofieiev <
>>>>> valentyn@google.com> wrote:
>>>>>
>>>>>> On Tue, Feb 7, 2023 at 2:35 PM Anand Inguva <an...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Yes, it is related to protobuf only. But I think the update of these
>>>>>>> dependencies are required for Python 3.11 since the newer versions have
>>>>>>> support for Python 3.11 wheels.
>>>>>>>
>>>>>> Assuming you refer to protobuf. Yes, there are no wheels for 3.10 for
>>>>>> protobuf==3.x.x and that can cause friction.
>>>>>> https://pypi.org/project/protobuf/3.20.3/#files
>>>>>>
>>>>>> I would probably narrow the problem further to demonstrate which
>>>>>> stubs are not being generated, and if reason not obvious we can also ask
>>>>>> for feedback from protobuf maintainers. Also - do we by chance need to
>>>>>> update some other deps from
>>>>>> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt#L28-L33
>>>>>> for this to work?
>>>>>>
>>>>>> Also: tracking issue for protobuf4 support in Beam:
>>>>>> https://github.com/apache/beam/issues/24569.
>>>>>>
>>>>>> If we use older versions of these packages, then we have to depend on
>>>>>>> installing those packages on Python 3.11 from source distributions which is
>>>>>>> not desired.
>>>>>>>
>>>>>>> I am working parallely on that issue in a different PR
>>>>>>> https://github.com/apache/beam/pull/24599 but I think this issue
>>>>>>> should be a blocker for Python 3.11 update.
>>>>>>>
>>>>>>> On Tue, Feb 7, 2023 at 5:25 PM Valentyn Tymofieiev <
>>>>>>> valentyn@google.com> wrote:
>>>>>>>
>>>>>>>> Hi Anand,
>>>>>>>>
>>>>>>>> On Tue, Feb 7, 2023 at 1:35 PM Anand Inguva via dev <
>>>>>>>> dev@beam.apache.org> wrote:
>>>>>>>>
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> We are planning to work on adding support for Python 3.11[1] to
>>>>>>>>> Apache Beam Python SDK.
>>>>>>>>>
>>>>>>>>> As part of this effort, we are going to update the python build
>>>>>>>>> dependencies defined at [2].
>>>>>>>>>
>>>>>>>>> Right now, there is an error with the newer version of
>>>>>>>>> protobuf(4.21.11). It is not generating _urn files.
>>>>>>>>>
>>>>>>>>> It can be reproduced by
>>>>>>>>>
>>>>>>>>
>>>>>>>>> 1. python setup.py sdist
>>>>>>>>> 2. pip install dist/apache-beam-x.xx.x.dev0.tar.gz
>>>>>>>>> 3. switch to python interpreter and run import apache_beam as beam
>>>>>>>>>
>>>>>>>> I think the error you are describing is related to protobuf 4, so
>>>>>>>> the repro should focus on the portion where generation of stubs is
>>>>>>>> happening. Presumably some stubs are not generated on protobuf 4 + Python
>>>>>>>> 3.11?
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> will lead to *ImportError: cannot import name
>>>>>>>>> 'beam_runner_api_pb2_urns' from 'apache_beam.portability.api'. *Running
>>>>>>>>> `python gen_protos.py` to forcefully generate files didn't help either.
>>>>>>>>>
>>>>>>>>> If you have encountered this error and found a resolution, please
>>>>>>>>> let me know(that would be super helpful).
>>>>>>>>>
>>>>>>>>> I am going to work on this soon. Please let me know if you want to
>>>>>>>>> collaborate.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Anand Inguva
>>>>>>>>>
>>>>>>>>> *[1] *https://github.com/apache/beam/pull/24721
>>>>>>>>> [2]
>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt
>>>>>>>>>
>>>>>>>>
Re: Python 3.11 support in Apache Beam
Posted by Ahmet Altay via dev <de...@beam.apache.org>.
Thank you, this is great!
Python 3.11 announcement had a claim about performance [1]:
"CPython 3.11 is an average of 25% faster than CPython 3.10 as measured
with the pyperformance benchmark suite, when compiled with GCC on Ubuntu
Linux. Depending on your workload, the overall speedup could be 10-60%."
Have we measured this in Beam? Are we seeing any benefits? If not, why? If
yes, this would be a cool blog post as well.
Ahmet
On Wed, Apr 5, 2023 at 1:12 PM Anand Inguva via dev <de...@beam.apache.org>
wrote:
> Python 3.11 support has been merged at
> https://github.com/apache/beam/pull/26121 targeting Beam 2.47.0 release.
>
> Please let me know if you have any questions.
>
> Thanks,
> Anand
>
> On Tue, Feb 21, 2023 at 6:04 PM Valentyn Tymofieiev <va...@google.com>
> wrote:
>
>> Thanks a lot Anand. I'll take a look at the PRs.
>>
>> On Tue, Feb 21, 2023 at 1:56 PM Anand Inguva <an...@google.com>
>> wrote:
>>
>>> I was able to spin up a PR: https://github.com/apache/beam/pull/24599
>>> that updates the build dependencies of Apache Beam.
>>>
>>> Several GCP dependencies needed to be updated as well. I covered them in
>>> the PR: https://github.com/apache/beam/pull/24599
>>>
>>> On Thu, Feb 9, 2023 at 3:29 PM Anand Inguva <an...@google.com>
>>> wrote:
>>>
>>>> Yes, we may need to update all of them
>>>> <https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt#L28-L33>.
>>>> I can add more information once I dig into the issue(most likely next
>>>> week). I will comment on my findings on the issue:
>>>> https://github.com/apache/beam/issues/24569 and will periodically
>>>> update this thread.
>>>>
>>>> On Tue, Feb 7, 2023 at 5:47 PM Valentyn Tymofieiev <va...@google.com>
>>>> wrote:
>>>>
>>>>> On Tue, Feb 7, 2023 at 2:35 PM Anand Inguva <an...@google.com>
>>>>> wrote:
>>>>>
>>>>>> Yes, it is related to protobuf only. But I think the update of these
>>>>>> dependencies are required for Python 3.11 since the newer versions have
>>>>>> support for Python 3.11 wheels.
>>>>>>
>>>>> Assuming you refer to protobuf. Yes, there are no wheels for 3.10 for
>>>>> protobuf==3.x.x and that can cause friction.
>>>>> https://pypi.org/project/protobuf/3.20.3/#files
>>>>>
>>>>> I would probably narrow the problem further to demonstrate which stubs
>>>>> are not being generated, and if reason not obvious we can also ask for
>>>>> feedback from protobuf maintainers. Also - do we by chance need to
>>>>> update some other deps from
>>>>> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt#L28-L33
>>>>> for this to work?
>>>>>
>>>>> Also: tracking issue for protobuf4 support in Beam:
>>>>> https://github.com/apache/beam/issues/24569.
>>>>>
>>>>> If we use older versions of these packages, then we have to depend on
>>>>>> installing those packages on Python 3.11 from source distributions which is
>>>>>> not desired.
>>>>>>
>>>>>> I am working parallely on that issue in a different PR
>>>>>> https://github.com/apache/beam/pull/24599 but I think this issue
>>>>>> should be a blocker for Python 3.11 update.
>>>>>>
>>>>>> On Tue, Feb 7, 2023 at 5:25 PM Valentyn Tymofieiev <
>>>>>> valentyn@google.com> wrote:
>>>>>>
>>>>>>> Hi Anand,
>>>>>>>
>>>>>>> On Tue, Feb 7, 2023 at 1:35 PM Anand Inguva via dev <
>>>>>>> dev@beam.apache.org> wrote:
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> We are planning to work on adding support for Python 3.11[1] to
>>>>>>>> Apache Beam Python SDK.
>>>>>>>>
>>>>>>>> As part of this effort, we are going to update the python build
>>>>>>>> dependencies defined at [2].
>>>>>>>>
>>>>>>>> Right now, there is an error with the newer version of
>>>>>>>> protobuf(4.21.11). It is not generating _urn files.
>>>>>>>>
>>>>>>>> It can be reproduced by
>>>>>>>>
>>>>>>>
>>>>>>>> 1. python setup.py sdist
>>>>>>>> 2. pip install dist/apache-beam-x.xx.x.dev0.tar.gz
>>>>>>>> 3. switch to python interpreter and run import apache_beam as beam
>>>>>>>>
>>>>>>> I think the error you are describing is related to protobuf 4, so
>>>>>>> the repro should focus on the portion where generation of stubs is
>>>>>>> happening. Presumably some stubs are not generated on protobuf 4 + Python
>>>>>>> 3.11?
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> will lead to *ImportError: cannot import name
>>>>>>>> 'beam_runner_api_pb2_urns' from 'apache_beam.portability.api'. *Running
>>>>>>>> `python gen_protos.py` to forcefully generate files didn't help either.
>>>>>>>>
>>>>>>>> If you have encountered this error and found a resolution, please
>>>>>>>> let me know(that would be super helpful).
>>>>>>>>
>>>>>>>> I am going to work on this soon. Please let me know if you want to
>>>>>>>> collaborate.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Anand Inguva
>>>>>>>>
>>>>>>>> *[1] *https://github.com/apache/beam/pull/24721
>>>>>>>> [2]
>>>>>>>> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt
>>>>>>>>
>>>>>>>