You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Anand Inguva via dev <de...@beam.apache.org> on 2023/02/07 21:34:53 UTC
Python 3.11 support in Apache Beam
Hi all,
We are planning to work on adding support for Python 3.11[1] to Apache Beam
Python SDK.
As part of this effort, we are going to update the python build
dependencies defined at [2].
Right now, there is an error with the newer version of protobuf(4.21.11).
It is not generating _urn files.
It can be reproduced by
1. python setup.py sdist
2. pip install dist/apache-beam-x.xx.x.dev0.tar.gz
3. switch to python interpreter and run import apache_beam as beam
will lead to *ImportError: cannot import name 'beam_runner_api_pb2_urns'
from 'apache_beam.portability.api'. *Running `python gen_protos.py` to
forcefully generate files didn't help either.
If you have encountered this error and found a resolution, please let me
know(that would be super helpful).
I am going to work on this soon. Please let me know if you want to
collaborate.
Thanks,
Anand Inguva
*[1] *https://github.com/apache/beam/pull/24721
[2]
https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt
Re: Python 3.11 support in Apache Beam
Posted by Anand Inguva via dev <de...@beam.apache.org>.
Thanks Ahmet. I will check them out soon.
On Thu, Apr 13, 2023 at 6:24 PM Ahmet Altay <al...@google.com> wrote:
> I forgot to add the link, [1] was meant to be :
> https://docs.python.org/3/whatsnew/3.11.html#faster-cpython
>
> On Thu, Apr 13, 2023 at 10:17 AM Anand Inguva <an...@google.com>
> wrote:
>
>> Yes Ahmet. That would be great.
>>
>> There are some load tests defined in the
>> https://github.com/apache/beam/blob/master/.test-infra which could be
>> useful for performance testing of Beam between 3.10 and 3.11. Do you
>> suggest any other tests?
>>
>
> I have not looked at the full list. I do not think we will see much in IO
> bound pipelines, or pipelines that do most of their work with a C extension
> library already. Maybe some of the load tests like pardo load tests?
>
> If feasible, we could convert benchmarks to run on 3.11 and see which ones
> will see a larger improvement.
>
> Also apparently there is a potential regression of using up to 20% more
> memory (
> https://docs.python.org/3/whatsnew/3.11.html#will-cpython-3-11-use-more-memory).
> I wonder if that will negatively impact us. If feasible, it would be useful
> to understand that as well.
>
>
>>
>> On Wed, Apr 12, 2023 at 8:04 PM Ahmet Altay <al...@google.com> wrote:
>>
>>> Thank you, this is great!
>>>
>>> Python 3.11 announcement had a claim about performance [1]:
>>>
>>> "CPython 3.11 is an average of 25% faster than CPython 3.10 as measured
>>> with the pyperformance benchmark suite, when compiled with GCC on Ubuntu
>>> Linux. Depending on your workload, the overall speedup could be 10-60%."
>>>
>>> Have we measured this in Beam? Are we seeing any benefits? If not, why?
>>> If yes, this would be a cool blog post as well.
>>>
>>> Ahmet
>>>
>>>
>>> On Wed, Apr 5, 2023 at 1:12 PM Anand Inguva via dev <de...@beam.apache.org>
>>> wrote:
>>>
>>>> Python 3.11 support has been merged at
>>>> https://github.com/apache/beam/pull/26121 targeting Beam 2.47.0
>>>> release.
>>>>
>>>> Please let me know if you have any questions.
>>>>
>>>> Thanks,
>>>> Anand
>>>>
>>>> On Tue, Feb 21, 2023 at 6:04 PM Valentyn Tymofieiev <
>>>> valentyn@google.com> wrote:
>>>>
>>>>> Thanks a lot Anand. I'll take a look at the PRs.
>>>>>
>>>>> On Tue, Feb 21, 2023 at 1:56 PM Anand Inguva <an...@google.com>
>>>>> wrote:
>>>>>
>>>>>> I was able to spin up a PR: https://github.com/apache/beam/pull/24599
>>>>>> that updates the build dependencies of Apache Beam.
>>>>>>
>>>>>> Several GCP dependencies needed to be updated as well. I covered them
>>>>>> in the PR: https://github.com/apache/beam/pull/24599
>>>>>>
>>>>>> On Thu, Feb 9, 2023 at 3:29 PM Anand Inguva <an...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Yes, we may need to update all of them
>>>>>>> <https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt#L28-L33>.
>>>>>>> I can add more information once I dig into the issue(most likely next
>>>>>>> week). I will comment on my findings on the issue:
>>>>>>> https://github.com/apache/beam/issues/24569 and will periodically
>>>>>>> update this thread.
>>>>>>>
>>>>>>> On Tue, Feb 7, 2023 at 5:47 PM Valentyn Tymofieiev <
>>>>>>> valentyn@google.com> wrote:
>>>>>>>
>>>>>>>> On Tue, Feb 7, 2023 at 2:35 PM Anand Inguva <an...@google.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Yes, it is related to protobuf only. But I think the update of
>>>>>>>>> these dependencies are required for Python 3.11 since the newer versions
>>>>>>>>> have support for Python 3.11 wheels.
>>>>>>>>>
>>>>>>>> Assuming you refer to protobuf. Yes, there are no wheels for 3.10
>>>>>>>> for protobuf==3.x.x and that can cause friction.
>>>>>>>> https://pypi.org/project/protobuf/3.20.3/#files
>>>>>>>>
>>>>>>>> I would probably narrow the problem further to demonstrate which
>>>>>>>> stubs are not being generated, and if reason not obvious we can also ask
>>>>>>>> for feedback from protobuf maintainers. Also - do we by chance need to
>>>>>>>> update some other deps from
>>>>>>>> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt#L28-L33
>>>>>>>> for this to work?
>>>>>>>>
>>>>>>>> Also: tracking issue for protobuf4 support in Beam:
>>>>>>>> https://github.com/apache/beam/issues/24569.
>>>>>>>>
>>>>>>>> If we use older versions of these packages, then we have to depend
>>>>>>>>> on installing those packages on Python 3.11 from source distributions which
>>>>>>>>> is not desired.
>>>>>>>>>
>>>>>>>>> I am working parallely on that issue in a different PR
>>>>>>>>> https://github.com/apache/beam/pull/24599 but I think this issue
>>>>>>>>> should be a blocker for Python 3.11 update.
>>>>>>>>>
>>>>>>>>> On Tue, Feb 7, 2023 at 5:25 PM Valentyn Tymofieiev <
>>>>>>>>> valentyn@google.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Anand,
>>>>>>>>>>
>>>>>>>>>> On Tue, Feb 7, 2023 at 1:35 PM Anand Inguva via dev <
>>>>>>>>>> dev@beam.apache.org> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi all,
>>>>>>>>>>>
>>>>>>>>>>> We are planning to work on adding support for Python 3.11[1] to
>>>>>>>>>>> Apache Beam Python SDK.
>>>>>>>>>>>
>>>>>>>>>>> As part of this effort, we are going to update the python build
>>>>>>>>>>> dependencies defined at [2].
>>>>>>>>>>>
>>>>>>>>>>> Right now, there is an error with the newer version of
>>>>>>>>>>> protobuf(4.21.11). It is not generating _urn files.
>>>>>>>>>>>
>>>>>>>>>>> It can be reproduced by
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> 1. python setup.py sdist
>>>>>>>>>>> 2. pip install dist/apache-beam-x.xx.x.dev0.tar.gz
>>>>>>>>>>> 3. switch to python interpreter and run import apache_beam as
>>>>>>>>>>> beam
>>>>>>>>>>>
>>>>>>>>>> I think the error you are describing is related to protobuf 4, so
>>>>>>>>>> the repro should focus on the portion where generation of stubs is
>>>>>>>>>> happening. Presumably some stubs are not generated on protobuf 4 + Python
>>>>>>>>>> 3.11?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> will lead to *ImportError: cannot import name
>>>>>>>>>>> 'beam_runner_api_pb2_urns' from 'apache_beam.portability.api'. *Running
>>>>>>>>>>> `python gen_protos.py` to forcefully generate files didn't help either.
>>>>>>>>>>>
>>>>>>>>>>> If you have encountered this error and found a resolution,
>>>>>>>>>>> please let me know(that would be super helpful).
>>>>>>>>>>>
>>>>>>>>>>> I am going to work on this soon. Please let me know if you want
>>>>>>>>>>> to collaborate.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Anand Inguva
>>>>>>>>>>>
>>>>>>>>>>> *[1] *https://github.com/apache/beam/pull/24721
>>>>>>>>>>> [2]
>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt
>>>>>>>>>>>
>>>>>>>>>>
Re: Python 3.11 support in Apache Beam
Posted by Ahmet Altay via dev <de...@beam.apache.org>.
I forgot to add the link, [1] was meant to be :
https://docs.python.org/3/whatsnew/3.11.html#faster-cpython
On Thu, Apr 13, 2023 at 10:17 AM Anand Inguva <an...@google.com>
wrote:
> Yes Ahmet. That would be great.
>
> There are some load tests defined in the
> https://github.com/apache/beam/blob/master/.test-infra which could be
> useful for performance testing of Beam between 3.10 and 3.11. Do you
> suggest any other tests?
>
I have not looked at the full list. I do not think we will see much in IO
bound pipelines, or pipelines that do most of their work with a C extension
library already. Maybe some of the load tests like pardo load tests?
If feasible, we could convert benchmarks to run on 3.11 and see which ones
will see a larger improvement.
Also apparently there is a potential regression of using up to 20% more
memory (
https://docs.python.org/3/whatsnew/3.11.html#will-cpython-3-11-use-more-memory).
I wonder if that will negatively impact us. If feasible, it would be useful
to understand that as well.
>
> On Wed, Apr 12, 2023 at 8:04 PM Ahmet Altay <al...@google.com> wrote:
>
>> Thank you, this is great!
>>
>> Python 3.11 announcement had a claim about performance [1]:
>>
>> "CPython 3.11 is an average of 25% faster than CPython 3.10 as measured
>> with the pyperformance benchmark suite, when compiled with GCC on Ubuntu
>> Linux. Depending on your workload, the overall speedup could be 10-60%."
>>
>> Have we measured this in Beam? Are we seeing any benefits? If not, why?
>> If yes, this would be a cool blog post as well.
>>
>> Ahmet
>>
>>
>> On Wed, Apr 5, 2023 at 1:12 PM Anand Inguva via dev <de...@beam.apache.org>
>> wrote:
>>
>>> Python 3.11 support has been merged at
>>> https://github.com/apache/beam/pull/26121 targeting Beam 2.47.0
>>> release.
>>>
>>> Please let me know if you have any questions.
>>>
>>> Thanks,
>>> Anand
>>>
>>> On Tue, Feb 21, 2023 at 6:04 PM Valentyn Tymofieiev <va...@google.com>
>>> wrote:
>>>
>>>> Thanks a lot Anand. I'll take a look at the PRs.
>>>>
>>>> On Tue, Feb 21, 2023 at 1:56 PM Anand Inguva <an...@google.com>
>>>> wrote:
>>>>
>>>>> I was able to spin up a PR: https://github.com/apache/beam/pull/24599
>>>>> that updates the build dependencies of Apache Beam.
>>>>>
>>>>> Several GCP dependencies needed to be updated as well. I covered them
>>>>> in the PR: https://github.com/apache/beam/pull/24599
>>>>>
>>>>> On Thu, Feb 9, 2023 at 3:29 PM Anand Inguva <an...@google.com>
>>>>> wrote:
>>>>>
>>>>>> Yes, we may need to update all of them
>>>>>> <https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt#L28-L33>.
>>>>>> I can add more information once I dig into the issue(most likely next
>>>>>> week). I will comment on my findings on the issue:
>>>>>> https://github.com/apache/beam/issues/24569 and will periodically
>>>>>> update this thread.
>>>>>>
>>>>>> On Tue, Feb 7, 2023 at 5:47 PM Valentyn Tymofieiev <
>>>>>> valentyn@google.com> wrote:
>>>>>>
>>>>>>> On Tue, Feb 7, 2023 at 2:35 PM Anand Inguva <an...@google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Yes, it is related to protobuf only. But I think the update of
>>>>>>>> these dependencies are required for Python 3.11 since the newer versions
>>>>>>>> have support for Python 3.11 wheels.
>>>>>>>>
>>>>>>> Assuming you refer to protobuf. Yes, there are no wheels for 3.10
>>>>>>> for protobuf==3.x.x and that can cause friction.
>>>>>>> https://pypi.org/project/protobuf/3.20.3/#files
>>>>>>>
>>>>>>> I would probably narrow the problem further to demonstrate which
>>>>>>> stubs are not being generated, and if reason not obvious we can also ask
>>>>>>> for feedback from protobuf maintainers. Also - do we by chance need to
>>>>>>> update some other deps from
>>>>>>> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt#L28-L33
>>>>>>> for this to work?
>>>>>>>
>>>>>>> Also: tracking issue for protobuf4 support in Beam:
>>>>>>> https://github.com/apache/beam/issues/24569.
>>>>>>>
>>>>>>> If we use older versions of these packages, then we have to depend
>>>>>>>> on installing those packages on Python 3.11 from source distributions which
>>>>>>>> is not desired.
>>>>>>>>
>>>>>>>> I am working parallely on that issue in a different PR
>>>>>>>> https://github.com/apache/beam/pull/24599 but I think this issue
>>>>>>>> should be a blocker for Python 3.11 update.
>>>>>>>>
>>>>>>>> On Tue, Feb 7, 2023 at 5:25 PM Valentyn Tymofieiev <
>>>>>>>> valentyn@google.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Anand,
>>>>>>>>>
>>>>>>>>> On Tue, Feb 7, 2023 at 1:35 PM Anand Inguva via dev <
>>>>>>>>> dev@beam.apache.org> wrote:
>>>>>>>>>
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>> We are planning to work on adding support for Python 3.11[1] to
>>>>>>>>>> Apache Beam Python SDK.
>>>>>>>>>>
>>>>>>>>>> As part of this effort, we are going to update the python build
>>>>>>>>>> dependencies defined at [2].
>>>>>>>>>>
>>>>>>>>>> Right now, there is an error with the newer version of
>>>>>>>>>> protobuf(4.21.11). It is not generating _urn files.
>>>>>>>>>>
>>>>>>>>>> It can be reproduced by
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> 1. python setup.py sdist
>>>>>>>>>> 2. pip install dist/apache-beam-x.xx.x.dev0.tar.gz
>>>>>>>>>> 3. switch to python interpreter and run import apache_beam as beam
>>>>>>>>>>
>>>>>>>>> I think the error you are describing is related to protobuf 4, so
>>>>>>>>> the repro should focus on the portion where generation of stubs is
>>>>>>>>> happening. Presumably some stubs are not generated on protobuf 4 + Python
>>>>>>>>> 3.11?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> will lead to *ImportError: cannot import name
>>>>>>>>>> 'beam_runner_api_pb2_urns' from 'apache_beam.portability.api'. *Running
>>>>>>>>>> `python gen_protos.py` to forcefully generate files didn't help either.
>>>>>>>>>>
>>>>>>>>>> If you have encountered this error and found a resolution, please
>>>>>>>>>> let me know(that would be super helpful).
>>>>>>>>>>
>>>>>>>>>> I am going to work on this soon. Please let me know if you want
>>>>>>>>>> to collaborate.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Anand Inguva
>>>>>>>>>>
>>>>>>>>>> *[1] *https://github.com/apache/beam/pull/24721
>>>>>>>>>> [2]
>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt
>>>>>>>>>>
>>>>>>>>>
Re: Python 3.11 support in Apache Beam
Posted by Anand Inguva via dev <de...@beam.apache.org>.
Yes Ahmet. That would be great.
There are some load tests defined in the
https://github.com/apache/beam/blob/master/.test-infra which could be
useful for performance testing of Beam between 3.10 and 3.11. Do you
suggest any other tests?
On Wed, Apr 12, 2023 at 8:04 PM Ahmet Altay <al...@google.com> wrote:
> Thank you, this is great!
>
> Python 3.11 announcement had a claim about performance [1]:
>
> "CPython 3.11 is an average of 25% faster than CPython 3.10 as measured
> with the pyperformance benchmark suite, when compiled with GCC on Ubuntu
> Linux. Depending on your workload, the overall speedup could be 10-60%."
>
> Have we measured this in Beam? Are we seeing any benefits? If not, why? If
> yes, this would be a cool blog post as well.
>
> Ahmet
>
>
> On Wed, Apr 5, 2023 at 1:12 PM Anand Inguva via dev <de...@beam.apache.org>
> wrote:
>
>> Python 3.11 support has been merged at
>> https://github.com/apache/beam/pull/26121 targeting Beam 2.47.0 release.
>>
>> Please let me know if you have any questions.
>>
>> Thanks,
>> Anand
>>
>> On Tue, Feb 21, 2023 at 6:04 PM Valentyn Tymofieiev <va...@google.com>
>> wrote:
>>
>>> Thanks a lot Anand. I'll take a look at the PRs.
>>>
>>> On Tue, Feb 21, 2023 at 1:56 PM Anand Inguva <an...@google.com>
>>> wrote:
>>>
>>>> I was able to spin up a PR: https://github.com/apache/beam/pull/24599
>>>> that updates the build dependencies of Apache Beam.
>>>>
>>>> Several GCP dependencies needed to be updated as well. I covered them
>>>> in the PR: https://github.com/apache/beam/pull/24599
>>>>
>>>> On Thu, Feb 9, 2023 at 3:29 PM Anand Inguva <an...@google.com>
>>>> wrote:
>>>>
>>>>> Yes, we may need to update all of them
>>>>> <https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt#L28-L33>.
>>>>> I can add more information once I dig into the issue(most likely next
>>>>> week). I will comment on my findings on the issue:
>>>>> https://github.com/apache/beam/issues/24569 and will periodically
>>>>> update this thread.
>>>>>
>>>>> On Tue, Feb 7, 2023 at 5:47 PM Valentyn Tymofieiev <
>>>>> valentyn@google.com> wrote:
>>>>>
>>>>>> On Tue, Feb 7, 2023 at 2:35 PM Anand Inguva <an...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Yes, it is related to protobuf only. But I think the update of these
>>>>>>> dependencies are required for Python 3.11 since the newer versions have
>>>>>>> support for Python 3.11 wheels.
>>>>>>>
>>>>>> Assuming you refer to protobuf. Yes, there are no wheels for 3.10 for
>>>>>> protobuf==3.x.x and that can cause friction.
>>>>>> https://pypi.org/project/protobuf/3.20.3/#files
>>>>>>
>>>>>> I would probably narrow the problem further to demonstrate which
>>>>>> stubs are not being generated, and if reason not obvious we can also ask
>>>>>> for feedback from protobuf maintainers. Also - do we by chance need to
>>>>>> update some other deps from
>>>>>> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt#L28-L33
>>>>>> for this to work?
>>>>>>
>>>>>> Also: tracking issue for protobuf4 support in Beam:
>>>>>> https://github.com/apache/beam/issues/24569.
>>>>>>
>>>>>> If we use older versions of these packages, then we have to depend on
>>>>>>> installing those packages on Python 3.11 from source distributions which is
>>>>>>> not desired.
>>>>>>>
>>>>>>> I am working parallely on that issue in a different PR
>>>>>>> https://github.com/apache/beam/pull/24599 but I think this issue
>>>>>>> should be a blocker for Python 3.11 update.
>>>>>>>
>>>>>>> On Tue, Feb 7, 2023 at 5:25 PM Valentyn Tymofieiev <
>>>>>>> valentyn@google.com> wrote:
>>>>>>>
>>>>>>>> Hi Anand,
>>>>>>>>
>>>>>>>> On Tue, Feb 7, 2023 at 1:35 PM Anand Inguva via dev <
>>>>>>>> dev@beam.apache.org> wrote:
>>>>>>>>
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> We are planning to work on adding support for Python 3.11[1] to
>>>>>>>>> Apache Beam Python SDK.
>>>>>>>>>
>>>>>>>>> As part of this effort, we are going to update the python build
>>>>>>>>> dependencies defined at [2].
>>>>>>>>>
>>>>>>>>> Right now, there is an error with the newer version of
>>>>>>>>> protobuf(4.21.11). It is not generating _urn files.
>>>>>>>>>
>>>>>>>>> It can be reproduced by
>>>>>>>>>
>>>>>>>>
>>>>>>>>> 1. python setup.py sdist
>>>>>>>>> 2. pip install dist/apache-beam-x.xx.x.dev0.tar.gz
>>>>>>>>> 3. switch to python interpreter and run import apache_beam as beam
>>>>>>>>>
>>>>>>>> I think the error you are describing is related to protobuf 4, so
>>>>>>>> the repro should focus on the portion where generation of stubs is
>>>>>>>> happening. Presumably some stubs are not generated on protobuf 4 + Python
>>>>>>>> 3.11?
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> will lead to *ImportError: cannot import name
>>>>>>>>> 'beam_runner_api_pb2_urns' from 'apache_beam.portability.api'. *Running
>>>>>>>>> `python gen_protos.py` to forcefully generate files didn't help either.
>>>>>>>>>
>>>>>>>>> If you have encountered this error and found a resolution, please
>>>>>>>>> let me know(that would be super helpful).
>>>>>>>>>
>>>>>>>>> I am going to work on this soon. Please let me know if you want to
>>>>>>>>> collaborate.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Anand Inguva
>>>>>>>>>
>>>>>>>>> *[1] *https://github.com/apache/beam/pull/24721
>>>>>>>>> [2]
>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt
>>>>>>>>>
>>>>>>>>
Re: Python 3.11 support in Apache Beam
Posted by Ahmet Altay via dev <de...@beam.apache.org>.
Thank you, this is great!
Python 3.11 announcement had a claim about performance [1]:
"CPython 3.11 is an average of 25% faster than CPython 3.10 as measured
with the pyperformance benchmark suite, when compiled with GCC on Ubuntu
Linux. Depending on your workload, the overall speedup could be 10-60%."
Have we measured this in Beam? Are we seeing any benefits? If not, why? If
yes, this would be a cool blog post as well.
Ahmet
On Wed, Apr 5, 2023 at 1:12 PM Anand Inguva via dev <de...@beam.apache.org>
wrote:
> Python 3.11 support has been merged at
> https://github.com/apache/beam/pull/26121 targeting Beam 2.47.0 release.
>
> Please let me know if you have any questions.
>
> Thanks,
> Anand
>
> On Tue, Feb 21, 2023 at 6:04 PM Valentyn Tymofieiev <va...@google.com>
> wrote:
>
>> Thanks a lot Anand. I'll take a look at the PRs.
>>
>> On Tue, Feb 21, 2023 at 1:56 PM Anand Inguva <an...@google.com>
>> wrote:
>>
>>> I was able to spin up a PR: https://github.com/apache/beam/pull/24599
>>> that updates the build dependencies of Apache Beam.
>>>
>>> Several GCP dependencies needed to be updated as well. I covered them in
>>> the PR: https://github.com/apache/beam/pull/24599
>>>
>>> On Thu, Feb 9, 2023 at 3:29 PM Anand Inguva <an...@google.com>
>>> wrote:
>>>
>>>> Yes, we may need to update all of them
>>>> <https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt#L28-L33>.
>>>> I can add more information once I dig into the issue(most likely next
>>>> week). I will comment on my findings on the issue:
>>>> https://github.com/apache/beam/issues/24569 and will periodically
>>>> update this thread.
>>>>
>>>> On Tue, Feb 7, 2023 at 5:47 PM Valentyn Tymofieiev <va...@google.com>
>>>> wrote:
>>>>
>>>>> On Tue, Feb 7, 2023 at 2:35 PM Anand Inguva <an...@google.com>
>>>>> wrote:
>>>>>
>>>>>> Yes, it is related to protobuf only. But I think the update of these
>>>>>> dependencies are required for Python 3.11 since the newer versions have
>>>>>> support for Python 3.11 wheels.
>>>>>>
>>>>> Assuming you refer to protobuf. Yes, there are no wheels for 3.10 for
>>>>> protobuf==3.x.x and that can cause friction.
>>>>> https://pypi.org/project/protobuf/3.20.3/#files
>>>>>
>>>>> I would probably narrow the problem further to demonstrate which stubs
>>>>> are not being generated, and if reason not obvious we can also ask for
>>>>> feedback from protobuf maintainers. Also - do we by chance need to
>>>>> update some other deps from
>>>>> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt#L28-L33
>>>>> for this to work?
>>>>>
>>>>> Also: tracking issue for protobuf4 support in Beam:
>>>>> https://github.com/apache/beam/issues/24569.
>>>>>
>>>>> If we use older versions of these packages, then we have to depend on
>>>>>> installing those packages on Python 3.11 from source distributions which is
>>>>>> not desired.
>>>>>>
>>>>>> I am working parallely on that issue in a different PR
>>>>>> https://github.com/apache/beam/pull/24599 but I think this issue
>>>>>> should be a blocker for Python 3.11 update.
>>>>>>
>>>>>> On Tue, Feb 7, 2023 at 5:25 PM Valentyn Tymofieiev <
>>>>>> valentyn@google.com> wrote:
>>>>>>
>>>>>>> Hi Anand,
>>>>>>>
>>>>>>> On Tue, Feb 7, 2023 at 1:35 PM Anand Inguva via dev <
>>>>>>> dev@beam.apache.org> wrote:
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> We are planning to work on adding support for Python 3.11[1] to
>>>>>>>> Apache Beam Python SDK.
>>>>>>>>
>>>>>>>> As part of this effort, we are going to update the python build
>>>>>>>> dependencies defined at [2].
>>>>>>>>
>>>>>>>> Right now, there is an error with the newer version of
>>>>>>>> protobuf(4.21.11). It is not generating _urn files.
>>>>>>>>
>>>>>>>> It can be reproduced by
>>>>>>>>
>>>>>>>
>>>>>>>> 1. python setup.py sdist
>>>>>>>> 2. pip install dist/apache-beam-x.xx.x.dev0.tar.gz
>>>>>>>> 3. switch to python interpreter and run import apache_beam as beam
>>>>>>>>
>>>>>>> I think the error you are describing is related to protobuf 4, so
>>>>>>> the repro should focus on the portion where generation of stubs is
>>>>>>> happening. Presumably some stubs are not generated on protobuf 4 + Python
>>>>>>> 3.11?
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> will lead to *ImportError: cannot import name
>>>>>>>> 'beam_runner_api_pb2_urns' from 'apache_beam.portability.api'. *Running
>>>>>>>> `python gen_protos.py` to forcefully generate files didn't help either.
>>>>>>>>
>>>>>>>> If you have encountered this error and found a resolution, please
>>>>>>>> let me know(that would be super helpful).
>>>>>>>>
>>>>>>>> I am going to work on this soon. Please let me know if you want to
>>>>>>>> collaborate.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Anand Inguva
>>>>>>>>
>>>>>>>> *[1] *https://github.com/apache/beam/pull/24721
>>>>>>>> [2]
>>>>>>>> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt
>>>>>>>>
>>>>>>>
Re: Python 3.11 support in Apache Beam
Posted by Anand Inguva via dev <de...@beam.apache.org>.
Python 3.11 support has been merged at
https://github.com/apache/beam/pull/26121 targeting Beam 2.47.0 release.
Please let me know if you have any questions.
Thanks,
Anand
On Tue, Feb 21, 2023 at 6:04 PM Valentyn Tymofieiev <va...@google.com>
wrote:
> Thanks a lot Anand. I'll take a look at the PRs.
>
> On Tue, Feb 21, 2023 at 1:56 PM Anand Inguva <an...@google.com>
> wrote:
>
>> I was able to spin up a PR: https://github.com/apache/beam/pull/24599
>> that updates the build dependencies of Apache Beam.
>>
>> Several GCP dependencies needed to be updated as well. I covered them in
>> the PR: https://github.com/apache/beam/pull/24599
>>
>> On Thu, Feb 9, 2023 at 3:29 PM Anand Inguva <an...@google.com>
>> wrote:
>>
>>> Yes, we may need to update all of them
>>> <https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt#L28-L33>.
>>> I can add more information once I dig into the issue(most likely next
>>> week). I will comment on my findings on the issue:
>>> https://github.com/apache/beam/issues/24569 and will periodically
>>> update this thread.
>>>
>>> On Tue, Feb 7, 2023 at 5:47 PM Valentyn Tymofieiev <va...@google.com>
>>> wrote:
>>>
>>>> On Tue, Feb 7, 2023 at 2:35 PM Anand Inguva <an...@google.com>
>>>> wrote:
>>>>
>>>>> Yes, it is related to protobuf only. But I think the update of these
>>>>> dependencies are required for Python 3.11 since the newer versions have
>>>>> support for Python 3.11 wheels.
>>>>>
>>>> Assuming you refer to protobuf. Yes, there are no wheels for 3.10 for
>>>> protobuf==3.x.x and that can cause friction.
>>>> https://pypi.org/project/protobuf/3.20.3/#files
>>>>
>>>> I would probably narrow the problem further to demonstrate which stubs
>>>> are not being generated, and if reason not obvious we can also ask for
>>>> feedback from protobuf maintainers. Also - do we by chance need to
>>>> update some other deps from
>>>> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt#L28-L33
>>>> for this to work?
>>>>
>>>> Also: tracking issue for protobuf4 support in Beam:
>>>> https://github.com/apache/beam/issues/24569.
>>>>
>>>> If we use older versions of these packages, then we have to depend on
>>>>> installing those packages on Python 3.11 from source distributions which is
>>>>> not desired.
>>>>>
>>>>> I am working parallely on that issue in a different PR
>>>>> https://github.com/apache/beam/pull/24599 but I think this issue
>>>>> should be a blocker for Python 3.11 update.
>>>>>
>>>>> On Tue, Feb 7, 2023 at 5:25 PM Valentyn Tymofieiev <
>>>>> valentyn@google.com> wrote:
>>>>>
>>>>>> Hi Anand,
>>>>>>
>>>>>> On Tue, Feb 7, 2023 at 1:35 PM Anand Inguva via dev <
>>>>>> dev@beam.apache.org> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> We are planning to work on adding support for Python 3.11[1] to
>>>>>>> Apache Beam Python SDK.
>>>>>>>
>>>>>>> As part of this effort, we are going to update the python build
>>>>>>> dependencies defined at [2].
>>>>>>>
>>>>>>> Right now, there is an error with the newer version of
>>>>>>> protobuf(4.21.11). It is not generating _urn files.
>>>>>>>
>>>>>>> It can be reproduced by
>>>>>>>
>>>>>>
>>>>>>> 1. python setup.py sdist
>>>>>>> 2. pip install dist/apache-beam-x.xx.x.dev0.tar.gz
>>>>>>> 3. switch to python interpreter and run import apache_beam as beam
>>>>>>>
>>>>>> I think the error you are describing is related to protobuf 4, so the
>>>>>> repro should focus on the portion where generation of stubs is happening.
>>>>>> Presumably some stubs are not generated on protobuf 4 + Python 3.11?
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> will lead to *ImportError: cannot import name
>>>>>>> 'beam_runner_api_pb2_urns' from 'apache_beam.portability.api'. *Running
>>>>>>> `python gen_protos.py` to forcefully generate files didn't help either.
>>>>>>>
>>>>>>> If you have encountered this error and found a resolution, please
>>>>>>> let me know(that would be super helpful).
>>>>>>>
>>>>>>> I am going to work on this soon. Please let me know if you want to
>>>>>>> collaborate.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Anand Inguva
>>>>>>>
>>>>>>> *[1] *https://github.com/apache/beam/pull/24721
>>>>>>> [2]
>>>>>>> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt
>>>>>>>
>>>>>>
Re: Python 3.11 support in Apache Beam
Posted by Valentyn Tymofieiev via dev <de...@beam.apache.org>.
Thanks a lot Anand. I'll take a look at the PRs.
On Tue, Feb 21, 2023 at 1:56 PM Anand Inguva <an...@google.com> wrote:
> I was able to spin up a PR: https://github.com/apache/beam/pull/24599
> that updates the build dependencies of Apache Beam.
>
> Several GCP dependencies needed to be updated as well. I covered them in
> the PR: https://github.com/apache/beam/pull/24599
>
> On Thu, Feb 9, 2023 at 3:29 PM Anand Inguva <an...@google.com>
> wrote:
>
>> Yes, we may need to update all of them
>> <https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt#L28-L33>.
>> I can add more information once I dig into the issue(most likely next
>> week). I will comment on my findings on the issue:
>> https://github.com/apache/beam/issues/24569 and will periodically update
>> this thread.
>>
>> On Tue, Feb 7, 2023 at 5:47 PM Valentyn Tymofieiev <va...@google.com>
>> wrote:
>>
>>> On Tue, Feb 7, 2023 at 2:35 PM Anand Inguva <an...@google.com>
>>> wrote:
>>>
>>>> Yes, it is related to protobuf only. But I think the update of these
>>>> dependencies are required for Python 3.11 since the newer versions have
>>>> support for Python 3.11 wheels.
>>>>
>>> Assuming you refer to protobuf. Yes, there are no wheels for 3.10 for
>>> protobuf==3.x.x and that can cause friction.
>>> https://pypi.org/project/protobuf/3.20.3/#files
>>>
>>> I would probably narrow the problem further to demonstrate which stubs
>>> are not being generated, and if reason not obvious we can also ask for
>>> feedback from protobuf maintainers. Also - do we by chance need to
>>> update some other deps from
>>> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt#L28-L33
>>> for this to work?
>>>
>>> Also: tracking issue for protobuf4 support in Beam:
>>> https://github.com/apache/beam/issues/24569.
>>>
>>> If we use older versions of these packages, then we have to depend on
>>>> installing those packages on Python 3.11 from source distributions which is
>>>> not desired.
>>>>
>>>> I am working parallely on that issue in a different PR
>>>> https://github.com/apache/beam/pull/24599 but I think this issue
>>>> should be a blocker for Python 3.11 update.
>>>>
>>>> On Tue, Feb 7, 2023 at 5:25 PM Valentyn Tymofieiev <va...@google.com>
>>>> wrote:
>>>>
>>>>> Hi Anand,
>>>>>
>>>>> On Tue, Feb 7, 2023 at 1:35 PM Anand Inguva via dev <
>>>>> dev@beam.apache.org> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> We are planning to work on adding support for Python 3.11[1] to
>>>>>> Apache Beam Python SDK.
>>>>>>
>>>>>> As part of this effort, we are going to update the python build
>>>>>> dependencies defined at [2].
>>>>>>
>>>>>> Right now, there is an error with the newer version of
>>>>>> protobuf(4.21.11). It is not generating _urn files.
>>>>>>
>>>>>> It can be reproduced by
>>>>>>
>>>>>
>>>>>> 1. python setup.py sdist
>>>>>> 2. pip install dist/apache-beam-x.xx.x.dev0.tar.gz
>>>>>> 3. switch to python interpreter and run import apache_beam as beam
>>>>>>
>>>>> I think the error you are describing is related to protobuf 4, so the
>>>>> repro should focus on the portion where generation of stubs is happening.
>>>>> Presumably some stubs are not generated on protobuf 4 + Python 3.11?
>>>>>
>>>>>
>>>>>>
>>>>>> will lead to *ImportError: cannot import name
>>>>>> 'beam_runner_api_pb2_urns' from 'apache_beam.portability.api'. *Running
>>>>>> `python gen_protos.py` to forcefully generate files didn't help either.
>>>>>>
>>>>>> If you have encountered this error and found a resolution, please let
>>>>>> me know(that would be super helpful).
>>>>>>
>>>>>> I am going to work on this soon. Please let me know if you want to
>>>>>> collaborate.
>>>>>>
>>>>>> Thanks,
>>>>>> Anand Inguva
>>>>>>
>>>>>> *[1] *https://github.com/apache/beam/pull/24721
>>>>>> [2]
>>>>>> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt
>>>>>>
>>>>>
Re: Python 3.11 support in Apache Beam
Posted by Anand Inguva via dev <de...@beam.apache.org>.
I was able to spin up a PR: https://github.com/apache/beam/pull/24599 that
updates the build dependencies of Apache Beam.
Several GCP dependencies needed to be updated as well. I covered them in
the PR: https://github.com/apache/beam/pull/24599
On Thu, Feb 9, 2023 at 3:29 PM Anand Inguva <an...@google.com> wrote:
> Yes, we may need to update all of them
> <https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt#L28-L33>.
> I can add more information once I dig into the issue(most likely next
> week). I will comment on my findings on the issue:
> https://github.com/apache/beam/issues/24569 and will periodically update
> this thread.
>
> On Tue, Feb 7, 2023 at 5:47 PM Valentyn Tymofieiev <va...@google.com>
> wrote:
>
>> On Tue, Feb 7, 2023 at 2:35 PM Anand Inguva <an...@google.com>
>> wrote:
>>
>>> Yes, it is related to protobuf only. But I think the update of these
>>> dependencies are required for Python 3.11 since the newer versions have
>>> support for Python 3.11 wheels.
>>>
>> Assuming you refer to protobuf. Yes, there are no wheels for 3.10 for
>> protobuf==3.x.x and that can cause friction.
>> https://pypi.org/project/protobuf/3.20.3/#files
>>
>> I would probably narrow the problem further to demonstrate which stubs
>> are not being generated, and if reason not obvious we can also ask for
>> feedback from protobuf maintainers. Also - do we by chance need to
>> update some other deps from
>> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt#L28-L33
>> for this to work?
>>
>> Also: tracking issue for protobuf4 support in Beam:
>> https://github.com/apache/beam/issues/24569.
>>
>> If we use older versions of these packages, then we have to depend on
>>> installing those packages on Python 3.11 from source distributions which is
>>> not desired.
>>>
>>> I am working parallely on that issue in a different PR
>>> https://github.com/apache/beam/pull/24599 but I think this issue should
>>> be a blocker for Python 3.11 update.
>>>
>>> On Tue, Feb 7, 2023 at 5:25 PM Valentyn Tymofieiev <va...@google.com>
>>> wrote:
>>>
>>>> Hi Anand,
>>>>
>>>> On Tue, Feb 7, 2023 at 1:35 PM Anand Inguva via dev <
>>>> dev@beam.apache.org> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> We are planning to work on adding support for Python 3.11[1] to Apache
>>>>> Beam Python SDK.
>>>>>
>>>>> As part of this effort, we are going to update the python build
>>>>> dependencies defined at [2].
>>>>>
>>>>> Right now, there is an error with the newer version of
>>>>> protobuf(4.21.11). It is not generating _urn files.
>>>>>
>>>>> It can be reproduced by
>>>>>
>>>>
>>>>> 1. python setup.py sdist
>>>>> 2. pip install dist/apache-beam-x.xx.x.dev0.tar.gz
>>>>> 3. switch to python interpreter and run import apache_beam as beam
>>>>>
>>>> I think the error you are describing is related to protobuf 4, so the
>>>> repro should focus on the portion where generation of stubs is happening.
>>>> Presumably some stubs are not generated on protobuf 4 + Python 3.11?
>>>>
>>>>
>>>>>
>>>>> will lead to *ImportError: cannot import name
>>>>> 'beam_runner_api_pb2_urns' from 'apache_beam.portability.api'. *Running
>>>>> `python gen_protos.py` to forcefully generate files didn't help either.
>>>>>
>>>>> If you have encountered this error and found a resolution, please let
>>>>> me know(that would be super helpful).
>>>>>
>>>>> I am going to work on this soon. Please let me know if you want to
>>>>> collaborate.
>>>>>
>>>>> Thanks,
>>>>> Anand Inguva
>>>>>
>>>>> *[1] *https://github.com/apache/beam/pull/24721
>>>>> [2]
>>>>> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt
>>>>>
>>>>
Re: Python 3.11 support in Apache Beam
Posted by Anand Inguva via dev <de...@beam.apache.org>.
Yes, we may need to update all of them
<https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt#L28-L33>.
I can add more information once I dig into the issue(most likely next
week). I will comment on my findings on the issue:
https://github.com/apache/beam/issues/24569 and will periodically update
this thread.
On Tue, Feb 7, 2023 at 5:47 PM Valentyn Tymofieiev <va...@google.com>
wrote:
> On Tue, Feb 7, 2023 at 2:35 PM Anand Inguva <an...@google.com>
> wrote:
>
>> Yes, it is related to protobuf only. But I think the update of these
>> dependencies are required for Python 3.11 since the newer versions have
>> support for Python 3.11 wheels.
>>
> Assuming you refer to protobuf. Yes, there are no wheels for 3.10 for
> protobuf==3.x.x and that can cause friction.
> https://pypi.org/project/protobuf/3.20.3/#files
>
> I would probably narrow the problem further to demonstrate which stubs are
> not being generated, and if reason not obvious we can also ask for feedback
> from protobuf maintainers. Also - do we by chance need to update some other
> deps from
> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt#L28-L33
> for this to work?
>
> Also: tracking issue for protobuf4 support in Beam:
> https://github.com/apache/beam/issues/24569.
>
> If we use older versions of these packages, then we have to depend on
>> installing those packages on Python 3.11 from source distributions which is
>> not desired.
>>
>> I am working parallely on that issue in a different PR
>> https://github.com/apache/beam/pull/24599 but I think this issue should
>> be a blocker for Python 3.11 update.
>>
>> On Tue, Feb 7, 2023 at 5:25 PM Valentyn Tymofieiev <va...@google.com>
>> wrote:
>>
>>> Hi Anand,
>>>
>>> On Tue, Feb 7, 2023 at 1:35 PM Anand Inguva via dev <de...@beam.apache.org>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> We are planning to work on adding support for Python 3.11[1] to Apache
>>>> Beam Python SDK.
>>>>
>>>> As part of this effort, we are going to update the python build
>>>> dependencies defined at [2].
>>>>
>>>> Right now, there is an error with the newer version of
>>>> protobuf(4.21.11). It is not generating _urn files.
>>>>
>>>> It can be reproduced by
>>>>
>>>
>>>> 1. python setup.py sdist
>>>> 2. pip install dist/apache-beam-x.xx.x.dev0.tar.gz
>>>> 3. switch to python interpreter and run import apache_beam as beam
>>>>
>>> I think the error you are describing is related to protobuf 4, so the
>>> repro should focus on the portion where generation of stubs is happening.
>>> Presumably some stubs are not generated on protobuf 4 + Python 3.11?
>>>
>>>
>>>>
>>>> will lead to *ImportError: cannot import name
>>>> 'beam_runner_api_pb2_urns' from 'apache_beam.portability.api'. *Running
>>>> `python gen_protos.py` to forcefully generate files didn't help either.
>>>>
>>>> If you have encountered this error and found a resolution, please let
>>>> me know(that would be super helpful).
>>>>
>>>> I am going to work on this soon. Please let me know if you want to
>>>> collaborate.
>>>>
>>>> Thanks,
>>>> Anand Inguva
>>>>
>>>> *[1] *https://github.com/apache/beam/pull/24721
>>>> [2]
>>>> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt
>>>>
>>>
Re: Python 3.11 support in Apache Beam
Posted by Valentyn Tymofieiev via dev <de...@beam.apache.org>.
On Tue, Feb 7, 2023 at 2:35 PM Anand Inguva <an...@google.com> wrote:
> Yes, it is related to protobuf only. But I think the update of these
> dependencies are required for Python 3.11 since the newer versions have
> support for Python 3.11 wheels.
>
Assuming you refer to protobuf. Yes, there are no wheels for 3.10 for
protobuf==3.x.x and that can cause friction.
https://pypi.org/project/protobuf/3.20.3/#files
I would probably narrow the problem further to demonstrate which stubs are
not being generated, and if reason not obvious we can also ask for feedback
from protobuf maintainers. Also - do we by chance need to update some other
deps from
https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt#L28-L33
for this to work?
Also: tracking issue for protobuf4 support in Beam:
https://github.com/apache/beam/issues/24569.
If we use older versions of these packages, then we have to depend on
> installing those packages on Python 3.11 from source distributions which is
> not desired.
>
> I am working parallely on that issue in a different PR
> https://github.com/apache/beam/pull/24599 but I think this issue should
> be a blocker for Python 3.11 update.
>
> On Tue, Feb 7, 2023 at 5:25 PM Valentyn Tymofieiev <va...@google.com>
> wrote:
>
>> Hi Anand,
>>
>> On Tue, Feb 7, 2023 at 1:35 PM Anand Inguva via dev <de...@beam.apache.org>
>> wrote:
>>
>>> Hi all,
>>>
>>> We are planning to work on adding support for Python 3.11[1] to Apache
>>> Beam Python SDK.
>>>
>>> As part of this effort, we are going to update the python build
>>> dependencies defined at [2].
>>>
>>> Right now, there is an error with the newer version of
>>> protobuf(4.21.11). It is not generating _urn files.
>>>
>>> It can be reproduced by
>>>
>>
>>> 1. python setup.py sdist
>>> 2. pip install dist/apache-beam-x.xx.x.dev0.tar.gz
>>> 3. switch to python interpreter and run import apache_beam as beam
>>>
>> I think the error you are describing is related to protobuf 4, so the
>> repro should focus on the portion where generation of stubs is happening.
>> Presumably some stubs are not generated on protobuf 4 + Python 3.11?
>>
>>
>>>
>>> will lead to *ImportError: cannot import name
>>> 'beam_runner_api_pb2_urns' from 'apache_beam.portability.api'. *Running
>>> `python gen_protos.py` to forcefully generate files didn't help either.
>>>
>>> If you have encountered this error and found a resolution, please let me
>>> know(that would be super helpful).
>>>
>>> I am going to work on this soon. Please let me know if you want to
>>> collaborate.
>>>
>>> Thanks,
>>> Anand Inguva
>>>
>>> *[1] *https://github.com/apache/beam/pull/24721
>>> [2]
>>> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt
>>>
>>
Re: Python 3.11 support in Apache Beam
Posted by Anand Inguva via dev <de...@beam.apache.org>.
Yes, it is related to protobuf only. But I think the update of these
dependencies are required for Python 3.11 since the newer versions have
support for Python 3.11 wheels. If we use older versions of these packages,
then we have to depend on installing those packages on Python 3.11 from
source distributions which is not desired.
I am working parallely on that issue in a different PR
https://github.com/apache/beam/pull/24599 but I think this issue should be
a blocker for Python 3.11 update.
On Tue, Feb 7, 2023 at 5:25 PM Valentyn Tymofieiev <va...@google.com>
wrote:
> Hi Anand,
>
> On Tue, Feb 7, 2023 at 1:35 PM Anand Inguva via dev <de...@beam.apache.org>
> wrote:
>
>> Hi all,
>>
>> We are planning to work on adding support for Python 3.11[1] to Apache
>> Beam Python SDK.
>>
>> As part of this effort, we are going to update the python build
>> dependencies defined at [2].
>>
>> Right now, there is an error with the newer version of protobuf(4.21.11).
>> It is not generating _urn files.
>>
>> It can be reproduced by
>>
>
>> 1. python setup.py sdist
>> 2. pip install dist/apache-beam-x.xx.x.dev0.tar.gz
>> 3. switch to python interpreter and run import apache_beam as beam
>>
> I think the error you are describing is related to protobuf 4, so the
> repro should focus on the portion where generation of stubs is happening.
> Presumably some stubs are not generated on protobuf 4 + Python 3.11?
>
>
>>
>> will lead to *ImportError: cannot import name 'beam_runner_api_pb2_urns'
>> from 'apache_beam.portability.api'. *Running `python gen_protos.py` to
>> forcefully generate files didn't help either.
>>
>> If you have encountered this error and found a resolution, please let me
>> know(that would be super helpful).
>>
>> I am going to work on this soon. Please let me know if you want to
>> collaborate.
>>
>> Thanks,
>> Anand Inguva
>>
>> *[1] *https://github.com/apache/beam/pull/24721
>> [2]
>> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt
>>
>
Re: Python 3.11 support in Apache Beam
Posted by Valentyn Tymofieiev via dev <de...@beam.apache.org>.
Hi Anand,
On Tue, Feb 7, 2023 at 1:35 PM Anand Inguva via dev <de...@beam.apache.org>
wrote:
> Hi all,
>
> We are planning to work on adding support for Python 3.11[1] to Apache
> Beam Python SDK.
>
> As part of this effort, we are going to update the python build
> dependencies defined at [2].
>
> Right now, there is an error with the newer version of protobuf(4.21.11).
> It is not generating _urn files.
>
> It can be reproduced by
>
> 1. python setup.py sdist
> 2. pip install dist/apache-beam-x.xx.x.dev0.tar.gz
> 3. switch to python interpreter and run import apache_beam as beam
>
I think the error you are describing is related to protobuf 4, so the repro
should focus on the portion where generation of stubs is happening.
Presumably some stubs are not generated on protobuf 4 + Python 3.11?
>
> will lead to *ImportError: cannot import name 'beam_runner_api_pb2_urns'
> from 'apache_beam.portability.api'. *Running `python gen_protos.py` to
> forcefully generate files didn't help either.
>
> If you have encountered this error and found a resolution, please let me
> know(that would be super helpful).
>
> I am going to work on this soon. Please let me know if you want to
> collaborate.
>
> Thanks,
> Anand Inguva
>
> *[1] *https://github.com/apache/beam/pull/24721
> [2]
> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt
>