You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Anand Inguva via dev <de...@beam.apache.org> on 2023/10/12 19:27:17 UTC
Re: Proposal for pyproject.toml Support in Apache Beam Python
The PR https://github.com/apache/beam/pull/28385 is merged today. If there
are any observed failures, please comment on the PR and I will follow up
with a forward fix. Thanks.
On Fri, Sep 1, 2023 at 2:30 PM Anand Inguva <an...@google.com> wrote:
> Since there is positive feedback from the dev community, I am going ahead
> and implementing this proposal for Python SDK.
>
> @austin@apache.org <au...@apache.org> Initially let's move forward with
> the setuptools as backend for building package and as part of the future
> work, we can find a better backend than setuptools.
>
> Thanks for the feedback.
> Anand
>
> On Mon, Aug 28, 2023 at 12:00 PM Austin Bennett <au...@apache.org> wrote:
>
>> I've thought about this a ton, but haven't been in a position to
>> undertake the work. Thanks for bringing this up, @Anand Inguva
>> <an...@google.com> !
>>
>> I'd point us to https://python-poetry.org/ ... [ which is where I'd
>> look take us, but I'm also not able to do all the work, so my
>> suggestion/preference doensn't matter that much ]
>>
>> https://python-poetry.org/docs/pyproject#the-pyprojecttoml-file <- for
>> info on pyproject.toml file.
>>
>> Notice the use of a 'lock' file is very valuable, ex:
>> https://python-poetry.org/docs/basic-usage/#committing-your-poetrylock-file-to-version-control
>>
>> I haven't come across `build`, that might be great too. I'd highlight
>> that Poetry is pretty common across industry these days, rock-solid,
>> ecosystem of interoperability, users, etc... If not familiar, PLEASE have
>> a look at that.
>>
>>
>>
>>
>> On Mon, Aug 28, 2023 at 8:04 AM Kerry Donny-Clark via dev <
>> dev@beam.apache.org> wrote:
>>
>>> +1
>>> Hi Anand,
>>> I appreciate this effort. Managing python dependencies has been a major
>>> pain point for me, and I think this approach would help.
>>> Kerry
>>>
>>> On Mon, Aug 28, 2023 at 10:14 AM Anand Inguva via dev <
>>> dev@beam.apache.org> wrote:
>>>
>>>> Hello Beam Dev Team,
>>>>
>>>> I've compiled a design document
>>>> <https://docs.google.com/document/d/17-y48WW25-VGBWZNyTdoN0WUN03k9ZhJjLp9wtyG1Wc/edit#heading=h.wskna8eurvjv>[1]
>>>> proposing the integration of pyproject.toml into Apache Beam's Python build
>>>> process. Your insights and feedback would be invaluable.
>>>>
>>>> What is pyproject.toml?
>>>> pyproject.toml is a configuration file that specifies a project's build
>>>> dependencies and other project-related metadata in a standardized
>>>> format. Before pyproject.toml, Python projects often had multiple
>>>> configuration files (like setup.py, setup.cfg, and requirements.txt).
>>>> pyproject.toml aims to centralize these configurations into one place,
>>>> making project setups more organized and straightforward. One of the
>>>> significant features enabled by pyproject.toml is the ability to perform
>>>> isolated builds. This ensures that build dependencies are separated from
>>>> the project's runtime dependencies, leading to more consistent and
>>>> reproducible builds.
>>>>
>>>> [1]
>>>> https://docs.google.com/document/d/17-y48WW25-VGBWZNyTdoN0WUN03k9ZhJjLp9wtyG1Wc/edit#heading=h.wskna8eurvjv
>>>>
>>>> Thanks,
>>>> Anand
>>>>
>>>
Re: Proposal for pyproject.toml Support in Apache Beam Python
Posted by Anand Inguva via dev <de...@beam.apache.org>.
The gen_protos.py will be called while building a sdist, wheel or an
editable installation. We use pytest through the tox package and during
the tox build process, gen_protos.py is called during either wheel or sdist
creation.
For building sdist, the process now changed from `python setup.py sdist` to
`python -m build --sdist` and editable installation still happens with `pip
install -e .` and rest of the development process follows more or less the
same old behavior.
On Thu, Oct 12, 2023 at 5:06 PM Robert Bradshaw <ro...@google.com> wrote:
> On Thu, Oct 12, 2023 at 2:04 PM Anand Inguva <an...@google.com>
> wrote:
>
>> I am in the process of updating the documentation at
>> https://cwiki.apache.org/confluence/display/BEAM/Python+Tips related to
>> setup.py/pyproject.toml changes, but yes you can't call setup.py
>> directly because it might fail due to the lack of presence of beam python's
>> build time dependencies.
>>
>> With regards to other files(eg:protos), we will follow the similar
>> behavior as before(generating proros using `gen_protos.py`).
>>
>
> Meaning this will be called automatically when needed (e.g. from pytest)?
>
>
>> On Thu, Oct 12, 2023 at 4:01 PM Robert Bradshaw <ro...@google.com>
>> wrote:
>>
>>> Does this change any development practices? E.g. if I clone the repo,
>>> I'm assuming I couldn't run "setup.py test" anymore. What about the
>>> generated files (like protos, or the yaml definitions copied from other
>>> parts of the repo)?
>>>
>>> On Thu, Oct 12, 2023 at 12:27 PM Anand Inguva via dev <
>>> dev@beam.apache.org> wrote:
>>>
>>>> The PR https://github.com/apache/beam/pull/28385 is merged today. If
>>>> there are any observed failures, please comment on the PR and I will follow
>>>> up with a forward fix. Thanks.
>>>>
>>>> On Fri, Sep 1, 2023 at 2:30 PM Anand Inguva <an...@google.com>
>>>> wrote:
>>>>
>>>>> Since there is positive feedback from the dev community, I am going
>>>>> ahead and implementing this proposal for Python SDK.
>>>>>
>>>>> @austin@apache.org <au...@apache.org> Initially let's move forward
>>>>> with the setuptools as backend for building package and as part of the
>>>>> future work, we can find a better backend than setuptools.
>>>>>
>>>>> Thanks for the feedback.
>>>>> Anand
>>>>>
>>>>> On Mon, Aug 28, 2023 at 12:00 PM Austin Bennett <au...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> I've thought about this a ton, but haven't been in a position to
>>>>>> undertake the work. Thanks for bringing this up, @Anand Inguva
>>>>>> <an...@google.com> !
>>>>>>
>>>>>> I'd point us to https://python-poetry.org/ ... [ which is where I'd
>>>>>> look take us, but I'm also not able to do all the work, so my
>>>>>> suggestion/preference doensn't matter that much ]
>>>>>>
>>>>>> https://python-poetry.org/docs/pyproject#the-pyprojecttoml-file <-
>>>>>> for info on pyproject.toml file.
>>>>>>
>>>>>> Notice the use of a 'lock' file is very valuable, ex:
>>>>>> https://python-poetry.org/docs/basic-usage/#committing-your-poetrylock-file-to-version-control
>>>>>>
>>>>>> I haven't come across `build`, that might be great too. I'd
>>>>>> highlight that Poetry is pretty common across industry these days,
>>>>>> rock-solid, ecosystem of interoperability, users, etc... If not familiar,
>>>>>> PLEASE have a look at that.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Aug 28, 2023 at 8:04 AM Kerry Donny-Clark via dev <
>>>>>> dev@beam.apache.org> wrote:
>>>>>>
>>>>>>> +1
>>>>>>> Hi Anand,
>>>>>>> I appreciate this effort. Managing python dependencies has been a
>>>>>>> major pain point for me, and I think this approach would help.
>>>>>>> Kerry
>>>>>>>
>>>>>>> On Mon, Aug 28, 2023 at 10:14 AM Anand Inguva via dev <
>>>>>>> dev@beam.apache.org> wrote:
>>>>>>>
>>>>>>>> Hello Beam Dev Team,
>>>>>>>>
>>>>>>>> I've compiled a design document
>>>>>>>> <https://docs.google.com/document/d/17-y48WW25-VGBWZNyTdoN0WUN03k9ZhJjLp9wtyG1Wc/edit#heading=h.wskna8eurvjv>[1]
>>>>>>>> proposing the integration of pyproject.toml into Apache Beam's Python build
>>>>>>>> process. Your insights and feedback would be invaluable.
>>>>>>>>
>>>>>>>> What is pyproject.toml?
>>>>>>>> pyproject.toml is a configuration file that specifies a project's
>>>>>>>> build dependencies and other project-related metadata in a standardized
>>>>>>>> format. Before pyproject.toml, Python projects often had multiple
>>>>>>>> configuration files (like setup.py, setup.cfg, and requirements.txt).
>>>>>>>> pyproject.toml aims to centralize these configurations into one place,
>>>>>>>> making project setups more organized and straightforward. One of the
>>>>>>>> significant features enabled by pyproject.toml is the ability to perform
>>>>>>>> isolated builds. This ensures that build dependencies are separated from
>>>>>>>> the project's runtime dependencies, leading to more consistent and
>>>>>>>> reproducible builds.
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> https://docs.google.com/document/d/17-y48WW25-VGBWZNyTdoN0WUN03k9ZhJjLp9wtyG1Wc/edit#heading=h.wskna8eurvjv
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Anand
>>>>>>>>
>>>>>>>
Re: Proposal for pyproject.toml Support in Apache Beam Python
Posted by Robert Bradshaw via dev <de...@beam.apache.org>.
On Thu, Oct 12, 2023 at 2:04 PM Anand Inguva <an...@google.com> wrote:
> I am in the process of updating the documentation at
> https://cwiki.apache.org/confluence/display/BEAM/Python+Tips related to
> setup.py/pyproject.toml changes, but yes you can't call setup.py directly
> because it might fail due to the lack of presence of beam python's build
> time dependencies.
>
> With regards to other files(eg:protos), we will follow the similar
> behavior as before(generating proros using `gen_protos.py`).
>
Meaning this will be called automatically when needed (e.g. from pytest)?
> On Thu, Oct 12, 2023 at 4:01 PM Robert Bradshaw <ro...@google.com>
> wrote:
>
>> Does this change any development practices? E.g. if I clone the repo, I'm
>> assuming I couldn't run "setup.py test" anymore. What about the generated
>> files (like protos, or the yaml definitions copied from other parts of the
>> repo)?
>>
>> On Thu, Oct 12, 2023 at 12:27 PM Anand Inguva via dev <
>> dev@beam.apache.org> wrote:
>>
>>> The PR https://github.com/apache/beam/pull/28385 is merged today. If
>>> there are any observed failures, please comment on the PR and I will follow
>>> up with a forward fix. Thanks.
>>>
>>> On Fri, Sep 1, 2023 at 2:30 PM Anand Inguva <an...@google.com>
>>> wrote:
>>>
>>>> Since there is positive feedback from the dev community, I am going
>>>> ahead and implementing this proposal for Python SDK.
>>>>
>>>> @austin@apache.org <au...@apache.org> Initially let's move forward
>>>> with the setuptools as backend for building package and as part of the
>>>> future work, we can find a better backend than setuptools.
>>>>
>>>> Thanks for the feedback.
>>>> Anand
>>>>
>>>> On Mon, Aug 28, 2023 at 12:00 PM Austin Bennett <au...@apache.org>
>>>> wrote:
>>>>
>>>>> I've thought about this a ton, but haven't been in a position to
>>>>> undertake the work. Thanks for bringing this up, @Anand Inguva
>>>>> <an...@google.com> !
>>>>>
>>>>> I'd point us to https://python-poetry.org/ ... [ which is where I'd
>>>>> look take us, but I'm also not able to do all the work, so my
>>>>> suggestion/preference doensn't matter that much ]
>>>>>
>>>>> https://python-poetry.org/docs/pyproject#the-pyprojecttoml-file <-
>>>>> for info on pyproject.toml file.
>>>>>
>>>>> Notice the use of a 'lock' file is very valuable, ex:
>>>>> https://python-poetry.org/docs/basic-usage/#committing-your-poetrylock-file-to-version-control
>>>>>
>>>>> I haven't come across `build`, that might be great too. I'd highlight
>>>>> that Poetry is pretty common across industry these days, rock-solid,
>>>>> ecosystem of interoperability, users, etc... If not familiar, PLEASE have
>>>>> a look at that.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Aug 28, 2023 at 8:04 AM Kerry Donny-Clark via dev <
>>>>> dev@beam.apache.org> wrote:
>>>>>
>>>>>> +1
>>>>>> Hi Anand,
>>>>>> I appreciate this effort. Managing python dependencies has been a
>>>>>> major pain point for me, and I think this approach would help.
>>>>>> Kerry
>>>>>>
>>>>>> On Mon, Aug 28, 2023 at 10:14 AM Anand Inguva via dev <
>>>>>> dev@beam.apache.org> wrote:
>>>>>>
>>>>>>> Hello Beam Dev Team,
>>>>>>>
>>>>>>> I've compiled a design document
>>>>>>> <https://docs.google.com/document/d/17-y48WW25-VGBWZNyTdoN0WUN03k9ZhJjLp9wtyG1Wc/edit#heading=h.wskna8eurvjv>[1]
>>>>>>> proposing the integration of pyproject.toml into Apache Beam's Python build
>>>>>>> process. Your insights and feedback would be invaluable.
>>>>>>>
>>>>>>> What is pyproject.toml?
>>>>>>> pyproject.toml is a configuration file that specifies a project's
>>>>>>> build dependencies and other project-related metadata in a standardized
>>>>>>> format. Before pyproject.toml, Python projects often had multiple
>>>>>>> configuration files (like setup.py, setup.cfg, and requirements.txt).
>>>>>>> pyproject.toml aims to centralize these configurations into one place,
>>>>>>> making project setups more organized and straightforward. One of the
>>>>>>> significant features enabled by pyproject.toml is the ability to perform
>>>>>>> isolated builds. This ensures that build dependencies are separated from
>>>>>>> the project's runtime dependencies, leading to more consistent and
>>>>>>> reproducible builds.
>>>>>>>
>>>>>>> [1]
>>>>>>> https://docs.google.com/document/d/17-y48WW25-VGBWZNyTdoN0WUN03k9ZhJjLp9wtyG1Wc/edit#heading=h.wskna8eurvjv
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Anand
>>>>>>>
>>>>>>
Re: Proposal for pyproject.toml Support in Apache Beam Python
Posted by Anand Inguva via dev <de...@beam.apache.org>.
I am in the process of updating the documentation at
https://cwiki.apache.org/confluence/display/BEAM/Python+Tips related to
setup.py/pyproject.toml changes, but yes you can't call setup.py directly
because it might fail due to the lack of presence of beam python's build
time dependencies.
With regards to other files(eg:protos), we will follow the similar behavior
as before(generating proros using `gen_protos.py`).
On Thu, Oct 12, 2023 at 4:01 PM Robert Bradshaw <ro...@google.com> wrote:
> Does this change any development practices? E.g. if I clone the repo, I'm
> assuming I couldn't run "setup.py test" anymore. What about the generated
> files (like protos, or the yaml definitions copied from other parts of the
> repo)?
>
> On Thu, Oct 12, 2023 at 12:27 PM Anand Inguva via dev <de...@beam.apache.org>
> wrote:
>
>> The PR https://github.com/apache/beam/pull/28385 is merged today. If
>> there are any observed failures, please comment on the PR and I will follow
>> up with a forward fix. Thanks.
>>
>> On Fri, Sep 1, 2023 at 2:30 PM Anand Inguva <an...@google.com>
>> wrote:
>>
>>> Since there is positive feedback from the dev community, I am going
>>> ahead and implementing this proposal for Python SDK.
>>>
>>> @austin@apache.org <au...@apache.org> Initially let's move forward
>>> with the setuptools as backend for building package and as part of the
>>> future work, we can find a better backend than setuptools.
>>>
>>> Thanks for the feedback.
>>> Anand
>>>
>>> On Mon, Aug 28, 2023 at 12:00 PM Austin Bennett <au...@apache.org>
>>> wrote:
>>>
>>>> I've thought about this a ton, but haven't been in a position to
>>>> undertake the work. Thanks for bringing this up, @Anand Inguva
>>>> <an...@google.com> !
>>>>
>>>> I'd point us to https://python-poetry.org/ ... [ which is where I'd
>>>> look take us, but I'm also not able to do all the work, so my
>>>> suggestion/preference doensn't matter that much ]
>>>>
>>>> https://python-poetry.org/docs/pyproject#the-pyprojecttoml-file <- for
>>>> info on pyproject.toml file.
>>>>
>>>> Notice the use of a 'lock' file is very valuable, ex:
>>>> https://python-poetry.org/docs/basic-usage/#committing-your-poetrylock-file-to-version-control
>>>>
>>>> I haven't come across `build`, that might be great too. I'd highlight
>>>> that Poetry is pretty common across industry these days, rock-solid,
>>>> ecosystem of interoperability, users, etc... If not familiar, PLEASE have
>>>> a look at that.
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Aug 28, 2023 at 8:04 AM Kerry Donny-Clark via dev <
>>>> dev@beam.apache.org> wrote:
>>>>
>>>>> +1
>>>>> Hi Anand,
>>>>> I appreciate this effort. Managing python dependencies has been a
>>>>> major pain point for me, and I think this approach would help.
>>>>> Kerry
>>>>>
>>>>> On Mon, Aug 28, 2023 at 10:14 AM Anand Inguva via dev <
>>>>> dev@beam.apache.org> wrote:
>>>>>
>>>>>> Hello Beam Dev Team,
>>>>>>
>>>>>> I've compiled a design document
>>>>>> <https://docs.google.com/document/d/17-y48WW25-VGBWZNyTdoN0WUN03k9ZhJjLp9wtyG1Wc/edit#heading=h.wskna8eurvjv>[1]
>>>>>> proposing the integration of pyproject.toml into Apache Beam's Python build
>>>>>> process. Your insights and feedback would be invaluable.
>>>>>>
>>>>>> What is pyproject.toml?
>>>>>> pyproject.toml is a configuration file that specifies a project's
>>>>>> build dependencies and other project-related metadata in a standardized
>>>>>> format. Before pyproject.toml, Python projects often had multiple
>>>>>> configuration files (like setup.py, setup.cfg, and requirements.txt).
>>>>>> pyproject.toml aims to centralize these configurations into one place,
>>>>>> making project setups more organized and straightforward. One of the
>>>>>> significant features enabled by pyproject.toml is the ability to perform
>>>>>> isolated builds. This ensures that build dependencies are separated from
>>>>>> the project's runtime dependencies, leading to more consistent and
>>>>>> reproducible builds.
>>>>>>
>>>>>> [1]
>>>>>> https://docs.google.com/document/d/17-y48WW25-VGBWZNyTdoN0WUN03k9ZhJjLp9wtyG1Wc/edit#heading=h.wskna8eurvjv
>>>>>>
>>>>>> Thanks,
>>>>>> Anand
>>>>>>
>>>>>
Re: Proposal for pyproject.toml Support in Apache Beam Python
Posted by Robert Bradshaw via dev <de...@beam.apache.org>.
Does this change any development practices? E.g. if I clone the repo, I'm
assuming I couldn't run "setup.py test" anymore. What about the generated
files (like protos, or the yaml definitions copied from other parts of the
repo)?
On Thu, Oct 12, 2023 at 12:27 PM Anand Inguva via dev <de...@beam.apache.org>
wrote:
> The PR https://github.com/apache/beam/pull/28385 is merged today. If
> there are any observed failures, please comment on the PR and I will follow
> up with a forward fix. Thanks.
>
> On Fri, Sep 1, 2023 at 2:30 PM Anand Inguva <an...@google.com>
> wrote:
>
>> Since there is positive feedback from the dev community, I am going ahead
>> and implementing this proposal for Python SDK.
>>
>> @austin@apache.org <au...@apache.org> Initially let's move forward
>> with the setuptools as backend for building package and as part of the
>> future work, we can find a better backend than setuptools.
>>
>> Thanks for the feedback.
>> Anand
>>
>> On Mon, Aug 28, 2023 at 12:00 PM Austin Bennett <au...@apache.org>
>> wrote:
>>
>>> I've thought about this a ton, but haven't been in a position to
>>> undertake the work. Thanks for bringing this up, @Anand Inguva
>>> <an...@google.com> !
>>>
>>> I'd point us to https://python-poetry.org/ ... [ which is where I'd
>>> look take us, but I'm also not able to do all the work, so my
>>> suggestion/preference doensn't matter that much ]
>>>
>>> https://python-poetry.org/docs/pyproject#the-pyprojecttoml-file <- for
>>> info on pyproject.toml file.
>>>
>>> Notice the use of a 'lock' file is very valuable, ex:
>>> https://python-poetry.org/docs/basic-usage/#committing-your-poetrylock-file-to-version-control
>>>
>>> I haven't come across `build`, that might be great too. I'd highlight
>>> that Poetry is pretty common across industry these days, rock-solid,
>>> ecosystem of interoperability, users, etc... If not familiar, PLEASE have
>>> a look at that.
>>>
>>>
>>>
>>>
>>> On Mon, Aug 28, 2023 at 8:04 AM Kerry Donny-Clark via dev <
>>> dev@beam.apache.org> wrote:
>>>
>>>> +1
>>>> Hi Anand,
>>>> I appreciate this effort. Managing python dependencies has been a major
>>>> pain point for me, and I think this approach would help.
>>>> Kerry
>>>>
>>>> On Mon, Aug 28, 2023 at 10:14 AM Anand Inguva via dev <
>>>> dev@beam.apache.org> wrote:
>>>>
>>>>> Hello Beam Dev Team,
>>>>>
>>>>> I've compiled a design document
>>>>> <https://docs.google.com/document/d/17-y48WW25-VGBWZNyTdoN0WUN03k9ZhJjLp9wtyG1Wc/edit#heading=h.wskna8eurvjv>[1]
>>>>> proposing the integration of pyproject.toml into Apache Beam's Python build
>>>>> process. Your insights and feedback would be invaluable.
>>>>>
>>>>> What is pyproject.toml?
>>>>> pyproject.toml is a configuration file that specifies a project's
>>>>> build dependencies and other project-related metadata in a standardized
>>>>> format. Before pyproject.toml, Python projects often had multiple
>>>>> configuration files (like setup.py, setup.cfg, and requirements.txt).
>>>>> pyproject.toml aims to centralize these configurations into one place,
>>>>> making project setups more organized and straightforward. One of the
>>>>> significant features enabled by pyproject.toml is the ability to perform
>>>>> isolated builds. This ensures that build dependencies are separated from
>>>>> the project's runtime dependencies, leading to more consistent and
>>>>> reproducible builds.
>>>>>
>>>>> [1]
>>>>> https://docs.google.com/document/d/17-y48WW25-VGBWZNyTdoN0WUN03k9ZhJjLp9wtyG1Wc/edit#heading=h.wskna8eurvjv
>>>>>
>>>>> Thanks,
>>>>> Anand
>>>>>
>>>>