You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Anand Inguva via dev <de...@beam.apache.org> on 2023/08/28 14:14:17 UTC
Proposal for pyproject.toml Support in Apache Beam Python
Hello Beam Dev Team,
I've compiled a design document
<https://docs.google.com/document/d/17-y48WW25-VGBWZNyTdoN0WUN03k9ZhJjLp9wtyG1Wc/edit#heading=h.wskna8eurvjv>[1]
proposing the integration of pyproject.toml into Apache Beam's Python build
process. Your insights and feedback would be invaluable.
What is pyproject.toml?
pyproject.toml is a configuration file that specifies a project's build
dependencies and other project-related metadata in a standardized
format. Before pyproject.toml, Python projects often had multiple
configuration files (like setup.py, setup.cfg, and requirements.txt).
pyproject.toml aims to centralize these configurations into one place,
making project setups more organized and straightforward. One of the
significant features enabled by pyproject.toml is the ability to perform
isolated builds. This ensures that build dependencies are separated from
the project's runtime dependencies, leading to more consistent and
reproducible builds.
[1]
https://docs.google.com/document/d/17-y48WW25-VGBWZNyTdoN0WUN03k9ZhJjLp9wtyG1Wc/edit#heading=h.wskna8eurvjv
Thanks,
Anand
Re: Proposal for pyproject.toml Support in Apache Beam Python
Posted by Anand Inguva via dev <de...@beam.apache.org>.
The gen_protos.py will be called while building a sdist, wheel or an
editable installation. We use pytest through the tox package and during
the tox build process, gen_protos.py is called during either wheel or sdist
creation.
For building sdist, the process now changed from `python setup.py sdist` to
`python -m build --sdist` and editable installation still happens with `pip
install -e .` and rest of the development process follows more or less the
same old behavior.
On Thu, Oct 12, 2023 at 5:06 PM Robert Bradshaw <ro...@google.com> wrote:
> On Thu, Oct 12, 2023 at 2:04 PM Anand Inguva <an...@google.com>
> wrote:
>
>> I am in the process of updating the documentation at
>> https://cwiki.apache.org/confluence/display/BEAM/Python+Tips related to
>> setup.py/pyproject.toml changes, but yes you can't call setup.py
>> directly because it might fail due to the lack of presence of beam python's
>> build time dependencies.
>>
>> With regards to other files(eg:protos), we will follow the similar
>> behavior as before(generating proros using `gen_protos.py`).
>>
>
> Meaning this will be called automatically when needed (e.g. from pytest)?
>
>
>> On Thu, Oct 12, 2023 at 4:01 PM Robert Bradshaw <ro...@google.com>
>> wrote:
>>
>>> Does this change any development practices? E.g. if I clone the repo,
>>> I'm assuming I couldn't run "setup.py test" anymore. What about the
>>> generated files (like protos, or the yaml definitions copied from other
>>> parts of the repo)?
>>>
>>> On Thu, Oct 12, 2023 at 12:27 PM Anand Inguva via dev <
>>> dev@beam.apache.org> wrote:
>>>
>>>> The PR https://github.com/apache/beam/pull/28385 is merged today. If
>>>> there are any observed failures, please comment on the PR and I will follow
>>>> up with a forward fix. Thanks.
>>>>
>>>> On Fri, Sep 1, 2023 at 2:30 PM Anand Inguva <an...@google.com>
>>>> wrote:
>>>>
>>>>> Since there is positive feedback from the dev community, I am going
>>>>> ahead and implementing this proposal for Python SDK.
>>>>>
>>>>> @austin@apache.org <au...@apache.org> Initially let's move forward
>>>>> with the setuptools as backend for building package and as part of the
>>>>> future work, we can find a better backend than setuptools.
>>>>>
>>>>> Thanks for the feedback.
>>>>> Anand
>>>>>
>>>>> On Mon, Aug 28, 2023 at 12:00 PM Austin Bennett <au...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> I've thought about this a ton, but haven't been in a position to
>>>>>> undertake the work. Thanks for bringing this up, @Anand Inguva
>>>>>> <an...@google.com> !
>>>>>>
>>>>>> I'd point us to https://python-poetry.org/ ... [ which is where I'd
>>>>>> look take us, but I'm also not able to do all the work, so my
>>>>>> suggestion/preference doensn't matter that much ]
>>>>>>
>>>>>> https://python-poetry.org/docs/pyproject#the-pyprojecttoml-file <-
>>>>>> for info on pyproject.toml file.
>>>>>>
>>>>>> Notice the use of a 'lock' file is very valuable, ex:
>>>>>> https://python-poetry.org/docs/basic-usage/#committing-your-poetrylock-file-to-version-control
>>>>>>
>>>>>> I haven't come across `build`, that might be great too. I'd
>>>>>> highlight that Poetry is pretty common across industry these days,
>>>>>> rock-solid, ecosystem of interoperability, users, etc... If not familiar,
>>>>>> PLEASE have a look at that.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Aug 28, 2023 at 8:04 AM Kerry Donny-Clark via dev <
>>>>>> dev@beam.apache.org> wrote:
>>>>>>
>>>>>>> +1
>>>>>>> Hi Anand,
>>>>>>> I appreciate this effort. Managing python dependencies has been a
>>>>>>> major pain point for me, and I think this approach would help.
>>>>>>> Kerry
>>>>>>>
>>>>>>> On Mon, Aug 28, 2023 at 10:14 AM Anand Inguva via dev <
>>>>>>> dev@beam.apache.org> wrote:
>>>>>>>
>>>>>>>> Hello Beam Dev Team,
>>>>>>>>
>>>>>>>> I've compiled a design document
>>>>>>>> <https://docs.google.com/document/d/17-y48WW25-VGBWZNyTdoN0WUN03k9ZhJjLp9wtyG1Wc/edit#heading=h.wskna8eurvjv>[1]
>>>>>>>> proposing the integration of pyproject.toml into Apache Beam's Python build
>>>>>>>> process. Your insights and feedback would be invaluable.
>>>>>>>>
>>>>>>>> What is pyproject.toml?
>>>>>>>> pyproject.toml is a configuration file that specifies a project's
>>>>>>>> build dependencies and other project-related metadata in a standardized
>>>>>>>> format. Before pyproject.toml, Python projects often had multiple
>>>>>>>> configuration files (like setup.py, setup.cfg, and requirements.txt).
>>>>>>>> pyproject.toml aims to centralize these configurations into one place,
>>>>>>>> making project setups more organized and straightforward. One of the
>>>>>>>> significant features enabled by pyproject.toml is the ability to perform
>>>>>>>> isolated builds. This ensures that build dependencies are separated from
>>>>>>>> the project's runtime dependencies, leading to more consistent and
>>>>>>>> reproducible builds.
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> https://docs.google.com/document/d/17-y48WW25-VGBWZNyTdoN0WUN03k9ZhJjLp9wtyG1Wc/edit#heading=h.wskna8eurvjv
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Anand
>>>>>>>>
>>>>>>>
Re: Proposal for pyproject.toml Support in Apache Beam Python
Posted by Robert Bradshaw via dev <de...@beam.apache.org>.
On Thu, Oct 12, 2023 at 2:04 PM Anand Inguva <an...@google.com> wrote:
> I am in the process of updating the documentation at
> https://cwiki.apache.org/confluence/display/BEAM/Python+Tips related to
> setup.py/pyproject.toml changes, but yes you can't call setup.py directly
> because it might fail due to the lack of presence of beam python's build
> time dependencies.
>
> With regards to other files(eg:protos), we will follow the similar
> behavior as before(generating proros using `gen_protos.py`).
>
Meaning this will be called automatically when needed (e.g. from pytest)?
> On Thu, Oct 12, 2023 at 4:01 PM Robert Bradshaw <ro...@google.com>
> wrote:
>
>> Does this change any development practices? E.g. if I clone the repo, I'm
>> assuming I couldn't run "setup.py test" anymore. What about the generated
>> files (like protos, or the yaml definitions copied from other parts of the
>> repo)?
>>
>> On Thu, Oct 12, 2023 at 12:27 PM Anand Inguva via dev <
>> dev@beam.apache.org> wrote:
>>
>>> The PR https://github.com/apache/beam/pull/28385 is merged today. If
>>> there are any observed failures, please comment on the PR and I will follow
>>> up with a forward fix. Thanks.
>>>
>>> On Fri, Sep 1, 2023 at 2:30 PM Anand Inguva <an...@google.com>
>>> wrote:
>>>
>>>> Since there is positive feedback from the dev community, I am going
>>>> ahead and implementing this proposal for Python SDK.
>>>>
>>>> @austin@apache.org <au...@apache.org> Initially let's move forward
>>>> with the setuptools as backend for building package and as part of the
>>>> future work, we can find a better backend than setuptools.
>>>>
>>>> Thanks for the feedback.
>>>> Anand
>>>>
>>>> On Mon, Aug 28, 2023 at 12:00 PM Austin Bennett <au...@apache.org>
>>>> wrote:
>>>>
>>>>> I've thought about this a ton, but haven't been in a position to
>>>>> undertake the work. Thanks for bringing this up, @Anand Inguva
>>>>> <an...@google.com> !
>>>>>
>>>>> I'd point us to https://python-poetry.org/ ... [ which is where I'd
>>>>> look take us, but I'm also not able to do all the work, so my
>>>>> suggestion/preference doensn't matter that much ]
>>>>>
>>>>> https://python-poetry.org/docs/pyproject#the-pyprojecttoml-file <-
>>>>> for info on pyproject.toml file.
>>>>>
>>>>> Notice the use of a 'lock' file is very valuable, ex:
>>>>> https://python-poetry.org/docs/basic-usage/#committing-your-poetrylock-file-to-version-control
>>>>>
>>>>> I haven't come across `build`, that might be great too. I'd highlight
>>>>> that Poetry is pretty common across industry these days, rock-solid,
>>>>> ecosystem of interoperability, users, etc... If not familiar, PLEASE have
>>>>> a look at that.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Aug 28, 2023 at 8:04 AM Kerry Donny-Clark via dev <
>>>>> dev@beam.apache.org> wrote:
>>>>>
>>>>>> +1
>>>>>> Hi Anand,
>>>>>> I appreciate this effort. Managing python dependencies has been a
>>>>>> major pain point for me, and I think this approach would help.
>>>>>> Kerry
>>>>>>
>>>>>> On Mon, Aug 28, 2023 at 10:14 AM Anand Inguva via dev <
>>>>>> dev@beam.apache.org> wrote:
>>>>>>
>>>>>>> Hello Beam Dev Team,
>>>>>>>
>>>>>>> I've compiled a design document
>>>>>>> <https://docs.google.com/document/d/17-y48WW25-VGBWZNyTdoN0WUN03k9ZhJjLp9wtyG1Wc/edit#heading=h.wskna8eurvjv>[1]
>>>>>>> proposing the integration of pyproject.toml into Apache Beam's Python build
>>>>>>> process. Your insights and feedback would be invaluable.
>>>>>>>
>>>>>>> What is pyproject.toml?
>>>>>>> pyproject.toml is a configuration file that specifies a project's
>>>>>>> build dependencies and other project-related metadata in a standardized
>>>>>>> format. Before pyproject.toml, Python projects often had multiple
>>>>>>> configuration files (like setup.py, setup.cfg, and requirements.txt).
>>>>>>> pyproject.toml aims to centralize these configurations into one place,
>>>>>>> making project setups more organized and straightforward. One of the
>>>>>>> significant features enabled by pyproject.toml is the ability to perform
>>>>>>> isolated builds. This ensures that build dependencies are separated from
>>>>>>> the project's runtime dependencies, leading to more consistent and
>>>>>>> reproducible builds.
>>>>>>>
>>>>>>> [1]
>>>>>>> https://docs.google.com/document/d/17-y48WW25-VGBWZNyTdoN0WUN03k9ZhJjLp9wtyG1Wc/edit#heading=h.wskna8eurvjv
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Anand
>>>>>>>
>>>>>>
Re: Proposal for pyproject.toml Support in Apache Beam Python
Posted by Anand Inguva via dev <de...@beam.apache.org>.
I am in the process of updating the documentation at
https://cwiki.apache.org/confluence/display/BEAM/Python+Tips related to
setup.py/pyproject.toml changes, but yes you can't call setup.py directly
because it might fail due to the lack of presence of beam python's build
time dependencies.
With regards to other files(eg:protos), we will follow the similar behavior
as before(generating proros using `gen_protos.py`).
On Thu, Oct 12, 2023 at 4:01 PM Robert Bradshaw <ro...@google.com> wrote:
> Does this change any development practices? E.g. if I clone the repo, I'm
> assuming I couldn't run "setup.py test" anymore. What about the generated
> files (like protos, or the yaml definitions copied from other parts of the
> repo)?
>
> On Thu, Oct 12, 2023 at 12:27 PM Anand Inguva via dev <de...@beam.apache.org>
> wrote:
>
>> The PR https://github.com/apache/beam/pull/28385 is merged today. If
>> there are any observed failures, please comment on the PR and I will follow
>> up with a forward fix. Thanks.
>>
>> On Fri, Sep 1, 2023 at 2:30 PM Anand Inguva <an...@google.com>
>> wrote:
>>
>>> Since there is positive feedback from the dev community, I am going
>>> ahead and implementing this proposal for Python SDK.
>>>
>>> @austin@apache.org <au...@apache.org> Initially let's move forward
>>> with the setuptools as backend for building package and as part of the
>>> future work, we can find a better backend than setuptools.
>>>
>>> Thanks for the feedback.
>>> Anand
>>>
>>> On Mon, Aug 28, 2023 at 12:00 PM Austin Bennett <au...@apache.org>
>>> wrote:
>>>
>>>> I've thought about this a ton, but haven't been in a position to
>>>> undertake the work. Thanks for bringing this up, @Anand Inguva
>>>> <an...@google.com> !
>>>>
>>>> I'd point us to https://python-poetry.org/ ... [ which is where I'd
>>>> look take us, but I'm also not able to do all the work, so my
>>>> suggestion/preference doensn't matter that much ]
>>>>
>>>> https://python-poetry.org/docs/pyproject#the-pyprojecttoml-file <- for
>>>> info on pyproject.toml file.
>>>>
>>>> Notice the use of a 'lock' file is very valuable, ex:
>>>> https://python-poetry.org/docs/basic-usage/#committing-your-poetrylock-file-to-version-control
>>>>
>>>> I haven't come across `build`, that might be great too. I'd highlight
>>>> that Poetry is pretty common across industry these days, rock-solid,
>>>> ecosystem of interoperability, users, etc... If not familiar, PLEASE have
>>>> a look at that.
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Aug 28, 2023 at 8:04 AM Kerry Donny-Clark via dev <
>>>> dev@beam.apache.org> wrote:
>>>>
>>>>> +1
>>>>> Hi Anand,
>>>>> I appreciate this effort. Managing python dependencies has been a
>>>>> major pain point for me, and I think this approach would help.
>>>>> Kerry
>>>>>
>>>>> On Mon, Aug 28, 2023 at 10:14 AM Anand Inguva via dev <
>>>>> dev@beam.apache.org> wrote:
>>>>>
>>>>>> Hello Beam Dev Team,
>>>>>>
>>>>>> I've compiled a design document
>>>>>> <https://docs.google.com/document/d/17-y48WW25-VGBWZNyTdoN0WUN03k9ZhJjLp9wtyG1Wc/edit#heading=h.wskna8eurvjv>[1]
>>>>>> proposing the integration of pyproject.toml into Apache Beam's Python build
>>>>>> process. Your insights and feedback would be invaluable.
>>>>>>
>>>>>> What is pyproject.toml?
>>>>>> pyproject.toml is a configuration file that specifies a project's
>>>>>> build dependencies and other project-related metadata in a standardized
>>>>>> format. Before pyproject.toml, Python projects often had multiple
>>>>>> configuration files (like setup.py, setup.cfg, and requirements.txt).
>>>>>> pyproject.toml aims to centralize these configurations into one place,
>>>>>> making project setups more organized and straightforward. One of the
>>>>>> significant features enabled by pyproject.toml is the ability to perform
>>>>>> isolated builds. This ensures that build dependencies are separated from
>>>>>> the project's runtime dependencies, leading to more consistent and
>>>>>> reproducible builds.
>>>>>>
>>>>>> [1]
>>>>>> https://docs.google.com/document/d/17-y48WW25-VGBWZNyTdoN0WUN03k9ZhJjLp9wtyG1Wc/edit#heading=h.wskna8eurvjv
>>>>>>
>>>>>> Thanks,
>>>>>> Anand
>>>>>>
>>>>>
Re: Proposal for pyproject.toml Support in Apache Beam Python
Posted by Robert Bradshaw via dev <de...@beam.apache.org>.
Does this change any development practices? E.g. if I clone the repo, I'm
assuming I couldn't run "setup.py test" anymore. What about the generated
files (like protos, or the yaml definitions copied from other parts of the
repo)?
On Thu, Oct 12, 2023 at 12:27 PM Anand Inguva via dev <de...@beam.apache.org>
wrote:
> The PR https://github.com/apache/beam/pull/28385 is merged today. If
> there are any observed failures, please comment on the PR and I will follow
> up with a forward fix. Thanks.
>
> On Fri, Sep 1, 2023 at 2:30 PM Anand Inguva <an...@google.com>
> wrote:
>
>> Since there is positive feedback from the dev community, I am going ahead
>> and implementing this proposal for Python SDK.
>>
>> @austin@apache.org <au...@apache.org> Initially let's move forward
>> with the setuptools as backend for building package and as part of the
>> future work, we can find a better backend than setuptools.
>>
>> Thanks for the feedback.
>> Anand
>>
>> On Mon, Aug 28, 2023 at 12:00 PM Austin Bennett <au...@apache.org>
>> wrote:
>>
>>> I've thought about this a ton, but haven't been in a position to
>>> undertake the work. Thanks for bringing this up, @Anand Inguva
>>> <an...@google.com> !
>>>
>>> I'd point us to https://python-poetry.org/ ... [ which is where I'd
>>> look take us, but I'm also not able to do all the work, so my
>>> suggestion/preference doensn't matter that much ]
>>>
>>> https://python-poetry.org/docs/pyproject#the-pyprojecttoml-file <- for
>>> info on pyproject.toml file.
>>>
>>> Notice the use of a 'lock' file is very valuable, ex:
>>> https://python-poetry.org/docs/basic-usage/#committing-your-poetrylock-file-to-version-control
>>>
>>> I haven't come across `build`, that might be great too. I'd highlight
>>> that Poetry is pretty common across industry these days, rock-solid,
>>> ecosystem of interoperability, users, etc... If not familiar, PLEASE have
>>> a look at that.
>>>
>>>
>>>
>>>
>>> On Mon, Aug 28, 2023 at 8:04 AM Kerry Donny-Clark via dev <
>>> dev@beam.apache.org> wrote:
>>>
>>>> +1
>>>> Hi Anand,
>>>> I appreciate this effort. Managing python dependencies has been a major
>>>> pain point for me, and I think this approach would help.
>>>> Kerry
>>>>
>>>> On Mon, Aug 28, 2023 at 10:14 AM Anand Inguva via dev <
>>>> dev@beam.apache.org> wrote:
>>>>
>>>>> Hello Beam Dev Team,
>>>>>
>>>>> I've compiled a design document
>>>>> <https://docs.google.com/document/d/17-y48WW25-VGBWZNyTdoN0WUN03k9ZhJjLp9wtyG1Wc/edit#heading=h.wskna8eurvjv>[1]
>>>>> proposing the integration of pyproject.toml into Apache Beam's Python build
>>>>> process. Your insights and feedback would be invaluable.
>>>>>
>>>>> What is pyproject.toml?
>>>>> pyproject.toml is a configuration file that specifies a project's
>>>>> build dependencies and other project-related metadata in a standardized
>>>>> format. Before pyproject.toml, Python projects often had multiple
>>>>> configuration files (like setup.py, setup.cfg, and requirements.txt).
>>>>> pyproject.toml aims to centralize these configurations into one place,
>>>>> making project setups more organized and straightforward. One of the
>>>>> significant features enabled by pyproject.toml is the ability to perform
>>>>> isolated builds. This ensures that build dependencies are separated from
>>>>> the project's runtime dependencies, leading to more consistent and
>>>>> reproducible builds.
>>>>>
>>>>> [1]
>>>>> https://docs.google.com/document/d/17-y48WW25-VGBWZNyTdoN0WUN03k9ZhJjLp9wtyG1Wc/edit#heading=h.wskna8eurvjv
>>>>>
>>>>> Thanks,
>>>>> Anand
>>>>>
>>>>
Re: Proposal for pyproject.toml Support in Apache Beam Python
Posted by Anand Inguva via dev <de...@beam.apache.org>.
The PR https://github.com/apache/beam/pull/28385 is merged today. If there
are any observed failures, please comment on the PR and I will follow up
with a forward fix. Thanks.
On Fri, Sep 1, 2023 at 2:30 PM Anand Inguva <an...@google.com> wrote:
> Since there is positive feedback from the dev community, I am going ahead
> and implementing this proposal for Python SDK.
>
> @austin@apache.org <au...@apache.org> Initially let's move forward with
> the setuptools as backend for building package and as part of the future
> work, we can find a better backend than setuptools.
>
> Thanks for the feedback.
> Anand
>
> On Mon, Aug 28, 2023 at 12:00 PM Austin Bennett <au...@apache.org> wrote:
>
>> I've thought about this a ton, but haven't been in a position to
>> undertake the work. Thanks for bringing this up, @Anand Inguva
>> <an...@google.com> !
>>
>> I'd point us to https://python-poetry.org/ ... [ which is where I'd
>> look take us, but I'm also not able to do all the work, so my
>> suggestion/preference doensn't matter that much ]
>>
>> https://python-poetry.org/docs/pyproject#the-pyprojecttoml-file <- for
>> info on pyproject.toml file.
>>
>> Notice the use of a 'lock' file is very valuable, ex:
>> https://python-poetry.org/docs/basic-usage/#committing-your-poetrylock-file-to-version-control
>>
>> I haven't come across `build`, that might be great too. I'd highlight
>> that Poetry is pretty common across industry these days, rock-solid,
>> ecosystem of interoperability, users, etc... If not familiar, PLEASE have
>> a look at that.
>>
>>
>>
>>
>> On Mon, Aug 28, 2023 at 8:04 AM Kerry Donny-Clark via dev <
>> dev@beam.apache.org> wrote:
>>
>>> +1
>>> Hi Anand,
>>> I appreciate this effort. Managing python dependencies has been a major
>>> pain point for me, and I think this approach would help.
>>> Kerry
>>>
>>> On Mon, Aug 28, 2023 at 10:14 AM Anand Inguva via dev <
>>> dev@beam.apache.org> wrote:
>>>
>>>> Hello Beam Dev Team,
>>>>
>>>> I've compiled a design document
>>>> <https://docs.google.com/document/d/17-y48WW25-VGBWZNyTdoN0WUN03k9ZhJjLp9wtyG1Wc/edit#heading=h.wskna8eurvjv>[1]
>>>> proposing the integration of pyproject.toml into Apache Beam's Python build
>>>> process. Your insights and feedback would be invaluable.
>>>>
>>>> What is pyproject.toml?
>>>> pyproject.toml is a configuration file that specifies a project's build
>>>> dependencies and other project-related metadata in a standardized
>>>> format. Before pyproject.toml, Python projects often had multiple
>>>> configuration files (like setup.py, setup.cfg, and requirements.txt).
>>>> pyproject.toml aims to centralize these configurations into one place,
>>>> making project setups more organized and straightforward. One of the
>>>> significant features enabled by pyproject.toml is the ability to perform
>>>> isolated builds. This ensures that build dependencies are separated from
>>>> the project's runtime dependencies, leading to more consistent and
>>>> reproducible builds.
>>>>
>>>> [1]
>>>> https://docs.google.com/document/d/17-y48WW25-VGBWZNyTdoN0WUN03k9ZhJjLp9wtyG1Wc/edit#heading=h.wskna8eurvjv
>>>>
>>>> Thanks,
>>>> Anand
>>>>
>>>
Re: Proposal for pyproject.toml Support in Apache Beam Python
Posted by Anand Inguva via dev <de...@beam.apache.org>.
Since there is positive feedback from the dev community, I am going ahead
and implementing this proposal for Python SDK.
@austin@apache.org <au...@apache.org> Initially let's move forward with
the setuptools as backend for building package and as part of the future
work, we can find a better backend than setuptools.
Thanks for the feedback.
Anand
On Mon, Aug 28, 2023 at 12:00 PM Austin Bennett <au...@apache.org> wrote:
> I've thought about this a ton, but haven't been in a position to undertake
> the work. Thanks for bringing this up, @Anand Inguva
> <an...@google.com> !
>
> I'd point us to https://python-poetry.org/ ... [ which is where I'd look
> take us, but I'm also not able to do all the work, so my
> suggestion/preference doensn't matter that much ]
>
> https://python-poetry.org/docs/pyproject#the-pyprojecttoml-file <- for
> info on pyproject.toml file.
>
> Notice the use of a 'lock' file is very valuable, ex:
> https://python-poetry.org/docs/basic-usage/#committing-your-poetrylock-file-to-version-control
>
> I haven't come across `build`, that might be great too. I'd highlight
> that Poetry is pretty common across industry these days, rock-solid,
> ecosystem of interoperability, users, etc... If not familiar, PLEASE have
> a look at that.
>
>
>
>
> On Mon, Aug 28, 2023 at 8:04 AM Kerry Donny-Clark via dev <
> dev@beam.apache.org> wrote:
>
>> +1
>> Hi Anand,
>> I appreciate this effort. Managing python dependencies has been a major
>> pain point for me, and I think this approach would help.
>> Kerry
>>
>> On Mon, Aug 28, 2023 at 10:14 AM Anand Inguva via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Hello Beam Dev Team,
>>>
>>> I've compiled a design document
>>> <https://docs.google.com/document/d/17-y48WW25-VGBWZNyTdoN0WUN03k9ZhJjLp9wtyG1Wc/edit#heading=h.wskna8eurvjv>[1]
>>> proposing the integration of pyproject.toml into Apache Beam's Python build
>>> process. Your insights and feedback would be invaluable.
>>>
>>> What is pyproject.toml?
>>> pyproject.toml is a configuration file that specifies a project's build
>>> dependencies and other project-related metadata in a standardized
>>> format. Before pyproject.toml, Python projects often had multiple
>>> configuration files (like setup.py, setup.cfg, and requirements.txt).
>>> pyproject.toml aims to centralize these configurations into one place,
>>> making project setups more organized and straightforward. One of the
>>> significant features enabled by pyproject.toml is the ability to perform
>>> isolated builds. This ensures that build dependencies are separated from
>>> the project's runtime dependencies, leading to more consistent and
>>> reproducible builds.
>>>
>>> [1]
>>> https://docs.google.com/document/d/17-y48WW25-VGBWZNyTdoN0WUN03k9ZhJjLp9wtyG1Wc/edit#heading=h.wskna8eurvjv
>>>
>>> Thanks,
>>> Anand
>>>
>>
Re: Proposal for pyproject.toml Support in Apache Beam Python
Posted by Austin Bennett <au...@apache.org>.
I've thought about this a ton, but haven't been in a position to undertake
the work. Thanks for bringing this up, @Anand Inguva
<an...@google.com> !
I'd point us to https://python-poetry.org/ ... [ which is where I'd look
take us, but I'm also not able to do all the work, so my
suggestion/preference doensn't matter that much ]
https://python-poetry.org/docs/pyproject#the-pyprojecttoml-file <- for info
on pyproject.toml file.
Notice the use of a 'lock' file is very valuable, ex:
https://python-poetry.org/docs/basic-usage/#committing-your-poetrylock-file-to-version-control
I haven't come across `build`, that might be great too. I'd highlight that
Poetry is pretty common across industry these days, rock-solid, ecosystem
of interoperability, users, etc... If not familiar, PLEASE have a look at
that.
On Mon, Aug 28, 2023 at 8:04 AM Kerry Donny-Clark via dev <
dev@beam.apache.org> wrote:
> +1
> Hi Anand,
> I appreciate this effort. Managing python dependencies has been a major
> pain point for me, and I think this approach would help.
> Kerry
>
> On Mon, Aug 28, 2023 at 10:14 AM Anand Inguva via dev <de...@beam.apache.org>
> wrote:
>
>> Hello Beam Dev Team,
>>
>> I've compiled a design document
>> <https://docs.google.com/document/d/17-y48WW25-VGBWZNyTdoN0WUN03k9ZhJjLp9wtyG1Wc/edit#heading=h.wskna8eurvjv>[1]
>> proposing the integration of pyproject.toml into Apache Beam's Python build
>> process. Your insights and feedback would be invaluable.
>>
>> What is pyproject.toml?
>> pyproject.toml is a configuration file that specifies a project's build
>> dependencies and other project-related metadata in a standardized
>> format. Before pyproject.toml, Python projects often had multiple
>> configuration files (like setup.py, setup.cfg, and requirements.txt).
>> pyproject.toml aims to centralize these configurations into one place,
>> making project setups more organized and straightforward. One of the
>> significant features enabled by pyproject.toml is the ability to perform
>> isolated builds. This ensures that build dependencies are separated from
>> the project's runtime dependencies, leading to more consistent and
>> reproducible builds.
>>
>> [1]
>> https://docs.google.com/document/d/17-y48WW25-VGBWZNyTdoN0WUN03k9ZhJjLp9wtyG1Wc/edit#heading=h.wskna8eurvjv
>>
>> Thanks,
>> Anand
>>
>
Re: Proposal for pyproject.toml Support in Apache Beam Python
Posted by Kerry Donny-Clark via dev <de...@beam.apache.org>.
+1
Hi Anand,
I appreciate this effort. Managing python dependencies has been a major
pain point for me, and I think this approach would help.
Kerry
On Mon, Aug 28, 2023 at 10:14 AM Anand Inguva via dev <de...@beam.apache.org>
wrote:
> Hello Beam Dev Team,
>
> I've compiled a design document
> <https://docs.google.com/document/d/17-y48WW25-VGBWZNyTdoN0WUN03k9ZhJjLp9wtyG1Wc/edit#heading=h.wskna8eurvjv>[1]
> proposing the integration of pyproject.toml into Apache Beam's Python build
> process. Your insights and feedback would be invaluable.
>
> What is pyproject.toml?
> pyproject.toml is a configuration file that specifies a project's build
> dependencies and other project-related metadata in a standardized
> format. Before pyproject.toml, Python projects often had multiple
> configuration files (like setup.py, setup.cfg, and requirements.txt).
> pyproject.toml aims to centralize these configurations into one place,
> making project setups more organized and straightforward. One of the
> significant features enabled by pyproject.toml is the ability to perform
> isolated builds. This ensures that build dependencies are separated from
> the project's runtime dependencies, leading to more consistent and
> reproducible builds.
>
> [1]
> https://docs.google.com/document/d/17-y48WW25-VGBWZNyTdoN0WUN03k9ZhJjLp9wtyG1Wc/edit#heading=h.wskna8eurvjv
>
> Thanks,
> Anand
>
Re: Proposal for pyproject.toml Support in Apache Beam Python
Posted by Danny McCormick via dev <de...@beam.apache.org>.
Thanks Anand! This generally sounds good to me. I left a few questions
before giving a full +1, but it definitely seems like we need some
migration effort here and this seems like a good route.
Thanks,
Danny
On Mon, Aug 28, 2023 at 10:14 AM Anand Inguva via dev <de...@beam.apache.org>
wrote:
> Hello Beam Dev Team,
>
> I've compiled a design document
> <https://docs.google.com/document/d/17-y48WW25-VGBWZNyTdoN0WUN03k9ZhJjLp9wtyG1Wc/edit#heading=h.wskna8eurvjv>[1]
> proposing the integration of pyproject.toml into Apache Beam's Python build
> process. Your insights and feedback would be invaluable.
>
> What is pyproject.toml?
> pyproject.toml is a configuration file that specifies a project's build
> dependencies and other project-related metadata in a standardized
> format. Before pyproject.toml, Python projects often had multiple
> configuration files (like setup.py, setup.cfg, and requirements.txt).
> pyproject.toml aims to centralize these configurations into one place,
> making project setups more organized and straightforward. One of the
> significant features enabled by pyproject.toml is the ability to perform
> isolated builds. This ensures that build dependencies are separated from
> the project's runtime dependencies, leading to more consistent and
> reproducible builds.
>
> [1]
> https://docs.google.com/document/d/17-y48WW25-VGBWZNyTdoN0WUN03k9ZhJjLp9wtyG1Wc/edit#heading=h.wskna8eurvjv
>
> Thanks,
> Anand
>