You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Boyuan Zhang <bo...@google.com> on 2018/08/03 20:16:38 UTC
Apache Beam Python Wheels Repository
Hey all,
I'm Boyuan Zhang from Google Dataflow Team, currently helping Release
Manager(Pablo Estrada) with 2.6.0 release. Since Beam decided to release
python wheels since 2.5.0, we need to create a wrapper repository(sample
repo <https://github.com/boyuanzz/apache-beam-wheels>) under apache to
build and stage released python wheels for each release. Anyone can help to
create this repository?
Thanks for all your help! Happy Friday~
Boyuan Zhang
Re: Apache Beam Python Wheels Repository
Posted by Maximilian Michels <mx...@apache.org>.
+1
Travis for building the Python wheels looks fine to me. Many Apache
projects use Travis in addition to Jenkins. Apache is also invested in
Travis [1] to ensure the build capacity is sufficient. In any case, we
could migrate away from Travis if it doesn't work out as expected.
We don't have to file a ticket with JIRA to create a new repository.
There is now: https://gitbox.apache.org/setup/newrepo.html
Cheers,
Max
[1] https://blogs.apache.org/infra/entry/apache_gains_additional_travis_ci
On 15.08.18 14:25, Robert Bradshaw wrote:
> For background, a separate repository that contains reference to Brett's
> Travis/Appveyer build scripts + project configuration information is the
> de facto standard for out of the box support for a wide variety of
> platform-specific Python wheels (aka binary builds).
>
> We also need a secure place to publish the artifacts, e.g. supplying
> (encrypted) credentials for Travis to commit the artifacts directly to
> SVN or any other place we can privately push/publicly pull ephemeral
> blobs as the build environments are torn down after the job completed.
> (Not sure what Apache infra has here that could act like a GCS bucket.)
>
> On Wed, Aug 8, 2018, 2:23 AM Boyuan Zhang <boyuanz@google.com
> <ma...@google.com>> wrote:
>
> Hey Pablo,
>
> For windows, I think Travis doesn't have a plan coming soon.
> But Matthew Brett's build scripts
> <https://github.com/matthew-brett/multibuild> actually supports
> windows platform in Appveyor <https://ci.appveyor.com/>, which we
> doesn't have chance to explore more.
>
> Boyuan
>
> On Tue, Aug 7, 2018 at 4:50 PM Pablo Estrada <pabloem@google.com
> <ma...@google.com>> wrote:
>
> Hi Boyuan,
> I think this is reasonable. I remember Robert supported this
> approach as well, and it is quite simple to support this. I
> am +1 on this.
>
> Do you know at all if there are any plans from Travis to support
> windows?
> Best
> -P.
>
> On Tue, Aug 7, 2018 at 2:04 PM Boyuan Zhang <boyuanz@google.com
> <ma...@google.com>> wrote:
>
> Thanks for Davor's suggestions. Following content discusses
> more about why we need a new repository. Hope they would be
> helpful!
>
> Purpose of this proposal
> - Facilitate building python wheels against multi-os
>
> Why a new repository is needed
> - Currently we chose Matthew Brett's build scripts
> <https://github.com/matthew-brett/multibuild> build Linux
> and macOS python wheels.
> - This build tool uses travis <https://travis-ci.com/> as
> Linux and maxOS build platform, which explains why we cannot
> use Jenkins.
> - In order to utilize this build tool based on their guide
> <https://github.com/matthew-brett/multibuild/blob/devel/README.rst>,
> we need to use a repo wrapper.
>
> Alternatives
> - Unfortunately, I didn't find any alternatives at this moment.
>
> Boyuan Zhang
>
> On Sat, Aug 4, 2018 at 9:50 AM Davor Bonaci
> <davor@apache.org <ma...@apache.org>> wrote:
>
> New repository is not a ticket, it is a self-serve thing.
>
> That said, you probably want to develop the proposal a
> bit further, understanding/educating others about the
> benefits of what you are proposing, any alternatives,
> why a repository is needed, why the sample repository
> has Travis CI when everything else is on Jenkins, how
> this fits into other decisions about repository
> management, and so on. Anything can be done, of course,
> but I'd suggest developing (or communicating, or
> educating) a bit more.
>
> (I'm fine with any approach.)
>
> On Fri, Aug 3, 2018 at 3:29 PM, Ahmet Altay
> <altay@google.com <ma...@google.com>> wrote:
>
> This LGTM, also greatly simplifies the creation of
> wheel files for multiple platforms.
>
> I can file an INFRA ticket to create a new repo to
> host wheel setup. Does anybody have experience with
> setting up a new repo similar to this?
>
> Ahmet
>
> On Fri, Aug 3, 2018 at 1:16 PM, Boyuan Zhang
> <boyuanz@google.com <ma...@google.com>> wrote:
>
> Hey all,
>
> I'm Boyuan Zhang from Google Dataflow Team,
> currently helping Release Manager(Pablo Estrada)
> with 2.6.0 release. Since Beam decided to
> release python wheels since 2.5.0, we need to
> create a wrapper repository(sample repo
> <https://github.com/boyuanzz/apache-beam-wheels>) under
> apache to build and stage released python wheels
> for each release. Anyone can help to create this
> repository?
>
> Thanks for all your help! Happy Friday~
>
> Boyuan Zhang
>
>
>
> --
> Got feedback? go/pabloem-feedback
> <https://goto.google.com/pabloem-feedback>
>
Re: Apache Beam Python Wheels Repository
Posted by Robert Bradshaw <ro...@google.com>.
For background, a separate repository that contains reference to Brett's
Travis/Appveyer build scripts + project configuration information is the de
facto standard for out of the box support for a wide variety of
platform-specific Python wheels (aka binary builds).
We also need a secure place to publish the artifacts, e.g. supplying
(encrypted) credentials for Travis to commit the artifacts directly to SVN
or any other place we can privately push/publicly pull ephemeral blobs as
the build environments are torn down after the job completed. (Not sure
what Apache infra has here that could act like a GCS bucket.)
On Wed, Aug 8, 2018, 2:23 AM Boyuan Zhang <bo...@google.com> wrote:
> Hey Pablo,
>
> For windows, I think Travis doesn't have a plan coming soon. But Matthew
> Brett's build scripts <https://github.com/matthew-brett/multibuild> actually
> supports windows platform in Appveyor <https://ci.appveyor.com/>, which
> we doesn't have chance to explore more.
>
> Boyuan
>
> On Tue, Aug 7, 2018 at 4:50 PM Pablo Estrada <pa...@google.com> wrote:
>
>> Hi Boyuan,
>> I think this is reasonable. I remember Robert supported this approach as
>> well, and it is quite simple to support this. I am +1 on this.
>>
>> Do you know at all if there are any plans from Travis to support windows?
>> Best
>> -P.
>>
>> On Tue, Aug 7, 2018 at 2:04 PM Boyuan Zhang <bo...@google.com> wrote:
>>
>>> Thanks for Davor's suggestions. Following content discusses more about
>>> why we need a new repository. Hope they would be helpful!
>>>
>>> Purpose of this proposal
>>> - Facilitate building python wheels against multi-os
>>>
>>> Why a new repository is needed
>>> - Currently we chose Matthew Brett's build scripts
>>> <https://github.com/matthew-brett/multibuild> build Linux and macOS
>>> python wheels.
>>> - This build tool uses travis <https://travis-ci.com/> as Linux and
>>> maxOS build platform, which explains why we cannot use Jenkins.
>>> - In order to utilize this build tool based on their guide
>>> <https://github.com/matthew-brett/multibuild/blob/devel/README.rst>, we
>>> need to use a repo wrapper.
>>>
>>> Alternatives
>>> - Unfortunately, I didn't find any alternatives at this moment.
>>>
>>> Boyuan Zhang
>>>
>>> On Sat, Aug 4, 2018 at 9:50 AM Davor Bonaci <da...@apache.org> wrote:
>>>
>>>> New repository is not a ticket, it is a self-serve thing.
>>>>
>>>> That said, you probably want to develop the proposal a bit further,
>>>> understanding/educating others about the benefits of what you are
>>>> proposing, any alternatives, why a repository is needed, why the sample
>>>> repository has Travis CI when everything else is on Jenkins, how this fits
>>>> into other decisions about repository management, and so on. Anything can
>>>> be done, of course, but I'd suggest developing (or communicating, or
>>>> educating) a bit more.
>>>>
>>>> (I'm fine with any approach.)
>>>>
>>>> On Fri, Aug 3, 2018 at 3:29 PM, Ahmet Altay <al...@google.com> wrote:
>>>>
>>>>> This LGTM, also greatly simplifies the creation of wheel files for
>>>>> multiple platforms.
>>>>>
>>>>> I can file an INFRA ticket to create a new repo to host wheel setup.
>>>>> Does anybody have experience with setting up a new repo similar to this?
>>>>>
>>>>> Ahmet
>>>>>
>>>>> On Fri, Aug 3, 2018 at 1:16 PM, Boyuan Zhang <bo...@google.com>
>>>>> wrote:
>>>>>
>>>>>> Hey all,
>>>>>>
>>>>>> I'm Boyuan Zhang from Google Dataflow Team, currently helping Release
>>>>>> Manager(Pablo Estrada) with 2.6.0 release. Since Beam decided to
>>>>>> release python wheels since 2.5.0, we need to create a wrapper repository(sample
>>>>>> repo <https://github.com/boyuanzz/apache-beam-wheels>) under apache
>>>>>> to build and stage released python wheels for each release. Anyone can help
>>>>>> to create this repository?
>>>>>>
>>>>>> Thanks for all your help! Happy Friday~
>>>>>>
>>>>>> Boyuan Zhang
>>>>>>
>>>>>
>>>>>
>>>> --
>> Got feedback? go/pabloem-feedback
>> <https://goto.google.com/pabloem-feedback>
>>
>
Re: Apache Beam Python Wheels Repository
Posted by Boyuan Zhang <bo...@google.com>.
Hey Pablo,
For windows, I think Travis doesn't have a plan coming soon. But Matthew
Brett's build scripts <https://github.com/matthew-brett/multibuild> actually
supports windows platform in Appveyor <https://ci.appveyor.com/>, which we
doesn't have chance to explore more.
Boyuan
On Tue, Aug 7, 2018 at 4:50 PM Pablo Estrada <pa...@google.com> wrote:
> Hi Boyuan,
> I think this is reasonable. I remember Robert supported this approach as
> well, and it is quite simple to support this. I am +1 on this.
>
> Do you know at all if there are any plans from Travis to support windows?
> Best
> -P.
>
> On Tue, Aug 7, 2018 at 2:04 PM Boyuan Zhang <bo...@google.com> wrote:
>
>> Thanks for Davor's suggestions. Following content discusses more about
>> why we need a new repository. Hope they would be helpful!
>>
>> Purpose of this proposal
>> - Facilitate building python wheels against multi-os
>>
>> Why a new repository is needed
>> - Currently we chose Matthew Brett's build scripts
>> <https://github.com/matthew-brett/multibuild> build Linux and macOS
>> python wheels.
>> - This build tool uses travis <https://travis-ci.com/> as Linux and
>> maxOS build platform, which explains why we cannot use Jenkins.
>> - In order to utilize this build tool based on their guide
>> <https://github.com/matthew-brett/multibuild/blob/devel/README.rst>, we
>> need to use a repo wrapper.
>>
>> Alternatives
>> - Unfortunately, I didn't find any alternatives at this moment.
>>
>> Boyuan Zhang
>>
>> On Sat, Aug 4, 2018 at 9:50 AM Davor Bonaci <da...@apache.org> wrote:
>>
>>> New repository is not a ticket, it is a self-serve thing.
>>>
>>> That said, you probably want to develop the proposal a bit further,
>>> understanding/educating others about the benefits of what you are
>>> proposing, any alternatives, why a repository is needed, why the sample
>>> repository has Travis CI when everything else is on Jenkins, how this fits
>>> into other decisions about repository management, and so on. Anything can
>>> be done, of course, but I'd suggest developing (or communicating, or
>>> educating) a bit more.
>>>
>>> (I'm fine with any approach.)
>>>
>>> On Fri, Aug 3, 2018 at 3:29 PM, Ahmet Altay <al...@google.com> wrote:
>>>
>>>> This LGTM, also greatly simplifies the creation of wheel files for
>>>> multiple platforms.
>>>>
>>>> I can file an INFRA ticket to create a new repo to host wheel setup.
>>>> Does anybody have experience with setting up a new repo similar to this?
>>>>
>>>> Ahmet
>>>>
>>>> On Fri, Aug 3, 2018 at 1:16 PM, Boyuan Zhang <bo...@google.com>
>>>> wrote:
>>>>
>>>>> Hey all,
>>>>>
>>>>> I'm Boyuan Zhang from Google Dataflow Team, currently helping Release
>>>>> Manager(Pablo Estrada) with 2.6.0 release. Since Beam decided to
>>>>> release python wheels since 2.5.0, we need to create a wrapper repository(sample
>>>>> repo <https://github.com/boyuanzz/apache-beam-wheels>) under apache
>>>>> to build and stage released python wheels for each release. Anyone can help
>>>>> to create this repository?
>>>>>
>>>>> Thanks for all your help! Happy Friday~
>>>>>
>>>>> Boyuan Zhang
>>>>>
>>>>
>>>>
>>> --
> Got feedback? go/pabloem-feedback
> <https://goto.google.com/pabloem-feedback>
>
Re: Apache Beam Python Wheels Repository
Posted by Pablo Estrada <pa...@google.com>.
Hi Boyuan,
I think this is reasonable. I remember Robert supported this approach as
well, and it is quite simple to support this. I am +1 on this.
Do you know at all if there are any plans from Travis to support windows?
Best
-P.
On Tue, Aug 7, 2018 at 2:04 PM Boyuan Zhang <bo...@google.com> wrote:
> Thanks for Davor's suggestions. Following content discusses more about why
> we need a new repository. Hope they would be helpful!
>
> Purpose of this proposal
> - Facilitate building python wheels against multi-os
>
> Why a new repository is needed
> - Currently we chose Matthew Brett's build scripts
> <https://github.com/matthew-brett/multibuild> build Linux and macOS
> python wheels.
> - This build tool uses travis <https://travis-ci.com/> as Linux and maxOS
> build platform, which explains why we cannot use Jenkins.
> - In order to utilize this build tool based on their guide
> <https://github.com/matthew-brett/multibuild/blob/devel/README.rst>, we
> need to use a repo wrapper.
>
> Alternatives
> - Unfortunately, I didn't find any alternatives at this moment.
>
> Boyuan Zhang
>
> On Sat, Aug 4, 2018 at 9:50 AM Davor Bonaci <da...@apache.org> wrote:
>
>> New repository is not a ticket, it is a self-serve thing.
>>
>> That said, you probably want to develop the proposal a bit further,
>> understanding/educating others about the benefits of what you are
>> proposing, any alternatives, why a repository is needed, why the sample
>> repository has Travis CI when everything else is on Jenkins, how this fits
>> into other decisions about repository management, and so on. Anything can
>> be done, of course, but I'd suggest developing (or communicating, or
>> educating) a bit more.
>>
>> (I'm fine with any approach.)
>>
>> On Fri, Aug 3, 2018 at 3:29 PM, Ahmet Altay <al...@google.com> wrote:
>>
>>> This LGTM, also greatly simplifies the creation of wheel files for
>>> multiple platforms.
>>>
>>> I can file an INFRA ticket to create a new repo to host wheel setup.
>>> Does anybody have experience with setting up a new repo similar to this?
>>>
>>> Ahmet
>>>
>>> On Fri, Aug 3, 2018 at 1:16 PM, Boyuan Zhang <bo...@google.com> wrote:
>>>
>>>> Hey all,
>>>>
>>>> I'm Boyuan Zhang from Google Dataflow Team, currently helping Release
>>>> Manager(Pablo Estrada) with 2.6.0 release. Since Beam decided to
>>>> release python wheels since 2.5.0, we need to create a wrapper repository(sample
>>>> repo <https://github.com/boyuanzz/apache-beam-wheels>) under apache to
>>>> build and stage released python wheels for each release. Anyone can help to
>>>> create this repository?
>>>>
>>>> Thanks for all your help! Happy Friday~
>>>>
>>>> Boyuan Zhang
>>>>
>>>
>>>
>> --
Got feedback? go/pabloem-feedback
Re: Apache Beam Python Wheels Repository
Posted by Boyuan Zhang <bo...@google.com>.
Thanks for Davor's suggestions. Following content discusses more about why
we need a new repository. Hope they would be helpful!
Purpose of this proposal
- Facilitate building python wheels against multi-os
Why a new repository is needed
- Currently we chose Matthew Brett's build scripts
<https://github.com/matthew-brett/multibuild> build Linux and macOS python
wheels.
- This build tool uses travis <https://travis-ci.com/> as Linux and maxOS
build platform, which explains why we cannot use Jenkins.
- In order to utilize this build tool based on their guide
<https://github.com/matthew-brett/multibuild/blob/devel/README.rst>, we
need to use a repo wrapper.
Alternatives
- Unfortunately, I didn't find any alternatives at this moment.
Boyuan Zhang
On Sat, Aug 4, 2018 at 9:50 AM Davor Bonaci <da...@apache.org> wrote:
> New repository is not a ticket, it is a self-serve thing.
>
> That said, you probably want to develop the proposal a bit further,
> understanding/educating others about the benefits of what you are
> proposing, any alternatives, why a repository is needed, why the sample
> repository has Travis CI when everything else is on Jenkins, how this fits
> into other decisions about repository management, and so on. Anything can
> be done, of course, but I'd suggest developing (or communicating, or
> educating) a bit more.
>
> (I'm fine with any approach.)
>
> On Fri, Aug 3, 2018 at 3:29 PM, Ahmet Altay <al...@google.com> wrote:
>
>> This LGTM, also greatly simplifies the creation of wheel files for
>> multiple platforms.
>>
>> I can file an INFRA ticket to create a new repo to host wheel setup. Does
>> anybody have experience with setting up a new repo similar to this?
>>
>> Ahmet
>>
>> On Fri, Aug 3, 2018 at 1:16 PM, Boyuan Zhang <bo...@google.com> wrote:
>>
>>> Hey all,
>>>
>>> I'm Boyuan Zhang from Google Dataflow Team, currently helping Release
>>> Manager(Pablo Estrada) with 2.6.0 release. Since Beam decided to
>>> release python wheels since 2.5.0, we need to create a wrapper repository(sample
>>> repo <https://github.com/boyuanzz/apache-beam-wheels>) under apache to
>>> build and stage released python wheels for each release. Anyone can help to
>>> create this repository?
>>>
>>> Thanks for all your help! Happy Friday~
>>>
>>> Boyuan Zhang
>>>
>>
>>
>
Re: Apache Beam Python Wheels Repository
Posted by Davor Bonaci <da...@apache.org>.
New repository is not a ticket, it is a self-serve thing.
That said, you probably want to develop the proposal a bit further,
understanding/educating others about the benefits of what you are
proposing, any alternatives, why a repository is needed, why the sample
repository has Travis CI when everything else is on Jenkins, how this fits
into other decisions about repository management, and so on. Anything can
be done, of course, but I'd suggest developing (or communicating, or
educating) a bit more.
(I'm fine with any approach.)
On Fri, Aug 3, 2018 at 3:29 PM, Ahmet Altay <al...@google.com> wrote:
> This LGTM, also greatly simplifies the creation of wheel files for
> multiple platforms.
>
> I can file an INFRA ticket to create a new repo to host wheel setup. Does
> anybody have experience with setting up a new repo similar to this?
>
> Ahmet
>
> On Fri, Aug 3, 2018 at 1:16 PM, Boyuan Zhang <bo...@google.com> wrote:
>
>> Hey all,
>>
>> I'm Boyuan Zhang from Google Dataflow Team, currently helping Release
>> Manager(Pablo Estrada) with 2.6.0 release. Since Beam decided to release
>> python wheels since 2.5.0, we need to create a wrapper repository(sample
>> repo <https://github.com/boyuanzz/apache-beam-wheels>) under apache to
>> build and stage released python wheels for each release. Anyone can help to
>> create this repository?
>>
>> Thanks for all your help! Happy Friday~
>>
>> Boyuan Zhang
>>
>
>
Re: Apache Beam Python Wheels Repository
Posted by Ahmet Altay <al...@google.com>.
This LGTM, also greatly simplifies the creation of wheel files for multiple
platforms.
I can file an INFRA ticket to create a new repo to host wheel setup. Does
anybody have experience with setting up a new repo similar to this?
Ahmet
On Fri, Aug 3, 2018 at 1:16 PM, Boyuan Zhang <bo...@google.com> wrote:
> Hey all,
>
> I'm Boyuan Zhang from Google Dataflow Team, currently helping Release
> Manager(Pablo Estrada) with 2.6.0 release. Since Beam decided to release
> python wheels since 2.5.0, we need to create a wrapper repository(sample
> repo <https://github.com/boyuanzz/apache-beam-wheels>) under apache to
> build and stage released python wheels for each release. Anyone can help to
> create this repository?
>
> Thanks for all your help! Happy Friday~
>
> Boyuan Zhang
>