You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Tobiasz Kędzierski <to...@polidea.com> on 2020/06/01 15:26:54 UTC
Re: Fwd: [DISCUSSION] Use github actions for python wheels ?
Hi everyone,
Austin Benett I would like cooperate with you with introducing GitHub actions for python wheels.
Since last week I am trying to figure out how to do this and I was able to build wheels for Linux and MacOS with help of https://github.com/joerick/cibuildwheel/ <https://github.com/joerick/cibuildwheel/>.
I’ve created draft PR to apache/beam: https://github.com/apache/beam/pull/11877 <https://github.com/apache/beam/pull/11877> and you can check GitHub actions logs on fork PR here:
https://github.com/TobKed/beam/pull/3 <https://github.com/TobKed/beam/pull/3>
I am struggling with building wheels for Windows due to the error:
> Cannot open include file: 'unistd.h': No such file or directory
caused by file sdks/python/apache_beam/runners/worker/statesampler_fast.pyx
Adding this file to be excluded by cythonize method in setup.py allowed build to be completed however I don’t think it is a solution.
Mentioned error on GitHub actions: https://github.com/TobKed/beam/runs/727709583?check_suite_focus=true#step:7:977 <https://github.com/TobKed/beam/runs/727709583?check_suite_focus=true#step:7:977> and my local PR: https://github.com/TobKed/beam/pull/2 <https://github.com/TobKed/beam/pull/2>
What do you think about it?
Tobiasz
On 2020/05/30 20:53:51, Austin Bennett <w....@gmail.com> wrote:
> There seems to be support here for this idea. After digging through>
> things, I *think* I understand the moving pieces and can address (reason I>
> was digging through the code, including beam-wheels repo [1]). Though, not>
> 100% that I have not overlooked bits.>
>
> Would be happy to pickup>
> https://issues.apache.org/jira/browse/BEAM-9388 (imagine>
> would get to it over the course of a month or few -- so also wouldn't want>
> to take it on if others would be looking to do it sooner) -- seems>
> worthwhile to reduce manual steps for the release process, as well as the>
> additional implications (daily builds) if can easily manage. I have some>
> experience with GitHub Actions, but not with our (BEAM) deployment/release>
> process, nor much with building wheels. I assume there are sufficient docs>
> available to go through for the mechanics of the latter. If picking up>
> this issue, I suspect may need to bother some of those that have done>
> recent releases -- that would best occur on the jira ticket, once at that>
> point?>
>
> @Ismail <ie...@gmail.com>-- given your experience with GitHub actions,>
> any reason to think GitHub Actions not appropriate/ready for this specific>
> task?>
>
> @Ahmet Altay <al...@google.com> happy to understand the extent of what you>
> had in mind, maybe the extensions are not as important to plan out, as>
> they're straightforwardly bolted on (ex: daily builds). More tactically>
> would be valuable to ensure I understand what all needs to occur. Any>
> other source of info to consume other than>
> https://github.com/apache/beam-wheels and>
> https://beam.apache.org/contribute/release-guide/.>
>
> Also, open to the thought that this might be taking on too much, without>
> more experience with the release process and such. Do advise...?>
>
>
> [1]>
> https://lists.apache.org/thread.html/r93884eb080297647207f7d2b8a393e224029fc2c3509017886e84051%40%3Cdev.beam.apache.org%3E>
>
>
>
> ---------- Forwarded message --------->
> From: Ismaël Mejía <ie...@gmail.com>>
> Date: Wed, Feb 26, 2020 at 12:09 PM>
> Subject: Re: [DISCUSSION] Use github actions for python wheels ?>
> To: dev <de...@beam.apache.org>>
>
>
> +1 I have been migrating multiple projects into github actions recently and>
> even if for some tasks it is not as mature and polished as travis it has>
> proven to be way more reliable.>
>
>
> On Wed, Feb 26, 2020 at 7:07 PM Ahmet Altay <al...@google.com> wrote:>
>
> > I created https://issues.apache.org/jira/browse/BEAM-9388 to explore>
> > this. To be explicit and not to do cookie licking, I would not be able to>
> > work on this at the moment. If anyone is interested please take it.>
> > Otherwise I will try to come back and explore this when I can.>
> >>
> > On Tue, Feb 25, 2020 at 2:57 PM Robert Bradshaw <ro...@google.com>>
> > wrote:>
> >>
> >> I'd be in favor of this, assuming it actually simplifies things.>
> >>
> >>
> > This is also my concern. I do think that it will simplify things, but I am>
> > not certain as I am not very familiar with the github actions.>
> >>
> >>
> >> (Note>
> >> that the wheels are for several variants of linux, presumably we could>
> >> do cross-compiles. Also, manylinux is a "minimal" linux specifically>
> >> built as to produce shared object libraries compatible with a wide>
> >> variety of distributions--we can't just assume that a shared object>
> >> library built on one modern linux will just work on another. (But>
> >> maybe it's sufficient to do this within a docker environment?)>
> >>>
> >>
> > There will be no change in this area. Both in Both Travis and github>
> > actions offer a comparable set of options.>
> >>
> >>
> >>>
> >> On Tue, Feb 25, 2020 at 2:23 PM Kenneth Knowles <ke...@apache.org> wrote:>
> >> >>
> >> > +1 to exploring this.>
> >> >>
> >> > On builds@apache.org there is lots of discussion and general approval>
> >> for trying it. It is enabled and used by some projects. Calcite uses it to>
> >> build their website, for example.>
> >>>
> >>
> > Great.>
> >>
> >>
> >> >>
> >> > Kenn>
> >> >>
> >> >>
> >> > On Tue, Feb 25, 2020 at 2:08 PM Ahmet Altay <al...@google.com> wrote:>
> >> >>>
> >> >> Hi all,>
> >> >>>
> >> >> I recently had a chance to look at the documentation for github>
> >> actions. I think we could use github actions instead of travis to for>
> >> building python wheels during releases. This will have the following>
> >> advantages:>
> >> >>>
> >> >> - We will eliminate one repo. (If you don't know, we have>
> >> https://github.com/apache/beam-wheels for the sole purpose of building>
> >> wheels file.)>
> >> >> - Workflow will be stored in the same repo. This will prevent bit rot>
> >> that is only discovered at release times. (happened a few times, although>
> >> usually easy to fix.)>
> >> >> - github actions supports ubuntu, mac, windows environments. We could>
> >> try to build wheels for windows as well. (Travis also supports the same>
> >> environments but we only use linux and mac environments. Maybe there are>
> >> other blockers for building wheels for Windows.)>
> >> >> - We could do more, like daily python builds.>
> >> >>>
> >> >> Downsides would be:>
> >> >> - I do not know if github actions will require some special set of>
> >> permissions that require an approval from infra.>
> >> >> - Travis works fine most of the time. This might be unnecessary work.>
> >> >>>
> >> >> What do you think? Is this feasible, would this be useful?>
> >> >>>
> >> >> Ahmet>
> >>>
> >>
>
Re: Fwd: [DISCUSSION] Use github actions for python wheels ?
Posted by Ahmet Altay <al...@google.com>.
Hi Tobiasz,
Thank you for your work on this. I think this is making great progress and
I agree that it will simplify the release process. I asked a few clarifying
questions on the PR. (I do not want to repeat it here to avoid diverging
the conversation.)
Thank you!
Ahmet
On Tue, Jun 9, 2020 at 2:39 AM Tobiasz Kędzierski <
tobiasz.kedzierski@polidea.com> wrote:
> Hi,
>
> I've added some important updates to
> https://github.com/apache/beam/pull/11877 and I wanted to share some
> thoughts with you about possible improvements:
>
> During releasing a new version of Beam the script
> *build_release_candidate.sh* is executed. It builds sources and puts them
> into the GCS staging bucket where they are consumed by separate repository
> CI jobs (beam-wheels). Then they are downloaded and processed by
> *sign_hash_python_wheels.sh* script.
>
> By using github actions this process could be simplified as follows:
> 1. Within *build_release_candidate.sh* *release* and *release candidate*
> branches are pushed to the remote repository (this is done by it now).
> 2. gh-actions will build sources and wheels based on these branches.
> 3. *build_release_candidate.sh* could verify status of the build by using
> github api and corresponding data (name of the branch, commit hash) and
> after successful build download sources and wheel files from gh-action
> artifacts and use them in further actions.
>
> Happy to know your opinion on this
>
> BR
> Tobiasz Kędzierski
>
> On Mon, Jun 1, 2020 at 11:02 PM Ahmet Altay <al...@google.com> wrote:
>
>> > @Ahmet Altay <al...@google.com> happy to understand the extent of what
>> you had in mind, maybe the extensions are not as important to plan out, as
>> they're straightforwardly bolted on (ex: daily builds). More tactically
>> would be valuable to ensure I understand what all needs to occur. Any
>> other source of info to consume other than
>> https://github.com/apache/beam-wheels and
>> https://beam.apache.org/contribute/release-guide/.
>>
>> I added a bit more details to
>> https://issues.apache.org/jira/browse/BEAM-9388 as a comment, so that it
>> is preserved in the JIRA. Thank you all for working on this.
>>
>> On Mon, Jun 1, 2020 at 9:20 AM Kamil Wasilewski <
>> kamil.wasilewski@polidea.com> wrote:
>>
>>> "unistd.h" C header is present on POSIX systems (MacOS and Linux), but
>>> not on Windows, therefore you can't build a wheel for Windows.
>>>
>>> I took a look and "statesampler_fast.pyx" uses "unistd.h" only because
>>> of the `usleep` function. Unless we use C++ which offers [1], the solution
>>> would be to search for the equivalent of `usleep` that works on Windows.
>>>
>>
>> +Robert Bradshaw <ro...@google.com> +Valentyn Tymofieiev
>> <va...@google.com> - Do you have any suggestions on how building
>> wheels could work on Windows?
>>
>>
>>>
>>> [1] https://en.cppreference.com/w/cpp/thread/sleep_for
>>>
>>
>
Re: Fwd: [DISCUSSION] Use github actions for python wheels ?
Posted by Tobiasz Kędzierski <to...@polidea.com>.
Hi,
I've added some important updates to
https://github.com/apache/beam/pull/11877 and I wanted to share some
thoughts with you about possible improvements:
During releasing a new version of Beam the script
*build_release_candidate.sh* is executed. It builds sources and puts them
into the GCS staging bucket where they are consumed by separate repository
CI jobs (beam-wheels). Then they are downloaded and processed by
*sign_hash_python_wheels.sh* script.
By using github actions this process could be simplified as follows:
1. Within *build_release_candidate.sh* *release* and *release candidate*
branches are pushed to the remote repository (this is done by it now).
2. gh-actions will build sources and wheels based on these branches.
3. *build_release_candidate.sh* could verify status of the build by using
github api and corresponding data (name of the branch, commit hash) and
after successful build download sources and wheel files from gh-action
artifacts and use them in further actions.
Happy to know your opinion on this
BR
Tobiasz Kędzierski
On Mon, Jun 1, 2020 at 11:02 PM Ahmet Altay <al...@google.com> wrote:
> > @Ahmet Altay <al...@google.com> happy to understand the extent of what
> you had in mind, maybe the extensions are not as important to plan out, as
> they're straightforwardly bolted on (ex: daily builds). More tactically
> would be valuable to ensure I understand what all needs to occur. Any
> other source of info to consume other than
> https://github.com/apache/beam-wheels and
> https://beam.apache.org/contribute/release-guide/.
>
> I added a bit more details to
> https://issues.apache.org/jira/browse/BEAM-9388 as a comment, so that it
> is preserved in the JIRA. Thank you all for working on this.
>
> On Mon, Jun 1, 2020 at 9:20 AM Kamil Wasilewski <
> kamil.wasilewski@polidea.com> wrote:
>
>> "unistd.h" C header is present on POSIX systems (MacOS and Linux), but
>> not on Windows, therefore you can't build a wheel for Windows.
>>
>> I took a look and "statesampler_fast.pyx" uses "unistd.h" only because of
>> the `usleep` function. Unless we use C++ which offers [1], the solution
>> would be to search for the equivalent of `usleep` that works on Windows.
>>
>
> +Robert Bradshaw <ro...@google.com> +Valentyn Tymofieiev
> <va...@google.com> - Do you have any suggestions on how building
> wheels could work on Windows?
>
>
>>
>> [1] https://en.cppreference.com/w/cpp/thread/sleep_for
>>
>
Re: Fwd: [DISCUSSION] Use github actions for python wheels ?
Posted by Ahmet Altay <al...@google.com>.
> @Ahmet Altay <al...@google.com> happy to understand the extent of what
you had in mind, maybe the extensions are not as important to plan out, as
they're straightforwardly bolted on (ex: daily builds). More tactically
would be valuable to ensure I understand what all needs to occur. Any
other source of info to consume other than
https://github.com/apache/beam-wheels and
https://beam.apache.org/contribute/release-guide/.
I added a bit more details to
https://issues.apache.org/jira/browse/BEAM-9388 as a comment, so that it is
preserved in the JIRA. Thank you all for working on this.
On Mon, Jun 1, 2020 at 9:20 AM Kamil Wasilewski <
kamil.wasilewski@polidea.com> wrote:
> "unistd.h" C header is present on POSIX systems (MacOS and Linux), but not
> on Windows, therefore you can't build a wheel for Windows.
>
> I took a look and "statesampler_fast.pyx" uses "unistd.h" only because of
> the `usleep` function. Unless we use C++ which offers [1], the solution
> would be to search for the equivalent of `usleep` that works on Windows.
>
+Robert Bradshaw <ro...@google.com> +Valentyn Tymofieiev
<va...@google.com> - Do you have any suggestions on how building wheels
could work on Windows?
>
> [1] https://en.cppreference.com/w/cpp/thread/sleep_for
>
Re: Fwd: [DISCUSSION] Use github actions for python wheels ?
Posted by Kamil Wasilewski <ka...@polidea.com>.
"unistd.h" C header is present on POSIX systems (MacOS and Linux), but not
on Windows, therefore you can't build a wheel for Windows.
I took a look and "statesampler_fast.pyx" uses "unistd.h" only because of
the `usleep` function. Unless we use C++ which offers [1], the solution
would be to search for the equivalent of `usleep` that works on Windows.
[1] https://en.cppreference.com/w/cpp/thread/sleep_for