Confronting Arrow packaging problems

[Wes McKinney <we...@gmail.com>]

hi folks,

So, I want to bring light to the problems we are having delivering
binary artifacts after Arrow releases.

We have some amount of packaging automation implemented in
https://github.com/apache/arrow-dist using Travis CI and Appveyor to
upload packages to Bintray, a packaging hosting service.

Unfortunately, we discovered a bunch of problems with these packaging
scripts after the release vote closed on Monday, and now 4 days later,
we still have been unable to post binaries to
https://pypi.python.org/pypi/pyarrow

This is no one's fault, but it highlights structural problems with our
development process:

* Why does producing packages after a release require error-prone manual labor?

* Why are we only finding out about packaging problem after a release
vote closes?

* Why is setting up nightly binary builds a brittle and bespoke process?

I hope all agree that:

* Packaging should not be a hardship or require a lot of manual labor

* Packaging problems on the master branch should be made known within
~24 hours, so they can be remedied immediately

* It should be straightforward to produce binary artifacts for all
supported platforms and programming languages

Eventually, we should include some binary artifacts in our release
votes, but we are pretty far away from suitable automation to make
this possible.

I don't know any easy solutions, but Apache Arrow has grown widely
used enough that I think it's worth our taking the time to plan and
execute some solutions to these problems, which I expect to pay
dividends in our community's productivity over time.

Thanks,
Wes

Re: Confronting Arrow packaging problems

[Wes McKinney <we...@gmail.com>]

Some responses inline. Would be good to get some more feedback from
folks in the Google doc. I think some of the particular details may
remain hazy until we begin to lay hands to code to make things more
concrete.

On Mon, Mar 26, 2018 at 3:49 PM, Phillip Cloud <cp...@gmail.com> wrote:
> On Mon, Mar 26, 2018 at 1:37 PM Wes McKinney <we...@gmail.com> wrote:
>
>> > Huge +1 on moving some of the packaging outside the scope of
>> responsibility of arrow dev, specifically I don't think we should be
>> responsible for anything except wheels and conda packages.
>>
>
> This is my ideal scenario, however unrealistic at the moment.
>
>
>>
>> In theory I agree, but until Apache Arrow grows popular enough and
>> important enough for other communities to assume responsibility for
>> timely packaging on standard platforms like major Linux distributions,
>> we are going to have to do it, otherwise it will harm the growth of
>> the community (since users will have a hard time installing the
>> software on various platforms).
>>
>> Before changing the scope of what we are committing ourselves to do, I
>> would like to see if we can develop suitable automation around the
>> things we already have implemented. I don't want to write some things
>> off as being "too hard" or "too much work for the Apache Arrow
>> community" without giving a concerted automation effort a try.
>>
>
> Then I think we need to have acknowledgement from committers willing to
> take on responsibility for packages when they fail to build during
> automated packaging and it needs to be documented in the Arrow repo. I
> don't think it's reasonable for all committers to be responsible for every
> package type on every platform when something goes wrong. Ideally
> automation will alleviate most or all of the issues here but things will
> still fail and need specific owners.
>
> For example, I am willing to own conda packaging for all platforms and pip
> packaging for windows. I am not willing to own debian, yum, or pip
> packaging for other platforms. When I say own, what I mean is that when the
> automated package build fails and I get an email, I will respond ASAP with
> either a fix or by contacting the appropriate person. Of course, this isn't
> set in stone. If I'm able to help in other areas and I have the time, then
> I will.

I think the simplest way to think about packaging is as a "feature".
The fact that we can "deliver Python wheels on macOS" is a feature of
Apache Arrow. So it turned out that we had a feature that we were not
testing often enough.

Any feature in the project generally will have one or more owners, so
packaging should be semantically the same from a maintenance
standpoint.

>
>
>> Note that we have other things we should be automating, but we are not:
>>
>> * Nightly performance benchmarking (e.g. ASV for Python)
>>
> * Nightly integration tests (Spark, HDFS, Dask, API docs, etc.)
>>
> * Running GPU-enabled tests in CI
>> * Building GPU-enabled binaries
>>
>> > +1. Trying to satsify everyone's downstream needs is an impossible task.
>>
>> I think it's really easy to say "let's remove options" and "let's make
>> the build system simpler" without assessing the consequences of this
>> to community/project growth. The reason there are a lot of options and
>> the build system has complexity is that we are trying to satisfy a
>> pretty large matrix of requirements developed organically over the
>> last 2+ years.
>>
>
> It's not clear to me what is not in scope. Where do we draw the line for
> testing projects downstream of arrow? How do we decide whether to test
> those projects or not? Do we test them on all platforms that they support?
>
> IMO testing dependents of Arrow should be the responsibility of those
> particular pieces of software, not the responsibility of the Arrow project.
> Testing Spark and Dask, for example, both seem like the responsibility of
> their respective projects. Does asking these projects to do this hurt
> community/project growth in some way?
>
> Are there other large, successful projects that take on significant testing
> of their dependents? If there are, we should look at how they have
> addressed this issue.
>

I think at some point the burden of testing will shift more to
downstream projects, but while Arrow support is experimental or new,
we'll need to take a more active role in at least monitoring and
reporting whether things are broken.

>
>>
>> The central problem we are having is that our continuous integration
>> and continuous delivery has not scaled to cover the diversity of use
>> cases that we have accumulated. If we punt on addressing our
>> automation problems and instead start removing build or packaging
>> functionality to make things simpler, eventually the project will grow
>> until we are dealing with a different kind of development workflow
>> crisis.
>>
>
> I don't want to punt on automation, we need to do that regardless.
>
> What do you think about having specific owners of packaging areas,
> documented in the repo?
>

Yep, let's definitely do that.

- Wes

>
>>
>> On Mon, Mar 26, 2018 at 11:58 AM, Phillip Cloud <cp...@gmail.com> wrote:
>> > Responses inline. This kind of information is extremely helpful and
>> > informative.
>> >
>> > On Mon, Mar 26, 2018 at 11:26 AM Antoine Pitrou <an...@python.org>
>> wrote:
>> >
>> >>
>> >> Hi,
>> >>
>> >> As someone who started contributing recently, I'd like to raise a few
>> >> points.  I hope this post doesn't come accross as rambling or clueless,
>> >> otherwise feel free to ignore / point it out :-)
>> >>
>> >>
>> >> What does a release require?
>> >> ============================
>> >>
>> >> I didn't find any official documentation answering that question.
>> >>
>> >
>> https://github.com/apache/arrow/blob/master/dev/release/RELEASE_MANAGEMENT.md
>> > is the one I used for the most recent release.
>> >
>> >>
>> >> Right now, the in-line CI in the arrow repository ensures we have the
>> >> following:
>> >> - the source base builds fine in *some* configurations on each of the
>> three
>> >>   major platforms (Linux, macOS, Windows)
>> >> - the various test suites run fine on each of those platforms
>> >>
>> >> Is it a requirement that binary packages can be produced reliably for
>> >> a number of platforms, and if so, which ones?  Is it a requirement that
>> >> binary packages are available from day one when a release is done, or
>> >> is that a best effort thing depending on the availability of specific
>> >> platform maintainers?  It would be useful to spell that out somewhere.
>> >>
>> >
>> > The release management doc doesn't spell out the specfic nitty gritty of
>> > what exactly the artifacts do/don't should/shouldn't contain, though it
>> > does contain some information about how to produce some of the artifacts.
>> > It's critical that we spell this out somewhere.
>> >
>> >
>> >>
>> >>
>> >> Who is responsible for producing packages?
>> >> ==========================================
>> >>
>> >> Right now it seems packages are all produced out of a single repository
>> >> "arrow-dist".  That repository handles production of binary artifacts:
>> >> Python wheels, Ubuntu / CentOS / Debian packages...
>> >>
>> >> It's not obvious if specific people are responsible for each of the
>> >> package production chains.  It's common in open source projects to have
>> >> dedicated persons (or teams) responsible for each platform target.
>> >> This ensures that 1) the packages are produced by motivated people
>> >> who are familiar enough with their platforms of interest 2) producing
>> >> packages does not otherwise drain the stamina of the development team.
>> >>
>> >
>> > Huge +1 on moving some of the packaging outside the scope of
>> responsibility
>> > of arrow dev, specifically I don't think we should be responsible for
>> > anything except wheels and conda packages.
>> >
>> > One question I have here is: are the separate package type
>> scripts/software
>> > maintained in different repositories?
>> >
>> > Also +1 on having a person reponsible for each platform. I wonder if
>> having
>> > a person responsible for a specific kind of artifact might spread the
>> > workload more evenly since there's likely a shortage of Windows
>> expertise.
>> >
>> >
>> >>
>> >> CI strategy
>> >> ===========
>> >>
>> >> We have two conflicting requirements:
>> >> 1) Test as much as possible as part of continuous integration
>> (including,
>> >>    possible, the production of viable binary packages)
>> >> 2) Keep CI times reasonable to avoid grinding.  Some significant work
>> >>    was done recently to cut down our build times on Travis-CI
>> >>    and AppVeyor, often by half (ARROW-2071, ARROW-2083, ARROW-2231).
>> >>
>> >> To give a point of comparison, CPython has a two-thronged approach:
>> >>
>> >> 1) in-line CI using Travis-CI and AppVeyor, with simple build matrices
>> >>    (1 build on AppVeyor, 2 required + 2 optional on Travis-CI).  In-line
>> >>    CI must validate for a PR to be merged.
>> >> 2) out-of-line CI using a farm of buildbots:
>> >>    http://buildbot.python.org/all/#/grid?branch=master
>> >
>> >
>> > Buildbot looks *a lot* better than the last time I looked at it :)
>> >
>> >
>> >>
>> >>
>> >> Each buildbot has a maintainer, interested in keeping that specific
>> >> platform
>> >> and configuration running.  Some buildbots are marked stable and
>> strongly
>> >> recommended to be green at all times (and especially when releasing).
>> Some
>> >> buildbots on the other hand are marked unstable and represent less
>> >> mainstream
>> >> configurations which are just "nice to fix".
>> >>
>> >> The take-aways here are:
>> >> * Mainline development isn't throttled by the production of binary
>> >> artifacts
>> >>   or testing on a myriad of (possible slow or busy) CI platforms.
>> >> * Each tested configuration has a maintainer willing to identify and
>> >> diagnose
>> >>   problems (either propose a solution themselves or notify the developer
>> >>   responsible for a regression).
>> >> * Some things are release blockers (the "stable" platforms), some are
>> not
>> >>   and just nice to have.
>> >>
>> >
>> > IMO I would like the "stable" platforms should be conda packages for
>> > arrow/pyarrow and pip wheels. We should discuss that more.
>> >
>> >
>> >>
>> >> Two side notes:
>> >> * CPython is a much simpler project than Arrow, since it's C99 with
>> minimal
>> >>   dependencies.
>> >> * I wouldn't necessarily recommend buildbot as a CI platform.
>> >>
>> >>
>> >> Build options
>> >> =============
>> >>
>> >> It may be useful to look into reducing the number of build options,
>> and/or
>> >> standardize on supported settings, per platform.  For example, we should
>> >> decide whether boost should be bundled or not, namespaced or not, on
>> each
>> >> platform.  People with specific development requirements can try to
>> >> override
>> >> that, but with no guarantee from us.
>> >>
>> >
>> > +1. Trying to satsify everyone's downstream needs is an impossible task.
>> >
>> >
>> >>
>> >> For example, on the llvmlite project we decided early on that we would
>> >> always
>> >> link LLVM statically.  Third-party maintainers may decide to do things
>> >> differently, but they would have to maintain their own build scripts or
>> >> patches.
>> >>
>> >>
>> >> Regards
>> >>
>> >> Antoine.
>> >>
>> >>
>> >> Le 23/03/2018 à 17:58, Wes McKinney a écrit :
>> >> > hi folks,
>> >> >
>> >> > So, I want to bring light to the problems we are having delivering
>> >> > binary artifacts after Arrow releases.
>> >> >
>> >> > We have some amount of packaging automation implemented in
>> >> > https://github.com/apache/arrow-dist using Travis CI and Appveyor to
>> >> > upload packages to Bintray, a packaging hosting service.
>> >> >
>> >> > Unfortunately, we discovered a bunch of problems with these packaging
>> >> > scripts after the release vote closed on Monday, and now 4 days later,
>> >> > we still have been unable to post binaries to
>> >> > https://pypi.python.org/pypi/pyarrow
>> >> >
>> >> > This is no one's fault, but it highlights structural problems with our
>> >> > development process:
>> >> >
>> >> > * Why does producing packages after a release require error-prone
>> manual
>> >> labor?
>> >> >
>> >> > * Why are we only finding out about packaging problem after a release
>> >> > vote closes?
>> >> >
>> >> > * Why is setting up nightly binary builds a brittle and bespoke
>> process?
>> >> >
>> >> > I hope all agree that:
>> >> >
>> >> > * Packaging should not be a hardship or require a lot of manual labor
>> >> >
>> >> > * Packaging problems on the master branch should be made known within
>> >> > ~24 hours, so they can be remedied immediately
>> >> >
>> >> > * It should be straightforward to produce binary artifacts for all
>> >> > supported platforms and programming languages
>> >> >
>> >> > Eventually, we should include some binary artifacts in our release
>> >> > votes, but we are pretty far away from suitable automation to make
>> >> > this possible.
>> >> >
>> >> > I don't know any easy solutions, but Apache Arrow has grown widely
>> >> > used enough that I think it's worth our taking the time to plan and
>> >> > execute some solutions to these problems, which I expect to pay
>> >> > dividends in our community's productivity over time.
>> >> >
>> >> > Thanks,
>> >> > Wes
>> >> >
>> >>
>>

Re: Confronting Arrow packaging problems

[Phillip Cloud <cp...@gmail.com>]

On Mon, Mar 26, 2018 at 1:37 PM Wes McKinney <we...@gmail.com> wrote:

> > Huge +1 on moving some of the packaging outside the scope of
> responsibility of arrow dev, specifically I don't think we should be
> responsible for anything except wheels and conda packages.
>

This is my ideal scenario, however unrealistic at the moment.


>
> In theory I agree, but until Apache Arrow grows popular enough and
> important enough for other communities to assume responsibility for
> timely packaging on standard platforms like major Linux distributions,
> we are going to have to do it, otherwise it will harm the growth of
> the community (since users will have a hard time installing the
> software on various platforms).
>
> Before changing the scope of what we are committing ourselves to do, I
> would like to see if we can develop suitable automation around the
> things we already have implemented. I don't want to write some things
> off as being "too hard" or "too much work for the Apache Arrow
> community" without giving a concerted automation effort a try.
>

Then I think we need to have acknowledgement from committers willing to
take on responsibility for packages when they fail to build during
automated packaging and it needs to be documented in the Arrow repo. I
don't think it's reasonable for all committers to be responsible for every
package type on every platform when something goes wrong. Ideally
automation will alleviate most or all of the issues here but things will
still fail and need specific owners.

For example, I am willing to own conda packaging for all platforms and pip
packaging for windows. I am not willing to own debian, yum, or pip
packaging for other platforms. When I say own, what I mean is that when the
automated package build fails and I get an email, I will respond ASAP with
either a fix or by contacting the appropriate person. Of course, this isn't
set in stone. If I'm able to help in other areas and I have the time, then
I will.


> Note that we have other things we should be automating, but we are not:
>
> * Nightly performance benchmarking (e.g. ASV for Python)
>
* Nightly integration tests (Spark, HDFS, Dask, API docs, etc.)
>
* Running GPU-enabled tests in CI
> * Building GPU-enabled binaries
>
> > +1. Trying to satsify everyone's downstream needs is an impossible task.
>
> I think it's really easy to say "let's remove options" and "let's make
> the build system simpler" without assessing the consequences of this
> to community/project growth. The reason there are a lot of options and
> the build system has complexity is that we are trying to satisfy a
> pretty large matrix of requirements developed organically over the
> last 2+ years.
>

It's not clear to me what is not in scope. Where do we draw the line for
testing projects downstream of arrow? How do we decide whether to test
those projects or not? Do we test them on all platforms that they support?

IMO testing dependents of Arrow should be the responsibility of those
particular pieces of software, not the responsibility of the Arrow project.
Testing Spark and Dask, for example, both seem like the responsibility of
their respective projects. Does asking these projects to do this hurt
community/project growth in some way?

Are there other large, successful projects that take on significant testing
of their dependents? If there are, we should look at how they have
addressed this issue.


>
> The central problem we are having is that our continuous integration
> and continuous delivery has not scaled to cover the diversity of use
> cases that we have accumulated. If we punt on addressing our
> automation problems and instead start removing build or packaging
> functionality to make things simpler, eventually the project will grow
> until we are dealing with a different kind of development workflow
> crisis.
>

I don't want to punt on automation, we need to do that regardless.

What do you think about having specific owners of packaging areas,
documented in the repo?


>
> On Mon, Mar 26, 2018 at 11:58 AM, Phillip Cloud <cp...@gmail.com> wrote:
> > Responses inline. This kind of information is extremely helpful and
> > informative.
> >
> > On Mon, Mar 26, 2018 at 11:26 AM Antoine Pitrou <an...@python.org>
> wrote:
> >
> >>
> >> Hi,
> >>
> >> As someone who started contributing recently, I'd like to raise a few
> >> points.  I hope this post doesn't come accross as rambling or clueless,
> >> otherwise feel free to ignore / point it out :-)
> >>
> >>
> >> What does a release require?
> >> ============================
> >>
> >> I didn't find any official documentation answering that question.
> >>
> >
> https://github.com/apache/arrow/blob/master/dev/release/RELEASE_MANAGEMENT.md
> > is the one I used for the most recent release.
> >
> >>
> >> Right now, the in-line CI in the arrow repository ensures we have the
> >> following:
> >> - the source base builds fine in *some* configurations on each of the
> three
> >>   major platforms (Linux, macOS, Windows)
> >> - the various test suites run fine on each of those platforms
> >>
> >> Is it a requirement that binary packages can be produced reliably for
> >> a number of platforms, and if so, which ones?  Is it a requirement that
> >> binary packages are available from day one when a release is done, or
> >> is that a best effort thing depending on the availability of specific
> >> platform maintainers?  It would be useful to spell that out somewhere.
> >>
> >
> > The release management doc doesn't spell out the specfic nitty gritty of
> > what exactly the artifacts do/don't should/shouldn't contain, though it
> > does contain some information about how to produce some of the artifacts.
> > It's critical that we spell this out somewhere.
> >
> >
> >>
> >>
> >> Who is responsible for producing packages?
> >> ==========================================
> >>
> >> Right now it seems packages are all produced out of a single repository
> >> "arrow-dist".  That repository handles production of binary artifacts:
> >> Python wheels, Ubuntu / CentOS / Debian packages...
> >>
> >> It's not obvious if specific people are responsible for each of the
> >> package production chains.  It's common in open source projects to have
> >> dedicated persons (or teams) responsible for each platform target.
> >> This ensures that 1) the packages are produced by motivated people
> >> who are familiar enough with their platforms of interest 2) producing
> >> packages does not otherwise drain the stamina of the development team.
> >>
> >
> > Huge +1 on moving some of the packaging outside the scope of
> responsibility
> > of arrow dev, specifically I don't think we should be responsible for
> > anything except wheels and conda packages.
> >
> > One question I have here is: are the separate package type
> scripts/software
> > maintained in different repositories?
> >
> > Also +1 on having a person reponsible for each platform. I wonder if
> having
> > a person responsible for a specific kind of artifact might spread the
> > workload more evenly since there's likely a shortage of Windows
> expertise.
> >
> >
> >>
> >> CI strategy
> >> ===========
> >>
> >> We have two conflicting requirements:
> >> 1) Test as much as possible as part of continuous integration
> (including,
> >>    possible, the production of viable binary packages)
> >> 2) Keep CI times reasonable to avoid grinding.  Some significant work
> >>    was done recently to cut down our build times on Travis-CI
> >>    and AppVeyor, often by half (ARROW-2071, ARROW-2083, ARROW-2231).
> >>
> >> To give a point of comparison, CPython has a two-thronged approach:
> >>
> >> 1) in-line CI using Travis-CI and AppVeyor, with simple build matrices
> >>    (1 build on AppVeyor, 2 required + 2 optional on Travis-CI).  In-line
> >>    CI must validate for a PR to be merged.
> >> 2) out-of-line CI using a farm of buildbots:
> >>    http://buildbot.python.org/all/#/grid?branch=master
> >
> >
> > Buildbot looks *a lot* better than the last time I looked at it :)
> >
> >
> >>
> >>
> >> Each buildbot has a maintainer, interested in keeping that specific
> >> platform
> >> and configuration running.  Some buildbots are marked stable and
> strongly
> >> recommended to be green at all times (and especially when releasing).
> Some
> >> buildbots on the other hand are marked unstable and represent less
> >> mainstream
> >> configurations which are just "nice to fix".
> >>
> >> The take-aways here are:
> >> * Mainline development isn't throttled by the production of binary
> >> artifacts
> >>   or testing on a myriad of (possible slow or busy) CI platforms.
> >> * Each tested configuration has a maintainer willing to identify and
> >> diagnose
> >>   problems (either propose a solution themselves or notify the developer
> >>   responsible for a regression).
> >> * Some things are release blockers (the "stable" platforms), some are
> not
> >>   and just nice to have.
> >>
> >
> > IMO I would like the "stable" platforms should be conda packages for
> > arrow/pyarrow and pip wheels. We should discuss that more.
> >
> >
> >>
> >> Two side notes:
> >> * CPython is a much simpler project than Arrow, since it's C99 with
> minimal
> >>   dependencies.
> >> * I wouldn't necessarily recommend buildbot as a CI platform.
> >>
> >>
> >> Build options
> >> =============
> >>
> >> It may be useful to look into reducing the number of build options,
> and/or
> >> standardize on supported settings, per platform.  For example, we should
> >> decide whether boost should be bundled or not, namespaced or not, on
> each
> >> platform.  People with specific development requirements can try to
> >> override
> >> that, but with no guarantee from us.
> >>
> >
> > +1. Trying to satsify everyone's downstream needs is an impossible task.
> >
> >
> >>
> >> For example, on the llvmlite project we decided early on that we would
> >> always
> >> link LLVM statically.  Third-party maintainers may decide to do things
> >> differently, but they would have to maintain their own build scripts or
> >> patches.
> >>
> >>
> >> Regards
> >>
> >> Antoine.
> >>
> >>
> >> Le 23/03/2018 à 17:58, Wes McKinney a écrit :
> >> > hi folks,
> >> >
> >> > So, I want to bring light to the problems we are having delivering
> >> > binary artifacts after Arrow releases.
> >> >
> >> > We have some amount of packaging automation implemented in
> >> > https://github.com/apache/arrow-dist using Travis CI and Appveyor to
> >> > upload packages to Bintray, a packaging hosting service.
> >> >
> >> > Unfortunately, we discovered a bunch of problems with these packaging
> >> > scripts after the release vote closed on Monday, and now 4 days later,
> >> > we still have been unable to post binaries to
> >> > https://pypi.python.org/pypi/pyarrow
> >> >
> >> > This is no one's fault, but it highlights structural problems with our
> >> > development process:
> >> >
> >> > * Why does producing packages after a release require error-prone
> manual
> >> labor?
> >> >
> >> > * Why are we only finding out about packaging problem after a release
> >> > vote closes?
> >> >
> >> > * Why is setting up nightly binary builds a brittle and bespoke
> process?
> >> >
> >> > I hope all agree that:
> >> >
> >> > * Packaging should not be a hardship or require a lot of manual labor
> >> >
> >> > * Packaging problems on the master branch should be made known within
> >> > ~24 hours, so they can be remedied immediately
> >> >
> >> > * It should be straightforward to produce binary artifacts for all
> >> > supported platforms and programming languages
> >> >
> >> > Eventually, we should include some binary artifacts in our release
> >> > votes, but we are pretty far away from suitable automation to make
> >> > this possible.
> >> >
> >> > I don't know any easy solutions, but Apache Arrow has grown widely
> >> > used enough that I think it's worth our taking the time to plan and
> >> > execute some solutions to these problems, which I expect to pay
> >> > dividends in our community's productivity over time.
> >> >
> >> > Thanks,
> >> > Wes
> >> >
> >>
>

Re: Confronting Arrow packaging problems

[Wes McKinney <we...@gmail.com>]

> Huge +1 on moving some of the packaging outside the scope of responsibility of arrow dev, specifically I don't think we should be responsible for anything except wheels and conda packages.

In theory I agree, but until Apache Arrow grows popular enough and
important enough for other communities to assume responsibility for
timely packaging on standard platforms like major Linux distributions,
we are going to have to do it, otherwise it will harm the growth of
the community (since users will have a hard time installing the
software on various platforms).

Before changing the scope of what we are committing ourselves to do, I
would like to see if we can develop suitable automation around the
things we already have implemented. I don't want to write some things
off as being "too hard" or "too much work for the Apache Arrow
community" without giving a concerted automation effort a try.

Note that we have other things we should be automating, but we are not:

* Nightly performance benchmarking (e.g. ASV for Python)
* Nightly integration tests (Spark, HDFS, Dask, API docs, etc.)
* Running GPU-enabled tests in CI
* Building GPU-enabled binaries

> +1. Trying to satsify everyone's downstream needs is an impossible task.

I think it's really easy to say "let's remove options" and "let's make
the build system simpler" without assessing the consequences of this
to community/project growth. The reason there are a lot of options and
the build system has complexity is that we are trying to satisfy a
pretty large matrix of requirements developed organically over the
last 2+ years.

The central problem we are having is that our continuous integration
and continuous delivery has not scaled to cover the diversity of use
cases that we have accumulated. If we punt on addressing our
automation problems and instead start removing build or packaging
functionality to make things simpler, eventually the project will grow
until we are dealing with a different kind of development workflow
crisis.

On Mon, Mar 26, 2018 at 11:58 AM, Phillip Cloud <cp...@gmail.com> wrote:
> Responses inline. This kind of information is extremely helpful and
> informative.
>
> On Mon, Mar 26, 2018 at 11:26 AM Antoine Pitrou <an...@python.org> wrote:
>
>>
>> Hi,
>>
>> As someone who started contributing recently, I'd like to raise a few
>> points.  I hope this post doesn't come accross as rambling or clueless,
>> otherwise feel free to ignore / point it out :-)
>>
>>
>> What does a release require?
>> ============================
>>
>> I didn't find any official documentation answering that question.
>>
> https://github.com/apache/arrow/blob/master/dev/release/RELEASE_MANAGEMENT.md
> is the one I used for the most recent release.
>
>>
>> Right now, the in-line CI in the arrow repository ensures we have the
>> following:
>> - the source base builds fine in *some* configurations on each of the three
>>   major platforms (Linux, macOS, Windows)
>> - the various test suites run fine on each of those platforms
>>
>> Is it a requirement that binary packages can be produced reliably for
>> a number of platforms, and if so, which ones?  Is it a requirement that
>> binary packages are available from day one when a release is done, or
>> is that a best effort thing depending on the availability of specific
>> platform maintainers?  It would be useful to spell that out somewhere.
>>
>
> The release management doc doesn't spell out the specfic nitty gritty of
> what exactly the artifacts do/don't should/shouldn't contain, though it
> does contain some information about how to produce some of the artifacts.
> It's critical that we spell this out somewhere.
>
>
>>
>>
>> Who is responsible for producing packages?
>> ==========================================
>>
>> Right now it seems packages are all produced out of a single repository
>> "arrow-dist".  That repository handles production of binary artifacts:
>> Python wheels, Ubuntu / CentOS / Debian packages...
>>
>> It's not obvious if specific people are responsible for each of the
>> package production chains.  It's common in open source projects to have
>> dedicated persons (or teams) responsible for each platform target.
>> This ensures that 1) the packages are produced by motivated people
>> who are familiar enough with their platforms of interest 2) producing
>> packages does not otherwise drain the stamina of the development team.
>>
>
> Huge +1 on moving some of the packaging outside the scope of responsibility
> of arrow dev, specifically I don't think we should be responsible for
> anything except wheels and conda packages.
>
> One question I have here is: are the separate package type scripts/software
> maintained in different repositories?
>
> Also +1 on having a person reponsible for each platform. I wonder if having
> a person responsible for a specific kind of artifact might spread the
> workload more evenly since there's likely a shortage of Windows expertise.
>
>
>>
>> CI strategy
>> ===========
>>
>> We have two conflicting requirements:
>> 1) Test as much as possible as part of continuous integration (including,
>>    possible, the production of viable binary packages)
>> 2) Keep CI times reasonable to avoid grinding.  Some significant work
>>    was done recently to cut down our build times on Travis-CI
>>    and AppVeyor, often by half (ARROW-2071, ARROW-2083, ARROW-2231).
>>
>> To give a point of comparison, CPython has a two-thronged approach:
>>
>> 1) in-line CI using Travis-CI and AppVeyor, with simple build matrices
>>    (1 build on AppVeyor, 2 required + 2 optional on Travis-CI).  In-line
>>    CI must validate for a PR to be merged.
>> 2) out-of-line CI using a farm of buildbots:
>>    http://buildbot.python.org/all/#/grid?branch=master
>
>
> Buildbot looks *a lot* better than the last time I looked at it :)
>
>
>>
>>
>> Each buildbot has a maintainer, interested in keeping that specific
>> platform
>> and configuration running.  Some buildbots are marked stable and strongly
>> recommended to be green at all times (and especially when releasing).  Some
>> buildbots on the other hand are marked unstable and represent less
>> mainstream
>> configurations which are just "nice to fix".
>>
>> The take-aways here are:
>> * Mainline development isn't throttled by the production of binary
>> artifacts
>>   or testing on a myriad of (possible slow or busy) CI platforms.
>> * Each tested configuration has a maintainer willing to identify and
>> diagnose
>>   problems (either propose a solution themselves or notify the developer
>>   responsible for a regression).
>> * Some things are release blockers (the "stable" platforms), some are not
>>   and just nice to have.
>>
>
> IMO I would like the "stable" platforms should be conda packages for
> arrow/pyarrow and pip wheels. We should discuss that more.
>
>
>>
>> Two side notes:
>> * CPython is a much simpler project than Arrow, since it's C99 with minimal
>>   dependencies.
>> * I wouldn't necessarily recommend buildbot as a CI platform.
>>
>>
>> Build options
>> =============
>>
>> It may be useful to look into reducing the number of build options, and/or
>> standardize on supported settings, per platform.  For example, we should
>> decide whether boost should be bundled or not, namespaced or not, on each
>> platform.  People with specific development requirements can try to
>> override
>> that, but with no guarantee from us.
>>
>
> +1. Trying to satsify everyone's downstream needs is an impossible task.
>
>
>>
>> For example, on the llvmlite project we decided early on that we would
>> always
>> link LLVM statically.  Third-party maintainers may decide to do things
>> differently, but they would have to maintain their own build scripts or
>> patches.
>>
>>
>> Regards
>>
>> Antoine.
>>
>>
>> Le 23/03/2018 à 17:58, Wes McKinney a écrit :
>> > hi folks,
>> >
>> > So, I want to bring light to the problems we are having delivering
>> > binary artifacts after Arrow releases.
>> >
>> > We have some amount of packaging automation implemented in
>> > https://github.com/apache/arrow-dist using Travis CI and Appveyor to
>> > upload packages to Bintray, a packaging hosting service.
>> >
>> > Unfortunately, we discovered a bunch of problems with these packaging
>> > scripts after the release vote closed on Monday, and now 4 days later,
>> > we still have been unable to post binaries to
>> > https://pypi.python.org/pypi/pyarrow
>> >
>> > This is no one's fault, but it highlights structural problems with our
>> > development process:
>> >
>> > * Why does producing packages after a release require error-prone manual
>> labor?
>> >
>> > * Why are we only finding out about packaging problem after a release
>> > vote closes?
>> >
>> > * Why is setting up nightly binary builds a brittle and bespoke process?
>> >
>> > I hope all agree that:
>> >
>> > * Packaging should not be a hardship or require a lot of manual labor
>> >
>> > * Packaging problems on the master branch should be made known within
>> > ~24 hours, so they can be remedied immediately
>> >
>> > * It should be straightforward to produce binary artifacts for all
>> > supported platforms and programming languages
>> >
>> > Eventually, we should include some binary artifacts in our release
>> > votes, but we are pretty far away from suitable automation to make
>> > this possible.
>> >
>> > I don't know any easy solutions, but Apache Arrow has grown widely
>> > used enough that I think it's worth our taking the time to plan and
>> > execute some solutions to these problems, which I expect to pay
>> > dividends in our community's productivity over time.
>> >
>> > Thanks,
>> > Wes
>> >
>>

Re: Confronting Arrow packaging problems

[Phillip Cloud <cp...@gmail.com>]

Responses inline. This kind of information is extremely helpful and
informative.

On Mon, Mar 26, 2018 at 11:26 AM Antoine Pitrou <an...@python.org> wrote:

>
> Hi,
>
> As someone who started contributing recently, I'd like to raise a few
> points.  I hope this post doesn't come accross as rambling or clueless,
> otherwise feel free to ignore / point it out :-)
>
>
> What does a release require?
> ============================
>
> I didn't find any official documentation answering that question.
>
https://github.com/apache/arrow/blob/master/dev/release/RELEASE_MANAGEMENT.md
is the one I used for the most recent release.

>
> Right now, the in-line CI in the arrow repository ensures we have the
> following:
> - the source base builds fine in *some* configurations on each of the three
>   major platforms (Linux, macOS, Windows)
> - the various test suites run fine on each of those platforms
>
> Is it a requirement that binary packages can be produced reliably for
> a number of platforms, and if so, which ones?  Is it a requirement that
> binary packages are available from day one when a release is done, or
> is that a best effort thing depending on the availability of specific
> platform maintainers?  It would be useful to spell that out somewhere.
>

The release management doc doesn't spell out the specfic nitty gritty of
what exactly the artifacts do/don't should/shouldn't contain, though it
does contain some information about how to produce some of the artifacts.
It's critical that we spell this out somewhere.


>
>
> Who is responsible for producing packages?
> ==========================================
>
> Right now it seems packages are all produced out of a single repository
> "arrow-dist".  That repository handles production of binary artifacts:
> Python wheels, Ubuntu / CentOS / Debian packages...
>
> It's not obvious if specific people are responsible for each of the
> package production chains.  It's common in open source projects to have
> dedicated persons (or teams) responsible for each platform target.
> This ensures that 1) the packages are produced by motivated people
> who are familiar enough with their platforms of interest 2) producing
> packages does not otherwise drain the stamina of the development team.
>

Huge +1 on moving some of the packaging outside the scope of responsibility
of arrow dev, specifically I don't think we should be responsible for
anything except wheels and conda packages.

One question I have here is: are the separate package type scripts/software
maintained in different repositories?

Also +1 on having a person reponsible for each platform. I wonder if having
a person responsible for a specific kind of artifact might spread the
workload more evenly since there's likely a shortage of Windows expertise.


>
> CI strategy
> ===========
>
> We have two conflicting requirements:
> 1) Test as much as possible as part of continuous integration (including,
>    possible, the production of viable binary packages)
> 2) Keep CI times reasonable to avoid grinding.  Some significant work
>    was done recently to cut down our build times on Travis-CI
>    and AppVeyor, often by half (ARROW-2071, ARROW-2083, ARROW-2231).
>
> To give a point of comparison, CPython has a two-thronged approach:
>
> 1) in-line CI using Travis-CI and AppVeyor, with simple build matrices
>    (1 build on AppVeyor, 2 required + 2 optional on Travis-CI).  In-line
>    CI must validate for a PR to be merged.
> 2) out-of-line CI using a farm of buildbots:
>    http://buildbot.python.org/all/#/grid?branch=master


Buildbot looks *a lot* better than the last time I looked at it :)


>
>
> Each buildbot has a maintainer, interested in keeping that specific
> platform
> and configuration running.  Some buildbots are marked stable and strongly
> recommended to be green at all times (and especially when releasing).  Some
> buildbots on the other hand are marked unstable and represent less
> mainstream
> configurations which are just "nice to fix".
>
> The take-aways here are:
> * Mainline development isn't throttled by the production of binary
> artifacts
>   or testing on a myriad of (possible slow or busy) CI platforms.
> * Each tested configuration has a maintainer willing to identify and
> diagnose
>   problems (either propose a solution themselves or notify the developer
>   responsible for a regression).
> * Some things are release blockers (the "stable" platforms), some are not
>   and just nice to have.
>

IMO I would like the "stable" platforms should be conda packages for
arrow/pyarrow and pip wheels. We should discuss that more.


>
> Two side notes:
> * CPython is a much simpler project than Arrow, since it's C99 with minimal
>   dependencies.
> * I wouldn't necessarily recommend buildbot as a CI platform.
>
>
> Build options
> =============
>
> It may be useful to look into reducing the number of build options, and/or
> standardize on supported settings, per platform.  For example, we should
> decide whether boost should be bundled or not, namespaced or not, on each
> platform.  People with specific development requirements can try to
> override
> that, but with no guarantee from us.
>

+1. Trying to satsify everyone's downstream needs is an impossible task.


>
> For example, on the llvmlite project we decided early on that we would
> always
> link LLVM statically.  Third-party maintainers may decide to do things
> differently, but they would have to maintain their own build scripts or
> patches.
>
>
> Regards
>
> Antoine.
>
>
> Le 23/03/2018 à 17:58, Wes McKinney a écrit :
> > hi folks,
> >
> > So, I want to bring light to the problems we are having delivering
> > binary artifacts after Arrow releases.
> >
> > We have some amount of packaging automation implemented in
> > https://github.com/apache/arrow-dist using Travis CI and Appveyor to
> > upload packages to Bintray, a packaging hosting service.
> >
> > Unfortunately, we discovered a bunch of problems with these packaging
> > scripts after the release vote closed on Monday, and now 4 days later,
> > we still have been unable to post binaries to
> > https://pypi.python.org/pypi/pyarrow
> >
> > This is no one's fault, but it highlights structural problems with our
> > development process:
> >
> > * Why does producing packages after a release require error-prone manual
> labor?
> >
> > * Why are we only finding out about packaging problem after a release
> > vote closes?
> >
> > * Why is setting up nightly binary builds a brittle and bespoke process?
> >
> > I hope all agree that:
> >
> > * Packaging should not be a hardship or require a lot of manual labor
> >
> > * Packaging problems on the master branch should be made known within
> > ~24 hours, so they can be remedied immediately
> >
> > * It should be straightforward to produce binary artifacts for all
> > supported platforms and programming languages
> >
> > Eventually, we should include some binary artifacts in our release
> > votes, but we are pretty far away from suitable automation to make
> > this possible.
> >
> > I don't know any easy solutions, but Apache Arrow has grown widely
> > used enough that I think it's worth our taking the time to plan and
> > execute some solutions to these problems, which I expect to pay
> > dividends in our community's productivity over time.
> >
> > Thanks,
> > Wes
> >
>

Re: Confronting Arrow packaging problems

[Antoine Pitrou <an...@python.org>]

Hi,

As someone who started contributing recently, I'd like to raise a few
points.  I hope this post doesn't come accross as rambling or clueless,
otherwise feel free to ignore / point it out :-)


What does a release require?
============================

I didn't find any official documentation answering that question.

Right now, the in-line CI in the arrow repository ensures we have the
following:
- the source base builds fine in *some* configurations on each of the three
  major platforms (Linux, macOS, Windows)
- the various test suites run fine on each of those platforms

Is it a requirement that binary packages can be produced reliably for
a number of platforms, and if so, which ones?  Is it a requirement that
binary packages are available from day one when a release is done, or
is that a best effort thing depending on the availability of specific
platform maintainers?  It would be useful to spell that out somewhere.


Who is responsible for producing packages?
==========================================

Right now it seems packages are all produced out of a single repository
"arrow-dist".  That repository handles production of binary artifacts:
Python wheels, Ubuntu / CentOS / Debian packages...

It's not obvious if specific people are responsible for each of the
package production chains.  It's common in open source projects to have
dedicated persons (or teams) responsible for each platform target.
This ensures that 1) the packages are produced by motivated people
who are familiar enough with their platforms of interest 2) producing
packages does not otherwise drain the stamina of the development team.


CI strategy
===========

We have two conflicting requirements:
1) Test as much as possible as part of continuous integration (including,
   possible, the production of viable binary packages)
2) Keep CI times reasonable to avoid grinding.  Some significant work
   was done recently to cut down our build times on Travis-CI
   and AppVeyor, often by half (ARROW-2071, ARROW-2083, ARROW-2231).

To give a point of comparison, CPython has a two-thronged approach:

1) in-line CI using Travis-CI and AppVeyor, with simple build matrices
   (1 build on AppVeyor, 2 required + 2 optional on Travis-CI).  In-line
   CI must validate for a PR to be merged.
2) out-of-line CI using a farm of buildbots:
   http://buildbot.python.org/all/#/grid?branch=master

Each buildbot has a maintainer, interested in keeping that specific platform
and configuration running.  Some buildbots are marked stable and strongly
recommended to be green at all times (and especially when releasing).  Some
buildbots on the other hand are marked unstable and represent less mainstream
configurations which are just "nice to fix".

The take-aways here are:
* Mainline development isn't throttled by the production of binary artifacts
  or testing on a myriad of (possible slow or busy) CI platforms.
* Each tested configuration has a maintainer willing to identify and diagnose
  problems (either propose a solution themselves or notify the developer
  responsible for a regression).
* Some things are release blockers (the "stable" platforms), some are not
  and just nice to have.

Two side notes:
* CPython is a much simpler project than Arrow, since it's C99 with minimal
  dependencies.
* I wouldn't necessarily recommend buildbot as a CI platform.


Build options
=============

It may be useful to look into reducing the number of build options, and/or
standardize on supported settings, per platform.  For example, we should
decide whether boost should be bundled or not, namespaced or not, on each
platform.  People with specific development requirements can try to override
that, but with no guarantee from us.

For example, on the llvmlite project we decided early on that we would always
link LLVM statically.  Third-party maintainers may decide to do things
differently, but they would have to maintain their own build scripts or patches.


Regards

Antoine.


Le 23/03/2018 à 17:58, Wes McKinney a écrit :
> hi folks,
> 
> So, I want to bring light to the problems we are having delivering
> binary artifacts after Arrow releases.
> 
> We have some amount of packaging automation implemented in
> https://github.com/apache/arrow-dist using Travis CI and Appveyor to
> upload packages to Bintray, a packaging hosting service.
> 
> Unfortunately, we discovered a bunch of problems with these packaging
> scripts after the release vote closed on Monday, and now 4 days later,
> we still have been unable to post binaries to
> https://pypi.python.org/pypi/pyarrow
> 
> This is no one's fault, but it highlights structural problems with our
> development process:
> 
> * Why does producing packages after a release require error-prone manual labor?
> 
> * Why are we only finding out about packaging problem after a release
> vote closes?
> 
> * Why is setting up nightly binary builds a brittle and bespoke process?
> 
> I hope all agree that:
> 
> * Packaging should not be a hardship or require a lot of manual labor
> 
> * Packaging problems on the master branch should be made known within
> ~24 hours, so they can be remedied immediately
> 
> * It should be straightforward to produce binary artifacts for all
> supported platforms and programming languages
> 
> Eventually, we should include some binary artifacts in our release
> votes, but we are pretty far away from suitable automation to make
> this possible.
> 
> I don't know any easy solutions, but Apache Arrow has grown widely
> used enough that I think it's worth our taking the time to plan and
> execute some solutions to these problems, which I expect to pay
> dividends in our community's productivity over time.
> 
> Thanks,
> Wes
> 

Re: Confronting Arrow packaging problems

["Uwe L. Korn" <uw...@xhochy.com>]

Hello all,

I strongly support Wes' points about having better automated feedback in our build chain is essential, independent from the tool we use to make the builds. As it sadly seems that our newly uploaded Arrow wheels for OSX are broken, I'm going to start there and add them to the build matrix: https://issues.apache.org/jira/browse/ARROW-2352. Looking at the build tools, I think that we could get a great benefit from using Bazel or something similar, it probably does not solve the problems we currently face in its full scope. Especially the wheel build are very brittle as they need other packages also build in a special way so that they are redistributable.

Uwe

On Sun, Mar 25, 2018, at 2:13 AM, Krisztián Szűcs wrote:
> Just want to mention two other build systems:
> 
> Pants: https://github.com/pantsbuild/pants 
> (https://link.getmailspring.com/link/1521936116.local-4d31e945-727a-v1.1.5-5834c99f@getmailspring.com/0?redirect=https%3A%2F%2Fgithub.com%2Fpantsbuild%2Fpants&recipient=ZGV2QGFycm93LmFwYWNoZS5vcmc%3D)
> Meson: https://github.com/mesonbuild/meson 
> (https://link.getmailspring.com/link/1521936116.local-4d31e945-727a-v1.1.5-5834c99f@getmailspring.com/1?redirect=https%3A%2F%2Fgithub.com%2Fmesonbuild%2Fmeson&recipient=ZGV2QGFycm93LmFwYWNoZS5vcmc%3D)
> Comparison http://mesonbuild.com/Comparisons.html 
> (https://link.getmailspring.com/link/1521936116.local-4d31e945-727a-v1.1.5-5834c99f@getmailspring.com/2?redirect=http%3A%2F%2Fmesonbuild.com%2FComparisons.html&recipient=ZGV2QGFycm93LmFwYWNoZS5vcmc%3D)
> Krisztian
> On Mar 24 2018, at 9:13 pm, Phillip Cloud <cp...@gmail.com> wrote:
> >
> > I think we need to use a tool that can perform every single step of the
> > deployment process, end-to-end. Right now, cmake isn't cutting it IMO
> > because it lends itself quite heavily to copy pasting and oodles of bash
> > scripts that are indecipherable by anyone except the original author.
> >
> > With that in mind, here's what I think the requirements are for the next
> > generation of arrow's build and deployment system for native code (C, C++,
> > Python):
> >
> > For every platform (Linux, OS X, and Windows):
> > 1. Build sources (these langauges are currently the most cumbersome at the
> > moment)
> > 2. Run all tests for each language (ideally integration tests as well, but
> > I'm not sure if that should be a hard requirement at the moment)
> > 3. Build the API documentation for each platform
> > 4. Build installable packages that we support:
> > * conda
> > * pip wheels
> > * deb
> > * rpm
> >
> > Additional requirements:
> > 1. Be able to run all of these steps on any platform with minimal
> > environment setup.
> > 2. The output of the doc build and package build steps **should be in a
> > release-ready state at all times**. If they are not then we should fail the
> > build
> > 3. Ideally, these run on every PR so that we can find out if a commit would
> > introduce a change that would break the release-ready status of arrow.
> >
> > These requirements indicate to me that a single, extensible, cross-platform
> > tool--as opposed to many tools that are tied together by a shell script--is
> > what we need.
> >
> > There are a few tools in this space that I'm aware of:
> > 1. Bazel (out of Google)
> > 2. Buck (out of Facebook)
> >
> > I'm not sure what others are out there, but I'm sure there must be some.
> > I don't really have a strong opinion on either Bazel or Buck, but I suspect
> > that since we follow Google's conventions in a few places integrating Bazel
> > into the arrow codebase would be less work.
> >
> > The main risk I see here is that it's possible that bazel isn't the right
> > tool. I'm not sure how to mitigate this risk other than to make sure that
> > our requirements can be met by it by scouring the Bazel docs.
> >
> > I do think that the fact that Bazel is extensible mitigates some risk here.
> > For example, we'd likely have to add rule for building conda packages and
> > pip wheels.
> >
> > I guess CMake is extensible too, but I don't think I've ever seen the
> > extensibility features of CMake as anything but a burden. Bazel's extension
> > language is a subset of Python and I would therefore expect it to be a lot
> > easier to use.
> >
> > I'm interested to hear others' experiences and opinions on similar
> > problems. Also, if I've missed anything in the requirements list, please
> > don't hestitate to respond.
> >
> > Let's fix our packaging!
> > -Phillip
> > On Fri, Mar 23, 2018 at 11:21 PM Holden Karau <ho...@pigscanfly.ca> wrote:
> > > I know in Spark we’ve benefited by having some of the different language
> > > devs act as RMs and each time that language dev has ended up improving a
> > > bunch of how their components packaging has been done. Not to suggest we
> > > should just do what other projects do, but maybe an idea to consider?
> > >
> > > On Fri, Mar 23, 2018 at 12:59 PM Wes McKinney <we...@gmail.com> wrote:
> > > > hi folks,
> > > > So, I want to bring light to the problems we are having delivering
> > > > binary artifacts after Arrow releases.
> > > >
> > > > We have some amount of packaging automation implemented in
> > > > https://github.com/apache/arrow-dist using Travis CI and Appveyor to
> > > > upload packages to Bintray, a packaging hosting service.
> > > >
> > > > Unfortunately, we discovered a bunch of problems with these packaging
> > > > scripts after the release vote closed on Monday, and now 4 days later,
> > > > we still have been unable to post binaries to
> > > > https://pypi.python.org/pypi/pyarrow
> > > >
> > > > This is no one's fault, but it highlights structural problems with our
> > > > development process:
> > > >
> > > > * Why does producing packages after a release require error-prone manual
> > > > labor?
> > > >
> > > > * Why are we only finding out about packaging problem after a release
> > > > vote closes?
> > > >
> > > > * Why is setting up nightly binary builds a brittle and bespoke process?
> > > > I hope all agree that:
> > > > * Packaging should not be a hardship or require a lot of manual labor
> > > > * Packaging problems on the master branch should be made known within
> > > > ~24 hours, so they can be remedied immediately
> > > >
> > > > * It should be straightforward to produce binary artifacts for all
> > > > supported platforms and programming languages
> > > >
> > > > Eventually, we should include some binary artifacts in our release
> > > > votes, but we are pretty far away from suitable automation to make
> > > > this possible.
> > > >
> > > > I don't know any easy solutions, but Apache Arrow has grown widely
> > > > used enough that I think it's worth our taking the time to plan and
> > > > execute some solutions to these problems, which I expect to pay
> > > > dividends in our community's productivity over time.
> > > >
> > > > Thanks,
> > > > Wes
> > > >
> > > --
> > > Twitter: https://twitter.com/holdenkarau
> >
> >
> 

Re: Confronting Arrow packaging problems

[Krisztián Szűcs <sz...@gmail.com>]

Just want to mention two other build systems:

Pants: https://github.com/pantsbuild/pants (https://link.getmailspring.com/link/1521936116.local-4d31e945-727a-v1.1.5-5834c99f@getmailspring.com/0?redirect=https%3A%2F%2Fgithub.com%2Fpantsbuild%2Fpants&recipient=ZGV2QGFycm93LmFwYWNoZS5vcmc%3D)
Meson: https://github.com/mesonbuild/meson (https://link.getmailspring.com/link/1521936116.local-4d31e945-727a-v1.1.5-5834c99f@getmailspring.com/1?redirect=https%3A%2F%2Fgithub.com%2Fmesonbuild%2Fmeson&recipient=ZGV2QGFycm93LmFwYWNoZS5vcmc%3D)
Comparison http://mesonbuild.com/Comparisons.html (https://link.getmailspring.com/link/1521936116.local-4d31e945-727a-v1.1.5-5834c99f@getmailspring.com/2?redirect=http%3A%2F%2Fmesonbuild.com%2FComparisons.html&recipient=ZGV2QGFycm93LmFwYWNoZS5vcmc%3D)
Krisztian
On Mar 24 2018, at 9:13 pm, Phillip Cloud <cp...@gmail.com> wrote:
>
> I think we need to use a tool that can perform every single step of the
> deployment process, end-to-end. Right now, cmake isn't cutting it IMO
> because it lends itself quite heavily to copy pasting and oodles of bash
> scripts that are indecipherable by anyone except the original author.
>
> With that in mind, here's what I think the requirements are for the next
> generation of arrow's build and deployment system for native code (C, C++,
> Python):
>
> For every platform (Linux, OS X, and Windows):
> 1. Build sources (these langauges are currently the most cumbersome at the
> moment)
> 2. Run all tests for each language (ideally integration tests as well, but
> I'm not sure if that should be a hard requirement at the moment)
> 3. Build the API documentation for each platform
> 4. Build installable packages that we support:
> * conda
> * pip wheels
> * deb
> * rpm
>
> Additional requirements:
> 1. Be able to run all of these steps on any platform with minimal
> environment setup.
> 2. The output of the doc build and package build steps **should be in a
> release-ready state at all times**. If they are not then we should fail the
> build
> 3. Ideally, these run on every PR so that we can find out if a commit would
> introduce a change that would break the release-ready status of arrow.
>
> These requirements indicate to me that a single, extensible, cross-platform
> tool--as opposed to many tools that are tied together by a shell script--is
> what we need.
>
> There are a few tools in this space that I'm aware of:
> 1. Bazel (out of Google)
> 2. Buck (out of Facebook)
>
> I'm not sure what others are out there, but I'm sure there must be some.
> I don't really have a strong opinion on either Bazel or Buck, but I suspect
> that since we follow Google's conventions in a few places integrating Bazel
> into the arrow codebase would be less work.
>
> The main risk I see here is that it's possible that bazel isn't the right
> tool. I'm not sure how to mitigate this risk other than to make sure that
> our requirements can be met by it by scouring the Bazel docs.
>
> I do think that the fact that Bazel is extensible mitigates some risk here.
> For example, we'd likely have to add rule for building conda packages and
> pip wheels.
>
> I guess CMake is extensible too, but I don't think I've ever seen the
> extensibility features of CMake as anything but a burden. Bazel's extension
> language is a subset of Python and I would therefore expect it to be a lot
> easier to use.
>
> I'm interested to hear others' experiences and opinions on similar
> problems. Also, if I've missed anything in the requirements list, please
> don't hestitate to respond.
>
> Let's fix our packaging!
> -Phillip
> On Fri, Mar 23, 2018 at 11:21 PM Holden Karau <ho...@pigscanfly.ca> wrote:
> > I know in Spark we’ve benefited by having some of the different language
> > devs act as RMs and each time that language dev has ended up improving a
> > bunch of how their components packaging has been done. Not to suggest we
> > should just do what other projects do, but maybe an idea to consider?
> >
> > On Fri, Mar 23, 2018 at 12:59 PM Wes McKinney <we...@gmail.com> wrote:
> > > hi folks,
> > > So, I want to bring light to the problems we are having delivering
> > > binary artifacts after Arrow releases.
> > >
> > > We have some amount of packaging automation implemented in
> > > https://github.com/apache/arrow-dist using Travis CI and Appveyor to
> > > upload packages to Bintray, a packaging hosting service.
> > >
> > > Unfortunately, we discovered a bunch of problems with these packaging
> > > scripts after the release vote closed on Monday, and now 4 days later,
> > > we still have been unable to post binaries to
> > > https://pypi.python.org/pypi/pyarrow
> > >
> > > This is no one's fault, but it highlights structural problems with our
> > > development process:
> > >
> > > * Why does producing packages after a release require error-prone manual
> > > labor?
> > >
> > > * Why are we only finding out about packaging problem after a release
> > > vote closes?
> > >
> > > * Why is setting up nightly binary builds a brittle and bespoke process?
> > > I hope all agree that:
> > > * Packaging should not be a hardship or require a lot of manual labor
> > > * Packaging problems on the master branch should be made known within
> > > ~24 hours, so they can be remedied immediately
> > >
> > > * It should be straightforward to produce binary artifacts for all
> > > supported platforms and programming languages
> > >
> > > Eventually, we should include some binary artifacts in our release
> > > votes, but we are pretty far away from suitable automation to make
> > > this possible.
> > >
> > > I don't know any easy solutions, but Apache Arrow has grown widely
> > > used enough that I think it's worth our taking the time to plan and
> > > execute some solutions to these problems, which I expect to pay
> > > dividends in our community's productivity over time.
> > >
> > > Thanks,
> > > Wes
> > >
> > --
> > Twitter: https://twitter.com/holdenkarau
>
>

Re: Confronting Arrow packaging problems

[Wes McKinney <we...@gmail.com>]

hi Phillip,

Thank you for these thoughts.

I am all for improving our build systems to be more
readable/maintainable and easier to debug and develop. I see our
problems are being different than improving the build systems for the
Python and C++ libraries: we need to set up a build and packaging
automation system for our build systems, whatever they may be. Whether
we use CMake or Bazel, we still need a system that will

* Either nightly, or on each commit, build all variants of all
packages on all platforms
* Upload the binary artifacts to a central location
* Capture build logs and upload to a central location for later inspection
* Notify Arrow community via e-mail in the event of any failure

Moving to Bazel will make the actual packaging process better in some
aspects, but it will _not_ help with the workflow automation and
nightly validation. Currently, it requires a human in the loop and
commit like

https://github.com/apache/arrow-dist/commit/2e99e83c352775f1ccde096242c4bec783340369

which took up collectively multiple days of development time in the last week.

We have some other structural problems, i.e. much of the code in
apache/arrow-dist should probably be in apache/arrow, and in the form
of templates so that we can _generate_ a build configuration given a
set of pinned dependencies and build parameters (e.g. in the case of
C++/Python, flags relating to linking and bundling of thirdparty
dependencies like Boost)

I think it's worth seriously considering the Bazel route, but we
should not start building a new build system without coming up with a
coherent and actionable plan to address the build and packaging
automation problem that we have now. Ideally this automation system
would be agnostic to the underlying build systems used by each
component. If we move to a better build system in the future, then the
build/packaging ("metabuild") system will have to do less work.

- Wes

On Sat, Mar 24, 2018 at 4:13 PM, Phillip Cloud <cp...@gmail.com> wrote:
> I think we need to use a tool that can perform every single step of the
> deployment process, end-to-end. Right now, cmake isn't cutting it IMO
> because it lends itself quite heavily to copy pasting and oodles of bash
> scripts that are indecipherable by anyone except the original author.
>
> With that in mind, here's what I think the requirements are for the next
> generation of arrow's build and deployment system for native code (C, C++,
> Python):
>
> For every platform (Linux, OS X, and Windows):
>
> 1. Build sources (these langauges are currently the most cumbersome at the
> moment)
> 2. Run all tests for each language (ideally integration tests as well, but
> I'm not sure if that should be a hard requirement at the moment)
> 3. Build the API documentation for each platform
> 4. Build installable packages that we support:
>   * conda
>   * pip wheels
>   * deb
>   * rpm
>
> Additional requirements:
> 1. Be able to run all of these steps on any platform with minimal
> environment setup.
> 2. The output of the doc build and package build steps **should be in a
> release-ready state at all times**. If they are not then we should fail the
> build
> 3. Ideally, these run on every PR so that we can find out if a commit would
> introduce a change that would break the release-ready status of arrow.
>
> These requirements indicate to me that a single, extensible, cross-platform
> tool--as opposed to many tools that are tied together by a shell script--is
> what we need.
>
> There are a few tools in this space that I'm aware of:
>
> 1. Bazel (out of Google)
> 2. Buck (out of Facebook)
>
> I'm not sure what others are out there, but I'm sure there must be some.
>
> I don't really have a strong opinion on either Bazel or Buck, but I suspect
> that since we follow Google's conventions in a few places integrating Bazel
> into the arrow codebase would be less work.
>
> The main risk I see here is that it's possible that bazel isn't the right
> tool. I'm not sure how to mitigate this risk other than to make sure that
> our requirements can be met by it by scouring the Bazel docs.
>
> I do think that the fact that Bazel is extensible mitigates some risk here.
> For example, we'd likely have to add rule for building conda packages and
> pip wheels.
>
> I guess CMake is extensible too, but I don't think I've ever seen the
> extensibility features of CMake as anything but a burden. Bazel's extension
> language is a subset of Python and I would therefore expect it to be a lot
> easier to use.
>
> I'm interested to hear others' experiences and opinions on similar
> problems. Also, if I've missed anything in the requirements list, please
> don't hestitate to respond.
>
> Let's fix our packaging!
>
> -Phillip
>
> On Fri, Mar 23, 2018 at 11:21 PM Holden Karau <ho...@pigscanfly.ca> wrote:
>
>> I know in Spark we’ve benefited by having some of the different language
>> devs act as RMs and each time that language dev has ended up improving a
>> bunch of how their components packaging has been done. Not to suggest we
>> should just do what other projects do, but maybe an idea to consider?
>>
>> On Fri, Mar 23, 2018 at 12:59 PM Wes McKinney <we...@gmail.com> wrote:
>>
>> > hi folks,
>> >
>> > So, I want to bring light to the problems we are having delivering
>> > binary artifacts after Arrow releases.
>> >
>> > We have some amount of packaging automation implemented in
>> > https://github.com/apache/arrow-dist using Travis CI and Appveyor to
>> > upload packages to Bintray, a packaging hosting service.
>> >
>> > Unfortunately, we discovered a bunch of problems with these packaging
>> > scripts after the release vote closed on Monday, and now 4 days later,
>> > we still have been unable to post binaries to
>> > https://pypi.python.org/pypi/pyarrow
>> >
>> > This is no one's fault, but it highlights structural problems with our
>> > development process:
>> >
>> > * Why does producing packages after a release require error-prone manual
>> > labor?
>> >
>> > * Why are we only finding out about packaging problem after a release
>> > vote closes?
>> >
>> > * Why is setting up nightly binary builds a brittle and bespoke process?
>> >
>> > I hope all agree that:
>> >
>> > * Packaging should not be a hardship or require a lot of manual labor
>> >
>> > * Packaging problems on the master branch should be made known within
>> > ~24 hours, so they can be remedied immediately
>> >
>> > * It should be straightforward to produce binary artifacts for all
>> > supported platforms and programming languages
>> >
>> > Eventually, we should include some binary artifacts in our release
>> > votes, but we are pretty far away from suitable automation to make
>> > this possible.
>> >
>> > I don't know any easy solutions, but Apache Arrow has grown widely
>> > used enough that I think it's worth our taking the time to plan and
>> > execute some solutions to these problems, which I expect to pay
>> > dividends in our community's productivity over time.
>> >
>> > Thanks,
>> > Wes
>> >
>> --
>> Twitter: https://twitter.com/holdenkarau
>>

Re: Confronting Arrow packaging problems

[Phillip Cloud <cp...@gmail.com>]

I think we need to use a tool that can perform every single step of the
deployment process, end-to-end. Right now, cmake isn't cutting it IMO
because it lends itself quite heavily to copy pasting and oodles of bash
scripts that are indecipherable by anyone except the original author.

With that in mind, here's what I think the requirements are for the next
generation of arrow's build and deployment system for native code (C, C++,
Python):

For every platform (Linux, OS X, and Windows):

1. Build sources (these langauges are currently the most cumbersome at the
moment)
2. Run all tests for each language (ideally integration tests as well, but
I'm not sure if that should be a hard requirement at the moment)
3. Build the API documentation for each platform
4. Build installable packages that we support:
  * conda
  * pip wheels
  * deb
  * rpm

Additional requirements:
1. Be able to run all of these steps on any platform with minimal
environment setup.
2. The output of the doc build and package build steps **should be in a
release-ready state at all times**. If they are not then we should fail the
build
3. Ideally, these run on every PR so that we can find out if a commit would
introduce a change that would break the release-ready status of arrow.

These requirements indicate to me that a single, extensible, cross-platform
tool--as opposed to many tools that are tied together by a shell script--is
what we need.

There are a few tools in this space that I'm aware of:

1. Bazel (out of Google)
2. Buck (out of Facebook)

I'm not sure what others are out there, but I'm sure there must be some.

I don't really have a strong opinion on either Bazel or Buck, but I suspect
that since we follow Google's conventions in a few places integrating Bazel
into the arrow codebase would be less work.

The main risk I see here is that it's possible that bazel isn't the right
tool. I'm not sure how to mitigate this risk other than to make sure that
our requirements can be met by it by scouring the Bazel docs.

I do think that the fact that Bazel is extensible mitigates some risk here.
For example, we'd likely have to add rule for building conda packages and
pip wheels.

I guess CMake is extensible too, but I don't think I've ever seen the
extensibility features of CMake as anything but a burden. Bazel's extension
language is a subset of Python and I would therefore expect it to be a lot
easier to use.

I'm interested to hear others' experiences and opinions on similar
problems. Also, if I've missed anything in the requirements list, please
don't hestitate to respond.

Let's fix our packaging!

-Phillip

On Fri, Mar 23, 2018 at 11:21 PM Holden Karau <ho...@pigscanfly.ca> wrote:

> I know in Spark we’ve benefited by having some of the different language
> devs act as RMs and each time that language dev has ended up improving a
> bunch of how their components packaging has been done. Not to suggest we
> should just do what other projects do, but maybe an idea to consider?
>
> On Fri, Mar 23, 2018 at 12:59 PM Wes McKinney <we...@gmail.com> wrote:
>
> > hi folks,
> >
> > So, I want to bring light to the problems we are having delivering
> > binary artifacts after Arrow releases.
> >
> > We have some amount of packaging automation implemented in
> > https://github.com/apache/arrow-dist using Travis CI and Appveyor to
> > upload packages to Bintray, a packaging hosting service.
> >
> > Unfortunately, we discovered a bunch of problems with these packaging
> > scripts after the release vote closed on Monday, and now 4 days later,
> > we still have been unable to post binaries to
> > https://pypi.python.org/pypi/pyarrow
> >
> > This is no one's fault, but it highlights structural problems with our
> > development process:
> >
> > * Why does producing packages after a release require error-prone manual
> > labor?
> >
> > * Why are we only finding out about packaging problem after a release
> > vote closes?
> >
> > * Why is setting up nightly binary builds a brittle and bespoke process?
> >
> > I hope all agree that:
> >
> > * Packaging should not be a hardship or require a lot of manual labor
> >
> > * Packaging problems on the master branch should be made known within
> > ~24 hours, so they can be remedied immediately
> >
> > * It should be straightforward to produce binary artifacts for all
> > supported platforms and programming languages
> >
> > Eventually, we should include some binary artifacts in our release
> > votes, but we are pretty far away from suitable automation to make
> > this possible.
> >
> > I don't know any easy solutions, but Apache Arrow has grown widely
> > used enough that I think it's worth our taking the time to plan and
> > execute some solutions to these problems, which I expect to pay
> > dividends in our community's productivity over time.
> >
> > Thanks,
> > Wes
> >
> --
> Twitter: https://twitter.com/holdenkarau
>

Re: Confronting Arrow packaging problems

[Holden Karau <ho...@pigscanfly.ca>]

I know in Spark we’ve benefited by having some of the different language
devs act as RMs and each time that language dev has ended up improving a
bunch of how their components packaging has been done. Not to suggest we
should just do what other projects do, but maybe an idea to consider?

On Fri, Mar 23, 2018 at 12:59 PM Wes McKinney <we...@gmail.com> wrote:

> hi folks,
>
> So, I want to bring light to the problems we are having delivering
> binary artifacts after Arrow releases.
>
> We have some amount of packaging automation implemented in
> https://github.com/apache/arrow-dist using Travis CI and Appveyor to
> upload packages to Bintray, a packaging hosting service.
>
> Unfortunately, we discovered a bunch of problems with these packaging
> scripts after the release vote closed on Monday, and now 4 days later,
> we still have been unable to post binaries to
> https://pypi.python.org/pypi/pyarrow
>
> This is no one's fault, but it highlights structural problems with our
> development process:
>
> * Why does producing packages after a release require error-prone manual
> labor?
>
> * Why are we only finding out about packaging problem after a release
> vote closes?
>
> * Why is setting up nightly binary builds a brittle and bespoke process?
>
> I hope all agree that:
>
> * Packaging should not be a hardship or require a lot of manual labor
>
> * Packaging problems on the master branch should be made known within
> ~24 hours, so they can be remedied immediately
>
> * It should be straightforward to produce binary artifacts for all
> supported platforms and programming languages
>
> Eventually, we should include some binary artifacts in our release
> votes, but we are pretty far away from suitable automation to make
> this possible.
>
> I don't know any easy solutions, but Apache Arrow has grown widely
> used enough that I think it's worth our taking the time to plan and
> execute some solutions to these problems, which I expect to pay
> dividends in our community's productivity over time.
>
> Thanks,
> Wes
>
-- 
Twitter: https://twitter.com/holdenkarau

Re: Confronting Arrow packaging problems

[Phillip Cloud <cp...@gmail.com>]

I've started a document on this issue here:
https://docs.google.com/document/d/1IyhbQpiElxTsI8HbMZ-g9EGPOtcFdtMBzEyDJv48BKc/edit?usp=sharing

On Sun, Mar 25, 2018 at 10:40 AM Wes McKinney <we...@gmail.com> wrote:

> @Kou, automated commits / PRs to trigger Appveyor/Travis CI are most
> likely part of the solution, but there are other issues.
>
> We should start a Google document or something to enumerate all the
> things we want to implement and identify the technical issues with
> each thing. For example, at present, among other things:
>
> * Building a newer version of arrow-dist requires manual tweaks to many
> files
> * Packages are deployed from different GitHub repos (apache/arrow-dist
> _and_ wesm/arrow-dist, due to a quirk with Appveyor)
> * Packages must be manually downloaded from BinTray (by clicking
> around a web UI) after a successful run
>
> At a high level goal, we need to figure out how to prevent the
> disaster of the last 7 days from occurring ever again. Easily 10-20
> hours of wasted time this past week on packaging issues, and some
> things are still broken -- this is not acceptable to me. It's like the
> frog slowly boiling in water; we have allowed this technical debt to
> accumulate, and we should not allow this to continue.
>
> It does not seem that many of the issues we are having are related to
> or caused by CMake, so I'd like to table the build system discussion
> until we identify solutions to the other problems.
>
> - Wes
>
> On Sun, Mar 25, 2018 at 9:38 AM, Kouhei Sutou <ko...@clear-code.com> wrote:
> > Hi,
> >
> > How about creating a pull request to apache/arrow-dist with
> > changes to use the latest apache/arrow? (A sample script
> > exists at the end.)
> >
> > If the pull request is created, we can test packaging with
> > the latest apache/arrow on Travis CI. If we add the
> > following configuration to .travis.yml in apache/arrow-dist,
> > we can get a notification via e-mail on failure:
> >
> > ---
> > notifications:
> >   email:
> >     recipients:
> >       - build-failure@arrow.apache.org # or something
> >     # See
> https://docs.travis-ci.com/user/notifications/#Changing-notification-frequency
> >     # for details
> >     on_success: never # or change
> >     on_failure: always
> > ---
> >
> > If the pull request is succeeded, we just close the pull
> > request. (It should be automated.) Or we merge it into
> > master and publish daily packages.
> >
> >
> > ---
> > #!/usr/bin/env ruby
> >
> > require "octokit"
> >
> > github_token = ENV["GITHUB_TOKEN"]
> > if github_token.nil?
> >   $stderr.puts("Must specify GitHub access token by GITHUB_TOKEN
> environment variable")
> >   exit(false)
> > end
> > client = Octokit::Client.new(:access_token => github_token)
> >
> > now = Time.now
> > branch = now.strftime("update-%Y-%m-%d")
> > message = now.strftime("Update at %Y-%m-%d")
> >
> > def run(*command_line)
> >   IO.pipe do |input, output|
> >     expanded_command_line = command_line.join(" ")
> >     puts(expanded_command_line)
> >     response = nil
> >     begin
> >       read_thread = Thread.new do
> >         response = input.read
> >       end
> >       unless system(*command_line, :out => output)
> >         raise "failed to run: #{expanded_command_line}"
> >       end
> >       output.close
> >     ensure
> >       read_thread.join
> >     end
> >     response
> >   end
> > end
> >
> > run("git", "fetch", "--all", "--prune")
> > run("git", "checkout", "master")
> > run("git", "rebase", "upstream/master")
> >
> > begin
> >   run("git", "checkout", "-b", branch)
> >   Dir.chdir("arrow") do
> >     current_version = run("git", "log", "-n1", "--format=%H",
> "HEAD").chomp
> >     run("git", "checkout", "master")
> >     run("git", "pull")
> >     new_version = run("git", "log", "-n1", "--format=%H", "HEAD").chomp
> >     exit(true) if current_version == new_version
> >   end
> >   Dir.chdir("cpp-linux") do
> >     run("rake", "version:update")
> >   end
> >   # And do something for other packages
> >   run("git", "commit", "-a", "-m", message)
> >   run("git", "push", "origin", branch)
> > ensure
> >   run("git", "checkout", "master")
> >   run("git", "submodule", "update", "arrow")
> >   run("git", "branch", "-D", branch)
> > end
> >
> > client.create_pull_request("apache/arrow-dist", "master", branch,
> message)
> > ---
> >
> > Thanks,
> > --
> > kou
> >
> > In <CA...@mail.gmail.com>
> >   "Confronting Arrow packaging problems" on Fri, 23 Mar 2018 12:58:54
> -0400,
> >   Wes McKinney <we...@gmail.com> wrote:
> >
> >> hi folks,
> >>
> >> So, I want to bring light to the problems we are having delivering
> >> binary artifacts after Arrow releases.
> >>
> >> We have some amount of packaging automation implemented in
> >> https://github.com/apache/arrow-dist using Travis CI and Appveyor to
> >> upload packages to Bintray, a packaging hosting service.
> >>
> >> Unfortunately, we discovered a bunch of problems with these packaging
> >> scripts after the release vote closed on Monday, and now 4 days later,
> >> we still have been unable to post binaries to
> >> https://pypi.python.org/pypi/pyarrow
> >>
> >> This is no one's fault, but it highlights structural problems with our
> >> development process:
> >>
> >> * Why does producing packages after a release require error-prone
> manual labor?
> >>
> >> * Why are we only finding out about packaging problem after a release
> >> vote closes?
> >>
> >> * Why is setting up nightly binary builds a brittle and bespoke process?
> >>
> >> I hope all agree that:
> >>
> >> * Packaging should not be a hardship or require a lot of manual labor
> >>
> >> * Packaging problems on the master branch should be made known within
> >> ~24 hours, so they can be remedied immediately
> >>
> >> * It should be straightforward to produce binary artifacts for all
> >> supported platforms and programming languages
> >>
> >> Eventually, we should include some binary artifacts in our release
> >> votes, but we are pretty far away from suitable automation to make
> >> this possible.
> >>
> >> I don't know any easy solutions, but Apache Arrow has grown widely
> >> used enough that I think it's worth our taking the time to plan and
> >> execute some solutions to these problems, which I expect to pay
> >> dividends in our community's productivity over time.
> >>
> >> Thanks,
> >> Wes
>

Re: Confronting Arrow packaging problems

[Wes McKinney <we...@gmail.com>]

@Kou, automated commits / PRs to trigger Appveyor/Travis CI are most
likely part of the solution, but there are other issues.

We should start a Google document or something to enumerate all the
things we want to implement and identify the technical issues with
each thing. For example, at present, among other things:

* Building a newer version of arrow-dist requires manual tweaks to many files
* Packages are deployed from different GitHub repos (apache/arrow-dist
_and_ wesm/arrow-dist, due to a quirk with Appveyor)
* Packages must be manually downloaded from BinTray (by clicking
around a web UI) after a successful run

At a high level goal, we need to figure out how to prevent the
disaster of the last 7 days from occurring ever again. Easily 10-20
hours of wasted time this past week on packaging issues, and some
things are still broken -- this is not acceptable to me. It's like the
frog slowly boiling in water; we have allowed this technical debt to
accumulate, and we should not allow this to continue.

It does not seem that many of the issues we are having are related to
or caused by CMake, so I'd like to table the build system discussion
until we identify solutions to the other problems.

- Wes

On Sun, Mar 25, 2018 at 9:38 AM, Kouhei Sutou <ko...@clear-code.com> wrote:
> Hi,
>
> How about creating a pull request to apache/arrow-dist with
> changes to use the latest apache/arrow? (A sample script
> exists at the end.)
>
> If the pull request is created, we can test packaging with
> the latest apache/arrow on Travis CI. If we add the
> following configuration to .travis.yml in apache/arrow-dist,
> we can get a notification via e-mail on failure:
>
> ---
> notifications:
>   email:
>     recipients:
>       - build-failure@arrow.apache.org # or something
>     # See https://docs.travis-ci.com/user/notifications/#Changing-notification-frequency
>     # for details
>     on_success: never # or change
>     on_failure: always
> ---
>
> If the pull request is succeeded, we just close the pull
> request. (It should be automated.) Or we merge it into
> master and publish daily packages.
>
>
> ---
> #!/usr/bin/env ruby
>
> require "octokit"
>
> github_token = ENV["GITHUB_TOKEN"]
> if github_token.nil?
>   $stderr.puts("Must specify GitHub access token by GITHUB_TOKEN environment variable")
>   exit(false)
> end
> client = Octokit::Client.new(:access_token => github_token)
>
> now = Time.now
> branch = now.strftime("update-%Y-%m-%d")
> message = now.strftime("Update at %Y-%m-%d")
>
> def run(*command_line)
>   IO.pipe do |input, output|
>     expanded_command_line = command_line.join(" ")
>     puts(expanded_command_line)
>     response = nil
>     begin
>       read_thread = Thread.new do
>         response = input.read
>       end
>       unless system(*command_line, :out => output)
>         raise "failed to run: #{expanded_command_line}"
>       end
>       output.close
>     ensure
>       read_thread.join
>     end
>     response
>   end
> end
>
> run("git", "fetch", "--all", "--prune")
> run("git", "checkout", "master")
> run("git", "rebase", "upstream/master")
>
> begin
>   run("git", "checkout", "-b", branch)
>   Dir.chdir("arrow") do
>     current_version = run("git", "log", "-n1", "--format=%H", "HEAD").chomp
>     run("git", "checkout", "master")
>     run("git", "pull")
>     new_version = run("git", "log", "-n1", "--format=%H", "HEAD").chomp
>     exit(true) if current_version == new_version
>   end
>   Dir.chdir("cpp-linux") do
>     run("rake", "version:update")
>   end
>   # And do something for other packages
>   run("git", "commit", "-a", "-m", message)
>   run("git", "push", "origin", branch)
> ensure
>   run("git", "checkout", "master")
>   run("git", "submodule", "update", "arrow")
>   run("git", "branch", "-D", branch)
> end
>
> client.create_pull_request("apache/arrow-dist", "master", branch, message)
> ---
>
> Thanks,
> --
> kou
>
> In <CA...@mail.gmail.com>
>   "Confronting Arrow packaging problems" on Fri, 23 Mar 2018 12:58:54 -0400,
>   Wes McKinney <we...@gmail.com> wrote:
>
>> hi folks,
>>
>> So, I want to bring light to the problems we are having delivering
>> binary artifacts after Arrow releases.
>>
>> We have some amount of packaging automation implemented in
>> https://github.com/apache/arrow-dist using Travis CI and Appveyor to
>> upload packages to Bintray, a packaging hosting service.
>>
>> Unfortunately, we discovered a bunch of problems with these packaging
>> scripts after the release vote closed on Monday, and now 4 days later,
>> we still have been unable to post binaries to
>> https://pypi.python.org/pypi/pyarrow
>>
>> This is no one's fault, but it highlights structural problems with our
>> development process:
>>
>> * Why does producing packages after a release require error-prone manual labor?
>>
>> * Why are we only finding out about packaging problem after a release
>> vote closes?
>>
>> * Why is setting up nightly binary builds a brittle and bespoke process?
>>
>> I hope all agree that:
>>
>> * Packaging should not be a hardship or require a lot of manual labor
>>
>> * Packaging problems on the master branch should be made known within
>> ~24 hours, so they can be remedied immediately
>>
>> * It should be straightforward to produce binary artifacts for all
>> supported platforms and programming languages
>>
>> Eventually, we should include some binary artifacts in our release
>> votes, but we are pretty far away from suitable automation to make
>> this possible.
>>
>> I don't know any easy solutions, but Apache Arrow has grown widely
>> used enough that I think it's worth our taking the time to plan and
>> execute some solutions to these problems, which I expect to pay
>> dividends in our community's productivity over time.
>>
>> Thanks,
>> Wes

Re: Confronting Arrow packaging problems

[Kouhei Sutou <ko...@clear-code.com>]

Hi,

How about creating a pull request to apache/arrow-dist with
changes to use the latest apache/arrow? (A sample script
exists at the end.)

If the pull request is created, we can test packaging with
the latest apache/arrow on Travis CI. If we add the
following configuration to .travis.yml in apache/arrow-dist,
we can get a notification via e-mail on failure:

---
notifications:
  email:
    recipients:
      - build-failure@arrow.apache.org # or something
    # See https://docs.travis-ci.com/user/notifications/#Changing-notification-frequency
    # for details
    on_success: never # or change
    on_failure: always
---

If the pull request is succeeded, we just close the pull
request. (It should be automated.) Or we merge it into
master and publish daily packages.


---
#!/usr/bin/env ruby

require "octokit"

github_token = ENV["GITHUB_TOKEN"]
if github_token.nil?
  $stderr.puts("Must specify GitHub access token by GITHUB_TOKEN environment variable")
  exit(false)
end
client = Octokit::Client.new(:access_token => github_token)

now = Time.now
branch = now.strftime("update-%Y-%m-%d")
message = now.strftime("Update at %Y-%m-%d")

def run(*command_line)
  IO.pipe do |input, output|
    expanded_command_line = command_line.join(" ")
    puts(expanded_command_line)
    response = nil
    begin
      read_thread = Thread.new do
        response = input.read
      end
      unless system(*command_line, :out => output)
        raise "failed to run: #{expanded_command_line}"
      end
      output.close
    ensure
      read_thread.join
    end
    response
  end
end

run("git", "fetch", "--all", "--prune")
run("git", "checkout", "master")
run("git", "rebase", "upstream/master")

begin
  run("git", "checkout", "-b", branch)
  Dir.chdir("arrow") do
    current_version = run("git", "log", "-n1", "--format=%H", "HEAD").chomp
    run("git", "checkout", "master")
    run("git", "pull")
    new_version = run("git", "log", "-n1", "--format=%H", "HEAD").chomp
    exit(true) if current_version == new_version
  end
  Dir.chdir("cpp-linux") do
    run("rake", "version:update")
  end
  # And do something for other packages
  run("git", "commit", "-a", "-m", message)
  run("git", "push", "origin", branch)
ensure
  run("git", "checkout", "master")
  run("git", "submodule", "update", "arrow")
  run("git", "branch", "-D", branch)
end

client.create_pull_request("apache/arrow-dist", "master", branch, message)
---

Thanks,
--
kou

In <CA...@mail.gmail.com>
  "Confronting Arrow packaging problems" on Fri, 23 Mar 2018 12:58:54 -0400,
  Wes McKinney <we...@gmail.com> wrote:

> hi folks,
> 
> So, I want to bring light to the problems we are having delivering
> binary artifacts after Arrow releases.
> 
> We have some amount of packaging automation implemented in
> https://github.com/apache/arrow-dist using Travis CI and Appveyor to
> upload packages to Bintray, a packaging hosting service.
> 
> Unfortunately, we discovered a bunch of problems with these packaging
> scripts after the release vote closed on Monday, and now 4 days later,
> we still have been unable to post binaries to
> https://pypi.python.org/pypi/pyarrow
> 
> This is no one's fault, but it highlights structural problems with our
> development process:
> 
> * Why does producing packages after a release require error-prone manual labor?
> 
> * Why are we only finding out about packaging problem after a release
> vote closes?
> 
> * Why is setting up nightly binary builds a brittle and bespoke process?
> 
> I hope all agree that:
> 
> * Packaging should not be a hardship or require a lot of manual labor
> 
> * Packaging problems on the master branch should be made known within
> ~24 hours, so they can be remedied immediately
> 
> * It should be straightforward to produce binary artifacts for all
> supported platforms and programming languages
> 
> Eventually, we should include some binary artifacts in our release
> votes, but we are pretty far away from suitable automation to make
> this possible.
> 
> I don't know any easy solutions, but Apache Arrow has grown widely
> used enough that I think it's worth our taking the time to plan and
> execute some solutions to these problems, which I expect to pay
> dividends in our community's productivity over time.
> 
> Thanks,
> Wes