[DISCUSS] Streamlining Builds

[Saad Ur Rahman <sa...@apache.org>]

Hello everyone,

With the 0.20.5 RC complete, I would like to take some time to discuss some
of the issues that were faced during the build. The vast majority of the
builds needed to be completed off of the ASF Infrastructure and we need to
work towards remediating this.

These are my thoughts and observations from the process:

[1] All build scripts should execute inside of Docker containers. This is
for consistency and control over the build system. Currently, some scripts
execute inside containers whilst others do not.
[2] The use of PEX in the build scripts is causing critical issues. This is
undoubtedly going to get worse over time. We require a solution to this
issue, the ideal one being to start unwinding the PEX from the build
scripts gradually so that we can verify changes.
[3] We should have a script that bootstraps the release candidate build
processes. We should not have to manually perform these steps nor should we
need to intervene in them. A script should execute and trigger the
generation of all the files we need. The only human intervention should be
signing the artifacts and releasing them.

What are your thoughts and observations?

Re: [DISCUSS] Streamlining Builds

[Ning Wang <wa...@gmail.com>]

Agreed with 2 & 3.

For 1, I feel the only concern is that docker is Linux friendly, but not
for other platforms (I mean MacOS as target OS). Personally, I don't feel
it is a real problem since it's not that bad to run with docker instead of
native MacOS, but I just would like to bring it up for us to make the call.

On Wed, Jan 19, 2022 at 6:20 PM Josh Fischer <jo...@joshfischer.io> wrote:

> inlined.
>
> On Tue, Jan 11, 2022 at 5:13 PM Saad Ur Rahman <sa...@apache.org>
> wrote:
>
> > Hello everyone,
> >
> > With the 0.20.5 RC complete, I would like to take some time to discuss
> some
> > of the issues that were faced during the build. The vast majority of the
> > builds needed to be completed off of the ASF Infrastructure and we need
> to
> > work towards remediating this.
> >
> > These are my thoughts and observations from the process:
> >
> > [1] All build scripts should execute inside of Docker containers. This is
> > for consistency and control over the build system. Currently, some
> scripts
> > execute inside containers whilst others do not.
> >
>  - Agree. However we saw that builds in containers were failing on Apache's
> Jenkins instance. We will need to work with infra to diagnose.
>
> > [2] The use of PEX in the build scripts is causing critical issues. This
> is
> > undoubtedly going to get worse over time. We require a solution to this
> > issue, the ideal one being to start unwinding the PEX from the build
> > scripts gradually so that we can verify changes.
> >
> - work is in progress on this.  It'll be slow but it's in the Heron repo
> under the branch joshfischer/native-python-rules
>
> > [3] We should have a script that bootstraps the release candidate build
> > processes. We should not have to manually perform these steps nor should
> we
> > need to intervene in them. A script should execute and trigger the
> > generation of all the files we need. The only human intervention should
> be
> > signing the artifacts and releasing them.
> >
> - Agreed.
>
> >
> > What are your thoughts and observations?
> >
>
> Biggest challenge I see is time.,. Do we work to get all of this done
> before attempting to graduate from the incubator or  take it in smaller
> steps and get done what we can while in the incubator?  Streamlining builds
> is one aspect of what needs to be done. Another is the community.  How can
> we make Heron easier to use for developers while moving forward with other
> pieces of development?
>

Re: [DISCUSS] Streamlining Builds

[Josh Fischer <jo...@joshfischer.io>]

inlined.

On Tue, Jan 11, 2022 at 5:13 PM Saad Ur Rahman <sa...@apache.org>
wrote:

> Hello everyone,
>
> With the 0.20.5 RC complete, I would like to take some time to discuss some
> of the issues that were faced during the build. The vast majority of the
> builds needed to be completed off of the ASF Infrastructure and we need to
> work towards remediating this.
>
> These are my thoughts and observations from the process:
>
> [1] All build scripts should execute inside of Docker containers. This is
> for consistency and control over the build system. Currently, some scripts
> execute inside containers whilst others do not.
>
 - Agree. However we saw that builds in containers were failing on Apache's
Jenkins instance. We will need to work with infra to diagnose.

> [2] The use of PEX in the build scripts is causing critical issues. This is
> undoubtedly going to get worse over time. We require a solution to this
> issue, the ideal one being to start unwinding the PEX from the build
> scripts gradually so that we can verify changes.
>
- work is in progress on this.  It'll be slow but it's in the Heron repo
under the branch joshfischer/native-python-rules

> [3] We should have a script that bootstraps the release candidate build
> processes. We should not have to manually perform these steps nor should we
> need to intervene in them. A script should execute and trigger the
> generation of all the files we need. The only human intervention should be
> signing the artifacts and releasing them.
>
- Agreed.

>
> What are your thoughts and observations?
>

Biggest challenge I see is time.,. Do we work to get all of this done
before attempting to graduate from the incubator or  take it in smaller
steps and get done what we can while in the incubator?  Streamlining builds
is one aspect of what needs to be done. Another is the community.  How can
we make Heron easier to use for developers while moving forward with other
pieces of development?