You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Raul Cumplido <ra...@voltrondata.com> on 2022/05/09 12:33:31 UTC
[DISC][Release] More control on Release Candidates commits
Hi,
I would like to propose a change in our release process.
The rationale for the change is to avoid introducing new issues once a
Release Candidate has already been cut by only merging specific commits to
new release candidates.
Currently once a new Release Candidate is required we drop the previous
version branch and create a new Release Candidate from the master branch on
the repository [1]. This has the problem that we might introduce new bugs
to the Release creating the need of cutting further release candidates. As
an example, for the release 7.0.0, 10 release candidates were required and
for the release 8.0.0 there was the need to remove a specific commit that
introduced some new issues [2]. For the release 8.0.0 we were able to find
it early but it could have potentially been introduced and created the need
for further RCs.
I would like to propose the following workflow.
When creating the initial RC, create both an rc1 branch and the version
branch from master.
release-x.0.0 and release-x.0.0.rc1
If a new RC is required, drop the release-x.0.0 (as we do today) and create
a new RC branch from the previous RC branch (instead of master), then
cherry pick only the specific commits that have been identified to be part
of the new release candidate. We can automate the cherrypick process via a
script specifying the JIRA tickets or the commit hashes that we want to add
to the new release candidate. Once the new RC branch is ready, create a new
version branch from it and proceed as today.
The commits to be added to the release once a release candidate has already
been cut will usually be fixes for the release but could also be features
if there is community consensus that a feature must be introduced to the
release.
This change will allow us to have a more granular control of what goes in
the release once a release candidate has been cut and speed up the release
by focusing both the release manager's and the community's efforts and
potentially reducing the number of RCs to be created and verified.
Thanks,
Raúl
[1]
https://cwiki.apache.org/confluence/display/ARROW/Release+Management+Guide
[2] https://github.com/apache/arrow/pull/12590#issuecomment-1116144088
Re: [DISC][Release] More control on Release Candidates commits
Posted by Antoine Pitrou <an...@python.org>.
+1 from me. I'm actually surprised that we didn't do something like that
already. Adding new features from one RC to another sounds like a very
bad idea.
Regards
Antoine.
Le 09/05/2022 à 14:33, Raul Cumplido a écrit :
> Hi,
>
> I would like to propose a change in our release process.
>
> The rationale for the change is to avoid introducing new issues once a
> Release Candidate has already been cut by only merging specific commits to
> new release candidates.
>
> Currently once a new Release Candidate is required we drop the previous
> version branch and create a new Release Candidate from the master branch on
> the repository [1]. This has the problem that we might introduce new bugs
> to the Release creating the need of cutting further release candidates. As
> an example, for the release 7.0.0, 10 release candidates were required and
> for the release 8.0.0 there was the need to remove a specific commit that
> introduced some new issues [2]. For the release 8.0.0 we were able to find
> it early but it could have potentially been introduced and created the need
> for further RCs.
>
> I would like to propose the following workflow.
> When creating the initial RC, create both an rc1 branch and the version
> branch from master.
> release-x.0.0 and release-x.0.0.rc1
>
> If a new RC is required, drop the release-x.0.0 (as we do today) and create
> a new RC branch from the previous RC branch (instead of master), then
> cherry pick only the specific commits that have been identified to be part
> of the new release candidate. We can automate the cherrypick process via a
> script specifying the JIRA tickets or the commit hashes that we want to add
> to the new release candidate. Once the new RC branch is ready, create a new
> version branch from it and proceed as today.
>
> The commits to be added to the release once a release candidate has already
> been cut will usually be fixes for the release but could also be features
> if there is community consensus that a feature must be introduced to the
> release.
>
> This change will allow us to have a more granular control of what goes in
> the release once a release candidate has been cut and speed up the release
> by focusing both the release manager's and the community's efforts and
> potentially reducing the number of RCs to be created and verified.
>
> Thanks,
> Raúl
>
> [1]
> https://cwiki.apache.org/confluence/display/ARROW/Release+Management+Guide
> [2] https://github.com/apache/arrow/pull/12590#issuecomment-1116144088
>
Re: [DISC][Release] More control on Release Candidates commits
Posted by Sutou Kouhei <ko...@clear-code.com>.
Hi,
> I can see two ways to approach this issue that are both based on extending
> our PR/commit title format with machine-readable information about the
> nature of the changes in the commit. We could take inspiration from [4] or
> something similar. This would allow us to easily update the overall version
> according to the actual semver spec[5] without requiring huge amounts of
> manual effort between releases to keep that info current. Additionally, it
> would also be possible to create an accurate semver for each component
> (e.g. pyarrow to alleviate issues brought up in [3]) as our title already
> contains component information. We already have a rather strict policy on
> commit titles, so I don't think it would impose too much additional work to
> add and verify this semver marker for the benefit it would bring us.
Thanks for bringing up the interesting idea.
Regarding separated version for each component:
* We need to update our release scripts
* We update version information by
dev/release/01-prepare.sh and
dev/release/post-12-bump-versions.sh on each release
* We reconsider tag name
* We always use apache-arrow-X.Y.Z
* PyArrow's version depends on the C++ implementation
* If PyArrow doesn't have any breaking change but the
C++ implementation has a breaking change, PyArrow
should bump major version.
How about only adding "breaking change" information to each
commit as the first step? We can reconsider major release
frequency with the information.
Thanks,
--
kou
In <CA...@mail.gmail.com>
"Re: [DISC][Release] More control on Release Candidates commits" on Fri, 20 May 2022 16:36:40 +0200,
Jacob Wujciak <ja...@voltrondata.com> wrote:
> Hello Everyone,
>
> --- Summary
> +1 (nb) on formalized release workflow with "feature-freeze" on the
> release branch (master can receive PRs as usual) and early & frequent
> automated reminders.
>
> -1 (nb) on bi-monthly releases in the short term due to limitations with
> CI. Solution: self-hosted ephemeral, auto-scaling runners but this requires
> planning, development, and long-term maintenance. (see [1][2])
> +1 (nb) on faster releases as a general idea but with changes to arrows
> versioning scheme -> provide machine-readable semver info in commit titles
> and use this to create an accurate semver for each release, possibly for
> each component (see [3] for user issues with major-only).
> ---
>
> I do like the idea of earlier, more frequent (automated) reminders for the
> "feature-freeze" on the release branch, while work can continue unhindered
> on master) as proposed by Kou. As Krisztian's existing workflow was pretty
> similar to the proposal, I think it is more a case of formalizing and
> documenting the process so that it is transparent for all contributors
> (especially the "feature-freeze" aspect).
>
> The idea to increase the release frequency so people don't feel that they
> missed out on shipping their new feature is interesting and I am not
> opposed to it in principle but I do not think it is something that we can
> implement in the short term without substantial negative impact on everyone
> involved in releases and requires more discussion and planning.
>
> On the one hand, we have technical issues. We simply do not have the CI
> resources at the moment to run the number of jobs on PRs that would be
> required to ensure that master is releasable at basically any point in
> time. The Apache Github org can have 180 concurrent jobs spread over all
> repositories, in practice, this means we get ~ 5 active jobs sometimes less
> or none (depending on the time of day but not much more). This problem has
> been discussed often and as far back as the initial release of GHA [1].
> Everyone has experienced this problem only getting worse in recent times.
>
> There are two ways to address this: reduce run times by optimizing
> workflows e.g. through caching but as pointed out in [1] this is not an
> actual solution as ASF has no way to assign runner quotas to repositories,
> so capacity we free by optimizing our workflows could very well be used by
> some other project leaving us with the issue unchanged. The real solution
> is to use self-hosted runners, which due to the nature of PR CI (running
> external code) are a security issue when attached to a public repo like
> arrow. While INFRA does allow self-hosted runners their safe and effective
> utilization requires planning, development, continued maintenance, and
> financial support (In [2] Jarek Potiuk talks about the system they use in
> apache/airflow). We should definitely strive to implement (ideally:
> ephemeral, auto-scaling) self-hosted runners as soon as possible and I am
> already working on a proposal for such a system but this is just not
> something we can achieve in a month or two.
>
> On the other hand, there are also ecosystem/downstream consequences that
> should be considered when discussing a bi-monthly release. There are
> already concerns about our releases only incrementing major versions (some
> discussion in [3]) which would become more severe with an even higher
> frequency of major releases. I do understand the difficulty of making sure
> there are no breaking changes in any of the components of the mono repo and
> thus being safe with major releases (which is one of the main reasons for
> major-only releases afaik?) but with bi-monthly (or monthly once the
> process is perfected?) releases we would be nearing Chrome territory pretty
> quickly version wise.
>
> I can see two ways to approach this issue that are both based on extending
> our PR/commit title format with machine-readable information about the
> nature of the changes in the commit. We could take inspiration from [4] or
> something similar. This would allow us to easily update the overall version
> according to the actual semver spec[5] without requiring huge amounts of
> manual effort between releases to keep that info current. Additionally, it
> would also be possible to create an accurate semver for each component
> (e.g. pyarrow to alleviate issues brought up in [3]) as our title already
> contains component information. We already have a rather strict policy on
> commit titles, so I don't think it would impose too much additional work to
> add and verify this semver marker for the benefit it would bring us.
>
> Best,
> Jacob
>
> [1]:
> https://cwiki.apache.org/confluence/display/BUILDS/GitHub+Actions+status
> [2]:
> https://issues.apache.org/jira/browse/INFRA-21646?focusedCommentId=17316108&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17316108
> [3]: https://github.com/apache/arrow/issues/13185
> [4]: https://www.conventionalcommits.org/en/v1.0.0/
> [5]: https://semver.org
>
>
>
> On Wed, May 11, 2022 at 12:28 PM Krisztián Szűcs <sz...@gmail.com>
> wrote:
>
>> On Wed, May 11, 2022 at 6:01 AM Sutou Kouhei <ko...@clear-code.com> wrote:
>> >
>> > Hi,
>> >
>> > In <CA...@mail.gmail.com>
>> > "Re: [DISC][Release] More control on Release Candidates commits" on
>> Tue, 10 May 2022 13:27:09 +0200,
>> > Raul Cumplido <ra...@voltrondata.com> wrote:
>> >
>> > > I still think there is some value in standardising the "feature
>> freeze" on
>> > > new release candidates once a first release candidate has been created
>> and
>> > > only add required fixes for the follow up RCs. What I would like to
>> avoid
>> > > with that is rushing big new features at the end that might be added
>> > > between release candidates.
>> > >
>> > >>> PROBLEM 1: Rush period before the release:
>> > >>
>> > > The only proposal I can think of around this is that I will try and
>> share
>> > > the release schedule earlier. I sent an email [2] with a ~1.5/~2 weeks
>> > > notice, maybe if all of us start being more aware that a release is
>> coming
>> > > with a little more time (1 month) we can plan better.
>> >
>> > How about releasing more frequently? If we release a new
>> > version frequently, stakeholders will be able to wait for
>> > future releases instead of pushing a new feature to the next
>> > release.
>> > If we release a new version in the middle of even
>> > months, how about the following schedule?
>>
>> That looks like a plan!
>>
>> I had a similar idea to be stricter about the release dates + a
>> feature freeze period, but wasn't thinking of more frequent releases
>> which is a nice trade-off.
>>
>> > 1. Set release target date (e.g. 2022-08-10/20 for 9.0.0)
>> > 2. Notice release target date at 2022-07-01
>> > (We can automate this.)
>> We may need more notices 2 weeks and 1 week before the feature freeze.
>>
>> > 3. Create a release branch for the next version (release-9.0.0) and
>> > use the next next version (10.0.0) as the default version
>> > for dev/merge_arrow_pr.py at 2022-08-01
>> > (We can automate this.)
>> > 4. Stabilize release-9.0.0 branch in 2022-08-01/10
>> > ("feature freeze")
>> > (We should verify this branch by our nightly CI.)
>> We should move the crossbow job triggers and reports to apache/arrow
>> so we have a tighter control over the nightlies.
>>
>> > 5. Vote and release 9.0.0 in 2022-08-10/20
>> > 6. Set release target date (e.g. 2022-10-10/20 for 10.0.0)
>> > 7. ...
>> >
>> >
>> > Thanks,
>> > --
>> > kou
>>
Re: [DISC][Release] More control on Release Candidates commits
Posted by Will Jones <wi...@gmail.com>.
+1 (nb) on "feature freeze". As someone who has been a part of that rush to
get last-minute features in before the release, I think an
explicitly "feature freeze" period sounds like a good idea.
Thanks Jacob for bringing up semantic versioning. It's likely deserving of
another thread, but I will just say that if we can find a way to do better
on that (at least for PyArrow) that would benefit a lot of downstream
package users.
Also, I like the idea of adopting conventional commits. Coincidentally, not
long after Jacob posted, I contributed to a project that enforced that
exact commit style and found it quite easy to learn and adapt my commit
messages.
On Fri, May 20, 2022 at 7:37 AM Jacob Wujciak <ja...@voltrondata.com> wrote:
> Hello Everyone,
>
> --- Summary
> +1 (nb) on formalized release workflow with "feature-freeze" on the
> release branch (master can receive PRs as usual) and early & frequent
> automated reminders.
>
> -1 (nb) on bi-monthly releases in the short term due to limitations with
> CI. Solution: self-hosted ephemeral, auto-scaling runners but this requires
> planning, development, and long-term maintenance. (see [1][2])
> +1 (nb) on faster releases as a general idea but with changes to arrows
> versioning scheme -> provide machine-readable semver info in commit titles
> and use this to create an accurate semver for each release, possibly for
> each component (see [3] for user issues with major-only).
> ---
>
> I do like the idea of earlier, more frequent (automated) reminders for the
> "feature-freeze" on the release branch, while work can continue unhindered
> on master) as proposed by Kou. As Krisztian's existing workflow was pretty
> similar to the proposal, I think it is more a case of formalizing and
> documenting the process so that it is transparent for all contributors
> (especially the "feature-freeze" aspect).
>
> The idea to increase the release frequency so people don't feel that they
> missed out on shipping their new feature is interesting and I am not
> opposed to it in principle but I do not think it is something that we can
> implement in the short term without substantial negative impact on everyone
> involved in releases and requires more discussion and planning.
>
> On the one hand, we have technical issues. We simply do not have the CI
> resources at the moment to run the number of jobs on PRs that would be
> required to ensure that master is releasable at basically any point in
> time. The Apache Github org can have 180 concurrent jobs spread over all
> repositories, in practice, this means we get ~ 5 active jobs sometimes less
> or none (depending on the time of day but not much more). This problem has
> been discussed often and as far back as the initial release of GHA [1].
> Everyone has experienced this problem only getting worse in recent times.
>
> There are two ways to address this: reduce run times by optimizing
> workflows e.g. through caching but as pointed out in [1] this is not an
> actual solution as ASF has no way to assign runner quotas to repositories,
> so capacity we free by optimizing our workflows could very well be used by
> some other project leaving us with the issue unchanged. The real solution
> is to use self-hosted runners, which due to the nature of PR CI (running
> external code) are a security issue when attached to a public repo like
> arrow. While INFRA does allow self-hosted runners their safe and effective
> utilization requires planning, development, continued maintenance, and
> financial support (In [2] Jarek Potiuk talks about the system they use in
> apache/airflow). We should definitely strive to implement (ideally:
> ephemeral, auto-scaling) self-hosted runners as soon as possible and I am
> already working on a proposal for such a system but this is just not
> something we can achieve in a month or two.
>
> On the other hand, there are also ecosystem/downstream consequences that
> should be considered when discussing a bi-monthly release. There are
> already concerns about our releases only incrementing major versions (some
> discussion in [3]) which would become more severe with an even higher
> frequency of major releases. I do understand the difficulty of making sure
> there are no breaking changes in any of the components of the mono repo and
> thus being safe with major releases (which is one of the main reasons for
> major-only releases afaik?) but with bi-monthly (or monthly once the
> process is perfected?) releases we would be nearing Chrome territory pretty
> quickly version wise.
>
> I can see two ways to approach this issue that are both based on extending
> our PR/commit title format with machine-readable information about the
> nature of the changes in the commit. We could take inspiration from [4] or
> something similar. This would allow us to easily update the overall version
> according to the actual semver spec[5] without requiring huge amounts of
> manual effort between releases to keep that info current. Additionally, it
> would also be possible to create an accurate semver for each component
> (e.g. pyarrow to alleviate issues brought up in [3]) as our title already
> contains component information. We already have a rather strict policy on
> commit titles, so I don't think it would impose too much additional work to
> add and verify this semver marker for the benefit it would bring us.
>
> Best,
> Jacob
>
> [1]:
> https://cwiki.apache.org/confluence/display/BUILDS/GitHub+Actions+status
> [2]:
>
> https://issues.apache.org/jira/browse/INFRA-21646?focusedCommentId=17316108&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17316108
> [3]: https://github.com/apache/arrow/issues/13185
> [4]: https://www.conventionalcommits.org/en/v1.0.0/
> [5]: https://semver.org
>
>
>
> On Wed, May 11, 2022 at 12:28 PM Krisztián Szűcs <
> szucs.krisztian@gmail.com>
> wrote:
>
> > On Wed, May 11, 2022 at 6:01 AM Sutou Kouhei <ko...@clear-code.com> wrote:
> > >
> > > Hi,
> > >
> > > In <CAGvDy=o3OHs5LfHqVRE3R0mo1if7O908v6wD_Oy0aqV5sdT3cA@mail.gmail.com
> >
> > > "Re: [DISC][Release] More control on Release Candidates commits" on
> > Tue, 10 May 2022 13:27:09 +0200,
> > > Raul Cumplido <ra...@voltrondata.com> wrote:
> > >
> > > > I still think there is some value in standardising the "feature
> > freeze" on
> > > > new release candidates once a first release candidate has been
> created
> > and
> > > > only add required fixes for the follow up RCs. What I would like to
> > avoid
> > > > with that is rushing big new features at the end that might be added
> > > > between release candidates.
> > > >
> > > >>> PROBLEM 1: Rush period before the release:
> > > >>
> > > > The only proposal I can think of around this is that I will try and
> > share
> > > > the release schedule earlier. I sent an email [2] with a ~1.5/~2
> weeks
> > > > notice, maybe if all of us start being more aware that a release is
> > coming
> > > > with a little more time (1 month) we can plan better.
> > >
> > > How about releasing more frequently? If we release a new
> > > version frequently, stakeholders will be able to wait for
> > > future releases instead of pushing a new feature to the next
> > > release.
> > > If we release a new version in the middle of even
> > > months, how about the following schedule?
> >
> > That looks like a plan!
> >
> > I had a similar idea to be stricter about the release dates + a
> > feature freeze period, but wasn't thinking of more frequent releases
> > which is a nice trade-off.
> >
> > > 1. Set release target date (e.g. 2022-08-10/20 for 9.0.0)
> > > 2. Notice release target date at 2022-07-01
> > > (We can automate this.)
> > We may need more notices 2 weeks and 1 week before the feature freeze.
> >
> > > 3. Create a release branch for the next version (release-9.0.0) and
> > > use the next next version (10.0.0) as the default version
> > > for dev/merge_arrow_pr.py at 2022-08-01
> > > (We can automate this.)
> > > 4. Stabilize release-9.0.0 branch in 2022-08-01/10
> > > ("feature freeze")
> > > (We should verify this branch by our nightly CI.)
> > We should move the crossbow job triggers and reports to apache/arrow
> > so we have a tighter control over the nightlies.
> >
> > > 5. Vote and release 9.0.0 in 2022-08-10/20
> > > 6. Set release target date (e.g. 2022-10-10/20 for 10.0.0)
> > > 7. ...
> > >
> > >
> > > Thanks,
> > > --
> > > kou
> >
>
Re: [DISC][Release] More control on Release Candidates commits
Posted by Jacob Wujciak <ja...@voltrondata.com>.
Hello Everyone,
--- Summary
+1 (nb) on formalized release workflow with "feature-freeze" on the
release branch (master can receive PRs as usual) and early & frequent
automated reminders.
-1 (nb) on bi-monthly releases in the short term due to limitations with
CI. Solution: self-hosted ephemeral, auto-scaling runners but this requires
planning, development, and long-term maintenance. (see [1][2])
+1 (nb) on faster releases as a general idea but with changes to arrows
versioning scheme -> provide machine-readable semver info in commit titles
and use this to create an accurate semver for each release, possibly for
each component (see [3] for user issues with major-only).
---
I do like the idea of earlier, more frequent (automated) reminders for the
"feature-freeze" on the release branch, while work can continue unhindered
on master) as proposed by Kou. As Krisztian's existing workflow was pretty
similar to the proposal, I think it is more a case of formalizing and
documenting the process so that it is transparent for all contributors
(especially the "feature-freeze" aspect).
The idea to increase the release frequency so people don't feel that they
missed out on shipping their new feature is interesting and I am not
opposed to it in principle but I do not think it is something that we can
implement in the short term without substantial negative impact on everyone
involved in releases and requires more discussion and planning.
On the one hand, we have technical issues. We simply do not have the CI
resources at the moment to run the number of jobs on PRs that would be
required to ensure that master is releasable at basically any point in
time. The Apache Github org can have 180 concurrent jobs spread over all
repositories, in practice, this means we get ~ 5 active jobs sometimes less
or none (depending on the time of day but not much more). This problem has
been discussed often and as far back as the initial release of GHA [1].
Everyone has experienced this problem only getting worse in recent times.
There are two ways to address this: reduce run times by optimizing
workflows e.g. through caching but as pointed out in [1] this is not an
actual solution as ASF has no way to assign runner quotas to repositories,
so capacity we free by optimizing our workflows could very well be used by
some other project leaving us with the issue unchanged. The real solution
is to use self-hosted runners, which due to the nature of PR CI (running
external code) are a security issue when attached to a public repo like
arrow. While INFRA does allow self-hosted runners their safe and effective
utilization requires planning, development, continued maintenance, and
financial support (In [2] Jarek Potiuk talks about the system they use in
apache/airflow). We should definitely strive to implement (ideally:
ephemeral, auto-scaling) self-hosted runners as soon as possible and I am
already working on a proposal for such a system but this is just not
something we can achieve in a month or two.
On the other hand, there are also ecosystem/downstream consequences that
should be considered when discussing a bi-monthly release. There are
already concerns about our releases only incrementing major versions (some
discussion in [3]) which would become more severe with an even higher
frequency of major releases. I do understand the difficulty of making sure
there are no breaking changes in any of the components of the mono repo and
thus being safe with major releases (which is one of the main reasons for
major-only releases afaik?) but with bi-monthly (or monthly once the
process is perfected?) releases we would be nearing Chrome territory pretty
quickly version wise.
I can see two ways to approach this issue that are both based on extending
our PR/commit title format with machine-readable information about the
nature of the changes in the commit. We could take inspiration from [4] or
something similar. This would allow us to easily update the overall version
according to the actual semver spec[5] without requiring huge amounts of
manual effort between releases to keep that info current. Additionally, it
would also be possible to create an accurate semver for each component
(e.g. pyarrow to alleviate issues brought up in [3]) as our title already
contains component information. We already have a rather strict policy on
commit titles, so I don't think it would impose too much additional work to
add and verify this semver marker for the benefit it would bring us.
Best,
Jacob
[1]:
https://cwiki.apache.org/confluence/display/BUILDS/GitHub+Actions+status
[2]:
https://issues.apache.org/jira/browse/INFRA-21646?focusedCommentId=17316108&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17316108
[3]: https://github.com/apache/arrow/issues/13185
[4]: https://www.conventionalcommits.org/en/v1.0.0/
[5]: https://semver.org
On Wed, May 11, 2022 at 12:28 PM Krisztián Szűcs <sz...@gmail.com>
wrote:
> On Wed, May 11, 2022 at 6:01 AM Sutou Kouhei <ko...@clear-code.com> wrote:
> >
> > Hi,
> >
> > In <CA...@mail.gmail.com>
> > "Re: [DISC][Release] More control on Release Candidates commits" on
> Tue, 10 May 2022 13:27:09 +0200,
> > Raul Cumplido <ra...@voltrondata.com> wrote:
> >
> > > I still think there is some value in standardising the "feature
> freeze" on
> > > new release candidates once a first release candidate has been created
> and
> > > only add required fixes for the follow up RCs. What I would like to
> avoid
> > > with that is rushing big new features at the end that might be added
> > > between release candidates.
> > >
> > >>> PROBLEM 1: Rush period before the release:
> > >>
> > > The only proposal I can think of around this is that I will try and
> share
> > > the release schedule earlier. I sent an email [2] with a ~1.5/~2 weeks
> > > notice, maybe if all of us start being more aware that a release is
> coming
> > > with a little more time (1 month) we can plan better.
> >
> > How about releasing more frequently? If we release a new
> > version frequently, stakeholders will be able to wait for
> > future releases instead of pushing a new feature to the next
> > release.
> > If we release a new version in the middle of even
> > months, how about the following schedule?
>
> That looks like a plan!
>
> I had a similar idea to be stricter about the release dates + a
> feature freeze period, but wasn't thinking of more frequent releases
> which is a nice trade-off.
>
> > 1. Set release target date (e.g. 2022-08-10/20 for 9.0.0)
> > 2. Notice release target date at 2022-07-01
> > (We can automate this.)
> We may need more notices 2 weeks and 1 week before the feature freeze.
>
> > 3. Create a release branch for the next version (release-9.0.0) and
> > use the next next version (10.0.0) as the default version
> > for dev/merge_arrow_pr.py at 2022-08-01
> > (We can automate this.)
> > 4. Stabilize release-9.0.0 branch in 2022-08-01/10
> > ("feature freeze")
> > (We should verify this branch by our nightly CI.)
> We should move the crossbow job triggers and reports to apache/arrow
> so we have a tighter control over the nightlies.
>
> > 5. Vote and release 9.0.0 in 2022-08-10/20
> > 6. Set release target date (e.g. 2022-10-10/20 for 10.0.0)
> > 7. ...
> >
> >
> > Thanks,
> > --
> > kou
>
Re: [DISC][Release] More control on Release Candidates commits
Posted by Krisztián Szűcs <sz...@gmail.com>.
On Wed, May 11, 2022 at 6:01 AM Sutou Kouhei <ko...@clear-code.com> wrote:
>
> Hi,
>
> In <CA...@mail.gmail.com>
> "Re: [DISC][Release] More control on Release Candidates commits" on Tue, 10 May 2022 13:27:09 +0200,
> Raul Cumplido <ra...@voltrondata.com> wrote:
>
> > I still think there is some value in standardising the "feature freeze" on
> > new release candidates once a first release candidate has been created and
> > only add required fixes for the follow up RCs. What I would like to avoid
> > with that is rushing big new features at the end that might be added
> > between release candidates.
> >
> >>> PROBLEM 1: Rush period before the release:
> >>
> > The only proposal I can think of around this is that I will try and share
> > the release schedule earlier. I sent an email [2] with a ~1.5/~2 weeks
> > notice, maybe if all of us start being more aware that a release is coming
> > with a little more time (1 month) we can plan better.
>
> How about releasing more frequently? If we release a new
> version frequently, stakeholders will be able to wait for
> future releases instead of pushing a new feature to the next
> release.
> If we release a new version in the middle of even
> months, how about the following schedule?
That looks like a plan!
I had a similar idea to be stricter about the release dates + a
feature freeze period, but wasn't thinking of more frequent releases
which is a nice trade-off.
> 1. Set release target date (e.g. 2022-08-10/20 for 9.0.0)
> 2. Notice release target date at 2022-07-01
> (We can automate this.)
We may need more notices 2 weeks and 1 week before the feature freeze.
> 3. Create a release branch for the next version (release-9.0.0) and
> use the next next version (10.0.0) as the default version
> for dev/merge_arrow_pr.py at 2022-08-01
> (We can automate this.)
> 4. Stabilize release-9.0.0 branch in 2022-08-01/10
> ("feature freeze")
> (We should verify this branch by our nightly CI.)
We should move the crossbow job triggers and reports to apache/arrow
so we have a tighter control over the nightlies.
> 5. Vote and release 9.0.0 in 2022-08-10/20
> 6. Set release target date (e.g. 2022-10-10/20 for 10.0.0)
> 7. ...
>
>
> Thanks,
> --
> kou
Re: [DISC][Release] More control on Release Candidates commits
Posted by Sutou Kouhei <ko...@clear-code.com>.
Hi,
In <CA...@mail.gmail.com>
"Re: [DISC][Release] More control on Release Candidates commits" on Tue, 10 May 2022 13:27:09 +0200,
Raul Cumplido <ra...@voltrondata.com> wrote:
> I still think there is some value in standardising the "feature freeze" on
> new release candidates once a first release candidate has been created and
> only add required fixes for the follow up RCs. What I would like to avoid
> with that is rushing big new features at the end that might be added
> between release candidates.
>
>>> PROBLEM 1: Rush period before the release:
>>
> The only proposal I can think of around this is that I will try and share
> the release schedule earlier. I sent an email [2] with a ~1.5/~2 weeks
> notice, maybe if all of us start being more aware that a release is coming
> with a little more time (1 month) we can plan better.
How about releasing more frequently? If we release a new
version frequently, stakeholders will be able to wait for
future releases instead of pushing a new feature to the next
release. If we release a new version in the middle of even
months, how about the following schedule?
1. Set release target date (e.g. 2022-08-10/20 for 9.0.0)
2. Notice release target date at 2022-07-01
(We can automate this.)
3. Create a release branch for the next version (release-9.0.0) and
use the next next version (10.0.0) as the default version
for dev/merge_arrow_pr.py at 2022-08-01
(We can automate this.)
4. Stabilize release-9.0.0 branch in 2022-08-01/10
("feature freeze")
(We should verify this branch by our nightly CI.)
5. Vote and release 9.0.0 in 2022-08-10/20
6. Set release target date (e.g. 2022-10-10/20 for 10.0.0)
7. ...
Thanks,
--
kou
Re: [DISC][Release] More control on Release Candidates commits
Posted by Antoine Pitrou <an...@python.org>.
Le 10/05/2022 à 13:27, Raul Cumplido a écrit :
>
> I still think there is some value in standardising the "feature freeze" on
> new release candidates once a first release candidate has been created and
> only add required fixes for the follow up RCs. What I would like to avoid
> with that is rushing big new features at the end that might be added
> between release candidates.
+1 on this. We really should discourage people from including features
between RCs, even if they seem extremely attractive. My past experience
is that granting exceptions often ends up costly both in time and quality.
Regards
Antoine.
Re: [DISC][Release] More control on Release Candidates commits
Posted by Raul Cumplido <ra...@voltrondata.com>.
Thanks for the feedback, Krisztián! Lots of good insights on the current
release process.
I can see that you were already taking actions towards the process I was
describing. I will write some notes to update the current process to
reflect that on the Release documentation [1] and will share.
I still think there is some value in standardising the "feature freeze" on
new release candidates once a first release candidate has been created and
only add required fixes for the follow up RCs. What I would like to avoid
with that is rushing big new features at the end that might be added
between release candidates.
>> PROBLEM 1: Rush period before the release:
>
> The only proposal I can think of around this is that I will try and share
the release schedule earlier. I sent an email [2] with a ~1.5/~2 weeks
notice, maybe if all of us start being more aware that a release is coming
with a little more time (1 month) we can plan better.
> >> PROBLEM 3: Lack of interest in nightly builds despite their importance
>
I have added more visibility to them by publishing them on Zulip [3] and I
am planning to work towards improving them further.
[1]
https://cwiki.apache.org/confluence/display/ARROW/Release+Management+Guide
[2] https://lists.apache.org/thread/zk8hhynvy0bqvqpxk0868n5g0nmzbzbn
[3]
https://ursalabs.zulipchat.com/#narrow/stream/181017-nightlies/topic/report
Re: [DISC][Release] More control on Release Candidates commits
Posted by Antoine Pitrou <an...@python.org>.
Well, in any case, the release manager should make the final call, so a
label would mostly be a sophisticated way of pinging them.
Le 09/05/2022 à 20:45, Weston Pace a écrit :
> How should we indicate whether a JIRA is a bugfix, which should be
> included in the next RC, or something else that shouldn't be included
> in the next RC? Right now I think this is a somewhat manual process
> with us dropping a note in the Github, or Zulip, or the person
> packaging the RC using their best judgement. Is that process working
> ok? Or would it be easier to have a label or version tag of some
> kind?
>
> On Mon, May 9, 2022 at 4:59 AM Krisztián Szűcs
> <sz...@gmail.com> wrote:
>>
>> Hi,
>>
>> Thanks Raúl for bringing this up since it's an important topic!
>> I'd like to provide more context for your proposal and share my
>> particular problems with the release process.
>>
>> On Mon, May 9, 2022 at 2:33 PM Raul Cumplido <ra...@voltrondata.com> wrote:
>>>
>>> Hi,
>>>
>>> I would like to propose a change in our release process.
>>>
>>> The rationale for the change is to avoid introducing new issues once a
>>> Release Candidate has already been cut by only merging specific commits to
>>> new release candidates.
>>>
>>> Currently once a new Release Candidate is required we drop the previous
>>> version branch and create a new Release Candidate from the master branch on
>>> the repository [1].
>> Actually dropping the previous "release-<version>" branch is not a
>> requirement, but it's indeed not clearly documented in the release
>> guidelines.
>>
>>> This has the problem that we might introduce new bugs
>>> to the Release creating the need of cutting further release candidates.
>> We introduced the release branches for this exact scenario, so we can
>> create releases independently from the master branch.
>>
>>> As an example, for the release 7.0.0, 10 release candidates were required
>> The reason for the notorious 7.0.0-RC10 is different, more on that later.
>>
>>> and for the release 8.0.0 there was the need to remove a specific commit that
>>> introduced some new issues [2]. For the release 8.0.0 we were able to find
>>> it early but it could have potentially been introduced and created the need
>>> for further RCs.
>>>
>>> I would like to propose the following workflow.
>>> When creating the initial RC, create both an rc1 branch and the version
>>> branch from master.
>>> release-x.0.0 and release-x.0.0.rc1
>>>
>>> If a new RC is required, drop the release-x.0.0 (as we do today) and create
>>> a new RC branch from the previous RC branch (instead of master), then
>>> cherry pick only the specific commits that have been identified to be part
>>> of the new release candidate. We can automate the cherrypick process via a
>>> script specifying the JIRA tickets or the commit hashes that we want to add
>>> to the new release candidate. Once the new RC branch is ready, create a new
>>> version branch from it and proceed as today.
>> This is why I manually cherry-picked 4 commits from the master branch
>> to the new release branch [1] excluding that specific patch.
>> Note, that there was a single blocker [2], but I still included 3
>> additional patches: 2 low-risk bug fixes and a patch for the
>> verification.
>>
>>> The commits to be added to the release once a release candidate has already
>>> been cut will usually be fixes for the release but could also be features
>>> if there is community consensus that a feature must be introduced to the
>>> release.
>> I'd have also included both the python UDF [3] and GCS [4] patches
>> since they are really valuable features.
>> In the first case we noticed the broken packaging builds from the
>> nightly report, this is why I had to cherry-pick commits from the
>> master rather than cutting RC3 directly from the master branch (there
>> is no other difference).
>> In the second case the PR simply didn't make it due to the same reason
>> [5] which we managed to catch before merging the patch.
>>
>>> This change will allow us to have a more granular control of what goes in
>>> the release once a release candidate has been cut and speed up the release
>> Since your proposal is already implemented, the actionable item I see
>> here is to properly document it in the release management guidelines.
>>
>>> by focusing both the release manager's and the community's efforts and
>>> potentially reducing the number of RCs to be created and verified.
>> Regarding the notorious 7.0.0-RC10 release candidate: I developed a
>> habit to execute the source verification tasks before calling a vote
>> while waiting for the packaging builds to finish. If there is an issue
>> it doesn't reach the VOTE phase. Just took a look and the 6th release
>> candidate (7.0.0-RC5) was the first one I managed to send out a VOTE
>> email for. Out of the 11 release candidates I created for the 7.0.0
>> release only 4 made it until the voting.
>>
>> Before that release the number of RC verification crossbow tasks kept
>> growing but without the ability to run them on a nightly basis.
>> Meaning that we were unable to tell whether the verification tasks
>> will pass for a certain commit and just noticing issues after creating
>> a release candidate.
>> Right after the 7.0.0 release we refactored [6] the source
>> verification scripts and crossbow tasks to support verifying specific
>> git commits, local checkouts and actual release candidates. Since then
>> we have nightly verification builds so we get notified about the
>> failing builds and haven't even tried to create the first release
>> candidate until we had failing verification tasks. This was the single
>> reason why we didn't have 10+ release candidates this time.
>>
>>
>> After spending countless sleepless nights with arrow releases I'd like
>> to raise awareness of three other problems bothering me:
>>
>> PROBLEM 1: Rush period before the release:
>> One or two weeks before the release we start to incrementally postpone
>> the issues which are unlikely to make it into the release but there
>> are features we would still like to squeeze in. There are too many
>> simultaneously moving parts right before the release, possibly
>> introducing new issues. Since we release many implementations at once
>> and there are multiple stakeholders focusing on different features
>> it's generally hard to "reach consensus" about what to exclude and
>> what to wait for. We're trying our best to include as much value to
>> each release as we can while trying to avoid significant delays in
>> delivery date.
>>
>> PROBLEM 2: Decoupled packaging and verification builds
>> Due to the on-demand nature of the crossbow tasks we often forget to
>> trigger crossbow builds before merging a PR resulting in nightly
>> failures which we need to fix in follow-up PRs. Ideally if we were
>> able to run all of our builds on all of the PRs before merging we
>> could keep the master branch in an always-relasable state.
>> This is a tradeoff we made to spare CI resources for the apache/arrow
>> repository but soon enough we will reach the capacity limits of
>> crossbow as well (for example I had to manually stop-and-restart macOS
>> crossbow builds during the release process to avoid waiting 12 more
>> hours).
>>
>> PROBLEM 3: Lack of interest in nightly builds despite their importance
>> We usually let nightly builds to continuously fail for days or even
>> weeks hiding more and more issues over time. This adds up before the
>> release making the rush period even worse. I'm not sure what's the
>> exact reason, probably the mixture of just a few subscribers to the
>> builds@ mailing list and the poor readability of nightly reports
>> (which keeps improving thanks to Raúl).
>>
>> Thanks, Krisztian
>>
>> [1]: https://github.com/apache/arrow/commits/release-8.0.0
>> [2]: https://github.com/apache/arrow/commit/0d30a05212b1448f53233f2ab325924311d76e54
>> [3]: https://github.com/apache/arrow/pull/12590
>> [4]: https://github.com/apache/arrow/pull/12763
>> [5]: https://github.com/apache/arrow/pull/12763#issuecomment-1109022291
>> [6]: https://github.com/apache/arrow/pull/12320
>>>
>>> Thanks,
>>> Raúl
>>>
>>> [1]
>>> https://cwiki.apache.org/confluence/display/ARROW/Release+Management+Guide
>>> [2] https://github.com/apache/arrow/pull/12590#issuecomment-1116144088
Re: [DISC][Release] More control on Release Candidates commits
Posted by Weston Pace <we...@gmail.com>.
How should we indicate whether a JIRA is a bugfix, which should be
included in the next RC, or something else that shouldn't be included
in the next RC? Right now I think this is a somewhat manual process
with us dropping a note in the Github, or Zulip, or the person
packaging the RC using their best judgement. Is that process working
ok? Or would it be easier to have a label or version tag of some
kind?
On Mon, May 9, 2022 at 4:59 AM Krisztián Szűcs
<sz...@gmail.com> wrote:
>
> Hi,
>
> Thanks Raúl for bringing this up since it's an important topic!
> I'd like to provide more context for your proposal and share my
> particular problems with the release process.
>
> On Mon, May 9, 2022 at 2:33 PM Raul Cumplido <ra...@voltrondata.com> wrote:
> >
> > Hi,
> >
> > I would like to propose a change in our release process.
> >
> > The rationale for the change is to avoid introducing new issues once a
> > Release Candidate has already been cut by only merging specific commits to
> > new release candidates.
> >
> > Currently once a new Release Candidate is required we drop the previous
> > version branch and create a new Release Candidate from the master branch on
> > the repository [1].
> Actually dropping the previous "release-<version>" branch is not a
> requirement, but it's indeed not clearly documented in the release
> guidelines.
>
> > This has the problem that we might introduce new bugs
> > to the Release creating the need of cutting further release candidates.
> We introduced the release branches for this exact scenario, so we can
> create releases independently from the master branch.
>
> > As an example, for the release 7.0.0, 10 release candidates were required
> The reason for the notorious 7.0.0-RC10 is different, more on that later.
>
> > and for the release 8.0.0 there was the need to remove a specific commit that
> > introduced some new issues [2]. For the release 8.0.0 we were able to find
> > it early but it could have potentially been introduced and created the need
> > for further RCs.
> >
> > I would like to propose the following workflow.
> > When creating the initial RC, create both an rc1 branch and the version
> > branch from master.
> > release-x.0.0 and release-x.0.0.rc1
> >
> > If a new RC is required, drop the release-x.0.0 (as we do today) and create
> > a new RC branch from the previous RC branch (instead of master), then
> > cherry pick only the specific commits that have been identified to be part
> > of the new release candidate. We can automate the cherrypick process via a
> > script specifying the JIRA tickets or the commit hashes that we want to add
> > to the new release candidate. Once the new RC branch is ready, create a new
> > version branch from it and proceed as today.
> This is why I manually cherry-picked 4 commits from the master branch
> to the new release branch [1] excluding that specific patch.
> Note, that there was a single blocker [2], but I still included 3
> additional patches: 2 low-risk bug fixes and a patch for the
> verification.
>
> > The commits to be added to the release once a release candidate has already
> > been cut will usually be fixes for the release but could also be features
> > if there is community consensus that a feature must be introduced to the
> > release.
> I'd have also included both the python UDF [3] and GCS [4] patches
> since they are really valuable features.
> In the first case we noticed the broken packaging builds from the
> nightly report, this is why I had to cherry-pick commits from the
> master rather than cutting RC3 directly from the master branch (there
> is no other difference).
> In the second case the PR simply didn't make it due to the same reason
> [5] which we managed to catch before merging the patch.
>
> > This change will allow us to have a more granular control of what goes in
> > the release once a release candidate has been cut and speed up the release
> Since your proposal is already implemented, the actionable item I see
> here is to properly document it in the release management guidelines.
>
> > by focusing both the release manager's and the community's efforts and
> > potentially reducing the number of RCs to be created and verified.
> Regarding the notorious 7.0.0-RC10 release candidate: I developed a
> habit to execute the source verification tasks before calling a vote
> while waiting for the packaging builds to finish. If there is an issue
> it doesn't reach the VOTE phase. Just took a look and the 6th release
> candidate (7.0.0-RC5) was the first one I managed to send out a VOTE
> email for. Out of the 11 release candidates I created for the 7.0.0
> release only 4 made it until the voting.
>
> Before that release the number of RC verification crossbow tasks kept
> growing but without the ability to run them on a nightly basis.
> Meaning that we were unable to tell whether the verification tasks
> will pass for a certain commit and just noticing issues after creating
> a release candidate.
> Right after the 7.0.0 release we refactored [6] the source
> verification scripts and crossbow tasks to support verifying specific
> git commits, local checkouts and actual release candidates. Since then
> we have nightly verification builds so we get notified about the
> failing builds and haven't even tried to create the first release
> candidate until we had failing verification tasks. This was the single
> reason why we didn't have 10+ release candidates this time.
>
>
> After spending countless sleepless nights with arrow releases I'd like
> to raise awareness of three other problems bothering me:
>
> PROBLEM 1: Rush period before the release:
> One or two weeks before the release we start to incrementally postpone
> the issues which are unlikely to make it into the release but there
> are features we would still like to squeeze in. There are too many
> simultaneously moving parts right before the release, possibly
> introducing new issues. Since we release many implementations at once
> and there are multiple stakeholders focusing on different features
> it's generally hard to "reach consensus" about what to exclude and
> what to wait for. We're trying our best to include as much value to
> each release as we can while trying to avoid significant delays in
> delivery date.
>
> PROBLEM 2: Decoupled packaging and verification builds
> Due to the on-demand nature of the crossbow tasks we often forget to
> trigger crossbow builds before merging a PR resulting in nightly
> failures which we need to fix in follow-up PRs. Ideally if we were
> able to run all of our builds on all of the PRs before merging we
> could keep the master branch in an always-relasable state.
> This is a tradeoff we made to spare CI resources for the apache/arrow
> repository but soon enough we will reach the capacity limits of
> crossbow as well (for example I had to manually stop-and-restart macOS
> crossbow builds during the release process to avoid waiting 12 more
> hours).
>
> PROBLEM 3: Lack of interest in nightly builds despite their importance
> We usually let nightly builds to continuously fail for days or even
> weeks hiding more and more issues over time. This adds up before the
> release making the rush period even worse. I'm not sure what's the
> exact reason, probably the mixture of just a few subscribers to the
> builds@ mailing list and the poor readability of nightly reports
> (which keeps improving thanks to Raúl).
>
> Thanks, Krisztian
>
> [1]: https://github.com/apache/arrow/commits/release-8.0.0
> [2]: https://github.com/apache/arrow/commit/0d30a05212b1448f53233f2ab325924311d76e54
> [3]: https://github.com/apache/arrow/pull/12590
> [4]: https://github.com/apache/arrow/pull/12763
> [5]: https://github.com/apache/arrow/pull/12763#issuecomment-1109022291
> [6]: https://github.com/apache/arrow/pull/12320
> >
> > Thanks,
> > Raúl
> >
> > [1]
> > https://cwiki.apache.org/confluence/display/ARROW/Release+Management+Guide
> > [2] https://github.com/apache/arrow/pull/12590#issuecomment-1116144088
Re: [DISC][Release] More control on Release Candidates commits
Posted by Krisztián Szűcs <sz...@gmail.com>.
Hi,
Thanks Raúl for bringing this up since it's an important topic!
I'd like to provide more context for your proposal and share my
particular problems with the release process.
On Mon, May 9, 2022 at 2:33 PM Raul Cumplido <ra...@voltrondata.com> wrote:
>
> Hi,
>
> I would like to propose a change in our release process.
>
> The rationale for the change is to avoid introducing new issues once a
> Release Candidate has already been cut by only merging specific commits to
> new release candidates.
>
> Currently once a new Release Candidate is required we drop the previous
> version branch and create a new Release Candidate from the master branch on
> the repository [1].
Actually dropping the previous "release-<version>" branch is not a
requirement, but it's indeed not clearly documented in the release
guidelines.
> This has the problem that we might introduce new bugs
> to the Release creating the need of cutting further release candidates.
We introduced the release branches for this exact scenario, so we can
create releases independently from the master branch.
> As an example, for the release 7.0.0, 10 release candidates were required
The reason for the notorious 7.0.0-RC10 is different, more on that later.
> and for the release 8.0.0 there was the need to remove a specific commit that
> introduced some new issues [2]. For the release 8.0.0 we were able to find
> it early but it could have potentially been introduced and created the need
> for further RCs.
>
> I would like to propose the following workflow.
> When creating the initial RC, create both an rc1 branch and the version
> branch from master.
> release-x.0.0 and release-x.0.0.rc1
>
> If a new RC is required, drop the release-x.0.0 (as we do today) and create
> a new RC branch from the previous RC branch (instead of master), then
> cherry pick only the specific commits that have been identified to be part
> of the new release candidate. We can automate the cherrypick process via a
> script specifying the JIRA tickets or the commit hashes that we want to add
> to the new release candidate. Once the new RC branch is ready, create a new
> version branch from it and proceed as today.
This is why I manually cherry-picked 4 commits from the master branch
to the new release branch [1] excluding that specific patch.
Note, that there was a single blocker [2], but I still included 3
additional patches: 2 low-risk bug fixes and a patch for the
verification.
> The commits to be added to the release once a release candidate has already
> been cut will usually be fixes for the release but could also be features
> if there is community consensus that a feature must be introduced to the
> release.
I'd have also included both the python UDF [3] and GCS [4] patches
since they are really valuable features.
In the first case we noticed the broken packaging builds from the
nightly report, this is why I had to cherry-pick commits from the
master rather than cutting RC3 directly from the master branch (there
is no other difference).
In the second case the PR simply didn't make it due to the same reason
[5] which we managed to catch before merging the patch.
> This change will allow us to have a more granular control of what goes in
> the release once a release candidate has been cut and speed up the release
Since your proposal is already implemented, the actionable item I see
here is to properly document it in the release management guidelines.
> by focusing both the release manager's and the community's efforts and
> potentially reducing the number of RCs to be created and verified.
Regarding the notorious 7.0.0-RC10 release candidate: I developed a
habit to execute the source verification tasks before calling a vote
while waiting for the packaging builds to finish. If there is an issue
it doesn't reach the VOTE phase. Just took a look and the 6th release
candidate (7.0.0-RC5) was the first one I managed to send out a VOTE
email for. Out of the 11 release candidates I created for the 7.0.0
release only 4 made it until the voting.
Before that release the number of RC verification crossbow tasks kept
growing but without the ability to run them on a nightly basis.
Meaning that we were unable to tell whether the verification tasks
will pass for a certain commit and just noticing issues after creating
a release candidate.
Right after the 7.0.0 release we refactored [6] the source
verification scripts and crossbow tasks to support verifying specific
git commits, local checkouts and actual release candidates. Since then
we have nightly verification builds so we get notified about the
failing builds and haven't even tried to create the first release
candidate until we had failing verification tasks. This was the single
reason why we didn't have 10+ release candidates this time.
After spending countless sleepless nights with arrow releases I'd like
to raise awareness of three other problems bothering me:
PROBLEM 1: Rush period before the release:
One or two weeks before the release we start to incrementally postpone
the issues which are unlikely to make it into the release but there
are features we would still like to squeeze in. There are too many
simultaneously moving parts right before the release, possibly
introducing new issues. Since we release many implementations at once
and there are multiple stakeholders focusing on different features
it's generally hard to "reach consensus" about what to exclude and
what to wait for. We're trying our best to include as much value to
each release as we can while trying to avoid significant delays in
delivery date.
PROBLEM 2: Decoupled packaging and verification builds
Due to the on-demand nature of the crossbow tasks we often forget to
trigger crossbow builds before merging a PR resulting in nightly
failures which we need to fix in follow-up PRs. Ideally if we were
able to run all of our builds on all of the PRs before merging we
could keep the master branch in an always-relasable state.
This is a tradeoff we made to spare CI resources for the apache/arrow
repository but soon enough we will reach the capacity limits of
crossbow as well (for example I had to manually stop-and-restart macOS
crossbow builds during the release process to avoid waiting 12 more
hours).
PROBLEM 3: Lack of interest in nightly builds despite their importance
We usually let nightly builds to continuously fail for days or even
weeks hiding more and more issues over time. This adds up before the
release making the rush period even worse. I'm not sure what's the
exact reason, probably the mixture of just a few subscribers to the
builds@ mailing list and the poor readability of nightly reports
(which keeps improving thanks to Raúl).
Thanks, Krisztian
[1]: https://github.com/apache/arrow/commits/release-8.0.0
[2]: https://github.com/apache/arrow/commit/0d30a05212b1448f53233f2ab325924311d76e54
[3]: https://github.com/apache/arrow/pull/12590
[4]: https://github.com/apache/arrow/pull/12763
[5]: https://github.com/apache/arrow/pull/12763#issuecomment-1109022291
[6]: https://github.com/apache/arrow/pull/12320
>
> Thanks,
> Raúl
>
> [1]
> https://cwiki.apache.org/confluence/display/ARROW/Release+Management+Guide
> [2] https://github.com/apache/arrow/pull/12590#issuecomment-1116144088