Release verification process

[Neal Richardson <ne...@gmail.com>]

Hi all,
[Splitting off from the [VOTE] discussion thread]

I wonder if there are things we can do to make for a smoother release next
time. This was the first release I participated in with the Arrow project,
and there were some aspects that were surprising to me. It seemed odd that
"release verification" involved manually running a bunch of tests that
weren't expected to pass, and then reporting that they didn't pass, and
then forging ahead anyway.

Perhaps we could, at the beginning of the release process and before making
the first RC, make clear what is expected in order to verify it, and also
state what is not required or expected. And we shouldn't spend time running
tests that we don't care whether they fail or not. That way, we can have
the discussion of what a successful release looks like before there's an RC
on the table and a sense of urgency to ship it, and then it will be clear
to everyone that the standards have been met when we run the tests.

Regardless of how we might want to improve the process for next time, I
appreciate all of the hard work that Kou (especially) and others put in to
making the release happen. Releasing a project of this size clearly takes a
lot of dedication and effort.

Neal


On Thu, Jul 4, 2019 at 7:05 AM Wes McKinney <we...@gmail.com> wrote:

> hi Jacques,
>
> I agree that we should allow ample time for discussions around issues
> raised by people validating the release.
>
> That being said, with Ryan's issue, he is using a feature
> (cross-language auth in Flight) that isn't being tested. The Flight
> integration tests do not use authentication AFAIK so I'm not surprised
> to hear that there may be an issue with it. It would be as if there
> were a cross-compatibility issues between Python and Rust. Again,
> would not surprise me because Rust is not included in the integration
> tests. Even with a reproduction of such an issue I don't think it
> should block the release in the future.
>
> I'd be glad to help make a point release in the near future since we
> uncovered quite a few small issues during this RC, and I think there's
> more value in getting the software into the hands of users than
> waiting multiple weeks to address them all.
>
> - Wes
>
>
> On Thu, Jul 4, 2019 at 8:50 AM Jacques Nadeau <ja...@apache.org> wrote:
> >
> > A release vote should last until we arrive at consensus. When an issue is
> > potentially identified, those that have voted should be given ample time
> to
> > change their vote and others that may have been lazy consenters should be
> > given time to chime in. There is no maximum amount of time a vote can be
> > open. Allowing at least 24 hours after an objection is raised is a pretty
> > minimum expectation unless the objector removes their objection.
> >
> > Note that Apache is more focused on consensus than timing (as opposed to
> > virtually other other organizations in the world).
> >
> > In this case I was waiting to see more information from the objector
> before
> > casting my vote. I see a -1 vote as community chilling and prefer for
> > consensus to be reached through discussions rather than contrary voting.
> >
> >
> >
> > On Thu, Jul 4, 2019, 6:35 AM Antoine Pitrou <an...@python.org> wrote:
> >
> > >
> > > That's an open question.  How long should a release vote last?
> > > Apparently this one lasted 3 days.  The 0.14.0 vote (after RC4) lasted
> 3
> > > days before a decision was made too.
> > >
> > > Regards
> > >
> > > Antoine.
> > >
> > >
> > > Le 04/07/2019 à 15:28, Jacques Nadeau a écrit :
> > > > There are two different questions here.
> > > >
> > > > 1) should this issue block the release.
> > > > 2) should a discussion with ample time for all parties to come to
> > > consensus
> > > > have occurred before closing the vote.
> > > >
> > > > The first question is a subjective question. The answer to the second
> > > > question should have been an unequivocal yes.
> > > >
> > > > I'll send a separate mail on point one but the problem I see here
> > > > fundamentally is point two.
> > > >
> > > > On Thu, Jul 4, 2019, 6:01 AM Antoine Pitrou <an...@python.org>
> wrote:
> > > >
> > > >>
> > > >> I agree with Kou here.  It's not a problem with the release per se,
> it's
> > > >> a problem with the Flight code in git master.  There is no
> regression
> > > >> AFAICT, and we have not promised that Flight was stable and
> > > >> production-ready yet.
> > > >>
> > > >> If we're ok releasing with bugs such as this (Java unable to read
> back
> > > >> union arrays?), then I think a Flight bug shouldn't hold a release.
> > > >> https://issues.apache.org/jira/browse/ARROW-5231
> > > >>
> > > >> Regards
> > > >>
> > > >> Antoine.
> > > >>
> > > >>
> > > >> Le 04/07/2019 à 14:46, Jacques Nadeau a écrit :
> > > >>> Im disappointed in this response. When someone finds an issue with
> a
> > > >>> release it should be triaged/validated rather than rushed past
> rather
> > > >>> quickly and closing the vote before others can chime in on the
> issue.
> > > >>>
> > > >>> I'm now left voting -1 on a closed vote. In the future, let's have
> a
> > > >>> discussion about issues like this. Your response and the close of
> the
> > > >> vote
> > > >>> were unilateral and all happened while some of us were sleeping.
> > > >>>
> > > >>> On Wed, Jul 3, 2019, 10:47 PM Sutou Kouhei <ko...@clear-code.com>
> wrote:
> > > >>>
> > > >>>> Hi,
> > > >>>>
> > > >>>> Flight isn't stable yet. So Flight problem isn't a blocker
> > > >>>> of this release. Could you open a JIRA issue for your
> > > >>>> problem? We'll be able to fix it until 1.0.0.
> > > >>>>
> > > >>>>
> > > >>>> Thanks,
> > > >>>> --
> > > >>>> kou
> > > >>>>
> > > >>>> In <
> > > CAKg4KDx8_Rr8xQOBaspW+U0LYPX21tPcWx6S4_RDdMk4VpDwcg@mail.gmail.com>
> > > >>>>   "Re: [VOTE] Release Apache Arrow 0.14.0 - RC0" on Wed, 3 Jul
> 2019
> > > >>>> 17:18:15 -0700,
> > > >>>>   Ryan Murray <ry...@dremio.com> wrote:
> > > >>>>
> > > >>>>> Hi All,
> > > >>>>>
> > > >>>>> I have noticed that a python Flight client can't authenticate to
> a
> > > Java
> > > >>>>> Flight server while doing some testing today. I am trying to
> confirm
> > > if
> > > >>>>> this is a bug in the flight implementation or in my own code. I
> > > >> thought I
> > > >>>>> would report here as it could potentially be a blocker to this
> > > release.
> > > >>>>>
> > > >>>>> I will confirm and come back to you shortly.
> > > >>>>>
> > > >>>>> Best,
> > > >>>>> Ryan
> > > >>>>>
>

Re: Release verification process

[Wes McKinney <we...@gmail.com>]

hi Neal,

On Thu, Jul 4, 2019 at 9:31 AM Neal Richardson
<ne...@gmail.com> wrote:
>
> Hi all,
> [Splitting off from the [VOTE] discussion thread]
>
> I wonder if there are things we can do to make for a smoother release next
> time. This was the first release I participated in with the Arrow project,
> and there were some aspects that were surprising to me. It seemed odd that
> "release verification" involved manually running a bunch of tests that
> weren't expected to pass, and then reporting that they didn't pass, and
> then forging ahead anyway.
>

Verification means different things to different people. The purpose
of the verification script is to try to build as many components of
the project as possible in a standalone way, using system environment
and dependencies in some cases. It helps turn up issues that do not
appear in the controlled CI environment.

Others will verify the release by testing their third party
applications using the software as a dependency.

> Perhaps we could, at the beginning of the release process and before making
> the first RC, make clear what is expected in order to verify it, and also
> state what is not required or expected. And we shouldn't spend time running
> tests that we don't care whether they fail or not. That way, we can have
> the discussion of what a successful release looks like before there's an RC
> on the table and a sense of urgency to ship it, and then it will be clear
> to everyone that the standards have been met when we run the tests.
>

I understand where you're coming from, but the release verification
process is intended to collect idiosyncratic information from people
attempting to use the software in different ways. I doubt that we
could write down satisfactory standards a priori or a cookie cutter
verification process that can be guaranteed to work for everyone. It's
been suggested in the past to convert the release verification to a
Dockerfile, but my view is that defeats the purpose of the script.

Note that our CI and Docker integration tests already provide a
certain level of security -- so we would hope to not have an RC unless
CI and docker-* tests are all passing (which they were here).

> Regardless of how we might want to improve the process for next time, I
> appreciate all of the hard work that Kou (especially) and others put in to
> making the release happen. Releasing a project of this size clearly takes a
> lot of dedication and effort.
>
> Neal
>
>
> On Thu, Jul 4, 2019 at 7:05 AM Wes McKinney <we...@gmail.com> wrote:
>
> > hi Jacques,
> >
> > I agree that we should allow ample time for discussions around issues
> > raised by people validating the release.
> >
> > That being said, with Ryan's issue, he is using a feature
> > (cross-language auth in Flight) that isn't being tested. The Flight
> > integration tests do not use authentication AFAIK so I'm not surprised
> > to hear that there may be an issue with it. It would be as if there
> > were a cross-compatibility issues between Python and Rust. Again,
> > would not surprise me because Rust is not included in the integration
> > tests. Even with a reproduction of such an issue I don't think it
> > should block the release in the future.
> >
> > I'd be glad to help make a point release in the near future since we
> > uncovered quite a few small issues during this RC, and I think there's
> > more value in getting the software into the hands of users than
> > waiting multiple weeks to address them all.
> >
> > - Wes
> >
> >
> > On Thu, Jul 4, 2019 at 8:50 AM Jacques Nadeau <ja...@apache.org> wrote:
> > >
> > > A release vote should last until we arrive at consensus. When an issue is
> > > potentially identified, those that have voted should be given ample time
> > to
> > > change their vote and others that may have been lazy consenters should be
> > > given time to chime in. There is no maximum amount of time a vote can be
> > > open. Allowing at least 24 hours after an objection is raised is a pretty
> > > minimum expectation unless the objector removes their objection.
> > >
> > > Note that Apache is more focused on consensus than timing (as opposed to
> > > virtually other other organizations in the world).
> > >
> > > In this case I was waiting to see more information from the objector
> > before
> > > casting my vote. I see a -1 vote as community chilling and prefer for
> > > consensus to be reached through discussions rather than contrary voting.
> > >
> > >
> > >
> > > On Thu, Jul 4, 2019, 6:35 AM Antoine Pitrou <an...@python.org> wrote:
> > >
> > > >
> > > > That's an open question.  How long should a release vote last?
> > > > Apparently this one lasted 3 days.  The 0.14.0 vote (after RC4) lasted
> > 3
> > > > days before a decision was made too.
> > > >
> > > > Regards
> > > >
> > > > Antoine.
> > > >
> > > >
> > > > Le 04/07/2019 à 15:28, Jacques Nadeau a écrit :
> > > > > There are two different questions here.
> > > > >
> > > > > 1) should this issue block the release.
> > > > > 2) should a discussion with ample time for all parties to come to
> > > > consensus
> > > > > have occurred before closing the vote.
> > > > >
> > > > > The first question is a subjective question. The answer to the second
> > > > > question should have been an unequivocal yes.
> > > > >
> > > > > I'll send a separate mail on point one but the problem I see here
> > > > > fundamentally is point two.
> > > > >
> > > > > On Thu, Jul 4, 2019, 6:01 AM Antoine Pitrou <an...@python.org>
> > wrote:
> > > > >
> > > > >>
> > > > >> I agree with Kou here.  It's not a problem with the release per se,
> > it's
> > > > >> a problem with the Flight code in git master.  There is no
> > regression
> > > > >> AFAICT, and we have not promised that Flight was stable and
> > > > >> production-ready yet.
> > > > >>
> > > > >> If we're ok releasing with bugs such as this (Java unable to read
> > back
> > > > >> union arrays?), then I think a Flight bug shouldn't hold a release.
> > > > >> https://issues.apache.org/jira/browse/ARROW-5231
> > > > >>
> > > > >> Regards
> > > > >>
> > > > >> Antoine.
> > > > >>
> > > > >>
> > > > >> Le 04/07/2019 à 14:46, Jacques Nadeau a écrit :
> > > > >>> Im disappointed in this response. When someone finds an issue with
> > a
> > > > >>> release it should be triaged/validated rather than rushed past
> > rather
> > > > >>> quickly and closing the vote before others can chime in on the
> > issue.
> > > > >>>
> > > > >>> I'm now left voting -1 on a closed vote. In the future, let's have
> > a
> > > > >>> discussion about issues like this. Your response and the close of
> > the
> > > > >> vote
> > > > >>> were unilateral and all happened while some of us were sleeping.
> > > > >>>
> > > > >>> On Wed, Jul 3, 2019, 10:47 PM Sutou Kouhei <ko...@clear-code.com>
> > wrote:
> > > > >>>
> > > > >>>> Hi,
> > > > >>>>
> > > > >>>> Flight isn't stable yet. So Flight problem isn't a blocker
> > > > >>>> of this release. Could you open a JIRA issue for your
> > > > >>>> problem? We'll be able to fix it until 1.0.0.
> > > > >>>>
> > > > >>>>
> > > > >>>> Thanks,
> > > > >>>> --
> > > > >>>> kou
> > > > >>>>
> > > > >>>> In <
> > > > CAKg4KDx8_Rr8xQOBaspW+U0LYPX21tPcWx6S4_RDdMk4VpDwcg@mail.gmail.com>
> > > > >>>>   "Re: [VOTE] Release Apache Arrow 0.14.0 - RC0" on Wed, 3 Jul
> > 2019
> > > > >>>> 17:18:15 -0700,
> > > > >>>>   Ryan Murray <ry...@dremio.com> wrote:
> > > > >>>>
> > > > >>>>> Hi All,
> > > > >>>>>
> > > > >>>>> I have noticed that a python Flight client can't authenticate to
> > a
> > > > Java
> > > > >>>>> Flight server while doing some testing today. I am trying to
> > confirm
> > > > if
> > > > >>>>> this is a bug in the flight implementation or in my own code. I
> > > > >> thought I
> > > > >>>>> would report here as it could potentially be a blocker to this
> > > > release.
> > > > >>>>>
> > > > >>>>> I will confirm and come back to you shortly.
> > > > >>>>>
> > > > >>>>> Best,
> > > > >>>>> Ryan
> > > > >>>>>
> >