[Discuss] Do we need a release verification script?

[Antoine Pitrou <an...@python.org>]

Hello,

Abiding by the Apache Software Foundation's guidelines, every Arrow 
release is voted on and requires at least 3 "binding" votes to be approved.

Also, every Arrow release vote is accompanied by a little ceremonial 
where contributors and core developers run a release verification script 
on their machine, wait for long minutes (sometimes an hour) and report 
the results.

This ceremonial has gone on for years, and it has not really been 
questioned. Yet, it's not obvious to me what it is achieving exactly. 
I've been here since 2018, but I don't really understand what the 
verification script is testing for, or, more importantly, *why* it is 
testing for what it is testing. I'm probably not the only one?

I would like to bring the following points:

* platform compatibility is (supposed to be) exercised on Continuous 
Integration; there is no understandable reason why it should be 
ceremoniously tested on each developer's machine before the release

* just before a release is probably the wrong time to be testing 
platform compatibility, and fixing compatibility bugs (though, of 
course, it might still be better than not noticing?)

* home environments are unstable, and not all developers run the 
verification script for each release, so each release is actually 
verified on different, uncontrolled, platforms

* as for sanity checks on binary packages, GPG signatures, etc., there 
shouldn't be any need to run them on multiple different machines, as 
they are (should be?) entirely deterministic and platform-agnostic

* maintaining the verification scripts is a thankless task, in part due 
to their nature (they need to track and mirror changes made in each 
implementation's build chain), in part due to implementation choices

* due to the existence of the verification scripts, the release vote is 
focussed on getting the script to run successfully (a very contextual 
and non-reproducible result), rather than the actual *contents* of the 
release

The most positive thing I can personally say about the verification 
scripts is that they *may* help us trust the release is not broken? But 
that's a very unqualified statement, and is very close to cargo-culting.

Regards

Antoine.

Re: [Discuss] Do we need a release verification script?

[Sutou Kouhei <ko...@clear-code.com>]

Hi,

I agree that we have problems in our release verification
script. But I think that it's useful for us.

> what the verification script is testing for

1. Verify cryptographic signatures of all artifacts
2. Compile source codes and run unit tests
3. Install binary packages and test them

> *why* it is testing for what it is testing

It's for legal protection. (1. only focus on it. I noticed
that license check by RAT isn't executed in the verification
script.)

https://www.apache.org/legal/release-policy.html#why

> The purpose of a clear line is to inform our legal
> strategy of providing protection for formal participants
> involved in producing releases

---

> * platform compatibility is (supposed to be) exercised on Continuous
>   Integration; there is no understandable reason why it should be
>   ceremoniously tested on each developer's machine before the release

Our CI covers many cases but we can't cover all cases. For
example, my environment (Debian GNU/Linux sid with GPU
without conda) isn't covered.

If a verifier's environment is covered by CI, I agree with
you. It's duplicated. The verifier will verify only
signatures and licenses.

Note that the verification script is used for CI too. So I
think that it's useful.

> * just before a release is probably the wrong time to be testing
>   platform compatibility, and fixing compatibility bugs (though, of
>   course, it might still be better than not noticing?)

It's better that we catch it in a timely manner.

I think that verifiers should verify an RC with their
downstream projects too. For example, I'm verifying an RC
with Groonga, a full-text search engine.

Some downstream projects such as pandas and Dask are covered
by our CI but we can't cover all downstream projects.

> * home environments are unstable, and not all developers run the
>   verification script for each release, so each release is actually
>   verified on different, uncontrolled, platforms

I think that it's an useful property for testing. We can
cover more cases by it. (Rare environments such as my
environment may not be valuable...) Users who use a release
use many different environments.

But it's better that we can catch it in timely manner.

> * maintaining the verification scripts is a thankless task, in part due
>   to their nature (they need to track and mirror changes made in each
>   implementation's build chain), in part due to implementation choices

I agree with you.

I think that one of useful tests in the verification script
is integration test without conda. (Our CI always uses conda
for integration test.) I hope that we can implement it with
low maintenance cost.

> * due to the existence of the verification scripts, the release vote is
>   focussed on getting the script to run successfully (a very contextual
>   and non-reproducible result), rather than the actual *contents* of the
>   release

Oh, really?
I added some checks to the verification script when I found
missing checks before we release a new version. For example,
I added .deb/.rpm install check.

But others may don't think about changing the verification
script...


Thanks,
-- 
kou

In <58...@python.org>
  "[Discuss] Do we need a release verification script?" on Tue, 22 Aug 2023 10:54:36 +0200,
  Antoine Pitrou <an...@python.org> wrote:

> 
> Hello,
> 
> Abiding by the Apache Software Foundation's guidelines, every Arrow
> release is voted on and requires at least 3 "binding" votes to be
> approved.
> 
> Also, every Arrow release vote is accompanied by a little ceremonial
> where contributors and core developers run a release verification
> script on their machine, wait for long minutes (sometimes an hour) and
> report the results.
> 
> This ceremonial has gone on for years, and it has not really been
> questioned. Yet, it's not obvious to me what it is achieving
> exactly. I've been here since 2018, but I don't really understand what
> the verification script is testing for, or, more importantly, *why* it
> is testing for what it is testing. I'm probably not the only one?
> 
> I would like to bring the following points:
> 
> * platform compatibility is (supposed to be) exercised on Continuous
> * Integration; there is no understandable reason why it should be
> * ceremoniously tested on each developer's machine before the release
> 
> * just before a release is probably the wrong time to be testing
> * platform compatibility, and fixing compatibility bugs (though, of
> * course, it might still be better than not noticing?)
> 
> * home environments are unstable, and not all developers run the
> * verification script for each release, so each release is actually
> * verified on different, uncontrolled, platforms
> 
> * as for sanity checks on binary packages, GPG signatures, etc., there
> * shouldn't be any need to run them on multiple different machines, as
> * they are (should be?) entirely deterministic and platform-agnostic
> 
> * maintaining the verification scripts is a thankless task, in part due
> * to their nature (they need to track and mirror changes made in each
> * implementation's build chain), in part due to implementation choices
> 
> * due to the existence of the verification scripts, the release vote is
> * focussed on getting the script to run successfully (a very contextual
> * and non-reproducible result), rather than the actual *contents* of the
> * release
> 
> The most positive thing I can personally say about the verification
> scripts is that they *may* help us trust the release is not broken?
> But that's a very unqualified statement, and is very close to
> cargo-culting.
> 
> Regards
> 
> Antoine.

Re: [Discuss] Do we need a release verification script?

[Dewey Dunnington <de...@voltrondata.com.INVALID>]

I can certainly empathize with the difficulty of maintaining a release
verification script and also the difficulty of remembering which
combination of environment variables are needed on my system to make
it work. The general sentiment of "anybody should be able to check
that this release actually works" is a good one, and (at least for R)
we have encountered many errors by running tests locally that were
never surfaced (and might never have been surfaced) on CI.

I wonder if the difficulty of maintaining the release script is at
least partially tied to the fact that we are trying to release so many
components simultaneously? Is it time to start releasing components
independently?

On Tue, Aug 22, 2023 at 10:11 PM Sutou Kouhei <ko...@clear-code.com> wrote:
>
> Hi,
>
> We can ask the ASF about this. members@apache.org?
>
>
> Thanks,
> --
> kou
>
> In <36...@python.org>
>   "Re: [Discuss] Do we need a release verification script?" on Tue, 22 Aug 2023 17:18:02 +0200,
>   Antoine Pitrou <an...@python.org> wrote:
>
> >
> > And of course this is a bit pedantic, and only important if we want to
> > comply with *the letter* of the ASF policies. My own personal opinion
> > is that complying in spirit is enough (but I'm also not sure I
> > understand the ASF's spirit :-)).
> >
> > Regards
> >
> > Antoine.
> >
> >
> > Le 22/08/2023 à 17:10, Antoine Pitrou a écrit :
> >> Hmm... perhaps Flatbuffers compilation is usually more deterministic
> >> than compiling C++ code into machine code, but that's mainly (AFAIK)
> >> because the transformation step is much simpler in the former case,
> >> than
> >> in the latter. The Flatbuffers compiler also has a range of options
> >> that
> >> influence code generation, certainly with less variation than a C++
> >> compiler, but still.
> >> In other words, I don't think being deterministic is a good criterion
> >> to
> >> know what "compiled code" means. There is a growing movement towards
> >> making generation of machine code artifacts deterministic:
> >> https://reproducible-builds.org/
> >> Regards
> >> Antoine.
> >> Le 22/08/2023 à 16:47, Adam Lippai a écrit :
> >>> Compiled code usually means binaries you can’t derive in a
> >>> deterministic,
> >>> verifiable way from the source code *shipped next to it*. So in this
> >>> case
> >>> any developer should be able to reproduce the flatbuffers output from
> >>> the
> >>> release package only.
> >>>
> >>> “Caches”, multi stage compilation etc should be ok.
> >>>
> >>> Best regards,
> >>> Adam Lippai
> >>>
> >>> On Tue, Aug 22, 2023 at 10:40 Antoine Pitrou <an...@python.org>
> >>> wrote:
> >>>
> >>>>
> >>>> If the main impetus for the verification script is to comply with ASF
> >>>> requirements, probably the script can be made much simpler, such as
> >>>> just
> >>>> verify the GPG signatures are valid? Or perhaps this can be achieved
> >>>> without a script at all.
> >>>>
> >>>> The irony is that, however complex, our verification script doesn't
> >>>> seem
> >>>> to check the actual ASF requirements on artifacts.
> >>>>
> >>>> For example, we don't check that """a source release SHOULD not
> >>>> contain
> >>>> compiled code""" (also, what does "compiled code" mean? does generated
> >>>> code, e.g. by the Flatbuffers compiler, apply?)
> >>>>
> >>>> Checking that the release """MUST be sufficient for a user to build
> >>>> and
> >>>> test the release provided they have access to the appropriate platform
> >>>> and tools""" is ill-defined and potentially tautologic, because the
> >>>> "appropriate platform and tools" is too imprecise and contextual (can
> >>>> the "appropriate platform and tools" contain a bunch of proprietary
> >>>> software that gets linked with the binaries? Well, it can, otherwise
> >>>> you
> >>>> can't build on Windows).
> >>>>
> >>>> Regards
> >>>>
> >>>> Antoine.
> >>>>
> >>>>
> >>>>
> >>>> Le 22/08/2023 à 12:31, Raúl Cumplido a écrit :
> >>>>> Hi,
> >>>>>
> >>>>> I do agree that currently verifying the release locally provides
> >>>>> little benefit for the effort we have to put in but I thought this was
> >>>>> required as per Apache policy:
> >>>>> https://www.apache.org/legal/release-policy.html#release-approval
> >>>>>
> >>>>> Copying the important bit:
> >>>>> """
> >>>>> Before casting +1 binding votes, individuals are REQUIRED to download
> >>>>> all signed source code packages onto their own hardware, verify that
> >>>>> they meet all requirements of ASF policy on releases as described
> >>>>> below, validate all cryptographic signatures, compile as provided, and
> >>>>> test the result on their own platform.
> >>>>> """
> >>>>>
> >>>>> I also think we should try and challenge those.
> >>>>>
> >>>>> In the past we have identified some minor issues on the local
> >>>>> verification but I don't recall any of them being blockers for the
> >>>>> release.
> >>>>>
> >>>>> Thanks,
> >>>>> Raúl
> >>>>>
> >>>>> El mar, 22 ago 2023 a las 11:46, Andrew Lamb (<al...@influxdata.com>)
> >>>> escribió:
> >>>>>>
> >>>>>> The Rust arrow implementation (arrow-rs) and DataFusion also use
> >>>>>> release
> >>>>>> verification scripts, mostly inherited from when they were split from
> >>>> the
> >>>>>> mono repo. They have found issues from time to time, for us, but those
> >>>>>> issues are often not platform related and have not been release
> >>>> blockers.
> >>>>>>
> >>>>>> Thankfully for Rust, the verification scripts don't need much
> >>>> maintenance
> >>>>>> so we just continue the ceremony. However, I certainly don't think we
> >>>> would
> >>>>>> lose much/any test coverage if we stopped their use.
> >>>>>>
> >>>>>> Andrew
> >>>>>>
> >>>>>> On Tue, Aug 22, 2023 at 4:54 AM Antoine Pitrou <an...@python.org>
> >>>> wrote:
> >>>>>>
> >>>>>>>
> >>>>>>> Hello,
> >>>>>>>
> >>>>>>> Abiding by the Apache Software Foundation's guidelines, every Arrow
> >>>>>>> release is voted on and requires at least 3 "binding" votes to be
> >>>> approved.
> >>>>>>>
> >>>>>>> Also, every Arrow release vote is accompanied by a little ceremonial
> >>>>>>> where contributors and core developers run a release verification
> >>>> script
> >>>>>>> on their machine, wait for long minutes (sometimes an hour) and report
> >>>>>>> the results.
> >>>>>>>
> >>>>>>> This ceremonial has gone on for years, and it has not really been
> >>>>>>> questioned. Yet, it's not obvious to me what it is achieving exactly.
> >>>>>>> I've been here since 2018, but I don't really understand what the
> >>>>>>> verification script is testing for, or, more importantly, *why* it is
> >>>>>>> testing for what it is testing. I'm probably not the only one?
> >>>>>>>
> >>>>>>> I would like to bring the following points:
> >>>>>>>
> >>>>>>> * platform compatibility is (supposed to be) exercised on Continuous
> >>>>>>> Integration; there is no understandable reason why it should be
> >>>>>>> ceremoniously tested on each developer's machine before the release
> >>>>>>>
> >>>>>>> * just before a release is probably the wrong time to be testing
> >>>>>>> platform compatibility, and fixing compatibility bugs (though, of
> >>>>>>> course, it might still be better than not noticing?)
> >>>>>>>
> >>>>>>> * home environments are unstable, and not all developers run the
> >>>>>>> verification script for each release, so each release is actually
> >>>>>>> verified on different, uncontrolled, platforms
> >>>>>>>
> >>>>>>> * as for sanity checks on binary packages, GPG signatures, etc., there
> >>>>>>> shouldn't be any need to run them on multiple different machines, as
> >>>>>>> they are (should be?) entirely deterministic and platform-agnostic
> >>>>>>>
> >>>>>>> * maintaining the verification scripts is a thankless task, in part due
> >>>>>>> to their nature (they need to track and mirror changes made in each
> >>>>>>> implementation's build chain), in part due to implementation choices
> >>>>>>>
> >>>>>>> * due to the existence of the verification scripts, the release vote is
> >>>>>>> focussed on getting the script to run successfully (a very contextual
> >>>>>>> and non-reproducible result), rather than the actual *contents* of the
> >>>>>>> release
> >>>>>>>
> >>>>>>> The most positive thing I can personally say about the verification
> >>>>>>> scripts is that they *may* help us trust the release is not broken?
> >>>>>>> But
> >>>>>>> that's a very unqualified statement, and is very close to
> >>>> cargo-culting.
> >>>>>>>
> >>>>>>> Regards
> >>>>>>>
> >>>>>>> Antoine.
> >>>>>>>
> >>>>
> >>>

Re: [Discuss] Do we need a release verification script?

[Sutou Kouhei <ko...@clear-code.com>]

Hi,

We can ask the ASF about this. members@apache.org?


Thanks,
-- 
kou

In <36...@python.org>
  "Re: [Discuss] Do we need a release verification script?" on Tue, 22 Aug 2023 17:18:02 +0200,
  Antoine Pitrou <an...@python.org> wrote:

> 
> And of course this is a bit pedantic, and only important if we want to
> comply with *the letter* of the ASF policies. My own personal opinion
> is that complying in spirit is enough (but I'm also not sure I
> understand the ASF's spirit :-)).
> 
> Regards
> 
> Antoine.
> 
> 
> Le 22/08/2023 à 17:10, Antoine Pitrou a écrit :
>> Hmm... perhaps Flatbuffers compilation is usually more deterministic
>> than compiling C++ code into machine code, but that's mainly (AFAIK)
>> because the transformation step is much simpler in the former case,
>> than
>> in the latter. The Flatbuffers compiler also has a range of options
>> that
>> influence code generation, certainly with less variation than a C++
>> compiler, but still.
>> In other words, I don't think being deterministic is a good criterion
>> to
>> know what "compiled code" means. There is a growing movement towards
>> making generation of machine code artifacts deterministic:
>> https://reproducible-builds.org/
>> Regards
>> Antoine.
>> Le 22/08/2023 à 16:47, Adam Lippai a écrit :
>>> Compiled code usually means binaries you can’t derive in a
>>> deterministic,
>>> verifiable way from the source code *shipped next to it*. So in this
>>> case
>>> any developer should be able to reproduce the flatbuffers output from
>>> the
>>> release package only.
>>>
>>> “Caches”, multi stage compilation etc should be ok.
>>>
>>> Best regards,
>>> Adam Lippai
>>>
>>> On Tue, Aug 22, 2023 at 10:40 Antoine Pitrou <an...@python.org>
>>> wrote:
>>>
>>>>
>>>> If the main impetus for the verification script is to comply with ASF
>>>> requirements, probably the script can be made much simpler, such as
>>>> just
>>>> verify the GPG signatures are valid? Or perhaps this can be achieved
>>>> without a script at all.
>>>>
>>>> The irony is that, however complex, our verification script doesn't
>>>> seem
>>>> to check the actual ASF requirements on artifacts.
>>>>
>>>> For example, we don't check that """a source release SHOULD not
>>>> contain
>>>> compiled code""" (also, what does "compiled code" mean? does generated
>>>> code, e.g. by the Flatbuffers compiler, apply?)
>>>>
>>>> Checking that the release """MUST be sufficient for a user to build
>>>> and
>>>> test the release provided they have access to the appropriate platform
>>>> and tools""" is ill-defined and potentially tautologic, because the
>>>> "appropriate platform and tools" is too imprecise and contextual (can
>>>> the "appropriate platform and tools" contain a bunch of proprietary
>>>> software that gets linked with the binaries? Well, it can, otherwise
>>>> you
>>>> can't build on Windows).
>>>>
>>>> Regards
>>>>
>>>> Antoine.
>>>>
>>>>
>>>>
>>>> Le 22/08/2023 à 12:31, Raúl Cumplido a écrit :
>>>>> Hi,
>>>>>
>>>>> I do agree that currently verifying the release locally provides
>>>>> little benefit for the effort we have to put in but I thought this was
>>>>> required as per Apache policy:
>>>>> https://www.apache.org/legal/release-policy.html#release-approval
>>>>>
>>>>> Copying the important bit:
>>>>> """
>>>>> Before casting +1 binding votes, individuals are REQUIRED to download
>>>>> all signed source code packages onto their own hardware, verify that
>>>>> they meet all requirements of ASF policy on releases as described
>>>>> below, validate all cryptographic signatures, compile as provided, and
>>>>> test the result on their own platform.
>>>>> """
>>>>>
>>>>> I also think we should try and challenge those.
>>>>>
>>>>> In the past we have identified some minor issues on the local
>>>>> verification but I don't recall any of them being blockers for the
>>>>> release.
>>>>>
>>>>> Thanks,
>>>>> Raúl
>>>>>
>>>>> El mar, 22 ago 2023 a las 11:46, Andrew Lamb (<al...@influxdata.com>)
>>>> escribió:
>>>>>>
>>>>>> The Rust arrow implementation (arrow-rs) and DataFusion also use
>>>>>> release
>>>>>> verification scripts, mostly inherited from when they were split from
>>>> the
>>>>>> mono repo. They have found issues from time to time, for us, but those
>>>>>> issues are often not platform related and have not been release
>>>> blockers.
>>>>>>
>>>>>> Thankfully for Rust, the verification scripts don't need much
>>>> maintenance
>>>>>> so we just continue the ceremony. However, I certainly don't think we
>>>> would
>>>>>> lose much/any test coverage if we stopped their use.
>>>>>>
>>>>>> Andrew
>>>>>>
>>>>>> On Tue, Aug 22, 2023 at 4:54 AM Antoine Pitrou <an...@python.org>
>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> Abiding by the Apache Software Foundation's guidelines, every Arrow
>>>>>>> release is voted on and requires at least 3 "binding" votes to be
>>>> approved.
>>>>>>>
>>>>>>> Also, every Arrow release vote is accompanied by a little ceremonial
>>>>>>> where contributors and core developers run a release verification
>>>> script
>>>>>>> on their machine, wait for long minutes (sometimes an hour) and report
>>>>>>> the results.
>>>>>>>
>>>>>>> This ceremonial has gone on for years, and it has not really been
>>>>>>> questioned. Yet, it's not obvious to me what it is achieving exactly.
>>>>>>> I've been here since 2018, but I don't really understand what the
>>>>>>> verification script is testing for, or, more importantly, *why* it is
>>>>>>> testing for what it is testing. I'm probably not the only one?
>>>>>>>
>>>>>>> I would like to bring the following points:
>>>>>>>
>>>>>>> * platform compatibility is (supposed to be) exercised on Continuous
>>>>>>> Integration; there is no understandable reason why it should be
>>>>>>> ceremoniously tested on each developer's machine before the release
>>>>>>>
>>>>>>> * just before a release is probably the wrong time to be testing
>>>>>>> platform compatibility, and fixing compatibility bugs (though, of
>>>>>>> course, it might still be better than not noticing?)
>>>>>>>
>>>>>>> * home environments are unstable, and not all developers run the
>>>>>>> verification script for each release, so each release is actually
>>>>>>> verified on different, uncontrolled, platforms
>>>>>>>
>>>>>>> * as for sanity checks on binary packages, GPG signatures, etc., there
>>>>>>> shouldn't be any need to run them on multiple different machines, as
>>>>>>> they are (should be?) entirely deterministic and platform-agnostic
>>>>>>>
>>>>>>> * maintaining the verification scripts is a thankless task, in part due
>>>>>>> to their nature (they need to track and mirror changes made in each
>>>>>>> implementation's build chain), in part due to implementation choices
>>>>>>>
>>>>>>> * due to the existence of the verification scripts, the release vote is
>>>>>>> focussed on getting the script to run successfully (a very contextual
>>>>>>> and non-reproducible result), rather than the actual *contents* of the
>>>>>>> release
>>>>>>>
>>>>>>> The most positive thing I can personally say about the verification
>>>>>>> scripts is that they *may* help us trust the release is not broken?
>>>>>>> But
>>>>>>> that's a very unqualified statement, and is very close to
>>>> cargo-culting.
>>>>>>>
>>>>>>> Regards
>>>>>>>
>>>>>>> Antoine.
>>>>>>>
>>>>
>>>

Re: [Discuss] Do we need a release verification script?

[Neal Richardson <ne...@gmail.com>]

I would be curious how old that language is in the ASF policy. In the era
of cloud infrastructure and public CI, "your own hardware" might mean
something different than it once did. IMO, it is essential that we download
and test the release artifacts in a separate context from where they are
built in order to verify that they work as intended, which seems like the
spirit of the policy. But if I can kick off a CI job that does that across
a matrix of platforms and environments, that seems more meaningful than
whether it works on a particular machine in my physical possession.

Neal

On Tue, Aug 22, 2023 at 11:18 AM Antoine Pitrou <an...@python.org> wrote:

>
> And of course this is a bit pedantic, and only important if we want to
> comply with *the letter* of the ASF policies. My own personal opinion is
> that complying in spirit is enough (but I'm also not sure I understand
> the ASF's spirit :-)).
>
> Regards
>
> Antoine.
>
>
> Le 22/08/2023 à 17:10, Antoine Pitrou a écrit :
> >
> > Hmm... perhaps Flatbuffers compilation is usually more deterministic
> > than compiling C++ code into machine code, but that's mainly (AFAIK)
> > because the transformation step is much simpler in the former case, than
> > in the latter. The Flatbuffers compiler also has a range of options that
> > influence code generation, certainly with less variation than a C++
> > compiler, but still.
> >
> > In other words, I don't think being deterministic is a good criterion to
> > know what "compiled code" means. There is a growing movement towards
> > making generation of machine code artifacts deterministic:
> > https://reproducible-builds.org/
> >
> > Regards
> >
> > Antoine.
> >
> >
> >
> > Le 22/08/2023 à 16:47, Adam Lippai a écrit :
> >> Compiled code usually means binaries you can’t derive in a
> deterministic,
> >> verifiable way from the source code *shipped next to it*. So in this
> case
> >> any developer should be able to reproduce the flatbuffers output from
> the
> >> release package only.
> >>
> >> “Caches”, multi stage compilation etc should be ok.
> >>
> >> Best regards,
> >> Adam Lippai
> >>
> >> On Tue, Aug 22, 2023 at 10:40 Antoine Pitrou <an...@python.org>
> wrote:
> >>
> >>>
> >>> If the main impetus for the verification script is to comply with ASF
> >>> requirements, probably the script can be made much simpler, such as
> just
> >>> verify the GPG signatures are valid? Or perhaps this can be achieved
> >>> without a script at all.
> >>>
> >>> The irony is that, however complex, our verification script doesn't
> seem
> >>> to check the actual ASF requirements on artifacts.
> >>>
> >>> For example, we don't check that """a source release SHOULD not contain
> >>> compiled code""" (also, what does "compiled code" mean? does generated
> >>> code, e.g. by the Flatbuffers compiler, apply?)
> >>>
> >>> Checking that the release """MUST be sufficient for a user to build and
> >>> test the release provided they have access to the appropriate platform
> >>> and tools""" is ill-defined and potentially tautologic, because the
> >>> "appropriate platform and tools" is too imprecise and contextual (can
> >>> the "appropriate platform and tools" contain a bunch of proprietary
> >>> software that gets linked with the binaries? Well, it can, otherwise
> you
> >>> can't build on Windows).
> >>>
> >>> Regards
> >>>
> >>> Antoine.
> >>>
> >>>
> >>>
> >>> Le 22/08/2023 à 12:31, Raúl Cumplido a écrit :
> >>>> Hi,
> >>>>
> >>>> I do agree that currently verifying the release locally provides
> >>>> little benefit for the effort we have to put in but I thought this was
> >>>> required as per Apache policy:
> >>>> https://www.apache.org/legal/release-policy.html#release-approval
> >>>>
> >>>> Copying the important bit:
> >>>> """
> >>>> Before casting +1 binding votes, individuals are REQUIRED to download
> >>>> all signed source code packages onto their own hardware, verify that
> >>>> they meet all requirements of ASF policy on releases as described
> >>>> below, validate all cryptographic signatures, compile as provided, and
> >>>> test the result on their own platform.
> >>>> """
> >>>>
> >>>> I also think we should try and challenge those.
> >>>>
> >>>> In the past we have identified some minor issues on the local
> >>>> verification but I don't recall any of them being blockers for the
> >>>> release.
> >>>>
> >>>> Thanks,
> >>>> Raúl
> >>>>
> >>>> El mar, 22 ago 2023 a las 11:46, Andrew Lamb (<al...@influxdata.com>)
> >>> escribió:
> >>>>>
> >>>>> The Rust arrow implementation (arrow-rs) and DataFusion also use
> release
> >>>>> verification scripts, mostly inherited from when they were split from
> >>> the
> >>>>> mono repo. They have found issues from time to time, for us, but
> those
> >>>>> issues are often not platform related and have not been release
> >>> blockers.
> >>>>>
> >>>>> Thankfully for Rust, the verification scripts don't need much
> >>> maintenance
> >>>>> so we just continue the ceremony. However, I certainly don't think we
> >>> would
> >>>>> lose much/any test coverage if we stopped their use.
> >>>>>
> >>>>> Andrew
> >>>>>
> >>>>> On Tue, Aug 22, 2023 at 4:54 AM Antoine Pitrou <an...@python.org>
> >>> wrote:
> >>>>>
> >>>>>>
> >>>>>> Hello,
> >>>>>>
> >>>>>> Abiding by the Apache Software Foundation's guidelines, every Arrow
> >>>>>> release is voted on and requires at least 3 "binding" votes to be
> >>> approved.
> >>>>>>
> >>>>>> Also, every Arrow release vote is accompanied by a little ceremonial
> >>>>>> where contributors and core developers run a release verification
> >>> script
> >>>>>> on their machine, wait for long minutes (sometimes an hour) and
> report
> >>>>>> the results.
> >>>>>>
> >>>>>> This ceremonial has gone on for years, and it has not really been
> >>>>>> questioned. Yet, it's not obvious to me what it is achieving
> exactly.
> >>>>>> I've been here since 2018, but I don't really understand what the
> >>>>>> verification script is testing for, or, more importantly, *why* it
> is
> >>>>>> testing for what it is testing. I'm probably not the only one?
> >>>>>>
> >>>>>> I would like to bring the following points:
> >>>>>>
> >>>>>> * platform compatibility is (supposed to be) exercised on Continuous
> >>>>>> Integration; there is no understandable reason why it should be
> >>>>>> ceremoniously tested on each developer's machine before the release
> >>>>>>
> >>>>>> * just before a release is probably the wrong time to be testing
> >>>>>> platform compatibility, and fixing compatibility bugs (though, of
> >>>>>> course, it might still be better than not noticing?)
> >>>>>>
> >>>>>> * home environments are unstable, and not all developers run the
> >>>>>> verification script for each release, so each release is actually
> >>>>>> verified on different, uncontrolled, platforms
> >>>>>>
> >>>>>> * as for sanity checks on binary packages, GPG signatures, etc.,
> there
> >>>>>> shouldn't be any need to run them on multiple different machines, as
> >>>>>> they are (should be?) entirely deterministic and platform-agnostic
> >>>>>>
> >>>>>> * maintaining the verification scripts is a thankless task, in part
> due
> >>>>>> to their nature (they need to track and mirror changes made in each
> >>>>>> implementation's build chain), in part due to implementation choices
> >>>>>>
> >>>>>> * due to the existence of the verification scripts, the release
> vote is
> >>>>>> focussed on getting the script to run successfully (a very
> contextual
> >>>>>> and non-reproducible result), rather than the actual *contents* of
> the
> >>>>>> release
> >>>>>>
> >>>>>> The most positive thing I can personally say about the verification
> >>>>>> scripts is that they *may* help us trust the release is not broken?
> But
> >>>>>> that's a very unqualified statement, and is very close to
> >>> cargo-culting.
> >>>>>>
> >>>>>> Regards
> >>>>>>
> >>>>>> Antoine.
> >>>>>>
> >>>
> >>
>

Re: [Discuss] Do we need a release verification script?

[Antoine Pitrou <an...@python.org>]

And of course this is a bit pedantic, and only important if we want to 
comply with *the letter* of the ASF policies. My own personal opinion is 
that complying in spirit is enough (but I'm also not sure I understand 
the ASF's spirit :-)).

Regards

Antoine.


Le 22/08/2023 à 17:10, Antoine Pitrou a écrit :
> 
> Hmm... perhaps Flatbuffers compilation is usually more deterministic
> than compiling C++ code into machine code, but that's mainly (AFAIK)
> because the transformation step is much simpler in the former case, than
> in the latter. The Flatbuffers compiler also has a range of options that
> influence code generation, certainly with less variation than a C++
> compiler, but still.
> 
> In other words, I don't think being deterministic is a good criterion to
> know what "compiled code" means. There is a growing movement towards
> making generation of machine code artifacts deterministic:
> https://reproducible-builds.org/
> 
> Regards
> 
> Antoine.
> 
> 
> 
> Le 22/08/2023 à 16:47, Adam Lippai a écrit :
>> Compiled code usually means binaries you can’t derive in a deterministic,
>> verifiable way from the source code *shipped next to it*. So in this case
>> any developer should be able to reproduce the flatbuffers output from the
>> release package only.
>>
>> “Caches”, multi stage compilation etc should be ok.
>>
>> Best regards,
>> Adam Lippai
>>
>> On Tue, Aug 22, 2023 at 10:40 Antoine Pitrou <an...@python.org> wrote:
>>
>>>
>>> If the main impetus for the verification script is to comply with ASF
>>> requirements, probably the script can be made much simpler, such as just
>>> verify the GPG signatures are valid? Or perhaps this can be achieved
>>> without a script at all.
>>>
>>> The irony is that, however complex, our verification script doesn't seem
>>> to check the actual ASF requirements on artifacts.
>>>
>>> For example, we don't check that """a source release SHOULD not contain
>>> compiled code""" (also, what does "compiled code" mean? does generated
>>> code, e.g. by the Flatbuffers compiler, apply?)
>>>
>>> Checking that the release """MUST be sufficient for a user to build and
>>> test the release provided they have access to the appropriate platform
>>> and tools""" is ill-defined and potentially tautologic, because the
>>> "appropriate platform and tools" is too imprecise and contextual (can
>>> the "appropriate platform and tools" contain a bunch of proprietary
>>> software that gets linked with the binaries? Well, it can, otherwise you
>>> can't build on Windows).
>>>
>>> Regards
>>>
>>> Antoine.
>>>
>>>
>>>
>>> Le 22/08/2023 à 12:31, Raúl Cumplido a écrit :
>>>> Hi,
>>>>
>>>> I do agree that currently verifying the release locally provides
>>>> little benefit for the effort we have to put in but I thought this was
>>>> required as per Apache policy:
>>>> https://www.apache.org/legal/release-policy.html#release-approval
>>>>
>>>> Copying the important bit:
>>>> """
>>>> Before casting +1 binding votes, individuals are REQUIRED to download
>>>> all signed source code packages onto their own hardware, verify that
>>>> they meet all requirements of ASF policy on releases as described
>>>> below, validate all cryptographic signatures, compile as provided, and
>>>> test the result on their own platform.
>>>> """
>>>>
>>>> I also think we should try and challenge those.
>>>>
>>>> In the past we have identified some minor issues on the local
>>>> verification but I don't recall any of them being blockers for the
>>>> release.
>>>>
>>>> Thanks,
>>>> Raúl
>>>>
>>>> El mar, 22 ago 2023 a las 11:46, Andrew Lamb (<al...@influxdata.com>)
>>> escribió:
>>>>>
>>>>> The Rust arrow implementation (arrow-rs) and DataFusion also use release
>>>>> verification scripts, mostly inherited from when they were split from
>>> the
>>>>> mono repo. They have found issues from time to time, for us, but those
>>>>> issues are often not platform related and have not been release
>>> blockers.
>>>>>
>>>>> Thankfully for Rust, the verification scripts don't need much
>>> maintenance
>>>>> so we just continue the ceremony. However, I certainly don't think we
>>> would
>>>>> lose much/any test coverage if we stopped their use.
>>>>>
>>>>> Andrew
>>>>>
>>>>> On Tue, Aug 22, 2023 at 4:54 AM Antoine Pitrou <an...@python.org>
>>> wrote:
>>>>>
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> Abiding by the Apache Software Foundation's guidelines, every Arrow
>>>>>> release is voted on and requires at least 3 "binding" votes to be
>>> approved.
>>>>>>
>>>>>> Also, every Arrow release vote is accompanied by a little ceremonial
>>>>>> where contributors and core developers run a release verification
>>> script
>>>>>> on their machine, wait for long minutes (sometimes an hour) and report
>>>>>> the results.
>>>>>>
>>>>>> This ceremonial has gone on for years, and it has not really been
>>>>>> questioned. Yet, it's not obvious to me what it is achieving exactly.
>>>>>> I've been here since 2018, but I don't really understand what the
>>>>>> verification script is testing for, or, more importantly, *why* it is
>>>>>> testing for what it is testing. I'm probably not the only one?
>>>>>>
>>>>>> I would like to bring the following points:
>>>>>>
>>>>>> * platform compatibility is (supposed to be) exercised on Continuous
>>>>>> Integration; there is no understandable reason why it should be
>>>>>> ceremoniously tested on each developer's machine before the release
>>>>>>
>>>>>> * just before a release is probably the wrong time to be testing
>>>>>> platform compatibility, and fixing compatibility bugs (though, of
>>>>>> course, it might still be better than not noticing?)
>>>>>>
>>>>>> * home environments are unstable, and not all developers run the
>>>>>> verification script for each release, so each release is actually
>>>>>> verified on different, uncontrolled, platforms
>>>>>>
>>>>>> * as for sanity checks on binary packages, GPG signatures, etc., there
>>>>>> shouldn't be any need to run them on multiple different machines, as
>>>>>> they are (should be?) entirely deterministic and platform-agnostic
>>>>>>
>>>>>> * maintaining the verification scripts is a thankless task, in part due
>>>>>> to their nature (they need to track and mirror changes made in each
>>>>>> implementation's build chain), in part due to implementation choices
>>>>>>
>>>>>> * due to the existence of the verification scripts, the release vote is
>>>>>> focussed on getting the script to run successfully (a very contextual
>>>>>> and non-reproducible result), rather than the actual *contents* of the
>>>>>> release
>>>>>>
>>>>>> The most positive thing I can personally say about the verification
>>>>>> scripts is that they *may* help us trust the release is not broken? But
>>>>>> that's a very unqualified statement, and is very close to
>>> cargo-culting.
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>> Antoine.
>>>>>>
>>>
>>

Re: [Discuss] Do we need a release verification script?

[Antoine Pitrou <an...@python.org>]

Hmm... perhaps Flatbuffers compilation is usually more deterministic 
than compiling C++ code into machine code, but that's mainly (AFAIK) 
because the transformation step is much simpler in the former case, than 
in the latter. The Flatbuffers compiler also has a range of options that 
influence code generation, certainly with less variation than a C++ 
compiler, but still.

In other words, I don't think being deterministic is a good criterion to 
know what "compiled code" means. There is a growing movement towards 
making generation of machine code artifacts deterministic:
https://reproducible-builds.org/

Regards

Antoine.



Le 22/08/2023 à 16:47, Adam Lippai a écrit :
> Compiled code usually means binaries you can’t derive in a deterministic,
> verifiable way from the source code *shipped next to it*. So in this case
> any developer should be able to reproduce the flatbuffers output from the
> release package only.
> 
> “Caches”, multi stage compilation etc should be ok.
> 
> Best regards,
> Adam Lippai
> 
> On Tue, Aug 22, 2023 at 10:40 Antoine Pitrou <an...@python.org> wrote:
> 
>>
>> If the main impetus for the verification script is to comply with ASF
>> requirements, probably the script can be made much simpler, such as just
>> verify the GPG signatures are valid? Or perhaps this can be achieved
>> without a script at all.
>>
>> The irony is that, however complex, our verification script doesn't seem
>> to check the actual ASF requirements on artifacts.
>>
>> For example, we don't check that """a source release SHOULD not contain
>> compiled code""" (also, what does "compiled code" mean? does generated
>> code, e.g. by the Flatbuffers compiler, apply?)
>>
>> Checking that the release """MUST be sufficient for a user to build and
>> test the release provided they have access to the appropriate platform
>> and tools""" is ill-defined and potentially tautologic, because the
>> "appropriate platform and tools" is too imprecise and contextual (can
>> the "appropriate platform and tools" contain a bunch of proprietary
>> software that gets linked with the binaries? Well, it can, otherwise you
>> can't build on Windows).
>>
>> Regards
>>
>> Antoine.
>>
>>
>>
>> Le 22/08/2023 à 12:31, Raúl Cumplido a écrit :
>>> Hi,
>>>
>>> I do agree that currently verifying the release locally provides
>>> little benefit for the effort we have to put in but I thought this was
>>> required as per Apache policy:
>>> https://www.apache.org/legal/release-policy.html#release-approval
>>>
>>> Copying the important bit:
>>> """
>>> Before casting +1 binding votes, individuals are REQUIRED to download
>>> all signed source code packages onto their own hardware, verify that
>>> they meet all requirements of ASF policy on releases as described
>>> below, validate all cryptographic signatures, compile as provided, and
>>> test the result on their own platform.
>>> """
>>>
>>> I also think we should try and challenge those.
>>>
>>> In the past we have identified some minor issues on the local
>>> verification but I don't recall any of them being blockers for the
>>> release.
>>>
>>> Thanks,
>>> Raúl
>>>
>>> El mar, 22 ago 2023 a las 11:46, Andrew Lamb (<al...@influxdata.com>)
>> escribió:
>>>>
>>>> The Rust arrow implementation (arrow-rs) and DataFusion also use release
>>>> verification scripts, mostly inherited from when they were split from
>> the
>>>> mono repo. They have found issues from time to time, for us, but those
>>>> issues are often not platform related and have not been release
>> blockers.
>>>>
>>>> Thankfully for Rust, the verification scripts don't need much
>> maintenance
>>>> so we just continue the ceremony. However, I certainly don't think we
>> would
>>>> lose much/any test coverage if we stopped their use.
>>>>
>>>> Andrew
>>>>
>>>> On Tue, Aug 22, 2023 at 4:54 AM Antoine Pitrou <an...@python.org>
>> wrote:
>>>>
>>>>>
>>>>> Hello,
>>>>>
>>>>> Abiding by the Apache Software Foundation's guidelines, every Arrow
>>>>> release is voted on and requires at least 3 "binding" votes to be
>> approved.
>>>>>
>>>>> Also, every Arrow release vote is accompanied by a little ceremonial
>>>>> where contributors and core developers run a release verification
>> script
>>>>> on their machine, wait for long minutes (sometimes an hour) and report
>>>>> the results.
>>>>>
>>>>> This ceremonial has gone on for years, and it has not really been
>>>>> questioned. Yet, it's not obvious to me what it is achieving exactly.
>>>>> I've been here since 2018, but I don't really understand what the
>>>>> verification script is testing for, or, more importantly, *why* it is
>>>>> testing for what it is testing. I'm probably not the only one?
>>>>>
>>>>> I would like to bring the following points:
>>>>>
>>>>> * platform compatibility is (supposed to be) exercised on Continuous
>>>>> Integration; there is no understandable reason why it should be
>>>>> ceremoniously tested on each developer's machine before the release
>>>>>
>>>>> * just before a release is probably the wrong time to be testing
>>>>> platform compatibility, and fixing compatibility bugs (though, of
>>>>> course, it might still be better than not noticing?)
>>>>>
>>>>> * home environments are unstable, and not all developers run the
>>>>> verification script for each release, so each release is actually
>>>>> verified on different, uncontrolled, platforms
>>>>>
>>>>> * as for sanity checks on binary packages, GPG signatures, etc., there
>>>>> shouldn't be any need to run them on multiple different machines, as
>>>>> they are (should be?) entirely deterministic and platform-agnostic
>>>>>
>>>>> * maintaining the verification scripts is a thankless task, in part due
>>>>> to their nature (they need to track and mirror changes made in each
>>>>> implementation's build chain), in part due to implementation choices
>>>>>
>>>>> * due to the existence of the verification scripts, the release vote is
>>>>> focussed on getting the script to run successfully (a very contextual
>>>>> and non-reproducible result), rather than the actual *contents* of the
>>>>> release
>>>>>
>>>>> The most positive thing I can personally say about the verification
>>>>> scripts is that they *may* help us trust the release is not broken? But
>>>>> that's a very unqualified statement, and is very close to
>> cargo-culting.
>>>>>
>>>>> Regards
>>>>>
>>>>> Antoine.
>>>>>
>>
> 

Re: [Discuss] Do we need a release verification script?

[Adam Lippai <ad...@rigo.sk>]

Compiled code usually means binaries you can’t derive in a deterministic,
verifiable way from the source code *shipped next to it*. So in this case
any developer should be able to reproduce the flatbuffers output from the
release package only.

“Caches”, multi stage compilation etc should be ok.

Best regards,
Adam Lippai

On Tue, Aug 22, 2023 at 10:40 Antoine Pitrou <an...@python.org> wrote:

>
> If the main impetus for the verification script is to comply with ASF
> requirements, probably the script can be made much simpler, such as just
> verify the GPG signatures are valid? Or perhaps this can be achieved
> without a script at all.
>
> The irony is that, however complex, our verification script doesn't seem
> to check the actual ASF requirements on artifacts.
>
> For example, we don't check that """a source release SHOULD not contain
> compiled code""" (also, what does "compiled code" mean? does generated
> code, e.g. by the Flatbuffers compiler, apply?)
>
> Checking that the release """MUST be sufficient for a user to build and
> test the release provided they have access to the appropriate platform
> and tools""" is ill-defined and potentially tautologic, because the
> "appropriate platform and tools" is too imprecise and contextual (can
> the "appropriate platform and tools" contain a bunch of proprietary
> software that gets linked with the binaries? Well, it can, otherwise you
> can't build on Windows).
>
> Regards
>
> Antoine.
>
>
>
> Le 22/08/2023 à 12:31, Raúl Cumplido a écrit :
> > Hi,
> >
> > I do agree that currently verifying the release locally provides
> > little benefit for the effort we have to put in but I thought this was
> > required as per Apache policy:
> > https://www.apache.org/legal/release-policy.html#release-approval
> >
> > Copying the important bit:
> > """
> > Before casting +1 binding votes, individuals are REQUIRED to download
> > all signed source code packages onto their own hardware, verify that
> > they meet all requirements of ASF policy on releases as described
> > below, validate all cryptographic signatures, compile as provided, and
> > test the result on their own platform.
> > """
> >
> > I also think we should try and challenge those.
> >
> > In the past we have identified some minor issues on the local
> > verification but I don't recall any of them being blockers for the
> > release.
> >
> > Thanks,
> > Raúl
> >
> > El mar, 22 ago 2023 a las 11:46, Andrew Lamb (<al...@influxdata.com>)
> escribió:
> >>
> >> The Rust arrow implementation (arrow-rs) and DataFusion also use release
> >> verification scripts, mostly inherited from when they were split from
> the
> >> mono repo. They have found issues from time to time, for us, but those
> >> issues are often not platform related and have not been release
> blockers.
> >>
> >> Thankfully for Rust, the verification scripts don't need much
> maintenance
> >> so we just continue the ceremony. However, I certainly don't think we
> would
> >> lose much/any test coverage if we stopped their use.
> >>
> >> Andrew
> >>
> >> On Tue, Aug 22, 2023 at 4:54 AM Antoine Pitrou <an...@python.org>
> wrote:
> >>
> >>>
> >>> Hello,
> >>>
> >>> Abiding by the Apache Software Foundation's guidelines, every Arrow
> >>> release is voted on and requires at least 3 "binding" votes to be
> approved.
> >>>
> >>> Also, every Arrow release vote is accompanied by a little ceremonial
> >>> where contributors and core developers run a release verification
> script
> >>> on their machine, wait for long minutes (sometimes an hour) and report
> >>> the results.
> >>>
> >>> This ceremonial has gone on for years, and it has not really been
> >>> questioned. Yet, it's not obvious to me what it is achieving exactly.
> >>> I've been here since 2018, but I don't really understand what the
> >>> verification script is testing for, or, more importantly, *why* it is
> >>> testing for what it is testing. I'm probably not the only one?
> >>>
> >>> I would like to bring the following points:
> >>>
> >>> * platform compatibility is (supposed to be) exercised on Continuous
> >>> Integration; there is no understandable reason why it should be
> >>> ceremoniously tested on each developer's machine before the release
> >>>
> >>> * just before a release is probably the wrong time to be testing
> >>> platform compatibility, and fixing compatibility bugs (though, of
> >>> course, it might still be better than not noticing?)
> >>>
> >>> * home environments are unstable, and not all developers run the
> >>> verification script for each release, so each release is actually
> >>> verified on different, uncontrolled, platforms
> >>>
> >>> * as for sanity checks on binary packages, GPG signatures, etc., there
> >>> shouldn't be any need to run them on multiple different machines, as
> >>> they are (should be?) entirely deterministic and platform-agnostic
> >>>
> >>> * maintaining the verification scripts is a thankless task, in part due
> >>> to their nature (they need to track and mirror changes made in each
> >>> implementation's build chain), in part due to implementation choices
> >>>
> >>> * due to the existence of the verification scripts, the release vote is
> >>> focussed on getting the script to run successfully (a very contextual
> >>> and non-reproducible result), rather than the actual *contents* of the
> >>> release
> >>>
> >>> The most positive thing I can personally say about the verification
> >>> scripts is that they *may* help us trust the release is not broken? But
> >>> that's a very unqualified statement, and is very close to
> cargo-culting.
> >>>
> >>> Regards
> >>>
> >>> Antoine.
> >>>
>

Re: [Discuss] Do we need a release verification script?

[Antoine Pitrou <an...@python.org>]

If the main impetus for the verification script is to comply with ASF 
requirements, probably the script can be made much simpler, such as just 
verify the GPG signatures are valid? Or perhaps this can be achieved 
without a script at all.

The irony is that, however complex, our verification script doesn't seem 
to check the actual ASF requirements on artifacts.

For example, we don't check that """a source release SHOULD not contain 
compiled code""" (also, what does "compiled code" mean? does generated 
code, e.g. by the Flatbuffers compiler, apply?)

Checking that the release """MUST be sufficient for a user to build and 
test the release provided they have access to the appropriate platform 
and tools""" is ill-defined and potentially tautologic, because the 
"appropriate platform and tools" is too imprecise and contextual (can 
the "appropriate platform and tools" contain a bunch of proprietary 
software that gets linked with the binaries? Well, it can, otherwise you 
can't build on Windows).

Regards

Antoine.



Le 22/08/2023 à 12:31, Raúl Cumplido a écrit :
> Hi,
> 
> I do agree that currently verifying the release locally provides
> little benefit for the effort we have to put in but I thought this was
> required as per Apache policy:
> https://www.apache.org/legal/release-policy.html#release-approval
> 
> Copying the important bit:
> """
> Before casting +1 binding votes, individuals are REQUIRED to download
> all signed source code packages onto their own hardware, verify that
> they meet all requirements of ASF policy on releases as described
> below, validate all cryptographic signatures, compile as provided, and
> test the result on their own platform.
> """
> 
> I also think we should try and challenge those.
> 
> In the past we have identified some minor issues on the local
> verification but I don't recall any of them being blockers for the
> release.
> 
> Thanks,
> Raúl
> 
> El mar, 22 ago 2023 a las 11:46, Andrew Lamb (<al...@influxdata.com>) escribió:
>>
>> The Rust arrow implementation (arrow-rs) and DataFusion also use release
>> verification scripts, mostly inherited from when they were split from the
>> mono repo. They have found issues from time to time, for us, but those
>> issues are often not platform related and have not been release blockers.
>>
>> Thankfully for Rust, the verification scripts don't need much maintenance
>> so we just continue the ceremony. However, I certainly don't think we would
>> lose much/any test coverage if we stopped their use.
>>
>> Andrew
>>
>> On Tue, Aug 22, 2023 at 4:54 AM Antoine Pitrou <an...@python.org> wrote:
>>
>>>
>>> Hello,
>>>
>>> Abiding by the Apache Software Foundation's guidelines, every Arrow
>>> release is voted on and requires at least 3 "binding" votes to be approved.
>>>
>>> Also, every Arrow release vote is accompanied by a little ceremonial
>>> where contributors and core developers run a release verification script
>>> on their machine, wait for long minutes (sometimes an hour) and report
>>> the results.
>>>
>>> This ceremonial has gone on for years, and it has not really been
>>> questioned. Yet, it's not obvious to me what it is achieving exactly.
>>> I've been here since 2018, but I don't really understand what the
>>> verification script is testing for, or, more importantly, *why* it is
>>> testing for what it is testing. I'm probably not the only one?
>>>
>>> I would like to bring the following points:
>>>
>>> * platform compatibility is (supposed to be) exercised on Continuous
>>> Integration; there is no understandable reason why it should be
>>> ceremoniously tested on each developer's machine before the release
>>>
>>> * just before a release is probably the wrong time to be testing
>>> platform compatibility, and fixing compatibility bugs (though, of
>>> course, it might still be better than not noticing?)
>>>
>>> * home environments are unstable, and not all developers run the
>>> verification script for each release, so each release is actually
>>> verified on different, uncontrolled, platforms
>>>
>>> * as for sanity checks on binary packages, GPG signatures, etc., there
>>> shouldn't be any need to run them on multiple different machines, as
>>> they are (should be?) entirely deterministic and platform-agnostic
>>>
>>> * maintaining the verification scripts is a thankless task, in part due
>>> to their nature (they need to track and mirror changes made in each
>>> implementation's build chain), in part due to implementation choices
>>>
>>> * due to the existence of the verification scripts, the release vote is
>>> focussed on getting the script to run successfully (a very contextual
>>> and non-reproducible result), rather than the actual *contents* of the
>>> release
>>>
>>> The most positive thing I can personally say about the verification
>>> scripts is that they *may* help us trust the release is not broken? But
>>> that's a very unqualified statement, and is very close to cargo-culting.
>>>
>>> Regards
>>>
>>> Antoine.
>>>

Re: [Discuss] Do we need a release verification script?

[Jacob Wujciak-Jens <ja...@voltrondata.com.INVALID>]

I am also in favour of changes to the verification scripts. This is a bit
of a tangent but has direct implications for our entire release process so
I will bring it up first. While re-reading the general release policy I
came across this new addition (made in late july):

> All supplied packages MUST be cryptographically signed with a detached
signature. It MUST be signed by either the Release Manager or __the
automated release infrastructure__

The details are described in [1] but we now have a clear way to completely
automate everything outside the vote.Combined with the current discussion
and the previous discussions about release frequency, versioning and the
need for a support policy, a re-thinking of our entire process might be in
order.

[1]: https://infra.apache.org/release-signing.html#automated-release-signing

On Tue, Aug 22, 2023 at 12:33 PM Raúl Cumplido <ra...@gmail.com>
wrote:

> Hi,
>
> I do agree that currently verifying the release locally provides
> little benefit for the effort we have to put in but I thought this was
> required as per Apache policy:
> https://www.apache.org/legal/release-policy.html#release-approval
>
> Copying the important bit:
> """
> Before casting +1 binding votes, individuals are REQUIRED to download
> all signed source code packages onto their own hardware, verify that
> they meet all requirements of ASF policy on releases as described
> below, validate all cryptographic signatures, compile as provided, and
> test the result on their own platform.
> """
>
> I also think we should try and challenge those.
>
> In the past we have identified some minor issues on the local
> verification but I don't recall any of them being blockers for the
> release.
>
> Thanks,
> Raúl
>
> El mar, 22 ago 2023 a las 11:46, Andrew Lamb (<al...@influxdata.com>)
> escribió:
> >
> > The Rust arrow implementation (arrow-rs) and DataFusion also use release
> > verification scripts, mostly inherited from when they were split from the
> > mono repo. They have found issues from time to time, for us, but those
> > issues are often not platform related and have not been release blockers.
> >
> > Thankfully for Rust, the verification scripts don't need much maintenance
> > so we just continue the ceremony. However, I certainly don't think we
> would
> > lose much/any test coverage if we stopped their use.
> >
> > Andrew
> >
> > On Tue, Aug 22, 2023 at 4:54 AM Antoine Pitrou <an...@python.org>
> wrote:
> >
> > >
> > > Hello,
> > >
> > > Abiding by the Apache Software Foundation's guidelines, every Arrow
> > > release is voted on and requires at least 3 "binding" votes to be
> approved.
> > >
> > > Also, every Arrow release vote is accompanied by a little ceremonial
> > > where contributors and core developers run a release verification
> script
> > > on their machine, wait for long minutes (sometimes an hour) and report
> > > the results.
> > >
> > > This ceremonial has gone on for years, and it has not really been
> > > questioned. Yet, it's not obvious to me what it is achieving exactly.
> > > I've been here since 2018, but I don't really understand what the
> > > verification script is testing for, or, more importantly, *why* it is
> > > testing for what it is testing. I'm probably not the only one?
> > >
> > > I would like to bring the following points:
> > >
> > > * platform compatibility is (supposed to be) exercised on Continuous
> > > Integration; there is no understandable reason why it should be
> > > ceremoniously tested on each developer's machine before the release
> > >
> > > * just before a release is probably the wrong time to be testing
> > > platform compatibility, and fixing compatibility bugs (though, of
> > > course, it might still be better than not noticing?)
> > >
> > > * home environments are unstable, and not all developers run the
> > > verification script for each release, so each release is actually
> > > verified on different, uncontrolled, platforms
> > >
> > > * as for sanity checks on binary packages, GPG signatures, etc., there
> > > shouldn't be any need to run them on multiple different machines, as
> > > they are (should be?) entirely deterministic and platform-agnostic
> > >
> > > * maintaining the verification scripts is a thankless task, in part due
> > > to their nature (they need to track and mirror changes made in each
> > > implementation's build chain), in part due to implementation choices
> > >
> > > * due to the existence of the verification scripts, the release vote is
> > > focussed on getting the script to run successfully (a very contextual
> > > and non-reproducible result), rather than the actual *contents* of the
> > > release
> > >
> > > The most positive thing I can personally say about the verification
> > > scripts is that they *may* help us trust the release is not broken? But
> > > that's a very unqualified statement, and is very close to
> cargo-culting.
> > >
> > > Regards
> > >
> > > Antoine.
> > >
>

Re: [Discuss] Do we need a release verification script?

[Raúl Cumplido <ra...@gmail.com>]

Hi,

I do agree that currently verifying the release locally provides
little benefit for the effort we have to put in but I thought this was
required as per Apache policy:
https://www.apache.org/legal/release-policy.html#release-approval

Copying the important bit:
"""
Before casting +1 binding votes, individuals are REQUIRED to download
all signed source code packages onto their own hardware, verify that
they meet all requirements of ASF policy on releases as described
below, validate all cryptographic signatures, compile as provided, and
test the result on their own platform.
"""

I also think we should try and challenge those.

In the past we have identified some minor issues on the local
verification but I don't recall any of them being blockers for the
release.

Thanks,
Raúl

El mar, 22 ago 2023 a las 11:46, Andrew Lamb (<al...@influxdata.com>) escribió:
>
> The Rust arrow implementation (arrow-rs) and DataFusion also use release
> verification scripts, mostly inherited from when they were split from the
> mono repo. They have found issues from time to time, for us, but those
> issues are often not platform related and have not been release blockers.
>
> Thankfully for Rust, the verification scripts don't need much maintenance
> so we just continue the ceremony. However, I certainly don't think we would
> lose much/any test coverage if we stopped their use.
>
> Andrew
>
> On Tue, Aug 22, 2023 at 4:54 AM Antoine Pitrou <an...@python.org> wrote:
>
> >
> > Hello,
> >
> > Abiding by the Apache Software Foundation's guidelines, every Arrow
> > release is voted on and requires at least 3 "binding" votes to be approved.
> >
> > Also, every Arrow release vote is accompanied by a little ceremonial
> > where contributors and core developers run a release verification script
> > on their machine, wait for long minutes (sometimes an hour) and report
> > the results.
> >
> > This ceremonial has gone on for years, and it has not really been
> > questioned. Yet, it's not obvious to me what it is achieving exactly.
> > I've been here since 2018, but I don't really understand what the
> > verification script is testing for, or, more importantly, *why* it is
> > testing for what it is testing. I'm probably not the only one?
> >
> > I would like to bring the following points:
> >
> > * platform compatibility is (supposed to be) exercised on Continuous
> > Integration; there is no understandable reason why it should be
> > ceremoniously tested on each developer's machine before the release
> >
> > * just before a release is probably the wrong time to be testing
> > platform compatibility, and fixing compatibility bugs (though, of
> > course, it might still be better than not noticing?)
> >
> > * home environments are unstable, and not all developers run the
> > verification script for each release, so each release is actually
> > verified on different, uncontrolled, platforms
> >
> > * as for sanity checks on binary packages, GPG signatures, etc., there
> > shouldn't be any need to run them on multiple different machines, as
> > they are (should be?) entirely deterministic and platform-agnostic
> >
> > * maintaining the verification scripts is a thankless task, in part due
> > to their nature (they need to track and mirror changes made in each
> > implementation's build chain), in part due to implementation choices
> >
> > * due to the existence of the verification scripts, the release vote is
> > focussed on getting the script to run successfully (a very contextual
> > and non-reproducible result), rather than the actual *contents* of the
> > release
> >
> > The most positive thing I can personally say about the verification
> > scripts is that they *may* help us trust the release is not broken? But
> > that's a very unqualified statement, and is very close to cargo-culting.
> >
> > Regards
> >
> > Antoine.
> >

Re: [Discuss] Do we need a release verification script?

[Andrew Lamb <al...@influxdata.com>]

The Rust arrow implementation (arrow-rs) and DataFusion also use release
verification scripts, mostly inherited from when they were split from the
mono repo. They have found issues from time to time, for us, but those
issues are often not platform related and have not been release blockers.

Thankfully for Rust, the verification scripts don't need much maintenance
so we just continue the ceremony. However, I certainly don't think we would
lose much/any test coverage if we stopped their use.

Andrew

On Tue, Aug 22, 2023 at 4:54 AM Antoine Pitrou <an...@python.org> wrote:

>
> Hello,
>
> Abiding by the Apache Software Foundation's guidelines, every Arrow
> release is voted on and requires at least 3 "binding" votes to be approved.
>
> Also, every Arrow release vote is accompanied by a little ceremonial
> where contributors and core developers run a release verification script
> on their machine, wait for long minutes (sometimes an hour) and report
> the results.
>
> This ceremonial has gone on for years, and it has not really been
> questioned. Yet, it's not obvious to me what it is achieving exactly.
> I've been here since 2018, but I don't really understand what the
> verification script is testing for, or, more importantly, *why* it is
> testing for what it is testing. I'm probably not the only one?
>
> I would like to bring the following points:
>
> * platform compatibility is (supposed to be) exercised on Continuous
> Integration; there is no understandable reason why it should be
> ceremoniously tested on each developer's machine before the release
>
> * just before a release is probably the wrong time to be testing
> platform compatibility, and fixing compatibility bugs (though, of
> course, it might still be better than not noticing?)
>
> * home environments are unstable, and not all developers run the
> verification script for each release, so each release is actually
> verified on different, uncontrolled, platforms
>
> * as for sanity checks on binary packages, GPG signatures, etc., there
> shouldn't be any need to run them on multiple different machines, as
> they are (should be?) entirely deterministic and platform-agnostic
>
> * maintaining the verification scripts is a thankless task, in part due
> to their nature (they need to track and mirror changes made in each
> implementation's build chain), in part due to implementation choices
>
> * due to the existence of the verification scripts, the release vote is
> focussed on getting the script to run successfully (a very contextual
> and non-reproducible result), rather than the actual *contents* of the
> release
>
> The most positive thing I can personally say about the verification
> scripts is that they *may* help us trust the release is not broken? But
> that's a very unqualified statement, and is very close to cargo-culting.
>
> Regards
>
> Antoine.
>