[DISCUSS] Helper tool / checklist to facilitate RC testing coverage?

[Misty Stanley-Jones <mi...@apache.org>]

I keep feeling like we need a checklist for all the different things that
should be tested when people vote on a release candidate. Not everyone
needs to do all the checks, but probably at least someone should be doing
each of the following (and maybe things I am forgetting):

- Binary .tar.gz and .zip extract
- Source .tar.gz and .zip extract
- All md5sums match
- The source .tar.gz matches the hash on the Git repo
- Everything is signed correctly
- The source compiles into exactly the binary
- The source compiles with both OpenJDK and Oracle JDK (?)
- The binaries from the binary .tar.gz and .zip run as expected
- The binaries built from the source .tar.gz and .zip run as expected
- The standard tests all run and pass, with the exception of known flakies
- LTT and PE tools run and look good


Would it be helpful to have some kind of dashboard where people could "sign
up" to test different things on that checklist, and maybe to require that
all the checks have been done by at least someone before the RC can be
released? Maybe it could be something silly like a Google Form, even. Or a
spreadsheet.

I think it might also be less intimidating for newbies who have never voted
on an RC before (and we really really want as many people as possible to
participate in the voting process).

What do y'all think?

Misty

Re: [DISCUSS] Helper tool / checklist to facilitate RC testing coverage?

[Dima Spivak <di...@apache.org>]

Big +1. I opened HBASE-16559 to start working on this last year, but then
life happened.

-Dima

On Mon, Sep 18, 2017 at 11:53 AM, Ted Yu <yu...@gmail.com> wrote:

> bq. have some kind of dashboard where people could "sign up" to test
> different things on that checklist
>
> This makes sense.
> +1
>
>
> On Mon, Sep 18, 2017 at 11:49 AM, Misty Stanley-Jones <mi...@apache.org>
> wrote:
>
> > I keep feeling like we need a checklist for all the different things that
> > should be tested when people vote on a release candidate. Not everyone
> > needs to do all the checks, but probably at least someone should be doing
> > each of the following (and maybe things I am forgetting):
> >
> > - Binary .tar.gz and .zip extract
> > - Source .tar.gz and .zip extract
> > - All md5sums match
> > - The source .tar.gz matches the hash on the Git repo
> > - Everything is signed correctly
> > - The source compiles into exactly the binary
> > - The source compiles with both OpenJDK and Oracle JDK (?)
> > - The binaries from the binary .tar.gz and .zip run as expected
> > - The binaries built from the source .tar.gz and .zip run as expected
> > - The standard tests all run and pass, with the exception of known
> flakies
> > - LTT and PE tools run and look good
> >
> >
> > Would it be helpful to have some kind of dashboard where people could
> "sign
> > up" to test different things on that checklist, and maybe to require that
> > all the checks have been done by at least someone before the RC can be
> > released? Maybe it could be something silly like a Google Form, even. Or
> a
> > spreadsheet.
> >
> > I think it might also be less intimidating for newbies who have never
> voted
> > on an RC before (and we really really want as many people as possible to
> > participate in the voting process).
> >
> > What do y'all think?
> >
> > Misty
> >
>

Re: [DISCUSS] Helper tool / checklist to facilitate RC testing coverage?

[Ted Yu <yu...@gmail.com>]

bq. have some kind of dashboard where people could "sign up" to test
different things on that checklist

This makes sense.
+1


On Mon, Sep 18, 2017 at 11:49 AM, Misty Stanley-Jones <mi...@apache.org>
wrote:

> I keep feeling like we need a checklist for all the different things that
> should be tested when people vote on a release candidate. Not everyone
> needs to do all the checks, but probably at least someone should be doing
> each of the following (and maybe things I am forgetting):
>
> - Binary .tar.gz and .zip extract
> - Source .tar.gz and .zip extract
> - All md5sums match
> - The source .tar.gz matches the hash on the Git repo
> - Everything is signed correctly
> - The source compiles into exactly the binary
> - The source compiles with both OpenJDK and Oracle JDK (?)
> - The binaries from the binary .tar.gz and .zip run as expected
> - The binaries built from the source .tar.gz and .zip run as expected
> - The standard tests all run and pass, with the exception of known flakies
> - LTT and PE tools run and look good
>
>
> Would it be helpful to have some kind of dashboard where people could "sign
> up" to test different things on that checklist, and maybe to require that
> all the checks have been done by at least someone before the RC can be
> released? Maybe it could be something silly like a Google Form, even. Or a
> spreadsheet.
>
> I think it might also be less intimidating for newbies who have never voted
> on an RC before (and we really really want as many people as possible to
> participate in the voting process).
>
> What do y'all think?
>
> Misty
>

Re: [DISCUSS] Helper tool / checklist to facilitate RC testing coverage?

[Umesh Agashe <ua...@cloudera.com>]

I think this is a good idea. Having a list documented itself is big plus.
Explicitly listing Required/ Mandatory checks and Optional checks for
release type (e.g. alpha, beta etc) will help too.

On Mon, Sep 18, 2017 at 1:10 PM, Andrew Purtell <ap...@apache.org> wrote:

> I also think it's fair to make allowances for folks who want to test the RC
> but might be short on personal time. So there should be a list of essential
> items (like sigs, sums, shas/tags) and then a list of optional items (ltt,
> pe, itbll, etc.)
>
> On Mon, Sep 18, 2017 at 12:52 PM, Stack <st...@duboce.net> wrote:
>
> > On Mon, Sep 18, 2017 at 11:49 AM, Misty Stanley-Jones <mi...@apache.org>
> > wrote:
> >
> > > I keep feeling like we need a checklist for all the different things
> that
> > > should be tested when people vote on a release candidate. Not everyone
> > > needs to do all the checks, but probably at least someone should be
> doing
> > > each of the following (and maybe things I am forgetting):
> > >
> > > - Binary .tar.gz and .zip extract
> > > - Source .tar.gz and .zip extract
> > > - All md5sums match
> > > - The source .tar.gz matches the hash on the Git repo
> > > - Everything is signed correctly
> > > - The source compiles into exactly the binary
> > > - The source compiles with both OpenJDK and Oracle JDK (?)
> > > - The binaries from the binary .tar.gz and .zip run as expected
> > > - The binaries built from the source .tar.gz and .zip run as expected
> > > - The standard tests all run and pass, with the exception of known
> > flakies
> > > - LTT and PE tools run and look good
> > >
> > >
> > > Would it be helpful to have some kind of dashboard where people could
> > "sign
> > > up" to test different things on that checklist, and maybe to require
> that
> > > all the checks have been done by at least someone before the RC can be
> > > released? Maybe it could be something silly like a Google Form, even.
> Or
> > a
> > > spreadsheet.
> > >
> > > I think it might also be less intimidating for newbies who have never
> > voted
> > > on an RC before (and we really really want as many people as possible
> to
> > > participate in the voting process).
> > >
> > > What do y'all think?
> > >
> > > Misty
> > >
> >
> >
> > Sounds great.
> >
> > St.Ack
> >
> > P.S.Here is a link to an old email that points at a LarsH doc. which
> cribs
> > from an old Dave Wang wiki page that tried to do similar:
> > http://search-hadoop.com/m/HBase/YGbb22RVtnt76g?subj=Re+Releases+tests+
> >
>
>
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
>    - A23, Crosstalk
>

Re: [DISCUSS] Helper tool / checklist to facilitate RC testing coverage?

[Mike Drob <md...@apache.org>]

One thing that I've seen in successful other communities is that when folks
run the long running tests they also document the specifics of the
configuration that they used. Things like your java version, operating
system, EC2 instance type. Then this all gets fed into a table published
with the release notes.

I think it does a lot to give users confidence in the releases when they
can see that there has been testing on a variety of platforms with a
variety of settings. It's also a super rough way to catch performance
regressions since the release notes will contain that ITBLL ran for 24
hours with X entries on Y machines and that can be compared across versions.

Mike

On Mon, Sep 18, 2017 at 3:10 PM, Andrew Purtell <ap...@apache.org> wrote:

> I also think it's fair to make allowances for folks who want to test the RC
> but might be short on personal time. So there should be a list of essential
> items (like sigs, sums, shas/tags) and then a list of optional items (ltt,
> pe, itbll, etc.)
>
> On Mon, Sep 18, 2017 at 12:52 PM, Stack <st...@duboce.net> wrote:
>
> > On Mon, Sep 18, 2017 at 11:49 AM, Misty Stanley-Jones <mi...@apache.org>
> > wrote:
> >
> > > I keep feeling like we need a checklist for all the different things
> that
> > > should be tested when people vote on a release candidate. Not everyone
> > > needs to do all the checks, but probably at least someone should be
> doing
> > > each of the following (and maybe things I am forgetting):
> > >
> > > - Binary .tar.gz and .zip extract
> > > - Source .tar.gz and .zip extract
> > > - All md5sums match
> > > - The source .tar.gz matches the hash on the Git repo
> > > - Everything is signed correctly
> > > - The source compiles into exactly the binary
> > > - The source compiles with both OpenJDK and Oracle JDK (?)
> > > - The binaries from the binary .tar.gz and .zip run as expected
> > > - The binaries built from the source .tar.gz and .zip run as expected
> > > - The standard tests all run and pass, with the exception of known
> > flakies
> > > - LTT and PE tools run and look good
> > >
> > >
> > > Would it be helpful to have some kind of dashboard where people could
> > "sign
> > > up" to test different things on that checklist, and maybe to require
> that
> > > all the checks have been done by at least someone before the RC can be
> > > released? Maybe it could be something silly like a Google Form, even.
> Or
> > a
> > > spreadsheet.
> > >
> > > I think it might also be less intimidating for newbies who have never
> > voted
> > > on an RC before (and we really really want as many people as possible
> to
> > > participate in the voting process).
> > >
> > > What do y'all think?
> > >
> > > Misty
> > >
> >
> >
> > Sounds great.
> >
> > St.Ack
> >
> > P.S.Here is a link to an old email that points at a LarsH doc. which
> cribs
> > from an old Dave Wang wiki page that tried to do similar:
> > http://search-hadoop.com/m/HBase/YGbb22RVtnt76g?subj=Re+Releases+tests+
> >
>
>
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
>    - A23, Crosstalk
>

Re: [DISCUSS] Helper tool / checklist to facilitate RC testing coverage?

[Andrew Purtell <ap...@apache.org>]

I also think it's fair to make allowances for folks who want to test the RC
but might be short on personal time. So there should be a list of essential
items (like sigs, sums, shas/tags) and then a list of optional items (ltt,
pe, itbll, etc.)

On Mon, Sep 18, 2017 at 12:52 PM, Stack <st...@duboce.net> wrote:

> On Mon, Sep 18, 2017 at 11:49 AM, Misty Stanley-Jones <mi...@apache.org>
> wrote:
>
> > I keep feeling like we need a checklist for all the different things that
> > should be tested when people vote on a release candidate. Not everyone
> > needs to do all the checks, but probably at least someone should be doing
> > each of the following (and maybe things I am forgetting):
> >
> > - Binary .tar.gz and .zip extract
> > - Source .tar.gz and .zip extract
> > - All md5sums match
> > - The source .tar.gz matches the hash on the Git repo
> > - Everything is signed correctly
> > - The source compiles into exactly the binary
> > - The source compiles with both OpenJDK and Oracle JDK (?)
> > - The binaries from the binary .tar.gz and .zip run as expected
> > - The binaries built from the source .tar.gz and .zip run as expected
> > - The standard tests all run and pass, with the exception of known
> flakies
> > - LTT and PE tools run and look good
> >
> >
> > Would it be helpful to have some kind of dashboard where people could
> "sign
> > up" to test different things on that checklist, and maybe to require that
> > all the checks have been done by at least someone before the RC can be
> > released? Maybe it could be something silly like a Google Form, even. Or
> a
> > spreadsheet.
> >
> > I think it might also be less intimidating for newbies who have never
> voted
> > on an RC before (and we really really want as many people as possible to
> > participate in the voting process).
> >
> > What do y'all think?
> >
> > Misty
> >
>
>
> Sounds great.
>
> St.Ack
>
> P.S.Here is a link to an old email that points at a LarsH doc. which cribs
> from an old Dave Wang wiki page that tried to do similar:
> http://search-hadoop.com/m/HBase/YGbb22RVtnt76g?subj=Re+Releases+tests+
>



-- 
Best regards,
Andrew

Words like orphans lost among the crosstalk, meaning torn from truth's
decrepit hands
   - A23, Crosstalk

Re: [DISCUSS] Helper tool / checklist to facilitate RC testing coverage?

[Stack <st...@duboce.net>]

On Mon, Sep 18, 2017 at 11:49 AM, Misty Stanley-Jones <mi...@apache.org>
wrote:

> I keep feeling like we need a checklist for all the different things that
> should be tested when people vote on a release candidate. Not everyone
> needs to do all the checks, but probably at least someone should be doing
> each of the following (and maybe things I am forgetting):
>
> - Binary .tar.gz and .zip extract
> - Source .tar.gz and .zip extract
> - All md5sums match
> - The source .tar.gz matches the hash on the Git repo
> - Everything is signed correctly
> - The source compiles into exactly the binary
> - The source compiles with both OpenJDK and Oracle JDK (?)
> - The binaries from the binary .tar.gz and .zip run as expected
> - The binaries built from the source .tar.gz and .zip run as expected
> - The standard tests all run and pass, with the exception of known flakies
> - LTT and PE tools run and look good
>
>
> Would it be helpful to have some kind of dashboard where people could "sign
> up" to test different things on that checklist, and maybe to require that
> all the checks have been done by at least someone before the RC can be
> released? Maybe it could be something silly like a Google Form, even. Or a
> spreadsheet.
>
> I think it might also be less intimidating for newbies who have never voted
> on an RC before (and we really really want as many people as possible to
> participate in the voting process).
>
> What do y'all think?
>
> Misty
>


Sounds great.

St.Ack

P.S.Here is a link to an old email that points at a LarsH doc. which cribs
from an old Dave Wang wiki page that tried to do similar:
http://search-hadoop.com/m/HBase/YGbb22RVtnt76g?subj=Re+Releases+tests+