Reduced testing burden for bug-fix releases

[Josh Elser <jo...@gmail.com>]

As we're starting to consider 1.5.2 and 1.6.1 coming out in the near 
future, I want to revisit a discussion[1] I started at the end of April 
regarding the "testing burden" that is currently set forth in our 
release document[2].

What I'm proposing is to modify the language of the release document to 
be explicit about the amount of testing needed. For bug-fix, "minor" 
releases (e.g. 1.5.2 and 1.6.1), the 7 days of testing using continuous 
ingest and randomwalk (with and without agitation) will be clearly 
defined as "may" instead of "should" or "must" language. If the 
resources are available, it is recommended that some longer, 
multi-process/node test is run against the release candidate; however, 
it is not required and should not prevent us from making the minor release.

I will also include language that strongly recommends that the changes 
included in the "minor" release be vetted/reviewed as a way to mitigate 
the risk of shipping new regressions.

I am not recommending that the language be changed for "major" releases 
(e.g. 1.7.0 and 2.0.0) as these releases still imply significant new 
features or internal changes.

Unless someone informs me otherwise, I will treat this as a normal 
lazy-consensus approval. Assuming we move closer to "proper" semantic 
versioning for 2.0.0, I believe these updated guidelines will change 
again. I do however think there is merit in making this change now so 
that we can get the good bugs that we've fixed out to our users.

Let me know what you think. I will wait, at least, the prescribed three 
days before changing any thing.

- Josh

[1] 
http://mail-archives.apache.org/mod_mbox/accumulo-dev/201404.mbox/%3C535931A7.30605%40gmail.com%3E
[2] http://accumulo.apache.org/governance/releasing.html

Re: Reduced testing burden for bug-fix releases

[Christopher <ct...@apache.org>]

+1 for reduced testing burden for bugfixes and a change in the guidelines
to reflect that.


--
Christopher L Tubbs II
http://gravatar.com/ctubbsii


On Thu, Jun 19, 2014 at 3:13 PM, Josh Elser <jo...@gmail.com> wrote:

> As we're starting to consider 1.5.2 and 1.6.1 coming out in the near
> future, I want to revisit a discussion[1] I started at the end of April
> regarding the "testing burden" that is currently set forth in our release
> document[2].
>
> What I'm proposing is to modify the language of the release document to be
> explicit about the amount of testing needed. For bug-fix, "minor" releases
> (e.g. 1.5.2 and 1.6.1), the 7 days of testing using continuous ingest and
> randomwalk (with and without agitation) will be clearly defined as "may"
> instead of "should" or "must" language. If the resources are available, it
> is recommended that some longer, multi-process/node test is run against the
> release candidate; however, it is not required and should not prevent us
> from making the minor release.
>
> I will also include language that strongly recommends that the changes
> included in the "minor" release be vetted/reviewed as a way to mitigate the
> risk of shipping new regressions.
>
> I am not recommending that the language be changed for "major" releases
> (e.g. 1.7.0 and 2.0.0) as these releases still imply significant new
> features or internal changes.
>
> Unless someone informs me otherwise, I will treat this as a normal
> lazy-consensus approval. Assuming we move closer to "proper" semantic
> versioning for 2.0.0, I believe these updated guidelines will change again.
> I do however think there is merit in making this change now so that we can
> get the good bugs that we've fixed out to our users.
>
> Let me know what you think. I will wait, at least, the prescribed three
> days before changing any thing.
>
> - Josh
>
> [1] http://mail-archives.apache.org/mod_mbox/accumulo-dev/
> 201404.mbox/%3C535931A7.30605%40gmail.com%3E
> [2] http://accumulo.apache.org/governance/releasing.html
>

Re: Reduced testing burden for bug-fix releases

[Josh Elser <jo...@gmail.com>]

On 6/23/14, 4:31 PM, Sean Busbey wrote:
>>> 3) As a point of principle, David's veto was completely valid as it stood.
>>> >>While we are individuals it's perfectly reasonable for the PMC to set
>>> >>requirements on what goes into a release we sign off on, even if the PMC
>>> >>members may not always have ready access to the resources needed to meet
>>> >>that standard.
>>> >>
>>> >>There's no reason David should have to volunteer resources to back up his
>>> >>veto, especially when it was merely calling for a continuation of the
>>> >>standard we already had set.
>>> >>
>> >
>> >That's why I said "Unless you are willing to supply the funds to pay to
>> >use some resources, I don't feel like this is a valid -1." If he, or
>> >anyone, is willing provide general resources for testing, that's a
>> >different story. Given his response, I assume that is not the case.
>> >
>> >
> But that's the opposite of what I just said. The opposition and the ability
> to find funds are not strongly coupled.

Oh it is. I apologize, I misread your initial statement.

> The lack of funds would hopefully be a convincing argument to try to sway
> someone that we should lower the testing barrier, but they aren't a
> legitimate excuse for invalidating their vote. What if Hypothetical David
> wanted to do fund raising to get resources together? Would you decide on a
> deadline that would allow his vote to be legitimate or not?

I wouldn't have a problem with anyone trying to raise funds for test 
resources, within reason. But, unless someone is actually planning to do 
this, it probably isn't much more than a hypothetical argument that we 
need to spend time discussing.

> The basis for a veto need merely be technical. "We've done this level of
> testing before and it protects our users. We should continue doing it." is
> a perfectly reasonable justification. (I admit I am taking some liberty in
> making parts of David's previous concern more explicit)
>
> BTW, situations like this are a part of why Majority Vote for governance
> decisions are my preference. While I fully agree with David's ability to
> veto I also agree that the community should be able to override him if no
> funding source could be found.

Re: Reduced testing burden for bug-fix releases

[Sean Busbey <bu...@cloudera.com>]

inline


On Mon, Jun 23, 2014 at 2:52 PM, Josh Elser <jo...@gmail.com> wrote:

> On 6/23/14, 3:31 PM, Sean Busbey wrote:
>
>>
>>
>> 2) We really could have used a [VOTE] on changing 1.x versions to use
>> bugfix for the last number. It would be an opportunity to formally adopt
>> our release governance docs into the PMC bylaws by reference.
>>
>>
> I don't see this as a huge issue since everyone seemed to be in agreement
> on it. Writing it down for reference would be good for future discussions
> on the subject.
>
>

I don't think everyone is in agreement on it. To wit, I remain opposed to
it.



>
>
>> 3) As a point of principle, David's veto was completely valid as it stood.
>> While we are individuals it's perfectly reasonable for the PMC to set
>> requirements on what goes into a release we sign off on, even if the PMC
>> members may not always have ready access to the resources needed to meet
>> that standard.
>>
>> There's no reason David should have to volunteer resources to back up his
>> veto, especially when it was merely calling for a continuation of the
>> standard we already had set.
>>
>
> That's why I said "Unless you are willing to supply the funds to pay to
> use some resources, I don't feel like this is a valid -1." If he, or
> anyone, is willing provide general resources for testing, that's a
> different story. Given his response, I assume that is not the case.
>
>
But that's the opposite of what I just said. The opposition and the ability
to find funds are not strongly coupled.

The lack of funds would hopefully be a convincing argument to try to sway
someone that we should lower the testing barrier, but they aren't a
legitimate excuse for invalidating their vote. What if Hypothetical David
wanted to do fund raising to get resources together? Would you decide on a
deadline that would allow his vote to be legitimate or not?

The basis for a veto need merely be technical. "We've done this level of
testing before and it protects our users. We should continue doing it." is
a perfectly reasonable justification. (I admit I am taking some liberty in
making parts of David's previous concern more explicit)

BTW, situations like this are a part of why Majority Vote for governance
decisions are my preference. While I fully agree with David's ability to
veto I also agree that the community should be able to override him if no
funding source could be found.



>
>
>> 4) While I'm not in a position to obligate Cloudera, the team I'm on
>> currently has been allocated sufficient resources to meet the current
>> testing standards for a release and I have no reason to believe we won't
>> have said resources when future release windows happen.
>>
>
> This would require time from you, Mike or Bill H, yes? While having some
> dedicated resources is nice, I'm worried about relying on other people (who
> have their own obligations) to fulfill the release manager's obligations.
>
> I think that gets into the details which the release manager can
> coordinate and other devs can express their concerns via the normal release
> voting process.
>
>
The release manager's only responsibility is to ensure the testing gets
done, not to do the testing themselves. That's why we're a community: so we
can rely on each other.

If a particular release manager needs help making sure the coverage
happens, they should just ask for that help when making the release plan.
Mike, Bill, and I are all capable of getting approval to block out time in
support of things that are good for Apache Accumulo.

(I am not actually sure it would require time from Mike, Bill, or me;
circumstances happen and "it depends." It's certainly easier for Mike,
Bill, or me to do it.)

-- 
Sean

Re: Reduced testing burden for bug-fix releases

[Josh Elser <jo...@gmail.com>]

On 6/23/14, 3:31 PM, Sean Busbey wrote:
> I realize we're past window here, but a few things:

It's ok. I only remembered about this early today.

>
> 1) I'm concerned about changing our testing standards prior to getting some
> practice in on holding to tighter bounds on what can go into a particular
> kind of fix number. However, I generally like the idea of just relying on
> people voting -1 for insufficient testing.
>
>
> 2) We really could have used a [VOTE] on changing 1.x versions to use
> bugfix for the last number. It would be an opportunity to formally adopt
> our release governance docs into the PMC bylaws by reference.
>

I don't see this as a huge issue since everyone seemed to be in 
agreement on it. Writing it down for reference would be good for future 
discussions on the subject.

>
> 3) As a point of principle, David's veto was completely valid as it stood.
> While we are individuals it's perfectly reasonable for the PMC to set
> requirements on what goes into a release we sign off on, even if the PMC
> members may not always have ready access to the resources needed to meet
> that standard.
>
> There's no reason David should have to volunteer resources to back up his
> veto, especially when it was merely calling for a continuation of the
> standard we already had set.

That's why I said "Unless you are willing to supply the funds to pay to 
use some resources, I don't feel like this is a valid -1." If he, or 
anyone, is willing provide general resources for testing, that's a 
different story. Given his response, I assume that is not the case.

>
> 4) While I'm not in a position to obligate Cloudera, the team I'm on
> currently has been allocated sufficient resources to meet the current
> testing standards for a release and I have no reason to believe we won't
> have said resources when future release windows happen.

This would require time from you, Mike or Bill H, yes? While having some 
dedicated resources is nice, I'm worried about relying on other people 
(who have their own obligations) to fulfill the release manager's 
obligations.

I think that gets into the details which the release manager can 
coordinate and other devs can express their concerns via the normal 
release voting process.

> -Sean
>
> On Thu, Jun 19, 2014 at 3:03 PM, David Medinets <da...@gmail.com>
> wrote:
>
>> +0 I'm changing my vote after some reconsideration. Having the ability
>> to vote -1 on a release as Keith mentioned is good enough for a
>> bug-fix release.
>>
>> On Thu, Jun 19, 2014 at 3:20 PM, David Medinets
>> <da...@gmail.com> wrote:
>>> -1 I hesitate to step into this discussion because I can't also step
>>> up and do the long-term testing even as I recommend that it must be
>>> done. There are at least four companies supporting Accumulo and
>>> contributing back to the project. Surely one of those companies can
>>> supply the resources to continue the existing test regimen? Is there
>>> some concern that those resources won't be available for the next
>>> release cycle?
>>>
>>> On Thu, Jun 19, 2014 at 3:13 PM, Josh Elser <jo...@gmail.com>
>> wrote:
>>>> As we're starting to consider 1.5.2 and 1.6.1 coming out in the near
>> future,
>>>> I want to revisit a discussion[1] I started at the end of April
>> regarding
>>>> the "testing burden" that is currently set forth in our release
>> document[2].
>>>>
>>>> What I'm proposing is to modify the language of the release document to
>> be
>>>> explicit about the amount of testing needed. For bug-fix, "minor"
>> releases
>>>> (e.g. 1.5.2 and 1.6.1), the 7 days of testing using continuous ingest
>> and
>>>> randomwalk (with and without agitation) will be clearly defined as "may"
>>>> instead of "should" or "must" language. If the resources are available,
>> it
>>>> is recommended that some longer, multi-process/node test is run against
>> the
>>>> release candidate; however, it is not required and should not prevent us
>>>> from making the minor release.
>>>>
>>>> I will also include language that strongly recommends that the changes
>>>> included in the "minor" release be vetted/reviewed as a way to mitigate
>> the
>>>> risk of shipping new regressions.
>>>>
>>>> I am not recommending that the language be changed for "major" releases
>>>> (e.g. 1.7.0 and 2.0.0) as these releases still imply significant new
>>>> features or internal changes.
>>>>
>>>> Unless someone informs me otherwise, I will treat this as a normal
>>>> lazy-consensus approval. Assuming we move closer to "proper" semantic
>>>> versioning for 2.0.0, I believe these updated guidelines will change
>> again.
>>>> I do however think there is merit in making this change now so that we
>> can
>>>> get the good bugs that we've fixed out to our users.
>>>>
>>>> Let me know what you think. I will wait, at least, the prescribed three
>> days
>>>> before changing any thing.
>>>>
>>>> - Josh
>>>>
>>>> [1]
>>>>
>> http://mail-archives.apache.org/mod_mbox/accumulo-dev/201404.mbox/%3C535931A7.30605%40gmail.com%3E
>>>> [2] http://accumulo.apache.org/governance/releasing.html
>>
>
>
>

Re: Reduced testing burden for bug-fix releases

[Sean Busbey <bu...@cloudera.com>]

I realize we're past window here, but a few things:


1) I'm concerned about changing our testing standards prior to getting some
practice in on holding to tighter bounds on what can go into a particular
kind of fix number. However, I generally like the idea of just relying on
people voting -1 for insufficient testing.


2) We really could have used a [VOTE] on changing 1.x versions to use
bugfix for the last number. It would be an opportunity to formally adopt
our release governance docs into the PMC bylaws by reference.


3) As a point of principle, David's veto was completely valid as it stood.
While we are individuals it's perfectly reasonable for the PMC to set
requirements on what goes into a release we sign off on, even if the PMC
members may not always have ready access to the resources needed to meet
that standard.

There's no reason David should have to volunteer resources to back up his
veto, especially when it was merely calling for a continuation of the
standard we already had set.

4) While I'm not in a position to obligate Cloudera, the team I'm on
currently has been allocated sufficient resources to meet the current
testing standards for a release and I have no reason to believe we won't
have said resources when future release windows happen.

-Sean

On Thu, Jun 19, 2014 at 3:03 PM, David Medinets <da...@gmail.com>
wrote:

> +0 I'm changing my vote after some reconsideration. Having the ability
> to vote -1 on a release as Keith mentioned is good enough for a
> bug-fix release.
>
> On Thu, Jun 19, 2014 at 3:20 PM, David Medinets
> <da...@gmail.com> wrote:
> > -1 I hesitate to step into this discussion because I can't also step
> > up and do the long-term testing even as I recommend that it must be
> > done. There are at least four companies supporting Accumulo and
> > contributing back to the project. Surely one of those companies can
> > supply the resources to continue the existing test regimen? Is there
> > some concern that those resources won't be available for the next
> > release cycle?
> >
> > On Thu, Jun 19, 2014 at 3:13 PM, Josh Elser <jo...@gmail.com>
> wrote:
> >> As we're starting to consider 1.5.2 and 1.6.1 coming out in the near
> future,
> >> I want to revisit a discussion[1] I started at the end of April
> regarding
> >> the "testing burden" that is currently set forth in our release
> document[2].
> >>
> >> What I'm proposing is to modify the language of the release document to
> be
> >> explicit about the amount of testing needed. For bug-fix, "minor"
> releases
> >> (e.g. 1.5.2 and 1.6.1), the 7 days of testing using continuous ingest
> and
> >> randomwalk (with and without agitation) will be clearly defined as "may"
> >> instead of "should" or "must" language. If the resources are available,
> it
> >> is recommended that some longer, multi-process/node test is run against
> the
> >> release candidate; however, it is not required and should not prevent us
> >> from making the minor release.
> >>
> >> I will also include language that strongly recommends that the changes
> >> included in the "minor" release be vetted/reviewed as a way to mitigate
> the
> >> risk of shipping new regressions.
> >>
> >> I am not recommending that the language be changed for "major" releases
> >> (e.g. 1.7.0 and 2.0.0) as these releases still imply significant new
> >> features or internal changes.
> >>
> >> Unless someone informs me otherwise, I will treat this as a normal
> >> lazy-consensus approval. Assuming we move closer to "proper" semantic
> >> versioning for 2.0.0, I believe these updated guidelines will change
> again.
> >> I do however think there is merit in making this change now so that we
> can
> >> get the good bugs that we've fixed out to our users.
> >>
> >> Let me know what you think. I will wait, at least, the prescribed three
> days
> >> before changing any thing.
> >>
> >> - Josh
> >>
> >> [1]
> >>
> http://mail-archives.apache.org/mod_mbox/accumulo-dev/201404.mbox/%3C535931A7.30605%40gmail.com%3E
> >> [2] http://accumulo.apache.org/governance/releasing.html
>



-- 
Sean

Re: Reduced testing burden for bug-fix releases

[David Medinets <da...@gmail.com>]

+0 I'm changing my vote after some reconsideration. Having the ability
to vote -1 on a release as Keith mentioned is good enough for a
bug-fix release.

On Thu, Jun 19, 2014 at 3:20 PM, David Medinets
<da...@gmail.com> wrote:
> -1 I hesitate to step into this discussion because I can't also step
> up and do the long-term testing even as I recommend that it must be
> done. There are at least four companies supporting Accumulo and
> contributing back to the project. Surely one of those companies can
> supply the resources to continue the existing test regimen? Is there
> some concern that those resources won't be available for the next
> release cycle?
>
> On Thu, Jun 19, 2014 at 3:13 PM, Josh Elser <jo...@gmail.com> wrote:
>> As we're starting to consider 1.5.2 and 1.6.1 coming out in the near future,
>> I want to revisit a discussion[1] I started at the end of April regarding
>> the "testing burden" that is currently set forth in our release document[2].
>>
>> What I'm proposing is to modify the language of the release document to be
>> explicit about the amount of testing needed. For bug-fix, "minor" releases
>> (e.g. 1.5.2 and 1.6.1), the 7 days of testing using continuous ingest and
>> randomwalk (with and without agitation) will be clearly defined as "may"
>> instead of "should" or "must" language. If the resources are available, it
>> is recommended that some longer, multi-process/node test is run against the
>> release candidate; however, it is not required and should not prevent us
>> from making the minor release.
>>
>> I will also include language that strongly recommends that the changes
>> included in the "minor" release be vetted/reviewed as a way to mitigate the
>> risk of shipping new regressions.
>>
>> I am not recommending that the language be changed for "major" releases
>> (e.g. 1.7.0 and 2.0.0) as these releases still imply significant new
>> features or internal changes.
>>
>> Unless someone informs me otherwise, I will treat this as a normal
>> lazy-consensus approval. Assuming we move closer to "proper" semantic
>> versioning for 2.0.0, I believe these updated guidelines will change again.
>> I do however think there is merit in making this change now so that we can
>> get the good bugs that we've fixed out to our users.
>>
>> Let me know what you think. I will wait, at least, the prescribed three days
>> before changing any thing.
>>
>> - Josh
>>
>> [1]
>> http://mail-archives.apache.org/mod_mbox/accumulo-dev/201404.mbox/%3C535931A7.30605%40gmail.com%3E
>> [2] http://accumulo.apache.org/governance/releasing.html

Re: Reduced testing burden for bug-fix releases

[Josh Elser <jo...@gmail.com>]

For context, I do not have the physical resources available to me at any 
point in time.

Time is also a concern, but usually less of one because I don't mind 
working into evenings and weekends to get this done, and I can usually 
work on my other priorities concurrently.

On 6/19/14, 12:38 PM, Christopher wrote:
> I don't know Josh's concerns, but the concern for me is both resources and
> time. No matter how much resources we have, it is still not infinite, and
> I'd rather we focus our testing efforts on the changeset between the
> previous release and the minor/bugfix release, rather than spend resources
> and time on all the exhaustive general testing, which mostly exercises code
> that has not changed.
>
> For instance, do we really need 72-hours of continuous ingest on a large
> cluster to release a bugfix which affects the shell?
> If the long running tests are what is necessary to exercise the changeset,
> that makes sense, but otherwise, no.
>
>
>
> --
> Christopher L Tubbs II
> http://gravatar.com/ctubbsii
>
>
> On Thu, Jun 19, 2014 at 3:20 PM, David Medinets <da...@gmail.com>
> wrote:
>
>> -1 I hesitate to step into this discussion because I can't also step
>> up and do the long-term testing even as I recommend that it must be
>> done. There are at least four companies supporting Accumulo and
>> contributing back to the project. Surely one of those companies can
>> supply the resources to continue the existing test regimen? Is there
>> some concern that those resources won't be available for the next
>> release cycle?
>>
>> On Thu, Jun 19, 2014 at 3:13 PM, Josh Elser <jo...@gmail.com> wrote:
>>> As we're starting to consider 1.5.2 and 1.6.1 coming out in the near
>> future,
>>> I want to revisit a discussion[1] I started at the end of April regarding
>>> the "testing burden" that is currently set forth in our release
>> document[2].
>>>
>>> What I'm proposing is to modify the language of the release document to
>> be
>>> explicit about the amount of testing needed. For bug-fix, "minor"
>> releases
>>> (e.g. 1.5.2 and 1.6.1), the 7 days of testing using continuous ingest and
>>> randomwalk (with and without agitation) will be clearly defined as "may"
>>> instead of "should" or "must" language. If the resources are available,
>> it
>>> is recommended that some longer, multi-process/node test is run against
>> the
>>> release candidate; however, it is not required and should not prevent us
>>> from making the minor release.
>>>
>>> I will also include language that strongly recommends that the changes
>>> included in the "minor" release be vetted/reviewed as a way to mitigate
>> the
>>> risk of shipping new regressions.
>>>
>>> I am not recommending that the language be changed for "major" releases
>>> (e.g. 1.7.0 and 2.0.0) as these releases still imply significant new
>>> features or internal changes.
>>>
>>> Unless someone informs me otherwise, I will treat this as a normal
>>> lazy-consensus approval. Assuming we move closer to "proper" semantic
>>> versioning for 2.0.0, I believe these updated guidelines will change
>> again.
>>> I do however think there is merit in making this change now so that we
>> can
>>> get the good bugs that we've fixed out to our users.
>>>
>>> Let me know what you think. I will wait, at least, the prescribed three
>> days
>>> before changing any thing.
>>>
>>> - Josh
>>>
>>> [1]
>>>
>> http://mail-archives.apache.org/mod_mbox/accumulo-dev/201404.mbox/%3C535931A7.30605%40gmail.com%3E
>>> [2] http://accumulo.apache.org/governance/releasing.html
>>
>

Re: Reduced testing burden for bug-fix releases

[Christopher <ct...@apache.org>]

I don't know Josh's concerns, but the concern for me is both resources and
time. No matter how much resources we have, it is still not infinite, and
I'd rather we focus our testing efforts on the changeset between the
previous release and the minor/bugfix release, rather than spend resources
and time on all the exhaustive general testing, which mostly exercises code
that has not changed.

For instance, do we really need 72-hours of continuous ingest on a large
cluster to release a bugfix which affects the shell?
If the long running tests are what is necessary to exercise the changeset,
that makes sense, but otherwise, no.



--
Christopher L Tubbs II
http://gravatar.com/ctubbsii


On Thu, Jun 19, 2014 at 3:20 PM, David Medinets <da...@gmail.com>
wrote:

> -1 I hesitate to step into this discussion because I can't also step
> up and do the long-term testing even as I recommend that it must be
> done. There are at least four companies supporting Accumulo and
> contributing back to the project. Surely one of those companies can
> supply the resources to continue the existing test regimen? Is there
> some concern that those resources won't be available for the next
> release cycle?
>
> On Thu, Jun 19, 2014 at 3:13 PM, Josh Elser <jo...@gmail.com> wrote:
> > As we're starting to consider 1.5.2 and 1.6.1 coming out in the near
> future,
> > I want to revisit a discussion[1] I started at the end of April regarding
> > the "testing burden" that is currently set forth in our release
> document[2].
> >
> > What I'm proposing is to modify the language of the release document to
> be
> > explicit about the amount of testing needed. For bug-fix, "minor"
> releases
> > (e.g. 1.5.2 and 1.6.1), the 7 days of testing using continuous ingest and
> > randomwalk (with and without agitation) will be clearly defined as "may"
> > instead of "should" or "must" language. If the resources are available,
> it
> > is recommended that some longer, multi-process/node test is run against
> the
> > release candidate; however, it is not required and should not prevent us
> > from making the minor release.
> >
> > I will also include language that strongly recommends that the changes
> > included in the "minor" release be vetted/reviewed as a way to mitigate
> the
> > risk of shipping new regressions.
> >
> > I am not recommending that the language be changed for "major" releases
> > (e.g. 1.7.0 and 2.0.0) as these releases still imply significant new
> > features or internal changes.
> >
> > Unless someone informs me otherwise, I will treat this as a normal
> > lazy-consensus approval. Assuming we move closer to "proper" semantic
> > versioning for 2.0.0, I believe these updated guidelines will change
> again.
> > I do however think there is merit in making this change now so that we
> can
> > get the good bugs that we've fixed out to our users.
> >
> > Let me know what you think. I will wait, at least, the prescribed three
> days
> > before changing any thing.
> >
> > - Josh
> >
> > [1]
> >
> http://mail-archives.apache.org/mod_mbox/accumulo-dev/201404.mbox/%3C535931A7.30605%40gmail.com%3E
> > [2] http://accumulo.apache.org/governance/releasing.html
>

Re: Reduced testing burden for bug-fix releases

[Josh Elser <jo...@gmail.com>]

I'm sorry, David, but you need to remember that we are individuals.

My employer is absolutely irrelevant from the equation. Unless you are 
willing to supply the funds to pay to use some resources, I don't feel 
like this is a valid -1.

On 6/19/14, 12:20 PM, David Medinets wrote:
> -1 I hesitate to step into this discussion because I can't also step
> up and do the long-term testing even as I recommend that it must be
> done. There are at least four companies supporting Accumulo and
> contributing back to the project. Surely one of those companies can
> supply the resources to continue the existing test regimen? Is there
> some concern that those resources won't be available for the next
> release cycle?
>
> On Thu, Jun 19, 2014 at 3:13 PM, Josh Elser <jo...@gmail.com> wrote:
>> As we're starting to consider 1.5.2 and 1.6.1 coming out in the near future,
>> I want to revisit a discussion[1] I started at the end of April regarding
>> the "testing burden" that is currently set forth in our release document[2].
>>
>> What I'm proposing is to modify the language of the release document to be
>> explicit about the amount of testing needed. For bug-fix, "minor" releases
>> (e.g. 1.5.2 and 1.6.1), the 7 days of testing using continuous ingest and
>> randomwalk (with and without agitation) will be clearly defined as "may"
>> instead of "should" or "must" language. If the resources are available, it
>> is recommended that some longer, multi-process/node test is run against the
>> release candidate; however, it is not required and should not prevent us
>> from making the minor release.
>>
>> I will also include language that strongly recommends that the changes
>> included in the "minor" release be vetted/reviewed as a way to mitigate the
>> risk of shipping new regressions.
>>
>> I am not recommending that the language be changed for "major" releases
>> (e.g. 1.7.0 and 2.0.0) as these releases still imply significant new
>> features or internal changes.
>>
>> Unless someone informs me otherwise, I will treat this as a normal
>> lazy-consensus approval. Assuming we move closer to "proper" semantic
>> versioning for 2.0.0, I believe these updated guidelines will change again.
>> I do however think there is merit in making this change now so that we can
>> get the good bugs that we've fixed out to our users.
>>
>> Let me know what you think. I will wait, at least, the prescribed three days
>> before changing any thing.
>>
>> - Josh
>>
>> [1]
>> http://mail-archives.apache.org/mod_mbox/accumulo-dev/201404.mbox/%3C535931A7.30605%40gmail.com%3E
>> [2] http://accumulo.apache.org/governance/releasing.html

Re: Reduced testing burden for bug-fix releases

[David Medinets <da...@gmail.com>]

-1 I hesitate to step into this discussion because I can't also step
up and do the long-term testing even as I recommend that it must be
done. There are at least four companies supporting Accumulo and
contributing back to the project. Surely one of those companies can
supply the resources to continue the existing test regimen? Is there
some concern that those resources won't be available for the next
release cycle?

On Thu, Jun 19, 2014 at 3:13 PM, Josh Elser <jo...@gmail.com> wrote:
> As we're starting to consider 1.5.2 and 1.6.1 coming out in the near future,
> I want to revisit a discussion[1] I started at the end of April regarding
> the "testing burden" that is currently set forth in our release document[2].
>
> What I'm proposing is to modify the language of the release document to be
> explicit about the amount of testing needed. For bug-fix, "minor" releases
> (e.g. 1.5.2 and 1.6.1), the 7 days of testing using continuous ingest and
> randomwalk (with and without agitation) will be clearly defined as "may"
> instead of "should" or "must" language. If the resources are available, it
> is recommended that some longer, multi-process/node test is run against the
> release candidate; however, it is not required and should not prevent us
> from making the minor release.
>
> I will also include language that strongly recommends that the changes
> included in the "minor" release be vetted/reviewed as a way to mitigate the
> risk of shipping new regressions.
>
> I am not recommending that the language be changed for "major" releases
> (e.g. 1.7.0 and 2.0.0) as these releases still imply significant new
> features or internal changes.
>
> Unless someone informs me otherwise, I will treat this as a normal
> lazy-consensus approval. Assuming we move closer to "proper" semantic
> versioning for 2.0.0, I believe these updated guidelines will change again.
> I do however think there is merit in making this change now so that we can
> get the good bugs that we've fixed out to our users.
>
> Let me know what you think. I will wait, at least, the prescribed three days
> before changing any thing.
>
> - Josh
>
> [1]
> http://mail-archives.apache.org/mod_mbox/accumulo-dev/201404.mbox/%3C535931A7.30605%40gmail.com%3E
> [2] http://accumulo.apache.org/governance/releasing.html

Re: Reduced testing burden for bug-fix releases

[Sean Busbey <bu...@cloudera.com>]

On Fri, Jun 27, 2014 at 4:34 PM, Christopher <ct...@apache.org> wrote:

> I like labels to identify testing needs, as well as which tests were
> performed. "needs-test", followed by "tested-24hr-ci", etc.
>
>

oh, yeeeeeessss. that would be nice. that would give us a better idea of
testing coverage without requiring explicit coordination pre-RC.

-- 
Sean

Re: Reduced testing burden for bug-fix releases

[Christopher <ct...@apache.org>]

I like labels to identify testing needs, as well as which tests were
performed. "needs-test", followed by "tested-24hr-ci", etc.


--
Christopher L Tubbs II
http://gravatar.com/ctubbsii


On Fri, Jun 27, 2014 at 3:44 PM, Sean Busbey <bu...@cloudera.com> wrote:

> There's a label "needs-test" that we don't already use for anything. We
> could use this to mean "needs more than bugfix level testing."
>
> I started looking at what it takes to add a field, but it's not obvious at
> first glance. If we figure it out, we could add a "test level" field that
> says "bugfix", "minor", "major" to indicate how much testing we need. Even
> if we currently don't define all three levels, where ever we explain the
> use of the field we could conflate as we desire.
>
>
>
> On Fri, Jun 27, 2014 at 2:34 PM, Josh Elser <jo...@gmail.com> wrote:
>
> > What did you have in mind? Please make a suggestion.
> >
> >
> > On 6/27/14, 2:28 PM, Sean Busbey wrote:
> >
> >> as a follow up, can we agree on a label or add a flag to jiras that can
> be
> >> used to signal when we think a particular issue introduces changes that
> >> warrant the higher level of testing scrutiny?
> >>
> >>
> >> On Fri, Jun 27, 2014 at 12:56 PM, Josh Elser <jo...@gmail.com>
> >> wrote:
> >>
> >>  I just pushed this to production. Thanks to those who gave their
> >>> opinions.
> >>>
> >>>
> >>> On 6/26/14, 2:59 PM, Josh Elser wrote:
> >>>
> >>>  I've update the language on the release guidelines on the staged
> site[1]
> >>>> for what was discussed here.
> >>>>
> >>>> Please let me know if there are any problems with it, otherwise I'll
> >>>> push the changes to production later today. If you miss this message
> >>>> before I push to production, please feel free to continue the
> discussion
> >>>> and we can revise the language and make sure we're all happy.
> >>>>
> >>>> - Josh
> >>>>
> >>>> [1] http://accumulo.staging.apache.org/governance/
> >>>> releasing.html#testing
> >>>>
> >>>> On 6/19/14, 3:40 PM, Josh Elser wrote:
> >>>>
> >>>>  On 6/19/14, 12:36 PM, Keith Turner wrote:
> >>>>>
> >>>>>  I am in favor of lowering the required threshhold.  If someone feels
> >>>>>> not
> >>>>>> enough testing was done, they can vote -1 the release vote.
> >>>>>>
> >>>>>>
> >>>>> I like that. If there is a specific area/reason to be concerned, it
> can
> >>>>> be addressed as a part of the regular release voting process.
> >>>>>
> >>>>>
> >>>>
> >>
> >>
>
>
> --
> Sean
>

Re: Reduced testing burden for bug-fix releases

[Sean Busbey <bu...@cloudera.com>]

There's a label "needs-test" that we don't already use for anything. We
could use this to mean "needs more than bugfix level testing."

I started looking at what it takes to add a field, but it's not obvious at
first glance. If we figure it out, we could add a "test level" field that
says "bugfix", "minor", "major" to indicate how much testing we need. Even
if we currently don't define all three levels, where ever we explain the
use of the field we could conflate as we desire.



On Fri, Jun 27, 2014 at 2:34 PM, Josh Elser <jo...@gmail.com> wrote:

> What did you have in mind? Please make a suggestion.
>
>
> On 6/27/14, 2:28 PM, Sean Busbey wrote:
>
>> as a follow up, can we agree on a label or add a flag to jiras that can be
>> used to signal when we think a particular issue introduces changes that
>> warrant the higher level of testing scrutiny?
>>
>>
>> On Fri, Jun 27, 2014 at 12:56 PM, Josh Elser <jo...@gmail.com>
>> wrote:
>>
>>  I just pushed this to production. Thanks to those who gave their
>>> opinions.
>>>
>>>
>>> On 6/26/14, 2:59 PM, Josh Elser wrote:
>>>
>>>  I've update the language on the release guidelines on the staged site[1]
>>>> for what was discussed here.
>>>>
>>>> Please let me know if there are any problems with it, otherwise I'll
>>>> push the changes to production later today. If you miss this message
>>>> before I push to production, please feel free to continue the discussion
>>>> and we can revise the language and make sure we're all happy.
>>>>
>>>> - Josh
>>>>
>>>> [1] http://accumulo.staging.apache.org/governance/
>>>> releasing.html#testing
>>>>
>>>> On 6/19/14, 3:40 PM, Josh Elser wrote:
>>>>
>>>>  On 6/19/14, 12:36 PM, Keith Turner wrote:
>>>>>
>>>>>  I am in favor of lowering the required threshhold.  If someone feels
>>>>>> not
>>>>>> enough testing was done, they can vote -1 the release vote.
>>>>>>
>>>>>>
>>>>> I like that. If there is a specific area/reason to be concerned, it can
>>>>> be addressed as a part of the regular release voting process.
>>>>>
>>>>>
>>>>
>>
>>


-- 
Sean

Re: Reduced testing burden for bug-fix releases

[Josh Elser <jo...@gmail.com>]

What did you have in mind? Please make a suggestion.

On 6/27/14, 2:28 PM, Sean Busbey wrote:
> as a follow up, can we agree on a label or add a flag to jiras that can be
> used to signal when we think a particular issue introduces changes that
> warrant the higher level of testing scrutiny?
>
>
> On Fri, Jun 27, 2014 at 12:56 PM, Josh Elser <jo...@gmail.com> wrote:
>
>> I just pushed this to production. Thanks to those who gave their opinions.
>>
>>
>> On 6/26/14, 2:59 PM, Josh Elser wrote:
>>
>>> I've update the language on the release guidelines on the staged site[1]
>>> for what was discussed here.
>>>
>>> Please let me know if there are any problems with it, otherwise I'll
>>> push the changes to production later today. If you miss this message
>>> before I push to production, please feel free to continue the discussion
>>> and we can revise the language and make sure we're all happy.
>>>
>>> - Josh
>>>
>>> [1] http://accumulo.staging.apache.org/governance/releasing.html#testing
>>>
>>> On 6/19/14, 3:40 PM, Josh Elser wrote:
>>>
>>>> On 6/19/14, 12:36 PM, Keith Turner wrote:
>>>>
>>>>> I am in favor of lowering the required threshhold.  If someone feels not
>>>>> enough testing was done, they can vote -1 the release vote.
>>>>>
>>>>
>>>> I like that. If there is a specific area/reason to be concerned, it can
>>>> be addressed as a part of the regular release voting process.
>>>>
>>>
>
>

Re: Reduced testing burden for bug-fix releases

[Sean Busbey <bu...@cloudera.com>]

as a follow up, can we agree on a label or add a flag to jiras that can be
used to signal when we think a particular issue introduces changes that
warrant the higher level of testing scrutiny?


On Fri, Jun 27, 2014 at 12:56 PM, Josh Elser <jo...@gmail.com> wrote:

> I just pushed this to production. Thanks to those who gave their opinions.
>
>
> On 6/26/14, 2:59 PM, Josh Elser wrote:
>
>> I've update the language on the release guidelines on the staged site[1]
>> for what was discussed here.
>>
>> Please let me know if there are any problems with it, otherwise I'll
>> push the changes to production later today. If you miss this message
>> before I push to production, please feel free to continue the discussion
>> and we can revise the language and make sure we're all happy.
>>
>> - Josh
>>
>> [1] http://accumulo.staging.apache.org/governance/releasing.html#testing
>>
>> On 6/19/14, 3:40 PM, Josh Elser wrote:
>>
>>> On 6/19/14, 12:36 PM, Keith Turner wrote:
>>>
>>>> I am in favor of lowering the required threshhold.  If someone feels not
>>>> enough testing was done, they can vote -1 the release vote.
>>>>
>>>
>>> I like that. If there is a specific area/reason to be concerned, it can
>>> be addressed as a part of the regular release voting process.
>>>
>>


-- 
Sean

Re: Reduced testing burden for bug-fix releases

[Josh Elser <jo...@gmail.com>]

I just pushed this to production. Thanks to those who gave their opinions.

On 6/26/14, 2:59 PM, Josh Elser wrote:
> I've update the language on the release guidelines on the staged site[1]
> for what was discussed here.
>
> Please let me know if there are any problems with it, otherwise I'll
> push the changes to production later today. If you miss this message
> before I push to production, please feel free to continue the discussion
> and we can revise the language and make sure we're all happy.
>
> - Josh
>
> [1] http://accumulo.staging.apache.org/governance/releasing.html#testing
>
> On 6/19/14, 3:40 PM, Josh Elser wrote:
>> On 6/19/14, 12:36 PM, Keith Turner wrote:
>>> I am in favor of lowering the required threshhold.  If someone feels not
>>> enough testing was done, they can vote -1 the release vote.
>>
>> I like that. If there is a specific area/reason to be concerned, it can
>> be addressed as a part of the regular release voting process.

Re: Reduced testing burden for bug-fix releases

[Josh Elser <jo...@gmail.com>]

I've update the language on the release guidelines on the staged site[1] 
for what was discussed here.

Please let me know if there are any problems with it, otherwise I'll 
push the changes to production later today. If you miss this message 
before I push to production, please feel free to continue the discussion 
and we can revise the language and make sure we're all happy.

- Josh

[1] http://accumulo.staging.apache.org/governance/releasing.html#testing

On 6/19/14, 3:40 PM, Josh Elser wrote:
> On 6/19/14, 12:36 PM, Keith Turner wrote:
>> I am in favor of lowering the required threshhold.  If someone feels not
>> enough testing was done, they can vote -1 the release vote.
>
> I like that. If there is a specific area/reason to be concerned, it can
> be addressed as a part of the regular release voting process.

Re: Reduced testing burden for bug-fix releases

[Josh Elser <jo...@gmail.com>]

On 6/19/14, 12:36 PM, Keith Turner wrote:
> I am in favor of lowering the required threshhold.  If someone feels not
> enough testing was done, they can vote -1 the release vote.

I like that. If there is a specific area/reason to be concerned, it can 
be addressed as a part of the regular release voting process.

Re: Reduced testing burden for bug-fix releases

[Keith Turner <ke...@deenlo.com>]

On Thu, Jun 19, 2014 at 3:13 PM, Josh Elser <jo...@gmail.com> wrote:

> As we're starting to consider 1.5.2 and 1.6.1 coming out in the near
> future, I want to revisit a discussion[1] I started at the end of April
> regarding the "testing burden" that is currently set forth in our release
> document[2].
>
> What I'm proposing is to modify the language of the release document to be
> explicit about the amount of testing needed. For bug-fix, "minor" releases
> (e.g. 1.5.2 and 1.6.1), the 7 days of testing using continuous ingest and
> randomwalk (with and without agitation) will be clearly defined as "may"
> instead of "should" or "must" language. If the resources are available, it
> is recommended that some longer, multi-process/node test is run against the
> release candidate; however, it is not required and should not prevent us
> from making the minor release.
>
> I will also include language that strongly recommends that the changes
> included in the "minor" release be vetted/reviewed as a way to mitigate the
> risk of shipping new regressions.
>
> I am not recommending that the language be changed for "major" releases
> (e.g. 1.7.0 and 2.0.0) as these releases still imply significant new
> features or internal changes.
>
> Unless someone informs me otherwise, I will treat this as a normal
> lazy-consensus approval. Assuming we move closer to "proper" semantic
> versioning for 2.0.0, I believe these updated guidelines will change again.
> I do however think there is merit in making this change now so that we can
> get the good bugs that we've fixed out to our users.
>
> Let me know what you think. I will wait, at least, the prescribed three
> days before changing any thing.
>

I am in favor of lowering the required threshhold.  If someone feels not
enough testing was done, they can vote -1 the release vote.


>
> - Josh
>
> [1] http://mail-archives.apache.org/mod_mbox/accumulo-dev/
> 201404.mbox/%3C535931A7.30605%40gmail.com%3E
> [2] http://accumulo.apache.org/governance/releasing.html
>