Branching and testing strategy for 0.22

[Owen O'Malley <om...@apache.org>]

I'd like to get started testing 0.22.

I plan to start making mini-branches for QA. These branches will be  
snapshots that QA can use for testing with an expected lifetime of two  
weeks each. Only bug fixes that are blocking QA will be applied to the  
mini-branches and every two weeks, the base of the branch will be  
moved to the head of trunk. This will allow QA to test a point in time  
(possibly with required bug fixes) with requiring development to  
continually maintain two branches.

To simplify automated builds, I'll call the branch the final name of  
"branch-0.22." But it will be rebased every two weeks or so.

Are there any concerns?

-- Owen

Re: Branching and testing strategy for 0.22

[Doug Cutting <cu...@apache.org>]

On 08/23/2010 03:19 PM, Owen O'Malley wrote:
> To simplify automated builds, I'll call the branch the final name of
> "branch-0.22." But it will be rebased every two weeks or so.
>
> Are there any concerns?

Since this will be used differently than what we've in the past named 
'branch-${version}", perhaps we should name it differently too, e.g., 
"qa-0.22" or "test-0.22".

Also, since this is branched from trunk, it might just be called 
'trunk-checkpoint' or somesuch rather than any particular version.

Doug

Re: Branching and testing strategy for 0.22

[Vinod KV <vi...@yahoo-inc.com>]

My first thought was also inline with Cos's. Any background we are 
missing about this?

+vinod


On Tuesday 24 August 2010 04:47 AM, Konstantin Boudnik wrote:
> If I may... why QA need source code branches rather than a sequential builds
> from the trunk (as it usually done)? Say:
>   - build qa1 is cut
>   - QA works on it and finds issues
>   - issues are reported against build qa1
>   - dev fixes some issues and mark them appropriately
>   - build qa2 is cut
>   - QA verifies that all issues from qa1 marked as 'fixed in qa2' are fixed and
>     keep going with their usual cycle.
>
> Is there anything wrong with that model? It also eliminates the need to
> maintain two branches at the same time.
>
> Cos
>
> On Mon, Aug 23, 2010 at 03:19PM, Owen O'Malley wrote:
>    
>> I'd like to get started testing 0.22.
>>
>> I plan to start making mini-branches for QA. These branches will be
>> snapshots that QA can use for testing with an expected lifetime of two
>> weeks each. Only bug fixes that are blocking QA will be applied to the
>> mini-branches and every two weeks, the base of the branch will be
>> moved to the head of trunk. This will allow QA to test a point in time
>> (possibly with required bug fixes) with requiring development to
>> continually maintain two branches.
>>
>> To simplify automated builds, I'll call the branch the final name of
>> "branch-0.22." But it will be rebased every two weeks or so.
>>
>> Are there any concerns?
>>
>> -- Owen
>>      

Re: Branching and testing strategy for 0.22

[Konstantin Boudnik <co...@yahoo-inc.com>]

As we have briefly discussed off-line I'll try to reiterate the assumptions
you have expressed:

 - testing cycle on a qa branch won't be reset after a blocker (B1) is fixed.
   The validation is continued from the point where the B1 has been
   discovered. In the real cluster test env. it means that only fresh
   deployment needs to be done.
 - in some cases a judgment call has to be made on whether or not testing
   cycle needs to be reset (e.g. in cases of massive fixes and so on). The
   chances are, however, that critical fixes are usually isolated and don't
   affect the rest of the code base.

It isn't like I disagree with the assumptions above. They are valid... under
two conditions:
 - a test harness in use (i.e. a team of test engineers running on manual
   testing errand or an automated solution similar to Herriot) supports the
   notion of interrupted test cycle
 - there's a sound regression tests suite which can guarantee the a fix B2
   doesn't invalidate fix B1 from before.

Cos

On Tue, Aug 24, 2010 at 12:08AM, Owen O'Malley wrote:
> 
> On Aug 23, 2010, at 4:17 PM, Konstantin Boudnik wrote:
> 
> > If I may... why QA need source code branches rather than a  
> > sequential builds
> > from the trunk (as it usually done)?
> 
> For QA to be effective, they can't handle a different artifact every  
> day with a new set of features. So the plan is to take a snapshot and  
> test that. However, if bugs are found that block testing they need  
> precisely those bugs fixed with out any additional features. So you  
> need branches to support that. So a sample cycle would look like:
> 
> 1. create build-1 and test it, discovering bugs B1, B2, and B3. Only  
> B1 is blocking further testing.
> 2. meanwhile, someone has checked in feature F1 into trunk
> 3. B1 is fixed in the branch and trunk. B2 and B3 are just fixed in  
> trunk
> 4. when build-2 is built it only has B1's fix
> 5. at the end of the cycle, we rebase the branch to trunk and pickup  
> B1, B2, B3 and F1 into build-3.
> 
> If you don't have a branch, you would get F1 whether you want it or not.
> 
> -- Owen

Re: Branching and testing strategy for 0.22

[Owen O'Malley <om...@apache.org>]

On Aug 23, 2010, at 4:17 PM, Konstantin Boudnik wrote:

> If I may... why QA need source code branches rather than a  
> sequential builds
> from the trunk (as it usually done)?

For QA to be effective, they can't handle a different artifact every  
day with a new set of features. So the plan is to take a snapshot and  
test that. However, if bugs are found that block testing they need  
precisely those bugs fixed with out any additional features. So you  
need branches to support that. So a sample cycle would look like:

1. create build-1 and test it, discovering bugs B1, B2, and B3. Only  
B1 is blocking further testing.
2. meanwhile, someone has checked in feature F1 into trunk
3. B1 is fixed in the branch and trunk. B2 and B3 are just fixed in  
trunk
4. when build-2 is built it only has B1's fix
5. at the end of the cycle, we rebase the branch to trunk and pickup  
B1, B2, B3 and F1 into build-3.

If you don't have a branch, you would get F1 whether you want it or not.

-- Owen

Re: Branching and testing strategy for 0.22

[Konstantin Boudnik <co...@yahoo-inc.com>]

If I may... why QA need source code branches rather than a sequential builds
from the trunk (as it usually done)? Say:
 - build qa1 is cut
 - QA works on it and finds issues
 - issues are reported against build qa1
 - dev fixes some issues and mark them appropriately
 - build qa2 is cut
 - QA verifies that all issues from qa1 marked as 'fixed in qa2' are fixed and
   keep going with their usual cycle.

Is there anything wrong with that model? It also eliminates the need to
maintain two branches at the same time.

Cos

On Mon, Aug 23, 2010 at 03:19PM, Owen O'Malley wrote:
> I'd like to get started testing 0.22.
> 
> I plan to start making mini-branches for QA. These branches will be  
> snapshots that QA can use for testing with an expected lifetime of two  
> weeks each. Only bug fixes that are blocking QA will be applied to the  
> mini-branches and every two weeks, the base of the branch will be  
> moved to the head of trunk. This will allow QA to test a point in time  
> (possibly with required bug fixes) with requiring development to  
> continually maintain two branches.
> 
> To simplify automated builds, I'll call the branch the final name of  
> "branch-0.22." But it will be rebased every two weeks or so.
> 
> Are there any concerns?
> 
> -- Owen

Re: Branching and testing strategy for 0.22

[Allen Wittenauer <aw...@linkedin.com>]

On Aug 23, 2010, at 3:19 PM, Owen O'Malley wrote:
> Are there any concerns?

Just that 0.21 isn't even out of RC yet and that patches to fix it may get missed.

Re: Branching and testing strategy for 0.22

[Owen O'Malley <om...@apache.org>]

On Aug 23, 2010, at 3:26 PM, Aaron Kimball wrote:

> Would it be worthwhile to give branches unique, persistent names?

I don't think so. Of course the old branches will always be there  
under the old revision. But it will be convenient to be able to do  
"svn up" on the given branch and have it track the current qa branch  
without worrying about what the "current" index is and doing a svn  
switch to the new url.

> branch-0.22-qa1, branch-0.22-qa2, etc. Then problems in a later  
> incarnation
> of the QA branch could be regression-tested against the previous one.

These are not intended to be long lived. In particular, any comparison  
testing will be done with the generated artifacts, not the source  
repository.

> Your point about automated builds is, however, noted. If this were  
> git,
> branch-0.22 could be a "floating" branch which is aliased with the  
> most
> recent qa branch name. Can we do something similar with svn?

I don't think so, since branches in subversion act like copies rather  
than markers.

-- Owen

Re: Branching and testing strategy for 0.22

[Aaron Kimball <aa...@cloudera.com>]

Would it be worthwhile to give branches unique, persistent names?
branch-0.22-qa1, branch-0.22-qa2, etc. Then problems in a later incarnation
of the QA branch could be regression-tested against the previous one.

Your point about automated builds is, however, noted. If this were git,
branch-0.22 could be a "floating" branch which is aliased with the most
recent qa branch name. Can we do something similar with svn?

We could remove all the QA branches when we officially cut for RC, if you're
concerned about clutter.

- Aaron

On Mon, Aug 23, 2010 at 3:19 PM, Owen O'Malley <om...@apache.org> wrote:

> I'd like to get started testing 0.22.
>
> I plan to start making mini-branches for QA. These branches will be
> snapshots that QA can use for testing with an expected lifetime of two weeks
> each. Only bug fixes that are blocking QA will be applied to the
> mini-branches and every two weeks, the base of the branch will be moved to
> the head of trunk. This will allow QA to test a point in time (possibly with
> required bug fixes) with requiring development to continually maintain two
> branches.
>
> To simplify automated builds, I'll call the branch the final name of
> "branch-0.22." But it will be rebased every two weeks or so.
>
> Are there any concerns?
>
> -- Owen
>

Re: Branching and testing strategy for 0.22

[Tom White <to...@cloudera.com>]

+1 Thanks for driving this, Owen.

Tom

On Mon, Aug 23, 2010 at 3:19 PM, Owen O'Malley <om...@apache.org> wrote:
> I'd like to get started testing 0.22.
>
> I plan to start making mini-branches for QA. These branches will be
> snapshots that QA can use for testing with an expected lifetime of two weeks
> each. Only bug fixes that are blocking QA will be applied to the
> mini-branches and every two weeks, the base of the branch will be moved to
> the head of trunk. This will allow QA to test a point in time (possibly with
> required bug fixes) with requiring development to continually maintain two
> branches.
>
> To simplify automated builds, I'll call the branch the final name of
> "branch-0.22." But it will be rebased every two weeks or so.
>
> Are there any concerns?
>
> -- Owen
>