Re: Proposal for reducing Subversion's lengthy (and unpredictable) release cycles.

[Daniel Shahaf <da...@elego.de>]

C. Michael Pilato wrote on Mon, Jun 17, 2013 at 11:24:45 -0400:
> On 06/17/2013 09:57 AM, Daniel Shahaf wrote:
> > On Mon, Jun 17, 2013 at 09:36:29AM -0400, C. Michael Pilato wrote:
> >> On 06/13/2013 10:30 PM, Greg Stein wrote:
> >>> Fair enough. But I think you're talking about Step Two. There is more
> >>> work on what "stable" means, what time schedules to use, etc. That's
> >>> Step One.
> >>
> >> In private mail, you also asked for tighter definition of the various
> >> trimesters of stability (which is an interesting choice of terminology in my
> >> own personal life right now, but I digress...).  Here's my thinking:
> >>
> >> Tri 1:  Trunk builds and passes tests, but may have crazy new,
> >> sweeping-change types of features on it.  We've tried to be
> >> forward-thinking, but who knows if these are the APIs/protocols/etc. that
> >> we'll wind up with in the release.  At the end of this period, we might say
> >> we're merely "build stable".  We could ship an alpha at the end of this
> >> period to get the crazy new features into the public's hands for user
> >> acceptance testing.
> >>
> > 
> > I like the idea of sprinkling alpha/beta releases along the way.
> > 
> >> Tri 2:  Trunk builds and passes tests, and the crazy stuff is still getting
> >> hammered into release-worthiness, but we're not allowing any more crazy
> >> stuff in.  Smallish features and enhancements are fine, but nothing like a
> >> WC-NG or Ev2 or FS-NG or....  At the end of this period, we would say we're
> >> "feature stable", and could ship a beta release.
> >>
> >> Tri 3:  Trunk is feature-complete.  Oh, and it builds and passes tests.  :-)
> >>  We're serious about getting this thing ready to release, now.  Strictly
> >> speaking, this "period" of trunk's life extends until the final release is
> >> cut by taking the "release branch" side of the fork in the road.  But we
> >> don't want to lock down the trunk indefinitely, so we get as much
> >> stabilization done on the trunk as we can before branching for release
> >> stabilization and reopening the trunk for a new "first trimester".
> > 
> > Which trimester is concurrent to the "1.N.x branched, but 1.N.0 not released
> > yet" period?
> 
> That was a point we didn't fully settle on in Berlin.  Today that period is
> a limbo of sorts.  Trunk is technically open to anything, but we don't like
> to codebomb it because it complicates backports of stabilization fixes to
> the release branch.
> 
> I suggest that it should instead be the first trimester of the new release
> cycle -- and as such, wide open to changes -- because if all goes as
> planned, there will be fewer things to backport anyway (since we will have
> already been sitting in a feature-frozen stabilization mode on the trunk for
> three months prior.)
> 
> Other ideas?

How about: at the end of the third trimester, the release branch is
created, a release candidate is cut, and the 1st trimester begins on
trunk?

I.e., make all of those simultaneous.  Trunk will be in Tri1 mode while
the release branch is in soak, so we won't have the problem of needing
to pay attention to two active branches.  That's also consistent with
doing alpha at the end of tri1 and beta at the end of tri2.

Re: Proposal for reducing Subversion's lengthy (and unpredictable) release cycles.

[Branko Čibej <br...@wandisco.com>]

On 17.06.2013 18:34, Andy Singleton wrote:
> On 6/17/2013 11:36 AM, Daniel Shahaf wrote:
>> C. Michael Pilato wrote on Mon, Jun 17, 2013 at 11:24:45 -0400:
>>> On 06/17/2013 09:57 AM, Daniel Shahaf wrote:
>>>> On Mon, Jun 17, 2013 at 09:36:29AM -0400, C. Michael Pilato wrote:
>>>>> On 06/13/2013 10:30 PM, Greg Stein wrote:
>>>>>> Fair enough. But I think you're talking about Step Two. There is more
>>>>>> work on what "stable" means, what time schedules to use, etc. That's
>>>>>> Step One.
>>>>> In private mail, you also asked for tighter definition of the various
>>>>> trimesters of stability (which is an interesting choice of terminology in my
>>>>> own personal life right now, but I digress...).  Here's my thinking:
>>>>>
>>>>> Tri 1:  Trunk builds and passes tests, but may have crazy new,
>>>>> sweeping-change types of features on it.  We've tried to be
>>>>> forward-thinking, but who knows if these are the APIs/protocols/etc. that
>>>>> we'll wind up with in the release.  At the end of this period, we might say
>>>>> we're merely "build stable".  We could ship an alpha at the end of this
>>>>> period to get the crazy new features into the public's hands for user
>>>>> acceptance testing.
>>>>>
>>>> I like the idea of sprinkling alpha/beta releases along the way.
>>>>
>>>>> Tri 2:  Trunk builds and passes tests, and the crazy stuff is still getting
>>>>> hammered into release-worthiness, but we're not allowing any more crazy
>>>>> stuff in.  Smallish features and enhancements are fine, but nothing like a
>>>>> WC-NG or Ev2 or FS-NG or....  At the end of this period, we would say we're
>>>>> "feature stable", and could ship a beta release.
>>>>>
>>>>> Tri 3:  Trunk is feature-complete.  Oh, and it builds and passes tests.  :-)
>>>>>  We're serious about getting this thing ready to release, now.  Strictly
>>>>> speaking, this "period" of trunk's life extends until the final release is
>>>>> cut by taking the "release branch" side of the fork in the road.  But we
>>>>> don't want to lock down the trunk indefinitely, so we get as much
>>>>> stabilization done on the trunk as we can before branching for release
>>>>> stabilization and reopening the trunk for a new "first trimester".
>>>> Which trimester is concurrent to the "1.N.x branched, but 1.N.0 not released
>>>> yet" period?
>>> That was a point we didn't fully settle on in Berlin.  Today that period is
>>> a limbo of sorts.  Trunk is technically open to anything, but we don't like
>>> to codebomb it because it complicates backports of stabilization fixes to
>>> the release branch.
>>>
>>> I suggest that it should instead be the first trimester of the new release
>>> cycle -- and as such, wide open to changes -- because if all goes as
>>> planned, there will be fewer things to backport anyway (since we will have
>>> already been sitting in a feature-frozen stabilization mode on the trunk for
>>> three months prior.)
>>>
>>> Other ideas?
>> How about: at the end of the third trimester, the release branch is
>> created, a release candidate is cut, and the 1st trimester begins on
>> trunk?
>>
>> I.e., make all of those simultaneous.  Trunk will be in Tri1 mode while
>> the release branch is in soak, so we won't have the problem of needing
>> to pay attention to two active branches.  That's also consistent with
>> doing alpha at the end of tri1 and beta at the end of tri2.
>
> Did you consider a process that produces a release every month, or
> every quarter?  If any engineering team can do it, the Apache
> Subversion team can do it.  There is some discipline around code
> review and a long history of automated testing.
>
> I like the trimester plan.  However, in my experience, when you get
> into a situation where it takes longer and longer to test and release
> a piece of software, it’s hard to reverse.  The system is becoming
> more complex, it has more pieces and use cases, and it naturally takes
> longer to test it.  Is it possible that this dynamic is at work with
> Subversion, and that the trimester plan has a high probability of
> falling into a situation where the third trimester takes substantially
> longer than one trimester?
>

I think we've considered all that. While I find the idea appealing, we
do have to consider that it would be an even more significant departure
from our current workflow. I'd personally prefer to avoid taking big steps.

> One reliable way to fix the problem is to move the development to
> continuous delivery and force testing much earlier in the process.  It
> might end up being less work.
>

I fear it's the "force testing much earlier" part that is hard to get
down peoples' throats, given that the majority of active developers are
volunteers.

On the other hand, we are in fact releasing often, it's just that most
releases are bug fixes.

> You could argue that Subversion is a difficult case for continuous
> delivery because it requires compatible upgrades to both client and
> server.  Yes, this is a difficult problem.  However, Subversion is
> already dealing with this problem.  There is an implicit requirement
> that new server releases be compatible with older clients.  Increasing
> the release frequency does not increase the difficulty of dealing with
> this requirement.
>
> You can also argue that Subversion is a difficult case because it
> isn’t just adding features.  The team is making deep structural
> reforms that can’t be finished in a month, and aren’t backwards
> compatible.  This is a problem that is faced by every continuous
> delivery team.  They often release two separate versions of a major
> subsystem in one piece of software, with one subsystem switched for
> most users off until it is ready.  This requires some extra
> engineering work during the transition period.  However, you get back
> a lot of the extra work when you go to do testing and deployment,
> because the new subsystem is already deployed and mostly tested.
>
> The team would be relying on some standard tactics of continuous
> development teams.
> * Feature switches.  Major changes are hidden with “IF
> FEATURE_IS_ON”.  This is standard practice and it's used in Subversion
> now.  It doesn’t create code clutter if you remove the switches later.
> * Code review
> * Automated tests
> * Beta testers.  People would need to make a commitment.  We would run
> the newest version on some of our servers.
>
> You would probably want to add automated error reporting from the new
> clients and servers.  That reporting can serve as a substitute for
> complete automated regression testing and reduces the investment in
> building automated tests.
>
> I believe that you would also want to think about the possibility that
> a release would not come from trunk.  Maybe a smaller group of
> contributors can prepare something for release on a branch.  In the
> very centralized development process that I see, the core maintainers
> are taking a lot of pressure in the ramp up to a release.  This could
> be distributed.
>

Yep, we did talk about non-trunk releases and experimental features. I
still believe they're viable options, just not on the short-range radar
right now. Mostly because regardless of where a tarball comes from, it
has to meet all the formal requirements of an Apache release; which
requires a significant involvement from the whole team.

> It's just an idea.  It works for us.


-- 
Branko Čibej | Director of Subversion
WANdisco // Non-Stop Data
e. brane@wandisco.com

Re: Proposal for reducing Subversion's lengthy (and unpredictable) release cycles.

[Andy Singleton <an...@assembla.com>]

  On 6/17/2013 11:36 AM, Daniel Shahaf wrote:
> C. Michael Pilato wrote on Mon, Jun 17, 2013 at 11:24:45 -0400:
>> On 06/17/2013 09:57 AM, Daniel Shahaf wrote:
>>> On Mon, Jun 17, 2013 at 09:36:29AM -0400, C. Michael Pilato wrote:
>>>> On 06/13/2013 10:30 PM, Greg Stein wrote:
>>>>> Fair enough. But I think you're talking about Step Two. There is more
>>>>> work on what "stable" means, what time schedules to use, etc. That's
>>>>> Step One.
>>>> In private mail, you also asked for tighter definition of the various
>>>> trimesters of stability (which is an interesting choice of terminology in my
>>>> own personal life right now, but I digress...).  Here's my thinking:
>>>>
>>>> Tri 1:  Trunk builds and passes tests, but may have crazy new,
>>>> sweeping-change types of features on it.  We've tried to be
>>>> forward-thinking, but who knows if these are the APIs/protocols/etc. that
>>>> we'll wind up with in the release.  At the end of this period, we might say
>>>> we're merely "build stable".  We could ship an alpha at the end of this
>>>> period to get the crazy new features into the public's hands for user
>>>> acceptance testing.
>>>>
>>> I like the idea of sprinkling alpha/beta releases along the way.
>>>
>>>> Tri 2:  Trunk builds and passes tests, and the crazy stuff is still getting
>>>> hammered into release-worthiness, but we're not allowing any more crazy
>>>> stuff in.  Smallish features and enhancements are fine, but nothing like a
>>>> WC-NG or Ev2 or FS-NG or....  At the end of this period, we would say we're
>>>> "feature stable", and could ship a beta release.
>>>>
>>>> Tri 3:  Trunk is feature-complete.  Oh, and it builds and passes tests.  :-)
>>>>   We're serious about getting this thing ready to release, now.  Strictly
>>>> speaking, this "period" of trunk's life extends until the final release is
>>>> cut by taking the "release branch" side of the fork in the road.  But we
>>>> don't want to lock down the trunk indefinitely, so we get as much
>>>> stabilization done on the trunk as we can before branching for release
>>>> stabilization and reopening the trunk for a new "first trimester".
>>> Which trimester is concurrent to the "1.N.x branched, but 1.N.0 not released
>>> yet" period?
>> That was a point we didn't fully settle on in Berlin.  Today that period is
>> a limbo of sorts.  Trunk is technically open to anything, but we don't like
>> to codebomb it because it complicates backports of stabilization fixes to
>> the release branch.
>>
>> I suggest that it should instead be the first trimester of the new release
>> cycle -- and as such, wide open to changes -- because if all goes as
>> planned, there will be fewer things to backport anyway (since we will have
>> already been sitting in a feature-frozen stabilization mode on the trunk for
>> three months prior.)
>>
>> Other ideas?
> How about: at the end of the third trimester, the release branch is
> created, a release candidate is cut, and the 1st trimester begins on
> trunk?
>
> I.e., make all of those simultaneous.  Trunk will be in Tri1 mode while
> the release branch is in soak, so we won't have the problem of needing
> to pay attention to two active branches.  That's also consistent with
> doing alpha at the end of tri1 and beta at the end of tri2.

Did you consider a process that produces a release every month, or every 
quarter? If any engineering team can do it, the Apache Subversion team 
can do it. There is some discipline around code review and a long 
history of automated testing.

I like the trimester plan. However, in my experience, when you get into 
a situation where it takes longer and longer to test and release a piece 
of software, it's hard to reverse. The system is becoming more complex, 
it has more pieces and use cases, and it naturally takes longer to test 
it. Is it possible that this dynamic is at work with Subversion, and 
that the trimester plan has a high probability of falling into a 
situation where the third trimester takes substantially longer than one 
trimester?

One reliable way to fix the problem is to move the development to 
continuous delivery and force testing much earlier in the process. It 
might end up being less work.

You could argue that Subversion is a difficult case for continuous 
delivery because it requires compatible upgrades to both client and 
server. Yes, this is a difficult problem. However, Subversion is already 
dealing with this problem. There is an implicit requirement that new 
server releases be compatible with older clients. Increasing the release 
frequency does not increase the difficulty of dealing with this requirement.

You can also argue that Subversion is a difficult case because it isn't 
just adding features. The team is making deep structural reforms that 
can't be finished in a month, and aren't backwards compatible. This is a 
problem that is faced by every continuous delivery team. They often 
release two separate versions of a major subsystem in one piece of 
software, with one subsystem switched for most users off until it is 
ready. This requires some extra engineering work during the transition 
period. However, you get back a lot of the extra work when you go to do 
testing and deployment, because the new subsystem is already deployed 
and mostly tested.

The team would be relying on some standard tactics of continuous 
development teams.
* Feature switches. Major changes are hidden with "IF FEATURE_IS_ON". 
This is standard practice and it's used in Subversion now. It doesn't 
create code clutter if you remove the switches later.
* Code review
* Automated tests
* Beta testers. People would need to make a commitment. We would run the 
newest version on some of our servers.

You would probably want to add automated error reporting from the new 
clients and servers. That reporting can serve as a substitute for 
complete automated regression testing and reduces the investment in 
building automated tests.

I believe that you would also want to think about the possibility that a 
release would not come from trunk. Maybe a smaller group of contributors 
can prepare something for release on a branch. In the very centralized 
development process that I see, the core maintainers are taking a lot of 
pressure in the ramp up to a release. This could be distributed.

It's just an idea.  It works for us.

-- 
Andy Singleton
Founder/CEO, Assembla Online: http://www.assembla.com
Phone: 781-328-2241
Skype: andysingleton