Re: A new release process?

[Jacob Champion <ch...@gmail.com>]

On 12/29/2016 08:16 PM, David Zuelke wrote:
> The tl;dr of this approach is that
>
> - any x.y.z release only introduces bugfixes. These releases are done
> every four weeks, like clockwork. If a fix doesn't make the cut for a
> release, it'll end up in the next one; - x.y.0 releases, on the other
> hand, may introduce new features, fixes, and deprecations, but no
> breaking changes; - x.0.0 releases are the big ones (think PHP 7.0.0
> in late 2015). where backward compatibility may be changed, etc.

My favorite pieces here are
- the introduction of a release cadence
- the separation of bugfix releases from feature releases

Both make life easier -- for us, for users, and for intermediate 
maintainers -- for the reasons you've mentioned.

> [...]
>
> There are a bunch of technicalities that would need adjusting to fit
> HTTPD, such as release intervals, release management (for PHP, every
> x.y.* series has two managers who jointly coordinate releases), etc,
> but overall the idea is, IMO, worth considering.

Our current process is also fairly labor-intensive, so moving to a quick 
release cadence without simultaneously fixing that is not likely to end 
up well, IMO. There are other discussions on the list for improving our 
automated QA, which I think is probably step 1.

> As a, more or less, "outside observer", I happen to think that the
> current method of voting on finals, instead of a practice of rolling
> out RCs (that are then left up for testing for at least a week), is
> fundamentally broken. The 2.4 changelog in particular is littered
> with releases that were never officially published. For users, that's
> really confusing.

To a certain extent I think this is something that people get used to 
easily... but for security patches in *particular*, I agree. It's 
disconcerting to hear that CVE 2099-BLAH was fixed in 2.4.Z and then 
find out that 2.4.Z was never released.

> For maintainers, it's painful to start over the
> process each time, and it sometimes leads to months and months
> without a release that contains certain fixes.

The pain of discarding and restarting your manual tests because someone 
else found a regression on another platform two hours into the release 
vote comes to mind, yes.

For the record, though, I think having strong automated QA processes 
will help both of these issues more than the release cadence will. It's 
just that the cadence reinforces the QA and the QA reinforces the 
cadence; they go hand-in-hand in a very nice positive feedback loop.

> Then a backport goes
> wrong (still using SVN, in my opinion, does not help there, but
> that's a whole different discussion :)),

I've started some discussions on how we might introduce feature- and 
fix-branches, and hopefully better backport workflows, into our current 
use of SVN. (I'm a git user myself, and the current backport process is 
more error-prone than I'm used to as well. But I have no desire to start 
a holy war, and I'm confident we can find an SVN workflow that works well.)

> and a regression is in the
> latest release until someone eventually picks up a fix.
>
> Much of this, and many of the "what do we backport from trunk" and
> "I'd like to squeeze in a change I've had sitting around locally,
> please wait with the new release, because who knows when the next one
> after that will be" are, from what I can tell, a significant source
> of discord on this mailing list. All these unnecessary distractions
> that deteriorate personal relationships, while at the same time
> slowing down the pace of the project (several people have already
> pointed out Nginx's rate of innovation in comparison) and raising the
> threshold for contributions, can be fixed. PHP is the perfect
> example, and I think HTTPD would be wise to at least consider
> following this example.
>
> Happy New Year!

Thanks for your input!

--Jacob

Re: A new release process?

[William A Rowe Jr <wr...@rowe-clan.net>]

On Tue, Jan 3, 2017 at 1:32 PM, Jacob Champion <ch...@gmail.com> wrote:
> On 12/29/2016 08:16 PM, David Zuelke wrote:
>>
>> The tl;dr of this approach is that
>>
>> - any x.y.z release only introduces bugfixes. These releases are done
>> every four weeks, like clockwork. If a fix doesn't make the cut for a
>> release, it'll end up in the next one; - x.y.0 releases, on the other
>> hand, may introduce new features, fixes, and deprecations, but no
>> breaking changes; - x.0.0 releases are the big ones (think PHP 7.0.0
>> in late 2015). where backward compatibility may be changed, etc.
>
> My favorite pieces here are
> - the introduction of a release cadence
> - the separation of bugfix releases from feature releases
>
> Both make life easier -- for us, for users, and for intermediate maintainers
> -- for the reasons you've mentioned.

Consider also our consumers, module authors who rely on us to provide an
API they can design to. All of our subsidiary components, be it apr, or expat,
or pcre, or openssl, or libxml2, or lua, or nghttp2... and most of the
expansions
to httpd, be it python or perl or whatnot, all follow essentially this model.

That isn't how httpd evolved; if you look far far back into the pre-1.0 history
and even through the evolution up to 1.3.12, there wasn't a particular rhyme
or reason other than what the major patch submitters wanted to have happen.
So when pools are introduced, minor version bump - a subversion bump was
obviously not going to cut it. But there was not necessarily any compatibility
at all with previously compiled modules.

This all changed radically with 1.3.14 and 2.0 later on. But the aversion from
back in the 1.2 and 1.3 days to declaring the version minors are precious
resources has not changed.

In the meantime, nghttp2 just published the 18th version minor by year end
last year. And remains solidly compatible due to good design from the get go,
it is still major version 1. Will be interesting to watch and see when enough
API issues have accumulated to ditch major v1 for v2.

We are certainly the enigma, and if there is one reason for developers to
disabused themselves from your API, that is likely making shit up as you
go along. I know a lot of folks frustrated back in the PHP 3-4 days, and it
seems like this solution has worked out very well for the PHP crew. And
there was an equal amount of chaos around Perl's versioning and lifecycle
evolution, but that also seems to be working out pretty well.


>> [...]
>>
>> There are a bunch of technicalities that would need adjusting to fit
>> HTTPD, such as release intervals, release management (for PHP, every
>> x.y.* series has two managers who jointly coordinate releases), etc,
>> but overall the idea is, IMO, worth considering.
>
> Our current process is also fairly labor-intensive, so moving to a quick
> release cadence without simultaneously fixing that is not likely to end up
> well, IMO. There are other discussions on the list for improving our
> automated QA, which I think is probably step 1.

Releases themselves are not terribly challenging. The investment, outside
of the reviews and testing we hope many committers are engaged in,
boils down to less than an hour of actual mechanics, and some time to
prepare specific announcement details or merge back in some sequestered
security patches if those have to be applied.


>> As a, more or less, "outside observer", I happen to think that the
>> current method of voting on finals, instead of a practice of rolling
>> out RCs (that are then left up for testing for at least a week), is
>> fundamentally broken. The 2.4 changelog in particular is littered
>> with releases that were never officially published. For users, that's
>> really confusing.
>
> To a certain extent I think this is something that people get used to
> easily... but for security patches in *particular*, I agree. It's
> disconcerting to hear that CVE 2099-BLAH was fixed in 2.4.Z and then find
> out that 2.4.Z was never released.

We say this, that subversion numbers are cheap, and the lack of 2.4.24
really isn't confusing. But I have to answer questions from at least 15
individuals over the course of the following month when that happens :)
So apparently it might not confuse us, but it does confuse the rest of
the world more than we might expect.


>> For maintainers, it's painful to start over the
>> process each time, and it sometimes leads to months and months
>> without a release that contains certain fixes.
>
> The pain of discarding and restarting your manual tests because someone else
> found a regression on another platform two hours into the release vote comes
> to mind, yes.

On that, adding features to a security release 4 days before tag and roll is
a sure fire path to a bad tag. But features-features-features, gotta collect
them all :)

Looking at a long term solution to the specific quirk that occurred in 2.4.24
and 2.4.25, but the general truth of throwing in one more late last feature
usually turns out how you would expect.


> For the record, though, I think having strong automated QA processes will
> help both of these issues more than the release cadence will. It's just that
> the cadence reinforces the QA and the QA reinforces the cadence; they go
> hand-in-hand in a very nice positive feedback loop.
>
>> Then a backport goes
>> wrong (still using SVN, in my opinion, does not help there, but
>> that's a whole different discussion :)),
>
> I've started some discussions on how we might introduce feature- and
> fix-branches, and hopefully better backport workflows, into our current use
> of SVN. (I'm a git user myself, and the current backport process is more
> error-prone than I'm used to as well. But I have no desire to start a holy
> war, and I'm confident we can find an SVN workflow that works well.)

Agreed, both svn and git have their own merits, but we believe strongly in
a centralized vc to ensure we all have collected the same collective effort.


>> and a regression is in the
>> latest release until someone eventually picks up a fix.
>>
>> Much of this, and many of the "what do we backport from trunk" and
>> "I'd like to squeeze in a change I've had sitting around locally,
>> please wait with the new release, because who knows when the next one
>> after that will be" are, from what I can tell, a significant source
>> of discord on this mailing list. All these unnecessary distractions
>> that deteriorate personal relationships, while at the same time
>> slowing down the pace of the project (several people have already
>> pointed out Nginx's rate of innovation in comparison) and raising the
>> threshold for contributions, can be fixed. PHP is the perfect
>> example, and I think HTTPD would be wise to at least consider
>> following this example.
>>
>> Happy New Year!
>
> Thanks for your input!

+1, thank you for sharing. Being the developer's tool for HTTP server
deployment, we can better meet developers' common expectations.
It will obviously rattle the cage that the gray beards have come to
expect (yes, including me), but these discussions are productive if
we collectively find good compromises which allow innovation,
predictability and stability. And which don't add make-work (hopefully
eliminate some) to contributor's efforts.

I'd rather ask RM's to spend 2 hrs on a release if it enables innovators
to conserve their development time and energy to actually coding and
not maintaining forks.