Thoughts on 2.5.0

[Jim Jagielski <ji...@jaguNET.com>]

I would like to start a discussion on 2.5.0 and give
some insights on my thoughts related to it.

First and foremost: there is cool stuff in 2.5.0
that I really, REALLY wish were in a releasable, usable
artifact. Up to now, we've been doing these as backports
to the 2.4.x tree with, imo at least, good success.

So I think the main questions regarding 2.5.0 is a list
of items/issues that simply *cannot* be backported, due
to API/ABI concerns. And then gauge the "value" of
the items on that list.

Another would be to look at some of the items currently
"on hold" for backporting, due to outstanding questions,
tech issues, more needed work/QA, etc... IMO, if these
backports lack "support" for 2.4.x, then I wonder how
"reliable" they are (or how tested they are) in the 2.5.o
tree. And if the answer is "we pull them out of 2.5.0"
then the question after that is what really *are* the
diffs between 2.5.0 and 2.4.x... If, however, the
answer is "tagging 2.5.0 will encourage us to address
those issues" then my question is "Why aren't we doing
that now... for 2.4.x".

And finally: 2.4.x is now, finally, being the default
version in distros, being the go-to version that people
are using, etc... I would like us to encourage that
momentum.

Re: Thoughts on 2.5.0

[William A Rowe Jr <wr...@rowe-clan.net>]

On Tue, Oct 24, 2017 at 12:39 PM, Jim Jagielski <ji...@jagunet.com> wrote:
> On Tue, Oct 24, 2017 at 11:20 AM, William A Rowe Jr <wr...@rowe-clan.net> wrote:
>> On Tue, Oct 24, 2017 at 8:42 AM, Jim Jagielski <ji...@jagunet.com> wrote:
>>>
>>> Another would be to look at some of the items currently
>>> "on hold" for backporting, due to outstanding questions,
>>> tech issues, more needed work/QA, etc... IMO, if these
>>> backports lack "support" for 2.4.x, then I wonder how
>>> "reliable" they are (or how tested they are) in the 2.5.o
>>> tree. And if the answer is "we pull them out of 2.5.0"
>>> then the question after that is what really *are* the
>>> diffs between 2.5.0 and 2.4.x... If, however, the
>>> answer is "tagging 2.5.0 will encourage us to address
>>> those issues" then my question is "Why aren't we doing
>>> that now... for 2.4.x".
>>
>> I am not doing it for 2.4.x, and will continue not to do it for 2.4.x,
>> because I won't contribute to further destabilizing 2.4.x current
>> releases.
>
> So you refuse to help stabilize the 2.4.x tree, but then complain
> that the 2.4.x tree is unstable.

Because introducing unwise enhancements makes it so; because branch
hygiene makes it so. I'm not going to battle either windmill, since
current practice dictates that it is a losing battle.

I'll continue to review and vote up bug fixes. I'll continue to push
bugfixes I write at 2.4.x. I'll continue to skim enhancements for
breakage that I can spot.

> OK. That's fine. Your cycles are yours to use as you wish.

There we go again with the personal swipes. Please speak to process
and code, not people.

This is exactly the point, of course. Those who wanted to maintain 2.2
did so, irrespective of what those other committers, who could not
have cared less, chose to work on. And so it will be after 2.4,
committers can keep maintaining 2.4, during the 2.5 cycle and beyond
2.6.0/3.0.0. Those who simply want to develop can develop.

Re: Thoughts on 2.5.0

[Jim Jagielski <ji...@jaguNET.com>]

> 
> I am not doing it for 2.4.x, and will continue not to do it for 2.4.x,
> because I won't contribute to further destabilizing 2.4.x current
> releases.

So you refuse to help stabilize the 2.4.x tree, but then complain
that the 2.4.x tree is unstable.

OK. That's fine. Your cycles are yours to use as you wish.

Re: Thoughts on 2.5.0

[Jim Jagielski <ji...@jaguNET.com>]

> On Oct 24, 2017, at 10:20 PM, Jacob Champion <ch...@gmail.com> wrote:
> 
> On 10/24/2017 11:45 AM, William A Rowe Jr wrote:
>> On Tue, Oct 24, 2017 at 12:35 PM, Jim Jagielski <ji...@jagunet.com> wrote:
>>> That is way when we backport we transition to RTC, because
>>> we want to ENSURE it's been reviewed.
>> Wrong. I was there. RTC was adopted in order to ensure both the
>> reliability but moreso, the compatibility of changes during a given
>> x.y major/minor release cycle. CTR existed to make forward progress
>> and get out of our developers' way.
> 
> I'm not really sure Jim's statement here is "wrong". Regardless of historical context. We use RTC when we want to guarantee review.
> 
>> You are suggesting a change of policy. It
>> is not policy to use RTC to get from trunk to 2.6.0, and will not
>> become policy without a vote for such a change by the PMC.
> 
> I'm not entirely convinced Jim is suggesting a change in policy, as you say. But in any case, I would be +1 to such a change. We should not be releasing a 2.6.x that contains unreviewed code.
> 

It's not a policy change, right. It's a clarification of the policy. CTR was to
allow for fast development. As Bill said, so the process got out of
the developers way. But we also ensured RTC such that anything
that made its way *into a release branch* was "formally" reviewed.

Thx!

Re: Thoughts on 2.5.0

[Jacob Champion <ch...@gmail.com>]

On 10/24/2017 11:45 AM, William A Rowe Jr wrote:
> On Tue, Oct 24, 2017 at 12:35 PM, Jim Jagielski <ji...@jagunet.com> wrote:
>> That is way when we backport we transition to RTC, because
>> we want to ENSURE it's been reviewed.
> 
> Wrong. I was there. RTC was adopted in order to ensure both the
> reliability but moreso, the compatibility of changes during a given
> x.y major/minor release cycle. CTR existed to make forward progress
> and get out of our developers' way.

I'm not really sure Jim's statement here is "wrong". Regardless of 
historical context. We use RTC when we want to guarantee review.

> You are suggesting a change of policy. It
> is not policy to use RTC to get from trunk to 2.6.0, and will not
> become policy without a vote for such a change by the PMC.

I'm not entirely convinced Jim is suggesting a change in policy, as you 
say. But in any case, I would be +1 to such a change. We should not be 
releasing a 2.6.x that contains unreviewed code.

--Jacob

Re: Thoughts on 2.5.0

[William A Rowe Jr <wr...@rowe-clan.net>]

On Tue, Oct 24, 2017 at 1:45 PM, William A Rowe Jr <wr...@rowe-clan.net> wrote:
>
> Proceeding as documented and practiced, between trunk and 2.6.0 tag,
> we operate under RTC until the committee adopts a rewritten policy.

under CTR, of course :)

Re: Thoughts on 2.5.0

[William A Rowe Jr <wr...@rowe-clan.net>]

On Tue, Oct 24, 2017 at 12:35 PM, Jim Jagielski <ji...@jagunet.com> wrote:
>
>>  That's what commit-then-review means. If you failed to
>> review it, you now have a six year knowledge gap and review to
>> conduct. That's not sustainable, nobody at the project has that kind
>> of time.
>
> "Review" does not have a time limit. Anyone can, and should, review
> whenever they wish.

Correct.

> Just because something has been folded
> in does not mean it has necessarily been reviewed.

Nor does it mean it hasn't.

> That is way when we backport we transition to RTC, because
> we want to ENSURE it's been reviewed.

Wrong. I was there. RTC was adopted in order to ensure both the
reliability but moreso, the compatibility of changes during a given
x.y major/minor release cycle. CTR existed to make forward progress
and get out of our developers' way.

We did not get to 2.0.35 GA under RTC. We did not get to 2.2.0 GA
under RTC, we did not get to 2.4.1 GA under RTC.

Proceeding as documented and practiced, between trunk and 2.6.0 tag,
we operate under RTC until the committee adopts a rewritten policy.

> Just tagging/releasing something that has been CTR does not
> automatically nor magically make all the commits "reviewed".

Nor does it indicate they were *not* reviewed. It is not our
responsibility to account to you what we collectively have reviewed
over the past six years because others failed to review what is
committed to the repository. You are suggesting a change of policy. It
is not policy to use RTC to get from trunk to 2.6.0, and will not
become policy without a vote for such a change by the PMC.

You are contriving policy; I don't know what your agenda is until you
put forward a policy change which we can debate as a community. But I
am proceeding under current policy and the mechanisms which ultimately
got us to 2.0, 2.2 and 2.4.

> Nor does it mean that people cannot re-review and even veto
> the commit.
>
> These are bits, not concrete.

Agreed.

Re: Thoughts on 2.5.0

[Jim Jagielski <ji...@jaguNET.com>]

>  That's what commit-then-review means. If you failed to
> review it, you now have a six year knowledge gap and review to
> conduct. That's not sustainable, nobody at the project has that kind
> of time.

"Review" does not have a time limit. Anyone can, and should, review
whenever they wish. Just because something has been folded
in does not mean it has necessarily been reviewed. That is way
when we backport we transition to RTC, because we want to ENSURE
it's been reviewed.

Just tagging/releasing something that has been CTR does not
automatically nor magically make all the commits "reviewed".
Nor does it mean that people cannot re-review and even veto
the commit.

These are bits, not concrete.

Re: Thoughts on 2.5.0

[William A Rowe Jr <wr...@rowe-clan.net>]

I will preface by stating that you are referring to 2.6.0 or 3.0.0,
our next GA, which is not yet what I've suggested on list. I'll start
another thread on 2.5.0 development branch, and run with your
discussion of the next GA...


On Tue, Oct 24, 2017 at 8:42 AM, Jim Jagielski <ji...@jagunet.com> wrote:
> I would like to start a discussion on 2.5.0 and give
> some insights on my thoughts related to it.
>
> First and foremost: there is cool stuff in 2.5.0
> that I really, REALLY wish were in a releasable, usable
> artifact. Up to now, we've been doing these as backports
> to the 2.4.x tree with, imo at least, good success.

The 2.4.x release series has a >50% regression rate release by
release, comparing released 2.4.n to 2.4.prev. I don't consider that
'good'. This is because 2.4 is not a maintenance branch, and is
therefore not stable. But we don't have to extend this conversation
because I brought up the concern previously and the concern was
rejected. A 2.5.x alpha preview branch *with community testers* would
have avoided a great number of these regressions, if the community had
a chance to test and provide feedback.

> So I think the main questions regarding 2.5.0 is a list
> of items/issues that simply *cannot* be backported, due
> to API/ABI concerns. And then gauge the "value" of
> the items on that list.

Disagreed. That the code was committed and passively vetoed due to an
unwillingness of the project to release it.

httpd 2.4.0 was forked at r1200449. This happened 6 years ago as of
this coming Nov 10th.

What is that code? Discounting all changes to docs/, all changes in
whitespace, line endings and propchanges, the remaining delta is;

 + 36654 lines + 1014607 non-whitespace characters
 - 17198 lines - 610777 non-whitespace characters

> Another would be to look at some of the items currently
> "on hold" for backporting, due to outstanding questions,
> tech issues, more needed work/QA, etc... IMO, if these
> backports lack "support" for 2.4.x, then I wonder how
> "reliable" they are (or how tested they are) in the 2.5.o
> tree. And if the answer is "we pull them out of 2.5.0"
> then the question after that is what really *are* the
> diffs between 2.5.0 and 2.4.x... If, however, the
> answer is "tagging 2.5.0 will encourage us to address
> those issues" then my question is "Why aren't we doing
> that now... for 2.4.x".

I am not doing it for 2.4.x, and will continue not to do it for 2.4.x,
because I won't contribute to further destabilizing 2.4.x current
releases. It takes a serious vulnerability for me to risk the
stability of 2.4.x, something on the order of the HTTP conformance
issues we faced over the past year. Once again, 4 1/2 year old code
had never been shared with the community, once again that ignored code
contained defects that would have been noticed in a public -alpha over
a four year period.

I am for doing this necessary review for 2.5.0, but all the code was
reviewed. That's what commit-then-review means. If you failed to
review it, you now have a six year knowledge gap and review to
conduct. That's not sustainable, nobody at the project has that kind
of time. As PMC and committers, we had these commits in our inboxes.
We questioned committers about changes that looked incorrect. We
vetoed some changes that didn't fit or wouldn't work.

It's over-past time for that code, those committers' efforts to see
the light of day, even as an alpha for the time being. Any argument to
the contrary or new layers of process is disingenuous at best, and
obstructionist at worst.

> And finally: 2.4.x is now, finally, being the default
> version in distros, being the go-to version that people
> are using, etc... I would like us to encourage that
> momentum.

No, 2.4.current is not the default. Our current n.n.n at the time
these distributions decided to freeze is the distributed version, then
they pick up security backports.

RHEL 7 and CentOS 7 are still 2.4.6. Ubuntu 16.04 is still 2.4.7, or
2.4.10 from backports, xenial is 2.4.18 and zesty is 2.4.25

https://w3techs.com/technologies/details/ws-apache/2.4/all

57% of all the deployed httpd 2.4 are one of the first four ancient
versions, and 67% if you count 2.4.25 (likely a mix of zesty and
non-zesty not updated in 4 months). Of that, 2.4 captures 51% and 49%
are older still (2.2/2.0); we can slice those adoption numbers in
half.

So 1/3rd of httpd deployments are on one of the five versions
distributed in these five operating system distributions.

Only 11% of the deployments have been refreshed since our June release.

If you are talking distributions, they will be at whatever we give
them once they ship. They won't pick up our new features until they
are ready to tag a new OS release, so 'distros', at least 'os distros'
is non sequitur. If you are talking httpd binaries, every creator has
their own schema; at Pivotal (and JBoss and similar, I suspect) we
prepared to pick up all releases but dropped the ones with quickly
identified regressions. Others, like Steffen's or CPanel's, attempt to
deliver every httpd release rain or shine AIUI. All of the later will
continue on into 2.6.0 or 3.0.0 or 4.0.0. Which OS distributors today
are still going to release a new OS on linux kernel 2.x?

But let's quantify in terms of actual data. In the time since we
forked at Nov 10 2011, httpd 2.x over 1.x grew from 90% to 99% of our
user base, 2.0.x legacy fell from 24% to 1%. Of the total webserver
footprint IIS fell from 18% to 11%, Apache httpd from 70% to 48%, and
ngnix grew from 6% to 36%, eating everyone's lunch.

Now this map https://w3techs.com/blog/entry/nginx_reaches_33_3_percent_web_server_market_share_while_apache_falls_below_50_percent
explains why this is; Microsoft has been much more proactive in
providing a server with documentation to the Chinese market, ngnix to
Russian speaking states. But consider in this time period that ngnix,
with an API started from scratch (no httpd 0.x legacy there, so little
deprecation) has run from 1.1 to 1.13... they also follow an
odds/evens schema, so counting only the development releases (our
2.{even}s, we have no stable per se until something hits legacy),
that's 6 significant minor releases (multiplied by many subversion
releases) to no minor releases at httpd. So while it may be native
language support and other good debates between the two technologies,
these minor releases obviously caused ngnix no pain, it ships what was
then-current in os distributions, and perhaps our unwillingness to
release code has contributed to the overall adoption of httpd.
Preserving the "2.4.x" designation has won us nothing.

Re: Thoughts on 2.5.0

[Mark Blackman <ma...@exonetric.com>]

> On 24 Oct 2017, at 14:42, Jim Jagielski <ji...@jaguNET.com> wrote:
> 
> I would like to start a discussion on 2.5.0 and give
> some insights on my thoughts related to it.
> 
> First and foremost: there is cool stuff in 2.5.0
> that I really, REALLY wish were in a releasable, usable
> artifact. Up to now, we've been doing these as backports
> to the 2.4.x tree with, imo at least, good success.
> 
> So I think the main questions regarding 2.5.0 is a list
> of items/issues that simply *cannot* be backported, due
> to API/ABI concerns. And then gauge the "value" of
> the items on that list.
> 
> Another would be to look at some of the items currently
> "on hold" for backporting, due to outstanding questions,
> tech issues, more needed work/QA, etc... IMO, if these
> backports lack "support" for 2.4.x, then I wonder how
> "reliable" they are (or how tested they are) in the 2.5.o
> tree. And if the answer is "we pull them out of 2.5.0"
> then the question after that is what really *are* the
> diffs between 2.5.0 and 2.4.x... If, however, the
> answer is "tagging 2.5.0 will encourage us to address
> those issues" then my question is "Why aren't we doing
> that now... for 2.4.x".
> 
> And finally: 2.4.x is now, finally, being the default
> version in distros, being the go-to version that people
> are using, etc... I would like us to encourage that
> momentum.

As an observer, I’d like to ask what your goals are for these branches and what kinds of expectations would you like consumers of these branches to have?

- Mark