Direction was Re: mod_custom_log exits too late?

[Justin Erenkrantz <je...@apache.org>]

On Sun, Sep 15, 2002 at 09:37:19AM -0700, Ian Holsman wrote:
> maybe we could sit down and think for 2 seconds and get a plan going
> on what we want to see in 2.0 in the next couple of months, and what 
> kind of API changes are required to do it, instead of just hacking or 
> changing something because (it's ugly, run's .001% faster, supports 
> feature XYZ, insert other reason here)

It's a noble cause, but I believe it's misguided.  Our goal is to
produce the best code we can - that means if we can make it run
.001% faster, we should do that.  If we want features, we can add
them.  If we think the code is ugly, we should rectify that.  There
should be no excuse for leaving code that isn't 100% right.

We honestly don't know what changes are going to be required or
what needs to be fixed.  Setting a deadline of 'get all of your
changes done in a month' is probably unrealistic.

We just can't force all of the developers to sit down and work on
the APIs - the best we can hope is to be open to people who do want
to change the APIs.  The only way to really address this is to get
people to write modules for 2.0 and tell us what is deficient in the
current APIs.  That means people need to use the code.  And, hence,
we must be patient rather than 'hurry up and freeze.'

Take for example, what would happen if we freeze and a PHP developer
rewrites the apache2filter and says, "Gee, this module could be
cleaner if we could only do XYZ?"  I think if XYZ is reasonable, we
should implement that change - even if it means breaking backwards
compatibility.  If PHP could take advantage of feature XYZ to be
cleaner, then I think other modules could benefit as well.

But, any type of freeze without a corresponding opening of 2.1 is
going to be detrimental to our development.  If you ask me, if we
really wanted a 2.1, the aaa rewrite was the perfect jumping-off
point.  Yet, the group wasn't supportive of that decision.  Fine.
But, we all have to live by consensus - even if we don't like the
end result.  That's how we work.

> all these constant changes just point to a lack of design & forethought 
> by us (the developers) going into 2.0 over the last 6 months.
>
> The proxy code shows this problem all the way through the changelog. 90% 
> of the changes done to it were due to API changes, and then reversing 
> out that change and re-reversing it back.

I don't think so.  I think our design process is evolutionary.  We
try things out, they work, or they don't.  We move on.  The guiding
belief is that we are somehow making forward progress when we commit.

We don't have a 'big' design document because it's impossible to
have one in a volunteer community.  The only thing we must have is
a willingness to be open to the fact that what we have isn't
optimal and being willing to change when proven wrong.

> as a module developer I don't have the time or patience to constantly 
> change things every month.

I look at it this way.

Is httpd-2.0 a stable web server?  Yes.
Is httpd-2.0 a mature platform?  No.

Two different issues.  When we say a release is GA, we mean that
it'll serve pages; not that it is a set-in-stone API.  We just
haven't had enough real-world experience and feedback on what has
worked and what hasn't to freeze the APIs.  2.0 is still a platform
in its infancy (we're around six months since we've gone GA).

Take a look at the 1.3 release cycle.  1.3.0 was released in June,
1998.  A lot of people said it stabilized around 1.3.9 - released
in August 16, 1999.  It was about that time we split off to 2.0 in
its current repository (httpd-2.0).  So, that seems about right - it
took over a year to reach stability.

I don't think we can make the stabilization go much faster than they
did for 1.3.  The one thing we can do better is making frequent
releases - which we're accomplishing.  We've been consistently
producing GA-quality releases about once a month since April.

And, if it helps, the features I'm probably going to be working on
or wanting to see in the next few months are (in no particular order):

- Finish off the aaa rewrite
- Merge in LDAP to aaa proper
- Add a bandwidth-monitoring module (similar to Bojan's logio)
  (It'd be awesome if people beat me to this - they just might!)
- Rewrite mod_dav to produce streamy results
- Add DAV ACL support to mod_dav

For better or worse, I can forsee how each of these issues could
change some set of APIs for the better as a result of experience.

Since I do this in my spare time (heh), I can't reasonably commit to
any type of deadlines - these changes will happen when the code
materializes.  Either I'll get to it, or someone else will.

And, I'm sure there will be other things, I just don't know what
they will be, and I'm not qualified to look beyond that.  And, I
doubt any of us are.  -- justin

Re: Direction was Re: mod_custom_log exits too late?

[Jeff Stuart <js...@computer-city.net>]

I'd also add the following: right now since both mod_perl AND PHP aren't 
"released" yet for Apache 2.0, a large (at least in MY opinion :)) number of 
sites/servers CAN'T upgrade.  IE all of my work is either PHP or mod_perl.  
With neither in stable status for Apache 2.0 ... I'm SOL.

Now why do I WANT to upgrade to 2.0?  Well, when mod_perl comes out for apache 
2.0 AND if it comes out according to the design docs they were talking about 
a while ago, then I can do mod_perl work using a threaded apache server which 
just acts as an app server which is REALLY REALLY REALLY slick (IMHO). :)  

At the moment, I'm stuck even worse since right now, Subversion "requires" 
Apache CVS, I can't even install SVN to test IT.  (IE I REALLY don't like the 
idea of grabbing a CVS version of the httpd code alogn with SVN bleeding edge 
code and trust my source code repository AND my netsaint output at the same 
time.  Just leaves me feeling REALLY scared!) 

My personal reason to upgrade will be to be able to process more requests on 
the same hardware.  IE the threaded models.  Also would be the POSSIBILITY of 
setting up apache as an application server that either does it's work via 
mod_perl pages directly OR via mod_perl/soap which I THINK Apache 2.0 will be 
able to handle quite well.

On Sunday 15 September 2002 07:41 pm, rbb@apache.org wrote:
[...large parts of message deleted...]

> I realize that my thoughts on Apache 2.0 adoption rates will not be
> popular, but I have not come up with a compelling reason for people to
> upgrade yet.  Without a compelling reason, the only people that you are
> going to get right now are early adopters.  Others will come in time, but
> you need to be patient.  Rasmus is convinced that PerChild is the
> compelling reason, I am not as convinced as he is, but I am willing to
> consider it.  What other compelling reasons do you see?  Why should people
> upgrade?  And, I would ask that you come at this from a users point of
> view, not a module authors point of view.
>
> Ryan

-- 
Jeff Stuart
jstuart@computer-city.net

Re: Direction was Re: mod_custom_log exits too late?

[rb...@apache.org]

> and as far as apache2 being stable... I don't think it is. The code might
> serve pages, but people are voting with their feet, and so far only 6000
> feet are saying that it is.

I am going to ask a rather harsh question.  Why do people think that
anybody would upgrade to 2.0?  I tried to convince some people a while ago
that 2.0 wasn't going to have a fast adoption rate.  My reasoning?  It
doesn't solve a real-world problem.  The reason for Apache 2.0 was simple
enough, a company wanted to speed up Apache on their version of Unix,
because it didn't handle the pre-forking model well.  However, for most
sites, Apache 1.3 does just work.

There are three types of people who really need Apache 2.0.

1) Incredibly large sites that need to serve a huge number of users.  This
is a VERY small set of users, especially since most sites aren't hung up
on what their web server can do, but rather how quickly their CGI or
servlet can generate pages.

2) Sites that would rather spend money on a few large machines rather than
a lot of small machines.  The world is moving to disposable server
machines.  In that model, you buy a lot of small boxes, and don't worry
about the speed of the software.  The reason for this migration, is that
disposable hardware is cheaper, and it has a lower cost when 1 machine
dies.

3) Developers.  Apache 2.0 is a lot more modular, and IMHO, is a nicer
platform to develop on.

This is VERY different from Apache 1.3, which solves some real problems
with the previous version of Apache.  Add to that, the threaded models of
Apache 2.0 doesn't work with some of the most popular Apache 1.3 modules.

As for the argument that people don't like having to re-compile your
modules whenever we release a new version of the server, I propose a few
things.  First of all, how often have you had to modify your module to
make it work?  Is it that we are changing APIs that you use, or that we
are bumping the MMN?  If it is the former, then I don't have a solution,
but for the later I do.

If more modules would use the Apache build system to do builds, this
problem wouldn't exist.  We created the build system to allow modules to
add their own config.m4 files.  If you use CVS to checkout Apache 2.0, you
can rely on CVS to merge the m4 files for you.  This means that as a part
of building the new version of Apache, your modules will also be built.

I realize that my thoughts on Apache 2.0 adoption rates will not be
popular, but I have not come up with a compelling reason for people to
upgrade yet.  Without a compelling reason, the only people that you are
going to get right now are early adopters.  Others will come in time, but
you need to be patient.  Rasmus is convinced that PerChild is the
compelling reason, I am not as convinced as he is, but I am willing to
consider it.  What other compelling reasons do you see?  Why should people
upgrade?  And, I would ask that you come at this from a users point of
view, not a module authors point of view.

Ryan

Re: Direction was Re: mod_custom_log exits too late?

[Ian Holsman <li...@holsman.net>]

On Sun, 15 Sep 2002 05:21:27 -0700, Justin Erenkrantz wrote:

> On Sun, Sep 15, 2002 at 09:37:19AM -0700, Ian Holsman wrote:
>> maybe we could sit down and think for 2 seconds and get a plan going on
>> what we want to see in 2.0 in the next couple of months, and what kind
>> of API changes are required to do it, instead of just hacking or
>> changing something because (it's ugly, run's .001% faster, supports
>> feature XYZ, insert other reason here)
> 
> It's a noble cause, but I believe it's misguided.  Our goal is to produce
> the best code we can - that means if we can make it run .001% faster, we
> should do that.  If we want features, we can add them.  If we think the
> code is ugly, we should rectify that.  There should be no excuse for
> leaving code that isn't 100% right.
> 
poppycock.

you just mentioned 4 things you would like to see happen in the next
couple of months each which would require a MMN jump.

all I am asking is that we start a list of these things, and possibly a
paragraph describing what needs to be changed (not a weighty tome as you
seem thing a design doc is) and stick it in the status.

Some of these changes to the API could be done before the new
functionality is added, combining the MMN jumps into a 'refactoring'
release.

as for we don't know what is coming ahead of us.. while we can never
anticipate all the things (like your PHP developer example) which are
going to happen, we CAN start making choices now for >WE< the apache
server see us going in the next 3-6 months so we can choose our direction
instead of just stumbling around in the dark and adding features and
functionaility in such a adhoc manner

I'm going to stop, as I know there are certain comitters who hate the
concept of thinking ahead, and will never even consider thinking of where
they want apache to be in 3 months.

and as far as apache2 being stable... I don't think it is. The code might
serve pages, but people are voting with their feet, and so far only 6000
feet are saying that it is.

--Ian


> We honestly don't know what changes are going to be required or what needs
> to be fixed.  Setting a deadline of 'get all of your changes done in a
> month' is probably unrealistic.
> 
> We just can't force all of the developers to sit down and work on the APIs
> - the best we can hope is to be open to people who do want to change the
> APIs.  The only way to really address this is to get people to write
> modules for 2.0 and tell us what is deficient in the current APIs.  That
> means people need to use the code.  And, hence, we must be patient rather
> than 'hurry up and freeze.'
> 
> Take for example, what would happen if we freeze and a PHP developer
> rewrites the apache2filter and says, "Gee, this module could be cleaner if
> we could only do XYZ?"  I think if XYZ is reasonable, we should implement
> that change - even if it means breaking backwards compatibility.  If PHP
> could take advantage of feature XYZ to be cleaner, then I think other
> modules could benefit as well.
> 
> But, any type of freeze without a corresponding opening of 2.1 is going to
> be detrimental to our development.  If you ask me, if we really wanted a
> 2.1, the aaa rewrite was the perfect jumping-off point.  Yet, the group
> wasn't supportive of that decision.  Fine. But, we all have to live by
> consensus - even if we don't like the end result.  That's how we work.
> 
>> all these constant changes just point to a lack of design & forethought
>> by us (the developers) going into 2.0 over the last 6 months.
>>
>> The proxy code shows this problem all the way through the changelog. 90%
>> of the changes done to it were due to API changes, and then reversing
>> out that change and re-reversing it back.
> 
> I don't think so.  I think our design process is evolutionary.  We try
> things out, they work, or they don't.  We move on.  The guiding belief is
> that we are somehow making forward progress when we commit.
> 
> We don't have a 'big' design document because it's impossible to have one
> in a volunteer community.  The only thing we must have is a willingness to
> be open to the fact that what we have isn't optimal and being willing to
> change when proven wrong.
> 
>> as a module developer I don't have the time or patience to constantly
>> change things every month.
> 
> I look at it this way.
> 
> Is httpd-2.0 a stable web server?  Yes. Is httpd-2.0 a mature platform? 
> No.
> 
> Two different issues.  When we say a release is GA, we mean that it'll
> serve pages; not that it is a set-in-stone API.  We just haven't had
> enough real-world experience and feedback on what has worked and what
> hasn't to freeze the APIs.  2.0 is still a platform in its infancy (we're
> around six months since we've gone GA).
> 
> Take a look at the 1.3 release cycle.  1.3.0 was released in June, 1998. 
> A lot of people said it stabilized around 1.3.9 - released in August 16,
> 1999.  It was about that time we split off to 2.0 in its current
> repository (httpd-2.0).  So, that seems about right - it took over a year
> to reach stability.
> 
> I don't think we can make the stabilization go much faster than they did
> for 1.3.  The one thing we can do better is making frequent releases -
> which we're accomplishing.  We've been consistently producing GA-quality
> releases about once a month since April.
> 
> And, if it helps, the features I'm probably going to be working on or
> wanting to see in the next few months are (in no particular order):
> 
> - Finish off the aaa rewrite
> - Merge in LDAP to aaa proper
> - Add a bandwidth-monitoring module (similar to Bojan's logio)
>   (It'd be awesome if people beat me to this - they just might!)
> - Rewrite mod_dav to produce streamy results - Add DAV ACL support to
> mod_dav
> 
> For better or worse, I can forsee how each of these issues could change
> some set of APIs for the better as a result of experience.
> 
> Since I do this in my spare time (heh), I can't reasonably commit to any
> type of deadlines - these changes will happen when the code materializes. 
> Either I'll get to it, or someone else will.
> 
> And, I'm sure there will be other things, I just don't know what they will
> be, and I'm not qualified to look beyond that.  And, I doubt any of us
> are.  -- justin