3.0 - Introduction

[Paul Querna <ch...@force-elite.com>]

I believe the httpd project is ready for a push towards the next major
version.

I believe everyone involved has learned many things from 2.x.  I wasn't
here for all of the early 2.x development, so it is very easy to say I
am naive in the scope of something like pushing for 3.0.

Today, I view the following as largely unsolvable problems with the 2.x
architecture:

- Async IO will not work in the core without committing more evil hacks,
that will make the code harder to understand and follow.

- Async IO will not work correctly with filters today.

- The module API exposes too many internals of how client IO is done to
make small incremental changes.

- The HTTP protocol is still married to the core, and there has been
very little progress in separating it out.

- Performance for many metrics will never be fully on par to async or
hybrid async/threaded based servers like lighttpd.

- The configuration subsystem does not enable runtime changes, or the
ability to easily programmatically extend it.

Could we try to address these issues and more with more constrained
changes?  Very likely.

Do I think it would be as fun for everyone involved? Nope.

I don't believe that a focus on 3.0 will replace 2.x overnight, just
like 2.x has taken many years to replace 1.3.x on new installs.  I
personally don't view this as a major problem; With time, and if the
project is truly better, users will migrate.  I do think we need a
better strategy for handling dynamic content generators, and things that
 are not thread safe.

I personally believe the push for 3.0 needs to be focused on how to
create a positive scratch your own itch for most of the developer. So,
in that spirit, what bothers everyone else about 2.x?

-Paul

Re: 3.0 - Introduction

[Jim Jagielski <ji...@jaguNET.com>]

On Feb 14, 2007, at 2:32 AM, Paul Querna wrote:

> I believe the httpd project is ready for a push towards the next major
> version.
>
>
> - Async IO will not work in the core without committing more evil  
> hacks,
> that will make the code harder to understand and follow.
>
> - Async IO will not work correctly with filters today.
>

An Async/Event-based core framework is likely the leading
design aspect of 3.0. Everything else will fall out from
that, imo.

Re: 3.0 - Introduction

[david reid <da...@jetnet.co.uk>]

Paul Querna wrote:
> I believe the httpd project is ready for a push towards the next major
> version.
> 
> I believe everyone involved has learned many things from 2.x.  I wasn't
> here for all of the early 2.x development, so it is very easy to say I
> am naive in the scope of something like pushing for 3.0.
> 
> Today, I view the following as largely unsolvable problems with the 2.x
> architecture:
> 
> - Async IO will not work in the core without committing more evil hacks,
> that will make the code harder to understand and follow.
> 
> - Async IO will not work correctly with filters today.
> 
> - The module API exposes too many internals of how client IO is done to
> make small incremental changes.
> 
> - The HTTP protocol is still married to the core, and there has been
> very little progress in separating it out.
> 
> - Performance for many metrics will never be fully on par to async or
> hybrid async/threaded based servers like lighttpd.
> 
> - The configuration subsystem does not enable runtime changes, or the
> ability to easily programmatically extend it.
> 
> Could we try to address these issues and more with more constrained
> changes?  Very likely.
> 
> Do I think it would be as fun for everyone involved? Nope.
> 
> I don't believe that a focus on 3.0 will replace 2.x overnight, just
> like 2.x has taken many years to replace 1.3.x on new installs.  I
> personally don't view this as a major problem; With time, and if the
> project is truly better, users will migrate.  I do think we need a
> better strategy for handling dynamic content generators, and things that
>  are not thread safe.
> 
> I personally believe the push for 3.0 needs to be focused on how to
> create a positive scratch your own itch for most of the developer. So,
> in that spirit, what bothers everyone else about 2.x?

While I appreciate the work that has been done on our auth model, it's
still a nightmare and not as flexible as we should have. I'd like to see
us fully support group auth and allowing "layered" approaches to auth.


-- 
david

http://feathercast.org/

Re: 3.0 - Introduction

[Shankar Unni <sh...@netscape.net>]

Paul Querna wrote:

> - The configuration subsystem does not enable runtime changes, or the
> ability to easily programmatically extend it.

Talking of configuration subsystem - it would be a good thing to expose 
the configuration mechanism to modules so that they can query core 
configuration parameters (e.g. User/Group, Doc base, Server name, Listen 
directives, etc.), as well as check for the presence of, and read the 
configuration of, other modules.

Re: 3.0 - Introduction

[Jorge Schrauwen <jo...@gmail.com>]

On 2/14/07, Paul Querna <ch...@force-elite.com> wrote:

> I personally believe the push for 3.0 needs to be focused on how to
> create a positive scratch your own itch for most of the developer. So,
> in that spirit, what bothers everyone else about 2.x?
>
>
- Things I find lacking in 2.x are like you said the httpd core tie-up.
projects like mod_ftp are great but you are stuck with a mpm's build for
http server.

- Config File Maintance! mod_macro does agreat job at making it easier.
  e.g.: you define a macro for a vhost and can just pass parameters to it.
  If you need to change something you don't have to edit all the vhost
blocks just one.

- Config File Layout: Its still a text file. This has it's advantage but
personaly I find xml much more readable and easier to follow the flow of a
huge file.

thats about it from what comes to my mind

-- 
~Jorge

Re: 3.0 - Introduction

[Brian Akins <br...@turner.com>]

Make everything possible into a hook or use the provider model.

Simple example: the way we determine if a connection can be kept alive is a 
monolithic function.  This should be a hook.

Disk I/O (read/write/seek, etc.) could be abstracted by providers, for example. 
  Maybe we need full blown VFS??



-- 
Brian Akins
Chief Operations Engineer
Turner Digital Media Technologies

Re: 3.0 - Introduction

[Justin Erenkrantz <ju...@erenkrantz.com>]

On 2/14/07, Paul Querna <ch...@force-elite.com> wrote:
> +1 to giving it a code name; '3.0' to me is just an easy way to explain
> 'this is a major departure from the current 2.x'.

I vote for using "Monsters Inc" characters...say Sulley or Boo.  =)
FWIW, I spawned this off Paul's suggestion on IRC of poking fun at
Debian's Toy Story codenames.  -- justin

Re: 3.0 - Introduction

[Jorge Schrauwen <jo...@gmail.com>]

On 2/19/07, Mads Toftum <ma...@toftum.dk> wrote:
>
> On Wed, Feb 14, 2007 at 07:26:57PM -0800, Justin Erenkrantz wrote:
> > As long as we first teach httpd/Subversion how to deal with all of the
> > spelling mistakes from us trying to spell out Dutch words.  I want
> > something easy to remember *and* type!  -- justin
>
> How about: D
> That whole fascination with Dutch things seem pretty silly random idea.
> Now that people are talking about waka support and http as a module, D
> seems more appropriate than ever.
>
>
D seems rather short IMHO. ApacheD maybe?

-- 
~Jorge

Re: 3.0 - Introduction

[Sander Temme <sc...@apache.org>]

On Feb 19, 2007, at 9:47 AM, Mads Toftum wrote:

> That whole fascination with Dutch things seem pretty silly random  
> idea.

Well, we, the community, *are* going there in just over two months,  
and I assume that we'll have at least a BOF to gather input/discussion.

S. (/me volunteers to put that BOF on the board if no one beats /me  
to it)

-- 
sctemme@apache.org            http://www.temme.net/sander/
PGP FP: 51B4 8727 466A 0BC3 69F4  B7B8 B2BE BC40 1529 24AF

Re: 3.0 - Introduction

[Mads Toftum <ma...@toftum.dk>]

On Wed, Feb 14, 2007 at 07:26:57PM -0800, Justin Erenkrantz wrote:
> As long as we first teach httpd/Subversion how to deal with all of the
> spelling mistakes from us trying to spell out Dutch words.  I want
> something easy to remember *and* type!  -- justin

How about: D
That whole fascination with Dutch things seem pretty silly random idea. 
Now that people are talking about waka support and http as a module, D
seems more appropriate than ever.

vh

Mads Toftum
-- 
http://soulfood.dk

Re: 3.0 - Introduction

[Justin Erenkrantz <ju...@erenkrantz.com>]

On 2/14/07, Sander Temme <sc...@apache.org> wrote:
> My delightful pleasure. We can really go to town on this. Like,
> neighborhoods and places in Amsterdam:

As long as we first teach httpd/Subversion how to deal with all of the
spelling mistakes from us trying to spell out Dutch words.  I want
something easy to remember *and* type!  -- justin

Re: 3.0 - Introduction

[Sander Temme <sc...@apache.org>]

On Feb 14, 2007, at 3:06 PM, William A. Rowe, Jr. wrote:

> Hey Dirk or Sander(s) - can we have a few cool sounding easy to spell
> place names from the Amsterdam area?

My delightful pleasure. We can really go to town on this. Like,  
neighborhoods and places in Amsterdam:

Jordaan
Vondelpark (most certainly minfrin)
De Pijp
Transvaal
Watergraafsmeer
Osdorp
Betondorp
Bijlmer
Zwanenburg

Or towns around Amsterdam

Zaandam
Purmerend
Amstelveen
Duijvendrecht
Abcoude (where I grew up)
Diemen

Of course we don't have to stay in Amsterdam. As the poet, and  
nighttime Mayor of the City of Rotterdam, Jules Deelder once put it:

Beknopte Topografie van de Rijnmond

Rotterdam
Schiedam
Vlaardingen
Maassluis
Hoekie om
Trappie af
Gekkenhuis

Of course we could call it "The Oosterdok Project" and be very  
circumspect about it for a year or so.

S. (Oosterdok is where our hotel is)

-- 
sctemme@apache.org            http://www.temme.net/sander/
PGP FP: 51B4 8727 466A 0BC3 69F4  B7B8 B2BE BC40 1529 24AF

Re: 3.0 - Introduction

["William A. Rowe, Jr." <wr...@rowe-clan.net>]

Roy T. Fielding wrote:
> 
> I was planning on creating
> 
>    repos/asf/httpd/sandbox/amsterdam/
> 
> and then moving the GSoC stuff to other subdirectories of that
> sandbox.  I prefer to think of branches as forks off of trunk,
> whereas the sandbox would not be.

Ok... that's another way.

http://svn.apache.org/repos/asf/httpd/httpd/branches/

already contains several sandboxes, but these could be moved as well.
I need to jump back into fips-dev now that openssl-fips-1.1.1 has it's
glossy certificate again.

> Although, we could do both, with a bunch of sandboxes to play with
> ideas and then a branch to act as the "what everyone has agreed to
> merge so far" tree.  That is, assuming we don't want to displace
> trunk yet.  *shrug*

If we do decide to open trunk to what become '3.0' changes, I'd only
ask that we take 5 minutes to branch 2.3.x first.

Bill

Re: 3.0 - Introduction

["Roy T. Fielding" <fi...@gbiv.com>]

On Feb 14, 2007, at 3:06 PM, William A. Rowe, Jr. wrote:

> Paul Querna wrote:
>>
>> +1 to moving goal and discussion to a SVN file and starting a sandbox
>> (same level as tags/branches?)
>
> Nooo - it is another sort of branch/, so belongs there.
>
> We could have a separate place, but why?  branches/n.n.x should always
> sort before branches/async-foo, branches/amsterdam and so on.

I was planning on creating

    repos/asf/httpd/sandbox/amsterdam/

and then moving the GSoC stuff to other subdirectories of that
sandbox.  I prefer to think of branches as forks off of trunk,
whereas the sandbox would not be.

Although, we could do both, with a bunch of sandboxes to play with
ideas and then a branch to act as the "what everyone has agreed to
merge so far" tree.  That is, assuming we don't want to displace
trunk yet.  *shrug*

....Roy

Re: 3.0 - Introduction

[Graham Leggett <mi...@sharp.fm>]

William A. Rowe, Jr. wrote:

> Hey Dirk or Sander(s) - can we have a few cool sounding easy to spell
> place names from the Amsterdam area?

The Vondelpark springs to mind - a very cool place to be in summer :)

Regards,
Graham
--

Re: 3.0 - Introduction

["William A. Rowe, Jr." <wr...@rowe-clan.net>]

Paul Querna wrote:
> 
> +1 to moving goal and discussion to a SVN file and starting a sandbox
> (same level as tags/branches?)

Nooo - it is another sort of branch/, so belongs there.

We could have a separate place, but why?  branches/n.n.x should always
sort before branches/async-foo, branches/amsterdam and so on.

Hey Dirk or Sander(s) - can we have a few cool sounding easy to spell
place names from the Amsterdam area?

Re: 3.0 - Introduction

[Paul Querna <ch...@force-elite.com>]

Roy T. Fielding wrote:
> On Feb 13, 2007, at 11:32 PM, Paul Querna wrote:
> 
>> I believe the httpd project is ready for a push towards the next major
>> version.
> 
> So do I.  In fact I was just about to create a sandbox for that purpose
> yesterday, but had to get the crypto stuff sorted out first.
> 
> But do we really want to start by calling it 3.0?  How about if we
> work off of a few code names first?  Say, for example, "amsterdam".
> The reason is because there will be some overlap between ideas of
> how to do certain things, with a variety of overlapping breakage
> that can get pretty annoying if you "just want to get one part working
> first".
> 

+1 to giving it a code name; '3.0' to me is just an easy way to explain
'this is a major departure from the current 2.x'.

> I want people to be able to break things in one tree without blocking
> others.  And then, say once a month, we all agree on what parts are
> "finished" enough to merge into all sandbox trees.
>
> The reason I was about to start the sandbox thing is because I've
> been thinking about moving away from the MPM design.  To be precise,
> I mean that we should get closer to the kernels on the more modern
> platforms and find a way to stay in kernel-land until a valid
> request is received (with load restrictions tested and ipfw applied
> automatically), transform the request into a waka message, and then
> dispatch that request to a process running under a userid that matches
> a prefix on the URI. That's pretty far out, though, and I wouldn't
> want it to stand in the way of any shorter term goals.
> 
> In the mean time, I would like to start the sandbox and move the
> goals and requirements discussion into subversion (where it can be
> saved and comments don't get lost).

+1 to moving goal and discussion to a SVN file and starting a sandbox
(same level as tags/branches?)

-Paul

Re: 3.0 - Introduction

[Garrett Rooney <ro...@electricjellyfish.net>]

On 2/14/07, Davanum Srinivas <da...@gmail.com> wrote:
> Dumb Question: Would all this mean a total(?) rewrite of APR as well?

A total rewrite of APR seems unlikely, but if there are changes people
want made to APR in order to better support new functionality in HTTPD
I don't see why it wouldn't be possible.

-garrett

Re: 3.0 - Introduction

[Davanum Srinivas <da...@gmail.com>]

Dumb Question: Would all this mean a total(?) rewrite of APR as well?

-- dims

On 2/14/07, Aaron Bannert <aa...@clove.org> wrote:
> On Wed, Feb 14, 2007 at 02:10:19PM -0800, Roy T. Fielding wrote:
>
> > But do we really want to start by calling it 3.0?  How about if we
> > work off of a few code names first?  Say, for example, "amsterdam".
> > The reason is because there will be some overlap between ideas of
> > how to do certain things, with a variety of overlapping breakage
> > that can get pretty annoying if you "just want to get one part working
> > first".
> >
> > I want people to be able to break things in one tree without blocking
> > others.  And then, say once a month, we all agree on what parts are
> > "finished" enough to merge into all sandbox trees.
>
> I prefer this rather than going straight to 3.0. Would each sandbox
> correspond to a single new feature prototype?
>
>
> > The reason I was about to start the sandbox thing is because I've
> > been thinking about moving away from the MPM design.  To be precise,
> > I mean that we should get closer to the kernels on the more modern
> > platforms and find a way to stay in kernel-land until a valid
> > request is received (with load restrictions tested and ipfw applied
> > automatically), transform the request into a waka message, and then
> > dispatch that request to a process running under a userid that matches
> > a prefix on the URI. That's pretty far out, though, and I wouldn't
> > want it to stand in the way of any shorter term goals.
>
> This may be too early to jump into design details, but the first thing
> that I like about this abstration is a direct mapping between URI-space
> and handlers. The second thing that's nice is multi-user support for
> any vhost or any portion of a URL path. I don't know how we would pass
> a request message containing a large body though. Also, how would this
> model gracefully fall back on older syscalls for legacy systems? Would
> we simply use a different kernel adapter (kind of like what we have now
> with the WinNT and BeOS MPMs)? Really we need to decouple low-level I/O
> (disk and network and pipes) from concurrency models (multi-process,
> multi-threaded, event-driven async) and also from our protocol handlers.
>
> -aaron
>


-- 
Davanum Srinivas :: http://wso2.org/ :: Oxygen for Web Services Developers

Re: 3.0 - Introduction

[Aaron Bannert <aa...@clove.org>]

On Wed, Feb 14, 2007 at 02:10:19PM -0800, Roy T. Fielding wrote:
 
> But do we really want to start by calling it 3.0?  How about if we
> work off of a few code names first?  Say, for example, "amsterdam".
> The reason is because there will be some overlap between ideas of
> how to do certain things, with a variety of overlapping breakage
> that can get pretty annoying if you "just want to get one part working
> first".
> 
> I want people to be able to break things in one tree without blocking
> others.  And then, say once a month, we all agree on what parts are
> "finished" enough to merge into all sandbox trees.

I prefer this rather than going straight to 3.0. Would each sandbox
correspond to a single new feature prototype?


> The reason I was about to start the sandbox thing is because I've
> been thinking about moving away from the MPM design.  To be precise,
> I mean that we should get closer to the kernels on the more modern
> platforms and find a way to stay in kernel-land until a valid
> request is received (with load restrictions tested and ipfw applied
> automatically), transform the request into a waka message, and then
> dispatch that request to a process running under a userid that matches
> a prefix on the URI. That's pretty far out, though, and I wouldn't
> want it to stand in the way of any shorter term goals.

This may be too early to jump into design details, but the first thing
that I like about this abstration is a direct mapping between URI-space
and handlers. The second thing that's nice is multi-user support for
any vhost or any portion of a URL path. I don't know how we would pass
a request message containing a large body though. Also, how would this
model gracefully fall back on older syscalls for legacy systems? Would
we simply use a different kernel adapter (kind of like what we have now
with the WinNT and BeOS MPMs)? Really we need to decouple low-level I/O
(disk and network and pipes) from concurrency models (multi-process,
multi-threaded, event-driven async) and also from our protocol handlers.

-aaron

Re: 3.0 - Introduction

["William A. Rowe, Jr." <wr...@rowe-clan.net>]

Roy T. Fielding wrote:
> On Feb 13, 2007, at 11:32 PM, Paul Querna wrote:
> 
>> I believe the httpd project is ready for a push towards the next major
>> version.
> 
> But do we really want to start by calling it 3.0?  How about if we
> work off of a few code names first?  Say, for example, "amsterdam".
> The reason is because there will be some overlap between ideas of
> how to do certain things, with a variety of overlapping breakage
> that can get pretty annoying if you "just want to get one part working
> first".

+1 to sandboxes (plural as needed).

> The reason I was about to start the sandbox thing is because I've
> been thinking about moving away from the MPM design.  To be precise,
> I mean that we should get closer to the kernels on the more modern
> platforms and find a way to stay in kernel-land until a valid
> request is received (with load restrictions tested and ipfw applied
> automatically), transform the request into a waka message, and then
> dispatch that request to a process running under a userid that matches
> a prefix on the URI. That's pretty far out, though, and I wouldn't
> want it to stand in the way of any shorter term goals.

Just a simple question, isn't it reasonable to do that sort of
encapsulation within the mpm's context?

MPM's don't have any rules about how clever they have to be - they
just need to offer the right hooks (lots of new hooks to accomplish
async stateless behavior) and all implemented them all, even if they
are only stubs on classic (simplistic) MPMs.

Bill

Re: 3.0 - Introduction

["Roy T. Fielding" <fi...@gbiv.com>]

On Feb 13, 2007, at 11:32 PM, Paul Querna wrote:

> I believe the httpd project is ready for a push towards the next major
> version.

So do I.  In fact I was just about to create a sandbox for that purpose
yesterday, but had to get the crypto stuff sorted out first.

But do we really want to start by calling it 3.0?  How about if we
work off of a few code names first?  Say, for example, "amsterdam".
The reason is because there will be some overlap between ideas of
how to do certain things, with a variety of overlapping breakage
that can get pretty annoying if you "just want to get one part working
first".

I want people to be able to break things in one tree without blocking
others.  And then, say once a month, we all agree on what parts are
"finished" enough to merge into all sandbox trees.

The reason I was about to start the sandbox thing is because I've
been thinking about moving away from the MPM design.  To be precise,
I mean that we should get closer to the kernels on the more modern
platforms and find a way to stay in kernel-land until a valid
request is received (with load restrictions tested and ipfw applied
automatically), transform the request into a waka message, and then
dispatch that request to a process running under a userid that matches
a prefix on the URI. That's pretty far out, though, and I wouldn't
want it to stand in the way of any shorter term goals.

In the mean time, I would like to start the sandbox and move the
goals and requirements discussion into subversion (where it can be
saved and comments don't get lost).

....Roy