3.0 - Proposed Requirements

[Paul Querna <ch...@force-elite.com>]

This proposed list of requirements for a 3.0 platform. this list enables
a 'base' level of performance and design decisions to be made. If others
can make designs work with 'lessor' requirements, all the better, but
I'm not worried about it.

Proposed Requirements:
- C99 Compiler.
- High Performance Event System Calls (KQueue, Event Ports, EPoll, I/O
Completion Ports).
- Async Socket and Disk IO available. (POSIX AIO?)
- Good Kernel Threading.

Based on this list, the following operating systems would be supported
TODAY:
- FreeBSD >= 6.x
- Solaris >= 10
- Linux >= 2.6
- Windows >= XP? (Maybe even 2003 or Vista -- I don't know this one well
enough)

Operating systems that would likely have problems with these
requirements today:
- AIX?
- NetWare?
- HP-UX?
- Many other older Unixy systems.

The key part to me for all of these is this is Today.  If you view any
3.0 project on a 1-3 year timeline, if we start pushing things like high
performance event system calls, there is time for these other operating
systems to pick them up.

Today, we have all of the major platforms with a good level of async IO,
events and threading support, so it makes sense to me to set these are
the base requirements.

-Paul

Re: 3.0 - Proposed Requirements

[Garrett Rooney <ro...@electricjellyfish.net>]

On 2/14/07, Greg Marr <gr...@alum.wpi.edu> wrote:
> At 08:33 AM 2/14/2007, Garrett Rooney wrote:
> >On 2/14/07, Paul Querna <ch...@force-elite.com> wrote:
> >>This proposed list of requirements for a 3.0 platform. this list
> >>enables
> >>a 'base' level of performance and design decisions to be made. If
> >>others
> >>can make designs work with 'lessor' requirements, all the better, but
> >>I'm not worried about it.
> >>
> >>Proposed Requirements:
> >>- C99 Compiler.
> >
> >Are there any C99 compilers?  I was under the impression that GCC was
> >close, but nobody else really seemed to be pushing for it (i.e.
> >Microsoft doesn't seem to care).
>
> According to all the information I've found, the only C99 features
> that Visual Studio 2005 supports are "long long", variadic macros,
> and C++ style comments (which they've supported for years because of
> requests from C++ customers).

Well, varardic macros are one of the really nice features of c99, but
I'm not sure that's really enough justification for requiring it...

-garrett

Re: 3.0 - Proposed Requirements

[Greg Marr <gr...@alum.wpi.edu>]

At 08:33 AM 2/14/2007, Garrett Rooney wrote:
>On 2/14/07, Paul Querna <ch...@force-elite.com> wrote:
>>This proposed list of requirements for a 3.0 platform. this list 
>>enables
>>a 'base' level of performance and design decisions to be made. If 
>>others
>>can make designs work with 'lessor' requirements, all the better, but
>>I'm not worried about it.
>>
>>Proposed Requirements:
>>- C99 Compiler.
>
>Are there any C99 compilers?  I was under the impression that GCC was
>close, but nobody else really seemed to be pushing for it (i.e.
>Microsoft doesn't seem to care).

According to all the information I've found, the only C99 features 
that Visual Studio 2005 supports are "long long", variadic macros, 
and C++ style comments (which they've supported for years because of 
requests from C++ customers).

Re: 3.0 - Proposed Requirements

[Garrett Rooney <ro...@electricjellyfish.net>]

On 2/14/07, Paul Querna <ch...@force-elite.com> wrote:
> This proposed list of requirements for a 3.0 platform. this list enables
> a 'base' level of performance and design decisions to be made. If others
> can make designs work with 'lessor' requirements, all the better, but
> I'm not worried about it.
>
> Proposed Requirements:
> - C99 Compiler.

Are there any C99 compilers?  I was under the impression that GCC was
close, but nobody else really seemed to be pushing for it (i.e.
Microsoft doesn't seem to care).

> - High Performance Event System Calls (KQueue, Event Ports, EPoll, I/O
> Completion Ports).
> - Async Socket and Disk IO available. (POSIX AIO?)

What kind of async I/O are you thinking of?  Does anyone actually use
posix aio?  I'm not all that thrilled with the idea of being the
canary in that coal mine ;-)

> - Good Kernel Threading.
>
> Based on this list, the following operating systems would be supported
> TODAY:
> - FreeBSD >= 6.x
> - Solaris >= 10
> - Linux >= 2.6
> - Windows >= XP? (Maybe even 2003 or Vista -- I don't know this one well
> enough)
>
> Operating systems that would likely have problems with these
> requirements today:
> - AIX?
> - NetWare?
> - HP-UX?
> - Many other older Unixy systems.
>
> The key part to me for all of these is this is Today.  If you view any
> 3.0 project on a 1-3 year timeline, if we start pushing things like high
> performance event system calls, there is time for these other operating
> systems to pick them up.
>
> Today, we have all of the major platforms with a good level of async IO,
> events and threading support, so it makes sense to me to set these are
> the base requirements.

I do think that providing a higher level of base requirements makes
sense, but I also expect that the devotees of systems that can't/don't
support those sort of things should be allowed to make things work on
their systems, so long as they don't require invasive changes in the
rest of the system.

-garrett

Re: 3.0 - Proposed Requirements

[Aaron Bannert <aa...@clove.org>]

On Mon, Mar 12, 2007 at 02:53:14PM -0700, Shankar Unni wrote:
> Paul Querna wrote:
> 
> >- High Performance Event System Calls (KQueue, Event Ports, EPoll, I/O
> >Completion Ports).
> 
> This is a tricky area. You definitely don't want to tie yourself to a 
> small subset of OSes.  The real magic trick, however, would be to come 
> up with an abstraction that can take advantage of these if available, 
> but still be able to fall back to conventional I/O if none of these are 
> available..

This is what libevent does. It has a clean and well-designed API, and
was released under a "3-clause BSD" license, which I hope is compatible
with our Apache License (v2).

-aaron

Re: 3.0 - Proposed Requirements

[Shankar Unni <sh...@netscape.net>]

Paul Querna wrote:

> - C99 Compiler.

Shrug. Everyone's there now.

> - High Performance Event System Calls (KQueue, Event Ports, EPoll, I/O
> Completion Ports).

This is a tricky area. You definitely don't want to tie yourself to a 
small subset of OSes.  The real magic trick, however, would be to come 
up with an abstraction that can take advantage of these if available, 
but still be able to fall back to conventional I/O if none of these are 
available..

> - Good Kernel Threading.

"Good"? Or merely "existence"? Older OSes often have some notion of 
threading support, though these may be implemented in userspace, or NxM, 
or 1-1, and may or may not be "good" depending on your viewpoint.

Nevertheless, taking advantage of that feature is generally always good, 
even if the implementation is somewhat suboptimal. I assume any design 
you create would follow today's Worker MPM design (a few dozen to maybe 
a hundred worker threads per process, not thousands or tens of 
thousands) - most threading implementations should be able to handle that.

Re: 3.0 - Proposed Requirements

[Jeff Trawick <tr...@gmail.com>]

On 2/14/07, Paul Querna <ch...@force-elite.com> wrote:
> This proposed list of requirements for a 3.0 platform. this list enables
> a 'base' level of performance and design decisions to be made. If others
> can make designs work with 'lessor' requirements, all the better, but
> I'm not worried about it.

I'm not sure what that means.  Other than perhaps the C99 stuff, I
think we all take it for granted that the design should take proper
advantage of the other features when available.  So the topic for
discussion is an absolute set of requirements, with the implementation
not complicated by logic to support a lower set of system
requirements.  Either we design it to require fancy features to work
at all or we don't.

>
> Proposed Requirements:
> - C99 Compiler.

shrug (for some reason it seems more natural to me to follow APR on
what language level is required)

> - High Performance Event System Calls (KQueue, Event Ports, EPoll, I/O
> Completion Ports).
> - Async Socket and Disk IO available. (POSIX AIO?)

Designing the server so that it can take advantage of these when
available but work fine without them is of tremendous value, not only
for supporting older or different platforms but also for providing a
simplified debugging/development environment for logic not directly
tied to these features.

I don't think we should force every module author to have to be aware
of these aspects of the design (though they may have to require the
use of a simpler MPM).  And we don't want to keep 2.x healthy forever
or chase people to other web servers.

> - Good Kernel Threading.

Dealing with threads/no-threads definitely introduces clutter.  Some
new Apache helper functions that are no-ops when !APR_HAS_THREADS or
the MPM has a single threaded process serving requests could clean up
a lot of the code that cares about this, perhaps at a very slight
performance cost for the non-threaded builds.

Re: 3.0 - Proposed Requirements

["William A. Rowe, Jr." <wr...@rowe-clan.net>]

Paul Querna wrote:
> This proposed list of requirements for a 3.0 platform. this list enables
> a 'base' level of performance and design decisions to be made. If others
> can make designs work with 'lessor' requirements, all the better, but
> I'm not worried about it.
> 
> Proposed Requirements:
> - C99 Compiler.

-.9 - what does it win us over todays requirements?

> - High Performance Event System Calls (KQueue, Event Ports, EPoll, I/O
> Completion Ports).

-1 - no way as a baseline.  Great if you have em, the modules should
remain oblivious, the mpm and underlying apr support should solve it.

> - Async Socket and Disk IO available. (POSIX AIO?)

-1 for the same reason as above.  We can provide every module author
the abstract appearance that their socket/disk IO is all async even
if it isn't.

> - Good Kernel Threading.
> 
> Based on this list, the following operating systems would be supported
> TODAY:
> - FreeBSD >= 6.x
> - Solaris >= 10
> - Linux >= 2.6
> - Windows >= XP? (Maybe even 2003 or Vista -- I don't know this one well
> enough)

This is the only one I'll bend on... I'm beginning to thing it is the
long-term solution to use threads-not-processes as baseline, even if
you want a one thread per process model.  All modules should behave as
if they are in the threaded async model, of course.

> Operating systems that would likely have problems with these
> requirements today:
> - AIX?
> - NetWare?
> - HP-UX?
> - Many other older Unixy systems.

Not your problem, drop this.

My point is that we can make this appear to be an entirely async,
preemptively multithreaded and even fiber-based design without actually
requiring any of these if their MPM is much simpler.

In 3.0

> The key part to me for all of these is this is Today.  If you view any
> 3.0 project on a 1-3 year timeline, if we start pushing things like high
> performance event system calls, there is time for these other operating
> systems to pick them up.

If you set a three year timeframe it won't happen.  If you set a one year
timeframe without better resolution of "what" it won't happen.

Chop this up into manageable pieces and get hacking :)

> Today, we have all of the major platforms with a good level of async IO,
> events and threading support, so it makes sense to me to set these are
> the base requirements.

The day you have the first accepted non-2.3'ish patch, we'll fork the
trunk to it's own branch for 2.x dev, give you 2.9.0 on trunk, and just
let everyone code away to either.  As you hint, another 2.some release
is likely before 3.n is ready.

Bill