[PATCH] event MPM nonblocking write completion

[Brian Pane <br...@apache.org>]

I've prototyped a modification to the event MPM that uses a (mostly)  
nonblocking
output filter in place of ap_core_output_filter().

This patch adds a new connection state, CONN_STATE_WRITE_COMPLETION.

The new network-level filter, event_output_filter(), does nonblocking  
writes
(except when it encounters a flush bucket that isn't immediately  
before or
after an EOS or EOC).  If an attempted write yields EAGAIN, the filter
does a setaside of the unwritten data.

If there still is unwritten data buffered in event_output_filter()  
upon completion
of a request, the filter puts the connection in write-completion state.

When the connection is in write-completion state, the main  
process_socket()
loop in event.c runs a loop to output the remaining data.  The basic  
logic is:

     while (state == CONN_STATE_WRITE_COMPLETION) {
         rv = event_output_filter(...);
         if (APR_STATUS_IS_EAGAIN(rv)) {
             block until writeability on client socket (or timeout);
         }
     }

The event_output_filter() is responsible for changing the connection  
state
to CONN_STATE_LINGER or CONN_STATE_CHECK_REQUEST_LINE_READABLE
(depending on the keepalive settings) when all of the output has been  
written.

Note that this patch keeps a worker thread waiting on the connection  
until
it exits CONN_STATE_WRITE_COMPLETION.  The real benefits of the
nonblocking output filter will come later, when the event MPM code is
enhanced to hand off the connection to the central poller thread  
whenever
it needs to wait for writeability.  I've purposely avoided making  
that change
so far; I want to make sure the nonblocking filter design is sound  
before
adding the complexity of truly async write completion.

Known bugs:
   - The new filter doesn't support nonblocking reads of pipe or  
socket buckets.
     (It's functionally correct, I think, but not optimal.)
   - It doesn't yet support "EnableSendfile off"; and it may not even  
compile
     on platforms that don't have sendfile.
   - Because event_output_filter() can return APR_EAGAIN, the few things
     that expect it to do a blocking write--like ap_rwrite()--are  
broken.  This
     could be fixed either by:
       1. Changing the callers to understand an EAGAIN response, or
       2. Changing event_output_filter() to return APR_SUCCESS instead
          of APR_EAGAIN if the connection is not in write-completion  
state.

Next steps:
I need a volunteer or two to review this patch.  Given the fundamental
nature of some of the design decisions involved, I'd like to get a  
second
opinion before committing this into 2.3-dev.

Thanks,
Brian