simple mpm's design and socket accepts

[Ryan Phillips <ry...@trolocsis.com>]

Hi all.

I've been reading into the guts of the new simple mpm and have some
thoughts.

Right now an accept() relies on a free thread in the worker thread pool. So
there is a 1-to-1 relationship on the accept and threadpool. I know that
lighttpd attempts to accept 100 client connections, which seems a bit more
efficient. There may be quite a bit of contention for an accepted client
and a worker thread with Simple.

I would like people's opinions on the current design and the following
approach (perhaps this is in Simple, but I'm missing it, this is my first
dive into a MPM):

Listener (one of these)

 1. Accept N connections and place them in a _new_ pollset
 2. Push pollset onto Queue

The pollset is specific to that N group of clients.

Dispatcher (Worker Threads)

 1. Pop a pollset of clients from the queue
 2. On state change of the pollset:
    a. Remove client from pollset
    b. fetch client work thread from pool
    c. have thread process client
    d. add socket back to pollset (if needed)
 3. Repeat poll on client sockets until empty

The current simple mpm seems to have contention on accepts. The way I
outlined has potentially a large thread overhead, but a bit less than the
1-1 ratio. Only active clients would have their own thread.

-Ryan

Re: simple mpm's design and socket accepts

[Paul Querna <ch...@force-elite.com>]

Ryan Phillips wrote:
> Right now an accept() relies on a free thread in the worker thread pool. So
> there is a 1-to-1 relationship on the accept and threadpool. I know that
> lighttpd attempts to accept 100 client connections, which seems a bit more
> efficient. There may be quite a bit of contention for an accepted client
> and a worker thread with Simple.

simple_io_accept actually calls accept() from the 'event thread', after 
the socket is accepted, yes, it is pushed right away to a thread, in 
simple_io_setup_conn(), which tries to do an initial read().

> I would like people's opinions on the current design and the following
> approach (perhaps this is in Simple, but I'm missing it, this is my first
> dive into a MPM):
> 
> Listener (one of these)
> 
>  1. Accept N connections and place them in a _new_ pollset
>  2. Push pollset onto Queue
> 
> The pollset is specific to that N group of clients.
> 
> Dispatcher (Worker Threads)
> 
>  1. Pop a pollset of clients from the queue
>  2. On state change of the pollset:
>     a. Remove client from pollset
>     b. fetch client work thread from pool
>     c. have thread process client
>     d. add socket back to pollset (if needed)
>  3. Repeat poll on client sockets until empty
> 
> The current simple mpm seems to have contention on accepts. The way I
> outlined has potentially a large thread overhead, but a bit less than the
> 1-1 ratio. Only active clients would have their own thread.

Yes, I agree that the approach you outlined above does make sense, 
however, it really depends on async *reads* working through the entire 
input filter stack, otherwise the Dispatcher is still going to block on 
different clients while its parsing the HTTP input headers.

Anyone have thoughts on making async reads work :-) ?

Thanks,

Paul