Event MPM accept() handling

[Saju Pillai <sa...@gmail.com>]

Greetings,

(On httpd-trunk event-mpm event.c)

 I can understand why serializing apr_pollset_poll() & accept() for the
listener threads doesn't make sense in the event-mpm. A quick look through
the code leaves me confused about the following ...

 It looks like all the listener threads epoll() simultaenously on the
listener sockets + their private set of sockets added to the pollset by
workers. Will apr_pollset_poll() return "success" to each listener if a new
connection arrives on a main listener socket ? If so won't each listener
attempt to accept() the new connection ?

How is this prevented from happening in the event mpm ?

regards
srp

Re: Event MPM accept() handling

[Greg Ames <gr...@apache.org>]

Paul Querna wrote:

> This is traditionally called the 'Thundering Herd' Problem.
> 
> When you have N worker processes, and all N of them are awoken for an 
> accept()'able new client.  Unlike the prefork MPM, N is usually a smaller
> number in Event, because you don't need that many EventThreads Per 
> Number of WorkerThreads,

I'll buy that.  you only need 2 processes to deal with 3rd party module reliability 
worries.  not much of a herd.  if you have reliable code and we scale well enough to 
handle the traffic, nothing stops you from using one process.

> I also reason that on a busy server, the place you most likely want to 
> put the event mpm, you will have many more non-listener sockets to deal 
> with, and those will fire more often than new clients are connecting, 

sure, assuming that there are typically multiple HTTP requests per connection or big 
files/slow network combinations to trigger Brian's async write logic.

> meaning you will already be coming out of the _poll() with 'real' 
> events.  So the 'cost' of being put into the Run Queue isn't a 'waste', 
> like it is on the Prefork MPM, where you just would go back into _poll() 
> without having done anything.

I'll buy that too.  once the poll pops, there is a loop to consume all of the poll events 
in one go.  judging by the comments in worker.c::listener_thread, I think Manoj meant to 
do that in worker's ancestor but never got a round tuit.

Greg

Re: Event MPM accept() handling

[Saju Pillai <sa...@gmail.com>]

On 01/03/06, Greg Ames <gr...@apache.org> wrote:
>
> Saju Pillai wrote:
>
> > For a brand new client connection, why should there be 2 exchanges
> > between the listener thread and the worker thread before the request is
> > actually read ?
> >
> > The client socket is accepted() and passed to a worker thread which runs
> > the create_connection hooks and marks the connection in the
> > CONN_STATE_CHECK_REQUEST_LINE_READABLE state.
>
> hmmm?
>
> in event.c::process_socket()
>
> if (cs == NULL) {           /* This is a new connection */
>      cs = apr_pcalloc(p, sizeof(conn_state_t));
> [...]
>      cs->state = CONN_STATE_READ_REQUEST_LINE;
>
> >   This is again pushed back
> > to the listener which will poll it for readability and mark it as
> > CONN_STATE_READ_REQUEST_LINE if it's readable. A worker now grabs this
> > client and performs the actual read.
>
> I want to walk thru this with gdb.  obviously you've been studying the
> code and I've been
> away from it for too long.



my bad ... The core create_connection hook does ...

 c->cs->expiration_time = 0;
 c->cs->state = CONN_STATE_CHECK_REQUEST_LINE_READABLE;
 c->cs->c = c;
 c->cs->p = ptrans;
 c->cs->bucket_alloc = alloc;

  return c;

but process_socket in event.c does the explicit

cs->state = CONN_STATE_READ_REQUEST_LINE;

for a new conenction.. Sorry, while reading I jumped from
create_connection() to the if .. else code in process_socket which handles
the CONN_STATE_CHECK_REQUEST_LINE_READABLE by pushing client socket back to
listener.

The worker really does accept() & read() without round trips to listener.

regards
srp




> Why do we need an explicit state CONN_STATE_CHECK_REQUEST_LINE_READABLE
> > ?.
>
> it is very useful when checking for requests after the first (Keep-Alive)
>
> >  Can the first worker not read() directly from the accepted client ?
>
> sure it can.  that is the intention.
>
> Greg
>

Re: Event MPM accept() handling

[Greg Ames <gr...@apache.org>]

Saju Pillai wrote:

> For a brand new client connection, why should there be 2 exchanges 
> between the listener thread and the worker thread before the request is 
> actually read ?
> 
> The client socket is accepted() and passed to a worker thread which runs 
> the create_connection hooks and marks the connection in the 
> CONN_STATE_CHECK_REQUEST_LINE_READABLE state. 

hmmm?

in event.c::process_socket()

if (cs == NULL) {           /* This is a new connection */
     cs = apr_pcalloc(p, sizeof(conn_state_t));
[...]
     cs->state = CONN_STATE_READ_REQUEST_LINE;

>   This is again pushed back 
> to the listener which will poll it for readability and mark it as  
> CONN_STATE_READ_REQUEST_LINE if it's readable. A worker now grabs this 
> client and performs the actual read.

I want to walk thru this with gdb.  obviously you've been studying the code and I've been 
away from it for too long.

> Why do we need an explicit state CONN_STATE_CHECK_REQUEST_LINE_READABLE 
> ?. 

it is very useful when checking for requests after the first (Keep-Alive)

>  Can the first worker not read() directly from the accepted client ? 

sure it can.  that is the intention.

Greg

Re: Event MPM accept() handling

[Saju Pillai <sa...@gmail.com>]

Final question on event_mpm

For a brand new client connection, why should there be 2 exchanges between
the listener thread and the worker thread before the request is actually
read ?

The client socket is accepted() and passed to a worker thread which runs the
create_connection hooks and marks the connection in the
CONN_STATE_CHECK_REQUEST_LINE_READABLE state. This is again pushed back to
the listener which will poll it for readability and mark it as
CONN_STATE_READ_REQUEST_LINE if it's readable. A worker now grabs this
client and performs the actual read.

Why do we need an explicit state CONN_STATE_CHECK_REQUEST_LINE_READABLE ?.
Can the first worker not read() directly from the accepted client ? Is it to
avoid a DoS attack where clients open connections but don't send a request ?

In cases of kernel (like freebsd) that support "http_ready" filters won't
the accept() only succeed once a request line is already available for
reading ? i.e on bsd (maybe) we can get better benchmarks if accept() &
read() at the same time.

regards
srp



On 01/03/06, Paul Querna <ch...@force-elite.com> wrote:
>
> Saju Pillai wrote:
> >
> >
> > On 01/03/06, *Paul Querna* <chip@force-elite.com
> > <ma...@force-elite.com>> wrote:
> >
> >     Greg Ames wrote:
> >      > Saju Pillai wrote:
> >      >
> >      >> I can understand why serializing apr_pollset_poll() & accept()
> >     for the
> >      >> listener threads doesn't make sense in the event-mpm.   A quick
> look
> >      >> through the code leaves me confused about the following ...
> >      >  >
> >      >>  It looks like all the listener threads epoll() simultaenously
> >     on the
> >      >> listener sockets + their private set of sockets added to the
> >     pollset
> >      >> by workers.
> >      >
> >      > looks like you are correct.
> >      >
> >      > originally there was a separate event thread for everything but
> new
> >      > connections and the listener thread's accept serialization was
> >     the same
> >      > as worker's.  then it seemed like a good idea to merge the
> >     listener and
> >      > event threads, and it only supported a single worker process
> briefly.
> >      > since there was only one merged listener/event thread in the
> whole
> >      > server there was nothing to serialize at that time.  then a few
> of us
> >      > grumbled about what happens if some 3rd party module seg faults
> >     or leaks
> >      > memory and we went back to multiple worker processes.
> >
> >
> >
> > Can you tell us what was the reasoning behind merging the event thread
> > and the listener thread ? A seperate event thread implies that the
> > listener threads could serialize for accept() for platforms that need
> > serialized accept(). The event thread never needs to accept() - it only
> > needs to check for read/write/close events.  ( or the platforms that
> > event mpm runs on - don't need serializing of accept ?)
>
> Because the Event MPM requires a moderm OS with a newer-epoll/kqueue
> type thing, none of our supported operating systems require the use of
> an accept lock :)
>
> -Paul
>
>

Re: Event MPM accept() handling

[Paul Querna <ch...@force-elite.com>]

Saju Pillai wrote:
> 
> 
> On 01/03/06, *Paul Querna* <chip@force-elite.com 
> <ma...@force-elite.com>> wrote:
> 
>     Greg Ames wrote:
>      > Saju Pillai wrote:
>      >
>      >> I can understand why serializing apr_pollset_poll() & accept()
>     for the
>      >> listener threads doesn't make sense in the event-mpm.   A quick look
>      >> through the code leaves me confused about the following ...
>      >  >
>      >>  It looks like all the listener threads epoll() simultaenously
>     on the
>      >> listener sockets + their private set of sockets added to the
>     pollset
>      >> by workers.
>      >
>      > looks like you are correct.
>      >
>      > originally there was a separate event thread for everything but new
>      > connections and the listener thread's accept serialization was
>     the same
>      > as worker's.  then it seemed like a good idea to merge the
>     listener and
>      > event threads, and it only supported a single worker process briefly.
>      > since there was only one merged listener/event thread in the whole
>      > server there was nothing to serialize at that time.  then a few of us
>      > grumbled about what happens if some 3rd party module seg faults
>     or leaks
>      > memory and we went back to multiple worker processes. 
> 
> 
> 
> Can you tell us what was the reasoning behind merging the event thread 
> and the listener thread ? A seperate event thread implies that the 
> listener threads could serialize for accept() for platforms that need 
> serialized accept(). The event thread never needs to accept() - it only 
> needs to check for read/write/close events.  ( or the platforms that 
> event mpm runs on - don't need serializing of accept ?)

Because the Event MPM requires a moderm OS with a newer-epoll/kqueue 
type thing, none of our supported operating systems require the use of 
an accept lock :)

-Paul

Re: Event MPM accept() handling

[Saju Pillai <sa...@gmail.com>]

On 01/03/06, Paul Querna <ch...@force-elite.com> wrote:
>
> Greg Ames wrote:
> > Saju Pillai wrote:
> >
> >> I can understand why serializing apr_pollset_poll() & accept() for the
> >> listener threads doesn't make sense in the event-mpm.   A quick look
> >> through the code leaves me confused about the following ...
> >  >
> >>  It looks like all the listener threads epoll() simultaenously on the
> >> listener sockets + their private set of sockets added to the pollset
> >> by workers.
> >
> > looks like you are correct.
> >
> > originally there was a separate event thread for everything but new
> > connections and the listener thread's accept serialization was the same
> > as worker's.  then it seemed like a good idea to merge the listener and
> > event threads, and it only supported a single worker process briefly.
> > since there was only one merged listener/event thread in the whole
> > server there was nothing to serialize at that time.  then a few of us
> > grumbled about what happens if some 3rd party module seg faults or leaks
> > memory and we went back to multiple worker processes.



Can you tell us what was the reasoning behind merging the event thread and
the listener thread ? A seperate event thread implies that the listener
threads could serialize for accept() for platforms that need serialized
accept(). The event thread never needs to accept() - it only needs to check
for read/write/close events.  ( or the platforms that event mpm runs on -
don't need serializing of accept ?)

regards
srp




>
> >>   Will apr_pollset_poll() return "success" to each listener if a new
> >> connection arrives on a main listener socket ? If so won't each
> >> listener attempt to accept() the new connection ?
> >
> > I think so, but I'm not a fancy poll expert.  Paul?
>
> Correct. This is on Purpose.  It actually turns out to be faster to call
> a nonblocking accept() and fail than it is to use the AcceptLock() that
> the other MPMs do. (Micro benchmarks I did back then seemed to show
> this, and just hammering a machine and comparing the results for Worker
> & Event MPMs seem to indicate this too).
>
> > then the question is how bad is it?
>
> Not that bad :)




This is traditionally called the 'Thundering Herd' Problem.
>
> When you have N worker processes, and all N of them are awoken for an
> accept()'able new client. Unlike the prefork MPM, N is usually a smaller
> number in Event, because you don't need that many EventThreads Per
> Number of WorkerThreads,
>
> I also reason that on a busy server, the place you most likely want to
> put the event mpm, you will have many more non-listener sockets to deal
> with, and those will fire more often than new clients are connecting,
> meaning you will already be coming out of the _poll() with 'real'
> events.  So the 'cost' of being put into the Run Queue isn't a 'waste',
> like it is on the Prefork MPM, where you just would go back into _poll()
> without having done anything.
>
> -Paul
>
>

Re: Event MPM accept() handling

[Paul Querna <ch...@force-elite.com>]

Greg Ames wrote:
> Saju Pillai wrote:
> 
>> I can understand why serializing apr_pollset_poll() & accept() for the 
>> listener threads doesn't make sense in the event-mpm.   A quick look
>> through the code leaves me confused about the following ...
>  >
>>  It looks like all the listener threads epoll() simultaenously on the 
>> listener sockets + their private set of sockets added to the pollset 
>> by workers. 
> 
> looks like you are correct.
> 
> originally there was a separate event thread for everything but new 
> connections and the listener thread's accept serialization was the same 
> as worker's.  then it seemed like a good idea to merge the listener and 
> event threads, and it only supported a single worker process briefly.  
> since there was only one merged listener/event thread in the whole 
> server there was nothing to serialize at that time.  then a few of us 
> grumbled about what happens if some 3rd party module seg faults or leaks 
> memory and we went back to multiple worker processes.
> 
>>   Will apr_pollset_poll() return "success" to each listener if a new 
>> connection arrives on a main listener socket ? If so won't each
>> listener attempt to accept() the new connection ?
> 
> I think so, but I'm not a fancy poll expert.  Paul?

Correct. This is on Purpose.  It actually turns out to be faster to call 
a nonblocking accept() and fail than it is to use the AcceptLock() that 
the other MPMs do. (Micro benchmarks I did back then seemed to show 
this, and just hammering a machine and comparing the results for Worker 
& Event MPMs seem to indicate this too).

> then the question is how bad is it?

Not that bad :)

This is traditionally called the 'Thundering Herd' Problem.

When you have N worker processes, and all N of them are awoken for an 
accept()'able new client. Unlike the prefork MPM, N is usually a smaller 
number in Event, because you don't need that many EventThreads Per 
Number of WorkerThreads,

I also reason that on a busy server, the place you most likely want to 
put the event mpm, you will have many more non-listener sockets to deal 
with, and those will fire more often than new clients are connecting, 
meaning you will already be coming out of the _poll() with 'real' 
events.  So the 'cost' of being put into the Run Queue isn't a 'waste', 
like it is on the Prefork MPM, where you just would go back into _poll() 
without having done anything.

-Paul

Re: Event MPM accept() handling

[Greg Ames <gr...@apache.org>]

Saju Pillai wrote:

> I can understand why serializing apr_pollset_poll() & accept() for the 
> listener threads doesn't make sense in the event-mpm.   A quick look
> through the code leaves me confused about the following ...
 >
>  It looks like all the listener threads epoll() simultaenously on the 
> listener sockets + their private set of sockets added to the pollset by 
> workers. 

looks like you are correct.

originally there was a separate event thread for everything but new connections and the 
listener thread's accept serialization was the same as worker's.  then it seemed like a 
good idea to merge the listener and event threads, and it only supported a single worker 
process briefly.  since there was only one merged listener/event thread in the whole 
server there was nothing to serialize at that time.  then a few of us grumbled about what 
happens if some 3rd party module seg faults or leaks memory and we went back to multiple 
worker processes.

>   Will apr_pollset_poll() return "success" to each listener if a 
> new connection arrives on a main listener socket ? If so won't each
> listener attempt to accept() the new connection ?

I think so, but I'm not a fancy poll expert.  Paul?

then the question is how bad is it?

Greg



Greg