You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Manoj Kasichainula <ma...@raleigh.ibm.com> on 1999/06/29 20:47:06 UTC
Getting rid of http_accept abstraction
Is there anybody with an objection to getting rid of the http_accept
abstraction? In the MPM code it doesn't really accomplish much, and
getting rid of it will make it much easier to solve a potential
thread-safeness problem with poll:
>From the STATUS file:
If SINGLE_LISTEN_UNSERIALIZED_ACCEPT is active, multiple threads are
allowed to call poll() at the same time, with the same pollfd array.
The only solution Manoj can think of right now is to have one pollfd
array per thread, or to just always require serialization (bleh on
both counts).
The only way I can see to fix this efficiently is to let each thread
have its own pollfd array. This breaks the accept abstraction (unless
we go through gymnastics that just aren't worth the effort now that
the MPM abstraction seems to have taken hold).
--
Manoj Kasichainula - manojk@raleigh.ibm.com
IBM, Apache Development
Re: poll by multiple threads at once
Posted by Manoj Kasichainula <ma...@raleigh.ibm.com>.
On Tue, Jun 29, 1999 at 05:37:00PM -0500, Me at IO wrote:
> Hmmm, should accept even be unserialized in the 1 socket case? We
> still have the poll call (because of the pipe of death). We could
> still have 10 threads waiting in poll and getting woken up, 9 of them
> blocking on accept(). Then, we're relying on a single thread to catch
> a graceful restart message on the pipe of death, which could take a
> while. It's not catastrophic, but it is annoying. Hmmm...
Actually, it is catastrophic, because we then rely on those other 9
threads getting kicked out of accept at some point, before allowing
the process to die.
For now, I've just turned off accept serialization completely.
And, the queued model suffers from the same problem, as long as there
are multiple processes. blah.
--
Manoj Kasichainula - manojk@raleigh.ibm.com
IBM, Apache Development
Re: poll by multiple threads at once
Posted by Manoj Kasichainula <ma...@io.com>.
On Tue, Jun 29, 1999 at 03:17:56PM -0700, Dean Gaudet wrote:
> multiple threads calling poll() on the same descriptors is mostly
> pointless...
>
> they'll all discover something is ready...
>
> hop down to a loop that does accept()...
>
> one will succeed, the rest will spin back up to poll...
>
> hello thundering herd.
Except that this is only a problem when we're not serializing accept.
This is only an issue really when we're only listening on one socket.
Hmmm, should accept even be unserialized in the 1 socket case? We
still have the poll call (because of the pipe of death). We could
still have 10 threads waiting in poll and getting woken up, 9 of them
blocking on accept(). Then, we're relying on a single thread to catch
a graceful restart message on the pipe of death, which could take a
while. It's not catastrophic, but it is annoying. Hmmm...
Sounds like the queued accept model may be better for those systems
with non-thundering herd accept. Though, maybe it should be tweaked to
use poll instead of multiple accept threads so that we don't limit IP
virtual hosting to the number of threads per process.
--
Manoj Kasichainula - manojk at io dot com - http://www.io.com/~manojk/
"Tandems are good if you need hardware which sucks reliably, 24x365."
-- Malcolm Ray.
Re: poll by multiple threads at once
Posted by Dean Gaudet <dg...@arctic.org>.
On Tue, 29 Jun 1999, Manoj Kasichainula wrote:
> Actually, I just implemented another solution, which is to just keep
> one pollfd array per thread. Sound good?
multiple threads calling poll() on the same descriptors is mostly
pointless...
they'll all discover something is ready...
hop down to a loop that does accept()...
one will succeed, the rest will spin back up to poll...
hello thundering herd.
It's a fundamental problem with poll/select. They're fine when you have a
single kernel context (i.e. one process). They suck for everything else.
Dean
Re: poll by multiple threads at once
Posted by Manoj Kasichainula <ma...@io.com>.
On Tue, Jun 29, 1999 at 03:00:21PM -0700, Dean Gaudet wrote:
> I think you'll find this is a non-portable assumption. (Consider that
> you've got to be able to set the bits while the kernel is possibly writing
> them in another processor... and that the events and revents are shorts,
> which are not atomic objects on some processors.)
That was my guess.
> You'll hate me for suggesting it, given I was against it before, but you
> may just want to return to having a poll thread and a worker thread pool.
> Or just putting in a pthread mutex and letting one thread at a time into
> poll.
Actually, I just implemented another solution, which is to just keep
one pollfd array per thread. Sound good?
This was one of the motivations for wanting to get rid of the
http_accept abstraction; the code in http_accept.c didn't know
anything about which worker thread is was running in, and it
shouldn't. So solving this would require either recreating the pollfd
array per request (uhhh, no), or passing it in from the worker thread,
which would break the abstraction. I have other motivations for
wanting to get rid of http_accept.c (maintaining two different accept
models, one of which is never used, having the MPM abstraction already
in place, etc.).
The code is done for the apache-apr tree. I'll do the MPM version soon
and hopefully commit them both tonight or tomorrow, so if anyone wants
http_accept.c for some reason, please speak up.
--
Manoj Kasichainula - manojk at io dot com - http://www.io.com/~manojk/
"This letter is clearly the result of too much spinning." -- The Tick
Re: poll by multiple threads at once
Posted by Dean Gaudet <dg...@arctic.org>.
On Tue, 29 Jun 1999, Manoj Kasichainula wrote:
> On Tue, Jun 29, 1999 at 02:47:06PM -0400, Me at IBM wrote:
> > If SINGLE_LISTEN_UNSERIALIZED_ACCEPT is active, multiple threads are
> > allowed to call poll() at the same time, with the same pollfd array.
> > The only solution Manoj can think of right now is to have one pollfd
> > array per thread, or to just always require serialization (bleh on
> > both counts).
>
> Actually, is poll called simultaneously by multiple on the same pollfd
> array actually a problem. I'd think there's a reasonable chance that
> either the libc/kernel would serialize for us if necessary, or that
> two different threads writing to the pollfd array at once wouldn't
> actually cause a problem because they'd be writing the same data.
I think you'll find this is a non-portable assumption. (Consider that
you've got to be able to set the bits while the kernel is possibly writing
them in another processor... and that the events and revents are shorts,
which are not atomic objects on some processors.)
You'll hate me for suggesting it, given I was against it before, but you
may just want to return to having a poll thread and a worker thread pool.
Or just putting in a pthread mutex and letting one thread at a time into
poll.
Dean
poll by multiple threads at once
Posted by Manoj Kasichainula <ma...@io.com>.
On Tue, Jun 29, 1999 at 02:47:06PM -0400, Me at IBM wrote:
> If SINGLE_LISTEN_UNSERIALIZED_ACCEPT is active, multiple threads are
> allowed to call poll() at the same time, with the same pollfd array.
> The only solution Manoj can think of right now is to have one pollfd
> array per thread, or to just always require serialization (bleh on
> both counts).
Actually, is poll called simultaneously by multiple on the same pollfd
array actually a problem. I'd think there's a reasonable chance that
either the libc/kernel would serialize for us if necessary, or that
two different threads writing to the pollfd array at once wouldn't
actually cause a problem because they'd be writing the same data.
--
Manoj Kasichainula - manojk at io dot com - http://www.io.com/~manojk/
"I am J. D. Falk, Sysadmin. I own a web-server and a LART." - Jeff Mercer