You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Manoj Kasichainula <ma...@raleigh.ibm.com> on 1999/05/07 22:45:02 UTC
Proposal: Get rid of most accept mutex calls on hybrid server.
Here's an idea for getting rid of the fcntl calls in the current
poll-accept model. Right now, the goop in the get_connection routine
looks like (taking Linux as the example to simplify the discussion)
pthread lock
fcntl lock
select
accept
fcntl unlock
pthread unlock
return
It would be nice to skip the fcntls if we know that another thread in
the same process will be handling the next connection.
So, here's a proposal for a way to avoid this. idle_thread_count and
have_accept_lock are initialized to 0.
idle_thread_count++;
pthread lock
if (!have_accept_lock) {
fcntl lock
have_accept_lock = 1
}
select
accept
idle_thread_count--;
if (idle_thread_count == 0) {
fcntl unlock
have_accept_lock = 0
}
pthread unlock
return
This will require an extra pthread lock to be sure that the increment
and decrement operations are atomic, but that's a lot cheaper than the
fcntl lock.
This will eliminate use of the fcntl lock when we still have threads
available in the current process to handle requests.
--
Manoj Kasichainula - manojk@raleigh.ibm.com
IBM, Apache Development
Re: Proposal: Get rid of most accept mutex calls on hybrid server.
Posted by Dean Gaudet <dg...@arctic.org>.
On Tue, 11 May 1999, Tony Finch wrote:
> Dean Gaudet <dg...@arctic.org> wrote:
> >On Mon, 10 May 1999, Tony Finch wrote:
> >> Dean Gaudet <dg...@arctic.org> wrote:
> >> >
> >> >Actually, I suspect that we don't really want to interprocess lock at all
> >> >in the multithreaded server. We use non-blocking listening sockets, and
> >> >pay the wake-all cost for the small number of processes (we're talking
> >> >like 8 processes, right?)
> >>
> >> If there's a select collision isn't *every* process woken up (not
> >> just the httpds)?
> >
> >I'm not sure what you mean... if a kernel had only one global sleeping
> >queue, yeah... but then there'd be no way for us to avoid thundering herd,
> >since everything would be awakened at all times. But kernels typically
> >have a lot of sleeping queues... including one on every socket.
>
> Sorry, I was too terse. The BSD network stack only has space for
> keeping track of one pid select()ing on each socket. If more than one
> child select()s on a listen socket at the same time the kernel cannot
> keep track of them so it doesn't even try: it just marks a collision
> and when a connection arrives on that socket it wakes up every process
> in select().
Hmm, that sucks. Linux keeps a list of all pids on each socket... it
doesn't cost much, the select() code allocates a page of memory to store
the wait list elements in. For stuff other than select, the wait list
elements are allocated on the kernel stack. So even though there's a
dynamic cost to allocating list elements, it's fairly cheap.
I wonder what solaris does (pretty much the only other platform I care
about ;)
Dean
Re: Proposal: Get rid of most accept mutex calls on hybrid server.
Posted by Tony Finch <do...@dotat.at>.
Dean Gaudet <dg...@arctic.org> wrote:
>On Mon, 10 May 1999, Tony Finch wrote:
>> Dean Gaudet <dg...@arctic.org> wrote:
>> >
>> >Actually, I suspect that we don't really want to interprocess lock at all
>> >in the multithreaded server. We use non-blocking listening sockets, and
>> >pay the wake-all cost for the small number of processes (we're talking
>> >like 8 processes, right?)
>>
>> If there's a select collision isn't *every* process woken up (not
>> just the httpds)?
>
>I'm not sure what you mean... if a kernel had only one global sleeping
>queue, yeah... but then there'd be no way for us to avoid thundering herd,
>since everything would be awakened at all times. But kernels typically
>have a lot of sleeping queues... including one on every socket.
Sorry, I was too terse. The BSD network stack only has space for
keeping track of one pid select()ing on each socket. If more than one
child select()s on a listen socket at the same time the kernel cannot
keep track of them so it doesn't even try: it just marks a collision
and when a connection arrives on that socket it wakes up every process
in select().
Tony.
--
f.a.n.finch dot@dotat.at fanf@demon.net
Arthur: "Oh, that sounds better, have you worked out the controls?"
Ford: "No, we just stopped playing with them."
Re: Proposal: Get rid of most accept mutex calls on hybrid server.
Posted by Dean Gaudet <dg...@arctic.org>.
On Mon, 10 May 1999, Tony Finch wrote:
> Dean Gaudet <dg...@arctic.org> wrote:
> >
> >Actually, I suspect that we don't really want to interprocess lock at all
> >in the multithreaded server. We use non-blocking listening sockets, and
> >pay the wake-all cost for the small number of processes (we're talking
> >like 8 processes, right?)
>
> If there's a select collision isn't *every* process woken up (not
> just the httpds)?
I'm not sure what you mean... if a kernel had only one global sleeping
queue, yeah... but then there'd be no way for us to avoid thundering herd,
since everything would be awakened at all times. But kernels typically
have a lot of sleeping queues... including one on every socket.
Dean
Re: Proposal: Get rid of most accept mutex calls on hybrid server.
Posted by Tony Finch <do...@dotat.at>.
Dean Gaudet <dg...@arctic.org> wrote:
>
>Actually, I suspect that we don't really want to interprocess lock at all
>in the multithreaded server. We use non-blocking listening sockets, and
>pay the wake-all cost for the small number of processes (we're talking
>like 8 processes, right?)
If there's a select collision isn't *every* process woken up (not
just the httpds)?
Tony.
--
f.a.n.finch dot@dotat.at fanf@demon.net
Arthur: "Oh, that sounds better, have you worked out the controls?"
Ford: "No, we just stopped playing with them."
Re: Proposal: Get rid of most accept mutex calls on hybrid server.
Posted by Manoj Kasichainula <ma...@io.com>.
On Fri, May 07, 1999 at 07:10:26PM -0700, unknown@riverstyx.net wrote:
> Well, conventional wisdom from those running Netscape Enterprise server
> holds that 32 threads/process is good, but I've been told that 16 is
> better. If you figure 900 concurrent connections, that's about 30-60
> processes. I don't have any idea why one would want to lower the number
> of threads though, and 32 sounds pretty reasonable to me.
The only reasons I can think of to decrease the threads per process
ratio are to reduce the affect of a process crash, or because we have
a lot of synchronization overhead between threads. The first reason
does have merit. With the current state of the hybrid server, I don't
think the second reason applies. That might not be true for Netscape,
though.
I'm envisioning two extremes in how server administrators might want
to configure things:
- The server admin without much in the way of resources, serving
mostly static content, who isn't too worried about child processes
crashing. This person will want very few processes running, with
lots of threads.
- A server with lots of dynamic content, not really needing the full
benefit of threading. The server admin wants child process crashes
to affect as few users as possible. This admin would have lots of
processes, each with few threads.
--
Manoj Kasichainula - manojk at io dot com - http://www.io.com/~manojk/
"The only thing worse than a few big, regulated telephone monopolies is
few, bigger, unregulated telephone monopolies." - Bob Metcalfe
Re: Proposal: Get rid of most accept mutex calls on hybrid server.
Posted by un...@riverstyx.net.
On Fri, 7 May 1999, Manoj Kasichainula wrote:
> Well, that's what HARD_SERVER_LIMIT is set to now, but it was just a
> random setting that sounded good. Could we ever want to run with just
> a few threads per process and a lot of processes? If so, I guess we
> could make accept locking depend on the number of processes, or make
> it user configurable.
>
> We also still have to retain the accept mutex for systems which croak
> when accept isn't serialized.
Well, conventional wisdom from those running Netscape Enterprise server
holds that 32 threads/process is good, but I've been told that 16 is
better. If you figure 900 concurrent connections, that's about 30-60
processes. I don't have any idea why one would want to lower the number
of threads though, and 32 sounds pretty reasonable to me. Does anyone
have any idea what good numbers for Apache would be, and why they might
differ from Netscape Server?
>
> --
> Manoj Kasichainula - manojk at io dot com - http://www.io.com/~manojk/
> "Would you die for The One?"
> "I wouldn't get pizza for the one. That ain't my job." - J.M. Straczynski
---
tani hosokawa
river styx internet
Re: Proposal: Get rid of most accept mutex calls on hybrid server.
Posted by Manoj Kasichainula <ma...@io.com>.
On Fri, May 07, 1999 at 03:20:29PM -0700, Dean Gaudet wrote:
> This works reasonably on single cpu systems, or when you use kernel
> threads... but if you're using user threads on multicpu systems... you
> really want other processes to get a chance in there.
>
> Actually, I suspect that we don't really want to interprocess lock at all
> in the multithreaded server. We use non-blocking listening sockets, and
> pay the wake-all cost for the small number of processes (we're talking
> like 8 processes, right?)
Well, that's what HARD_SERVER_LIMIT is set to now, but it was just a
random setting that sounded good. Could we ever want to run with just
a few threads per process and a lot of processes? If so, I guess we
could make accept locking depend on the number of processes, or make
it user configurable.
We also still have to retain the accept mutex for systems which croak
when accept isn't serialized.
--
Manoj Kasichainula - manojk at io dot com - http://www.io.com/~manojk/
"Would you die for The One?"
"I wouldn't get pizza for the one. That ain't my job." - J.M. Straczynski
Re: Proposal: Get rid of most accept mutex calls on hybrid server.
Posted by Dean Gaudet <dg...@arctic.org>.
This works reasonably on single cpu systems, or when you use kernel
threads... but if you're using user threads on multicpu systems... you
really want other processes to get a chance in there.
Actually, I suspect that we don't really want to interprocess lock at all
in the multithreaded server. We use non-blocking listening sockets, and
pay the wake-all cost for the small number of processes (we're talking
like 8 processes, right?)
Dean
On Fri, 7 May 1999, Manoj Kasichainula wrote:
> Here's an idea for getting rid of the fcntl calls in the current
> poll-accept model. Right now, the goop in the get_connection routine
> looks like (taking Linux as the example to simplify the discussion)
>
> pthread lock
> fcntl lock
> select
> accept
> fcntl unlock
> pthread unlock
> return
>
> It would be nice to skip the fcntls if we know that another thread in
> the same process will be handling the next connection.
>
> So, here's a proposal for a way to avoid this. idle_thread_count and
> have_accept_lock are initialized to 0.
>
> idle_thread_count++;
> pthread lock
> if (!have_accept_lock) {
> fcntl lock
> have_accept_lock = 1
> }
> select
> accept
> idle_thread_count--;
> if (idle_thread_count == 0) {
> fcntl unlock
> have_accept_lock = 0
> }
> pthread unlock
> return
>
> This will require an extra pthread lock to be sure that the increment
> and decrement operations are atomic, but that's a lot cheaper than the
> fcntl lock.
>
> This will eliminate use of the fcntl lock when we still have threads
> available in the current process to handle requests.
>
> --
> Manoj Kasichainula - manojk@raleigh.ibm.com
> IBM, Apache Development
>