You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apr.apache.org by "Roy T. Fielding" <fi...@ebuilt.com> on 2001/04/26 06:49:58 UTC
thread locking within apr file io
APR file io, when compiled with APR_HAS_THREADS (the default unless it
is specifically disabled or on a non-thread platform), is acquiring
mutex locks on every buffered file read or write. That is lunacy.
Whether or not locking is necessary must be decided by the caller,
not by the library. The normal behavior is for buffers to be
thread-specific, and thus never need mutexing.
BTW, the same can be said for pools -- right now it is locking all
pool operations, even though the only ones that need locking are those
shared by multiple threads (the global pools). Bogus.
This stuff came up while Justin and Aaron were performance testing
mod_mbox -- right now you can double the requests per second simply
by recompiling with --disable-threads.
....Roy
Re: thread locking within apr file io
Posted by "Paul J. Reder" <re...@raleigh.ibm.com>.
Bill Stoddard wrote:
> Couldn't this just be a variation of problem 2? Threads will not go away until the connection
> closes. Could instrument the join code to see if it is worker threads hanging around.
ps shows no worker threads, just the server processes in defunct state. I checked the
pids before, during, and after. The workers all go away, then the delay begins.
The most likely case, in my opinion, is related to the join. I am in the process of
using Jeff's logging code to check timing related to the join to see if there is
some sort of timeout happening.
--
Paul J. Reder
-----------------------------------------------------------
"The strength of the Constitution lies entirely in the determination of each
citizen to defend it. Only if every single citizen feels duty bound to do
his share in this defense are the constitutional rights secure."
-- Albert Einstein
Re: thread locking within apr file io
Posted by Bill Stoddard <bi...@wstoddard.com>.
> "Paul J. Reder" wrote:
> >
> > Greg Ames wrote:
> > > It would be great if somebody could beat it up on a live
> > > non-FreeBSD system, and tell us what happens.
> >
> > I'll run it through my battery of abuse tests tonight. If it survives the carnage
> > then we'll be in good shape.
>
> After running the threaded mpm through a variety of abuse tests it seems
> to be running fine except in two cases.
>
> Startup, SIGWINCH, and normal request processing under a variety of loads
> run as expected.
>
> Problem 1:
> SIGHUP and SIGTERM take a few seconds to clear out the workers, then takes an
> additional 20 to 30 seconds to clear out the server processes. After the 20-30
> second delay it does what it is supposed to (restart or shutdown). I am looking
> into the reason for the delay. During the 20-30 second delay after the SIGHUP the
> server does not serve any pages until it restarts. Once it restarts, it performs
> normally.
Couldn't this just be a variation of problem 2? Threads will not go away until the connection
closes. Could instrument the join code to see if it is worker threads hanging around.
Bill
Re: thread locking within apr file io
Posted by "Paul J. Reder" <re...@raleigh.ibm.com>.
"Paul J. Reder" wrote:
>
> Greg Ames wrote:
> > It would be great if somebody could beat it up on a live
> > non-FreeBSD system, and tell us what happens.
>
> I'll run it through my battery of abuse tests tonight. If it survives the carnage
> then we'll be in good shape.
After running the threaded mpm through a variety of abuse tests it seems
to be running fine except in two cases.
Startup, SIGWINCH, and normal request processing under a variety of loads
run as expected.
Problem 1:
SIGHUP and SIGTERM take a few seconds to clear out the workers, then takes an
additional 20 to 30 seconds to clear out the server processes. After the 20-30
second delay it does what it is supposed to (restart or shutdown). I am looking
into the reason for the delay. During the 20-30 second delay after the SIGHUP the
server does not serve any pages until it restarts. Once it restarts, it performs
normally.
Problem 2:
The problem related to perform_idle_server_maintenance still exists. This is as
expected since we haven't done anything to fix it yet.
I did experience one anomaly which I cannot reproduce. At one point while
testing SIGHUP and SIGWINCH under mild load I ended up with 1300+ workers
(noticeably higher than the configured 10*32 max). Apache was still spawning
more when I checked and killed it. All of the server processes were owned by
pid=1, all of the workers were owned by their respective server process. The
main Apache process was still intact. There was nothing interesting in the log.
Try as I might I could not get it to happen again, so I must assume it has
something to do with the Indian Burial ground that my house was built on
(**insert suitably spooky music here**).
All in all (except for the delay) threaded mpm works well. Feel free to bang on it.
It should perform well with a suitably high (but not 0) MaxRequestsPerChild setting
(perhaps 50000).
Good luck all.
--
Paul J. Reder
-----------------------------------------------------------
"The strength of the Constitution lies entirely in the determination of each
citizen to defend it. Only if every single citizen feels duty bound to do
his share in this defense are the constitutional rights secure."
-- Albert Einstein
Re: thread locking within apr file io
Posted by Greg Ames <gr...@remulak.net>.
Greg Ames wrote:
>
> > That said, I guess the threaded mpm really isn't ready for prime time,
> > unless:
> > * you are prepared to tune it (we can help), or
> > * you know that your download times are all pretty uniform.
> >
>
> I should probably elaborate some on the tuning, in case folks want to
> play. Basic stuff first, more advanced stuff after more sleep:
>
D'oh! forgot to mention the simplest way to bypass this problem: set
MaxRequestsPerChild to 0. If you can get by with that, you're done,
ignore the rest.
> The threaded MPM currently uses MaxRequestsPerChild just like Apache 1.3
> and 2.0 prefork do, tha it, when a process hits the magic number, it
> quits accepting new connections. But since ThreadsPerChild will usually
> be > 1 , one thread can be serving something that takes a really long
> time,
[much stuff deleted here]
Greg
Re: thread locking within apr file io
Posted by Greg Ames <gr...@remulak.net>.
Greg Ames wrote:
>
>
> Paul, since you have quite a few monster files mixed in with the little
> stuff, please tune the threaded parameters judiciously. We don't have
> any automatic protection yet for low MaxRequestsPerChild combined with
> high ThreadsPerChild combined with small scoreboards combined with long
> & short downloads. Don't assume the threaded parms in the config you
> got from me and/or apache.org make any sense at all - apache.org has
> only run prefork in production (FreeBSD).
>
> That said, I guess the threaded mpm really isn't ready for prime time,
> unless:
> * you are prepared to tune it (we can help), or
> * you know that your download times are all pretty uniform.
>
I should probably elaborate some on the tuning, in case folks want to
play. Basic stuff first, more advanced stuff after more sleep:
The threaded MPM currently uses MaxRequestsPerChild just like Apache 1.3
and 2.0 prefork do, tha it, when a process hits the magic number, it
quits accepting new connections. But since ThreadsPerChild will usually
be > 1 , one thread can be serving something that takes a really long
time, while the rest of the threads run thru the little stuff
lickety-split. That process goes into what I call a "quiescing" state,
and is useless as far as serving any new requsts although it takes up
space in the scoreboard.
So you can give yourself some immunity to this phenomenon by starting
with the number of threads per process really small (e.g. 2), and the
max number of processes really big (e.g. whatever you use with prefork
or 1.3), and increasing MaxRequestsPerChild. Bump up MaxClients in your
config file to the max number of processes you want, and lower
ThreadsPerClient. While you're in there, if you don't think you have
memory leaks, bump MaxRequestsPerChild way up. Change HARD_SERVER_LIMIT
in server/threaded/mpm_default.h to give the scoreboard enough room for
your processes. You can lower HARD_THREAD_LIMIT to match
ThreadsPerChild if you want to minimize the shared memory used by the
scoreboard, or just leave it as-is if shared memory is not causing
problems.
If this works out for you, you can gradually increase the number of
threads and decrease the max number of processes. I would keep an eye
on things using ps and mod_status. (yeah, I know it needs work for
threaded, but it's something...)
Greg
Re: thread locking within apr file io
Posted by "Paul J. Reder" <re...@raleigh.ibm.com>.
Greg Ames wrote:
> Paul, since you have quite a few monster files mixed in with the little
> stuff, please tune the threaded parameters judiciously. We don't have
> any automatic protection yet for low MaxRequestsPerChild combined with
> high ThreadsPerChild combined with small scoreboards combined with long
> & short downloads. Don't assume the threaded parms in the config you
> got from me and/or apache.org make any sense at all - apache.org has
> only run prefork in production (FreeBSD).
>
> That said, I guess the threaded mpm really isn't ready for prime time,
> unless:
> * you are prepared to tune it (we can help), or
> * you know that your download times are all pretty uniform.
Yeah, I figured out early on that the existing values were bogus. I have
already done some tuning, but am always open to suggestions. I can forstall
problems, but not eliminate them. But after my original patch it no longer
dies or hangs, just slows down periodically.
--
Paul J. Reder
-----------------------------------------------------------
"The strength of the Constitution lies entirely in the determination of each
citizen to defend it. Only if every single citizen feels duty bound to do
his share in this defense are the constitutional rights secure."
-- Albert Einstein
Re: thread locking within apr file io
Posted by Greg Ames <gr...@remulak.net>.
"Paul J. Reder" wrote:
>
> Greg Ames wrote:
> > It would be great if somebody could beat it up on a live
> > non-FreeBSD system, and tell us what happens.
>
> I'll run it through my battery of abuse tests tonight. If it survives the carnage
> then we'll be in good shape.
>
>
Paul, since you have quite a few monster files mixed in with the little
stuff, please tune the threaded parameters judiciously. We don't have
any automatic protection yet for low MaxRequestsPerChild combined with
high ThreadsPerChild combined with small scoreboards combined with long
& short downloads. Don't assume the threaded parms in the config you
got from me and/or apache.org make any sense at all - apache.org has
only run prefork in production (FreeBSD).
That said, I guess the threaded mpm really isn't ready for prime time,
unless:
* you are prepared to tune it (we can help), or
* you know that your download times are all pretty uniform.
Greg
Re: thread locking within apr file io
Posted by David Reid <dr...@jetnet.co.uk>.
On a "vanilla" plain install of FreeBSD you'll see this behaviour as well.
It's basically that we have a lot of sockets in FIN_WAIT2 which is normal
and expected behaviour. Adjusting the config should fix it.
david
> On Thu, Apr 26, 2001 at 01:56:56PM -0700, Justin Erenkrantz wrote:
> > Greg Ames wrote:
> > > It would be great if somebody could beat it up on a live
> > > non-FreeBSD system, and tell us what happens.
> >
> > Looks fine here with Linux 2.2. Takes a few seconds (5 or so) with
> > threaded MPM to shut down completely. This is after hitting it with
> > about 5000 static requests via ab.
> >
> > Still only getting one active request at a time with mod_mbox. Hmph.
>
> Spoke too soon...
>
> Linux 2.2 - threaded MPM - throw about 2000 requests at it and it runs
> out of file descriptors (didn't pay attention to ab's error messages).
> It seems to happen around the 1750-1950 request plateau.
>
> I set ulimit -n to be 1024 - can't set it any higher in userspace -
> /proc/sys/fs/file-max is 8192. The result is that no new connections
> can be created.
>
> httpd.conf:
>
> <IfModule threaded.c>
> StartServers 2
> MaxClients 8
> MinSpareThreads 50
> MaxSpareThreads 128
> ThreadsPerChild 64
> MaxRequestsPerChild 0
> </IfModule>
>
> Triggered by:
> ab -k -c 100 -n 2000 http://target.machine.yours/index.html
>
> Are we not closing our fds after we are done with them? This is just
> hitting the static pages - nothing to do with mod_mbox or anything
> else. Guessing this would be restricted to mod_core.
>
> Can anyone else duplicate this? Or, is this a symptom of something
> else I need to tweak with Linux? This couldn't be triggered by
> something bogus in ab, could it?
>
> I have to go now. Will be back later tonight to dig around mod_core
> if no one has any ideas. It is a pain to debug under Linux with
> gdb because of sigsuspend usage... -- justin
>
Re: thread locking within apr file io
Posted by "Victor J. Orlikowski" <v....@gte.net>.
Don't forget to raise:
/proc/sys/fs/inode-max (should be ~4 times file-max since sockets
count against these too)
/proc/sys/net/ipv4/ip_local_port_range (I like 32768-61000)
Furthermore, the file descriptor limit of 1024 only applies to earlier 2.2
kernels; what kernel are you using? Alan Cox changed the per-process
limit to about 32k descriptors in 2.2.11 or so, as long as the
system's file limit was set high enough...
Further, ulimit limits normal users to 1024 by default; root is not
limited in the regard, and can permit users a higher "hard" ulimit -n.
Retry with the above numbers tuned.
This should allow you to bypass the leak Ryan describes.
Victor
--
Victor J. Orlikowski
======================
v.j.orlikowski@gte.net
orlikowski@apache.org
vjo@us.ibm.com
Re: thread locking within apr file io
Posted by rb...@covalent.net.
> Linux 2.2 - threaded MPM - throw about 2000 requests at it and it runs
> out of file descriptors (didn't pay attention to ab's error messages).
> It seems to happen around the 1750-1950 request plateau.
>
> I set ulimit -n to be 1024 - can't set it any higher in userspace -
> /proc/sys/fs/file-max is 8192. The result is that no new connections
> can be created.
>
> httpd.conf:
>
> <IfModule threaded.c>
> StartServers 2
> MaxClients 8
> MinSpareThreads 50
> MaxSpareThreads 128
> ThreadsPerChild 64
> MaxRequestsPerChild 0
> </IfModule>
>
> Triggered by:
> ab -k -c 100 -n 2000 http://target.machine.yours/index.html
>
> Are we not closing our fds after we are done with them? This is just
> hitting the static pages - nothing to do with mod_mbox or anything
> else. Guessing this would be restricted to mod_core.
>
> Can anyone else duplicate this? Or, is this a symptom of something
> else I need to tweak with Linux? This couldn't be triggered by
> something bogus in ab, could it?
>
> I have to go now. Will be back later tonight to dig around mod_core
> if no one has any ideas. It is a pain to debug under Linux with
> gdb because of sigsuspend usage... -- justin
We have a small resource leak with file descriptors and keepalives.
Basically, we keep the file open until the connection is closed. This is
completely bogus. It was mentioned on the list a few months ago, but I
don't believe anything has happened with it.
The last time it was brought up, I suggested that we could solve the
problem by allocating files out of the request_rec's pool, and ensuring
that any data that was left at the end of the request was a heap bucket.
The second half of that is important. Basically, we had to bump the files
up to the conn_rec, because we would MMAP out of the request_rec, and then
we wouldn't send the file, because there wasn't enough data, so we would
lose the MMAP.
Ryan
_______________________________________________________________________________
Ryan Bloom rbb@apache.org
406 29th St.
San Francisco, CA 94131
-------------------------------------------------------------------------------
Re: thread locking within apr file io
Posted by Justin Erenkrantz <je...@ebuilt.com>.
On Thu, Apr 26, 2001 at 01:56:56PM -0700, Justin Erenkrantz wrote:
> Greg Ames wrote:
> > It would be great if somebody could beat it up on a live
> > non-FreeBSD system, and tell us what happens.
>
> Looks fine here with Linux 2.2. Takes a few seconds (5 or so) with
> threaded MPM to shut down completely. This is after hitting it with
> about 5000 static requests via ab.
>
> Still only getting one active request at a time with mod_mbox. Hmph.
Spoke too soon...
Linux 2.2 - threaded MPM - throw about 2000 requests at it and it runs
out of file descriptors (didn't pay attention to ab's error messages).
It seems to happen around the 1750-1950 request plateau.
I set ulimit -n to be 1024 - can't set it any higher in userspace -
/proc/sys/fs/file-max is 8192. The result is that no new connections
can be created.
httpd.conf:
<IfModule threaded.c>
StartServers 2
MaxClients 8
MinSpareThreads 50
MaxSpareThreads 128
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>
Triggered by:
ab -k -c 100 -n 2000 http://target.machine.yours/index.html
Are we not closing our fds after we are done with them? This is just
hitting the static pages - nothing to do with mod_mbox or anything
else. Guessing this would be restricted to mod_core.
Can anyone else duplicate this? Or, is this a symptom of something
else I need to tweak with Linux? This couldn't be triggered by
something bogus in ab, could it?
I have to go now. Will be back later tonight to dig around mod_core
if no one has any ideas. It is a pain to debug under Linux with
gdb because of sigsuspend usage... -- justin
Re: thread locking within apr file io
Posted by Justin Erenkrantz <je...@ebuilt.com>.
Greg Ames wrote:
> It would be great if somebody could beat it up on a live
> non-FreeBSD system, and tell us what happens.
Looks fine here with Linux 2.2. Takes a few seconds (5 or so) with
threaded MPM to shut down completely. This is after hitting it with
about 5000 static requests via ab.
Still only getting one active request at a time with mod_mbox. Hmph.
-- justin
Re: thread locking within apr file io
Posted by "Paul J. Reder" <re...@raleigh.ibm.com>.
Greg Ames wrote:
> It would be great if somebody could beat it up on a live
> non-FreeBSD system, and tell us what happens.
I'll run it through my battery of abuse tests tonight. If it survives the carnage
then we'll be in good shape.
--
Paul J. Reder
-----------------------------------------------------------
"The strength of the Constitution lies entirely in the determination of each
citizen to defend it. Only if every single citizen feels duty bound to do
his share in this defense are the constitutional rights secure."
-- Albert Einstein
Re: thread locking within apr file io
Posted by Greg Ames <gr...@remulak.net>.
Justin Erenkrantz wrote:
>
> >
> > Wow, which MPM are you using? That is actually quite a suprise to me as Apache 2.0 performance is
> > close to that of Apache 1.3 (on AIX at least) which does not use pool locking or buffer locking for
> > file i/o.
>
> We are currently using prefork because pthread is completely useless at
> this point. Hopefully, the signals stuff gets fixed soon...
assuming you mean "threaded", do a "cvs up" and you'll be happier :)
I believe the threaded MPM is ready for prime time after my last
commit. It would be great if somebody could beat it up on a live
non-FreeBSD system, and tell us what happens.
> Actually,
> on FreeBSD, threaded MPMs are disabled, but no one tells APR, so it
> defines APR_HAS_THREADS - we need to fix that. I may submit a patch for
> that - just add --disable-threads to APRs configure for these platforms?
>
huh? I thought we fixed APR quite a while ago, then at the hack-athon
Ryan fixed the default mpm choice logic. Why am I confused?
Greg
Re: thread locking within apr file io
Posted by Greg Stein <gs...@lyra.org>.
On Thu, Apr 26, 2001 at 11:57:26AM -0700, Justin Erenkrantz wrote:
>...
> We are currently using prefork because pthread is completely useless at
> this point. Hopefully, the signals stuff gets fixed soon... Actually,
> on FreeBSD, threaded MPMs are disabled, but no one tells APR, so it
> defines APR_HAS_THREADS - we need to fix that. I may submit a patch for
> that - just add --disable-threads to APRs configure for these platforms?
There should already be some logic here and there for disabling threads on
FreeBSD. I'm not sure whether we try to disable all threading, or just the
threaded MPM. In any case, APR's configure should detect the problem and
shut things down. Apache should then notice "hey. no threads." and disable
the selection of the threaded MPM.
>...
> Somehow there is an exclusive cross-process lock going on somewhere
> (upon further review, we don't think it is fcntl - but it still might
> be...). The net result is that only one process is running at a time.
Maybe APR only has an exclusive lock available in your setup? Thus, SDBM
ends up using an exclusive lock because who knows if some *other* process
might end up trying to write to the thing while the reading is occurring.
Take a look at apr-util/dbm/sdbm/sdbm_lock.c for more info. The two
functions are called from sdbm.c. In apr_file_io.h, the SHARED lock is a
reader lock, and EXCLUSIVE is a writer lock.
As a side note: exporting the SDBM headers from apr-util has been deemed as
a mistake. Rough consensus has said to keep SDBM private and have people
only use the apr_dbm interfaces. (so we can try to avoid name conflicts on
the sdbm code)
> There seems to be a lot of lock contention going on within SDBM - we
> ONLY want read access.
Yup. But *somebody* is going to be writing to those dbms. Thus, you need
*some* kind of lock to say, "hey, I'm reading. nobody should write to the
file right now". Take a look at the APR file locking for your platform and
see what is happening.
> As Roy has pointed out, there should be very
> little locking going on in the mainline read code ANYWHERE within Apache
> (and hence within APR). This is why each request gets its own
> thread/process/whatever-the-MPM-decides. Requests shouldn't spawn
> threads themselves. r->pool is allocated on a per-request-basis.
> Therefore, it is pointless to lock that. By its very nature, we
> know that we are exclusive.
That is true, and you can feel free to fix it. It is simply that nobody else
has to this point. I'd bet that we all agree with you.
Except for the bit about requests spawning threads. *That* should be doable
and should be allowed.
>...
> mod_mbox doesn't give a hoot for threads anywhere. Actually IIRC,
> SDBM doesn't either explicitly - but the pools, IO, and locks do.
Right. SDBM just relies on the file locking, which is provided by APR. It
doesn't care about threads because it assumes that you won't use the same
SDBM structure from two threads.
> I'm not exactly sure where to start with removing the locks. I guess
> we could just remove ALL of the locking calls in read and pools (unless
> global pools) and see if we can get threaded APR to be closer to
> non-threaded APR. Roy might have some ideas... -- justin
I just looked at APR's pool locks. Basically, it is a mutex around the free
block list. The implementation sucks, though, because the locking is not
focused around access to "block_freelist", but instead to the new_block()
function on the assumption that whenever you call that, it needs the lock
for block_freelist. The "right" answer is to have new_block() manipulate the
mutex (just like free_blocks() does). There are two other accesses to
block_freelist, one is protected by the mutex, and one debug function which
isn't.
Now... given that the pools use the (global) block freelist all the time,
you can't escape the mutex without a lot of work. You can easily have
multiple threads accessing that free list.
One possible answer is to have each pool maintain a list of free blocks.
When a pool is destroyed, its blocks are returned to the *parent* rather
than a global. You can then add a flag to a pool to state whether it needs a
mutex to protect its accesses. All the subpools just operate with their
blocks, periodically requesting blocks from the parent (which may float
upwards to the global pool, who then malloc's a new block).
Overall, that should reduce lock contention, at the cost of a possible free
block imbalance -- one connection could have 1000 free blocks, and another
could have 2. With the global pool, this kind of imbalance doesn't happen.
Cheers,
-g
--
Greg Stein, http://www.lyra.org/
Re: thread locking within apr file io
Posted by Greg Stein <gs...@lyra.org>.
On Thu, Apr 26, 2001 at 11:57:26AM -0700, Justin Erenkrantz wrote:
>...
> We are currently using prefork because pthread is completely useless at
> this point. Hopefully, the signals stuff gets fixed soon... Actually,
> on FreeBSD, threaded MPMs are disabled, but no one tells APR, so it
> defines APR_HAS_THREADS - we need to fix that. I may submit a patch for
> that - just add --disable-threads to APRs configure for these platforms?
There should already be some logic here and there for disabling threads on
FreeBSD. I'm not sure whether we try to disable all threading, or just the
threaded MPM. In any case, APR's configure should detect the problem and
shut things down. Apache should then notice "hey. no threads." and disable
the selection of the threaded MPM.
>...
> Somehow there is an exclusive cross-process lock going on somewhere
> (upon further review, we don't think it is fcntl - but it still might
> be...). The net result is that only one process is running at a time.
Maybe APR only has an exclusive lock available in your setup? Thus, SDBM
ends up using an exclusive lock because who knows if some *other* process
might end up trying to write to the thing while the reading is occurring.
Take a look at apr-util/dbm/sdbm/sdbm_lock.c for more info. The two
functions are called from sdbm.c. In apr_file_io.h, the SHARED lock is a
reader lock, and EXCLUSIVE is a writer lock.
As a side note: exporting the SDBM headers from apr-util has been deemed as
a mistake. Rough consensus has said to keep SDBM private and have people
only use the apr_dbm interfaces. (so we can try to avoid name conflicts on
the sdbm code)
> There seems to be a lot of lock contention going on within SDBM - we
> ONLY want read access.
Yup. But *somebody* is going to be writing to those dbms. Thus, you need
*some* kind of lock to say, "hey, I'm reading. nobody should write to the
file right now". Take a look at the APR file locking for your platform and
see what is happening.
> As Roy has pointed out, there should be very
> little locking going on in the mainline read code ANYWHERE within Apache
> (and hence within APR). This is why each request gets its own
> thread/process/whatever-the-MPM-decides. Requests shouldn't spawn
> threads themselves. r->pool is allocated on a per-request-basis.
> Therefore, it is pointless to lock that. By its very nature, we
> know that we are exclusive.
That is true, and you can feel free to fix it. It is simply that nobody else
has to this point. I'd bet that we all agree with you.
Except for the bit about requests spawning threads. *That* should be doable
and should be allowed.
>...
> mod_mbox doesn't give a hoot for threads anywhere. Actually IIRC,
> SDBM doesn't either explicitly - but the pools, IO, and locks do.
Right. SDBM just relies on the file locking, which is provided by APR. It
doesn't care about threads because it assumes that you won't use the same
SDBM structure from two threads.
> I'm not exactly sure where to start with removing the locks. I guess
> we could just remove ALL of the locking calls in read and pools (unless
> global pools) and see if we can get threaded APR to be closer to
> non-threaded APR. Roy might have some ideas... -- justin
I just looked at APR's pool locks. Basically, it is a mutex around the free
block list. The implementation sucks, though, because the locking is not
focused around access to "block_freelist", but instead to the new_block()
function on the assumption that whenever you call that, it needs the lock
for block_freelist. The "right" answer is to have new_block() manipulate the
mutex (just like free_blocks() does). There are two other accesses to
block_freelist, one is protected by the mutex, and one debug function which
isn't.
Now... given that the pools use the (global) block freelist all the time,
you can't escape the mutex without a lot of work. You can easily have
multiple threads accessing that free list.
One possible answer is to have each pool maintain a list of free blocks.
When a pool is destroyed, its blocks are returned to the *parent* rather
than a global. You can then add a flag to a pool to state whether it needs a
mutex to protect its accesses. All the subpools just operate with their
blocks, periodically requesting blocks from the parent (which may float
upwards to the global pool, who then malloc's a new block).
Overall, that should reduce lock contention, at the cost of a possible free
block imbalance -- one connection could have 1000 free blocks, and another
could have 2. With the global pool, this kind of imbalance doesn't happen.
Cheers,
-g
--
Greg Stein, http://www.lyra.org/
Re: thread locking within apr file io
Posted by "William A. Rowe, Jr." <ad...@rowe-clan.net>.
From: "Greg Ames" <gr...@remulak.net>
Sent: Thursday, April 26, 2001 2:46 PM
>
> >
> > Hmmm... notice that APR_HAS_THREADS is irrelevant to the MPM selected... you
> > _can_ using threading support in APR with a prefork MPM,
>
> if you get this combo to work on FreeBSD, please let us know ASAP. But
> do a "top" when it comes up, and be ready to do a "killall -9
> <path/to/httpd>".
>
> >
> > So you can't indiscriminantly disable APR threads 'just because', there is
> > going to need to be a bit more logic.
>
> On FreeBSD, disabling threads by default is not indiscriminant, the last
> I heard.
No arguments, that's a platform determination, and should first occur within APR.
Apache must fall back to Prefork if handed a non-threaded APR.
Bill
Re: thread locking within apr file io
Posted by Greg Ames <gr...@remulak.net>.
"William A. Rowe, Jr." wrote:
>
>
> Hmmm... notice that APR_HAS_THREADS is irrelevant to the MPM selected... you
> _can_ using threading support in APR with a prefork MPM,
if you get this combo to work on FreeBSD, please let us know ASAP. But
do a "top" when it comes up, and be ready to do a "killall -9
<path/to/httpd>".
>
> So you can't indiscriminantly disable APR threads 'just because', there is
> going to need to be a bit more logic.
On FreeBSD, disabling threads by default is not indiscriminant, the last
I heard.
Greg
Re: thread locking within apr file io
Posted by Chuck Murcko <ch...@topsail.org>.
But we are still trying to link in pthreads, as witnessed by me getting
"pthread_sigmask" not found link errors on Darwin when building using
the prefork mpm. Or else it's just autoconf crud.
Chuck
On Thursday, April 26, 2001, at 03:18 PM, William A. Rowe, Jr. wrote:
> From: "Justin Erenkrantz" <je...@ebuilt.com>
> Sent: Thursday, April 26, 2001 1:57 PM
>
>
>>>>> This stuff came up while Justin and Aaron were performance testing
>>>>> mod_mbox -- right now you can double the requests per second simply
>>>>> by recompiling with --disable-threads.
>>>
>>> Wow, which MPM are you using? That is actually quite a suprise to me
>>> as Apache 2.0 performance is
>>> close to that of Apache 1.3 (on AIX at least) which does not use
>>> pool locking or buffer locking for
>>> file i/o.
>>
>> We are currently using prefork because pthread is completely useless at
>> this point. Hopefully, the signals stuff gets fixed soon... Actually,
>> on FreeBSD, threaded MPMs are disabled, but no one tells APR, so it
>> defines APR_HAS_THREADS - we need to fix that. I may submit a patch
>> for
>> that - just add --disable-threads to APRs configure for these
>> platforms?
>
> Hmmm... notice that APR_HAS_THREADS is irrelevant to the MPM
> selected... you
> _can_ using threading support in APR with a prefork MPM, and there may
> be
> modules that need just that configuration someday soon (say, child
> threads
> in a mod_proxy or servlet environment).
>
> So you can't indiscriminantly disable APR threads 'just because', there
> is
> going to need to be a bit more logic. Contrawise, just because the user
> enables APR threads does _not_ mean they are choosing pthreads ;-)
>
> Bill
>
>
Chuck Murcko
Topsail Group
http://www.topsail.org/
Re: thread locking within apr file io
Posted by "William A. Rowe, Jr." <ad...@rowe-clan.net>.
From: "Justin Erenkrantz" <je...@ebuilt.com>
Sent: Thursday, April 26, 2001 1:57 PM
> > > > This stuff came up while Justin and Aaron were performance testing
> > > > mod_mbox -- right now you can double the requests per second simply
> > > > by recompiling with --disable-threads.
> >
> > Wow, which MPM are you using? That is actually quite a suprise to me as Apache 2.0 performance is
> > close to that of Apache 1.3 (on AIX at least) which does not use pool locking or buffer locking for
> > file i/o.
>
> We are currently using prefork because pthread is completely useless at
> this point. Hopefully, the signals stuff gets fixed soon... Actually,
> on FreeBSD, threaded MPMs are disabled, but no one tells APR, so it
> defines APR_HAS_THREADS - we need to fix that. I may submit a patch for
> that - just add --disable-threads to APRs configure for these platforms?
Hmmm... notice that APR_HAS_THREADS is irrelevant to the MPM selected... you
_can_ using threading support in APR with a prefork MPM, and there may be
modules that need just that configuration someday soon (say, child threads
in a mod_proxy or servlet environment).
So you can't indiscriminantly disable APR threads 'just because', there is
going to need to be a bit more logic. Contrawise, just because the user
enables APR threads does _not_ mean they are choosing pthreads ;-)
Bill
Re: thread locking within apr file io
Posted by Chuck Murcko <ch...@topsail.org>.
Totally +1 from me. This likely gets 2.0 building on Darwin, which has
somewhat incomplete pthreads at the moment. 8^( Forcing APR_HAS_THREADS
off works.
On Thursday, April 26, 2001, at 02:57 PM, Justin Erenkrantz wrote:
>>>> This stuff came up while Justin and Aaron were performance testing
>>>> mod_mbox -- right now you can double the requests per second simply
>>>> by recompiling with --disable-threads.
>>
>> Wow, which MPM are you using? That is actually quite a suprise to me
>> as Apache 2.0 performance is
>> close to that of Apache 1.3 (on AIX at least) which does not use pool
>> locking or buffer locking for
>> file i/o.
>
> We are currently using prefork because pthread is completely useless at
> this point. Hopefully, the signals stuff gets fixed soon... Actually,
> on FreeBSD, threaded MPMs are disabled, but no one tells APR, so it
> defines APR_HAS_THREADS - we need to fix that. I may submit a patch for
> that - just add --disable-threads to APRs configure for these platforms?
>
Chuck Murcko
Topsail Group
http://www.topsail.org/
Re: thread locking within apr file io
Posted by rb...@covalent.net.
On Thu, 26 Apr 2001, Justin Erenkrantz wrote:
> > > > This stuff came up while Justin and Aaron were performance testing
> > > > mod_mbox -- right now you can double the requests per second simply
> > > > by recompiling with --disable-threads.
> >
> > Wow, which MPM are you using? That is actually quite a suprise to me as Apache 2.0 performance is
> > close to that of Apache 1.3 (on AIX at least) which does not use pool locking or buffer locking for
> > file i/o.
>
> We are currently using prefork because pthread is completely useless at
> this point. Hopefully, the signals stuff gets fixed soon... Actually,
> on FreeBSD, threaded MPMs are disabled, but no one tells APR, so it
> defines APR_HAS_THREADS - we need to fix that. I may submit a patch for
> that - just add --disable-threads to APRs configure for these platforms?
Uh, on FreeBSD, APR doesn't allow threads at all. We have specifically
disabled threads on FreeBSD in the APR configure script, which is why
Apache won't use the threaded MPMs.
Ryan
_______________________________________________________________________________
Ryan Bloom rbb@apache.org
406 29th St.
San Francisco, CA 94131
-------------------------------------------------------------------------------
Re: thread locking within apr file io
Posted by Justin Erenkrantz <je...@ebuilt.com>.
> > > This stuff came up while Justin and Aaron were performance testing
> > > mod_mbox -- right now you can double the requests per second simply
> > > by recompiling with --disable-threads.
>
> Wow, which MPM are you using? That is actually quite a suprise to me as Apache 2.0 performance is
> close to that of Apache 1.3 (on AIX at least) which does not use pool locking or buffer locking for
> file i/o.
We are currently using prefork because pthread is completely useless at
this point. Hopefully, the signals stuff gets fixed soon... Actually,
on FreeBSD, threaded MPMs are disabled, but no one tells APR, so it
defines APR_HAS_THREADS - we need to fix that. I may submit a patch for
that - just add --disable-threads to APRs configure for these platforms?
The traditional static pages bypass most of the locks because they only
deal with ap_send_fd (apr_bucket_file.c). They also aren't stressing
the locking. Lots of apr_pallocs in the SDBM code (and to some extent
in mod_mbox). Each request needs to load the SDBM file into memory.
Somehow there is an exclusive cross-process lock going on somewhere
(upon further review, we don't think it is fcntl - but it still might
be...). The net result is that only one process is running at a time.
There seems to be a lot of lock contention going on within SDBM - we
ONLY want read access. As Roy has pointed out, there should be very
little locking going on in the mainline read code ANYWHERE within Apache
(and hence within APR). This is why each request gets its own
thread/process/whatever-the-MPM-decides. Requests shouldn't spawn
threads themselves. r->pool is allocated on a per-request-basis.
Therefore, it is pointless to lock that. By its very nature, we
know that we are exclusive.
I was actually shocked, but mod_mbox's performance *doubles* when we
disable APR threading. We were able to duplicate this on FreeBSD
and Linux (haven't seen what the performance on Solaris is yet).
mod_mbox doesn't give a hoot for threads anywhere. Actually IIRC,
SDBM doesn't either explicitly - but the pools, IO, and locks do.
I'm not exactly sure where to start with removing the locks. I guess
we could just remove ALL of the locking calls in read and pools (unless
global pools) and see if we can get threaded APR to be closer to
non-threaded APR. Roy might have some ideas... -- justin
Re: thread locking within apr file io
Posted by Bill Stoddard <bi...@wstoddard.com>.
> On Wed, 25 Apr 2001, Roy T. Fielding wrote:
>
> > APR file io, when compiled with APR_HAS_THREADS (the default unless it
> > is specifically disabled or on a non-thread platform), is acquiring
> > mutex locks on every buffered file read or write. That is lunacy.
> > Whether or not locking is necessary must be decided by the caller,
> > not by the library. The normal behavior is for buffers to be
> > thread-specific, and thus never need mutexing.
> >
> > BTW, the same can be said for pools -- right now it is locking all
> > pool operations, even though the only ones that need locking are those
> > shared by multiple threads (the global pools). Bogus.
> >
> > This stuff came up while Justin and Aaron were performance testing
> > mod_mbox -- right now you can double the requests per second simply
> > by recompiling with --disable-threads.
Wow, which MPM are you using? That is actually quite a suprise to me as Apache 2.0 performance is
close to that of Apache 1.3 (on AIX at least) which does not use pool locking or buffer locking for
file i/o.
>
> Well, that's a pretty major bump. This makes complete sense to me. I say
> let's remove the locks. Make the application lock the code itself, which
> makes a bit more sense to me.
>
> The thread locks in the pools came from the Windows pool code, I would
> love to see those locks go away ASAP.
>
> Ryan
>
We need -some- pool locking. For instance, all the threads allocate subpools out of a pool common to
the process the threads are running in. It would be easy enough to mutex those cases (all of which
are in the MPM I believe).
Bill
Re: thread locking within apr file io
Posted by rb...@covalent.net.
On Wed, 25 Apr 2001, Roy T. Fielding wrote:
> APR file io, when compiled with APR_HAS_THREADS (the default unless it
> is specifically disabled or on a non-thread platform), is acquiring
> mutex locks on every buffered file read or write. That is lunacy.
> Whether or not locking is necessary must be decided by the caller,
> not by the library. The normal behavior is for buffers to be
> thread-specific, and thus never need mutexing.
>
> BTW, the same can be said for pools -- right now it is locking all
> pool operations, even though the only ones that need locking are those
> shared by multiple threads (the global pools). Bogus.
>
> This stuff came up while Justin and Aaron were performance testing
> mod_mbox -- right now you can double the requests per second simply
> by recompiling with --disable-threads.
Well, that's a pretty major bump. This makes complete sense to me. I say
let's remove the locks. Make the application lock the code itself, which
makes a bit more sense to me.
The thread locks in the pools came from the Windows pool code, I would
love to see those locks go away ASAP.
Ryan
_______________________________________________________________________________
Ryan Bloom rbb@apache.org
406 29th St.
San Francisco, CA 94131
-------------------------------------------------------------------------------