You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Cliff Woolley <cl...@yahoo.com> on 2001/08/09 22:27:28 UTC
are we ready to roll?
So are we ready to roll 2.0.23? I haven't heard any showstopper
complaints about it that haven't been addressed already...
--------------------------------------------------------------
Known issues with 2.0.23:
1) Win9x, WinME, and Netware do not yet work.
2) Unix: The threaded MPM might take longer than expected to perform
a graceful restart if an insufficient number of incoming requests
are being received at the time of the restart. This is due to the
accept model for connections used by the threaded MPM and the fact that
workers cannot be awakened to receive the shutdown signal until they
receive an incoming connection. A new MPM (temporarily called "worker")
is being developed to correct this problem using a different accept
model, though it is not ready yet. For now, the prefork MPM is
considered the most stable (and is therefore currently the default),
though threaded has made quite a bit of progress since 2.0.22. We are
very interested in real-world performance comparisons between
prefork and threaded, particularly on heavily-loaded servers.
Any feedback along those lines would be greatly appreciated.
3) mod_auth_dbm, mod_auth_db, and mod_auth_digest might not compile on
some systems due to missing headers or libraries which are not
correctly flagged as missing by configure. Using --enable-modules=most,
--enable-shared=most, etc, can enable some of these modules even
on platforms where they will not compile. If you have trouble
compiling any of them, you can disable the offending module by
rerunning configure and adding --disable-auth-dbm, --disable-auth-db,
or --disable-auth-digest to the end of your configure line.
4) mod_ssl is still in the process of being ported to Apache 2.0 and
is considered alpha quality. Considerable progress has been made
on it since 2.0.22, though it still might not work on all systems and
not all functionality has yet been enabled. One particular problem of
note is that some people have experienced problems POSTing data to
mod_ssl and mod_tls.
5) There is a known build problem when using GNU make version 3.77
on some systems, which appears to be a bug in that version of gmake.
Upgrading to a newer version of gmake fixes the problem. However,
this bug is apparently being triggered by some non-portable features
being used in one of our makefiles and has been seen to cause problems
with some platforms' native makes as well (OS/390 in particular).
--------------------------------------------------------------
The following patches have been committed to 2.0.24-dev but not to 2.0.23:
a) Ryan's http_protocol.c ap_http_filter() patches
b) a four line patch to ssl_engine_io.c (rev 1.15) removing the hardcoded
APR_BLOCK_READ/AP_MODE_BLOCKING settings
c) s/socket/skt/ in apr_network_io.h for apr_socket_(un)set_inherit
d) the patch from an hour or so ago to apr_pools.c to not destroy a
mutex that was never created
e) Remove "sinclude(../apr/build/libtool.m4)" from apr-util/configure.in
Do I need to get any of these things tagged in to 2.0.23, or can they wait
until 2.0.24? If the answer is no to all, then as far as I know we're
ready to roll sometime this evening...
Thanks,
Cliff
--------------------------------------------------------------
Cliff Woolley
cliffwoolley@yahoo.com
Charlottesville, VA
Re: are we ready to roll?
Posted by Greg Ames <gr...@remulak.net>.
Ryan Bloom wrote:
>
> On Thursday 09 August 2001 22:17, Greg Ames wrote:
> > > Known issues with 2.0.23:
> > >
> > > 1) Win9x, WinME, and Netware do not yet work.
> > > 2) Unix: The threaded MPM might take longer than expected to perform
> > > a graceful restart if an insufficient number of incoming requests
> > > are being received at the time of the restart.
> >
> > Sorry I missed it when you first wrote this, but the sentence above
> > isn't stricly true, and I believe it may be causing some folks to be
> > overly alarmed about threaded.
> >
> > Graceful restarts happen just as quickly in threaded as with prefork, as
> > far as parsing the config file, serving new http requests with the new
> > config, and finishing old requests are concerned. The one and only
> > thing that may take longer under low load conditions is cleaning up idle
> > threads and their processes from the old generation. I don't consider
> > that much of a concern, especially since any new requests help resolve
> > the situation.
>
> Greg, I'm sorry, but this is just not true. Please take a look at the code.
I'm quite familiar with it, as you are well aware.
> The
> way that graceful restarts work in the threaded mode, is that we accept the
> connection,
bzzzzt! wrong. Now we're getting somewhere - I think I understand why
you think threaded has a problem.
The way that graceful restarts work in threaded, is that one thread is
blocked in apr_poll. When the parent initiates graceful restart, it
writes at least one character per child process to the PoD. This wakes
up the thread blocked in poll, who sets workers_may_exit, releases the
mutex, sets ps->quiescing and exits.
If the next thread to get the mutex happens to be in the same process,
it doesn't do much beside release the mutex once more, decrement the
thread count and exit. If the current mutex holder is in a different
process, then it goes into the poll etc as described in the paragraph
above.
If the mutex holder happens to get awakened by a new connection before
the PoD is readable, it will server the request using the old
generation's config, just like in prefork.
> serve the request, and then shutdown the thread. Since threads
> serve a request before they are shutdown, having them sitting around means that
> the graceful restart hasn't been finished yet. It also means that a site could be
> serving pages with the old config long after the graceful restart was signalled.
A site with threaded won't be serving pages with the old config any
longer than it will with prefork.
> Try doing graceful restarts on a server that isn't being hit continuosly.
not a problem.
> If you
> think this case doesn't matter, consider slashdot, and what will happen if you
> are running a server that is usually very quiet, but you believe that you will be
> being hit very hard soon. Most people would re-config their server, and do a
> graceful restart to handle the load they fear. With the threaded MPM, the first
> set of requests will be served with the old config instead of the new one.
no sir, it won't. Any new requests receive after the PoD is seen are
served in new generation processes, just like in prefork
> I have been on site at a user who wanted to re-config their server everyday at
> 3:00pm, because they always got a big spike from 3:00 - 5:00, as people got
> ready to leave the office at the end of the day. At other times, they didn't have the
> same load, so they wanted the server to use less resources. With today's threaded
> MPM, I couldn't say to them that their server would only finish the current requests
> with the old config, and all new requests would be served with the new config.
If you see any evidence of new requests being served with the old
config, please let us know ASAP. AFAIK, it just doesn't happen.
Greg
Re: are we ready to roll?
Posted by Ryan Bloom <rb...@covalent.net>.
On Thursday 09 August 2001 22:17, Greg Ames wrote:
> > Known issues with 2.0.23:
> >
> > 1) Win9x, WinME, and Netware do not yet work.
> > 2) Unix: The threaded MPM might take longer than expected to perform
> > a graceful restart if an insufficient number of incoming requests
> > are being received at the time of the restart.
>
> Sorry I missed it when you first wrote this, but the sentence above
> isn't stricly true, and I believe it may be causing some folks to be
> overly alarmed about threaded.
>
> Graceful restarts happen just as quickly in threaded as with prefork, as
> far as parsing the config file, serving new http requests with the new
> config, and finishing old requests are concerned. The one and only
> thing that may take longer under low load conditions is cleaning up idle
> threads and their processes from the old generation. I don't consider
> that much of a concern, especially since any new requests help resolve
> the situation.
Greg, I'm sorry, but this is just not true. Please take a look at the code. The
way that graceful restarts work in the threaded mode, is that we accept the
connection, serve the request, and then shutdown the thread. Since threads
serve a request before they are shutdown, having them sitting around means that
the graceful restart hasn't been finished yet. It also means that a site could be
serving pages with the old config long after the graceful restart was signalled.
Yes, on a heavily hit server, this is much less of an issue, but it doesn't go
away, because we rely on the OS to ensure that every thread get's its fair
share of requests.
> As you know from personal experience, you can do several graceful
> restarts per second with threaded in 2.0.23, without problems.
Try doing graceful restarts on a server that isn't being hit continuosly. If you
think this case doesn't matter, consider slashdot, and what will happen if you
are running a server that is usually very quiet, but you believe that you will be
being hit very hard soon. Most people would re-config their server, and do a
graceful restart to handle the load they fear. With the threaded MPM, the first
set of requests will be served with the old config instead of the new one.
I have been on site at a user who wanted to re-config their server everyday at
3:00pm, because they always got a big spike from 3:00 - 5:00, as people got
ready to leave the office at the end of the day. At other times, they didn't have the
same load, so they wanted the server to use less resources. With today's threaded
MPM, I couldn't say to them that their server would only finish the current requests
with the old config, and all new requests would be served with the new config.
Ryan
_____________________________________________________________________________
Ryan Bloom rbb@apache.org
Covalent Technologies rbb@covalent.net
-----------------------------------------------------------------------------
Re: are we ready to roll?
Posted by Greg Ames <gr...@remulak.net>.
> Known issues with 2.0.23:
>
> 1) Win9x, WinME, and Netware do not yet work.
> 2) Unix: The threaded MPM might take longer than expected to perform
> a graceful restart if an insufficient number of incoming requests
> are being received at the time of the restart.
Sorry I missed it when you first wrote this, but the sentence above
isn't stricly true, and I believe it may be causing some folks to be
overly alarmed about threaded.
Graceful restarts happen just as quickly in threaded as with prefork, as
far as parsing the config file, serving new http requests with the new
config, and finishing old requests are concerned. The one and only
thing that may take longer under low load conditions is cleaning up idle
threads and their processes from the old generation. I don't consider
that much of a concern, especially since any new requests help resolve
the situation.
As you know from personal experience, you can do several graceful
restarts per second with threaded in 2.0.23, without problems.
> This is due to the
> accept model for connections used by the threaded MPM and the fact that
> workers cannot be awakened to receive the shutdown signal until they
> receive an incoming connection. A new MPM (temporarily called "worker")
> is being developed to correct this problem using a different accept
> model, though it is not ready yet. For now, the prefork MPM is
> considered the most stable (and is therefore currently the default),
> though threaded has made quite a bit of progress since 2.0.22. We are
> very interested in real-world performance comparisons between
> prefork and threaded, particularly on heavily-loaded servers.
> Any feedback along those lines would be greatly appreciated.
Re: are we ready to roll?
Posted by Cliff Woolley <cl...@yahoo.com>.
On Fri, 10 Aug 2001, Sander Temme wrote:
> on 8/9/01 10:52 PM, Cliff Woolley at jwoolley@virginia.edu wrote:
>
> > I've been trying to figure out how it is that the config.guess's and
> > config.sub's that were in the 2.0.22 tarball didn't work on Darwin but DO
> > work from CVS. All three copies of each that make it into the tarball
>
> Two problematic ones left in the tarball:
>
> [monalisa:~/projects/httpd-2_0_23] sctemme% find . -name config.guess -print
> -exec sh {} \;
> ./srclib/apr/build/config.guess
> powerpc-apple-darwin1.3.7
> ./srclib/apr/shmem/unix/mm/config.guess
> powerpc-apple-darwin1.3.7
> ./srclib/apr-util/xml/expat/conftools/config.guess
> ./srclib/pcre/config.guess
> [monalisa:~/projects/httpd-2_0_23] sctemme%
>
> However, only the expat one causes trouble, and if I replace only that one
> with the download from the FSF, configure completes. PCRE, which has the
> other old config.guess, doesn't seem to use it:
Hmmm... okay. Well, until this is fixed (which we should be able to do in
our CVS copies of config.guess), we'll just tell people on Darwin to
update their copies of config.guess. <sigh> Oh well.
--------------------------------------------------------------
Cliff Woolley
cliffwoolley@yahoo.com
Charlottesville, VA
Re: are we ready to roll?
Posted by Sander Temme <sc...@covalent.net>.
on 8/9/01 10:52 PM, Cliff Woolley at jwoolley@virginia.edu wrote:
> I've been trying to figure out how it is that the config.guess's and
> config.sub's that were in the 2.0.22 tarball didn't work on Darwin but DO
> work from CVS. All three copies of each that make it into the tarball
Two problematic ones left in the tarball:
[monalisa:~/projects/httpd-2_0_23] sctemme% find . -name config.guess -print
-exec sh {} \;
./srclib/apr/build/config.guess
powerpc-apple-darwin1.3.7
./srclib/apr/shmem/unix/mm/config.guess
powerpc-apple-darwin1.3.7
./srclib/apr-util/xml/expat/conftools/config.guess
./srclib/pcre/config.guess
[monalisa:~/projects/httpd-2_0_23] sctemme%
However, only the expat one causes trouble, and if I replace only that one
with the download from the FSF, configure completes. PCRE, which has the
other old config.guess, doesn't seem to use it:
Configuring PCRE regular expression library ...
configuring package in srclib/pcre now
loading cache ../.././config.cache
checking for gcc... (cached) cc
checking whether the C compiler (cc ) works... yes
checking whether the C compiler (cc ) is a cross-compiler... no
checking whether we are using GNU C... (cached) yes
checking whether cc accepts -g... (cached) yes
checking for ranlib... (cached) ranlib
checking how to run the C preprocessor... (cached) cc -E -traditional-cpp
checking for ANSI C header files... (cached) yes
checking for limits.h... (cached) yes
checking for working const... (cached) yes
checking for size_t... (cached) yes
checking for bcopy... (cached) yes
checking for memmove... (cached) yes
checking for strerror... yes
updating cache ../.././config.cache
creating ./config.status
creating Makefile
creating pcre.h
creating pcre-config
creating config.h
srclib/pcre configured properly
loading cache ./config.cache
Forward to a fresh checkout of apr-util: there is no config.guess in its
conftools directory. Instead, it is pulled in by the buildconf.sh script
which runs libtoolize:
[monalisa:apr-util/xml/expat] sctemme% pwd
/Users/sctemme/projects/apr-util/xml/expat
[monalisa:apr-util/xml/expat] sctemme% glibtoolize --force --copy --dry-run
You should add the contents of `/usr/share/aclocal/libtool.m4' to
`aclocal.m4'.
Putting files in AC_CONFIG_AUX_DIR, `conftools'.
rm -f config.guess
cp -f /usr/share/libtool/config.guess config.guess
rm -f config.sub
cp -f /usr/share/libtool/config.sub config.sub
rm -f ltconfig
cp -f /usr/share/libtool/ltconfig ltconfig
rm -f ltmain.sh
cp -f /usr/share/libtool/ltmain.sh ltmain.sh
[monalisa:apr-util/xml/expat] sctemme%
... so that's where all those come from.
How is expat pulled into the tarball? Is it checked out from CVS (in which
case it has to be massaged into tarrable shape) or is a distribution used
with whatever material is in there?
Should I send this to another mailinglist like one for APR?
S.
--
Covalent Technologies sctemme@covalent.net
Engineering group Voice: (415) 536 5214
645 Howard St. Fax: (415) 536 5210
San Francisco CA 94105
PGP Fingerprint: 1E74 4E58 DFAC 2CF5 6A03 5531 AFB1 96AF B584 0AB1
=======================================================
This email message is for the sole use of the intended recipient(s) and may
contain confidential and privileged information. Any unauthorized review,
use, disclosure or distribution is prohibited. If you are not the intended
recipient, please contact the sender by reply email and destroy all copies
of the original message
=======================================================
Re: are we ready to roll?
Posted by Cliff Woolley <jw...@virginia.edu>.
On Thu, 9 Aug 2001, Sander Temme wrote:
> Tag checkout, updated today. OS:
>
> [monalisa:/usr/local/apache2] sctemme% uname -a
> Darwin MonaLisa 1.3.7 Darwin Kernel Version 1.3.7: Sat Jun 9 11:12:48 PDT
> 2001; root:xnu/xnu-124.13.obj~1/RELEASE_PPC Power Macintosh powerpc
>
> Configured without options, makes, installs cleanly. Prefork MPM.
I've been trying to figure out how it is that the config.guess's and
config.sub's that were in the 2.0.22 tarball didn't work on Darwin but DO
work from CVS. All three copies of each that make it into the tarball
*are* in CVS... granted, only two of the three actually have an entry for
Darwin. But if it works from CVS, it really ought to work from the
tarball. Grrr. Anyway, I'm going to pretend this issue doesn't exist and
roll 2.0.23 anyway. If it turns out to be a problem again as it was with
2.0.22, we'll just stick the same known-problem entry on as we did before.
It's an easy workaround, assuming it's even necessary.
> One observation: in httpd.conf ll 115-121:
>
> <IfModule prefork.c>
> StartServers 5
> MinSpareServers 5
> MaxSpareServers 10
> MaxClients 20
> MaxRequestsPerChild 0
> </IfModule>
>
> Isn't that awful low for MaxClients?
The default MPM parameters for most/all of the MPMs are either on the low
side or just completely wacky. The defaults for threaded cause it to spin
up three processes and immediately kill two of them off because there are
too many idle workers. =-) I figure we'll tweak these later.
--Cliff
--------------------------------------------------------------
Cliff Woolley
cliffwoolley@yahoo.com
Charlottesville, VA
Re: are we ready to roll?
Posted by Sander Temme <sc...@covalent.net>.
on 8/9/01 1:27 PM, Cliff Woolley at cliffwoolley@yahoo.com wrote:
> So are we ready to roll 2.0.23? I haven't heard any showstopper
> complaints about it that haven't been addressed already...
Tag checkout, updated today. OS:
[monalisa:/usr/local/apache2] sctemme% uname -a
Darwin MonaLisa 1.3.7 Darwin Kernel Version 1.3.7: Sat Jun 9 11:12:48 PDT
2001; root:xnu/xnu-124.13.obj~1/RELEASE_PPC Power Macintosh powerpc
Configured without options, makes, installs cleanly. Prefork MPM.
ab'ed from localhost, no problemo. Looking good...
One observation: in httpd.conf ll 115-121:
<IfModule prefork.c>
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 20
MaxRequestsPerChild 0
</IfModule>
Isn't that awful low for MaxClients?
S.
--
Covalent Technologies sctemme@covalent.net
Engineering group Voice: (415) 536 5214
645 Howard St. Fax: (415) 536 5210
San Francisco CA 94105
PGP Fingerprint: 1E74 4E58 DFAC 2CF5 6A03 5531 AFB1 96AF B584 0AB1
=======================================================
This email message is for the sole use of the intended recipient(s) and may
contain confidential and privileged information. Any unauthorized review,
use, disclosure or distribution is prohibited. If you are not the intended
recipient, please contact the sender by reply email and destroy all copies
of the original message
=======================================================
Re: are we ready to roll?
Posted by Doug MacEachern <do...@covalent.net>.
On Thu, 9 Aug 2001, Cliff Woolley wrote:
> Are these problems in 2.0.23 or in 2.0.24-dev? The ap_http_filter changes
> are in 2.0.24-dev...
2.0.24-dev
Re: are we ready to roll?
Posted by Cliff Woolley <cl...@yahoo.com>.
On Thu, 9 Aug 2001, Doug MacEachern wrote:
> On Thu, 9 Aug 2001, Cliff Woolley wrote:
>
> >
> > So are we ready to roll 2.0.23? I haven't heard any showstopper
> > complaints about it that haven't been addressed already...
>
> there's still the issue of mod_ssl not working with POST after the recent
> ap_http_filter changes. and another issue where input filter on POST data
> is not working (modperl-2.0/t/filter/TestFilter/input_body.pm). i've
> tried debugging, but of course the test works when stepping with
> gdb. socket_read gets all the headers and POST data in one gulp when i'm
> stepping, otherwise gets it in two (headers and post data), and somehow
> the post data gets lost. dunno if either is considered a showstopper
> though.
Are these problems in 2.0.23 or in 2.0.24-dev? The ap_http_filter changes
are in 2.0.24-dev...
--Cliff
--------------------------------------------------------------
Cliff Woolley
cliffwoolley@yahoo.com
Charlottesville, VA
Re: are we ready to roll?
Posted by Doug MacEachern <do...@covalent.net>.
On Thu, 9 Aug 2001, Cliff Woolley wrote:
> APR_BRIGADE_FOREACH is finicky... in order to access the next bucket on
> the next iteration, it assumes you didn't mess with the bucket pointer it
> gave you on the last iteration. If you fiddle with the bucket pointer
> that APR_BRIGADE_FOREACH gave you rather than just treating as a constant
> value, you're almost certain to break the assumptions made by that macro.
yikes. i guess i was doing that.
> Given the example you list above of what will work and what won't, I'm
> guessing that what your loop is doing is taking each bucket in the brigade
> one at a time, dinking with the contents (unencrypting them, perhaps),
> removing the bucket from the brigade, and re-injecting the data either
> into either a completely different bucket or possibly even into a
> different ("output") brigade. That just won't work with
> APR_BRIGADE_FOREACH, because when you destroy the bucket it gave you
> during the course of one iteration, it has no way to find out what the
> APR_BUCKET_NEXT(bucket) value is in order to move on to the next bucket in
> the brigade. (Did that make sense? I'm rambling...)
>
> The "while (!APR_BRIGADE_EMPTY(bb)) {}" trick is a very commonly-used way
> to avoid these sorts of problems when doing these sorts of things. It
> probably deserves a big honkin @tip or @warning in the docco for
> APR_BRIGADE_FOREACH. I'll go add one. =-)
thanks for explaining. big honking warning would be good.
Re: are we ready to roll?
Posted by Cliff Woolley <cl...@yahoo.com>.
On Thu, 9 Aug 2001, Doug MacEachern wrote:
> well, i found a solution when writing a c version of the test.
>
> APR_BRIGADE_FOREACH(bucket, bb) {
> ...
> }
>
> does not work, but this does:
>
> while (!APR_BRIGADE_EMPTY(bb)) {
> bucket = APR_BRIGADE_FIRST(bb);
> ...
> if (APR_BUCKET_IS_EOS(bucket)) {
> break;
> }
> }
>
> i've seen other cases of this, in ssl_engine_io.c for example:
>
> /* XXX: shame that APR_BRIGADE_FOREACH doesn't work here */
>
> what's up with that?
The question is what's in the '...' in the code snippet above.
APR_BRIGADE_FOREACH is finicky... in order to access the next bucket on
the next iteration, it assumes you didn't mess with the bucket pointer it
gave you on the last iteration. If you fiddle with the bucket pointer
that APR_BRIGADE_FOREACH gave you rather than just treating as a constant
value, you're almost certain to break the assumptions made by that macro.
Given the example you list above of what will work and what won't, I'm
guessing that what your loop is doing is taking each bucket in the brigade
one at a time, dinking with the contents (unencrypting them, perhaps),
removing the bucket from the brigade, and re-injecting the data either
into either a completely different bucket or possibly even into a
different ("output") brigade. That just won't work with
APR_BRIGADE_FOREACH, because when you destroy the bucket it gave you
during the course of one iteration, it has no way to find out what the
APR_BUCKET_NEXT(bucket) value is in order to move on to the next bucket in
the brigade. (Did that make sense? I'm rambling...)
The "while (!APR_BRIGADE_EMPTY(bb)) {}" trick is a very commonly-used way
to avoid these sorts of problems when doing these sorts of things. It
probably deserves a big honkin @tip or @warning in the docco for
APR_BRIGADE_FOREACH. I'll go add one. =-)
--Cliff
--------------------------------------------------------------
Cliff Woolley
cliffwoolley@yahoo.com
Charlottesville, VA
Re: are we ready to roll?
Posted by Doug MacEachern <do...@covalent.net>.
On Thu, 9 Aug 2001, Doug MacEachern wrote:
> and another issue where input filter on POST data
> is not working (modperl-2.0/t/filter/TestFilter/input_body.pm). i've
> tried debugging, but of course the test works when stepping with
> gdb. socket_read gets all the headers and POST data in one gulp when i'm
> stepping, otherwise gets it in two (headers and post data), and somehow
> the post data gets lost. dunno if either is considered a showstopper
> though.
well, i found a solution when writing a c version of the test.
APR_BRIGADE_FOREACH(bucket, bb) {
...
}
does not work, but this does:
while (!APR_BRIGADE_EMPTY(bb)) {
bucket = APR_BRIGADE_FIRST(bb);
...
if (APR_BUCKET_IS_EOS(bucket)) {
break;
}
}
i've seen other cases of this, in ssl_engine_io.c for example:
/* XXX: shame that APR_BRIGADE_FOREACH doesn't work here */
what's up with that?
Re: are we ready to roll?
Posted by Doug MacEachern <do...@covalent.net>.
On Thu, 9 Aug 2001, Cliff Woolley wrote:
>
> So are we ready to roll 2.0.23? I haven't heard any showstopper
> complaints about it that haven't been addressed already...
there's still the issue of mod_ssl not working with POST after the recent
ap_http_filter changes. and another issue where input filter on POST data
is not working (modperl-2.0/t/filter/TestFilter/input_body.pm). i've
tried debugging, but of course the test works when stepping with
gdb. socket_read gets all the headers and POST data in one gulp when i'm
stepping, otherwise gets it in two (headers and post data), and somehow
the post data gets lost. dunno if either is considered a showstopper
though.