You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Cliff Woolley <cl...@yahoo.com> on 2001/08/08 08:53:47 UTC
Currently known issues with 2.0.23 (very few! :)
--------------------------------------------------------------
Known issues with 2.0.23:
1) Win9x, WinME, and Netware do not yet work.
2) Unix: The threaded MPM might take longer than expected to restart or
shutdown on very-low-traffic (near idle) servers. This is due to the
accept model for connections used by the threaded MPM and the fact that
workers cannot be awakened to receive the shutdown signal until they
receive an incoming connection. A new MPM (temporarily called "worker")
is being developed to correct this problem using a different accept
model, though it is not ready yet. For now, the prefork MPM is
considered the most stable (and is therefore currently the default),
though threaded has made quite a bit of progress since 2.0.22. We are
very interested in real-world performance comparisons between
prefork and threaded, particularly on heavily-loaded servers.
Any feedback along those lines would be greatly appreciated.
3) mod_auth_dbm, mod_auth_db, and mod_auth_digest might not compile on
some systems due to missing headers or libraries which are not
correctly flagged as missing by configure. Using --enable-modules=most,
--enable-shared=most, etc, can enable some of these modules even
on platforms where they will not compile. If you have trouble
compiling any of them, you can disable the offending module by
rerunning configure and adding --disable-auth-dbm, --disable-auth-db,
or --disable-auth-digest to the end of your configure line.
4) mod_ssl is still in the process of being ported to Apache 2.0 and
is considered alpha quality. Considerable progress has been made
on it since 2.0.22, though it still might not work on all systems and
not all functionality has yet been enabled.
5) There is a known build problem when using GNU make version 3.77
on some systems, which appears to be a bug in that version of gmake.
Upgrading to a newer version of gmake fixes the problem.
--------------------------------------------------------------
Is #5 still a problem? Are my explanations for #2-4 satisfactory? Most
importantly, are there any known problems that are NOT on this list?
Thanks,
Cliff
--------------------------------------------------------------
Cliff Woolley
cliffwoolley@yahoo.com
Charlottesville, VA
Re: Currently known issues with 2.0.23 (very few! :)
Posted by Jerry Baker <je...@weirdness.com>.
Ryan Bloom wrote:
>
> This was actually just fixed by Doug M. :-)
>
> Ryan
Still crashes. Stacks attached.
BIO_write(bio_st * 0x00615818, const void * 0x1067bee0, int 476) line 184 + 13 bytes
churn(SSLFilterRec * 0x005fae70, int 0, __int64 * 0x1046dcc0) line 281 + 26 bytes
ssl_io_filter_Input(ap_filter_t * 0x005fae90, apr_bucket_brigade * 0x005faf60, int 0, __int64 * 0x1046dcc0) line 370 + 17 bytes
ap_get_brigade(ap_filter_t * 0x005fae90, apr_bucket_brigade * 0x005faf60, int 0, __int64 * 0x1046dcc0) line 217 + 24 bytes
ap_http_filter(ap_filter_t * 0x005faf10, apr_bucket_brigade * 0x0057c428, int 0, __int64 * 0x1046dcc0) line 644 + 26 bytes
ap_get_brigade(ap_filter_t * 0x005faf10, apr_bucket_brigade * 0x0057c428, int 0, __int64 * 0x1046dcc0) line 217 + 24 bytes
ap_getline(char * 0x1046dd7c, int 8192, request_rec * 0x0057bca8, int 0) line 223 + 22 bytes
read_request_line(request_rec * 0x0057bca8) line 396 + 23 bytes
ap_read_request(conn_rec * 0x00544670) line 593 + 9 bytes
ap_process_http_connection(conn_rec * 0x00544670) line 281 + 9 bytes
ap_run_process_connection(conn_rec * 0x00544670) line 82 + 81 bytes
ap_process_connection(conn_rec * 0x00544670) line 221
worker_main(int 249) line 908
_threadstartex(void * 0x005f86c8) line 212 + 13 bytes
KERNEL32! 77e8758a()
SSL_CTX_ctrl(ssl_ctx_st * 0x00000000, int 32, long 1048575, char * 0x00000000) line 871 + 3 bytes
ssl_init_ConfigureServer(server_rec * 0x00578b48, apr_pool_t * 0x005198a0, SSLSrvConfigRec * 0x00540c30) line 527 + 18 bytes
ssl_init_Module(apr_pool_t * 0x005198a0, apr_pool_t * 0x00548618, apr_pool_t * 0x0054e6f0, server_rec * 0x0051a6c8) line 247 + 17 bytes
ap_run_post_config(apr_pool_t * 0x005198a0, apr_pool_t * 0x00548618, apr_pool_t * 0x0054e6f0, server_rec * 0x0051a6c8) line 126 + 90 bytes
main(int 3, const char * const * 0x00513730) line 423
mainCRTStartup() line 338 + 17 bytes
KERNEL32! 77e97d08()
--
Jerry Baker
PGP Key: http://www.jerrybaker.org/pgp.html
LAME MP3 Encoder Binaries: http://www.jerrybaker.org/lame/
Apache 2.0 Web server Installer: http://www.jerrybaker.org/apache/
Re: Currently known issues with 2.0.23 (very few! :)
Posted by Ryan Bloom <rb...@covalent.net>.
This was actually just fixed by Doug M. :-)
Ryan
On Tuesday 07 August 2001 23:57, Jerry Baker wrote:
> Cliff Woolley wrote:
> > --------------------------------------------------------------
> >
> > Is #5 still a problem? Are my explanations for #2-4 satisfactory? Most
> > importantly, are there any known problems that are NOT on this list?
> >
> > Thanks,
> > Cliff
> >
> > --------------------------------------------------------------
> > Cliff Woolley
> > cliffwoolley@yahoo.com
> > Charlottesville, VA
>
> Might mention the whole mod_ssl and mod_tls thing where you cannot use a
> form to POST or GET more than about $ENV{CONTENT_LENGTH} = 570 bytes.
--
_____________________________________________________________________________
Ryan Bloom rbb@apache.org
Covalent Technologies rbb@covalent.net
-----------------------------------------------------------------------------
Re: Currently known issues with 2.0.23 (very few! :)
Posted by Doug MacEachern <do...@covalent.net>.
On Wed, 8 Aug 2001, Jerry Baker wrote:
> Sure. Using Netscape 4.78, IE 5.5, or Mozilla 0.9.2. The POST consists
> of a simple form containing one text area. Fill the text area with a few
> kilobytes of plain text and post it. POST is handled by a Perl script as
> follows:
that works fine for me with prefork mpm and mod_cgi. could be a win32
issue, i don't have a win32 environment to test with.
Re: Currently known issues with 2.0.23 (very few! :)
Posted by Jerry Baker <je...@weirdness.com>.
Doug MacEachern wrote:
>
> On Wed, 8 Aug 2001, Jerry Baker wrote:
> ....
> > There are two crashes in serial fashion immediately as soon as the form
> > is posted via mod_ssl. Here they are in the order they appear on the
> > machine:
>
> if you could post your test case (your client, whats posted from the
> client, what handles the post data, etc.), i'll look at in the
> morning. i've just been testing with:
>
Sure. Using Netscape 4.78, IE 5.5, or Mozilla 0.9.2. The POST consists
of a simple form containing one text area. Fill the text area with a few
kilobytes of plain text and post it. POST is handled by a Perl script as
follows:
-----------------------------------------------
#!C:\perl\bin\perl.exe
use CGI qw("param");
$variable = param("textarea");
open (FILE, ">test.txt") || die $!;
print FILE $variable;
close FILE;
------------------------------------------------
--
Jerry Baker
PGP Key: http://www.jerrybaker.org/pgp.html
LAME MP3 Encoder Binaries: http://www.jerrybaker.org/lame/
Apache 2.0 Web server Installer: http://www.jerrybaker.org/apache/
Re: Currently known issues with 2.0.23 (very few! :)
Posted by Doug MacEachern <do...@covalent.net>.
On Wed, 8 Aug 2001, Jerry Baker wrote:
...
> There are two crashes in serial fashion immediately as soon as the form
> is posted via mod_ssl. Here they are in the order they appear on the
> machine:
if you could post your test case (your client, whats posted from the
client, what handles the post data, etc.), i'll look at in the
morning. i've just been testing with:
% perl -e 'print "x" x 12048' | POST https://127.0.0.1:8443/echo_post
using various random sizes against the test handler below.
static int echo_post_handler(request_rec *r)
{
int rc;
long nrd;
char buff[BUFSIZ];
if (strcmp(r->handler, "echo-post")) {
return DECLINED;
}
if (r->method_number != M_POST) {
return DECLINED;
}
if ((rc = ap_setup_client_block(r, REQUEST_CHUNKED_ERROR)) != OK) {
ap_log_error(APLOG_MARK, APLOG_ERR|APLOG_NOERRNO, 0,
r->server,
"ap_setup_client_block failed: %d", rc);
return 0;
}
if (!ap_should_client_block(r)) {
return OK;
}
fprintf(stderr, "going to echo %d bytes\n", r->remaining);
while ((nrd = ap_get_client_block(r, buff, sizeof(buff))) > 0) {
fprintf(stderr, "read %d bytes (wanted %d, remaining=%d)\n",
nrd, sizeof(buff), r->remaining);
ap_rwrite(buff, nrd, r);
}
fprintf(stderr, "done reading, %d bytes remain\n", r->remaining);
return OK;
}
Re: Currently known issues with 2.0.23 (very few! :)
Posted by Jerry Baker <je...@weirdness.com>.
Cliff Woolley wrote:
>
> On Wed, 8 Aug 2001, Jerry Baker wrote:
>
> > I compiled it without changing anything except to change it to build
> > Installbin Release. mod_tls still has the POST problem. mod_ssl now
> > crashes upon POST.
>
> The patch needs to be ported to mod_tls. As for the crash, I don't
> know... I haven't had much of a chance to look at it. Do you have a
> backtrace we can look at?
>
> Doug? Any ideas?
>
> Thanks,
> Cliff
>
> --------------------------------------------------------------
> Cliff Woolley
> cliffwoolley@yahoo.com
> Charlottesville, VA
There are two crashes in serial fashion immediately as soon as the form
is posted via mod_ssl. Here they are in the order they appear on the
machine:
BIO_write(bio_st * 0x00723f70, const void * 0x005ca700, int 476) line 184 + 13 bytes
churn(SSLFilterRec * 0x005bf6d0, int 0, __int64 * 0x1056dcc0) line 285 + 26 bytes
ssl_io_filter_Input(ap_filter_t * 0x005bf6f0, apr_bucket_brigade * 0x005bf7c0, int 0, __int64 * 0x1056dcc0) line 374 + 17 bytes
ap_get_brigade(ap_filter_t * 0x005bf6f0, apr_bucket_brigade * 0x005bf7c0, int 0, __int64 * 0x1056dcc0) line 217 + 24 bytes
ap_http_filter(ap_filter_t * 0x005bf770, apr_bucket_brigade * 0x0057bbe0, int 0, __int64 * 0x1056dcc0) line 644 + 26 bytes
ap_get_brigade(ap_filter_t * 0x005bf770, apr_bucket_brigade * 0x0057bbe0, int 0, __int64 * 0x1056dcc0) line 217 + 24 bytes
ap_getline(char * 0x1056dd7c, int 8192, request_rec * 0x0057b460, int 0) line 223 + 22 bytes
read_request_line(request_rec * 0x0057b460) line 396 + 23 bytes
ap_read_request(conn_rec * 0x005442f0) line 593 + 9 bytes
ap_process_http_connection(conn_rec * 0x005442f0) line 281 + 9 bytes
ap_run_process_connection(conn_rec * 0x005442f0) line 82 + 81 bytes
ap_process_connection(conn_rec * 0x005442f0) line 221
worker_main(int 249) line 908
_threadstartex(void * 0x0070e050) line 212 + 13 bytes
KERNEL32! 77e8758a()
SSL_CTX_ctrl(ssl_ctx_st * 0x00000000, int 32, long 1048575, char * 0x00000000) line 871 + 3 bytes
ssl_init_ConfigureServer(server_rec * 0x00578238, apr_pool_t * 0x002f9788, SSLSrvConfigRec * 0x00540910) line 527 + 18 bytes
ssl_init_Module(apr_pool_t * 0x002f9788, apr_pool_t * 0x00548238, apr_pool_t * 0x0054e280, server_rec * 0x002fa5b0) line 247 + 17 bytes
ap_run_post_config(apr_pool_t * 0x002f9788, apr_pool_t * 0x00548238, apr_pool_t * 0x0054e280, server_rec * 0x002fa5b0) line 126 + 90 bytes
main(int 3, const char * const * 0x00512580) line 423
mainCRTStartup() line 338 + 17 bytes
KERNEL32! 77e97d08()
--
Jerry Baker
PGP Key: http://www.jerrybaker.org/pgp.html
LAME MP3 Encoder Binaries: http://www.jerrybaker.org/lame/
Apache 2.0 Web server Installer: http://www.jerrybaker.org/apache/
Re: Currently known issues with 2.0.23 (very few! :)
Posted by Cliff Woolley <cl...@yahoo.com>.
On Wed, 8 Aug 2001, Jerry Baker wrote:
> I compiled it without changing anything except to change it to build
> Installbin Release. mod_tls still has the POST problem. mod_ssl now
> crashes upon POST.
The patch needs to be ported to mod_tls. As for the crash, I don't
know... I haven't had much of a chance to look at it. Do you have a
backtrace we can look at?
Doug? Any ideas?
Thanks,
Cliff
--------------------------------------------------------------
Cliff Woolley
cliffwoolley@yahoo.com
Charlottesville, VA
Re: Currently known issues with 2.0.23 (very few! :)
Posted by Jerry Baker <je...@weirdness.com>.
Cliff Woolley wrote:
>
> On Wed, 8 Aug 2001, Jerry Baker wrote:
>
> > Might mention the whole mod_ssl and mod_tls thing where you cannot use a
> > form to POST or GET more than about $ENV{CONTENT_LENGTH} = 570 bytes.
>
> That should be fixed now, thanks to Doug M's commit about an hour and a
> half ago, which I then retagged into 2_0_23 (see rev 1.14 of
> ssl_engine_io.c). By all means, let us know if you're still seeing a
> problem there after doing a cvs up.
>
> Thanks,
> Cliff
>
> --------------------------------------------------------------
> Cliff Woolley
> cliffwoolley@yahoo.com
> Charlottesville, VA
I pulled the following:
cvs co -r APACHE_2_0_23 httpd-2.0
cvs co -r APACHE_2_0_23 apr
cvs co -r APACHE_2_0_23 apr-util
I compiled it without changing anything except to change it to build
Installbin Release. mod_tls still has the POST problem. mod_ssl now
crashes upon POST.
--
Jerry Baker
PGP Key: http://www.jerrybaker.org/pgp.html
LAME MP3 Encoder Binaries: http://www.jerrybaker.org/lame/
Apache 2.0 Web server Installer: http://www.jerrybaker.org/apache/
Re: Currently known issues with 2.0.23 (very few! :)
Posted by Cliff Woolley <cl...@yahoo.com>.
On Wed, 8 Aug 2001, Jerry Baker wrote:
> Might mention the whole mod_ssl and mod_tls thing where you cannot use a
> form to POST or GET more than about $ENV{CONTENT_LENGTH} = 570 bytes.
That should be fixed now, thanks to Doug M's commit about an hour and a
half ago, which I then retagged into 2_0_23 (see rev 1.14 of
ssl_engine_io.c). By all means, let us know if you're still seeing a
problem there after doing a cvs up.
Thanks,
Cliff
--------------------------------------------------------------
Cliff Woolley
cliffwoolley@yahoo.com
Charlottesville, VA
Re: Currently known issues with 2.0.23 (very few! :)
Posted by Jerry Baker <je...@weirdness.com>.
Cliff Woolley wrote:
>
> --------------------------------------------------------------
>
> Is #5 still a problem? Are my explanations for #2-4 satisfactory? Most
> importantly, are there any known problems that are NOT on this list?
>
> Thanks,
> Cliff
>
> --------------------------------------------------------------
> Cliff Woolley
> cliffwoolley@yahoo.com
> Charlottesville, VA
Might mention the whole mod_ssl and mod_tls thing where you cannot use a
form to POST or GET more than about $ENV{CONTENT_LENGTH} = 570 bytes.
--
Jerry Baker
PGP Key: http://www.jerrybaker.org/pgp.html
LAME MP3 Encoder Binaries: http://www.jerrybaker.org/lame/
Apache 2.0 Web server Installer: http://www.jerrybaker.org/apache/
Re: Currently known issues with 2.0.23 (very few! :)
Posted by Cliff Woolley <cl...@yahoo.com>.
On Wed, 8 Aug 2001, Greg Ames wrote:
> Greg Ames wrote:
>
> > As I mentioned to you off-list yesterday PM before rushing off to band
> > practice, replaying apache.org workload on my ThinkPad took it to its
> > knees. ab on a simple static file was fine. I haven't tried it yet
> > with the tag, but will ASAP.
>
> No problem running apache.org workload today with the tagged tree. It
> may have been debugging code I was running yesterday logging its brains
> out. Sorry for the false alarm.
No problem... glad to hear it. =-)
--Cliff
--------------------------------------------------------------
Cliff Woolley
cliffwoolley@yahoo.com
Charlottesville, VA
Re: Currently known issues with 2.0.23 (very few! :)
Posted by Greg Ames <gr...@remulak.net>.
Greg Ames wrote:
> As I mentioned to you off-list yesterday PM before rushing off to band
> practice, replaying apache.org workload on my ThinkPad took it to its
> knees. ab on a simple static file was fine. I haven't tried it yet
> with the tag, but will ASAP.
No problem running apache.org workload today with the tagged tree. It
may have been debugging code I was running yesterday logging its brains
out. Sorry for the false alarm.
Greg
Re: Currently known issues with 2.0.23 (very few! :)
Posted by Greg Ames <gr...@remulak.net>.
Cliff Woolley wrote:
>
> --------------------------------------------------------------
> Known issues with 2.0.23:
>
> 1) Win9x, WinME, and Netware do not yet work.
> 2) Unix: The threaded MPM might take longer than expected to restart or
> shutdown on very-low-traffic (near idle) servers. This is due to the
> accept model for connections used by the threaded MPM and the fact that
> workers cannot be awakened to receive the shutdown signal until they
> receive an incoming connection. A new MPM (temporarily called "worker")
> is being developed to correct this problem using a different accept
> model, though it is not ready yet. For now, the prefork MPM is
> considered the most stable (and is therefore currently the default),
> though threaded has made quite a bit of progress since 2.0.22. We are
> very interested in real-world performance comparisons between
> prefork and threaded, particularly on heavily-loaded servers.
> Any feedback along those lines would be greatly appreciated.
I think this is an accurate, unemotional way to explain the issue. Good
job!
hadn't thought much about shutdown. Checking the code, I believe it
could easily be make speedier.
If you're planning to put this in a README or whatever, you may want to
break it up into 2 or 3 paragraphs to make it easier on the eyeballs.
> 3) mod_auth_dbm, mod_auth_db, and mod_auth_digest might not compile on
> some systems due to missing headers or libraries which are not
> correctly flagged as missing by configure. Using --enable-modules=most,
> --enable-shared=most, etc, can enable some of these modules even
> on platforms where they will not compile. If you have trouble
> compiling any of them, you can disable the offending module by
> rerunning configure and adding --disable-auth-dbm, --disable-auth-db,
> or --disable-auth-digest to the end of your configure line.
> 4) mod_ssl is still in the process of being ported to Apache 2.0 and
> is considered alpha quality. Considerable progress has been made
> on it since 2.0.22, though it still might not work on all systems and
> not all functionality has yet been enabled.
> 5) There is a known build problem when using GNU make version 3.77
> on some systems, which appears to be a bug in that version of gmake.
> Upgrading to a newer version of gmake fixes the problem.
> --------------------------------------------------------------
>
> Is #5 still a problem? Are my explanations for #2-4 satisfactory? Most
> importantly, are there any known problems that are NOT on this list?
As I mentioned to you off-list yesterday PM before rushing off to band
practice, replaying apache.org workload on my ThinkPad took it to its
knees. ab on a simple static file was fine. I haven't tried it yet
with the tag, but will ASAP.
The hard drive was going constantly on my TP before it crashed, like
when we use uncontrolled quantities of virtual memory. The apache.org
workload has ssi's, autoindex, and some huge files.
Greg
Re: Currently known issues with 2.0.23 (very few! :)
Posted by Jeff Trawick <tr...@attglobal.net>.
Justin Erenkrantz <je...@ebuilt.com> writes:
> > 5) There is a known build problem when using GNU make version 3.77
> > on some systems, which appears to be a bug in that version of gmake.
> > Upgrading to a newer version of gmake fixes the problem.
>
> AFAIK, this is a problem. I don't care to spend any time fixing this
> when an upgraded version solves the problem. If someone wants to
> submit a patch, but this is really just a broken make. We should
> have a "Platform Notes" section in the README or INSTALL (where?).
> It'd be nice to also add the note about the broken Linux 2.4.3 stuff
> I stumbled across a few weeks ago. -- justin
I still hope to eventually fix server/Makefile.in so it will work on
the two systems where I've had problems (any OS/390 and a particular
Solaris 8/gmake ??? combination*). We're using a non-portable feature:
some/any form of expansion of wildcards in a dependency list.
I dunno exactly which problem Cliff refers to; I'm guessing he refers
to the one where running make twice works around the problem.
*maybe this is a gmake bug where an upgrade is appropriate, but since
it has to be fixed anyway for another platform's make it doesn't make
much difference to me
--
Jeff Trawick | trawick@attglobal.net | PGP public key at web site:
http://www.geocities.com/SiliconValley/Park/9289/
Born in Roswell... married an alien...
Re: Currently known issues with 2.0.23 (very few! :)
Posted by Justin Erenkrantz <je...@ebuilt.com>.
On Wed, Aug 08, 2001 at 02:53:47AM -0400, Cliff Woolley wrote:
> 3) mod_auth_dbm, mod_auth_db, and mod_auth_digest might not compile on
> some systems due to missing headers or libraries which are not
> correctly flagged as missing by configure. Using --enable-modules=most,
> --enable-shared=most, etc, can enable some of these modules even
> on platforms where they will not compile. If you have trouble
> compiling any of them, you can disable the offending module by
> rerunning configure and adding --disable-auth-dbm, --disable-auth-db,
> or --disable-auth-digest to the end of your configure line.
I'll focus on this next week (assuming I have time) unless someone
wants to jump in. I'm enough of an autoconf wimp to get this working
probably. If someone wants to submit a patch, I'll try to review
and commit (again, assumming no one beats me to it).
> 4) mod_ssl is still in the process of being ported to Apache 2.0 and
> is considered alpha quality. Considerable progress has been made
> on it since 2.0.22, though it still might not work on all systems and
> not all functionality has yet been enabled.
One note that Madhu (?) may not have considered is the special things
that need to happen w.r.t. OpenSSL for multi-threaded apps. Check
out flood in httpd-test. I've been mucking with the OpenSSL and
thread safety there for the past few days. I've finally got the right
voodoo happening in that code. flood is all APR-based so you could
take the code directly from there (see flood_net_ssl.c).
If no one has gotten to it by next week (see the commented out part
of ssl_engine_init for locking_callback), I'll try and commit the
relevant bits. I'd bet mod_ssl doesn't play nice with threaded MPM (or
worker for that matter). But, mod_ssl is a designated non-showstopper.
> 5) There is a known build problem when using GNU make version 3.77
> on some systems, which appears to be a bug in that version of gmake.
> Upgrading to a newer version of gmake fixes the problem.
AFAIK, this is a problem. I don't care to spend any time fixing this
when an upgraded version solves the problem. If someone wants to
submit a patch, but this is really just a broken make. We should
have a "Platform Notes" section in the README or INSTALL (where?).
It'd be nice to also add the note about the broken Linux 2.4.3 stuff
I stumbled across a few weeks ago. -- justin
Re: Currently known issues with 2.0.23
Posted by Greg Ames <gr...@remulak.net>.
Ian Holsman wrote:
> how about a line like:
> "gracefull restarts are currently not working properly on the threaded
> MPM. The work-around at the moment is to do a full restart instead of a
> graceful one."
actually, graceful restarts work pretty well on threaded as long as you
have some incoming connections, and don't hurt anything when there are
no connections coming in. OTOH, as it stands today, apachectl restart
won't be any faster than graceful on both threaded and worker if any
threads are tied up on long requests. Jeff has an idea which should
take care a lot of that.
I would recommend that admins for sites with traffic who want to use
threaded, use apachectl graceful with 2.0.23.
Greg
Re: Currently known issues with 2.0.23
Posted by Cliff Woolley <cl...@yahoo.com>.
On 8 Aug 2001, Ian Holsman wrote:
> how about a line like:
> "gracefull restarts are currently not working properly on the threaded
> MPM. The work-around at the moment is to do a full restart instead of a
> graceful one."
>
> That way admins have a workaround if they want to use threaded, and have
> been warned as well.
They _do_ work... as long as you have enough traffic hitting your server
at the time of the graceful to awaken all of the workers.
But your point is well taken... a mention of full restart as an
alternative is probably worthwhile.
--Cliff
--------------------------------------------------------------
Cliff Woolley
cliffwoolley@yahoo.com
Charlottesville, VA
Re: Currently known issues with 2.0.23
Posted by Ian Holsman <ia...@cnet.com>.
On 08 Aug 2001 08:29:13 -0700, Ryan Bloom wrote:
>
> > > > 2) Unix: The threaded MPM might take longer than expected to restart or
> > > > shutdown on very-low-traffic (near idle) servers. This is due to
> > > > the
> > >
> > > I dislike this. It's the "near idle" that bothers me. We rely on the
> > > OS to never starve a thread to get any restart or shutdown. The
> > > server could easily be getting hit moderately, and not shutdown
> > > correctly. I don't want to see us down-play the problem, and have
> > > people think that the server is working when it isn't. There is a
> > > real chance that an admin can do a graceful restart, and still be
> > > serving requests off the old config for a very long time.
> >
> > How about "servers getting very few requests" instead of
> > "very-low-traffic (near idle) servers"?
> >
> > I threw the "near idle" bit in there to emphasize that (a) it effects
> > people running test servers that have nobody connecting to them at all,
> > and (b) that our definition here of "very low traffic" is not 100 conn/sec
> > as opposed to 1000 conn/sec, but more on the order of just a handful of
> > conn/sec, since 100 conn/sec might be seen as very low traffic by some
> > administrators. ;-)
>
> a) is just not true. This doesn't affect only test servers, it affects anybody who
> isn't seeing a log of hits at a time. And we haven't got a clue what the threshold
> is. Again, we rely on the OS to make sure that every single thread gets
> the it's share of the connections.
>
> The real issue is that this MPM doesn't do what the admin expects. Admins
> expect, that if you issue a graceful restart, then any requests that are in the
> middle of being served will be the last requests served with the current
> config. The code doesn't do that. Today's code continues to serve off of the
> current config until EVERY thread has had a chance to serve one request. It
> doesn't matter if that takes 1 second or 1000 seconds. We advertise graceful
> restart as doing one thing, and it does something else.
>
> > As for the shutdown issue... is it really an issue? Generally speaking,
> > when I do an apachectl graceful, I can see the issue of which you speak,
> > but when I do an apachectl shutdown, it just goes. Maybe I'm missing it.
> > <shrug>
>
> As far as the threaded child process is concerned, a graceful restart should
> be the exact same thing as a graceful shutdown. If they aren't, then there is
> a bigger problem with this code than we know about.
how about a line like:
"gracefull restarts are currently not working properly on the threaded
MPM. The work-around at the moment is to do a full restart instead of a
graceful one."
That way admins have a workaround if they want to use threaded, and have
been warned as well.
..Ian
>
> Ryan
> _____________________________________________________________________________
> Ryan Bloom rbb@apache.org
> Covalent Technologies rbb@covalent.net
> -----------------------------------------------------------------------------
--
Ian Holsman IanH@cnet.com
Performance Measurement & Analysis
CNET Networks - (415) 364-8608
Re: graceful restart Re: Currently known issues with 2.0.23
Posted by Greg Ames <gr...@remulak.net>.
Brian Pane wrote:
> Yep. And I think the 'server not getting many hits' case should be expected
> even on high-traffic sites. A cautious operations team might reasonably do
> something like this when making config changes:
> - Configure the load balancer to stop sending new connections to one
> of n web servers
> - Install a new httpd.conf on that server
> - Graceful restart of apache (to give any open connections a chance to
> complete)
> - Verify that the server handles requests as expected
> - Configure the load balancer to start sending traffic to that server
> again
Not a problem, sir. AFAIK, threaded will handle this scenario just
fine.
Greg
graceful restart Re: Currently known issues with 2.0.23
Posted by Brian Pane <bp...@pacbell.net>.
Ryan Bloom wrote:
>>>>2) Unix: The threaded MPM might take longer than expected to restart or
>>>> shutdown on very-low-traffic (near idle) servers. This is due to
>>>>the
>>>>
>>>I dislike this. It's the "near idle" that bothers me. We rely on the
>>>OS to never starve a thread to get any restart or shutdown. The
>>>server could easily be getting hit moderately, and not shutdown
>>>correctly. I don't want to see us down-play the problem, and have
>>>people think that the server is working when it isn't. There is a
>>>real chance that an admin can do a graceful restart, and still be
>>>serving requests off the old config for a very long time.
>>>
>>How about "servers getting very few requests" instead of
>>"very-low-traffic (near idle) servers"?
>>
>>I threw the "near idle" bit in there to emphasize that (a) it effects
>>people running test servers that have nobody connecting to them at all,
>>and (b) that our definition here of "very low traffic" is not 100 conn/sec
>>as opposed to 1000 conn/sec, but more on the order of just a handful of
>>conn/sec, since 100 conn/sec might be seen as very low traffic by some
>>administrators. ;-)
>>
>
>a) is just not true. This doesn't affect only test servers, it affects anybody who
>isn't seeing a log of hits at a time. And we haven't got a clue what the threshold
>is. Again, we rely on the OS to make sure that every single thread gets
>the it's share of the connections.
>
Yep. And I think the 'server not getting many hits' case should be expected
even on high-traffic sites. A cautious operations team might reasonably do
something like this when making config changes:
- Configure the load balancer to stop sending new connections to one
of n web servers
- Install a new httpd.conf on that server
- Graceful restart of apache (to give any open connections a chance to
complete)
- Verify that the server handles requests as expected
- Configure the load balancer to start sending traffic to that server
again
--Brian
Re: Currently known issues with 2.0.23
Posted by Ryan Bloom <rb...@covalent.net>.
> > > As for the shutdown issue... is it really an issue? Generally
> > > speaking, when I do an apachectl graceful, I can see the issue of which
> > > you speak, but when I do an apachectl shutdown, it just goes. Maybe
> > > I'm missing it. <shrug>
>
> s/apachectl shutdown/apachectl stop/
>
> > As far as the threaded child process is concerned, a graceful restart
> > should be the exact same thing as a graceful shutdown. If they
> > aren't, then there is a bigger problem with this code than we know
> > about.
>
> Graceful shutdown? There is such a thing?
I always thought so, but I just checked 1.3, and SIGTERM does a graceless shutdown.
Hmmmmmm, I must be remembering something else. That is why you aren't seeing the
shutdown problem though, we don't support graceful shutdowns. :-)
Ryan
_____________________________________________________________________________
Ryan Bloom rbb@apache.org
Covalent Technologies rbb@covalent.net
-----------------------------------------------------------------------------
Re: Currently known issues with 2.0.23
Posted by Cliff Woolley <cl...@yahoo.com>.
On Wed, 8 Aug 2001, Ryan Bloom wrote:
> a) is just not true. This doesn't affect only test servers, it
> affects anybody who isn't seeing a log of hits at a time. And we
> haven't got a clue what the threshold is. Again, we rely on the OS to
> make sure that every single thread gets the it's share of the
> connections.
I didn't mean _only_ (a), I meant _also_ (a). It's true that we don't
know what the threshold is.
> has had a chance to serve one request. It doesn't matter if that
> takes 1 second or 1000 seconds. We advertise graceful restart as
> doing one thing, and it does something else.
Clearly.
> > As for the shutdown issue... is it really an issue? Generally speaking,
> > when I do an apachectl graceful, I can see the issue of which you speak,
> > but when I do an apachectl shutdown, it just goes. Maybe I'm missing it.
> > <shrug>
s/apachectl shutdown/apachectl stop/
> As far as the threaded child process is concerned, a graceful restart
> should be the exact same thing as a graceful shutdown. If they
> aren't, then there is a bigger problem with this code than we know
> about.
Graceful shutdown? There is such a thing?
--Cliff
--------------------------------------------------------------
Cliff Woolley
cliffwoolley@yahoo.com
Charlottesville, VA
Re: Currently known issues with 2.0.23
Posted by Ryan Bloom <rb...@covalent.net>.
> > > 2) Unix: The threaded MPM might take longer than expected to restart or
> > > shutdown on very-low-traffic (near idle) servers. This is due to
> > > the
> >
> > I dislike this. It's the "near idle" that bothers me. We rely on the
> > OS to never starve a thread to get any restart or shutdown. The
> > server could easily be getting hit moderately, and not shutdown
> > correctly. I don't want to see us down-play the problem, and have
> > people think that the server is working when it isn't. There is a
> > real chance that an admin can do a graceful restart, and still be
> > serving requests off the old config for a very long time.
>
> How about "servers getting very few requests" instead of
> "very-low-traffic (near idle) servers"?
>
> I threw the "near idle" bit in there to emphasize that (a) it effects
> people running test servers that have nobody connecting to them at all,
> and (b) that our definition here of "very low traffic" is not 100 conn/sec
> as opposed to 1000 conn/sec, but more on the order of just a handful of
> conn/sec, since 100 conn/sec might be seen as very low traffic by some
> administrators. ;-)
a) is just not true. This doesn't affect only test servers, it affects anybody who
isn't seeing a log of hits at a time. And we haven't got a clue what the threshold
is. Again, we rely on the OS to make sure that every single thread gets
the it's share of the connections.
The real issue is that this MPM doesn't do what the admin expects. Admins
expect, that if you issue a graceful restart, then any requests that are in the
middle of being served will be the last requests served with the current
config. The code doesn't do that. Today's code continues to serve off of the
current config until EVERY thread has had a chance to serve one request. It
doesn't matter if that takes 1 second or 1000 seconds. We advertise graceful
restart as doing one thing, and it does something else.
> As for the shutdown issue... is it really an issue? Generally speaking,
> when I do an apachectl graceful, I can see the issue of which you speak,
> but when I do an apachectl shutdown, it just goes. Maybe I'm missing it.
> <shrug>
As far as the threaded child process is concerned, a graceful restart should
be the exact same thing as a graceful shutdown. If they aren't, then there is
a bigger problem with this code than we know about.
Ryan
_____________________________________________________________________________
Ryan Bloom rbb@apache.org
Covalent Technologies rbb@covalent.net
-----------------------------------------------------------------------------
Re: Currently known issues with 2.0.23
Posted by Greg Ames <gr...@remulak.net>.
Cliff Woolley wrote:
> Also, as I mentioned last night, ungraceful for threaded seems to be
> broken.
Thanks...I'll take a look.
Greg
Re: Currently known issues with 2.0.23
Posted by Cliff Woolley <cl...@yahoo.com>.
On Wed, 8 Aug 2001, Ryan Bloom wrote:
> On Tuesday 07 August 2001 23:53, Cliff Woolley wrote:
> > --------------------------------------------------------------
> > Known issues with 2.0.23:
> >
> > 1) Win9x, WinME, and Netware do not yet work.
> > 2) Unix: The threaded MPM might take longer than expected to restart or
> > shutdown on very-low-traffic (near idle) servers. This is due to the
>
> I dislike this. It's the "near idle" that bothers me. We rely on the
> OS to never starve a thread to get any restart or shutdown. The
> server could easily be getting hit moderately, and not shutdown
> correctly. I don't want to see us down-play the problem, and have
> people think that the server is working when it isn't. There is a
> real chance that an admin can do a graceful restart, and still be
> serving requests off the old config for a very long time.
How about "servers getting very few requests" instead of
"very-low-traffic (near idle) servers"?
I threw the "near idle" bit in there to emphasize that (a) it effects
people running test servers that have nobody connecting to them at all,
and (b) that our definition here of "very low traffic" is not 100 conn/sec
as opposed to 1000 conn/sec, but more on the order of just a handful of
conn/sec, since 100 conn/sec might be seen as very low traffic by some
administrators. ;-)
As per Greg's suggestion, I'll also try to split up that paragraph into
two to make it easier to read.
As for the shutdown issue... is it really an issue? Generally speaking,
when I do an apachectl graceful, I can see the issue of which you speak,
but when I do an apachectl shutdown, it just goes. Maybe I'm missing it.
<shrug>
Also, as I mentioned last night, ungraceful for threaded seems to be
broken.
Thanks,
Cliff
--------------------------------------------------------------
Cliff Woolley
cliffwoolley@yahoo.com
Charlottesville, VA
Re: Currently known issues with 2.0.23 (very few! :)
Posted by Ryan Bloom <rb...@covalent.net>.
On Tuesday 07 August 2001 23:53, Cliff Woolley wrote:
> --------------------------------------------------------------
> Known issues with 2.0.23:
>
> 1) Win9x, WinME, and Netware do not yet work.
> 2) Unix: The threaded MPM might take longer than expected to restart or
> shutdown on very-low-traffic (near idle) servers. This is due to the
I dislike this. It's the "near idle" that bothers me. We rely on the OS to never
starve a thread to get any restart or shutdown. The server could easily be getting
hit moderately, and not shutdown correctly. I don't want to see us down-play the
problem, and have people think that the server is working when it isn't. There is a
real chance that an admin can do a graceful restart, and still be serving requests
off the old config for a very long time.
Ryan
_____________________________________________________________________________
Ryan Bloom rbb@apache.org
Covalent Technologies rbb@covalent.net
-----------------------------------------------------------------------------
Re: Currently known issues with 2.0.23
Posted by Greg Ames <gr...@remulak.net>.
Greg Ames wrote:
> Program received signal SIGSEGV, Segmentation fault.
> apr_pool_clear (a=0x0) at apr_pools.c:869
> 869 while (a->sub_pools) {
> (gdb) bt
> #0 apr_pool_clear (a=0x0) at apr_pools.c:869
> #1 0x08090320 in apr_pool_destroy (a=0x0) at apr_pools.c:920
> #2 0x4027d2ff in cgid_maint (reason=0, data=0x811a724, status=15)
> at mod_cgid.c:238
> #3 0x0808dd69 in apr_proc_other_child_check () at otherchild.c:208
> #4 0x0806ccfc in ap_reclaim_child_processes (terminate=1) at
> mpm_common.c:175
> #5 0x08064553 in ap_mpm_run (_pconf=0x80c443c, plog=0x80e453c,
> s=0x80c4984)
> at threaded.c:1329
> #6 0x08068e50 in main (argc=1, argv=0xbffff734) at main.c:427
> #7 0x4013f0de in __libc_start_main () from /lib/libc.so.6
>
> I've taken mod_cgid out of my build for now (didn't realize I had it
> actually) and see what happens.
OK, graceless restarts are working again in threaded with mod_cgid out
of the picture.
Greg
Re: Currently known issues with 2.0.23
Posted by Greg Ames <gr...@remulak.net>.
Greg Ames wrote:
>
> Cliff Woolley wrote:
> >
> > It looks like there might be a problem with _un_graceful restarts on
> > threaded, namely that the whole server just vaporizes.
> After adding many ap_log_errors, looks like things go normally until we
> hit the code in ap_mpm_run responsible for graceless restart. Then it
> sure looks like the parent catches a SIGTERM that it intended to send
> the children in ap_start_shutdown (the normal SIGTERM handler).
confirmed w/gdb:
1310 wake_up_and_die();
(gdb) n
1312 if (is_graceful) {
(gdb) p is_graceful
$3 = 0
(gdb) n
1326 if (unixd_killpg(getpgrp(), SIGTERM) < 0) {
(gdb) s
0x4000d500 in _dl_runtime_resolve () at dl-runtime.c:203
203 dl-runtime.c: No such file or directory.
in dl-runtime.c
(gdb) n
Program received signal SIGTERM, Terminated.
0x4014fd11 in kill () from /lib/libc.so.6
...but then this cleanup stuff for mod_cgid seg faults
(gdb) finish
Run till exit from #0 0x4014fb46 in killpg () from /lib/libc.so.6
0x08064518 in ap_mpm_run (_pconf=0x80c443c, plog=0x80e453c, s=0x80c4984)
at threaded.c:1326
1326 if (unixd_killpg(getpgrp(), SIGTERM) < 0) {
(gdb) n
1329 ap_reclaim_child_processes(1); /* Start with
SIGTERM
(gdb) p shutdown_pending
$4 = 1 <=== gets set in ap_start_shutdown, wasn't on
before
(gdb) n
Program received signal SIGSEGV, Segmentation fault.
apr_pool_clear (a=0x0) at apr_pools.c:869
869 while (a->sub_pools) {
(gdb) bt
#0 apr_pool_clear (a=0x0) at apr_pools.c:869
#1 0x08090320 in apr_pool_destroy (a=0x0) at apr_pools.c:920
#2 0x4027d2ff in cgid_maint (reason=0, data=0x811a724, status=15)
at mod_cgid.c:238
#3 0x0808dd69 in apr_proc_other_child_check () at otherchild.c:208
#4 0x0806ccfc in ap_reclaim_child_processes (terminate=1) at
mpm_common.c:175
#5 0x08064553 in ap_mpm_run (_pconf=0x80c443c, plog=0x80e453c,
s=0x80c4984)
at threaded.c:1329
#6 0x08068e50 in main (argc=1, argv=0xbffff734) at main.c:427
#7 0x4013f0de in __libc_start_main () from /lib/libc.so.6
I've taken mod_cgid out of my build for now (didn't realize I had it
actually) and see what happens. Catching the SIGTERM in the parent
can't be good though.
Greg
Re: Currently known issues with 2.0.23
Posted by Greg Ames <gr...@remulak.net>.
Cliff Woolley wrote:
>
> It looks like there might be a problem with _un_graceful restarts on
> threaded, namely that the whole server just vaporizes. It doesn't do a
> clean shutdown because the pidfile is left behind, but it's gone
> nevertheless. I've yet to find evidence of a segfault happening, but that
> remains a possibility. I just checked and it works fine on prefork (as
> expected). Will investigate further tomorrow.
You're right - I get exactly the same sympton on "apachectl restart"
now. It sure isn't hanging - it vanishes without a trace. hmmmm, I've
been the only one breaking^H^H^H^H^H^H^H^H putting good stuff into
threaded lately, so I must have done it somehow.
After adding many ap_log_errors, looks like things go normally until we
hit the code in ap_mpm_run responsible for graceless restart. Then it
sure looks like the parent catches a SIGTERM that it intended to send
the children in ap_start_shutdown (the normal SIGTERM handler).
I'm going to try strace on the parent (thanks Jeff!), and/or backing
out/further scrutinizing my last patch to threaded.
Greg
Re: Currently known issues with 2.0.23
Posted by Jeff Trawick <tr...@attglobal.net>.
Greg Ames <gr...@remulak.net> writes:
> Jeff Trawick wrote:
> >
> > Cliff Woolley <cl...@yahoo.com> writes:
> >
> > > It looks like there might be a problem with _un_graceful restarts on
> > > threaded, namely that the whole server just vaporizes. It doesn't do a
> > > clean shutdown because the pidfile is left behind, but it's gone
> > > nevertheless. I've yet to find evidence of a segfault happening, but that
> >
> > Unlike the child/server processes, the parent process can't rely on
> > the parent (itself) to write the log message for the segfault.
> >
> > Maybe sig_coredump() needs to call ap_log_error() iff called in the
> > parent.
>
> Something like that would clearly help.
>
> But, just to be difficult, what if ap_log_error() seg faults? We don't
> want recursive seg faults. Maybe some kind of "I'm trying to log a
> parent seg fault" footprint in sig_coredump to stop the recursion?
AFAIK, it shouldn't be a problem. Do the ap_log_error() call after we
remove our handler ("apr_signal(sig, SIG_DFL)"), and of course ?GOVRR
from mainline as well as from the handler while testing.
--
Jeff Trawick | trawick@attglobal.net | PGP public key at web site:
http://www.geocities.com/SiliconValley/Park/9289/
Born in Roswell... married an alien...
Re: Currently known issues with 2.0.23
Posted by Greg Ames <gr...@remulak.net>.
Jeff Trawick wrote:
>
> Cliff Woolley <cl...@yahoo.com> writes:
>
> > It looks like there might be a problem with _un_graceful restarts on
> > threaded, namely that the whole server just vaporizes. It doesn't do a
> > clean shutdown because the pidfile is left behind, but it's gone
> > nevertheless. I've yet to find evidence of a segfault happening, but that
>
> Unlike the child/server processes, the parent process can't rely on
> the parent (itself) to write the log message for the segfault.
>
> Maybe sig_coredump() needs to call ap_log_error() iff called in the
> parent.
Something like that would clearly help.
But, just to be difficult, what if ap_log_error() seg faults? We don't
want recursive seg faults. Maybe some kind of "I'm trying to log a
parent seg fault" footprint in sig_coredump to stop the recursion?
hmmmm, I'll play with it
Greg
Re: Currently known issues with 2.0.23
Posted by Jeff Trawick <tr...@attglobal.net>.
Cliff Woolley <cl...@yahoo.com> writes:
> It looks like there might be a problem with _un_graceful restarts on
> threaded, namely that the whole server just vaporizes. It doesn't do a
> clean shutdown because the pidfile is left behind, but it's gone
> nevertheless. I've yet to find evidence of a segfault happening, but that
Unlike the child/server processes, the parent process can't rely on
the parent (itself) to write the log message for the segfault.
Maybe sig_coredump() needs to call ap_log_error() iff called in the
parent.
--
Jeff Trawick | trawick@attglobal.net | PGP public key at web site:
http://www.geocities.com/SiliconValley/Park/9289/
Born in Roswell... married an alien...
Re: Currently known issues with 2.0.23
Posted by Cliff Woolley <cl...@yahoo.com>.
On 9 Aug 2001, Jeff Trawick wrote:
> Greg Ames and I have been playing with this. See a patch I just
> committed to mod_cgid. [Apparently] since 8/6/2001 when dougm fixed a
> leak problem, ungraceful restart was segfaulting in the parent
> process. If you retag this as 2.0.23 just wipe the CHANGES entry
> since the problem appeared and went away in 2.0.23.
I tested it and it works great for me, so I went ahead and pulled it in to
the 2.0.23 tag. Thanks, guys.
PS: I'd originally pulled the CHANGES entry along with me, but then I saw
your note here so I went back and stripped out the CHANGES entry I'd
committed to the 2.0.23-branch and the one you committed to 2.0.24-dev.
--Cliff
--------------------------------------------------------------
Cliff Woolley
cliffwoolley@yahoo.com
Charlottesville, VA
Re: Currently known issues with 2.0.23
Posted by Jeff Trawick <tr...@attglobal.net>.
Cliff Woolley <cl...@yahoo.com> writes:
> It looks like there might be a problem with _un_graceful restarts on
> threaded, namely that the whole server just vaporizes. It doesn't do a
> clean shutdown because the pidfile is left behind, but it's gone
> nevertheless. I've yet to find evidence of a segfault happening, but that
> remains a possibility. I just checked and it works fine on prefork (as
> expected). Will investigate further tomorrow.
Greg Ames and I have been playing with this. See a patch I just
committed to mod_cgid. [Apparently] since 8/6/2001 when dougm fixed a
leak problem, ungraceful restart was segfaulting in the parent
process. If you retag this as 2.0.23 just wipe the CHANGES entry
since the problem appeared and went away in 2.0.23.
--
Jeff Trawick | trawick@attglobal.net | PGP public key at web site:
http://www.geocities.com/SiliconValley/Park/9289/
Born in Roswell... married an alien...
Re: Currently known issues with 2.0.23
Posted by Cliff Woolley <cl...@yahoo.com>.
It looks like there might be a problem with _un_graceful restarts on
threaded, namely that the whole server just vaporizes. It doesn't do a
clean shutdown because the pidfile is left behind, but it's gone
nevertheless. I've yet to find evidence of a segfault happening, but that
remains a possibility. I just checked and it works fine on prefork (as
expected). Will investigate further tomorrow.
For now, it's 3:45am in my part of the world... I need to get to bed. =-)
--Cliff
--------------------------------------------------------------
Cliff Woolley
cliffwoolley@yahoo.com
Charlottesville, VA