You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Rainer Jung <ra...@kippdata.de> on 2009/12/02 21:09:13 UTC
Crash in scoreboard for 2.3.4 after restart (register_hooks)
Platform: Solaris 8 (sic!)
MPM: worker dynamically loaded
APR etc: Bundled
PCRE: 7.8
During testing of 2.3.4 I noticed crashes after restart.
I did a build with lots of modules, especially including mod_logio. The
scoreboard uses in ap_increment_counts() the optional function
ap_logio_get_last_bytes() from mod_logio if available.
In my case after restart the memory location of mod_logio and the
address of the optional function changed, but the scoreboard still tries
to call to the original address retrieved after the initial start.
I don't know about the full implementation of the optional functions,
but it seems either
APR_REGISTER_OPTIONAL_FN(ap_logio_get_last_bytes);
in register_hooks in mod_logio needs to run after restarts too, or there
is a problem resulting in an unwanted change of load order of the
modules during restart. I did not edit the config files between start
and restart.
The problem happens with normal restarts and graceful restarts.
Wild guess: it might have to do with dynamic MPM loading.
The diff between pmap of the main process after initial start and after
restart is:
--- pmap_22741_a 2009-12-02 21:02:46.000000000 +0100
+++ pmap_22741_c 2009-12-02 21:03:44.000000000 +0100
@@ -12,19 +12,19 @@
FDF0C000 8K read/write/exec [ anon ]
FDF10000 8K read/write/exec [ anon ]
FE00E000 8K read/write/exec [ anon ]
-FE370000 24K read/exec
/shared/build/autobuild/workdirs/20091202a-httpd-2.3.4/run/modules/mod_status.so
-FE384000 8K read/write/exec
/shared/build/autobuild/workdirs/20091202a-httpd-2.3.4/run/modules/mod_status.so
-FE390000 8K read/exec
/shared/build/autobuild/workdirs/20091202a-httpd-2.3.4/run/modules/mod_unixd.so
-FE3A0000 8K read/write/exec
/shared/build/autobuild/workdirs/20091202a-httpd-2.3.4/run/modules/mod_unixd.so
-FE3B0000 40K read/exec
/shared/build/autobuild/workdirs/20091202a-httpd-2.3.4/run/modules/mod_mpm_worker.so
-FE3C8000 8K read/write/exec
/shared/build/autobuild/workdirs/20091202a-httpd-2.3.4/run/modules/mod_mpm_worker.so
-FE3D0000 8K read/exec
/shared/build/autobuild/workdirs/20091202a-httpd-2.3.4/run/modules/mod_logio.so
-FE3E0000 8K read/write/exec
/shared/build/autobuild/workdirs/20091202a-httpd-2.3.4/run/modules/mod_logio.so
FE3F0000 8K read/write/exec [ anon ]
+FEA80000 496K read/write/shared [ anon ]
FEB20000 240K read/write/shared [ anon ]
FEB60000 8K read/exec
/shared/build/autobuild/workdirs/20091202a-httpd-2.3.4/run/lib/apr-util-1/apr_ldap-1.so
FEB70000 8K read/write/exec
/shared/build/autobuild/workdirs/20091202a-httpd-2.3.4/run/lib/apr-util-1/apr_ldap-1.so
-FEB80000 496K read/write/shared [ anon ]
+FEBA0000 24K read/exec
/shared/build/autobuild/workdirs/20091202a-httpd-2.3.4/run/modules/mod_status.so
+FEBB4000 8K read/write/exec
/shared/build/autobuild/workdirs/20091202a-httpd-2.3.4/run/modules/mod_status.so
+FEBC0000 8K read/exec
/shared/build/autobuild/workdirs/20091202a-httpd-2.3.4/run/modules/mod_unixd.so
+FEBD0000 8K read/write/exec
/shared/build/autobuild/workdirs/20091202a-httpd-2.3.4/run/modules/mod_unixd.so
+FEBE0000 40K read/exec
/shared/build/autobuild/workdirs/20091202a-httpd-2.3.4/run/modules/mod_mpm_worker.so
+FEBF8000 8K read/write/exec
/shared/build/autobuild/workdirs/20091202a-httpd-2.3.4/run/modules/mod_mpm_worker.so
+FEC00000 8K read/exec
/shared/build/autobuild/workdirs/20091202a-httpd-2.3.4/run/modules/mod_logio.so
+FEC10000 8K read/write/exec
/shared/build/autobuild/workdirs/20091202a-httpd-2.3.4/run/modules/mod_logio.so
FEC20000 24K read/exec
/shared/build/autobuild/workdirs/20091202a-httpd-2.3.4/run/modules/mod_log_config.so
FEC34000 8K read/write/exec
/shared/build/autobuild/workdirs/20091202a-httpd-2.3.4/run/modules/mod_log_config.so
FEC40000 48K read/exec
/shared/build/autobuild/workdirs/20091202a-httpd-2.3.4/run/modules/mod_ldap.so
@@ -38,14 +38,14 @@
FECD0000 8K read/exec
/shared/build/autobuild/workdirs/20091202a-httpd-2.3.4/run/modules/mod_auth_basic.so
FECE0000 8K read/write/exec
/shared/build/autobuild/workdirs/20091202a-httpd-2.3.4/run/modules/mod_auth_basic.so
FECF0000 8K read/write/exec [ anon ]
-FED00000 8K read/exec
/shared/build/autobuild/workdirs/20091202a-httpd-2.3.4/run/modules/mod_access_compat.so
-FED10000 8K read/write/exec
/shared/build/autobuild/workdirs/20091202a-httpd-2.3.4/run/modules/mod_access_compat.so
+FED00000 32K read/exec
/shared/build/autobuild/workdirs/20091202a-httpd-2.3.4/run/modules/mod_authnz_ldap.so
+FED16000 8K read/write/exec
/shared/build/autobuild/workdirs/20091202a-httpd-2.3.4/run/modules/mod_authnz_ldap.so
FED20000 216K read/exec /usr/lib/libresolv.so.2
FED66000 24K read/write/exec /usr/lib/libresolv.so.2
FED70000 144K read/exec /usr/lib/libldap.so.4
FEDA4000 16K read/write/exec /usr/lib/libldap.so.4
-FEDB0000 32K read/exec
/shared/build/autobuild/workdirs/20091202a-httpd-2.3.4/run/modules/mod_authnz_ldap.so
-FEDC6000 8K read/write/exec
/shared/build/autobuild/workdirs/20091202a-httpd-2.3.4/run/modules/mod_authnz_ldap.so
+FEDB0000 8K read/exec
/shared/build/autobuild/workdirs/20091202a-httpd-2.3.4/run/modules/mod_access_compat.so
+FEDC0000 8K read/write/exec
/shared/build/autobuild/workdirs/20091202a-httpd-2.3.4/run/modules/mod_access_compat.so
FEDD0000 16K read/exec
/shared/build/autobuild/workdirs/20091202a-httpd-2.3.4/run/modules/mod_authz_core.so
FEDE2000 8K read/write/exec
/shared/build/autobuild/workdirs/20091202a-httpd-2.3.4/run/modules/mod_authz_core.so
FEDF0000 8K read/exec
/shared/build/autobuild/workdirs/20091202a-httpd-2.3.4/run/modules/mod_authz_dbd.so
@@ -72,7 +72,7 @@
FEF40000 8K read/write/exec
/shared/build/autobuild/workdirs/20091202a-httpd-2.3.4/run/modules/mod_authn_dbm.so
FEF50000 8K read/exec
/shared/build/autobuild/workdirs/20091202a-httpd-2.3.4/run/modules/mod_authn_file.so
FEF60000 8K read/write/exec
/shared/build/autobuild/workdirs/20091202a-httpd-2.3.4/run/modules/mod_authn_file.so
-FEF70000 8K read/write/exec/shared [ shmid=0xc02 ]
+FEF70000 8K read/write/exec/shared [ shmid=0xd02 ]
FEF84000 8K read/write/exec [ anon ]
FEF90000 8K read/write/exec [ anon ]
FEFA0000 112K read/exec /usr/lib/libthread.so.1
The address used for pfn_ap_logio_get_last_bytes is '0xfe3d0ad4' after
start and after restart (which is wrong).
Any ideas?
I can easily reproduce and debug further once there is an idea which
code to look at.
Regards,
Rainer
Re: Crash in scoreboard for 2.3.4 after restart (register_hooks)
Posted by Ruediger Pluem <rp...@apache.org>.
On 12/02/2009 09:53 PM, Rainer Jung wrote:
>
> Yes, that fixes it, makes a lot of sense.
>
> What's interesting is, that the function address changes during the
> first restart, but not any more when doing further restarts. So the
> initial start ends up with a different load order than all of the
> following restarts. Something to keep in mind at least in case it's new
> in 2.3 - and about 60 other places to check which use optional
> functions, whether the function pointers are correctly reinitialized.
>
> Thanks!
>
> Will you commit?
Done as r886324.
Regards
Rüdiger
Re: Crash in scoreboard for 2.3.4 after restart (register_hooks)
Posted by Rainer Jung <ra...@kippdata.de>.
Hi Rüdiger,
On 02.12.2009 21:33, Ruediger Pluem wrote:
>
>
> On 12/02/2009 09:09 PM, Rainer Jung wrote:
>> Platform: Solaris 8 (sic!)
>> MPM: worker dynamically loaded
>> APR etc: Bundled
>> PCRE: 7.8
>>
>> During testing of 2.3.4 I noticed crashes after restart.
>>
>> I did a build with lots of modules, especially including mod_logio. The
>> scoreboard uses in ap_increment_counts() the optional function
>> ap_logio_get_last_bytes() from mod_logio if available.
>>
>> In my case after restart the memory location of mod_logio and the
>> address of the optional function changed, but the scoreboard still tries
>> to call to the original address retrieved after the initial start.
>>
>> I don't know about the full implementation of the optional functions,
>> but it seems either
>>
>> APR_REGISTER_OPTIONAL_FN(ap_logio_get_last_bytes);
>>
>> in register_hooks in mod_logio needs to run after restarts too, or there
>> is a problem resulting in an unwanted change of load order of the
>> modules during restart. I did not edit the config files between start
>> and restart.
>>
>> The problem happens with normal restarts and graceful restarts.
>>
>> Wild guess: it might have to do with dynamic MPM loading.
>
> I am not so sure. I guess the problem is that we call
>
> APR_RETRIEVE_OPTIONAL_FN(ap_logio_get_last_bytes);
>
> only in ap_calc_scoreboard_size which is IMHO not called at restarts.
> Does the following totally untested patch fix your issue?
>
> Index: scoreboard.c
> ===================================================================
> --- scoreboard.c (Revision 886294)
> +++ scoreboard.c (Arbeitskopie)
> @@ -284,6 +284,8 @@
> apr_status_t rv;
> #endif
>
> + pfn_ap_logio_get_last_bytes = APR_RETRIEVE_OPTIONAL_FN(ap_logio_get_last_bytes);
> +
> if (ap_scoreboard_image) {
> running_gen = ap_scoreboard_image->global->running_generation;
> ap_scoreboard_image->global->restart_time = apr_time_now();
>
>
Yes, that fixes it, makes a lot of sense.
What's interesting is, that the function address changes during the
first restart, but not any more when doing further restarts. So the
initial start ends up with a different load order than all of the
following restarts. Something to keep in mind at least in case it's new
in 2.3 - and about 60 other places to check which use optional
functions, whether the function pointers are correctly reinitialized.
Thanks!
Will you commit?
Rainer
Re: Crash in scoreboard for 2.3.4 after restart (register_hooks)
Posted by Jeff Trawick <tr...@gmail.com>.
On Wed, Dec 2, 2009 at 3:33 PM, Ruediger Pluem <rp...@apache.org> wrote:
>
>
> On 12/02/2009 09:09 PM, Rainer Jung wrote:
>> Platform: Solaris 8 (sic!)
>> MPM: worker dynamically loaded
>> APR etc: Bundled
>> PCRE: 7.8
>>
>> During testing of 2.3.4 I noticed crashes after restart.
>>
>> I did a build with lots of modules, especially including mod_logio. The
>> scoreboard uses in ap_increment_counts() the optional function
>> ap_logio_get_last_bytes() from mod_logio if available.
>>
>> In my case after restart the memory location of mod_logio and the
>> address of the optional function changed, but the scoreboard still tries
>> to call to the original address retrieved after the initial start.
>>
>> I don't know about the full implementation of the optional functions,
>> but it seems either
>>
>> APR_REGISTER_OPTIONAL_FN(ap_logio_get_last_bytes);
>>
>> in register_hooks in mod_logio needs to run after restarts too, or there
>> is a problem resulting in an unwanted change of load order of the
>> modules during restart. I did not edit the config files between start
>> and restart.
>>
>> The problem happens with normal restarts and graceful restarts.
>>
>> Wild guess: it might have to do with dynamic MPM loading.
>
> I am not so sure. I guess the problem is that we call
>
> APR_RETRIEVE_OPTIONAL_FN(ap_logio_get_last_bytes);
>
> only in ap_calc_scoreboard_size which is IMHO not called at restarts.
That was my guess. Which reminds me of this old challenge for a module I wrote:
* XXX
* our daemon must be created in the pre-mpm hook so that the scoreboard
* is available in the daemon process; the scoreboard isn't created until
* the core module's pre-mpm hook
* BUT: the pre-mpm hook is not called for graceful restart, so we can't
* start a new instance of the daemon then
* BUT: request processing from the daemon MUST use the new configuration,
* so we'd be busted for graceful restart
Re: Crash in scoreboard for 2.3.4 after restart (register_hooks)
Posted by Ruediger Pluem <rp...@apache.org>.
On 12/02/2009 09:09 PM, Rainer Jung wrote:
> Platform: Solaris 8 (sic!)
> MPM: worker dynamically loaded
> APR etc: Bundled
> PCRE: 7.8
>
> During testing of 2.3.4 I noticed crashes after restart.
>
> I did a build with lots of modules, especially including mod_logio. The
> scoreboard uses in ap_increment_counts() the optional function
> ap_logio_get_last_bytes() from mod_logio if available.
>
> In my case after restart the memory location of mod_logio and the
> address of the optional function changed, but the scoreboard still tries
> to call to the original address retrieved after the initial start.
>
> I don't know about the full implementation of the optional functions,
> but it seems either
>
> APR_REGISTER_OPTIONAL_FN(ap_logio_get_last_bytes);
>
> in register_hooks in mod_logio needs to run after restarts too, or there
> is a problem resulting in an unwanted change of load order of the
> modules during restart. I did not edit the config files between start
> and restart.
>
> The problem happens with normal restarts and graceful restarts.
>
> Wild guess: it might have to do with dynamic MPM loading.
I am not so sure. I guess the problem is that we call
APR_RETRIEVE_OPTIONAL_FN(ap_logio_get_last_bytes);
only in ap_calc_scoreboard_size which is IMHO not called at restarts.
Does the following totally untested patch fix your issue?
Index: scoreboard.c
===================================================================
--- scoreboard.c (Revision 886294)
+++ scoreboard.c (Arbeitskopie)
@@ -284,6 +284,8 @@
apr_status_t rv;
#endif
+ pfn_ap_logio_get_last_bytes = APR_RETRIEVE_OPTIONAL_FN(ap_logio_get_last_bytes);
+
if (ap_scoreboard_image) {
running_gen = ap_scoreboard_image->global->running_generation;
ap_scoreboard_image->global->restart_time = apr_time_now();
Regards
Rüdiger