You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Jeff Trawick <tr...@attglobal.net> on 2003/12/01 13:52:03 UTC
Re: Apache 1.3.28 SEGFAULTS and doesn't produce a core file
FWIW, it segfaults on a jsp request... I suppose that this is handled by a
third party module such as mod_jk? See the final snippet:
[pid 32119] read(11, "GET /messaging/businessObject.js"..., 4096) = 775
[pid 32119] rt_sigaction(SIGUSR1, {SIG_IGN}, {0x4002127c, [],
SA_INTERRUPT|0x4000000}, 8) = 0
[pid 32119] time(NULL) = 1069347777
[pid 32119] stat64("/etc/httpd/htdocs/email/messaging/businessObject.jsp",
{st_mode=S_IFREG|0644, st_size=12754, ...}) = 0
[pid 32119] getpid() = 32119
[pid 32119] getpid() = 32119
[pid 32119] --- SIGSEGV (Segmentation fault) ---
[pid 10610] wait4(-1, [WIFSIGNALED(s) && WTERMSIG(s) == SIGSEGV], WNOHANG,
Common issues with getting coredumps from Apache 1.3 on Linux 2.4 kernel:
1) prctl() call, resolved by mod_prctl or patch like that posted on the list
2) ulimit -c setting in the shell used to start Apache
3) CoredumpDirectory directive MUST BE SPECIFIED and must point to a directory
that the web server user id (e.g., "nobody") has write access to; additionally,
there must be plenty of free space in that directory; without the
CoredumpDirectory, the default directory (serverroot) will be used, and the web
server user id doesn't have write access there
Check #3 in particular. I don't think it was mentioned in the responses to
your post.
Another thing: To verify that the problem with core dump is not your
system/Apache configuration but instead something horrible that happens due to
the way the child process crashes, send SIGSEGV to some random child process*
and verify that the kernel is able to write a coredump and that the message
written to the Apache error log has "possible coredump in /some/dir" as part of
the "child pid XXX exit signal YYY" message
*obviously you may not want to do that if the server is in production :)
Re: Apache 1.3.28 SEGFAULTS and doesn't produce a core file
Posted by ma...@bellsouth.net.
On Mon, 1 Dec 2003, Jeff Trawick wrote:
> FWIW, it segfaults on a jsp request... I suppose that this is handled by a
> third party module such as mod_jk? See the final snippet:
>
We are using the BEA weblogic plugin to broker *.jsp to our application
server. We are using the latest QE'ed build of the plugin.
> [pid 32119] read(11, "GET /messaging/businessObject.js"..., 4096) = 775
> [pid 32119] rt_sigaction(SIGUSR1, {SIG_IGN}, {0x4002127c, [],
> SA_INTERRUPT|0x4000000}, 8) = 0
> [pid 32119] time(NULL) = 1069347777
> [pid 32119] stat64("/etc/httpd/htdocs/email/messaging/businessObject.jsp",
> {st_mode=S_IFREG|0644, st_size=12754, ...}) = 0
> [pid 32119] getpid() = 32119
> [pid 32119] getpid() = 32119
> [pid 32119] --- SIGSEGV (Segmentation fault) ---
> [pid 10610] wait4(-1, [WIFSIGNALED(s) && WTERMSIG(s) == SIGSEGV], WNOHANG,
We are also using auth_ldap to authenticate users, and authentication
looks to be occuring right before the programs SEGFAULTS. Since I cannot
associate the getpid()'s with a module, I cannot pinpoint which module is
causing Apache grief.
>
> Common issues with getting coredumps from Apache 1.3 on Linux 2.4 kernel:
>
> 1) prctl() call, resolved by mod_prctl or patch like that posted on the list
Even with the prtctl module Apache refuses to dump core. Once I start
Apache as the user "apache," it dumps core perfectly. It looks like
the Linux kernel refuses to let any program that changes it's uid
core. I wish this option could be disabled through "/proc" or sysctl
until I get a core to analyze.
> 2) ulimit -c setting in the shell used to start Apache
Our ulimits are cool.
> 3) CoredumpDirectory directive MUST BE SPECIFIED and must point to a directory
> that the web server user id (e.g., "nobody") has write access to; additionally,
> there must be plenty of free space in that directory; without the
> CoredumpDirectory, the default directory (serverroot) will be used, and the web
> server user id doesn't have write access there
Got all three of these items taken care of.
>
> Check #3 in particular. I don't think it was mentioned in the responses to
> your post.
>
> Another thing: To verify that the problem with core dump is not your
> system/Apache configuration but instead something horrible that happens due to
> the way the child process crashes, send SIGSEGV to some random child process*
> and verify that the kernel is able to write a coredump and that the message
> written to the Apache error log has "possible coredump in /some/dir" as part of
> the "child pid XXX exit signal YYY" message
>
> *obviously you may not want to do that if the server is in production :)
>
Thanks for your feedback. I am going to reconfigure my web servers to
start Apache as a non-root user, and run the web servers on port 8000.
Since we have a pair of load-balancers in front of the web servers, this
shoulnd't cause use too much grief.
Re: Apache 1.3.28 SEGFAULTS and doesn't produce a core file
Posted by Joe Orton <jo...@redhat.com>.
On Mon, Dec 01, 2003 at 07:52:03AM -0500, Jeff Trawick wrote:
> FWIW, it segfaults on a jsp request... I suppose that this is handled by a
> third party module such as mod_jk? See the final snippet:
>
> [pid 32119] read(11, "GET /messaging/businessObject.js"..., 4096) = 775
> [pid 32119] rt_sigaction(SIGUSR1, {SIG_IGN}, {0x4002127c, [],
> SA_INTERRUPT|0x4000000}, 8) = 0
> [pid 32119] time(NULL) = 1069347777
> [pid 32119] stat64("/etc/httpd/htdocs/email/messaging/businessObject.jsp",
> {st_mode=S_IFREG|0644, st_size=12754, ...}) = 0
> [pid 32119] getpid() = 32119
> [pid 32119] getpid() = 32119
> [pid 32119] --- SIGSEGV (Segmentation fault) ---
> [pid 10610] wait4(-1, [WIFSIGNALED(s) && WTERMSIG(s) == SIGSEGV], WNOHANG,
>
> Common issues with getting coredumps from Apache 1.3 on Linux 2.4 kernel:
>
> 1) prctl() call, resolved by mod_prctl or patch like that posted on the list
Unfortunately some of our kernels still have broken
prctl(PR_SET_DUMPABLE, 1) support after some over-aggressive ptrace
security fixes were added, the 7.3 kernel appears to be one.
joe
Re: Apache 1.3.28 SEGFAULTS and doesn't produce a core file
Posted by ma...@bellsouth.net.
On Mon, 1 Dec 2003, Jeff Trawick wrote:
> Jeff Trawick wrote:
>
> > FWIW, it segfaults on a jsp request... I suppose that this is handled
> > by a third party module such as mod_jk? See the final snippet:
>
> I was reminded by a little bird that you had mentioned in your original message
> that you were using the WebLogic plug-in. Maybe some hints here will help you
> get a core dump, but regardless: If you're using a commercially supported
> product and that product was likely doing something around the time of the
> crash (which seems likely), make sure you report it to them. They can at least
I have had a case open with BEA since the issue started. They don't think (based
on a large strace I sent in) that the SEGFAULTS are occuring within their module.
Once I startup apache as the user "apache," I should be able to get a stack trace
to see what's making Apache angry. I also plan to strip all modules but the following:
LoadModule config_log_module libexec/mod_log_config.so
LoadModule mime_module libexec/mod_mime.so
LoadModule dir_module libexec/mod_dir.so
LoadModule access_module libexec/mod_access.so
LoadModule auth_module libexec/mod_auth.so
ClearModuleList
AddModule mod_log_config.c
AddModule mod_mime.c
AddModule mod_dir.c
AddModule mod_access.c
AddModule mod_auth.c
AddModule mod_so.c
I will also need to add the weblogic and auth_ldap modules to the mix
> make sure you have their latest and greatest code, and if you're lucky even
> verify that there was some known issue that could explain the crashes. Another
I have the latest source build.
> general debug technique that applies here: If you can verify that a crash only
> occurs for requests handled by that plug-in, that is more information they can
> use as well.
Thanks for the feedback.
Re: Apache 1.3.28 SEGFAULTS and doesn't produce a core file
Posted by Jeff Trawick <tr...@attglobal.net>.
Jeff Trawick wrote:
> FWIW, it segfaults on a jsp request... I suppose that this is handled
> by a third party module such as mod_jk? See the final snippet:
I was reminded by a little bird that you had mentioned in your original message
that you were using the WebLogic plug-in. Maybe some hints here will help you
get a core dump, but regardless: If you're using a commercially supported
product and that product was likely doing something around the time of the
crash (which seems likely), make sure you report it to them. They can at least
make sure you have their latest and greatest code, and if you're lucky even
verify that there was some known issue that could explain the crashes. Another
general debug technique that applies here: If you can verify that a crash only
occurs for requests handled by that plug-in, that is more information they can
use as well.