You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Dirk-Willem van Gulik <di...@webweaving.org> on 1999/12/02 21:57:57 UTC
Solaris 2.7 and getting a clean coredump
Anyone have any hints as how to proceed... apache bombs out with a
bus/segv's quite often.. and I'd like a core.
As we have 200+ active processes and 50 req/second attaching a debuger to
PID's is not an option.. kind of difficult to guess which one :-)
Now I've set/tried
ulimit -c unlimited
cd to the core-directory before startup
CoreDumpDirectory ...
the coredump directory is rw for the world. There is
enough disk space.
A 'kill -SEGV' does not yield core's either, but a CGI like
#/usr/local/bin/perl
$|++;
print "Content-type: text/plain\n\n"
exec "ulimit -a; kill -SIGV $$";
produces one (of sh) just fine. Kind of suggesting that the inherited
child environment is fine.
Of course the log nicely log's the child SIGV-ing in both cases. Even
though the SEGV of apache causes no core.
Any hints. Same setup dumps fine on BSD.
Any and all suggestions appreciated !
Dw
Re: Solaris 2.7 and getting a clean coredump
Posted by Sam Talebbeik <sa...@ix.netcom.com>.
I am glad that the problem is resolved. Last that I checked, the issue
of setuid() was indeed discussed in Apache's debugging page. That
is why I pointed you to the page. Here is a cut and paste from that page
about setuid(). The "coreadm" utility is something that I was not
aware of. Thanks for pointing it out.
ST.
-----------------------------------------------------------------------------
Getting the server to dump core
Strangely enough, sometimes you actually want to force the server to crash
so that you can get a
look at some nutty behavior. Normally this can be done simply by using the
gcore command.
However, for security reasons, most Unix systems do not allow a setuid
process to dump core, since
the file contents might reveal something that is supposed to be protected
in memory.
Here is one way to get a core file from a setuid Apache httpd process on
Solaris, without knowing
which httpd child might be the one to die [note: it is probably easier to
use the MaxClients trick in the
first section above].
# for pid in `ps -eaf | fgrep httpd | cut -d' ' -f4`
do
truss -f -l -t\!all -S SIGSEGV -p $pid 2>&1 | egrep SIGSEGV &
done
The undocumented '-S' flag to truss will halt the process in place upon
receipt of a given signal
(SIGSEGV in this case). At this point you can use:
# gcore PID
and then look at the backtrace as discussed above for gdb.
-----------------------------------------------------------------------------
Dirk-Willem van Gulik wrote:
> On Thu, 2 Dec 1999, Sam Talebbeik wrote:
>
> > You will fine some very good debugging tips at Apache's
> > own developer web site. Try the following link:
> >
> > http://dev.apache.org/debugging.html
>
> But not the setuid() and coreadm information which actually solved the
> problem. I've commet an updated version to cvs.
>
> DW.
Re: Solaris 2.7 and getting a clean coredump
Posted by Dirk-Willem van Gulik <di...@webweaving.org>.
On Thu, 2 Dec 1999, Sam Talebbeik wrote:
> You will fine some very good debugging tips at Apache's
> own developer web site. Try the following link:
>
> http://dev.apache.org/debugging.html
But not the setuid() and coreadm information which actually solved the
problem. I've commet an updated version to cvs.
DW.
Re: Solaris 2.7 and getting a clean coredump
Posted by Sam Talebbeik <sa...@ix.netcom.com>.
You will fine some very good debugging tips at Apache's
own developer web site. Try the following link:
http://dev.apache.org/debugging.html
ST.
Eli Marmor wrote:
> Solaris processes can't core dump after calling setuid().
> Try to run it as non-root (with port number > 1023) and a dir
> which is writable by the uid. You can also run it with "User"
> root (httpd.conf), or copy /usr/bin/gcore to a temporary and
> secret place, change its mode to 4755, and call it explitcitly
> from Apache before crashing.
>
> In addition, don't forget to ensure that limit coredumpsize is
> unlimited, and the directory where core is going to be dumped
> is writable by the user.
>
> DON'T FORGET TO RESTORE EVERYTHING LATER! (httpd.conf, or the
> secret copy of gcore, etc.).
>
> Dirk-Willem van Gulik wrote:
> >
> > Anyone have any hints as how to proceed... apache bombs out with a
> > bus/segv's quite often.. and I'd like a core.
> >
> > As we have 200+ active processes and 50 req/second attaching a debuger to
> > PID's is not an option.. kind of difficult to guess which one :-)
> >
> > Now I've set/tried
> >
> > ulimit -c unlimited
> >
> > cd to the core-directory before startup
> >
> > CoreDumpDirectory ...
> >
> >
> > the coredump directory is rw for the world. There is
> > enough disk space.
> >
> > A 'kill -SEGV' does not yield core's either, but a CGI like
> >
> > #/usr/local/bin/perl
> > $|++;
> > print "Content-type: text/plain\n\n"
> > exec "ulimit -a; kill -SIGV $$";
> >
> > produces one (of sh) just fine. Kind of suggesting that the inherited
> > child environment is fine.
> >
> > Of course the log nicely log's the child SIGV-ing in both cases. Even
> > though the SEGV of apache causes no core.
> >
> > Any hints. Same setup dumps fine on BSD.
> >
> > Any and all suggestions appreciated !
> >
> > Dw
>
> --
> Eli Marmor
Re: Solaris 2.7 and getting a clean coredump
Posted by Eli Marmor <ma...@elmar.co.il>.
Solaris processes can't core dump after calling setuid().
Try to run it as non-root (with port number > 1023) and a dir
which is writable by the uid. You can also run it with "User"
root (httpd.conf), or copy /usr/bin/gcore to a temporary and
secret place, change its mode to 4755, and call it explitcitly
from Apache before crashing.
In addition, don't forget to ensure that limit coredumpsize is
unlimited, and the directory where core is going to be dumped
is writable by the user.
DON'T FORGET TO RESTORE EVERYTHING LATER! (httpd.conf, or the
secret copy of gcore, etc.).
Dirk-Willem van Gulik wrote:
>
> Anyone have any hints as how to proceed... apache bombs out with a
> bus/segv's quite often.. and I'd like a core.
>
> As we have 200+ active processes and 50 req/second attaching a debuger to
> PID's is not an option.. kind of difficult to guess which one :-)
>
> Now I've set/tried
>
> ulimit -c unlimited
>
> cd to the core-directory before startup
>
> CoreDumpDirectory ...
>
>
> the coredump directory is rw for the world. There is
> enough disk space.
>
> A 'kill -SEGV' does not yield core's either, but a CGI like
>
> #/usr/local/bin/perl
> $|++;
> print "Content-type: text/plain\n\n"
> exec "ulimit -a; kill -SIGV $$";
>
> produces one (of sh) just fine. Kind of suggesting that the inherited
> child environment is fine.
>
> Of course the log nicely log's the child SIGV-ing in both cases. Even
> though the SEGV of apache causes no core.
>
> Any hints. Same setup dumps fine on BSD.
>
> Any and all suggestions appreciated !
>
> Dw
--
Eli Marmor
Re: Solaris 2.7 and getting a clean coredump
Posted by Dirk-Willem van Gulik <di...@webweaving.org>.
On Thu, 2 Dec 1999, Jens-Uwe Mager wrote:
> # coreadm
> global core file pattern: /var/core/core.%f.%p.u%u
> init core file pattern: core
> global core dumps: enabled
> per-process core dumps: enabled
> global setid core dumps: enabled
> per-process setid core dumps: enabled
> global core dump logging: disabled
Yes! Thanks a million !!.
Dw.
Re: Solaris 2.7 and getting a clean coredump
Posted by Jens-Uwe Mager <ju...@helios.de>.
On Thu, Dec 02, 1999 at 02:33:31PM -0700, Marc Slemko wrote:
> If a process has setuid(), it can't dump core.
Solaris 7 August 99 has a nice coreadm command that allows one to
configure this, for example I am using:
# coreadm
global core file pattern: /var/core/core.%f.%p.u%u
init core file pattern: core
global core dumps: enabled
per-process core dumps: enabled
global setid core dumps: enabled
per-process setid core dumps: enabled
global core dump logging: disabled
--
Jens-Uwe Mager
HELIOS Software GmbH
Steinriede 3
30827 Garbsen
Germany
Phone: +49 5131 709320
FAX: +49 5131 709325
Internet: jum@helios.de
Re: Solaris 2.7 and getting a clean coredump
Posted by Marc Slemko <ma...@znep.com>.
On Thu, 2 Dec 1999, Dirk-Willem van Gulik wrote:
>
> Anyone have any hints as how to proceed... apache bombs out with a
> bus/segv's quite often.. and I'd like a core.
>
> As we have 200+ active processes and 50 req/second attaching a debuger to
> PID's is not an option.. kind of difficult to guess which one :-)
Is that really true? If it happens often, and as long as you can have your
MaxRequestsPerChild up high enough, then why not just pick one, attach
to it, and hope it chokes? It will slow thing down a little bit, may take
a while, but it often works. You have to muck a bit with gdb to
get it to not stop on various signals.k
>
> Now I've set/tried
>
> ulimit -c unlimited
>
> cd to the core-directory before startup
>
> CoreDumpDirectory ...
>
>
> the coredump directory is rw for the world. There is
> enough disk space.
If a process has setuid(), it can't dump core.
So you have to make it not setuid(). If you can reproduce this problem
in testing, running on a port >1024 and starting as the user it should
run as can do this.
Otherwise, you can perhaps hack your kernel to either allow anyone
to bind to low ports or to allow programs that have setuid() to
core dump or use a hack to pass the listening descriptor in as a
command line option.