You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Dirk-Willem van Gulik <di...@webweaving.org> on 1999/12/02 21:57:57 UTC

Solaris 2.7 and getting a clean coredump

Anyone have any hints as how to proceed... apache bombs out with a
bus/segv's quite often.. and I'd like a core.

As we have 200+ active processes and 50 req/second attaching a debuger to
PID's is not an option.. kind of difficult to guess which one :-)

Now I've set/tried

	ulimit -c unlimited

	cd to the core-directory before startup

	CoreDumpDirectory ...
		

	the coredump directory is rw for the world. There is
	enough disk space.
	
A 'kill -SEGV' does not yield core's either, but a CGI like

	#/usr/local/bin/perl
	$|++;
	print "Content-type: text/plain\n\n"
	exec "ulimit -a; kill -SIGV $$";

produces one (of sh) just fine. Kind of suggesting that the inherited
child environment is fine. 

Of course the log nicely log's the child SIGV-ing in both cases. Even
though the SEGV of apache causes no core.

Any hints. Same setup dumps fine on BSD.

Any and all suggestions appreciated !

Dw


Re: Solaris 2.7 and getting a clean coredump

Posted by Sam Talebbeik <sa...@ix.netcom.com>.
I am glad that the problem is resolved. Last that I checked, the issue
of setuid() was indeed discussed in Apache's debugging page. That
is why I pointed you to the page. Here is a cut and paste from that page
about setuid(). The "coreadm" utility is something that I was not
aware of. Thanks for pointing it out.

    ST.

-----------------------------------------------------------------------------

Getting the server to dump core

Strangely enough, sometimes you actually want to force the server to crash
so that you can get a
look at some nutty behavior. Normally this can be done simply by using the
gcore command.
However, for security reasons, most Unix systems do not allow a setuid
process to dump core, since
the file contents might reveal something that is supposed to be protected
in memory.

Here is one way to get a core file from a setuid Apache httpd process on
Solaris, without knowing
which httpd child might be the one to die [note: it is probably easier to
use the MaxClients trick in the
first section above].

    # for pid in `ps -eaf | fgrep httpd | cut -d' ' -f4`
    do
      truss -f -l -t\!all -S SIGSEGV -p $pid 2>&1 | egrep SIGSEGV &
    done

The undocumented '-S' flag to truss will halt the process in place upon
receipt of a given signal
(SIGSEGV in this case). At this point you can use:

    # gcore PID

and then look at the backtrace as discussed above for gdb.
-----------------------------------------------------------------------------

Dirk-Willem van Gulik wrote:

> On Thu, 2 Dec 1999, Sam Talebbeik wrote:
>
> > You will fine some very good debugging tips at Apache's
> > own developer web site. Try the following link:
> >
> > http://dev.apache.org/debugging.html
>
> But not the setuid() and coreadm information which actually solved the
> problem. I've commet an updated version to cvs.
>
> DW.


Re: Solaris 2.7 and getting a clean coredump

Posted by Dirk-Willem van Gulik <di...@webweaving.org>.

On Thu, 2 Dec 1999, Sam Talebbeik wrote:

> You will fine some very good debugging tips at Apache's
> own developer web site. Try the following link:
> 
> http://dev.apache.org/debugging.html

But not the setuid() and coreadm information which actually solved the
problem. I've commet an updated version to cvs.

DW.


Re: Solaris 2.7 and getting a clean coredump

Posted by Sam Talebbeik <sa...@ix.netcom.com>.
You will fine some very good debugging tips at Apache's
own developer web site. Try the following link:

http://dev.apache.org/debugging.html

    ST.

Eli Marmor wrote:

> Solaris processes can't core dump after calling setuid().
> Try to run it as non-root (with port number > 1023) and a dir
> which is writable by the uid. You can also run it with "User"
> root (httpd.conf), or copy /usr/bin/gcore to a temporary and
> secret place, change its mode to 4755, and call it explitcitly
> from Apache before crashing.
>
> In addition, don't forget to ensure that limit coredumpsize is
> unlimited, and the directory where core is going to be dumped
> is writable by the user.
>
> DON'T FORGET TO RESTORE EVERYTHING LATER!  (httpd.conf, or the
> secret copy of gcore, etc.).
>
> Dirk-Willem van Gulik wrote:
> >
> > Anyone have any hints as how to proceed... apache bombs out with a
> > bus/segv's quite often.. and I'd like a core.
> >
> > As we have 200+ active processes and 50 req/second attaching a debuger to
> > PID's is not an option.. kind of difficult to guess which one :-)
> >
> > Now I've set/tried
> >
> >         ulimit -c unlimited
> >
> >         cd to the core-directory before startup
> >
> >         CoreDumpDirectory ...
> >
> >
> >         the coredump directory is rw for the world. There is
> >         enough disk space.
> >
> > A 'kill -SEGV' does not yield core's either, but a CGI like
> >
> >         #/usr/local/bin/perl
> >         $|++;
> >         print "Content-type: text/plain\n\n"
> >         exec "ulimit -a; kill -SIGV $$";
> >
> > produces one (of sh) just fine. Kind of suggesting that the inherited
> > child environment is fine.
> >
> > Of course the log nicely log's the child SIGV-ing in both cases. Even
> > though the SEGV of apache causes no core.
> >
> > Any hints. Same setup dumps fine on BSD.
> >
> > Any and all suggestions appreciated !
> >
> > Dw
>
> --
> Eli Marmor


Re: Solaris 2.7 and getting a clean coredump

Posted by Eli Marmor <ma...@elmar.co.il>.
Solaris processes can't core dump after calling setuid().
Try to run it as non-root (with port number > 1023) and a dir
which is writable by the uid. You can also run it with "User"
root (httpd.conf), or copy /usr/bin/gcore to a temporary and
secret place, change its mode to 4755, and call it explitcitly
from Apache before crashing.

In addition, don't forget to ensure that limit coredumpsize is
unlimited, and the directory where core is going to be dumped
is writable by the user.

DON'T FORGET TO RESTORE EVERYTHING LATER!  (httpd.conf, or the
secret copy of gcore, etc.).

Dirk-Willem van Gulik wrote:
> 
> Anyone have any hints as how to proceed... apache bombs out with a
> bus/segv's quite often.. and I'd like a core.
> 
> As we have 200+ active processes and 50 req/second attaching a debuger to
> PID's is not an option.. kind of difficult to guess which one :-)
> 
> Now I've set/tried
> 
>         ulimit -c unlimited
> 
>         cd to the core-directory before startup
> 
>         CoreDumpDirectory ...
> 
> 
>         the coredump directory is rw for the world. There is
>         enough disk space.
> 
> A 'kill -SEGV' does not yield core's either, but a CGI like
> 
>         #/usr/local/bin/perl
>         $|++;
>         print "Content-type: text/plain\n\n"
>         exec "ulimit -a; kill -SIGV $$";
> 
> produces one (of sh) just fine. Kind of suggesting that the inherited
> child environment is fine.
> 
> Of course the log nicely log's the child SIGV-ing in both cases. Even
> though the SEGV of apache causes no core.
> 
> Any hints. Same setup dumps fine on BSD.
> 
> Any and all suggestions appreciated !
> 
> Dw

-- 
Eli Marmor

Re: Solaris 2.7 and getting a clean coredump

Posted by Dirk-Willem van Gulik <di...@webweaving.org>.

On Thu, 2 Dec 1999, Jens-Uwe Mager wrote:

> # coreadm
>      global core file pattern: /var/core/core.%f.%p.u%u
>        init core file pattern: core
>             global core dumps: enabled
>        per-process core dumps: enabled
>       global setid core dumps: enabled
>  per-process setid core dumps: enabled
>      global core dump logging: disabled

Yes! Thanks a million !!.

Dw.


Re: Solaris 2.7 and getting a clean coredump

Posted by Jens-Uwe Mager <ju...@helios.de>.
On Thu, Dec 02, 1999 at 02:33:31PM -0700, Marc Slemko wrote:

> If a process has setuid(), it can't dump core.

Solaris 7 August 99 has a nice coreadm command that allows one to
configure this, for example I am using:

# coreadm
     global core file pattern: /var/core/core.%f.%p.u%u
       init core file pattern: core
            global core dumps: enabled
       per-process core dumps: enabled
      global setid core dumps: enabled
 per-process setid core dumps: enabled
     global core dump logging: disabled


-- 
Jens-Uwe Mager

HELIOS Software GmbH
Steinriede 3
30827 Garbsen
Germany

Phone:		+49 5131 709320
FAX:		+49 5131 709325
Internet:	jum@helios.de

Re: Solaris 2.7 and getting a clean coredump

Posted by Marc Slemko <ma...@znep.com>.
On Thu, 2 Dec 1999, Dirk-Willem van Gulik wrote:

> 
> Anyone have any hints as how to proceed... apache bombs out with a
> bus/segv's quite often.. and I'd like a core.
> 
> As we have 200+ active processes and 50 req/second attaching a debuger to
> PID's is not an option.. kind of difficult to guess which one :-)

Is that really true?  If it happens often, and as long as you can have your
MaxRequestsPerChild up high enough, then why not just pick one, attach 
to it, and hope it chokes?  It will slow thing down a little bit, may take
a while, but it often works.  You have to muck a bit with gdb to 
get it to not stop on various signals.k

> 
> Now I've set/tried
> 
> 	ulimit -c unlimited
> 
> 	cd to the core-directory before startup
> 
> 	CoreDumpDirectory ...
> 		
> 
> 	the coredump directory is rw for the world. There is
> 	enough disk space.

If a process has setuid(), it can't dump core.

So you have to make it not setuid().  If you can reproduce this problem
in testing, running on a port >1024 and starting as the user it should
run as can do this.

Otherwise, you can perhaps hack your kernel to either allow anyone
to bind to low ports or to allow programs that have setuid() to
core dump or use a hack to pass the listening descriptor in as a
command line option.