You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@trafficserver.apache.org by "Alan M. Carroll" <am...@network-geographics.com> on 2011/08/27 07:32:32 UTC
Getting traffic_server to dump core
I want to have traffic_server leave behind a core file when it crashes but can't seem to make it happen. I have
* Used ulimit -c
* Set the core limit high in /etc/security/limits.conf
* Set proxy.config.core_limit to -1
* Verified that setrlimit is being called with a very large limit.
* Set the permissions to 777 on INSTALL and INSTALL/bin.
But still no core file.
Re: Getting traffic_server to dump core
Posted by "Alan M. Carroll" <am...@network-geographics.com>.
It's definitely a problem with calling setuid() and then not re-invoking PR_SET_DUMPABLE. I think it must have been broken for a long time as the version before using POSIX capabilities didn't call setuid until after invoking PR_SET_DUMPABLE. I put some extra debugging output in and verified that after calling change_uid_gid the PR_SET_DUMPABLE flag was off (core dumps disabled). I put a check in to the original call in set_core_size to validate and that did not trigger, indicating that the value was set at that point.
I made a patch that I am testing that sets a flag if the core file size is non-zero and calls PR_SET_DUMPABLE after calling change_uid_gid() if that flag is set. At that point I could generate core dumps.
Thanks for the note, Rayson, it set me on the right path.
Re: Getting traffic_server to dump core
Posted by Leif Hedstrom <zw...@apache.org>.
Btw, we have a Conflunce wiki page with some of this info already. Please update that page with the missing info from this thread.
Cheers,
-- Leif
On Aug 27, 2011, at 10:42 AM, Rayson Ho <ra...@gmail.com> wrote:
> Hi Alan,
>
> I could only find 3 places that reset the flag to PR_SET_DUMPABLE... I
> was wondering if the control flows into mgmt/LocalManager.cc
> (removeRootPriv(), restoreRootPriv()... as main() calls
> listenForProxy() after calling setup_coredump()) and other places
> that calls seteuid but without resetting PR_SET_DUMPABLE would affect
> the core dump flag.
>
> (I've joined this list for a while but still haven't have time to hack
> the Traffic Server code - so this is the first time I really read the
> TS code... I'm usually too busy with the Open Grid Scheduler project).
>
> Or just set sysctl and see if traffic server dumps core or not, if it
> does then it should be this issue and we can just add the missing
> PR_SET_DUMPABLE calls. If not, then it is something else on your
> system. (It is an easy hack but we don't need to enable it
> permanently.)
>
> Rayson
>
>
>
> On Sat, Aug 27, 2011 at 12:13 PM, Alan M. Carroll
> <am...@network-geographics.com> wrote:
>> Saturday, August 27, 2011, 10:01:19 AM, you wrote:
>>
>>> There's also the setuid(2)/seteuid(2)/setguid(2)/seteguid(2) issue on
>>> Linux (the kernel does not dump core setXid programs).
>>
>> I saw that but thought it meant only setuid at the file system level. However, ATS uses
>>
>> prctl(PR_SET_DUMPABLE, 1, 0, 0, 0);
>>
>> presumably to get around that problem. I checked the return value and it claims to have executed correctly. However, perhaps I changed the ordering too much when I fixed the libcap problems. Definitely something to check.
>>
>>> 2) There is also a easier (but a bit less secure way), and enabling it
>>> could cause sensitive data to be dumped to disk as it is a system-wide
>>> setting:
>>
>>> # sysctl -w kernel.core_setuid_ok=1
>>
>> I'll keep that in mind, although as you write that's a much less preferable solution.
>>
>>
>
>
>
> --
> Rayson
>
> ==================================================
> Open Grid Scheduler - The Official Open Source Grid Engine
> http://gridscheduler.sourceforge.net/
Re: Getting traffic_server to dump core
Posted by Rayson Ho <ra...@gmail.com>.
Hi Alan,
I could only find 3 places that reset the flag to PR_SET_DUMPABLE... I
was wondering if the control flows into mgmt/LocalManager.cc
(removeRootPriv(), restoreRootPriv()... as main() calls
listenForProxy() after calling setup_coredump()) and other places
that calls seteuid but without resetting PR_SET_DUMPABLE would affect
the core dump flag.
(I've joined this list for a while but still haven't have time to hack
the Traffic Server code - so this is the first time I really read the
TS code... I'm usually too busy with the Open Grid Scheduler project).
Or just set sysctl and see if traffic server dumps core or not, if it
does then it should be this issue and we can just add the missing
PR_SET_DUMPABLE calls. If not, then it is something else on your
system. (It is an easy hack but we don't need to enable it
permanently.)
Rayson
On Sat, Aug 27, 2011 at 12:13 PM, Alan M. Carroll
<am...@network-geographics.com> wrote:
> Saturday, August 27, 2011, 10:01:19 AM, you wrote:
>
>> There's also the setuid(2)/seteuid(2)/setguid(2)/seteguid(2) issue on
>> Linux (the kernel does not dump core setXid programs).
>
> I saw that but thought it meant only setuid at the file system level. However, ATS uses
>
> prctl(PR_SET_DUMPABLE, 1, 0, 0, 0);
>
> presumably to get around that problem. I checked the return value and it claims to have executed correctly. However, perhaps I changed the ordering too much when I fixed the libcap problems. Definitely something to check.
>
>> 2) There is also a easier (but a bit less secure way), and enabling it
>> could cause sensitive data to be dumped to disk as it is a system-wide
>> setting:
>
>> # sysctl -w kernel.core_setuid_ok=1
>
> I'll keep that in mind, although as you write that's a much less preferable solution.
>
>
--
Rayson
==================================================
Open Grid Scheduler - The Official Open Source Grid Engine
http://gridscheduler.sourceforge.net/
Re: Getting traffic_server to dump core
Posted by "Alan M. Carroll" <am...@network-geographics.com>.
Saturday, August 27, 2011, 10:01:19 AM, you wrote:
> There's also the setuid(2)/seteuid(2)/setguid(2)/seteguid(2) issue on
> Linux (the kernel does not dump core setXid programs).
I saw that but thought it meant only setuid at the file system level. However, ATS uses
prctl(PR_SET_DUMPABLE, 1, 0, 0, 0);
presumably to get around that problem. I checked the return value and it claims to have executed correctly. However, perhaps I changed the ordering too much when I fixed the libcap problems. Definitely something to check.
> 2) There is also a easier (but a bit less secure way), and enabling it
> could cause sensitive data to be dumped to disk as it is a system-wide
> setting:
> # sysctl -w kernel.core_setuid_ok=1
I'll keep that in mind, although as you write that's a much less preferable solution.
Re: Getting traffic_server to dump core
Posted by Rayson Ho <ra...@gmail.com>.
There's also the setuid(2)/seteuid(2)/setguid(2)/seteguid(2) issue on
Linux (the kernel does not dump core setXid programs).
There are a few places that trafficserver calls those system calls
(eg. in runAsUser()).
There are 2 ways of working around this:
1) In Sun Grid Engine (now the commercial Oracle Grid Engine and the
Open-Source Grid Engine project called Open Grid Scheduler) we use an
LD_PRELOAD wrapper to reset the core-dump bit after those system calls
are called.
You might want to take a look at our wrapper:
http://gridscheduler.sourceforge.net/patches/libcore.c
% gcc -D_GNU_SOURCE -fPIC -c libcore.c
% gcc -shared -o libcore.so libcore.o -lpthread
Then LD_PRELOAD libcore.so into the application to enable core dump.
2) There is also a easier (but a bit less secure way), and enabling it
could cause sensitive data to be dumped to disk as it is a system-wide
setting:
# sysctl -w kernel.core_setuid_ok=1
Rayson
On Sat, Aug 27, 2011 at 10:24 AM, Leif Hedstrom <le...@ogre.com> wrote:
> On 08/26/2011 11:32 PM, Alan M. Carroll wrote:
>>
>> I want to have traffic_server leave behind a core file when it crashes but
>> can't seem to make it happen. I have
>>
>> * Used ulimit -c
>> * Set the core limit high in /etc/security/limits.conf
>> * Set proxy.config.core_limit to -1
>> * Verified that setrlimit is being called with a very large limit.
>> * Set the permissions to 777 on INSTALL and INSTALL/bin.
>>
>
>
> Did you specify where it's should core dump? There's a sysctl for that.
>
> -- Leif
>
>
==================================================
Open Grid Scheduler - The Official Open Source Grid Engine
http://gridscheduler.sourceforge.net/
Re: Getting traffic_server to dump core
Posted by Leif Hedstrom <le...@ogre.com>.
On 08/26/2011 11:32 PM, Alan M. Carroll wrote:
> I want to have traffic_server leave behind a core file when it crashes but can't seem to make it happen. I have
>
> * Used ulimit -c
> * Set the core limit high in /etc/security/limits.conf
> * Set proxy.config.core_limit to -1
> * Verified that setrlimit is being called with a very large limit.
> * Set the permissions to 777 on INSTALL and INSTALL/bin.
>
Did you specify where it's should core dump? There's a sysctl for that.
-- Leif