You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@trafficserver.apache.org by Reindl Harald <h....@thelounge.net> on 2013/02/06 15:02:22 UTC
trafficserver-3.2.4 unstable?
hm - i have never seen this with 3.2.0 since it was
released (my first used version) and after update to
3.2.4 it happend three times
raw-storage if it matters
cat /etc/trafficserver/storage.config
# Storage Configuration file
/dev/sdc
_______________________________________
[ET_NET 2][7426]: segfault at 20 ip 0000000000645d13 sp 00002b9849f74940 error 4 in traffic_server[400000+349000
NOTE: Traffic Server received Sig 11: Segmentation fault
/usr/bin/traffic_server - STACK TRACE:
/lib64/libpthread.so.0(+0xefe0)[0x2b984574ffe0]
/usr/bin/traffic_server(_ZN7CacheVC12openReadMainEiP5Event+0x2b3)[0x645d13]
/usr/bin/traffic_server(_ZN7CacheVC8callcontEi+0x73)[0x64cf43]
/usr/bin/traffic_server(_ZN7CacheVC21openReadStartEarliestEiP5Event+0xa74)[0x649764]
/usr/bin/traffic_server(_ZN7CacheVC14handleReadDoneEiP5Event+0x144)[0x624194]
/usr/bin/traffic_server(_ZN19AIOCallbackInternal11io_completeEiPv+0x22)[0x625a02]
/usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x97)[0x69ef37]
/usr/bin/traffic_server(_ZN7EThread7executeEv+0x643)[0x69fa23]
/usr/bin/traffic_server[0x69dd72]
/lib64/libpthread.so.0(+0x7d14)[0x2b9845748d14]
/lib64/libc.so.6(clone+0x6d)[0x2b9847e8b68d]
[Feb 6 14:44:14.403] Manager {0x7f65a23f4880} ERROR: [LocalManager::pollMgmtProcessServer] Server Process
terminated due to Sig 11: Segmentation fault
[Feb 6 14:44:14.404] Manager {0x7f65a23f4880} ERROR: (last system error 2: No such file or directory)
[Feb 6 14:44:14.404] Manager {0x7f65a23f4880} ERROR: [Alarms::signalAlarm] Server Process was reset
[Feb 6 14:44:14.404] Manager {0x7f65a23f4880} ERROR: (last system error 2: No such file or directory)
[Feb 6 14:44:15.422] Manager {0x7f65a23f4880} NOTE: [LocalManager::startProxy] Launching ts process
Re: trafficserver-3.2.4 unstable?
Posted by Reindl Harald <h....@thelounge.net>.
Am 06.02.2013 17:11, schrieb Leif Hedstrom:
> On 2/6/13 9:08 AM, Leif Hedstrom wrote:
>> I wonder if we don't deal with some (reverted) changes to the storage structures? Can you try reinitialize the
>>>> cache, and see if that helps?
>>> ok, i will try a "systemctl stop trafficserver.service; dd if=/dev/zero of=/dev/sdc bs=1M count=20; systemctl start
>>> trafficserver.service" and give feedback if it happens again or until next week no longer
>>>
>>
>> Oh, it's easier than that. After stopping the service, run
>>
>> sudo traffic_server -Cclear_cache
>>
>>
>> that will clear and initialize the document cache(s).
>
>
> Note that I'm merely guessing here, if you have important data in your cache, don't blow it away yet (go back to
> the version of TS that works)
no problem
reverse-proxy only, there is nothing which is not filled up at request
BTW:
i wonder that TS idle needs around 2-3% CPU on a Xeon E5-2640 0 @ 2.50GH
under high load it does not get really high but what it is doing in
idle times? :-)
Re: trafficserver-3.2.4 unstable?
Posted by Leif Hedstrom <zw...@apache.org>.
On 2/6/13 9:08 AM, Leif Hedstrom wrote:
> I wonder if we don't deal with some (reverted) changes to the storage
> structures? Can you try reinitialize the
>>> cache, and see if that helps?
>> ok, i will try a "systemctl stop trafficserver.service; dd if=/dev/zero
>> of=/dev/sdc bs=1M count=20; systemctl start
>> trafficserver.service" and give feedback if it happens again or until
>> next week no longer
>>
>
> Oh, it's easier than that. After stopping the service, run
>
> sudo traffic_server -Cclear_cache
>
>
> that will clear and initialize the document cache(s).
Note that I'm merely guessing here, if you have important data in your
cache, don't blow it away yet (go back to the version of TS that works).
-- Leif
Re: trafficserver-3.2.4 unstable?
Posted by Leif Hedstrom <zw...@apache.org>.
On 2/6/13 8:23 AM, Reindl Harald wrote:
>
> Am 06.02.2013 16:18, schrieb Leif Hedstrom:
>> On 2/6/13 7:02 AM, Reindl Harald wrote:
>>> hm - i have never seen this with 3.2.0 since it was
>>> released (my first used version) and after update to
>>> 3.2.4 it happend three times
>> I wonder if we don't deal with some (reverted) changes to the storage structures? Can you try reinitialize the
>> cache, and see if that helps?
> ok, i will try a "systemctl stop trafficserver.service; dd if=/dev/zero of=/dev/sdc bs=1M count=20; systemctl start
> trafficserver.service" and give feedback if it happens again or until next week no longer
>
Oh, it's easier than that. After stopping the service, run
sudo traffic_server -Cclear_cache
that will clear and initialize the document cache(s).
-- Leif
Re: trafficserver-3.2.4 unstable?
Posted by Reindl Harald <h....@thelounge.net>.
Am 06.02.2013 16:23, schrieb Reindl Harald:
>
>
> Am 06.02.2013 16:18, schrieb Leif Hedstrom:
>> On 2/6/13 7:02 AM, Reindl Harald wrote:
>>> hm - i have never seen this with 3.2.0 since it was
>>> released (my first used version) and after update to
>>> 3.2.4 it happend three times
>>
>> I wonder if we don't deal with some (reverted) changes to the storage structures? Can you try reinitialize the
>> cache, and see if that helps?
>
> ok, i will try a "systemctl stop trafficserver.service; dd if=/dev/zero of=/dev/sdc bs=1M count=20; systemctl start
> trafficserver.service" and give feedback if it happens again or until next week no longer
sorry - but it still happens from time to time
[Feb 7 00:00:00.406] Server {0x2b3343241700} STATUS: The logfile /var/log/trafficserver/error.log was rolled to
/var/log/trafficserver/error.log_proxy.thelounge.net.20130206.00h25m16s-20130207.00h00m00s.old.
NOTE: Traffic Server received Sig 11: Segmentation fault
/usr/bin/traffic_server - STACK TRACE:
/lib64/libpthread.so.0(+0xefe0)[0x2b333005cfe0]
/usr/bin/traffic_server(_ZN7CacheVC12openReadMainEiP5Event+0x2b3)[0x645d13]
/usr/bin/traffic_server(_ZN7CacheVC8callcontEi+0x73)[0x64cf43]
/usr/bin/traffic_server(_ZN7CacheVC21openReadStartEarliestEiP5Event+0xa74)[0x649764]
/usr/bin/traffic_server(_ZN7CacheVC14handleReadDoneEiP5Event+0x144)[0x624194]
/usr/bin/traffic_server(_ZN19AIOCallbackInternal11io_completeEiPv+0x22)[0x625a02]
/usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x97)[0x69ef37]
/usr/bin/traffic_server(_ZN7EThread7executeEv+0x643)[0x69fa23]
/usr/bin/traffic_server(main+0x1439)[0x483679]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x2b33326c8735]
/usr/bin/traffic_server[0x48808d]
[Feb 7 10:23:46.538] Manager {0x7f147029e880} FATAL: [LocalManager::pollMgmtProcessServer] Error in read (errno:
104)
[Feb 7 10:23:46.539] Manager {0x7f147029e880} FATAL: (last system error 104: Connection reset by peer)
[Feb 7 10:23:46.539] Manager {0x7f147029e880} NOTE: [LocalManager::mgmtShutdown] Executing shutdown request.
[Feb 7 10:23:46.539] Manager {0x7f147029e880} NOTE: [LocalManager::processShutdown] Executing process shutdown
request.
[Feb 7 10:23:46.539] Manager {0x7f147029e880} ERROR: [LocalManager::sendMgmtMsgToProcesses] Error writing message
[Feb 7 10:23:46.539] Manager {0x7f147029e880} ERROR: (last system error 32: Broken pipe)
[E. Mgmt] log ==> [TrafficManager] using root directory '/usr'
[Feb 7 10:23:46.579] {0x7f1a36b59880} STATUS: opened /var/log/trafficserver/manager.log
[Feb 7 10:23:46.579] {0x7f1a36b59880} NOTE: updated diags config
[Feb 7 10:23:46.582] Manager {0x7f1a36b59880} NOTE: [ClusterCom::ClusterCom] Node running on OS: 'Linux' Release:
'3.7.3-101.fc17.x86_64'
[Feb 7 10:23:46.582] Manager {0x7f1a36b59880} NOTE: [LocalManager::listenForProxy] Listening on port: 80
[Feb 7 10:23:46.582] Manager {0x7f1a36b59880} NOTE: [TrafficManager] Setup complete
[Feb 7 10:23:47.613] Manager {0x7f1a36b59880} NOTE: [LocalManager::startProxy] Launching ts process
[TrafficServer] using root directory '/usr'
[Feb 7 10:23:47.627] Manager {0x7f1a36b59880} NOTE: [LocalManager::pollMgmtProcessServer] New process connecting
fd '13'
[Feb 7 10:23:47.627] Manager {0x7f1a36b59880} NOTE: [Alarms::signalAlarm] Server Process born
[Feb 7 10:23:48.648] {0x2b95c7585880} STATUS: opened /var/log/trafficserver/diags.log
[Feb 7 10:23:48.649] {0x2b95c7585880} NOTE: updated diags config
[Feb 7 10:23:48.653] Server {0x2b95c7585880} NOTE: cache clustering disabled
[Feb 7 10:23:48.666] Server {0x2b95c7585880} NOTE: cache clustering disabled
[Feb 7 10:23:48.723] Server {0x2b95c7585880} NOTE: logging initialized[15], logging_mode = 1
[Feb 7 10:23:48.879] Server {0x2b95c7585880} NOTE: traffic server running
[Feb 7 10:23:49.202] Server {0x2b95c7f08700} NOTE: cache enabled
NOTE: Traffic Server received Sig 11: Segmentation fault
/usr/bin/traffic_server - STACK TRACE:
/lib64/libpthread.so.0(+0xefe0)[0x2b95c37e3fe0]
/usr/bin/traffic_server(_ZN7CacheVC12openReadMainEiP5Event+0x2b3)[0x645d13]
/usr/bin/traffic_server(_ZN7CacheVC8callcontEi+0x73)[0x64cf43]
/usr/bin/traffic_server(_ZN7CacheVC21openReadStartEarliestEiP5Event+0xa74)[0x649764]
/usr/bin/traffic_server(_ZN7CacheVC14handleReadDoneEiP5Event+0x144)[0x624194]
/usr/bin/traffic_server(_ZN19AIOCallbackInternal11io_completeEiPv+0x22)[0x625a02]
/usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x97)[0x69ef37]
/usr/bin/traffic_server(_ZN7EThread7executeEv+0x643)[0x69fa23]
/usr/bin/traffic_server(main+0x1439)[0x483679]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x2b95c5e4f735]
/usr/bin/traffic_server[0x48808d]
[Feb 7 10:26:13.874] Manager {0x7f1a36b59880} ERROR: [LocalManager::pollMgmtProcessServer] Server Process
terminated due to Sig 11: Segmentation fault
[Feb 7 10:26:13.876] Manager {0x7f1a36b59880} ERROR: (last system error 2: No such file or directory)
[Feb 7 10:26:13.878] Manager {0x7f1a36b59880} ERROR: [Alarms::signalAlarm] Server Process was reset
[Feb 7 10:26:13.879] Manager {0x7f1a36b59880} ERROR: (last system error 2: No such file or directory)
[Feb 7 10:26:14.888] Manager {0x7f1a36b59880} NOTE: [LocalManager::startProxy] Launching ts process
[TrafficServer] using root directory '/usr'
[Feb 7 10:26:14.906] Manager {0x7f1a36b59880} NOTE: [LocalManager::pollMgmtProcessServer] New process connecting
fd '13'
[Feb 7 10:26:14.906] Manager {0x7f1a36b59880} NOTE: [Alarms::signalAlarm] Server Process born
[Feb 7 10:26:15.931] {0x2af21c7a7880} STATUS: opened /var/log/trafficserver/diags.log
[Feb 7 10:26:15.931] {0x2af21c7a7880} NOTE: updated diags config
[Feb 7 10:26:15.937] Server {0x2af21c7a7880} NOTE: cache clustering disabled
[Feb 7 10:26:15.949] Server {0x2af21c7a7880} NOTE: cache clustering disabled
[Feb 7 10:26:16.010] Server {0x2af21c7a7880} NOTE: logging initialized[15], logging_mode = 1
[Feb 7 10:26:16.105] Server {0x2af21c7a7880} NOTE: traffic server running
[Feb 7 10:26:16.446] Server {0x2af21d22b700} NOTE: cache enabled
Re: trafficserver-3.2.4 unstable?
Posted by Reindl Harald <h....@thelounge.net>.
Am 06.02.2013 16:18, schrieb Leif Hedstrom:
> On 2/6/13 7:02 AM, Reindl Harald wrote:
>> hm - i have never seen this with 3.2.0 since it was
>> released (my first used version) and after update to
>> 3.2.4 it happend three times
>
> I wonder if we don't deal with some (reverted) changes to the storage structures? Can you try reinitialize the
> cache, and see if that helps?
ok, i will try a "systemctl stop trafficserver.service; dd if=/dev/zero of=/dev/sdc bs=1M count=20; systemctl start
trafficserver.service" and give feedback if it happens again or until next week no longer
Re: trafficserver-3.2.4 unstable?
Posted by Leif Hedstrom <zw...@apache.org>.
On 2/6/13 7:02 AM, Reindl Harald wrote:
> hm - i have never seen this with 3.2.0 since it was
> released (my first used version) and after update to
> 3.2.4 it happend three times
I wonder if we don't deal with some (reverted) changes to the storage
structures? Can you try reinitialize the cache, and see if that helps?
-- Leif
>
> raw-storage if it matters
>
> cat /etc/trafficserver/storage.config
> # Storage Configuration file
> /dev/sdc
> _______________________________________
>
> [ET_NET 2][7426]: segfault at 20 ip 0000000000645d13 sp 00002b9849f74940 error 4 in traffic_server[400000+349000
>
> NOTE: Traffic Server received Sig 11: Segmentation fault
> /usr/bin/traffic_server - STACK TRACE:
> /lib64/libpthread.so.0(+0xefe0)[0x2b984574ffe0]
> /usr/bin/traffic_server(_ZN7CacheVC12openReadMainEiP5Event+0x2b3)[0x645d13]
> /usr/bin/traffic_server(_ZN7CacheVC8callcontEi+0x73)[0x64cf43]
> /usr/bin/traffic_server(_ZN7CacheVC21openReadStartEarliestEiP5Event+0xa74)[0x649764]
> /usr/bin/traffic_server(_ZN7CacheVC14handleReadDoneEiP5Event+0x144)[0x624194]
> /usr/bin/traffic_server(_ZN19AIOCallbackInternal11io_completeEiPv+0x22)[0x625a02]
> /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x97)[0x69ef37]
> /usr/bin/traffic_server(_ZN7EThread7executeEv+0x643)[0x69fa23]
> /usr/bin/traffic_server[0x69dd72]
> /lib64/libpthread.so.0(+0x7d14)[0x2b9845748d14]
> /lib64/libc.so.6(clone+0x6d)[0x2b9847e8b68d]
> [Feb 6 14:44:14.403] Manager {0x7f65a23f4880} ERROR: [LocalManager::pollMgmtProcessServer] Server Process
> terminated due to Sig 11: Segmentation fault
> [Feb 6 14:44:14.404] Manager {0x7f65a23f4880} ERROR: (last system error 2: No such file or directory)
> [Feb 6 14:44:14.404] Manager {0x7f65a23f4880} ERROR: [Alarms::signalAlarm] Server Process was reset
> [Feb 6 14:44:14.404] Manager {0x7f65a23f4880} ERROR: (last system error 2: No such file or directory)
> [Feb 6 14:44:15.422] Manager {0x7f65a23f4880} NOTE: [LocalManager::startProxy] Launching ts process
>
>
Re: trafficserver-3.2.4 unstable?
Posted by Reindl Harald <h....@thelounge.net>.
Am 07.02.2013 15:14, schrieb Jan-Frode Myklebust:
> On Thu, Feb 07, 2013 at 02:19:55PM +0100, Reindl Harald wrote:
>>>>
>>>> map http://www.rhsoft.net http://www.rhsoft.net
>>>> reverse_map http://www.rhsoft.net http://www.rhsoft.net
>>>
>>> Does that reverse_map make any sense ?
>>
>> it makes pretty much sense
>> you missed the part with dnsmasq :-)
>>
>> * trafficserver is using DNS 127.0.0.1
>> * this is dnsmasq configured with /etc/hosts.dnsmasq
>> * /etc/hosts.dnsmasq and the mappings are configured based on a webservice
>> * this way i can decide with the public DNS if a host should use
>> the trafficserver or directly the origin because trafficserver
>> here is useed to reduce image-loads fro high-traffic projects
>> by caching them for 60 seconds which makes not much sense
>> for small sites
>
> No, I didn't miss the dnsmasq part, but I might not fully understand
> reverse_maps. As far as I understand it, your origin server should
> return "Location: http://www.rhsoft.net" both when it's accessed
> directly, and when it's accessed trough the traffic server. And mapping
> http://www.rhsoft.net to http://www.rhsoft.net seems kind of redundant :-)
i try to explain it again by a example
[root@proxy:~]$ nslookup www.rhsoft.net
Non-authoritative answer:
www.rhsoft.net canonical name = proxy.thelounge.net.
Name: proxy.thelounge.net
Address: 91.118.73.4
[root@proxy:~]$ nslookup www.rhsoft.net 127.0.0.1
Server: 127.0.0.1
Address: 127.0.0.1#53
Name: www.rhsoft.net
Address: 10.0.0.6
so if DNS points to 91.118.73.4 = Trafficserver, well it will
be happy and fetch content from 10.0.0.6, if DNS points
to 10.0.0.6 origin is already happy too
the mapping are 100% automated generated so i can at any
moment in time point a DNS server to the trafficserver-IP
and it will just work - no maintainance ever needed
>>>> LimitNOFILE=100000
>>>> LimitMEMLOCK=infinity
>>>> OOMScoreAdjust=-1000
>>>> PrivateTmp=yes
>>>> CapabilityBoundingSet=~CAP_SYS_PTRACE
>>>> InaccessibleDirectories=/boot
>>>> InaccessibleDirectories=/home
>>>> InaccessibleDirectories=/usr/local/scripts
>>>> InaccessibleDirectories=/var/lib/rpm
>>>> InaccessibleDirectories=/var/spool
>>>
>>
>> the first 3 values where already there, we will see
>> for me "stack_dump_enabled" is new and unclear what
>> it is supposed to do
>
> I would rather try a less restrictive systemd environment. Drop
> CapabilityBoundingSet, PrivateTmp and limits, just to make sure they're
> not influencing
uhm the limits raise them up
OOMScoreAdjust=-1000 -> OOM-killer will never kill trafficserver
LimitNOFILE=100000 -> 100000 open files is normally not allowed
LimitMEMLOCK=infinity -> do not restrict memory usage for trafficserver
PrivateTmp should not matter because it is transparent
InaccessibleDirectories should never be touch by TS
CapabilityBoundingSet is the only questionable but should
never be needed by a reverse-proxy
Re: trafficserver-3.2.4 unstable?
Posted by Jan-Frode Myklebust <ja...@tanso.net>.
On Thu, Feb 07, 2013 at 02:19:55PM +0100, Reindl Harald wrote:
> >>
> >> map http://www.rhsoft.net http://www.rhsoft.net
> >> reverse_map http://www.rhsoft.net http://www.rhsoft.net
> >
> > Does that reverse_map make any sense ?
>
> it makes pretty much sense
> you missed the part with dnsmasq :-)
>
> * trafficserver is using DNS 127.0.0.1
> * this is dnsmasq configured with /etc/hosts.dnsmasq
> * /etc/hosts.dnsmasq and the mappings are configured based on a webservice
> * this way i can decide with the public DNS if a host should use
> the trafficserver or directly the origin because trafficserver
> here is useed to reduce image-loads fro high-traffic projects
> by caching them for 60 seconds which makes not much sense
> for small sites
No, I didn't miss the dnsmasq part, but I might not fully understand
reverse_maps. As far as I understand it, your origin server should
return "Location: http://www.rhsoft.net" both when it's accessed
directly, and when it's accessed trough the traffic server. And mapping
http://www.rhsoft.net to http://www.rhsoft.net seems kind of redundant :-)
> >> LimitNOFILE=100000
> >> LimitMEMLOCK=infinity
> >> OOMScoreAdjust=-1000
> >> PrivateTmp=yes
> >> CapabilityBoundingSet=~CAP_SYS_PTRACE
> >> InaccessibleDirectories=/boot
> >> InaccessibleDirectories=/home
> >> InaccessibleDirectories=/usr/local/scripts
> >> InaccessibleDirectories=/var/lib/rpm
> >> InaccessibleDirectories=/var/spool
> >
>
> the first 3 values where already there, we will see
> for me "stack_dump_enabled" is new and unclear what
> it is supposed to do
I would rather try a less restrictive systemd environment. Drop
CapabilityBoundingSet, PrivateTmp and limits, just to make sure they're
not influencing.
-jf
Re: trafficserver-3.2.4 unstable?
Posted by Reindl Harald <h....@thelounge.net>.
Am 15.02.2013 11:48, schrieb Jan-Frode Myklebust:
> On Thu, Feb 14, 2013 at 07:18:49PM +0100, Reindl Harald wrote:
>>
>>
>>>>> CapabilityBoundingSet=~CAP_SYS_PTRACE
>>>
>>> CONFIG proxy.config.stack_dump_enabled 0
>>>
>> FYI: "CONFIG proxy.config.stack_dump_enabled 0" resolves the
>> instability in context of "CapabilityBoundingSet=~CAP_SYS_PTRACE"
>
> Are you saying that changing either of these fixed your crashes ? If so,
> great to know so that I don't push this into the EPEL/Fedora packages!
CONFIG proxy.config.stack_dump_enabled 0
and all is fine
> I should probably do the PrivateTmp and a list of InaccessibleDirectories
be careful with "InaccessibleDirectories"
currently if they not existing the service won't start
there is a discussion on the systemd-list and Lennart
seems to be interested to support the minus sign as
for commandos to ignore the directory in such cases
> Not quite sure about OOMScoreAdjust and the limits.. I haven't needed
> setting any limits myself, and don't really care too much if the
> OOM-killer kills my ATS (as this will lead to IP-address failover and
> recovery in my config)
well, i simply took this from my mysld.service on a dbmail
machine where if some client / imap-process runs crazy the
lastz you want to see is mysqld killed and since i use
a dedicated virtual machine with only trafficserver it
can't hurt
Re: trafficserver-3.2.4 unstable?
Posted by Jan-Frode Myklebust <ja...@tanso.net>.
On Thu, Feb 14, 2013 at 07:18:49PM +0100, Reindl Harald wrote:
>
>
> >>> CapabilityBoundingSet=~CAP_SYS_PTRACE
> >
> > CONFIG proxy.config.stack_dump_enabled 0
> >
> FYI: "CONFIG proxy.config.stack_dump_enabled 0" resolves the
> instability in context of "CapabilityBoundingSet=~CAP_SYS_PTRACE"
Are you saying that changing either of these fixed your crashes ? If so,
great to know so that I don't push this into the EPEL/Fedora packages!
I should probably do the PrivateTmp and a list of InaccessibleDirectories.
Not quite sure about OOMScoreAdjust and the limits.. I haven't needed
setting any limits myself, and don't really care too much if the
OOM-killer kills my ATS (as this will lead to IP-address failover and
recovery in my config).
-jf
Re: trafficserver-3.2.4 unstable?
Posted by Reindl Harald <h....@thelounge.net>.
Am 07.02.2013 14:19, schrieb Reindl Harald:
>>> LimitNOFILE=100000
>>> LimitMEMLOCK=infinity
>>> OOMScoreAdjust=-1000
>>> PrivateTmp=yes
>>> CapabilityBoundingSet=~CAP_SYS_PTRACE
>>> InaccessibleDirectories=/boot
>>> InaccessibleDirectories=/home
>>> InaccessibleDirectories=/usr/local/scripts
>>> InaccessibleDirectories=/var/lib/rpm
>>> InaccessibleDirectories=/var/spool
>
> CONFIG proxy.config.diags.debug.enabled INT 0
> CONFIG proxy.config.diags.debug.tags STRING http.*|dns.*
> CONFIG proxy.config.dump_mem_info_frequency INT 0
> CONFIG proxy.config.stack_dump_enabled 0
>
> the first 3 values where already there, we will see
> for me "stack_dump_enabled" is new and unclear what
> it is supposed to do
FYI: "CONFIG proxy.config.stack_dump_enabled 0" resolves the
instability in context of "CapabilityBoundingSet=~CAP_SYS_PTRACE"
maybe this should not be implicitly enabled - not because my
systemd-settings, more because in context of servers any code
which does not run can not make troubles in a security-context
Re: trafficserver-3.2.4 unstable?
Posted by Reindl Harald <h....@thelounge.net>.
Am 07.02.2013 13:55, schrieb Jan-Frode Myklebust:
> On Thu, Feb 07, 2013 at 11:54:24AM +0100, Reindl Harald wrote:
>>
>> /etc/trafficserver/remap.config has some hundret hosts like below
>> on the machine is running dnsmasq on 127.0.0.1 to feed the target
>> IP's and dnsmasq-config is also as "remap.config" automatically
>> created with any hostname in the infrastructure
>>
>> map http://www.rhsoft.net http://www.rhsoft.net
>> reverse_map http://www.rhsoft.net http://www.rhsoft.net
>
> Does that reverse_map make any sense ?
it makes pretty much sense
you missed the part with dnsmasq :-)
* trafficserver is using DNS 127.0.0.1
* this is dnsmasq configured with /etc/hosts.dnsmasq
* /etc/hosts.dnsmasq and the mappings are configured based on a webservice
* this way i can decide with the public DNS if a host should use
the trafficserver or directly the origin because trafficserver
here is useed to reduce image-loads fro high-traffic projects
by caching them for 60 seconds which makes not much sense
for small sites
>> [Service]
>> Type=simple
>> ExecStart=/usr/bin/traffic_cop
>> ExecReload=/usr/bin/traffic_line -x
>> Restart=always
>> RestartSec=1
>> LimitNOFILE=100000
>> LimitMEMLOCK=infinity
>> OOMScoreAdjust=-1000
>> PrivateTmp=yes
>> CapabilityBoundingSet=~CAP_SYS_PTRACE
>> InaccessibleDirectories=/boot
>> InaccessibleDirectories=/home
>> InaccessibleDirectories=/usr/local/scripts
>> InaccessibleDirectories=/var/lib/rpm
>> InaccessibleDirectories=/var/spool
>
> Cool that you're using systemd to restrict ATS like that, but could it
> cause problems? Does maybe the default proxy.config.stack_dump_enabled=1
> conflict with disabling CAP_SYS_PTRACE ?
>
> Sorry, HTH, but I'm just guessing here..
CONFIG proxy.config.diags.debug.enabled INT 0
CONFIG proxy.config.diags.debug.tags STRING http.*|dns.*
CONFIG proxy.config.dump_mem_info_frequency INT 0
CONFIG proxy.config.stack_dump_enabled 0
the first 3 values where already there, we will see
for me "stack_dump_enabled" is new and unclear what
it is supposed to do
Re: trafficserver-3.2.4 unstable?
Posted by Jan-Frode Myklebust <ja...@tanso.net>.
On Thu, Feb 07, 2013 at 11:54:24AM +0100, Reindl Harald wrote:
>
> /etc/trafficserver/remap.config has some hundret hosts like below
> on the machine is running dnsmasq on 127.0.0.1 to feed the target
> IP's and dnsmasq-config is also as "remap.config" automatically
> created with any hostname in teh infrastructure
>
> map http://www.rhsoft.net http://www.rhsoft.net
> reverse_map http://www.rhsoft.net http://www.rhsoft.net
Does that reverse_map make any sense ?
> [Service]
> Type=simple
> ExecStart=/usr/bin/traffic_cop
> ExecReload=/usr/bin/traffic_line -x
> Restart=always
> RestartSec=1
> LimitNOFILE=100000
> LimitMEMLOCK=infinity
> OOMScoreAdjust=-1000
> PrivateTmp=yes
> CapabilityBoundingSet=~CAP_SYS_PTRACE
> InaccessibleDirectories=/boot
> InaccessibleDirectories=/home
> InaccessibleDirectories=/usr/local/scripts
> InaccessibleDirectories=/var/lib/rpm
> InaccessibleDirectories=/var/spool
Cool that you're using systemd to restrict ATS like that, but could it
cause problems? Does maybe the default proxy.config.stack_dump_enabled=1
conflict with disabling CAP_SYS_PTRACE ?
Sorry, HTH, but I'm just guessing here..
-jf
Re: trafficserver-3.2.4 unstable?
Posted by Reindl Harald <h....@thelounge.net>.
Am 07.02.2013 11:44, schrieb Jan-Frode Myklebust:
> On Wed, Feb 06, 2013 at 03:02:22PM +0100, Reindl Harald wrote:
>> hm - i have never seen this with 3.2.0 since it was
>> released (my first used version) and after update to
>> 3.2.4 it happend three times
>
> v3.2.4 has been perfectly stable for me on RHEL6, while I had lots of crashes
> with earlier v3.2.x versions. Looks like you're on fedora-17, so might be worth
> trying this build:
>
> http://koji.fedoraproject.org/koji/taskinfo?taskID=4904824
>
> I can't push this to f17 because of the incompatible changes from v3.0,
> ref: https://cwiki.apache.org/confluence/display/TS/Upgrading+to+3.2 , but it
> was pushed as an update to f18 today I think.
i have this build, 3.2.0 was the first version i ever used
and last summer a rebuild of the F17 src.rpm on F16
> Are you doing anything special with your config? I'm running in reverse
> proxy mode, using multiple ipv4 and ipv6-addresses, SSL certificates with both
> wildcard and SAN certs on multiple ip-adresses.
nothing special
________________________________________________________________________
/etc/trafficserver/remap.config has some hundret hosts like below
on the machine is running dnsmasq on 127.0.0.1 to feed the target
IP's and dnsmasq-config is also as "remap.config" automatically
created with any hostname in teh infrastructure
map http://www.rhsoft.net http://www.rhsoft.net
reverse_map http://www.rhsoft.net http://www.rhsoft.net
________________________________________________________________________
[root@proxy:~]$ cat /etc/systemd/system/trafficserver.service
[Unit]
Description=Apache Traffic Server
After=network.target dnsmasq.service
[Service]
Type=simple
ExecStart=/usr/bin/traffic_cop
ExecReload=/usr/bin/traffic_line -x
Restart=always
RestartSec=1
LimitNOFILE=100000
LimitMEMLOCK=infinity
OOMScoreAdjust=-1000
PrivateTmp=yes
CapabilityBoundingSet=~CAP_SYS_PTRACE
InaccessibleDirectories=/boot
InaccessibleDirectories=/home
InaccessibleDirectories=/usr/local/scripts
InaccessibleDirectories=/var/lib/rpm
InaccessibleDirectories=/var/spool
[Install]
WantedBy=multi-user.target
[root@proxy:~]$ cat /etc/trafficserver/cache.config
url_regex=. suffix=css ttl-in-cache=30 cache-responses-to-cookies=1
url_regex=. suffix=js ttl-in-cache=30 cache-responses-to-cookies=1
url_regex=. suffix=txt ttl-in-cache=60 cache-responses-to-cookies=1
url_regex=. suffix=htm ttl-in-cache=60 cache-responses-to-cookies=1
url_regex=. suffix=html ttl-in-cache=60 cache-responses-to-cookies=1
url_regex=. suffix=xml ttl-in-cache=60 cache-responses-to-cookies=1
url_regex=. suffix=htc ttl-in-cache=60 cache-responses-to-cookies=1
[root@proxy:~]$ cat records.config
##############################################################################
# System Variables #
##############################################################################
CONFIG proxy.config.proxy_name STRING proxy.thelounge.net
CONFIG proxy.config.config_dir STRING /etc/trafficserver
CONFIG proxy.config.proxy_binary_opts STRING -M
CONFIG proxy.config.temp_dir STRING /tmp
CONFIG proxy.config.alarm_email STRING ats
CONFIG proxy.config.syslog_facility STRING LOG_DAEMON
CONFIG proxy.config.output.logfile STRING traffic.out
CONFIG proxy.config.snapshot_dir STRING snapshots
CONFIG proxy.config.system.mmap_max INT 2097152
##############################################################################
# Main threads configuration (worker threads). Also see configurations for #
# SSL threads, disk I/O threads and task threads in their respective areas #
##############################################################################
CONFIG proxy.config.exec_thread.autoconfig INT 1
CONFIG proxy.config.exec_thread.autoconfig.scale FLOAT 1.500000
CONFIG proxy.config.exec_thread.limit INT 4
CONFIG proxy.config.accept_threads INT 1
##############################################################################
# Local Manager #
##############################################################################
CONFIG proxy.config.admin.admin_user STRING admin
CONFIG proxy.config.admin.number_config_bak INT 0
CONFIG proxy.config.admin.user_id STRING ats
##############################################################################
# Process Manager #
##############################################################################
CONFIG proxy.config.admin.autoconf_port INT 8083
CONFIG proxy.config.process_manager.mgmt_port INT 8084
##############################################################################
# HTTP Engine #
##############################################################################
CONFIG proxy.config.http.server_ports STRING 80
CONFIG proxy.config.http.connect_ports STRING 80
CONFIG proxy.config.http.insert_request_via_str INT 0
CONFIG proxy.config.http.insert_response_via_str INT 0
CONFIG proxy.config.http.response_server_enabled INT 0
CONFIG proxy.config.http.insert_age_in_response INT 1
CONFIG proxy.config.http.enable_url_expandomatic INT 0
CONFIG proxy.config.http.no_dns_just_forward_to_parent INT 0
CONFIG proxy.config.http.uncacheable_requests_bypass_parent INT 0
CONFIG proxy.config.http.keep_alive_enabled_in INT 1
CONFIG proxy.config.http.keep_alive_enabled_out INT 0
CONFIG proxy.config.http.chunking_enabled INT 1
CONFIG proxy.config.http.send_http11_requests INT 1
CONFIG proxy.config.http.share_server_sessions INT 1
CONFIG proxy.config.http.origin_server_pipeline INT 1
CONFIG proxy.config.http.user_agent_pipeline INT 8
CONFIG proxy.config.http.referer_filter INT 0
##############################################################################
# parent proxy configuration #
##############################################################################
CONFIG proxy.config.http.parent_proxy_routing_enable INT 0
##############################################################################
# HTTP connection timeouts (secs) #
# out: proxy -> origin server connection #
# in : ua -> proxy connection #
##############################################################################
CONFIG proxy.config.http.keep_alive_no_activity_timeout_in INT 1
CONFIG proxy.config.http.keep_alive_no_activity_timeout_out INT 0
CONFIG proxy.config.http.transaction_no_activity_timeout_in INT 60
CONFIG proxy.config.http.transaction_no_activity_timeout_out INT 60
CONFIG proxy.config.http.transaction_active_timeout_in INT 900
CONFIG proxy.config.http.transaction_active_timeout_out INT 0
CONFIG proxy.config.http.accept_no_activity_timeout INT 30
CONFIG proxy.config.http.background_fill_active_timeout INT 60
CONFIG proxy.config.http.background_fill_completed_threshold FLOAT 0.000000
##############################################################################
# origin server connect attempts #
##############################################################################
CONFIG proxy.config.http.connect_attempts_max_retries INT 10
CONFIG proxy.config.http.connect_attempts_max_retries_dead_server INT 10
CONFIG proxy.config.http.connect_attempts_rr_retries INT 10
CONFIG proxy.config.http.connect_attempts_timeout INT 30
CONFIG proxy.config.http.post_connect_attempts_timeout INT 1800
CONFIG proxy.config.http.down_server.cache_time INT 5
CONFIG proxy.config.http.down_server.abort_threshold INT 30
##############################################################################
# congestion control #
##############################################################################
CONFIG proxy.config.http.congestion_control.enabled INT 0
##############################################################################
# negative response caching #
##############################################################################
CONFIG proxy.config.http.negative_caching_enabled INT 1
CONFIG proxy.config.http.negative_caching_lifetime INT 60
##############################################################################
# proxy users variables #
##############################################################################
CONFIG proxy.config.http.anonymize_remove_from INT 0
CONFIG proxy.config.http.anonymize_remove_referer INT 0
CONFIG proxy.config.http.anonymize_remove_user_agent INT 0
CONFIG proxy.config.http.anonymize_remove_cookie INT 0
CONFIG proxy.config.http.anonymize_remove_client_ip INT 0
CONFIG proxy.config.http.anonymize_insert_client_ip INT 0
CONFIG proxy.config.http.anonymize_other_header_list STRING NULL
CONFIG proxy.config.http.insert_squid_x_forwarded_for INT 1
##############################################################################
# security #
##############################################################################
CONFIG proxy.config.http.push_method_enabled INT 0
##############################################################################
# cache control #
##############################################################################
CONFIG proxy.config.http.normalize_ae_gzip INT 1
CONFIG proxy.config.http.cache.http INT 1
CONFIG proxy.config.http.cache.cache_responses_to_cookies INT 3
CONFIG proxy.config.http.cache.cache_urls_that_look_dynamic INT 1
CONFIG proxy.config.http.cache.ims_on_client_no_cache INT 1
CONFIG proxy.config.http.cache.ignore_client_cc_max_age INT 0
CONFIG proxy.config.http.cache.ignore_client_no_cache INT 0
CONFIG proxy.config.http.cache.ignore_accept_mismatch INT 1
CONFIG proxy.config.http.cache.ignore_accept_language_mismatch INT 1
CONFIG proxy.config.http.cache.ignore_accept_charset_mismatch INT 1
CONFIG proxy.config.http.cache.ignore_server_no_cache INT 0
CONFIG proxy.config.http.cache.ignore_authentication INT 0
CONFIG proxy.config.http.cache.enable_default_vary_headers INT 0
CONFIG proxy.config.http.cache.when_to_revalidate INT 0
CONFIG proxy.config.http.cache.when_to_add_no_cache_to_msie_requests INT 0
CONFIG proxy.config.http.cache.required_headers INT 0
CONFIG proxy.config.http.cache.max_stale_age INT 1800
CONFIG proxy.config.http.cache.range.lookup INT 1
CONFIG proxy.config.cache.vary_on_user_agent INT 0
##############################################################################
# heuristic expiration #
##############################################################################
CONFIG proxy.config.http.cache.heuristic_min_lifetime INT 60
CONFIG proxy.config.http.cache.heuristic_max_lifetime INT 60
CONFIG proxy.config.http.cache.heuristic_lm_factor FLOAT 0.100000
CONFIG proxy.config.http.cache.fuzz.time INT 60
CONFIG proxy.config.http.cache.fuzz.probability FLOAT 0.005000
##############################################################################
# dynamic content & content negotiation #
##############################################################################
CONFIG proxy.config.http.cache.vary_default_text STRING Accept-Encoding
CONFIG proxy.config.http.cache.vary_default_images STRING NULL
CONFIG proxy.config.http.cache.vary_default_other STRING NULL
##############################################################################
# The HTTP stats are expensive, turn off you dont need them #
##############################################################################
CONFIG proxy.config.http.enable_http_stats INT 0
##############################################################################
# Customizable User Response Pages #
##############################################################################
CONFIG proxy.config.body_factory.enable_customizations INT 1
CONFIG proxy.config.body_factory.enable_logging INT 0
CONFIG proxy.config.body_factory.response_suppression_mode INT 0
##############################################################################
# Net Subsystem #
##############################################################################
CONFIG proxy.config.net.connections_throttle INT 50000
CONFIG proxy.config.net.defer_accept INT 15
##############################################################################
# Cluster Subsystem #
##############################################################################
LOCAL proxy.local.cluster.type INT 3
##############################################################################
# Cache #
##############################################################################
CONFIG proxy.config.cache.permit.pinning INT 0
CONFIG proxy.config.cache.ram_cache.size INT 2415919104
CONFIG proxy.config.cache.ram_cache_cutoff INT 7340032
CONFIG proxy.config.cache.ram_cache.algorithm INT 0
CONFIG proxy.config.cache.ram_cache.use_seen_filter INT 0
CONFIG proxy.config.cache.ram_cache.compress INT 0
CONFIG proxy.config.cache.limits.http.max_alts INT 10
CONFIG proxy.config.cache.target_fragment_size INT 262144
CONFIG proxy.config.cache.max_doc_size INT 0
CONFIG proxy.config.cache.enable_read_while_writer INT 1
CONFIG proxy.config.connection_collapsing.hashtable_enabled INT 1
CONFIG proxy.config.cache.min_average_object_size INT 8000
CONFIG proxy.config.cache.threads_per_disk INT 16
CONFIG proxy.config.cache.mutex_retry_delay INT 2
##############################################################################
# DNS #
##############################################################################
CONFIG proxy.config.dns.search_default_domains INT 1
CONFIG proxy.config.dns.splitDNS.enabled INT 0
CONFIG proxy.config.dns.max_dns_in_flight INT 2048
CONFIG proxy.config.dns.url_expansions STRING NULL
CONFIG proxy.config.dns.round_robin_nameservers INT 0
CONFIG proxy.config.dns.nameservers STRING 127.0.0.1
CONFIG proxy.config.dns.resolv_conf STRING NULL
CONFIG proxy.config.dns.validate_query_name INT 0
##############################################################################
# HostDB #
##############################################################################
CONFIG proxy.config.hostdb.size INT 50000
CONFIG proxy.config.hostdb.storage_size INT 14680064
CONFIG proxy.config.hostdb.ttl_mode INT 0
CONFIG proxy.config.hostdb.timeout INT 1440
CONFIG proxy.config.hostdb.strict_round_robin INT 0
##############################################################################
# Logging Config #
# #
# 0: no logging at all #
# 1: log errors only #
# 2: log transactions only #
# 3: full logging (errors + transactions) #
##############################################################################
LOCAL proxy.local.log.collation_mode INT 0
CONFIG proxy.config.log.logging_enabled INT 1
CONFIG proxy.config.log.max_secs_per_buffer INT 5
CONFIG proxy.config.log.max_space_mb_for_logs INT 25000
CONFIG proxy.config.log.max_space_mb_for_orphan_logs INT 25
CONFIG proxy.config.log.max_space_mb_headroom INT 1000
CONFIG proxy.config.log.hostname STRING localhost
CONFIG proxy.config.log.logfile_dir STRING /var/log/trafficserver
CONFIG proxy.config.log.logfile_perm STRING rw-r--r--
CONFIG proxy.config.log.custom_logs_enabled INT 0
CONFIG proxy.config.log.squid_log_enabled INT 1
CONFIG proxy.config.log.squid_log_is_ascii INT 0
CONFIG proxy.config.log.squid_log_name STRING squid
CONFIG proxy.config.log.squid_log_header STRING NULL
CONFIG proxy.config.log.common_log_enabled INT 0
CONFIG proxy.config.log.common_log_is_ascii INT 1
CONFIG proxy.config.log.common_log_name STRING common
CONFIG proxy.config.log.common_log_header STRING NULL
CONFIG proxy.config.log.extended_log_enabled INT 0
CONFIG proxy.config.log.extended_log_is_ascii INT 0
CONFIG proxy.config.log.extended_log_name STRING extended
CONFIG proxy.config.log.extended_log_header STRING NULL
CONFIG proxy.config.log.extended2_log_enabled INT 0
CONFIG proxy.config.log.extended2_log_is_ascii INT 1
CONFIG proxy.config.log.extended2_log_name STRING extended2
CONFIG proxy.config.log.extended2_log_header STRING NULL
CONFIG proxy.config.log.separate_icp_logs INT 0
CONFIG proxy.config.log.separate_host_logs INT 0
CONFIG proxy.config.log.collation_host STRING NULL
CONFIG proxy.config.log.collation_port INT 8085
CONFIG proxy.config.log.collation_secret STRING foobar
CONFIG proxy.config.log.collation_host_tagged INT 0
CONFIG proxy.config.log.collation_retry_sec INT 5
CONFIG proxy.config.log.rolling_enabled INT 1
CONFIG proxy.config.log.rolling_interval_sec INT 86400
CONFIG proxy.config.log.rolling_offset_hr INT 0
CONFIG proxy.config.log.rolling_size_mb INT 10
CONFIG proxy.config.log.auto_delete_rolled_files INT 1
CONFIG proxy.config.log.sampling_frequency INT 1
##############################################################################
# Reverse Proxy #
##############################################################################
CONFIG proxy.config.reverse_proxy.enabled INT 1
CONFIG proxy.config.header.parse.no_host_url_redirect STRING NULL
##############################################################################
# URL Remap Rules #
##############################################################################
CONFIG proxy.config.url_remap.default_to_server_pac INT 0
CONFIG proxy.config.url_remap.default_to_server_pac_port INT -1
CONFIG proxy.config.url_remap.remap_required INT 1
CONFIG proxy.config.url_remap.pristine_host_hdr INT 1
##############################################################################
# ICP Configuration #
##############################################################################
CONFIG proxy.config.icp.enabled INT 0
##############################################################################
# Scheduled Update Configuration #
##############################################################################
CONFIG proxy.config.update.enabled INT 0
CONFIG proxy.config.update.force INT 0
CONFIG proxy.config.update.retry_count INT 10
CONFIG proxy.config.update.retry_interval INT 2
CONFIG proxy.config.update.concurrent_updates INT 100
##############################################################################
# Socket send/recv buffer sizes 0 == dont call setsockopt() #
# out: proxy -> os connection #
# in : ua -> proxy connection #
##############################################################################
CONFIG proxy.config.net.sock_send_buffer_size_in INT 32768
CONFIG proxy.config.net.sock_recv_buffer_size_in INT 32768
CONFIG proxy.config.net.sock_send_buffer_size_out INT 32768
CONFIG proxy.config.net.sock_recv_buffer_size_out INT 32768
CONFIG proxy.config.net.sock_option_flag_in INT 1
CONFIG proxy.config.net.sock_option_flag_out INT 1
##############################################################################
# User Overridden Configurations Below #
##############################################################################
CONFIG proxy.config.core_limit INT -1
##############################################################################
# Debugging #
##############################################################################
CONFIG proxy.config.diags.debug.enabled INT 0
CONFIG proxy.config.diags.debug.tags STRING http.*|dns.*
CONFIG proxy.config.dump_mem_info_frequency INT 0
##############################################################################
# Log any request that takes more then x number of milliseconds, needs #
# to be > 0 to be enabled #
##############################################################################
CONFIG proxy.config.http.slow.log.threshold INT 0
##############################################################################
# Thread pool for "misc" tasks, plugins etc. 2 is a good minimum #
##############################################################################
CONFIG proxy.config.task_threads INT 4
CONFIG proxy.config.cluster.cluster_configuration STRING cluster.config
CONFIG proxy.config.body_factory.template_sets_dir STRING /etc/trafficserver/body_factory
Re: trafficserver-3.2.4 unstable?
Posted by Jan-Frode Myklebust <ja...@tanso.net>.
On Wed, Feb 06, 2013 at 03:02:22PM +0100, Reindl Harald wrote:
> hm - i have never seen this with 3.2.0 since it was
> released (my first used version) and after update to
> 3.2.4 it happend three times
v3.2.4 has been perfectly stable for me on RHEL6, while I had lots of crashes
with earlier v3.2.x versions. Looks like you're on fedora-17, so might be worth
trying this build:
http://koji.fedoraproject.org/koji/taskinfo?taskID=4904824
I can't push this to f17 because of the incompatible changes from v3.0,
ref: https://cwiki.apache.org/confluence/display/TS/Upgrading+to+3.2 , but it
was pushed as an update to f18 today I think.
Are you doing anything special with your config? I'm running in reverse
proxy mode, using multiple ipv4 and ipv6-addresses, SSL certificates with both
wildcard and SAN certs on multiple ip-adresses.
-jf
Re: trafficserver-3.2.4 unstable?
Posted by Reindl Harald <h....@thelounge.net>.
if you followed the thread
CONFIG proxy.config.stack_dump_enabled 0
after that i never saw a ATS process crashing
Am 17.02.2013 01:14, schrieb David Carlin:
> I upgraded one box to 3.2.4, and approx. 24 hours after deploying it I saw
> a seg fault with an identical stack trace.
>
> From /var/log/messages:
>
> Feb 16 11:43:26 l4 kernel: [ET_NET 18][16876]: segfault at 155 ip
> 0000000000567571 sp 00002b04d1412b40 error 4 in
> traffic_server[400000+341000]
>
> From traffic.out:
>
> NOTE: Traffic Server received Sig 11: Segmentation fault
> /home/y/bin/traffic_server - STACK TRACE:
> /lib64/libpthread.so.0[0x31f8c0f500]
> /home/y/bin/traffic_server(_ZN10HttpTunnel15chain_abort_allEP18HttpTunnelPr
> oducer+0xe1)[0x567571]
> /home/y/bin/traffic_server(_ZN6HttpSM17tunnel_handler_uaEiP18HttpTunnelCons
> umer+0x354)[0x5205b4]
> /home/y/bin/traffic_server(_ZN10HttpTunnel16consumer_handlerEiP18HttpTunnel
> Consumer+0x96)[0x56b716]
> /home/y/bin/traffic_server(_ZN10HttpTunnel12main_handlerEiPv+0x11d)[0x56bf6
> d]
> /home/y/bin/traffic_server(_ZN6HttpSM28state_watch_for_client_abortEiPv+0x2
> aa)[0x523d2a]
> /home/y/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0xe8)[0x52b2f8]
> /home/y/bin/traffic_server[0x672b5b]
> /home/y/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x5df
> )[0x674f5f]
> /home/y/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0xb4)[0x6970
> f4]
> /home/y/bin/traffic_server(_ZN7EThread7executeEv+0x633)[0x697bf3]
> /home/y/bin/traffic_server[0x6960c2]
> /lib64/libpthread.so.0[0x31f8c07851]
> /lib64/libc.so.6(clone+0x6d)[0x31f88e76dd]
> [Feb 16 11:43:27.240] Manager {0x7f217b8d07e0} FATAL:
> [LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
> [Feb 16 11:43:27.240] Manager {0x7f217b8d07e0} FATAL: (last system error
> 104: Connection reset by peer)
> [Feb 16 11:43:27.240] Manager {0x7f217b8d07e0} NOTE:
> [LocalManager::mgmtShutdown] Executing shutdown request.
> [Feb 16 11:43:27.240] Manager {0x7f217b8d07e0} NOTE:
> [LocalManager::processShutdown] Executing process shutdown request.
> [Feb 16 11:43:27.241] Manager {0x7f217b8d07e0} ERROR:
> [LocalManager::sendMgmtMsgToProcesses] Error writing message
> [Feb 16 11:43:27.241] Manager {0x7f217b8d07e0} ERROR: (last system error
> 32: Broken pipe)
>
>
>
> On 2/6/13 8:02 AM, "Reindl Harald" <h....@thelounge.net> wrote:
>
>> hm - i have never seen this with 3.2.0 since it was
>> released (my first used version) and after update to
>> 3.2.4 it happend three times
>>
>> raw-storage if it matters
>>
>> cat /etc/trafficserver/storage.config
>> # Storage Configuration file
>> /dev/sdc
>> _______________________________________
>>
>> [ET_NET 2][7426]: segfault at 20 ip 0000000000645d13 sp 00002b9849f74940
>> error 4 in traffic_server[400000+349000
>>
>> NOTE: Traffic Server received Sig 11: Segmentation fault
>> /usr/bin/traffic_server - STACK TRACE:
>> /lib64/libpthread.so.0(+0xefe0)[0x2b984574ffe0]
>> /usr/bin/traffic_server(_ZN7CacheVC12openReadMainEiP5Event+0x2b3)[0x645d13
>> ]
>> /usr/bin/traffic_server(_ZN7CacheVC8callcontEi+0x73)[0x64cf43]
>> /usr/bin/traffic_server(_ZN7CacheVC21openReadStartEarliestEiP5Event+0xa74)
>> [0x649764]
>> /usr/bin/traffic_server(_ZN7CacheVC14handleReadDoneEiP5Event+0x144)[0x6241
>> 94]
>> /usr/bin/traffic_server(_ZN19AIOCallbackInternal11io_completeEiPv+0x22)[0x
>> 625a02]
>> /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x97)[0x69ef37
>> ]
>> /usr/bin/traffic_server(_ZN7EThread7executeEv+0x643)[0x69fa23]
>> /usr/bin/traffic_server[0x69dd72]
>> /lib64/libpthread.so.0(+0x7d14)[0x2b9845748d14]
>> /lib64/libc.so.6(clone+0x6d)[0x2b9847e8b68d]
>> [Feb 6 14:44:14.403] Manager {0x7f65a23f4880} ERROR:
>> [LocalManager::pollMgmtProcessServer] Server Process
>> terminated due to Sig 11: Segmentation fault
>> [Feb 6 14:44:14.404] Manager {0x7f65a23f4880} ERROR: (last system error
>> 2: No such file or directory)
>> [Feb 6 14:44:14.404] Manager {0x7f65a23f4880} ERROR:
>> [Alarms::signalAlarm] Server Process was reset
>> [Feb 6 14:44:14.404] Manager {0x7f65a23f4880} ERROR: (last system error
>> 2: No such file or directory)
>> [Feb 6 14:44:15.422] Manager {0x7f65a23f4880} NOTE:
>> [LocalManager::startProxy] Launching ts process
Re: trafficserver-3.2.4 unstable?
Posted by David Carlin <dc...@yahoo-inc.com>.
I upgraded one box to 3.2.4, and approx. 24 hours after deploying it I saw
a seg fault with an identical stack trace.
>From /var/log/messages:
Feb 16 11:43:26 l4 kernel: [ET_NET 18][16876]: segfault at 155 ip
0000000000567571 sp 00002b04d1412b40 error 4 in
traffic_server[400000+341000]
>From traffic.out:
NOTE: Traffic Server received Sig 11: Segmentation fault
/home/y/bin/traffic_server - STACK TRACE:
/lib64/libpthread.so.0[0x31f8c0f500]
/home/y/bin/traffic_server(_ZN10HttpTunnel15chain_abort_allEP18HttpTunnelPr
oducer+0xe1)[0x567571]
/home/y/bin/traffic_server(_ZN6HttpSM17tunnel_handler_uaEiP18HttpTunnelCons
umer+0x354)[0x5205b4]
/home/y/bin/traffic_server(_ZN10HttpTunnel16consumer_handlerEiP18HttpTunnel
Consumer+0x96)[0x56b716]
/home/y/bin/traffic_server(_ZN10HttpTunnel12main_handlerEiPv+0x11d)[0x56bf6
d]
/home/y/bin/traffic_server(_ZN6HttpSM28state_watch_for_client_abortEiPv+0x2
aa)[0x523d2a]
/home/y/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0xe8)[0x52b2f8]
/home/y/bin/traffic_server[0x672b5b]
/home/y/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x5df
)[0x674f5f]
/home/y/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0xb4)[0x6970
f4]
/home/y/bin/traffic_server(_ZN7EThread7executeEv+0x633)[0x697bf3]
/home/y/bin/traffic_server[0x6960c2]
/lib64/libpthread.so.0[0x31f8c07851]
/lib64/libc.so.6(clone+0x6d)[0x31f88e76dd]
[Feb 16 11:43:27.240] Manager {0x7f217b8d07e0} FATAL:
[LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
[Feb 16 11:43:27.240] Manager {0x7f217b8d07e0} FATAL: (last system error
104: Connection reset by peer)
[Feb 16 11:43:27.240] Manager {0x7f217b8d07e0} NOTE:
[LocalManager::mgmtShutdown] Executing shutdown request.
[Feb 16 11:43:27.240] Manager {0x7f217b8d07e0} NOTE:
[LocalManager::processShutdown] Executing process shutdown request.
[Feb 16 11:43:27.241] Manager {0x7f217b8d07e0} ERROR:
[LocalManager::sendMgmtMsgToProcesses] Error writing message
[Feb 16 11:43:27.241] Manager {0x7f217b8d07e0} ERROR: (last system error
32: Broken pipe)
On 2/6/13 8:02 AM, "Reindl Harald" <h....@thelounge.net> wrote:
>hm - i have never seen this with 3.2.0 since it was
>released (my first used version) and after update to
>3.2.4 it happend three times
>
>raw-storage if it matters
>
>cat /etc/trafficserver/storage.config
># Storage Configuration file
>/dev/sdc
>_______________________________________
>
>[ET_NET 2][7426]: segfault at 20 ip 0000000000645d13 sp 00002b9849f74940
>error 4 in traffic_server[400000+349000
>
>NOTE: Traffic Server received Sig 11: Segmentation fault
>/usr/bin/traffic_server - STACK TRACE:
>/lib64/libpthread.so.0(+0xefe0)[0x2b984574ffe0]
>/usr/bin/traffic_server(_ZN7CacheVC12openReadMainEiP5Event+0x2b3)[0x645d13
>]
>/usr/bin/traffic_server(_ZN7CacheVC8callcontEi+0x73)[0x64cf43]
>/usr/bin/traffic_server(_ZN7CacheVC21openReadStartEarliestEiP5Event+0xa74)
>[0x649764]
>/usr/bin/traffic_server(_ZN7CacheVC14handleReadDoneEiP5Event+0x144)[0x6241
>94]
>/usr/bin/traffic_server(_ZN19AIOCallbackInternal11io_completeEiPv+0x22)[0x
>625a02]
>/usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x97)[0x69ef37
>]
>/usr/bin/traffic_server(_ZN7EThread7executeEv+0x643)[0x69fa23]
>/usr/bin/traffic_server[0x69dd72]
>/lib64/libpthread.so.0(+0x7d14)[0x2b9845748d14]
>/lib64/libc.so.6(clone+0x6d)[0x2b9847e8b68d]
>[Feb 6 14:44:14.403] Manager {0x7f65a23f4880} ERROR:
>[LocalManager::pollMgmtProcessServer] Server Process
>terminated due to Sig 11: Segmentation fault
>[Feb 6 14:44:14.404] Manager {0x7f65a23f4880} ERROR: (last system error
>2: No such file or directory)
>[Feb 6 14:44:14.404] Manager {0x7f65a23f4880} ERROR:
>[Alarms::signalAlarm] Server Process was reset
>[Feb 6 14:44:14.404] Manager {0x7f65a23f4880} ERROR: (last system error
>2: No such file or directory)
>[Feb 6 14:44:15.422] Manager {0x7f65a23f4880} NOTE:
>[LocalManager::startProxy] Launching ts process
>
>