You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@trafficserver.apache.org by "Ricky Chan (JIRA)" <ji...@apache.org> on 2011/04/01 10:55:06 UTC
[jira] [Created] (TS-729) Seg faults within
HttpTransactHeaders::insert_via_header_in_request
Seg faults within HttpTransactHeaders::insert_via_header_in_request
-------------------------------------------------------------------
Key: TS-729
URL: https://issues.apache.org/jira/browse/TS-729
Project: Traffic Server
Issue Type: Bug
Components: MIME
Affects Versions: 2.0.1
Environment: X6240 Sun Blade (AMD64), 64G of RAM.
Reporter: Ricky Chan
Getting reports of regular seg faults on the traffic server CDN farm, in 1 site 7 out of 10 servers have segfault at least once in 24 hour period. Traffic volume is low around 6 MB/s across all 10 servers. The idea was to put more traffic onto but I need to resolve this first.
Okay I stipped the binaries so the core dump information isn't as help as it could be but it indicates the issue happened inside insert_via_header_in_request.
(gdb) bt
#0 0x00000000006f2ef3 in ink_stack_trace_dump ()
#1 0x0000000000502f8a in ?? ()
#2 <signal handler called>
#3 0x0000000000580903 in HttpTransactHeaders::insert_via_header_in_request ()
#4 0x0000000000562a3a in HttpTransact::build_request ()
#5 0x00000000005790f6 in HttpTransact::HandleCacheOpenReadHit ()
#6 0x0000000000534485 in HttpSM::call_transact_and_set_next_state ()
#7 0x000000000054b35b in HttpSM::set_next_state ()
#8 0x0000000000534469 in HttpSM::call_transact_and_set_next_state ()
#9 0x0000000000536a62 in HttpSM::do_hostdb_lookup ()
#10 0x000000000054b5db in HttpSM::set_next_state ()
#11 0x0000000000534469 in HttpSM::call_transact_and_set_next_state ()
#12 0x000000000054b35b in HttpSM::set_next_state ()
#13 0x0000000000534469 in HttpSM::call_transact_and_set_next_state ()
#14 0x000000000054b35b in HttpSM::set_next_state ()
#15 0x0000000000534469 in HttpSM::call_transact_and_set_next_state ()
#16 0x0000000000549491 in HttpSM::state_cache_open_read ()
#17 0x0000000000547e3b in HttpSM::main_handler ()
#18 0x000000000052457e in HttpCacheSM::state_cache_open_read ()
#19 0x0000000000694dc4 in CacheVC::openReadStartEarliest ()
#20 0x000000000066e3c0 in CacheVC::handleReadDone ()
#21 0x00000000006756fb in AIOCallbackInternal::io_complete ()
#22 0x00000000006e3614 in EThread::process_event ()
#23 0x00000000006e3f5b in EThread::execute ()
#24 0x00000000006e1a72 in ?? ()
#25 0x00002b9c8dd73fc7 in start_thread () from /lib/libpthread.so.0
#26 0x00002b9c8f4e564d in clone () from /lib/libc.so.6
#27 0x0000000000000000 in ?? ()
The segfault issue was the same for all concerned servers.
I'll see if I can put the unstripped binaries back on so we can get more information.
Ricky
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (TS-729) Seg faults within
HttpTransactHeaders::insert_via_header_in_request
Posted by "Ricky Chan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ricky Chan updated TS-729:
--------------------------
Looking at the source code.
Both
HttpTransactHeaders::insert_via_header_in_request
HttpTransactHeaders::insert_via_header_in_response
Doesn't seem to assert that data add is not larger than 8192.
It could be possible I have a client which had made a request near that limit and when it appened it buffer overflowed triggering the segfault.
For now I am setting:
proxy.config.http.insert_request_via_str INT 0
I have left request via as I "trust" my origins not to return silly Via headers.
I'll see if that causes the segfaults to go away.
Ricky
> Seg faults within HttpTransactHeaders::insert_via_header_in_request
> -------------------------------------------------------------------
>
> Key: TS-729
> URL: https://issues.apache.org/jira/browse/TS-729
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 Sun Blade (AMD64), 64G of RAM.
> Reporter: Ricky Chan
>
> Getting reports of regular seg faults on the traffic server CDN farm, in 1 site 7 out of 10 servers have segfault at least once in 24 hour period. Traffic volume is low around 6 MB/s across all 10 servers. The idea was to put more traffic onto but I need to resolve this first.
> Okay I stipped the binaries so the core dump information isn't as help as it could be but it indicates the issue happened inside insert_via_header_in_request.
> (gdb) bt
> #0 0x00000000006f2ef3 in ink_stack_trace_dump ()
> #1 0x0000000000502f8a in ?? ()
> #2 <signal handler called>
> #3 0x0000000000580903 in HttpTransactHeaders::insert_via_header_in_request ()
> #4 0x0000000000562a3a in HttpTransact::build_request ()
> #5 0x00000000005790f6 in HttpTransact::HandleCacheOpenReadHit ()
> #6 0x0000000000534485 in HttpSM::call_transact_and_set_next_state ()
> #7 0x000000000054b35b in HttpSM::set_next_state ()
> #8 0x0000000000534469 in HttpSM::call_transact_and_set_next_state ()
> #9 0x0000000000536a62 in HttpSM::do_hostdb_lookup ()
> #10 0x000000000054b5db in HttpSM::set_next_state ()
> #11 0x0000000000534469 in HttpSM::call_transact_and_set_next_state ()
> #12 0x000000000054b35b in HttpSM::set_next_state ()
> #13 0x0000000000534469 in HttpSM::call_transact_and_set_next_state ()
> #14 0x000000000054b35b in HttpSM::set_next_state ()
> #15 0x0000000000534469 in HttpSM::call_transact_and_set_next_state ()
> #16 0x0000000000549491 in HttpSM::state_cache_open_read ()
> #17 0x0000000000547e3b in HttpSM::main_handler ()
> #18 0x000000000052457e in HttpCacheSM::state_cache_open_read ()
> #19 0x0000000000694dc4 in CacheVC::openReadStartEarliest ()
> #20 0x000000000066e3c0 in CacheVC::handleReadDone ()
> #21 0x00000000006756fb in AIOCallbackInternal::io_complete ()
> #22 0x00000000006e3614 in EThread::process_event ()
> #23 0x00000000006e3f5b in EThread::execute ()
> #24 0x00000000006e1a72 in ?? ()
> #25 0x00002b9c8dd73fc7 in start_thread () from /lib/libpthread.so.0
> #26 0x00002b9c8f4e564d in clone () from /lib/libc.so.6
> #27 0x0000000000000000 in ?? ()
> The segfault issue was the same for all concerned servers.
> I'll see if I can put the unstripped binaries back on so we can get more information.
> Ricky
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-729) Seg faults within
HttpTransactHeaders::insert_via_header_in_request
Posted by "Ricky Chan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017451#comment-13017451 ]
Ricky Chan commented on TS-729:
-------------------------------
I've managed to re-create the bug using the method details in TS-733.
As such I was able to get a trace in the lab:
(gdb) bt
#0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
#1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
#2 <signal handler called>
#3 0x0000000000580903 in HttpTransactHeaders::insert_via_header_in_request (http_config_param=<value optimized out>, scheme=<value optimized out>,
cache_info=<value optimized out>, header=0x2aaaba35ad30, incoming_via=<value optimized out>, proxy_ip_address=<value optimized out>)
at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1290
#4 0x0000000000562a3a in HttpTransact::build_request (s=0x2aaaba35a650, base_request=0x2aaaba35acc0, outgoing_request=0x2aaaba35ad30,
outgoing_version=<value optimized out>) at HttpTransact.cc:9106
#5 0x000000000056bec8 in HttpTransact::HandleCacheOpenReadMiss (s=0x2aaaba35a650) at HttpTransact.cc:3445
#6 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba35a5d0, f=0x1037f28) at HttpSM.cc:7190
#7 0x000000000054b35b in HttpSM::set_next_state (this=0x2aaaba35a5d0) at HttpSM.cc:7232
#8 0x0000000000534469 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba35a5d0, f=<value optimized out>) at HttpSM.cc:7198
#9 0x0000000000536a62 in HttpSM::do_hostdb_lookup (this=0x2aaaba35a5d0) at HttpSM.cc:4079
#10 0x000000000054b5db in HttpSM::set_next_state (this=0x2aaaba35a5d0) at HttpSM.cc:7278
#11 0x0000000000534469 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba35a5d0, f=<value optimized out>) at HttpSM.cc:7198
#12 0x00000000005494ff in HttpSM::state_cache_open_read (this=0x2aaaba35a5d0, event=<value optimized out>, data=0xffffffffffffb049) at HttpSM.cc:535
#13 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba35a5d0, event=1103, data=0xffffffffffffb049) at HttpSM.cc:2683
#14 0x0000000000524516 in HttpCacheSM::state_cache_open_read (this=0x2aaaba35c4b8, event=1103, data=0xffffffffffffb049)
at ../../iocore/eventsystem/I_Continuation.h:147
#15 0x00000000006933e3 in Cache::open_read (this=<value optimized out>, cont=0x2aaaba35c4b8, key=0x41b81570, request=0x2aaaba35acc0, params=0x2aaaba35a6f8,
type=25,
hostname=0x1d83fc0 "test1.isp.sky.com0lient-ip127.0.0.1X-Forwarded-For127.0.0.11.1 AKmdrL2CacheBC10.telecom.co.nz. (BUG ME), HTTP/1.0 SkyCDN[0AF482C5] (ApacheTrafficServer/2.0.1 [uScM])", host_len=17) at ../../iocore/eventsystem/I_Continuation.h:147
#16 0x000000000066e074 in CacheProcessor::open_read (this=<value optimized out>, cont=0x2aaaba35c4b8, url=0x2aaaba35a690, request=0x2aaaba35acc0,
params=0x2aaaba35a6f8, pin_in_cache=<value optimized out>, type=CACHE_FRAG_TYPE_HTTP) at P_CacheInternal.h:940
#17 0x0000000000523b34 in HttpCacheSM::open_read (this=0x2aaaba35c4b8, url=<value optimized out>, hdr=<value optimized out>, params=<value optimized out>,
pin_in_cache=<value optimized out>) at HttpCacheSM.cc:223
#18 0x0000000000537f0d in HttpSM::do_cache_lookup_and_read (this=0x2aaaba35a5d0) at HttpSM.cc:4240
#19 0x000000000054b480 in HttpSM::set_next_state (this=0x2aaaba35a5d0) at HttpSM.cc:7306
#20 0x0000000000534469 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba35a5d0, f=<value optimized out>) at HttpSM.cc:7198
#21 0x000000000054b35b in HttpSM::set_next_state (this=0x2aaaba35a5d0) at HttpSM.cc:7232
#22 0x0000000000534469 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba35a5d0, f=<value optimized out>) at HttpSM.cc:7198
#23 0x000000000054b415 in HttpSM::set_next_state (this=0x2aaaba35a5d0) at HttpSM.cc:7242
#24 0x0000000000534469 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba35a5d0, f=<value optimized out>) at HttpSM.cc:7198
#25 0x000000000054b35b in HttpSM::set_next_state (this=0x2aaaba35a5d0) at HttpSM.cc:7232
#26 0x0000000000534469 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba35a5d0, f=<value optimized out>) at HttpSM.cc:7198
#27 0x000000000053c012 in HttpSM::state_read_client_request_header (this=0x2aaaba35a5d0, event=100, data=<value optimized out>) at HttpSM.cc:852
#28 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba35a5d0, event=100, data=0x1e23608) at HttpSM.cc:2683
#29 0x00000000006c1a17 in read_from_net (nh=0x2aaaaacf4098, vc=0x1e23530, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
#30 0x00000000006b9472 in NetHandler::mainNetEvent (this=0x2aaaaacf4098, event=<value optimized out>, e=0x1062150) at UnixNet.cc:292
#31 0x00000000006e3634 in EThread::process_event (this=0x2aaaaacf3010, e=0x1062150, calling_code=5) at I_Continuation.h:147
#32 0x00000000006e3e70 in EThread::execute (this=0x2aaaaacf3010) at UnixEThread.cc:249
#33 0x00000000006e1a92 in spawn_thread_internal (a=0x1048ee0) at Thread.cc:85
#34 0x00002b7889e16fc7 in start_thread () from /lib/libpthread.so.0
#35 0x00002b788b58864d in clone () from /lib/libc.so.6
#36 0x0000000000000000 in ?? ()
(gdb)
(gdb) frame 3
#3 0x0000000000580903 in HttpTransactHeaders::insert_via_header_in_request (http_config_param=<value optimized out>, scheme=<value optimized out>,
cache_info=<value optimized out>, header=0x2aaaba35ad30, incoming_via=<value optimized out>, proxy_ip_address=<value optimized out>)
at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1290
1290 while (field->m_next_dup)
(gdb) p *header
$1 = {<MIMEHdr> = {<HdrHeapSDKHandle> = {m_sdk_alloc = {<DLL<SDKAllocHdr, -1>> = {head = 0x0}, <No data fields>}, m_heap = 0x1d61390}, m_mime = 0x1d61448},
m_http = 0x1d61418, m_url_cached = {<HdrHeapSDKHandle> = {m_sdk_alloc = {<DLL<SDKAllocHdr, -1>> = {head = 0x0}, <No data fields>}, m_heap = 0x0},
m_url_impl = 0x0}}
> Seg faults within HttpTransactHeaders::insert_via_header_in_request
> -------------------------------------------------------------------
>
> Key: TS-729
> URL: https://issues.apache.org/jira/browse/TS-729
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 Sun Blade (AMD64), 64G of RAM.
> Reporter: Ricky Chan
> Assignee: Leif Hedstrom
> Fix For: 2.1.8
>
>
> Getting reports of regular seg faults on the traffic server CDN farm, in 1 site 7 out of 10 servers have segfault at least once in 24 hour period. Traffic volume is low around 6 MB/s across all 10 servers. The idea was to put more traffic onto but I need to resolve this first.
> Okay I stipped the binaries so the core dump information isn't as help as it could be but it indicates the issue happened inside insert_via_header_in_request.
> (gdb) bt
> #0 0x00000000006f2ef3 in ink_stack_trace_dump ()
> #1 0x0000000000502f8a in ?? ()
> #2 <signal handler called>
> #3 0x0000000000580903 in HttpTransactHeaders::insert_via_header_in_request ()
> #4 0x0000000000562a3a in HttpTransact::build_request ()
> #5 0x00000000005790f6 in HttpTransact::HandleCacheOpenReadHit ()
> #6 0x0000000000534485 in HttpSM::call_transact_and_set_next_state ()
> #7 0x000000000054b35b in HttpSM::set_next_state ()
> #8 0x0000000000534469 in HttpSM::call_transact_and_set_next_state ()
> #9 0x0000000000536a62 in HttpSM::do_hostdb_lookup ()
> #10 0x000000000054b5db in HttpSM::set_next_state ()
> #11 0x0000000000534469 in HttpSM::call_transact_and_set_next_state ()
> #12 0x000000000054b35b in HttpSM::set_next_state ()
> #13 0x0000000000534469 in HttpSM::call_transact_and_set_next_state ()
> #14 0x000000000054b35b in HttpSM::set_next_state ()
> #15 0x0000000000534469 in HttpSM::call_transact_and_set_next_state ()
> #16 0x0000000000549491 in HttpSM::state_cache_open_read ()
> #17 0x0000000000547e3b in HttpSM::main_handler ()
> #18 0x000000000052457e in HttpCacheSM::state_cache_open_read ()
> #19 0x0000000000694dc4 in CacheVC::openReadStartEarliest ()
> #20 0x000000000066e3c0 in CacheVC::handleReadDone ()
> #21 0x00000000006756fb in AIOCallbackInternal::io_complete ()
> #22 0x00000000006e3614 in EThread::process_event ()
> #23 0x00000000006e3f5b in EThread::execute ()
> #24 0x00000000006e1a72 in ?? ()
> #25 0x00002b9c8dd73fc7 in start_thread () from /lib/libpthread.so.0
> #26 0x00002b9c8f4e564d in clone () from /lib/libc.so.6
> #27 0x0000000000000000 in ?? ()
> The segfault issue was the same for all concerned servers.
> I'll see if I can put the unstripped binaries back on so we can get more information.
> Ricky
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-729) Seg faults within
HttpTransactHeaders::insert_via_header_in_request
Posted by "Leif Hedstrom (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017765#comment-13017765 ]
Leif Hedstrom commented on TS-729:
----------------------------------
Did we conclude that this is the same crasher as TS-733 ? If so, can we close this one? I think TS-733 has more details.
> Seg faults within HttpTransactHeaders::insert_via_header_in_request
> -------------------------------------------------------------------
>
> Key: TS-729
> URL: https://issues.apache.org/jira/browse/TS-729
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 Sun Blade (AMD64), 64G of RAM.
> Reporter: Ricky Chan
> Assignee: Leif Hedstrom
> Fix For: 2.1.8
>
>
> Getting reports of regular seg faults on the traffic server CDN farm, in 1 site 7 out of 10 servers have segfault at least once in 24 hour period. Traffic volume is low around 6 MB/s across all 10 servers. The idea was to put more traffic onto but I need to resolve this first.
> Okay I stipped the binaries so the core dump information isn't as help as it could be but it indicates the issue happened inside insert_via_header_in_request.
> (gdb) bt
> #0 0x00000000006f2ef3 in ink_stack_trace_dump ()
> #1 0x0000000000502f8a in ?? ()
> #2 <signal handler called>
> #3 0x0000000000580903 in HttpTransactHeaders::insert_via_header_in_request ()
> #4 0x0000000000562a3a in HttpTransact::build_request ()
> #5 0x00000000005790f6 in HttpTransact::HandleCacheOpenReadHit ()
> #6 0x0000000000534485 in HttpSM::call_transact_and_set_next_state ()
> #7 0x000000000054b35b in HttpSM::set_next_state ()
> #8 0x0000000000534469 in HttpSM::call_transact_and_set_next_state ()
> #9 0x0000000000536a62 in HttpSM::do_hostdb_lookup ()
> #10 0x000000000054b5db in HttpSM::set_next_state ()
> #11 0x0000000000534469 in HttpSM::call_transact_and_set_next_state ()
> #12 0x000000000054b35b in HttpSM::set_next_state ()
> #13 0x0000000000534469 in HttpSM::call_transact_and_set_next_state ()
> #14 0x000000000054b35b in HttpSM::set_next_state ()
> #15 0x0000000000534469 in HttpSM::call_transact_and_set_next_state ()
> #16 0x0000000000549491 in HttpSM::state_cache_open_read ()
> #17 0x0000000000547e3b in HttpSM::main_handler ()
> #18 0x000000000052457e in HttpCacheSM::state_cache_open_read ()
> #19 0x0000000000694dc4 in CacheVC::openReadStartEarliest ()
> #20 0x000000000066e3c0 in CacheVC::handleReadDone ()
> #21 0x00000000006756fb in AIOCallbackInternal::io_complete ()
> #22 0x00000000006e3614 in EThread::process_event ()
> #23 0x00000000006e3f5b in EThread::execute ()
> #24 0x00000000006e1a72 in ?? ()
> #25 0x00002b9c8dd73fc7 in start_thread () from /lib/libpthread.so.0
> #26 0x00002b9c8f4e564d in clone () from /lib/libc.so.6
> #27 0x0000000000000000 in ?? ()
> The segfault issue was the same for all concerned servers.
> I'll see if I can put the unstripped binaries back on so we can get more information.
> Ricky
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-729) Seg faults within
HttpTransactHeaders::insert_via_header_in_request
Posted by "Ricky Chan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13014687#comment-13014687 ]
Ricky Chan commented on TS-729:
-------------------------------
I'm currently see if I can emulate the segfault in the lab, but because I didn't have debugging on I can't see what the request header was like prior to crash and also with stripped binaries I lose the line numbers.
If I can *somehow* re-create in the lab, I will post line numbers and conditions to re-create, in production system I've had to turn of insert/append of via string for requests which should stop that part of the code running (and hence not trigger the bug).
I also should be free to start testing with v2.1.7, with all my reported issues as soon as I get the lab back from other testers.
Ricky
> Seg faults within HttpTransactHeaders::insert_via_header_in_request
> -------------------------------------------------------------------
>
> Key: TS-729
> URL: https://issues.apache.org/jira/browse/TS-729
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 Sun Blade (AMD64), 64G of RAM.
> Reporter: Ricky Chan
>
> Getting reports of regular seg faults on the traffic server CDN farm, in 1 site 7 out of 10 servers have segfault at least once in 24 hour period. Traffic volume is low around 6 MB/s across all 10 servers. The idea was to put more traffic onto but I need to resolve this first.
> Okay I stipped the binaries so the core dump information isn't as help as it could be but it indicates the issue happened inside insert_via_header_in_request.
> (gdb) bt
> #0 0x00000000006f2ef3 in ink_stack_trace_dump ()
> #1 0x0000000000502f8a in ?? ()
> #2 <signal handler called>
> #3 0x0000000000580903 in HttpTransactHeaders::insert_via_header_in_request ()
> #4 0x0000000000562a3a in HttpTransact::build_request ()
> #5 0x00000000005790f6 in HttpTransact::HandleCacheOpenReadHit ()
> #6 0x0000000000534485 in HttpSM::call_transact_and_set_next_state ()
> #7 0x000000000054b35b in HttpSM::set_next_state ()
> #8 0x0000000000534469 in HttpSM::call_transact_and_set_next_state ()
> #9 0x0000000000536a62 in HttpSM::do_hostdb_lookup ()
> #10 0x000000000054b5db in HttpSM::set_next_state ()
> #11 0x0000000000534469 in HttpSM::call_transact_and_set_next_state ()
> #12 0x000000000054b35b in HttpSM::set_next_state ()
> #13 0x0000000000534469 in HttpSM::call_transact_and_set_next_state ()
> #14 0x000000000054b35b in HttpSM::set_next_state ()
> #15 0x0000000000534469 in HttpSM::call_transact_and_set_next_state ()
> #16 0x0000000000549491 in HttpSM::state_cache_open_read ()
> #17 0x0000000000547e3b in HttpSM::main_handler ()
> #18 0x000000000052457e in HttpCacheSM::state_cache_open_read ()
> #19 0x0000000000694dc4 in CacheVC::openReadStartEarliest ()
> #20 0x000000000066e3c0 in CacheVC::handleReadDone ()
> #21 0x00000000006756fb in AIOCallbackInternal::io_complete ()
> #22 0x00000000006e3614 in EThread::process_event ()
> #23 0x00000000006e3f5b in EThread::execute ()
> #24 0x00000000006e1a72 in ?? ()
> #25 0x00002b9c8dd73fc7 in start_thread () from /lib/libpthread.so.0
> #26 0x00002b9c8f4e564d in clone () from /lib/libc.so.6
> #27 0x0000000000000000 in ?? ()
> The segfault issue was the same for all concerned servers.
> I'll see if I can put the unstripped binaries back on so we can get more information.
> Ricky
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (TS-729) Seg faults within
HttpTransactHeaders::insert_via_header_in_request
Posted by "Leif Hedstrom (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Leif Hedstrom reassigned TS-729:
--------------------------------
Assignee: Leif Hedstrom
> Seg faults within HttpTransactHeaders::insert_via_header_in_request
> -------------------------------------------------------------------
>
> Key: TS-729
> URL: https://issues.apache.org/jira/browse/TS-729
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 Sun Blade (AMD64), 64G of RAM.
> Reporter: Ricky Chan
> Assignee: Leif Hedstrom
>
> Getting reports of regular seg faults on the traffic server CDN farm, in 1 site 7 out of 10 servers have segfault at least once in 24 hour period. Traffic volume is low around 6 MB/s across all 10 servers. The idea was to put more traffic onto but I need to resolve this first.
> Okay I stipped the binaries so the core dump information isn't as help as it could be but it indicates the issue happened inside insert_via_header_in_request.
> (gdb) bt
> #0 0x00000000006f2ef3 in ink_stack_trace_dump ()
> #1 0x0000000000502f8a in ?? ()
> #2 <signal handler called>
> #3 0x0000000000580903 in HttpTransactHeaders::insert_via_header_in_request ()
> #4 0x0000000000562a3a in HttpTransact::build_request ()
> #5 0x00000000005790f6 in HttpTransact::HandleCacheOpenReadHit ()
> #6 0x0000000000534485 in HttpSM::call_transact_and_set_next_state ()
> #7 0x000000000054b35b in HttpSM::set_next_state ()
> #8 0x0000000000534469 in HttpSM::call_transact_and_set_next_state ()
> #9 0x0000000000536a62 in HttpSM::do_hostdb_lookup ()
> #10 0x000000000054b5db in HttpSM::set_next_state ()
> #11 0x0000000000534469 in HttpSM::call_transact_and_set_next_state ()
> #12 0x000000000054b35b in HttpSM::set_next_state ()
> #13 0x0000000000534469 in HttpSM::call_transact_and_set_next_state ()
> #14 0x000000000054b35b in HttpSM::set_next_state ()
> #15 0x0000000000534469 in HttpSM::call_transact_and_set_next_state ()
> #16 0x0000000000549491 in HttpSM::state_cache_open_read ()
> #17 0x0000000000547e3b in HttpSM::main_handler ()
> #18 0x000000000052457e in HttpCacheSM::state_cache_open_read ()
> #19 0x0000000000694dc4 in CacheVC::openReadStartEarliest ()
> #20 0x000000000066e3c0 in CacheVC::handleReadDone ()
> #21 0x00000000006756fb in AIOCallbackInternal::io_complete ()
> #22 0x00000000006e3614 in EThread::process_event ()
> #23 0x00000000006e3f5b in EThread::execute ()
> #24 0x00000000006e1a72 in ?? ()
> #25 0x00002b9c8dd73fc7 in start_thread () from /lib/libpthread.so.0
> #26 0x00002b9c8f4e564d in clone () from /lib/libc.so.6
> #27 0x0000000000000000 in ?? ()
> The segfault issue was the same for all concerned servers.
> I'll see if I can put the unstripped binaries back on so we can get more information.
> Ricky
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-729) Seg faults within
HttpTransactHeaders::insert_via_header_in_request
Posted by "Leif Hedstrom (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13014654#comment-13014654 ]
Leif Hedstrom commented on TS-729:
----------------------------------
I assume there's no chance you will be able to test v2.1.7 ? Also, do you know which line it's crashing on ? And what the values on parameters and local variables are?
> Seg faults within HttpTransactHeaders::insert_via_header_in_request
> -------------------------------------------------------------------
>
> Key: TS-729
> URL: https://issues.apache.org/jira/browse/TS-729
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 Sun Blade (AMD64), 64G of RAM.
> Reporter: Ricky Chan
>
> Getting reports of regular seg faults on the traffic server CDN farm, in 1 site 7 out of 10 servers have segfault at least once in 24 hour period. Traffic volume is low around 6 MB/s across all 10 servers. The idea was to put more traffic onto but I need to resolve this first.
> Okay I stipped the binaries so the core dump information isn't as help as it could be but it indicates the issue happened inside insert_via_header_in_request.
> (gdb) bt
> #0 0x00000000006f2ef3 in ink_stack_trace_dump ()
> #1 0x0000000000502f8a in ?? ()
> #2 <signal handler called>
> #3 0x0000000000580903 in HttpTransactHeaders::insert_via_header_in_request ()
> #4 0x0000000000562a3a in HttpTransact::build_request ()
> #5 0x00000000005790f6 in HttpTransact::HandleCacheOpenReadHit ()
> #6 0x0000000000534485 in HttpSM::call_transact_and_set_next_state ()
> #7 0x000000000054b35b in HttpSM::set_next_state ()
> #8 0x0000000000534469 in HttpSM::call_transact_and_set_next_state ()
> #9 0x0000000000536a62 in HttpSM::do_hostdb_lookup ()
> #10 0x000000000054b5db in HttpSM::set_next_state ()
> #11 0x0000000000534469 in HttpSM::call_transact_and_set_next_state ()
> #12 0x000000000054b35b in HttpSM::set_next_state ()
> #13 0x0000000000534469 in HttpSM::call_transact_and_set_next_state ()
> #14 0x000000000054b35b in HttpSM::set_next_state ()
> #15 0x0000000000534469 in HttpSM::call_transact_and_set_next_state ()
> #16 0x0000000000549491 in HttpSM::state_cache_open_read ()
> #17 0x0000000000547e3b in HttpSM::main_handler ()
> #18 0x000000000052457e in HttpCacheSM::state_cache_open_read ()
> #19 0x0000000000694dc4 in CacheVC::openReadStartEarliest ()
> #20 0x000000000066e3c0 in CacheVC::handleReadDone ()
> #21 0x00000000006756fb in AIOCallbackInternal::io_complete ()
> #22 0x00000000006e3614 in EThread::process_event ()
> #23 0x00000000006e3f5b in EThread::execute ()
> #24 0x00000000006e1a72 in ?? ()
> #25 0x00002b9c8dd73fc7 in start_thread () from /lib/libpthread.so.0
> #26 0x00002b9c8f4e564d in clone () from /lib/libc.so.6
> #27 0x0000000000000000 in ?? ()
> The segfault issue was the same for all concerned servers.
> I'll see if I can put the unstripped binaries back on so we can get more information.
> Ricky
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (TS-729) Seg faults within
HttpTransactHeaders::insert_via_header_in_request
Posted by "Leif Hedstrom (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Leif Hedstrom resolved TS-729.
------------------------------
Resolution: Duplicate
Fix Version/s: (was: 2.1.8)
Marking this as a duplicate of TS-733.
> Seg faults within HttpTransactHeaders::insert_via_header_in_request
> -------------------------------------------------------------------
>
> Key: TS-729
> URL: https://issues.apache.org/jira/browse/TS-729
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 Sun Blade (AMD64), 64G of RAM.
> Reporter: Ricky Chan
> Assignee: Leif Hedstrom
>
> Getting reports of regular seg faults on the traffic server CDN farm, in 1 site 7 out of 10 servers have segfault at least once in 24 hour period. Traffic volume is low around 6 MB/s across all 10 servers. The idea was to put more traffic onto but I need to resolve this first.
> Okay I stipped the binaries so the core dump information isn't as help as it could be but it indicates the issue happened inside insert_via_header_in_request.
> (gdb) bt
> #0 0x00000000006f2ef3 in ink_stack_trace_dump ()
> #1 0x0000000000502f8a in ?? ()
> #2 <signal handler called>
> #3 0x0000000000580903 in HttpTransactHeaders::insert_via_header_in_request ()
> #4 0x0000000000562a3a in HttpTransact::build_request ()
> #5 0x00000000005790f6 in HttpTransact::HandleCacheOpenReadHit ()
> #6 0x0000000000534485 in HttpSM::call_transact_and_set_next_state ()
> #7 0x000000000054b35b in HttpSM::set_next_state ()
> #8 0x0000000000534469 in HttpSM::call_transact_and_set_next_state ()
> #9 0x0000000000536a62 in HttpSM::do_hostdb_lookup ()
> #10 0x000000000054b5db in HttpSM::set_next_state ()
> #11 0x0000000000534469 in HttpSM::call_transact_and_set_next_state ()
> #12 0x000000000054b35b in HttpSM::set_next_state ()
> #13 0x0000000000534469 in HttpSM::call_transact_and_set_next_state ()
> #14 0x000000000054b35b in HttpSM::set_next_state ()
> #15 0x0000000000534469 in HttpSM::call_transact_and_set_next_state ()
> #16 0x0000000000549491 in HttpSM::state_cache_open_read ()
> #17 0x0000000000547e3b in HttpSM::main_handler ()
> #18 0x000000000052457e in HttpCacheSM::state_cache_open_read ()
> #19 0x0000000000694dc4 in CacheVC::openReadStartEarliest ()
> #20 0x000000000066e3c0 in CacheVC::handleReadDone ()
> #21 0x00000000006756fb in AIOCallbackInternal::io_complete ()
> #22 0x00000000006e3614 in EThread::process_event ()
> #23 0x00000000006e3f5b in EThread::execute ()
> #24 0x00000000006e1a72 in ?? ()
> #25 0x00002b9c8dd73fc7 in start_thread () from /lib/libpthread.so.0
> #26 0x00002b9c8f4e564d in clone () from /lib/libc.so.6
> #27 0x0000000000000000 in ?? ()
> The segfault issue was the same for all concerned servers.
> I'll see if I can put the unstripped binaries back on so we can get more information.
> Ricky
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (TS-729) Seg faults within
HttpTransactHeaders::insert_via_header_in_request
Posted by "Leif Hedstrom (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Leif Hedstrom updated TS-729:
-----------------------------
Fix Version/s: 2.1.8
> Seg faults within HttpTransactHeaders::insert_via_header_in_request
> -------------------------------------------------------------------
>
> Key: TS-729
> URL: https://issues.apache.org/jira/browse/TS-729
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 Sun Blade (AMD64), 64G of RAM.
> Reporter: Ricky Chan
> Assignee: Leif Hedstrom
> Fix For: 2.1.8
>
>
> Getting reports of regular seg faults on the traffic server CDN farm, in 1 site 7 out of 10 servers have segfault at least once in 24 hour period. Traffic volume is low around 6 MB/s across all 10 servers. The idea was to put more traffic onto but I need to resolve this first.
> Okay I stipped the binaries so the core dump information isn't as help as it could be but it indicates the issue happened inside insert_via_header_in_request.
> (gdb) bt
> #0 0x00000000006f2ef3 in ink_stack_trace_dump ()
> #1 0x0000000000502f8a in ?? ()
> #2 <signal handler called>
> #3 0x0000000000580903 in HttpTransactHeaders::insert_via_header_in_request ()
> #4 0x0000000000562a3a in HttpTransact::build_request ()
> #5 0x00000000005790f6 in HttpTransact::HandleCacheOpenReadHit ()
> #6 0x0000000000534485 in HttpSM::call_transact_and_set_next_state ()
> #7 0x000000000054b35b in HttpSM::set_next_state ()
> #8 0x0000000000534469 in HttpSM::call_transact_and_set_next_state ()
> #9 0x0000000000536a62 in HttpSM::do_hostdb_lookup ()
> #10 0x000000000054b5db in HttpSM::set_next_state ()
> #11 0x0000000000534469 in HttpSM::call_transact_and_set_next_state ()
> #12 0x000000000054b35b in HttpSM::set_next_state ()
> #13 0x0000000000534469 in HttpSM::call_transact_and_set_next_state ()
> #14 0x000000000054b35b in HttpSM::set_next_state ()
> #15 0x0000000000534469 in HttpSM::call_transact_and_set_next_state ()
> #16 0x0000000000549491 in HttpSM::state_cache_open_read ()
> #17 0x0000000000547e3b in HttpSM::main_handler ()
> #18 0x000000000052457e in HttpCacheSM::state_cache_open_read ()
> #19 0x0000000000694dc4 in CacheVC::openReadStartEarliest ()
> #20 0x000000000066e3c0 in CacheVC::handleReadDone ()
> #21 0x00000000006756fb in AIOCallbackInternal::io_complete ()
> #22 0x00000000006e3614 in EThread::process_event ()
> #23 0x00000000006e3f5b in EThread::execute ()
> #24 0x00000000006e1a72 in ?? ()
> #25 0x00002b9c8dd73fc7 in start_thread () from /lib/libpthread.so.0
> #26 0x00002b9c8f4e564d in clone () from /lib/libc.so.6
> #27 0x0000000000000000 in ?? ()
> The segfault issue was the same for all concerned servers.
> I'll see if I can put the unstripped binaries back on so we can get more information.
> Ricky
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-729) Seg faults within
HttpTransactHeaders::insert_via_header_in_request
Posted by "Leif Hedstrom (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13014730#comment-13014730 ]
Leif Hedstrom commented on TS-729:
----------------------------------
Hmmm, looking into this, I'm not sure you could easily have had a buffer overrun here. However, we found a couple of bugs in this code, which I'm fixing (but I doubt that would cause this crash).
> Seg faults within HttpTransactHeaders::insert_via_header_in_request
> -------------------------------------------------------------------
>
> Key: TS-729
> URL: https://issues.apache.org/jira/browse/TS-729
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 Sun Blade (AMD64), 64G of RAM.
> Reporter: Ricky Chan
>
> Getting reports of regular seg faults on the traffic server CDN farm, in 1 site 7 out of 10 servers have segfault at least once in 24 hour period. Traffic volume is low around 6 MB/s across all 10 servers. The idea was to put more traffic onto but I need to resolve this first.
> Okay I stipped the binaries so the core dump information isn't as help as it could be but it indicates the issue happened inside insert_via_header_in_request.
> (gdb) bt
> #0 0x00000000006f2ef3 in ink_stack_trace_dump ()
> #1 0x0000000000502f8a in ?? ()
> #2 <signal handler called>
> #3 0x0000000000580903 in HttpTransactHeaders::insert_via_header_in_request ()
> #4 0x0000000000562a3a in HttpTransact::build_request ()
> #5 0x00000000005790f6 in HttpTransact::HandleCacheOpenReadHit ()
> #6 0x0000000000534485 in HttpSM::call_transact_and_set_next_state ()
> #7 0x000000000054b35b in HttpSM::set_next_state ()
> #8 0x0000000000534469 in HttpSM::call_transact_and_set_next_state ()
> #9 0x0000000000536a62 in HttpSM::do_hostdb_lookup ()
> #10 0x000000000054b5db in HttpSM::set_next_state ()
> #11 0x0000000000534469 in HttpSM::call_transact_and_set_next_state ()
> #12 0x000000000054b35b in HttpSM::set_next_state ()
> #13 0x0000000000534469 in HttpSM::call_transact_and_set_next_state ()
> #14 0x000000000054b35b in HttpSM::set_next_state ()
> #15 0x0000000000534469 in HttpSM::call_transact_and_set_next_state ()
> #16 0x0000000000549491 in HttpSM::state_cache_open_read ()
> #17 0x0000000000547e3b in HttpSM::main_handler ()
> #18 0x000000000052457e in HttpCacheSM::state_cache_open_read ()
> #19 0x0000000000694dc4 in CacheVC::openReadStartEarliest ()
> #20 0x000000000066e3c0 in CacheVC::handleReadDone ()
> #21 0x00000000006756fb in AIOCallbackInternal::io_complete ()
> #22 0x00000000006e3614 in EThread::process_event ()
> #23 0x00000000006e3f5b in EThread::execute ()
> #24 0x00000000006e1a72 in ?? ()
> #25 0x00002b9c8dd73fc7 in start_thread () from /lib/libpthread.so.0
> #26 0x00002b9c8f4e564d in clone () from /lib/libc.so.6
> #27 0x0000000000000000 in ?? ()
> The segfault issue was the same for all concerned servers.
> I'll see if I can put the unstripped binaries back on so we can get more information.
> Ricky
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira