You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@trafficserver.apache.org by "Ricky Chan (JIRA)" <ji...@apache.org> on 2011/04/08 11:10:05 UTC
[jira] [Created] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
segfault in mime_hdr_set_accelerators_and_presence_bits
-------------------------------------------------------
Key: TS-733
URL: https://issues.apache.org/jira/browse/TS-733
Project: Traffic Server
Issue Type: Bug
Components: MIME
Affects Versions: 2.0.1
Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
Reporter: Ricky Chan
We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
I got this segfault 6 times this morning, and it appears from the same client too.
I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
#0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
66 fp = (void **) (*fp);
(gdb) bt
#0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
#1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
#2 <signal handler called>
#3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
#4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
#5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
#6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
#7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
#8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
#9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
#10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
#11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
#12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
#13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
#14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
#15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
#16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
#17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
#18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
#19 0x00002abbb74f064d in clone () from /lib/libc.so.6
#20 0x0000000000000000 in ?? ()
(gdb) frame 3
#3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
warning: Source file is more recent than executable.
482 if (field->m_wks_idx < 0)
(gdb) p *field
$1 = {
m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
Posted by "Ricky Chan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ricky Chan updated TS-733:
--------------------------
Attachment: disable_dup.patch
This is a patch for 2.0.1 which on the headers being copied disabled any reference to duplication and set's itself as DUP_HEAD.
This prevents a condition in 2.0.1 where m_next_dup points to a location which is no longer valid and crashes the server (segfault). Although it does mean duplicates are no longer handled.
> segfault in mime_hdr_set_accelerators_and_presence_bits
> -------------------------------------------------------
>
> Key: TS-733
> URL: https://issues.apache.org/jira/browse/TS-733
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
> Reporter: Ricky Chan
> Labels: MIME, segfault
> Fix For: 2.1.8
>
> Attachments: disable_dup.patch
>
>
> We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
> Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
> I got this segfault 6 times this morning, and it appears from the same client too.
> I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> 66 fp = (void **) (*fp);
> (gdb) bt
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> #1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
> #2 <signal handler called>
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> #4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
> #5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
> response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
> #6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
> #7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
> #8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
> #9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
> #10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
> #11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
> #12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
> #13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
> #14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
> #15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
> #16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
> #17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
> #18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
> #19 0x00002abbb74f064d in clone () from /lib/libc.so.6
> #20 0x0000000000000000 in ?? ()
> (gdb) frame 3
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> warning: Source file is more recent than executable.
> 482 if (field->m_wks_idx < 0)
> (gdb) p *field
> $1 = {
> m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
> m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
> m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
Posted by "Ricky Chan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13018756#comment-13018756 ]
Ricky Chan commented on TS-733:
-------------------------------
24 hours since new patch applied and no incidents in production.
I'm not familiar with how ATS isolates threads not to access/modify shared data, but it seems likely that under certain conditions an MIMEField address is being read AND changed by different concurrent threads.
I saw this in one of the last segfauits, when I can see I clearly set m_next_dup in the _adjust function to NULL (0x0) then next thing it does is set m_flags to DUP_HEAD, however it the dump it shows m_next_dup as NULL but no flags had been set yet.
So I decided to do a new patch which forces _attach and _detach to treat it as if it is DUP_HEAD and m_next_dup as as if the former is always set and the later as if it's always NULL.
> segfault in mime_hdr_set_accelerators_and_presence_bits
> -------------------------------------------------------
>
> Key: TS-733
> URL: https://issues.apache.org/jira/browse/TS-733
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
> Reporter: Ricky Chan
> Labels: MIME, segfault
> Fix For: 2.1.8
>
> Attachments: disable_dup.patch-2
>
>
> We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
> Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
> I got this segfault 6 times this morning, and it appears from the same client too.
> I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> 66 fp = (void **) (*fp);
> (gdb) bt
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> #1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
> #2 <signal handler called>
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> #4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
> #5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
> response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
> #6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
> #7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
> #8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
> #9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
> #10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
> #11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
> #12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
> #13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
> #14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
> #15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
> #16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
> #17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
> #18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
> #19 0x00002abbb74f064d in clone () from /lib/libc.so.6
> #20 0x0000000000000000 in ?? ()
> (gdb) frame 3
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> warning: Source file is more recent than executable.
> 482 if (field->m_wks_idx < 0)
> (gdb) p *field
> $1 = {
> m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
> m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
> m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
Posted by "Ricky Chan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ricky Chan updated TS-733:
--------------------------
Attachment: disable_dup.patch
Improved version of the patch attached.
> segfault in mime_hdr_set_accelerators_and_presence_bits
> -------------------------------------------------------
>
> Key: TS-733
> URL: https://issues.apache.org/jira/browse/TS-733
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
> Reporter: Ricky Chan
> Labels: MIME, segfault
> Fix For: 2.1.8
>
> Attachments: disable_dup.patch
>
>
> We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
> Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
> I got this segfault 6 times this morning, and it appears from the same client too.
> I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> 66 fp = (void **) (*fp);
> (gdb) bt
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> #1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
> #2 <signal handler called>
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> #4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
> #5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
> response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
> #6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
> #7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
> #8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
> #9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
> #10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
> #11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
> #12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
> #13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
> #14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
> #15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
> #16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
> #17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
> #18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
> #19 0x00002abbb74f064d in clone () from /lib/libc.so.6
> #20 0x0000000000000000 in ?? ()
> (gdb) frame 3
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> warning: Source file is more recent than executable.
> 482 if (field->m_wks_idx < 0)
> (gdb) p *field
> $1 = {
> m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
> m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
> m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
Posted by "Ricky Chan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13018109#comment-13018109 ]
Ricky Chan commented on TS-733:
-------------------------------
I mean it hasn't crashed in over 8 hours with the minimal patch.
> segfault in mime_hdr_set_accelerators_and_presence_bits
> -------------------------------------------------------
>
> Key: TS-733
> URL: https://issues.apache.org/jira/browse/TS-733
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
> Reporter: Ricky Chan
> Labels: MIME, segfault
> Fix For: 2.1.8
>
> Attachments: disable_dup.patch
>
>
> We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
> Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
> I got this segfault 6 times this morning, and it appears from the same client too.
> I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> 66 fp = (void **) (*fp);
> (gdb) bt
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> #1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
> #2 <signal handler called>
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> #4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
> #5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
> response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
> #6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
> #7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
> #8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
> #9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
> #10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
> #11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
> #12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
> #13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
> #14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
> #15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
> #16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
> #17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
> #18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
> #19 0x00002abbb74f064d in clone () from /lib/libc.so.6
> #20 0x0000000000000000 in ?? ()
> (gdb) frame 3
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> warning: Source file is more recent than executable.
> 482 if (field->m_wks_idx < 0)
> (gdb) p *field
> $1 = {
> m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
> m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
> m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
Posted by "Ricky Chan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017432#comment-13017432 ]
Ricky Chan commented on TS-733:
-------------------------------
Okay I'm able to replicate this, this looks like it could be related to the insert via request bug I logged earlier:
I made 20 concurrent request for this asset endlessly and with in a few seconds, ATS will segfault:
GET /1K HTTP/1.1.
Host: mytest.lab.com
User-Agent: Mozilla/4.0 (compatible;).
Accept: text/html, */*.
Accept-Encoding: gzip, deflate, identity.
If-Modified-Since: Fri, 08 Apr 2011 11:09:54 GMT.
Via: 1.1 AKmdrL2CacheBC10.telecom.co.nz.
Cache-Control: max-stale=0.
Connection: Keep-Alive.
Via: 1.1 AKmdrL2CacheBC10.telecom.co.nz.
X-BlueCoat-Via: B8C344C089BFFBD7.
It looks like it's the dual Via header that's the problem!! It's probably the root cause of many of the errors I've reported.
Of course this is a client (forward proxy) bug, but it should be handled gracefully.
Ricky
> segfault in mime_hdr_set_accelerators_and_presence_bits
> -------------------------------------------------------
>
> Key: TS-733
> URL: https://issues.apache.org/jira/browse/TS-733
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
> Reporter: Ricky Chan
> Labels: MIME, segfault
>
> We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
> Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
> I got this segfault 6 times this morning, and it appears from the same client too.
> I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> 66 fp = (void **) (*fp);
> (gdb) bt
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> #1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
> #2 <signal handler called>
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> #4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
> #5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
> response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
> #6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
> #7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
> #8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
> #9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
> #10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
> #11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
> #12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
> #13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
> #14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
> #15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
> #16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
> #17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
> #18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
> #19 0x00002abbb74f064d in clone () from /lib/libc.so.6
> #20 0x0000000000000000 in ?? ()
> (gdb) frame 3
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> warning: Source file is more recent than executable.
> 482 if (field->m_wks_idx < 0)
> (gdb) p *field
> $1 = {
> m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
> m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
> m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
Posted by "Ricky Chan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13018053#comment-13018053 ]
Ricky Chan commented on TS-733:
-------------------------------
I was able to crash it on 2.1.7 using ab instead of my script. I used this ab setting to make it more likely to happen.
/usr/sbin/ab -X 127.0.0.1:80 -H 'Via: abc' -H 'Via: abc' -H 'Via: abc' -H 'Via: abc' -H 'Via: abc' -H 'Via: abc' -H 'Via: abc' -H 'Via: abc' -c 200 -n 1000000000 http://www.test1.com/1K
while true; do curl -X PURGE -v -H 'Host: www.test1.com' http://127.0.0.1/1K; done
AB aborted when it crashed. (Yes I am doing the PURGE's locally and taking up some CPU cycles for that process, but I would expect it to slow down rather than crash).
The error is the same as before in core dump. This time ATS was able to also to a STACK Trace in traffic.out:
NOTE: Traffic Server received Sig 11: Segmentation fault
/usr/bin/traffic_server - STACK TRACE:
/usr/bin/traffic_server[0x53616f]
/lib/libpthread.so.0[0x2b86c9042a80]
[0x2aaac0126048]
/usr/bin/traffic_server(_ZN12Continuation11handleEventEiPv+0x6f)[0x4ee0ef]
/usr/bin/traffic_server(_ZN5Cache9open_readEP12ContinuationP7INK_MD5P7HTTPHdrP21CacheLookupHttpConfig13CacheFragTypePci+0x584)[0x6d2914]
/usr/bin/traffic_server(_ZN5Cache9open_readEP12ContinuationP3URLP7HTTPHdrP21CacheLookupHttpConfig13CacheFragType+0x85)[0x6b3b97]
/usr/bin/traffic_server(_ZN14CacheProcessor9open_readEP12ContinuationP3URLP7HTTPHdrP21CacheLookupHttpConfigl13CacheFragType+0xb3)[0x6a44f7]
/usr/bin/traffic_server(_ZN11HttpCacheSM18do_cache_open_readEv+0xdf)[0x5691d5]
/usr/bin/traffic_server(_ZN11HttpCacheSM9open_readEP3URLP7HTTPHdrP21CacheLookupHttpConfigl+0xe2)[0x569328]
/usr/bin/traffic_server(_ZN6HttpSM24do_cache_lookup_and_readEv+0x1b1)[0x584acf]
/usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x5dd)[0x590053]
/usr/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x14a)[0x57d5e0]
/usr/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x130)[0x59178e]
/usr/bin/traffic_server(_ZN6HttpSM14do_api_calloutEv+0x39)[0x5958a9]
/usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x6f)[0x58fae5]
/usr/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x14a)[0x57d5e0]
/usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x19c)[0x58fc12]
/usr/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x14a)[0x57d5e0]
/usr/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x130)[0x59178e]
/usr/bin/traffic_server(_ZN6HttpSM14do_api_calloutEv+0x39)[0x5958a9]
/usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x6f)[0x58fae5]
/usr/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x14a)[0x57d5e0]
/usr/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x130)[0x59178e]
/usr/bin/traffic_server(_ZN6HttpSM14do_api_calloutEv+0x39)[0x5958a9]
/usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x6f)[0x58fae5]
/usr/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x14a)[0x57d5e0]
/usr/bin/traffic_server(_ZN6HttpSM32state_read_client_request_headerEiPv+0x940)[0x591534]
/usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x223)[0x58ae67]
/usr/bin/traffic_server(_ZN12Continuation11handleEventEiPv+0x6f)[0x4ee0ef]
/usr/bin/traffic_server[0x6faff1]
/usr/bin/traffic_server[0x6fb952]
/usr/bin/traffic_server(_ZN18UnixNetVConnection11net_read_ioEP10NetHandlerP7EThread+0x25)[0x6fbaaf]
/usr/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x758)[0x6f546c]
/usr/bin/traffic_server(_ZN12Continuation11handleEventEiPv+0x6f)[0x4ee0ef]
/usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x129)[0x71e9df]
/usr/bin/traffic_server(_ZN7EThread7executeEv+0x3b9)[0x71ef0d]
/usr/bin/traffic_server[0x71e26e]
/lib/libpthread.so.0[0x2b86c903afc7]
/lib/libc.so.6(clone+0x6d)[0x2b86cb22164d]
/lib/libc.so.6(clone+0x6d)[0x2b86cb22164d]
> segfault in mime_hdr_set_accelerators_and_presence_bits
> -------------------------------------------------------
>
> Key: TS-733
> URL: https://issues.apache.org/jira/browse/TS-733
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
> Reporter: Ricky Chan
> Labels: MIME, segfault
> Fix For: 2.1.8
>
>
> We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
> Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
> I got this segfault 6 times this morning, and it appears from the same client too.
> I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> 66 fp = (void **) (*fp);
> (gdb) bt
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> #1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
> #2 <signal handler called>
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> #4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
> #5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
> response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
> #6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
> #7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
> #8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
> #9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
> #10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
> #11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
> #12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
> #13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
> #14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
> #15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
> #16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
> #17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
> #18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
> #19 0x00002abbb74f064d in clone () from /lib/libc.so.6
> #20 0x0000000000000000 in ?? ()
> (gdb) frame 3
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> warning: Source file is more recent than executable.
> 482 if (field->m_wks_idx < 0)
> (gdb) p *field
> $1 = {
> m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
> m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
> m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
Posted by "Ricky Chan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ricky Chan updated TS-733:
--------------------------
Attachment: (was: disable_dup.patch)
> segfault in mime_hdr_set_accelerators_and_presence_bits
> -------------------------------------------------------
>
> Key: TS-733
> URL: https://issues.apache.org/jira/browse/TS-733
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
> Reporter: Ricky Chan
> Labels: MIME, segfault
> Fix For: 2.1.8
>
> Attachments: disable_dup.patch-2
>
>
> We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
> Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
> I got this segfault 6 times this morning, and it appears from the same client too.
> I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> 66 fp = (void **) (*fp);
> (gdb) bt
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> #1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
> #2 <signal handler called>
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> #4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
> #5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
> response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
> #6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
> #7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
> #8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
> #9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
> #10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
> #11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
> #12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
> #13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
> #14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
> #15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
> #16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
> #17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
> #18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
> #19 0x00002abbb74f064d in clone () from /lib/libc.so.6
> #20 0x0000000000000000 in ?? ()
> (gdb) frame 3
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> warning: Source file is more recent than executable.
> 482 if (field->m_wks_idx < 0)
> (gdb) p *field
> $1 = {
> m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
> m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
> m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
Posted by "Leif Hedstrom (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017569#comment-13017569 ]
Leif Hedstrom commented on TS-733:
----------------------------------
Yeah, stable is a "misnomer", it means nothing is changing ;). V2.1.7 ought to be a heck of a lot better than v2.0.1 is, but, it probably has other bugs.
Please try v2.1.7 (or trunk) and let us know how it goes.
> segfault in mime_hdr_set_accelerators_and_presence_bits
> -------------------------------------------------------
>
> Key: TS-733
> URL: https://issues.apache.org/jira/browse/TS-733
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
> Reporter: Ricky Chan
> Labels: MIME, segfault
>
> We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
> Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
> I got this segfault 6 times this morning, and it appears from the same client too.
> I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> 66 fp = (void **) (*fp);
> (gdb) bt
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> #1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
> #2 <signal handler called>
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> #4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
> #5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
> response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
> #6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
> #7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
> #8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
> #9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
> #10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
> #11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
> #12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
> #13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
> #14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
> #15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
> #16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
> #17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
> #18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
> #19 0x00002abbb74f064d in clone () from /lib/libc.so.6
> #20 0x0000000000000000 in ?? ()
> (gdb) frame 3
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> warning: Source file is more recent than executable.
> 482 if (field->m_wks_idx < 0)
> (gdb) p *field
> $1 = {
> m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
> m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
> m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
Posted by "Ricky Chan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017846#comment-13017846 ]
Ricky Chan commented on TS-733:
-------------------------------
I got a new segfault, for a different m_wks_idx index. This one is 36.
Below is slightly modified to hide real host header :-)
#0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
#1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
#2 <signal handler called>
#3 mime_hdr_field_detach (mh=0x65eccc8, field=0x65ecde0, detach_all_dups=false) at MIME.cc:482
#4 0x0000000000601eae in mime_hdr_field_delete (heap=0x65ecc10, mh=0x65eccc8, field=0x65ecde0, delete_all_dups=true) at MIME.cc:1737
#5 0x000000000057e796 in HttpTransactHeaders::copy_header_fields (src_hdr=<value optimized out>, new_hdr=0x2aaaba53a090, retain_proxy_auth_hdrs=false,
date=0) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
#6 0x00000000005629ae in HttpTransact::build_request (s=0x2aaaba5399b0, base_request=0x2aaaba53a020, outgoing_request=0x2aaaba53a090,
outgoing_version=<value optimized out>) at HttpTransact.cc:9086
#7 0x00000000005790f6 in HttpTransact::HandleCacheOpenReadHit (s=0x2aaaba5399b0) at HttpTransact.cc:3009
#8 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba539930, f=0x65ecde0) at HttpSM.cc:7190
#9 0x000000000054b35b in HttpSM::set_next_state (this=0x2aaaba539930) at HttpSM.cc:7232
#10 0x0000000000534469 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba539930, f=<value optimized out>) at HttpSM.cc:7198
#11 0x0000000000536a62 in HttpSM::do_hostdb_lookup (this=0x2aaaba539930) at HttpSM.cc:4079
#12 0x000000000054b5db in HttpSM::set_next_state (this=0x2aaaba539930) at HttpSM.cc:7278
#13 0x0000000000534469 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba539930, f=<value optimized out>) at HttpSM.cc:7198
#14 0x000000000054b35b in HttpSM::set_next_state (this=0x2aaaba539930) at HttpSM.cc:7232
#15 0x0000000000534469 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba539930, f=<value optimized out>) at HttpSM.cc:7198
#16 0x000000000054b35b in HttpSM::set_next_state (this=0x2aaaba539930) at HttpSM.cc:7232
#17 0x0000000000534469 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba539930, f=<value optimized out>) at HttpSM.cc:7198
#18 0x0000000000549491 in HttpSM::state_cache_open_read (this=0x2aaaba539930, event=1102, data=0xbf281f0) at HttpSM.cc:2605
#19 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba539930, event=1102, data=0xbf281f0) at HttpSM.cc:2683
#20 0x000000000052457e in HttpCacheSM::state_cache_open_read (this=0x2aaaba53b818, event=<value optimized out>, data=0xbf281f0)
at ../../iocore/eventsystem/I_Continuation.h:147
#21 0x0000000000694df4 in CacheVC::openReadStartEarliest (this=0xbf281f0, event=<value optimized out>, e=<value optimized out>) at CacheRead.cc:761
#22 0x000000000066e3f0 in CacheVC::handleReadDone (this=0xbf281f0, event=<value optimized out>, e=<value optimized out>)
at ../../iocore/eventsystem/I_Continuation.h:147
#23 0x000000000067572b in AIOCallbackInternal::io_complete (this=<value optimized out>, event=<value optimized out>, data=<value optimized out>)
at ../../iocore/eventsystem/I_Continuation.h:147
#24 0x00000000006e3644 in EThread::process_event (this=0x2aaaac64c010, e=0x33c8200, calling_code=1) at I_Continuation.h:147
#25 0x00000000006e3f8b in EThread::execute (this=0x2aaaac64c010) at UnixEThread.cc:168
#26 0x00000000006e1aa2 in spawn_thread_internal (a=0x2a93210) at Thread.cc:85
#27 0x00002b6dca298fc7 in start_thread () from /lib/libpthread.so.0
#28 0x00002b6dcba0a64d in clone () from /lib/libc.so.6
#29 0x0000000000000000 in ?? ()
(gdb) frame 3
#3 mime_hdr_field_detach (mh=0x65eccc8, field=0x65ecde0, detach_all_dups=false) at MIME.cc:482
482 MIME.cc: No such file or directory.
in MIME.cc
(gdb) p *field
$3 = {
m_ptr_name = 0x2aaaceca918e "Keep-Alive: 115\r\nUser-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.16) Gecko/20110319 Firefox/3.6.16\r\nConnection: keep-alive\r\nKeep-Alive: 115\r\n\r\n90/c_3.gif HTTP/1.1\r\nHost: www.teste"...,
m_ptr_value = 0x2aaaceca919a "115\r\nUser-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.16) Gecko/20110319 Firefox/3.6.16\r\nConnection: keep-alive\r\nKeep-Alive: 115\r\n\r\n90/c_3.gif HTTP/1.1\r\nHost: www.testetf1.com\r\nUse"..., m_next_dup = 0x2aab065ece40, m_wks_idx = 36,
m_len_name = 10, m_len_value = 3, m_n_v_raw_printable = 1 '\001', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
(gdb) p *mh
$4 = {<HdrHeapObjImpl> = {m_type = 4, m_length = 600, m_obj_flags = 0}, m_presence_bits = 2251800904730639, m_slot_accelerators = {4294914576, 4283432952,
4294967295, 4294443007}, m_cooked_stuff = {m_cache_control = {m_mask = 0, m_secs_max_age = 0, m_secs_s_maxage = 0, m_secs_max_stale = 0,
m_secs_min_fresh = 0}, m_pragma = {m_no_cache = false}}, m_fblock_list_tail = 0x65ecd08, m_first_fblock = {<HdrHeapObjImpl> = {m_type = 5,
m_length = 536, m_obj_flags = 0}, m_next = 0x0, m_freetop = 10, m_field_slots = {{
m_ptr_name = 0x2aaaceca901b "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept-Encoding: deflate, gzip, identity\r\nAccept-Language: en-gb,en;q=0.5\r\nCook"...,
m_ptr_value = 0x2aaaceca9023 "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept-Encoding: deflate, gzip, identity\r\nAccept-Language: en-gb,en;q=0.5\r\nCookie: rsi_"..., m_next_dup = 0x0, m_wks_idx = 0, m_len_name = 6,
m_len_value = 63, m_n_v_raw_printable = 1 '\001', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}, {
m_ptr_name = 0x2aaaceca9064 "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept-Encoding: deflate, gzip, identity\r\nAccept-Language: en-gb,en;q=0.5\r\nCookie: rsi_segs=G07611_10219|G07611_10469|G07611_10484|G07611_10487|G07611_1"...,
m_ptr_value = 0x2aaaceca9074 "ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept-Encoding: deflate, gzip, identity\r\nAccept-Language: en-gb,en;q=0.5\r\nCookie: rsi_segs=G07611_10219|G07611_10469|G07611_10484|G07611_10487|G07611_10491; _em_vt=0a9"..., m_next_dup = 0x0, m_wks_idx = 1, m_len_name = 14,
m_len_value = 30, m_n_v_raw_printable = 1 '\001', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}, {
m_ptr_name = 0x2aaaceca9094 "Accept-Encoding: deflate, gzip, identity\r\nAccept-Language: en-gb,en;q=0.5\r\nCookie: rsi_segs=G07611_10219|G07611_10469|G07611_10484|G07611_10487|G07611_10491; _em_vt=0a9dde6ceeab7a2d189dd364a4784d9197e"...,
m_ptr_value = 0x2aaaceca90a5 "deflate, gzip, identity\r\nAccept-Language: en-gb,en;q=0.5\r\nCookie: rsi_segs=G07611_10219|G07611_10469|G07611_10484|G07611_10487|G07611_10491; _em_vt=0a9dde6ceeab7a2d189dd364a4784d9197ea8a91a2-977621484"..., m_next_dup = 0x0, m_wks_idx = 2, m_len_name = 15,
m_len_value = 23, m_n_v_raw_printable = 1 '\001', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}, {
m_ptr_name = 0x2aaaceca90be "Accept-Language: en-gb,en;q=0.5\r\nCookie: rsi_segs=G07611_10219|G07611_10469|G07611_10484|G07611_10487|G07611_10491; _em_vt=0a9dde6ceeab7a2d189dd364a4784d9197ea8a91a2-977621484da018ae\r\nHost: www.testet"...,
m_ptr_value = 0x2aaaceca90cf "en-gb,en;q=0.5\r\nCookie: rsi_segs=G07611_10219|G07611_10469|G07611_10484|G07611_10487|G07611_10491; _em_vt=0a9dde6ceeab7a2d189dd364a4784d9197ea8a91a2-977621484da018ae\r\nHost: www.testetf1.com\r\nKeep-Aliv"..., m_next_dup = 0x0, m_wks_idx = 3, m_len_name = 15,
m_len_value = 14, m_n_v_raw_printable = 1 '\001', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}, {
m_ptr_name = 0x2aaaceca90df "Cookie: rsi_segs=G07611_10219|G07611_10469|G07611_10484|G07611_10487|G07611_10491; _em_vt=0a9dde6ceeab7a2d189dd364a4784d9197ea8a91a2-977621484da018ae\r\nHost: www.testetf1.com\r\nKeep-Alive: 115\r\nUser-Age"...,
m_ptr_value = 0x2aaaceca90e7 "rsi_segs=G07611_10219|G07611_10469|G07611_10484|G07611_10487|G07611_10491; _em_vt=0a9dde6ceeab7a2d189dd364a4784d9197ea8a91a2-977621484da018ae\r\nHost: www.testetf1.com\r\nKeep-Alive: 115\r\nUser-Agent: Mozi"..., m_next_dup = 0x0, m_wks_idx = 22, m_len_name = 6,
m_len_value = 141, m_n_v_raw_printable = 1 '\001', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}, {
m_ptr_name = 0x2aaaceca9176 "Host: www.testetf1.com\r\nKeep-Alive: 115\r\nUser-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.16) Gecko/20110319 Firefox/3.6.16\r\nConnection: keep-alive\r\nKeep-Alive: 115\r\n\r\n90/c_3.gif H"...,
m_ptr_value = 0x2aaaceca917c "www.testetf1.com\r\nKeep-Alive: 115\r\nUser-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.16) Gecko/20110319 Firefox/3.6.16\r\nConnection: keep-alive\r\nKeep-Alive: 115\r\n\r\n90/c_3.gif HTTP/1."..., m_next_dup = 0x0, m_wks_idx = 30, m_len_name = 4,
m_len_value = 16, m_n_v_raw_printable = 1 '\001', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}, {
m_ptr_name = 0x2aaaceca918e "Keep-Alive: 115\r\nUser-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.16) Gecko/20110319 Firefox/3.6.16\r\nConnection: keep-alive\r\nKeep-Alive: 115\r\n\r\n90/c_3.gif HTTP/1.1\r\nHost: www.teste"...,
m_ptr_value = 0x2aaaceca919a "115\r\nUser-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.16) Gecko/20110319 Firefox/3.6.16\r\nConnection: keep-alive\r\nKeep-Alive: 115\r\n\r\n90/c_3.gif HTTP/1.1\r\nHost: www.testetf1.com\r\nUse"..., m_next_dup = 0x2aab065ece40, m_wks_idx = 36,
m_len_name = 10, m_len_value = 3, m_n_v_raw_printable = 1 '\001', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}, {
m_ptr_name = 0x2aaaceca919f "User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.16) Gecko/20110319 Firefox/3.6.16\r\nConnection: keep-alive\r\nKeep-Alive: 115\r\n\r\n90/c_3.gif HTTP/1.1\r\nHost: www.testetf1.com\r\nUser-Age"...,
m_ptr_value = 0x2aaaceca91ab "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.16) Gecko/20110319 Firefox/3.6.16\r\nConnection: keep-alive\r\nKeep-Alive: 115\r\n\r\n90/c_3.gif HTTP/1.1\r\nHost: www.testetf1.com\r\nUser-Agent: Mozilla/"..., m_next_dup = 0x0, m_wks_idx = 65, m_len_name = 10,
m_len_value = 90, m_n_v_raw_printable = 1 '\001', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}, {
m_ptr_name = 0x2aaaceca9207 "Connection: keep-alive\r\nKeep-Alive: 115\r\n\r\n90/c_3.gif HTTP/1.1\r\nHost: www.testetf1.com\r\nUser-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.16) Gecko/20110319 Firefox/3.6.16\r\nAccept: "...,
m_ptr_value = 0x2aaaceca9213 "keep-alive\r\nKeep-Alive: 115\r\n\r\n90/c_3.gif HTTP/1.1\r\nHost: www.testetf1.com\r\nUser-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.16) Gecko/20110319 Firefox/3.6.16\r\nAccept: image/png,im"..., m_next_dup = 0x0, m_wks_idx = 12, m_len_name = 10,
m_len_value = 10, m_n_v_raw_printable = 1 '\001', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}, {
m_ptr_name = 0x2aaaceca921f "Keep-Alive: 115\r\n\r\n90/c_3.gif HTTP/1.1\r\nHost: www.testetf1.com\r\nUser-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.16) Gecko/20110319 Firefox/3.6.16\r\nAccept: image/png,image/*;q=0.8,"...,
m_ptr_value = 0x2aaaceca922b "115\r\n\r\n90/c_3.gif HTTP/1.1\r\nHost: www.testetf1.com\r\nUser-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.16) Gecko/20110319 Firefox/3.6.16\r\nAccept: image/png,image/*;q=0.8,*/*;q=0.5\r\nA"..., m_next_dup = 0x0, m_wks_idx = 36, m_len_name = 10,
m_len_value = 3, m_n_v_raw_printable = 1 '\001', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 0 '\0'}, {
m_ptr_name = 0x7cc6168 "Transfer-Encoding: chunked\r\nConnection: keep-alive\r\n\r\nApr 2011 09:42:01 GMT\r\nVia: 1.0 squid15.teamtalk.com:80 (squid/2.HEAD-CVS)\r\nConnection: keep-alive\r\n\r\n/*\n * Thickbox 3.1 - One Box To Rule Them Al"...,
m_ptr_value = 0x7cc617b "chunked\r\nConnection: keep-alive\r\n\r\nApr 2011 09:42:01 GMT\r\nVia: 1.0 squid15.teamtalk.com:80 (squid/2.HEAD-CVS)\r\nConnection: keep-alive\r\n\r\n/*\n * Thickbox 3.1 - One Box To Rule Them All.\n * By Cody Lindl"..., m_next_dup = 0x0, m_wks_idx = 63, m_len_name = 17,
m_len_value = 7, m_n_v_raw_printable = 1 '\001', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 3 '\003', m_flags = 1 '\001'}, {
m_ptr_name = 0x7cc6184 "Connection: keep-alive\r\n\r\nApr 2011 09:42:01 GMT\r\nVia: 1.0 squid15.teamtalk.com:80 (squid/2.HEAD-CVS)\r\nConnection: keep-alive\r\n\r\n/*\n * Thickbox 3.1 - One Box To Rule Them All.\n * By Cody Lindley (http:"...,
m_ptr_value = 0x7cc6190 "keep-alive\r\n\r\nApr 2011 09:42:01 GMT\r\nVia: 1.0 squid15.teamtalk.com:80 (squid/2.HEAD-CVS)\r\nConnection: keep-alive\r\n\r\n/*\n * Thickbox 3.1 - One Box To Rule Them All.\n * By Cody Lindley (http://www.codyli"..., m_next_dup = 0x0, m_wks_idx = 12, m_len_name = 10,
m_len_value = 10, m_n_v_raw_printable = 1 '\001', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}, {
m_ptr_name = 0x2aaab46d4b81 "0.244.132.255omClient-ip212.183.128.69X-Forwarded-For212.183.128.69If-Modified-SinceMon, 04 Apr 2011 10:48:28 GMTIf-None-Match\"20d1a-b011-4a0157ff6c7c5\"2.gif",
m_ptr_value = 0x2aaab46d4b92 "ient-ip212.183.128.69X-Forwarded-For212.183.128.69If-Modified-SinceMon, 04 Apr 2011 10:48:28 GMTIf-None-Match\"20d1a-b011-4a0157ff6c7c5\"2.gif", m_next_dup = 0x0, m_wks_idx = 63, m_len_name = 17, m_len_value = 7, m_n_v_raw_printable = 0 '\0',
m_n_v_raw_printable_pad = 0 '\0', m_readiness = 2 '\002', m_flags = 1 '\001'}, {
m_ptr_name = 0x2aaab46d4b99 "212.183.128.69X-Forwarded-For212.183.128.69If-Modified-SinceMon, 04 Apr 2011 10:48:28 GMTIf-None-Match\"20d1a-b011-4a0157ff6c7c5\"2.gif",
m_ptr_value = 0x2aaab46d4ba3 "8.69X-Forwarded-For212.183.128.69If-Modified-SinceMon, 04 Apr 2011 10:48:28 GMTIf-None-Match\"20d1a-b011-4a0157ff6c7c5\"2.gif", m_next_dup = 0x0, m_wks_idx = 12, m_len_name = 10, m_len_value = 10, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 0 '\0',
m_readiness = 2 '\002', m_flags = 1 '\001'}, {
m_ptr_name = 0x2aaab46d03a5 "liveATS/2.0.125d7c-2c1c-490d5d2540c80\"Connectionkeep-aliveServerATS/2.0.1MTIf-None-Match\"1ed1d-c88-4a01598a14c14\"/c_3.gif", m_ptr_value = 0x2aaab46d03b4 "d7c-2c1c-490d5d2540c80\"Connectionkeep-aliveServerATS/2.0.1MTIf-None-Match\"1ed1d-c88-4a01598a14c14\"/c_3.gif",
m_next_dup = 0x0, m_wks_idx = 122, m_len_name = 15, m_len_value = 12, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 0 '\0',
m_readiness = 2 '\002', m_flags = 1 '\001'}, {
m_ptr_name = 0x2aaab46d03c0 "d5d2540c80\"Connectionkeep-aliveServerATS/2.0.1MTIf-None-Match\"1ed1d-c88-4a01598a14c14\"/c_3.gif",
m_ptr_value = 0x2aaab46d03d1 "tionkeep-aliveServerATS/2.0.1MTIf-None-Match\"1ed1d-c88-4a01598a14c14\"/c_3.gif", m_next_dup = 0x0, m_wks_idx = 32,
m_len_name = 17, m_len_value = 29, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 0 '\0', m_readiness = 2 '\002', m_flags = 1 '\001'}}}}
> segfault in mime_hdr_set_accelerators_and_presence_bits
> -------------------------------------------------------
>
> Key: TS-733
> URL: https://issues.apache.org/jira/browse/TS-733
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
> Reporter: Ricky Chan
> Labels: MIME, segfault
> Fix For: 2.1.8
>
>
> We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
> Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
> I got this segfault 6 times this morning, and it appears from the same client too.
> I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> 66 fp = (void **) (*fp);
> (gdb) bt
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> #1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
> #2 <signal handler called>
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> #4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
> #5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
> response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
> #6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
> #7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
> #8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
> #9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
> #10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
> #11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
> #12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
> #13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
> #14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
> #15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
> #16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
> #17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
> #18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
> #19 0x00002abbb74f064d in clone () from /lib/libc.so.6
> #20 0x0000000000000000 in ?? ()
> (gdb) frame 3
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> warning: Source file is more recent than executable.
> 482 if (field->m_wks_idx < 0)
> (gdb) p *field
> $1 = {
> m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
> m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
> m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
Posted by "Leif Hedstrom (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13019453#comment-13019453 ]
Leif Hedstrom commented on TS-733:
----------------------------------
Ricky: We have a fix checked in on trunk, that fixes the problem in v2.1.7. I have no idea if it would help you v2.0.1 problems, but if possible, can you try latest trunk? Also, you really should start working on getting v2.1.7 or v2.1.8 (out in a few weeks hopefully) setup, too much have changed since v2.0 to make it reasonable to backport fixes any more (we really need to get v3.0 released soon, and EOLife v2.0).
> segfault in mime_hdr_set_accelerators_and_presence_bits
> -------------------------------------------------------
>
> Key: TS-733
> URL: https://issues.apache.org/jira/browse/TS-733
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
> Reporter: Ricky Chan
> Labels: MIME, segfault
> Fix For: 2.1.8
>
> Attachments: disable_dup.patch-2
>
>
> We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
> Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
> I got this segfault 6 times this morning, and it appears from the same client too.
> I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> 66 fp = (void **) (*fp);
> (gdb) bt
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> #1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
> #2 <signal handler called>
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> #4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
> #5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
> response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
> #6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
> #7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
> #8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
> #9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
> #10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
> #11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
> #12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
> #13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
> #14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
> #15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
> #16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
> #17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
> #18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
> #19 0x00002abbb74f064d in clone () from /lib/libc.so.6
> #20 0x0000000000000000 in ?? ()
> (gdb) frame 3
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> warning: Source file is more recent than executable.
> 482 if (field->m_wks_idx < 0)
> (gdb) p *field
> $1 = {
> m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
> m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
> m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
Posted by "Ricky Chan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017927#comment-13017927 ]
Ricky Chan commented on TS-733:
-------------------------------
I spoke to early I was able to crash it in 2.1.7 as well using my method details above.
#0 0x00000000006d02e9 in CacheVC::openReadChooseWriter (this=0x297e8e0, event=1, e=0x0) at CacheRead.cc:209
209 if (w->start_time > start_time || w->closed < 0)
(gdb) bt
#0 0x00000000006d02e9 in CacheVC::openReadChooseWriter (this=0x297e8e0, event=1, e=0x0) at CacheRead.cc:209
#1 0x00000000006d0e2c in CacheVC::openReadFromWriter (this=0x297e8e0, event=1, e=0x0) at CacheRead.cc:326
#2 0x00000000004ee0ef in Continuation::handleEvent (this=0x297e8e0, event=1, data=0x0) at I_Continuation.h:146
#3 0x00000000006d2914 in Cache::open_read (this=0x1f6f9b0, cont=0x2aaab4d16b90, key=0x4229ce50, request=0x2aaab4d15210, params=0x2aaab4d14a48, type=CACHE_FRAG_TYPE_HTTP,
hostname=0x2aaab475c83e "test1.isp.sky.com0", '�' <repeats 182 times>..., host_len=17) at CacheRead.cc:149
#4 0x00000000006b3b97 in Cache::open_read (this=0x1f6f9b0, cont=0x2aaab4d16b90, url=0x2aaab4d149f8, request=0x2aaab4d15210, params=0x2aaab4d14a48, type=CACHE_FRAG_TYPE_HTTP)
at P_CacheInternal.h:1006
#5 0x00000000006a44f7 in CacheProcessor::open_read (this=0xf750a0, cont=0x2aaab4d16b90, url=0x2aaab4d149f8, request=0x2aaab4d15210, params=0x2aaab4d14a48, pin_in_cache=0,
type=CACHE_FRAG_TYPE_HTTP) at Cache.cc:2787
#6 0x00000000005691d5 in HttpCacheSM::do_cache_open_read (this=0x2aaab4d16b90) at HttpCacheSM.cc:215
#7 0x0000000000569328 in HttpCacheSM::open_read (this=0x2aaab4d16b90, url=0x2aaab4d149f8, hdr=0x2aaab4d15210, params=0x2aaab4d14a48, pin_in_cache=0) at HttpCacheSM.cc:247
#8 0x0000000000584acf in HttpSM::do_cache_lookup_and_read (this=0x2aaab4d14950) at HttpSM.cc:3898
#9 0x0000000000590053 in HttpSM::set_next_state (this=0x2aaab4d14950) at HttpSM.cc:6440
#10 0x000000000057d5e0 in HttpSM::call_transact_and_set_next_state (this=0x2aaab4d14950, f=0) at HttpSM.cc:6332
#11 0x000000000059178e in HttpSM::handle_api_return (this=0x2aaab4d14950) at HttpSM.cc:1523
#12 0x00000000005958a9 in HttpSM::do_api_callout (this=0x2aaab4d14950) at HttpSM.cc:506
#13 0x000000000058fae5 in HttpSM::set_next_state (this=0x2aaab4d14950) at HttpSM.cc:6366
#14 0x000000000057d5e0 in HttpSM::call_transact_and_set_next_state (this=0x2aaab4d14950, f=0) at HttpSM.cc:6332
#15 0x000000000058fc12 in HttpSM::set_next_state (this=0x2aaab4d14950) at HttpSM.cc:6382
#16 0x000000000057d5e0 in HttpSM::call_transact_and_set_next_state (this=0x2aaab4d14950, f=0) at HttpSM.cc:6332
#17 0x000000000059178e in HttpSM::handle_api_return (this=0x2aaab4d14950) at HttpSM.cc:1523
#18 0x00000000005958a9 in HttpSM::do_api_callout (this=0x2aaab4d14950) at HttpSM.cc:506
#19 0x000000000058fae5 in HttpSM::set_next_state (this=0x2aaab4d14950) at HttpSM.cc:6366
#20 0x000000000057d5e0 in HttpSM::call_transact_and_set_next_state (this=0x2aaab4d14950, f=0) at HttpSM.cc:6332
#21 0x000000000059178e in HttpSM::handle_api_return (this=0x2aaab4d14950) at HttpSM.cc:1523
#22 0x00000000005958a9 in HttpSM::do_api_callout (this=0x2aaab4d14950) at HttpSM.cc:506
#23 0x000000000058fae5 in HttpSM::set_next_state (this=0x2aaab4d14950) at HttpSM.cc:6366
#24 0x000000000057d5e0 in HttpSM::call_transact_and_set_next_state (this=0x2aaab4d14950, f=0x5bcb70 <HttpTransact::ModifyRequest(HttpTransact::State*)>) at HttpSM.cc:6332
#25 0x0000000000591534 in HttpSM::state_read_client_request_header (this=0x2aaab4d14950, event=100, data=0x20efea8) at HttpSM.cc:787
#26 0x000000000058ae67 in HttpSM::main_handler (this=0x2aaab4d14950, event=100, data=0x20efea8) at HttpSM.cc:2443
#27 0x00000000004ee0ef in Continuation::handleEvent (this=0x2aaab4d14950, event=100, data=0x20efea8) at I_Continuation.h:146
#28 0x00000000006faff1 in read_signal_and_update (event=100, vc=0x20efce0) at UnixNetVConnection.cc:146
#29 0x00000000006fb952 in read_from_net (nh=0x2aaaab517628, vc=0x20efce0, thread=0x2aaaab516010) at UnixNetVConnection.cc:337
#30 0x00000000006fbaaf in UnixNetVConnection::net_read_io (this=0x20efce0, nh=0x2aaaab517628, lthread=0x2aaaab516010) at UnixNetVConnection.cc:822
#31 0x00000000006f546c in NetHandler::mainNetEvent (this=0x2aaaab517628, event=5, e=0x1f4b130) at UnixNet.cc:401
#32 0x00000000004ee0ef in Continuation::handleEvent (this=0x2aaaab517628, event=5, data=0x1f4b130) at I_Continuation.h:146
#33 0x000000000071e9df in EThread::process_event (this=0x2aaaab516010, e=0x1f4b130, calling_code=5) at UnixEThread.cc:140
#34 0x000000000071ef0d in EThread::execute (this=0x2aaaab516010) at UnixEThread.cc:262
#35 0x000000000071e26e in spawn_thread_internal (a=0x1f282b0) at Thread.cc:85
#36 0x00002b0c50d1afc7 in start_thread () from /lib/libpthread.so.0
#37 0x00002b0c52f0164d in clone () from /lib/libc.so.6
#38 0x0000000000000000 in ?? ()
I have compiled with --enable-debug as request.
So feel free to ask me any extra details from the dump you require.
(gdb) p *this
$1 = {<CacheVConnection> = {<VConnection> = {<Continuation> = {<force_VFPT_to_top> = {_vptr.force_VFPT_to_top = 0x7a0250},
handler = 0x6d0a80 <CacheVC::openReadFromWriter(int, Event*)>, handler_name = 0x7a5bb8 "&CacheVC::openReadFromWriter", mutex = {m_ptr = 0x2176df0},
link = {<SLink<Continuation>> = {next = 0x0}, prev = 0x0}}, lerrno = 0}, <No data fields>}, static size_to_init = 376, key = {b = {9862551532286235992,
860322339962582156}}, first_key = {b = {9862551532286235992, 860322339962582156}}, earliest_key = {b = {9862551532286235992, 860322339962582156}}, update_key = {b = {
1964747881772749290, 15406874007834800518}}, dir = {w = {0, 0, 0, 0, 0}}, earliest_dir = {w = {21215, 39936, 10382, 0, 0}}, overwrite_dir = {w = {0, 0, 0, 0, 0}},
first_dir = {w = {22192, 39936, 9356, 0, 0}}, _action = {_vptr.Action = 0x73fb30, continuation = 0x2aaab4d16b90, mutex = {m_ptr = 0x2176df0}, cancelled = 0},
request = {<MIMEHdr> = {<HdrHeapSDKHandle> = {m_heap = 0x2aaaaaaba010}, m_mime = 0x2aaaaaaba0c8}, m_http = 0x2aaaaaaba098, m_url_cached = {<HdrHeapSDKHandle> = {m_heap = 0x0},
m_url_impl = 0x0}, m_host_length = 0, m_host = 0x0, m_port = 0, m_target_cached = false, m_target_in_url = false, m_port_in_header = false}, vector = {magic = 0x0, data = {
data = 0x297e9f8, fast_data = {{alternate = {m_alt = 0x2aaab4f3c048}}, {alternate = {m_alt = 0x2aaab4f3d268}}, {alternate = {m_alt = 0x2aaab4f3e088}}, {alternate = {
m_alt = 0x2aaab4f3eea8}}}, default_val = 0xf758c8, size = 0, pos = -1}, xcount = 0, vector_buf = {m_ptr = 0x0}}, alternate = {m_alt = 0x0}, buf = {m_ptr = 0x0},
first_buf = {m_ptr = 0x0}, blocks = {m_ptr = 0x0}, writer_buf = {m_ptr = 0x0}, od = 0x29da510, io = {<AIOCallback> = {<Continuation> = {<force_VFPT_to_top> = {
_vptr.force_VFPT_to_top = 0x7a0390}, handler = 0x6b189a <AIOCallbackInternal::io_complete(int, void*)>, handler_name = 0x79dcd0 "&AIOCallbackInternal::io_complete",
mutex = {m_ptr = 0x0}, link = {<SLink<Continuation>> = {next = 0x0}, prev = 0x0}}, aiocb = {aio_fildes = 0, aio_buf = 0x0, aio_nbytes = 0, aio_offset = 0, aio_reqprio = 0,
aio_lio_opcode = 0, aio_state = 0, aio__pad = {0}}, action = {_vptr.Action = 0x0, continuation = 0x0, mutex = {m_ptr = 0x0}, cancelled = 0}, thread = 0x0, then = 0x0,
aio_result = 0}, first = 0x0, aio_req = 0x0, sleep_time = 0}, alternate_index = -1, opendir_link = {<SLink<CacheVC>> = {next = 0x0}, prev = 0x0}, vio = {_cont = 0x0,
nbytes = 0, ndone = 0, op = 1, buffer = {mbuf = 0x0, entry = 0x0, name = 0x0}, vc_server = 0x0, mutex = {m_ptr = 0x0}}, initial_thread = 0x2aaaab516010,
frag_type = CACHE_FRAG_TYPE_HTTP, info = 0x0, write_vector = 0x29da520, params = 0x2aaab4d14a48, header_len = 0, frag_len = 0, write_len = 0, agg_len = 0, write_serial = 0,
frag = 0x0, integral_frags = {{offset = 0}, {offset = 0}, {offset = 0}, {offset = 0}}, part = 0x1f6f9e0, last_collision = 0x0, trigger = 0x0, read_key = 0x0, save_handler = NULL,
pin_in_cache = 0, start_time = 1302371993498180000, base_stat = 13, recursive = 0, closed = 0, seek_to = 0, offset = 0, writer_offset = 0, length = 0, doc_pos = 0, write_pos = 0,
total_len = 0, doc_len = 0, update_len = 0, fragment = 0, scan_msec_delay = 0, write_vc = 0x0, hostname = 0x0, host_len = 0, header_to_write_len = 0, header_to_write = 0x0,
writer_lock_retry = 0, {flags = 8192, f = {use_first_key = 0, overwrite = 0, close_complete = 0, sync = 0, evacuator = 0, single_fragment = 0, evac_vector = 0, lookup = 0,
update = 0, remove = 0, remove_aborted_writers = 0, open_read_timeout = 0, data_done = 0, read_from_writer_called = 1, not_from_ram_cache = 0, rewrite_resident_alt = 0,
readers = 0, doc_from_ram_cache = 0}}}
> segfault in mime_hdr_set_accelerators_and_presence_bits
> -------------------------------------------------------
>
> Key: TS-733
> URL: https://issues.apache.org/jira/browse/TS-733
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
> Reporter: Ricky Chan
> Labels: MIME, segfault
> Fix For: 2.1.8
>
>
> We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
> Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
> I got this segfault 6 times this morning, and it appears from the same client too.
> I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> 66 fp = (void **) (*fp);
> (gdb) bt
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> #1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
> #2 <signal handler called>
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> #4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
> #5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
> response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
> #6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
> #7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
> #8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
> #9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
> #10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
> #11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
> #12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
> #13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
> #14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
> #15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
> #16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
> #17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
> #18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
> #19 0x00002abbb74f064d in clone () from /lib/libc.so.6
> #20 0x0000000000000000 in ?? ()
> (gdb) frame 3
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> warning: Source file is more recent than executable.
> 482 if (field->m_wks_idx < 0)
> (gdb) p *field
> $1 = {
> m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
> m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
> m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
Posted by "Leif Hedstrom (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017974#comment-13017974 ]
Leif Hedstrom commented on TS-733:
----------------------------------
Fwiw, this is what I'm running
ab -k -c 20 -n 1000000 -H "User-Agent: Mozilla/4.0 (compatible;)" -H "Accept: text/html, */*" -H "Accept-Encoding: gzip, deflate, identity" -H "Via: 1.1 AKmdrL2CacheBC10.telecom.co.nz" -H "Cache-Control: max-stale=0" -H "If-Modified-Since: Fri, 08 Apr 2011 11:09:54 GMT" -H "Via: 1.1 AKmdrL2CacheBC10.telecom.co.nz" -H "X-BlueCoat-Via: B8C344C089BFFBD7" -H "Connection: keep-alive" http://loki.ogre.com/bench/1k.bmp
and at the same time I run
while [ 1 ] ; do curlhead -X PURGE http://loki.ogre.com/bench/1k.bmp; usleep 300000 ; done
Still no crash :/. I'm wondering if there are some other configuration options at play here, what have you changed from the default records.config as shipped with v2.1.7? And, are you using a raw device for cache, or is it a file on the filesystem?
> segfault in mime_hdr_set_accelerators_and_presence_bits
> -------------------------------------------------------
>
> Key: TS-733
> URL: https://issues.apache.org/jira/browse/TS-733
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
> Reporter: Ricky Chan
> Labels: MIME, segfault
> Fix For: 2.1.8
>
>
> We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
> Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
> I got this segfault 6 times this morning, and it appears from the same client too.
> I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> 66 fp = (void **) (*fp);
> (gdb) bt
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> #1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
> #2 <signal handler called>
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> #4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
> #5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
> response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
> #6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
> #7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
> #8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
> #9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
> #10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
> #11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
> #12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
> #13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
> #14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
> #15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
> #16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
> #17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
> #18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
> #19 0x00002abbb74f064d in clone () from /lib/libc.so.6
> #20 0x0000000000000000 in ?? ()
> (gdb) frame 3
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> warning: Source file is more recent than executable.
> 482 if (field->m_wks_idx < 0)
> (gdb) p *field
> $1 = {
> m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
> m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
> m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
Posted by "Leif Hedstrom (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017554#comment-13017554 ]
Leif Hedstrom commented on TS-733:
----------------------------------
Ricky,
thanks for the very detailed bug report, much appreciated!
I've tried to reproduce this, using the headers above, but I'm unable to do so :/. I've sent thousands of requests (in parallel) through it, and it's not causing a crash, while at the same time sending a PURGE request on the same URL. I have both request and response Via: headers enabled in records.config.
Have you tried this with ATS v2.1.7, or trunk?
> segfault in mime_hdr_set_accelerators_and_presence_bits
> -------------------------------------------------------
>
> Key: TS-733
> URL: https://issues.apache.org/jira/browse/TS-733
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
> Reporter: Ricky Chan
> Labels: MIME, segfault
>
> We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
> Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
> I got this segfault 6 times this morning, and it appears from the same client too.
> I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> 66 fp = (void **) (*fp);
> (gdb) bt
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> #1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
> #2 <signal handler called>
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> #4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
> #5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
> response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
> #6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
> #7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
> #8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
> #9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
> #10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
> #11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
> #12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
> #13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
> #14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
> #15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
> #16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
> #17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
> #18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
> #19 0x00002abbb74f064d in clone () from /lib/libc.so.6
> #20 0x0000000000000000 in ?? ()
> (gdb) frame 3
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> warning: Source file is more recent than executable.
> 482 if (field->m_wks_idx < 0)
> (gdb) p *field
> $1 = {
> m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
> m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
> m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
Posted by "Ricky Chan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017862#comment-13017862 ]
Ricky Chan commented on TS-733:
-------------------------------
In my production environment as I have live customers, I'm going to mitigate the chances of this bug taking place by restoring the fuzzy heuristics settings (at a cost to origin hits, which I have mitigate by another ATS layer).
In the LAB I will apply a different approach to the m_next_dup pointing outside itself, instead of ignoring it, I will try setting m_next_dup and seeing if ATS will just treat the dup's as seperate entities, which as it seems to be an edge case I can live with as I think ATS will work fine (limited understanding of code).
> segfault in mime_hdr_set_accelerators_and_presence_bits
> -------------------------------------------------------
>
> Key: TS-733
> URL: https://issues.apache.org/jira/browse/TS-733
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
> Reporter: Ricky Chan
> Labels: MIME, segfault
> Fix For: 2.1.8
>
>
> We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
> Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
> I got this segfault 6 times this morning, and it appears from the same client too.
> I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> 66 fp = (void **) (*fp);
> (gdb) bt
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> #1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
> #2 <signal handler called>
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> #4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
> #5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
> response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
> #6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
> #7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
> #8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
> #9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
> #10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
> #11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
> #12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
> #13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
> #14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
> #15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
> #16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
> #17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
> #18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
> #19 0x00002abbb74f064d in clone () from /lib/libc.so.6
> #20 0x0000000000000000 in ?? ()
> (gdb) frame 3
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> warning: Source file is more recent than executable.
> 482 if (field->m_wks_idx < 0)
> (gdb) p *field
> $1 = {
> m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
> m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
> m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
Posted by "Ricky Chan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ricky Chan updated TS-733:
--------------------------
Comment: was deleted
(was: I've just download 2.1.7 and compared MIME.cc and MIME.h between that and 2.0.1.
I (could be wrong but I) can't see any real code change apart from (mainly) moving away from inkomti properitary types.
I just want to mention in my version of 2.0.1 in about line 1250 in MIME.cc
ink_release_assert(j < block_count);
This is disabled, because it get's trigger and causes an abort (I've already reported this) and I don't believe this is fixed in 2.1.7 (speculation) based on the content of 2.1.7 MIME.cc code isn't much different from 2.0.1. All my reported errors seems to originate from here.
It's disabled because in my lab test it didn't affect the system although the next_dup is pointing to something beyond it's self and therefore it could be triggering all the subsequent headers.
I may need to add a different patch (such as don't treat it as a dup when it points beyond it self as the RFC seems to indicate multiple tags of the same name is valid) and may be collapsed or not. From my (very limited) understanding of the code I believe this will be fine.
The core bug I believe is the dup code or the data it points to becomes invalid (by another thread perhaps?).
I don't believe this (again I could be wrong) is addressed by 2.1.7.
)
> segfault in mime_hdr_set_accelerators_and_presence_bits
> -------------------------------------------------------
>
> Key: TS-733
> URL: https://issues.apache.org/jira/browse/TS-733
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
> Reporter: Ricky Chan
> Labels: MIME, segfault
> Fix For: 2.1.8
>
>
> We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
> Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
> I got this segfault 6 times this morning, and it appears from the same client too.
> I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> 66 fp = (void **) (*fp);
> (gdb) bt
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> #1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
> #2 <signal handler called>
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> #4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
> #5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
> response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
> #6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
> #7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
> #8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
> #9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
> #10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
> #11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
> #12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
> #13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
> #14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
> #15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
> #16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
> #17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
> #18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
> #19 0x00002abbb74f064d in clone () from /lib/libc.so.6
> #20 0x0000000000000000 in ?? ()
> (gdb) frame 3
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> warning: Source file is more recent than executable.
> 482 if (field->m_wks_idx < 0)
> (gdb) p *field
> $1 = {
> m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
> m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
> m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
Posted by "Ricky Chan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017450#comment-13017450 ]
Ricky Chan commented on TS-733:
-------------------------------
I mean keep purging that cache asset.
> segfault in mime_hdr_set_accelerators_and_presence_bits
> -------------------------------------------------------
>
> Key: TS-733
> URL: https://issues.apache.org/jira/browse/TS-733
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
> Reporter: Ricky Chan
> Labels: MIME, segfault
>
> We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
> Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
> I got this segfault 6 times this morning, and it appears from the same client too.
> I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> 66 fp = (void **) (*fp);
> (gdb) bt
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> #1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
> #2 <signal handler called>
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> #4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
> #5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
> response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
> #6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
> #7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
> #8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
> #9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
> #10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
> #11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
> #12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
> #13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
> #14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
> #15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
> #16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
> #17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
> #18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
> #19 0x00002abbb74f064d in clone () from /lib/libc.so.6
> #20 0x0000000000000000 in ?? ()
> (gdb) frame 3
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> warning: Source file is more recent than executable.
> 482 if (field->m_wks_idx < 0)
> (gdb) p *field
> $1 = {
> m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
> m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
> m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
Posted by "Leif Hedstrom (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Leif Hedstrom updated TS-733:
-----------------------------
Fix Version/s: 2.1.8
> segfault in mime_hdr_set_accelerators_and_presence_bits
> -------------------------------------------------------
>
> Key: TS-733
> URL: https://issues.apache.org/jira/browse/TS-733
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
> Reporter: Ricky Chan
> Labels: MIME, segfault
> Fix For: 2.1.8
>
>
> We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
> Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
> I got this segfault 6 times this morning, and it appears from the same client too.
> I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> 66 fp = (void **) (*fp);
> (gdb) bt
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> #1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
> #2 <signal handler called>
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> #4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
> #5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
> response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
> #6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
> #7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
> #8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
> #9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
> #10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
> #11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
> #12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
> #13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
> #14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
> #15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
> #16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
> #17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
> #18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
> #19 0x00002abbb74f064d in clone () from /lib/libc.so.6
> #20 0x0000000000000000 in ?? ()
> (gdb) frame 3
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> warning: Source file is more recent than executable.
> 482 if (field->m_wks_idx < 0)
> (gdb) p *field
> $1 = {
> m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
> m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
> m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
Posted by "Ricky Chan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13019243#comment-13019243 ]
Ricky Chan commented on TS-733:
-------------------------------
I completely agree, I just needed it to stop crashing in production for now :-)
It seems likely (to me anyway) that mime_hdr_copy_onto isn't working correctly occasionally, it uses memcpy (via _mime_field_block_copy) from the one MIMEHdrImp* to another MIMEHdrImp*, as the original s_mh had pointers to MIMEField* in m_next_dup, the mime_hdr_field_block_list_adjust is called to fix it so the new data d_mh has m_next_dup's pointing to it's own copy of the data.
However the 4 issues (1 abort and 3 segfaults) I had all stemmed from some of the m_next_dup in s_mh *somehow* having invalid fields in it.
I don't know what is causing m_next_dup in the original list to point to invalid address.
> segfault in mime_hdr_set_accelerators_and_presence_bits
> -------------------------------------------------------
>
> Key: TS-733
> URL: https://issues.apache.org/jira/browse/TS-733
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
> Reporter: Ricky Chan
> Labels: MIME, segfault
> Fix For: 2.1.8
>
> Attachments: disable_dup.patch-2
>
>
> We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
> Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
> I got this segfault 6 times this morning, and it appears from the same client too.
> I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> 66 fp = (void **) (*fp);
> (gdb) bt
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> #1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
> #2 <signal handler called>
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> #4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
> #5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
> response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
> #6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
> #7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
> #8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
> #9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
> #10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
> #11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
> #12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
> #13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
> #14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
> #15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
> #16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
> #17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
> #18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
> #19 0x00002abbb74f064d in clone () from /lib/libc.so.6
> #20 0x0000000000000000 in ?? ()
> (gdb) frame 3
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> warning: Source file is more recent than executable.
> 482 if (field->m_wks_idx < 0)
> (gdb) p *field
> $1 = {
> m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
> m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
> m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
Posted by "Ricky Chan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017357#comment-13017357 ]
Ricky Chan commented on TS-733:
-------------------------------
ignore:
warning: Source file is more recent than executable.
I saved the file after reading MIME.cc previously. I don't think I was clear, I got a feeling line 482 in MIME.cc isn't the cause. (field isn't NULL, and the trace shows m_wks_idx has a value, 67 in this case).
However I don't think 67 is valid value.
Ricky
> segfault in mime_hdr_set_accelerators_and_presence_bits
> -------------------------------------------------------
>
> Key: TS-733
> URL: https://issues.apache.org/jira/browse/TS-733
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
> Reporter: Ricky Chan
> Labels: MIME, segfault
>
> We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
> Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
> I got this segfault 6 times this morning, and it appears from the same client too.
> I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> 66 fp = (void **) (*fp);
> (gdb) bt
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> #1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
> #2 <signal handler called>
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> #4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
> #5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
> response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
> #6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
> #7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
> #8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
> #9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
> #10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
> #11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
> #12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
> #13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
> #14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
> #15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
> #16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
> #17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
> #18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
> #19 0x00002abbb74f064d in clone () from /lib/libc.so.6
> #20 0x0000000000000000 in ?? ()
> (gdb) frame 3
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> warning: Source file is more recent than executable.
> 482 if (field->m_wks_idx < 0)
> (gdb) p *field
> $1 = {
> m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
> m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
> m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
Posted by "Ricky Chan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017568#comment-13017568 ]
Ricky Chan commented on TS-733:
-------------------------------
2.0.1 Stable :)
Of course, it may be working in 2.1.7 or trunk.
I'll stick in the lab on Monday and come back to you.
Ricky
> segfault in mime_hdr_set_accelerators_and_presence_bits
> -------------------------------------------------------
>
> Key: TS-733
> URL: https://issues.apache.org/jira/browse/TS-733
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
> Reporter: Ricky Chan
> Labels: MIME, segfault
>
> We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
> Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
> I got this segfault 6 times this morning, and it appears from the same client too.
> I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> 66 fp = (void **) (*fp);
> (gdb) bt
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> #1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
> #2 <signal handler called>
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> #4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
> #5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
> response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
> #6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
> #7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
> #8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
> #9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
> #10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
> #11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
> #12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
> #13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
> #14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
> #15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
> #16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
> #17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
> #18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
> #19 0x00002abbb74f064d in clone () from /lib/libc.so.6
> #20 0x0000000000000000 in ?? ()
> (gdb) frame 3
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> warning: Source file is more recent than executable.
> 482 if (field->m_wks_idx < 0)
> (gdb) p *field
> $1 = {
> m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
> m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
> m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
Posted by "Leif Hedstrom (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13018149#comment-13018149 ]
Leif Hedstrom commented on TS-733:
----------------------------------
Does this patch fix the crasher problem in v2.1.7 ?
> segfault in mime_hdr_set_accelerators_and_presence_bits
> -------------------------------------------------------
>
> Key: TS-733
> URL: https://issues.apache.org/jira/browse/TS-733
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
> Reporter: Ricky Chan
> Labels: MIME, segfault
> Fix For: 2.1.8
>
> Attachments: disable_dup.patch
>
>
> We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
> Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
> I got this segfault 6 times this morning, and it appears from the same client too.
> I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> 66 fp = (void **) (*fp);
> (gdb) bt
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> #1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
> #2 <signal handler called>
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> #4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
> #5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
> response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
> #6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
> #7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
> #8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
> #9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
> #10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
> #11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
> #12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
> #13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
> #14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
> #15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
> #16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
> #17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
> #18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
> #19 0x00002abbb74f064d in clone () from /lib/libc.so.6
> #20 0x0000000000000000 in ?? ()
> (gdb) frame 3
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> warning: Source file is more recent than executable.
> 482 if (field->m_wks_idx < 0)
> (gdb) p *field
> $1 = {
> m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
> m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
> m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
Posted by "Ricky Chan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13018043#comment-13018043 ]
Ricky Chan commented on TS-733:
-------------------------------
I'm struggling to re-produce it myself at the moment! I'm not using ab but there's no reason why that won't work, although I would have disabled keep-alive (-k) because my threaded perl script effectively doesn't use keep-alive. (in 2.0.1 anyway, without keep-alive it is a lot easier to crash it).
Your purging is similar to what I am doing, except I am using curl and not curlhead (although args look the same) and I don't bother sleeping, allowing it to completely thrash a single CPU ;-)
For 2.1.7, I installed from source with --enable-debug --enable-layount=Debian --with-group=nogroup --with-user=nobody (my config.layout has binaries for Debian in /usr/bin instead of /usr/sbin which I know isn't strictly correct for debian but it's a legacy location).
records.config is the default from the source, except port 8080 changed to port 80 (that's how I knew my 2.0.1 config was overridden and it chosen an invalid cluster interface, even though cluster is disabled (known issue where ATS won't start up with invalid interface even if clustering not used)).
I thought it would be good to try to crash on defaults before using my own settings.
I'll get back if I managed to crash it again.
> segfault in mime_hdr_set_accelerators_and_presence_bits
> -------------------------------------------------------
>
> Key: TS-733
> URL: https://issues.apache.org/jira/browse/TS-733
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
> Reporter: Ricky Chan
> Labels: MIME, segfault
> Fix For: 2.1.8
>
>
> We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
> Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
> I got this segfault 6 times this morning, and it appears from the same client too.
> I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> 66 fp = (void **) (*fp);
> (gdb) bt
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> #1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
> #2 <signal handler called>
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> #4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
> #5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
> response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
> #6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
> #7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
> #8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
> #9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
> #10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
> #11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
> #12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
> #13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
> #14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
> #15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
> #16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
> #17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
> #18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
> #19 0x00002abbb74f064d in clone () from /lib/libc.so.6
> #20 0x0000000000000000 in ?? ()
> (gdb) frame 3
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> warning: Source file is more recent than executable.
> 482 if (field->m_wks_idx < 0)
> (gdb) p *field
> $1 = {
> m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
> m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
> m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
Posted by "Ricky Chan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017366#comment-13017366 ]
Ricky Chan commented on TS-733:
-------------------------------
Sorry I mis-read the code, 67 is the index for hdrroken_strs which in this case is Via.
so 67 is valid.
> segfault in mime_hdr_set_accelerators_and_presence_bits
> -------------------------------------------------------
>
> Key: TS-733
> URL: https://issues.apache.org/jira/browse/TS-733
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
> Reporter: Ricky Chan
> Labels: MIME, segfault
>
> We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
> Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
> I got this segfault 6 times this morning, and it appears from the same client too.
> I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> 66 fp = (void **) (*fp);
> (gdb) bt
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> #1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
> #2 <signal handler called>
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> #4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
> #5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
> response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
> #6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
> #7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
> #8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
> #9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
> #10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
> #11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
> #12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
> #13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
> #14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
> #15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
> #16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
> #17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
> #18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
> #19 0x00002abbb74f064d in clone () from /lib/libc.so.6
> #20 0x0000000000000000 in ?? ()
> (gdb) frame 3
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> warning: Source file is more recent than executable.
> 482 if (field->m_wks_idx < 0)
> (gdb) p *field
> $1 = {
> m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
> m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
> m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
Posted by "Ricky Chan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017433#comment-13017433 ]
Ricky Chan commented on TS-733:
-------------------------------
Obviously the trailing dot should be removed for entry in the header and the smiley ;) is actually ; followed by ).
> segfault in mime_hdr_set_accelerators_and_presence_bits
> -------------------------------------------------------
>
> Key: TS-733
> URL: https://issues.apache.org/jira/browse/TS-733
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
> Reporter: Ricky Chan
> Labels: MIME, segfault
>
> We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
> Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
> I got this segfault 6 times this morning, and it appears from the same client too.
> I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> 66 fp = (void **) (*fp);
> (gdb) bt
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> #1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
> #2 <signal handler called>
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> #4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
> #5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
> response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
> #6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
> #7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
> #8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
> #9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
> #10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
> #11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
> #12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
> #13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
> #14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
> #15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
> #16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
> #17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
> #18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
> #19 0x00002abbb74f064d in clone () from /lib/libc.so.6
> #20 0x0000000000000000 in ?? ()
> (gdb) frame 3
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> warning: Source file is more recent than executable.
> 482 if (field->m_wks_idx < 0)
> (gdb) p *field
> $1 = {
> m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
> m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
> m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
Posted by "Ricky Chan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13018047#comment-13018047 ]
Ricky Chan commented on TS-733:
-------------------------------
I had deleted some incorrect info from this report earlier but just want to mention I am seeing segfaults in 2.0.1 not only for multiple Via's (hdrtoken index 67) but also for Multiple Keep-Alive's (hdrtoken index 36). (In production I don't see it for 67 anymore because I patched the 2.0.1 code not handle Via headers in the accelerators code but that is a work around and not real solution as the keep-alive is showing).
> segfault in mime_hdr_set_accelerators_and_presence_bits
> -------------------------------------------------------
>
> Key: TS-733
> URL: https://issues.apache.org/jira/browse/TS-733
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
> Reporter: Ricky Chan
> Labels: MIME, segfault
> Fix For: 2.1.8
>
>
> We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
> Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
> I got this segfault 6 times this morning, and it appears from the same client too.
> I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> 66 fp = (void **) (*fp);
> (gdb) bt
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> #1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
> #2 <signal handler called>
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> #4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
> #5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
> response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
> #6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
> #7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
> #8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
> #9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
> #10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
> #11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
> #12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
> #13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
> #14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
> #15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
> #16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
> #17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
> #18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
> #19 0x00002abbb74f064d in clone () from /lib/libc.so.6
> #20 0x0000000000000000 in ?? ()
> (gdb) frame 3
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> warning: Source file is more recent than executable.
> 482 if (field->m_wks_idx < 0)
> (gdb) p *field
> $1 = {
> m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
> m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
> m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
Posted by "Ricky Chan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13018107#comment-13018107 ]
Ricky Chan commented on TS-733:
-------------------------------
Probably could comment out the build block translation table as well, as the patch makes it not used.
It isn't resource friendly and a lot of code paths now don't get run, but so far in the lab 2.0.1 has crashed in 8 hours under the trigger conditions, without this patch, the trigger can cause it to happen within 5 minutes.
> segfault in mime_hdr_set_accelerators_and_presence_bits
> -------------------------------------------------------
>
> Key: TS-733
> URL: https://issues.apache.org/jira/browse/TS-733
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
> Reporter: Ricky Chan
> Labels: MIME, segfault
> Fix For: 2.1.8
>
> Attachments: disable_dup.patch
>
>
> We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
> Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
> I got this segfault 6 times this morning, and it appears from the same client too.
> I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> 66 fp = (void **) (*fp);
> (gdb) bt
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> #1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
> #2 <signal handler called>
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> #4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
> #5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
> response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
> #6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
> #7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
> #8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
> #9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
> #10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
> #11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
> #12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
> #13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
> #14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
> #15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
> #16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
> #17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
> #18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
> #19 0x00002abbb74f064d in clone () from /lib/libc.so.6
> #20 0x0000000000000000 in ?? ()
> (gdb) frame 3
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> warning: Source file is more recent than executable.
> 482 if (field->m_wks_idx < 0)
> (gdb) p *field
> $1 = {
> m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
> m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
> m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
Posted by "Ricky Chan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13018045#comment-13018045 ]
Ricky Chan commented on TS-733:
-------------------------------
Here the the information request from the core dump.
#0 0x00000000006d02e9 in CacheVC::openReadChooseWriter (this=0x297e8e0, event=1, e=0x0) at CacheRead.cc:209
209 if (w->start_time > start_time || w->closed < 0)
(gdb) p w
$1 = (class CacheVC *) 0xff002648ce8fa70
(gdb) p *w
Cannot access memory at address 0xff002648ce8fa70
That's not good and explains the segfault.. maybe un-initialised or freed?
> segfault in mime_hdr_set_accelerators_and_presence_bits
> -------------------------------------------------------
>
> Key: TS-733
> URL: https://issues.apache.org/jira/browse/TS-733
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
> Reporter: Ricky Chan
> Labels: MIME, segfault
> Fix For: 2.1.8
>
>
> We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
> Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
> I got this segfault 6 times this morning, and it appears from the same client too.
> I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> 66 fp = (void **) (*fp);
> (gdb) bt
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> #1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
> #2 <signal handler called>
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> #4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
> #5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
> response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
> #6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
> #7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
> #8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
> #9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
> #10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
> #11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
> #12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
> #13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
> #14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
> #15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
> #16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
> #17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
> #18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
> #19 0x00002abbb74f064d in clone () from /lib/libc.so.6
> #20 0x0000000000000000 in ?? ()
> (gdb) frame 3
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> warning: Source file is more recent than executable.
> 482 if (field->m_wks_idx < 0)
> (gdb) p *field
> $1 = {
> m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
> m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
> m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
Posted by "Leif Hedstrom (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13018928#comment-13018928 ]
Leif Hedstrom commented on TS-733:
----------------------------------
I finally got 2.1.7 to segfault with your method! So, at least there's some hope here to debug this.
It takes quite a while though to reproduce (hours), but this is a step in the right direction. As for the v2.0.1 patches, I'm a little concerned about it "fixing" the symptoms and not the cause.
> segfault in mime_hdr_set_accelerators_and_presence_bits
> -------------------------------------------------------
>
> Key: TS-733
> URL: https://issues.apache.org/jira/browse/TS-733
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
> Reporter: Ricky Chan
> Labels: MIME, segfault
> Fix For: 2.1.8
>
> Attachments: disable_dup.patch-2
>
>
> We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
> Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
> I got this segfault 6 times this morning, and it appears from the same client too.
> I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> 66 fp = (void **) (*fp);
> (gdb) bt
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> #1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
> #2 <signal handler called>
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> #4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
> #5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
> response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
> #6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
> #7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
> #8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
> #9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
> #10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
> #11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
> #12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
> #13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
> #14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
> #15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
> #16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
> #17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
> #18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
> #19 0x00002abbb74f064d in clone () from /lib/libc.so.6
> #20 0x0000000000000000 in ?? ()
> (gdb) frame 3
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> warning: Source file is more recent than executable.
> 482 if (field->m_wks_idx < 0)
> (gdb) p *field
> $1 = {
> m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
> m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
> m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
Posted by "Leif Hedstrom (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13018174#comment-13018174 ]
Leif Hedstrom commented on TS-733:
----------------------------------
Running your exact commands, using ab with a forward proxy config (except a different URL, which is 1kB). I made the following changes to records.config:
CONFIG proxy.config.http.server_port INT 80
CONFIG proxy.config.url_remap.remap_required INT 0
CONFIG proxy.config.http.cache.required_headers INT 1
Everything else is set to defaults with v2.1.7. I'm still not getting the crasher :-/. How long does it take until it will crash ?
> segfault in mime_hdr_set_accelerators_and_presence_bits
> -------------------------------------------------------
>
> Key: TS-733
> URL: https://issues.apache.org/jira/browse/TS-733
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
> Reporter: Ricky Chan
> Labels: MIME, segfault
> Fix For: 2.1.8
>
> Attachments: disable_dup.patch
>
>
> We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
> Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
> I got this segfault 6 times this morning, and it appears from the same client too.
> I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> 66 fp = (void **) (*fp);
> (gdb) bt
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> #1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
> #2 <signal handler called>
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> #4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
> #5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
> response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
> #6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
> #7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
> #8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
> #9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
> #10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
> #11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
> #12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
> #13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
> #14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
> #15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
> #16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
> #17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
> #18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
> #19 0x00002abbb74f064d in clone () from /lib/libc.so.6
> #20 0x0000000000000000 in ?? ()
> (gdb) frame 3
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> warning: Source file is more recent than executable.
> 482 if (field->m_wks_idx < 0)
> (gdb) p *field
> $1 = {
> m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
> m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
> m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
Posted by "Leif Hedstrom (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017571#comment-13017571 ]
Leif Hedstrom commented on TS-733:
----------------------------------
Also, if you get it to crash with v2.1.7 (which I'm currently unable to do, so can't get a good stack trace), please compile ATS with the --enable-debug option. i.e.
./configure --enable-debug
that will avoid having all those parameter values etc. be optimized out.
> segfault in mime_hdr_set_accelerators_and_presence_bits
> -------------------------------------------------------
>
> Key: TS-733
> URL: https://issues.apache.org/jira/browse/TS-733
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
> Reporter: Ricky Chan
> Labels: MIME, segfault
>
> We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
> Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
> I got this segfault 6 times this morning, and it appears from the same client too.
> I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> 66 fp = (void **) (*fp);
> (gdb) bt
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> #1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
> #2 <signal handler called>
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> #4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
> #5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
> response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
> #6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
> #7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
> #8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
> #9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
> #10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
> #11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
> #12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
> #13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
> #14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
> #15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
> #16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
> #17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
> #18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
> #19 0x00002abbb74f064d in clone () from /lib/libc.so.6
> #20 0x0000000000000000 in ?? ()
> (gdb) frame 3
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> warning: Source file is more recent than executable.
> 482 if (field->m_wks_idx < 0)
> (gdb) p *field
> $1 = {
> m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
> m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
> m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
Posted by "Ricky Chan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ricky Chan updated TS-733:
--------------------------
Attachment: (was: disable_dup.patch)
> segfault in mime_hdr_set_accelerators_and_presence_bits
> -------------------------------------------------------
>
> Key: TS-733
> URL: https://issues.apache.org/jira/browse/TS-733
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
> Reporter: Ricky Chan
> Labels: MIME, segfault
> Fix For: 2.1.8
>
> Attachments: disable_dup.patch
>
>
> We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
> Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
> I got this segfault 6 times this morning, and it appears from the same client too.
> I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> 66 fp = (void **) (*fp);
> (gdb) bt
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> #1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
> #2 <signal handler called>
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> #4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
> #5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
> response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
> #6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
> #7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
> #8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
> #9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
> #10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
> #11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
> #12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
> #13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
> #14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
> #15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
> #16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
> #17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
> #18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
> #19 0x00002abbb74f064d in clone () from /lib/libc.so.6
> #20 0x0000000000000000 in ?? ()
> (gdb) frame 3
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> warning: Source file is more recent than executable.
> 482 if (field->m_wks_idx < 0)
> (gdb) p *field
> $1 = {
> m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
> m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
> m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
Posted by "Ricky Chan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017924#comment-13017924 ]
Ricky Chan commented on TS-733:
-------------------------------
Do the hack mentioned above in 2.0.1 didn't really help. I still got segfaults using my duplicate via test.
I've put 2.1.7 in the lab and like you said, I was unable to re-produce the bug and create any of the segfaults.
So it looks like an issue specifically with 2.0.x
> segfault in mime_hdr_set_accelerators_and_presence_bits
> -------------------------------------------------------
>
> Key: TS-733
> URL: https://issues.apache.org/jira/browse/TS-733
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
> Reporter: Ricky Chan
> Labels: MIME, segfault
> Fix For: 2.1.8
>
>
> We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
> Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
> I got this segfault 6 times this morning, and it appears from the same client too.
> I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> 66 fp = (void **) (*fp);
> (gdb) bt
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> #1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
> #2 <signal handler called>
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> #4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
> #5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
> response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
> #6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
> #7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
> #8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
> #9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
> #10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
> #11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
> #12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
> #13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
> #14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
> #15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
> #16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
> #17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
> #18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
> #19 0x00002abbb74f064d in clone () from /lib/libc.so.6
> #20 0x0000000000000000 in ?? ()
> (gdb) frame 3
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> warning: Source file is more recent than executable.
> 482 if (field->m_wks_idx < 0)
> (gdb) p *field
> $1 = {
> m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
> m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
> m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
Posted by "Ricky Chan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017847#comment-13017847 ]
Ricky Chan commented on TS-733:
-------------------------------
Hmm.. _hdrtoken_strs[36] seems to be Keep-Alive
Looking the headers it seems to be multiple Keep-Alive headers...
A bit suspicious of the dup code grep of above shows:
m_ptr_name = 0x2aaaceca918e "Keep-Alive: 115\r\nUser-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.16) Gecko/20110319 Firefox/3.6.16\r\nConnection: keep-alive\r\nKeep-Alive: 115\r\n\r\n90/c_3.gif HTTP/1.1\r\nHost: www.teste"...,
m_ptr_value = 0x2aaaceca919a "115\r\nUser-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.16) Gecko/20110319 Firefox/3.6.16\r\nConnection: keep-alive\r\nKeep-Alive: 115\r\n\r\n90/c_3.gif HTTP/1.1\r\nHost: www.testetf1.com\r\nUse"..., m_next_dup = 0x2aab065ece40, m_wks_idx = 36,
--
m_ptr_name = 0x2aaaceca921f "Keep-Alive: 115\r\n\r\n90/c_3.gif HTTP/1.1\r\nHost: www.testetf1.com\r\nUser-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.16) Gecko/20110319 Firefox/3.6.16\r\nAccept: image/png,image/*;q=0.8,"...,
m_ptr_value = 0x2aaaceca922b "115\r\n\r\n90/c_3.gif HTTP/1.1\r\nHost: www.testetf1.com\r\nUser-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.16) Gecko/20110319 Firefox/3.6.16\r\nAccept: image/png,image/*;q=0.8,*/*;q=0.5\r\nA"..., m_next_dup = 0x0, m_wks_idx = 36, m_len_name = 10,
I'd assume m_next_dup would point to either m_ptr_name or m_ptr_value of the dup right?
Ricky
> segfault in mime_hdr_set_accelerators_and_presence_bits
> -------------------------------------------------------
>
> Key: TS-733
> URL: https://issues.apache.org/jira/browse/TS-733
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
> Reporter: Ricky Chan
> Labels: MIME, segfault
> Fix For: 2.1.8
>
>
> We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
> Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
> I got this segfault 6 times this morning, and it appears from the same client too.
> I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> 66 fp = (void **) (*fp);
> (gdb) bt
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> #1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
> #2 <signal handler called>
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> #4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
> #5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
> response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
> #6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
> #7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
> #8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
> #9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
> #10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
> #11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
> #12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
> #13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
> #14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
> #15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
> #16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
> #17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
> #18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
> #19 0x00002abbb74f064d in clone () from /lib/libc.so.6
> #20 0x0000000000000000 in ?? ()
> (gdb) frame 3
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> warning: Source file is more recent than executable.
> 482 if (field->m_wks_idx < 0)
> (gdb) p *field
> $1 = {
> m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
> m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
> m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
Posted by "Ricky Chan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017849#comment-13017849 ]
Ricky Chan commented on TS-733:
-------------------------------
The second "Keep-Alive: 115\r\n\r\n" seems to have trashed part of the address of another request?
> segfault in mime_hdr_set_accelerators_and_presence_bits
> -------------------------------------------------------
>
> Key: TS-733
> URL: https://issues.apache.org/jira/browse/TS-733
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
> Reporter: Ricky Chan
> Labels: MIME, segfault
> Fix For: 2.1.8
>
>
> We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
> Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
> I got this segfault 6 times this morning, and it appears from the same client too.
> I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> 66 fp = (void **) (*fp);
> (gdb) bt
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> #1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
> #2 <signal handler called>
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> #4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
> #5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
> response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
> #6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
> #7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
> #8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
> #9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
> #10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
> #11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
> #12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
> #13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
> #14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
> #15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
> #16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
> #17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
> #18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
> #19 0x00002abbb74f064d in clone () from /lib/libc.so.6
> #20 0x0000000000000000 in ?? ()
> (gdb) frame 3
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> warning: Source file is more recent than executable.
> 482 if (field->m_wks_idx < 0)
> (gdb) p *field
> $1 = {
> m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
> m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
> m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
Posted by "Ricky Chan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13018166#comment-13018166 ]
Ricky Chan commented on TS-733:
-------------------------------
Leif,
I'll try it for 2.1.7 in the lab on Monday (UK time). I've just been running it against 2.0.1 for over 12 hours now against my crash scripts to see if it fixes it.
> segfault in mime_hdr_set_accelerators_and_presence_bits
> -------------------------------------------------------
>
> Key: TS-733
> URL: https://issues.apache.org/jira/browse/TS-733
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
> Reporter: Ricky Chan
> Labels: MIME, segfault
> Fix For: 2.1.8
>
> Attachments: disable_dup.patch
>
>
> We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
> Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
> I got this segfault 6 times this morning, and it appears from the same client too.
> I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> 66 fp = (void **) (*fp);
> (gdb) bt
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> #1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
> #2 <signal handler called>
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> #4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
> #5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
> response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
> #6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
> #7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
> #8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
> #9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
> #10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
> #11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
> #12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
> #13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
> #14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
> #15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
> #16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
> #17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
> #18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
> #19 0x00002abbb74f064d in clone () from /lib/libc.so.6
> #20 0x0000000000000000 in ?? ()
> (gdb) frame 3
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> warning: Source file is more recent than executable.
> 482 if (field->m_wks_idx < 0)
> (gdb) p *field
> $1 = {
> m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
> m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
> m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
Posted by "Leif Hedstrom (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Leif Hedstrom updated TS-733:
-----------------------------
Fix Version/s: (was: 2.1.8)
2.0.2
Moving this over to v2.0.2, unless you can reproduce this crasher with current trunk (then please move it back).
> segfault in mime_hdr_set_accelerators_and_presence_bits
> -------------------------------------------------------
>
> Key: TS-733
> URL: https://issues.apache.org/jira/browse/TS-733
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
> Reporter: Ricky Chan
> Labels: MIME, segfault
> Fix For: 2.0.2
>
> Attachments: disable_dup.patch-2
>
>
> We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
> Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
> I got this segfault 6 times this morning, and it appears from the same client too.
> I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> 66 fp = (void **) (*fp);
> (gdb) bt
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> #1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
> #2 <signal handler called>
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> #4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
> #5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
> response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
> #6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
> #7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
> #8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
> #9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
> #10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
> #11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
> #12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
> #13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
> #14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
> #15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
> #16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
> #17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
> #18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
> #19 0x00002abbb74f064d in clone () from /lib/libc.so.6
> #20 0x0000000000000000 in ?? ()
> (gdb) frame 3
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> warning: Source file is more recent than executable.
> 482 if (field->m_wks_idx < 0)
> (gdb) p *field
> $1 = {
> m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
> m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
> m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
Posted by "Ricky Chan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ricky Chan updated TS-733:
--------------------------
Comment: was deleted
(was: I got a new segfault, for a different m_wks_idx index. This one is 36.
Below is slightly modified to hide real host header :-)
#0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
#1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
#2 <signal handler called>
#3 mime_hdr_field_detach (mh=0x65eccc8, field=0x65ecde0, detach_all_dups=false) at MIME.cc:482
#4 0x0000000000601eae in mime_hdr_field_delete (heap=0x65ecc10, mh=0x65eccc8, field=0x65ecde0, delete_all_dups=true) at MIME.cc:1737
#5 0x000000000057e796 in HttpTransactHeaders::copy_header_fields (src_hdr=<value optimized out>, new_hdr=0x2aaaba53a090, retain_proxy_auth_hdrs=false,
date=0) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
#6 0x00000000005629ae in HttpTransact::build_request (s=0x2aaaba5399b0, base_request=0x2aaaba53a020, outgoing_request=0x2aaaba53a090,
outgoing_version=<value optimized out>) at HttpTransact.cc:9086
#7 0x00000000005790f6 in HttpTransact::HandleCacheOpenReadHit (s=0x2aaaba5399b0) at HttpTransact.cc:3009
#8 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba539930, f=0x65ecde0) at HttpSM.cc:7190
#9 0x000000000054b35b in HttpSM::set_next_state (this=0x2aaaba539930) at HttpSM.cc:7232
#10 0x0000000000534469 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba539930, f=<value optimized out>) at HttpSM.cc:7198
#11 0x0000000000536a62 in HttpSM::do_hostdb_lookup (this=0x2aaaba539930) at HttpSM.cc:4079
#12 0x000000000054b5db in HttpSM::set_next_state (this=0x2aaaba539930) at HttpSM.cc:7278
#13 0x0000000000534469 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba539930, f=<value optimized out>) at HttpSM.cc:7198
#14 0x000000000054b35b in HttpSM::set_next_state (this=0x2aaaba539930) at HttpSM.cc:7232
#15 0x0000000000534469 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba539930, f=<value optimized out>) at HttpSM.cc:7198
#16 0x000000000054b35b in HttpSM::set_next_state (this=0x2aaaba539930) at HttpSM.cc:7232
#17 0x0000000000534469 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba539930, f=<value optimized out>) at HttpSM.cc:7198
#18 0x0000000000549491 in HttpSM::state_cache_open_read (this=0x2aaaba539930, event=1102, data=0xbf281f0) at HttpSM.cc:2605
#19 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba539930, event=1102, data=0xbf281f0) at HttpSM.cc:2683
#20 0x000000000052457e in HttpCacheSM::state_cache_open_read (this=0x2aaaba53b818, event=<value optimized out>, data=0xbf281f0)
at ../../iocore/eventsystem/I_Continuation.h:147
#21 0x0000000000694df4 in CacheVC::openReadStartEarliest (this=0xbf281f0, event=<value optimized out>, e=<value optimized out>) at CacheRead.cc:761
#22 0x000000000066e3f0 in CacheVC::handleReadDone (this=0xbf281f0, event=<value optimized out>, e=<value optimized out>)
at ../../iocore/eventsystem/I_Continuation.h:147
#23 0x000000000067572b in AIOCallbackInternal::io_complete (this=<value optimized out>, event=<value optimized out>, data=<value optimized out>)
at ../../iocore/eventsystem/I_Continuation.h:147
#24 0x00000000006e3644 in EThread::process_event (this=0x2aaaac64c010, e=0x33c8200, calling_code=1) at I_Continuation.h:147
#25 0x00000000006e3f8b in EThread::execute (this=0x2aaaac64c010) at UnixEThread.cc:168
#26 0x00000000006e1aa2 in spawn_thread_internal (a=0x2a93210) at Thread.cc:85
#27 0x00002b6dca298fc7 in start_thread () from /lib/libpthread.so.0
#28 0x00002b6dcba0a64d in clone () from /lib/libc.so.6
#29 0x0000000000000000 in ?? ()
(gdb) frame 3
#3 mime_hdr_field_detach (mh=0x65eccc8, field=0x65ecde0, detach_all_dups=false) at MIME.cc:482
482 MIME.cc: No such file or directory.
in MIME.cc
(gdb) p *field
$3 = {
m_ptr_name = 0x2aaaceca918e "Keep-Alive: 115\r\nUser-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.16) Gecko/20110319 Firefox/3.6.16\r\nConnection: keep-alive\r\nKeep-Alive: 115\r\n\r\n90/c_3.gif HTTP/1.1\r\nHost: www.teste"...,
m_ptr_value = 0x2aaaceca919a "115\r\nUser-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.16) Gecko/20110319 Firefox/3.6.16\r\nConnection: keep-alive\r\nKeep-Alive: 115\r\n\r\n90/c_3.gif HTTP/1.1\r\nHost: www.testetf1.com\r\nUse"..., m_next_dup = 0x2aab065ece40, m_wks_idx = 36,
m_len_name = 10, m_len_value = 3, m_n_v_raw_printable = 1 '\001', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
(gdb) p *mh
$4 = {<HdrHeapObjImpl> = {m_type = 4, m_length = 600, m_obj_flags = 0}, m_presence_bits = 2251800904730639, m_slot_accelerators = {4294914576, 4283432952,
4294967295, 4294443007}, m_cooked_stuff = {m_cache_control = {m_mask = 0, m_secs_max_age = 0, m_secs_s_maxage = 0, m_secs_max_stale = 0,
m_secs_min_fresh = 0}, m_pragma = {m_no_cache = false}}, m_fblock_list_tail = 0x65ecd08, m_first_fblock = {<HdrHeapObjImpl> = {m_type = 5,
m_length = 536, m_obj_flags = 0}, m_next = 0x0, m_freetop = 10, m_field_slots = {{
m_ptr_name = 0x2aaaceca901b "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept-Encoding: deflate, gzip, identity\r\nAccept-Language: en-gb,en;q=0.5\r\nCook"...,
m_ptr_value = 0x2aaaceca9023 "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept-Encoding: deflate, gzip, identity\r\nAccept-Language: en-gb,en;q=0.5\r\nCookie: rsi_"..., m_next_dup = 0x0, m_wks_idx = 0, m_len_name = 6,
m_len_value = 63, m_n_v_raw_printable = 1 '\001', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}, {
m_ptr_name = 0x2aaaceca9064 "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept-Encoding: deflate, gzip, identity\r\nAccept-Language: en-gb,en;q=0.5\r\nCookie: rsi_segs=G07611_10219|G07611_10469|G07611_10484|G07611_10487|G07611_1"...,
m_ptr_value = 0x2aaaceca9074 "ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept-Encoding: deflate, gzip, identity\r\nAccept-Language: en-gb,en;q=0.5\r\nCookie: rsi_segs=G07611_10219|G07611_10469|G07611_10484|G07611_10487|G07611_10491; _em_vt=0a9"..., m_next_dup = 0x0, m_wks_idx = 1, m_len_name = 14,
m_len_value = 30, m_n_v_raw_printable = 1 '\001', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}, {
m_ptr_name = 0x2aaaceca9094 "Accept-Encoding: deflate, gzip, identity\r\nAccept-Language: en-gb,en;q=0.5\r\nCookie: rsi_segs=G07611_10219|G07611_10469|G07611_10484|G07611_10487|G07611_10491; _em_vt=0a9dde6ceeab7a2d189dd364a4784d9197e"...,
m_ptr_value = 0x2aaaceca90a5 "deflate, gzip, identity\r\nAccept-Language: en-gb,en;q=0.5\r\nCookie: rsi_segs=G07611_10219|G07611_10469|G07611_10484|G07611_10487|G07611_10491; _em_vt=0a9dde6ceeab7a2d189dd364a4784d9197ea8a91a2-977621484"..., m_next_dup = 0x0, m_wks_idx = 2, m_len_name = 15,
m_len_value = 23, m_n_v_raw_printable = 1 '\001', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}, {
m_ptr_name = 0x2aaaceca90be "Accept-Language: en-gb,en;q=0.5\r\nCookie: rsi_segs=G07611_10219|G07611_10469|G07611_10484|G07611_10487|G07611_10491; _em_vt=0a9dde6ceeab7a2d189dd364a4784d9197ea8a91a2-977621484da018ae\r\nHost: www.testet"...,
m_ptr_value = 0x2aaaceca90cf "en-gb,en;q=0.5\r\nCookie: rsi_segs=G07611_10219|G07611_10469|G07611_10484|G07611_10487|G07611_10491; _em_vt=0a9dde6ceeab7a2d189dd364a4784d9197ea8a91a2-977621484da018ae\r\nHost: www.testetf1.com\r\nKeep-Aliv"..., m_next_dup = 0x0, m_wks_idx = 3, m_len_name = 15,
m_len_value = 14, m_n_v_raw_printable = 1 '\001', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}, {
m_ptr_name = 0x2aaaceca90df "Cookie: rsi_segs=G07611_10219|G07611_10469|G07611_10484|G07611_10487|G07611_10491; _em_vt=0a9dde6ceeab7a2d189dd364a4784d9197ea8a91a2-977621484da018ae\r\nHost: www.testetf1.com\r\nKeep-Alive: 115\r\nUser-Age"...,
m_ptr_value = 0x2aaaceca90e7 "rsi_segs=G07611_10219|G07611_10469|G07611_10484|G07611_10487|G07611_10491; _em_vt=0a9dde6ceeab7a2d189dd364a4784d9197ea8a91a2-977621484da018ae\r\nHost: www.testetf1.com\r\nKeep-Alive: 115\r\nUser-Agent: Mozi"..., m_next_dup = 0x0, m_wks_idx = 22, m_len_name = 6,
m_len_value = 141, m_n_v_raw_printable = 1 '\001', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}, {
m_ptr_name = 0x2aaaceca9176 "Host: www.testetf1.com\r\nKeep-Alive: 115\r\nUser-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.16) Gecko/20110319 Firefox/3.6.16\r\nConnection: keep-alive\r\nKeep-Alive: 115\r\n\r\n90/c_3.gif H"...,
m_ptr_value = 0x2aaaceca917c "www.testetf1.com\r\nKeep-Alive: 115\r\nUser-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.16) Gecko/20110319 Firefox/3.6.16\r\nConnection: keep-alive\r\nKeep-Alive: 115\r\n\r\n90/c_3.gif HTTP/1."..., m_next_dup = 0x0, m_wks_idx = 30, m_len_name = 4,
m_len_value = 16, m_n_v_raw_printable = 1 '\001', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}, {
m_ptr_name = 0x2aaaceca918e "Keep-Alive: 115\r\nUser-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.16) Gecko/20110319 Firefox/3.6.16\r\nConnection: keep-alive\r\nKeep-Alive: 115\r\n\r\n90/c_3.gif HTTP/1.1\r\nHost: www.teste"...,
m_ptr_value = 0x2aaaceca919a "115\r\nUser-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.16) Gecko/20110319 Firefox/3.6.16\r\nConnection: keep-alive\r\nKeep-Alive: 115\r\n\r\n90/c_3.gif HTTP/1.1\r\nHost: www.testetf1.com\r\nUse"..., m_next_dup = 0x2aab065ece40, m_wks_idx = 36,
m_len_name = 10, m_len_value = 3, m_n_v_raw_printable = 1 '\001', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}, {
m_ptr_name = 0x2aaaceca919f "User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.16) Gecko/20110319 Firefox/3.6.16\r\nConnection: keep-alive\r\nKeep-Alive: 115\r\n\r\n90/c_3.gif HTTP/1.1\r\nHost: www.testetf1.com\r\nUser-Age"...,
m_ptr_value = 0x2aaaceca91ab "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.16) Gecko/20110319 Firefox/3.6.16\r\nConnection: keep-alive\r\nKeep-Alive: 115\r\n\r\n90/c_3.gif HTTP/1.1\r\nHost: www.testetf1.com\r\nUser-Agent: Mozilla/"..., m_next_dup = 0x0, m_wks_idx = 65, m_len_name = 10,
m_len_value = 90, m_n_v_raw_printable = 1 '\001', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}, {
m_ptr_name = 0x2aaaceca9207 "Connection: keep-alive\r\nKeep-Alive: 115\r\n\r\n90/c_3.gif HTTP/1.1\r\nHost: www.testetf1.com\r\nUser-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.16) Gecko/20110319 Firefox/3.6.16\r\nAccept: "...,
m_ptr_value = 0x2aaaceca9213 "keep-alive\r\nKeep-Alive: 115\r\n\r\n90/c_3.gif HTTP/1.1\r\nHost: www.testetf1.com\r\nUser-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.16) Gecko/20110319 Firefox/3.6.16\r\nAccept: image/png,im"..., m_next_dup = 0x0, m_wks_idx = 12, m_len_name = 10,
m_len_value = 10, m_n_v_raw_printable = 1 '\001', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}, {
m_ptr_name = 0x2aaaceca921f "Keep-Alive: 115\r\n\r\n90/c_3.gif HTTP/1.1\r\nHost: www.testetf1.com\r\nUser-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.16) Gecko/20110319 Firefox/3.6.16\r\nAccept: image/png,image/*;q=0.8,"...,
m_ptr_value = 0x2aaaceca922b "115\r\n\r\n90/c_3.gif HTTP/1.1\r\nHost: www.testetf1.com\r\nUser-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.16) Gecko/20110319 Firefox/3.6.16\r\nAccept: image/png,image/*;q=0.8,*/*;q=0.5\r\nA"..., m_next_dup = 0x0, m_wks_idx = 36, m_len_name = 10,
m_len_value = 3, m_n_v_raw_printable = 1 '\001', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 0 '\0'}, {
m_ptr_name = 0x7cc6168 "Transfer-Encoding: chunked\r\nConnection: keep-alive\r\n\r\nApr 2011 09:42:01 GMT\r\nVia: 1.0 squid15.teamtalk.com:80 (squid/2.HEAD-CVS)\r\nConnection: keep-alive\r\n\r\n/*\n * Thickbox 3.1 - One Box To Rule Them Al"...,
m_ptr_value = 0x7cc617b "chunked\r\nConnection: keep-alive\r\n\r\nApr 2011 09:42:01 GMT\r\nVia: 1.0 squid15.teamtalk.com:80 (squid/2.HEAD-CVS)\r\nConnection: keep-alive\r\n\r\n/*\n * Thickbox 3.1 - One Box To Rule Them All.\n * By Cody Lindl"..., m_next_dup = 0x0, m_wks_idx = 63, m_len_name = 17,
m_len_value = 7, m_n_v_raw_printable = 1 '\001', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 3 '\003', m_flags = 1 '\001'}, {
m_ptr_name = 0x7cc6184 "Connection: keep-alive\r\n\r\nApr 2011 09:42:01 GMT\r\nVia: 1.0 squid15.teamtalk.com:80 (squid/2.HEAD-CVS)\r\nConnection: keep-alive\r\n\r\n/*\n * Thickbox 3.1 - One Box To Rule Them All.\n * By Cody Lindley (http:"...,
m_ptr_value = 0x7cc6190 "keep-alive\r\n\r\nApr 2011 09:42:01 GMT\r\nVia: 1.0 squid15.teamtalk.com:80 (squid/2.HEAD-CVS)\r\nConnection: keep-alive\r\n\r\n/*\n * Thickbox 3.1 - One Box To Rule Them All.\n * By Cody Lindley (http://www.codyli"..., m_next_dup = 0x0, m_wks_idx = 12, m_len_name = 10,
m_len_value = 10, m_n_v_raw_printable = 1 '\001', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}, {
m_ptr_name = 0x2aaab46d4b81 "0.244.132.255omClient-ip212.183.128.69X-Forwarded-For212.183.128.69If-Modified-SinceMon, 04 Apr 2011 10:48:28 GMTIf-None-Match\"20d1a-b011-4a0157ff6c7c5\"2.gif",
m_ptr_value = 0x2aaab46d4b92 "ient-ip212.183.128.69X-Forwarded-For212.183.128.69If-Modified-SinceMon, 04 Apr 2011 10:48:28 GMTIf-None-Match\"20d1a-b011-4a0157ff6c7c5\"2.gif", m_next_dup = 0x0, m_wks_idx = 63, m_len_name = 17, m_len_value = 7, m_n_v_raw_printable = 0 '\0',
m_n_v_raw_printable_pad = 0 '\0', m_readiness = 2 '\002', m_flags = 1 '\001'}, {
m_ptr_name = 0x2aaab46d4b99 "212.183.128.69X-Forwarded-For212.183.128.69If-Modified-SinceMon, 04 Apr 2011 10:48:28 GMTIf-None-Match\"20d1a-b011-4a0157ff6c7c5\"2.gif",
m_ptr_value = 0x2aaab46d4ba3 "8.69X-Forwarded-For212.183.128.69If-Modified-SinceMon, 04 Apr 2011 10:48:28 GMTIf-None-Match\"20d1a-b011-4a0157ff6c7c5\"2.gif", m_next_dup = 0x0, m_wks_idx = 12, m_len_name = 10, m_len_value = 10, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 0 '\0',
m_readiness = 2 '\002', m_flags = 1 '\001'}, {
m_ptr_name = 0x2aaab46d03a5 "liveATS/2.0.125d7c-2c1c-490d5d2540c80\"Connectionkeep-aliveServerATS/2.0.1MTIf-None-Match\"1ed1d-c88-4a01598a14c14\"/c_3.gif", m_ptr_value = 0x2aaab46d03b4 "d7c-2c1c-490d5d2540c80\"Connectionkeep-aliveServerATS/2.0.1MTIf-None-Match\"1ed1d-c88-4a01598a14c14\"/c_3.gif",
m_next_dup = 0x0, m_wks_idx = 122, m_len_name = 15, m_len_value = 12, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 0 '\0',
m_readiness = 2 '\002', m_flags = 1 '\001'}, {
m_ptr_name = 0x2aaab46d03c0 "d5d2540c80\"Connectionkeep-aliveServerATS/2.0.1MTIf-None-Match\"1ed1d-c88-4a01598a14c14\"/c_3.gif",
m_ptr_value = 0x2aaab46d03d1 "tionkeep-aliveServerATS/2.0.1MTIf-None-Match\"1ed1d-c88-4a01598a14c14\"/c_3.gif", m_next_dup = 0x0, m_wks_idx = 32,
m_len_name = 17, m_len_value = 29, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 0 '\0', m_readiness = 2 '\002', m_flags = 1 '\001'}}}}
)
> segfault in mime_hdr_set_accelerators_and_presence_bits
> -------------------------------------------------------
>
> Key: TS-733
> URL: https://issues.apache.org/jira/browse/TS-733
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
> Reporter: Ricky Chan
> Labels: MIME, segfault
> Fix For: 2.1.8
>
>
> We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
> Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
> I got this segfault 6 times this morning, and it appears from the same client too.
> I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> 66 fp = (void **) (*fp);
> (gdb) bt
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> #1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
> #2 <signal handler called>
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> #4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
> #5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
> response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
> #6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
> #7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
> #8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
> #9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
> #10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
> #11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
> #12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
> #13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
> #14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
> #15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
> #16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
> #17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
> #18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
> #19 0x00002abbb74f064d in clone () from /lib/libc.so.6
> #20 0x0000000000000000 in ?? ()
> (gdb) frame 3
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> warning: Source file is more recent than executable.
> 482 if (field->m_wks_idx < 0)
> (gdb) p *field
> $1 = {
> m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
> m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
> m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
Posted by "Ricky Chan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017440#comment-13017440 ]
Ricky Chan commented on TS-733:
-------------------------------
The accept line came out incorrect it is meant to be <star>/<star> not /
> segfault in mime_hdr_set_accelerators_and_presence_bits
> -------------------------------------------------------
>
> Key: TS-733
> URL: https://issues.apache.org/jira/browse/TS-733
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
> Reporter: Ricky Chan
> Labels: MIME, segfault
>
> We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
> Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
> I got this segfault 6 times this morning, and it appears from the same client too.
> I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> 66 fp = (void **) (*fp);
> (gdb) bt
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> #1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
> #2 <signal handler called>
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> #4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
> #5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
> response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
> #6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
> #7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
> #8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
> #9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
> #10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
> #11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
> #12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
> #13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
> #14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
> #15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
> #16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
> #17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
> #18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
> #19 0x00002abbb74f064d in clone () from /lib/libc.so.6
> #20 0x0000000000000000 in ?? ()
> (gdb) frame 3
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> warning: Source file is more recent than executable.
> 482 if (field->m_wks_idx < 0)
> (gdb) p *field
> $1 = {
> m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
> m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
> m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
Posted by "Ricky Chan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017449#comment-13017449 ]
Ricky Chan commented on TS-733:
-------------------------------
To trigger bug, just need to keep purge cache, as it's seems some sort of race condition.
> segfault in mime_hdr_set_accelerators_and_presence_bits
> -------------------------------------------------------
>
> Key: TS-733
> URL: https://issues.apache.org/jira/browse/TS-733
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
> Reporter: Ricky Chan
> Labels: MIME, segfault
>
> We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
> Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
> I got this segfault 6 times this morning, and it appears from the same client too.
> I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> 66 fp = (void **) (*fp);
> (gdb) bt
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> #1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
> #2 <signal handler called>
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> #4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
> #5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
> response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
> #6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
> #7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
> #8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
> #9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
> #10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
> #11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
> #12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
> #13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
> #14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
> #15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
> #16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
> #17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
> #18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
> #19 0x00002abbb74f064d in clone () from /lib/libc.so.6
> #20 0x0000000000000000 in ?? ()
> (gdb) frame 3
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> warning: Source file is more recent than executable.
> 482 if (field->m_wks_idx < 0)
> (gdb) p *field
> $1 = {
> m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
> m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
> m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
Posted by "Leif Hedstrom (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017967#comment-13017967 ]
Leif Hedstrom commented on TS-733:
----------------------------------
This is a totally different stack trace, right? Can describe exactly how you reproduced it? I tried with the multiple Via headers etc, but did not reproduce.
Also, what value is w when you segfault?
Thanks for all the work, I think we're making some progress (and as you have noticed we certainly should handle multiple instances of a header, so I suspect that's just a side effect in 2.0.1, of something else going wrong)
> segfault in mime_hdr_set_accelerators_and_presence_bits
> -------------------------------------------------------
>
> Key: TS-733
> URL: https://issues.apache.org/jira/browse/TS-733
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
> Reporter: Ricky Chan
> Labels: MIME, segfault
> Fix For: 2.1.8
>
>
> We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
> Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
> I got this segfault 6 times this morning, and it appears from the same client too.
> I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> 66 fp = (void **) (*fp);
> (gdb) bt
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> #1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
> #2 <signal handler called>
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> #4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
> #5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
> response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
> #6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
> #7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
> #8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
> #9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
> #10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
> #11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
> #12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
> #13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
> #14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
> #15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
> #16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
> #17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
> #18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
> #19 0x00002abbb74f064d in clone () from /lib/libc.so.6
> #20 0x0000000000000000 in ?? ()
> (gdb) frame 3
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> warning: Source file is more recent than executable.
> 482 if (field->m_wks_idx < 0)
> (gdb) p *field
> $1 = {
> m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
> m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
> m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
Posted by "Ricky Chan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017855#comment-13017855 ]
Ricky Chan commented on TS-733:
-------------------------------
I've just download 2.1.7 and compared MIME.cc and MIME.h between that and 2.0.1.
I (could be wrong but I) can't see any real code change apart from (mainly) moving away from inkomti properitary types.
I just want to mention in my version of 2.0.1 in about line 1250 in MIME.cc
ink_release_assert(j < block_count);
This is disabled, because it get's trigger and causes an abort (I've already reported this) and I don't believe this is fixed in 2.1.7 (speculation) based on the content of 2.1.7 MIME.cc code isn't much different from 2.0.1. All my reported errors seems to originate from here.
It's disabled because in my lab test it didn't affect the system although the next_dup is pointing to something beyond it's self and therefore it could be triggering all the subsequent headers.
I may need to add a different patch (such as don't treat it as a dup when it points beyond it self as the RFC seems to indicate multiple tags of the same name is valid) and may be collapsed or not. From my (very limited) understanding of the code I believe this will be fine.
The core bug I believe is the dup code or the data it points to becomes invalid (by another thread perhaps?).
I don't believe this (again I could be wrong) is addressed by 2.1.7.
> segfault in mime_hdr_set_accelerators_and_presence_bits
> -------------------------------------------------------
>
> Key: TS-733
> URL: https://issues.apache.org/jira/browse/TS-733
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
> Reporter: Ricky Chan
> Labels: MIME, segfault
> Fix For: 2.1.8
>
>
> We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
> Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
> I got this segfault 6 times this morning, and it appears from the same client too.
> I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> 66 fp = (void **) (*fp);
> (gdb) bt
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> #1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
> #2 <signal handler called>
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> #4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
> #5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
> response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
> #6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
> #7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
> #8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
> #9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
> #10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
> #11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
> #12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
> #13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
> #14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
> #15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
> #16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
> #17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
> #18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
> #19 0x00002abbb74f064d in clone () from /lib/libc.so.6
> #20 0x0000000000000000 in ?? ()
> (gdb) frame 3
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> warning: Source file is more recent than executable.
> 482 if (field->m_wks_idx < 0)
> (gdb) p *field
> $1 = {
> m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
> m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
> m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
Posted by "Ricky Chan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ricky Chan updated TS-733:
--------------------------
Attachment: disable_dup.patch-2
This should more reliably not run the duplication patching code.
The previous patch, I can see in attach and detach it still had m_flags with out MIME_FIELD_SLOT_FLAGS_DUP_HEAD set and caused a crash finding dups to delete (when there isn't any).
Probably could also remove the delete_all_dups part of mime_hdr_delete as well, as it *should* only run in the loop once.
> segfault in mime_hdr_set_accelerators_and_presence_bits
> -------------------------------------------------------
>
> Key: TS-733
> URL: https://issues.apache.org/jira/browse/TS-733
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
> Reporter: Ricky Chan
> Labels: MIME, segfault
> Fix For: 2.1.8
>
> Attachments: disable_dup.patch-2
>
>
> We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
> Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
> I got this segfault 6 times this morning, and it appears from the same client too.
> I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> 66 fp = (void **) (*fp);
> (gdb) bt
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> #1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
> #2 <signal handler called>
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> #4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
> #5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
> response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
> #6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
> #7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
> #8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
> #9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
> #10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
> #11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
> #12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
> #13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
> #14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
> #15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
> #16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
> #17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
> #18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
> #19 0x00002abbb74f064d in clone () from /lib/libc.so.6
> #20 0x0000000000000000 in ?? ()
> (gdb) frame 3
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> warning: Source file is more recent than executable.
> 482 if (field->m_wks_idx < 0)
> (gdb) p *field
> $1 = {
> m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
> m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
> m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
Posted by "Ricky Chan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13018343#comment-13018343 ]
Ricky Chan commented on TS-733:
-------------------------------
Annoyingly another small bug appeared in production, so I had to apply another fix and use the lab for testing.
#3 mime_hdr_field_detach (mh=0x20e3df8, field=0x20e3f90, detach_all_dups=false) at MIME.cc:1639
(gdb) p *field
$1 = {
m_ptr_name = 0x2aaabf1188e2 "Keep-Alive: 115\r\n\r\nNT 5.1; msn OptimizedIE8;ENGB)\r\nHost: www.planet-f1.com\r\nCookie: rsi_segs=G07611_10219; _em_vt=a3caed7c960fca32a042dc54f1504d7890244ab394-068710124d9f0a50; CP=null*\r\nAccept: */*\r\nRe"...,
m_ptr_value = 0x2aaabf1188ee "115\r\n\r\nNT 5.1; msn OptimizedIE8;ENGB)\r\nHost: www.planet-f1.com\r\nCookie: rsi_segs=G07611_10219; _em_vt=a3caed7c960fca32a042dc54f1504d7890244ab394-068710124d9f0a50; CP=null*\r\nAccept: */*\r\nReferer: http:"..., m_next_dup = 0x0, m_wks_idx = 36, m_len_name = 10,
m_len_value = 3, m_n_v_raw_printable = 1 '\001', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 0 '\0'}
m_next_dup is set to NULL as per my patch, but m_flags has been had MIME_FIELD_SLOT_FLAGS_DUP_HEAD isn't set!!
So I added this patch into the "mime_hdr_field_detach" function..
if(!next_dup)
field->m_flags |= MIME_FIELD_SLOT_FLAGS_DUP_HEAD;
I was trying to get away with minimal amount of changes to disable looking at dups, but I may have to remove it from "mime_hdr_field_attach" as well.
This is more a workaround than a fix. Something (another thread perhaps?) is resetting m_flags which is probably the same thing which is messing up m_next_dup.
I'll test 2.1.7 once 2.0.1 stabilises.
> segfault in mime_hdr_set_accelerators_and_presence_bits
> -------------------------------------------------------
>
> Key: TS-733
> URL: https://issues.apache.org/jira/browse/TS-733
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
> Reporter: Ricky Chan
> Labels: MIME, segfault
> Fix For: 2.1.8
>
> Attachments: disable_dup.patch
>
>
> We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
> Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
> I got this segfault 6 times this morning, and it appears from the same client too.
> I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> 66 fp = (void **) (*fp);
> (gdb) bt
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> #1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
> #2 <signal handler called>
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> #4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
> #5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
> response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
> #6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
> #7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
> #8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
> #9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
> #10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
> #11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
> #12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
> #13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
> #14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
> #15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
> #16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
> #17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
> #18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
> #19 0x00002abbb74f064d in clone () from /lib/libc.so.6
> #20 0x0000000000000000 in ?? ()
> (gdb) frame 3
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> warning: Source file is more recent than executable.
> 482 if (field->m_wks_idx < 0)
> (gdb) p *field
> $1 = {
> m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
> m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
> m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
Posted by "Ricky Chan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ricky Chan updated TS-733:
--------------------------
Comment: was deleted
(was: The second "Keep-Alive: 115\r\n\r\n" seems to have trashed part of the address of another request?)
> segfault in mime_hdr_set_accelerators_and_presence_bits
> -------------------------------------------------------
>
> Key: TS-733
> URL: https://issues.apache.org/jira/browse/TS-733
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
> Reporter: Ricky Chan
> Labels: MIME, segfault
> Fix For: 2.1.8
>
>
> We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
> Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
> I got this segfault 6 times this morning, and it appears from the same client too.
> I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> 66 fp = (void **) (*fp);
> (gdb) bt
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> #1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
> #2 <signal handler called>
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> #4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
> #5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
> response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
> #6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
> #7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
> #8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
> #9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
> #10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
> #11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
> #12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
> #13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
> #14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
> #15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
> #16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
> #17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
> #18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
> #19 0x00002abbb74f064d in clone () from /lib/libc.so.6
> #20 0x0000000000000000 in ?? ()
> (gdb) frame 3
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> warning: Source file is more recent than executable.
> 482 if (field->m_wks_idx < 0)
> (gdb) p *field
> $1 = {
> m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
> m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
> m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-733) segfault in
mime_hdr_set_accelerators_and_presence_bits
Posted by "Ricky Chan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017360#comment-13017360 ]
Ricky Chan commented on TS-733:
-------------------------------
Another note, the 6 segfaults on the core dumps show the crash was caused by the same
cluster of forward proxies based in NZ.
Namely a BC forward proxy called: "1.1 AKmdrL2CacheBC10.telecom.co.nz"
This proxy looks a bit buggy it sends a lot of multiple Via headers (as seen above) and I suspect it is the cause of the insert via request str segfaults (this I can't prove as this feature is turned off).
> segfault in mime_hdr_set_accelerators_and_presence_bits
> -------------------------------------------------------
>
> Key: TS-733
> URL: https://issues.apache.org/jira/browse/TS-733
> Project: Traffic Server
> Issue Type: Bug
> Components: MIME
> Affects Versions: 2.0.1
> Environment: X6240 AMD64 Debian Lenny (2.6.26) 64G of Ram.
> Reporter: Ricky Chan
> Labels: MIME, segfault
>
> We are seeing segfault and I have now put back unstripped binaries so I can get line numbers are frame traces.
> Below is the trace, although GDB claims it's line 482, I believe it's now actually there (a short int comparison won't crash it). My interest is the fact that m_wks_idx is 67 which is larger than the MAX amount of slots which I believe is 16 (0 - 15) right?
> I got this segfault 6 times this morning, and it appears from the same client too.
> I'm thinking of patching the code to make sure m_wks_idx isn't > MAX_FIELD_SLOTNUM_MAX for now.
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> 66 fp = (void **) (*fp);
> (gdb) bt
> #0 ink_stack_trace_dump (sighandler_frame=2) at ink_stack_trace.cc:66
> #1 0x0000000000502f8a in signal_handler (sig=<value optimized out>) at signals.cc:332
> #2 <signal handler called>
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> #4 0x0000000000601e8e in mime_hdr_field_delete (heap=0x2aaab46c11e0, mh=0x2aaab46c1298, field=0x2aaab46c1390, delete_all_dups=true) at MIME.cc:1737
> #5 0x000000000056cee9 in HttpTransact::set_headers_for_cache_write (s=0x2aaaba56d8b0, cache_info=0x2aaaba56d948, request=0x2aaaba56df90,
> response=0x2aaaba56dfc8) at ../../iocore/cache/../../proxy/http2/../hdrs/MIME.h:1071
> #6 0x000000000056ec29 in HttpTransact::handle_cache_operation_on_forward_server_response (s=0x2aaaba56d8b0) at HttpTransact.cc:5270
> #7 0x000000000056ff99 in HttpTransact::handle_forward_server_connection_open (s=0x2aaaba56d8b0) at HttpTransact.cc:4732
> #8 0x0000000000572370 in HttpTransact::handle_response_from_server (s=0x2aaaba56d8b0) at HttpTransact.cc:4255
> #9 0x0000000000578a5d in HttpTransact::HandleResponse (s=0x2aaaba56d8b0) at HttpTransact.cc:3937
> #10 0x0000000000534485 in HttpSM::call_transact_and_set_next_state (this=0x2aaaba56d830, f=0x2aaab46c1390) at HttpSM.cc:7190
> #11 0x0000000000549aa0 in HttpSM::state_read_server_response_header (this=0x2aaaba56d830, event=<value optimized out>, data=0x2232e28) at HttpSM.cc:535
> #12 0x0000000000547e3b in HttpSM::main_handler (this=0x2aaaba56d830, event=100, data=0x2232e28) at HttpSM.cc:2683
> #13 0x00000000006c19f7 in read_from_net (nh=0x2aaaac950098, vc=0x2232d50, thread=<value optimized out>) at ../../iocore/eventsystem/I_Continuation.h:147
> #14 0x00000000006b9452 in NetHandler::mainNetEvent (this=0x2aaaac950098, event=<value optimized out>, e=0xfa9130) at UnixNet.cc:292
> #15 0x00000000006e3614 in EThread::process_event (this=0x2aaaac94f010, e=0xfa9130, calling_code=5) at I_Continuation.h:147
> #16 0x00000000006e3e50 in EThread::execute (this=0x2aaaac94f010) at UnixEThread.cc:249
> #17 0x00000000006e1a72 in spawn_thread_internal (a=0xf93270) at Thread.cc:85
> #18 0x00002abbb5d7efc7 in start_thread () from /lib/libpthread.so.0
> #19 0x00002abbb74f064d in clone () from /lib/libc.so.6
> #20 0x0000000000000000 in ?? ()
> (gdb) frame 3
> #3 mime_hdr_field_detach (mh=0x2aaab46c1298, field=0x2aaab46c1390, detach_all_dups=false) at MIME.cc:482
> warning: Source file is more recent than executable.
> 482 if (field->m_wks_idx < 0)
> (gdb) p *field
> $1 = {
> m_ptr_name = 0x19618e4 "Via1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om"...,
> m_ptr_value = 0x19618e7 "1.1 AKmdrL2CacheBC10.telecom.co.nzCache-Controlmax-stale=0Via1.1 AKmdrL2CacheBC10.telecom.co.nzX-BlueCoat-ViaB8C344C089BFFBD7Client-ip210.55.215.151X-Forwarded-For210.55.215.151http10.244.132.255om0yo"..., m_next_dup = 0xffffffffb46c87f0, m_wks_idx = 67, m_len_name = 3,
> m_len_value = 34, m_n_v_raw_printable = 0 '\0', m_n_v_raw_printable_pad = 4 '\004', m_readiness = 2 '\002', m_flags = 1 '\001'}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira