[jira] [Created] (TS-3078) MIMEHdrImpl::unmarshal crash caused by cache corruption

["kang li (JIRA)" <ji...@apache.org>]

kang li created TS-3078:
---------------------------

             Summary: MIMEHdrImpl::unmarshal crash caused by cache corruption
                 Key: TS-3078
                 URL: https://issues.apache.org/jira/browse/TS-3078
             Project: Traffic Server
          Issue Type: Bug
          Components: Cache, MIME
            Reporter: kang li



{code}
(gdb) bt
#0  0x00000000005c22c6 in unmarshal (this=0x2aaed05f61f8, offset=46930308587840) at MIME.cc:3534
#1  MIMEHdrImpl::unmarshal (this=0x2aaed05f61f8, offset=46930308587840) at MIME.cc:3590
#2  0x00000000005b4bcb in HdrHeap::unmarshal (this=0x2aaed05f6140, buf_length=<value optimized out>, obj_type=<value optimized out>, found_obj=<value optimized out>, 
    block_ref=<value optimized out>) at HdrHeap.cc:926
#3  0x00000000005b8be1 in HTTPInfo::unmarshal (buf=0x2aaed05f6048 "\355\336\315\253", len=3280, block_ref=0x2aae080feec0) at HTTP.cc:1948
#4  0x0000000000635e43 in unmarshal_helper (this=0x2aae10081b90, event=<value optimized out>, e=<value optimized out>) at Cache.cc:2066
#5  CacheVC::handleReadDone (this=0x2aae10081b90, event=<value optimized out>, e=<value optimized out>) at Cache.cc:2195
#6  0x00000000005f1845 in handleEvent (this=<value optimized out>, event=<value optimized out>, data=<value optimized out>)
    at ../../iocore/eventsystem/I_Continuation.h:146
#7  AIOCallbackInternal::io_complete (this=<value optimized out>, event=<value optimized out>, data=<value optimized out>) at ../../iocore/aio/P_AIO.h:123
#8  0x00000000006a990f in handleEvent (this=0x2aadf0809010, e=0x2aae3812b710, calling_code=1) at I_Continuation.h:146
#9  EThread::process_event (this=0x2aadf0809010, e=0x2aae3812b710, calling_code=1) at UnixEThread.cc:141
#10 0x00000000006aa48b in EThread::execute (this=0x2aadf0809010) at UnixEThread.cc:192
#11 0x00000000006a87aa in spawn_thread_internal (a=0x13e0a80) at Thread.cc:88
#12 0x00002aadeaf3c9d1 in start_thread () from /lib64/libpthread.so.0
#13 0x00000034cfce8b6d in clone () from /lib64/libc.so.6
(gdb) p m_freetop
$1 = 892220471
(gdb) p m_field_slots
$2 = {{m_ptr_name = 0x33203a2274616c22 <Address 0x33203a2274616c22 out of bounds>, m_ptr_value = 0x3939333338312e33 <Address 0x3939333338312e33 out of bounds>, 
    m_next_dup = 0x3a226e6f6c22202c, m_wks_idx = 11552, m_len_name = 14137, m_len_value = 3289390, m_n_v_raw_printable = 0 '\000', m_n_v_raw_printable_pad = 3 '\003', 
    m_readiness = 3 '\003', m_flags = 0 '\000'}, {m_ptr_name = 0x766c4ccefc939174 <Address 0x766c4ccefc939174 out of bounds>, 
    m_ptr_value = 0x222056def09983ac <Address 0x222056def09983ac out of bounds>, m_next_dup = 0x203a4d1444c0d5b3, m_wks_idx = 21538, m_len_name = 9048, 
    m_len_value = 7890260, m_n_v_raw_printable = 1 '\001', m_n_v_raw_printable_pad = 0 '\000', m_readiness = 2 '\002', m_flags = 1 '\001'}, {
    m_ptr_name = 0x656f7722202c2273 <Address 0x656f7722202c2273 out of bounds>, m_ptr_value = 0x3039373231203a22 <Address 0x3039373231203a22 out of bounds>, 
    m_next_dup = 0x697a22202c393739, m_wks_idx = 8816, m_len_name = 8250, m_len_value = 3553058, m_n_v_raw_printable = 0 '\000', m_n_v_raw_printable_pad = 1 '\001', 
    m_readiness = 3 '\003', m_flags = 0 '\000'}, {m_ptr_name = 0x7d5d4b2bf0819670 <Address 0x7d5d4b2bf0819670 out of bounds>, 
    m_ptr_value = 0x796b8e1844d2836c <Address 0x796b8e1844d2836c out of bounds>, m_next_dup = 0x756c8c24f2da9b62, m_wks_idx = 8805, m_len_name = 31546, 
    m_len_value = 7152160, m_n_v_raw_printable = 1 '\001', m_n_v_raw_printable_pad = 0 '\000', m_readiness = 2 '\002', m_flags = 1 '\001'}, {
    m_ptr_name = 0x22207b203a227370 <Address 0x22207b203a227370 out of bounds>, m_ptr_value = 0x22203a2272646461 <Address 0x22203a2272646461 out of bounds>, 
    m_next_dup = 0x626d452034323332, m_wks_idx = 29285, m_len_name = 22304, m_len_value = 6582127, m_n_v_raw_printable = 1 '\001', m_n_v_raw_printable_pad = 1 '\001', 
    m_readiness = 3 '\003', m_flags = 1 '\001'}, {m_ptr_name = 0x6c654b202c724420 <Address 0x6c654b202c724420 out of bounds>, 

{code}

This only happened in one machine in four nodes. The frequency is about several days one time. After disable plugins and restart  ATS the problem still exist. And there are also some warning in traffic.out. One interesting thing is that the warning all happened "after 776 bytes".
{code}
WARNING: Unmarshal failed due to unknow obj type 173 after 776 bytes---- Dumping header heap @ 0x2b37b251e140 - len 2250 ------
0x2b37b251e140: 0xdcbafeed 0x0 0xb251e4b8 0x2b37
0x2b37b251e150: 0xb251e1c8 0x2b37 0x378 0x0
0x2b37b251e160: 0x0 0x0 0x0 0x0
0x2b37b251e170: 0x0 0x0 0x956b2af0 0x2b39
0x2b37b251e180: 0xb251e4b8 0x2b37 0x552 0x312e312f
0x2b37b251e190: 0x31323420 0x63655220 0x0 0x0
0x2b37b251e1a0: 0xd646e75 0x6361430a 0x432d6568 0x72746e6f
0x2b37b251e1b0: 0x0 0x0 0xd657461 0x6e6f430a
0x2b37b251e1c0: 0x89 0x7079542d 0x3003 0x1
0x2b37b251e1d0: 0x10001 0x480004 0xb251e448 0x2b37
0x2b37b251e1e0: 0xb251e4b8 0x2b37 0x6e0003 0x0
0x2b37b251e1f0: 0xb251e1f8 0x2b37 0x25004 0x2b35
0x2b37b251e200: 0x1000401 0x0 0x6ffffff1 0xfffffffe
0x2b37b251e210: 0xffffffff 0xfeffffff 0x0 0x0
0x2b37b251e220: 0x0 0x0 0x0 0x2b00
0x2b37b251e230: 0xb251e238 0x2b37 0x21005 0xb
0x2b37b251e240: 0x0 0x0 0xb251e4bb 0x2b37
0x2b37b251e250: 0xb251e4bf 0x2b37 0x0 0x0
0x2b37b251e260: 0x4001e 0x68000024 0xb251e4e3 0x2b37
0x2b37b251e270: 0xb251e4e9 0x2b37 0x0 0x0
0x2b37b251e280: 0x60004 0x68000003 0xb251e4ec 0x2b37
0x2b37b251e290: 0xb251e4fa 0x2b37 0x0 0x0
0x2b37b251e2a0: 0xeffff 0x6800012e 0xb251e628 0x2b37
0x2b37b251e2b0: 0xb251e63c 0x2b37 0xb251e388 0x2b37
0x2b37b251e2c0: 0x14ffff 0x68000018 0xb251e654 0x2b37
0x2b37b251e2d0: 0xb251e663 0x2b37 0x0 0x0
0x2b37b251e2e0: 0xfffff 0x68000001 0xb251e664 0x2b37
0x2b37b251e2f0: 0xb251e674 0x2b37 0x0 0x0
0x2b37b251e300: 0x10ffff 0x68000105 0xb251e779 0x2b37
0x2b37b251e310: 0xb251e782 0x2b37 0x0 0x0
0x2b37b251e320: 0x9000b 0x6800000c 0xb251e78e 0x2b37
0x2b37b251e330: 0xb251e79d 0x2b37 0x0 0x0
0x2b37b251e340: 0xf0078 0x6800001c 0x4dd2e5e1 0x2b35
0x2b37b251e350: 0x4dd2e5e4 0x2b35 0x0 0x0
0x2b37b251e360: 0x30042 0x78000086 0x6b5e2416 0x2b3a
0x2b37b251e370: 0x6b5e2422 0x2b3a 0x0 0x0
0x2b37b251e380: 0xa000c 0x79000005 0xb251e842 0x2b37
0x2b37b251e390: 0xb251e856 0x2b37 0x0 0x0
0x2b37b251e3a0: 0x14ffff 0x20000018 0xec9d3913 0x2b37
0x2b37b251e3b0: 0xec9d391a 0x2b37 0x0 0x0
0x2b37b251e3c0: 0x70043 0x60000015 0x0 0x0
0x2b37b251e3d0: 0x0 0x0 0x0 0x0
0x2b37b251e3e0: 0x0 0x0 0x0 0x0
0x2b37b251e3f0: 0x0 0x0 0x0 0x0
0x2b37b251e400: 0x42a53c96 0x4a4246a8 0x515e4ead 0x5aea5690
0x2b37b251e410: 0x605d5c77 0x6a726583 0x70606ccd 0x794174c1
0x2b37b251e420: 0x737f01 0x9510525 0x10c70ff4 0x1a6e174f
0x2b37b251e430: 0x23821c2f 0x28fd244b 0x30412ca1 0x3be137e2
0x2b37b251e440: 0x42a53c96 0x4a4246a8 0x515e4ead 0x5aea5690
0x2b37b251e450: 0x605d5c77 0x6a726583 0x70606ccd 0x794174c1
0x2b37b251e460: 0x737f01 0x9510525 0x1c2f1a6e 0x244b2382
0x2b37b251e470: 0x2ca12aa3 0x37e23041 0x3c963be1 0x46a842a5
0x2b37b251e480: 0x4ead4a42 0x5690515e 0x5c775aea 0x6583605d
0x2b37b251e490: 0x6ccd6a72 0x74c17060 0x7f017941 0x5250044
0x2b37b251e4a0: 0xff40951 0x244b10c7 0x2ca12870 0x37e23135
0x2b37b251e4b0: 0x3fbc3b89 0x45424236 0x4ead48a0 0x5690535e
0x2b37b251e4c0: 0x5c775aea 0x6583624d 0x6ccd6a7b 0x74c17060
0x2b37b251e4d0: 0x7ef97941 0x5250045 0xebe0951 0x164a12ea
0x2b37b251e4e0: 0x1c2f1a6e 0x244b2240 0x2ca12870 0x37e23135
0x2b37b251e4f0: 0x3fb83b89 0x45424331 0x4ead48a0 0x5690535e
0x2b37b251e500: 0x5c775b70 0x65f4624d 0x6ccd6a7b 0x74c17060
0x2b37b251e510: 0x7ef97941 0x5250045 0xebe0951 0x164a12ea
0x2b37b251e520: 0x1c2f1a6e 0x244b2240 0x2ca12870 0x36ab3135
0x2b37b251e530: 0x3fb83b89 0x45424331 0x4ead48a0 0x5690535e
0x2b37b251e540: 0x5c775b70 0x65f4624d 0x6ccd6a7b 0x755d7060
0x2b37b251e550: 0x7ef97941 0x5250045 0xebe0951 0x164a12ea
0x2b37b251e560: 0x1c2f1a6e 0x244b2240 0x2ca12870 0x36ab3135
0x2b37b251e570: 0x3fb83b89 0x45424331 0x4ead48a0 0x5690535e
0x2b37b251e580: 0x5c775829 0x65f4624d 0x6ccd6a7b 0x755d7060
0x2b37b251e590: 0x7ef97941 0x5250045 0xebe095e 0x164a12ea
0x2b37b251e5a0: 0x1c2f1a6e 0xe402240 0x45914591 0x460d4591
0x2b37b251e5b0: 0x5a7b0cd3 0x460d460d 0x5455460d 0x460d0cd3
0x2b37b251e5c0: 0xcd35455 0x5455460d 0x54550cd3 0x543e0cd3
0x2b37b251e5d0: 0x543e0e7e 0x543e460d 0x543e0c4d 0x543c0c4d
0x2b37b251e5e0: 0x543c0c4d 0x460d0fc4 0xfc433b0 0xfc45685
0x2b37b251e5f0: 0x460d5685 0x5a7b5685 0x61ae5e94 0x4004434f
0x2b37b251e600: 0x56854bde 0x56850fc4 0x5e945a7b 0x68d72780
0x2b37b251e610: 0x33b06f7c 0x5a7b5685 0x5e942780 0x278061ae
0x2b37b251e620: 0x40044004 0x56850c67 0x40041b4d 0x5a7b5685
0x2b37b251e630: 0x22885e94 0x5a7b5685 0x4bde1ea0 0x56850c50
0x2b37b251e640: 0x5e945a7b 0x56852288 0x1ea05a7b 0xcbd4bde
0x2b37b251e650: 0x5a7b5685 0x22885e94 0x5a7b5685 0x61ae5fd1
0x2b37b251e660: 0x134f4ebf 0x5a7b57ec 0x5fd1329b 0x1b4d1782
...
{code}






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)