You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@trafficserver.apache.org by "kang li (JIRA)" <ji...@apache.org> on 2014/09/16 12:03:33 UTC
[jira] [Updated] (TS-3078) MIMEHdrImpl::unmarshal crash caused by
cache corruption
[ https://issues.apache.org/jira/browse/TS-3078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
kang li updated TS-3078:
------------------------
Affects Version/s: 4.0.2
> MIMEHdrImpl::unmarshal crash caused by cache corruption
> -------------------------------------------------------
>
> Key: TS-3078
> URL: https://issues.apache.org/jira/browse/TS-3078
> Project: Traffic Server
> Issue Type: Bug
> Components: Cache, MIME
> Affects Versions: 4.0.2
> Reporter: kang li
>
> {code}
> (gdb) bt
> #0 0x00000000005c22c6 in unmarshal (this=0x2aaed05f61f8, offset=46930308587840) at MIME.cc:3534
> #1 MIMEHdrImpl::unmarshal (this=0x2aaed05f61f8, offset=46930308587840) at MIME.cc:3590
> #2 0x00000000005b4bcb in HdrHeap::unmarshal (this=0x2aaed05f6140, buf_length=<value optimized out>, obj_type=<value optimized out>, found_obj=<value optimized out>,
> block_ref=<value optimized out>) at HdrHeap.cc:926
> #3 0x00000000005b8be1 in HTTPInfo::unmarshal (buf=0x2aaed05f6048 "\355\336\315\253", len=3280, block_ref=0x2aae080feec0) at HTTP.cc:1948
> #4 0x0000000000635e43 in unmarshal_helper (this=0x2aae10081b90, event=<value optimized out>, e=<value optimized out>) at Cache.cc:2066
> #5 CacheVC::handleReadDone (this=0x2aae10081b90, event=<value optimized out>, e=<value optimized out>) at Cache.cc:2195
> #6 0x00000000005f1845 in handleEvent (this=<value optimized out>, event=<value optimized out>, data=<value optimized out>)
> at ../../iocore/eventsystem/I_Continuation.h:146
> #7 AIOCallbackInternal::io_complete (this=<value optimized out>, event=<value optimized out>, data=<value optimized out>) at ../../iocore/aio/P_AIO.h:123
> #8 0x00000000006a990f in handleEvent (this=0x2aadf0809010, e=0x2aae3812b710, calling_code=1) at I_Continuation.h:146
> #9 EThread::process_event (this=0x2aadf0809010, e=0x2aae3812b710, calling_code=1) at UnixEThread.cc:141
> #10 0x00000000006aa48b in EThread::execute (this=0x2aadf0809010) at UnixEThread.cc:192
> #11 0x00000000006a87aa in spawn_thread_internal (a=0x13e0a80) at Thread.cc:88
> #12 0x00002aadeaf3c9d1 in start_thread () from /lib64/libpthread.so.0
> #13 0x00000034cfce8b6d in clone () from /lib64/libc.so.6
> (gdb) p m_freetop
> $1 = 892220471
> (gdb) p m_field_slots
> $2 = {{m_ptr_name = 0x33203a2274616c22 <Address 0x33203a2274616c22 out of bounds>, m_ptr_value = 0x3939333338312e33 <Address 0x3939333338312e33 out of bounds>,
> m_next_dup = 0x3a226e6f6c22202c, m_wks_idx = 11552, m_len_name = 14137, m_len_value = 3289390, m_n_v_raw_printable = 0 '\000', m_n_v_raw_printable_pad = 3 '\003',
> m_readiness = 3 '\003', m_flags = 0 '\000'}, {m_ptr_name = 0x766c4ccefc939174 <Address 0x766c4ccefc939174 out of bounds>,
> m_ptr_value = 0x222056def09983ac <Address 0x222056def09983ac out of bounds>, m_next_dup = 0x203a4d1444c0d5b3, m_wks_idx = 21538, m_len_name = 9048,
> m_len_value = 7890260, m_n_v_raw_printable = 1 '\001', m_n_v_raw_printable_pad = 0 '\000', m_readiness = 2 '\002', m_flags = 1 '\001'}, {
> m_ptr_name = 0x656f7722202c2273 <Address 0x656f7722202c2273 out of bounds>, m_ptr_value = 0x3039373231203a22 <Address 0x3039373231203a22 out of bounds>,
> m_next_dup = 0x697a22202c393739, m_wks_idx = 8816, m_len_name = 8250, m_len_value = 3553058, m_n_v_raw_printable = 0 '\000', m_n_v_raw_printable_pad = 1 '\001',
> m_readiness = 3 '\003', m_flags = 0 '\000'}, {m_ptr_name = 0x7d5d4b2bf0819670 <Address 0x7d5d4b2bf0819670 out of bounds>,
> m_ptr_value = 0x796b8e1844d2836c <Address 0x796b8e1844d2836c out of bounds>, m_next_dup = 0x756c8c24f2da9b62, m_wks_idx = 8805, m_len_name = 31546,
> m_len_value = 7152160, m_n_v_raw_printable = 1 '\001', m_n_v_raw_printable_pad = 0 '\000', m_readiness = 2 '\002', m_flags = 1 '\001'}, {
> m_ptr_name = 0x22207b203a227370 <Address 0x22207b203a227370 out of bounds>, m_ptr_value = 0x22203a2272646461 <Address 0x22203a2272646461 out of bounds>,
> m_next_dup = 0x626d452034323332, m_wks_idx = 29285, m_len_name = 22304, m_len_value = 6582127, m_n_v_raw_printable = 1 '\001', m_n_v_raw_printable_pad = 1 '\001',
> m_readiness = 3 '\003', m_flags = 1 '\001'}, {m_ptr_name = 0x6c654b202c724420 <Address 0x6c654b202c724420 out of bounds>,
> {code}
> This only happened in one machine in four nodes. The frequency is about several days one time. After disable plugins and restart ATS the problem still exist. And there are also some warning in traffic.out. One interesting thing is that the warning all happened "after 776 bytes".
> {code}
> WARNING: Unmarshal failed due to unknow obj type 173 after 776 bytes---- Dumping header heap @ 0x2b37b251e140 - len 2250 ------
> 0x2b37b251e140: 0xdcbafeed 0x0 0xb251e4b8 0x2b37
> 0x2b37b251e150: 0xb251e1c8 0x2b37 0x378 0x0
> 0x2b37b251e160: 0x0 0x0 0x0 0x0
> 0x2b37b251e170: 0x0 0x0 0x956b2af0 0x2b39
> 0x2b37b251e180: 0xb251e4b8 0x2b37 0x552 0x312e312f
> 0x2b37b251e190: 0x31323420 0x63655220 0x0 0x0
> 0x2b37b251e1a0: 0xd646e75 0x6361430a 0x432d6568 0x72746e6f
> 0x2b37b251e1b0: 0x0 0x0 0xd657461 0x6e6f430a
> 0x2b37b251e1c0: 0x89 0x7079542d 0x3003 0x1
> 0x2b37b251e1d0: 0x10001 0x480004 0xb251e448 0x2b37
> 0x2b37b251e1e0: 0xb251e4b8 0x2b37 0x6e0003 0x0
> 0x2b37b251e1f0: 0xb251e1f8 0x2b37 0x25004 0x2b35
> 0x2b37b251e200: 0x1000401 0x0 0x6ffffff1 0xfffffffe
> 0x2b37b251e210: 0xffffffff 0xfeffffff 0x0 0x0
> 0x2b37b251e220: 0x0 0x0 0x0 0x2b00
> 0x2b37b251e230: 0xb251e238 0x2b37 0x21005 0xb
> 0x2b37b251e240: 0x0 0x0 0xb251e4bb 0x2b37
> 0x2b37b251e250: 0xb251e4bf 0x2b37 0x0 0x0
> 0x2b37b251e260: 0x4001e 0x68000024 0xb251e4e3 0x2b37
> 0x2b37b251e270: 0xb251e4e9 0x2b37 0x0 0x0
> 0x2b37b251e280: 0x60004 0x68000003 0xb251e4ec 0x2b37
> 0x2b37b251e290: 0xb251e4fa 0x2b37 0x0 0x0
> 0x2b37b251e2a0: 0xeffff 0x6800012e 0xb251e628 0x2b37
> 0x2b37b251e2b0: 0xb251e63c 0x2b37 0xb251e388 0x2b37
> 0x2b37b251e2c0: 0x14ffff 0x68000018 0xb251e654 0x2b37
> 0x2b37b251e2d0: 0xb251e663 0x2b37 0x0 0x0
> 0x2b37b251e2e0: 0xfffff 0x68000001 0xb251e664 0x2b37
> 0x2b37b251e2f0: 0xb251e674 0x2b37 0x0 0x0
> 0x2b37b251e300: 0x10ffff 0x68000105 0xb251e779 0x2b37
> 0x2b37b251e310: 0xb251e782 0x2b37 0x0 0x0
> 0x2b37b251e320: 0x9000b 0x6800000c 0xb251e78e 0x2b37
> 0x2b37b251e330: 0xb251e79d 0x2b37 0x0 0x0
> 0x2b37b251e340: 0xf0078 0x6800001c 0x4dd2e5e1 0x2b35
> 0x2b37b251e350: 0x4dd2e5e4 0x2b35 0x0 0x0
> 0x2b37b251e360: 0x30042 0x78000086 0x6b5e2416 0x2b3a
> 0x2b37b251e370: 0x6b5e2422 0x2b3a 0x0 0x0
> 0x2b37b251e380: 0xa000c 0x79000005 0xb251e842 0x2b37
> 0x2b37b251e390: 0xb251e856 0x2b37 0x0 0x0
> 0x2b37b251e3a0: 0x14ffff 0x20000018 0xec9d3913 0x2b37
> 0x2b37b251e3b0: 0xec9d391a 0x2b37 0x0 0x0
> 0x2b37b251e3c0: 0x70043 0x60000015 0x0 0x0
> 0x2b37b251e3d0: 0x0 0x0 0x0 0x0
> 0x2b37b251e3e0: 0x0 0x0 0x0 0x0
> 0x2b37b251e3f0: 0x0 0x0 0x0 0x0
> 0x2b37b251e400: 0x42a53c96 0x4a4246a8 0x515e4ead 0x5aea5690
> 0x2b37b251e410: 0x605d5c77 0x6a726583 0x70606ccd 0x794174c1
> 0x2b37b251e420: 0x737f01 0x9510525 0x10c70ff4 0x1a6e174f
> 0x2b37b251e430: 0x23821c2f 0x28fd244b 0x30412ca1 0x3be137e2
> 0x2b37b251e440: 0x42a53c96 0x4a4246a8 0x515e4ead 0x5aea5690
> 0x2b37b251e450: 0x605d5c77 0x6a726583 0x70606ccd 0x794174c1
> 0x2b37b251e460: 0x737f01 0x9510525 0x1c2f1a6e 0x244b2382
> 0x2b37b251e470: 0x2ca12aa3 0x37e23041 0x3c963be1 0x46a842a5
> 0x2b37b251e480: 0x4ead4a42 0x5690515e 0x5c775aea 0x6583605d
> 0x2b37b251e490: 0x6ccd6a72 0x74c17060 0x7f017941 0x5250044
> 0x2b37b251e4a0: 0xff40951 0x244b10c7 0x2ca12870 0x37e23135
> 0x2b37b251e4b0: 0x3fbc3b89 0x45424236 0x4ead48a0 0x5690535e
> 0x2b37b251e4c0: 0x5c775aea 0x6583624d 0x6ccd6a7b 0x74c17060
> 0x2b37b251e4d0: 0x7ef97941 0x5250045 0xebe0951 0x164a12ea
> 0x2b37b251e4e0: 0x1c2f1a6e 0x244b2240 0x2ca12870 0x37e23135
> 0x2b37b251e4f0: 0x3fb83b89 0x45424331 0x4ead48a0 0x5690535e
> 0x2b37b251e500: 0x5c775b70 0x65f4624d 0x6ccd6a7b 0x74c17060
> 0x2b37b251e510: 0x7ef97941 0x5250045 0xebe0951 0x164a12ea
> 0x2b37b251e520: 0x1c2f1a6e 0x244b2240 0x2ca12870 0x36ab3135
> 0x2b37b251e530: 0x3fb83b89 0x45424331 0x4ead48a0 0x5690535e
> 0x2b37b251e540: 0x5c775b70 0x65f4624d 0x6ccd6a7b 0x755d7060
> 0x2b37b251e550: 0x7ef97941 0x5250045 0xebe0951 0x164a12ea
> 0x2b37b251e560: 0x1c2f1a6e 0x244b2240 0x2ca12870 0x36ab3135
> 0x2b37b251e570: 0x3fb83b89 0x45424331 0x4ead48a0 0x5690535e
> 0x2b37b251e580: 0x5c775829 0x65f4624d 0x6ccd6a7b 0x755d7060
> 0x2b37b251e590: 0x7ef97941 0x5250045 0xebe095e 0x164a12ea
> 0x2b37b251e5a0: 0x1c2f1a6e 0xe402240 0x45914591 0x460d4591
> 0x2b37b251e5b0: 0x5a7b0cd3 0x460d460d 0x5455460d 0x460d0cd3
> 0x2b37b251e5c0: 0xcd35455 0x5455460d 0x54550cd3 0x543e0cd3
> 0x2b37b251e5d0: 0x543e0e7e 0x543e460d 0x543e0c4d 0x543c0c4d
> 0x2b37b251e5e0: 0x543c0c4d 0x460d0fc4 0xfc433b0 0xfc45685
> 0x2b37b251e5f0: 0x460d5685 0x5a7b5685 0x61ae5e94 0x4004434f
> 0x2b37b251e600: 0x56854bde 0x56850fc4 0x5e945a7b 0x68d72780
> 0x2b37b251e610: 0x33b06f7c 0x5a7b5685 0x5e942780 0x278061ae
> 0x2b37b251e620: 0x40044004 0x56850c67 0x40041b4d 0x5a7b5685
> 0x2b37b251e630: 0x22885e94 0x5a7b5685 0x4bde1ea0 0x56850c50
> 0x2b37b251e640: 0x5e945a7b 0x56852288 0x1ea05a7b 0xcbd4bde
> 0x2b37b251e650: 0x5a7b5685 0x22885e94 0x5a7b5685 0x61ae5fd1
> 0x2b37b251e660: 0x134f4ebf 0x5a7b57ec 0x5fd1329b 0x1b4d1782
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)