[jira] [Comment Edited] (TS-3948) crashed at json_out_stat() with sprintf format mismatch

["Leif Hedstrom (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TS-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15085784#comment-15085784 ] 

Leif Hedstrom edited comment on TS-3948 at 1/6/16 4:37 PM:
-----------------------------------------------------------

I'm fairly certain it's correct, it's just not complete. The issue is that in the past, you could not add new metrics to a running system, so the lock was just not necessary (because the hash table couldn't change past startup). But long ago, I added APIs to add / delete metrics at runtime, via plugin APIs, but that didn't take into account these locking issues.

The reason why this likely trips with the iterator API used by stats_over_http is that it's "slow", whereas in the other places of the code, the odds of tripping over the changes to the hash is so small, it never happens.

Even so, that chance of it tripping is small, we very rarely saw the crasher.


was (Author: zwoop):
I'm fairly certain it's correct, it's just not complete. The issue is that in the past, you could not add new metrics to a running system, so the lock was just not necessary (because the hash table couldn't change past startup). But long ago, I added APIs to add / delete metrics at runtime, via plugin APIs, but that didn't take into account these locking issues.

> crashed at json_out_stat() with sprintf format mismatch
> -------------------------------------------------------
>
>                 Key: TS-3948
>                 URL: https://issues.apache.org/jira/browse/TS-3948
>             Project: Traffic Server
>          Issue Type: Bug
>            Reporter: Masa Sekimura
>            Assignee: Masa Sekimura
>              Labels: A, crash
>             Fix For: 6.1.0
>
>
> We ran into a core dump with the below stack trace.  Record data_type was RECD_INT(0x1) in RecDumpRecords():RecCore.cc:963 but somehow it became TS_RECORDDATATYPE_STRING(0x3) which caused sprintf format error in json_out_stat():stats_over_http.c:176
> {code}
> (gdb) where
> #0  0x00002aaaae7d662e in _IO_default_xsputn_internal () from /lib64/libc.so.6
> #1  0x00002aaaae7a732a in vfprintf () from /lib64/libc.so.6
> #2  0x00002aaaae861aa0 in __vsnprintf_chk () from /lib64/libc.so.6
> #3  0x00002aaaae8619da in __snprintf_chk () from /lib64/libc.so.6
> #4  0x00002aaab77f2617 in snprintf (rec_type=<value optimized out>, edata=0x2ae56bb16c80, registered=<value optimized out>, name=<value optimized out>, data_type=<value optimized out>,
>     datum=<value optimized out>) at /usr/include/bits/stdio2.h:65
> #5  json_out_stat (rec_type=<value optimized out>, edata=0x2ae56bb16c80, registered=<value optimized out>, name=<value optimized out>, data_type=<value optimized out>, datum=<value optimized out>)
>     at stats_over_http.c:176
> #6  0x00002aaaaae46ac5 in RecDumpRecords (rec_type=38, callback=0x2aaab77f2540 <json_out_stat>, edata=0x2ae56bb16c80) at RecCore.cc:963
> #7  0x00002aaab77f2891 in json_out_stats (contp=<value optimized out>, event=TS_EVENT_VCONN_WRITE_READY, edata=0x2abca167bf70) at stats_over_http.c:189
> #8  stats_process_write (contp=<value optimized out>, event=TS_EVENT_VCONN_WRITE_READY, edata=0x2abca167bf70) at stats_over_http.c:204
> #9  stats_dostuff (contp=<value optimized out>, event=TS_EVENT_VCONN_WRITE_READY, edata=0x2abca167bf70) at stats_over_http.c:227
> #10 0x00002aaaaabc7b1a in PluginVC::process_write_side (this=0x2abca167be20, other_side_call=false) at PluginVC.cc:547
> #11 0x00002aaaaabc9be5 in PluginVC::main_handler (this=0x2abca167be20, event=<value optimized out>, data=0x2aaab2c26840) at PluginVC.cc:208
> #12 0x00002aaaaae545b8 in handleEvent (this=0x2aaab23a0000, e=0x2aaab2c26840, calling_code=1) at I_Continuation.h:145
> #13 EThread::process_event (this=0x2aaab23a0000, e=0x2aaab2c26840, calling_code=1) at UnixEThread.cc:128
> #14 0x00002aaaaae54edb in EThread::execute (this=0x2aaab23a0000) at UnixEThread.cc:179
> #15 0x00002aaaaae5398a in spawn_thread_internal (a=0x2aaaab8124f0) at Thread.cc:85
> #16 0x00002aaaac9439d1 in start_thread () from /lib64/libpthread.so.0
> #17 0x00002aaaae84a8fd in clone () from /lib64/libc.so.6
> (gdb) f 6
> #6  0x00002aaaaae46ac5 in RecDumpRecords (rec_type=38, callback=0x2aaab77f2540 <json_out_stat>, edata=0x2ae56bb16c80) at RecCore.cc:963
> 963	in RecCore.cc
> (gdb) p rec_type
> $20 = 38
> (gdb) p edata
> $21 = (void *) 0x2ae56bb16c80
> (gdb) p r->registered
> $22 = true
> (gdb) p r->name
> $23 = 0x2aaaab7ba580 "proxy.process.ssl.origin_server_decryption_failed"
> (gdb) p r->data_type
> $24 = RECD_INT
> (gdb) p &r->data
> $25 = (RecData *) 0x2aaaab76f380
> (gdb)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)