You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@trafficserver.apache.org by "Manjesh Nilange (JIRA)" <ji...@apache.org> on 2012/04/24 00:58:32 UTC
[jira] [Created] (TS-1219) Crash report: ink_freelist_new causing
core dumps
Manjesh Nilange created TS-1219:
-----------------------------------
Summary: Crash report: ink_freelist_new causing core dumps
Key: TS-1219
URL: https://issues.apache.org/jira/browse/TS-1219
Project: Traffic Server
Issue Type: Bug
Affects Versions: 3.0.2
Reporter: Manjesh Nilange
Our TS based production boxes are seeing a couple of crashes per day with stack traces ending in
#3 0x00000000004c85ea in signal_handler (sig=11) at signals.cc:225
#4 <signal handler called>
#5 0x0000003312414160 in ink_freelist_new (f=0x3312629ce0) at ink_queue.cc:326
#6 0x000000331240d47a in alloc_void (this=0x7fff80173cd8, size=511, alignment=1) at Allocator.h:208
#7 blk_alloc (this=0x7fff80173cd8, size=511, alignment=1) at Arena.cc:45
#8 Arena::alloc (this=0x7fff80173cd8, size=511, alignment=1) at Arena.cc:118
#9 0x0000000000569b93 in str_alloc (arena=<value optimized out>,
url=0x31be6a8 "*************"...,
len_in=<value optimized out>, len_out=0x7fff80173d20) at ../../lib/ts/Arena.h:123
#10 LogUtils::escapify_url (arena=<value optimized out>,
url=0x31be6a8 "*************"...,
len_in=<value optimized out>, len_out=0x7fff80173d20) at LogUtils.cc:392
#11 0x00000000005482e9 in LogAccessHttp::init (this=0x7fff80173cc0) at LogAccessHttp.cc:98
#12 0x000000000054b436 in Log::access (lad=0x7fff80173cc0) at Log.cc:1055
#13 0x00000000004f3085 in HttpSM::update_stats (this=0x2ae5480651e0) at HttpSM.cc:6044
#14 0x00000000004f8918 in HttpSM::kill_this (this=0x2ae5480651e0) at HttpSM.cc:6005
#15 0x00000000004f8c08 in HttpSM::main_handler (this=0x2ae5480651e0, event=2301, data=0x2ae548066ec8)
at HttpSM.cc:2452
The URL was valid, I just "anonymized" it. This is our environment
$ uname -a
Linux xxx.prod 2.6.32-131.4.1.el6.x86_64 #1 SMP Fri Jun 10 10:54:26 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
$ file /usr/bin/traffic_server
/usr/bin/traffic_server: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, stripped
There doesn't seem to be more useful information in the frame:
(gdb) f 5
#5 0x0000003312414160 in ink_freelist_new (f=0x3312629ce0) at ink_queue.cc:326
326 FREELIST_VERSION(item) + 1);
(gdb) list
321 fastalloc_mem_in_use += f->chunk_size * f->type_size;
322 #endif
323
324 } else {
325 SET_FREELIST_POINTER_VERSION(next, *ADDRESS_OF_NEXT(TO_PTR(FREELIST_POINTER(item)), f->offset),
326 FREELIST_VERSION(item) + 1);
327 #if !defined(INK_USE_MUTEX_FOR_FREELISTS)
328 result = ink_atomic_cas64((int64_t *) & f->head.data, item.data, next.data);
329 #else
330 f->head.data = next.data;
(gdb) p next
$2 = <value optimized out>
(gdb) p f
$3 = (InkFreeList *) 0x3312629ce0
(gdb) p *f
$4 = {head = {data = -8941773651046140890}, name = 0x331241e92f "ArenaBlock", type_size = 1024, chunk_size = 128,
count = 519, allocated = 1536, offset = 0, alignment = 8, allocated_base = 0, count_base = 0}
(gdb) p item
$5 = {data = -8953314125091277786}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (TS-1219) Crash report: ink_freelist_new causing
core dumps
Posted by "Manjesh Nilange (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Manjesh Nilange updated TS-1219:
--------------------------------
Affects Version/s: 3.0.4
Issue is seen on 3.0.4 as well.
> Crash report: ink_freelist_new causing core dumps
> -------------------------------------------------
>
> Key: TS-1219
> URL: https://issues.apache.org/jira/browse/TS-1219
> Project: Traffic Server
> Issue Type: Bug
> Affects Versions: 3.0.4, 3.0.2
> Reporter: Manjesh Nilange
>
> Our TS based production boxes are seeing a couple of crashes per day with stack traces ending in
> #3 0x00000000004c85ea in signal_handler (sig=11) at signals.cc:225
> #4 <signal handler called>
> #5 0x0000003312414160 in ink_freelist_new (f=0x3312629ce0) at ink_queue.cc:326
> #6 0x000000331240d47a in alloc_void (this=0x7fff80173cd8, size=511, alignment=1) at Allocator.h:208
> #7 blk_alloc (this=0x7fff80173cd8, size=511, alignment=1) at Arena.cc:45
> #8 Arena::alloc (this=0x7fff80173cd8, size=511, alignment=1) at Arena.cc:118
> #9 0x0000000000569b93 in str_alloc (arena=<value optimized out>,
> url=0x31be6a8 "*************"...,
> len_in=<value optimized out>, len_out=0x7fff80173d20) at ../../lib/ts/Arena.h:123
> #10 LogUtils::escapify_url (arena=<value optimized out>,
> url=0x31be6a8 "*************"...,
> len_in=<value optimized out>, len_out=0x7fff80173d20) at LogUtils.cc:392
> #11 0x00000000005482e9 in LogAccessHttp::init (this=0x7fff80173cc0) at LogAccessHttp.cc:98
> #12 0x000000000054b436 in Log::access (lad=0x7fff80173cc0) at Log.cc:1055
> #13 0x00000000004f3085 in HttpSM::update_stats (this=0x2ae5480651e0) at HttpSM.cc:6044
> #14 0x00000000004f8918 in HttpSM::kill_this (this=0x2ae5480651e0) at HttpSM.cc:6005
> #15 0x00000000004f8c08 in HttpSM::main_handler (this=0x2ae5480651e0, event=2301, data=0x2ae548066ec8)
> at HttpSM.cc:2452
> The URL was valid, I just "anonymized" it. This is our environment
> $ uname -a
> Linux xxx.prod 2.6.32-131.4.1.el6.x86_64 #1 SMP Fri Jun 10 10:54:26 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
> $ file /usr/bin/traffic_server
> /usr/bin/traffic_server: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, stripped
> There doesn't seem to be more useful information in the frame:
> (gdb) f 5
> #5 0x0000003312414160 in ink_freelist_new (f=0x3312629ce0) at ink_queue.cc:326
> 326 FREELIST_VERSION(item) + 1);
> (gdb) list
> 321 fastalloc_mem_in_use += f->chunk_size * f->type_size;
> 322 #endif
> 323
> 324 } else {
> 325 SET_FREELIST_POINTER_VERSION(next, *ADDRESS_OF_NEXT(TO_PTR(FREELIST_POINTER(item)), f->offset),
> 326 FREELIST_VERSION(item) + 1);
> 327 #if !defined(INK_USE_MUTEX_FOR_FREELISTS)
> 328 result = ink_atomic_cas64((int64_t *) & f->head.data, item.data, next.data);
> 329 #else
> 330 f->head.data = next.data;
> (gdb) p next
> $2 = <value optimized out>
> (gdb) p f
> $3 = (InkFreeList *) 0x3312629ce0
> (gdb) p *f
> $4 = {head = {data = -8941773651046140890}, name = 0x331241e92f "ArenaBlock", type_size = 1024, chunk_size = 128,
> count = 519, allocated = 1536, offset = 0, alignment = 8, allocated_base = 0, count_base = 0}
> (gdb) p item
> $5 = {data = -8953314125091277786}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (TS-1219) Crash report:
ink_freelist_new causing core dumps
Posted by "Brian Geffon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262443#comment-13262443 ]
Brian Geffon edited comment on TS-1219 at 4/26/12 8:03 AM:
-----------------------------------------------------------
I have a possible theory for the crash, but I think jplevyak might be the only one who could say for sure. In 3.0.x inkfreelist_new follows the standard pattern which does a load followed by a CAS:
{noformat}
int result = 0;
do {
INK_QUEUE_LD64(item, f->head);
if (TO_PTR(FREELIST_POINTER(item)) == NULL) {
...
} else {\n
// crash happens on the following line
SET_FREELIST_POINTER_VERSION(next, *ADDRESS_OF_NEXT(TO_PTR(FREELIST_POINTER(item)), f->offset),
FREELIST_VERSION(item) + 1);
result = ink_atomic_cas64((int64_t *) & f->head.data, item.data, next.data);
}
}
while(result == 0)
{noformat}
I believe this might be a race in the event that the head is freed between the time of the load to item and the SET_FREELIST_POINTER_VERSION, then we would potentially be dereferencing item, which is no longer valid if it's freed as part of a dequeue operation. Does anyone who understands the freelists think this might be the situation?
Also, to support this idea, I see the following in the crash:
(gdb) p f->head
$17 = {data = -6044627300393456023}
(gdb) p item
$18 = {data = -6053071549694775703}
(gdb)
Which means that item != f->head meaning that head has changed since we first read it, and if it was changed as a result of a dequeue then the memory might have been freed explaining the crash.
was (Author: briang):
I have a possible theory for the crash, but I think jplevyak might be the only one who could say for sure. In 3.0.x inkfreelist_new follows the standard pattern which does a load followed by a CAS:
{noformat}
int result = 0;
do {
INK_QUEUE_LD64(item, f->head);
if (TO_PTR(FREELIST_POINTER(item)) == NULL) {
...
} else {\n
// crash happens on the following line
SET_FREELIST_POINTER_VERSION(next, *ADDRESS_OF_NEXT(TO_PTR(FREELIST_POINTER(item)), f->offset),
FREELIST_VERSION(item) + 1);
result = ink_atomic_cas64((int64_t *) & f->head.data, item.data, next.data);
}
}
while(result == 0)
{noformat}
I believe this might be a race in the event that the head is freed between the time of the load to item and the SET_FREELIST_POINTER_VERSION, then we would potentially be dereferencing item, which is no longer valid if it's freed as part of a dequeue operation. Does anyone who understands the freelists think this might be the situation?
> Crash report: ink_freelist_new causing core dumps
> -------------------------------------------------
>
> Key: TS-1219
> URL: https://issues.apache.org/jira/browse/TS-1219
> Project: Traffic Server
> Issue Type: Bug
> Affects Versions: 3.0.2
> Reporter: Manjesh Nilange
>
> Our TS based production boxes are seeing a couple of crashes per day with stack traces ending in
> #3 0x00000000004c85ea in signal_handler (sig=11) at signals.cc:225
> #4 <signal handler called>
> #5 0x0000003312414160 in ink_freelist_new (f=0x3312629ce0) at ink_queue.cc:326
> #6 0x000000331240d47a in alloc_void (this=0x7fff80173cd8, size=511, alignment=1) at Allocator.h:208
> #7 blk_alloc (this=0x7fff80173cd8, size=511, alignment=1) at Arena.cc:45
> #8 Arena::alloc (this=0x7fff80173cd8, size=511, alignment=1) at Arena.cc:118
> #9 0x0000000000569b93 in str_alloc (arena=<value optimized out>,
> url=0x31be6a8 "*************"...,
> len_in=<value optimized out>, len_out=0x7fff80173d20) at ../../lib/ts/Arena.h:123
> #10 LogUtils::escapify_url (arena=<value optimized out>,
> url=0x31be6a8 "*************"...,
> len_in=<value optimized out>, len_out=0x7fff80173d20) at LogUtils.cc:392
> #11 0x00000000005482e9 in LogAccessHttp::init (this=0x7fff80173cc0) at LogAccessHttp.cc:98
> #12 0x000000000054b436 in Log::access (lad=0x7fff80173cc0) at Log.cc:1055
> #13 0x00000000004f3085 in HttpSM::update_stats (this=0x2ae5480651e0) at HttpSM.cc:6044
> #14 0x00000000004f8918 in HttpSM::kill_this (this=0x2ae5480651e0) at HttpSM.cc:6005
> #15 0x00000000004f8c08 in HttpSM::main_handler (this=0x2ae5480651e0, event=2301, data=0x2ae548066ec8)
> at HttpSM.cc:2452
> The URL was valid, I just "anonymized" it. This is our environment
> $ uname -a
> Linux xxx.prod 2.6.32-131.4.1.el6.x86_64 #1 SMP Fri Jun 10 10:54:26 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
> $ file /usr/bin/traffic_server
> /usr/bin/traffic_server: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, stripped
> There doesn't seem to be more useful information in the frame:
> (gdb) f 5
> #5 0x0000003312414160 in ink_freelist_new (f=0x3312629ce0) at ink_queue.cc:326
> 326 FREELIST_VERSION(item) + 1);
> (gdb) list
> 321 fastalloc_mem_in_use += f->chunk_size * f->type_size;
> 322 #endif
> 323
> 324 } else {
> 325 SET_FREELIST_POINTER_VERSION(next, *ADDRESS_OF_NEXT(TO_PTR(FREELIST_POINTER(item)), f->offset),
> 326 FREELIST_VERSION(item) + 1);
> 327 #if !defined(INK_USE_MUTEX_FOR_FREELISTS)
> 328 result = ink_atomic_cas64((int64_t *) & f->head.data, item.data, next.data);
> 329 #else
> 330 f->head.data = next.data;
> (gdb) p next
> $2 = <value optimized out>
> (gdb) p f
> $3 = (InkFreeList *) 0x3312629ce0
> (gdb) p *f
> $4 = {head = {data = -8941773651046140890}, name = 0x331241e92f "ArenaBlock", type_size = 1024, chunk_size = 128,
> count = 519, allocated = 1536, offset = 0, alignment = 8, allocated_base = 0, count_base = 0}
> (gdb) p item
> $5 = {data = -8953314125091277786}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-1219) Crash report: ink_freelist_new causing
core dumps
Posted by "Brian Geffon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262443#comment-13262443 ]
Brian Geffon commented on TS-1219:
----------------------------------
I have a possible theory for the crash, but I think jplevyak might be the only one who could say for sure. In 3.0.x inkfreelist_new follows the standard pattern which does a load followed by a CAS:
int result = 0;
do {
INK_QUEUE_LD64(item, f->head);
if (TO_PTR(FREELIST_POINTER(item)) == NULL) {
...
} else {
// crash happens on the following line
SET_FREELIST_POINTER_VERSION(next, *ADDRESS_OF_NEXT(TO_PTR(FREELIST_POINTER(item)), f->offset),
FREELIST_VERSION(item) + 1);
result = ink_atomic_cas64((int64_t *) & f->head.data, item.data, next.data);
}
}
while(result == 0)
I believe this might be a race in the event that the head is freed between the time of the load to item and the SET_FREELIST_POINTER_VERSION, then we would potentially be dereferencing item, which is no longer valid if it's freed as part of a dequeue operation. Does anyone who understands the freelists think this might be the situation?
> Crash report: ink_freelist_new causing core dumps
> -------------------------------------------------
>
> Key: TS-1219
> URL: https://issues.apache.org/jira/browse/TS-1219
> Project: Traffic Server
> Issue Type: Bug
> Affects Versions: 3.0.2
> Reporter: Manjesh Nilange
>
> Our TS based production boxes are seeing a couple of crashes per day with stack traces ending in
> #3 0x00000000004c85ea in signal_handler (sig=11) at signals.cc:225
> #4 <signal handler called>
> #5 0x0000003312414160 in ink_freelist_new (f=0x3312629ce0) at ink_queue.cc:326
> #6 0x000000331240d47a in alloc_void (this=0x7fff80173cd8, size=511, alignment=1) at Allocator.h:208
> #7 blk_alloc (this=0x7fff80173cd8, size=511, alignment=1) at Arena.cc:45
> #8 Arena::alloc (this=0x7fff80173cd8, size=511, alignment=1) at Arena.cc:118
> #9 0x0000000000569b93 in str_alloc (arena=<value optimized out>,
> url=0x31be6a8 "*************"...,
> len_in=<value optimized out>, len_out=0x7fff80173d20) at ../../lib/ts/Arena.h:123
> #10 LogUtils::escapify_url (arena=<value optimized out>,
> url=0x31be6a8 "*************"...,
> len_in=<value optimized out>, len_out=0x7fff80173d20) at LogUtils.cc:392
> #11 0x00000000005482e9 in LogAccessHttp::init (this=0x7fff80173cc0) at LogAccessHttp.cc:98
> #12 0x000000000054b436 in Log::access (lad=0x7fff80173cc0) at Log.cc:1055
> #13 0x00000000004f3085 in HttpSM::update_stats (this=0x2ae5480651e0) at HttpSM.cc:6044
> #14 0x00000000004f8918 in HttpSM::kill_this (this=0x2ae5480651e0) at HttpSM.cc:6005
> #15 0x00000000004f8c08 in HttpSM::main_handler (this=0x2ae5480651e0, event=2301, data=0x2ae548066ec8)
> at HttpSM.cc:2452
> The URL was valid, I just "anonymized" it. This is our environment
> $ uname -a
> Linux xxx.prod 2.6.32-131.4.1.el6.x86_64 #1 SMP Fri Jun 10 10:54:26 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
> $ file /usr/bin/traffic_server
> /usr/bin/traffic_server: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, stripped
> There doesn't seem to be more useful information in the frame:
> (gdb) f 5
> #5 0x0000003312414160 in ink_freelist_new (f=0x3312629ce0) at ink_queue.cc:326
> 326 FREELIST_VERSION(item) + 1);
> (gdb) list
> 321 fastalloc_mem_in_use += f->chunk_size * f->type_size;
> 322 #endif
> 323
> 324 } else {
> 325 SET_FREELIST_POINTER_VERSION(next, *ADDRESS_OF_NEXT(TO_PTR(FREELIST_POINTER(item)), f->offset),
> 326 FREELIST_VERSION(item) + 1);
> 327 #if !defined(INK_USE_MUTEX_FOR_FREELISTS)
> 328 result = ink_atomic_cas64((int64_t *) & f->head.data, item.data, next.data);
> 329 #else
> 330 f->head.data = next.data;
> (gdb) p next
> $2 = <value optimized out>
> (gdb) p f
> $3 = (InkFreeList *) 0x3312629ce0
> (gdb) p *f
> $4 = {head = {data = -8941773651046140890}, name = 0x331241e92f "ArenaBlock", type_size = 1024, chunk_size = 128,
> count = 519, allocated = 1536, offset = 0, alignment = 8, allocated_base = 0, count_base = 0}
> (gdb) p item
> $5 = {data = -8953314125091277786}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-1219) Crash report: ink_freelist_new causing
core dumps
Posted by "Manjesh Nilange (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261818#comment-13261818 ]
Manjesh Nilange commented on TS-1219:
-------------------------------------
We are planning to move to a newer version. I'll try to fast-track it.
> Crash report: ink_freelist_new causing core dumps
> -------------------------------------------------
>
> Key: TS-1219
> URL: https://issues.apache.org/jira/browse/TS-1219
> Project: Traffic Server
> Issue Type: Bug
> Affects Versions: 3.0.2
> Reporter: Manjesh Nilange
>
> Our TS based production boxes are seeing a couple of crashes per day with stack traces ending in
> #3 0x00000000004c85ea in signal_handler (sig=11) at signals.cc:225
> #4 <signal handler called>
> #5 0x0000003312414160 in ink_freelist_new (f=0x3312629ce0) at ink_queue.cc:326
> #6 0x000000331240d47a in alloc_void (this=0x7fff80173cd8, size=511, alignment=1) at Allocator.h:208
> #7 blk_alloc (this=0x7fff80173cd8, size=511, alignment=1) at Arena.cc:45
> #8 Arena::alloc (this=0x7fff80173cd8, size=511, alignment=1) at Arena.cc:118
> #9 0x0000000000569b93 in str_alloc (arena=<value optimized out>,
> url=0x31be6a8 "*************"...,
> len_in=<value optimized out>, len_out=0x7fff80173d20) at ../../lib/ts/Arena.h:123
> #10 LogUtils::escapify_url (arena=<value optimized out>,
> url=0x31be6a8 "*************"...,
> len_in=<value optimized out>, len_out=0x7fff80173d20) at LogUtils.cc:392
> #11 0x00000000005482e9 in LogAccessHttp::init (this=0x7fff80173cc0) at LogAccessHttp.cc:98
> #12 0x000000000054b436 in Log::access (lad=0x7fff80173cc0) at Log.cc:1055
> #13 0x00000000004f3085 in HttpSM::update_stats (this=0x2ae5480651e0) at HttpSM.cc:6044
> #14 0x00000000004f8918 in HttpSM::kill_this (this=0x2ae5480651e0) at HttpSM.cc:6005
> #15 0x00000000004f8c08 in HttpSM::main_handler (this=0x2ae5480651e0, event=2301, data=0x2ae548066ec8)
> at HttpSM.cc:2452
> The URL was valid, I just "anonymized" it. This is our environment
> $ uname -a
> Linux xxx.prod 2.6.32-131.4.1.el6.x86_64 #1 SMP Fri Jun 10 10:54:26 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
> $ file /usr/bin/traffic_server
> /usr/bin/traffic_server: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, stripped
> There doesn't seem to be more useful information in the frame:
> (gdb) f 5
> #5 0x0000003312414160 in ink_freelist_new (f=0x3312629ce0) at ink_queue.cc:326
> 326 FREELIST_VERSION(item) + 1);
> (gdb) list
> 321 fastalloc_mem_in_use += f->chunk_size * f->type_size;
> 322 #endif
> 323
> 324 } else {
> 325 SET_FREELIST_POINTER_VERSION(next, *ADDRESS_OF_NEXT(TO_PTR(FREELIST_POINTER(item)), f->offset),
> 326 FREELIST_VERSION(item) + 1);
> 327 #if !defined(INK_USE_MUTEX_FOR_FREELISTS)
> 328 result = ink_atomic_cas64((int64_t *) & f->head.data, item.data, next.data);
> 329 #else
> 330 f->head.data = next.data;
> (gdb) p next
> $2 = <value optimized out>
> (gdb) p f
> $3 = (InkFreeList *) 0x3312629ce0
> (gdb) p *f
> $4 = {head = {data = -8941773651046140890}, name = 0x331241e92f "ArenaBlock", type_size = 1024, chunk_size = 128,
> count = 519, allocated = 1536, offset = 0, alignment = 8, allocated_base = 0, count_base = 0}
> (gdb) p item
> $5 = {data = -8953314125091277786}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (TS-1219) Crash report:
ink_freelist_new causing core dumps
Posted by "Brian Geffon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262443#comment-13262443 ]
Brian Geffon edited comment on TS-1219 at 4/26/12 7:51 AM:
-----------------------------------------------------------
I have a possible theory for the crash, but I think jplevyak might be the only one who could say for sure. In 3.0.x inkfreelist_new follows the standard pattern which does a load followed by a CAS:
{noformat}
int result = 0;
do {
INK_QUEUE_LD64(item, f->head);
if (TO_PTR(FREELIST_POINTER(item)) == NULL) {
...
} else {\n
// crash happens on the following line
SET_FREELIST_POINTER_VERSION(next, *ADDRESS_OF_NEXT(TO_PTR(FREELIST_POINTER(item)), f->offset),
FREELIST_VERSION(item) + 1);
result = ink_atomic_cas64((int64_t *) & f->head.data, item.data, next.data);
}
}
while(result == 0)
{noformat}
I believe this might be a race in the event that the head is freed between the time of the load to item and the SET_FREELIST_POINTER_VERSION, then we would potentially be dereferencing item, which is no longer valid if it's freed as part of a dequeue operation. Does anyone who understands the freelists think this might be the situation?
was (Author: briang):
I have a possible theory for the crash, but I think jplevyak might be the only one who could say for sure. In 3.0.x inkfreelist_new follows the standard pattern which does a load followed by a CAS:
int result = 0;
do {
INK_QUEUE_LD64(item, f->head);
if (TO_PTR(FREELIST_POINTER(item)) == NULL) {
...
} else {
// crash happens on the following line
SET_FREELIST_POINTER_VERSION(next, *ADDRESS_OF_NEXT(TO_PTR(FREELIST_POINTER(item)), f->offset),
FREELIST_VERSION(item) + 1);
result = ink_atomic_cas64((int64_t *) & f->head.data, item.data, next.data);
}
}
while(result == 0)
I believe this might be a race in the event that the head is freed between the time of the load to item and the SET_FREELIST_POINTER_VERSION, then we would potentially be dereferencing item, which is no longer valid if it's freed as part of a dequeue operation. Does anyone who understands the freelists think this might be the situation?
> Crash report: ink_freelist_new causing core dumps
> -------------------------------------------------
>
> Key: TS-1219
> URL: https://issues.apache.org/jira/browse/TS-1219
> Project: Traffic Server
> Issue Type: Bug
> Affects Versions: 3.0.2
> Reporter: Manjesh Nilange
>
> Our TS based production boxes are seeing a couple of crashes per day with stack traces ending in
> #3 0x00000000004c85ea in signal_handler (sig=11) at signals.cc:225
> #4 <signal handler called>
> #5 0x0000003312414160 in ink_freelist_new (f=0x3312629ce0) at ink_queue.cc:326
> #6 0x000000331240d47a in alloc_void (this=0x7fff80173cd8, size=511, alignment=1) at Allocator.h:208
> #7 blk_alloc (this=0x7fff80173cd8, size=511, alignment=1) at Arena.cc:45
> #8 Arena::alloc (this=0x7fff80173cd8, size=511, alignment=1) at Arena.cc:118
> #9 0x0000000000569b93 in str_alloc (arena=<value optimized out>,
> url=0x31be6a8 "*************"...,
> len_in=<value optimized out>, len_out=0x7fff80173d20) at ../../lib/ts/Arena.h:123
> #10 LogUtils::escapify_url (arena=<value optimized out>,
> url=0x31be6a8 "*************"...,
> len_in=<value optimized out>, len_out=0x7fff80173d20) at LogUtils.cc:392
> #11 0x00000000005482e9 in LogAccessHttp::init (this=0x7fff80173cc0) at LogAccessHttp.cc:98
> #12 0x000000000054b436 in Log::access (lad=0x7fff80173cc0) at Log.cc:1055
> #13 0x00000000004f3085 in HttpSM::update_stats (this=0x2ae5480651e0) at HttpSM.cc:6044
> #14 0x00000000004f8918 in HttpSM::kill_this (this=0x2ae5480651e0) at HttpSM.cc:6005
> #15 0x00000000004f8c08 in HttpSM::main_handler (this=0x2ae5480651e0, event=2301, data=0x2ae548066ec8)
> at HttpSM.cc:2452
> The URL was valid, I just "anonymized" it. This is our environment
> $ uname -a
> Linux xxx.prod 2.6.32-131.4.1.el6.x86_64 #1 SMP Fri Jun 10 10:54:26 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
> $ file /usr/bin/traffic_server
> /usr/bin/traffic_server: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, stripped
> There doesn't seem to be more useful information in the frame:
> (gdb) f 5
> #5 0x0000003312414160 in ink_freelist_new (f=0x3312629ce0) at ink_queue.cc:326
> 326 FREELIST_VERSION(item) + 1);
> (gdb) list
> 321 fastalloc_mem_in_use += f->chunk_size * f->type_size;
> 322 #endif
> 323
> 324 } else {
> 325 SET_FREELIST_POINTER_VERSION(next, *ADDRESS_OF_NEXT(TO_PTR(FREELIST_POINTER(item)), f->offset),
> 326 FREELIST_VERSION(item) + 1);
> 327 #if !defined(INK_USE_MUTEX_FOR_FREELISTS)
> 328 result = ink_atomic_cas64((int64_t *) & f->head.data, item.data, next.data);
> 329 #else
> 330 f->head.data = next.data;
> (gdb) p next
> $2 = <value optimized out>
> (gdb) p f
> $3 = (InkFreeList *) 0x3312629ce0
> (gdb) p *f
> $4 = {head = {data = -8941773651046140890}, name = 0x331241e92f "ArenaBlock", type_size = 1024, chunk_size = 128,
> count = 519, allocated = 1536, offset = 0, alignment = 8, allocated_base = 0, count_base = 0}
> (gdb) p item
> $5 = {data = -8953314125091277786}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-1219) Crash report: ink_freelist_new causing
core dumps
Posted by "Leif Hedstrom (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261813#comment-13261813 ]
Leif Hedstrom commented on TS-1219:
-----------------------------------
Any chance you can test either 3.0.4 or better, trunk ? I don't know that this has been fixed, but I know there's been a number of fixes on the freelist code since 3.0.
> Crash report: ink_freelist_new causing core dumps
> -------------------------------------------------
>
> Key: TS-1219
> URL: https://issues.apache.org/jira/browse/TS-1219
> Project: Traffic Server
> Issue Type: Bug
> Affects Versions: 3.0.2
> Reporter: Manjesh Nilange
>
> Our TS based production boxes are seeing a couple of crashes per day with stack traces ending in
> #3 0x00000000004c85ea in signal_handler (sig=11) at signals.cc:225
> #4 <signal handler called>
> #5 0x0000003312414160 in ink_freelist_new (f=0x3312629ce0) at ink_queue.cc:326
> #6 0x000000331240d47a in alloc_void (this=0x7fff80173cd8, size=511, alignment=1) at Allocator.h:208
> #7 blk_alloc (this=0x7fff80173cd8, size=511, alignment=1) at Arena.cc:45
> #8 Arena::alloc (this=0x7fff80173cd8, size=511, alignment=1) at Arena.cc:118
> #9 0x0000000000569b93 in str_alloc (arena=<value optimized out>,
> url=0x31be6a8 "*************"...,
> len_in=<value optimized out>, len_out=0x7fff80173d20) at ../../lib/ts/Arena.h:123
> #10 LogUtils::escapify_url (arena=<value optimized out>,
> url=0x31be6a8 "*************"...,
> len_in=<value optimized out>, len_out=0x7fff80173d20) at LogUtils.cc:392
> #11 0x00000000005482e9 in LogAccessHttp::init (this=0x7fff80173cc0) at LogAccessHttp.cc:98
> #12 0x000000000054b436 in Log::access (lad=0x7fff80173cc0) at Log.cc:1055
> #13 0x00000000004f3085 in HttpSM::update_stats (this=0x2ae5480651e0) at HttpSM.cc:6044
> #14 0x00000000004f8918 in HttpSM::kill_this (this=0x2ae5480651e0) at HttpSM.cc:6005
> #15 0x00000000004f8c08 in HttpSM::main_handler (this=0x2ae5480651e0, event=2301, data=0x2ae548066ec8)
> at HttpSM.cc:2452
> The URL was valid, I just "anonymized" it. This is our environment
> $ uname -a
> Linux xxx.prod 2.6.32-131.4.1.el6.x86_64 #1 SMP Fri Jun 10 10:54:26 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
> $ file /usr/bin/traffic_server
> /usr/bin/traffic_server: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, stripped
> There doesn't seem to be more useful information in the frame:
> (gdb) f 5
> #5 0x0000003312414160 in ink_freelist_new (f=0x3312629ce0) at ink_queue.cc:326
> 326 FREELIST_VERSION(item) + 1);
> (gdb) list
> 321 fastalloc_mem_in_use += f->chunk_size * f->type_size;
> 322 #endif
> 323
> 324 } else {
> 325 SET_FREELIST_POINTER_VERSION(next, *ADDRESS_OF_NEXT(TO_PTR(FREELIST_POINTER(item)), f->offset),
> 326 FREELIST_VERSION(item) + 1);
> 327 #if !defined(INK_USE_MUTEX_FOR_FREELISTS)
> 328 result = ink_atomic_cas64((int64_t *) & f->head.data, item.data, next.data);
> 329 #else
> 330 f->head.data = next.data;
> (gdb) p next
> $2 = <value optimized out>
> (gdb) p f
> $3 = (InkFreeList *) 0x3312629ce0
> (gdb) p *f
> $4 = {head = {data = -8941773651046140890}, name = 0x331241e92f "ArenaBlock", type_size = 1024, chunk_size = 128,
> count = 519, allocated = 1536, offset = 0, alignment = 8, allocated_base = 0, count_base = 0}
> (gdb) p item
> $5 = {data = -8953314125091277786}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (TS-1219) Crash report:
ink_freelist_new causing core dumps
Posted by "Brian Geffon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TS-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262443#comment-13262443 ]
Brian Geffon edited comment on TS-1219 at 4/26/12 7:49 AM:
-----------------------------------------------------------
I have a possible theory for the crash, but I think jplevyak might be the only one who could say for sure. In 3.0.x inkfreelist_new follows the standard pattern which does a load followed by a CAS:
int result = 0;
do {
INK_QUEUE_LD64(item, f->head);
if (TO_PTR(FREELIST_POINTER(item)) == NULL) {
...
} else {
// crash happens on the following line
SET_FREELIST_POINTER_VERSION(next, *ADDRESS_OF_NEXT(TO_PTR(FREELIST_POINTER(item)), f->offset),
FREELIST_VERSION(item) + 1);
result = ink_atomic_cas64((int64_t *) & f->head.data, item.data, next.data);
}
}
while(result == 0)
I believe this might be a race in the event that the head is freed between the time of the load to item and the SET_FREELIST_POINTER_VERSION, then we would potentially be dereferencing item, which is no longer valid if it's freed as part of a dequeue operation. Does anyone who understands the freelists think this might be the situation?
was (Author: briang):
I have a possible theory for the crash, but I think jplevyak might be the only one who could say for sure. In 3.0.x inkfreelist_new follows the standard pattern which does a load followed by a CAS:
int result = 0;
do {
INK_QUEUE_LD64(item, f->head);
if (TO_PTR(FREELIST_POINTER(item)) == NULL) {
...
} else {
// crash happens on the following line
SET_FREELIST_POINTER_VERSION(next, *ADDRESS_OF_NEXT(TO_PTR(FREELIST_POINTER(item)), f->offset),
FREELIST_VERSION(item) + 1);
result = ink_atomic_cas64((int64_t *) & f->head.data, item.data, next.data);
}
}
while(result == 0)
I believe this might be a race in the event that the head is freed between the time of the load to item and the SET_FREELIST_POINTER_VERSION, then we would potentially be dereferencing item, which is no longer valid if it's freed as part of a dequeue operation. Does anyone who understands the freelists think this might be the situation?
> Crash report: ink_freelist_new causing core dumps
> -------------------------------------------------
>
> Key: TS-1219
> URL: https://issues.apache.org/jira/browse/TS-1219
> Project: Traffic Server
> Issue Type: Bug
> Affects Versions: 3.0.2
> Reporter: Manjesh Nilange
>
> Our TS based production boxes are seeing a couple of crashes per day with stack traces ending in
> #3 0x00000000004c85ea in signal_handler (sig=11) at signals.cc:225
> #4 <signal handler called>
> #5 0x0000003312414160 in ink_freelist_new (f=0x3312629ce0) at ink_queue.cc:326
> #6 0x000000331240d47a in alloc_void (this=0x7fff80173cd8, size=511, alignment=1) at Allocator.h:208
> #7 blk_alloc (this=0x7fff80173cd8, size=511, alignment=1) at Arena.cc:45
> #8 Arena::alloc (this=0x7fff80173cd8, size=511, alignment=1) at Arena.cc:118
> #9 0x0000000000569b93 in str_alloc (arena=<value optimized out>,
> url=0x31be6a8 "*************"...,
> len_in=<value optimized out>, len_out=0x7fff80173d20) at ../../lib/ts/Arena.h:123
> #10 LogUtils::escapify_url (arena=<value optimized out>,
> url=0x31be6a8 "*************"...,
> len_in=<value optimized out>, len_out=0x7fff80173d20) at LogUtils.cc:392
> #11 0x00000000005482e9 in LogAccessHttp::init (this=0x7fff80173cc0) at LogAccessHttp.cc:98
> #12 0x000000000054b436 in Log::access (lad=0x7fff80173cc0) at Log.cc:1055
> #13 0x00000000004f3085 in HttpSM::update_stats (this=0x2ae5480651e0) at HttpSM.cc:6044
> #14 0x00000000004f8918 in HttpSM::kill_this (this=0x2ae5480651e0) at HttpSM.cc:6005
> #15 0x00000000004f8c08 in HttpSM::main_handler (this=0x2ae5480651e0, event=2301, data=0x2ae548066ec8)
> at HttpSM.cc:2452
> The URL was valid, I just "anonymized" it. This is our environment
> $ uname -a
> Linux xxx.prod 2.6.32-131.4.1.el6.x86_64 #1 SMP Fri Jun 10 10:54:26 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
> $ file /usr/bin/traffic_server
> /usr/bin/traffic_server: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, stripped
> There doesn't seem to be more useful information in the frame:
> (gdb) f 5
> #5 0x0000003312414160 in ink_freelist_new (f=0x3312629ce0) at ink_queue.cc:326
> 326 FREELIST_VERSION(item) + 1);
> (gdb) list
> 321 fastalloc_mem_in_use += f->chunk_size * f->type_size;
> 322 #endif
> 323
> 324 } else {
> 325 SET_FREELIST_POINTER_VERSION(next, *ADDRESS_OF_NEXT(TO_PTR(FREELIST_POINTER(item)), f->offset),
> 326 FREELIST_VERSION(item) + 1);
> 327 #if !defined(INK_USE_MUTEX_FOR_FREELISTS)
> 328 result = ink_atomic_cas64((int64_t *) & f->head.data, item.data, next.data);
> 329 #else
> 330 f->head.data = next.data;
> (gdb) p next
> $2 = <value optimized out>
> (gdb) p f
> $3 = (InkFreeList *) 0x3312629ce0
> (gdb) p *f
> $4 = {head = {data = -8941773651046140890}, name = 0x331241e92f "ArenaBlock", type_size = 1024, chunk_size = 128,
> count = 519, allocated = 1536, offset = 0, alignment = 8, allocated_base = 0, count_base = 0}
> (gdb) p item
> $5 = {data = -8953314125091277786}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira