You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@trafficserver.apache.org by aniket ray <an...@gmail.com> on 2011/05/24 18:07:29 UTC
Is ink_freelist_new thread safe?
Hi,
I'm getting a crasher with the following call stack. Apparently it
happens only when multiple threads have the same call stack.
#0 0x4001e424 in __kernel_vsyscall ()
#1 0x40527941 in raise () from /lib/libc.so.6
#2 0x4052ae42 in abort () from /lib/libc.so.6
#3 0x082f2cbc in ink_die_die_die (return_code=1,
message_format=0x834fc18 "ink_memalign: couldn't allocate %d bytes at
alignment %d - insufficient memory", ap=0x415b3e68 "") at
ink_error.cc:43
#4 ink_fatal_va (return_code=1, message_format=0x834fc18
"ink_memalign: couldn't allocate %d bytes at alignment %d -
insufficient memory", ap=0x415b3e68 "") at ink_error.cc:65
#5 0x082f2cf7 in ink_fatal (return_code=1, message_format=0x834fc18
"ink_memalign: couldn't allocate %d bytes at alignment %d -
insufficient memory") at ink_error.cc:73
#6 0x082f3fc5 in ink_memalign (alignment=8192, size=67108864) at
ink_memory.cc:124
#7 0x082f4762 in ink_freelist_new (f=0x886ec28) at ink_queue.cc:260
#8 0x080e60f8 in MIOBuffer::add_block() ()
#9 0x082e7ae4 in MIOBuffer::write (this=0x9b35fd0, abuf=0x8334750,
alen=2) at IOBuffer.cc:102
#10 0x0819eb38 in ChunkedHandler::generate_chunked_content
(this=0x47af1c24) at HttpTunnel.cc:426
#11 0x081a0847 in producer_handler_dechunked (this=0x47af1a28,
event=100, p=0x47af1c00) at HttpTunnel.cc:953
#12 HttpTunnel::producer_handler (this=0x47af1a28, event=100,
p=0x47af1c00) at HttpTunnel.cc:1051
#13 0x081a14fe in HttpTunnel::main_handler (this=0x47af1a28,
event=3155, data=0x9b578fc) at HttpTunnel.cc:1453
#14 0x08119741 in TransformTerminus::handle_event (this=0x9b578b0,
event=1, edata=0x479aec90) at Transform.cc:279
#15 0x082ea8c2 in handleEvent (this=0x40dad008, e=0x479aec90,
calling_code=1) at I_Continuation.h:147
#16 EThread::process_event (this=0x40dad008, e=0x479aec90,
calling_code=1) at UnixEThread.cc:149
#17 0x082eb2ec in EThread::execute (this=0x40dad008) at UnixEThread.cc:198
#18 0x082e888c in spawn_thread_internal (a=0x9ab75e8) at Thread.cc:85
#19 0x4012acc9 in start_thread () from /lib/libpthread.so.0
#20 0x405cd69e in clone () from /lib/libc.so.6
I'm trying to understand the code. What is a freelist ? What is it
being used for?
My question is whether ink_freelist_new is thread safe?
In trying to understand this I got a few more doubts.
What is a micro-TS build?
Also, I tried adding a "#define INK_USE_MUTEX_FOR_FREELISTS 1" to
the ink_config.h.in file. Traffic_server does not even launch when
built with this flag.
Any ideas that might help me understand the code and track this crasher?
Thanks,
Aniket Ray
Re: Is ink_freelist_new thread safe?
Posted by aniket ray <an...@gmail.com>.
On Thu, May 26, 2011 at 5:11 PM, aniket ray <an...@gmail.com> wrote:
> On Thu, May 26, 2011 at 7:39 AM, John Plevyak <jp...@acm.org> wrote:
>> Ah, yes, that is 64MB. The freelist is thread safe, although a missing
>> "volatile"
>> declaration caused it to fail to be so for the 32-bit version with recent
>> versions
>> of gcc. That is the reason to update to 2.1.8. Are you seeing your problem
>> with
>> a 32-bit or 64-bit build?
>>
>> john
>
> I'm seeing it with the 32-bit version but I'm also seeing it with 2.1.8.
> Based on smaps, it seems the heap is pretty quickly using up the
> virtual memory address space with a very low Rss usage.
> After this posix_memalign starts failing with an ENOMEM.
> I think this implies memory fragmentation.
> Any ideas? Is this a known issue?
>
> -aniket
>
>>
I think the issue is being caused by two optimizations at HttpTunnel.cc.
First At HttpTunnel.cc:421,
chunked_buffer->write(dechunked_reader, write_val);
This clones existing bufferblocks and adds them to the writer list.
Second at HttpTunnel.cc:426, chunked_buffer->write("\r\n", 2);
This allocates a 64MB memory buffer, expecting that more things would
be written to bufferblock . But instead of writing our first
optimization gets called again. Thus this huge buffer gets wasted,
i.e. only 2 bytes out of 64MB get used.
I'm sure I'm missing something but not sure what.
Thanks,
-aniket
>> On Wed, May 25, 2011 at 11:36 AM, aniket ray <an...@gmail.com> wrote:
>>
>>> On Wed, May 25, 2011 at 8:12 PM, John Plevyak <jp...@acm.org> wrote:
>>> > There is no 64MB freelist unless you have created one yourself. The
>>> buffer
>>> > freelists max out at 2MB and unless you are caching files you should not
>>> see
>>> > them allocated.
>>> >
>>> > john
>>> >
>>>
>>> I might have been mistaken but it seems ioBufAllocator[14] has a
>>> type_size of 2M and a chunk_size of 32 (DEFAULT_HUGE_BUFFER_NUMBER) at
>>> init_buffer_allocators().
>>>
>>> This initializes a freelist that tries to memalign 2M * 32 each time a
>>> MIOBuffer::write is called.
>>> Is my understanding incorrect? Just to add, I haven't created any
>>> freelists.
>>>
>>> Thanks,
>>> -aniket
>>>
>>>
>>> > On Wed, May 25, 2011 at 2:24 AM, aniket ray <an...@gmail.com>
>>> wrote:
>>> >
>>> >> On Tue, May 24, 2011 at 9:52 PM, Leif Hedstrom <zw...@apache.org>
>>> wrote:
>>> >> > On 05/24/2011 10:14 AM, aniket ray wrote:
>>> >> >>
>>> >> >> It's trafficserver 2.1.5-unstable
>>> >> >
>>> >> > Can you try v2.1.8, or even better, current trunk, I know we fixed
>>> >> several
>>> >> > bugs related to this code since v2.1.5.
>>> >> >
>>> >> I ported my plugin to the new version of ATS to get it running to test
>>> >> the crasher on v2.1.8. I'm getting the exact same crash with the same
>>> >> call stack.
>>> >>
>>> >> It seems that my thread safety diagnosis might not have been correct.
>>> >> It is trying to allocate 64MB blocks at a time off the freelist till
>>> >> memalign starts failing. Is this the expected behaviour?
>>> >>
>>> >> > -- leif
>>> >> >
>>> >> > Getting trunk:
>>> >> >
>>> >> > svn co http://svn.apache.org/repos/asf/trafficserver/traffic/trunk/
>>> >> >
>>> >>
>>> >> I could not get this working. Does this need a specific version of
>>> >> automake? I have 1.11.
>>> >>
>>> >> Thanks,
>>> >> -aniket
>>> >>
>>> >
>>>
>>
>
Re: Is ink_freelist_new thread safe?
Posted by aniket ray <an...@gmail.com>.
On Thu, May 26, 2011 at 7:39 AM, John Plevyak <jp...@acm.org> wrote:
> Ah, yes, that is 64MB. The freelist is thread safe, although a missing
> "volatile"
> declaration caused it to fail to be so for the 32-bit version with recent
> versions
> of gcc. That is the reason to update to 2.1.8. Are you seeing your problem
> with
> a 32-bit or 64-bit build?
>
> john
I'm seeing it with the 32-bit version but I'm also seeing it with 2.1.8.
Based on smaps, it seems the heap is pretty quickly using up the
virtual memory address space with a very low Rss usage.
After this posix_memalign starts failing with an ENOMEM.
I think this implies memory fragmentation.
Any ideas? Is this a known issue?
-aniket
>
> On Wed, May 25, 2011 at 11:36 AM, aniket ray <an...@gmail.com> wrote:
>
>> On Wed, May 25, 2011 at 8:12 PM, John Plevyak <jp...@acm.org> wrote:
>> > There is no 64MB freelist unless you have created one yourself. The
>> buffer
>> > freelists max out at 2MB and unless you are caching files you should not
>> see
>> > them allocated.
>> >
>> > john
>> >
>>
>> I might have been mistaken but it seems ioBufAllocator[14] has a
>> type_size of 2M and a chunk_size of 32 (DEFAULT_HUGE_BUFFER_NUMBER) at
>> init_buffer_allocators().
>>
>> This initializes a freelist that tries to memalign 2M * 32 each time a
>> MIOBuffer::write is called.
>> Is my understanding incorrect? Just to add, I haven't created any
>> freelists.
>>
>> Thanks,
>> -aniket
>>
>>
>> > On Wed, May 25, 2011 at 2:24 AM, aniket ray <an...@gmail.com>
>> wrote:
>> >
>> >> On Tue, May 24, 2011 at 9:52 PM, Leif Hedstrom <zw...@apache.org>
>> wrote:
>> >> > On 05/24/2011 10:14 AM, aniket ray wrote:
>> >> >>
>> >> >> It's trafficserver 2.1.5-unstable
>> >> >
>> >> > Can you try v2.1.8, or even better, current trunk, I know we fixed
>> >> several
>> >> > bugs related to this code since v2.1.5.
>> >> >
>> >> I ported my plugin to the new version of ATS to get it running to test
>> >> the crasher on v2.1.8. I'm getting the exact same crash with the same
>> >> call stack.
>> >>
>> >> It seems that my thread safety diagnosis might not have been correct.
>> >> It is trying to allocate 64MB blocks at a time off the freelist till
>> >> memalign starts failing. Is this the expected behaviour?
>> >>
>> >> > -- leif
>> >> >
>> >> > Getting trunk:
>> >> >
>> >> > svn co http://svn.apache.org/repos/asf/trafficserver/traffic/trunk/
>> >> >
>> >>
>> >> I could not get this working. Does this need a specific version of
>> >> automake? I have 1.11.
>> >>
>> >> Thanks,
>> >> -aniket
>> >>
>> >
>>
>
Re: Is ink_freelist_new thread safe?
Posted by John Plevyak <jp...@acm.org>.
Ah, yes, that is 64MB. The freelist is thread safe, although a missing
"volatile"
declaration caused it to fail to be so for the 32-bit version with recent
versions
of gcc. That is the reason to update to 2.1.8. Are you seeing your problem
with
a 32-bit or 64-bit build?
john
On Wed, May 25, 2011 at 11:36 AM, aniket ray <an...@gmail.com> wrote:
> On Wed, May 25, 2011 at 8:12 PM, John Plevyak <jp...@acm.org> wrote:
> > There is no 64MB freelist unless you have created one yourself. The
> buffer
> > freelists max out at 2MB and unless you are caching files you should not
> see
> > them allocated.
> >
> > john
> >
>
> I might have been mistaken but it seems ioBufAllocator[14] has a
> type_size of 2M and a chunk_size of 32 (DEFAULT_HUGE_BUFFER_NUMBER) at
> init_buffer_allocators().
>
> This initializes a freelist that tries to memalign 2M * 32 each time a
> MIOBuffer::write is called.
> Is my understanding incorrect? Just to add, I haven't created any
> freelists.
>
> Thanks,
> -aniket
>
>
> > On Wed, May 25, 2011 at 2:24 AM, aniket ray <an...@gmail.com>
> wrote:
> >
> >> On Tue, May 24, 2011 at 9:52 PM, Leif Hedstrom <zw...@apache.org>
> wrote:
> >> > On 05/24/2011 10:14 AM, aniket ray wrote:
> >> >>
> >> >> It's trafficserver 2.1.5-unstable
> >> >
> >> > Can you try v2.1.8, or even better, current trunk, I know we fixed
> >> several
> >> > bugs related to this code since v2.1.5.
> >> >
> >> I ported my plugin to the new version of ATS to get it running to test
> >> the crasher on v2.1.8. I'm getting the exact same crash with the same
> >> call stack.
> >>
> >> It seems that my thread safety diagnosis might not have been correct.
> >> It is trying to allocate 64MB blocks at a time off the freelist till
> >> memalign starts failing. Is this the expected behaviour?
> >>
> >> > -- leif
> >> >
> >> > Getting trunk:
> >> >
> >> > svn co http://svn.apache.org/repos/asf/trafficserver/traffic/trunk/
> >> >
> >>
> >> I could not get this working. Does this need a specific version of
> >> automake? I have 1.11.
> >>
> >> Thanks,
> >> -aniket
> >>
> >
>
Re: Is ink_freelist_new thread safe?
Posted by aniket ray <an...@gmail.com>.
On Wed, May 25, 2011 at 8:12 PM, John Plevyak <jp...@acm.org> wrote:
> There is no 64MB freelist unless you have created one yourself. The buffer
> freelists max out at 2MB and unless you are caching files you should not see
> them allocated.
>
> john
>
I might have been mistaken but it seems ioBufAllocator[14] has a
type_size of 2M and a chunk_size of 32 (DEFAULT_HUGE_BUFFER_NUMBER) at
init_buffer_allocators().
This initializes a freelist that tries to memalign 2M * 32 each time a
MIOBuffer::write is called.
Is my understanding incorrect? Just to add, I haven't created any freelists.
Thanks,
-aniket
> On Wed, May 25, 2011 at 2:24 AM, aniket ray <an...@gmail.com> wrote:
>
>> On Tue, May 24, 2011 at 9:52 PM, Leif Hedstrom <zw...@apache.org> wrote:
>> > On 05/24/2011 10:14 AM, aniket ray wrote:
>> >>
>> >> It's trafficserver 2.1.5-unstable
>> >
>> > Can you try v2.1.8, or even better, current trunk, I know we fixed
>> several
>> > bugs related to this code since v2.1.5.
>> >
>> I ported my plugin to the new version of ATS to get it running to test
>> the crasher on v2.1.8. I'm getting the exact same crash with the same
>> call stack.
>>
>> It seems that my thread safety diagnosis might not have been correct.
>> It is trying to allocate 64MB blocks at a time off the freelist till
>> memalign starts failing. Is this the expected behaviour?
>>
>> > -- leif
>> >
>> > Getting trunk:
>> >
>> > svn co http://svn.apache.org/repos/asf/trafficserver/traffic/trunk/
>> >
>>
>> I could not get this working. Does this need a specific version of
>> automake? I have 1.11.
>>
>> Thanks,
>> -aniket
>>
>
Re: Is ink_freelist_new thread safe?
Posted by John Plevyak <jp...@acm.org>.
There is no 64MB freelist unless you have created one yourself. The buffer
freelists max out at 2MB and unless you are caching files you should not see
them allocated.
john
On Wed, May 25, 2011 at 2:24 AM, aniket ray <an...@gmail.com> wrote:
> On Tue, May 24, 2011 at 9:52 PM, Leif Hedstrom <zw...@apache.org> wrote:
> > On 05/24/2011 10:14 AM, aniket ray wrote:
> >>
> >> It's trafficserver 2.1.5-unstable
> >
> > Can you try v2.1.8, or even better, current trunk, I know we fixed
> several
> > bugs related to this code since v2.1.5.
> >
> I ported my plugin to the new version of ATS to get it running to test
> the crasher on v2.1.8. I'm getting the exact same crash with the same
> call stack.
>
> It seems that my thread safety diagnosis might not have been correct.
> It is trying to allocate 64MB blocks at a time off the freelist till
> memalign starts failing. Is this the expected behaviour?
>
> > -- leif
> >
> > Getting trunk:
> >
> > svn co http://svn.apache.org/repos/asf/trafficserver/traffic/trunk/
> >
>
> I could not get this working. Does this need a specific version of
> automake? I have 1.11.
>
> Thanks,
> -aniket
>
Re: Is ink_freelist_new thread safe?
Posted by aniket ray <an...@gmail.com>.
On Tue, May 24, 2011 at 9:52 PM, Leif Hedstrom <zw...@apache.org> wrote:
> On 05/24/2011 10:14 AM, aniket ray wrote:
>>
>> It's trafficserver 2.1.5-unstable
>
> Can you try v2.1.8, or even better, current trunk, I know we fixed several
> bugs related to this code since v2.1.5.
>
I ported my plugin to the new version of ATS to get it running to test
the crasher on v2.1.8. I'm getting the exact same crash with the same
call stack.
It seems that my thread safety diagnosis might not have been correct.
It is trying to allocate 64MB blocks at a time off the freelist till
memalign starts failing. Is this the expected behaviour?
> -- leif
>
> Getting trunk:
>
> svn co http://svn.apache.org/repos/asf/trafficserver/traffic/trunk/
>
I could not get this working. Does this need a specific version of
automake? I have 1.11.
Thanks,
-aniket
Re: Is ink_freelist_new thread safe?
Posted by Leif Hedstrom <zw...@apache.org>.
On 05/24/2011 10:14 AM, aniket ray wrote:
> It's trafficserver 2.1.5-unstable
Can you try v2.1.8, or even better, current trunk, I know we fixed
several bugs related to this code since v2.1.5.
-- leif
Getting trunk:
svn co http://svn.apache.org/repos/asf/trafficserver/traffic/trunk/
Re: Is ink_freelist_new thread safe?
Posted by aniket ray <an...@gmail.com>.
It's trafficserver 2.1.5-unstable
Thanks,
-aniket
On Tue, May 24, 2011 at 9:42 PM, Leif Hedstrom <zw...@apache.org> wrote:
> On 05/24/2011 10:07 AM, aniket ray wrote:
>>
>> Hi,
>>
>> I'm getting a crasher with the following call stack. Apparently it
>> happens only when multiple threads have the same call stack.
>
> What version?
>
> -- leif
>
>
Re: Is ink_freelist_new thread safe?
Posted by Leif Hedstrom <zw...@apache.org>.
On 05/24/2011 10:07 AM, aniket ray wrote:
> Hi,
>
> I'm getting a crasher with the following call stack. Apparently it
> happens only when multiple threads have the same call stack.
What version?
-- leif