You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@trafficserver.apache.org by Vamsi Ambati <va...@neumob.com> on 2016/11/10 21:43:52 UTC
Address Sanitizer issue in detecting memory leak.
I am trying to resolve a memory leak issue with ATS 7.0 version.
We are using ATS from the CDN perspective and developed 3 plugins.
DevOps complains that traffic_server process gradually consumes lot of memory and does release back which is a sign of memory leak.
I tried with Valgrind but it didn’t work because of LUA and one of our ATS community member suggested to use Address Sanitizer.
I followed exactly the steps mentioned Debugging Traffic Server using ASAN <https://cwiki.apache.org/confluence/download/attachments/56066455/summit_asan.pptx?version=1&modificationDate=1429916307000&api=v2>.
I have build the traffic server with CXXFLAGS=-fno-omit-frame-pointer -fsanitize=address
Verified that traffic_server is build with ASAN library by 'ldd bin/traffic_server’
Restarted the traffic server with 'service nm-trafficserver restart'.
I have adopted three approaches to catch the memory leak
Approach 1:
Just running the traffic server(with -f option) and continuously sending HTTP requests using wrk (from another machine) by attaching
to a gdb .
' gdb $(pidof traffic_server)’
break __asan_report_error.
Issue: I didn’t hit this function from the memory leak perspective.
Approach 2:
I have run the traffic server with PROXY_AUTO_EXIT and sending the http requests
PROXY_AUTO_EXIT=30 ASAN_OPTIONS=detect_leaks=1:verbosity=2 bin/traffic_server -f
Issue: Tool didn’t complain any thing about the memory leak.
Approach 3:
I have created simple code explicitly introducing the memory leak
#include <stdlib.h>
void dummy() {
malloc(20);
}
int main() {
dummy();
return 0;
}
Compiled with g++ -fsanitize=address add.cpp
Executed like ASAN_OPTIONS=detect_leaks=1 ./a.out
Issue: Tool didn’t detect this memory leak.
Production System Configuration:
16Gb Ram with 8 cores VM.
gcc --version
gcc (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4
Did any one encounter this ASAN issue on Ubuntu 14.04 ? If so, how did they resolve it ?
Any other suggestions how to narrow down the memory leak issue with ATS ?
Regards
Vamsi
PS: I am new to the community and new to Apache traffic server open source module too.
Re: Address Sanitizer issue in detecting memory leak.
Posted by Bryan Call <bc...@apache.org>.
It is working on my Fedora 24. You might need to upgrade your compiler or OS.
[bcall@homer ~]$ cat test.cc
#include <stdlib.h>
void dummy() {
void *x = malloc(20);
}
int main() {
dummy();
return 0;
}
[bcall@homer ~]$ g++ test.cc -fno-omit-frame-pointer -fsanitize=address
[bcall@homer ~]$ ./a.out
=================================================================
==9344==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 20 byte(s) in 1 object(s) allocated from:
#0 0x7f00500b3e60 in malloc (/lib64/libasan.so.3+0xc6e60)
#1 0x400777 in dummy() (/home/bcall/a.out+0x400777)
#2 0x400787 in main (/home/bcall/a.out+0x400787)
#3 0x7f004f3a3730 in __libc_start_main (/lib64/libc.so.6+0x20730)
SUMMARY: AddressSanitizer: 20 byte(s) leaked in 1 allocation(s).
-Bryan
> On Nov 10, 2016, at 2:22 PM, Vamsi Ambati <va...@neumob.com> wrote:
>
> Hi Sudheer,
>
> Thanks for the quick response.
> I tried with your suggested code snippet but Address sanitizer doesn’t detect the leak.
> I even tried with gdb by introducing break point at __asan_report_error.
>
>
> Vamsi
>
>> On Nov 10, 2016, at 1:58 PM, Sudheer Vinukonda <sudheervinukonda@yahoo.com <ma...@yahoo.com>> wrote:
>>
>> Not sure if the *leak* in your test code below is detectable..since your program is exiting after allocating the memory (system should automatically reclaim the memory on process exit).
>>
>> You may try to modify it to something more explicit like the below -
>>
>>> #include <stdlib.h>
>>> void dummy() {
>>> char *c = (char*) malloc(20);
>> c = (char*) malloc(100);
>>> }
>>> int main() {
>>> dummy();
>>> while(1) {}
>>> }
>>
>>
>>
>> On Nov 10, 2016, at 1:43 PM, Vamsi Ambati <vamsi@neumob.com <ma...@neumob.com>> wrote:
>>
>>>
>>> I am trying to resolve a memory leak issue with ATS 7.0 version.
>>> We are using ATS from the CDN perspective and developed 3 plugins.
>>> DevOps complains that traffic_server process gradually consumes lot of memory and does release back which is a sign of memory leak.
>>> I tried with Valgrind but it didn’t work because of LUA and one of our ATS community member suggested to use Address Sanitizer.
>>> I followed exactly the steps mentioned Debugging Traffic Server using ASAN <https://cwiki.apache.org/confluence/download/attachments/56066455/summit_asan.pptx?version=1&modificationDate=1429916307000&api=v2>.
>>>
>>> I have build the traffic server with CXXFLAGS=-fno-omit-frame-pointer -fsanitize=address
>>> Verified that traffic_server is build with ASAN library by 'ldd bin/traffic_server’
>>> Restarted the traffic server with 'service nm-trafficserver restart'.
>>>
>>> I have adopted three approaches to catch the memory leak
>>>
>>> Approach 1:
>>> Just running the traffic server(with -f option) and continuously sending HTTP requests using wrk (from another machine) by attaching
>>> to a gdb .
>>> ' gdb $(pidof traffic_server)’
>>> break __asan_report_error.
>>>
>>> Issue: I didn’t hit this function from the memory leak perspective.
>>>
>>> Approach 2:
>>> I have run the traffic server with PROXY_AUTO_EXIT and sending the http requests
>>> PROXY_AUTO_EXIT=30 ASAN_OPTIONS=detect_leaks=1:verbosity=2 bin/traffic_server -f
>>>
>>> Issue: Tool didn’t complain any thing about the memory leak.
>>>
>>>
>>> Approach 3:
>>> I have created simple code explicitly introducing the memory leak
>>>
>>> #include <stdlib.h>
>>> void dummy() {
>>> malloc(20);
>>> }
>>> int main() {
>>> dummy();
>>> return 0;
>>> }
>>>
>>> Compiled with g++ -fsanitize=address add.cpp
>>> Executed like ASAN_OPTIONS=detect_leaks=1 ./a.out
>>>
>>> Issue: Tool didn’t detect this memory leak.
>>>
>>>
>>> Production System Configuration:
>>>
>>> 16Gb Ram with 8 cores VM.
>>> gcc --version
>>> gcc (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4
>>>
>>> Did any one encounter this ASAN issue on Ubuntu 14.04 ? If so, how did they resolve it ?
>>> Any other suggestions how to narrow down the memory leak issue with ATS ?
>>>
>>>
>>>
>>> Regards
>>> Vamsi
>>>
>>> PS: I am new to the community and new to Apache traffic server open source module too.
>>>
>>>
>>>
>>>
>
Re: Address Sanitizer issue in detecting memory leak.
Posted by Bryan Call <bc...@apache.org>.
It is working on my Fedora 24. You might need to upgrade your compiler or OS.
[bcall@homer ~]$ cat test.cc
#include <stdlib.h>
void dummy() {
void *x = malloc(20);
}
int main() {
dummy();
return 0;
}
[bcall@homer ~]$ g++ test.cc -fno-omit-frame-pointer -fsanitize=address
[bcall@homer ~]$ ./a.out
=================================================================
==9344==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 20 byte(s) in 1 object(s) allocated from:
#0 0x7f00500b3e60 in malloc (/lib64/libasan.so.3+0xc6e60)
#1 0x400777 in dummy() (/home/bcall/a.out+0x400777)
#2 0x400787 in main (/home/bcall/a.out+0x400787)
#3 0x7f004f3a3730 in __libc_start_main (/lib64/libc.so.6+0x20730)
SUMMARY: AddressSanitizer: 20 byte(s) leaked in 1 allocation(s).
-Bryan
> On Nov 10, 2016, at 2:22 PM, Vamsi Ambati <va...@neumob.com> wrote:
>
> Hi Sudheer,
>
> Thanks for the quick response.
> I tried with your suggested code snippet but Address sanitizer doesn’t detect the leak.
> I even tried with gdb by introducing break point at __asan_report_error.
>
>
> Vamsi
>
>> On Nov 10, 2016, at 1:58 PM, Sudheer Vinukonda <sudheervinukonda@yahoo.com <ma...@yahoo.com>> wrote:
>>
>> Not sure if the *leak* in your test code below is detectable..since your program is exiting after allocating the memory (system should automatically reclaim the memory on process exit).
>>
>> You may try to modify it to something more explicit like the below -
>>
>>> #include <stdlib.h>
>>> void dummy() {
>>> char *c = (char*) malloc(20);
>> c = (char*) malloc(100);
>>> }
>>> int main() {
>>> dummy();
>>> while(1) {}
>>> }
>>
>>
>>
>> On Nov 10, 2016, at 1:43 PM, Vamsi Ambati <vamsi@neumob.com <ma...@neumob.com>> wrote:
>>
>>>
>>> I am trying to resolve a memory leak issue with ATS 7.0 version.
>>> We are using ATS from the CDN perspective and developed 3 plugins.
>>> DevOps complains that traffic_server process gradually consumes lot of memory and does release back which is a sign of memory leak.
>>> I tried with Valgrind but it didn’t work because of LUA and one of our ATS community member suggested to use Address Sanitizer.
>>> I followed exactly the steps mentioned Debugging Traffic Server using ASAN <https://cwiki.apache.org/confluence/download/attachments/56066455/summit_asan.pptx?version=1&modificationDate=1429916307000&api=v2>.
>>>
>>> I have build the traffic server with CXXFLAGS=-fno-omit-frame-pointer -fsanitize=address
>>> Verified that traffic_server is build with ASAN library by 'ldd bin/traffic_server’
>>> Restarted the traffic server with 'service nm-trafficserver restart'.
>>>
>>> I have adopted three approaches to catch the memory leak
>>>
>>> Approach 1:
>>> Just running the traffic server(with -f option) and continuously sending HTTP requests using wrk (from another machine) by attaching
>>> to a gdb .
>>> ' gdb $(pidof traffic_server)’
>>> break __asan_report_error.
>>>
>>> Issue: I didn’t hit this function from the memory leak perspective.
>>>
>>> Approach 2:
>>> I have run the traffic server with PROXY_AUTO_EXIT and sending the http requests
>>> PROXY_AUTO_EXIT=30 ASAN_OPTIONS=detect_leaks=1:verbosity=2 bin/traffic_server -f
>>>
>>> Issue: Tool didn’t complain any thing about the memory leak.
>>>
>>>
>>> Approach 3:
>>> I have created simple code explicitly introducing the memory leak
>>>
>>> #include <stdlib.h>
>>> void dummy() {
>>> malloc(20);
>>> }
>>> int main() {
>>> dummy();
>>> return 0;
>>> }
>>>
>>> Compiled with g++ -fsanitize=address add.cpp
>>> Executed like ASAN_OPTIONS=detect_leaks=1 ./a.out
>>>
>>> Issue: Tool didn’t detect this memory leak.
>>>
>>>
>>> Production System Configuration:
>>>
>>> 16Gb Ram with 8 cores VM.
>>> gcc --version
>>> gcc (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4
>>>
>>> Did any one encounter this ASAN issue on Ubuntu 14.04 ? If so, how did they resolve it ?
>>> Any other suggestions how to narrow down the memory leak issue with ATS ?
>>>
>>>
>>>
>>> Regards
>>> Vamsi
>>>
>>> PS: I am new to the community and new to Apache traffic server open source module too.
>>>
>>>
>>>
>>>
>
Re: Address Sanitizer issue in detecting memory leak.
Posted by Vamsi Ambati <va...@neumob.com>.
Address Sanitizer issue is resolved at my end. I am providing the details how it was solved , may be helpful for others.
Address sanitizer doesn’t detect memory leaks in Ubuntu 14.04 LTS with gcc version 4.8.4 .
I have found a patch to gcc for Address Sanitizer at https://gcc.gnu.org/ml/gcc-patches/2013-11/msg01874.html.
I have found the code changes related to this patch in the official gcc 4.9.X release.
When I upgraded to gcc 4.9, then address sanitizer started detecting memory leaks .
One more observation is that Address sanitizer throws the memory leak log only when program exits , if it is running in a for ever loop then it doesn’t
show the leak log.
Regards
Vamsi
> On Nov 15, 2016, at 11:57 PM, Bryan Call <bc...@apache.org> wrote:
>
> Don’t know why Apple Mail didn’t send this. I noticed after restarting:
>
> By default you should see some leaks from CRYPTO_malloc even if you don’t have TLS/SSL configured. I am running Fedora 24.
>
> [bcall@homer trafficserver]$ sudo /usr/local/bin/traffic_server
> traffic_server: using root directory '/usr/local'
> ^Ctraffic_server: Interrupt (Signal sent by the kernel 0 0)
>
> =================================================================
> ==9081==ERROR: LeakSanitizer: detected memory leaks
>
> Direct leak of 48 byte(s) in 1 object(s) allocated from:
> #0 0x7f2a242f7e60 in malloc (/lib64/libasan.so.3+0xc6e60)
> #1 0x7f2a231320b7 in CRYPTO_malloc (/lib64/libcrypto.so.10+0x6e0b7)
>
> Indirect leak of 120 byte(s) in 1 object(s) allocated from:
> #0 0x7f2a242f7e60 in malloc (/lib64/libasan.so.3+0xc6e60)
> #1 0x7f2a231320b7 in CRYPTO_malloc (/lib64/libcrypto.so.10+0x6e0b7)
>
> SUMMARY: AddressSanitizer: 168 byte(s) leaked in 2 allocation(s).
>
> -Bryan
>
>
>
>
>> On Nov 10, 2016, at 2:22 PM, Vamsi Ambati <vamsi@neumob.com <ma...@neumob.com>> wrote:
>>
>> Hi Sudheer,
>>
>> Thanks for the quick response.
>> I tried with your suggested code snippet but Address sanitizer doesn’t detect the leak.
>> I even tried with gdb by introducing break point at __asan_report_error.
>>
>>
>> Vamsi
>>
>>> On Nov 10, 2016, at 1:58 PM, Sudheer Vinukonda <sudheervinukonda@yahoo.com <ma...@yahoo.com>> wrote:
>>>
>>> Not sure if the *leak* in your test code below is detectable..since your program is exiting after allocating the memory (system should automatically reclaim the memory on process exit).
>>>
>>> You may try to modify it to something more explicit like the below -
>>>
>>>> #include <stdlib.h>
>>>> void dummy() {
>>>> char *c = (char*) malloc(20);
>>> c = (char*) malloc(100);
>>>> }
>>>> int main() {
>>>> dummy();
>>>> while(1) {}
>>>> }
>>>
>>>
>>>
>>> On Nov 10, 2016, at 1:43 PM, Vamsi Ambati <vamsi@neumob.com <ma...@neumob.com>> wrote:
>>>
>>>>
>>>> I am trying to resolve a memory leak issue with ATS 7.0 version.
>>>> We are using ATS from the CDN perspective and developed 3 plugins.
>>>> DevOps complains that traffic_server process gradually consumes lot of memory and does release back which is a sign of memory leak.
>>>> I tried with Valgrind but it didn’t work because of LUA and one of our ATS community member suggested to use Address Sanitizer.
>>>> I followed exactly the steps mentioned Debugging Traffic Server using ASAN <https://cwiki.apache.org/confluence/download/attachments/56066455/summit_asan.pptx?version=1&modificationDate=1429916307000&api=v2>.
>>>>
>>>> I have build the traffic server with CXXFLAGS=-fno-omit-frame-pointer -fsanitize=address
>>>> Verified that traffic_server is build with ASAN library by 'ldd bin/traffic_server’
>>>> Restarted the traffic server with 'service nm-trafficserver restart'.
>>>>
>>>> I have adopted three approaches to catch the memory leak
>>>>
>>>> Approach 1:
>>>> Just running the traffic server(with -f option) and continuously sending HTTP requests using wrk (from another machine) by attaching
>>>> to a gdb .
>>>> ' gdb $(pidof traffic_server)’
>>>> break __asan_report_error.
>>>>
>>>> Issue: I didn’t hit this function from the memory leak perspective.
>>>>
>>>> Approach 2:
>>>> I have run the traffic server with PROXY_AUTO_EXIT and sending the http requests
>>>> PROXY_AUTO_EXIT=30 ASAN_OPTIONS=detect_leaks=1:verbosity=2 bin/traffic_server -f
>>>>
>>>> Issue: Tool didn’t complain any thing about the memory leak.
>>>>
>>>>
>>>> Approach 3:
>>>> I have created simple code explicitly introducing the memory leak
>>>>
>>>> #include <stdlib.h>
>>>> void dummy() {
>>>> malloc(20);
>>>> }
>>>> int main() {
>>>> dummy();
>>>> return 0;
>>>> }
>>>>
>>>> Compiled with g++ -fsanitize=address add.cpp
>>>> Executed like ASAN_OPTIONS=detect_leaks=1 ./a.out
>>>>
>>>> Issue: Tool didn’t detect this memory leak.
>>>>
>>>>
>>>> Production System Configuration:
>>>>
>>>> 16Gb Ram with 8 cores VM.
>>>> gcc --version
>>>> gcc (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4
>>>>
>>>> Did any one encounter this ASAN issue on Ubuntu 14.04 ? If so, how did they resolve it ?
>>>> Any other suggestions how to narrow down the memory leak issue with ATS ?
>>>>
>>>>
>>>>
>>>> Regards
>>>> Vamsi
>>>>
>>>> PS: I am new to the community and new to Apache traffic server open source module too.
>>>>
>>>>
>>>>
>>>>
>>
>
Re: Address Sanitizer issue in detecting memory leak.
Posted by Vamsi Ambati <va...@neumob.com>.
Address Sanitizer issue is resolved at my end. I am providing the details how it was solved , may be helpful for others.
Address sanitizer doesn’t detect memory leaks in Ubuntu 14.04 LTS with gcc version 4.8.4 .
I have found a patch to gcc for Address Sanitizer at https://gcc.gnu.org/ml/gcc-patches/2013-11/msg01874.html.
I have found the code changes related to this patch in the official gcc 4.9.X release.
When I upgraded to gcc 4.9, then address sanitizer started detecting memory leaks .
One more observation is that Address sanitizer throws the memory leak log only when program exits , if it is running in a for ever loop then it doesn’t
show the leak log.
Regards
Vamsi
> On Nov 15, 2016, at 11:57 PM, Bryan Call <bc...@apache.org> wrote:
>
> Don’t know why Apple Mail didn’t send this. I noticed after restarting:
>
> By default you should see some leaks from CRYPTO_malloc even if you don’t have TLS/SSL configured. I am running Fedora 24.
>
> [bcall@homer trafficserver]$ sudo /usr/local/bin/traffic_server
> traffic_server: using root directory '/usr/local'
> ^Ctraffic_server: Interrupt (Signal sent by the kernel 0 0)
>
> =================================================================
> ==9081==ERROR: LeakSanitizer: detected memory leaks
>
> Direct leak of 48 byte(s) in 1 object(s) allocated from:
> #0 0x7f2a242f7e60 in malloc (/lib64/libasan.so.3+0xc6e60)
> #1 0x7f2a231320b7 in CRYPTO_malloc (/lib64/libcrypto.so.10+0x6e0b7)
>
> Indirect leak of 120 byte(s) in 1 object(s) allocated from:
> #0 0x7f2a242f7e60 in malloc (/lib64/libasan.so.3+0xc6e60)
> #1 0x7f2a231320b7 in CRYPTO_malloc (/lib64/libcrypto.so.10+0x6e0b7)
>
> SUMMARY: AddressSanitizer: 168 byte(s) leaked in 2 allocation(s).
>
> -Bryan
>
>
>
>
>> On Nov 10, 2016, at 2:22 PM, Vamsi Ambati <vamsi@neumob.com <ma...@neumob.com>> wrote:
>>
>> Hi Sudheer,
>>
>> Thanks for the quick response.
>> I tried with your suggested code snippet but Address sanitizer doesn’t detect the leak.
>> I even tried with gdb by introducing break point at __asan_report_error.
>>
>>
>> Vamsi
>>
>>> On Nov 10, 2016, at 1:58 PM, Sudheer Vinukonda <sudheervinukonda@yahoo.com <ma...@yahoo.com>> wrote:
>>>
>>> Not sure if the *leak* in your test code below is detectable..since your program is exiting after allocating the memory (system should automatically reclaim the memory on process exit).
>>>
>>> You may try to modify it to something more explicit like the below -
>>>
>>>> #include <stdlib.h>
>>>> void dummy() {
>>>> char *c = (char*) malloc(20);
>>> c = (char*) malloc(100);
>>>> }
>>>> int main() {
>>>> dummy();
>>>> while(1) {}
>>>> }
>>>
>>>
>>>
>>> On Nov 10, 2016, at 1:43 PM, Vamsi Ambati <vamsi@neumob.com <ma...@neumob.com>> wrote:
>>>
>>>>
>>>> I am trying to resolve a memory leak issue with ATS 7.0 version.
>>>> We are using ATS from the CDN perspective and developed 3 plugins.
>>>> DevOps complains that traffic_server process gradually consumes lot of memory and does release back which is a sign of memory leak.
>>>> I tried with Valgrind but it didn’t work because of LUA and one of our ATS community member suggested to use Address Sanitizer.
>>>> I followed exactly the steps mentioned Debugging Traffic Server using ASAN <https://cwiki.apache.org/confluence/download/attachments/56066455/summit_asan.pptx?version=1&modificationDate=1429916307000&api=v2>.
>>>>
>>>> I have build the traffic server with CXXFLAGS=-fno-omit-frame-pointer -fsanitize=address
>>>> Verified that traffic_server is build with ASAN library by 'ldd bin/traffic_server’
>>>> Restarted the traffic server with 'service nm-trafficserver restart'.
>>>>
>>>> I have adopted three approaches to catch the memory leak
>>>>
>>>> Approach 1:
>>>> Just running the traffic server(with -f option) and continuously sending HTTP requests using wrk (from another machine) by attaching
>>>> to a gdb .
>>>> ' gdb $(pidof traffic_server)’
>>>> break __asan_report_error.
>>>>
>>>> Issue: I didn’t hit this function from the memory leak perspective.
>>>>
>>>> Approach 2:
>>>> I have run the traffic server with PROXY_AUTO_EXIT and sending the http requests
>>>> PROXY_AUTO_EXIT=30 ASAN_OPTIONS=detect_leaks=1:verbosity=2 bin/traffic_server -f
>>>>
>>>> Issue: Tool didn’t complain any thing about the memory leak.
>>>>
>>>>
>>>> Approach 3:
>>>> I have created simple code explicitly introducing the memory leak
>>>>
>>>> #include <stdlib.h>
>>>> void dummy() {
>>>> malloc(20);
>>>> }
>>>> int main() {
>>>> dummy();
>>>> return 0;
>>>> }
>>>>
>>>> Compiled with g++ -fsanitize=address add.cpp
>>>> Executed like ASAN_OPTIONS=detect_leaks=1 ./a.out
>>>>
>>>> Issue: Tool didn’t detect this memory leak.
>>>>
>>>>
>>>> Production System Configuration:
>>>>
>>>> 16Gb Ram with 8 cores VM.
>>>> gcc --version
>>>> gcc (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4
>>>>
>>>> Did any one encounter this ASAN issue on Ubuntu 14.04 ? If so, how did they resolve it ?
>>>> Any other suggestions how to narrow down the memory leak issue with ATS ?
>>>>
>>>>
>>>>
>>>> Regards
>>>> Vamsi
>>>>
>>>> PS: I am new to the community and new to Apache traffic server open source module too.
>>>>
>>>>
>>>>
>>>>
>>
>
Re: Address Sanitizer issue in detecting memory leak.
Posted by Bryan Call <bc...@apache.org>.
Don’t know why Apple Mail didn’t send this. I noticed after restarting:
By default you should see some leaks from CRYPTO_malloc even if you don’t have TLS/SSL configured. I am running Fedora 24.
[bcall@homer trafficserver]$ sudo /usr/local/bin/traffic_server
traffic_server: using root directory '/usr/local'
^Ctraffic_server: Interrupt (Signal sent by the kernel 0 0)
=================================================================
==9081==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 48 byte(s) in 1 object(s) allocated from:
#0 0x7f2a242f7e60 in malloc (/lib64/libasan.so.3+0xc6e60)
#1 0x7f2a231320b7 in CRYPTO_malloc (/lib64/libcrypto.so.10+0x6e0b7)
Indirect leak of 120 byte(s) in 1 object(s) allocated from:
#0 0x7f2a242f7e60 in malloc (/lib64/libasan.so.3+0xc6e60)
#1 0x7f2a231320b7 in CRYPTO_malloc (/lib64/libcrypto.so.10+0x6e0b7)
SUMMARY: AddressSanitizer: 168 byte(s) leaked in 2 allocation(s).
-Bryan
> On Nov 10, 2016, at 2:22 PM, Vamsi Ambati <va...@neumob.com> wrote:
>
> Hi Sudheer,
>
> Thanks for the quick response.
> I tried with your suggested code snippet but Address sanitizer doesn’t detect the leak.
> I even tried with gdb by introducing break point at __asan_report_error.
>
>
> Vamsi
>
>> On Nov 10, 2016, at 1:58 PM, Sudheer Vinukonda <sudheervinukonda@yahoo.com <ma...@yahoo.com>> wrote:
>>
>> Not sure if the *leak* in your test code below is detectable..since your program is exiting after allocating the memory (system should automatically reclaim the memory on process exit).
>>
>> You may try to modify it to something more explicit like the below -
>>
>>> #include <stdlib.h>
>>> void dummy() {
>>> char *c = (char*) malloc(20);
>> c = (char*) malloc(100);
>>> }
>>> int main() {
>>> dummy();
>>> while(1) {}
>>> }
>>
>>
>>
>> On Nov 10, 2016, at 1:43 PM, Vamsi Ambati <vamsi@neumob.com <ma...@neumob.com>> wrote:
>>
>>>
>>> I am trying to resolve a memory leak issue with ATS 7.0 version.
>>> We are using ATS from the CDN perspective and developed 3 plugins.
>>> DevOps complains that traffic_server process gradually consumes lot of memory and does release back which is a sign of memory leak.
>>> I tried with Valgrind but it didn’t work because of LUA and one of our ATS community member suggested to use Address Sanitizer.
>>> I followed exactly the steps mentioned Debugging Traffic Server using ASAN <https://cwiki.apache.org/confluence/download/attachments/56066455/summit_asan.pptx?version=1&modificationDate=1429916307000&api=v2>.
>>>
>>> I have build the traffic server with CXXFLAGS=-fno-omit-frame-pointer -fsanitize=address
>>> Verified that traffic_server is build with ASAN library by 'ldd bin/traffic_server’
>>> Restarted the traffic server with 'service nm-trafficserver restart'.
>>>
>>> I have adopted three approaches to catch the memory leak
>>>
>>> Approach 1:
>>> Just running the traffic server(with -f option) and continuously sending HTTP requests using wrk (from another machine) by attaching
>>> to a gdb .
>>> ' gdb $(pidof traffic_server)’
>>> break __asan_report_error.
>>>
>>> Issue: I didn’t hit this function from the memory leak perspective.
>>>
>>> Approach 2:
>>> I have run the traffic server with PROXY_AUTO_EXIT and sending the http requests
>>> PROXY_AUTO_EXIT=30 ASAN_OPTIONS=detect_leaks=1:verbosity=2 bin/traffic_server -f
>>>
>>> Issue: Tool didn’t complain any thing about the memory leak.
>>>
>>>
>>> Approach 3:
>>> I have created simple code explicitly introducing the memory leak
>>>
>>> #include <stdlib.h>
>>> void dummy() {
>>> malloc(20);
>>> }
>>> int main() {
>>> dummy();
>>> return 0;
>>> }
>>>
>>> Compiled with g++ -fsanitize=address add.cpp
>>> Executed like ASAN_OPTIONS=detect_leaks=1 ./a.out
>>>
>>> Issue: Tool didn’t detect this memory leak.
>>>
>>>
>>> Production System Configuration:
>>>
>>> 16Gb Ram with 8 cores VM.
>>> gcc --version
>>> gcc (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4
>>>
>>> Did any one encounter this ASAN issue on Ubuntu 14.04 ? If so, how did they resolve it ?
>>> Any other suggestions how to narrow down the memory leak issue with ATS ?
>>>
>>>
>>>
>>> Regards
>>> Vamsi
>>>
>>> PS: I am new to the community and new to Apache traffic server open source module too.
>>>
>>>
>>>
>>>
>
Re: Address Sanitizer issue in detecting memory leak.
Posted by Bryan Call <bc...@apache.org>.
Don’t know why Apple Mail didn’t send this. I noticed after restarting:
By default you should see some leaks from CRYPTO_malloc even if you don’t have TLS/SSL configured. I am running Fedora 24.
[bcall@homer trafficserver]$ sudo /usr/local/bin/traffic_server
traffic_server: using root directory '/usr/local'
^Ctraffic_server: Interrupt (Signal sent by the kernel 0 0)
=================================================================
==9081==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 48 byte(s) in 1 object(s) allocated from:
#0 0x7f2a242f7e60 in malloc (/lib64/libasan.so.3+0xc6e60)
#1 0x7f2a231320b7 in CRYPTO_malloc (/lib64/libcrypto.so.10+0x6e0b7)
Indirect leak of 120 byte(s) in 1 object(s) allocated from:
#0 0x7f2a242f7e60 in malloc (/lib64/libasan.so.3+0xc6e60)
#1 0x7f2a231320b7 in CRYPTO_malloc (/lib64/libcrypto.so.10+0x6e0b7)
SUMMARY: AddressSanitizer: 168 byte(s) leaked in 2 allocation(s).
-Bryan
> On Nov 10, 2016, at 2:22 PM, Vamsi Ambati <va...@neumob.com> wrote:
>
> Hi Sudheer,
>
> Thanks for the quick response.
> I tried with your suggested code snippet but Address sanitizer doesn’t detect the leak.
> I even tried with gdb by introducing break point at __asan_report_error.
>
>
> Vamsi
>
>> On Nov 10, 2016, at 1:58 PM, Sudheer Vinukonda <sudheervinukonda@yahoo.com <ma...@yahoo.com>> wrote:
>>
>> Not sure if the *leak* in your test code below is detectable..since your program is exiting after allocating the memory (system should automatically reclaim the memory on process exit).
>>
>> You may try to modify it to something more explicit like the below -
>>
>>> #include <stdlib.h>
>>> void dummy() {
>>> char *c = (char*) malloc(20);
>> c = (char*) malloc(100);
>>> }
>>> int main() {
>>> dummy();
>>> while(1) {}
>>> }
>>
>>
>>
>> On Nov 10, 2016, at 1:43 PM, Vamsi Ambati <vamsi@neumob.com <ma...@neumob.com>> wrote:
>>
>>>
>>> I am trying to resolve a memory leak issue with ATS 7.0 version.
>>> We are using ATS from the CDN perspective and developed 3 plugins.
>>> DevOps complains that traffic_server process gradually consumes lot of memory and does release back which is a sign of memory leak.
>>> I tried with Valgrind but it didn’t work because of LUA and one of our ATS community member suggested to use Address Sanitizer.
>>> I followed exactly the steps mentioned Debugging Traffic Server using ASAN <https://cwiki.apache.org/confluence/download/attachments/56066455/summit_asan.pptx?version=1&modificationDate=1429916307000&api=v2>.
>>>
>>> I have build the traffic server with CXXFLAGS=-fno-omit-frame-pointer -fsanitize=address
>>> Verified that traffic_server is build with ASAN library by 'ldd bin/traffic_server’
>>> Restarted the traffic server with 'service nm-trafficserver restart'.
>>>
>>> I have adopted three approaches to catch the memory leak
>>>
>>> Approach 1:
>>> Just running the traffic server(with -f option) and continuously sending HTTP requests using wrk (from another machine) by attaching
>>> to a gdb .
>>> ' gdb $(pidof traffic_server)’
>>> break __asan_report_error.
>>>
>>> Issue: I didn’t hit this function from the memory leak perspective.
>>>
>>> Approach 2:
>>> I have run the traffic server with PROXY_AUTO_EXIT and sending the http requests
>>> PROXY_AUTO_EXIT=30 ASAN_OPTIONS=detect_leaks=1:verbosity=2 bin/traffic_server -f
>>>
>>> Issue: Tool didn’t complain any thing about the memory leak.
>>>
>>>
>>> Approach 3:
>>> I have created simple code explicitly introducing the memory leak
>>>
>>> #include <stdlib.h>
>>> void dummy() {
>>> malloc(20);
>>> }
>>> int main() {
>>> dummy();
>>> return 0;
>>> }
>>>
>>> Compiled with g++ -fsanitize=address add.cpp
>>> Executed like ASAN_OPTIONS=detect_leaks=1 ./a.out
>>>
>>> Issue: Tool didn’t detect this memory leak.
>>>
>>>
>>> Production System Configuration:
>>>
>>> 16Gb Ram with 8 cores VM.
>>> gcc --version
>>> gcc (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4
>>>
>>> Did any one encounter this ASAN issue on Ubuntu 14.04 ? If so, how did they resolve it ?
>>> Any other suggestions how to narrow down the memory leak issue with ATS ?
>>>
>>>
>>>
>>> Regards
>>> Vamsi
>>>
>>> PS: I am new to the community and new to Apache traffic server open source module too.
>>>
>>>
>>>
>>>
>
Re: Address Sanitizer issue in detecting memory leak.
Posted by Vamsi Ambati <va...@neumob.com>.
Hi Sudheer,
Thanks for the quick response.
I tried with your suggested code snippet but Address sanitizer doesn’t detect the leak.
I even tried with gdb by introducing break point at __asan_report_error.
Vamsi
> On Nov 10, 2016, at 1:58 PM, Sudheer Vinukonda <su...@yahoo.com> wrote:
>
> Not sure if the *leak* in your test code below is detectable..since your program is exiting after allocating the memory (system should automatically reclaim the memory on process exit).
>
> You may try to modify it to something more explicit like the below -
>
>> #include <stdlib.h>
>> void dummy() {
>> char *c = (char*) malloc(20);
> c = (char*) malloc(100);
>> }
>> int main() {
>> dummy();
>> while(1) {}
>> }
>
>
>
> On Nov 10, 2016, at 1:43 PM, Vamsi Ambati <vamsi@neumob.com <ma...@neumob.com>> wrote:
>
>>
>> I am trying to resolve a memory leak issue with ATS 7.0 version.
>> We are using ATS from the CDN perspective and developed 3 plugins.
>> DevOps complains that traffic_server process gradually consumes lot of memory and does release back which is a sign of memory leak.
>> I tried with Valgrind but it didn’t work because of LUA and one of our ATS community member suggested to use Address Sanitizer.
>> I followed exactly the steps mentioned Debugging Traffic Server using ASAN <https://cwiki.apache.org/confluence/download/attachments/56066455/summit_asan.pptx?version=1&modificationDate=1429916307000&api=v2>.
>>
>> I have build the traffic server with CXXFLAGS=-fno-omit-frame-pointer -fsanitize=address
>> Verified that traffic_server is build with ASAN library by 'ldd bin/traffic_server’
>> Restarted the traffic server with 'service nm-trafficserver restart'.
>>
>> I have adopted three approaches to catch the memory leak
>>
>> Approach 1:
>> Just running the traffic server(with -f option) and continuously sending HTTP requests using wrk (from another machine) by attaching
>> to a gdb .
>> ' gdb $(pidof traffic_server)’
>> break __asan_report_error.
>>
>> Issue: I didn’t hit this function from the memory leak perspective.
>>
>> Approach 2:
>> I have run the traffic server with PROXY_AUTO_EXIT and sending the http requests
>> PROXY_AUTO_EXIT=30 ASAN_OPTIONS=detect_leaks=1:verbosity=2 bin/traffic_server -f
>>
>> Issue: Tool didn’t complain any thing about the memory leak.
>>
>>
>> Approach 3:
>> I have created simple code explicitly introducing the memory leak
>>
>> #include <stdlib.h>
>> void dummy() {
>> malloc(20);
>> }
>> int main() {
>> dummy();
>> return 0;
>> }
>>
>> Compiled with g++ -fsanitize=address add.cpp
>> Executed like ASAN_OPTIONS=detect_leaks=1 ./a.out
>>
>> Issue: Tool didn’t detect this memory leak.
>>
>>
>> Production System Configuration:
>>
>> 16Gb Ram with 8 cores VM.
>> gcc --version
>> gcc (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4
>>
>> Did any one encounter this ASAN issue on Ubuntu 14.04 ? If so, how did they resolve it ?
>> Any other suggestions how to narrow down the memory leak issue with ATS ?
>>
>>
>>
>> Regards
>> Vamsi
>>
>> PS: I am new to the community and new to Apache traffic server open source module too.
>>
>>
>>
>>
Re: Address Sanitizer issue in detecting memory leak.
Posted by Sudheer Vinukonda <su...@yahoo.com>.
Not sure if the *leak* in your test code below is detectable..since your program is exiting after allocating the memory (system should automatically reclaim the memory on process exit).
You may try to modify it to something more explicit like the below -
> #include <stdlib.h>
> void dummy() {
> char *c = (char*) malloc(20);
c = (char*) malloc(100);
> }
> int main() {
> dummy();
> while(1) {}
> }
> On Nov 10, 2016, at 1:43 PM, Vamsi Ambati <va...@neumob.com> wrote:
>
>
> I am trying to resolve a memory leak issue with ATS 7.0 version.
> We are using ATS from the CDN perspective and developed 3 plugins.
> DevOps complains that traffic_server process gradually consumes lot of memory and does release back which is a sign of memory leak.
> I tried with Valgrind but it didn’t work because of LUA and one of our ATS community member suggested to use Address Sanitizer.
> I followed exactly the steps mentioned Debugging Traffic Server using ASAN.
>
> I have build the traffic server with CXXFLAGS=-fno-omit-frame-pointer -fsanitize=address
> Verified that traffic_server is build with ASAN library by 'ldd bin/traffic_server’
> Restarted the traffic server with 'service nm-trafficserver restart'.
>
> I have adopted three approaches to catch the memory leak
>
> Approach 1:
> Just running the traffic server(with -f option) and continuously sending HTTP requests using wrk (from another machine) by attaching
> to a gdb .
> ' gdb $(pidof traffic_server)’
> break __asan_report_error.
>
> Issue: I didn’t hit this function from the memory leak perspective.
>
> Approach 2:
> I have run the traffic server with PROXY_AUTO_EXIT and sending the http requests
> PROXY_AUTO_EXIT=30 ASAN_OPTIONS=detect_leaks=1:verbosity=2 bin/traffic_server -f
>
> Issue: Tool didn’t complain any thing about the memory leak.
>
>
> Approach 3:
> I have created simple code explicitly introducing the memory leak
>
> #include <stdlib.h>
> void dummy() {
> malloc(20);
> }
> int main() {
> dummy();
> return 0;
> }
>
> Compiled with g++ -fsanitize=address add.cpp
> Executed like ASAN_OPTIONS=detect_leaks=1 ./a.out
>
> Issue: Tool didn’t detect this memory leak.
>
>
> Production System Configuration:
>
> 16Gb Ram with 8 cores VM.
> gcc --version
> gcc (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4
>
> Did any one encounter this ASAN issue on Ubuntu 14.04 ? If so, how did they resolve it ?
> Any other suggestions how to narrow down the memory leak issue with ATS ?
>
>
>
> Regards
> Vamsi
>
> PS: I am new to the community and new to Apache traffic server open source module too.
>
>
>
>
Re: Address Sanitizer issue in detecting memory leak.
Posted by Sudheer Vinukonda <su...@yahoo.com.INVALID>.
Not sure if the *leak* in your test code below is detectable..since your program is exiting after allocating the memory (system should automatically reclaim the memory on process exit).
You may try to modify it to something more explicit like the below -
> #include <stdlib.h>
> void dummy() {
> char *c = (char*) malloc(20);
c = (char*) malloc(100);
> }
> int main() {
> dummy();
> while(1) {}
> }
> On Nov 10, 2016, at 1:43 PM, Vamsi Ambati <va...@neumob.com> wrote:
>
>
> I am trying to resolve a memory leak issue with ATS 7.0 version.
> We are using ATS from the CDN perspective and developed 3 plugins.
> DevOps complains that traffic_server process gradually consumes lot of memory and does release back which is a sign of memory leak.
> I tried with Valgrind but it didn’t work because of LUA and one of our ATS community member suggested to use Address Sanitizer.
> I followed exactly the steps mentioned Debugging Traffic Server using ASAN.
>
> I have build the traffic server with CXXFLAGS=-fno-omit-frame-pointer -fsanitize=address
> Verified that traffic_server is build with ASAN library by 'ldd bin/traffic_server’
> Restarted the traffic server with 'service nm-trafficserver restart'.
>
> I have adopted three approaches to catch the memory leak
>
> Approach 1:
> Just running the traffic server(with -f option) and continuously sending HTTP requests using wrk (from another machine) by attaching
> to a gdb .
> ' gdb $(pidof traffic_server)’
> break __asan_report_error.
>
> Issue: I didn’t hit this function from the memory leak perspective.
>
> Approach 2:
> I have run the traffic server with PROXY_AUTO_EXIT and sending the http requests
> PROXY_AUTO_EXIT=30 ASAN_OPTIONS=detect_leaks=1:verbosity=2 bin/traffic_server -f
>
> Issue: Tool didn’t complain any thing about the memory leak.
>
>
> Approach 3:
> I have created simple code explicitly introducing the memory leak
>
> #include <stdlib.h>
> void dummy() {
> malloc(20);
> }
> int main() {
> dummy();
> return 0;
> }
>
> Compiled with g++ -fsanitize=address add.cpp
> Executed like ASAN_OPTIONS=detect_leaks=1 ./a.out
>
> Issue: Tool didn’t detect this memory leak.
>
>
> Production System Configuration:
>
> 16Gb Ram with 8 cores VM.
> gcc --version
> gcc (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4
>
> Did any one encounter this ASAN issue on Ubuntu 14.04 ? If so, how did they resolve it ?
> Any other suggestions how to narrow down the memory leak issue with ATS ?
>
>
>
> Regards
> Vamsi
>
> PS: I am new to the community and new to Apache traffic server open source module too.
>
>
>
>