You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@arrow.apache.org by James Van Alstine <jr...@gmail.com> on 2021/08/13 21:51:47 UTC
Constructor leaking memory? Allocators perhaps?
Greetings,
When running a simple C++ hello world program, simply linking against
Arrow produces a valgrind report that looks like the one below. Obviously,
there's some constructors allocating memory before main(). Is this
intentional like the allocators in STL? If it is, is there some way to free
whatever is being allocated? If not, then how do people typically profile
their program's allocations when using Arrow?
==1768== Memcheck, a memory error detector
==1768== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==1768== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==1768== Command: ./hello
==1768==
Hello World!
==1768==
==1768== HEAP SUMMARY:
==1768== in use at exit: 7,094 bytes in 100 blocks
==1768== total heap usage: 3,550 allocs, 3,450 frees, 347,875 bytes
allocated
==1768==
==1768== LEAK SUMMARY:
==1768== definitely lost: 0 bytes in 0 blocks
==1768== indirectly lost: 0 bytes in 0 blocks
==1768== possibly lost: 368 bytes in 1 blocks
==1768== still reachable: 6,726 bytes in 99 blocks
==1768== suppressed: 0 bytes in 0 blocks
==1768== Rerun with --leak-check=full to see details of leaked memory
==1768==
==1768== For lists of detected and suppressed errors, rerun with: -s
==1768== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
Re: Constructor leaking memory? Allocators perhaps?
Posted by James Van Alstine <jr...@gmail.com>.
Yes. I’m comfortable using a valgrind suppression file to get rid of it now
that I know what it is.
Cheers,
James
On Sat, Aug 14, 2021 at 1:19 PM Antoine Pitrou <an...@python.org> wrote:
>
> Ok, so this is simple jemalloc creating its background thread by
> default. Not a problem per se.
>
> Regards
>
> Antoine.
>
>
> On Fri, 13 Aug 2021 20:25:22 -0700
> James Van Alstine <jr...@gmail.com> wrote:
>
> > I ended up just rebuilding arrow with debug symbols.
> >
> > It is indeed a duplicate of
> > https://issues.apache.org/jira/browse/ARROW-9530
> >
> > ==13317== Memcheck, a memory error detector
> > ==13317== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
> > ==13317== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright
> info
> > ==13317== Command: ./app/example/example
> > ==13317==
> > Hello World!
> > ==13317==
> > ==13317== HEAP SUMMARY:
> > ==13317== in use at exit: 304 bytes in 1 blocks
> > ==13317== total heap usage: 13 allocs, 12 frees, 75,181 bytes allocated
> > ==13317==
> > ==13317== 304 bytes in 1 blocks are possibly lost in loss record 1 of 1
> > ==13317== at 0x483DD99: calloc (in
> > /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
> > ==13317== by 0x40149CA: allocate_dtv (dl-tls.c:286)
> > ==13317== by 0x40149CA: _dl_allocate_tls (dl-tls.c:532)
> > ==13317== by 0x5A70322: allocate_stack (allocatestack.c:622)
> > ==13317== by 0x5A70322: pthread_create@@GLIBC_2.2.5
> > (pthread_create.c:660)
> > ==13317== by 0x554B443: je_arrow_private_je_pthread_create_wrapper
> > (background_thread.c:48)
> > ==13317== by 0x554B443: background_thread_create_signals_masked
> > (background_thread.c:365)
> > ==13317== by 0x554B443: background_thread_create_locked
> > (background_thread.c:573)
> > ==13317== by 0x554B63B: je_arrow_private_je_background_thread_create
> > (background_thread.c:598)
> > ==13317== by 0x4011B89: call_init.part.0 (dl-init.c:72)
> > ==13317== by 0x4011C90: call_init (dl-init.c:30)
> > ==13317== by 0x4011C90: _dl_init (dl-init.c:119)
> > ==13317== by 0x4001139: ??? (in /usr/lib/x86_64-linux-gnu/ld-2.31.so)
> > ==13317==
> > ==13317== LEAK SUMMARY:
> > ==13317== definitely lost: 0 bytes in 0 blocks
> > ==13317== indirectly lost: 0 bytes in 0 blocks
> > ==13317== possibly lost: 304 bytes in 1 blocks
> > ==13317== still reachable: 0 bytes in 0 blocks
> > ==13317== suppressed: 0 bytes in 0 blocks
> > ==13317==
> > ==13317== For lists of detected and suppressed errors, rerun with: -s
> > ==13317== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
> >
> > On Fri, Aug 13, 2021 at 6:32 PM James Van Alstine <
> jrvanalstine@gmail.com>
> > wrote:
> >
> > > Is there a libarrow deb with debug symbols so I can resolve the (???)s
> > > valgrind returns?
> > >
> > > On Fri, Aug 13, 2021 at 5:34 PM James Van Alstine <
> jrvanalstine@gmail.com>
> > > wrote:
> > >
> > >> Yes,
> > >>
> > >> For you can repro:
> > >>
> > >> Program:
> > >> ```
> > >> #include <iostream>
> > >>
> > >> int main() {
> > >> std::cout << "Hello World!" << std::endl;
> > >>
> > >> return 0;
> > >> }
> > >> ```
> > >> Executable:
> > >> ```
> > >> clang++-10 -g -Wall -O2 -l:libarrow.so main.cc -o hello
> > >> ```
> > >>
> > >> with --leak-check=full
> > >> ```
> > >> ind --leak-check=full ./hello
> > >> ==6456== Memcheck, a memory error detector
> > >> ==6456== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et
> al.
> > >> ==6456== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright
> > >> info
> > >> ==6456== Command: ./hello
> > >> ==6456==
> > >> Hello World!
> > >> ==6456==
> > >> ==6456== HEAP SUMMARY:
> > >> ==6456== in use at exit: 7,094 bytes in 100 blocks
> > >> ==6456== total heap usage: 3,550 allocs, 3,450 frees, 347,875 bytes
> > >> allocated
> > >> ==6456==
> > >> ==6456== 368 bytes in 1 blocks are possibly lost in loss record 12 of
> 15
> > >> ==6456== at 0x483DD99: calloc (in
> > >> /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
> > >> ==6456== by 0x40149CA: allocate_dtv (dl-tls.c:286)
> > >> ==6456== by 0x40149CA: _dl_allocate_tls (dl-tls.c:532)
> > >> ==6456== by 0x6A1D322: allocate_stack (allocatestack.c:622)
> > >> ==6456== by 0x6A1D322: pthread_create@@GLIBC_2.2.5
> > >> (pthread_create.c:660)
> > >> ==6456== by 0x58E5AB5: ??? (in
> > >> /usr/lib/x86_64-linux-gnu/libarrow.so.500.0.0)
> > >> ==6456== by 0x58E6B62: ??? (in
> > >> /usr/lib/x86_64-linux-gnu/libarrow.so.500.0.0)
> > >> ==6456== by 0x58E6CEB: ??? (in
> > >> /usr/lib/x86_64-linux-gnu/libarrow.so.500.0.0)
> > >> ==6456== by 0x4011B89: call_init.part.0 (dl-init.c:72)
> > >> ==6456== by 0x4011C90: call_init (dl-init.c:30)
> > >> ==6456== by 0x4011C90: _dl_init (dl-init.c:119)
> > >> ==6456== by 0x4001139: ??? (in /usr/lib/x86_64-linux-gnu/
> ld-2.31.so)
> > >> ==6456==
> > >> ==6456== LEAK SUMMARY:
> > >> ==6456== definitely lost: 0 bytes in 0 blocks
> > >> ==6456== indirectly lost: 0 bytes in 0 blocks
> > >> ==6456== possibly lost: 368 bytes in 1 blocks
> > >> ==6456== still reachable: 6,726 bytes in 99 blocks
> > >> ==6456== suppressed: 0 bytes in 0 blocks
> > >> ==6456== Reachable blocks (those to which a pointer was found) are not
> > >> shown.
> > >> ==6456== To see them, rerun with: --leak-check=full
> --show-leak-kinds=all
> > >> ==6456==
> > >> ==6456== For lists of detected and suppressed errors, rerun with: -s
> > >> ==6456== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from
> 0)
> > >> ```
> > >>
> > >> If by `ARROW_DEFAULT_MEMORY_POOL=system` you mean setting that as an
> env
> > >> var and then running the same executable, then
> > >> the valgrind output is the same.
> > >>
> > >> I think what is happening here is glibc is running _dl_init() when
> > >> linking, which is kicking off the constructor of some global object in
> > >> Arrow.
> > >>
> > >>
> > >> On Fri, Aug 13, 2021 at 5:20 PM Weston Pace <we...@gmail.com>
> > >> wrote:
> > >>
> > >>> Can you try running your program with
> ARROW_DEFAULT_MEMORY_POOL=system
> > >>>
> > >>> I am wondering if you are encountering
> > >>> https://issues.apache.org/jira/browse/ARROW-9530
> > >>>
> > >>> Otherwise, can you run with `--leak-check=full` which should give us
> a
> > >>> hint where the possibly lost memory is coming from?
> > >>>
> > >>> On Fri, Aug 13, 2021 at 11:52 AM James Van Alstine
> > >>> <jr...@gmail.com> wrote:
> > >>> >
> > >>> > Greetings,
> > >>> >
> > >>> > When running a simple C++ hello world program, simply linking
> against
> > >>> Arrow produces a valgrind report that looks like the one below.
> Obviously,
> > >>> there's some constructors allocating memory before main(). Is this
> > >>> intentional like the allocators in STL? If it is, is there some way
> to free
> > >>> whatever is being allocated? If not, then how do people typically
> profile
> > >>> their program's allocations when using Arrow?
> > >>> >
> > >>> > ==1768== Memcheck, a memory error detector
> > >>> > ==1768== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward
> et
> > >>> al.
> > >>> > ==1768== Using Valgrind-3.15.0 and LibVEX; rerun with -h for
> copyright
> > >>> info
> > >>> > ==1768== Command: ./hello
> > >>> > ==1768==
> > >>> > Hello World!
> > >>> > ==1768==
> > >>> > ==1768== HEAP SUMMARY:
> > >>> > ==1768== in use at exit: 7,094 bytes in 100 blocks
> > >>> > ==1768== total heap usage: 3,550 allocs, 3,450 frees, 347,875
> bytes
> > >>> allocated
> > >>> > ==1768==
> > >>> > ==1768== LEAK SUMMARY:
> > >>> > ==1768== definitely lost: 0 bytes in 0 blocks
> > >>> > ==1768== indirectly lost: 0 bytes in 0 blocks
> > >>> > ==1768== possibly lost: 368 bytes in 1 blocks
> > >>> > ==1768== still reachable: 6,726 bytes in 99 blocks
> > >>> > ==1768== suppressed: 0 bytes in 0 blocks
> > >>> > ==1768== Rerun with --leak-check=full to see details of leaked
> memory
> > >>> > ==1768==
> > >>> > ==1768== For lists of detected and suppressed errors, rerun with:
> -s
> > >>> > ==1768== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0
> from 0)
> > >>>
> > >>
> >
>
>
>
>
Re: Constructor leaking memory? Allocators perhaps?
Posted by Antoine Pitrou <an...@python.org>.
Ok, so this is simple jemalloc creating its background thread by
default. Not a problem per se.
Regards
Antoine.
On Fri, 13 Aug 2021 20:25:22 -0700
James Van Alstine <jr...@gmail.com> wrote:
> I ended up just rebuilding arrow with debug symbols.
>
> It is indeed a duplicate of
> https://issues.apache.org/jira/browse/ARROW-9530
>
> ==13317== Memcheck, a memory error detector
> ==13317== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
> ==13317== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
> ==13317== Command: ./app/example/example
> ==13317==
> Hello World!
> ==13317==
> ==13317== HEAP SUMMARY:
> ==13317== in use at exit: 304 bytes in 1 blocks
> ==13317== total heap usage: 13 allocs, 12 frees, 75,181 bytes allocated
> ==13317==
> ==13317== 304 bytes in 1 blocks are possibly lost in loss record 1 of 1
> ==13317== at 0x483DD99: calloc (in
> /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
> ==13317== by 0x40149CA: allocate_dtv (dl-tls.c:286)
> ==13317== by 0x40149CA: _dl_allocate_tls (dl-tls.c:532)
> ==13317== by 0x5A70322: allocate_stack (allocatestack.c:622)
> ==13317== by 0x5A70322: pthread_create@@GLIBC_2.2.5
> (pthread_create.c:660)
> ==13317== by 0x554B443: je_arrow_private_je_pthread_create_wrapper
> (background_thread.c:48)
> ==13317== by 0x554B443: background_thread_create_signals_masked
> (background_thread.c:365)
> ==13317== by 0x554B443: background_thread_create_locked
> (background_thread.c:573)
> ==13317== by 0x554B63B: je_arrow_private_je_background_thread_create
> (background_thread.c:598)
> ==13317== by 0x4011B89: call_init.part.0 (dl-init.c:72)
> ==13317== by 0x4011C90: call_init (dl-init.c:30)
> ==13317== by 0x4011C90: _dl_init (dl-init.c:119)
> ==13317== by 0x4001139: ??? (in /usr/lib/x86_64-linux-gnu/ld-2.31.so)
> ==13317==
> ==13317== LEAK SUMMARY:
> ==13317== definitely lost: 0 bytes in 0 blocks
> ==13317== indirectly lost: 0 bytes in 0 blocks
> ==13317== possibly lost: 304 bytes in 1 blocks
> ==13317== still reachable: 0 bytes in 0 blocks
> ==13317== suppressed: 0 bytes in 0 blocks
> ==13317==
> ==13317== For lists of detected and suppressed errors, rerun with: -s
> ==13317== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
>
> On Fri, Aug 13, 2021 at 6:32 PM James Van Alstine <jr...@gmail.com>
> wrote:
>
> > Is there a libarrow deb with debug symbols so I can resolve the (???)s
> > valgrind returns?
> >
> > On Fri, Aug 13, 2021 at 5:34 PM James Van Alstine <jr...@gmail.com>
> > wrote:
> >
> >> Yes,
> >>
> >> For you can repro:
> >>
> >> Program:
> >> ```
> >> #include <iostream>
> >>
> >> int main() {
> >> std::cout << "Hello World!" << std::endl;
> >>
> >> return 0;
> >> }
> >> ```
> >> Executable:
> >> ```
> >> clang++-10 -g -Wall -O2 -l:libarrow.so main.cc -o hello
> >> ```
> >>
> >> with --leak-check=full
> >> ```
> >> ind --leak-check=full ./hello
> >> ==6456== Memcheck, a memory error detector
> >> ==6456== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
> >> ==6456== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright
> >> info
> >> ==6456== Command: ./hello
> >> ==6456==
> >> Hello World!
> >> ==6456==
> >> ==6456== HEAP SUMMARY:
> >> ==6456== in use at exit: 7,094 bytes in 100 blocks
> >> ==6456== total heap usage: 3,550 allocs, 3,450 frees, 347,875 bytes
> >> allocated
> >> ==6456==
> >> ==6456== 368 bytes in 1 blocks are possibly lost in loss record 12 of 15
> >> ==6456== at 0x483DD99: calloc (in
> >> /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
> >> ==6456== by 0x40149CA: allocate_dtv (dl-tls.c:286)
> >> ==6456== by 0x40149CA: _dl_allocate_tls (dl-tls.c:532)
> >> ==6456== by 0x6A1D322: allocate_stack (allocatestack.c:622)
> >> ==6456== by 0x6A1D322: pthread_create@@GLIBC_2.2.5
> >> (pthread_create.c:660)
> >> ==6456== by 0x58E5AB5: ??? (in
> >> /usr/lib/x86_64-linux-gnu/libarrow.so.500.0.0)
> >> ==6456== by 0x58E6B62: ??? (in
> >> /usr/lib/x86_64-linux-gnu/libarrow.so.500.0.0)
> >> ==6456== by 0x58E6CEB: ??? (in
> >> /usr/lib/x86_64-linux-gnu/libarrow.so.500.0.0)
> >> ==6456== by 0x4011B89: call_init.part.0 (dl-init.c:72)
> >> ==6456== by 0x4011C90: call_init (dl-init.c:30)
> >> ==6456== by 0x4011C90: _dl_init (dl-init.c:119)
> >> ==6456== by 0x4001139: ??? (in /usr/lib/x86_64-linux-gnu/ld-2.31.so)
> >> ==6456==
> >> ==6456== LEAK SUMMARY:
> >> ==6456== definitely lost: 0 bytes in 0 blocks
> >> ==6456== indirectly lost: 0 bytes in 0 blocks
> >> ==6456== possibly lost: 368 bytes in 1 blocks
> >> ==6456== still reachable: 6,726 bytes in 99 blocks
> >> ==6456== suppressed: 0 bytes in 0 blocks
> >> ==6456== Reachable blocks (those to which a pointer was found) are not
> >> shown.
> >> ==6456== To see them, rerun with: --leak-check=full --show-leak-kinds=all
> >> ==6456==
> >> ==6456== For lists of detected and suppressed errors, rerun with: -s
> >> ==6456== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
> >> ```
> >>
> >> If by `ARROW_DEFAULT_MEMORY_POOL=system` you mean setting that as an env
> >> var and then running the same executable, then
> >> the valgrind output is the same.
> >>
> >> I think what is happening here is glibc is running _dl_init() when
> >> linking, which is kicking off the constructor of some global object in
> >> Arrow.
> >>
> >>
> >> On Fri, Aug 13, 2021 at 5:20 PM Weston Pace <we...@gmail.com>
> >> wrote:
> >>
> >>> Can you try running your program with ARROW_DEFAULT_MEMORY_POOL=system
> >>>
> >>> I am wondering if you are encountering
> >>> https://issues.apache.org/jira/browse/ARROW-9530
> >>>
> >>> Otherwise, can you run with `--leak-check=full` which should give us a
> >>> hint where the possibly lost memory is coming from?
> >>>
> >>> On Fri, Aug 13, 2021 at 11:52 AM James Van Alstine
> >>> <jr...@gmail.com> wrote:
> >>> >
> >>> > Greetings,
> >>> >
> >>> > When running a simple C++ hello world program, simply linking against
> >>> Arrow produces a valgrind report that looks like the one below. Obviously,
> >>> there's some constructors allocating memory before main(). Is this
> >>> intentional like the allocators in STL? If it is, is there some way to free
> >>> whatever is being allocated? If not, then how do people typically profile
> >>> their program's allocations when using Arrow?
> >>> >
> >>> > ==1768== Memcheck, a memory error detector
> >>> > ==1768== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et
> >>> al.
> >>> > ==1768== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright
> >>> info
> >>> > ==1768== Command: ./hello
> >>> > ==1768==
> >>> > Hello World!
> >>> > ==1768==
> >>> > ==1768== HEAP SUMMARY:
> >>> > ==1768== in use at exit: 7,094 bytes in 100 blocks
> >>> > ==1768== total heap usage: 3,550 allocs, 3,450 frees, 347,875 bytes
> >>> allocated
> >>> > ==1768==
> >>> > ==1768== LEAK SUMMARY:
> >>> > ==1768== definitely lost: 0 bytes in 0 blocks
> >>> > ==1768== indirectly lost: 0 bytes in 0 blocks
> >>> > ==1768== possibly lost: 368 bytes in 1 blocks
> >>> > ==1768== still reachable: 6,726 bytes in 99 blocks
> >>> > ==1768== suppressed: 0 bytes in 0 blocks
> >>> > ==1768== Rerun with --leak-check=full to see details of leaked memory
> >>> > ==1768==
> >>> > ==1768== For lists of detected and suppressed errors, rerun with: -s
> >>> > ==1768== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
> >>>
> >>
>
Re: Constructor leaking memory? Allocators perhaps?
Posted by James Van Alstine <jr...@gmail.com>.
I ended up just rebuilding arrow with debug symbols.
It is indeed a duplicate of
https://issues.apache.org/jira/browse/ARROW-9530
==13317== Memcheck, a memory error detector
==13317== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==13317== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==13317== Command: ./app/example/example
==13317==
Hello World!
==13317==
==13317== HEAP SUMMARY:
==13317== in use at exit: 304 bytes in 1 blocks
==13317== total heap usage: 13 allocs, 12 frees, 75,181 bytes allocated
==13317==
==13317== 304 bytes in 1 blocks are possibly lost in loss record 1 of 1
==13317== at 0x483DD99: calloc (in
/usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==13317== by 0x40149CA: allocate_dtv (dl-tls.c:286)
==13317== by 0x40149CA: _dl_allocate_tls (dl-tls.c:532)
==13317== by 0x5A70322: allocate_stack (allocatestack.c:622)
==13317== by 0x5A70322: pthread_create@@GLIBC_2.2.5
(pthread_create.c:660)
==13317== by 0x554B443: je_arrow_private_je_pthread_create_wrapper
(background_thread.c:48)
==13317== by 0x554B443: background_thread_create_signals_masked
(background_thread.c:365)
==13317== by 0x554B443: background_thread_create_locked
(background_thread.c:573)
==13317== by 0x554B63B: je_arrow_private_je_background_thread_create
(background_thread.c:598)
==13317== by 0x4011B89: call_init.part.0 (dl-init.c:72)
==13317== by 0x4011C90: call_init (dl-init.c:30)
==13317== by 0x4011C90: _dl_init (dl-init.c:119)
==13317== by 0x4001139: ??? (in /usr/lib/x86_64-linux-gnu/ld-2.31.so)
==13317==
==13317== LEAK SUMMARY:
==13317== definitely lost: 0 bytes in 0 blocks
==13317== indirectly lost: 0 bytes in 0 blocks
==13317== possibly lost: 304 bytes in 1 blocks
==13317== still reachable: 0 bytes in 0 blocks
==13317== suppressed: 0 bytes in 0 blocks
==13317==
==13317== For lists of detected and suppressed errors, rerun with: -s
==13317== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
On Fri, Aug 13, 2021 at 6:32 PM James Van Alstine <jr...@gmail.com>
wrote:
> Is there a libarrow deb with debug symbols so I can resolve the (???)s
> valgrind returns?
>
> On Fri, Aug 13, 2021 at 5:34 PM James Van Alstine <jr...@gmail.com>
> wrote:
>
>> Yes,
>>
>> For you can repro:
>>
>> Program:
>> ```
>> #include <iostream>
>>
>> int main() {
>> std::cout << "Hello World!" << std::endl;
>>
>> return 0;
>> }
>> ```
>> Executable:
>> ```
>> clang++-10 -g -Wall -O2 -l:libarrow.so main.cc -o hello
>> ```
>>
>> with --leak-check=full
>> ```
>> ind --leak-check=full ./hello
>> ==6456== Memcheck, a memory error detector
>> ==6456== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
>> ==6456== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright
>> info
>> ==6456== Command: ./hello
>> ==6456==
>> Hello World!
>> ==6456==
>> ==6456== HEAP SUMMARY:
>> ==6456== in use at exit: 7,094 bytes in 100 blocks
>> ==6456== total heap usage: 3,550 allocs, 3,450 frees, 347,875 bytes
>> allocated
>> ==6456==
>> ==6456== 368 bytes in 1 blocks are possibly lost in loss record 12 of 15
>> ==6456== at 0x483DD99: calloc (in
>> /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
>> ==6456== by 0x40149CA: allocate_dtv (dl-tls.c:286)
>> ==6456== by 0x40149CA: _dl_allocate_tls (dl-tls.c:532)
>> ==6456== by 0x6A1D322: allocate_stack (allocatestack.c:622)
>> ==6456== by 0x6A1D322: pthread_create@@GLIBC_2.2.5
>> (pthread_create.c:660)
>> ==6456== by 0x58E5AB5: ??? (in
>> /usr/lib/x86_64-linux-gnu/libarrow.so.500.0.0)
>> ==6456== by 0x58E6B62: ??? (in
>> /usr/lib/x86_64-linux-gnu/libarrow.so.500.0.0)
>> ==6456== by 0x58E6CEB: ??? (in
>> /usr/lib/x86_64-linux-gnu/libarrow.so.500.0.0)
>> ==6456== by 0x4011B89: call_init.part.0 (dl-init.c:72)
>> ==6456== by 0x4011C90: call_init (dl-init.c:30)
>> ==6456== by 0x4011C90: _dl_init (dl-init.c:119)
>> ==6456== by 0x4001139: ??? (in /usr/lib/x86_64-linux-gnu/ld-2.31.so)
>> ==6456==
>> ==6456== LEAK SUMMARY:
>> ==6456== definitely lost: 0 bytes in 0 blocks
>> ==6456== indirectly lost: 0 bytes in 0 blocks
>> ==6456== possibly lost: 368 bytes in 1 blocks
>> ==6456== still reachable: 6,726 bytes in 99 blocks
>> ==6456== suppressed: 0 bytes in 0 blocks
>> ==6456== Reachable blocks (those to which a pointer was found) are not
>> shown.
>> ==6456== To see them, rerun with: --leak-check=full --show-leak-kinds=all
>> ==6456==
>> ==6456== For lists of detected and suppressed errors, rerun with: -s
>> ==6456== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
>> ```
>>
>> If by `ARROW_DEFAULT_MEMORY_POOL=system` you mean setting that as an env
>> var and then running the same executable, then
>> the valgrind output is the same.
>>
>> I think what is happening here is glibc is running _dl_init() when
>> linking, which is kicking off the constructor of some global object in
>> Arrow.
>>
>>
>> On Fri, Aug 13, 2021 at 5:20 PM Weston Pace <we...@gmail.com>
>> wrote:
>>
>>> Can you try running your program with ARROW_DEFAULT_MEMORY_POOL=system
>>>
>>> I am wondering if you are encountering
>>> https://issues.apache.org/jira/browse/ARROW-9530
>>>
>>> Otherwise, can you run with `--leak-check=full` which should give us a
>>> hint where the possibly lost memory is coming from?
>>>
>>> On Fri, Aug 13, 2021 at 11:52 AM James Van Alstine
>>> <jr...@gmail.com> wrote:
>>> >
>>> > Greetings,
>>> >
>>> > When running a simple C++ hello world program, simply linking against
>>> Arrow produces a valgrind report that looks like the one below. Obviously,
>>> there's some constructors allocating memory before main(). Is this
>>> intentional like the allocators in STL? If it is, is there some way to free
>>> whatever is being allocated? If not, then how do people typically profile
>>> their program's allocations when using Arrow?
>>> >
>>> > ==1768== Memcheck, a memory error detector
>>> > ==1768== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et
>>> al.
>>> > ==1768== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright
>>> info
>>> > ==1768== Command: ./hello
>>> > ==1768==
>>> > Hello World!
>>> > ==1768==
>>> > ==1768== HEAP SUMMARY:
>>> > ==1768== in use at exit: 7,094 bytes in 100 blocks
>>> > ==1768== total heap usage: 3,550 allocs, 3,450 frees, 347,875 bytes
>>> allocated
>>> > ==1768==
>>> > ==1768== LEAK SUMMARY:
>>> > ==1768== definitely lost: 0 bytes in 0 blocks
>>> > ==1768== indirectly lost: 0 bytes in 0 blocks
>>> > ==1768== possibly lost: 368 bytes in 1 blocks
>>> > ==1768== still reachable: 6,726 bytes in 99 blocks
>>> > ==1768== suppressed: 0 bytes in 0 blocks
>>> > ==1768== Rerun with --leak-check=full to see details of leaked memory
>>> > ==1768==
>>> > ==1768== For lists of detected and suppressed errors, rerun with: -s
>>> > ==1768== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
>>>
>>
Re: Constructor leaking memory? Allocators perhaps?
Posted by James Van Alstine <jr...@gmail.com>.
Is there a libarrow deb with debug symbols so I can resolve the (???)s
valgrind returns?
On Fri, Aug 13, 2021 at 5:34 PM James Van Alstine <jr...@gmail.com>
wrote:
> Yes,
>
> For you can repro:
>
> Program:
> ```
> #include <iostream>
>
> int main() {
> std::cout << "Hello World!" << std::endl;
>
> return 0;
> }
> ```
> Executable:
> ```
> clang++-10 -g -Wall -O2 -l:libarrow.so main.cc -o hello
> ```
>
> with --leak-check=full
> ```
> ind --leak-check=full ./hello
> ==6456== Memcheck, a memory error detector
> ==6456== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
> ==6456== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
> ==6456== Command: ./hello
> ==6456==
> Hello World!
> ==6456==
> ==6456== HEAP SUMMARY:
> ==6456== in use at exit: 7,094 bytes in 100 blocks
> ==6456== total heap usage: 3,550 allocs, 3,450 frees, 347,875 bytes
> allocated
> ==6456==
> ==6456== 368 bytes in 1 blocks are possibly lost in loss record 12 of 15
> ==6456== at 0x483DD99: calloc (in
> /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
> ==6456== by 0x40149CA: allocate_dtv (dl-tls.c:286)
> ==6456== by 0x40149CA: _dl_allocate_tls (dl-tls.c:532)
> ==6456== by 0x6A1D322: allocate_stack (allocatestack.c:622)
> ==6456== by 0x6A1D322: pthread_create@@GLIBC_2.2.5
> (pthread_create.c:660)
> ==6456== by 0x58E5AB5: ??? (in
> /usr/lib/x86_64-linux-gnu/libarrow.so.500.0.0)
> ==6456== by 0x58E6B62: ??? (in
> /usr/lib/x86_64-linux-gnu/libarrow.so.500.0.0)
> ==6456== by 0x58E6CEB: ??? (in
> /usr/lib/x86_64-linux-gnu/libarrow.so.500.0.0)
> ==6456== by 0x4011B89: call_init.part.0 (dl-init.c:72)
> ==6456== by 0x4011C90: call_init (dl-init.c:30)
> ==6456== by 0x4011C90: _dl_init (dl-init.c:119)
> ==6456== by 0x4001139: ??? (in /usr/lib/x86_64-linux-gnu/ld-2.31.so)
> ==6456==
> ==6456== LEAK SUMMARY:
> ==6456== definitely lost: 0 bytes in 0 blocks
> ==6456== indirectly lost: 0 bytes in 0 blocks
> ==6456== possibly lost: 368 bytes in 1 blocks
> ==6456== still reachable: 6,726 bytes in 99 blocks
> ==6456== suppressed: 0 bytes in 0 blocks
> ==6456== Reachable blocks (those to which a pointer was found) are not
> shown.
> ==6456== To see them, rerun with: --leak-check=full --show-leak-kinds=all
> ==6456==
> ==6456== For lists of detected and suppressed errors, rerun with: -s
> ==6456== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
> ```
>
> If by `ARROW_DEFAULT_MEMORY_POOL=system` you mean setting that as an env
> var and then running the same executable, then
> the valgrind output is the same.
>
> I think what is happening here is glibc is running _dl_init() when
> linking, which is kicking off the constructor of some global object in
> Arrow.
>
>
> On Fri, Aug 13, 2021 at 5:20 PM Weston Pace <we...@gmail.com> wrote:
>
>> Can you try running your program with ARROW_DEFAULT_MEMORY_POOL=system
>>
>> I am wondering if you are encountering
>> https://issues.apache.org/jira/browse/ARROW-9530
>>
>> Otherwise, can you run with `--leak-check=full` which should give us a
>> hint where the possibly lost memory is coming from?
>>
>> On Fri, Aug 13, 2021 at 11:52 AM James Van Alstine
>> <jr...@gmail.com> wrote:
>> >
>> > Greetings,
>> >
>> > When running a simple C++ hello world program, simply linking against
>> Arrow produces a valgrind report that looks like the one below. Obviously,
>> there's some constructors allocating memory before main(). Is this
>> intentional like the allocators in STL? If it is, is there some way to free
>> whatever is being allocated? If not, then how do people typically profile
>> their program's allocations when using Arrow?
>> >
>> > ==1768== Memcheck, a memory error detector
>> > ==1768== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
>> > ==1768== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright
>> info
>> > ==1768== Command: ./hello
>> > ==1768==
>> > Hello World!
>> > ==1768==
>> > ==1768== HEAP SUMMARY:
>> > ==1768== in use at exit: 7,094 bytes in 100 blocks
>> > ==1768== total heap usage: 3,550 allocs, 3,450 frees, 347,875 bytes
>> allocated
>> > ==1768==
>> > ==1768== LEAK SUMMARY:
>> > ==1768== definitely lost: 0 bytes in 0 blocks
>> > ==1768== indirectly lost: 0 bytes in 0 blocks
>> > ==1768== possibly lost: 368 bytes in 1 blocks
>> > ==1768== still reachable: 6,726 bytes in 99 blocks
>> > ==1768== suppressed: 0 bytes in 0 blocks
>> > ==1768== Rerun with --leak-check=full to see details of leaked memory
>> > ==1768==
>> > ==1768== For lists of detected and suppressed errors, rerun with: -s
>> > ==1768== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
>>
>
Re: Constructor leaking memory? Allocators perhaps?
Posted by James Van Alstine <jr...@gmail.com>.
Yes,
For you can repro:
Program:
```
#include <iostream>
int main() {
std::cout << "Hello World!" << std::endl;
return 0;
}
```
Executable:
```
clang++-10 -g -Wall -O2 -l:libarrow.so main.cc -o hello
```
with --leak-check=full
```
ind --leak-check=full ./hello
==6456== Memcheck, a memory error detector
==6456== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==6456== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==6456== Command: ./hello
==6456==
Hello World!
==6456==
==6456== HEAP SUMMARY:
==6456== in use at exit: 7,094 bytes in 100 blocks
==6456== total heap usage: 3,550 allocs, 3,450 frees, 347,875 bytes
allocated
==6456==
==6456== 368 bytes in 1 blocks are possibly lost in loss record 12 of 15
==6456== at 0x483DD99: calloc (in
/usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==6456== by 0x40149CA: allocate_dtv (dl-tls.c:286)
==6456== by 0x40149CA: _dl_allocate_tls (dl-tls.c:532)
==6456== by 0x6A1D322: allocate_stack (allocatestack.c:622)
==6456== by 0x6A1D322: pthread_create@@GLIBC_2.2.5 (pthread_create.c:660)
==6456== by 0x58E5AB5: ??? (in
/usr/lib/x86_64-linux-gnu/libarrow.so.500.0.0)
==6456== by 0x58E6B62: ??? (in
/usr/lib/x86_64-linux-gnu/libarrow.so.500.0.0)
==6456== by 0x58E6CEB: ??? (in
/usr/lib/x86_64-linux-gnu/libarrow.so.500.0.0)
==6456== by 0x4011B89: call_init.part.0 (dl-init.c:72)
==6456== by 0x4011C90: call_init (dl-init.c:30)
==6456== by 0x4011C90: _dl_init (dl-init.c:119)
==6456== by 0x4001139: ??? (in /usr/lib/x86_64-linux-gnu/ld-2.31.so)
==6456==
==6456== LEAK SUMMARY:
==6456== definitely lost: 0 bytes in 0 blocks
==6456== indirectly lost: 0 bytes in 0 blocks
==6456== possibly lost: 368 bytes in 1 blocks
==6456== still reachable: 6,726 bytes in 99 blocks
==6456== suppressed: 0 bytes in 0 blocks
==6456== Reachable blocks (those to which a pointer was found) are not
shown.
==6456== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==6456==
==6456== For lists of detected and suppressed errors, rerun with: -s
==6456== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
```
If by `ARROW_DEFAULT_MEMORY_POOL=system` you mean setting that as an env
var and then running the same executable, then
the valgrind output is the same.
I think what is happening here is glibc is running _dl_init() when linking,
which is kicking off the constructor of some global object in Arrow.
On Fri, Aug 13, 2021 at 5:20 PM Weston Pace <we...@gmail.com> wrote:
> Can you try running your program with ARROW_DEFAULT_MEMORY_POOL=system
>
> I am wondering if you are encountering
> https://issues.apache.org/jira/browse/ARROW-9530
>
> Otherwise, can you run with `--leak-check=full` which should give us a
> hint where the possibly lost memory is coming from?
>
> On Fri, Aug 13, 2021 at 11:52 AM James Van Alstine
> <jr...@gmail.com> wrote:
> >
> > Greetings,
> >
> > When running a simple C++ hello world program, simply linking against
> Arrow produces a valgrind report that looks like the one below. Obviously,
> there's some constructors allocating memory before main(). Is this
> intentional like the allocators in STL? If it is, is there some way to free
> whatever is being allocated? If not, then how do people typically profile
> their program's allocations when using Arrow?
> >
> > ==1768== Memcheck, a memory error detector
> > ==1768== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
> > ==1768== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright
> info
> > ==1768== Command: ./hello
> > ==1768==
> > Hello World!
> > ==1768==
> > ==1768== HEAP SUMMARY:
> > ==1768== in use at exit: 7,094 bytes in 100 blocks
> > ==1768== total heap usage: 3,550 allocs, 3,450 frees, 347,875 bytes
> allocated
> > ==1768==
> > ==1768== LEAK SUMMARY:
> > ==1768== definitely lost: 0 bytes in 0 blocks
> > ==1768== indirectly lost: 0 bytes in 0 blocks
> > ==1768== possibly lost: 368 bytes in 1 blocks
> > ==1768== still reachable: 6,726 bytes in 99 blocks
> > ==1768== suppressed: 0 bytes in 0 blocks
> > ==1768== Rerun with --leak-check=full to see details of leaked memory
> > ==1768==
> > ==1768== For lists of detected and suppressed errors, rerun with: -s
> > ==1768== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
>
Re: Constructor leaking memory? Allocators perhaps?
Posted by Weston Pace <we...@gmail.com>.
Can you try running your program with ARROW_DEFAULT_MEMORY_POOL=system
I am wondering if you are encountering
https://issues.apache.org/jira/browse/ARROW-9530
Otherwise, can you run with `--leak-check=full` which should give us a
hint where the possibly lost memory is coming from?
On Fri, Aug 13, 2021 at 11:52 AM James Van Alstine
<jr...@gmail.com> wrote:
>
> Greetings,
>
> When running a simple C++ hello world program, simply linking against Arrow produces a valgrind report that looks like the one below. Obviously, there's some constructors allocating memory before main(). Is this intentional like the allocators in STL? If it is, is there some way to free whatever is being allocated? If not, then how do people typically profile their program's allocations when using Arrow?
>
> ==1768== Memcheck, a memory error detector
> ==1768== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
> ==1768== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
> ==1768== Command: ./hello
> ==1768==
> Hello World!
> ==1768==
> ==1768== HEAP SUMMARY:
> ==1768== in use at exit: 7,094 bytes in 100 blocks
> ==1768== total heap usage: 3,550 allocs, 3,450 frees, 347,875 bytes allocated
> ==1768==
> ==1768== LEAK SUMMARY:
> ==1768== definitely lost: 0 bytes in 0 blocks
> ==1768== indirectly lost: 0 bytes in 0 blocks
> ==1768== possibly lost: 368 bytes in 1 blocks
> ==1768== still reachable: 6,726 bytes in 99 blocks
> ==1768== suppressed: 0 bytes in 0 blocks
> ==1768== Rerun with --leak-check=full to see details of leaked memory
> ==1768==
> ==1768== For lists of detected and suppressed errors, rerun with: -s
> ==1768== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)