You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "eddelbuettel (via GitHub)" <gi...@apache.org> on 2023/05/21 16:03:50 UTC
[GitHub] [arrow-nanoarrow] eddelbuettel opened a new issue, #200: Possible valgrind issue ?
eddelbuettel opened a new issue, #200:
URL: https://github.com/apache/arrow-nanoarrow/issues/200
Converting from the C API data structure for which `nanoarrow` is so useful seems to tickle something from `valgrind` which may make CRAN unhappy. (Though I am unsure at this moment if they act us to change things on _possibly lost_. In any event, working on a related issue on my side I notice the 'residual' reported here (see below) that can be tickled via a simple variant around one your README example, namely a conversion to `arrow`:
```sh
R -q -d valgrind -e 'arrow::as_arrow_table(nanoarrow::as_nanoarrow_array(data.frame(col1= c(1.1,2.2))))'
```
Full logs below. This on Ubuntu 22.10 with all packages at current CRAN versions. (`nanoarrow` may be from repo, I could check.)
I think this may be a false positive: I have seen similar 'possibly lost' before.
Or could it be related to how arrow is instantiated? Other uses (as e.g. an export into a tibble) do not exhibit this (cf second example below).
<details>
```
$ R -q -d valgrind -e 'as.data.frame(nanoarrow::as_nanoarrow_array(data.frame(col1= c(1.1,2.2))))'
==2812420== Memcheck, a memory error detector
==2812420== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==2812420== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
==2812420== Command: /usr/lib/R/bin/exec/R -q -e as.data.frame(nanoarrow::as_nanoarrow_array(data.frame(col1=~+~c(1.1,2.2))))
==2812420==
> as.data.frame(nanoarrow::as_nanoarrow_array(data.frame(col1= c(1.1,2.2))))
col1
1 1.1
2 2.2
>
>
==2812420==
==2812420== HEAP SUMMARY:
==2812420== in use at exit: 55,176,410 bytes in 12,118 blocks
==2812420== total heap usage: 28,209 allocs, 16,091 frees, 83,130,127 bytes allocated
==2812420==
==2812420== LEAK SUMMARY:
==2812420== definitely lost: 0 bytes in 0 blocks
==2812420== indirectly lost: 0 bytes in 0 blocks
==2812420== possibly lost: 0 bytes in 0 blocks
==2812420== still reachable: 55,176,410 bytes in 12,118 blocks
==2812420== of which reachable via heuristic:
==2812420== newarray : 4,264 bytes in 1 blocks
==2812420== suppressed: 0 bytes in 0 blocks
==2812420== Reachable blocks (those to which a pointer was found) are not shown.
==2812420== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==2812420==
==2812420== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
edd@rob:~/git/rcppnanoarrow(master)$ R -q -d valgrind -e 'arrow::as_arrow_table(nanoarrow::as_nanoarrow_array(data.frame(col1= c(1.1,2.2))))'
==2817635== Memcheck, a memory error detector
==2817635== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==2817635== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
==2817635== Command: /usr/lib/R/bin/exec/R -q -e arrow::as_arrow_table(nanoarrow::as_nanoarrow_array(data.frame(col1=~+~c(1.1,2.2))))
==2817635==
> arrow::as_arrow_table(nanoarrow::as_nanoarrow_array(data.frame(col1= c(1.1,2.2))))
Table
2 rows x 1 columns
$col1 <double>
>
>
==2817635==
==2817635== HEAP SUMMARY:
==2817635== in use at exit: 66,601,761 bytes in 15,000 blocks
==2817635== total heap usage: 90,088 allocs, 75,088 frees, 153,000,130 bytes allocated
==2817635==
==2817635== 416 bytes in 1 blocks are possibly lost in loss record 170 of 1,580
==2817635== at 0x4849A83: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==2817635== by 0x4012949: calloc (rtld-malloc.h:44)
==2817635== by 0x4012949: allocate_dtv (dl-tls.c:375)
==2817635== by 0x4012949: _dl_allocate_tls (dl-tls.c:634)
==2817635== by 0x4D4BF99: allocate_stack (allocatestack.c:423)
==2817635== by 0x4D4BF99: pthread_create@@GLIBC_2.34 (pthread_create.c:650)
==2817635== by 0x14899D93: ??? (in /usr/local/lib/R/site-library/arrow/libs/arrow.so)
==2817635== by 0x1489B01C: ??? (in /usr/local/lib/R/site-library/arrow/libs/arrow.so)
==2817635== by 0x4004FBD: call_init.part.0 (dl-init.c:70)
==2817635== by 0x40050A7: call_init (dl-init.c:33)
==2817635== by 0x40050A7: _dl_init (dl-init.c:117)
==2817635== by 0x4E287F3: _dl_catch_exception (dl-error-skeleton.c:182)
==2817635== by 0x400BE95: dl_open_worker (dl-open.c:808)
==2817635== by 0x400BE95: dl_open_worker (dl-open.c:771)
==2817635== by 0x4E28799: _dl_catch_exception (dl-error-skeleton.c:208)
==2817635== by 0x400C20B: _dl_open (dl-open.c:886)
==2817635== by 0x4D46E6B: dlopen_doit (dlopen.c:56)
==2817635==
==2817635== 2,920 bytes in 1 blocks are possibly lost in loss record 279 of 1,580
==2817635== at 0x4844899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==2817635== by 0x401207D: malloc (rtld-malloc.h:56)
==2817635== by 0x401207D: allocate_dtv_entry (dl-tls.c:684)
==2817635== by 0x401207D: allocate_and_init (dl-tls.c:709)
==2817635== by 0x401207D: tls_get_addr_tail (dl-tls.c:907)
==2817635== by 0x4015D6B: __tls_get_addr (tls_get_addr.S:55)
==2817635== by 0x14899F93: ??? (in /usr/local/lib/R/site-library/arrow/libs/arrow.so)
==2817635== by 0x4D4B401: start_thread (pthread_create.c:442)
==2817635== by 0x4DD9743: clone (clone.S:100)
==2817635==
==2817635==
==2817635== LEAK SUMMARY:
==2817635== definitely lost: 0 bytes in 0 blocks
==2817635== indirectly lost: 0 bytes in 0 blocks
==2817635== possibly lost: 3,336 bytes in 2 blocks
==2817635== still reachable: 66,598,425 bytes in 14,998 blocks
==2817635== of which reachable via heuristic:
==2817635== newarray : 4,264 bytes in 1 blocks
==2817635== suppressed: 0 bytes in 0 blocks
==2817635== Reachable blocks (those to which a pointer was found) are not shown.
==2817635== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==2817635==
==2817635== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 0 from 0)
$
```
</details>
<details>
```sh
$ R -q -d valgrind -e 'tibble::as_tibble(nanoarrow::as_nanoarrow_array(data.frame(col1= c(1.1,2.2))))'
==2851061== Memcheck, a memory error detector
==2851061== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==2851061== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
==2851061== Command: /usr/lib/R/bin/exec/R -q -e tibble::as_tibble(nanoarrow::as_nanoarrow_array(data.frame(col1=~+~c(1.1,2.2))))
==2851061==
> tibble::as_tibble(nanoarrow::as_nanoarrow_array(data.frame(col1= c(1.1,2.2))))
# A tibble: 2 × 1
col1
<dbl>
1 1.1
2 2.2
>
>
==2851061==
==2851061== HEAP SUMMARY:
==2851061== in use at exit: 61,673,573 bytes in 13,974 blocks
==2851061== total heap usage: 60,305 allocs, 46,331 frees, 134,526,038 bytes allocated
==2851061==
==2851061== LEAK SUMMARY:
==2851061== definitely lost: 0 bytes in 0 blocks
==2851061== indirectly lost: 0 bytes in 0 blocks
==2851061== possibly lost: 0 bytes in 0 blocks
==2851061== still reachable: 61,673,573 bytes in 13,974 blocks
==2851061== of which reachable via heuristic:
==2851061== newarray : 4,264 bytes in 1 blocks
==2851061== suppressed: 0 bytes in 0 blocks
==2851061== Reachable blocks (those to which a pointer was found) are not shown.
==2851061== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==2851061==
==2851061== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
$
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] eddelbuettel commented on issue #200: Possible valgrind issue ?
Posted by "eddelbuettel (via GitHub)" <gi...@apache.org>.
eddelbuettel commented on issue #200:
URL: https://github.com/apache/arrow-nanoarrow/issues/200#issuecomment-1556253424
Thanks for the very prompt reply! I had seen the .supp on your repo, but never perused documentation. I am mostly fearful of getting my wrist slapped by CRAN if I use the similar pattern on nanoarrow/arrow conversion. Any thoughts? Have you had any issues with your packages?
(I had a real issue where I wasn't relasing a small subcomponent of a larger data structure so I am remain grateful for what `valrgrind` does but I also had some headscrathers. Again, thanks for the quick follow-up!)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] paleolimbot commented on issue #200: Possible valgrind issue ?
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on issue #200:
URL: https://github.com/apache/arrow-nanoarrow/issues/200#issuecomment-1556425471
I think you are safe to blame the arrow package if CRAN does send you a note, and I doubt they will since the pattern you showed is one that nanoarrow uses in its own tests (and the arrow package, of course). When buffers or arrays actually leak (and there were plenty in the development of the R bindings!) the stack trace very clearly shows something like `ArrowMalloc()` or `MemoryPool::Allocate()`.
I'll keep this open as a note to reinvestigate the thread-local storage jemalloc issue (either add that to the suppressions file or figure out how to make it disappear).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] eddelbuettel commented on issue #200: Possible valgrind issue ?
Posted by "eddelbuettel (via GitHub)" <gi...@apache.org>.
eddelbuettel commented on issue #200:
URL: https://github.com/apache/arrow-nanoarrow/issues/200#issuecomment-1556267374
Hm, with
```
ARROW_DEFAULT_MEMORY_POOL=system R -q -d valgrind \
-e 'arrow::as_arrow_table(nanoarrow::as_nanoarrow_array(data.frame(col1= c(1.1,2.2))))'
```
I get the same result as above. Using it from my checkout of your repo and pointing to `valgrind.supp` one of the two disappears:
```sh
$ R -q -d valgrind --debugger-args="--suppressions=valgrind.supp" -e 'arrow::as_arrow_table(nanoarrow::as_nanoarrow_array(data.frame(col1= c(1.1,2.2))))'
==3525498== Memcheck, a memory error detector
==3525498== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==3525498== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
==3525498== Command: /usr/lib/R/bin/exec/R -q -e arrow::as_arrow_table(nanoarrow::as_nanoarrow_array(data.frame(col1=~+~c(1.1,2.2))))
==3525498==
> arrow::as_arrow_table(nanoarrow::as_nanoarrow_array(data.frame(col1= c(1.1,2.2))))
Table
2 rows x 1 columns
$col1 <double>
>
>
==3525498==
==3525498== HEAP SUMMARY:
==3525498== in use at exit: 66,601,762 bytes in 15,000 blocks
==3525498== total heap usage: 90,088 allocs, 75,088 frees, 153,000,132 bytes allocated
==3525498==
==3525498== 2,920 bytes in 1 blocks are possibly lost in loss record 279 of 1,580
==3525498== at 0x4844899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==3525498== by 0x401207D: malloc (rtld-malloc.h:56)
==3525498== by 0x401207D: allocate_dtv_entry (dl-tls.c:684)
==3525498== by 0x401207D: allocate_and_init (dl-tls.c:709)
==3525498== by 0x401207D: tls_get_addr_tail (dl-tls.c:907)
==3525498== by 0x4015D6B: __tls_get_addr (tls_get_addr.S:55)
==3525498== by 0x14899F93: ??? (in /usr/local/lib/R/site-library/arrow/libs/arrow.so)
==3525498== by 0x4D4B401: start_thread (pthread_create.c:442)
==3525498== by 0x4DD9743: clone (clone.S:100)
==3525498==
==3525498== LEAK SUMMARY:
==3525498== definitely lost: 0 bytes in 0 blocks
==3525498== indirectly lost: 0 bytes in 0 blocks
==3525498== possibly lost: 2,920 bytes in 1 blocks
==3525498== still reachable: 66,598,426 bytes in 14,998 blocks
==3525498== of which reachable via heuristic:
==3525498== newarray : 4,264 bytes in 1 blocks
==3525498== suppressed: 416 bytes in 1 blocks
==3525498== Reachable blocks (those to which a pointer was found) are not shown.
==3525498== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==3525498==
==3525498== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 1 from 1)
--3525498--
--3525498-- used_suppression: 1 <jemalloc>:Thread locals don't appear to be freed valgrind.supp:19 suppressed: 416 bytes in 1 blocks
==3525498==
==3525498== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 1 from 1)
$
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] paleolimbot closed issue #200: Possible valgrind issue ?
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot closed issue #200: Possible valgrind issue ?
URL: https://github.com/apache/arrow-nanoarrow/issues/200
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] paleolimbot commented on issue #200: Possible valgrind issue ?
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on issue #200:
URL: https://github.com/apache/arrow-nanoarrow/issues/200#issuecomment-1556247653
I believe those are both thread-local storage allocated by jemalloc, which is Arrow C++'s default allocator on Linux. I forget all the details but I think you can set `ARROW_DEFAULT_MEMORY_POOL=system` ( https://arrow.apache.org/docs/cpp/env_vars.html#envvar-ARROW_DEFAULT_MEMORY_POOL ) or use https://github.com/apache/arrow-nanoarrow/blob/main/valgrind.supp .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org